s3readerat

README

s3readerat

An implementation of io.ReaderAt that works using S3 GetObject and Range. Inspired by seekinghttp and httpreaderat.

Example

You can try out the included command-line program, seek-s3. It will let you fetch ranges of an S3 object, using the io.ReaderAt interface provided by seekings3.

$ go build ./cmd/seek-s3
$ ./seek-s3 -help
Usage of ./seek-s3:
  -debug
    	enable verbose output
  -limit int
    	limit the bytes to print (-1 is unlimited) (default -1)
  -offset int
    	offset parameter to seek (default -8)
  -whence int
    	whence parameter to seek (0 is start, 1 is current and 2 is end) (default 2)

For example, assuming your S3 object is a Parquet file, you can read the last 4 bytes.

$ ./seek-s3 -offset -4 -whence 2 -limit 4 s3://$BUCKET/$KEY | xxd
00000000: 5041 5231                                PAR1

Single- and multi-region modes

If you call NewWithOptions passing an s3.Client, then the S3ReaderAt will be in "single-region" mode. What that means is that, if the S3 bucket you are trying to access is in another region, you may fail with a 3xx response from S3.

If you call NewWithOptions passing s3.Options, then the S3ReaderAt will be in "multi-region" mode. What that means is that, if the S3 bucket you are trying to access is in another region, the S3ReaderAt will construct an s3.Client for you in the appropriate region, thereby avoiding the 3xx response from S3.

Documentation

Overview

Package s3readerat implements io.ReaderAt using S3 GetObject and Range.

Index

• type Options
• type S3ReaderAt
  • func New(client *s3.Client, bucket string, key string) (*S3ReaderAt, error)
  • func NewWithOptions(options Options) (*S3ReaderAt, error)
  • func NewWithSize(client *s3.Client, bucket string, key string, size int64) (*S3ReaderAt, error)
  • func (ra *S3ReaderAt) ReadAt(p []byte, off int64) (int, error)
  • func (ra *S3ReaderAt) Size() (int64, error)
  • func (ra *S3ReaderAt) WithContext(ctx context.Context) *S3ReaderAt

Types

type Options

type Options struct {
	// Debug indicates whether to enable debug logging.
	Debug bool

	// Context is the context.Context to use.
	Context context.Context

	// Client is the s3.Client to use when running in single-region mode. You can instead pass s3.Options to run in
	// multi-region mode.
	Client *s3.Client

	// Options are the s3.Options to use when running in multi-region mode. In this mode, S3ReaderAt constructs the
	// s3.Client for you in the appropriate region(s). You can instead pass s3.Client to run in single-region mode.
	Options *s3.Options

	// Bucket is the AWS S3 bucket to use.
	Bucket string

	// Key is the key to use within the AWS S3 bucket. It should not start with a leading slash.
	Key string

	// Size is the size in bytes to use, if known in advance. This is an optimization that avoids calling "HeadObject".
	Size *int64
}

type S3ReaderAt

type S3ReaderAt struct {
	Debug bool
	// contains filtered or unexported fields
}

S3ReaderAt is io.ReaderAt implementation that makes HTTP Range Requests. New instances must be created with the New() function. It is safe for concurrent use.

func New

func New(client *s3.Client, bucket string, key string) (*S3ReaderAt, error)

New creates a new S3ReaderAt.

func NewWithOptions

func NewWithOptions(options Options) (*S3ReaderAt, error)

func NewWithSize

func NewWithSize(client *s3.Client, bucket string, key string, size int64) (*S3ReaderAt, error)

NewWithSize creates a new S3ReaderAt that skips checking the S3 object's size.

func (*S3ReaderAt) ReadAt

func (ra *S3ReaderAt) ReadAt(p []byte, off int64) (int, error)

ReadAt reads len(b) bytes from the remote file starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF. It is safe for concurrent use.

func (*S3ReaderAt) Size

func (ra *S3ReaderAt) Size() (int64, error)

func (*S3ReaderAt) WithContext

func (ra *S3ReaderAt) WithContext(ctx context.Context) *S3ReaderAt