s3

Documentation

Index

• func NewS3FileReader(ctx context.Context, bucket string, key string, cfgs ...*aws.Config) (source.ParquetFile, error)
• func NewS3FileReaderVersioned(ctx context.Context, bucket string, key string, version *string, ...) (source.ParquetFile, error)
• func NewS3FileReaderVersionedWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string, ...) (source.ParquetFile, error)
• func NewS3FileReaderWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string) (source.ParquetFile, error)
• func NewS3FileReaderWithParams(ctx context.Context, params S3FileReaderParams) (source.ParquetFile, error)
• func NewS3FileWriter(ctx context.Context, bucket string, key string, acl string, ...) (source.ParquetFile, error)
• func NewS3FileWriterWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string, ...) (source.ParquetFile, error)
• func SetActiveSession(sess *session.Session)
• type S3File
  • func (s *S3File) Close() error
  • func (s *S3File) Create(key string) (source.ParquetFile, error)
  • func (s *S3File) Open(name string) (source.ParquetFile, error)
  • func (s *S3File) Read(p []byte) (n int, err error)
  • func (s *S3File) Seek(offset int64, whence int) (int64, error)
  • func (s *S3File) Write(p []byte) (n int, err error)
• type S3FileReaderParams

Functions

func NewS3FileReader

func NewS3FileReader(ctx context.Context, bucket string, key string, cfgs ...*aws.Config) (source.ParquetFile, error)

NewS3FileReader creates an S3 FileReader, to be used with NewParquetReader

func NewS3FileReaderVersioned

func NewS3FileReaderVersioned(ctx context.Context, bucket string, key string, version *string, cfgs ...*aws.Config) (source.ParquetFile, error)

NewS3FileReaderVersioned creates an S3 FileReader for a versioned of S3 object, to be used with NewParquetReader

func NewS3FileReaderVersionedWithClient

func NewS3FileReaderVersionedWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string, version *string) (source.ParquetFile, error)

NewS3FileReaderVersionedWithClient is the same as NewS3FileReaderVersioned but allows passing your own S3 client

func NewS3FileReaderWithClient

func NewS3FileReaderWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string) (source.ParquetFile, error)

NewS3FileReaderWithClient is the same as NewS3FileReader but allows passing your own S3 client

func NewS3FileReaderWithParams

func NewS3FileReaderWithParams(ctx context.Context, params S3FileReaderParams) (source.ParquetFile, error)

NewS3FileReaderWithParams creates an S3 FileReader for an object identified by and configured using the S3FileReaderParams object.

func NewS3FileWriter

func NewS3FileWriter(
	ctx context.Context,
	bucket string,
	key string,
	acl string,
	uploaderOptions []func(*s3manager.Uploader),
	cfgs ...*aws.Config,
) (source.ParquetFile, error)

NewS3FileWriter creates an S3 FileWriter, to be used with NewParquetWriter

func NewS3FileWriterWithClient

func NewS3FileWriterWithClient(
	ctx context.Context,
	s3Client s3iface.S3API,
	bucket string,
	key string,
	acl string,
	uploaderOptions []func(*s3manager.Uploader),
) (source.ParquetFile, error)

NewS3FileWriterWithClient is the same as NewS3FileWriter but allows passing your own S3 client.

func SetActiveSession

func SetActiveSession(sess *session.Session)

SetActiveSession sets the current session. If this is unset, the functions of this package will implicitly create a new session.Session and use that. This allows you to control what session is used, particularly useful for testing with a system like localstack.

Types

type S3File

type S3File struct {
	BucketName string
	Key        string
	VersionId  *string
	ACL        string
	// contains filtered or unexported fields
}

S3File is ParquetFile for AWS S3

func (*S3File) Close

func (s *S3File) Close() error

Close signals write completion and cleans up any open streams. Will block until pending uploads are complete.

func (*S3File) Create

func (s *S3File) Create(key string) (source.ParquetFile, error)

Create creates a new S3 File instance to perform writes

func (*S3File) Open

func (s *S3File) Open(name string) (source.ParquetFile, error)

Open creates a new S3 File instance to perform concurrent reads

func (*S3File) Read

func (s *S3File) Read(p []byte) (n int, err error)

Read up to len(p) bytes into p and return the number of bytes read

func (*S3File) Seek

func (s *S3File) Seek(offset int64, whence int) (int64, error)

Seek tracks the offset for the next Read. Has no effect on Write.

func (*S3File) Write

func (s *S3File) Write(p []byte) (n int, err error)

Write len(p) bytes from p to the S3 data stream

type S3FileReaderParams

type S3FileReaderParams struct {
	Bucket string
	Key    string

	// S3Client will be used to issue requests to S3. If not set, a new one one will be
	// created. Optional.
	S3Client s3iface.S3API
	// Configs are the configs used to construct a new S3Client. If S3Client is provided,
	// Configs are ignored. Optional.
	Configs []*aws.Config
	// Version is the version of the S3 object that will be read. If not set, the newest
	// version will be read. Optional.
	Version *string
	// MinRequestSize controls the amount of data per request that the S3File will ask for
	// from S3. Optional.
	// A large MinRequestSize should improve performance and reduce AWS costs due to
	// number of request. However, in some cases it may increase AWS costs due to data
	// processing or data transfer. For best results, set it at or above the largest of
	// the footer size and the biggest chunk size in the parquet file.
	// S3File will not buffer a large amount of data in memory at one time, regardless
	// of the value of MinRequestSize.
	MinRequestSize int
}

S3FileReaderParams contains fields used to initialize and configure an S3File object for reading.