s3

Documentation

Index

• func NewS3FileReader(ctx context.Context, bucket string, key string, cfgs ...*aws.Config) (source.ParquetFile, error)
• func NewS3FileReaderWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string) (source.ParquetFile, error)
• func NewS3FileWriter(ctx context.Context, bucket string, key string, acl string, ...) (source.ParquetFile, error)
• func NewS3FileWriterWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string, ...) (source.ParquetFile, error)
• func SetActiveSession(sess *session.Session)
• type S3File
  • func (s *S3File) Close() error
  • func (s *S3File) Create(key string) (source.ParquetFile, error)
  • func (s *S3File) Open(name string) (source.ParquetFile, error)
  • func (s *S3File) Read(p []byte) (n int, err error)
  • func (s *S3File) Seek(offset int64, whence int) (int64, error)
  • func (s *S3File) Write(p []byte) (n int, err error)

Functions

func NewS3FileReader

func NewS3FileReader(ctx context.Context, bucket string, key string, cfgs ...*aws.Config) (source.ParquetFile, error)

NewS3FileReader creates an S3 FileReader, to be used with NewParquetReader

func NewS3FileReaderWithClient

func NewS3FileReaderWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string) (source.ParquetFile, error)

NewS3FileReaderWithClient is the same as NewS3FileReader but allows passing your own S3 client

func NewS3FileWriter

func NewS3FileWriter(
	ctx context.Context,
	bucket string,
	key string,
	acl string,
	uploaderOptions []func(*s3manager.Uploader),
	cfgs ...*aws.Config,
) (source.ParquetFile, error)

NewS3FileWriter creates an S3 FileWriter, to be used with NewParquetWriter

func NewS3FileWriterWithClient

func NewS3FileWriterWithClient(
	ctx context.Context,
	s3Client s3iface.S3API,
	bucket string,
	key string,
	acl string,
	uploaderOptions []func(*s3manager.Uploader),
) (source.ParquetFile, error)

NewS3FileWriterWithClient is the same as NewS3FileWriter but allows passing your own S3 client.

func SetActiveSession

func SetActiveSession(sess *session.Session)

SetActiveSession sets the current session. If this is unset, the functions of this package will implicitly create a new session.Session and use that. This allows you to control what session is used, particularly useful for testing with a system like localstack.

Types

type S3File

type S3File struct {
	BucketName string
	Key        string
	ACL        string
	// contains filtered or unexported fields
}

S3File is ParquetFile for AWS S3

func (*S3File) Close

func (s *S3File) Close() error

Close signals write completion and cleans up any open streams. Will block until pending uploads are complete.

func (*S3File) Create

func (s *S3File) Create(key string) (source.ParquetFile, error)

Create creates a new S3 File instance to perform writes

func (*S3File) Open

func (s *S3File) Open(name string) (source.ParquetFile, error)

Open creates a new S3 File instance to perform concurrent reads

func (*S3File) Read

func (s *S3File) Read(p []byte) (n int, err error)

Read up to len(p) bytes into p and return the number of bytes read

func (*S3File) Seek

func (s *S3File) Seek(offset int64, whence int) (int64, error)

Seek tracks the offset for the next Read. Has no effect on Write.

func (*S3File) Write

func (s *S3File) Write(p []byte) (n int, err error)

Write len(p) bytes from p to the S3 data stream