s3

Documentation

Overview

Package s3 is for working with AWS S3 buckets.

Index

• type ConcurrencyManager
  • func (cm *ConcurrencyManager) Process(asyncProcessor FileProcessor, objects []types.Object) chan HydratedFile
• type Event
• type EventBucket
• type EventObject
• type EventRecord
• type EventS3
• type FileProcessor
• type HydratedFile
• type Meta
• type S3
  • func New() (S3, error)
  • func NewWithMaxRetries(maxRetries int) (S3, error)
  • func NewWithOptions(optFns ...func(*s3.Options)) (S3, error)
  • func (s3 *S3) AddDownloader() error
  • func (s3 *S3) AddUploader() error
  • func (s *S3) CheckBucket(bucket string) error
  • func (s *S3) Copy(bucket, key, source string) error
  • func (s *S3) Delete(bucket, key string) error
  • func (s *S3) Download(bucket, key string, f *os.File) (int64, error)
  • func (s *S3) Exists(bucket, key string) (bool, error)
  • func (s *S3) Get(bucket, key, version string, b *bytes.Buffer) error
  • func (s *S3) GetByteRange(bucket, key, version, byteRange string, b *bytes.Buffer) error
  • func (s *S3) GetContentSizeTime(bucket, key string) (int64, time.Time, error)
  • func (s *S3) GetMeta(bucket, key, version string) (Meta, error)
  • func (s *S3) GetWithLastModified(bucket, key, version string, b *bytes.Buffer) (time.Time, error)
  • func (s *S3) LastModified(bucket, key, version string) (time.Time, error)
  • func (s *S3) List(bucket, prefix string, max int32) ([]string, error)
  • func (s *S3) ListAll(bucket, prefix string) ([]string, error)
  • func (s *S3) ListAllObjects(bucket, prefix string) ([]types.Object, error)
  • func (s *S3) ListAllObjectsConcurrently(bucket string, prefixes []string) ([]types.Object, error)
  • func (s *S3) ListCommonPrefixes(bucket, prefix, delimiter string) ([]string, error)
  • func (s *S3) ListObjects(bucket, prefix string, max int32) ([]types.Object, error)
  • func (s *S3) PrefixExists(bucket, prefix string) (bool, error)
  • func (s *S3) Put(bucket, key string, object []byte) error
  • func (s *S3) PutStream(bucket, key string, reader io.ReadCloser) error
  • func (s *S3) PutWithMetadata(bucket, key string, object []byte, metadata Meta) error
  • func (s *S3) Ready() bool
• type S3Concurrent
  • func NewConcurrent(maxConnections, maxConnectionsPerRequest, maxBytes int) (S3Concurrent, error)
  • func (s *S3Concurrent) GetAllConcurrently(bucket, version string, objects []types.Object) chan HydratedFile

Types

type ConcurrencyManager

type ConcurrencyManager struct {
	// contains filtered or unexported fields
}

func (*ConcurrencyManager) Process

func (cm *ConcurrencyManager) Process(asyncProcessor FileProcessor, objects []types.Object) chan HydratedFile

Functions for providing a fan-out/fan-in operation. Workers are taken from the worker pool and added to a WorkerGroup. All workers are returned to the pool once the jobs have finished.

type Event

type Event struct {
	Records []EventRecord
}

type EventBucket

type EventBucket struct {
	Name string `json:"name,omitempty"`
}

type EventObject

type EventObject struct {
	Key       string `json:"key,omitempty"`
	VersionId string `json:"versionId,omitempty"`
}

type EventRecord

type EventRecord struct {
	EventName string  `json:"eventName,omitempty"`
	S3        EventS3 `json:"s3,omitempty"`
}

type EventS3

type EventS3 struct {
	Object EventObject `json:"object,omitempty"`
	Bucket EventBucket `json:"bucket,omitempty"`
}

type FileProcessor

type FileProcessor func(types.Object) HydratedFile

type HydratedFile

type HydratedFile struct {
	Key   string
	Data  []byte
	Error error
}

type Meta

type Meta map[string]string

type S3

type S3 struct {
	// contains filtered or unexported fields
}

func New

func New() (S3, error)

New returns an S3 struct which wraps an S3 client using the default AWS credentials chain. This consults (in order) environment vars, config files, EC2 and ECS roles. It is an error if the AWS_REGION environment variable is not set. Requests with recoverable errors will be retried with the default retrier.

func NewWithMaxRetries

func NewWithMaxRetries(maxRetries int) (S3, error)

NewWithMaxRetries returns the same as New(), but with the back off set to up to maxRetries times.

func NewWithOptions

func NewWithOptions(optFns ...func(*s3.Options)) (S3, error)

NewWithOptions returns the same as New(), but with the additional option functions applied.

func (*S3) AddDownloader

func (s3 *S3) AddDownloader() error

AddDownloader creates an s3manager downloader and sets it to the S3 struct's downloader field. This can be used for downloading objects in concurrent chunks.

func (*S3) AddUploader

func (s3 *S3) AddUploader() error

AddUploader creates an s3manager uploader and sets it to the S3 struct's uploader field. This can be used for streaming uploading.

func (*S3) CheckBucket

func (s *S3) CheckBucket(bucket string) error

CheckBucket checks if the given S3 bucket exists and is accessible.

func (*S3) Copy

func (s *S3) Copy(bucket, key, source string) error

Copy copies from the source to the bucket with key as the new name. source should include the bucket name eg: "mybucket/objectkey.pdf"

func (*S3) Delete

func (s *S3) Delete(bucket, key string) error

Delete deletes an object from a bucket.

func (*S3) Download

func (s *S3) Download(bucket, key string, f *os.File) (int64, error)

Download uses the downloader to download file from bucket. File is split up into parts and downloaded concurrently into an os.File, so is useful for getting large files. Returns number of bytes downloaded.

func (*S3) Exists

func (s *S3) Exists(bucket, key string) (bool, error)

Exists checks if an object for key already exists in the bucket.

func (*S3) Get

func (s *S3) Get(bucket, key, version string, b *bytes.Buffer) error

Get gets the object referred to by key and version from bucket and writes it into b. Version can be empty.

func (*S3) GetByteRange

func (s *S3) GetByteRange(bucket, key, version, byteRange string, b *bytes.Buffer) error

GetByteRange gets the specified byte range of an object referred to by key and version from bucket and writes it into b. Version can be empty. See https://www.rfc-editor.org/rfc/rfc9110.html#name-byte-ranges for examples

func (*S3) GetContentSizeTime

func (s *S3) GetContentSizeTime(bucket, key string) (int64, time.Time, error)

GetContentSize returns the content length and last modified time of the specified key

func (*S3) GetMeta

func (s *S3) GetMeta(bucket, key, version string) (Meta, error)

GetMeta returns the metadata for an object. Version can be empty.

func (*S3) GetWithLastModified

func (s *S3) GetWithLastModified(bucket, key, version string, b *bytes.Buffer) (time.Time, error)

GetWithLastModified behaves the same as Get(), but also returns the time that the object was last modified.

func (*S3) LastModified

func (s *S3) LastModified(bucket, key, version string) (time.Time, error)

LastModified returns the time that the specified object was last modified.

func (*S3) List

func (s *S3) List(bucket, prefix string, max int32) ([]string, error)

List returns a list of object keys that match the provided prefix. It will not return more than the specified max number of keys. Keys are in alphabetical order.

func (*S3) ListAll

func (s *S3) ListAll(bucket, prefix string) ([]string, error)

ListAll returns a list of ALL object keys that match the provided prefix. Keys are in alphabetical order.

func (*S3) ListAllObjects

func (s *S3) ListAllObjects(bucket, prefix string) ([]types.Object, error)

ListAllObjects returns a list of ALL objects that match the provided prefix. Keys are in alphabetical order.

func (*S3) ListAllObjectsConcurrently

func (s *S3) ListAllObjectsConcurrently(bucket string, prefixes []string) ([]types.Object, error)

ListAllObjectsConcurrently returns a list of ALL objects that match the provided prefixes. Keys are NOT in alphabetical order.

func (*S3) ListCommonPrefixes

func (s *S3) ListCommonPrefixes(bucket, prefix, delimiter string) ([]string, error)

ListCommonPrefixes returns a list of ALL common prefixes (no 1000 limit).

func (*S3) ListObjects

func (s *S3) ListObjects(bucket, prefix string, max int32) ([]types.Object, error)

ListObjects returns a list of objects that match the provided prefix. It will not return more than the specified max number of keys.

func (*S3) PrefixExists

func (s *S3) PrefixExists(bucket, prefix string) (bool, error)

Returns whether there is an object in bucket with specified prefix.

func (*S3) Put

func (s *S3) Put(bucket, key string, object []byte) error

Put puts the object in bucket using specified key.

func (*S3) PutStream

func (s *S3) PutStream(bucket, key string, reader io.ReadCloser) error

PutStream puts the data stream to key in bucket.

func (*S3) PutWithMetadata

func (s *S3) PutWithMetadata(bucket, key string, object []byte, metadata Meta) error

Put puts the object in bucket with metadata using specified key.

func (*S3) Ready

func (s *S3) Ready() bool

Ready returns whether the S3 client has been initialised.

type S3Concurrent

type S3Concurrent struct {
	S3
	// contains filtered or unexported fields
}

func NewConcurrent

func NewConcurrent(maxConnections, maxConnectionsPerRequest, maxBytes int) (S3Concurrent, error)

NewConcurrent returns an S3Concurrent client, which embeds an S3 client, and has a ConcurrencyManager to allow the use of the GetAllConcurrently function. The GetAllConcurrently function can download multiple files at once while retaining order. The S3 client is configured to make use of the specified maxConnections. Also, ensure that the S3 Client has access to maxBytes in memory to avoid out of memory errors.

func (*S3Concurrent) GetAllConcurrently

func (s *S3Concurrent) GetAllConcurrently(bucket, version string, objects []types.Object) chan HydratedFile

GetAllConcurrently gets the objects specified from bucket and writes the resulting HydratedFiles to the returned output channel. The closure of this channel is handled, however it's the caller's responsibility to purge the channel, and handle any errors present in the HydratedFiles. If the ConcurrencyManager is not initialised before calling GetAllConcurrently, an output channel containing a single HydratedFile with an error is returned. Version can be empty, but must be the same for all objects.