s3wrapper

Documentation

Index

• func FormatS3Uri(bucket string, key string) string
• type ListOutput
• type S3Wrapper
  • func New(svc *s3.S3, maxParallel int) *S3Wrapper
  • func (w *S3Wrapper) CopyAll(keys chan *ListOutput, source, dest string, delimiter string, ...) chan *ListOutput
  • func (w *S3Wrapper) DeleteObjects(keys chan *ListOutput) chan *ListOutput
  • func (w *S3Wrapper) GetAll(keys chan *ListOutput, skipExisting bool) chan *ListOutput
  • func (w *S3Wrapper) GetReader(bucket string, key string) (io.ReadCloser, error)
  • func (w *S3Wrapper) List(s3Uri string, recursive bool, delimiter string, keyRegex *string) chan *ListOutput
  • func (w *S3Wrapper) ListAll(s3Uris []string, recursive bool, delimiter string, keyRegex *string) chan *ListOutput
  • func (w *S3Wrapper) ListBuckets(s3Uri string) ([]string, error)
  • func (w *S3Wrapper) Stream(keys chan *ListOutput, includeKeyName bool, raw bool) chan string
  • func (w *S3Wrapper) WithMaxConcurrency(maxConcurrency int) *S3Wrapper

Functions

func FormatS3Uri

func FormatS3Uri(bucket string, key string) string

FormatS3Uri takes a bucket and a prefix and turns it into a S3 URI

Types

type ListOutput

type ListOutput struct {
	IsPrefix     bool
	Size         int64
	Key          string
	LastModified time.Time
	Bucket       string
	FullKey      string
}

ListOutput represents the pruned and normalized result of a list call to S3, this is meant to cut down on memory and overhead being used in the channels

type S3Wrapper

type S3Wrapper struct {
	// contains filtered or unexported fields
}

S3Wrapper is a wrapper for the S3 library which aims to make some of it's functions faster

func New

func New(svc *s3.S3, maxParallel int) *S3Wrapper

New creates a new S3Wrapper

func (*S3Wrapper) CopyAll

func (w *S3Wrapper) CopyAll(keys chan *ListOutput, source, dest string, delimiter string, recurse, flat bool) chan *ListOutput

CopyAll copies keys to the dest, source defines what the base prefix is

func (*S3Wrapper) DeleteObjects

func (w *S3Wrapper) DeleteObjects(keys chan *ListOutput) chan *ListOutput

DeleteObjects deletes all keys in the given keys channel

func (*S3Wrapper) GetAll

func (w *S3Wrapper) GetAll(keys chan *ListOutput, skipExisting bool) chan *ListOutput

GetAll retrieves all keys to the local filesystem, it repurposes ListOutput as it's output which contains the local paths to the keys

func (*S3Wrapper) GetReader

func (w *S3Wrapper) GetReader(bucket string, key string) (io.ReadCloser, error)

GetReader retrieves an appropriate reader for the given bucket and key

func (*S3Wrapper) List

func (w *S3Wrapper) List(s3Uri string, recursive bool, delimiter string, keyRegex *string) chan *ListOutput

List is a wrapping function to parallelize listings and normalize the results from the API

func (*S3Wrapper) ListAll

func (w *S3Wrapper) ListAll(s3Uris []string, recursive bool, delimiter string, keyRegex *string) chan *ListOutput

ListAll is a convienience function for listing and collating all the results for multiple S3 URIs

func (*S3Wrapper) ListBuckets

func (w *S3Wrapper) ListBuckets(s3Uri string) ([]string, error)

ListBuckets returns a list of bucket names and does a prefix filter based on s3Uri (of the form s3://<bucket-prefix>)

func (*S3Wrapper) Stream

func (w *S3Wrapper) Stream(keys chan *ListOutput, includeKeyName bool, raw bool) chan string

Stream provides a channel with data from the keys

func (*S3Wrapper) WithMaxConcurrency

func (w *S3Wrapper) WithMaxConcurrency(maxConcurrency int) *S3Wrapper

WithMaxConcurrency sets the maximum concurrency for the S3 operations