s3wrapper

Documentation

Index

• func FormatS3Uri(bucket string, key string) string
• func GetErrorMapFromRegex(regEx, url string) (errorsMap map[string]string)
• type ListOutput
• type S3Wrapper
  • func New(svc *s3.S3, maxParallel int) *S3Wrapper
  • func (w *S3Wrapper) GetReader(bucket string, key string) (io.ReadCloser, error)
  • func (w *S3Wrapper) List(s3Uri string, recursive bool, delimiter string, keyRegex string) chan *ListOutput
  • func (w *S3Wrapper) ListAll(s3Uris []string, recursive bool, delimiter string, keyRegex string) chan *ListOutput
  • func (w *S3Wrapper) ListBuckets(s3Uri string) ([]string, error)
  • func (w *S3Wrapper) WithMaxConcurrency(maxConcurrency int) *S3Wrapper

Functions

func FormatS3Uri

func FormatS3Uri(bucket string, key string) string

FormatS3Uri takes a bucket and a prefix and turns it into a S3 URI

func GetErrorMapFromRegex

func GetErrorMapFromRegex(regEx, url string) (errorsMap map[string]string)

GetErrorMapFromRegex takes an error message and a regex for key parts of the error message and returns a map of the error messages as strings

Types

type ListOutput

type ListOutput struct {
	IsPrefix     bool
	Size         int64
	Key          string
	LastModified time.Time
	Bucket       string
	FullKey      string
}

ListOutput represents the pruned and normalized result of a list call to S3, this is meant to cut down on memory and overhead being used in the channels

type S3Wrapper

type S3Wrapper struct {
	// contains filtered or unexported fields
}

S3Wrapper is a wrapper for the S3 library which aims to make some of it's functions faster

func New

func New(svc *s3.S3, maxParallel int) *S3Wrapper

New creates a new S3Wrapper

func (*S3Wrapper) GetReader

func (w *S3Wrapper) GetReader(bucket string, key string) (io.ReadCloser, error)

GetReader retrieves an appropriate reader for the given bucket and key

func (*S3Wrapper) List

func (w *S3Wrapper) List(s3Uri string, recursive bool, delimiter string, keyRegex string) chan *ListOutput

List is a wrapping function to parallelize listings and normalize the results from the API

func (*S3Wrapper) ListAll

func (w *S3Wrapper) ListAll(s3Uris []string, recursive bool, delimiter string, keyRegex string) chan *ListOutput

ListAll is a convienience function for listing and collating all the results for multiple S3 URIs

func (*S3Wrapper) ListBuckets

func (w *S3Wrapper) ListBuckets(s3Uri string) ([]string, error)

ListBuckets returns a list of bucket names and does a prefix filter based on s3Uri (of the form s3://<bucket-prefix>)

func (*S3Wrapper) WithMaxConcurrency

func (w *S3Wrapper) WithMaxConcurrency(maxConcurrency int) *S3Wrapper

WithMaxConcurrency sets the maximum concurrency for the S3 operations