s3file

Documentation

Overview

Package s3file implements grail file interface for S3.

Index

• Constants
• Variables
• func FindBucketRegion(ctx context.Context, bucket string) (string, error)
• func NewImplementation(provider SessionProvider, opts Options) file.Implementation
• func ParseURL(url string) (scheme, bucket, key string, err error)
• func ReadChunkBytes() int
• type Options
• type SessionProvider
  • func NewDefaultProvider(configs ...*aws.Config) SessionProvider
• type SessionProviderInput

Examples

• ParseURL

Constants

const (
	Scheme = "s3"
)

Variables

var (
	// BackoffPolicy defines backoff timing parameters. It's exported for unit tests only.
	// TODO(josh): Rename to `RetryPolicy`.
	// TODO(josh): Create `retry.ThrottlePolicy` and `retry.AIMDPolicy` and use here.
	BackoffPolicy = retry.Jitter(retry.Backoff(500*time.Millisecond, time.Minute, 1.2), 0.2)

	// WithDeadline allows faking context.WithDeadline. It's exported for unit tests only.
	WithDeadline = context.WithDeadline

	// MaxRetryDuration defines the max amount of time a request can spend
	// retrying on errors.
	//
	// Requirements:
	//
	// - The value must be >5 minutes. 5 min is the S3 negative-cache TTL.  If
	//   less than 5 minutes, an Open() call w/ RetryWhenNotFound may fail.
	//
	// - It must be long enough to allow CompleteMultiPartUpload to finish after a
	//   retry. The doc says it may take a few minutes even in a successful case.
	MaxRetryDuration = 60 * time.Minute
)

var UploadPartSize = 16 << 20

UploadPartSize is the size of a chunk during multi-part uploads. It is exposed only for unittests.

var (
	VersionsFunc versionsFunc
)

Functions

func FindBucketRegion

func FindBucketRegion(ctx context.Context, bucket string) (string, error)

FindBucketRegion locates the AWS region in which bucket is located. The lookup is cached internally.

It assumes the region is in the "aws" partition, not other partitions like "aws-us-gov". See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html

func NewImplementation

func NewImplementation(provider SessionProvider, opts Options) file.Implementation

NewImplementation creates a new file.Implementation for S3. The provider is called to create s3 client objects.

func ParseURL

func ParseURL(url string) (scheme, bucket, key string, err error)

ParseURL parses a path of form "s3://grail-bucket/dir/file" and returns ("s3", "grail-bucket", "dir/file", nil).

Example

scheme, bucket, key, err := ParseURL("s3://grail-bucket/dir/file")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
scheme, bucket, key, err = ParseURL("s3://grail-bucket/dir/")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
scheme, bucket, key, err = ParseURL("s3://grail-bucket")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)

Output:

scheme: s3, bucket: grail-bucket, key: dir/file, err: <nil>
scheme: s3, bucket: grail-bucket, key: dir/, err: <nil>
scheme: s3, bucket: grail-bucket, key: , err: <nil>

func ReadChunkBytes

func ReadChunkBytes() int

ReadChunkBytes is the size for individual S3 API read operations, guided by S3 docs:

As a general rule, when you download large objects within a Region from Amazon S3 to
Amazon EC2, we suggest making concurrent requests for byte ranges of an object at the
granularity of 8–16 MB.
https://web.archive.org/web/20220325121400/https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html

Types

type Options

type Options struct {
	// ServerSideEncryption allows you to set the `ServerSideEncryption` value to use when
	// uploading files (e.g.  "AES256")
	ServerSideEncryption string
}

Options defines options that can be given when creating an s3Impl

type SessionProvider

type SessionProvider interface {
	// Get returns AWS sessions that can be used to perform in.S3IAMAction on
	// s3://{in.bucket}/{in.key}.
	//
	// s3file maintains an internal cache keyed by *session.Session that is only pruned
	// occasionally. Get() is called for every S3 operation so it should be very fast. Caching
	// (that is, reusing *session.Session whenever possible) is strongly encouraged.
	//
	// Get() must return >= 1 session, or error. If > 1, the S3 operation will be tried
	// on each session in unspecified order until it succeeds.
	//
	// Note: Some implementations will not need SessionProviderInput and can just ignore it.
	//
	// TODO: Consider passing chan<- *session.Session (implementer sends and then closes)
	// so s3file can try credentials as soon as they're available.
	Get(_ context.Context, in SessionProviderInput) ([]*session.Session, error)
}

SessionProvider provides Sessions for making AWS API calls. Get() is called whenever s3file needs to access a file. The provider should cache and reuse the sessions, if needed. The implementation must be thread safe.

func NewDefaultProvider

func NewDefaultProvider(configs ...*aws.Config) SessionProvider

NewDefaultProvider returns a SessionProvider that calls session.NewSession(configs...) once.

type SessionProviderInput

type SessionProviderInput struct {
	// S3IAMAction is an action name from this list:
	// https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html
	//
	// Note: There is no `s3:` prefix.
	//
	// Note: This is different from the notion of "action" in the S3 API documentation:
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html
	// Some names, like GetObject, appear in both; others, like HeadObject, do not.
	S3IAMAction string
	// Bucket and Key describe the API operation to be performed, if applicable.
	Bucket, Key string
}