s3blob

Documentation

Overview

Package s3blob implements the blob interfaces for S3.

Index

• Variables
• type Bucket
  • func NewBucket(name string, client s3iface.S3API) *Bucket
  • func (b *Bucket) Copy(ctx context.Context, src, dst string, contentHash string) error
  • func (b *Bucket) CopyFrom(ctx context.Context, srcBucket blob.Bucket, src, dst string) error
  • func (b *Bucket) Delete(ctx context.Context, keys ...string) (err error)
  • func (b *Bucket) Download(ctx context.Context, key, etag string, size int64, w io.WriterAt) (int64, error)
  • func (b *Bucket) File(ctx context.Context, key string) (reflow.File, error)
  • func (b *Bucket) Get(ctx context.Context, key, etag string) (io.ReadCloser, reflow.File, error)
  • func (b *Bucket) Location() string
  • func (b *Bucket) Put(ctx context.Context, key string, size int64, body io.Reader, ...) error
  • func (b *Bucket) Scan(prefix string, withMetadata bool) blob.Scanner
  • func (b *Bucket) Snapshot(ctx context.Context, prefix string) (reflow.Fileset, error)
• type Store
  • func New(sess *session.Session) *Store
  • func (s *Store) Bucket(ctx context.Context, bucket string) (blob.Bucket, error)

Variables

var DefaultRegion = "us-east-1"

DefaultRegion is the region used for s3 requests if a bucket's region is undiscoverable (e.g., lacking permissions for the GetBucketLocation API call.)

Amazon generally defaults to us-east-1 when regions are unspecified (or undiscoverable), but this can be overridden if a different default is desired.

Types

type Bucket

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket represents an s3 bucket; it implements blob.Bucket.

func NewBucket

func NewBucket(name string, client s3iface.S3API) *Bucket

NewBucket returns a new S3 bucket that uses the provided client for SDK calls. NewBucket is primarily intended for testing.

func (*Bucket) Copy

func (b *Bucket) Copy(ctx context.Context, src, dst string, contentHash string) error

Copy copies the key src to the key dst. This is done directly without streaming the data through the client. If a non-empty contentHash is provided, it is stored in the object's metadata.

func (*Bucket) CopyFrom

func (b *Bucket) CopyFrom(ctx context.Context, srcBucket blob.Bucket, src, dst string) error

CopyFrom copies from bucket src and key srcKey into this bucket. This is done directly without streaming the data through the client.

func (*Bucket) Delete

func (b *Bucket) Delete(ctx context.Context, keys ...string) (err error)

Delete removes the provided keys in bulk.

func (*Bucket) Download

func (b *Bucket) Download(ctx context.Context, key, etag string, size int64, w io.WriterAt) (int64, error)

Download downloads the object named by the provided key. Download uses the AWS SDK's download manager, performing concurrent downloads to the provided io.WriterAt.

func (*Bucket) File

func (b *Bucket) File(ctx context.Context, key string) (reflow.File, error)

File returns metadata for the provided key.

func (*Bucket) Get

func (b *Bucket) Get(ctx context.Context, key, etag string) (io.ReadCloser, reflow.File, error)

Get retrieves the object at the provided key.

func (*Bucket) Location

func (b *Bucket) Location() string

Location returns the s3 URL of this bucket, e.g., s3://grail-reflow/.

func (*Bucket) Put

func (b *Bucket) Put(ctx context.Context, key string, size int64, body io.Reader, contentHash string) error

Put stores the contents of the provided io.Reader at the provided key and attaches the given contentHash to the object's metadata.

func (*Bucket) Scan

func (b *Bucket) Scan(prefix string, withMetadata bool) blob.Scanner

Scan returns a scanner that iterates over all objects in the provided prefix. If withMetadata is true, the scanner is configured to make a best-effort attempt to fetch each object's metadata.

func (*Bucket) Snapshot

func (b *Bucket) Snapshot(ctx context.Context, prefix string) (reflow.Fileset, error)

Snapshot returns an un-loaded Reflow fileset of the contents at the provided prefix.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store implements blob.Store for S3. Buckets in store correspond exactly with buckets in S3. Store manages region discovery and session maintenance so that S3 access can be treated uniformly across regions.

func New

func New(sess *session.Session) *Store

New returns a new store that uses the provided session for API access.

func (*Store) Bucket

func (s *Store) Bucket(ctx context.Context, bucket string) (blob.Bucket, error)

Bucket returns the s3 bucket with the provided name. An errors.NotExist error is returned if the bucket does not exist.