s3

README

S3 Storage Backend

The S3 backend provides support for Amazon S3 and compatible services.

Configuration

To use the S3 backend, you need to specify the following options in your connection string:

• endpoint: The endpoint URL of your S3 service.
• region: The AWS region.
• accessKey: Your AWS access key.
• secretKey: Your AWS secret key.
• sse: Server Side Encryption setting.

A connection string for the S3 backend looks like this:

s3://myBucketName/my/prefix?region=region&accessKey=accessKey&secretKey=secretKey&sse=sse

Replace myBucketName/my/prefix, region, accessKey, secretKey, and sse with your actual parameters.

Documentation

Index

• func New(opts Options) (types.Storage, error)
• func NewCredentials(accessKey, secretKey string) *credentials.Credentials
• type Backend
  • func (b *Backend) AbortMultipartUploadWithContext(ctx context.Context, path, uploadID string) error
  • func (b *Backend) CompleteMultipartUploadWithContext(ctx context.Context, path, uploadID string, ...) error
  • func (b *Backend) DeleteWithContext(ctx context.Context, path string) error
  • func (b *Backend) InitiateMultipartUploadWithContext(ctx context.Context, path string) (string, error)
  • func (b *Backend) ListWithContext(ctx context.Context, prefix string) (*[]types.Object, error)
  • func (b *Backend) MoveToBucketWithContext(ctx context.Context, srcPath, dstPath, dstBucket string) error
  • func (b *Backend) MoveWithContext(ctx context.Context, fromPath string, toPath string) error
  • func (b *Backend) ReadWithContext(ctx context.Context, path string, start int64, end int64) (io.ReadCloser, error)
  • func (b *Backend) StatWithContext(ctx context.Context, path string) (*types.Object, error)
  • func (b *Backend) WriteMultipartWithContext(ctx context.Context, path, uploadID string, partNumber int64, ...) (int64, *types.CompletedPart, error)
  • func (b *Backend) WriteWithContext(ctx context.Context, path string, reader io.Reader, size int64) (int64, error)
• type Options

Functions

func New

func New(opts Options) (types.Storage, error)

func NewCredentials

func NewCredentials(accessKey, secretKey string) *credentials.Credentials

Types

type Backend

type Backend struct {
	Bucket     string
	Client     *s3.S3
	Downloader *s3manager.Downloader
	Prefix     string
	Uploader   *s3manager.Uploader
	SSE        string
}

Backend is a storage backend for S3

func (*Backend) AbortMultipartUploadWithContext

func (b *Backend) AbortMultipartUploadWithContext(ctx context.Context, path, uploadID string) error

AbortMultipartUpload aborts the multipart upload

func (*Backend) CompleteMultipartUploadWithContext

func (b *Backend) CompleteMultipartUploadWithContext(ctx context.Context, path, uploadID string, completedParts []*types.CompletedPart) error

CompleteMultipartUpload completes a multipart upload

func (*Backend) DeleteWithContext

func (b *Backend) DeleteWithContext(ctx context.Context, path string) error

DeleteObject removes an object from a S3 bucket, at prefix

func (*Backend) InitiateMultipartUploadWithContext

func (b *Backend) InitiateMultipartUploadWithContext(ctx context.Context, path string) (string, error)

InitiateMultipartUpload initiates a multipart upload.

Use WriteWithContext over this method if the full file is already on the local machine as it will do the multipart upload for you.

func (*Backend) ListWithContext

func (b *Backend) ListWithContext(ctx context.Context, prefix string) (*[]types.Object, error)

ListObjects lists all objects in a S3 bucket, at prefix Note: This function does not handle pagination and will return a maximum of 1000 objects. If there are more than 1000 objects with the specified prefix, consider using pagination to retrieve all objects.

func (*Backend) MoveToBucketWithContext

func (b *Backend) MoveToBucketWithContext(ctx context.Context, srcPath, dstPath, dstBucket string) error

MoveToBucket moves an object from one S3 bucket to another

func (*Backend) MoveWithContext

func (b *Backend) MoveWithContext(ctx context.Context, fromPath string, toPath string) error

MoveObject moves an object from one path to another within a S3 bucket

func (*Backend) ReadWithContext

func (b *Backend) ReadWithContext(ctx context.Context, path string, start int64, end int64) (io.ReadCloser, error)

ReadWithContext reads an object from S3 bucket, at given path

func (*Backend) StatWithContext

func (b *Backend) StatWithContext(ctx context.Context, path string) (*types.Object, error)

Stat returns information about an object in a S3 bucket, at given path

func (*Backend) WriteMultipartWithContext

func (b *Backend) WriteMultipartWithContext(ctx context.Context, path, uploadID string, partNumber int64, reader io.ReadSeeker, size int64) (int64, *types.CompletedPart, error)

WriteMultipartWithContext writes a part of a multipart upload

func (*Backend) WriteWithContext

func (b *Backend) WriteWithContext(ctx context.Context, path string, reader io.Reader, size int64) (int64, error)

WriteObject uploads an object to S3, intelligently buffering large files into smaller chunks and sending them in parallel across multiple goroutines.

Always use this method if your full file is already on disk or in memory.

type Options

type Options struct {
	Endpoint   string
	Prefix     string
	Region     string
	AccessKey  string
	SecretKey  string
	BucketName string
	SSE        string
}

Options is a struct for S3 options