s3blob

Documentation

Overview

Package s3blob provides a blob implementation that uses S3. Use OpenBucket to construct a *blob.Bucket.

Open URLs

For blob.Open URLs, s3blob registers for the scheme "s3"; URLs start with "s3://" like "s3://mybucket". blob.Open will create a new AWS session with the default options. If you want to use a different session or find details on the format of the URL, see URLOpener.

Escaping

Go CDK supports all UTF-8 strings; to make this work with providers lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for s3blob:

• Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and the trailing "/" in "//" are escaped in the same way.
• Metadata keys: Escaped using URL encoding, then additionally "@:=" are escaped using "__0x<hex>__". These characters were determined by experimentation.
• Metadata values: Escaped using URL encoding.

As

s3blob exposes the following types for As:

• Bucket: *s3.S3
• Error: awserr.Error
• ListObject: s3.Object for objects, s3.CommonPrefix for "directories"
• ListOptions.BeforeList: *s3.ListObjectsV2Input
• Reader: s3.GetObjectOutput
• Attributes: s3.HeadObjectOutput
• WriterOptions.BeforeWrite: *s3manager.UploadInput

Example

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go/aws/session"
	"gocloud.dev/blob/s3blob"
)

func main() {
	// Establish an AWS session.
	// See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more info.
	session, err := session.NewSession(nil)
	if err != nil {
		log.Fatal(err)
	}

	// Create a *blob.Bucket.
	ctx := context.Background()
	b, err := s3blob.OpenBucket(ctx, session, "my-bucket", nil)
	if err != nil {
		log.Fatal(err)
	}

	// Now we can use b to read or write files to the container.
	data, err := b.ReadAll(ctx, "my-key")
	if err != nil {
		log.Fatal(err)
	}
	_ = data
}

Example (Open)

package main

import (
	"context"

	"gocloud.dev/blob"
)

func main() {
	ctx := context.Background()

	// Open creates a *blob.Bucket from a URL.
	b, err := blob.OpenBucket(ctx, "s3://my-bucket")
	_, _ = b, err
}

Index

• Constants
• func OpenBucket(ctx context.Context, sess client.ConfigProvider, bucketName string, ...) (*blob.Bucket, error)
• type Options
• type URLOpener
  • func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

Examples

• Package
• Package (Open)

Constants

const Scheme = "s3"

Scheme is the URL scheme s3blob registers its URLOpener under on blob.DefaultMux.

Functions

func OpenBucket

func OpenBucket(ctx context.Context, sess client.ConfigProvider, bucketName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by S3. See the package documentation for an example.

Types

type Options

type Options struct{}

Options sets options for constructing a *blob.Bucket backed by fileblob.

type URLOpener

type URLOpener struct {
	// ConfigProvider must be set to a non-nil value.
	ConfigProvider client.ConfigProvider

	// Options specifies the options to pass to OpenBucket.
	Options Options
}

URLOpener opens S3 URLs like "s3://mybucket".

func (*URLOpener) OpenBucketURL

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens the S3 bucket with the same name as the host in the URL.