s3blob

Documentation

Overview

Package s3blob provides a blob implementation that uses S3. Use OpenBucket to construct a *blob.Bucket.

URLs

For blob.OpenBucket, s3blob registers for the scheme "s3". The default URL opener will use an AWS session with the default credentials and configuration; see https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more details. To customize the URL opener, or for more details on the URL format, see URLOpener. See https://sraphs.github.io/gdk/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for s3blob:

• Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and the trailing "/" in "//" are escaped in the same way.
• Metadata keys: Escaped using URL encoding, then additionally "@:=" are escaped using "__0x<hex>__". These characters were determined by experimentation.
• Metadata values: Escaped using URL encoding.

As

s3blob exposes the following types for As:

• Bucket: *s3.Client
• Error: any error type returned by the service, notably smithy.APIError
• ListObject: typesv2.Object for objects, typesv2.CommonPrefix for "directories
• ListOptions.BeforeList: *s3.ListObjectsV2Input, or *s3.ListObjectsInput when Options.UseLegacyList == true
• Reader: s3.GetObjectInput
• ReaderOptions.BeforeRead: *s3.GetObjectInput
• Attributes: s3.HeadObjectOutput
• CopyOptions.BeforeCopy: s3.CopyObjectInput
• WriterOptions.BeforeWrite: *s3.PutObjectInput, *s3manager.Uploader
• SignedURLOptions.BeforeSign: *s3.GetObjectInput, when Options.Method == http.MethodGet, or *s3.PutObjectInput, when Options.Method == http.MethodPut, or [not supported] when Options.Method == http.MethodDelete

Example (OpenBucketFromURL)

package main

import (
	"context"
	"log"

	"github.com/sraphs/gdk/blob"
)

func main() {
	// PRAGMA: This example is used on github.com/sraphs/gdk; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On github.com/sraphs/gdk, add a blank import: _ "github.com/sraphs/gdk/blob/s3blob"
	// PRAGMA: On github.com/sraphs/gdk, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	bucket, err := blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()

	// Forcing AWS SDK V2.
	bucket, err = blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1&awssdk=2")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}

Index

• Constants
• Variables
• func OpenBucket(ctx context.Context, client *s3.Client, bucketName string, opts *Options) (*blob.Bucket, error)
• type Options
• type URLOpener
  • func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

Examples

• Package (OpenBucketFromURL)
• OpenBucket

Constants

const Scheme = "s3"

Scheme is the URL scheme s3blob registers its URLOpener under on blob.DefaultMux.

Variables

var Set = wire.NewSet(
	wire.Struct(new(URLOpener), "ConfigProvider"),
)

Set holds Wire providers for this package.

Functions

func OpenBucket

func OpenBucket(ctx context.Context, client *s3.Client, bucketName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by S3, using AWS SDK v2.

Example

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	"github.com/sraphs/gdk/blob/s3blob"
)

func main() {
	// PRAGMA: This example is used on github.com/sraphs/gdk; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On github.com/sraphs/gdk, hide lines until the next blank line.

	// Establish a AWS V2 Config.
	// See https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/ for more info.
	ctx := context.Background()
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create a *blob.Bucket.
	client := s3.NewFromConfig(cfg)
	bucket, err := s3blob.OpenBucket(ctx, client, "my-bucket", nil)
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}

Types

type Options

type Options struct {
	// UseLegacyList forces the use of ListObjects instead of ListObjects.
	// Some S3-compatible services (like CEPH) do not currently support
	// ListObjects.
	UseLegacyList bool
}

Options sets options for constructing a *blob.Bucket backed by fileblob.

type URLOpener

type URLOpener struct {
	// Options specifies the options to pass to OpenBucket.
	Options Options
}

URLOpener opens S3 URLs like "s3://my-bucket".

The URL host is used as the bucket name.

func (*URLOpener) OpenBucketURL

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.