s3

README

S3 Stow Implementation

Location = Amazon S3

Container = Bucket

Item = File

Helpful Links:

http://docs.aws.amazon.com/sdk-for-go/api/service/s3/#example_S3_ListBuckets

---

SDK Notes:

• Metadata of an S3 Object can only be set when the Object is created.

---

Concerns:

• An AWS account may have credentials which temporarily modifies permissions. This is specified by a token value. This feature is implemented but disabled and added as a TODO.

---

Things to know:

• Paging for the list of containers doesn't exist yet, this is because there's a hard limit of about 100 containers for every account.

• A client is required to provide a region. Manipulating buckets that reside within other regions isn't possible.

---

Dev Notes

The init function of every implementation of stow must call stow.Register.

stow.Register accepts a few things:

Kind, a string argument respresenting the name of the location.

makefn a function that accepts any type that conforms to the stow.Config interface. It first validates the values of the Config argument, and then attempts to use the configuration to create a new client. If successful, An instance of a data type that conforms to the stow.Location interface is created. This Location should have fields that contain the client and configuration.

Further calls in the hierarchy of a Location, Container, and Item depend on the values of the configuration + the client to send and receive information.

• kingmatchfn a function that ensures that a given URL matches the Kind of the type of storage.

---

*stow.Register(kind string, makefn func(Config) (Locaion, error), kindmatchfn func(url.URL) bool)

• Adds kind and makefn into a map that contains a list of locations.

• Adds kind to a slice that contains all of the different kinds.

• Adds kind as part of an anonymous function which validates the scheme of the url.URL

Once the stow.Register function is completed, a location of the given kind is returned.

---

Documentation

Overview

Package s3 provides an abstraction of Amazon S3 (Simple Storage Service). An S3 Bucket is a Stow Container and an S3 Object is a Stow Item. Recall that nested directories exist within S3.

Usage and Credentials

There are three separate pieces of information required by Stow to have access to an S3 Stow Location: an AWS User's ACCESS_KEY_ID and SECRET_KEY fields, as well as the physical region of the S3 Endpoint. Ensure that the AWS User whose credentials are used to manipulate the S3 endpoint has permissions to do so.

stow.Dial requires both a string value ("s3") of the particular Stow Location Kind and a stow.Config instance. The stow.Config instance requires three entries with the specific key value attributes:

- a key of s3.ConfigAccessKeyID with a value of the AWS account's Access Key ID - a key of s3.ConfigSecretKey with a value of the AWS account's Secret Key - a key of s3.ConfigRegion with a value of the S3 endpoint's region (in all lowercase)

Location

The s3.location methods allow the retrieval of an S3 endpoint's Bucket or list of Buckets (Container or Containers). A stow.Item representation of an S3 Object can also be retrieved based on the Object's URL (ItemByURL).

Additional s3.location methods provide capabilities to create and remove S3 Buckets (CreateContainer or RemoveContainer, respectively).

Container

There are s3.container methods which can retrieve an S3 Bucket's:

- name (ID or Name) - Object or complete list of Objects (Item or Items) - region

Additional s3.container methods give Stow the ability to:

- remove an S3 Bucket (RemoveItem) - update or create an S3 Object (Put)

Item

Methods within an s3.item allow the retrieval of an S3 Object's: - name (ID or name) - URL (ItemByUrl) - size in bytes (Size) - S3 specific metadata (Metadata, key value pairs usually found within the console) - last modified date (LastMod) - Etag (Etag) - content (Open)

Index

• Constants

Constants

const (
	// ConfigAuthType is an optional argument that defines whether to use an IAM role or access key based auth
	ConfigAuthType = "auth_type"

	// ConfigAccessKeyID is one key of a pair of AWS credentials.
	ConfigAccessKeyID = "access_key_id"

	// ConfigSecretKey is one key of a pair of AWS credentials.
	ConfigSecretKey = "secret_key"

	// ConfigRegion represents the region/availability zone of the session.
	ConfigRegion = "region"

	// ConfigEndpoint is optional config value for changing s3 endpoint
	// used for e.g. minio.io
	ConfigEndpoint = "endpoint"

	// ConfigHTTPTracing enable verbose logs for http requests
	ConfigHTTPTracing = "false"
)

const Kind = "s3"

Kind represents the name of the location/storage type.