gs

Documentation

Overview

Package gs implements a versatile Google Storage client on top of the standard Google Storage Go API. It adds:

• The ability to read from specific byte offsets.
• Exponential backoff retries on transient errors.
• Logging
• The ability to easily stub a Google Storage interface.

Index

• Variables
• type Client
  • func NewProdClient(ctx context.Context, rt http.RoundTripper) (Client, error)
• type LimitedClient
  • func (lc *LimitedClient) NewReader(p Path, offset, length int64) (io.ReadCloser, error)
• type Path
  • func MakePath(bucket, filename string) Path
  • func (p Path) Bucket() string
  • func (p Path) Concat(v string, parts ...string) Path
  • func (p Path) Filename() string
  • func (p Path) IsFullPath() bool
  • func (p Path) Split() (bucket string, filename string)
• type Writer

Variables

var (
	// ReadWriteScopes is the set of scopes needed for read/write Google Storage
	// access.
	ReadWriteScopes = []string{gs.ScopeReadWrite}

	// ReadOnlyScopes is the set of scopes needed for read/write Google Storage
	// read-only access.
	ReadOnlyScopes = []string{gs.ScopeReadOnly}
)

Types

type Client

type Client interface {
	io.Closer

	// Attrs retrieves Object attributes for a given path.
	Attrs(p Path) (*gs.ObjectAttrs, error)

	// NewReader instantiates a new Reader instance for the named bucket/path.
	//
	// The supplied offset must be >= 0, or else this function will panic.
	//
	// If the supplied length is <0, no upper byte bound will be set.
	NewReader(p Path, offset, length int64) (io.ReadCloser, error)

	// NewWriter instantiates a new Writer instance for the named bucket/path.
	NewWriter(p Path) (Writer, error)

	// Delete deletes the object at the specified path.
	//
	// If the object does not exist, it is considered a success.
	Delete(p Path) error

	// Rename renames an object from one path to another.
	//
	// NOTE: The object should be removed from its original path, but current
	// implementation uses two operations (Copy + Delete), so it may
	// occasionally fail.
	Rename(src, dst Path) error
}

Client abstracts funcitonality to connect with and use Google Storage from the actual Google Storage client.

Non-production implementations are used primarily for testing.

func NewProdClient

func NewProdClient(ctx context.Context, rt http.RoundTripper) (Client, error)

NewProdClient creates a new Client instance that uses production Cloud Storage.

The supplied RoundTripper will be used to make connections. If nil, the default HTTP client will be used.

type LimitedClient

type LimitedClient struct {
	// Base is the base Client instance.
	Client

	// MaxReadBytes, if >0, is the maximum number of bytes that can be read at a
	// time. If more bytes are required, additional read calls will be made.
	MaxReadBytes int64
}

LimitedClient wraps a base Client, allowing additional limits to be applied to its calls.

func (*LimitedClient) NewReader

func (lc *LimitedClient) NewReader(p Path, offset, length int64) (io.ReadCloser, error)

NewReader implements Client.

type Path

type Path string

Path is a Google Storage path. A full path consists of a Google storage bucket and a series of path components.

An example of a Path is:

gs://test-bucket/path/to/thing.txt

func MakePath

func MakePath(bucket, filename string) Path

MakePath constructs a Google Storage path from optional bucket and filename components.

Trailing forward slashes will be removed from the bucket name, if present.

func (Path) Bucket

func (p Path) Bucket() string

Bucket returns the Google Storage bucket component of the Path. If there is no bucket, an empty string will be returned.

func (Path) Concat

func (p Path) Concat(v string, parts ...string) Path

Concat concatenates a filename component to the end of Path.

Multiple components may be specified. In this case, each will be added as a "/"-delimited component, and will have any present trailing slashes stripped.

func (Path) Filename

func (p Path) Filename() string

Filename returns the filename component of the Path. If there is no filename component, an empty string will be returned.

Leading and trailing slashes will be truncated.

func (Path) IsFullPath

func (p Path) IsFullPath() bool

IsFullPath returns true if the Path contains both a bucket and file name.

func (Path) Split

func (p Path) Split() (bucket string, filename string)

Split returns the bucket and filename components of the Path.

If a bucket is not defined (doesn't begin with "gs://"), the remainder will be considered to be the filename component. If a filename is not defined, an empty string will be returned.

type Writer

type Writer interface {
	io.WriteCloser

	// Count returns the number of bytes written by the object.
	Count() int64
}

Writer is an augmented io.WriteCloser instance.