storage

Documentation

Overview

Package storage wraps Google Storage routines into a simpler interface.

Index

• type Metadata
  • func ParseMetadata(m map[string]string) *Metadata
  • func (m *Metadata) Add(md Metadatum)
  • func (m *Metadata) Assemble() map[string]string
  • func (m *Metadata) Clone() *Metadata
  • func (m *Metadata) Empty() bool
  • func (m *Metadata) Equal(o *Metadata) bool
  • func (m *Metadata) Keys() []string
  • func (m *Metadata) ToPretty(now time.Time, limit int) string
  • func (m *Metadata) TrimUnimportant(keep int)
  • func (m *Metadata) Values(k string) []Metadatum
• type Metadatum
• type Object
  • func (o *Object) Log(ctx context.Context)
  • func (o *Object) String() string
• type Storage
  • func New(ctx context.Context, ts oauth2.TokenSource, location string) (*Storage, error)
  • func (s *Storage) Check(ctx context.Context, name string) (*Object, error)
  • func (s *Storage) UpdateMetadata(ctx context.Context, obj *Object, cb func(m *Metadata) error) error
  • func (s *Storage) Upload(ctx context.Context, name, digest string, r io.Reader) (*Object, error)
• type Timestamp
  • func TimestampFromTime(t time.Time) Timestamp
  • func (ts Timestamp) Time() time.Time

Types

type Metadata

type Metadata struct {
	// contains filtered or unexported fields
}

Metadata is an interpreted metadata dict.

Cloud Storage object metadata is nominally key=>value map, but we need a multi-map. This is accomplished by embedding timestamps inside keys (as "<key>@<microsec timestamp>"). Metadata struct handled such representation.

func ParseMetadata

func ParseMetadata(m map[string]string) *Metadata

ParseMetadata convert key[@timestamp]=>value map into Metadata object.

func (*Metadata) Add

func (m *Metadata) Add(md Metadatum)

Add adds (or overrides) an entry.

func (*Metadata) Assemble

func (m *Metadata) Assemble() map[string]string

Assemble reconstructs back the metadata map.

func (*Metadata) Clone

func (m *Metadata) Clone() *Metadata

Clone returns a deep copy of 'm'.

func (*Metadata) Empty

func (m *Metadata) Empty() bool

Empty is true if 'm' has no entries.

func (*Metadata) Equal

func (m *Metadata) Equal(o *Metadata) bool

Equal returns true if 'm' is equal to 'o'.

func (*Metadata) Keys

func (m *Metadata) Keys() []string

Keys returns a sorted list of keys.

func (*Metadata) ToPretty

func (m *Metadata) ToPretty(now time.Time, limit int) string

ToPretty returns multi-line pretty printed contents of 'm'.

func (*Metadata) TrimUnimportant

func (m *Metadata) TrimUnimportant(keep int)

TrimUnimportant for each individual key removes values until only 'keep' entries remain.

Always keeps the oldest entry and most recent ones, removing only ones in the middle.

Helps to keep metadata small by "forgetting" uninteresting entries.

func (*Metadata) Values

func (m *Metadata) Values(k string) []Metadatum

Values returns all values of some key, most recent first.

Should not be mutated directly. If you need to modify Metadata, use Add. There's no way to delete entries (other than trimming old ones via Trim).

type Metadatum

type Metadatum struct {
	Key       string    // key with the timestamp stripped, if it had any
	Timestamp Timestamp // timestamp extracted from the key or 0 if it had none
	Value     string    // value associated with the key
}

Metadatum is one parsed key-value metadata pair.

type Object

type Object struct {
	Bucket         string    // e.g. "bucket-name"
	Name           string    // e.g. "dir/dir/123456.tar.gz"
	Generation     int64     // e.g. 12312423452345
	Metageneration int64     // e.g. 12312423452345
	Created        time.Time // when it was created
	Owner          string    // FYI who uploaded it
	MD5            string    // FYI for logs only
	Metadata       *Metadata // custom metadata
	// contains filtered or unexported fields
}

Object is a pointer to a concrete version of a file in Google Storage, along with its custom metadata.

func (*Object) Log

func (o *Object) Log(ctx context.Context)

Log pretty-prints fields of Object at info logging level.

func (*Object) String

func (o *Object) String() string

String returns "gs://<bucket>/<name>#<generation>" string.

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage knows how to upload tarballs to a predefined Google Storage prefix.

func New

func New(ctx context.Context, ts oauth2.TokenSource, location string) (*Storage, error)

New returns a Storage that uploads tarballs Google Storage.

'location' should have form "gs://<bucket>[/<path>]".

func (*Storage) Check

func (s *Storage) Check(ctx context.Context, name string) (*Object, error)

Check fetches information about existing Google Storage object.

Returns:

(*Object, nil) if such object already exists.
(nil, nil) if such object doesn't exist.
(nil, error) on errors.

func (*Storage) UpdateMetadata

func (s *Storage) UpdateMetadata(ctx context.Context, obj *Object, cb func(m *Metadata) error) error

UpdateMetadata fetches existing metadata, calls 'cb' to mutate it, and pushes it back if it has changed.

'obj' must have come from Check or Upload. Panics otherwise.

May call 'cb' multiple times if someone is updating the metadata concurrently with us. If 'cb' returns an error, returns exact same error as is.

func (*Storage) Upload

func (s *Storage) Upload(ctx context.Context, name, digest string, r io.Reader) (*Object, error)

Upload uploads the tarball with the given hex SHA256 digest to the storage.

Its body is produced by 'r' and the Storage will double check that the body matches the digest. On mismatch the upload operation is abandoned (i.e. the bucket is left unchanged).

The object is named as "<prefix>/<name>", where <prefix> is taken from <location> passed to New(...) when the Storage was initialized.

Unconditionally overwrites an existing object, if any. Doesn't try to set any ACLs. The bucket must exist already.

Respects context deadlines and cancellation. On success returns a pointer to the uploaded object (including its generation number).

type Timestamp

type Timestamp int64

Timestamp is a unix timestamp in microsecond precision.

func TimestampFromTime

func TimestampFromTime(t time.Time) Timestamp

TimestampFromTime converts `t` to Timestamp (trimming precision accordingly).

func (Timestamp) Time

func (ts Timestamp) Time() time.Time

Time converts the timestamp to local time.Time.