google

README

Google Cloud Storage Stow Implementation

Location = Google Cloud Storage

Container = Bucket

Item = File

How to access underlying service types

Use a type conversion to extract the underlying Location, Container, or Item implementations. Then use the Google-specific getters to access the internal Google Cloud Storage Service, Bucket, and Object values.

import (
  "log"
  "github.com/graymeta/stow"
  stowgs "github.com/graymeta/stow/google"
)

stowLoc, err := stow.Dial(stowgs.Kind, stow.ConfigMap{
	stowgs.ConfigJSON:      "<json config>",
	stowgs.ConfigProjectId: "<project id>",
})
if err != nil {
  log.Fatal(err)
}

stowBucket, err = stowLoc.Container("mybucket")
if err != nil {
  log.Fatal(err)
}

if gsBucket, ok := stowBucket.(*stowgs.Container); ok {
  if gsLoc, ok := stowLoc.(*stowgs.Location); ok {

    googleService := gsLoc.Service()
    googleBucket, err := gsBucket.Bucket()

    // < Send platform-specific commands here >

  }
}

By default, Stow uses https://www.googleapis.com/auth/devstorage.read_write scope. Different scopes can be used by passing a comma separated list of scopes, like below:

stowLoc, err := stow.Dial(stowgs.Kind, stow.ConfigMap{
	stowgs.ConfigJSON:      "<json config>",
	stowgs.ConfigProjectId: "<project id>",
	stowgs.ConfigScopes:    "<scope_1>,<scope_2>",
})

---

Configuration... You need to create a project in google, and then create a service account in google tied to that project. You will need to download a .json file with the configuration for the service account. To run the test suite, the service account will need edit privileges inside the project.

To run the test suite, set the GOOGLE_CREDENTIALS_FILE environment variable to point to the location of the .json file containing the service account credentials and set GOOGLE_PROJECT_ID to the project ID, otherwise the test suite will not be run.

---

Concerns:

• Google's storage plaform is more eventually consistent than other platforms. Sometimes, the tests appear to be flaky because of this. One example is when deleting files from a bucket, then immediately deleting the bucket...sometimes the bucket delete will fail saying that the bucket isn't empty simply because the file delete messages haven't propagated through Google's infrastructure. We may need to add some delay into the test suite to account for this.

Documentation

Overview

Package google provides an abstraction of Google Cloud Storage. In this package, a Google Cloud Storage Bucket is represented by a Stow Container and a Google Cloud Storage Object is represented by a Stow Item. Note that directories may exist within a Bucket.

Usage and Credentials

A path to the JSON file representing configuration information for the service account is needed, as well as the Project ID that it is tied to.

stow.Dial requires both a string value of the particular Stow Location Kind ("google") and a stow.Config instance. The stow.Config instance requires two entries with the specific key value attributes:

- a key of google.ConfigJSON with a value of the path of the JSON configuration file - a key of google.ConfigProjectID with a value of the Project ID

Location

There are google.location methods which allow the retrieval of a Google Cloud Storage Object (Container or Containers). An Object can also be retrieved based on the its URL (ItemByURL).

Additional google.location methods provide capabilities to create and remove Google Cloud Storage Buckets (CreateContainer or RemoveContainer).

Container

Methods of stow.container allow the retrieval of a Google Bucket's:

- name(ID or Name) - object or complete list of objects (Item or Items, respectively)

Additional methods of google.container allow Stow to:

- remove an Object (RemoveItem) - update or create an Object (Put)

Item

Methods of google.Item allow the retrieval of a Google Cloud Storage Object's: - name (ID or name) - URL - size in bytes - Object specific metadata (information stored within the Google Cloud Service) - last modified date - Etag - content

Index

• Constants
• type Container
  • func (c *Container) Bucket() *storage.BucketHandle
  • func (c *Container) ID() string
  • func (c *Container) Item(id string) (stow.Item, error)
  • func (c *Container) Items(prefix string, cursor string, count int) ([]stow.Item, string, error)
  • func (c *Container) Name() string
  • func (c *Container) Put(name string, r io.Reader, size int64, metadata map[string]interface{}) (stow.Item, error)
  • func (c *Container) RemoveItem(id string) error
• type Item
  • func (i *Item) ETag() (string, error)
  • func (i *Item) ID() string
  • func (i *Item) LastMod() (time.Time, error)
  • func (i *Item) Metadata() (map[string]interface{}, error)
  • func (i *Item) Name() string
  • func (i *Item) Open() (io.ReadCloser, error)
  • func (i *Item) OpenRange(start, end uint64) (io.ReadCloser, error)
  • func (i *Item) Size() (int64, error)
  • func (i *Item) StorageObject() *storage.ObjectAttrs
  • func (i *Item) URL() *url.URL
• type Location
  • func (l *Location) Close() error
  • func (l *Location) Container(id string) (stow.Container, error)
  • func (l *Location) Containers(prefix string, cursor string, count int) ([]stow.Container, string, error)
  • func (l *Location) CreateContainer(containerName string) (stow.Container, error)
  • func (l *Location) ItemByURL(url *url.URL) (stow.Item, error)
  • func (l *Location) RemoveContainer(id string) error
  • func (l *Location) Service() *storage.Client

Constants

const (
	// The service account json blob
	ConfigJSON      = "json"
	ConfigProjectId = "project_id"
	ConfigScopes    = "scopes"
)

const Kind = "google"

Kind represents the name of the location/storage type.

Types

type Container

type Container struct {
	// contains filtered or unexported fields
}

func (*Container) Bucket

func (c *Container) Bucket() *storage.BucketHandle

Bucket returns the google bucket attributes

func (*Container) ID

func (c *Container) ID() string

ID returns a string value which represents the name of the container.

func (*Container) Item

func (c *Container) Item(id string) (stow.Item, error)

Item returns a stow.Item instance of a container based on the name of the container

func (*Container) Items

func (c *Container) Items(prefix string, cursor string, count int) ([]stow.Item, string, error)

Items retrieves a list of items that are prepended with the prefix argument. The 'cursor' variable facilitates pagination.

func (*Container) Name

func (c *Container) Name() string

Name returns a string value which represents the name of the container.

func (*Container) Put

func (c *Container) Put(name string, r io.Reader, size int64, metadata map[string]interface{}) (stow.Item, error)

Put sends a request to upload content to the container. The arguments received are the name of the item, a reader representing the content, and the size of the file.

func (*Container) RemoveItem

func (c *Container) RemoveItem(id string) error

RemoveItem will delete a google storage Object

type Item

type Item struct {
	// contains filtered or unexported fields
}

func (*Item) ETag

func (i *Item) ETag() (string, error)

ETag returns the ETag value

func (*Item) ID

func (i *Item) ID() string

ID returns a string value that represents the name of a file.

func (*Item) LastMod

func (i *Item) LastMod() (time.Time, error)

LastMod returns the last modified date of the item.

func (*Item) Metadata

func (i *Item) Metadata() (map[string]interface{}, error)

Metadata returns a nil map and no error.

func (*Item) Name

func (i *Item) Name() string

Name returns a string value that represents the name of the file.

func (*Item) Open

func (i *Item) Open() (io.ReadCloser, error)

Open returns an io.ReadCloser to the object. Useful for downloading/streaming the object.

func (*Item) OpenRange

func (i *Item) OpenRange(start, end uint64) (io.ReadCloser, error)

OpenRange returns an io.Reader to the object for a specific byte range

func (*Item) Size

func (i *Item) Size() (int64, error)

Size returns the size of an item in bytes.

func (*Item) StorageObject

func (i *Item) StorageObject() *storage.ObjectAttrs

Object returns the Google Storage Object

func (*Item) URL

func (i *Item) URL() *url.URL

URL returns a url which follows the predefined format

type Location

type Location struct {
	// contains filtered or unexported fields
}

A Location contains a client + the configurations used to create the client.

func (*Location) Close

func (l *Location) Close() error

Close simply satisfies the Location interface. There's nothing that needs to be done in order to satisfy the interface.

func (*Location) Container

func (l *Location) Container(id string) (stow.Container, error)

Container retrieves a stow.Container based on its name which must be exact.

func (*Location) Containers

func (l *Location) Containers(prefix string, cursor string, count int) ([]stow.Container, string, error)

Containers returns a slice of the Container interface, a cursor, and an error.

func (*Location) CreateContainer

func (l *Location) CreateContainer(containerName string) (stow.Container, error)

CreateContainer creates a new container, in this case a bucket.

func (*Location) ItemByURL

func (l *Location) ItemByURL(url *url.URL) (stow.Item, error)

ItemByURL retrieves a stow.Item by parsing the URL, in this case an item is an object.

func (*Location) RemoveContainer

func (l *Location) RemoveContainer(id string) error

RemoveContainer removes a container simply by name.

func (*Location) Service

func (l *Location) Service() *storage.Client