stow

README

Cloud storage abstraction package for Go.

Documentation

Overview

Package stow provides an abstraction on cloud storage capabilities.

Index

• Variables
• func IsCursorEnd(cursor string) bool
• func IsNotSupported(err error) bool
• func KindByURL(u *url.URL) (string, error)
• func Kinds() []string
• func NotSupported(feature string) error
• func Register(kind string, makefn func(Config) (Location, error), ...)
• func Validate(kind string, config Config) error
• func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error
• func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error
• type ClientMethod
  • func ClientMethodString(s string) (ClientMethod, error)
  • func ClientMethodValues() []ClientMethod
  • func (i ClientMethod) IsAClientMethod() bool
  • func (i ClientMethod) MarshalJSON() ([]byte, error)
  • func (i ClientMethod) String() string
  • func (i *ClientMethod) UnmarshalJSON(data []byte) error
• type Config
• type ConfigMap
  • func (c ConfigMap) Config(name string) (string, bool)
  • func (c ConfigMap) Set(name, value string)
• type Container
• type HttpMethod
• type Item
• type ItemRanger
• type Location
  • func Dial(kind string, config Config) (Location, error)
• type PresignRequestParams
• type Taggable
• type WalkContainersFunc
• type WalkFunc

Variables

var (
	// ErrNotFound is returned when something could not be found.
	ErrNotFound = errors.New("not found")
	// ErrBadCursor is returned by paging methods when the specified
	// cursor is invalid.
	ErrBadCursor = errors.New("bad cursor")
)

var (
	// CursorStart is a string representing a cursor pointing
	// to the first page of items or containers.
	CursorStart = ""

	// NoPrefix is a string representing no prefix. It can be used
	// in any function that asks for a prefix value, but where one is
	// not appropriate.
	NoPrefix = ""
)

Functions

func IsCursorEnd

func IsCursorEnd(cursor string) bool

IsCursorEnd checks whether the cursor indicates there are no more items or not.

func IsNotSupported

func IsNotSupported(err error) bool

IsNotSupported gets whether the error is due to a feature not being supported by a specific implementation.

func KindByURL

func KindByURL(u *url.URL) (string, error)

KindByURL gets the kind represented by the given URL. It consults all registered locations. Error returned if no match is found.

func Kinds

func Kinds() []string

Kinds gets a list of installed location kinds.

func NotSupported

func NotSupported(feature string) error

NotSupported gets an error describing the feature as not supported by this implementation.

func Register

func Register(kind string, makefn func(Config) (Location, error), kindmatchfn func(*url.URL) bool, validatefn func(Config) error)

Register adds a Location implementation, with two helper functions. makefn should make a Location with the given Config. kindmatchfn should inspect a URL and return whether it represents a Location of this kind or not. Code can call KindByURL to get a kind string for any given URL and all registered implementations will be consulted. Register is usually called in an implementation package's init method.

func Validate

func Validate(kind string, config Config) error

Validate validates the config for a location

func Walk

func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error

Walk walks all Items in the Container. Returns the first error returned by the WalkFunc or nil if no errors were returned. The pageSize is the number of Items to get per request.

func WalkContainers

func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error

WalkContainers walks all Containers in the Location. Returns the first error returned by the WalkContainersFunc or nil if no errors were returned. The pageSize is the number of Containers to get per request.

Types

type ClientMethod

type ClientMethod int

ClientMethod defines common client methods across storage providers

const (
	ClientMethodGet ClientMethod = iota
	ClientMethodPut
)

func ClientMethodString

func ClientMethodString(s string) (ClientMethod, error)

ClientMethodString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func ClientMethodValues

func ClientMethodValues() []ClientMethod

ClientMethodValues returns all values of the enum

func (ClientMethod) IsAClientMethod

func (i ClientMethod) IsAClientMethod() bool

IsAClientMethod returns "true" if the value is listed in the enum definition. "false" otherwise

func (ClientMethod) MarshalJSON

func (i ClientMethod) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for ClientMethod

func (ClientMethod) String

func (i ClientMethod) String() string

func (*ClientMethod) UnmarshalJSON

func (i *ClientMethod) UnmarshalJSON(data []byte) error

UnmarshalJSON implements the json.Unmarshaler interface for ClientMethod

type Config

type Config interface {
	// Config gets a string configuration value and a
	// bool indicating whether the value was present or not.
	Config(name string) (string, bool)
	// Set sets the configuration name to specified value
	Set(name, value string)
}

Config represents key/value configuration.

type ConfigMap

type ConfigMap map[string]string

ConfigMap is a map[string]string that implements the Config method.

func (ConfigMap) Config

func (c ConfigMap) Config(name string) (string, bool)

Config gets a string configuration value and a bool indicating whether the value was present or not.

func (ConfigMap) Set

func (c ConfigMap) Set(name, value string)

Set sets name configuration to value

type Container

type Container interface {
	// ID gets a unique string describing this Container.
	ID() string
	// Name gets a human-readable name describing this Container.
	Name() string
	// Item gets an item by its ID.
	Item(id string) (Item, error)
	// Items gets a page of items with the specified
	// prefix for this Container.
	// The specified cursor is a pointer to the start of
	// the items to get. It should be obtained from a previous
	// call to this method, or should be CursorStart for the
	// first page.
	// count is the number of items to return per page.
	// The returned cursor can be checked with IsCursorEnd to
	// decide if there are any more items or not.
	Items(prefix, cursor string, count int) ([]Item, string, error)
	// RemoveItem removes the Item with the specified ID.
	RemoveItem(id string) error
	// Put creates a new Item with the specified name, and contents
	// read from the reader.
	Put(name string, r io.Reader, size int64, metadata map[string]interface{}) (Item, error)
	// PreSignRequest generates a pre-signed url for the given id (key after bucket/container) and a given clientMethod.
	PreSignRequest(ctx context.Context, clientMethod ClientMethod, id string, params PresignRequestParams) (url string, err error)
}

Container represents a container.

type HttpMethod

type HttpMethod = string

HttpMethod defines an alias type for string to represent http Methods. These are defined in RFC 7231 section 4.3.

type Item

type Item interface {
	// ID gets a unique string describing this Item.
	ID() string
	// Name gets a human-readable name describing this Item.
	Name() string
	// URL gets a URL for this item.
	// For example:
	// local: file:///path/to/something
	// azure: azure://host:port/api/something
	//    s3: s3://host:post/etc
	URL() *url.URL
	// Size gets the size of the Item's contents in bytes.
	Size() (int64, error)
	// Open opens the Item for reading.
	// Calling code must close the io.ReadCloser.
	Open() (io.ReadCloser, error)
	// ETag is a string that is different when the Item is
	// different, and the same when the item is the same.
	// Usually this is the last modified datetime.
	ETag() (string, error)
	// LastMod returns the last modified date of the file.
	LastMod() (time.Time, error)
	// Metadata gets a map of key/values that belong
	// to this Item.
	Metadata() (map[string]interface{}, error)
}

Item represents an item inside a Container. Such as a file.

type ItemRanger

type ItemRanger interface {
	// OpenRange opens the item for reading starting at byte start and ending
	// at byte end.
	OpenRange(start, end uint64) (io.ReadCloser, error)
}

ItemRanger represents an item that can be partially downloaded.

type Location

type Location interface {
	io.Closer
	// CreateContainer creates a new Container with the
	// specified name.
	CreateContainer(name string) (Container, error)
	// Containers gets a page of containers
	// with the specified prefix from this Location.
	// The specified cursor is a pointer to the start of
	// the containers to get. It it obtained from a previous
	// call to this method, or should be CursorStart for the
	// first page.
	// count is the number of items to return per page.
	// The returned cursor can be checked with IsCursorEnd to
	// decide if there are any more items or not.
	Containers(prefix string, cursor string, count int) ([]Container, string, error)
	// Container gets the Container with the specified
	// identifier.
	Container(id string) (Container, error)
	// RemoveContainer removes the container with the specified ID.
	RemoveContainer(id string) error
	// ItemByURL gets an Item at this location with the
	// specified URL.
	ItemByURL(url *url.URL) (Item, error)
}

Location represents a storage location.

func Dial

func Dial(kind string, config Config) (Location, error)

Dial gets a new Location with the given kind and configuration.

type PresignRequestParams

type PresignRequestParams struct {
	ExpiresIn   time.Duration
	ContentMD5  string
	ExtraParams map[string]interface{}
	HttpMethod  HttpMethod
}

type Taggable

type Taggable interface {
	// Tags returns a list of tags that belong to a given Item
	Tags() (map[string]interface{}, error)
}

Taggable represents a taggable Item

type WalkContainersFunc

type WalkContainersFunc func(container Container, err error) error

WalkContainersFunc is a function called for each Container visited by WalkContainers. If there was a problem, the incoming error will describe the problem and the function can decide how to handle that error. If an error is returned, processing stops.

type WalkFunc

type WalkFunc func(item Item, err error) error

WalkFunc is a function called for each Item visited by Walk. If there was a problem, the incoming error will describe the problem and the function can decide how to handle that error. If an error is returned, processing stops.