storage

Documentation

Overview

Package storage provides types and functionality for abstracting storage systems (local, in memory, S3, Google Cloud storage) into a common interface.

Index

• Constants
• Variables
• func Exists(ctx context.Context, fs FS, path string) bool
• func IsNotExist(err error) bool
• func List(ctx context.Context, w Walker, path string) ([]string, error)
• func NewMockFS() *mockFS
• func Read(ctx context.Context, fs FS, path string, options *ReaderOptions) ([]byte, error)
• func WalkN(ctx context.Context, w Walker, path string, n int, fn WalkFn) error
• func Write(ctx context.Context, fs FS, path string, data []byte, options *WriterOptions) error
• type Attributes
• type CacheOptions
• type FS
  • func FromURL(path string) FS
  • func NewCacheWrapper(src, cache FS, options *CacheOptions) FS
  • func NewCloudStorageFS(bucket string, credentials *google.Credentials) FS
  • func NewHashWrapper(h hash.Hash, fs FS, gs GetSetter) FS
  • func NewLocalFS(path string) FS
  • func NewLoggerWrapper(fs FS, name string, l Logger) FS
  • func NewMemoryFS() FS
  • func NewPrefixWrapper(fs FS, prefix string) FS
  • func NewS3FS(bucket string, cfgs ...*aws.Config) FS
  • func NewSlowWrapper(fs FS, readDelay time.Duration, writeDelay time.Duration) FS
  • func NewStatsWrapper(fs FS, name string) FS
  • func NewTimeoutWrapper(fs FS, read time.Duration, write time.Duration) FS
  • func NewTraceWrapper(fs FS, name string) FS
• type File
• type GetSetter
• type Logger
• type ReaderOptions
• type SignedURLOptions
• type WalkFn
• type Walker
• type WriterOptions

Constants

const (
	StatOpenTotal    = "open.total"
	StatOpenErrors   = "open.errors"
	StatAttrsTotal   = "attrs.total"
	StatAttrsErrors  = "attrs.errors"
	StatCreateTotal  = "create.total"
	StatCreateErrors = "create.errors"
	StatDeleteTotal  = "delete.total"
	StatDeleteErrors = "delete.errors"
	StatURLTotal     = "url.total"
	StatURLErrors    = "url.errors"
)

const DefaultLocalCreatePathMode = os.FileMode(0755)

DefaultLocalCreatePathMode is the default os.FileMode used when creating directories during a localFS.Create call.

Variables

var ErrCredentialsMissing = errors.New("credentials missing")

var ErrNotImplemented = errors.New("not implemented")

var LocalCreatePathMode = DefaultLocalCreatePathMode

LocalCreatePathMode is the os.FileMode used when creating directories via localFS.Create

Functions

func Exists

func Exists(ctx context.Context, fs FS, path string) bool

func IsNotExist

func IsNotExist(err error) bool

IsNotExist returns a boolean indicating whether the error is known to report that a path does not exist.

func List

func List(ctx context.Context, w Walker, path string) ([]string, error)

List runs the Walker on the given path and returns the list of visited paths.

func NewMockFS

func NewMockFS() *mockFS

NewMockFS creates an FS where each method can be mocked. To be used in tests.

func Read

func Read(ctx context.Context, fs FS, path string, options *ReaderOptions) ([]byte, error)

func WalkN

func WalkN(ctx context.Context, w Walker, path string, n int, fn WalkFn) error

WalkN creates n workers which accept paths from the Walker. If a WalkFn returns non-nil error we wait for other running WalkFns to finish before returning.

func Write

func Write(ctx context.Context, fs FS, path string, data []byte, options *WriterOptions) error

Types

type Attributes

type Attributes struct {
	// ContentType is the MIME type of the blob object. It will not be empty.
	ContentType string
	// Metadata holds key/value pairs associated with the blob.
	// Keys are guaranteed to be in lowercase, even if the backend provider
	// has case-sensitive keys (although note that Metadata written via
	// this package will always be lowercased). If there are duplicate
	// case-insensitive keys (e.g., "foo" and "FOO"), only one value
	// will be kept, and it is undefined which one.
	Metadata map[string]string
	// ModTime is the time the blob object was last modified.
	ModTime time.Time
	// CreationTime is the time the blob object was created.
	CreationTime time.Time
	// Size is the size of the object in bytes.
	Size int64
}

Attributes represents the metadata of a File Inspired from gocloud.dev/blob.Attributes

type CacheOptions

type CacheOptions struct {
	// MaxAge is the maximum time allowed since the underlying File's ModTime
	// This means that if the cache is older than MaxAge, the Cache will fetch from the src again.
	// If the expired File is still present on the src (i.e. not updated), it will be ignored.
	MaxAge time.Duration

	// DefaultExpired makes the cache treat a File as expired if its CreationTime cannot be checked.
	// By default, it is false, which means the cache will treat zero-CreationTime files as valid.
	// Only useful if MaxAge is set.
	DefaultExpired bool

	// NoData disables caching of the contents of the entries, it only stores the metadata.
	NoData bool
}

type FS

type FS interface {
	Walker

	// Open opens an existing file at path in the filesystem.  Callers must close the
	// File when done to release all underlying resources.
	Open(ctx context.Context, path string, options *ReaderOptions) (*File, error)

	// Attributes returns attributes about a path
	Attributes(ctx context.Context, path string, options *ReaderOptions) (*Attributes, error)

	// Create makes a new file at path in the filesystem.  Callers must close the
	// returned WriteCloser and check the error to be sure that the file
	// was successfully written.
	Create(ctx context.Context, path string, options *WriterOptions) (io.WriteCloser, error)

	// Delete removes a path from the filesystem.
	Delete(ctx context.Context, path string) error

	// URL resolves a path to an addressable URL
	URL(ctx context.Context, path string, options *SignedURLOptions) (string, error)
}

FS is an interface which defines a virtual filesystem.

func FromURL

func FromURL(path string) FS

FromURL takes a file system path and returns a FSWalker corresponding to a supported storage system (cloudStorageFS, s3FS, or localFS if no platform-specific prefix is used).

func NewCacheWrapper

func NewCacheWrapper(src, cache FS, options *CacheOptions) FS

NewCacheWrapper creates an FS implementation which caches files opened from src into cache.

func NewCloudStorageFS

func NewCloudStorageFS(bucket string, credentials *google.Credentials) FS

NewCloudStorageFS creates a Google Cloud Storage FS credentials can be nil to use the default GOOGLE_APPLICATION_CREDENTIALS

func NewHashWrapper

func NewHashWrapper(h hash.Hash, fs FS, gs GetSetter) FS

NewHashWrapper creates a content addressable filesystem using hash.Hash to sum the content and store it using that name.

func NewLocalFS

func NewLocalFS(path string) FS

func NewLoggerWrapper

func NewLoggerWrapper(fs FS, name string, l Logger) FS

NewLoggerWrapper creates a new FS which logs all calls to FS.

func NewMemoryFS

func NewMemoryFS() FS

NewMemoryFS creates a a basic in-memory implementation of FS.

func NewPrefixWrapper

func NewPrefixWrapper(fs FS, prefix string) FS

NewPrefixWrapper creates a FS which wraps fs and prefixes all paths with prefix.

func NewS3FS

func NewS3FS(bucket string, cfgs ...*aws.Config) FS

func NewSlowWrapper

func NewSlowWrapper(fs FS, readDelay time.Duration, writeDelay time.Duration) FS

NewSlowWrapper creates an artificially slow FS. Probably only useful for testing.

func NewStatsWrapper

func NewStatsWrapper(fs FS, name string) FS

NewStatsWrapper creates an FS which records accesses for an FS. To retrieve the stats: stats := expvar.Get(name).(*expvar.Map) total := stats.Get("open.total").(*expvar.Int).Value() // int64

func NewTimeoutWrapper

func NewTimeoutWrapper(fs FS, read time.Duration, write time.Duration) FS

NewTimeoutWrapper creates a FS which wraps fs and adds a timeout to most operations: read: Open, Attributes, URL write: Create, Delete

Note that the Open and Create methods are only for resolving the object, NOT actually reading or writing the contents. These operations should be fairly quick, on the same order as Attribute and Delete, respectively.

This depends on the underlying implementation to honour context's errors. It is at least supported on the CloudStorageFS.

Walk is not covered, since its duration is highly unpredictable.

func NewTraceWrapper

func NewTraceWrapper(fs FS, name string) FS

NewTraceWrapper creates a new FS which wraps an FS and records calls using golang.org/x/net/trace.

type File

type File struct {
	io.ReadCloser // Underlying data.
	Attributes
}

File contains the metadata required to define a file (for reading).

type GetSetter

type GetSetter interface {
	Get(key string) (string, error)
	Set(key string, value string) error
	Delete(key string) error
}

GetSetter implements a key-value store which is concurrency safe (can be used in multiple go-routines concurrently).

type Logger

type Logger interface {
	Print(v ...interface{})
}

Logger can be a *log.Logger

type ReaderOptions

type ReaderOptions struct{}

ReaderOptions are used to modify the behaviour of read operations. Inspired from gocloud.dev/blob.ReaderOptions It is provided for future extensibility.

type SignedURLOptions

type SignedURLOptions = blob.SignedURLOptions

type WalkFn

type WalkFn func(path string) error

WalkFn is a function type which is passed to Walk.

type Walker

type Walker interface {
	// Walk traverses a path listing by prefix, calling fn with each path.
	Walk(ctx context.Context, path string, fn WalkFn) error
}

Walker is an interface which defines the Walk method.

type WriterOptions

type WriterOptions struct {
	Attributes Attributes

	// BufferSize changes the default size in bytes of the chunks that
	// Writer will upload in a single request; larger blobs will be split into
	// multiple requests.
	//
	// This option may be ignored by some drivers.
	//
	// If 0, the driver will choose a reasonable default.
	//
	// If the Writer is used to do many small writes concurrently, using a
	// smaller BufferSize may reduce memory usage.
	BufferSize int
}

WriterOptions are used to modify the behaviour of write operations. Inspired from gocloud.dev/blob.WriterOptions Not all options are supported by all FS