storage

package module
v0.0.0-...-58767fd Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 13, 2020 License: MIT Imports: 19 Imported by: 0

README

Storage

Build Status GoDoc

storage is a Go package which abstracts file systems (local, in-memory, Google Cloud Storage, S3) into a few interfaces. It includes convenience wrappers for simplifying common file system use cases such as caching, prefix isolation and more!

Requirements

Installation

$ go get code.sajari.com/storage

Usage

For full documentation see: http://godoc.org/code.sajari.com/storage/.

All storage in this package follow two simple interfaces designed for using file systems.

type FS interface {
	Walker

	// Open opens an existing file at path in the filesystem.  Callers must close the
	// File when done to release all underlying resources.
	Open(ctx context.Context, path string) (*File, error)

	// Create makes a new file in the filesystem.  Callers must close the
	// returned WriteCloser and check the error to be sure that the file
	// was successfully written.
	Create(ctx context.Context, path string) (io.WriteCloser, error)

	// Delete removes a file from the filesystem.
	Delete(ctx context.Context, path string) error
}

// WalkFn is a function type which is passed to Walk.
type WalkFn func(path string) error

// Walker is an interface which defines the Walk method.
type Walker interface {
	// Walk traverses a path listing by prefix, calling fn with each object path rewritten
	// to be relative to the underlying filesystem and provided path.
	Walk(ctx context.Context, path string, fn WalkFn) error
}

Local

Local is the default implementation of a local file system (i.e. using os.Open etc).

local := storage.Local("/some/root/path")
f, err := local.Open(context.Background(), "file.json") // will open "/some/root/path/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

Memory

Mem is the default in-memory implementation of a file system.

mem := storage.Mem()
wc, err := mem.Create(context.Background(), "file.txt")
if err != nil {
	// ...
}
if _, err := io.WriteString(wc, "Hello World!"); err != nil {
	// ...
}
if err := wc.Close(); err != nil {
	// ...
}

And now:

f, err := mem.Open(context.Background(), "file.txt")
if err != nil {
	// ...
}
// ...
f.Close()

Google Cloud Storage

CloudStorage is the default implementation of Google Cloud Storage. This uses https://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource for autentication.

store := storage.CloudStorage{Bucket:"some-bucket"}
f, err := store.Open(context.Background(), "file.json") // will fetch "gs://some-bucket/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

S3

Not yet implemented! Watch this space.

Wrappers and Helpers

Simple Caching

To use Cloud Storage as a source file system, but cache all opened files in a local filesystem:

src := storage.CloudStorage{Bucket:"some-bucket"}
local := storage.Local("/scratch-space")

fs := storage.Cache(src, local)
f, err := fs.Open(context.Background(), "file.json") // will try src then jump to cache ("gs://some-bucket/file.json")
if err != nil {
	// ...
}
// ...
f.Close()

f, err := fs.Open(context.Background(), "file.json") // should now be cached ("/scratch-space/file.json")
if err != nil {
	// ...
}
// ...
f.Close()

This is particularly useful when distributing files across multiple regions or between cloud providers. For instance, we could add the following code to the previous example:

mainSrc := storage.CloudStorage{Bucket:"some-bucket-in-another-region"}
fs2 := storage.Cache(mainSrc, fs) // fs is from previous snippet

// Open will:
// 1. Try local (see above)
// 2. Try gs://some-bucket
// 3. Try gs://some-bucket-in-another-region, which will be cached in gs://some-bucket and then local on its
//    way back to the caller.
f, err := fs2.Open(context.Background(), "file.json") // will fetch "gs://some-bucket-in-another-region/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

f, err := fs2.Open(context.Background(), "file.json") // will fetch "/scratch-space/file.json"
if err != nil {
	// ...
}
// ...
f.Close()
Adding prefixes to paths

If you're writing code that relies on a set directory structure, it can be very messy to have to pass path-patterns around. You can avoid this by wrapping storage.FS implementations with storage.Prefix that rewrites all incoming paths.

modelFS := storage.Prefix(rootFS, "models/")
f, err := modelFS.Open(context.Background(), "file.json") // will call rootFS.Open with path "models/file.json"
if err != nil {
	// ...
}
// ...
f.Close()

It's also now simple to write wrapper functions to abstract out more complex directory structures.

func UserFS(fs storage.FS, userID, mediaType string) FS {
	return storage.Prefix(fs, fmt.Sprintf("%v/%v", userID, userType))
}

userFS := UserFS(rootFS, "1111", "pics")
f, err := userFS.Open(context.Background(), "beach.png") // will call rootFS.Open with path "1111/pics/beach.png"
if err != nil {
	// ...
}
// ...
f.Close()

Documentation

Overview

Package storage provides types and functionality for abstracting storage systems (Local, in memory, S3, Google Cloud storage) into a common interface.

Index

Constants

View Source 
const DefaultLocalCreatePathMode = os.FileMode(0755)

DefaultLocalCreatePathMode is the default os.FileMode used when creating directories during a Local.Create call.

Variables

View Source 
var LocalCreatePathMode = DefaultLocalCreatePathMode

LocalCreatePathMode is the os.FileMode used when creating directories via Local.Create

Functions

func IsNotExist 

func IsNotExist(err error) bool

IsNotExist returns a boolean indicating whether the error is known to report that a path does not exist.

func List 

func List(ctx context.Context, w Walker, path string) ([]string, error)

List runs the Walker on the given path and returns the list of visited paths.

func WalkN 

func WalkN(ctx context.Context, w Walker, path string, n int, fn WalkFn) error

WalkN creates n workers which accept paths from the Walker. If a WalkFn returns non-nil error we wait for other running WalkFns to finish before returning.

Types

type CloudStorage 

type CloudStorage struct {
	Bucket string
}

CloudStorage implements FS and uses Google Cloud Storage as the underlying file storage.

func (*CloudStorage) Create 

func (c *CloudStorage) Create(ctx context.Context, path string) (io.WriteCloser, error)

Create implements FS.

func (*CloudStorage) Delete 

func (c *CloudStorage) Delete(ctx context.Context, path string) error

Delete implements FS.

func (*CloudStorage) Open 

func (c *CloudStorage) Open(ctx context.Context, path string) (*File, error)

Open implements FS.

func (*CloudStorage) Walk 

func (c *CloudStorage) Walk(ctx context.Context, path string, fn WalkFn) error

Walk implements FS.

type ErrCountFS 

type ErrCountFS struct {
	// contains filtered or unexported fields
}

ErrCountFS is an FS which records error counts for an FS.

func NewErrCountFS 

func NewErrCountFS(fs FS, name string, err error) *ErrCountFS

NewErrCountFS creates an FS which records stats based on usage.

func (ErrCountFS) Create 

func (s ErrCountFS) Create(ctx context.Context, path string) (io.WriteCloser, error)

Create implements FS. All errors from Create are counted.

func (ErrCountFS) Delete 

func (s ErrCountFS) Delete(ctx context.Context, path string) error

Delete implements FS. All errors from Delete are counted.

func (ErrCountFS) Open 

func (s ErrCountFS) Open(ctx context.Context, path string) (*File, error)

Open implements FS. All errors from Open are counted.

func (ErrCountFS) Walk 

func (s ErrCountFS) Walk(ctx context.Context, path string, fn WalkFn) error

Walk implements FS. No stats are recorded at this time.

type FS 

type FS interface {
	Walker

	// Open opens an existing file at path in the filesystem.  Callers must close the
	// File when done to release all underlying resources.
	Open(ctx context.Context, path string) (*File, error)

	// Create makes a new file at path in the filesystem.  Callers must close the
	// returned WriteCloser and check the error to be sure that the file
	// was successfully written.
	Create(ctx context.Context, path string) (io.WriteCloser, error)

	// Delete removes a path from the filesystem.
	Delete(ctx context.Context, path string) error
}

FS is an interface which defines a virtual filesystem.

func Cache 

func Cache(src, cache FS) FS

Cache creates an FS implementation which caches files opened from src into cache.

func FSFromURL 

func FSFromURL(path string) FS

FSFromURL takes a file system path and returns a FSWalker corresponding to a supported storage system (CloudStorage, S3, or Local if no platform-specific prefix is used).

func HashFS 

func HashFS(h hash.Hash, fs FS, gs GetSetter) FS

HashFS creates a content addressable filesystem using hash.Hash to sum the content and store it using that name.

func Mem 

func Mem() FS

Mem creates a a basic in-memory implementation of FS.

func Prefix 

func Prefix(fs FS, prefix string) FS

Prefix creates a FS which wraps fs and prefixes all paths with prefix.

type File 

type File struct {
	io.ReadCloser           // Underlying data.
	Name          string    // Name of the file (likely basename).
	ModTime       time.Time // Modified time of the file.
	Size          int64     // Size of the file.
}

File contains the metadata required to define a file (for reading).

type GetSetter 

type GetSetter interface {
	Get(key string) (string, error)
	Set(key string, value string) error
	Delete(key string) error
}

GetSetter implements a key-value store which is concurrency safe (can be used in multiple go-routines concurrently).

type Local 

type Local string

Local is a local FS and Walker implementation.

func (Local) Create 

func (l Local) Create(_ context.Context, path string) (io.WriteCloser, error)

Create implements FS. If the path contains any directories which do not already exist then Create will try to make them, returning an error if it fails.

func (Local) Delete 

func (l Local) Delete(_ context.Context, path string) error

Delete implements FS. All files underneath path will be removed.

func (Local) Open 

func (l Local) Open(_ context.Context, path string) (*File, error)

Open implements FS.

func (Local) Walk 

func (l Local) Walk(_ context.Context, path string, fn WalkFn) error

Walk implements Walker.

type LogFS 

type LogFS struct {
	// contains filtered or unexported fields
}

LogFS is an FS implementation which logs all filesystem calls.

func NewLogFS 

func NewLogFS(fs FS, name string, l *log.Logger) *LogFS

NewLogFS creates a new FS which logs all calls to FS.

func (*LogFS) Create 

func (l *LogFS) Create(ctx context.Context, path string) (io.WriteCloser, error)

Create implements FS. All calls to Create are logged and errors are logged seperately.

func (*LogFS) Delete 

func (l *LogFS) Delete(ctx context.Context, path string) error

Delete implements FS. All calls to Delete are logged and errors are logged seperately.

func (*LogFS) Open 

func (l *LogFS) Open(ctx context.Context, path string) (*File, error)

Open implements FS. All calls to Open are logged and errors are logged seperately.

func (*LogFS) Walk 

func (l *LogFS) Walk(ctx context.Context, path string, fn WalkFn) error

Walk implements FS. No logs are written at this time.

type S3 

type S3 struct {
	Bucket string // Bucket is the name of the bucket to use as the underlying storage.
}

S3 is an implementation of FS which uses AWS S3 as the underlying storage layer.

func (*S3) Create 

func (s *S3) Create(ctx context.Context, path string) (io.WriteCloser, error)

Create implements FS.

func (*S3) Delete 

func (s *S3) Delete(ctx context.Context, path string) error

Delete implements FS.

func (*S3) Open 

func (s *S3) Open(ctx context.Context, path string) (*File, error)

Open implements FS.

func (*S3) Walk 

func (s *S3) Walk(ctx context.Context, path string, fn WalkFn) error

Walk implements FS.

type TraceFS 

type TraceFS struct {
	// contains filtered or unexported fields
}

TraceFS is a FS implementation which wraps an FS and records calls using golang.org/x/net/trace.

func NewTraceFS 

func NewTraceFS(fs FS, name string) *TraceFS

NewTraceFS creates a new FS which wraps an FS and records calls using golang.org/x/net/trace.

func (*TraceFS) Create 

func (t *TraceFS) Create(ctx context.Context, path string) (wc io.WriteCloser, err error)

Create implements FS. All calls to Create are logged via golang.org/x/net/trace.

func (*TraceFS) Delete 

func (t *TraceFS) Delete(ctx context.Context, path string) (err error)

Delete implements FS. All calls to Delete are logged via golang.org/x/net/trace.

func (*TraceFS) Open 

func (t *TraceFS) Open(ctx context.Context, path string) (f *File, err error)

Open implements FS. All calls to Open are logged via golang.org/x/net/trace.

func (*TraceFS) Walk 

func (t *TraceFS) Walk(ctx context.Context, path string, fn WalkFn) error

Walk implements FS. Nothing is traced at this time.

type WalkFn 

type WalkFn func(path string) error

WalkFn is a function type which is passed to Walk.

type Walker 

type Walker interface {
	// Walk traverses a path listing by prefix, calling fn with each path.
	Walk(ctx context.Context, path string, fn WalkFn) error
}

Walker is an interface which defines the Walk method.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.