storage

Documentation

Overview

Package storage implements a simple storage abstraction.

This is meant to abstract filesystem calls, as well as be a wrapper for in-memory or remote storage. It also provides a smaller attack vector as implementations can do verifications as to what is accessed and what is not.

Index

• Variables
• func Copy(ctx context.Context, from ReadBucket, to ReadWriteBucket, ...) (int, error)
• func CopyPaths(ctx context.Context, from ReadBucket, to ReadWriteBucket, paths []string, ...) (int, error)
• func CopyPrefix(ctx context.Context, from ReadBucket, to ReadWriteBucket, prefix string, ...) (int, error)
• func Exists(ctx context.Context, readBucket ReadBucket, path string) (bool, error)
• func IsNotExist(err error) bool
• func NewErrNotExist(path string) error
• func ReadPath(ctx context.Context, readBucket ReadBucket, path string) (_ []byte, retErr error)
• type BucketType
• type ObjectInfo
• type ReadBucket
• type ReadBucketCloser
• type ReadObject
• type ReadWriteBucket
• type ReadWriteBucketCloser
• type WriteObject

Variables

var (
	// ErrIncompleteWrite is the error returned if a write is not complete
	ErrIncompleteWrite = errors.New("incomplete write")
	// ErrClosed is the error returned if a bucket or object is already closed.
	ErrClosed = errors.New("already closed")
)

Functions

func Copy

func Copy(
	ctx context.Context,
	from ReadBucket,
	to ReadWriteBucket,
	options ...normalpath.TransformerOption,
) (int, error)

Copy copies the bucket at from to the bucket at to.

Copies done concurrently. Paths from the source bucket will be transformed before being added to the destination bucket. Returns the number of files copied.

func CopyPaths

func CopyPaths(
	ctx context.Context,
	from ReadBucket,
	to ReadWriteBucket,
	paths []string,
	options ...normalpath.TransformerOption,
) (int, error)

CopyPaths copies the paths from the bucket at from to the bucket at to.

Paths ignored if they do not exists in from. Copies done concurrently. Paths from the source bucket will be transformed before being added to the destination bucket. Paths will be normalized within this function. Returns the number of files copied.

func CopyPrefix

func CopyPrefix(
	ctx context.Context,
	from ReadBucket,
	to ReadWriteBucket,
	prefix string,
	options ...normalpath.TransformerOption,
) (int, error)

CopyPrefix copies the bucket at from to the bucket at to for the given prefix.

Copies done concurrently. Paths from the source bucket will be transformed before being added to the destination bucket. Returns the number of files copied.

func Exists

func Exists(ctx context.Context, readBucket ReadBucket, path string) (bool, error)

Exists returns true if the path exists, false otherwise.

Returns error on system error.

func IsNotExist

func IsNotExist(err error) bool

IsNotExist returns true for a error that is for a path not existing.

func NewErrNotExist

func NewErrNotExist(path string) error

NewErrNotExist returns a new error for a path not existing.

func ReadPath

func ReadPath(ctx context.Context, readBucket ReadBucket, path string) (_ []byte, retErr error)

ReadPath is analogous to ioutil.ReadFile.

Returns an error that fufills IsNotExist if the path does not exist.

Types

type BucketType

type BucketType int

BucketType is a bucket type.

const (
	// BucketTypeUnspecified says that a bucket's type is unspecified.
	BucketTypeUnspecified BucketType = iota
	// BucketTypeMem says that a bucket is in-memory.
	BucketTypeMem
	// BucketTypeDir says that a bucket is a local filesystem directory.
	BucketTypeDir
)

type ObjectInfo

type ObjectInfo interface {
	// Size is the size of the object.
	//
	// For writes, the write size must sum up to this size when closed, otherwise ErrIncompleteWrite is returned.
	// For writes, any write over this size will return io.EOF.
	Size() uint32
}

ObjectInfo contains object info.

type ReadBucket

type ReadBucket interface {
	// Type is the bucket type.
	//
	// Returns BucketTypeUnspecified if the bucket type is unspecified.
	Type() BucketType

	// Get gets the path.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Get(ctx context.Context, path string) (ReadObject, error)
	// Stat gets info in the object.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Stat(ctx context.Context, path string) (ObjectInfo, error)
	// Walk walks the bucket with the prefix, calling f on each path.
	//
	// Note that foo/barbaz will not be called for foo/bar, but will
	// be called for foo/bar/baz.
	//
	// All paths given to f are normalized and validated.
	// If f returns error, Walk will stop short and return this error.
	// Returns other error on system error.
	Walk(ctx context.Context, prefix string, f func(string) error) error
}

ReadBucket is a simple read-only bucket.

All paths regular files - Buckets do not handle directories. All paths must be relative. All paths are cleaned and ToSlash'ed by each function. Paths must not jump the bucket context, that is after clean, they cannot contain "..".

type ReadBucketCloser

type ReadBucketCloser interface {
	io.Closer
	ReadBucket
}

ReadBucketCloser is a read-only bucket that must be closed.

type ReadObject

type ReadObject interface {
	ObjectInfo
	io.ReadCloser
}

ReadObject is a read-only object.

It must be closed when done.

type ReadWriteBucket

type ReadWriteBucket interface {
	ReadBucket

	// Put returns a WriteCloser to write to the path.
	//
	// The path is truncated beforehand.
	//
	// Returns error on system error.
	Put(ctx context.Context, path string, size uint32) (WriteObject, error)
}

ReadWriteBucket is a simple read/write bucket.

type ReadWriteBucketCloser

type ReadWriteBucketCloser interface {
	io.Closer
	ReadWriteBucket
}

ReadWriteBucketCloser is a read/write bucket that must be closed.

type WriteObject

type WriteObject interface {
	ObjectInfo
	io.WriteCloser
}

WriteObject is a write-only object.

It must be closed when done.