objstore

Documentation

Index

• Constants
• func DownloadDir(ctx context.Context, logger log.Logger, bkt BucketReader, src, dst string) error
• func DownloadFile(ctx context.Context, logger log.Logger, bkt BucketReader, src, dst string) (err error)
• func EmptyBucket(t testing.TB, ctx context.Context, bkt Bucket)
• func Exists(ctx context.Context, bkt Bucket, src string) (bool, error)
• func UploadDir(ctx context.Context, logger log.Logger, bkt Bucket, srcdir, dstdir string) error
• func UploadFile(ctx context.Context, logger log.Logger, bkt Bucket, src, dst string) error
• type Bucket
  • func BucketWithMetrics(name string, b Bucket, r prometheus.Registerer) Bucket
• type BucketReader

Constants

const DirDelim = "/"

DirDelim is the delimiter used to model a directory structure in an object store bucket.

Functions

func DownloadDir

func DownloadDir(ctx context.Context, logger log.Logger, bkt BucketReader, src, dst string) error

DownloadDir downloads all object found in the directory into the local directory.

func DownloadFile

func DownloadFile(ctx context.Context, logger log.Logger, bkt BucketReader, src, dst string) (err error)

DownloadFile downloads the src file from the bucket to dst. If dst is an existing directory, a file with the same name as the source is created in dst. If destination file is already existing, download file will overwrite it.

func EmptyBucket

func EmptyBucket(t testing.TB, ctx context.Context, bkt Bucket)

EmptyBucket deletes all objects from bucket. This operation is required to properly delete bucket as a whole. It is used for testing only. TODO(bplotka): Add retries.

func Exists

func Exists(ctx context.Context, bkt Bucket, src string) (bool, error)

Exists returns true, if file exists, otherwise false and nil error if presence IsObjNotFoundErr, otherwise false with returning error.

func UploadDir

func UploadDir(ctx context.Context, logger log.Logger, bkt Bucket, srcdir, dstdir string) error

UploadDir uploads all files in srcdir to the bucket with into a top-level directory named dstdir. It is a caller responsibility to clean partial upload in case of failure.

func UploadFile

func UploadFile(ctx context.Context, logger log.Logger, bkt Bucket, src, dst string) error

UploadFile uploads the file with the given name to the bucket. It is a caller responsibility to clean partial upload in case of failure.

Types

type Bucket

type Bucket interface {
	io.Closer
	BucketReader

	// Upload the contents of the reader as an object into the bucket.
	// Upload should be idempotent.
	Upload(ctx context.Context, name string, r io.Reader) error

	// Delete removes the object with the given name.
	// If object does not exists in the moment of deletion, Delete should throw error.
	Delete(ctx context.Context, name string) error

	// Name returns the bucket name for the provider.
	Name() string
}

Bucket provides read and write access to an object storage bucket. NOTE: We assume strong consistency for write-read flow.

func BucketWithMetrics

func BucketWithMetrics(name string, b Bucket, r prometheus.Registerer) Bucket

BucketWithMetrics takes a bucket and registers metrics with the given registry for operations run against the bucket.

type BucketReader

type BucketReader interface {
	// Iter calls f for each entry in the given directory (not recursive.). The argument to f is the full
	// object name including the prefix of the inspected directory.
	Iter(ctx context.Context, dir string, f func(string) error) error

	// Get returns a reader for the given object name.
	Get(ctx context.Context, name string) (io.ReadCloser, error)

	// GetRange returns a new range reader for the given object name and range.
	GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

	// Exists checks if the given object exists in the bucket.
	// TODO(bplotka): Consider removing Exists in favor of helper that do Get & IsObjNotFoundErr (less code to maintain).
	Exists(ctx context.Context, name string) (bool, error)

	// IsObjNotFoundErr returns true if error means that object is not found. Relevant to Get operations.
	IsObjNotFoundErr(err error) bool
}

BucketReader provides read access to an object storage bucket.