objstore

package
v0.21.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 3, 2021 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	OpIter       = "iter"
	OpGet        = "get"
	OpGetRange   = "get_range"
	OpExists     = "exists"
	OpUpload     = "upload"
	OpDelete     = "delete"
	OpAttributes = "attributes"
)
View Source 
const DirDelim = "/"

DirDelim is the delimiter used to model a directory structure in an object store bucket.

Variables

This section is empty.

Functions

func AcceptanceTest added in v0.12.0 

func AcceptanceTest(t *testing.T, bkt Bucket)

func BucketWithMetrics 

func BucketWithMetrics(name string, b Bucket, reg prometheus.Registerer) *metricBucket

BucketWithMetrics takes a bucket and registers metrics with the given registry for operations run against the bucket.

func CreateTemporaryTestBucketName added in v0.9.0 

func CreateTemporaryTestBucketName(t testing.TB) string

func DownloadDir 

func DownloadDir(ctx context.Context, logger log.Logger, bkt BucketReader, originalSrc, src, dst string, ignoredPaths ...string) error

DownloadDir downloads all object found in the directory into the local directory.

func DownloadFile 

func DownloadFile(ctx context.Context, logger log.Logger, bkt BucketReader, src, dst string) (err error)

DownloadFile downloads the src file from the bucket to dst. If dst is an existing directory, a file with the same name as the source is created in dst. If destination file is already existing, download file will overwrite it.

func EmptyBucket 

func EmptyBucket(t testing.TB, ctx context.Context, bkt Bucket)

EmptyBucket deletes all objects from bucket. This operation is required to properly delete bucket as a whole. It is used for testing only. TODO(bplotka): Add retries.

func NopCloserWithSize added in v0.20.0 

func NopCloserWithSize(r io.Reader) io.ReadCloser

NopCloserWithSize returns a ReadCloser with a no-op Close method wrapping the provided Reader r. Returned ReadCloser also implements Size method.

func TryToGetSize added in v0.12.0 

func TryToGetSize(r io.Reader) (int64, error)

TryToGetSize tries to get upfront size from reader. Some implementations may return only size of unread data in the reader, so it's best to call this method before doing any reading.

TODO(https://github.com/thanos-io/thanos/issues/678): Remove guessing length when minio provider will support multipart upload without this.

func UploadDir 

func UploadDir(ctx context.Context, logger log.Logger, bkt Bucket, srcdir, dstdir string) error

UploadDir uploads all files in srcdir to the bucket with into a top-level directory named dstdir. It is a caller responsibility to clean partial upload in case of failure.

func UploadFile 

func UploadFile(ctx context.Context, logger log.Logger, bkt Bucket, src, dst string) error

UploadFile uploads the file with the given name to the bucket. It is a caller responsibility to clean partial upload in case of failure.

func WithRecursiveIter added in v0.19.0 

func WithRecursiveIter(params *IterParams)

WithRecursiveIter is an option that can be applied to Iter() to recursively list objects in the bucket.

Types

type Bucket 

type Bucket interface {
	io.Closer
	BucketReader

	// Upload the contents of the reader as an object into the bucket.
	// Upload should be idempotent.
	Upload(ctx context.Context, name string, r io.Reader) error

	// Delete removes the object with the given name.
	// If object does not exists in the moment of deletion, Delete should throw error.
	Delete(ctx context.Context, name string) error

	// Name returns the bucket name for the provider.
	Name() string
}

Bucket provides read and write access to an object storage bucket. NOTE: We assume strong consistency for write-read flow.

type BucketReader 

type BucketReader interface {
	// Iter calls f for each entry in the given directory (not recursive.). The argument to f is the full
	// object name including the prefix of the inspected directory.
	// Entries are passed to function in sorted order.
	Iter(ctx context.Context, dir string, f func(string) error, options ...IterOption) error

	// Get returns a reader for the given object name.
	Get(ctx context.Context, name string) (io.ReadCloser, error)

	// GetRange returns a new range reader for the given object name and range.
	GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

	// Exists checks if the given object exists in the bucket.
	Exists(ctx context.Context, name string) (bool, error)

	// IsObjNotFoundErr returns true if error means that object is not found. Relevant to Get operations.
	IsObjNotFoundErr(err error) bool

	// Attributes returns information about the specified object.
	Attributes(ctx context.Context, name string) (ObjectAttributes, error)
}

BucketReader provides read access to an object storage bucket.

type InMemBucket added in v0.12.0 

type InMemBucket struct {
	// contains filtered or unexported fields
}

InMemBucket implements the objstore.Bucket interfaces against local memory. Methods from Bucket interface are thread-safe. Objects are assumed to be immutable.

func NewInMemBucket added in v0.12.0 

func NewInMemBucket() *InMemBucket

NewInMemBucket returns a new in memory Bucket. NOTE: Returned bucket is just a naive in memory bucket implementation. For test use cases only.

func (*InMemBucket) Attributes added in v0.14.0 

func (b *InMemBucket) Attributes(_ context.Context, name string) (ObjectAttributes, error)

Attributes returns information about the specified object.

func (*InMemBucket) Close added in v0.12.0 

func (b *InMemBucket) Close() error

func (*InMemBucket) Delete added in v0.12.0 

func (b *InMemBucket) Delete(_ context.Context, name string) error

Delete removes all data prefixed with the dir.

func (*InMemBucket) Exists added in v0.12.0 

func (b *InMemBucket) Exists(_ context.Context, name string) (bool, error)

Exists checks if the given directory exists in memory.

func (*InMemBucket) Get added in v0.12.0 

func (b *InMemBucket) Get(_ context.Context, name string) (io.ReadCloser, error)

Get returns a reader for the given object name.

func (*InMemBucket) GetRange added in v0.12.0 

func (b *InMemBucket) GetRange(_ context.Context, name string, off, length int64) (io.ReadCloser, error)

GetRange returns a new range reader for the given object name and range.

func (*InMemBucket) IsObjNotFoundErr added in v0.12.0 

func (b *InMemBucket) IsObjNotFoundErr(err error) bool

IsObjNotFoundErr returns true if error means that object is not found. Relevant to Get operations.

func (*InMemBucket) Iter added in v0.12.0 

func (b *InMemBucket) Iter(_ context.Context, dir string, f func(string) error, options ...IterOption) error

Iter calls f for each entry in the given directory. The argument to f is the full object name including the prefix of the inspected directory.

func (*InMemBucket) Name added in v0.12.0 

func (b *InMemBucket) Name() string

Name returns the bucket name.

func (*InMemBucket) Objects added in v0.12.0 

func (b *InMemBucket) Objects() map[string][]byte

Objects returns internally stored objects. NOTE: For assert purposes.

func (*InMemBucket) Upload added in v0.12.0 

func (b *InMemBucket) Upload(_ context.Context, name string, r io.Reader) error

Upload writes the file specified in src to into the memory.

type InstrumentedBucket added in v0.12.0 

type InstrumentedBucket interface {
	Bucket

	// WithExpectedErrs allows to specify a filter that marks certain errors as expected, so it will not increment
	// thanos_objstore_bucket_operation_failures_total metric.
	WithExpectedErrs(IsOpFailureExpectedFunc) Bucket

	// ReaderWithExpectedErrs allows to specify a filter that marks certain errors as expected, so it will not increment
	// thanos_objstore_bucket_operation_failures_total metric.
	// TODO(bwplotka): Remove this when moved to Go 1.14 and replace with InstrumentedBucketReader.
	ReaderWithExpectedErrs(IsOpFailureExpectedFunc) BucketReader
}

InstrumentedBucket is a Bucket with optional instrumentation control on reader.

func NewTracingBucket added in v0.14.0 

func NewTracingBucket(bkt Bucket) InstrumentedBucket

func WithNoopInstr added in v0.12.0 

func WithNoopInstr(bkt Bucket) InstrumentedBucket

type InstrumentedBucketReader added in v0.12.0 

type InstrumentedBucketReader interface {
	BucketReader

	// ReaderWithExpectedErrs allows to specify a filter that marks certain errors as expected, so it will not increment
	// thanos_objstore_bucket_operation_failures_total metric.
	ReaderWithExpectedErrs(IsOpFailureExpectedFunc) BucketReader
}

InstrumentedBucket is a BucketReader with optional instrumentation control.

type IsOpFailureExpectedFunc added in v0.12.0 

type IsOpFailureExpectedFunc func(error) bool

IsOpFailureExpectedFunc allows to mark certain errors as expected, so they will not increment thanos_objstore_bucket_operation_failures_total metric.

type IterOption added in v0.19.0 

type IterOption func(params *IterParams)

IterOption configures the provided params.

type IterParams added in v0.19.0 

type IterParams struct {
	Recursive bool
}

IterParams holds the Iter() parameters and is used by objstore clients implementations.

func ApplyIterOptions added in v0.19.0 

func ApplyIterOptions(options ...IterOption) IterParams

type ObjectAttributes added in v0.14.0 

type ObjectAttributes struct {
	// Size is the object size in bytes.
	Size int64 `json:"size"`

	// LastModified is the timestamp the object was last modified.
	LastModified time.Time `json:"last_modified"`
}

type ObjectSizer added in v0.20.0 

type ObjectSizer interface {
	// ObjectSize returns the size of the object in bytes, or error if it is not available.
	ObjectSize() (int64, error)
}

ObjectSizer can return size of object.

type TracingBucket added in v0.14.0 

type TracingBucket struct {
	// contains filtered or unexported fields
}

TracingBucket includes bucket operations in the traces.

func (TracingBucket) Attributes added in v0.14.0 

func (t TracingBucket) Attributes(ctx context.Context, name string) (attrs ObjectAttributes, err error)

func (TracingBucket) Close added in v0.14.0 

func (t TracingBucket) Close() error

func (TracingBucket) Delete added in v0.14.0 

func (t TracingBucket) Delete(ctx context.Context, name string) (err error)

func (TracingBucket) Exists added in v0.14.0 

func (t TracingBucket) Exists(ctx context.Context, name string) (exists bool, err error)

func (TracingBucket) Get added in v0.14.0 

func (t TracingBucket) Get(ctx context.Context, name string) (io.ReadCloser, error)

func (TracingBucket) GetRange added in v0.14.0 

func (t TracingBucket) GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

func (TracingBucket) IsObjNotFoundErr added in v0.14.0 

func (t TracingBucket) IsObjNotFoundErr(err error) bool

func (TracingBucket) Iter added in v0.14.0 

func (t TracingBucket) Iter(ctx context.Context, dir string, f func(string) error, options ...IterOption) (err error)

func (TracingBucket) Name added in v0.14.0 

func (t TracingBucket) Name() string

func (TracingBucket) ReaderWithExpectedErrs added in v0.14.0 

func (t TracingBucket) ReaderWithExpectedErrs(expectedFunc IsOpFailureExpectedFunc) BucketReader

func (TracingBucket) Upload added in v0.14.0 

func (t TracingBucket) Upload(ctx context.Context, name string, r io.Reader) (err error)

func (TracingBucket) WithExpectedErrs added in v0.14.0 

func (t TracingBucket) WithExpectedErrs(expectedFunc IsOpFailureExpectedFunc) Bucket

Source Files

View all Source files

Directories

Path Synopsis
Package gcs implements common object storage abstractions against Google Cloud Storage.
 Package gcs implements common object storage abstractions against Google Cloud Storage.
Package s3 implements common object storage abstractions against s3-compatible APIs.
 Package s3 implements common object storage abstractions against s3-compatible APIs.
Package swift implements common object storage abstractions against OpenStack swift APIs.
 Package swift implements common object storage abstractions against OpenStack swift APIs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.