blob

package
v0.16.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 24, 2024 License: Apache-2.0 Imports: 9 Imported by: 6

Documentation

Overview

Package blob implements simple storage of immutable, unstructured binary large objects (BLOBs).

Index

Constants

View Source 
const InvalidCredentialsErrStr = "The provided token has expired"

InvalidCredentialsErrStr is the error string returned by the provider when a token has expired.

Variables

View Source 
var ErrBlobAlreadyExists = errors.New("blob already exists")

ErrBlobAlreadyExists is returned when attempting to put a blob that already exists.

View Source 
var ErrBlobNotFound = errors.New("BLOB not found")

ErrBlobNotFound is returned when a BLOB cannot be found in storage.

View Source 
var ErrInvalidCredentials = errors.Errorf(InvalidCredentialsErrStr)

ErrInvalidCredentials is returned when the token used for authenticating with a storage provider has expired.

View Source 
var ErrInvalidRange = errors.Errorf("invalid blob offset or length")

ErrInvalidRange is returned when the requested blob offset or length is invalid.

View Source 
var ErrNotAVolume = errors.New("unsupported method, storage is not a volume")

ErrNotAVolume is returned when attempting to use a Volume method against a storage implementation that does not support the intended functionality.

View Source 
var ErrSetTimeUnsupported = errors.Errorf("SetTime is not supported")

ErrSetTimeUnsupported is returned by implementations of Storage that don't support SetTime.

View Source 
var ErrUnsupportedObjectLock = errors.New("object locking unsupported")

ErrUnsupportedObjectLock is returned when attempting to use an Object Lock specific function on a storage implementation that does not have the intended functionality.

View Source 
var ErrUnsupportedPutBlobOption = errors.New("unsupported put-blob option")

ErrUnsupportedPutBlobOption is returned when a PutBlob option that is not supported by an implementation of Storage is specified in a PutBlob call.

Functions

func AddSupportedStorage 

func AddSupportedStorage[T any](
	urlScheme string,
	defaultConfig T,
	createStorageFunc func(ctx context.Context, options *T, isCreate bool) (Storage, error),
)

AddSupportedStorage registers factory function to create storage with a given type name.

func DeleteMultiple added in v0.9.0 

func DeleteMultiple(ctx context.Context, st Storage, ids []ID, parallelism int) error

DeleteMultiple deletes multiple blobs in parallel.

func EnsureLengthExactly added in v0.8.0 

func EnsureLengthExactly(gotLength int, length int64) error

EnsureLengthExactly validates that length of the given slice is exactly the provided value. and returns ErrInvalidRange if the length is of the slice if not. As a special case length < 0 disables validation.

func IterateAllPrefixesInParallel 

func IterateAllPrefixesInParallel(ctx context.Context, parallelism int, st Storage, prefixes []ID, callback func(Metadata) error) error

IterateAllPrefixesInParallel invokes the provided callback and returns the first error returned by the callback or nil.

func MaxTimestamp added in v0.9.0 

func MaxTimestamp(mds []Metadata) time.Time

MaxTimestamp returns maximum timestamp for blobs in Metadata slice.

func MinTimestamp added in v0.9.0 

func MinTimestamp(mds []Metadata) time.Time

MinTimestamp returns minimum timestamp for blobs in Metadata slice.

func ReadBlobMap added in v0.11.0 

func ReadBlobMap(ctx context.Context, br Reader) (map[ID]Metadata, error)

ReadBlobMap reads the map of all the blobs indexed by ID.

func TotalLength added in v0.9.0 

func TotalLength(mds []Metadata) int64

TotalLength returns minimum timestamp for blobs in Metadata slice.

Types

type Bytes added in v0.6.0 

type Bytes interface {
	io.WriterTo

	Length() int
	Reader() io.ReadSeekCloser
}

Bytes encapsulates a sequence of bytes, possibly stored in a non-contiguous buffers, which can be written sequentially or treated as a io.Reader.

type Capacity added in v0.10.7 

type Capacity struct {
	// Size of volume in bytes.
	SizeB uint64 `json:"capacity,omitempty"`
	// Available (writeable) space in bytes.
	FreeB uint64 `json:"available"`
}

Capacity describes the storage capacity and usage of a Volume.

type ConnectionInfo 

type ConnectionInfo struct {
	Type   string
	Config interface{}
}

ConnectionInfo represents JSON-serializable configuration of a blob storage.

func (ConnectionInfo) MarshalJSON 

func (c ConnectionInfo) MarshalJSON() ([]byte, error)

MarshalJSON returns JSON-encoded storage configuration.

func (*ConnectionInfo) UnmarshalJSON 

func (c *ConnectionInfo) UnmarshalJSON(b []byte) error

UnmarshalJSON parses the JSON-encoded data into ConnectionInfo.

type DefaultProviderImplementation added in v0.14.0 

type DefaultProviderImplementation struct{}

DefaultProviderImplementation provides a default implementation for common functions that are mostly provider independent and have a sensible default.

Storage providers should embed this struct and override functions that they have different return values for.

func (DefaultProviderImplementation) Close added in v0.14.0 

func (s DefaultProviderImplementation) Close(context.Context) error

Close complies with the Storage interface.

func (DefaultProviderImplementation) ExtendBlobRetention added in v0.14.0 

func (s DefaultProviderImplementation) ExtendBlobRetention(context.Context, ID, ExtendOptions) error

ExtendBlobRetention provides a common implementation for unsupported blob retention storage.

func (DefaultProviderImplementation) FlushCaches added in v0.14.0 

func (s DefaultProviderImplementation) FlushCaches(context.Context) error

FlushCaches complies with the Storage interface.

func (DefaultProviderImplementation) GetCapacity added in v0.14.0 

func (s DefaultProviderImplementation) GetCapacity(context.Context) (Capacity, error)

GetCapacity complies with the Storage interface.

func (DefaultProviderImplementation) IsReadOnly added in v0.14.0 

func (s DefaultProviderImplementation) IsReadOnly() bool

IsReadOnly complies with the Storage interface.

type ExtendOptions added in v0.14.0 

type ExtendOptions struct {
	RetentionMode   RetentionMode
	RetentionPeriod time.Duration
}

ExtendOptions represents retention options for extending object locks.

type ID 

type ID string

ID is a string that represents blob identifier.

func IDsFromMetadata added in v0.9.0 

func IDsFromMetadata(mds []Metadata) []ID

IDsFromMetadata returns IDs for blobs in Metadata slice.

type Metadata 

type Metadata struct {
	BlobID    ID        `json:"id"`
	Length    int64     `json:"length"`
	Timestamp time.Time `json:"timestamp"`
}

Metadata represents metadata about a single BLOB in a storage.

func ListAllBlobs 

func ListAllBlobs(ctx context.Context, st Reader, prefix ID) ([]Metadata, error)

ListAllBlobs returns Metadata for all blobs in a given storage that have the provided name prefix.

func PutBlobAndGetMetadata added in v0.9.8 

func PutBlobAndGetMetadata(ctx context.Context, st Storage, blobID ID, data Bytes, opts PutOptions) (Metadata, error)

PutBlobAndGetMetadata invokes PutBlob and returns the resulting Metadata.

func (*Metadata) String added in v0.6.0 

func (m *Metadata) String() string

type OutputBuffer added in v0.9.5 

type OutputBuffer interface {
	io.Writer

	Reset()
	Length() int
}

OutputBuffer is implemented by *gather.WriteBuffer.

type PutOptions added in v0.9.6 

type PutOptions struct {
	RetentionMode   RetentionMode
	RetentionPeriod time.Duration

	// if true, PutBlob will fail with ErrBlobAlreadyExists if a blob with the same ID exists.
	DoNotRecreate bool

	// if not empty, set the provided timestamp on the blob instead of server-assigned,
	// if unsupported by the server return ErrSetTimeUnsupported
	SetModTime time.Time
	GetModTime *time.Time // if != nil, populate the value pointed at with the actual modification time
}

PutOptions represents put-options for a single BLOB in a storage.

func (PutOptions) HasRetentionOptions added in v0.9.6 

func (o PutOptions) HasRetentionOptions() bool

HasRetentionOptions returns true when blob-retention settings have been specified, otherwise returns false.

type Reader added in v0.8.0 

type Reader interface {
	// GetBlob returns full or partial contents of a blob with given ID.
	// If length>0, the function retrieves a range of bytes [offset,offset+length)
	// If length<0, the entire blob must be fetched.
	// Returns ErrInvalidRange if the fetched blob length is invalid.
	GetBlob(ctx context.Context, blobID ID, offset, length int64, output OutputBuffer) error

	// GetMetadata returns Metadata about single blob.
	GetMetadata(ctx context.Context, blobID ID) (Metadata, error)

	// ListBlobs invokes the provided callback for each blob in the storage.
	// Iteration continues until the callback returns an error or until all matching blobs have been reported.
	ListBlobs(ctx context.Context, blobIDPrefix ID, cb func(bm Metadata) error) error

	// ConnectionInfo returns JSON-serializable data structure containing information required to
	// connect to storage.
	ConnectionInfo() ConnectionInfo

	// DisplayName Name of the storage used for quick identification by humans.
	DisplayName() string
}

Reader defines read access API to blob storage.

type RetentionMode added in v0.10.2 

type RetentionMode string

RetentionMode - object retention mode. 

const (
	// Governance - governance mode.
	Governance RetentionMode = "GOVERNANCE"

	// Compliance - compliance mode.
	Compliance RetentionMode = "COMPLIANCE"

	// Locked - Locked policy mode for Azure.
	Locked RetentionMode = RetentionMode(blob.ImmutabilityPolicyModeLocked)
)

func (RetentionMode) IsValid added in v0.10.2 

func (r RetentionMode) IsValid() bool

IsValid - check whether this retention mode is valid or not.

func (RetentionMode) String added in v0.10.2 

func (r RetentionMode) String() string

type Storage 

type Storage interface {
	Volume
	Reader

	// PutBlob uploads the blob with given data to the repository or replaces existing blob with the provided
	// id with contents gathered from the specified list of slices.
	PutBlob(ctx context.Context, blobID ID, data Bytes, opts PutOptions) error

	// DeleteBlob removes the blob from storage. Future Get() operations will fail with ErrNotFound.
	DeleteBlob(ctx context.Context, blobID ID) error

	// Close releases all resources associated with storage.
	Close(ctx context.Context) error

	// FlushCaches flushes any local caches associated with storage.
	FlushCaches(ctx context.Context) error

	// ExtendBlobRetention extends the retention time of a blob (when blob retention is enabled)
	ExtendBlobRetention(ctx context.Context, blobID ID, opts ExtendOptions) error

	// IsReadOnly returns whether this Storage is in read-only mode. When in
	// read-only mode all mutation operations will fail.
	IsReadOnly() bool
}

Storage encapsulates API for connecting to blob storage.

The underlying storage system must provide:

* high durability, availability and bit-rot protection * read-after-write - blob written using PubBlob() must be immediately readable using GetBlob() and ListBlobs() * atomicity - it mustn't be possible to observe partial results of PubBlob() via either GetBlob() or ListBlobs() * timestamps that don't go back in time (small clock skew up to minutes is allowed) * reasonably low latency for retrievals

The required semantics are provided by existing commercial cloud storage products (Google Cloud, AWS, Azure).

func NewStorage 

func NewStorage(ctx context.Context, cfg ConnectionInfo, isCreate bool) (Storage, error)

NewStorage creates new storage based on ConnectionInfo. The storage type must be previously registered using AddSupportedStorage.

type Volume added in v0.10.7 

type Volume interface {
	// GetCapacity returns the capacity of a given volume.
	GetCapacity(ctx context.Context) (Capacity, error)
}

Volume defines disk/volume access API to blob storage.

Source Files

View all Source files

Directories

Path Synopsis
Package azure implements Azure Blob Storage.
 Package azure implements Azure Blob Storage.
Package b2 implements Storage based on an Backblaze B2 bucket.
 Package b2 implements Storage based on an Backblaze B2 bucket.
Package beforeop implements wrapper around blob.Storage that run a given callback before all operations.
 Package beforeop implements wrapper around blob.Storage that run a given callback before all operations.
Package filesystem implements filesystem-based Storage.
 Package filesystem implements filesystem-based Storage.
Package gcs implements Storage based on Google Cloud Storage bucket.
 Package gcs implements Storage based on Google Cloud Storage bucket.
Package gdrive implements Storage based on Google Drive.
 Package gdrive implements Storage based on Google Drive.
Package logging implements wrapper around Storage that logs all activity.
 Package logging implements wrapper around Storage that logs all activity.
Package rclone implements blob storage provider proxied by rclone (http://rclone.org)
 Package rclone implements blob storage provider proxied by rclone (http://rclone.org)
Package readonly implements wrapper around readonlyStorage that prevents all mutations.
 Package readonly implements wrapper around readonlyStorage that prevents all mutations.
Package retrying implements wrapper around blob.Storage that adds retry loop around all operations in case they return unexpected errors.
 Package retrying implements wrapper around blob.Storage that adds retry loop around all operations in case they return unexpected errors.
Package s3 implements Storage based on an S3 bucket.
 Package s3 implements Storage based on an S3 bucket.
Package sftp implements blob storage provided for SFTP/SSH.
 Package sftp implements blob storage provided for SFTP/SSH.
Package sharded implements common support for sharded blob providers, such as filesystem or webdav.
 Package sharded implements common support for sharded blob providers, such as filesystem or webdav.
Package storagemetrics implements wrapper around Storage that adds metrics around all activity.
 Package storagemetrics implements wrapper around Storage that adds metrics around all activity.
Package throttling implements wrapper around blob.Storage that adds throttling to all calls.
 Package throttling implements wrapper around blob.Storage that adds throttling to all calls.
Package webdav implements WebDAV-based Storage.
 Package webdav implements WebDAV-based Storage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.