storage

package
v1.61.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 12, 2022 License: AGPL-3.0 Imports: 9 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultLookupLimit = 500

DefaultLookupLimit is the default lookup limit for storage implementations.

View Source 
const Delimiter = '/'

Delimiter separates nested paths in storage.

Variables

View Source 
var ErrEmptyKey = errs.Class("empty key")

ErrEmptyKey is returned when an empty key is used in Put or in CompareAndSwap.

View Source 
var ErrEmptyQueue = errs.Class("empty queue")

ErrEmptyQueue is returned when attempting to Dequeue from an empty queue.

View Source 
var ErrInvalidBlobRef = errs.Class("invalid blob ref")

ErrInvalidBlobRef is returned when an blob reference is invalid.

View Source 
var ErrKeyNotFound = errs.Class("key not found")

ErrKeyNotFound used when something doesn't exist.

View Source 
var ErrLimitExceeded = errs.Class("limit exceeded")

ErrLimitExceeded is returned when request limit is exceeded.

View Source 
var ErrValueChanged = errs.Class("value changed")

ErrValueChanged is returned when the current value of the key does not match the oldValue in CompareAndSwap.

Functions

func ListV2Iterate added in v1.3.2 

func ListV2Iterate(ctx context.Context, store KeyValueStore, opts ListOptions, fn func(context.Context, *ListItem) error) (more bool, err error)

ListV2Iterate lists all keys corresponding to ListOptions. limit is capped to LookupLimit.

more indicates if the result was truncated. If false then the result []ListItem includes all requested keys. If true then the caller must call List again to get more results by setting `StartAfter` appropriately.

The opts.IncludeValue is ignored for this func. The callback item will be reused for next calls. If the user needs the preserve the value, it must call storage.CloneValue or storage.CloneKey.

func PutAll 

func PutAll(ctx context.Context, store KeyValueStore, items ...ListItem) (err error)

PutAll adds multiple values to the store.

Types

type BlobInfo added in v0.18.0 

type BlobInfo interface {
	// BlobRef returns the relevant BlobRef for the blob.
	BlobRef() BlobRef
	// StorageFormatVersion indicates the storage format version used to store the piece.
	StorageFormatVersion() FormatVersion
	// FullPath gives the full path to the on-disk blob file.
	FullPath(ctx context.Context) (string, error)
	// Stat does a stat on the on-disk blob file.
	Stat(ctx context.Context) (os.FileInfo, error)
}

BlobInfo allows lazy inspection of a blob and its underlying file during iteration with WalkNamespace-type methods.

type BlobReader 

type BlobReader interface {
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Closer
	// Size returns the size of the blob.
	Size() (int64, error)
	// StorageFormatVersion returns the storage format version associated with the blob.
	StorageFormatVersion() FormatVersion
}

BlobReader is an interface that groups Read, ReadAt, Seek and Close.

type BlobRef 

type BlobRef struct {
	Namespace []byte
	Key       []byte
}

BlobRef is a reference to a blob.

func (*BlobRef) IsValid 

func (ref *BlobRef) IsValid() bool

IsValid returns whether both namespace and key are specified.

type BlobWriter 

type BlobWriter interface {
	io.Writer
	io.Seeker
	// Cancel discards the blob.
	Cancel(context.Context) error
	// Commit ensures that the blob is readable by others.
	Commit(context.Context) error
	// Size returns the size of the blob
	Size() (int64, error)
	// StorageFormatVersion returns the storage format version associated with the blob.
	StorageFormatVersion() FormatVersion
}

BlobWriter defines the interface that must be satisfied for a general blob storage provider. BlobWriter instances are returned by the Create() method on Blobs instances.

type Blobs 

type Blobs interface {
	// Create creates a new blob that can be written.
	// Optionally takes a size argument for performance improvements, -1 is unknown size.
	Create(ctx context.Context, ref BlobRef, size int64) (BlobWriter, error)
	// Open opens a reader with the specified namespace and key.
	Open(ctx context.Context, ref BlobRef) (BlobReader, error)
	// OpenWithStorageFormat opens a reader for the already-located blob, avoiding the potential
	// need to check multiple storage formats to find the blob.
	OpenWithStorageFormat(ctx context.Context, ref BlobRef, formatVer FormatVersion) (BlobReader, error)
	// Delete deletes the blob with the namespace and key.
	Delete(ctx context.Context, ref BlobRef) error
	// DeleteWithStorageFormat deletes a blob of a specific storage format.
	DeleteWithStorageFormat(ctx context.Context, ref BlobRef, formatVer FormatVersion) error
	// DeleteNamespace deletes blobs folder for a specific namespace.
	DeleteNamespace(ctx context.Context, ref []byte) (err error)
	// Trash marks a file for pending deletion.
	Trash(ctx context.Context, ref BlobRef) error
	// RestoreTrash restores all files in the trash for a given namespace and returns the keys restored.
	RestoreTrash(ctx context.Context, namespace []byte) ([][]byte, error)
	// EmptyTrash removes all files in trash that were moved to trash prior to trashedBefore and returns the total bytes emptied and keys deleted.
	EmptyTrash(ctx context.Context, namespace []byte, trashedBefore time.Time) (int64, [][]byte, error)
	// Stat looks up disk metadata on the blob file.
	Stat(ctx context.Context, ref BlobRef) (BlobInfo, error)
	// StatWithStorageFormat looks up disk metadata for the blob file with the given storage format
	// version. This avoids the potential need to check multiple storage formats for the blob
	// when the format is already known.
	StatWithStorageFormat(ctx context.Context, ref BlobRef, formatVer FormatVersion) (BlobInfo, error)
	// FreeSpace return how much free space is available to the blobstore.
	FreeSpace(ctx context.Context) (int64, error)
	// CheckWritability tests writability of the storage directory by creating and deleting a file.
	CheckWritability(ctx context.Context) error
	// SpaceUsedForTrash returns the total space used by the trash.
	SpaceUsedForTrash(ctx context.Context) (int64, error)
	// SpaceUsedForBlobs adds up how much is used in all namespaces.
	SpaceUsedForBlobs(ctx context.Context) (int64, error)
	// SpaceUsedForBlobsInNamespace adds up how much is used in the given namespace.
	SpaceUsedForBlobsInNamespace(ctx context.Context, namespace []byte) (int64, error)
	// ListNamespaces finds all namespaces in which keys might currently be stored.
	ListNamespaces(ctx context.Context) ([][]byte, error)
	// WalkNamespace executes walkFunc for each locally stored blob, stored with
	// storage format V1 or greater, in the given namespace. If walkFunc returns a non-nil
	// error, WalkNamespace will stop iterating and return the error immediately. The ctx
	// parameter is intended to allow canceling iteration early.
	WalkNamespace(ctx context.Context, namespace []byte, walkFunc func(BlobInfo) error) error
	// CreateVerificationFile creates a file to be used for storage directory verification.
	CreateVerificationFile(ctx context.Context, id storj.NodeID) error
	// VerifyStorageDir verifies that the storage directory is correct by checking for the existence and validity
	// of the verification file.
	VerifyStorageDir(ctx context.Context, id storj.NodeID) error
	// Close closes the blob store and any resources associated with it.
	Close() error
}

Blobs is a blob storage interface.

architecture: Database

type FormatVersion added in v0.18.0 

type FormatVersion int

FormatVersion represents differing storage format version values. Different Blobs implementors might interpret different FormatVersion values differently, but they share a type so that there can be a common StorageFormatVersion() call on the interface.

Changes in FormatVersion might affect how a Blobs or BlobReader or BlobWriter instance works, or they might only be relevant to some higher layer. A FormatVersion must be specified when writing a new blob, and the blob storage interface must store that value with the blob somehow, so that the same FormatVersion is returned later when reading that stored blob.

type Items 

type Items []ListItem

Items keeps all ListItem.

func CloneItems 

func CloneItems(items Items) Items

CloneItems creates a deep copy of items.

func ListV2 

func ListV2(ctx context.Context, store KeyValueStore, opts ListOptions) (result Items, more bool, err error)

ListV2 lists all keys corresponding to ListOptions. limit is capped to LookupLimit.

more indicates if the result was truncated. If false then the result []ListItem includes all requested keys. If true then the caller must call List again to get more results by setting `StartAfter` appropriately.

func (Items) GetKeys 

func (items Items) GetKeys() Keys

GetKeys gets all the Keys in []ListItem and converts them to Keys.

func (Items) Len 

func (items Items) Len() int

Len is the number of elements in the collection.

func (Items) Less 

func (items Items) Less(i, k int) bool

Less reports whether the element with index i should sort before the element with index j.

func (Items) Swap 

func (items Items) Swap(i, k int)

Swap swaps the elements with indexes i and j.

type IterateOptions 

type IterateOptions struct {
	// Prefix ensure.
	Prefix Key
	// First will be the first item iterator returns or the next item (previous when reverse).
	First Key
	// Recurse, do not collapse items based on Delimiter.
	Recurse bool
	// The maximum number of elements to be returned.
	Limit int
}

IterateOptions contains options for iterator.

type Iterator 

type Iterator interface {
	// Next prepares the next list item.
	// It returns true on success, or false if there is no next result row or an error happened while preparing it.
	Next(ctx context.Context, item *ListItem) bool
}

Iterator iterates over a sequence of ListItems.

type IteratorFunc 

type IteratorFunc func(ctx context.Context, item *ListItem) bool

IteratorFunc implements basic iterator.

func (IteratorFunc) Next 

func (next IteratorFunc) Next(ctx context.Context, item *ListItem) bool

Next returns the next item.

type Key 

type Key []byte

Key is the type for the keys in a `KeyValueStore`.

func AfterPrefix 

func AfterPrefix(key Key) Key

AfterPrefix returns the key after prefix.

func CloneKey 

func CloneKey(key Key) Key

CloneKey creates a copy of key.

func NextKey 

func NextKey(key Key) Key

NextKey returns the successive key.

func (Key) Equal 

func (key Key) Equal(b Key) bool

Equal returns whether key and b are equal.

func (Key) IsZero 

func (key Key) IsZero() bool

IsZero returns true if the key struct is it's zero value.

func (Key) Less 

func (key Key) Less(b Key) bool

Less returns whether key should be sorted before b.

func (Key) MarshalBinary 

func (key Key) MarshalBinary() ([]byte, error)

MarshalBinary implements the encoding.BinaryMarshaler interface for the Key type.

func (Key) String 

func (key Key) String() string

String implements the Stringer interface.

type KeyValueStore 

type KeyValueStore interface {
	// Put adds a value to store.
	Put(context.Context, Key, Value) error
	// Get gets a value to store.
	Get(context.Context, Key) (Value, error)
	// GetAll gets all values from the store.
	GetAll(context.Context, Keys) (Values, error)
	// Delete deletes key and the value.
	Delete(context.Context, Key) error
	// DeleteMultiple deletes keys and returns nil for.
	DeleteMultiple(context.Context, []Key) (Items, error)
	// List lists all keys starting from start and upto limit items.
	List(ctx context.Context, start Key, limit int) (Keys, error)
	// Iterate iterates over items based on opts.
	Iterate(ctx context.Context, opts IterateOptions, fn func(context.Context, Iterator) error) error
	// IterateWithoutLookupLimit calls the callback with an iterator over the keys, but doesn't enforce default limit on opts.
	IterateWithoutLookupLimit(ctx context.Context, opts IterateOptions, fn func(context.Context, Iterator) error) error
	// CompareAndSwap atomically compares and swaps oldValue with newValue.
	CompareAndSwap(ctx context.Context, key Key, oldValue, newValue Value) error
	// Close closes the store.
	Close() error

	// LookupLimit returns the maximum limit that is allowed.
	LookupLimit() int
}

KeyValueStore describes key/value stores like redis and boltdb.

type Keys 

type Keys []Key

Keys is the type for a slice of keys in a `KeyValueStore`.

func ListKeys 

func ListKeys(ctx context.Context, store KeyValueStore, first Key, limit int) (_ Keys, err error)

ListKeys returns keys starting from first and upto limit. limit is capped to LookupLimit.

func (Keys) ByteSlices 

func (keys Keys) ByteSlices() [][]byte

ByteSlices converts a `Keys` struct to a slice of byte-slices (i.e. `[][]byte`).

func (Keys) Strings 

func (keys Keys) Strings() []string

Strings returns everything as strings.

type ListItem 

type ListItem struct {
	Key      Key
	Value    Value
	IsPrefix bool
}

ListItem returns Key, Value, IsPrefix.

func CloneItem 

func CloneItem(item ListItem) ListItem

CloneItem creates a deep copy of item.

func (ListItem) Less 

func (item ListItem) Less(b ListItem) bool

Less returns whether item should be sorted before b.

type ListOptions 

type ListOptions struct {
	Prefix       Key
	StartAfter   Key // StartAfter is relative to Prefix
	Recursive    bool
	IncludeValue bool
	Limit        int
}

ListOptions are items that are optional for the LIST method.

type Value 

type Value []byte

Value is the type for the values in a `ValueValueStore`.

func CloneValue 

func CloneValue(value Value) Value

CloneValue creates a copy of value.

func (Value) IsZero 

func (value Value) IsZero() bool

IsZero returns true if the value struct is it's zero value.

func (Value) MarshalBinary 

func (value Value) MarshalBinary() ([]byte, error)

MarshalBinary implements the encoding.BinaryMarshaler interface for the Value type.

type Values 

type Values []Value

Values is the type for a slice of Values in a `KeyValueStore`.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.