Documentation
¶
Overview ¶
Package blob implements an interface and support code for persistent storage of opaque (untyped) binary blobs.
Summary ¶
A Store represents a collection of disjoint named key-value namespaces backed by a shared pool of storage. A store may further be partititioned into named "substores", each of which manages its own collection of keyspaces within its enclosing store. While stores and their keyspaces are logically distinct, they are intended to represent partitions of a single underlying storage layer.
Keyspaces are either arbitrary (KV) or content-addressed (CAS). Both types implement the common KVCore interface. Arbitrary keyspaces allow writing of values under user-chosen keys ("Put"), while content-addressed keyspaces write values under their content address only ("CASPut"). An arbitrary keyspace can be converted into a content addressed keyspace using CASFromKV.
Implementation Notes ¶
The Store and KV interfaces defined here are intended to be implementable on a variety of concrete substrates (files, databases, key-value stores) in a straightforward manner. The API of these types is intended to support blobs of a "reasonable" size, where any individual blob can be efficiently processed in memory without streaming or chunking.
While in principle blobs of arbitrary size may be stored, an implementation may reject "very large" blobs. Practically an implementation should try to accept blobs on the order of (up to) ~100MIB, but may reject blobs much larger than that. This interface is intended to store data that is partitioned at a higher level in the protocol, and may not be a good fit for use cases that require large individual blobs.
The memstore package provides an implementation suitable for use in testing. The filestore package provides an implementation that uses files and directories on a local filesystem. More interesting implementations using other storage libraries can be found in other repositories.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrKeyExists is reported by Put when writing a key that already exists in // the store. ErrKeyExists = errors.New("key already exists") // ErrKeyNotFound is reported by Get or Size when given a key that does not // exist in the store. ErrKeyNotFound = errors.New("key not found") )
Functions ¶
func IsKeyExists ¶
IsKeyExists reports whether err is or wraps ErrKeyExists.
func IsKeyNotFound ¶
IsKeyNotFound reports whether err or is or wraps ErrKeyNotFound. It is false if err == nil.
func KeyExists ¶
KeyExists returns an ErrKeyExists error reporting that key exists in the store. The concrete type is *blob.KeyError.
func KeyNotFound ¶
KeyNotFound returns an ErrKeyNotFound error reporting that key was not found. The concrete type is *blob.KeyError.
Types ¶
type CAS ¶
type CAS interface {
KVCore
// CASPut writes data to a content-addressed blob in the underlying store,
// and returns the assigned key. The target key is returned even in case of
// error.
CASPut(ctx context.Context, data []byte) (string, error)
// CASKey returns the content address of data without modifying the store.
// This must be the same value that would be returned by a successful call
// to CASPut on data.
CASKey(ctx context.Context, data []byte) string
}
CAS represents a mutable set of content-addressed key-value pairs in which each value is identified by a unique, opaque string key.
type Closer ¶ added in v0.9.0
Closer is an extension interface representing the ability to close and release resources claimed by a storage component.
type KV ¶ added in v0.7.0
type KV interface {
KVCore
// Put writes a blob to the store. If the store already contains the
// specified key and opts.Replace is true, the existing value is replaced
// without error; otherwise Put must report an ErrKeyExists error.
Put(ctx context.Context, opts PutOptions) error
}
A KV represents a mutable set of key-value pairs in which each value is identified by a unique, opaque string key. An implementation of KV is permitted (but not required) to report an error from Put when given an empty key. If the implementation cannot store empty keys, it must report ErrKeyNotFound when operating on an empty key.
Implementations of this interface must be safe for concurrent use by multiple goroutines. Moreover, any sequence of operations on a KV that does not overlap with any Delete executions must be linearizable.
type KVCore ¶ added in v0.9.0
type KVCore interface {
// Get fetches the contents of a blob from the store. If the key is not
// found in the store, Get must report an ErrKeyNotFound error.
Get(ctx context.Context, key string) ([]byte, error)
// Has reports which of the specified keys are present in the store.
// The result set contains one entry for each requested key that is present
// in the store. If none of the requested keys is present, the resulting set
// may be either empty or nil.
Has(ctx context.Context, keys ...string) (KeySet, error)
// Delete atomically removes a blob from the store. If the key is not found
// in the store, Delete must report an ErrKeyNotFound error.
Delete(ctx context.Context, key string) error
// List returns an iterator over each key in the store greater than or equal
// to start, in lexicographic order.
//
// Requirements:
//
// Each pair reported by the iterator MUST be either a valid key and a nil
// error, or an empty key and a non-nil error.
//
// After the iterator reports an error, it MUST immediately return, even if
// the yield function reports true.
//
// The caller should check the error as part of iteration:
//
// for key, err := range kv.List(ctx, start) {
// if err != nil {
// return fmt.Errorf("list: %w", err)
// }
// // ... process key
// }
//
// It must be safe to call Get, Has, List, and Len during iteration.
// A caller should not attempt to modify the store while listing, unless the
// storage implementation documents that it is safe to do so.
List(ctx context.Context, start string) iter.Seq2[string, error]
// Len reports the number of keys currently in the store.
Len(ctx context.Context) (int64, error)
}
KVCore is the common interface shared by implementations of a key-value namespace. Users will generally not use this interface directly; it is included by reference in KV and CAS.
type KeyError ¶
type KeyError struct {
Err error // the underlying error
Key string // the key implicated by the error
}
KeyError is the concrete type of errors involving a blob key. The caller may type-assert to *blob.KeyError to recover the key.
type KeySet ¶ added in v0.11.0
KeySet represents a set of keys. It is aliased here so the caller does not need to explicitly import mapset.
type PutOptions ¶
type PutOptions struct {
Key string // the key to associate with the data
Data []byte // the data to write
Replace bool // whether to replace an existing value for this key
}
PutOptions regulate the behaviour of the Put method of a KV implementation.
type Store ¶
type Store interface {
// KV returns a key space on the store.
//
// Multiple calls to KV with the same name are not required to return
// exactly the same KV value, but should return values that will converge
// (eventually) to the same view of the storage.
KV(ctx context.Context, name string) (KV, error)
// CAS returns a content-addressed key space on the store.
//
// Multiple calls to CAS with the same name are not required to return
// exactly the same CAS value, but should return values that will converge
// (eventually) to the same view of the storage.
//
// Implementations of this method that do not require special handling are
// encouraged to use CASFromKV to derive a CAS from a KV.
CAS(ctx context.Context, name string) (CAS, error)
// Sub returns a new Store subordinate to the receiver (a "substore").
// A substore shares logical storage with its parent store, but keyspaces
// derived from the substore are distinct from keyspaces of the parent store
// or any other substores derived from it.
//
// Multiple calls to Sub with the same name are not required to return
// exactly the same [Store] value, but should return values that will
// converge (eventually) to the same view of the storage.
Sub(ctx context.Context, name string) (Store, error)
}
A Store represents a collection of key-value namespaces ("keyspaces") identified by string labels. Each keyspace in a store is logically distinct; the keys from one space are independent of the keys in another.
Implementations of this interface must be safe for concurrent use by multiple goroutines.
The KV and CAS methods share a namespace, meaning that a KV and a CAS derived from the same Store and using the same name must share the same underlying key-value space. In particular a Put to a KV or a CASPut to a CAS must be visible to a Get or List from either.
type StoreCloser ¶ added in v0.8.0
StoreCloser combines a Store with a Close method that settles state and releases any resources from the store when it is no longer in use.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package memstore implements the blob.Store and blob.KV interfaces using in-memory dictionaries.
|
Package memstore implements the blob.Store and blob.KV interfaces using in-memory dictionaries. |
|
Package storetest provides correctness tests for implementations of the blob.KV interface.
|
Package storetest provides correctness tests for implementations of the blob.KV interface. |