storage

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2023 License: BSD-3-Clause Imports: 17 Imported by: 30

Documentation

Overview

Package storage provides a unified interface to a number of storage engines. Since each storage engine has different capabilities, this package defines a number of interfaces in addition to the core Engine interface, which all storage engines should satisfy.

Keys are specified as a combination of Context and a datatype-specific byte slice, typically called an "type-specific key" (TKey) in DVID docs and code. The Context provides DVID-wide namespacing and as such, must use one of the Context implementations within the storage package. (This is enforced by making Context a Go opaque interface.) The type-specific key formatting is entirely up to the datatype designer, although use of dvid.Index is suggested.

Initially we are concentrating on key-value backends but expect to support graph and perhaps relational databases, either using specialized databases or software layers on top of an ordered key-value store.

Although we assume lexicographically ordering for range queries, there is some variation in how variable size keys are treated. We assume all storage engines, after appropriate DVID drivers, use the following definition of ordering: 

A string s precedes a string t in lexicographic order if:

* s is a prefix of t, or
* if c and d are respectively the first character of s and t in which s and t differ,
  then c precedes d in character order.
* if s and t are equivalent for all of s, but t is longer

Note: For the characters that are alphabetical letters, the character order coincides
with the alphabetical order. Digits precede letters, and uppercase letters precede
lowercase ones.

Examples:

composer precedes computer
house precedes household
Household precedes house
H2O precedes HOTEL
mydex precedes mydexterity

Note that the above is different than shortlex order, which would group strings
based on length first.

The above lexicographical ordering is used by default for levedb variants.

Index

Constants

View Source 
const (
	// MarkData is a byte indicating real data stored and should be the last byte of any
	// versioned key.
	MarkData = 0x03

	// MarkTombstone is a byte indicating a tombstone -- a marker for deleted data.
	MarkTombstone = 0x4F
)
View Source 
const (
	TKeyMinClass = 0x00
	TKeyMaxClass = 0xFF
)
View Source 
const KafkaMaxMessageSize = 980 * dvid.Kilo

KafkaMaxMessageSize is the max message size in bytes for a Kafka message.

View Source 
const MonitorBuffer = 10000

Variables

View Source 
var (
	// Number of bytes read in last second from storage engine.
	StoreKeyBytesReadPerSec int

	// Number of bytes written in last second to storage engine.
	StoreKeyBytesWrittenPerSec int

	// Number of bytes read in last second from storage engine.
	StoreValueBytesReadPerSec int

	// Number of bytes written in last second to storage engine.
	StoreValueBytesWrittenPerSec int

	// Number of bytes read in last second from file system.
	FileBytesReadPerSec int

	// Number of bytes written in last second to file system.
	FileBytesWrittenPerSec int

	// Number of key-value GET calls in last second.
	GetsPerSec int

	// Number of key-value PUT calls in last second.
	PutsPerSec int

	// Channel to notify bytes read from a storage engine.
	StoreKeyBytesRead chan int

	// Channel to notify bytes written to a storage engine.
	StoreKeyBytesWritten chan int

	// Channel to notify bytes read from a storage engine.
	StoreValueBytesRead chan int

	// Channel to notify bytes written to a storage engine.
	StoreValueBytesWritten chan int

	// Channel to notify bytes read from file system.
	FileBytesRead chan int

	// Channel to notify bytes written to file system.
	FileBytesWritten chan int
)
View Source 
var ErrLogEOF = fmt.Errorf("log EOF")

ErrLogEOF is an error returned by a log Read() on an end of file

View Source 
var (
	// KafkaTopicPrefix is the kafka topic prefix for mutation logging
	KafkaTopicPrefix string
)

Functions

func AllStores 

func AllStores() (map[Alias]dvid.Store, error)

func ChangeDataKeyInstance 

func ChangeDataKeyInstance(k Key, instance dvid.InstanceID) error

ChangeDataKeyInstance modifies the passed Key to use the given data instance ID.

func ChangeDataKeyVersion 

func ChangeDataKeyVersion(k Key, v dvid.VersionID) error

ChangeDataKeyVersion modifies the passed Key to use the given version.

func DataFromFile 

func DataFromFile(filename string) ([]byte, error)

DataFromFile returns data from a file.

func DataKeyToLocalIDs 

func DataKeyToLocalIDs(k Key) (dvid.InstanceID, dvid.VersionID, dvid.ClientID, error)

DataKeyToLocalIDs parses a key and returns instance, version and client ids.

func DefaultKVStore 

func DefaultKVStore() (dvid.Store, error)

func DefaultLogStore 

func DefaultLogStore() (dvid.Store, error)

func DeleteDataInstance 

func DeleteDataInstance(data dvid.Data) error

DeleteDataInstance removes a data instance.

func EnginesAvailable 

func EnginesAvailable() string

EnginesAvailable returns a description of the available storage engines.

func GetAssignedLog 

func GetAssignedLog(ds DataSpec) (dvid.Store, error)

GetAssignedLog returns the append-only log based on DataUUID, (Name, RootUUID), tag, or type, in that order.

func GetAssignedStore 

func GetAssignedStore(ds DataSpec) (dvid.Store, error)

GetAssignedStore returns the store based on DataUUID, (Name, RootUUID), tag, or type, in that order. In some cases, this store may include a caching wrapper if the data instance has been configured to use groupcache.

func GetDataSizes 

func GetDataSizes(store dvid.Store, instances []dvid.InstanceID) (map[dvid.InstanceID]uint64, error)

GetDataSizes returns a list of storage sizes in bytes for each data instance in the store. A list of InstanceID can be optionally supplied so only those instances are queried. This requires some scanning of the database so could take longer than normal requests, particularly if a list of instances is not given. Note that the underlying store must support both the OrderedKeyValueGetter and SizeViewer interface, else this function returns nil.

func GetStoreByAlias 

func GetStoreByAlias(alias Alias) (dvid.Store, error)

GetStoreByAlias returns a store by the alias given to it in the configuration TOML file, e.g., "raid6".

func Initialize 

func Initialize(cmdline dvid.Config, backend *Backend, compiledTypes map[dvid.TypeString]struct{}) (createdMetadata bool, err error)

Initialize the storage systems. Returns a bool + error where the bool is true if the metadata store is newly created and needs initialization. The map of store configurations should be keyed by either a datatype name, "default", or "metadata".

func KafkaActivityTopic added in v0.9.17 

func KafkaActivityTopic() string

KafkaActivityTopic returns the topic name used for logging activity for this server.

func KafkaProduceMsg 

func KafkaProduceMsg(value []byte, topicName string) (err error)

KafkaProduceMsg sends a message to kafka

func KafkaShutdown 

func KafkaShutdown()

KafkaShutdown makes sure that the kafka queue is flushed before stopping.

func KafkaTopicSuffix 

func KafkaTopicSuffix(dataUUID dvid.UUID) string

KafkaTopicSuffix returns any configured suffix for the given data UUID or the empty string.

func LogActivityToKafka 

func LogActivityToKafka(activity map[string]interface{})

LogActivityToKafka publishes activity

func MetadataKeyPrefix 

func MetadataKeyPrefix() []byte

func NewStore 

func NewStore(c dvid.StoreConfig) (db dvid.Store, created bool, err error)

NewStore checks if a given engine is available and if so, returns a store opened with the configuration.

func RegisterEngine 

func RegisterEngine(e Engine)

RegisterEngine registers an Engine for DVID use.

func Repair 

func Repair(name, path string) error

Repair repairs a named engine's store at given path.

func Shutdown 

func Shutdown()

Shutdown handles any storage-specific shutdown procedures.

func SplitKey 

func SplitKey(k Key) (unversioned Key, versioned Key, err error)

SplitKey returns key components depending on whether the passed Key is a metadata or data key. If metadata, it returns the key and a 0 version id. If it is a data key, it returns the unversioned portion of the Key and the version id.

func UpdateDataKey 

func UpdateDataKey(k Key, instance dvid.InstanceID, version dvid.VersionID, client dvid.ClientID) error

UpdateDataKey modifies the passed Key to use the passed identifiers.

func VersionFromDataKey 

func VersionFromDataKey(key Key) (dvid.VersionID, error)

VersionFromDataKey returns a version ID from a data key.

Types

type Alias 

type Alias string

Alias is a nickname for a storage configuration, e.g., "raid6" for basholeveldb on RAID6 drives. It is used in the DVID TOML configuration file like [storage.alias]

type Backend 

type Backend struct {
	Metadata    Alias // The store that should be used for metadata storage.
	DefaultLog  Alias // The log that should be used by default.
	DefaultKVDB Alias // The key-value datastore that should be used by default.
	Stores      map[Alias]dvid.StoreConfig
	KVAssign    DataMap
	LogAssign   DataMap
	Groupcache  GroupcacheConfig
}

Backend provide data instance to store mappings gleaned from DVID configuration. Currently, a data instance can be mapped to one KV and one Log store.

func GetTestableBackend 

func GetTestableBackend(kvMap, logMap DataMap) (map[Alias]TestableEngine, *Backend, error)

GetTestableBackend returns a testable engine and backend.

type Batch 

type Batch interface {
	// Delete removes from the batch a put using the given key.
	Delete(TKey)

	// Put adds to the batch a put using the given key-value.
	Put(k TKey, v []byte)

	// Commits a batch of operations and closes the write batch.
	Commit() error
}

Batch groups operations into a transaction. Clear() and Close() were removed due to how other key-value stores implement batches. It's easier to implement cross-database handling of a simple write/delete batch that commits then closes rather than something that clears.

type BlobStore 

type BlobStore interface {
	PutBlob([]byte) (ref string, err error)
	GetBlob(ref string) ([]byte, error)
}

BlobStore allows writing and retrieving unversioned data. It differs from KeyValueDB in that it auto-generates and returns a key on writing, typically the content hash.

type Block 

type Block struct {
	*BlockOp
	Coord dvid.ChunkPoint3d
	Value []byte
}

Block is the unit passed down channels to chunk handlers. Chunks can be passed from lower-level database access functions to type-specific chunk processing.

type BlockFunc 

type BlockFunc func(*Block) error

BlockFunc is a function that accepts a Block.

type BlockOp 

type BlockOp struct {
	Op interface{}
	Wg *sync.WaitGroup
}

BlockOp is a type-specific operation with an optional WaitGroup to sync mapping before reduce.

type BufferableOps 

type BufferableOps interface {
	KeyValueSetter

	// Put key-value pairs.  Note that it could be more efficient to use the Batcher
	// interface so you don't have to create and keep a slice of KeyValue.  Some
	// databases like leveldb will copy on batch put anyway.
	PutRange(Context, []TKeyValue) error

	// DeleteRange removes all key-value pairs with keys in the given range.
	// If versioned data in mutable stores, this will create tombstones in the version
	// unlike RawDelete or DeleteAll.
	DeleteRange(ctx Context, kStart, kEnd TKey) error

	// ProcessRange will process all gets when flush is called
	ProcessRange(ctx Context, kStart, kEnd TKey, op *ChunkOp, f ChunkFunc) error
}

RequestBufferSubset implements a subset of the ordered key/value interface. It declares interface common to both ordered key value and RequestBuffer

type Chunk 

type Chunk struct {
	*ChunkOp
	*TKeyValue
}

Chunk is the unit passed down channels to chunk handlers. Chunks can be passed from lower-level database access functions to type-specific chunk processing.

type ChunkFunc 

type ChunkFunc func(*Chunk) error

ChunkFunc is a function that accepts a Chunk.

type ChunkOp 

type ChunkOp struct {
	Op interface{}
	Wg *sync.WaitGroup
}

ChunkOp is a type-specific operation with an optional WaitGroup to sync mapping before reduce.

type Context 

type Context interface {
	// VersionID returns the local version ID of the DAG node being operated on.
	// If not versioned, the version is the root ID.
	VersionID() dvid.VersionID

	// RepoRoot returns the root uuid.
	RepoRoot() (dvid.UUID, error)

	// ConstructKey takes a type-specific key component, and generates a
	// namespaced key that fits with the DVID-wide key space partitioning.
	ConstructKey(TKey) Key

	// ConstructKeyVersion constructs a key like ConstructKey
	// but using specified version
	ConstructKeyVersion(TKey, dvid.VersionID) Key

	// KeyRange returns the minimum and maximum keys for this context.
	KeyRange() (min, max Key)

	// String prints a description of the Context
	String() string

	// Returns a sync.Mutex specific to this context.
	Mutex() *sync.Mutex

	// Versioned is true if this Context is also a VersionedCtx.
	Versioned() bool

	// SplitKey returns key components useful to store all versiones in a familyColumn if the storage engine supports it
	SplitKey(tk TKey) (Key, []byte, error)
}

Context allows encapsulation of data that defines the partitioning of the DVID key space.

type DAGStoreGetter 

type DAGStoreGetter interface {
	// Get returns a value given a key.
	Get(ctx Context, k TKey) ([]byte, error)

	// GetRange returns a range of values spanning (kStart, kEnd) keys.
	GetRange(ctx Context, kStart, kEnd TKey) ([]*TKeyValue, error)

	// GetRangeStream sends range of values spanning (kStart, kEnd) key-values.
	GetRangeStream(ctx Context, kStart, kEnd TKey, ordered, keysOnly bool, out chan *KeyValue) error

	// ProcessRange sends a range of type key-value pairs to type-specific chunk handlers,
	// allowing chunk processing to be concurrent with key-value reads.
	// If the ChunkFunc returns an error, it is expected that the ProcessRange should
	// immediately terminate and propagate the error.
	ProcessRange(ctx Context, kStart, kEnd TKey, ordered bool, op *ChunkOp, f ChunkFunc) error
}

DAGStoreGetter provides ways to read key-value pairs individually and in groups, where the latter can also be ordered by key or more quickly streamed in unordered fashion.

type DataContext 

type DataContext struct {
	// contains filtered or unexported fields
}

DataContext supports both unversioned and versioned data persistence.

func NewDataContext 

func NewDataContext(data dvid.Data, versionID dvid.VersionID) *DataContext

NewDataContext provides a way for datatypes to create a Context that adheres to DVID key space partitioning. Since Context and VersionedCtx interfaces are opaque, i.e., can only be implemented within package storage, we force compatible implementations to embed DataContext and initialize it via this function.

func (*DataContext) ClientFromKey 

func (ctx *DataContext) ClientFromKey(key Key) (dvid.ClientID, error)

ClientFromKey returns a clientID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) ClientID 

func (ctx *DataContext) ClientID() dvid.ClientID

func (*DataContext) ConstructKey 

func (ctx *DataContext) ConstructKey(tk TKey) Key

func (*DataContext) ConstructKeyVersion 

func (ctx *DataContext) ConstructKeyVersion(tk TKey, version dvid.VersionID) Key

func (*DataContext) Data 

func (ctx *DataContext) Data() dvid.Data

func (*DataContext) DataName 

func (ctx *DataContext) DataName() dvid.InstanceName

func (*DataContext) GetRequestID 

func (ctx *DataContext) GetRequestID() string

GetRequestID returns a string identifier or the empty string if none have been set.

func (*DataContext) InstanceFromKey 

func (ctx *DataContext) InstanceFromKey(key Key) (dvid.InstanceID, error)

InstanceFromKey returns an InstanceID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) InstanceID 

func (ctx *DataContext) InstanceID() dvid.InstanceID

func (*DataContext) InstanceVersion 

func (ctx *DataContext) InstanceVersion() dvid.InstanceVersion

func (*DataContext) KeyRange 

func (ctx *DataContext) KeyRange() (min, max Key)

KeyRange returns the min and max full keys. The DataContext can have any version since min/max keys for a data instance is independent of the current context's version.

func (*DataContext) MaxVersionKey 

func (ctx *DataContext) MaxVersionKey(tk TKey) (Key, error)

Returns upper bound key for versions of given byte slice key representation.

func (*DataContext) MinVersionKey 

func (ctx *DataContext) MinVersionKey(tk TKey) (Key, error)

Returns lower bound key for versions of given byte slice key representation.

func (*DataContext) Mutex 

func (ctx *DataContext) Mutex() *sync.Mutex

func (*DataContext) RepoRoot 

func (ctx *DataContext) RepoRoot() (dvid.UUID, error)

func (*DataContext) RequestID 

func (ctx *DataContext) RequestID() string

func (*DataContext) SetRequestID 

func (ctx *DataContext) SetRequestID(id string)

SetRequestID sets a string identifier.

func (*DataContext) SplitKey 

func (ctx *DataContext) SplitKey(tk TKey) (Key, []byte, error)

Split the key in two parts: the first one call unversioned key, and the second one called versioned key

func (*DataContext) String 

func (ctx *DataContext) String() string

func (*DataContext) TKeyClassRange 

func (ctx *DataContext) TKeyClassRange(c TKeyClass) (min, max Key)

TKeyClassRange returns the min and max full keys for this class of TKeys.

func (*DataContext) TKeyRange 

func (ctx *DataContext) TKeyRange() (min, max TKey)

TKeyRange returns min and max type-specific keys. The max key is not guaranteed to be the theoretical maximum TKey but should be so for any TKey of 128 bytes or less. The DataContext can be nil.

func (*DataContext) TombstoneKey 

func (ctx *DataContext) TombstoneKey(tk TKey) Key

func (*DataContext) TombstoneKeyVersion 

func (ctx *DataContext) TombstoneKeyVersion(tk TKey, version dvid.VersionID) Key

func (*DataContext) UnversionedKey 

func (ctx *DataContext) UnversionedKey(tk TKey) (Key, dvid.VersionID, error)

UnversionedKey returns a unversioned Key and the version id as separate components. This can be useful for storage systems like column stores where the row key is the unversioned Key and the column qualifier is the version id.

func (*DataContext) UnversionedKeyPrefix 

func (ctx *DataContext) UnversionedKeyPrefix(tk TKey) Key

UnversionedKeyPrefix returns a byte slice that is the common prefix for all versions of this particular TKey.

func (*DataContext) UpdateInstance 

func (ctx *DataContext) UpdateInstance(k Key) error

func (*DataContext) ValidKV 

func (ctx *DataContext) ValidKV(kv *KeyValue, versions map[dvid.VersionID]struct{}) bool

ValidKV returns true if a key-value pair is in an allowed set of versions. A nil kv always returns true. An uninterpretable key returns false.

func (*DataContext) VersionFromKey 

func (ctx *DataContext) VersionFromKey(key Key) (dvid.VersionID, error)

VersionFromKey returns a version ID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) VersionID 

func (ctx *DataContext) VersionID() dvid.VersionID

func (*DataContext) Versioned 

func (ctx *DataContext) Versioned() bool

Versioned returns false. This can be overriden by embedding DataContext in structures that will support the VersionedCtx interface.

type DataMap 

type DataMap map[dvid.DataSpecifier]Alias

DataMap describes how data instances and types are mapped to available storage systems.

type DataSpec added in v1.0.0 

type DataSpec interface {
	DataName() dvid.InstanceName
	DataUUID() dvid.UUID
	RootUUID() dvid.UUID
	Tags() map[string]string
	TypeName() dvid.TypeString
}

DataSpec is an interface for properties of data instances useful for store assignment.

type Engine 

type Engine interface {
	fmt.Stringer

	// GetName returns a simple driver identifier like "basholeveldb", "kvautobus" or "bigtable".
	GetName() string

	// IsDistributed returns whether the engine is a distributed DB (engine should manage request throttling)
	IsDistributed() bool

	// GetSemVer returns the semantic versioning info.
	GetSemVer() semver.Version

	// NewStore returns a new storage engine given the passed configuration.
	// Should return true for initMetadata if the store needs initialization of metadata.
	NewStore(dvid.StoreConfig) (db dvid.Store, initMetadata bool, err error)
}

Engine is a storage engine that can create a storage instance, dvid.Store, which could be a database directory in the case of an embedded database Engine implementation. Engine implementations can fulfill a variety of interfaces, checkable by runtime cast checks, e.g., myGetter, ok := myEngine.(OrderedKeyValueGetter) Data types can throw a warning at init time if the backend doesn't support required interfaces, or they can choose to implement multiple ways of handling data. Each Engine implementation should call storage.Register() to register its availability.

func GetEngine 

func GetEngine(name string) Engine

GetEngine returns an Engine of the given name.

type Filter 

type Filter interface {
	// Check filters type-specific key-value pairs.
	Check(tkv *TKeyValue) (skip bool, err error)
}

Filter can filter key-value pairs based on some criteria.

type FilterSpec 

type FilterSpec string

FilterSpec is a string specification of type-specific filters to apply to key-value pairs before sending them to a remote DVID. For example, a FilterSpec could look like: 

roi:seven_column,3f8a/tile:xy,xz

The above specifies two filters joined by a forward slash. The first is an "roi" filters that lists a ROI data instance name ("seven_column") and its version as a partial, unique UUID. The second is a "tile" filter that specifies two types of tile plane: xy and xz.

func (FilterSpec) GetFilterSpec 

func (f FilterSpec) GetFilterSpec(ftype string) (value string, found bool)

GetFilterSpec parses a FilterSpec and returns the filter spec of given type. If no filter spec of ftype is available, the second argument is false.

type Filterer 

type Filterer interface {
	NewFilter(FilterSpec) (Filter, error)
}

Filterer is an interface that can provide a send filter given a spec. Datatypes can fulfill this interface if they want to filter key-values sent to peer DVID servers. An example is the use of ROIs to filter kv pairs based on spatial constraints.

See datatype/imageblk/distributed.go

type GridProps 

type GridProps struct {
	VolumeSize dvid.Point3d
	ChunkSize  dvid.Point3d
	Encoding   string     // "raw", "jpeg", or "compressed_segmentation"
	Resolution [3]float64 // resolution in nm for a voxel along dimensions
}

GridProps describes the properties of a GridStore. This matches neuroglancer precomputed volume specs.

type GridStoreGetter 

type GridStoreGetter interface {
	GridProperties(scaleLevel int) (GridProps, error)
	GridGet(scaleLevel int, blockCoord dvid.ChunkPoint3d) ([]byte, error)
	GridGetVolume(scaleLevel int, minBlock, maxBlock dvid.ChunkPoint3d, ordered bool, op *BlockOp, f BlockFunc) error
}

GridStoreGetter describes nD block getter functions

type GroupcacheConfig 

type GroupcacheConfig struct {
	GB        int
	Host      string   // The http address of this DVID server's groupcache port.
	Peers     []string // The http addresses of the peer groupcache group.
	Instances []string // Data instances that use groupcache in form "<name>:<uuid>""
}

GroupcacheConfig handles settings for the groupcache library.

type GroupcacheStats 

type GroupcacheStats struct {
	Gets           int64 // any GET request, including from peers
	CacheHits      int64 // either cache was good
	PeerLoads      int64 // either remote load or remote cache hit (not an error)
	PeerErrors     int64
	Loads          int64 // gets - cacheHits
	LoadsDeduped   int64 // after singleflight
	LocalLoads     int64 // total good local loads
	LocalLoadErrs  int64 // total bad local loads
	ServerRequests int64 // gets that came over network from peers

	// Cache stats for items owned by the host.
	MainCache groupcache.CacheStats

	// Cache stats for replicated items from a peer.
	HotCache groupcache.CacheStats
}

GroupcacheStats returns all the stats of interest for this groupcache service.

func GetGroupcacheStats 

func GetGroupcacheStats() (stats GroupcacheStats, err error)

GetGroupcacheStats returns all kinds of stats about the groupcache system.

type KafkaConfig 

type KafkaConfig struct {
	TopicActivity string   // if supplied, will be override topic for activity log
	TopicPrefix   string   // if supplied, will be prefixed to any mutation logging
	TopicSuffixes []string // optional topic suffixes per data UUID
	Servers       []string
	BufferSize    int // queue.buffering.max.messages
}

KafkaConfig describes kafka servers and an optional local file directory into which failed messages will be stored.

func (KafkaConfig) Initialize 

func (kc KafkaConfig) Initialize(hostID string) error

Initialize sets up default activity topic and support for on-the-fly mutation topics

type Key 

type Key []byte

Key is the slice of bytes used to store a value in a storage engine. It internally represents a number of DVID features like a data instance ID, version, and a type-specific key component.

func ConstructBlobKey 

func ConstructBlobKey(k []byte) Key

ConstructBlobKey returns a blob Key, partitioned from other key spaces, for a given key.

func DataInstanceKeyRange 

func DataInstanceKeyRange(d dvid.InstanceID) (minKey, maxKey Key)

DataInstanceKeyRange returns the min and max Key across all keys for a data instance.

func DataKeyRange 

func DataKeyRange() (minKey, maxKey Key)

DataKeyRange returns the min and max Key across all data keys.

func MaxDataKey 

func MaxDataKey() Key

func MaxVersionDataKey 

func MaxVersionDataKey(d dvid.InstanceID, tk TKey) (Key, error)

MaxVersionDataKey returns upper bound key for given data instance.

func MergeKey 

func MergeKey(unvKey Key, verKey []byte) Key

func MinDataKey 

func MinDataKey() Key

func (Key) IsBlobKey 

func (k Key) IsBlobKey() bool

IsBlobKey returns true if the given key is in the metadata keyspace (instead of data or blob).

func (Key) IsDataKey 

func (k Key) IsDataKey() bool

IsDataKey returns true if the given key is in the data keyspace (instead of metadata or blob).

func (Key) IsMetadataKey 

func (k Key) IsMetadataKey() bool

IsMetadataKey returns true if the given key is in the metadata keyspace (instead of data or blob).

func (Key) IsTombstone 

func (k Key) IsTombstone() bool

IsTombstone returns true if the given key is a tombstone key.

type KeyChan 

type KeyChan chan Key

KeyChan is a channel of full (not type-specific) keys.

type KeyRange 

type KeyRange struct {
	Start   Key // Range includes this Key.
	OpenEnd Key // Range extend to but does not include this Key.
}

KeyRange is a range of keys that is closed at the beginning and open at the end.

type KeyValue 

type KeyValue struct {
	K Key
	V []byte
}

KeyValue stores a full storage key-value pair.

type KeyValueBatcher 

type KeyValueBatcher interface {
	NewBatch(ctx Context) Batch
}

KeyValueBatcher allow batching operations into an atomic update or transaction. For example: "Atomic Updates" in http://leveldb.googlecode.com/svn/trunk/doc/index.html

type KeyValueDB 

type KeyValueDB interface {
	dvid.Store
	KeyValueGetter
	KeyValueSetter
}

KeyValueDB provides an interface to the simplest storage API: a key-value store.

func DefaultKVDB 

func DefaultKVDB() (KeyValueDB, error)

type KeyValueGetter 

type KeyValueGetter interface {
	// Get returns a value given a key.
	Get(ctx Context, k TKey) ([]byte, error)

	// Exists returns true if a key has been set.
	Exists(ctx Context, k TKey) (bool, error)
}

type KeyValueIngestable 

type KeyValueIngestable interface {
	// KeyValueIngest accepts mutations without any guarantee that the ingested key value
	// will be immediately readable.  This allows immutable stores to accept data that will
	// be later processed into its final readable, immutable form.
	KeyValueIngest(Context, TKey, []byte) error
}

KeyValueIngestable implementations allow ingestion of data without necessarily allowing immediate reads of the ingested data.

type KeyValueRequester 

type KeyValueRequester interface {
	NewBuffer(ctx Context) RequestBuffer
}

KeyValueRequester allows operations to be queued so that they can be handled as a batch job. (See RequestBuffer for more information.)

type KeyValueSetter 

type KeyValueSetter interface {
	// Put writes a value with given key in a possibly versioned context.
	Put(Context, TKey, []byte) error

	// Delete deletes a key-value pair so that subsequent Get on the key returns nil.
	// For versioned data in mutable stores, Delete() will create a tombstone for the version
	// unlike RawDelete or DeleteAll.
	Delete(Context, TKey) error

	// RawPut is a low-level function that puts a key-value pair using full keys.
	// This can be used in conjunction with RawRangeQuery.  It does not automatically
	// delete any associated tombstone, unlike the Delete() function, so tombstone
	// deletion must be handled via RawDelete().
	RawPut(Key, []byte) error

	// RawDelete is a low-level function.  It deletes a key-value pair using full keys
	// without any context.  This can be used in conjunction with RawRangeQuery.
	RawDelete(Key) error
}

type KeyValueTimestampGetter 

type KeyValueTimestampGetter interface {
	// Get returns a value as well as last modification time
	GetWithTimestamp(ctx Context, k TKey) ([]byte, time.Time, error)
}

type KeyValues 

type KeyValues []KeyValue

KeyValues is a slice of key-value pairs that can be sorted.

func (KeyValues) Len 

func (kv KeyValues) Len() int

func (KeyValues) Less 

func (kv KeyValues) Less(i, j int) bool

func (KeyValues) Swap 

func (kv KeyValues) Swap(i, j int)

type LogMessage 

type LogMessage struct {
	EntryType uint16
	Data      []byte
}

LogMessage is a single message to a log

type LogReadable 

type LogReadable interface {
	ReadLogRequired() bool
	GetReadLog() ReadLog
}

type LogWritable 

type LogWritable interface {
	WriteLogRequired() bool
	GetWriteLog() WriteLog
}

type Logable 

type Logable interface {
	LogReadable
	LogWritable
}

type MetadataContext 

type MetadataContext struct{}

MetadataContext is an implementation of Context for MetadataContext persistence.

func NewMetadataContext 

func NewMetadataContext() MetadataContext

func (MetadataContext) ConstructKey 

func (ctx MetadataContext) ConstructKey(tk TKey) Key

func (MetadataContext) ConstructKeyVersion 

func (ctx MetadataContext) ConstructKeyVersion(tk TKey, version dvid.VersionID) Key

Note: needed to satisfy interface but should not be called

func (MetadataContext) KeyRange 

func (ctx MetadataContext) KeyRange() (min, max Key)

func (MetadataContext) Mutex 

func (ctx MetadataContext) Mutex() *sync.Mutex

func (MetadataContext) RepoRoot 

func (ctx MetadataContext) RepoRoot() (dvid.UUID, error)

func (MetadataContext) RequestID 

func (ctx MetadataContext) RequestID() string

func (MetadataContext) SplitKey 

func (ctx MetadataContext) SplitKey(tk TKey) (Key, []byte, error)

func (MetadataContext) String 

func (ctx MetadataContext) String() string

func (MetadataContext) VersionID 

func (ctx MetadataContext) VersionID() dvid.VersionID

func (MetadataContext) Versioned 

func (ctx MetadataContext) Versioned() bool

type Op 

type Op uint8

Op enumerates the types of single key-value operations that can be performed for storage engines. 

const (
	GetOp Op = iota
	PutOp
	DeleteOp
	CommitOp
)

type OrderedKeyValueDB 

type OrderedKeyValueDB interface {
	dvid.Store
	OrderedKeyValueGetter
	OrderedKeyValueSetter
}

OrderedKeyValueDB addes range queries and range puts to a base KeyValueDB.

func DefaultOrderedKVDB 

func DefaultOrderedKVDB() (OrderedKeyValueDB, error)

func MetaDataKVStore 

func MetaDataKVStore() (OrderedKeyValueDB, error)

type OrderedKeyValueGetter 

type OrderedKeyValueGetter interface {
	KeyValueGetter

	// GetRange returns a range of values spanning (kStart, kEnd) keys.
	GetRange(ctx Context, kStart, kEnd TKey) ([]*TKeyValue, error)

	// KeysInRange returns a range of type-specific key components spanning (kStart, kEnd).
	KeysInRange(ctx Context, kStart, kEnd TKey) ([]TKey, error)

	// SendKeysInRange sends a range of keys down a key channel.
	SendKeysInRange(ctx Context, kStart, kEnd TKey, ch KeyChan) error

	// ProcessRange sends a range of type key-value pairs to type-specific chunk handlers,
	// allowing chunk processing to be concurrent with key-value sequential reads.
	// Since the chunks are typically sent during sequential read iteration, the
	// receiving function can be organized as a pool of chunk handling goroutines.
	// See datatype/imageblk.ProcessChunk() for an example.  If the ChunkFunc returns
	// an error, it is expected that the ProcessRange should immediately terminate and
	// propagate the error.
	ProcessRange(ctx Context, kStart, kEnd TKey, op *ChunkOp, f ChunkFunc) error

	// RawRangeQuery sends a range of full keys.  This is to be used for low-level data
	// retrieval like DVID-to-DVID communication and should not be used by data type
	// implementations if possible because each version's key-value pairs are sent
	// without filtering by the current version and its ancestor graph.  A nil is sent
	// down the channel when the range is complete.  The query can be cancelled by sending
	// a value down the cancel channel.
	RawRangeQuery(kStart, kEnd Key, keysOnly bool, out chan *KeyValue, cancel <-chan struct{}) error
}

type OrderedKeyValueSetter 

type OrderedKeyValueSetter interface {
	KeyValueSetter

	// Put key-value pairs.  Note that it could be more efficient to use the Batcher
	// interface so you don't have to create and keep a slice of KeyValue.  Some
	// databases like leveldb will copy on batch put anyway.
	PutRange(Context, []TKeyValue) error

	// DeleteRange removes all key-value pairs with keys in the given range.
	// If versioned data in mutable stores, this will create tombstones in the version
	// unlike RawDelete or DeleteAll.
	DeleteRange(ctx Context, kStart, kEnd TKey) error

	// DeleteAll removes all key-value pairs for the context.
	DeleteAll(ctx Context) error
}

type PatchFunc 

type PatchFunc func([]byte) ([]byte, error)

PatchFunc is a function that accepts a value and patches that value in place

type ReadLog 

type ReadLog interface {
	dvid.Store
	ReadBinary(dataID, version dvid.UUID) ([]byte, error)
	ReadAll(dataID, version dvid.UUID) ([]LogMessage, error)
	StreamAll(dataID, version dvid.UUID, ch chan LogMessage) error
}

type RepairableEngine 

type RepairableEngine interface {
	Engine
	Repair(path string) error
}

RepairableEngine is a storage engine that can be repaired.

type RequestBuffer 

type RequestBuffer interface {
	BufferableOps

	// ProcessList will process all gets when flush is called
	ProcessList(ctx Context, tkeys []TKey, op *ChunkOp, f ChunkFunc) error

	// PutCallback writes a value with given key in a possibly versioned context and signals the callback
	PutCallback(Context, TKey, []byte, chan error) error

	// Flush processes all the queued jobs
	Flush() error
}

RequestBuffer allows one to queue up several requests and then process them all as a batch operation (if the driver supports batch operations). There is no guarantee on the order or atomicity of these operations.

type RequestCtx 

type RequestCtx interface {
	// GetRequestID returns a string identifier or the empty string if none have been set.
	GetRequestID() string

	// SetRequestID sets a string identifier.
	SetRequestID(id string)
}

RequestCtx is associated with a particular request (typically a web request) and can set and retrieve information about it.

type Requirements 

type Requirements struct {
	BulkIniter bool
	BulkWriter bool
	Batcher    bool
}

Requirements lists required backend interfaces for a type.

type SizeViewer 

type SizeViewer interface {
	GetApproximateSizes(ranges []KeyRange) ([]uint64, error)
}

SizeViewer stores are able to return the size in bytes stored for a given range of Key.

type TKey 

type TKey []byte

TKey is the type-specific component of a key. Each data instance will insert key components into a class of TKey. Typically, the first byte is unique for a given class of keys within a datatype. For a given class of type-specific keys, they must either be (1) identical length or (2) if varying length, not share a common prefix. The second case can be trivially fulfilled by ending a type-specific key with a unique (not occurring within these type-specific keys) termination byte like 0x00.

func MaxTKey 

func MaxTKey(class TKeyClass) TKey

MaxTKey returns the lexicographically largest TKey for this class.

func MinTKey 

func MinTKey(class TKeyClass) TKey

MinTKey returns the lexicographically smallest TKey for this class.

func NewTKey 

func NewTKey(class TKeyClass, tkey []byte) TKey

func TKeyFromKey 

func TKeyFromKey(key Key) (TKey, error)

TKeyFromKey returns a type-specific key from a full key. An error is returned if the key is not one with a type-specific key component.

func (TKey) Class 

func (tk TKey) Class() (TKeyClass, error)

Class returns the TKeyClass of a TKey.

func (TKey) ClassBytes 

func (tk TKey) ClassBytes(class TKeyClass) ([]byte, error)

ClassBytes returns the bytes for a class of TKey, suitable for decoding by each data instance.

type TKeyClass 

type TKeyClass byte

TKeyClass partitions the TKey space into a maximum of 256 classes.

type TKeyValue 

type TKeyValue struct {
	K TKey
	V []byte
}

TKeyValue stores a type-specific key-value pair.

func (TKeyValue) Deserialize 

func (kv TKeyValue) Deserialize(uncompress bool) (TKeyValue, error)

Deserialize returns a type key-value pair where the value has been deserialized.

type TKeyValues 

type TKeyValues []TKeyValue

TKeyValues is a slice of type-specific key-value pairs that can be sorted.

func (TKeyValues) Len 

func (kv TKeyValues) Len() int

func (TKeyValues) Less 

func (kv TKeyValues) Less(i, j int) bool

func (TKeyValues) Swap 

func (kv TKeyValues) Swap(i, j int)

type TestableEngine 

type TestableEngine interface {
	Engine
	AddTestConfig(*Backend) (Alias, error)
	Delete(dvid.StoreConfig) error
}

TestableEngine is a storage engine that allows creation of temporary stores for testing.

type VersionedCtx 

type VersionedCtx interface {
	Context

	// UnversionedKeyPrefix returns a byte slice that is the common prefix
	// for all versions of this particular TKey.
	UnversionedKeyPrefix(TKey) Key

	// UnversionedKey returns a unversioned Key and the version id
	// as separate components.  This can be useful for storage systems
	// like column stores where the row key is the unversioned Key and
	// the column qualifier is the version id.
	UnversionedKey(TKey) (Key, dvid.VersionID, error)

	// VersionFromKey returns a version ID from a full key.  Any VersionedContext is sufficient as receiver
	VersionFromKey(Key) (dvid.VersionID, error)

	// TombstoneKey takes a type-specific key component and returns a key that
	// signals a deletion of any ancestor values.  The returned key must have
	// as its last byte storage.MarkTombstone.
	TombstoneKey(TKey) Key

	// TombstoneKeyVersion implement TombstoneKey but for the specified version
	TombstoneKeyVersion(TKey, dvid.VersionID) Key

	// Head checks whether this is the open head of the master branch
	Head() bool

	// MasterVersion checks whether current version is on master branch
	MasterVersion(dvid.VersionID) bool

	// NumVersions returns the number of version in the current DAG
	NumVersions() int32

	// Returns lower bound key for versions.
	MinVersionKey(TKey) (Key, error)

	// Returns upper bound key for versions.
	MaxVersionKey(TKey) (Key, error)

	// GetBestKeyVersion returns the key that most closely matches this context's version.
	// If no suitable key or a tombstone is encountered closed to the version, nil is returned.
	GetBestKeyVersion([]Key) (Key, error)

	// VersionedKeyValue returns the key-value pair corresponding to this key's version
	// given a list of key-value pairs across many versions.  If no suitable key-value
	// pair is found, nil is returned.
	VersionedKeyValue([]*KeyValue) (*KeyValue, error)

	// Data returns the data associated with this context
	Data() dvid.Data
}

VersionedCtx extends a Context with the minimal functions necessary to handle versioning in storage engines.

type WriteLog 

type WriteLog interface {
	dvid.Store
	Append(dataID, version dvid.UUID, msg LogMessage) error
	CloseLog(dataID, version dvid.UUID) error
	TopicAppend(topic string, msg LogMessage) error
	TopicClose(topic string) error
}

WriteLog is an append-only log of messages specific to a topic or a data instance and UUID.

Source Files

View all Source files

Directories

Path Synopsis
Package swift adds Openstack Swift support to DVID.
 Package swift adds Openstack Swift support to DVID.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.