storage

package
v0.8.18 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2018 License: BSD-3-Clause Imports: 15 Imported by: 0

Documentation

Overview

Package storage provides a unified interface to a number of storage engines. Since each storage engine has different capabilities, this package defines a number of interfaces in addition to the core Engine interface, which all storage engines should satisfy.

Keys are specified as a combination of Context and a datatype-specific byte slice, typically called an "type-specific key" (TKey) in DVID docs and code. The Context provides DVID-wide namespacing and as such, must use one of the Context implementations within the storage package. (This is enforced by making Context a Go opaque interface.) The type-specific key formatting is entirely up to the datatype designer, although use of dvid.Index is suggested.

Initially we are concentrating on key-value backends but expect to support graph and perhaps relational databases, either using specialized databases or software layers on top of an ordered key-value store.

Although we assume lexicographically ordering for range queries, there is some variation in how variable size keys are treated. We assume all storage engines, after appropriate DVID drivers, use the following definition of ordering: 

A string s precedes a string t in lexicographic order if:

* s is a prefix of t, or
* if c and d are respectively the first character of s and t in which s and t differ,
  then c precedes d in character order.
* if s and t are equivalent for all of s, but t is longer

Note: For the characters that are alphabetical letters, the character order coincides
with the alphabetical order. Digits precede letters, and uppercase letters precede
lowercase ones.

Examples:

composer precedes computer
house precedes household
Household precedes house
H2O precedes HOTEL
mydex precedes mydexterity

Note that the above is different than shortlex order, which would group strings
based on length first.

The above lexicographical ordering is used by default for levedb variants.

Index

Constants

View Source 
const (
	// MarkData is a byte indicating real data stored and should be the last byte of any
	// versioned key.
	MarkData = 0x03

	// MarkTombstone is a byte indicating a tombstone -- a marker for deleted data.
	MarkTombstone = 0x4F
)
View Source 
const (
	TKeyMinClass = 0x00
	TKeyMaxClass = 0xFF
)
View Source 
const MonitorBuffer = 10000

Variables

View Source 
var (
	// Number of bytes read in last second from storage engine.
	StoreKeyBytesReadPerSec int

	// Number of bytes written in last second to storage engine.
	StoreKeyBytesWrittenPerSec int

	// Number of bytes read in last second from storage engine.
	StoreValueBytesReadPerSec int

	// Number of bytes written in last second to storage engine.
	StoreValueBytesWrittenPerSec int

	// Number of bytes read in last second from file system.
	FileBytesReadPerSec int

	// Number of bytes written in last second to file system.
	FileBytesWrittenPerSec int

	// Number of key-value GET calls in last second.
	GetsPerSec int

	// Number of key-value PUT calls in last second.
	PutsPerSec int

	// Channel to notify bytes read from a storage engine.
	StoreKeyBytesRead chan int

	// Channel to notify bytes written to a storage engine.
	StoreKeyBytesWritten chan int

	// Channel to notify bytes read from a storage engine.
	StoreValueBytesRead chan int

	// Channel to notify bytes written to a storage engine.
	StoreValueBytesWritten chan int

	// Channel to notify bytes read from file system.
	FileBytesRead chan int

	// Channel to notify bytes written to file system.
	FileBytesWritten chan int
)
View Source 
var ErrLogEOF = fmt.Errorf("log EOF")

ErrLogEOF is an error returned by a log Read() on an end of file

Functions

func AllStores added in v0.8.5 

func AllStores() (map[Alias]dvid.Store, error)

func DataFromFile 

func DataFromFile(filename string) ([]byte, error)

DataFromFile returns data from a file.

func DataKeyToLocalIDs 

func DataKeyToLocalIDs(k Key) (dvid.InstanceID, dvid.VersionID, dvid.ClientID, error)

KeyToLocalIDs parses a key under a DataContext and returns instance, version and client ids.

func DefaultKVStore added in v0.8.1 

func DefaultKVStore() (dvid.Store, error)

func DefaultLogStore added in v0.8.13 

func DefaultLogStore() (dvid.Store, error)

func DeleteDataInstance 

func DeleteDataInstance(data dvid.Data) error

DeleteDataInstance removes a data instance.

func EnginesAvailable 

func EnginesAvailable() string

EnginesAvailable returns a description of the available storage engines.

func GetAssignedLog added in v0.8.13 

func GetAssignedLog(dataname dvid.InstanceName, root dvid.UUID, tags map[string]string, typename dvid.TypeString) (dvid.Store, error)

GetAssignedLog returns the append-only log assigned based on (instance name, root uuid) or type.

func GetAssignedStore added in v0.8.1 

func GetAssignedStore(dataname dvid.InstanceName, root dvid.UUID, tags map[string]string, typename dvid.TypeString) (dvid.Store, error)

GetAssignedStore returns the store assigned based on (instance name, root uuid), tag, or type, in that order. In some cases, this store may include a caching wrapper if the data instance has been configured to use groupcache.

func GetDataSizes added in v0.8.5 

func GetDataSizes(store dvid.Store, instances []dvid.InstanceID) (map[dvid.InstanceID]uint64, error)

GetDataSizes returns a list of storage sizes in bytes for each data instance in the store. A list of InstanceID can be optionally supplied so only those instances are queried. This requires some scanning of the database so could take longer than normal requests, particularly if a list of instances is not given. Note that the underlying store must support both the OrderedKeyValueGetter and SizeViewer interface, else this function returns nil.

func GetStoreByAlias added in v0.8.2 

func GetStoreByAlias(alias Alias) (dvid.Store, error)

GetStoreByAlias returns a store by the alias given to it in the configuration TOML file, e.g., "raid6".

func Initialize 

func Initialize(cmdline dvid.Config, backend *Backend) (createdMetadata bool, err error)

Initialize the storage systems. Returns a bool + error where the bool is true if the metadata store is newly created and needs initialization. The map of store configurations should be keyed by either a datatype name, "default", or "metadata".

func KafkaProduceMsg added in v0.8.18 

func KafkaProduceMsg(msg []byte, topic string) error

KafkaProduceMsg sends a message to kafka

func NewStore added in v0.8.1 

func NewStore(c dvid.StoreConfig) (db dvid.Store, created bool, err error)

NewStore checks if a given engine is available and if so, returns a store opened with the configuration.

func RegisterEngine 

func RegisterEngine(e Engine)

RegisterEngine registers an Engine for DVID use.

func Repair 

func Repair(name, path string) error

Repair repairs a named engine's store at given path.

func Shutdown added in v0.8.17 

func Shutdown()

Shutdown handles any storage-specific shutdown procedures.

func SplitKey added in v0.8.0 

func SplitKey(k Key) (unversioned Key, versioned Key, err error)

SplitKey returns key components depending on whether the passed Key is a metadata or data key. If metadata, it returns the key and a 0 version id. If it is a data key, it returns the unversioned portion of the Key and the version id.

func UpdateDataKey 

func UpdateDataKey(k Key, instance dvid.InstanceID, version dvid.VersionID, client dvid.ClientID) error

Types

type Alias added in v0.8.2 

type Alias string

Alias is a nickname for a storage configuration, e.g., "raid6" for basholeveldb on RAID6 drives. It is used in the DVID TOML configuration file like [storage.alias]

type Backend added in v0.8.2 

type Backend struct {
	Metadata    Alias // The store that should be used for metadata storage.
	DefaultLog  Alias // The log that should be used by default.
	DefaultKVDB Alias // The key-value datastore that should be used by default.
	Stores      map[Alias]dvid.StoreConfig
	KVStore     DataMap
	LogStore    DataMap
	Groupcache  GroupcacheConfig
}

Backend provide data instance to store mappings gleaned from DVID configuration. Currently, a data instance can be mapped to one KV and one Log store.

func GetTestableBackend added in v0.8.13 

func GetTestableBackend(kvMap, logMap DataMap) (map[Alias]TestableEngine, *Backend, error)

GetTestableBackend returns testable enines and a storage backend that combines all testable engine configurations.

func (Backend) StoreConfig added in v0.8.2 

func (b Backend) StoreConfig(d dvid.DataSpecifier) (config dvid.StoreConfig, found bool)

StoreConfig returns a data specifier's assigned store configuration. The DataSpecifier can be "default" or "metadata" as well as datatype names and data instance specifications.

type Batch 

type Batch interface {
	// Delete removes from the batch a put using the given key.
	Delete(TKey)

	// Put adds to the batch a put using the given key-value.
	Put(k TKey, v []byte)

	// Commits a batch of operations and closes the write batch.
	Commit() error
}

Batch groups operations into a transaction. Clear() and Close() were removed due to how other key-value stores implement batches. It's easier to implement cross-database handling of a simple write/delete batch that commits then closes rather than something that clears.

type BlobStore added in v0.8.17 

type BlobStore interface {
	PutBlob([]byte) (ref string, err error)
	GetBlob(ref string) ([]byte, error)
}

BlobStore allows writing and retrieving unversioned data. It differs from KeyValueDB in that it auto-generates and returns a key on writing, typically the content hash.

type BufferableOps added in v0.8.0 

type BufferableOps interface {
	KeyValueSetter

	// Put key-value pairs.  Note that it could be more efficient to use the Batcher
	// interface so you don't have to create and keep a slice of KeyValue.  Some
	// databases like leveldb will copy on batch put anyway.
	PutRange(Context, []TKeyValue) error

	// DeleteRange removes all key-value pairs with keys in the given range.
	// If versioned data in mutable stores, this will create tombstones in the version
	// unlike RawDelete or DeleteAll.
	DeleteRange(ctx Context, kStart, kEnd TKey) error

	// ProcessRange will process all gets when flush is called
	ProcessRange(ctx Context, kStart, kEnd TKey, op *ChunkOp, f ChunkFunc) error
}

RequestBufferSubset implements a subset of the ordered key/value interface. It declares interface common to both ordered key value and RequestBuffer

type Chunk 

type Chunk struct {
	*ChunkOp
	*TKeyValue
}

Chunk is the unit passed down channels to chunk handlers. Chunks can be passed from lower-level database access functions to type-specific chunk processing.

type ChunkFunc 

type ChunkFunc func(*Chunk) error

ChunkFunc is a function that accepts a Chunk.

type ChunkOp 

type ChunkOp struct {
	Op interface{}
	Wg *sync.WaitGroup
}

ChunkOp is a type-specific operation with an optional WaitGroup to sync mapping before reduce.

type Context 

type Context interface {
	// VersionID returns the local version ID of the DAG node being operated on.
	// If not versioned, the version is the root ID.
	VersionID() dvid.VersionID

	// RepoRoot returns the root uuid.
	RepoRoot() (dvid.UUID, error)

	// ConstructKey takes a type-specific key component, and generates a
	// namespaced key that fits with the DVID-wide key space partitioning.
	ConstructKey(TKey) Key

	// ConstructKeyVersion constructs a key like ConstructKey
	// but using specified version
	ConstructKeyVersion(TKey, dvid.VersionID) Key

	// KeyRange returns the minimum and maximum keys for this context.
	KeyRange() (min, max Key)

	// String prints a description of the Context
	String() string

	// Returns a sync.Mutex specific to this context.
	Mutex() *sync.Mutex

	// Versioned is true if this Context is also a VersionedCtx.
	Versioned() bool

	// SplitKey returns key components useful to store all versiones in a familyColumn if the storage engine supports it
	SplitKey(tk TKey) (Key, []byte, error)
}

Context allows encapsulation of data that defines the partitioning of the DVID key space.

type DataContext 

type DataContext struct {
	// contains filtered or unexported fields
}

DataContext supports both unversioned and versioned data persistence.

func NewDataContext 

func NewDataContext(data dvid.Data, versionID dvid.VersionID) *DataContext

NewDataContext provides a way for datatypes to create a Context that adheres to DVID key space partitioning. Since Context and VersionedCtx interfaces are opaque, i.e., can only be implemented within package storage, we force compatible implementations to embed DataContext and initialize it via this function.

func (*DataContext) ClientFromKey 

func (ctx *DataContext) ClientFromKey(key Key) (dvid.ClientID, error)

ClientFromKey returns a clientID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) ClientID added in v0.8.2 

func (ctx *DataContext) ClientID() dvid.ClientID

func (*DataContext) ConstructKey 

func (ctx *DataContext) ConstructKey(tk TKey) Key

func (*DataContext) ConstructKeyVersion added in v0.8.13 

func (ctx *DataContext) ConstructKeyVersion(tk TKey, version dvid.VersionID) Key

func (*DataContext) Data added in v0.8.1 

func (ctx *DataContext) Data() dvid.Data

func (*DataContext) DataName 

func (ctx *DataContext) DataName() dvid.InstanceName

func (*DataContext) GetRequestID added in v0.8.0 

func (ctx *DataContext) GetRequestID() string

GetRequestID returns a string identifier or the empty string if none have been set.

func (*DataContext) InstanceFromKey added in v0.8.5 

func (ctx *DataContext) InstanceFromKey(key Key) (dvid.InstanceID, error)

InstanceFromKey returns an InstanceID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) InstanceID 

func (ctx *DataContext) InstanceID() dvid.InstanceID

func (*DataContext) InstanceVersion 

func (ctx *DataContext) InstanceVersion() dvid.InstanceVersion

func (*DataContext) KeyRange 

func (ctx *DataContext) KeyRange() (min, max Key)

KeyRange returns the min and max full keys. The DataContext can have any version since min/max keys for a data instance is independent of the current context's version.

func (*DataContext) MaxVersionKey 

func (ctx *DataContext) MaxVersionKey(tk TKey) (Key, error)

Returns upper bound key for versions of given byte slice key representation.

func (*DataContext) MinVersionKey 

func (ctx *DataContext) MinVersionKey(tk TKey) (Key, error)

Returns lower bound key for versions of given byte slice key representation.

func (*DataContext) Mutex 

func (ctx *DataContext) Mutex() *sync.Mutex

func (*DataContext) RepoRoot added in v0.8.13 

func (ctx *DataContext) RepoRoot() (dvid.UUID, error)

func (*DataContext) RequestID added in v0.8.0 

func (ctx *DataContext) RequestID() string

func (*DataContext) SetRequestID added in v0.8.0 

func (ctx *DataContext) SetRequestID(id string)

SetRequestID sets a string identifier.

func (*DataContext) SplitKey added in v0.8.0 

func (ctx *DataContext) SplitKey(tk TKey) (Key, []byte, error)

Split the key in two parts: the first one call unversioned key, and the second one called versioned key

func (*DataContext) String 

func (ctx *DataContext) String() string

func (*DataContext) TKeyClassRange added in v0.8.6 

func (ctx *DataContext) TKeyClassRange(c TKeyClass) (min, max Key)

TKeyClassRange returns the min and max full keys for this class of TKeys.

func (*DataContext) TKeyRange added in v0.8.1 

func (ctx *DataContext) TKeyRange() (min, max TKey)

TKeyRange returns min and max type-specific keys. The max key is not guaranteed to be the theoretical maximum TKey but should be so for any TKey of 128 bytes or less. The DataContext can be nil.

func (*DataContext) TombstoneKey 

func (ctx *DataContext) TombstoneKey(tk TKey) Key

func (*DataContext) TombstoneKeyVersion added in v0.8.13 

func (ctx *DataContext) TombstoneKeyVersion(tk TKey, version dvid.VersionID) Key

func (*DataContext) UnversionedKey added in v0.7.1 

func (ctx *DataContext) UnversionedKey(tk TKey) (Key, dvid.VersionID, error)

UnversionedKey returns a unversioned Key and the version id as separate components. This can be useful for storage systems like column stores where the row key is the unversioned Key and the column qualifier is the version id.

func (*DataContext) UpdateInstance added in v0.8.2 

func (ctx *DataContext) UpdateInstance(k Key) error

func (*DataContext) ValidKV added in v0.8.1 

func (ctx *DataContext) ValidKV(kv *KeyValue, versions map[dvid.VersionID]struct{}) bool

ValidKV returns true if a key-value pair is in an allowed set of versions. A nil kv always returns true. An uninterpretable key returns false.

func (*DataContext) VersionFromKey 

func (ctx *DataContext) VersionFromKey(key Key) (dvid.VersionID, error)

VersionFromKey returns a version ID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) VersionID 

func (ctx *DataContext) VersionID() dvid.VersionID

func (*DataContext) Versioned 

func (ctx *DataContext) Versioned() bool

Versioned returns false. This can be overriden by embedding DataContext in structures that will support the VersionedCtx interface.

type DataMap added in v0.8.18 

type DataMap map[dvid.DataSpecifier]Alias

DataMap describes how data instances and types are mapped to available storage systems.

type Engine 

type Engine interface {
	fmt.Stringer

	// GetName returns a simple driver identifier like "basholeveldb", "kvautobus" or "bigtable".
	GetName() string

	// IsDistributed returns whether the engine is a distributed DB (engine should manage request throttling)
	IsDistributed() bool

	// GetSemVer returns the semantic versioning info.
	GetSemVer() semver.Version

	// NewStore returns a new storage engine given the passed configuration.
	// Should return true for initMetadata if the store needs initialization of metadata.
	NewStore(dvid.StoreConfig) (db dvid.Store, initMetadata bool, err error)
}

Engine is a storage engine that can create a storage instance, dvid.Store, which could be a database directory in the case of an embedded database Engine implementation. Engine implementations can fulfill a variety of interfaces, checkable by runtime cast checks, e.g., myGetter, ok := myEngine.(OrderedKeyValueGetter) Data types can throw a warning at init time if the backend doesn't support required interfaces, or they can choose to implement multiple ways of handling data. Each Engine implementation should call storage.Register() to register its availability.

func GetEngine 

func GetEngine(name string) Engine

GetEngine returns an Engine of the given name.

type Filter added in v0.8.2 

type Filter interface {
	// Check filters type-specific key-value pairs.
	Check(tkv *TKeyValue) (skip bool, err error)
}

Filter can filter key-value pairs based on some criteria.

type FilterSpec added in v0.8.2 

type FilterSpec string

FilterSpec is a string specification of type-specific filters to apply to key-value pairs before sending them to a remote DVID. For example, a FilterSpec could look like: 

roi:seven_column,3f8a/tile:xy,xz

The above specifies two filters joined by a forward slash. The first is an "roi" filters that lists a ROI data instance name ("seven_column") and its version as a partial, unique UUID. The second is a "tile" filter that specifies two types of tile plane: xy and xz.

func (FilterSpec) GetFilterSpec added in v0.8.2 

func (f FilterSpec) GetFilterSpec(ftype string) (value string, found bool)

GetFilterSpec parses a FilterSpec and returns the filter spec of given type. If no filter spec of ftype is available, the second argument is false.

type Filterer added in v0.8.2 

type Filterer interface {
	NewFilter(FilterSpec) (Filter, error)
}

Filterer is an interface that can provide a send filter given a spec. Datatypes can fulfill this interface if they want to filter key-values sent to peer DVID servers. An example is the use of ROIs to filter kv pairs based on spatial constraints.

See datatype/imageblk/distributed.go

type GraphDB 

type GraphDB interface {
	GraphSetter
	GraphGetter
	Close()
}

GraphDB defines the entire interface that a graph database should support

func GraphStore 

func GraphStore() (GraphDB, error)

func NewGraphStore added in v0.8.2 

func NewGraphStore(kvdb OrderedKeyValueDB) (GraphDB, error)

NewGraphStore returns a graph backend that uses the provided keyvalue datastore. If a first-class graph store is used in the future, this function must be changed.

type GraphGetter 

type GraphGetter interface {
	// GetVertices retrieves a list of all vertices in the graph
	GetVertices(ctx Context) ([]dvid.GraphVertex, error)
	// GetEdges retrieves a list of all edges in the graph
	GetEdges(ctx Context) ([]dvid.GraphEdge, error)
	// GetVertex retrieves a vertex given a vertex id
	GetVertex(ctx Context, id dvid.VertexID) (dvid.GraphVertex, error)
	// GetVertex retrieves an edges between two vertex IDs
	GetEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) (dvid.GraphEdge, error)
	// GetVertexProperty retrieves a property as a byte array given a vertex id
	GetVertexProperty(ctx Context, id dvid.VertexID, key string) ([]byte, error)
	// GetEdgeProperty retrieves a property as a byte array given an edge defined by id1 and id2
	GetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) ([]byte, error)
}

GraphGetter defines operations that retrieve information from a graph

type GraphKeyValueDB added in v0.8.2 

type GraphKeyValueDB struct {
	OrderedKeyValueDB
	// contains filtered or unexported fields
}

GraphKeyValueDB defines a type that embeds a KeyValueDB using that engine to store all graph objects

func (*GraphKeyValueDB) AddEdge added in v0.8.2 

func (db *GraphKeyValueDB) AddEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

AddEdge reads both vertices, modifies the vertex edge lists, and then creates an edge (2 read ops, 3 write ops)

func (*GraphKeyValueDB) AddVertex added in v0.8.2 

func (db *GraphKeyValueDB) AddVertex(ctx Context, id dvid.VertexID, weight float64) error

AddVertex requires one key-value put

func (*GraphKeyValueDB) Close added in v0.8.2 

func (db *GraphKeyValueDB) Close()

Close does nothing, explicitly close the key value DB instead

func (*GraphKeyValueDB) CreateGraph added in v0.8.2 

func (db *GraphKeyValueDB) CreateGraph(ctx Context) error

CreateGraph does nothing as the graph keyspace uniquely defines a graph when using a single key-value datastore

func (*GraphKeyValueDB) GetEdge added in v0.8.2 

func (db *GraphKeyValueDB) GetEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) (dvid.GraphEdge, error)

GetEdge performs 1 read

func (*GraphKeyValueDB) GetEdgeProperty added in v0.8.2 

func (db *GraphKeyValueDB) GetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) ([]byte, error)

GetEdgeProperty performs 1 read (property name with vertex ids encoded in key)

func (*GraphKeyValueDB) GetEdges added in v0.8.2 

func (db *GraphKeyValueDB) GetEdges(ctx Context) ([]dvid.GraphEdge, error)

GetEdges uses a range query to get all edges (#reads = #edges)

func (*GraphKeyValueDB) GetVertex added in v0.8.2 

func (db *GraphKeyValueDB) GetVertex(ctx Context, id dvid.VertexID) (dvid.GraphVertex, error)

GetVertex performs 1 read

func (*GraphKeyValueDB) GetVertexProperty added in v0.8.2 

func (db *GraphKeyValueDB) GetVertexProperty(ctx Context, id dvid.VertexID, key string) ([]byte, error)

GetVertexProperty performs 1 read (property name with vertex id encoded in key)

func (*GraphKeyValueDB) GetVertices added in v0.8.2 

func (db *GraphKeyValueDB) GetVertices(ctx Context) ([]dvid.GraphVertex, error)

GetVertices uses a range query to get all vertices (#reads = #vertices)

func (*GraphKeyValueDB) RemoveEdge added in v0.8.2 

func (db *GraphKeyValueDB) RemoveEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) error

RemoveEdge removes the edge and all of its properties and modifies affected vertices (2 read, 3 + num properties writes)

func (*GraphKeyValueDB) RemoveEdgeProperty added in v0.8.2 

func (db *GraphKeyValueDB) RemoveEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) error

RemoveEdgeProperty retrieves edge and batch removes property and modifies edge (1 read, 2 writes)

func (*GraphKeyValueDB) RemoveGraph added in v0.8.2 

func (db *GraphKeyValueDB) RemoveGraph(ctx Context) error

RemoveGraph retrieves all keys with the graph prefix and deletes them

func (*GraphKeyValueDB) RemoveVertex added in v0.8.2 

func (db *GraphKeyValueDB) RemoveVertex(ctx Context, id dvid.VertexID) error

RemoveVertex removes the vertex and all of its edges and properties (1 read, 1 + num edges + num properties writes)

func (*GraphKeyValueDB) RemoveVertexProperty added in v0.8.2 

func (db *GraphKeyValueDB) RemoveVertexProperty(ctx Context, id dvid.VertexID, key string) error

RemoveVertexProperty retrieves vertex and batch removes property and modifies vertex (1 read, 2 writes)

func (*GraphKeyValueDB) SetEdgeProperty added in v0.8.2 

func (db *GraphKeyValueDB) SetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string, value []byte) error

SetEdgeProperty modifies the edge and adds a property vertex key (1 read, 2 writes)

func (*GraphKeyValueDB) SetEdgeWeight added in v0.8.2 

func (db *GraphKeyValueDB) SetEdgeWeight(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

SetEdgeWeight requires 1 read, 1 write

func (*GraphKeyValueDB) SetVertexProperty added in v0.8.2 

func (db *GraphKeyValueDB) SetVertexProperty(ctx Context, id dvid.VertexID, key string, value []byte) error

SetVertexProperty modifies the vertex and adds a property vertex key (1 read, 2 writes)

func (*GraphKeyValueDB) SetVertexWeight added in v0.8.2 

func (db *GraphKeyValueDB) SetVertexWeight(ctx Context, id dvid.VertexID, weight float64) error

SetVertexWeight requires 1 read, 1 write

func (*GraphKeyValueDB) String added in v0.8.2 

func (db *GraphKeyValueDB) String() string

GetName returns the name of theengien

type GraphSetter 

type GraphSetter interface {
	// CreateGraph creates a graph with the given context.
	CreateGraph(ctx Context) error

	// AddVertex inserts an id of a given weight into the graph
	AddVertex(ctx Context, id dvid.VertexID, weight float64) error

	// AddEdge adds an edge between vertex id1 and id2 with the provided weight
	AddEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

	// SetVertexWeight modifies the weight of vertex id
	SetVertexWeight(ctx Context, id dvid.VertexID, weight float64) error

	// SetEdgeWeight modifies the weight of the edge defined by id1 and id2
	SetEdgeWeight(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

	// SetVertexProperty adds arbitrary data to a vertex using a string key
	SetVertexProperty(ctx Context, id dvid.VertexID, key string, value []byte) error
	// SetEdgeProperty adds arbitrary data to an edge using a string key
	SetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string, value []byte) error
	// RemoveVertex removes the vertex and its properties and edges
	RemoveVertex(ctx Context, id dvid.VertexID) error
	// RemoveEdge removes the edge defined by id1 and id2 and its properties
	RemoveEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) error
	// RemoveGraph removes the entire graph including all vertices, edges, and properties
	RemoveGraph(ctx Context) error
	// RemoveVertexProperty removes the property data for vertex id at the key
	RemoveVertexProperty(ctx Context, id dvid.VertexID, key string) error
	// RemoveEdgeProperty removes the property data for edge at the key
	RemoveEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) error
}

GraphSetter defines operations that modify a graph

type GroupcacheConfig added in v0.8.4 

type GroupcacheConfig struct {
	GB        int
	Host      string   // The http address of this DVID server's groupcache port.
	Peers     []string // The http addresses of the peer groupcache group.
	Instances []string // Data instances that use groupcache in form "<name>:<uuid>""
}

GroupcacheConfig handles settings for the groupcache library.

type GroupcacheCtx added in v0.8.4 

type GroupcacheCtx struct {
	Context
	KeyValueDB
}

GroupcacheCtx packages both a storage.Context and a KeyValueDB for GETs.

type GroupcacheStats added in v0.8.4 

type GroupcacheStats struct {
	Gets           int64 // any GET request, including from peers
	CacheHits      int64 // either cache was good
	PeerLoads      int64 // either remote load or remote cache hit (not an error)
	PeerErrors     int64
	Loads          int64 // gets - cacheHits
	LoadsDeduped   int64 // after singleflight
	LocalLoads     int64 // total good local loads
	LocalLoadErrs  int64 // total bad local loads
	ServerRequests int64 // gets that came over network from peers

	// Cache stats for items owned by the host.
	MainCache groupcache.CacheStats

	// Cache stats for replicated items from a peer.
	HotCache groupcache.CacheStats
}

GroupcacheStats returns all the stats of interest for this groupcache service.

func GetGroupcacheStats added in v0.8.4 

func GetGroupcacheStats() (stats GroupcacheStats, err error)

GetGroupcacheStats returns all kinds of stats about the groupcache system.

type KafkaConfig added in v0.8.18 

type KafkaConfig struct {
	Servers []string
}

KafkaConfig describes kafka servers and an optional local file directory into which failed messages will be stored.

func (*KafkaConfig) Initialize added in v0.8.18 

func (c *KafkaConfig) Initialize() error

Initialize intializes kafka connection

type Key 

type Key []byte

Key is the slice of bytes used to store a value in a storage engine. It internally represents a number of DVID features like a data instance ID, version, and a type-specific key component.

func ConstructBlobKey added in v0.8.17 

func ConstructBlobKey(k []byte) Key

ConstructBlobKey returns a blob Key, partitioned from other key spaces, for a given key.

func MergeKey added in v0.8.0 

func MergeKey(unvKey Key, verKey []byte) Key

func (Key) IsDataKey added in v0.8.17 

func (k Key) IsDataKey() bool

IsDataKey returns true if the given key is in the data keyspace (instead of metadata or blob).

func (Key) IsTombstone 

func (k Key) IsTombstone() bool

IsTombstone returns true if the given key is a tombstone key.

type KeyChan 

type KeyChan chan Key

KeyChan is a channel of full (not type-specific) keys.

type KeyRange added in v0.8.5 

type KeyRange struct {
	Start   Key // Range includes this Key.
	OpenEnd Key // Range extend to but does not include this Key.
}

KeyRange is a range of keys that is closed at the beginning and open at the end.

type KeyValue 

type KeyValue struct {
	K Key
	V []byte
}

KeyValue stores a full storage key-value pair.

type KeyValueBatcher 

type KeyValueBatcher interface {
	NewBatch(ctx Context) Batch
}

KeyValueBatcher allow batching operations into an atomic update or transaction. For example: "Atomic Updates" in http://leveldb.googlecode.com/svn/trunk/doc/index.html

type KeyValueDB 

type KeyValueDB interface {
	dvid.Store
	KeyValueGetter
	KeyValueSetter
}

KeyValueDB provides an interface to the simplest storage API: a key-value store.

func DefaultKVDB added in v0.8.13 

func DefaultKVDB() (KeyValueDB, error)

type KeyValueGetter 

type KeyValueGetter interface {
	// Get returns a value given a key.
	Get(ctx Context, k TKey) ([]byte, error)
}

type KeyValueIngestable added in v0.8.1 

type KeyValueIngestable interface {
	// KeyValueIngest accepts mutations without any guarantee that the ingested key value
	// will be immediately readable.  This allows immutable stores to accept data that will
	// be later processed into its final readable, immutable form.
	KeyValueIngest(Context, TKey, []byte) error
}

KeyValueIngestable implementations allow ingestion of data without necessarily allowing immediate reads of the ingested data.

type KeyValueRequester added in v0.8.0 

type KeyValueRequester interface {
	NewBuffer(ctx Context) RequestBuffer
}

KeyValueRequester allows operations to be queued so that they can be handled as a batch job. (See RequestBuffer for more information.)

type KeyValueSetter 

type KeyValueSetter interface {
	// Put writes a value with given key in a possibly versioned context.
	Put(Context, TKey, []byte) error

	// Delete deletes a key-value pair so that subsequent Get on the key returns nil.
	// For versioned data in mutable stores, Delete() will create a tombstone for the version
	// unlike RawDelete or DeleteAll.
	Delete(Context, TKey) error

	// RawPut is a low-level function that puts a key-value pair using full keys.
	// This can be used in conjunction with RawRangeQuery.  It does not automatically
	// delete any associated tombstone, unlike the Delete() function, so tombstone
	// deletion must be handled via RawDelete().
	RawPut(Key, []byte) error

	// RawDelete is a low-level function.  It deletes a key-value pair using full keys
	// without any context.  This can be used in conjunction with RawRangeQuery.
	RawDelete(Key) error
}

type KeyValueTimestampGetter added in v0.8.18 

type KeyValueTimestampGetter interface {
	// Get returns a value as well as last modification time
	GetWithTimestamp(ctx Context, k TKey) ([]byte, time.Time, error)
}

type LogMessage added in v0.8.17 

type LogMessage struct {
	EntryType uint16
	Data      []byte
}

LogMessage is a single message to a log

type LogReadable added in v0.8.17 

type LogReadable interface {
	GetReadLog() ReadLog
}

type LogWritable added in v0.8.17 

type LogWritable interface {
	GetWriteLog() WriteLog
}

type Logable added in v0.8.13 

type Logable interface {
	LogReadable
	LogWritable
}

type MetadataContext 

type MetadataContext struct{}

MetadataContext is an implementation of Context for MetadataContext persistence.

func NewMetadataContext 

func NewMetadataContext() MetadataContext

func (MetadataContext) ConstructKey 

func (ctx MetadataContext) ConstructKey(tk TKey) Key

func (MetadataContext) ConstructKeyVersion added in v0.8.13 

func (ctx MetadataContext) ConstructKeyVersion(tk TKey, version dvid.VersionID) Key

Note: needed to satisfy interface but should not be called

func (MetadataContext) KeyRange 

func (ctx MetadataContext) KeyRange() (min, max Key)

func (MetadataContext) Mutex 

func (ctx MetadataContext) Mutex() *sync.Mutex

func (MetadataContext) RepoRoot added in v0.8.13 

func (ctx MetadataContext) RepoRoot() (dvid.UUID, error)

func (MetadataContext) RequestID added in v0.8.0 

func (ctx MetadataContext) RequestID() string

func (MetadataContext) SplitKey added in v0.8.0 

func (ctx MetadataContext) SplitKey(tk TKey) (Key, []byte, error)

func (MetadataContext) String 

func (ctx MetadataContext) String() string

func (MetadataContext) VersionID 

func (ctx MetadataContext) VersionID() dvid.VersionID

func (MetadataContext) Versioned 

func (ctx MetadataContext) Versioned() bool

type Op 

type Op uint8

Op enumerates the types of single key-value operations that can be performed for storage engines. 

const (
	GetOp Op = iota
	PutOp
	DeleteOp
	CommitOp
)

type OrderedKeyValueDB 

type OrderedKeyValueDB interface {
	dvid.Store
	OrderedKeyValueGetter
	OrderedKeyValueSetter
}

OrderedKeyValueDB addes range queries and range puts to a base KeyValueDB.

func DefaultOrderedKVDB added in v0.8.13 

func DefaultOrderedKVDB() (OrderedKeyValueDB, error)

func MetaDataKVStore added in v0.8.1 

func MetaDataKVStore() (OrderedKeyValueDB, error)

type OrderedKeyValueGetter 

type OrderedKeyValueGetter interface {
	KeyValueGetter

	// GetRange returns a range of values spanning (kStart, kEnd) keys.
	GetRange(ctx Context, kStart, kEnd TKey) ([]*TKeyValue, error)

	// KeysInRange returns a range of type-specific key components spanning (kStart, kEnd).
	KeysInRange(ctx Context, kStart, kEnd TKey) ([]TKey, error)

	// SendKeysInRange sends a range of keys down a key channel.
	SendKeysInRange(ctx Context, kStart, kEnd TKey, ch KeyChan) error

	// ProcessRange sends a range of type key-value pairs to type-specific chunk handlers,
	// allowing chunk processing to be concurrent with key-value sequential reads.
	// Since the chunks are typically sent during sequential read iteration, the
	// receiving function can be organized as a pool of chunk handling goroutines.
	// See datatype/imageblk.ProcessChunk() for an example.  If the ChunkFunc returns
	// an error, it is expected that the ProcessRange should immediately terminate and
	// propagate the error.
	ProcessRange(ctx Context, kStart, kEnd TKey, op *ChunkOp, f ChunkFunc) error

	// RawRangeQuery sends a range of full keys.  This is to be used for low-level data
	// retrieval like DVID-to-DVID communication and should not be used by data type
	// implementations if possible because each version's key-value pairs are sent
	// without filtering by the current version and its ancestor graph.  A nil is sent
	// down the channel when the range is complete.  The query can be cancelled by sending
	// a value down the cancel channel.
	RawRangeQuery(kStart, kEnd Key, keysOnly bool, out chan *KeyValue, cancel <-chan struct{}) error
}

type OrderedKeyValueSetter 

type OrderedKeyValueSetter interface {
	KeyValueSetter

	// Put key-value pairs.  Note that it could be more efficient to use the Batcher
	// interface so you don't have to create and keep a slice of KeyValue.  Some
	// databases like leveldb will copy on batch put anyway.
	PutRange(Context, []TKeyValue) error

	// DeleteRange removes all key-value pairs with keys in the given range.
	// If versioned data in mutable stores, this will create tombstones in the version
	// unlike RawDelete or DeleteAll.
	DeleteRange(ctx Context, kStart, kEnd TKey) error

	// DeleteAll removes all key-value pairs for the context.  If allVersions is true,
	// then all versions of the data instance are deleted.
	DeleteAll(ctx Context, allVersions bool) error
}

type PatchFunc added in v0.8.13 

type PatchFunc func([]byte) ([]byte, error)

PatchFunc is a function that accepts a value and patches that value in place

type ReadLog added in v0.8.13 

type ReadLog interface {
	dvid.Store
	ReadAll(dataID, version dvid.UUID) ([]LogMessage, error)
	StreamAll(dataID, version dvid.UUID, ch chan LogMessage, wg *sync.WaitGroup) error
}

type RepairableEngine 

type RepairableEngine interface {
	Engine
	Repair(path string) error
}

RepairableEngine is a storage engine that can be repaired.

type RequestBuffer added in v0.8.0 

type RequestBuffer interface {
	BufferableOps

	// ProcessList will process all gets when flush is called
	ProcessList(ctx Context, tkeys []TKey, op *ChunkOp, f ChunkFunc) error

	// PutCallback writes a value with given key in a possibly versioned context and signals the callback
	PutCallback(Context, TKey, []byte, chan error) error

	// Flush processes all the queued jobs
	Flush() error
}

RequestBuffer allows one to queue up several requests and then process them all as a batch operation (if the driver supports batch operations). There is no guarantee on the order or atomicity of these operations.

type RequestCtx added in v0.8.0 

type RequestCtx interface {
	// GetRequestID returns a string identifier or the empty string if none have been set.
	GetRequestID() string

	// SetRequestID sets a string identifier.
	SetRequestID(id string)
}

RequestCtx is associated with a particular request (typically a web request) and can set and retrieve information about it.

type Requirements 

type Requirements struct {
	BulkIniter bool
	BulkWriter bool
	Batcher    bool
	GraphDB    bool
}

Requirements lists required backend interfaces for a type.

type SizeViewer added in v0.8.5 

type SizeViewer interface {
	GetApproximateSizes(ranges []KeyRange) ([]uint64, error)
}

SizeViewer stores are able to return the size in bytes stored for a given range of Key.

type TKey 

type TKey []byte

TKey is the type-specific component of a key. Each data instance will insert key components into a class of TKey. Typically, the first byte is unique for a given class of keys within a datatype. For a given class of type-specific keys, they must either be (1) identical length or (2) if varying length, not share a common prefix. The second case can be trivially fulfilled by ending a type-specific key with a unique (not occuring within these type-specific keys) termination byte like 0x00.

func MaxTKey 

func MaxTKey(class TKeyClass) TKey

MaxTKey returns the lexicographically largest TKey for this class.

func MinTKey 

func MinTKey(class TKeyClass) TKey

MinTKey returns the lexicographically smallest TKey for this class.

func NewTKey 

func NewTKey(class TKeyClass, tkey []byte) TKey

func TKeyFromKey added in v0.8.2 

func TKeyFromKey(key Key) (TKey, error)

TKeyFromKey returns a type-specific key from a full key. An error is returned if the key is not one with a type-specific key component.

func (TKey) Class added in v0.8.2 

func (tk TKey) Class() (TKeyClass, error)

Class returns the TKeyClass of a TKey.

func (TKey) ClassBytes 

func (tk TKey) ClassBytes(class TKeyClass) ([]byte, error)

ClassBytes returns the bytes for a class of TKey, suitable for decoding by each data instance.

type TKeyClass 

type TKeyClass byte

TKeyClass partitions the TKey space into a maximum of 256 classes.

type TKeyValue 

type TKeyValue struct {
	K TKey
	V []byte
}

TKeyValue stores a type-specific key-value pair.

func (TKeyValue) Deserialize 

func (kv TKeyValue) Deserialize(uncompress bool) (TKeyValue, error)

Deserialize returns a type key-value pair where the value has been deserialized.

type TKeyValues 

type TKeyValues []TKeyValue

TKeyValues is a slice of type-specific key-value pairs that can be sorted.

func (TKeyValues) Len 

func (kv TKeyValues) Len() int

func (TKeyValues) Less 

func (kv TKeyValues) Less(i, j int) bool

func (TKeyValues) Swap 

func (kv TKeyValues) Swap(i, j int)

type TestableEngine 

type TestableEngine interface {
	Engine
	AddTestConfig(*Backend) (Alias, error)
	Delete(dvid.StoreConfig) error
}

TestableEngine is a storage engine that allows creation of temporary stores for testing.

type TransactionDB added in v0.8.13 

type TransactionDB interface {
	// LockKey uses atomic get/put to use the key as a shared lock.
	// If the lock is being used, the lock will be queried with some backoff.
	LockKey(Key) error

	// Release lock on key
	UnlockKey(Key) error

	// Patch patches the value at the given key with function f
	// The patching function should work on uninitialized data.
	Patch(Context, TKey, PatchFunc) error
}

TransactionDB allows multiple database operations to execute as an atomic transaction

type VersionedCtx 

type VersionedCtx interface {
	Context

	// UnversionedKey returns a unversioned Key and the version id
	// as separate components.  This can be useful for storage systems
	// like column stores where the row key is the unversioned Key and
	// the column qualifier is the version id.
	UnversionedKey(TKey) (Key, dvid.VersionID, error)

	// VersionFromKey returns a version ID from a full key.  Any VersionedContext is sufficient as receiver
	VersionFromKey(Key) (dvid.VersionID, error)

	// TombstoneKey takes a type-specific key component and returns a key that
	// signals a deletion of any ancestor values.  The returned key must have
	// as its last byte storage.MarkTombstone.
	TombstoneKey(TKey) Key

	// TombstoneKeyVersion implement TombstoneKey but for the specified version
	TombstoneKeyVersion(TKey, dvid.VersionID) Key

	// Head checks whether this the open head of the master branch
	Head() bool

	// MasterVersion checks whether current version is on master branch
	MasterVersion(dvid.VersionID) bool

	// NumVersions returns the number of version in the current DAG
	NumVersions() int32
	// Returns lower bound key for versions.
	MinVersionKey(TKey) (Key, error)

	// Returns upper bound key for versions.
	MaxVersionKey(TKey) (Key, error)

	// VersionedKeyValue returns the key-value pair corresponding to this key's version
	// given a list of key-value pairs across many versions.  If no suitable key-value
	// pair is found, nil is returned.
	VersionedKeyValue([]*KeyValue) (*KeyValue, error)

	// Data returns the data associated with this context
	Data() dvid.Data
}

VersionedCtx extends a Context with the minimal functions necessary to handle versioning in storage engines.

type WriteLog added in v0.8.13 

type WriteLog interface {
	dvid.Store
	Append(dataID, version dvid.UUID, msg LogMessage) error
	CloseLog(dataID, version dvid.UUID) error
	TopicAppend(topic string, msg LogMessage) error
	TopicClose(topic string) error
}

WriteLog is an append-only log of messages specific to a topic or a data instance and UUID.

Source Files

View all Source files

Directories

Path Synopsis
Package swift adds Openstack Swift support to DVID.
 Package swift adds Openstack Swift support to DVID.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.