storage

package
v0.0.0-...-e545f54 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 6, 2014 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Overview

Package storage provides a unified interface to a number of storage engines. Since each storage engine has different capabilities, this package defines a number of interfaces in addition to the core Engine interface, which all storage engines should satisfy.

Initially we are concentrating on key-value backends but expect to support graph and perhaps relational databases.

Each storage engine must implement the following package function:

func NewStore(path string, create bool, options *Options) (Engine, error)

Currently only one storage engine file is compiled through the use of build tags like "leveldb", "hyperleveldb", and "basholeveldb".

Index

Constants

View Source 
const MonitorBuffer = 10000

Variables

View Source 
var (
	// Number of bytes read in last second from storage engine.
	StoreKeyBytesReadPerSec int

	// Number of bytes written in last second to storage engine.
	StoreKeyBytesWrittenPerSec int

	// Number of bytes read in last second from storage engine.
	StoreValueBytesReadPerSec int

	// Number of bytes written in last second to storage engine.
	StoreValueBytesWrittenPerSec int

	// Number of bytes read in last second from file system.
	FileBytesReadPerSec int

	// Number of bytes written in last second to file system.
	FileBytesWrittenPerSec int

	// Number of key-value GET calls in last second.
	GetsPerSec int

	// Number of key-value PUT calls in last second.
	PutsPerSec int

	// Channel to notify bytes read from a storage engine.
	StoreKeyBytesRead chan int

	// Channel to notify bytes written to a storage engine.
	StoreKeyBytesWritten chan int

	// Channel to notify bytes read from a storage engine.
	StoreValueBytesRead chan int

	// Channel to notify bytes written to a storage engine.
	StoreValueBytesWritten chan int

	// Channel to notify bytes read from file system.
	FileBytesRead chan int

	// Channel to notify bytes written to file system.
	FileBytesWritten chan int
)

Functions

func DataFromFile 

func DataFromFile(filename string) ([]byte, error)

DataFromFile returns data from a file.

func GenerateInode 

func GenerateInode(dID dvid.DataLocalID, vID dvid.VersionLocalID, index dvid.Index) uint64

Inode numbers are uint64 which are composed from most to least significant bits: 1) uint16 describing local data ID 2) uint16 describing local version ID 3) uint32 fnv hash of the filename/key.

func OpenFUSE 

func OpenFUSE(dir string, data Mountable, vinfo VersionInfo) error

OpenFUSE mounts the given directory as a FUSE file system. The FUSE system is a singleton with only one FUSE server operable.

func Shutdown 

func Shutdown()

func ShutdownFUSE 

func ShutdownFUSE()

ShutdownFUSE shuts down any FUSE server that's running.

Types

type Batch 

type Batch interface {
	// Delete removes from the batch a put using the given key.
	Delete(k Key)

	// Put adds to the batch a put using the given key/value.
	Put(k Key, v []byte)

	// Commits a batch of operations and closes the write batch.
	Commit() error
}

Batch groups operations into a transaction. Clear() and Close() were removed due to how other key-value stores implement batches. It's easier to implement cross-database handling of a simple write/delete batch that commits then closes rather than something that clears.

type Batcher 

type Batcher interface {
	NewBatch() Batch
}

Batchers allow batching operations into an atomic update or transaction. For example: "Atomic Updates" in http://leveldb.googlecode.com/svn/trunk/doc/index.html

type BulkIniter 

type BulkIniter interface {
}

BulkIniters can employ even more aggressive optimization in loading large data since they can assume an uninitialized blank database.

type BulkWriter 

type BulkWriter interface {
}

BulkWriter employ some sort of optimization to efficiently write large amount of data. For some key-value databases, this requires keys to be presorted.

type Chunk 

type Chunk struct {
	*ChunkOp
	KeyValue
}

Chunk is the unit passed down channels to chunk handlers. Chunks can be passed from lower-level database access functions to type-specific chunk processing.

type ChunkOp 

type ChunkOp struct {
	Op interface{}
	Wg *sync.WaitGroup
}

ChunkOp is a type-specific operation with an optional WaitGroup to sync mapping before reduce.

type Data 

type Data []Mountable

Data is a slice of mountable data services for a particular version.

type Engine 

type Engine interface {
	GetName() string
	GetConfig() dvid.Config
	Close()
}

Engine implementations can fulfill a variety of interfaces and can be checked by runtime cast checks, e.g., myGetter, ok := myEngine.(OrderedKeyValueGetter) Data types can throw a warning at init time if the backend doesn't support required interfaces, or they can choose to implement multiple ways of handling data.

type FUSEDirer 

type FUSEDirer interface {
	Attr() fuse.Attr
	Lookup(string, fs.Intr) (fs.Node, fuse.Error)
	ReadDir(fs.Intr) ([]fuse.Dirent, fuse.Error)
}

FUSEDirer can return a type that implements FUSE directory functions.

type FUSEFiler 

type FUSEFiler interface {
	Attr() fuse.Attr
	ReadAll(fs.Intr) ([]byte, fuse.Error)
}

FUSEFiler can return a type that implements Node and Handle.

type FUSEHandler 

type FUSEHandler interface {
	FUSESystem() FUSERooter
}

type FUSERooter 

type FUSERooter interface {
	Root() (fs.Node, fuse.Error)
}

type FUSEServer 

type FUSEServer struct {
	// contains filtered or unexported fields
}

FUSEServer manages mounted UUIDs for specified data.

func (FUSEServer) Root 

func (s FUSEServer) Root() (fs.Node, fuse.Error)

type Key 

type Key interface {
	KeyType() KeyType
	BytesToKey([]byte) (Key, error)
	Bytes() []byte
	BytesString() string
	String() string
}

type KeyType 

type KeyType byte

KeyType is a portion of a serialized Key that partitions key space into distinct areas and makes sure keys don't conflict. 

const (
	// Key group that hold data for Datasets
	KeyDatasets KeyType = iota

	// Key group that holds Dataset structs.  There can be many Dataset structs
	// persisted to a particular DVID datastore.
	KeyDataset

	// Key group that holds the Data.  Each Datatype figures out how to partition
	// its own key space using some type-specific indexing scheme.
	KeyData

	// Key group that holds Sync links between Data.  Sync key/value pairs designate
	// what values need to be updated when its linked data changes.
	KeySync
)

func (KeyType) String 

func (t KeyType) String() string

type KeyValue 

type KeyValue struct {
	K Key
	V []byte
}

KeyValue stores a key-value pair.

func (KeyValue) Deserialize 

func (kv KeyValue) Deserialize(uncompress bool) (KeyValue, error)

Deserialize returns a key-value pair where the value has been deserialized.

type KeyValueDB 

type KeyValueDB interface {
	KeyValueGetter
	KeyValueSetter
}

KeyValueDB provides an interface to the simplest storage API: a key/value store.

type KeyValueGetter 

type KeyValueGetter interface {
	// Get returns a value given a key.
	Get(k Key) (v []byte, err error)
}

type KeyValueSetter 

type KeyValueSetter interface {
	// Put writes a value with given key.
	Put(k Key, v []byte) error

	// Delete removes an entry given key.
	Delete(k Key) error
}

type KeyValues 

type KeyValues []KeyValue

KeyValues is a slice of key-value pairs that can be sorted.

func (KeyValues) Len 

func (kv KeyValues) Len() int

func (KeyValues) Less 

func (kv KeyValues) Less(i, j int) bool

func (KeyValues) Swap 

func (kv KeyValues) Swap(i, j int)

type Mount 

type Mount struct {
	// contains filtered or unexported fields
}

func (*Mount) AddData 

func (m *Mount) AddData(d Mountable, vinfo VersionInfo) error

type MountDir 

type MountDir struct{}

MountDir implements both Node and Handle for the root mount directory.

func (MountDir) Attr 

func (MountDir) Attr() fuse.Attr

func (MountDir) Lookup 

func (MountDir) Lookup(name string, intr fs.Intr) (fs.Node, fuse.Error)

Lookup returns a mounted UUID Node.

func (MountDir) ReadDir 

func (MountDir) ReadDir(intr fs.Intr) ([]fuse.Dirent, fuse.Error)

ReadDir returns a list of version directories.

type Mountable 

type Mountable interface {
	// DataName returns the name of the data (e.g., grayscale data that is grayscale8 data type).
	DataName() dvid.DataString

	// LocalDataID returns the server-specific ID (fewer bytes) for this particular data.
	LocalID() dvid.DataLocalID

	// FUSEDir returns a data-specific implementation of a FUSE Directory
	FUSEDir(VersionInfo) FUSEDirer
}

Mountable is an interface for data that can be mounted as FUSE Files.

type Op 

type Op uint8

Op enumerates the types of single key-value operations that can be performed for storage engines. 

const (
	GetOp Op = iota
	PutOp
	DeleteOp
	CommitOp
)

type OrderedKeyValueDB 

type OrderedKeyValueDB interface {
	OrderedKeyValueGetter
	OrderedKeyValueSetter
}

OrderedKeyValueDB addes range queries and range puts to a base KeyValueDB.

type OrderedKeyValueGetter 

type OrderedKeyValueGetter interface {
	KeyValueGetter

	// GetRange returns a range of values spanning (kStart, kEnd) keys.
	GetRange(kStart, kEnd Key) (values []KeyValue, err error)

	// KeysInRange returns a range of keys spanning (kStart, kEnd).
	KeysInRange(kStart, kEnd Key) (keys []Key, err error)

	// ProcessRange sends a range of key/value pairs to type-specific chunk handlers,
	// allowing chunk processing to be concurrent with key/value sequential reads.
	// Since the chunks are typically sent during sequential read iteration, the
	// receiving function can be organized as a pool of chunk handling goroutines.
	// See datatype.voxels.ProcessChunk() for an example.
	ProcessRange(kStart, kEnd Key, op *ChunkOp, f func(*Chunk)) (err error)
}

type OrderedKeyValueSetter 

type OrderedKeyValueSetter interface {
	KeyValueSetter

	// Put key-value pairs.  Note that it could be more efficient to use the Batcher
	// interface so you don't have to create and keep a slice of KeyValue.  Some
	// databases like leveldb will copy on batch put anyway.
	PutRange(values []KeyValue) error
}

type Requirements 

type Requirements struct {
	BulkIniter bool
	BulkWriter bool
	Batcher    bool
}

Requirements lists required backend interfaces for a type.

type VersionDir 

type VersionDir struct {
	Mount
}

VersionDir implements both Node and Handle for the version directories. Each directory is specific to one data in a dataset.

func (VersionDir) Attr 

func (v VersionDir) Attr() fuse.Attr

func (VersionDir) Lookup 

func (v VersionDir) Lookup(name string, intr fs.Intr) (fs.Node, fuse.Error)

func (VersionDir) ReadDir 

func (v VersionDir) ReadDir(intr fs.Intr) ([]fuse.Dirent, fuse.Error)

ReadDir returns a list of mounted data for this version.

type VersionInfo 

type VersionInfo struct {
	// contains filtered or unexported fields
}

VersionInfo holds both local and global IDs for a node.

func NewVersionInfo 

func NewVersionInfo(uuid dvid.UUID, dID dvid.DatasetLocalID, vID dvid.VersionLocalID) VersionInfo

func (VersionInfo) GetDatasetID 

func (v VersionInfo) GetDatasetID() dvid.DatasetLocalID

func (VersionInfo) GetUUID 

func (v VersionInfo) GetUUID() dvid.UUID

func (VersionInfo) GetVersionID 

func (v VersionInfo) GetVersionID() dvid.VersionLocalID

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.