datastore

package
v0.8.18 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2018 License: BSD-3-Clause Imports: 20 Imported by: 0

Documentation

Overview

Package datastore provides versioning and persisting supported data types using one of the supported storage engines.

Index

Constants

View Source 
const (
	StartDataMsg = "datastore.startData"
	PutKVMsg     = "datastore.putKV"
)
View Source 
const DataShutdownTime = 20

DataShutdownTime is the maximum number of seconds a data instance can delay when terminating goroutines during Shutdown.

View Source 
const RepoFormatVersion = 1

The current repo metadata format version

View Source 
const (
	ServerLockKey storage.TKeyClass // name of key for locking metadata globally
)

Key space handling for metadata

View Source 
const (
	Version = "0.10.0"
)

Variables

View Source 
var (
	ErrManagerNotInitialized = errors.New("datastore repo manager not initialized")
	ErrBadMergeType          = errors.New("bad merge type")

	ErrInvalidUUID         = errors.New("UUID is not present in datastore")
	ErrInvalidVersion      = errors.New("server-specific version id is invalid")
	ErrInvalidRepoID       = errors.New("server-specific repo id is invalid")
	ErrExistingUUID        = errors.New("UUID already exists in datastore")
	ErrExistingDataName    = errors.New("data name already exists")
	ErrInvalidDataName     = errors.New("invalid data instance name")
	ErrInvalidDataInstance = errors.New("invalid data instance id")
	ErrInvalidDataUUID     = errors.New("invalid data instance UUID")

	ErrInvalidStore = errors.New("invalid data store")

	ErrModifyLockedNode   = errors.New("can't modify locked node")
	ErrBranchUnlockedNode = errors.New("can't branch an unlocked node")
	ErrBranchUnique       = errors.New("branch already exists with given name")
)
View Source 
var (
	// Compiled is the set of registered datatypes compiled into DVID and
	// held as a global variable initialized at runtime.
	Compiled map[dvid.URLString]TypeService
)

Functions

func AddToNodeLog 

func AddToNodeLog(uuid dvid.UUID, msgs []string) error

func AddToRepoLog 

func AddToRepoLog(uuid dvid.UUID, msgs []string) error

func BlockOnUpdating added in v0.8.4 

func BlockOnUpdating(uuid dvid.UUID, name dvid.InstanceName) error

BlockOnUpdating blocks until the given data is not updating from syncs or has events waiting in sync channels. Primarily used during testing.

func CloseReopenTest 

func CloseReopenTest(datamap ...DataStorageMap)

CloseReopenTest forces close and then reopening of the datastore, useful for testing persistence. We only allow close/reopen when all tests not avaiting close/reopen are finished.

func CloseTest 

func CloseTest()

func Commit 

func Commit(uuid dvid.UUID, note string, log []string) error

func CompiledChart 

func CompiledChart() string

CompiledChart returns a chart (names/urls) of datatypes compiled into this DVID.

func CompiledTypes added in v0.8.0 

func CompiledTypes() map[dvid.TypeString]TypeService

CompiledTypes returns a map of datatypes compiled into this DVID.

func CompiledURLs 

func CompiledURLs() string

CompiledURLs returns a list of datatype urls supported by this DVID.

func CopyInstance added in v0.8.2 

func CopyInstance(uuid dvid.UUID, source, target dvid.InstanceName, c dvid.Config) error

CopyInstance copies a data instance locally, perhaps to a different storage engine if the new instance uses a different backend per a data instance-specific configuration. (See sample config.example.toml file in root dvid source directory.)

func DeleteConflicts 

func DeleteConflicts(uuid dvid.UUID, data DataService, oldParents, newParents []dvid.UUID) error

DeleteConflicts removes all conflicted kv pairs for the given data instance using the priority established by parents. As a side effect, newParents are modified by new children of parents.

func DeleteDataByName added in v0.8.3 

func DeleteDataByName(uuid dvid.UUID, name dvid.InstanceName, passcode string) error

DeleteDataByName returns a data service given an instance name and UUID.

func DeleteDataByVersion 

func DeleteDataByVersion(v dvid.VersionID, name dvid.InstanceName, passcode string) error

DeleteDataByVersion returns a data service given an instance name and UUID.

func DeleteRepo 

func DeleteRepo(uuid dvid.UUID, passcode string) error

DeleteRepo deletes a Repo holding a node with UUID.

func GetAncestry added in v0.8.17 

func GetAncestry(v dvid.VersionID) ([]dvid.VersionID, error)

GetAncestry returns a list of ancestor versions from the given version to the root.

func GetBlobStore added in v0.8.17 

func GetBlobStore(d dvid.Data) (store storage.BlobStore, err error)

GetBlobStore returns a blob store or nil if not available. TODO -- Add configurable option to use a file system.

func GetChildrenByVersion 

func GetChildrenByVersion(v dvid.VersionID) ([]dvid.VersionID, error)

GetChildren returns the child nodes of the given version id.

func GetGraphDB added in v0.8.17 

func GetGraphDB(d dvid.Data) (db storage.GraphDB, err error)

GetGraphDB returns a graph store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetKeyValueBatcher added in v0.8.17 

func GetKeyValueBatcher(d dvid.Data) (db storage.KeyValueBatcher, err error)

GetKeyValueBatcher returns a batch-capable kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetKeyValueDB added in v0.8.17 

func GetKeyValueDB(d dvid.Data) (db storage.KeyValueDB, err error)

GetKeyValueDB returns a kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetNodeLog 

func GetNodeLog(uuid dvid.UUID) ([]string, error)

func GetNodeNote added in v0.8.7 

func GetNodeNote(uuid dvid.UUID) (string, error)

func GetOrderedKeyValueDB added in v0.8.17 

func GetOrderedKeyValueDB(d dvid.Data) (db storage.OrderedKeyValueDB, err error)

GetOrderedKeyValueDB returns the ordered kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetParentsByVersion 

func GetParentsByVersion(v dvid.VersionID) ([]dvid.VersionID, error)

GetParents returns the parent nodes of the given version id.

func GetRepoAlias 

func GetRepoAlias(uuid dvid.UUID) (string, error)

func GetRepoDescription 

func GetRepoDescription(uuid dvid.UUID) (string, error)

func GetRepoJSON 

func GetRepoJSON(uuid dvid.UUID) (string, error)

func GetRepoLog 

func GetRepoLog(uuid dvid.UUID) ([]string, error)

func GetRepoRoot 

func GetRepoRoot(uuid dvid.UUID) (dvid.UUID, error)

func GetRepoRootVersion 

func GetRepoRootVersion(v dvid.VersionID) (dvid.VersionID, error)

func GetStorageBreakdown added in v0.8.5 

func GetStorageBreakdown() (string, error)

GetStorageBreakdown returns JSON for all the data instances in the stores.

func Initialize 

func Initialize(initMetadata bool, iconfig InstanceConfig) error

Initialize creates a repositories manager that is handled through package functions.

func LockedUUID 

func LockedUUID(uuid dvid.UUID) (bool, error)

LockedUUID returns true if a given UUID is locked.

func LockedVersion 

func LockedVersion(v dvid.VersionID) (bool, error)

LockedVersion returns true if a given version is locked.

func LogRepoOpToKafka added in v0.8.18 

func LogRepoOpToKafka(uuid dvid.UUID, b []byte) error

LogRepoOpToKafka logs a repo operation to kafka

func MarshalJSON 

func MarshalJSON() ([]byte, error)

MarshalJSON returns JSON of object where each repo is a property with root UUID name and value corresponding to repo info.

func MatchingUUID 

func MatchingUUID(uuidStr string) (dvid.UUID, dvid.VersionID, error)

MatchingUUID returns version identifiers that uniquely matches a uuid string.

func Merge 

func Merge(parents []dvid.UUID, note string, mt MergeType) (dvid.UUID, error)

func MetadataUniversalLock added in v0.8.13 

func MetadataUniversalLock() error

MetadataUniversalLock locks shared databases (currently those implementing transactions)

func MetadataUniversalUnlock added in v0.8.13 

func MetadataUniversalUnlock()

MetadataUniversalUnlock releases the shared lock

func MigrateInstance added in v0.8.2 

func MigrateInstance(uuid dvid.UUID, source dvid.InstanceName, oldStore dvid.Store, c dvid.Config) error

MigrateInstance migrates a data instance locally from an old storage engine to the current configured storage. After completion of the copy, the data instance in the old storage is deleted.

func ModifyDataConfigByName added in v0.8.3 

func ModifyDataConfigByName(uuid dvid.UUID, name dvid.InstanceName, c dvid.Config) error

func NewRepo 

func NewRepo(alias, description string, assign *dvid.UUID, passcode string) (dvid.UUID, error)

NewRepo creates a new Repo and returns its UUID, either an assigned UUID if provided or creating a new UUID.

func NewTestRepo 

func NewTestRepo() (dvid.UUID, dvid.VersionID)

NewTestRepo returns a new datastore.Repo suitable for testing.

func NewUUID 

func NewUUID(assign *dvid.UUID) (dvid.UUID, dvid.VersionID, error)

func NewVersion 

func NewVersion(parent dvid.UUID, note string, branchname string, assign *dvid.UUID) (dvid.UUID, error)

NewVersion creates a new version as a child of the given parent. If the assign parameter is not nil, the new node is given the UUID.

func NotifySubscribers 

func NotifySubscribers(e SyncEvent, m SyncMessage) error

NotifySubscribers sends a message to any data instances subscribed to the event.

func OpenTest 

func OpenTest(datamap ...DataStorageMap)

func PushData added in v0.8.2 

func PushData(d dvid.Data, p *PushSession) error

PushData transfers all key-value pairs pertinent to the given data instance. Each datatype can implement filters that can restrict the transmitted key-value pairs based on the given FilterSpec. Note that because of the generality of this function, a particular datatype implementation could be much more efficient when implementing filtering. For example, the imageblk datatype could scan its key-values using the ROI to generate keys (since imageblk keys will likely be a vast superset of ROI spans), while this generic routine will scan every key-value pair for a data instance and query the ROI to see if this key is ok to send.

func PushRepo added in v0.8.2 

func PushRepo(uuid dvid.UUID, target string, config dvid.Config) error

PushRepo pushes a Repo to a remote DVID server at the target address.

func Register 

func Register(t TypeService)

Register registers a datatype for DVID use.

func ReloadMetadata added in v0.8.0 

func ReloadMetadata() error

ReloadMetadata reloads the repositories manager from an existing metadata store. This shuts off all other dvid requests.

func RenameData added in v0.8.3 

func RenameData(uuid dvid.UUID, oldname, newname dvid.InstanceName, passcode string) error

RenameData renames a data service given an old instance name and UUID.

func SaveDataByUUID 

func SaveDataByUUID(uuid dvid.UUID, data DataService) error

SaveDataByUUID persists metadata for a data instance with given uuid. TODO -- Make this more efficient by storing data metadata separately from repo. 

Currently we save entire repo.

func SaveDataByVersion 

func SaveDataByVersion(v dvid.VersionID, data DataService) error

SaveDataByVersion persists metadata for a data instance with given version. TODO -- Make this more efficient by storing data metadata separately from repo. 

Currently we save entire repo.

func SetNodeNote 

func SetNodeNote(uuid dvid.UUID, note string) error

func SetRepoAlias 

func SetRepoAlias(uuid dvid.UUID, alias string) error

func SetRepoDescription 

func SetRepoDescription(uuid dvid.UUID, desc string) error

func SetSyncByJSON added in v0.8.3 

func SetSyncByJSON(d dvid.Data, uuid dvid.UUID, replace bool, in io.ReadCloser) error

SetSyncByJSON takes a JSON object of sync names and UUID, and creates the sync graph and sets the data instance's sync. If replace is false (default), the new sync is appended to the current syncs.

func SetSyncData added in v0.8.3 

func SetSyncData(data dvid.Data, syncs dvid.UUIDSet, replace bool) error

SetSyncData modfies the manager sync graphs and data instance's sync list. If replace is false (default), the new sync is appended to the current syncs.

func Shutdown 

func Shutdown()

Shutdown sends signal for all goroutines for data processing to be terminated.

func Types 

func Types() (map[dvid.URLString]TypeService, error)

Types returns the types currently within the DVID server.

func UUIDFromVersion 

func UUIDFromVersion(v dvid.VersionID) (dvid.UUID, error)

func VersionFromUUID 

func VersionFromUUID(uuid dvid.UUID) (dvid.VersionID, error)

Types

type BlobService added in v0.8.17 

type BlobService interface {
	GetBlob(ref string) (data []byte, err error)
	PutBlob(data []byte) (ref string, err error)
}

BlobService is an interface for storing and retrieving data based on its content.

type CommitSyncer 

type CommitSyncer interface {
	// SyncOnCommit is an asynchronous function that should be called when a node is committed.
	SyncOnCommit(dvid.UUID, dvid.VersionID)
}

CommitSyncer want to be notified when a node is committed.

type Data 

type Data struct {
	// contains filtered or unexported fields
}

Data is the base struct of repo-specific data instances. It should be embedded in a datatype's DataService implementation and handle datastore-wide key partitioning.

func NewDataService 

func NewDataService(t TypeService, rootUUID dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewDataService returns a new Data instance that fulfills the DataService interface. The UUID passed in corresponds to the root UUID of the DAG subgraph that should hold the data. This returned Data struct is usually embedded by datatype-specific data instances. By default, LZ4 and the default checksum is used.

func (*Data) Checksum 

func (d *Data) Checksum() dvid.Checksum

func (*Data) Compression 

func (d *Data) Compression() dvid.Compression

func (*Data) DAGRootUUID added in v0.8.13 

func (d *Data) DAGRootUUID() (dvid.UUID, error)

func (*Data) DataName 

func (d *Data) DataName() dvid.InstanceName

func (*Data) DataUUID added in v0.8.3 

func (d *Data) DataUUID() dvid.UUID

func (*Data) Equals 

func (d *Data) Equals(d2 *Data) bool

Equals returns true if the two data instances are identical, not just referring to the same data instance (e.g., different names but same data UUID, etc).

func (*Data) GetBlob added in v0.8.17 

func (d *Data) GetBlob(ref string) (data []byte, err error)

GetBlob retrieves data given a reference that was received during PutBlob.

func (*Data) GetReadLog added in v0.8.17 

func (d *Data) GetReadLog() storage.ReadLog

func (*Data) GetType 

func (d *Data) GetType() TypeService

func (*Data) GetWriteLog added in v0.8.13 

func (d *Data) GetWriteLog() storage.WriteLog

func (*Data) GobDecode 

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode 

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InstanceID 

func (d *Data) InstanceID() dvid.InstanceID

func (*Data) IsDeleted added in v0.8.18 

func (d *Data) IsDeleted() bool

IsDeleted returns true if data has been deleted or is deleting.

func (*Data) IsMutationRequest added in v0.8.5 

func (d *Data) IsMutationRequest(action, endpoint string) bool

IsMutationRequest is the default definition of mutation requests. All POST, PUT, and DELETE actions return true, while others return false.

func (*Data) KVStore added in v0.8.13 

func (d *Data) KVStore() (dvid.Store, error)

func (*Data) MarshalJSON 

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) ModifyConfig 

func (d *Data) ModifyConfig(config dvid.Config) error

func (*Data) MutAdd added in v0.8.5 

func (d *Data) MutAdd(mutID uint64) (newOp bool)

MutAdd adds a wait to this operation. Returns true if this is a new operation for this Data

func (*Data) MutDelete added in v0.8.5 

func (d *Data) MutDelete(mutID uint64)

MutDelete removes a wait for this operation. Should only be done after all MutWait() have completed.

func (*Data) MutDone added in v0.8.5 

func (d *Data) MutDone(mutID uint64)

MutDone marks the end of operations corresponding to MutAdd.

func (*Data) MutWait added in v0.8.5 

func (d *Data) MutWait(mutID uint64)

MutWait blocks until all operations with the given ID are completed.

func (*Data) NewMutationID added in v0.8.5 

func (d *Data) NewMutationID() uint64

func (*Data) ProduceKafkaMsg added in v0.8.17 

func (d *Data) ProduceKafkaMsg(b []byte) error

func (*Data) PushData added in v0.8.2 

func (d *Data) PushData(p *PushSession) error

PushData is the base implementation of pushing data instance key-value pairs to a remote DVID without any datatype-specific filtering of data.

func (*Data) PutBlob added in v0.8.17 

func (d *Data) PutBlob(b []byte) (ref string, err error)

PutBlob stores data and returns a reference to that data.

func (*Data) RootUUID added in v0.8.3 

func (d *Data) RootUUID() dvid.UUID

func (*Data) RootVersionID added in v0.8.7 

func (d *Data) RootVersionID() (dvid.VersionID, error)

func (*Data) SetDataUUID added in v0.8.3 

func (d *Data) SetDataUUID(uuid dvid.UUID)

func (*Data) SetDeleted added in v0.8.18 

func (d *Data) SetDeleted(deleted bool)

SetDeleted is used to indicate whether data has been deleted or is deleting.

func (*Data) SetInstanceID 

func (d *Data) SetInstanceID(id dvid.InstanceID)

func (*Data) SetKVStore added in v0.8.13 

func (d *Data) SetKVStore(kvStore dvid.Store)

func (*Data) SetLogStore added in v0.8.13 

func (d *Data) SetLogStore(logStore storage.WriteLog)

func (*Data) SetName added in v0.8.3 

func (d *Data) SetName(name dvid.InstanceName)

func (*Data) SetRootUUID added in v0.8.3 

func (d *Data) SetRootUUID(uuid dvid.UUID)

func (*Data) SetSync added in v0.8.0 

func (d *Data) SetSync(syncs dvid.UUIDSet)

SetSync sets the list of synced UUIDs for this data instance. It does not modify the sync graph.

func (*Data) SyncPending added in v0.8.5 

func (d *Data) SyncPending() bool

SyncPending returns true if there are outstanding sync events in this instance's subscription.

func (*Data) SyncedData added in v0.8.3 

func (d *Data) SyncedData() dvid.UUIDSet

SyncedData returns a set of data UUIDs to which it is synced.

func (*Data) SyncedNames 

func (d *Data) SyncedNames() []dvid.InstanceName

SyncedNames returns a set of data instance names to which it is synced. Legacy and will be removed after metadata refactor.

func (*Data) Tags added in v0.8.17 

func (d *Data) Tags() map[string]string

func (*Data) TypeName 

func (d *Data) TypeName() dvid.TypeString

func (*Data) TypeURL 

func (d *Data) TypeURL() dvid.URLString

func (*Data) TypeVersion 

func (d *Data) TypeVersion() string

func (*Data) Versioned 

func (d *Data) Versioned() bool

type DataAvail 

type DataAvail uint8

DataAvail gives the availability of data within a node or whether parent nodes must be traversed to check for key-value pairs. 

const (
	// For any query, we must also traverse this node's ancestors in the DAG
	// up to any DataComplete ancestor.  Used if a node is marked as archived.
	DataDelta DataAvail = iota

	// All key-value pairs are available within this node.
	DataComplete

	// Queries are redirected to Root since this is unversioned.
	DataRoot

	// Data has been explicitly deleted at this node and is no longer available.
	DataDeleted
)

func (DataAvail) String 

func (avail DataAvail) String() string

type DataInitializer added in v0.8.5 

type DataInitializer interface {
	InitDataHandlers() error
}

DataInitializer is a data instance that needs to be initialized, e.g., start long-lived goroutines that handle data syncs, etc. Initialization should only constitute supporting data and goroutines and not change the data itself like InstanceMutator, so no saveNeeded flag needs to be returned.

type DataService 

type DataService interface {
	dvid.Data
	json.Marshaler
	BlobService

	Help() string
	GetType() TypeService

	// ModifyConfig modifies a configuration in a type-specific way.
	ModifyConfig(dvid.Config) error

	// IsMutationRequest returns true if the given HTTP method on the endpoint results in mutations.
	// This allows datatypes to define actions on an endpoint as immutable even if they use POST.
	// The endpoint is the keyword immediately following a data instance name in the URL:
	//    /api/node/483f/grayscale/raw/xy/...
	// In the above URL, "raw" is the endpoint.
	IsMutationRequest(action, endpoint string) bool

	// DoRPC handles command line and RPC commands specific to a data type
	DoRPC(Request, *Response) error

	// ServeHTTP handles HTTP requests in the context of a particular version.
	ServeHTTP(dvid.UUID, *VersionedCtx, http.ResponseWriter, *http.Request)

	// PushData handles DVID-to-DVID transmission of key-value pairs, optionally
	// delimited by type-specific filter specifications (e.g., "roi:roiname,uuid")
	// and a set of versions.  DataService implementations can automatically embed
	// this via datastore.Data or can add filters by providing their own PushData(),
	// Filterer, and Filter implementations.  (datatype/imageblk/distributed.go)
	PushData(*PushSession) error
}

DataService is an interface for operations on an instance of a supported datatype.

func GetDataByDataUUID added in v0.8.3 

func GetDataByDataUUID(dataUUID dvid.UUID) (DataService, error)

GetDataByDataUUID returns a data service given a data UUID.

func GetDataByUUIDName added in v0.8.3 

func GetDataByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (DataService, error)

GetDataByUUIDName returns a data service given an instance name and UUID.

func GetDataByVersionName added in v0.8.3 

func GetDataByVersionName(v dvid.VersionID, name dvid.InstanceName) (DataService, error)

GetDataByVersionName returns a data service given an instance name and version.

func NewData 

func NewData(uuid dvid.UUID, t TypeService, name dvid.InstanceName, c dvid.Config) (DataService, error)

NewData adds a new, named instance of a datatype to repo. Settings can be passed via the 'config' argument. For example, config["versioned"] with a bool value will specify whether the data is versioned.

type DataStorageMap added in v0.8.18 

type DataStorageMap struct {
	KVStores  storage.DataMap
	LogStores storage.DataMap
}

DataStorageMap describes mappings from various instance and data type specifications to KV and Log stores.

type DataTxInit added in v0.8.1 

type DataTxInit struct {
	Session    rpc.SessionID
	DataName   dvid.InstanceName
	TypeName   dvid.TypeString
	InstanceID dvid.InstanceID
	Tags       map[string]string
}

type Initializer added in v0.8.17 

type Initializer interface {
	Initialize()
}

Initializer is a data instance that can be initialized. Typically this sets up any goroutines necessary after server configuration, etc.

type InstanceConfig 

type InstanceConfig struct {
	Gen   string
	Start dvid.InstanceID
}

InstanceConfig specifies how new instance IDs are generated

type InstanceMutator 

type InstanceMutator interface {
	// Loads all mutable properties and applies any necessary migration to
	// transform the internal data from the stored to expected version.
	LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (saveNeeded bool, err error)
}

InstanceMutator provides a hook for data instances to load mutable data on startup. It is assumed that the data instances store data whenever its data mutates, e.g., extents for labelblk or max label for labelvol.

type KVMessage added in v0.8.1 

type KVMessage struct {
	Session   rpc.SessionID
	KV        storage.KeyValue
	Terminate bool // true if this message is the last txn for this data instance and KV is invalid.
}

KVMessage packages a key-value pair for transmission to a remote DVID as well as control of the receiving FSM.

type MergeType 

type MergeType uint8

MergeType describes the expectation of processing for the merge, e.g., is it expected to be free of conflicts at the key-value level, require automated type-specific key-value merging, or require external guidance. 

const (
	// MergeConflictFree assumes that changes in nodes-to-be-merged are disjoint
	// at the key-value level.
	MergeConflictFree MergeType = iota

	// MergeTypeSpecificAuto requires datatype-specific code for merging at each
	// key-value pair.
	MergeTypeSpecificAuto

	// MergeExternalData requires external data to reconcile merging of nodes.
	MergeExternalData
)

type MutationMutexer added in v0.8.13 

type MutationMutexer interface {
	MutAdd(mutID uint64) (newOp bool)
	MutDone(mutID uint64)
	MutWait(mutID uint64)
	MutDelete(mutID uint64)
}

MutationMutexer is an interface for mutexes on particular mutation IDs.

type PropertyCopier added in v0.8.2 

type PropertyCopier interface {
	CopyPropertiesFrom(DataService, storage.FilterSpec) error
}

PropertyCopier are types that can copy data instance properties from another (typically identically typed) data instance with an optional filter. This is used to create copies of data instances locally or when pushing to a remote DVID.

type PushSession added in v0.8.2 

type PushSession struct {
	Filter   storage.FilterSpec
	Versions map[dvid.VersionID]struct{}
	// contains filtered or unexported fields
}

PushSession encapsulates parameters necessary for DVID-to-DVID push/pull processing.

func (*PushSession) EndInstancePush added in v0.8.5 

func (p *PushSession) EndInstancePush() error

EndInstancePush terminates a data instance push.

func (*PushSession) SendKV added in v0.8.5 

func (p *PushSession) SendKV(kv *storage.KeyValue) error

SendKV sends a key-value pair. The key-values may be buffered before sending for efficiency of transmission.

func (*PushSession) StartInstancePush added in v0.8.5 

func (p *PushSession) StartInstancePush(d dvid.Data) error

StartInstancePush initiates a data instance push. After some number of Send calls, the EndInstancePush must be called.

type Request 

type Request struct {
	dvid.Command
	Input []byte
}

Request supports RPC requests to the DVID server.

type Response 

type Response struct {
	dvid.Response
	Output []byte
}

Response supports RPC responses from DVID.

func (*Response) Write 

func (r *Response) Write(w io.Writer) error

Writes a RPC response to a writer.

type Shutdowner added in v0.8.5 

type Shutdowner interface {
	Shutdown(wg *sync.WaitGroup) // Expects wg.Done() to be called in Shutdown function.
}

Shutdowner is a data instance that has a function to call during shutdown. Typically, this exits goroutines used for background data processing.

type SyncEvent 

type SyncEvent struct {
	Data  dvid.UUID
	Event string
}

SyncEvent identifies an event in which a data instance has modified its data

type SyncMessage 

type SyncMessage struct {
	Event   string
	Version dvid.VersionID
	Delta   interface{}
}

SyncMessage describes changes to a data instance for a given version.

type SyncSub 

type SyncSub struct {
	Event  SyncEvent
	Notify dvid.UUID // the data UUID of data instance to notify
	Ch     chan SyncMessage
}

SyncSub is a subscription request from an instance to be notified via a channel when a given data instance has a given event.

type SyncSubs added in v0.8.0 

type SyncSubs []SyncSub

SyncSubs is a slice of sync subscriptions.

func (SyncSubs) Add added in v0.8.0 

func (subs SyncSubs) Add(added SyncSub) SyncSubs

Add returns a SyncSubs with the added SyncSub, making sure that only one subscription exists for any (Event, Notify) tuple. If a previous (Event, Notify) exists, it is replaced by the passed SyncSub.

type Syncer 

type Syncer interface {
	// GetSyncSubs returns the subscriptions that need to be created to keep this data
	// synced and may launch goroutines that will consume inbound channels of changes
	// from associated data.
	GetSyncSubs(dvid.Data) (SyncSubs, error)

	// SyncedNames returns the set of data instance UUIDs to which the data is synced.
	SyncedNames() []dvid.InstanceName

	// SyncedNames returns the set of data instance UUIDs to which the data is synced.
	SyncedData() dvid.UUIDSet

	// SyncPending returns true if there are outstanding sync events in this instance's subscription.
	SyncPending() bool
}

Syncer types are typically DataService that know how to sync with other data.

type Type 

type Type struct {
	// Data type name and may not be unique.
	Name dvid.TypeString

	// The unique package name that fulfills the DVID Data interface
	URL dvid.URLString

	// The version identifier of this datatype code
	Version string

	// A list of interface requirements for the backend datastore
	Requirements *storage.Requirements
}

Type identifies the datatype underlying a DataService.

func (*Type) Do added in v0.8.0 

func (t *Type) Do(cmd Request, reply *Response) error

Do error message. Do should be overridden

func (*Type) GetStorageRequirements 

func (t *Type) GetStorageRequirements() *storage.Requirements

func (*Type) GetTypeName 

func (t *Type) GetTypeName() dvid.TypeString

func (*Type) GetTypeURL 

func (t *Type) GetTypeURL() dvid.URLString

func (*Type) GetTypeVersion 

func (t *Type) GetTypeVersion() string

type TypeMigrator added in v0.8.0 

type TypeMigrator interface {
	MigrateData([]dvid.VersionID) (DataService, error)
}

TypeMigrator is an interface for a DataService that can migrate itself to another DataService. A deprecated DataService implementation can implement this interface to auto-convert on metadata load.

type TypeService 

type TypeService interface {
	GetTypeName() dvid.TypeString

	GetTypeURL() dvid.URLString

	GetTypeVersion() string

	GetStorageRequirements() *storage.Requirements

	// Create an instance of this datatype in the given repo (identified by its root UUID)
	// with local instance ID and name, passing configuration parameters via dvid.Config.
	NewDataService(dvid.UUID, dvid.InstanceID, dvid.InstanceName, dvid.Config) (DataService, error)

	// Help returns a string explaining how to use a datatype's service
	Help() string

	// Do executes type-specific commands from command line.
	Do(Request, *Response) error
}

TypeService is an interface all datatype implementations must fulfill. New types can be made by embedding Type and will automatically fulfill the Get* functions.

func TypeServiceByName 

func TypeServiceByName(name dvid.TypeString) (TypeService, error)

TypeServiceByName returns a TypeService given a type name. Note that the type name is possibly ambiguous, particularly if using type names across different DVID servers.

func TypeServiceByURL 

func TypeServiceByURL(url dvid.URLString) (TypeService, error)

TypeServiceByURL returns a TypeService given its URL. This is the preferred method for accessing datatype implementations since they should work across different DVID servers.

type TypeUpgrader added in v0.8.4 

type TypeUpgrader interface {
	DataService
	UpgradeData() (upgraded bool, err error)
}

TypeUpgrader is an interface for a DataService that can upgrade itself to another version of data storage. A deprecated DataService implementation can implement this interface to auto-convert on metadata load.

type Updater added in v0.7.1 

type Updater struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

func (*Updater) StartUpdate added in v0.7.1 

func (u *Updater) StartUpdate()

func (*Updater) StopUpdate added in v0.7.1 

func (u *Updater) StopUpdate()

func (*Updater) Updating added in v0.7.1 

func (u *Updater) Updating() bool

Returns true if the data is currently being updated.

type VersionInitializer 

type VersionInitializer interface {
	InitVersion(dvid.UUID, dvid.VersionID) error
}

VersionInitializer provides a hook for data instances to receive branch (new version) events and modify their properties as needed.

type VersionRemapper added in v0.8.2 

type VersionRemapper interface {
	RemapVersions(dvid.VersionMap) error
}

VersionRemapper provides a hook for data instances to remap properties that depend on server-specific version ids. During DVID-to-DVID transfer of data, these version ids need to be remapped as part of the push/pull.

type VersionedCtx 

type VersionedCtx struct {
	*storage.DataContext
}

VersionedCtx implements storage.VersionedCtx for data instances that have a version DAG.

func NewVersionedCtx 

func NewVersionedCtx(data dvid.Data, versionID dvid.VersionID) *VersionedCtx

func (*VersionedCtx) GetGraphDB added in v0.8.1 

func (vctx *VersionedCtx) GetGraphDB() (storage.GraphDB, error)

GetGraphDB returns a graph store associated with this context or an error if one is not available.

func (*VersionedCtx) GetKeyValueDB added in v0.8.1 

func (vctx *VersionedCtx) GetKeyValueDB() (storage.KeyValueDB, error)

GetKeyValueDB returns a key-value store associated with this context on an error if one is unavailable.

func (*VersionedCtx) GetOrderedKeyValueDB added in v0.8.1 

func (vctx *VersionedCtx) GetOrderedKeyValueDB() (storage.OrderedKeyValueDB, error)

GetOrderedKeyValueDB returns an ordered key-value store associated with this context or an error if one is unavailable.

func (*VersionedCtx) Head added in v0.8.13 

func (vctx *VersionedCtx) Head() bool

Head checks whether this the open head of the master branch

func (*VersionedCtx) MasterVersion added in v0.8.13 

func (vctx *VersionedCtx) MasterVersion(version dvid.VersionID) bool

Head checks whether specified version is on the master branch

func (*VersionedCtx) NumVersions added in v0.8.13 

func (vctx *VersionedCtx) NumVersions() int32

NumVersions returns the number of versions for a given

func (*VersionedCtx) RepoRoot added in v0.8.13 

func (vctx *VersionedCtx) RepoRoot() (dvid.UUID, error)

RepoRoot returns the root uuid.

func (*VersionedCtx) String 

func (vctx *VersionedCtx) String() string

func (*VersionedCtx) Versioned 

func (vctx *VersionedCtx) Versioned() bool

func (*VersionedCtx) VersionedKeyValue 

func (vctx *VersionedCtx) VersionedKeyValue(values []*storage.KeyValue) (*storage.KeyValue, error)

VersionedKeyValue returns the key-value pair corresponding to this key's version given a list of key-value pairs across many versions. If no suitable key-value pair is found or a tombstone is encounterd closest to version, nil is returned.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.