storage

Documentation

Index

• func CreateTempFile(dir string, prefix string) (*os.File, error)
• func CreateTempFolder(dir string, prefix string) (string, error)
• func RenameFolder(src, dst string) error
• type Backupable
• type ChangeApplier
• type ChangePropagator
• type Iteration
  • func NewIteration(kvs KVStore, iterReq *serverpb.IterateRequest) Iteration
• type IterationOption
  • func IterationPrefixKey(prefix []byte) IterationOption
  • func IterationStartKey(start []byte) IterationOption
• type IterationOptions
  • func NewIteratorOptions(opts ...IterationOption) (IterationOptions, error)
• type Iterator
• type KVStore
• type Stat
  • func NewStat(engine string) *Stat

Functions

func CreateTempFile

func CreateTempFile(dir string, prefix string) (*os.File, error)

CreateTempFile creates a temporary folder with the given prefix. It attempts to also appends a timestamp to the given prefix so as to better avoid collisions. Under the hood, it delegates to the GoLang API for temporary folder creation.

func CreateTempFolder

func CreateTempFolder(dir string, prefix string) (string, error)

CreateTempFolder creates a temporary folder with the given prefix. It attempts to also appends a timestamp to the given prefix so as to better avoid collisions. Under the hood, it delegates to the GoLang API for temporary folder creation.

func RenameFolder

func RenameFolder(src, dst string) error

RenameFolder moves the given src path onto the given dst path by first removing the dst path and then performing the actual movement.

Types

type Backupable

type Backupable interface {
	// BackupTo backs up the entire state of the underlying store
	// as one or more files into the given `path`.
	// Note that it is upto the implementation to interpret the
	// provided path as a file or a folder.
	BackupTo(path string) error
	// RestoreFrom restores the entire state of the underlying store
	// from one or more files belonging to the given `path`. Typically
	// these files must have been generated by a previous invocation
	// of the `BackupTo` method using this `path` by the same
	// implementation of this interface.
	// Note that it is upto the implementation to interpret the
	// provided path as a file or a folder.
	// Returns the various traits of the newly restored store upon
	// successful restoration, else an error
	RestoreFrom(path string) (KVStore, Backupable, ChangePropagator, ChangeApplier, error)
}

A Backupable represents the capability of the underlying store to be backed up and restored using filesystem as the medium.

type ChangeApplier

type ChangeApplier interface {
	// GetLatestAppliedChangeNumber retrieves the change number of
	// the latest committed change applied. Returns an error if unable
	// to load this number.
	GetLatestAppliedChangeNumber() (uint64, error)
	// SaveChanges commits to local key space the given changes and
	// returns the change number of the last committed change along
	// with an error that might have occurred during the process.
	// Note that implementors must treat every change on its own and
	// return the first error that occurs during the process. Remaining
	// changes if any must NOT be applied in order to ensure sequential
	// consistency.
	SaveChanges(changes []*serverpb.ChangeRecord) (uint64, error)
}

A ChangeApplier represents the capability of the underlying store to apply changes directly onto its key space. This is typically used for replication purposes to indicate that the implementor assumes the role of a slave node in a master-slave setup.

type ChangePropagator

type ChangePropagator interface {
	// GetLatestCommittedChangeNumber retrieves the change number of
	// the latest committed change. Returns an error if unable to
	// load this number.
	GetLatestCommittedChangeNumber() (uint64, error)
	// LoadChanges retrieves all the changes committed since the given
	// `fromChangeNumber`. Also, `maxChanges` can be used to limit the
	// number of changes returned in the response.
	LoadChanges(fromChangeNumber uint64, maxChanges int) ([]*serverpb.ChangeRecord, error)
}

A ChangePropagator represents the capability of the underlying store from which committed changes can be retrieved for replication purposes. The implementor of this interface assumes the role of a master node in a typical master-slave setup.

type Iteration

type Iteration interface {
	ForEach(func(*serverpb.KVPair) error) error
}

Iteration is a convenience wrapper around `Iterator` that allows for a given handler to be invoked exactly once for every key value pair iterated.

func NewIteration

func NewIteration(kvs KVStore, iterReq *serverpb.IterateRequest) Iteration

NewIteration allows for the creation of an `Iteration` instance that uses the underlying store's Iterator to callback for every key value pair iterated.

type IterationOption

type IterationOption func(*iterOpts)

IterationOption captures an iteration option.

func IterationPrefixKey

func IterationPrefixKey(prefix []byte) IterationOption

IterationPrefixKey is used to indicate the prefix of the keys that are to be iterated. In other words, only those keys that begin with this prefix are returned by the iterator.

func IterationStartKey

func IterationStartKey(start []byte) IterationOption

IterationStartKey sets the start key for iteration. All keys starting with this key are returned by the iterator.

type IterationOptions

type IterationOptions interface {
	KeyPrefix() ([]byte, bool)
	StartKey() ([]byte, bool)
}

IterationOptions captures the various options that can be set to control iteration process.

func NewIteratorOptions

func NewIteratorOptions(opts ...IterationOption) (IterationOptions, error)

NewIteratorOptions allows for the creation of `IterationOptions` instance that is used for controlling the behavior of iterating through keyspace.

type Iterator

type Iterator interface {
	io.Closer
	HasNext() bool
	Next() *serverpb.KVPair
	Err() error
}

Iterator represents the behavior of a key space iterator that allows for iterating though keys using the `HasNext` and `Next` methods.

type KVStore

type KVStore interface {
	io.Closer
	// Put stores the association between the given key and value and
	// optionally sets the expireTS of the key to the provided epoch in seconds
	Put(pairs ...*serverpb.KVPair) error
	// Get bulk fetches the associated values for the given keys.
	// Note that during partial failures, any successful results
	// are discarded and an error is returned instead.
	Get(keys ...[]byte) ([]*serverpb.KVPair, error)
	// Delete deletes the given key.
	Delete(key []byte) error
	// GetSnapshot retrieves the entire keyspace representation
	// with latest value against every key.
	GetSnapshot() (io.ReadCloser, error)
	// PutSnapshot ingests the given keyspace representation wholly
	// into the current state. Any existing state will be discarded
	// or replaced with the given state.
	PutSnapshot(io.ReadCloser) error
	// Iterate iterates through the entire keyspace in no particular
	// order. IterationOptions can be used to control where to begin
	// iteration as well as what keys are iterated by their prefix.
	Iterate(IterationOptions) Iterator
	// CompareAndSet compares the current value of the given key with
	// that of the given value. In case of a match, it updates that
	// key with the new value and returns true. Else, it returns false.
	// All this is done atomically from the caller's point of view and
	// hence is safe from a concurrency perspective.
	// If the expected value is `nil`, then the key is created and
	// initialized with the given value, atomically.
	CompareAndSet(request *serverpb.CompareAndSetRequest) (bool, error)
}

A KVStore represents the key value store that provides the underlying storage implementation for the various DKV operations.

type Stat

type Stat struct {
	RequestLatency        *prometheus.SummaryVec
	ResponseError         *prometheus.CounterVec
	StoreMetricsCollector prometheus.Collector
}

func NewStat

func NewStat(engine string) *Stat