storage

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 9, 2022 License: Apache-2.0 Imports: 5 Imported by: 3

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrAborted indicates that the request operation was aborted as the data
	// storage couldn't handle the request at the time of request. The client
	// is suggested to retry later.
	ErrAborted = errors.New("operation aborted")
	// ErrShardNotFound is returned by the data storage to indicate that the
	// requested shard is not found.
	ErrShardNotFound = errors.New("shard not found")
)

Functions

This section is empty.

Types

type BaseStorage added in v0.2.0 

type BaseStorage interface {
	Closeable
	StatsKeeper
	WriteBatchCreator
	// CreateSnapshot creates a snapshot from the specified shard and store the
	// generated snapshot into the directory specified by the path parameter. It
	// returns the encountered error if there is any.
	CreateSnapshot(shardID uint64, path string) error
	// ApplySnapshot applies the snapshort stored in the given path into the
	// specified shard.
	ApplySnapshot(shardID uint64, path string) error
}

BaseStorage is the interface to be implemented by all storage types.

type Batch added in v0.2.0 

type Batch struct {
	// Index is the corresponding raft log index of the batch.
	Index uint64
	// Requests is the requests included in the batch.
	Requests []Request
}

Batch contains a list of requests. For write batches, all requests are from the same raft log specified by the Index value. They must be atomically applied into the data storage together with the Index value itself. For read operation, each Batch contains multiple read requests.

type Closeable added in v0.2.0 

type Closeable interface {
	// Close closes the instance.
	Close() error
}

Closeable is an instance that can be closed.

type DataStorage 

type DataStorage interface {
	BaseStorage
	// Write applies write requests into the underlying data storage. The
	// `WriteContext` holds all requests involved and packs as many requests from
	// multiple Raft logs together as possible. The implementation must ensure
	// that the content of each LogRequest instance provided by the WriteContext
	// is atomically applied into the underlying storage. The implementation
	// should call the `SetWrittenBytes` and `SetDiffBytes` methods of the
	// `WriteContext` to report the statistical changes involved in applying
	// the specified `WriteContext` before returning.
	Write(WriteContext) error

	// Read execute read requests and returns the read result. The `ReadContext`
	// holds the read request to invoked. The implementation should call the
	// `SetReadBytes` method of `ReadContext` to report the statistical changes
	// involved in this execution before returning.
	Read(ReadContext) ([]byte, error)
	// GetInitialStates returns the most recent shard states of all shards known
	// to the DataStorage instance that are consistent with their related table
	// shards data. The shard metadata is last changed by the raft log identified
	// by the LogIndex value.
	GetInitialStates() ([]meta.ShardMetadata, error)
	// GetPersistentLogIndex returns the most recent raft log index that is known
	// to have its update persistently stored. This means all updates made by Raft
	// logs no greater than the returned index value have been persistently stored,
	// they are guaranteed to be available after reboot.
	GetPersistentLogIndex(shardID uint64) (uint64, error)
	// SaveShardMetadata saves the provided shards metadata into the DataStorage.
	// It is up to the storage engine to determine whether to synchronize the
	// saved content to persistent storage or not. It is also the responsibility
	// of the data storage to ensure that a consistent view of shard data and
	// metadata is always available on restart.
	SaveShardMetadata([]meta.ShardMetadata) error
	// RemoveShard is used for notifying the data storage that a shard has been
	// removed by MatrixCube. The removeData parameter indicates whether shard
	// data should be removed by the data storage. When removeData is set to true,
	// the data storage is required do the cleaning asynchronously.
	RemoveShard(shard meta.Shard, removeData bool) error
	// Sync persistently saves table shards data and shards metadata of the
	// specified shards to the underlying persistent storage.
	Sync([]uint64) error
	// SplitCheck finds keys within the [start, end) range so that the sum of bytes
	// of each value is no greater than the specified size in bytes. It returns the
	// current bytes(approximate) and the total number of keys(approximate) in [start,end),
	// the founded split keys. The ctx is context information of this check will be passed
	// to the engine by cube in the subsequent split operation.
	SplitCheck(shard meta.Shard, size uint64) (currentApproximateSize uint64,
		currentApproximateKeys uint64, splitKeys [][]byte, ctx []byte, err error)
	// Split After the split request completes raft consensus, it is used to save the
	// metadata after the Shard has executed the split, metadata needs atomically saved
	// into the underlying storage.
	Split(old meta.ShardMetadata, news []meta.ShardMetadata, ctx []byte) error
}

DataStorage is the interface to be implemented by data engines for storing both table shards data and shards metadata. We assume that data engines are WAL-less engines meaning some of its most recent writes will be lost on restarts. On such restart, table shards data in the DataStorage will rollback to a certain point in time state. GetPersistentLogIndex() can be invoked to query the most recent persistent raft log index that can be used to identify such point in time state. DataStorage guarantees that its table shards data and shards metadata will never rollback to any earlier state in feuture restarts. GetInitialStates() is invoked immediate after each restart, it returns ShardMetadata of all known shards consistent to the above mentioned persistent table shards data state. This means the state of the data storage will be consistent as long as raft logs are replayed from the GetPersistentLogIndex() + 1.

type Executor added in v0.2.0 

type Executor interface {
	// UpdateWriteBatch applies write requests into the provided `WriteContext`.
	// No writes is allowed to be written to the actual underlying data storage.
	UpdateWriteBatch(WriteContext) error
	// ApplyWriteBatch atomically applies the write batch into the underlying
	// data storage.
	ApplyWriteBatch(wb Resetable) error
	// Read executes the read request and returns the result. The `ReadContext`
	// holds the read request to be invoked in this execution. The implementation
	// should call the `SetReadBytes` method of `Context` to report the
	// statistical changes involved in this execution before returning.
	Read(ReadContext) ([]byte, error)
}

Executor is used to execute read/write requests.

type KVBaseStorage added in v0.2.0 

type KVBaseStorage interface {
	BaseStorage
	KVStore
}

KVBaseStorage is a KV based base storage.

type KVMetadataStore added in v0.2.0 

type KVMetadataStore interface {
	// not allowed to close the store
	WriteBatchCreator
	KVStore
}

KVMetadataStore is a KV based data store for storing MatrixCube metadata.

type KVStorage 

type KVStorage interface {
	Closeable
	WriteBatchCreator
	StatsKeeper
	KVStore
}

KVStorage is key-value based storage.

type KVStorageWrapper added in v0.2.0 

type KVStorageWrapper interface {
	// GetKVStorage returns the wrapped KVStorage
	GetKVStorage() KVStorage
}

KVStorageWrapper is a KVStorage wrapper

type KVStore added in v0.2.0 

type KVStore interface {
	// GetView returns a point in time view of the KVStore.
	GetView() View
	// Write writes the data in batch to the storage.
	Write(wb util.WriteBatch, sync bool) error
	// Set puts the key-value pair to the storage.
	Set(key []byte, value []byte, sync bool) error
	// Get returns the value associated with the key.
	Get(key []byte) ([]byte, error)
	// Delete removes the key-value pair specified by the key.
	Delete(key []byte, sync bool) error
	// Scan scans the key-value paire in the specified [start, end) range, the
	// specified handler function is invoked on each key-value pair until the
	// handler function returns false. Depending on the copy parameter, the
	// handler function will be provided a cloned key-value pair that can be
	// retained after the return of the handler function or a pair of temporary
	// key-value slices that could change after the return of the handler
	// function.
	Scan(start, end []byte,
		handler func(key, value []byte) (bool, error), copy bool) error
	// ScanInView is similar to Scan, it performs the Scan operation on the
	// specified view.
	ScanInView(view View, start, end []byte,
		handler func(key, value []byte) (bool, error), copy bool) error
	// PrefixScan scans all key-value pairs that share the specified prefix, the
	// specified handler function will be invoked on each such key-value pairs
	// until false is returned by the handler function. Depending on the copy
	// parameter, the handler function will be provided a cloned key-value pair
	// that can be retained after the return of the handler function or a pair
	// of temporary key-value slices that could change after the return of the
	// handler function.
	PrefixScan(prefix []byte,
		handler func(key, value []byte) (bool, error), copy bool) error
	// RangeDelete delete data within the specified [start,end) range.
	RangeDelete(start, end []byte, sync bool) error
	// Seek returns the first key-value pair that has the key component no less
	// than the specified key.
	Seek(key []byte) ([]byte, []byte, error)
	// Sync synchronize the storage's in-core state with that on disk.
	Sync() error
}

KVStore is the interface for supported key-value based data store operations.

type ReadContext added in v0.2.0 

type ReadContext interface {
	// ByteBuf returns the bytebuf that can be used to avoid memory allocation.
	ByteBuf() *buf.ByteBuf
	// Shard returns the current shard details.
	Shard() meta.Shard
	// Requeset returns the read request to be processed on the storage engine.
	Request() Request
	// SetReadBytes set the number of bytes read from storage for all requests in
	// the current context. This is an approximation value that contributes to the
	// scheduler's auto-rebalancing feature.
	SetReadBytes(uint64)
}

type Request added in v0.2.0 

type Request struct {
	// CmdType is the request type.
	CmdType uint64
	// Key is the key of the request.
	Key []byte
	// Cmd is the content of the request.
	Cmd []byte
}

Request is the custom request type.

type Resetable added in v0.2.0 

type Resetable interface {
	// Reset makes the instance ready to be reused.
	Reset()
}

Resetable is an instance that can be reset and reused.

type SimpleReadContext added in v0.2.0 

type SimpleReadContext struct {
	// contains filtered or unexported fields
}

func NewSimpleReadContext added in v0.2.0 

func NewSimpleReadContext(shardID uint64, req Request) *SimpleReadContext

NewSimpleReadContext returns a testing context.

func (*SimpleReadContext) ByteBuf added in v0.2.0 

func (c *SimpleReadContext) ByteBuf() *buf.ByteBuf

func (*SimpleReadContext) GetReadBytes added in v0.2.0 

func (c *SimpleReadContext) GetReadBytes() uint64

func (*SimpleReadContext) Request added in v0.2.0 

func (c *SimpleReadContext) Request() Request

func (*SimpleReadContext) SetReadBytes added in v0.2.0 

func (c *SimpleReadContext) SetReadBytes(readBytes uint64)

func (*SimpleReadContext) Shard added in v0.2.0 

func (c *SimpleReadContext) Shard() meta.Shard

type SimpleWriteContext added in v0.2.0 

type SimpleWriteContext struct {
	// contains filtered or unexported fields
}

SimpleWriteContext is a simple WriteContext implementation used for testing.

func NewSimpleWriteContext added in v0.2.0 

func NewSimpleWriteContext(shardID uint64,
	base KVStorage, batch Batch) *SimpleWriteContext

NewSimpleWriteContext returns a testing context.

func (*SimpleWriteContext) AppendResponse added in v0.2.0 

func (ctx *SimpleWriteContext) AppendResponse(value []byte)

func (*SimpleWriteContext) Batch added in v0.2.0 

func (ctx *SimpleWriteContext) Batch() Batch

func (*SimpleWriteContext) ByteBuf added in v0.2.0 

func (ctx *SimpleWriteContext) ByteBuf() *buf.ByteBuf

func (*SimpleWriteContext) GetDiffBytes added in v0.2.0 

func (ctx *SimpleWriteContext) GetDiffBytes() int64

func (*SimpleWriteContext) GetWrittenBytes added in v0.2.0 

func (ctx *SimpleWriteContext) GetWrittenBytes() uint64

func (*SimpleWriteContext) Responses added in v0.2.0 

func (ctx *SimpleWriteContext) Responses() [][]byte

func (*SimpleWriteContext) SetDiffBytes added in v0.2.0 

func (ctx *SimpleWriteContext) SetDiffBytes(value int64)

func (*SimpleWriteContext) SetWrittenBytes added in v0.2.0 

func (ctx *SimpleWriteContext) SetWrittenBytes(value uint64)

func (*SimpleWriteContext) Shard added in v0.2.0 

func (ctx *SimpleWriteContext) Shard() meta.Shard

func (*SimpleWriteContext) WriteBatch added in v0.2.0 

func (ctx *SimpleWriteContext) WriteBatch() Resetable

type StatsKeeper added in v0.2.0 

type StatsKeeper interface {
	// Stats returns the stats of the instance.
	Stats() stats.Stats
}

StatsKeeper is an instance that can provide stats.

type View added in v0.2.0 

type View interface {
	Close() error
	Raw() interface{}
}

View is a point in time view of the KVStore.

type WriteBatchCreator added in v0.2.0 

type WriteBatchCreator interface {
	// NewWriteBatch creates and returns a new write batch that can be used with
	// the base storage instance.
	NewWriteBatch() Resetable
}

WriteBatchCreator is capable of creating new write batches.

type WriteContext added in v0.2.0 

type WriteContext interface {
	// ByteBuf returns the bytebuf that can be used to avoid memory allocation.
	// Note, the will be reset after each Read or Write execution, so it is safe
	// to use bytebuf in one Read or Write call. Multiple calls to `ByteBuf` in
	// a single Read or Write return the same instance.
	ByteBuf() *buf.ByteBuf
	// WriteBatch returns a write batch which will be used to hold a sequence of
	// updates to be atomically made into the underlying storage engine. A
	// resetable instance is returned by this method and it is up to the user to
	// cast it to the actual write batch type compatible with the intended data
	// storage.
	WriteBatch() Resetable
	// Shard returns the current shard details.
	Shard() meta.Shard
	// Batch returns the Batch instance transformed from a single Raft log.
	Batch() Batch
	// AppendResponse is used for appending responses once each request is handled.
	AppendResponse([]byte)
	// SetWrittenBytes set the number of bytes written to storage for all requests
	// in the current Context instance. This is an approximation value that
	// contributes to the scheduler's auto-rebalancing feature.
	// This method must be called before `Read` or `Write` returns.
	SetWrittenBytes(uint64)
	// SetDiffBytes set the diff of the bytes stored in storage after Write is
	// executed. This is an approximation value used to modify the approximate
	// amount of data in the `Shard` which is used for triggering the auto-split
	// procedure.
	SetDiffBytes(int64)
}

WriteContext contains the details of write requests to be handled by the data storage.

Source Files

View all Source files

Directories

Path Synopsis
simple
kv
mem
pebble

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.