README
¶
turing
Turing is a framework for building domain specific databases on top of a replicated key value store. Database commands are implemented as instructions that are managed and executed by a cluster of turing nodes. The framework can be used to build client/server style databases or embedded databases within application/services. The goal is to provide a simple API and toolkit that can be used from standalone in-memory databases up to clusters consisting of many nodes. Under the hood, turing uses the pebble (alpha) for storing the data and dragonboat for reaching consensus among the nodes.
Examples
- An example implementing a simple counter can be found here.
- The
stdsetpackage implements a set of basic instructions.
License
The MIT License (MIT)
Copyright (c) 2020 Joël Gähwiler
Documentation
¶
Overview ¶
Package turing provides a framework to build domain specific databases.
Index ¶
- Constants
- Variables
- func Clone(src []byte) []byte
- func PrefixRange(prefix []byte) ([]byte, []byte)
- func SetLogger(sink io.Writer)
- type Cache
- type Config
- type Description
- type Instruction
- type Iterator
- type Machine
- func (m *Machine) Execute(ins Instruction, opts ...Options) error
- func (m *Machine) ExecuteAsync(ins Instruction, fn func(error), opts ...Options) error
- func (m *Machine) Status() Status
- func (m *Machine) Stop()
- func (m *Machine) Subscribe(observer Observer)
- func (m *Machine) Unsubscribe(observer Observer)
- type Member
- type Memory
- type Observer
- type Operator
- type Options
- type Ref
- type Role
- type Status
Constants ¶
const UnboundedEffect = -1
UnboundedEffect can be used as an effect if the instruction potentially modifies more keys than MaxEffect allows. These instructions may be applied using multiple transactions.
Variables ¶
var ErrDatabaseClosed = errors.New("turing: database closed")
ErrDatabaseClosed is returned if the database has been closed.
var ErrMaxEffect = errors.New("turing: max effect")
ErrMaxEffect is returned by a transaction if the effect limit has been reached. The instruction should return with this error to have the current changes persistent and be executed again to persist the remaining changes.
var ErrReadOnly = errors.New("turing: read only")
ErrReadOnly is returned by a transaction on write operations if the instruction has been flagged as read only.
Functions ¶
func PrefixRange ¶
PrefixRange will compute the lower and upper bound of a prefix range.
Types ¶
type Cache ¶
type Cache interface {
// Set will set the specified key to the new value.
Set(key, value interface{})
// Get will lookup the the specified key and return the value and whether
// it has been found.
Get(key interface{}) (interface{}, bool)
}
Cache is used by instructions to cache concrete values.
type Config ¶
type Config struct {
// The id of this member.
ID uint64
// All cluster members.
Members []Member
// The storage directory. If empty, an in-memory filesystem is used.
Directory string
// The used instructions.
Instructions []Instruction
// In standalone mode the database is not replicated.
Standalone bool
// The maximum effect that can be reported by an instruction. Instructions
// with a bigger effect must report an unbounded effect. Increasing the
// value will allow more throughput as more instructions are executed using
// the same transaction.
//
// Default: 10_000.
MaxEffect int
// The average round trip time.
//
// Default: 10ms.
RoundTripTime time.Duration
// The number of concurrent database readers.
//
// Default: min(NumCPUs - 3, 2).
ConcurrentReaders int
// The number of concurrent raft proposers.
//
// Default: NumCPUs.
ConcurrentProposers int
// The maximum instruction batch sizes.
//
// Default: 200, 200, 200.
UpdateBatchSize int
LookupBatchSize int
ProposalBatchSize int
// The time after a proposal times out.
//
// Default: 10s.
ProposalTimeout time.Duration
// The time after a linear read times out.
//
// Default: 10s.
LinearReadTimeout time.Duration
}
Config is used to configure a machine.
func (Config) DatabaseDir ¶
DatabaseDir returns the directory used for the database files.
func (Config) DatabaseFS ¶
DatabaseFS returns the filesystem used for the database files.
type Description ¶
type Description struct {
// The unique name of the instruction. The notation "path/package/Type" is
// recommended to ease discoverability.
Name string
// The builder can be set to provide a custom builder. If not set, the
// default reflect based builder will be used.
Builder func() Instruction
// The recycler can be used in conjunction with the custom builder to
// recycle built instructions.
Recycler func(ins Instruction)
// The operators used by this instruction. Deprecated operators must be
// retained to ensure they can be used to compact older database levels.
Operators []*Operator
// NoResult may be set to true to indicate that the write instruction does
// not carry a result. This potentially reduces some RPC traffic.
NoResult bool
// contains filtered or unexported fields
}
Description is a description of an instruction.
func (Description) Validate ¶
func (d Description) Validate() error
Validate will validate the instruction description.
type Instruction ¶
type Instruction interface {
// Describe should return a description of the instruction. This method is
// called often, therefore it should just return a pointer to a statically
// allocated object and never build the object on request.
Describe() *Description
// Effect should return the amount of modifications this instruction will
// make. A positive number is interpreted as the maximum amount of set,
// unset merged and deleted keys during the execution. A zero value
// indicates that the instruction is read only and will not set or delete
// any keys. A negative number indicates that the effect is unbounded and
// may modify many keys.
Effect() int
// Execute should execute the instruction using the provided memory.
Execute(mem Memory, cache Cache) error
// Encode should encode the instruction.
Encode() ([]byte, Ref, error)
// Decode should decode the instruction.
Decode([]byte) error
}
Instruction is the interface that is implemented by instructions that are executed by the machine.
type Iterator ¶
type Iterator interface {
// SeekGE will seek to the exact key or the next greater key.
SeekGE(key []byte) bool
// SeekLT will seek to the exact key or the next smaller key.
SeekLT(key []byte) bool
// First will seek to the first key in the range.
First() bool
// Last will seek to the last key in the range.
Last() bool
// Valid will return whether a valid key/value pair is present.
Valid() bool
// Next will move on to the next key.
Next() bool
// Prev will go back to the previous key.
Prev() bool
// Key will return a buffered key that can be used until released. Leaking
// the buffer brings a performance penalty. Use Clone and TempKey instead.
Key() ([]byte, Ref)
// TempKey will return the temporary key which is only valid until the next
// iteration or until the iterator is closed.
TempKey() []byte
// Value will return a buffered value that can be used until released. Leaking
// the buffer brings a performance penalty. Use Clone and TempValue instead.
Value() ([]byte, Ref, error)
// TempValue will return the temporary value which is only valid until the
// next iteration or the iterator is closed.
TempValue() ([]byte, error)
// Use will yield the temporary key and value to the provided function.
Use(fn func(key, value []byte) error) error
// Error will return the error.
Error() error
// Close will close the iterator.
Close() error
}
Iterator is used to iterate over the memory.
type Machine ¶
type Machine struct {
// contains filtered or unexported fields
}
Machine maintains a raft cluster with members and maintains consensus about the executed instructions on the replicated database.
func Test ¶
func Test(ins ...Instruction) *Machine
Test will start and return a machine for testing purposes.
func (*Machine) Execute ¶
func (m *Machine) Execute(ins Instruction, opts ...Options) error
Execute will execute the specified instruction.
func (*Machine) ExecuteAsync ¶
func (m *Machine) ExecuteAsync(ins Instruction, fn func(error), opts ...Options) error
ExecuteAsync will execute the specified instruction asynchronously. The specified function is called once the instruction has been executed.
func (*Machine) Unsubscribe ¶
Unsubscribe will unsubscribe the provided observer.
type Member ¶
Member specifies a cluster member.
func ParseMember ¶
ParseMember will parse the provided string and return a member. The string is expected to have the form "id@[host]:port".
func ParseMembers ¶
ParseMembers will parse the provided string and return a list of members. The string is expected to have the form "id@[host]:port,...".
type Memory ¶
type Memory interface {
// Iterate will construct and return a new iterator. The iterator must be
// closed as soon as it is not used anymore.
Iterate(prefix []byte) Iterator
// Get will lookup the specified key. The returned slice must not be modified
// by the caller. A closer is returned that must be closed once the value is
// not used anymore. Consider using Use() if the value is only used temporarily.
Get(key []byte) ([]byte, bool, io.Closer, error)
// Use will lookup the specified key and yield it to the provided function if
// it exists.
Use(key []byte, fn func(value []byte) error) error
// Set will set the specified key to the new value. This operation will count
// as one towards the effect of the backing transaction.
Set(key, value []byte) error
// Unset will remove the specified key. This operation will count as one towards
// the effect of the backing transaction.
Unset(key []byte) error
// Delete deletes all of the keys in the range [start, end] (inclusive on start,
// exclusive on end). This operation will count as one towards the effect of the
// backing transaction.
Delete(start, end []byte) error
// Merge merges existing values with the provided value using the specified
// operator.
Merge(key, value []byte, operator *Operator) error
// Effect will return the current effect of the backing transaction.
Effect() int
}
Memory is used by instructions to read and write to the database.
type Observer ¶
type Observer interface {
// Init is called when the source instruction stream has been (re)opened.
// This happens when the machine starts and whenever the node fails and
// restarts.
Init()
// Process is called repeatedly with every instruction processed by the
// machine. The implementation must ensure that the function returns as fast
// as possible as it blocks the execution of other instructions. If false is
// returned, the observer will be unsubscribed.
Process(ins Instruction) bool
}
Observer is the interface implemented by observers that want to observe the stream of instructions processed by the machine.
type Operator ¶
type Operator struct {
// The name of the operator.
Name string
// The zero value used as the base value if there is none.
Zero []byte
// The function called to apply operands to a value.
Apply func(value []byte, ops [][]byte) ([]byte, Ref, error)
// An optional function called to combine operands.
Combine func(ops [][]byte) ([]byte, Ref, error)
// contains filtered or unexported fields
}
Operator describes a merge operator.
type Options ¶
type Options struct {
// StaleRead can be set to execute a stale read. While this is much faster
// the instruction might read stale data in relationship to the current
// state of the cluster. In other words, settings this value will reduce
// the default linearizable guarantee to a serializable guarantee.
StaleRead bool
}
Options define options used during instruction execution.
type Ref ¶
type Ref interface {
Release()
}
Ref manages the reference to buffer that can be released.
type Role ¶
type Role int
Role specifies the role of a cluster member.
const ( // RoleLeader is the elected leader of a cluster. RoleLeader Role // RoleFollower is a electable cluster member. RoleFollower // RoleObserver is non-electable cluster member. RoleObserver )
Source Files
Directories