Documentation ¶
Overview ¶
Package logdb provides an efficient log-structured database supporting efficient insertion of new entries and removal from either end of the log.
This provides a number of interfaces and types, and a lot of errors.
- 'LogDB' is the main interface for a log-structured database.
- 'PersistDB' is an interface for databases which can be persisted in some way.
- 'BoundedDB' is an interface for databases with a fixed maximum entry size.
- 'CloseDB' is an interface for databases which can be closed.
The 'LockFreeChunkDB' and 'ChunkDB' types implement all of these interfaces, and are created with 'Open' and 'WrapForConcurrency' respectively. As the names suggest, the difference is the thread-safety. A 'LockFreeChunkDB' is only safe for single-threaded access, where a 'ChunkDB' wraps it and adds locking, for safe concurrent access. Additionally, the 'InMemDB' type implements the 'LogDB' interface using a purely in-memory store.
Index ¶
- Variables
- type AtomicityError
- type BoundedDB
- type ChunkContinuityError
- type ChunkDB
- func (db *ChunkDB) AppendEntries(entries [][]byte) (uint64, error)
- func (db *ChunkDB) Close() error
- func (db *ChunkDB) Forget(newOldestID uint64) error
- func (db *ChunkDB) Get(id uint64) ([]byte, error)
- func (db *ChunkDB) Rollback(newNewestID uint64) error
- func (db *ChunkDB) SetSync(every int) error
- func (db *ChunkDB) Sync() error
- func (db *ChunkDB) Truncate(newOldestID, newNewestID uint64) error
- type ChunkFileNameError
- type ChunkMetaError
- type ChunkSizeError
- type CloseDB
- type CodingDB
- type CompressingDB
- type DeleteError
- type FormatError
- type InMemDB
- func (db *InMemDB) Append(entry []byte) (uint64, error)
- func (db *InMemDB) AppendEntries(entries [][]byte) (uint64, error)
- func (db *InMemDB) Forget(newOldestID uint64) error
- func (db *InMemDB) Get(id uint64) ([]byte, error)
- func (db *InMemDB) NewestID() uint64
- func (db *InMemDB) OldestID() uint64
- func (db *InMemDB) Rollback(newNewestID uint64) error
- func (db *InMemDB) Truncate(newOldestID, newNewestID uint64) error
- type LockError
- type LockFreeChunkDB
- func (db *LockFreeChunkDB) Append(entry []byte) (uint64, error)
- func (db *LockFreeChunkDB) AppendEntries(entries [][]byte) (uint64, error)
- func (db *LockFreeChunkDB) Close() error
- func (db *LockFreeChunkDB) Forget(newOldestID uint64) error
- func (db *LockFreeChunkDB) Get(id uint64) ([]byte, error)
- func (db *LockFreeChunkDB) MaxEntrySize() uint64
- func (db *LockFreeChunkDB) NewestID() uint64
- func (db *LockFreeChunkDB) OldestID() uint64
- func (db *LockFreeChunkDB) Rollback(newNewestID uint64) error
- func (db *LockFreeChunkDB) SetSync(every int) error
- func (db *LockFreeChunkDB) Sync() error
- func (db *LockFreeChunkDB) Truncate(newOldestID, newNewestID uint64) error
- type LogDB
- type MetaContinuityError
- type MetaOffsetError
- type PathError
- type PersistDB
- type ReadError
- type SyncError
- type WriteError
Constants ¶
This section is empty.
Variables ¶
var ( // ErrIDOutOfRange means that the requested ID is not present in the log. ErrIDOutOfRange = errors.New("log ID out of range") // ErrUnknownVersion means that the disk format version of an opened database is unknown. ErrUnknownVersion = errors.New("unknown disk format version") // ErrNotDirectory means that the path given to 'Open' exists and is not a directory. ErrNotDirectory = errors.New("database path not a directory") // ErrPathDoesntExist means that the path given to 'Open' does not exist and the 'create' flag was // false. ErrPathDoesntExist = errors.New("database directory does not exist") // ErrTooBig means that an entry could not be appended because it is larger than the chunk size. ErrTooBig = errors.New("entry larger than chunksize") // ErrClosed means that the database handle is closed. ErrClosed = errors.New("database is closed") // ErrEmptyNonfinalChunk means that the metadata for a non-final chunk has zero entries. ErrEmptyNonfinalChunk = errors.New("metadata of non-final chunk contains no entries") )
var ErrNotValueSlice = errors.New("AppendValues must be called with a slice argument")
ErrNotValueSlice means that AppendValues was called with a non-slice argument.
Functions ¶
This section is empty.
Types ¶
type AtomicityError ¶
AtomicityError means that an error occurred while appending an entry in an 'AppendEntries' call, and attempting to rollback also gave an error. It wraps the actual errors.
func (*AtomicityError) Error ¶
func (e *AtomicityError) Error() string
func (*AtomicityError) WrappedErrors ¶
func (e *AtomicityError) WrappedErrors() []error
type BoundedDB ¶
type BoundedDB interface { // 'BoundedDB' is an extension of 'LogDB'. LogDB // The maximum size of an entry. It is an error to try to insert an entry larger than this. MaxEntrySize() uint64 }
A BoundedDB has a maximum entry size. In addition to defining methods, a 'BoundedDB' changes some the behaviour of 'Append' and 'AppendEntries': they now return 'ErrTooBig' if an entry appended is larger than the maximum size.
type ChunkContinuityError ¶
ChunkContinuityError means that two adjacent chunks do not contain a contiguous sequence of entries.
func (*ChunkContinuityError) Error ¶
func (e *ChunkContinuityError) Error() string
type ChunkDB ¶
type ChunkDB struct { // The underlying 'LockFreeChunkDB'. This is not safe for concurrent use with the 'ChunkDB'. *LockFreeChunkDB // contains filtered or unexported fields }
ChunkDB is a 'LogDB' implementation using an on-disk format where entries are stored in fixed-size "chunks". It provides data consistency guarantees in the event of program failure: database entries are guaranteed to be contiguous, and if the database thinks an entry is there, it will contain non-corrupt data.
In addition to implementing the behaviour specified in the 'LogDB', 'PersistDB', 'BoundedDB', and 'CloseDB' interfaces, a 'Sync' is always performed if an entire chunk is forgotten, rolled back, or truncated; or if an append creates a new on-disk chunk.
func WrapForConcurrency ¶
func WrapForConcurrency(db *LockFreeChunkDB) *ChunkDB
Wrap a 'LockFreeChunkDB' into a 'ChunkDB', which is safe for concurrent use. The underlying 'LockFreeChunkDB' should not be used while the returned 'ChunkDB' is live.
func (*ChunkDB) AppendEntries ¶
AppendEntries implements the 'LogDB', 'PersistDB', 'BoundedDB', and 'CloseDB' interfaces.
func (*ChunkDB) Close ¶
Close implements the 'CloseDB' interface. This also closes the underlying 'LockFreeChunkDB'.
type ChunkFileNameError ¶
type ChunkFileNameError struct {
FilePath string
}
ChunkFileNameError means that a filename is not valid for a chunk file.
func (*ChunkFileNameError) Error ¶
func (e *ChunkFileNameError) Error() string
type ChunkMetaError ¶
ChunkMetaError means that the metadata for a chunk could not be read. It wraps the actual error.
func (*ChunkMetaError) Error ¶
func (e *ChunkMetaError) Error() string
func (*ChunkMetaError) WrappedErrors ¶
func (e *ChunkMetaError) WrappedErrors() []error
type ChunkSizeError ¶
ChunkSizeError means that a chunk file is not the expected size.
func (*ChunkSizeError) Error ¶
func (e *ChunkSizeError) Error() string
type CloseDB ¶
type CloseDB interface { // 'CloseDB' is an extension of 'LogDB'. LogDB // Close performs some database-specific clean-up. It is an error to try to use a database after // closing it. // // Returns 'ErrClosed' if the database is already closed. Close() error }
A CloseDB can be closed, which may perform some clean-up.
If a 'CloseDB' is also a 'PersistDB', then 'Sync' should be called during 'Close'. In addition, all 'LogDB' and 'PersistDB' with an error return will fail after calling 'Close', with 'ErrClosed'.
type CodingDB ¶
type CodingDB struct { LogDB Encode func(interface{}) ([]byte, error) Decode func([]byte, interface{}) error }
A CodingDB wraps a 'LogDB' with functions to encode and decode values of some sort, giving a higher-level interface than raw byte slices.
func BinaryCoder ¶
BinaryCoder creates a 'CodingDB' with the binary encoder/decoder. Values must be valid input for the 'binary.Write' function.
func GobCoder ¶
GobCoder creates a 'CodingDB' with the gob encoder/decoder. Values must be valid input for the 'god.Encode' function.
func IdentityCoder ¶
IdentityCoder creates a 'CodingDB' with the identity encoder/decoder. It is an error to append a value which is not a '[]byte'.
func (*CodingDB) AppendValue ¶
AppendValue encodes a value using the encoder, and stores it in the underlying 'LogDB' is there is no error.
func (*CodingDB) AppendValues ¶
AppendValues encodes a slice of values (represented as an 'interface{}', to make the casting simpler), and stores them in the underlying 'LogDB' if there is no error.
Returns 'ErrNotValueSlice' if called with a non-slice argument.
type CompressingDB ¶
type CompressingDB struct { LogDB Compress func([]byte) ([]byte, error) Decompress func([]byte) ([]byte, error) }
A CompressingDB wraps a 'LogDB' with functions to compress and decompress entries, applied transparently during 'Append', 'AppendEntries', and 'Get'.
func CompressDEFLATE ¶
func CompressDEFLATE(logdb LogDB, level int) (*CompressingDB, error)
CompressDEFLATE creates a 'CompressingDB' with DEFLATE compression at the given level.
Returns an error if the level is < -2 or > 9.
func CompressIdentity ¶
func CompressIdentity(logdb LogDB) *CompressingDB
CompressIdentity create a 'CompressingDB' with the identity compressor/decompressor.
func CompressLZW ¶
CompressLZW creates a 'CompressingDB' with LZW compression with the given order and literal width.
Returns an error if the lit width is < 2 or > 8.
func (*CompressingDB) Append ¶
func (db *CompressingDB) Append(entry []byte) (uint64, error)
Append implements the 'LogDB' interface. If the underlying 'LogDB' is also a 'PersistDB', 'BoundedDB', or 'CloseDB' then those interfaces are also implemented.
With 'BoundedDB' the bounded entry size is that of the compressed byte array; so it may be possible to insert entries which are larger than the bound.
func (*CompressingDB) AppendEntries ¶
func (db *CompressingDB) AppendEntries(entries [][]byte) (uint64, error)
AppendEntries implements the 'LogDB' interface. If the underlying 'LogDB' is also a 'PersistDB', 'BoundedDB', or 'CloseDB' then those interfaces are also implemented.
With 'BoundedDB' the bounded entry size is that of the compressed byte array; so it may be possible to insert entries which are larger than the bound.
func (*CompressingDB) Get ¶
func (db *CompressingDB) Get(id uint64) ([]byte, error)
Get implements the 'LogDB' interface. If the underlying 'LogDB; is also a 'CloseDB' then that interface is also implemented.
With 'BoundedDB' the bounded entry size is that of the compressed byte array; so it may be possible to retrieve entries which are larger than the bound.
type DeleteError ¶
type DeleteError struct{ Err error }
DeleteError means that a file could not be deleted from disk. It wraps the actual error.
func (*DeleteError) Error ¶
func (e *DeleteError) Error() string
func (*DeleteError) WrappedErrors ¶
func (e *DeleteError) WrappedErrors() []error
type FormatError ¶
FormatError means that there is a problem with the database files. It wraps the actual error.
func (*FormatError) Error ¶
func (e *FormatError) Error() string
func (*FormatError) WrappedErrors ¶
func (e *FormatError) WrappedErrors() []error
type InMemDB ¶
type InMemDB struct {
// contains filtered or unexported fields
}
InMemDB is an in-memory 'LogDB' implementation. As does not support persistence, it shouldn't be used in a production system. It is, however, helpful for benchmark comparisons as an absolute best case to compare against.
func (*InMemDB) AppendEntries ¶
AppendEntries implements the 'LogDB' interface.
type LockError ¶
type LockError struct{ Err error }
LockError means that the database files could not be locked. It wraps the actual error.
func (*LockError) WrappedErrors ¶
type LockFreeChunkDB ¶
type LockFreeChunkDB struct {
// contains filtered or unexported fields
}
A LockFreeChunkDB is a 'ChunkDB' with no internal locks. It is NOT safe for concurrent use.
func Open ¶
func Open(path string, chunkSize uint32, create bool) (*LockFreeChunkDB, error)
Open a 'LockFreeChunkDB' database.
It is not possible to have multiple open references to the same database, as the files are locked. Concurrent usage of one open handle in a single process is safe.
The log is stored on disk in fixed-size files, controlled by the 'chunkSize' parameter. Entries are not split over chunks, and so if entries are a fixed size, the chunk size should be a multiple of that to avoid wasting space. Furthermore, no entry can be larger than the chunk size. There is a trade-off to be made: a chunk is only deleted when its entries do not overlap with the live entries at all (this happens through calls to 'Forget' and 'Rollback'), so a larger chunk size means fewer files, but longer persistence.
If the 'create' flag is true and the database doesn't already exist, the database is created using the given chunk size. If the database does exist, the chunk size parameter is ignored, and detected automatically from the chunk files.
func (*LockFreeChunkDB) Append ¶
func (db *LockFreeChunkDB) Append(entry []byte) (uint64, error)
Append implements the 'LogDB', 'PersistDB', 'BoundedDB', and 'CloseDB' interfaces.
func (*LockFreeChunkDB) AppendEntries ¶
func (db *LockFreeChunkDB) AppendEntries(entries [][]byte) (uint64, error)
AppendEntries implements the 'LogDB', 'PersistDB', 'BoundedDB', and 'CloseDB' interfaces.
func (*LockFreeChunkDB) Close ¶
func (db *LockFreeChunkDB) Close() error
Close implements the 'CloseDB' interface.
func (*LockFreeChunkDB) Forget ¶
func (db *LockFreeChunkDB) Forget(newOldestID uint64) error
Forget implements the 'LogDB', 'PersistDB', and 'CloseDB' interfaces.
func (*LockFreeChunkDB) Get ¶
func (db *LockFreeChunkDB) Get(id uint64) ([]byte, error)
Get implements the 'LogDB' and 'CloseDB' interfaces.
func (*LockFreeChunkDB) MaxEntrySize ¶
func (db *LockFreeChunkDB) MaxEntrySize() uint64
MaxEntrySize implements the 'BoundedDB' interface.
func (*LockFreeChunkDB) NewestID ¶
func (db *LockFreeChunkDB) NewestID() uint64
NewestID implements the 'LogDB' interface.
func (*LockFreeChunkDB) OldestID ¶
func (db *LockFreeChunkDB) OldestID() uint64
OldestID implements the 'LogDB' interface.
func (*LockFreeChunkDB) Rollback ¶
func (db *LockFreeChunkDB) Rollback(newNewestID uint64) error
Rollback implements the 'LogDB', 'PersistDB', and 'CloseDB' interfaces.
func (*LockFreeChunkDB) SetSync ¶
func (db *LockFreeChunkDB) SetSync(every int) error
SetSync implements the 'PersistDB' and 'CloseDB' interface.
func (*LockFreeChunkDB) Sync ¶
func (db *LockFreeChunkDB) Sync() error
Sync implements the 'PersistDB' and 'CloseDB' interface.
func (*LockFreeChunkDB) Truncate ¶
func (db *LockFreeChunkDB) Truncate(newOldestID, newNewestID uint64) error
Truncate implements the 'LogDB', 'PersistDB', and 'CloseDB' interfaces.
type LogDB ¶
type LogDB interface { // Append writes a new entry to the log and returns its ID. // // Returns 'WriteError' value if the database files could not be written to. Append(entry []byte) (uint64, error) // AppendEntries atomically writes a collection of new entries to the log and returns the ID of the // first (the IDs are contiguous). If the slice is empty or nil, the returned ID is meaningless. // // Returns the same errors as 'Append', and an 'AtomicityError' value if any entry fails to // append and rolling back the log failed. AppendEntries(entries [][]byte) (uint64, error) // Get looks up an entry by ID. // // Returns 'ErrIDOutOfRange' if the requested ID is lesser than the oldest or greater than the // newest. Get(id uint64) ([]byte, error) // Forget removes entries from the end of the log. // // If the new "oldest" ID is older than the current, this is a no-op. // // Returns 'ErrIDOutOfRange' if the ID is newer than the "newest" ID. Forget(newOldestID uint64) error // Rollback removes entries from the head of the log. // // If the new "newest" ID is newer than the current, this is a no-op. // // Returns the same errors as 'Forget', with 'ErrIDOutOfRange' being returned if the ID is older // than the "oldest" ID. Rollback(newNewestID uint64) error // Truncate performs a 'Forget' followed by a 'Rollback' atomically. The semantics are that if // the 'Forget' fails, the 'Rollback' is not performed; but the 'Forget' is not undone either. // // Returns the same errors as 'Forget' and 'Rollback', and also an 'ErrIDOutOfRange' if the new // newest < the new oldest. Truncate(newOldestID, newNewestID uint64) error // OldestID gets the ID of the oldest log entry. // // For an empty database, this will return 0. OldestID() uint64 // NewestID gets the ID of the newest log entry. // // For an empty database, this will return 0. NewestID() uint64 }
A LogDB is a log-structured database.
type MetaContinuityError ¶
MetaContinuityError means that the metadata for a chunk does not contain a contiguous sequence of entries.
func (*MetaContinuityError) Error ¶
func (e *MetaContinuityError) Error() string
type MetaOffsetError ¶
MetaOffsetError means that the metadata for a chunk does not contain a monotonically increasing sequence of entry ending offsets.
func (*MetaOffsetError) Error ¶
func (e *MetaOffsetError) Error() string
type PathError ¶
type PathError struct{ Err error }
PathError means that a directory could not be created. It wraps the actual error.
func (*PathError) WrappedErrors ¶
type PersistDB ¶
type PersistDB interface { // 'PersistDB' is an extension of 'LogDB'. LogDB // SetSync configures the database to synchronise the data after touching (appending, forgetting, // or rolling back) at most this many entries. // // <0 disables periodic syncing, and 'Sync' must be called instead. The default value is 256. // Both 0 and 1 cause a 'Sync' after every write. // // Returns a 'SyncError' value if this triggered an immediate synchronisation which failed, and // 'ErrClosed' if the handle is closed. SetSync(every int) error // Sync persists the data now. // // May return a SyncError value, and 'ErrClosed' if the handle is closed. Sync() error }
A PersistDB is a database which can be persisted in some fashion. In addition to defining methods, a 'PersistDB' changes some existing behaviours:
'Append', 'AppendEntries', 'Forget', 'Rollback', and 'Truncate' can now cause a 'Sync', if 'SetSync' has been called.
The above may return a 'SyncError' value if a periodic synchronisation failed.
type ReadError ¶
type ReadError struct{ Err error }
ReadError means that a read failed. It wraps the actual error.
func (*ReadError) WrappedErrors ¶
type SyncError ¶
type SyncError struct{ Err error }
SyncError means that a file could not be synced to disk. It wraps the actual error.
func (*SyncError) WrappedErrors ¶
type WriteError ¶
type WriteError struct{ Err error }
WriteError means that a write failed. It wraps the actual error.
func (*WriteError) Error ¶
func (e *WriteError) Error() string
func (*WriteError) WrappedErrors ¶
func (e *WriteError) WrappedErrors() []error