README
¶
EagleDB
EagleDB is a fast and simple key-value store written in Golang. Design choices have been inspired by Bitcask.
Usage
From a code perspective, you can use Eagle just like a simple map.
import (
"log"
"github.com/ostafen/eagle"
)
func main() {
db, _ := eagle.Open(eagle.DefaultOptions("/tmp/eagle-db"))
defer db.Close()
db.Put([]byte("name"), []byte("Bob"))
val, _ := db.Get([]byte("name"))
log.Println(string(val))
}
Features
Eagle has been designed to handle an high volume of read/write operations. It shares serveral concepts and principles with log-structured databases. Records are stored in append-only files, while keys are held in an in-memory structure directly pointing to the locations on disk where the data resides. Because of this design choice, the system must have enough memory to hold the entire key space. However, this enables several major advantages. First of all, each read and write operation has a predictable performance, since a single seek is required to retrieve any value or to append a new record to the end of the current file that is open for writing. Moreover, since it is not required for data to be sorted, I/O and disk bandwidth are not satured that easily. Also, in the event of a power loss and data corruption, at most two files must be repaired (the one used by the write thread and the one used by the compaction thread), thus allowing for fast recovery times.
In-memory structure
Eagle uses an hash table to store the key space, where collisions are solved by chaining. Hence, each lookup, insert and remove operation is espected to take O(1) time on the average. When the number of elements stored in the table grows, buckets are resized through the incremental rehashing technique used by Redis. Moreover, to allow for concurrent usage, the hash table is partitioned into multiple indipendent thread-safe tables.
Strong encryption with minimal overhead
Encryption is a fundamental requirement for protecting the privacy of data. Hence, any modern database is expected to natively support encryption. When enabled, Eagle encrypts data through the AES algorithm, using the Counter (CTR) mode of operation, which is an industry standard. Such mode implies an additional 16 bytes initialization vector (IV), which ensures that the cyphertext will be different, even when encrypting the same block multiple times. However, generating a new IV for each piece of data which is appended to the log or index files would waste a lot of space. If you consider that the average size of an index entry is around 20 bytes, prepending a new IV to each entry would cause a storage overhead of 80%. To avoid wasting so much space, Eagle makes use of a technique which have been introduced in the Badger database. For each log file (or index file), a unique 12-byte IV is used to encrypt all the entries in the file. Before an entry is actually encrypted, the 12-byte IV is attached a 4-byte unique value, representing the file offset of the value being written, thus forming a 16-byte full IV. Note that the overhead of this method is less than adding an additional entry to each file.
Notes
This project is still under development and not ready for production use. If you want, feel free to contribute.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidFile = errors.New("invalid file")
Functions ¶
func SaveToFile ¶
Types ¶
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
func Open ¶
Open create a new Eagle instance in the target directory and loads existing data from disk.
func (*DB) Close ¶
Close closed the current instance of the db. Any subsequent map operation will no more be synched on disk
func (*DB) ContainsKey ¶
ContainsKey returns true if this map contains a mapping for the specified key.
func (*DB) Get ¶
Get returns the value to which the specified key is mapped, or null if this map contains no mapping for the key.
func (*DB) Put ¶
Associates the specified value with the specified key in this map (optional operation). If the map previously contained a mapping for the key, the old value is replaced by the specified value.
type Options ¶
type Options struct {
MaxFileSize uint32
MaxTombstoneFileSize uint32
RootDir string
FileCompactionThreshold float64
EncryptKey []byte
}
func DefaultOptions ¶
DefaultOptions returns a list of recommended options for good performance.
func (*Options) UseEncryption ¶
func (Options) WithEncryptionKey ¶
WithEncryptionKey sets the encryption key which will be used for crypting and decrypting each frame.
func (Options) WithFileCompactionThreshold ¶
WithMaxStaleDataThreshold represents the threshold of stale data above which a bucket is rewritten from scratch. In particular, for a given file f, if staleSize(f) / size(f) > threshold and size(f) > minRewriteFileSize, then a bucket is entirely rewritten, and the old file is discarded.
func (Options) WithMaxFileSize ¶
type ValuePointer ¶
type ValuePointer struct {
FileId uint32
// contains filtered or unexported fields
}
Source Files
Directories