README
¶
lazydb
Package lazydb is a storage of byte sequences for Go with automatic key generation. 
Documentation
¶
Overview ¶
Package lazydb is a storage of byte sequences for Go with automatic key generation.
Basics ¶
LazyDB is a key/value database library, where keys are 32 bit unsigned integer numbers and values are slotted to contain multiple byte sequences of arbitrary length.
Instead of directly implementing indexing algorithms, LazyDB takes an experimental approach where filesystem directory structure is used for indexing data saved on disk (hence the "lazy" part of its name).
Key Mapping Internals ¶
Apart from storage implementations that map a single file as the database, LazyDB relies on subdirectories of the filesystem for managing keys. Therefore the filesystem chosen for storage is the real engine that maps keys to values, and their designers are the ones who must be given credit if package LazyDB happens to achieve satisfactory performance.
Each key-slot pair is uniquely associated with a distinct file in the filesystem. The path to the file is derived from the key-slot, eg. slot 0 of key 0x12345678, assuming the numeric base for key mapping is set to 16, is file 1/2/3/4/5/6/7/8/0 under the database directory. The data associated with the slot is the content of the file. Conversely, keys in the database are retrieved by parsing paths to existent files.
When creating a new database, user may choose the numeric base for internal key mapping. The base can range from MinBase to MaxBase, and it was designed to allow LazyDB to be tuned for the filesystem at use. The default base is 16, meaning that a uint32 key requires 8 subdirectories to be mapped.
Whether the numeric base chosen for internal key mapping, LazyDB uses single unicode characters to name files and subdirectories whithin the filesystem and manage key mapping. The first 10 characters in the mapping range are decimal digits from 0 to 9, and the next 26 ones are upper case letters from A to Z.
Issues ¶
Although LazyDB write methods commit changes to filesystem immediately on successful return, commited data may reside temporarily in on memory filesystem's caches. Users may need to manually flush updates to disk (eg sync, umount) to guarantee that all updates to the database are written to disk.
Wipe method can take a long time to return.
Wish List ¶
Document filesystem guidelines for better performance with package lazydb.
Example ¶
package main
import (
"github.com/coolparadox/go/storage/lazydb"
"bytes"
"os"
"io"
"fmt"
)
func main() {
// error handling purposely ignored
// in some places for didactic purposes.
// Create an empty database
dbPath := "/tmp/my_db"
os.MkdirAll(dbPath, 0755)
lazydb.Wipe(dbPath)
db, _ := lazydb.New(dbPath, 0)
// Save values in new keys
k1, _, _ := db.Save([]io.Reader{bytes.NewReader([]byte("goodbye"))})
k2, _, _ := db.Save([]io.Reader{bytes.NewReader([]byte("cruel"))})
k3, _, _ := db.Save([]io.Reader{bytes.NewReader([]byte("world"))})
// Update, remove
db.SaveAs(k1, []io.Reader{bytes.NewReader([]byte("hello"))})
db.Erase(k2)
db.SaveAs(k3, []io.Reader{bytes.NewReader([]byte("folks"))})
// Loop through keys
key, err := db.FindKey(0, true)
for err == nil {
// Print value
val := new(bytes.Buffer)
db.Load(key, []io.Writer{val})
fmt.Printf("key %v: %s\n", key, string(val.Bytes()))
if key >= lazydb.MaxKey {
// Maximum key reached
break
}
// Find next existent key
key, err = db.FindKey(key+1, true)
}
if err != nil && err != lazydb.KeyNotFoundError {
// An abnormal error occurred
panic(err)
}
}
Output: key 0: hello key 2: folks
Index ¶
- Constants
- Variables
- func Wipe(dir string) error
- type LazyDB
- func (db LazyDB) Erase(key uint32) error
- func (db LazyDB) Exists(key uint32, slot uint32) (bool, error)
- func (db LazyDB) FindKey(key uint32, ascending bool) (uint32, error)
- func (db LazyDB) Load(key uint32, dst []io.Writer) ([]int64, error)
- func (db LazyDB) Save(src []io.Reader) (uint32, []int64, error)
- func (db LazyDB) SaveAs(key uint32, src []io.Reader) ([]int64, error)
Examples ¶
Constants ¶
const ( MinBase = 2 MaxBase = 0x10000 )
MinBase and MaxBase define the range of possible values of the numeric base for internal key mapping in the filesystem (see parameter base in New).
const ( Depth2Base = 0x10000 Depth4Base = 0x100 Depth8Base = 0x10 Depth16Base = 0x4 Depth32Base = 0x2 )
Depth*Base are convenience values of numeric bases for internal key mapping to be used when creating a new database. These values give the most efficient occupation of subdirectories in the filesystem (see Key Mapping Internals).
const MaxKey = 0xFFFFFFFF
MaxKey represents the maximum value of a key.
Variables ¶
var KeyNotFoundError = errors.New("key not found")
KeyNotFoundError is returned by FindKey when there are no keys available.
Functions ¶
Types ¶
type LazyDB ¶
type LazyDB struct {
// contains filtered or unexported fields
}
LazyDB is a handler to a LazyDB database in the filesystem.
func New ¶
New creates a new LazyDB database in the filesystem, or opens an existent one.
Parameter dir is an absolute path to a directory in the filesystem for storing the database. If it's the first time this directory is used by package lazydb, it must be empty.
Parameter keyBase is the numeric base for naming files and subdirectories under the database (see: Key Mapping Internals, Depth*Base constants). It has effect only during creation of a new database (it's ignored when opening an existent database). Pass zero for a sane default.
func (LazyDB) FindKey ¶
FindKey takes a key and returns it if it exists. If key does not exist, the closest key in ascending (or descending) order is returned instead.
KeyNotFoundError is returned if there are no keys to be answered.
func (LazyDB) Load ¶
Load retrieves data from previously saved value slots.
All non nil elements of dst are written with data from corresponding slots.
Returns the number of bytes written to dst elements, and the first error encountered during operation.
func (LazyDB) Save ¶
Save creates a new key and updates it with given value slots. Key is automatically assigned and guaranteed to be new.
For all non nil elements of src, data is read until EOF is reached, and corresponding slots are initialized with read data.
Returns the assigned key, the number of bytes read from src elements, and the first error encountered during operation.
func (LazyDB) SaveAs ¶
SaveAs updates value slots for a given key. Key is created if absent.
For all non nil elements of src, data is read until EOF is reached, and corresponding slots are updated with read data. Previously existent slots corresponding to nil or missing elements of src are left untouched.
Returns the numbers of bytes read from src elements, and the first error encountered during operation.
Source Files