leveldb

README

leveldb

high performance key value database written in Go. The api is based on Google's leveldb. The implementation is based on the http://github.com/robaho/keydb.

Note: snapshot support is currently under development.

bulk insert and sequential read < 1 micro sec

random access read of disk based record < 4 micro secs

uses LSM trees, see https://en.wikipedia.org/wiki/Log-structured_merge-tree

limitation of max 1024 byte keys, to allow efficient on disk index searching, but has compressed keys which allows for very efficient storage of time series data (market tick data) in the same table

use the dbdump and dbload utilities to save/restore databases to a single file, but just zipping up the directory works as well...

see the related http://github.com/robaho/leveldbr which allows remote access to a leveldb instance, and allows a leveldb database to be shared by multiple processes

TODOs

make some settings configurable

purge removed key/value, it currently stores an empty []byte

snapshot support

How To Use

db, err := level.Open("test/mydb", leveldb.Options{})
if err != nil {
	t.Fatal("unable to create database", err)
}
err = db.Put([]byte("mykey"), []byte("myvalue"))
if err != nil {
	t.Fatal("unable to put key/Value", err)
}
err = db.Close()
if err != nil {
    t.Fatal("unable to close database", err)
}

Performance

Using example/performance.go

Using Go 1.15.5:

insert time  10000000 records =  17890 ms, usec per op  1.7890143
close time  8477 ms
scan time  2887 ms, usec per op  0.2887559
scan time 50%  81 ms, usec per op  0.162584
random access time  3.508029 us per get
close with merge 1 time  0.148 ms
scan time  2887 ms, usec per op  0.2887248
scan time 50%  85 ms, usec per op  0.171406
random access time  3.487226 us per get

Documentation

Index

• Variables
• func IsValidDatabase(path string) error
• func KeyValueCompare(a KeyValue, b KeyValue) int
• func Remove(path string) error
• type Database
  • func Open(path string, options Options) (*Database, error)
  • func (db *Database) Close() error
  • func (db *Database) CloseWithMerge(segmentCount int) error
  • func (db *Database) Get(key []byte) (value []byte, err error)
  • func (db *Database) Lookup(lower []byte, upper []byte) (LookupIterator, error)
  • func (db *Database) Put(key []byte, value []byte) error
  • func (db *Database) Remove(key []byte) ([]byte, error)
  • func (db *Database) Write(wb WriteBatch) error
• type Deleter
• type KeyValue
  • func Key(key []byte) KeyValue
• type LookupIterator
• type Options
• type SkipList
  • func NewSkipList[K any](cmp func(K, K) int) SkipList[K]
  • func (s *SkipList[K]) Contains(key K) bool
  • func (s *SkipList[K]) Iterator() iterator[K]
  • func (s *SkipList[K]) Put(key K) K
• type WriteBatch
  • func (wb *WriteBatch) Put(key []byte, value []byte)
  • func (wb *WriteBatch) Remove(key []byte)

Variables

var DatabaseClosed = errors.New("database closed")

var DatabaseCorrupted = errors.New("database corrupted, run repair")

var DatabaseInUse = errors.New("database in use")

var EmptyKey = errors.New("key is empty")

var EndOfIterator = errors.New("end of iterator")

var KeyNotFound = errors.New("key not found")

var KeyTooLong = errors.New("key too long, max 1024")

var NoDatabaseFound = errors.New("no database found")

var NotADirectory = errors.New("path is not a directory")

var NotValidDatabase = errors.New("path is not a valid database")

var ReadOnlySegment = errors.New("read only segment")

Functions

func IsValidDatabase

func IsValidDatabase(path string) error

IsValidDatabase checks if the path points to a valid database or empty directory (which is also valid)

func KeyValueCompare

func KeyValueCompare(a KeyValue, b KeyValue) int

func Remove

func Remove(path string) error

Remove the database, deleting all files. the caller must be able to gain exclusive multi to the database

Types

type Database

type Database struct {
	sync.Mutex
	// contains filtered or unexported fields
}

Database reference is obtained via Open()

func Open

func Open(path string, options Options) (*Database, error)

Open a database. The database can only be opened by a single process, but the *Database reference can be shared across Go routines. The path is a directory name. if createIfNeeded is true, them if the db doesn't exist it will be created.

func (*Database) Close

func (db *Database) Close() error

Close the database. any memory segments are persisted to disk. The resulting segments are merged until the default maxSegments is reached

func (*Database) CloseWithMerge

func (db *Database) CloseWithMerge(segmentCount int) error

CloseWithMerge closes the database with control of the segment count. if segmentCount is 0, then the merge process is skipped

func (*Database) Get

func (db *Database) Get(key []byte) (value []byte, err error)

Get a value for a key, error is non-nil if the key was not found or an error occurred

func (*Database) Lookup

func (db *Database) Lookup(lower []byte, upper []byte) (LookupIterator, error)

Lookup finds matching record between lower and upper inclusive. lower or upper can be nil and then the range is unbounded on that side. Using the iterator after the transaction has been Commit/Rollback is not supported.

func (*Database) Put

func (db *Database) Put(key []byte, value []byte) error

Put a key/value pair into the table, overwriting any existing entry. empty keys are not supported.

func (*Database) Remove

func (db *Database) Remove(key []byte) ([]byte, error)

Remove a key and its value from the table. empty keys are not supported.

func (*Database) Write

func (db *Database) Write(wb WriteBatch) error

type Deleter

type Deleter interface {
	// contains filtered or unexported methods
}

type KeyValue

type KeyValue struct {
	// contains filtered or unexported fields
}

func Key

func Key(key []byte) KeyValue

type LookupIterator

type LookupIterator interface {
	// Next returns EndOfIterator when complete, if err is nil, then key and value are valid
	Next() (key []byte, value []byte, err error)
	// contains filtered or unexported methods
}

LookupIterator iterator interface for table scanning. all iterators should be read until completion

type Options

type Options struct {
	CreateIfNeeded bool
	DisableBgMerge bool
	// Maximum number of segments per database. Controls the number of open files.
	MaxSegments int
	// Maximum size of memory segment in bytes.
	MaxMemoryBytes int
}

type SkipList

type SkipList[K any] struct {
	// contains filtered or unexported fields
}

func NewSkipList

func NewSkipList[K any](cmp func(K, K) int) SkipList[K]

func (*SkipList[K]) Contains

func (s *SkipList[K]) Contains(key K) bool

func (*SkipList[K]) Iterator

func (s *SkipList[K]) Iterator() iterator[K]

func (*SkipList[K]) Put

func (s *SkipList[K]) Put(key K) K

type WriteBatch

type WriteBatch struct {
	// contains filtered or unexported fields
}

func (*WriteBatch) Put

func (wb *WriteBatch) Put(key []byte, value []byte)

func (*WriteBatch) Remove

func (wb *WriteBatch) Remove(key []byte)