leveldb

README

leveldb

high performance key value database written in Go. The api is based on Google's leveldb. The implementation is based on the http://github.com/robaho/keydb.

Note: snapshot support is currently under development.

bulk insert and sequential read < 1 micro sec

random access read of disk based record < 4 micro secs

uses LSM trees, see https://en.wikipedia.org/wiki/Log-structured_merge-tree

limitation of max 1024 byte keys, to allow efficient on disk index searching, but has compressed keys which allows for very efficient storage of time series data (market tick data) in the same table

use the dbdump and dbload utilities to save/restore databases to a single file, but just zipping up the directory works as well...

see the related http://github.com/robaho/leveldbr which allows remote access to a leveldb instance, and allows a leveldb database to be shared by multiple processes

TODOs

make some settings configurable

purge removed key/value, it currently stores an empty []byte

snapshot support

seek to end, backwards iteration

How To Use

db, err := level.Open("test/mydb", leveldb.Options{})
if err != nil {
	t.Fatal("unable to create database", err)
}
err = db.Put([]byte("mykey"), []byte("myvalue"))
if err != nil {
	t.Fatal("unable to put key/Value", err)
}
err = db.Close()
if err != nil {
    t.Fatal("unable to close database", err)
}

Performance

Using example/performance.go

Using Go 1.19:
insert  time  10000000 records =  48406 ms, usec per op  4.8406189
close time  2535 ms
scan time  3943 ms, usec per op  0.3943393
scan time 50%  69 ms, usec per op  0.138772
random access time  3.490679 us per get
insert NoFlush time  10000000 records =  12235 ms, usec per op  1.2235342
close time  6556 ms
scan time  3442 ms, usec per op  0.3442262
scan time 50%  70 ms, usec per op  0.141112
random access time  3.636962 us per get
insert batch time  10000000 records =  11967 ms, usec per op  1.1967498
close time  8024 ms
scan time  3354 ms, usec per op  0.3354182
scan time 50%  65 ms, usec per op  0.13129
random access time  3.641579 us per get

Documentation

Index

• Constants
• Variables
• func IsValidDatabase(path string) error
• func MapError(err string) error
• func Remove(path string) error
• type Database
  • func Open(path string, options Options) (*Database, error)
  • func (db *Database) Close() error
  • func (db *Database) CloseWithMerge(segmentCount uint) error
  • func (db *Database) Get(key []byte) (value []byte, err error)
  • func (db *Database) Lookup(lower []byte, upper []byte) (LookupIterator, error)
  • func (db *Database) Put(key []byte, value []byte) error
  • func (db *Database) Remove(key []byte) ([]byte, error)
  • func (db *Database) Snapshot() (*Snapshot, error)
  • func (db *Database) Stats() Statistics
  • func (db *Database) Write(wb WriteBatch) error
• type Deleter
• type KeyComparison
• type KeyValue
  • func Key(key []byte) KeyValue
• type LookupIterator
• type Options
• type Snapshot
  • func (s *Snapshot) Close()
  • func (s *Snapshot) Get(key []byte) ([]byte, error)
  • func (s *Snapshot) Lookup(lower []byte, upper []byte) (LookupIterator, error)
• type Statistics
• type WriteBatch
  • func (wb *WriteBatch) Put(key []byte, value []byte)
  • func (wb *WriteBatch) Remove(key []byte)

Constants

const (
	DiscardPartial  batchReadMode = 0
	ApplyPartial    batchReadMode = 1
	ReturnOpenError batchReadMode = 2
)

Variables

var DatabaseClosed = errors.New("database closed")

var DatabaseCorrupted = errors.New("database corrupted, run repair")

var DatabaseInUse = errors.New("database in use")

var EmptyKey = errors.New("key is empty")

var EndOfIterator = errors.New("end of iterator")

var KeyNotFound = errors.New("key not found")

var KeyTooLong = errors.New("key too long, max 1024")

var NoDatabaseFound = errors.New("no database found")

var NotADirectory = errors.New("path is not a directory")

var NotValidDatabase = errors.New("path is not a valid database")

var ReadOnlySegment = errors.New("read only segment")

var SnapshotClosed = errors.New("snapshot closed")

Functions

func IsValidDatabase

func IsValidDatabase(path string) error

IsValidDatabase checks if the path points to a valid database or empty directory (which is also valid)

func MapError

func MapError(err string) error

MapError maps err to a leveldb error, or returns a new error for err

func Remove

func Remove(path string) error

Remove the database, deleting all files. the caller must be able to gain exclusive multi to the database

Types

type Database

type Database struct {
	sync.Mutex
	// contains filtered or unexported fields
}

Database reference is obtained via Open()

func Open

func Open(path string, options Options) (*Database, error)

Open a database. The database can only be opened by a single process, but the *Database reference can be shared across Go routines. The path is a directory name. if createIfNeeded is true, them if the db doesn't exist it will be created.

func (*Database) Close

func (db *Database) Close() error

Close the database. any memory segments are persisted to disk. The resulting segments are merged until the default maxSegments is reached

func (*Database) CloseWithMerge

func (db *Database) CloseWithMerge(segmentCount uint) error

CloseWithMerge closes the database with control of the segment count. if segmentCount is 0, then the merge process is skipped

func (*Database) Get

func (db *Database) Get(key []byte) (value []byte, err error)

Get a value for a key, error is non-nil if the key was not found or an error occurred

func (*Database) Lookup

func (db *Database) Lookup(lower []byte, upper []byte) (LookupIterator, error)

Lookup finds matching records between lower and upper inclusive. lower or upper can be nil and then the range is unbounded on that side. If the database is mutated during iteration, the returned results are undefined and are likely to be invalid in conjunction with a large number of mutations. A Snapshot should be used in this case.

func (*Database) Put

func (db *Database) Put(key []byte, value []byte) error

Put a key/value pair into the table, overwriting any existing entry. empty keys are not supported.

func (*Database) Remove

func (db *Database) Remove(key []byte) ([]byte, error)

Remove a key and its value from the table. empty keys are not supported.

func (*Database) Snapshot

func (db *Database) Snapshot() (*Snapshot, error)

Snapshot creates a read-only view of the database at a moment in time.

func (*Database) Stats

func (db *Database) Stats() Statistics

func (*Database) Write

func (db *Database) Write(wb WriteBatch) error

type Deleter

type Deleter interface {
	// contains filtered or unexported methods
}

type KeyComparison

type KeyComparison func([]byte, []byte) int

type KeyValue

type KeyValue struct {
	// contains filtered or unexported fields
}

func Key

func Key(key []byte) KeyValue

type LookupIterator

type LookupIterator interface {
	// Next returns EndOfIterator when complete, if err is nil, then key and value are valid
	Next() (key []byte, value []byte, err error)
	// contains filtered or unexported methods
}

LookupIterator iterator interface for table scanning. all iterators should be read until completion

type Options

type Options struct {
	// If true, then if the database does not exist on Open() it will be created.
	CreateIfNeeded bool
	// The database segments are periodically merged to enforce MaxSegments.
	// If this is true, the merging only occurs during Close().
	DisableAutoMerge bool
	// Maximum number of segments per database which controls the number of open files.
	// If the number of segments exceeds 2x this value, producers are paused while the
	// segments are merged.
	MaxSegments uint
	// Maximum size of memory segment in bytes. Maximum memory usage per database is
	// roughly MaxSegments * MaxMemoryBytes but can be higher based on producer rate.
	MaxMemoryBytes uint64
	// Disable flush to disk when writing to increase performance.
	DisableWriteFlush bool
	// Force sync to disk when writing. If true, then DisableWriteFlush is ignored.
	EnableSyncWrite bool
	// Determines handling of partial batches during Open()
	BatchReadMode batchReadMode
	// Key comparison function or nil to use standard bytes.Compare
	UserKeyCompare KeyComparison
}

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is a read-only view of the database at a moment in time. A Snapshot can be used by multiple go routines, but access across Close() and other operations must be externally synchronized.

func (*Snapshot) Close

func (s *Snapshot) Close()

Close frees any resources used by the Snapshot. This is optional and instead simply setting the Snapshot reference to nil will eventually free the resources.

func (*Snapshot) Get

func (s *Snapshot) Get(key []byte) ([]byte, error)

func (*Snapshot) Lookup

func (s *Snapshot) Lookup(lower []byte, upper []byte) (LookupIterator, error)

type Statistics

type Statistics struct {
	NumberOfSegments int
}

type WriteBatch

type WriteBatch struct {
	// contains filtered or unexported fields
}

func (*WriteBatch) Put

func (wb *WriteBatch) Put(key []byte, value []byte)

func (*WriteBatch) Remove

func (wb *WriteBatch) Remove(key []byte)