levigo

package
v2.1.3+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 16, 2019 License: Apache-2.0, MIT Imports: 4 Imported by: 0

README

LevelDB

This repo is a fork from DataDog/leveldb, while DataDog/leveldb itself is a fork of jmhodges/levigo.

This fork makes levigo go-gettable by including the C++ source for LevelDB and building it with the Go code. LevelDB and Snappy are both compiled and linked statically, so while you will not need them installed on your target machine, you should have a roughly compatible version of libstdc++.

This package currently includes LevelDB 1.19, and supports Linux and OS X. It should work on Go 1.5 or greater.

godoc documentation.

Building

This package supports Linux and OS X.

You'll need gcc and g++. Then:

go get github.com/DataDog/leveldb

This will take a long time as it compiles the LevelDB source code along with the Go library.

To avoid waiting for compilation every time you want to build your project, you can run:

go install github.com/DataDog/leveldb

Updating Vendored Snappy

  1. Overwrite source in ./deps/snappy/
  2. make link_snappy

If nothing significant in snappy has changed, things should JustWork™

Updating Vendored LevelDB

The embedded LevelDB source code is located in deps/leveldb. In order to get go build to compile LevelDB, we symlink the required LevelDB source files to the root directory of the project. These symlinks are prefixed with deps_.

To change the embedded version of LevelDB, do the following:

  1. rm deps_*.cc
  2. Replace deps/leveldb with the source code of the desired version of LevelDB
  3. On Linux, run ./deps/leveldb/build_detect_platform linux.mk build_flags
  4. On OS X, run ./deps/leveldb/build_detect_platform darwin.mk build_flags
  5. cat build_flags/*, take a quick look at the new compiler and linker flags and see if there are any flags we're missing in cgo_flags_*.go. If so, add them.
  6. make link_leveldb
  7. On OS X and Linux, run go build and verify that everything compiles.

note I couldn't get build_detect_platform to run as advertised above -- j

Caveats

Comparators and WriteBatch iterators must be written in C in your own library. This seems like a pain in the ass, but remember that you'll have the LevelDB C API available to your in your client package when you import levigo.

An example of writing your own Comparator can be found in https://github.com/DataDog/leveldb/blob/master/examples.

Documentation

Overview

Package levigo provides the ability to create and access LevelDB databases.

levigo.Open opens and creates databases.

opts := levigo.NewOptions()
opts.SetCache(levigo.NewLRUCache(3<<30))
opts.SetCreateIfMissing(true)
db, err := levigo.Open("/path/to/db", opts)

The DB struct returned by Open provides DB.Get, DB.Put and DB.Delete to modify and query the database.

ro := levigo.NewReadOptions()
wo := levigo.NewWriteOptions()
// if ro and wo are not used again, be sure to Close them.
data, err := db.Get(ro, []byte("key"))
...
err = db.Put(wo, []byte("anotherkey"), data)
...
err = db.Delete(wo, []byte("key"))

For bulk reads, use an Iterator. If you want to avoid disturbing your live traffic while doing the bulk read, be sure to call SetFillCache(false) on the ReadOptions you use when creating the Iterator.

ro := levigo.NewReadOptions()
ro.SetFillCache(false)
it := db.NewIterator(ro)
defer it.Close()
for it.Seek(mykey); it.Valid(); it.Next() {
	munge(it.Key(), it.Value())
}
if err := it.GetError(); err != nil {
	...
}

Batched, atomic writes can be performed with a WriteBatch and DB.Write.

wb := levigo.NewWriteBatch()
// defer wb.Close or use wb.Clear and reuse.
wb.Delete([]byte("removed"))
wb.Put([]byte("added"), []byte("data"))
wb.Put([]byte("anotheradded"), []byte("more"))
err := db.Write(wo, wb)

If your working dataset does not fit in memory, you'll want to add a bloom filter to your database. NewBloomFilter and Options.SetFilterPolicy is what you want. NewBloomFilter is amount of bits in the filter to use per key in your database.

filter := levigo.NewBloomFilter(10)
opts.SetFilterPolicy(filter)
db, err := levigo.Open("/path/to/db", opts)

If you're using a custom comparator in your code, be aware you may have to make your own filter policy object.

This documentation is not a complete discussion of LevelDB. Please read the LevelDB documentation <http://code.google.com/p/leveldb> for information on its operation. You'll find lots of goodies there.

Index

Constants

View Source
const (
	NoCompression     = CompressionOpt(0)
	SnappyCompression = CompressionOpt(1)
	LZ4Compression    = CompressionOpt(2)
)

Known compression arguments for Options.SetCompression.

Variables

View Source
var ErrDBClosed = errors.New("database is closed")

Functions

func DestroyComparator

func DestroyComparator(cmp *C.leveldb_comparator_t)

DestroyComparator deallocates a *C.leveldb_comparator_t.

This is provided as a convienience to advanced users that have implemented their own comparators in C in their own code.

func DestroyDatabase

func DestroyDatabase(dbname string, o *Options) error

DestroyDatabase removes a database entirely, removing everything from the filesystem.

func Exists

func Exists(it *Iterator, key []byte) bool

Exists will seek the iterator to the given key and return true if it exists.

func GetLevelDBMajorVersion

func GetLevelDBMajorVersion() int

func GetLevelDBMinorVersion

func GetLevelDBMinorVersion() int

func RepairDatabase

func RepairDatabase(dbname string, o *Options) error

RepairDatabase attempts to repair a database.

If the database is unrepairable, an error is returned.

func SnappySupported

func SnappySupported() bool

Types

type Cache

type Cache struct {
	Cache *C.leveldb_cache_t
}

Cache is a cache used to store data read from data in memory.

Typically, NewLRUCache is all you will need, but advanced users may implement their own *C.leveldb_cache_t and create a Cache.

To prevent memory leaks, a Cache must have Close called on it when it is no longer needed by the program. Note: if the process is shutting down, this may not be necessary and could be avoided to shorten shutdown time.

func NewLRUCache

func NewLRUCache(capacity int) *Cache

NewLRUCache creates a new Cache object with the capacity given.

To prevent memory leaks, Close should be called on the Cache when the program no longer needs it. Note: if the process is shutting down, this may not be necessary and could be avoided to shorten shutdown time.

func (*Cache) Close

func (c *Cache) Close()

Close deallocates the underlying memory of the Cache object.

type CompressionOpt

type CompressionOpt int

CompressionOpt is a value for Options.SetCompression.

type DB

type DB struct {
	Ldb *C.leveldb_t
	// contains filtered or unexported fields
}

DB is a reusable handle to a LevelDB database on disk, created by Open.

To avoid memory and file descriptor leaks, call Close when the process no longer needs the handle. Calls to any DB method made after Close will panic.

The DB instance may be shared between goroutines. The usual data race conditions will occur if the same key is written to from more than one, of course.

func Open

func Open(dbname string, o *Options) (*DB, error)

Open opens a database.

Creating a new database is done by calling SetCreateIfMissing(true) on the Options passed to Open.

It is usually wise to set a Cache object on the Options with SetCache to keep recently used data from that database in memory.

func (*DB) Close

func (db *DB) Close()

Close closes the database, rendering it unusable for I/O, by deallocating the underlying handle.

Any attempts to use the DB after Close is called will panic.

func (*DB) CompactRange

func (db *DB) CompactRange(r Range)

CompactRange runs a manual compaction on the Range of keys given. This is not likely to be needed for typical usage.

func (*DB) Delete

func (db *DB) Delete(wo *WriteOptions, key []byte) error

Delete removes the data associated with the key from the database.

The key byte slice may be reused safely. Delete takes a copy of them before returning. The WriteOptions passed in can be reused by multiple calls to this and if the WriteOptions is left unchanged.

func (*DB) Get

func (db *DB) Get(ro *ReadOptions, key []byte) ([]byte, error)

Get returns the data associated with the key from the database.

If the key does not exist in the database, a nil []byte is returned. If the key does exist, but the data is zero-length in the database, a zero-length []byte will be returned.

The key byte slice may be reused safely. Get takes a copy of them before returning.

func (*DB) GetApproximateSizes

func (db *DB) GetApproximateSizes(ranges []Range) []uint64

GetApproximateSizes returns the approximate number of bytes of file system space used by one or more key ranges.

The keys counted will begin at Range.Start and end on the key before Range.Limit.

func (*DB) NewIterator

func (db *DB) NewIterator(ro *ReadOptions) *Iterator

NewIterator returns an Iterator over the the database that uses the ReadOptions given.

Often, this is used for large, offline bulk reads while serving live traffic. In that case, it may be wise to disable caching so that the data processed by the returned Iterator does not displace the already cached data. This can be done by calling SetFillCache(false) on the ReadOptions before passing it here.

Similarly, ReadOptions.SetSnapshot is also useful.

The ReadOptions passed in can be reused by multiple calls to this and other methods if the ReadOptions is left unchanged.

func (*DB) NewSnapshot

func (db *DB) NewSnapshot() *Snapshot

NewSnapshot creates a new snapshot of the database.

The Snapshot, when used in a ReadOptions, provides a consistent view of state of the database at the the snapshot was created.

To prevent memory leaks and resource strain in the database, the snapshot returned must be released with DB.ReleaseSnapshot method on the DB that created it.

See the LevelDB documentation for details.

func (*DB) PropertyValue

func (db *DB) PropertyValue(propName string) string

PropertyValue returns the value of a database property.

Examples of properties include "leveldb.stats", "leveldb.sstables", and "leveldb.num-files-at-level0".

func (*DB) Put

func (db *DB) Put(wo *WriteOptions, key, value []byte) error

Put writes data associated with a key to the database.

If a nil []byte is passed in as value, it will be returned by Get as an zero-length slice. The WriteOptions passed in can be reused by multiple calls to this and if the WriteOptions is left unchanged.

The key and value byte slices may be reused safely. Put takes a copy of them before returning.

func (*DB) ReleaseSnapshot

func (db *DB) ReleaseSnapshot(snap *Snapshot)

ReleaseSnapshot removes the snapshot from the database's list of snapshots, and deallocates it.

func (*DB) Write

func (db *DB) Write(wo *WriteOptions, w *WriteBatch) error

Write atomically writes a WriteBatch to disk. The WriteOptions passed in can be reused by multiple calls to this and other methods.

type DatabaseError

type DatabaseError string

func (DatabaseError) Error

func (e DatabaseError) Error() string

type Env

type Env struct {
	Env *C.leveldb_env_t
}

Env is a system call environment used by a database.

Typically, NewDefaultEnv is all you need. Advanced users may create their own Env with a *C.leveldb_env_t of their own creation.

To prevent memory leaks, an Env must have Close called on it when it is no longer needed by the program.

func NewDefaultEnv

func NewDefaultEnv() *Env

NewDefaultEnv creates a default environment for use in an Options.

To prevent memory leaks, the Env returned should be deallocated with Close.

func (*Env) Close

func (env *Env) Close()

Close deallocates the Env, freeing the underlying struct.

type FilterPolicy

type FilterPolicy struct {
	Policy *C.leveldb_filterpolicy_t
}

FilterPolicy is a factory type that allows the LevelDB database to create a filter, such as a bloom filter, that is stored in the sstables and used by DB.Get to reduce reads.

An instance of this struct may be supplied to Options when opening a DB. Typical usage is to call NewBloomFilter to get an instance.

To prevent memory leaks, a FilterPolicy must have Close called on it when it is no longer needed by the program.

func NewBloomFilter

func NewBloomFilter(bitsPerKey int) *FilterPolicy

NewBloomFilter creates a filter policy that will create a bloom filter when necessary with the given number of bits per key.

See the FilterPolicy documentation for more.

func (*FilterPolicy) Close

func (fp *FilterPolicy) Close()

type Iterator

type Iterator struct {
	Iter *C.leveldb_iterator_t
}

Iterator is a read-only iterator through a LevelDB database. It provides a way to seek to specific keys and iterate through the keyspace from that point, as well as access the values of those keys.

Care must be taken when using an Iterator. If the method Valid returns false, calls to Key, Value, Next, and Prev will result in panics. However, Seek, SeekToFirst, SeekToLast, GetError, Valid, and Close will still be safe to call.

GetError will only return an error in the event of a LevelDB error. It will return a nil on iterators that are simply invalid. Given that behavior, GetError is not a replacement for a Valid.

A typical use looks like:

db := levigo.Open(...)

it := db.NewIterator(readOpts)
defer it.Close()
for it.Seek(mykey); it.Valid(); it.Next() {
	useKeyAndValue(it.Key(), it.Value())
}
if err := it.GetError() {
	...
}

To prevent memory leaks, an Iterator must have Close called on it when it is no longer needed by the program.

func (*Iterator) Close

func (it *Iterator) Close()

Close deallocates the given Iterator, freeing the underlying C struct.

func (*Iterator) GetError

func (it *Iterator) GetError() error

GetError returns an IteratorError from LevelDB if it had one during iteration.

This method is safe to call when Valid returns false.

func (*Iterator) Key

func (it *Iterator) Key() []byte

Key returns a copy the key in the database the iterator currently holds.

If Valid returns false, this method will panic.

func (*Iterator) Next

func (it *Iterator) Next()

Next moves the iterator to the next sequential key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

If Valid returns false, this method will panic.

func (*Iterator) Prev

func (it *Iterator) Prev()

Prev moves the iterator to the previous sequential key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

If Valid returns false, this method will panic.

func (*Iterator) Seek

func (it *Iterator) Seek(key []byte)

Seek moves the iterator the position of the key given or, if the key doesn't exist, the next key that does exist in the database. If the key doesn't exist, and there is no next key, the Iterator becomes invalid.

This method is safe to call when Valid returns false.

func (*Iterator) SeekToFirst

func (it *Iterator) SeekToFirst()

SeekToFirst moves the iterator to the first key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

This method is safe to call when Valid returns false.

func (*Iterator) SeekToLast

func (it *Iterator) SeekToLast()

SeekToLast moves the iterator to the last key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

This method is safe to call when Valid returns false.

func (*Iterator) Valid

func (it *Iterator) Valid() bool

Valid returns false only when an Iterator has iterated past either the first or the last key in the database.

func (*Iterator) Value

func (it *Iterator) Value() []byte

Value returns a copy of the value in the database the iterator currently holds.

If Valid returns false, this method will panic.

type IteratorError

type IteratorError string

func (IteratorError) Error

func (e IteratorError) Error() string

type MultiKeyError

type MultiKeyError struct {
	// contains filtered or unexported fields
}

MultiKeyError encapsulates multiple errors encountered in a Get/Put-Many() call, behind the errors.error interface. Caller may cast the generic error object to a MultiKeyError and call ErrorsByKeyIdx() to get all the errors by the failed key index.

func (*MultiKeyError) Error

func (mke *MultiKeyError) Error() string

func (*MultiKeyError) ErrorsByKeyIdx

func (mke *MultiKeyError) ErrorsByKeyIdx() map[int]error

ErrorsByKeyIdx() returns a result map of keyIdx => error. e.g. If one calls GetMany(keys) and keys[2], keys[3], keys[7] failed, then result[2] will hold the error for keys[2], result[3] for keys[3] and result[7] the error for keys[7]

func (*MultiKeyError) GoString

func (mke *MultiKeyError) GoString() string

type Options

type Options struct {
	Opt *C.leveldb_options_t
}

Options represent all of the available options when opening a database with Open. Options should be created with NewOptions.

It is usually with to call SetCache with a cache object. Otherwise, all data will be read off disk.

To prevent memory leaks, Close must be called on an Options when the program no longer needs it.

func NewOptions

func NewOptions() *Options

NewOptions allocates a new Options object. Note that the CompressionLevel is set to NoCompression by default rather than SnappyCompression.

func (*Options) Close

func (o *Options) Close()

Close deallocates the Options, freeing its underlying C struct.

func (*Options) SetBlockRestartInterval

func (o *Options) SetBlockRestartInterval(n int)

SetBlockRestartInterval is the number of keys between restarts points for delta encoding keys.

Most clients should leave this parameter alone. See the LevelDB documentation for details.

func (*Options) SetBlockSize

func (o *Options) SetBlockSize(s int)

SetBlockSize sets the approximate size of user data packed per block.

The default is roughly 4096 uncompressed bytes. A better setting depends on your use case. See the LevelDB documentation for details.

func (*Options) SetCache

func (o *Options) SetCache(cache *Cache)

SetCache places a cache object in the database when a database is opened.

This is usually wise to use. See also ReadOptions.SetFillCache.

func (*Options) SetComparator

func (o *Options) SetComparator(cmp *C.leveldb_comparator_t)

SetComparator sets the comparator to be used for all read and write operations.

The comparator that created a database must be the same one (technically, one with the same name string) that is used to perform read and write operations.

The default comparator is usually sufficient.

func (*Options) SetCompression

func (o *Options) SetCompression(t CompressionOpt)

SetCompression sets whether to compress blocks using the specified compresssion algorithm.

The default value is NoCompression, but either SnappyCompression or LZ4Compression are both fast enough that it is unlikely you want to keep them off.

If the LevelDB library was built without Snappy compression enabled, the SnappyCompression setting will be ignored.

func (*Options) SetCreateIfMissing

func (o *Options) SetCreateIfMissing(b bool)

SetCreateIfMissing causes Open to create a new database on disk if it does not already exist.

func (*Options) SetEnv

func (o *Options) SetEnv(env *Env)

SetEnv sets the Env object for the new database handle.

func (*Options) SetErrorIfExists

func (o *Options) SetErrorIfExists(error_if_exists bool)

SetErrorIfExists, if passed true, will cause the opening of a database that already exists to throw an error.

func (*Options) SetFilterPolicy

func (o *Options) SetFilterPolicy(fp *FilterPolicy)

SetFilterPolicy causes Open to create a new database that will uses filter created from the filter policy passed in.

func (*Options) SetInfoLog

func (o *Options) SetInfoLog(log *C.leveldb_logger_t)

SetInfoLog sets a *C.leveldb_logger_t object as the informational logger for the database.

func (*Options) SetMaxOpenFiles

func (o *Options) SetMaxOpenFiles(n int)

SetMaxOpenFiles sets the number of files than can be used at once by the database.

See the LevelDB documentation for details.

func (*Options) SetParanoidChecks

func (o *Options) SetParanoidChecks(pc bool)

SetParanoidChecks, when called with true, will cause the database to do aggressive checking of the data it is processing and will stop early if it detects errors.

See the LevelDB documentation docs for details.

func (*Options) SetWriteBufferSize

func (o *Options) SetWriteBufferSize(s int)

SetWriteBufferSize sets the number of bytes the database will build up in memory (backed by an unsorted log on disk) before converting to a sorted on-disk file.

type Range

type Range struct {
	Start []byte
	Limit []byte
}

Range is a range of keys in the database. GetApproximateSizes calls with it begin at the key Start and end right before the key Limit.

type ReadOptions

type ReadOptions struct {
	Opt *C.leveldb_readoptions_t
}

ReadOptions represent all of the available options when reading from a database.

To prevent memory leaks, Close must called on a ReadOptions when the program no longer needs it.

func NewReadOptions

func NewReadOptions() *ReadOptions

NewReadOptions allocates a new ReadOptions object.

func (*ReadOptions) Close

func (ro *ReadOptions) Close()

Close deallocates the ReadOptions, freeing its underlying C struct.

func (*ReadOptions) SetFillCache

func (ro *ReadOptions) SetFillCache(b bool)

SetFillCache controls whether reads performed with this ReadOptions will fill the Cache of the server. It defaults to true.

It is useful to turn this off on ReadOptions for DB.Iterator (and DB.Get) calls used in offline threads to prevent bulk scans from flushing out live user data in the cache.

See also Options.SetCache

func (*ReadOptions) SetSnapshot

func (ro *ReadOptions) SetSnapshot(snap *Snapshot)

SetSnapshot causes reads to provided as they were when the passed in Snapshot was created by DB.NewSnapshot. This is useful for getting consistent reads during a bulk operation.

See the LevelDB documentation for details.

func (*ReadOptions) SetVerifyChecksums

func (ro *ReadOptions) SetVerifyChecksums(b bool)

SetVerifyChecksums controls whether all data read with this ReadOptions will be verified against corresponding checksums.

It defaults to false. See the LevelDB documentation for details.

type SeekResult

type SeekResult struct {
	Valid bool
	Equal bool
	Value []byte
}

SeekResult is the result of a SeekTo.

func SeekTo

func SeekTo(it *Iterator, key []byte) SeekResult

SeekTo will seek the given iterator to the given key and return a result about it. It will check the validity of the key, if they're equal and if so, it will also fetch the value. The value slice is a direct reference to the underlying memory so it can only be referenced while the iterator is valid and has not moved. All of this exists because it eliminates a bunch fo cgo overhead and is faster.

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot provides a consistent view of read operations in a DB.

Snapshot is used in read operations by setting it on a ReadOptions. Snapshots are created by calling DB.NewSnapshot.

To prevent memory leaks and resource strain in the database, the snapshot returned must be released with DB.ReleaseSnapshot method on the DB that created it.

type WriteBatch

type WriteBatch struct {
	// contains filtered or unexported fields
}

WriteBatch is a batching of Puts, and Deletes to be written atomically to a database. A WriteBatch is written when passed to DB.Write.

To prevent memory leaks, call Close when the program no longer needs the WriteBatch object.

func NewWriteBatch

func NewWriteBatch() *WriteBatch

NewWriteBatch creates a fully allocated WriteBatch.

func (*WriteBatch) Clear

func (w *WriteBatch) Clear()

Clear removes all the enqueued Put and Deletes in the WriteBatch.

func (*WriteBatch) Close

func (w *WriteBatch) Close()

Close releases the underlying memory of a WriteBatch.

func (*WriteBatch) Delete

func (w *WriteBatch) Delete(key []byte)

Delete queues a deletion of the data at key to be deleted later.

The key byte slice may be reused safely. Delete takes a copy of them before returning.

func (*WriteBatch) Put

func (w *WriteBatch) Put(key, value []byte)

Put places a key-value pair into the WriteBatch for writing later.

Both the key and value byte slices may be reused as WriteBatch takes a copy of them before returning.

type WriteOptions

type WriteOptions struct {
	Opt *C.leveldb_writeoptions_t
}

WriteOptions represent all of the available options when writing from a database.

To prevent memory leaks, Close must called on a WriteOptions when the program no longer needs it.

func NewWriteOptions

func NewWriteOptions() *WriteOptions

NewWriteOptions allocates a new WriteOptions object.

func (*WriteOptions) Close

func (wo *WriteOptions) Close()

Close deallocates the WriteOptions, freeing its underlying C struct.

func (*WriteOptions) SetSync

func (wo *WriteOptions) SetSync(b bool)

SetSync controls whether each write performed with this WriteOptions will be flushed from the operating system buffer cache before the write is considered complete.

If called with true, this will significantly slow down writes. If called with false, and the host machine crashes, some recent writes may be lost. The default is false.

See the LevelDB documentation for details.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL