levigo

package
v2.1.7+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 7, 2019 License: Apache-2.0, MIT Imports: 4 Imported by: 0

README

LevelDB

This repo is a fork from DataDog/leveldb, while DataDog/leveldb itself is a fork of jmhodges/levigo.

This fork makes levigo go-gettable by including the C++ source for LevelDB and building it with the Go code. LevelDB and Snappy are both compiled and linked statically, so while you will not need them installed on your target machine, you should have a roughly compatible version of libstdc++.

This package currently includes LevelDB 1.19, and supports Linux and OS X. It should work on Go 1.5 or greater.

godoc documentation.

Building

This package supports Linux and OS X.

You'll need gcc and g++. Then:

go get github.com/DataDog/leveldb

This will take a long time as it compiles the LevelDB source code along with the Go library.

To avoid waiting for compilation every time you want to build your project, you can run:

go install github.com/DataDog/leveldb

Updating Vendored Snappy

  1. Overwrite source in ./deps/snappy/
  2. make link_snappy

If nothing significant in snappy has changed, things should JustWork™

Updating Vendored LevelDB

The embedded LevelDB source code is located in deps/leveldb. In order to get go build to compile LevelDB, we symlink the required LevelDB source files to the root directory of the project. These symlinks are prefixed with deps_.

To change the embedded version of LevelDB, do the following:

  1. rm deps_*.cc
  2. Replace deps/leveldb with the source code of the desired version of LevelDB
  3. On Linux, run ./deps/leveldb/build_detect_platform linux.mk build_flags
  4. On OS X, run ./deps/leveldb/build_detect_platform darwin.mk build_flags
  5. cat build_flags/*, take a quick look at the new compiler and linker flags and see if there are any flags we're missing in cgo_flags_*.go. If so, add them.
  6. make link_leveldb
  7. On OS X and Linux, run go build and verify that everything compiles.

note I couldn't get build_detect_platform to run as advertised above -- j

Caveats

Comparators and WriteBatch iterators must be written in C in your own library. This seems like a pain in the ass, but remember that you'll have the LevelDB C API available to your in your client package when you import levigo.

An example of writing your own Comparator can be found in https://github.com/DataDog/leveldb/blob/master/examples.

Documentation

Overview

Package levigo provides the ability to create and access LevelDB databases.

levigo.Open opens and creates databases. 

opts := levigo.NewOptions()
opts.SetCache(levigo.NewLRUCache(3<<30))
opts.SetCreateIfMissing(true)
db, err := levigo.Open("/path/to/db", opts)

The DB struct returned by Open provides DB.Get, DB.Put and DB.Delete to modify and query the database. 

ro := levigo.NewReadOptions()
wo := levigo.NewWriteOptions()
// if ro and wo are not used again, be sure to Close them.
data, err := db.Get(ro, []byte("key"))
...
err = db.Put(wo, []byte("anotherkey"), data)
...
err = db.Delete(wo, []byte("key"))

For bulk reads, use an Iterator. If you want to avoid disturbing your live traffic while doing the bulk read, be sure to call SetFillCache(false) on the ReadOptions you use when creating the Iterator. 

ro := levigo.NewReadOptions()
ro.SetFillCache(false)
it := db.NewIterator(ro)
defer it.Close()
for it.Seek(mykey); it.Valid(); it.Next() {
	munge(it.Key(), it.Value())
}
if err := it.GetError(); err != nil {
	...
}

Batched, atomic writes can be performed with a WriteBatch and DB.Write. 

wb := levigo.NewWriteBatch()
// defer wb.Close or use wb.Clear and reuse.
wb.Delete([]byte("removed"))
wb.Put([]byte("added"), []byte("data"))
wb.Put([]byte("anotheradded"), []byte("more"))
err := db.Write(wo, wb)

If your working dataset does not fit in memory, you'll want to add a bloom filter to your database. NewBloomFilter and Options.SetFilterPolicy is what you want. NewBloomFilter is amount of bits in the filter to use per key in your database. 

filter := levigo.NewBloomFilter(10)
opts.SetFilterPolicy(filter)
db, err := levigo.Open("/path/to/db", opts)

If you're using a custom comparator in your code, be aware you may have to make your own filter policy object.

This documentation is not a complete discussion of LevelDB. Please read the LevelDB documentation <http://code.google.com/p/leveldb> for information on its operation. You'll find lots of goodies there.

Index

Constants

View Source 
const (
	NoCompression     = CompressionOpt(0)
	SnappyCompression = CompressionOpt(1)
	LZ4Compression    = CompressionOpt(2)
)

Known compression arguments for Options.SetCompression.

Variables

View Source 
var ErrDBClosed = errors.New("database is closed")

Functions

func DestroyComparator 

func DestroyComparator(cmp *C.leveldb_comparator_t)

DestroyComparator deallocates a *C.leveldb_comparator_t.

This is provided as a convienience to advanced users that have implemented their own comparators in C in their own code.

func DestroyDatabase 

func DestroyDatabase(dbname string, o *Options) error

DestroyDatabase removes a database entirely, removing everything from the filesystem.

func Exists 

func Exists(it *Iterator, key []byte) bool

Exists will seek the iterator to the given key and return true if it exists.

func GetLevelDBMajorVersion 

func GetLevelDBMajorVersion() int

func GetLevelDBMinorVersion 

func GetLevelDBMinorVersion() int

func RepairDatabase 

func RepairDatabase(dbname string, o *Options) error

RepairDatabase attempts to repair a database.

If the database is unrepairable, an error is returned.

func SnappySupported 

func SnappySupported() bool

Types

type Cache 

type Cache struct {
	Cache *C.leveldb_cache_t
}

Cache is a cache used to store data read from data in memory.

Typically, NewLRUCache is all you will need, but advanced users may implement their own *C.leveldb_cache_t and create a Cache.

To prevent memory leaks, a Cache must have Close called on it when it is no longer needed by the program. Note: if the process is shutting down, this may not be necessary and could be avoided to shorten shutdown time.

func NewLRUCache 

func NewLRUCache(capacity int) *Cache

NewLRUCache creates a new Cache object with the capacity given.

To prevent memory leaks, Close should be called on the Cache when the program no longer needs it. Note: if the process is shutting down, this may not be necessary and could be avoided to shorten shutdown time.

func (*Cache) Close 

func (c *Cache) Close()

Close deallocates the underlying memory of the Cache object.

type CompressionOpt 

type CompressionOpt int

CompressionOpt is a value for Options.SetCompression.

type DB 

type DB struct {
	Ldb *C.leveldb_t
	// contains filtered or unexported fields
}

DB is a reusable handle to a LevelDB database on disk, created by Open.

To avoid memory and file descriptor leaks, call Close when the process no longer needs the handle. Calls to any DB method made after Close will panic.

The DB instance may be shared between goroutines. The usual data race conditions will occur if the same key is written to from more than one, of course.

func Open 

func Open(dbname string, o *Options) (*DB, error)

Open opens a database.

Creating a new database is done by calling SetCreateIfMissing(true) on the Options passed to Open.

It is usually wise to set a Cache object on the Options with SetCache to keep recently used data from that database in memory.

func (*DB) Close 

func (db *DB) Close()

Close closes the database, rendering it unusable for I/O, by deallocating the underlying handle.

Any attempts to use the DB after Close is called will panic.

func (*DB) CompactRange 

func (db *DB) CompactRange(r Range)

CompactRange runs a manual compaction on the Range of keys given. This is not likely to be needed for typical usage.

func (*DB) Delete 

func (db *DB) Delete(wo *WriteOptions, key []byte) error

Delete removes the data associated with the key from the database.

The key byte slice may be reused safely. Delete takes a copy of them before returning. The WriteOptions passed in can be reused by multiple calls to this and if the WriteOptions is left unchanged.

func (*DB) Get 

func (db *DB) Get(ro *ReadOptions, key []byte) ([]byte, error)

Get returns the data associated with the key from the database.

If the key does not exist in the database, a nil []byte is returned. If the key does exist, but the data is zero-length in the database, a zero-length []byte will be returned.

The key byte slice may be reused safely. Get takes a copy of them before returning.

func (*DB) GetApproximateSizes 

func (db *DB) GetApproximateSizes(ranges []Range) []uint64

GetApproximateSizes returns the approximate number of bytes of file system space used by one or more key ranges.

The keys counted will begin at Range.Start and end on the key before Range.Limit.

func (*DB) NewIterator 

func (db *DB) NewIterator(ro *ReadOptions) *Iterator

NewIterator returns an Iterator over the the database that uses the ReadOptions given.

Often, this is used for large, offline bulk reads while serving live traffic. In that case, it may be wise to disable caching so that the data processed by the returned Iterator does not displace the already cached data. This can be done by calling SetFillCache(false) on the ReadOptions before passing it here.

Similarly, ReadOptions.SetSnapshot is also useful.

The ReadOptions passed in can be reused by multiple calls to this and other methods if the ReadOptions is left unchanged.

func (*DB) NewSnapshot 

func (db *DB) NewSnapshot() *Snapshot

NewSnapshot creates a new snapshot of the database.

The Snapshot, when used in a ReadOptions, provides a consistent view of state of the database at the the snapshot was created.

To prevent memory leaks and resource strain in the database, the snapshot returned must be released with DB.ReleaseSnapshot method on the DB that created it.

See the LevelDB documentation for details.

func (*DB) PropertyValue 

func (db *DB) PropertyValue(propName string) string

PropertyValue returns the value of a database property.

Examples of properties include "leveldb.stats", "leveldb.sstables", and "leveldb.num-files-at-level0".

func (*DB) Put 

func (db *DB) Put(wo *WriteOptions, key, value []byte) error

Put writes data associated with a key to the database.

If a nil []byte is passed in as value, it will be returned by Get as an zero-length slice. The WriteOptions passed in can be reused by multiple calls to this and if the WriteOptions is left unchanged.

The key and value byte slices may be reused safely. Put takes a copy of them before returning.

func (*DB) ReleaseSnapshot 

func (db *DB) ReleaseSnapshot(snap *Snapshot)

ReleaseSnapshot removes the snapshot from the database's list of snapshots, and deallocates it.

func (*DB) Write 

func (db *DB) Write(wo *WriteOptions, w *WriteBatch) error

Write atomically writes a WriteBatch to disk. The WriteOptions passed in can be reused by multiple calls to this and other methods.

type DatabaseError 

type DatabaseError string

func (DatabaseError) Error 

func (e DatabaseError) Error() string

type Env 

type Env struct {
	Env *C.leveldb_env_t
}

Env is a system call environment used by a database.

Typically, NewDefaultEnv is all you need. Advanced users may create their own Env with a *C.leveldb_env_t of their own creation.

To prevent memory leaks, an Env must have Close called on it when it is no longer needed by the program.

func NewDefaultEnv 

func NewDefaultEnv() *Env

NewDefaultEnv creates a default environment for use in an Options.

To prevent memory leaks, the Env returned should be deallocated with Close.

func (*Env) Close 

func (env *Env) Close()

Close deallocates the Env, freeing the underlying struct.

type FilterPolicy 

type FilterPolicy struct {
	Policy *C.leveldb_filterpolicy_t
}

FilterPolicy is a factory type that allows the LevelDB database to create a filter, such as a bloom filter, that is stored in the sstables and used by DB.Get to reduce reads.

An instance of this struct may be supplied to Options when opening a DB. Typical usage is to call NewBloomFilter to get an instance.

To prevent memory leaks, a FilterPolicy must have Close called on it when it is no longer needed by the program.

func NewBloomFilter 

func NewBloomFilter(bitsPerKey int) *FilterPolicy

NewBloomFilter creates a filter policy that will create a bloom filter when necessary with the given number of bits per key.

See the FilterPolicy documentation for more.

func (*FilterPolicy) Close 

func (fp *FilterPolicy) Close()

type Iterator 

type Iterator struct {
	Iter *C.leveldb_iterator_t
}

Iterator is a read-only iterator through a LevelDB database. It provides a way to seek to specific keys and iterate through the keyspace from that point, as well as access the values of those keys.

Care must be taken when using an Iterator. If the method Valid returns false, calls to Key, Value, Next, and Prev will result in panics. However, Seek, SeekToFirst, SeekToLast, GetError, Valid, and Close will still be safe to call.

GetError will only return an error in the event of a LevelDB error. It will return a nil on iterators that are simply invalid. Given that behavior, GetError is not a replacement for a Valid.

A typical use looks like: 

db := levigo.Open(...)

it := db.NewIterator(readOpts)
defer it.Close()
for it.Seek(mykey); it.Valid(); it.Next() {
	useKeyAndValue(it.Key(), it.Value())
}
if err := it.GetError() {
	...
}

To prevent memory leaks, an Iterator must have Close called on it when it is no longer needed by the program.

func (*Iterator) Close 

func (it *Iterator) Close()

Close deallocates the given Iterator, freeing the underlying C struct.

func (*Iterator) GetError 

func (it *Iterator) GetError() error

GetError returns an IteratorError from LevelDB if it had one during iteration.

This method is safe to call when Valid returns false.

func (*Iterator) Key 

func (it *Iterator) Key() []byte

Key returns a copy the key in the database the iterator currently holds.

If Valid returns false, this method will panic.

func (*Iterator) Next 

func (it *Iterator) Next()

Next moves the iterator to the next sequential key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

If Valid returns false, this method will panic.

func (*Iterator) Prev 

func (it *Iterator) Prev()

Prev moves the iterator to the previous sequential key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

If Valid returns false, this method will panic.

func (*Iterator) Seek 

func (it *Iterator) Seek(key []byte)

Seek moves the iterator the position of the key given or, if the key doesn't exist, the next key that does exist in the database. If the key doesn't exist, and there is no next key, the Iterator becomes invalid.

This method is safe to call when Valid returns false.

func (*Iterator) SeekToFirst 

func (it *Iterator) SeekToFirst()

SeekToFirst moves the iterator to the first key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

This method is safe to call when Valid returns false.

func (*Iterator) SeekToLast 

func (it *Iterator) SeekToLast()

SeekToLast moves the iterator to the last key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

This method is safe to call when Valid returns false.

func (*Iterator) Valid 

func (it *Iterator) Valid() bool

Valid returns false only when an Iterator has iterated past either the first or the last key in the database.

func (*Iterator) Value 

func (it *Iterator) Value() []byte

Value returns a copy of the value in the database the iterator currently holds.

If Valid returns false, this method will panic.

type IteratorError 

type IteratorError string

func (IteratorError) Error 

func (e IteratorError) Error() string

type MultiKeyError 

type MultiKeyError struct {
	// contains filtered or unexported fields
}

MultiKeyError encapsulates multiple errors encountered in a Get/Put-Many() call, behind the errors.error interface. Caller may cast the generic error object to a MultiKeyError and call ErrorsByKeyIdx() to get all the errors by the failed key index.

func (*MultiKeyError) Error 

func (mke *MultiKeyError) Error() string

func (*MultiKeyError) ErrorsByKeyIdx 

func (mke *MultiKeyError) ErrorsByKeyIdx() map[int]error

ErrorsByKeyIdx() returns a result map of keyIdx => error. e.g. If one calls GetMany(keys) and keys[2], keys[3], keys[7] failed, then result[2] will hold the error for keys[2], result[3] for keys[3] and result[7] the error for keys[7]

func (*MultiKeyError) GoString 

func (mke *MultiKeyError) GoString() string

type Options 

type Options struct {
	Opt *C.leveldb_options_t
}

Options represent all of the available options when opening a database with Open. Options should be created with NewOptions.

It is usually with to call SetCache with a cache object. Otherwise, all data will be read off disk.

To prevent memory leaks, Close must be called on an Options when the program no longer needs it.

func NewOptions 

func NewOptions() *Options

NewOptions allocates a new Options object. Note that the CompressionLevel is set to NoCompression by default rather than SnappyCompression.

func (*Options) Close 

func (o *Options) Close()

Close deallocates the Options, freeing its underlying C struct.

func (*Options) SetBlockRestartInterval 

func (o *Options) SetBlockRestartInterval(n int)

SetBlockRestartInterval is the number of keys between restarts points for delta encoding keys.

Most clients should leave this parameter alone. See the LevelDB documentation for details.

func (*Options) SetBlockSize 

func (o *Options) SetBlockSize(s int)

SetBlockSize sets the approximate size of user data packed per block.

The default is roughly 4096 uncompressed bytes. A better setting depends on your use case. See the LevelDB documentation for details.

func (*Options) SetCache 

func (o *Options) SetCache(cache *Cache)

SetCache places a cache object in the database when a database is opened.

This is usually wise to use. See also ReadOptions.SetFillCache.

func (*Options) SetComparator 

func (o *Options) SetComparator(cmp *C.leveldb_comparator_t)

SetComparator sets the comparator to be used for all read and write operations.

The comparator that created a database must be the same one (technically, one with the same name string) that is used to perform read and write operations.

The default comparator is usually sufficient.

func (*Options) SetCompression 

func (o *Options) SetCompression(t CompressionOpt)

SetCompression sets whether to compress blocks using the specified compresssion algorithm.

The default value is NoCompression, but either SnappyCompression or LZ4Compression are both fast enough that it is unlikely you want to keep them off.

If the LevelDB library was built without Snappy compression enabled, the SnappyCompression setting will be ignored.

func (*Options) SetCreateIfMissing 

func (o *Options) SetCreateIfMissing(b bool)

SetCreateIfMissing causes Open to create a new database on disk if it does not already exist.

func (*Options) SetEnv 

func (o *Options) SetEnv(env *Env)

SetEnv sets the Env object for the new database handle.

func (*Options) SetErrorIfExists 

func (o *Options) SetErrorIfExists(error_if_exists bool)

SetErrorIfExists, if passed true, will cause the opening of a database that already exists to throw an error.

func (*Options) SetFilterPolicy 

func (o *Options) SetFilterPolicy(fp *FilterPolicy)

SetFilterPolicy causes Open to create a new database that will uses filter created from the filter policy passed in.

func (*Options) SetInfoLog 

func (o *Options) SetInfoLog(log *C.leveldb_logger_t)

SetInfoLog sets a *C.leveldb_logger_t object as the informational logger for the database.

func (*Options) SetMaxOpenFiles 

func (o *Options) SetMaxOpenFiles(n int)

SetMaxOpenFiles sets the number of files than can be used at once by the database.

See the LevelDB documentation for details.

func (*Options) SetParanoidChecks 

func (o *Options) SetParanoidChecks(pc bool)

SetParanoidChecks, when called with true, will cause the database to do aggressive checking of the data it is processing and will stop early if it detects errors.

See the LevelDB documentation docs for details.

func (*Options) SetWriteBufferSize 

func (o *Options) SetWriteBufferSize(s int)

SetWriteBufferSize sets the number of bytes the database will build up in memory (backed by an unsorted log on disk) before converting to a sorted on-disk file.

type Range 

type Range struct {
	Start []byte
	Limit []byte
}

Range is a range of keys in the database. GetApproximateSizes calls with it begin at the key Start and end right before the key Limit.

type ReadOptions 

type ReadOptions struct {
	Opt *C.leveldb_readoptions_t
}

ReadOptions represent all of the available options when reading from a database.

To prevent memory leaks, Close must called on a ReadOptions when the program no longer needs it.

func NewReadOptions 

func NewReadOptions() *ReadOptions

NewReadOptions allocates a new ReadOptions object.

func (*ReadOptions) Close 

func (ro *ReadOptions) Close()

Close deallocates the ReadOptions, freeing its underlying C struct.

func (*ReadOptions) SetFillCache 

func (ro *ReadOptions) SetFillCache(b bool)

SetFillCache controls whether reads performed with this ReadOptions will fill the Cache of the server. It defaults to true.

It is useful to turn this off on ReadOptions for DB.Iterator (and DB.Get) calls used in offline threads to prevent bulk scans from flushing out live user data in the cache.

See also Options.SetCache

func (*ReadOptions) SetSnapshot 

func (ro *ReadOptions) SetSnapshot(snap *Snapshot)

SetSnapshot causes reads to provided as they were when the passed in Snapshot was created by DB.NewSnapshot. This is useful for getting consistent reads during a bulk operation.

See the LevelDB documentation for details.

func (*ReadOptions) SetVerifyChecksums 

func (ro *ReadOptions) SetVerifyChecksums(b bool)

SetVerifyChecksums controls whether all data read with this ReadOptions will be verified against corresponding checksums.

It defaults to false. See the LevelDB documentation for details.

type SeekResult 

type SeekResult struct {
	Valid bool
	Equal bool
	Value []byte
}

SeekResult is the result of a SeekTo.

func SeekTo 

func SeekTo(it *Iterator, key []byte) SeekResult

SeekTo will seek the given iterator to the given key and return a result about it. It will check the validity of the key, if they're equal and if so, it will also fetch the value. The value slice is a direct reference to the underlying memory so it can only be referenced while the iterator is valid and has not moved. All of this exists because it eliminates a bunch fo cgo overhead and is faster.

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot provides a consistent view of read operations in a DB.

Snapshot is used in read operations by setting it on a ReadOptions. Snapshots are created by calling DB.NewSnapshot.

To prevent memory leaks and resource strain in the database, the snapshot returned must be released with DB.ReleaseSnapshot method on the DB that created it.

type WriteBatch 

type WriteBatch struct {
	// contains filtered or unexported fields
}

WriteBatch is a batching of Puts, and Deletes to be written atomically to a database. A WriteBatch is written when passed to DB.Write.

To prevent memory leaks, call Close when the program no longer needs the WriteBatch object.

func NewWriteBatch 

func NewWriteBatch() *WriteBatch

NewWriteBatch creates a fully allocated WriteBatch.

func (*WriteBatch) Clear 

func (w *WriteBatch) Clear()

Clear removes all the enqueued Put and Deletes in the WriteBatch.

func (*WriteBatch) Close 

func (w *WriteBatch) Close()

Close releases the underlying memory of a WriteBatch.

func (*WriteBatch) Delete 

func (w *WriteBatch) Delete(key []byte)

Delete queues a deletion of the data at key to be deleted later.

The key byte slice may be reused safely. Delete takes a copy of them before returning.

func (*WriteBatch) Put 

func (w *WriteBatch) Put(key, value []byte)

Put places a key-value pair into the WriteBatch for writing later.

Both the key and value byte slices may be reused as WriteBatch takes a copy of them before returning.

type WriteOptions 

type WriteOptions struct {
	Opt *C.leveldb_writeoptions_t
}

WriteOptions represent all of the available options when writing from a database.

To prevent memory leaks, Close must called on a WriteOptions when the program no longer needs it.

func NewWriteOptions 

func NewWriteOptions() *WriteOptions

NewWriteOptions allocates a new WriteOptions object.

func (*WriteOptions) Close 

func (wo *WriteOptions) Close()

Close deallocates the WriteOptions, freeing its underlying C struct.

func (*WriteOptions) SetSync 

func (wo *WriteOptions) SetSync(b bool)

SetSync controls whether each write performed with this WriteOptions will be flushed from the operating system buffer cache before the write is considered complete.

If called with true, this will significantly slow down writes. If called with false, and the host machine crashes, some recent writes may be lost. The default is false.

See the LevelDB documentation for details.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.