lmdb

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2015 License: BSD-3-Clause Imports: 9 Imported by: 17

Documentation

Overview

Package lmdb provides bindings to the lmdb C API. The package bindings are fairly low level and are designed to provide a minimal interface that prevents misuse to a reasonable extent. When in doubt refer to the C documentation as a reference. 

http://symas.com/mdb/doc/group__mdb.html

Environment

An LMDB environment holds named databases (key-value stores). An environment is represented as one file on the filesystem (though often a corresponding lock file exists).

LMDB recommends setting an environment's size as large as possible at the time of creation. On filesystems that support sparse files this should not adversely affect disk usage. Resizing an environment is possible but must be handled with care when concurrent access is involved.

Databases

A database in an LMDB environment is an ordered key-value store that holds arbitrary binary data. Typically the keys are unique but duplicate keys may be allowed (DupSort), in which case the values for each duplicate key are ordered.

A single LMDB environment can have multiple named databases. But there is also a 'root' (unnamed) database that can be used to store data. Use caution storing data in the root database when named databases are in use. The root database serves as an index for named databases.

A database is referenced by an opaque handle known as its DBI which must be opened inside a transaction with the OpenDBI or OpenRoot methods. DBIs may be closed but it is not required. Typically, applications aquire handles for all their databases immediately after opening an environment and retain them for the lifetime of the process.

Transactions

View (readonly) transactions in LMDB operate on a snapshot of the database at the time the transaction began. The number of simultaneously active view transactions is bounded and configured when the environment is initialized.

Update (read-write) transactions are serialized in LMDB. Attempts to create update transactions block until a lock may be obtained. Update transactions can create subtransactions which may be rolled back independently from their parent.

The lmdb package supplies managed and unmanaged transactions. Managed transactions do not require explicit calling of Abort/Commit and are provided through the Env methods Update, View, and RunTxn. The BeginTxn method on Env creates an unmanaged transaction but its use is not advised in most applications.

Index

Examples

Constants

View Source 
const (
	// Flags for Cursor.Get
	//
	// See MDB_cursor_op.
	First        = C.MDB_FIRST          // The first item.
	FirstDup     = C.MDB_FIRST_DUP      // The first value of current key (DupSort).
	GetBoth      = C.MDB_GET_BOTH       // Get the key as well as the value (DupSort).
	GetBothRange = C.MDB_GET_BOTH_RANGE // Get the key and the nearsest value (DupSort).
	GetCurrent   = C.MDB_GET_CURRENT    // Get the key and value at the current position.
	GetMultiple  = C.MDB_GET_MULTIPLE   // Get up to a page dup values for key at current position (DupFixed).
	Last         = C.MDB_LAST           // Last item.
	LastDup      = C.MDB_LAST_DUP       // Position at last value of current key (DupSort).
	Next         = C.MDB_NEXT           // Next value.
	NextDup      = C.MDB_NEXT_DUP       // Next value of the current key (DupSort).
	NextMultiple = C.MDB_NEXT_MULTIPLE  // Get key and up to a page of values from the next cursor position (DupFixed).
	NextNoDup    = C.MDB_NEXT_NODUP     // The first value of the next key (DupSort).
	Prev         = C.MDB_PREV           // The previous item.
	PrevDup      = C.MDB_PREV_DUP       // The previous item of the current key (DupSort).
	PrevNoDup    = C.MDB_PREV_NODUP     // The last data item of the previous key (DupSort).
	Set          = C.MDB_SET            // The specified key.
	SetKey       = C.MDB_SET_KEY        // Get key and data at the specified key.
	SetRange     = C.MDB_SET_RANGE      // The first key no less than the specified key.
)
View Source 
const (
	// Flags for Txn.Put and Cursor.Put.
	//
	// See mdb_put and mdb_cursor_put.
	Current     = C.MDB_CURRENT     // Replace the item at the current key position (Cursor only)
	NoDupData   = C.MDB_NODUPDATA   // Store the key-value pair only if key is not present (DupSort).
	NoOverwrite = C.MDB_NOOVERWRITE // Store a new key-value pair only if key is not present.
	Append      = C.MDB_APPEND      // Append an item to the database.
	AppendDup   = C.MDB_APPENDDUP   // Append an item to the database (DupSort).
)

The MDB_MULTIPLE and MDB_RESERVE flags are special and do not fit the calling pattern of other calls to Put. They are not exported because they require special methods, PutMultiple and PutReserve in which the flag is implied and does not need to be passed.

View Source 
const (
	// Flags for Env.Open.
	//
	// See mdb_env_open
	FixedMap    = C.MDB_FIXEDMAP   // Danger zone. Map memory at a fixed address.
	NoSubdir    = C.MDB_NOSUBDIR   // Argument to Open is a file, not a directory.
	Readonly    = C.MDB_RDONLY     // Used in several functions to denote an object as readonly.
	WriteMap    = C.MDB_WRITEMAP   // Use a writable memory map.
	NoMetaSync  = C.MDB_NOMETASYNC // Don't fsync metapage after commit.
	NoSync      = C.MDB_NOSYNC     // Don't fsync after commit.
	MapAsync    = C.MDB_MAPASYNC   // Flush asynchronously when using the WriteMap flag.
	NoTLS       = C.MDB_NOTLS      // Danger zone. When unset reader locktable slots are tied to their thread.
	NoLock      = C.MDB_NOLOCK     // Danger zone. LMDB does not use any locks.
	NoReadahead = C.MDB_NORDAHEAD  // Disable readahead. Requires OS support.
	NoMemInit   = C.MDB_NOMEMINIT  // Disable LMDB memory initialization.
)
View Source 
const (
	// Flags for Txn.OpenDBI.
	ReverseKey = C.MDB_REVERSEKEY // Use reverse string keys.
	DupSort    = C.MDB_DUPSORT    // Use sorted duplicates.
	DupFixed   = C.MDB_DUPFIXED   // Duplicate items have a fixed size (DupSort).
	ReverseDup = C.MDB_REVERSEDUP // Reverse duplicate values (DupSort).
	Create     = C.MDB_CREATE     // Create DB if not already existing.
)

BUG(bmatsuo): MDB_INTEGERKEY and MDB_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.

View Source 
const (
	// Flags for Env.CopyFlags
	//
	// See mdb_env_copy2
	CopyCompact = C.MDB_CP_COMPACT // Perform compaction while copying
)

Variables

This section is empty.

Functions

func IsErrno 

func IsErrno(err error, errno Errno) bool

IsErrno returns true if err's errno is the given errno.

func IsErrnoFn 

func IsErrnoFn(err error, fn func(error) bool) bool

IsErrnoFn calls fn on the error underlying err and returns the result. If err is an *OpError then err.Errno is passed to fn. Otherwise err is passed directly to fn.

func IsErrnoSys 

func IsErrnoSys(err error, errno syscall.Errno) bool

IsErrnoSys returns true if err's errno is the given errno.

func IsMapFull 

func IsMapFull(err error) bool

IsMapFull returns true if the environment map size has been reached.

func IsMapResized 

func IsMapResized(err error) bool

IsResized returns true if the environment has grown.

func IsNotExist 

func IsNotExist(err error) bool

IsNotExist returns true the path passed to the Env.Open method does not exist.

func IsNotFound 

func IsNotFound(err error) bool

IsNotFound returns true if the key requested in Txn.Get or Cursor.Get does not exist or if the Cursor reached the end of the database without locating a value (EOF).

func Version 

func Version() (major, minor, patch int, s string)

Version return the major, minor, and patch version numbers of the LMDB C library and a string representation of the version.

See mdb_version.

func VersionString 

func VersionString() string

VersionString returns a string representation of the LMDB C library version.

See mdb_version.

Types

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor operates on data inside a transaction and holds a position in the database.

See MDB_cursor.

Example

This example shows how to read and write data using a Cursor. Errors are ignored for brevity. Real code should check and handle are errors which may require more modular code. 

package main

import (
	"fmt"
	"io/ioutil"
	"os"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// create a directory to hold the database
	path, _ := ioutil.TempDir("", "mdb_test")
	defer os.RemoveAll(path)

	// open the LMDB environment
	env, _ := lmdb.NewEnv()
	env.SetMaxDBs(1)
	env.Open(path, 0, 0664)
	defer env.Close()

	// open a database.
	var dbi lmdb.DBI
	err := env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("exampledb", lmdb.Create)
		return
	})
	if err != nil {
		panic(err)
	}

	// write some data and print the number of items written
	err = env.Update(func(txn *lmdb.Txn) error {
		cursor, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cursor.Close()

		err = cursor.Put([]byte("key0"), []byte("val0"), 0)
		if err != nil {
			return err
		}
		err = cursor.Put([]byte("key1"), []byte("val1"), 0)
		if err != nil {
			return err
		}
		err = cursor.Put([]byte("key2"), []byte("val2"), 0)
		if err != nil {
			return err
		}

		// inspect the transaction
		stat, err := txn.Stat(dbi)
		if err != nil {
			return err
		}
		fmt.Println(stat.Entries)

		return nil
	})
	if err != nil {
		panic(err)
	}

	// scan the database and print all key-value pairs
	err = env.View(func(txn *lmdb.Txn) error {
		cursor, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cursor.Close()

		for {
			bkey, bval, err := cursor.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				break
			}
			if err != nil {
				return err
			}
			fmt.Printf("%s: %s\n", bkey, bval)
		}

		return nil
	})
	if err != nil {
		panic(err)
	}

}

Output:

3
key0: val0
key1: val1
key2: val2

func (*Cursor) Close 

func (c *Cursor) Close()

Close the cursor handle. A runtime panic occurs if a the cursor is used after Close is called.

Cursors in write transactions must be closed before their transaction is terminated.

See mdb_cursor_close.

func (*Cursor) Count 

func (c *Cursor) Count() (uint64, error)

Count returns the number of duplicates for the current key.

See mdb_cursor_count.

func (*Cursor) DBI 

func (c *Cursor) DBI() DBI

DBI returns the cursor's database handle.

func (*Cursor) Del 

func (c *Cursor) Del(flags uint) error

Del deletes the item referred to by the cursor from the database.

See mdb_cursor_del.

func (*Cursor) Get 

func (c *Cursor) Get(setkey, setval []byte, op uint) (key, val []byte, err error)

Get retrieves items from the database. If c.Txn().RawRead is true the slices returned by Get reference readonly sections of memory that must not be accessed after the transaction has terminated.

See mdb_cursor_get.

func (*Cursor) Put 

func (c *Cursor) Put(key, val []byte, flags uint) error

Put stores an item in the database.

See mdb_cursor_put.

func (*Cursor) PutMulti 

func (c *Cursor) PutMulti(key []byte, page []byte, stride int, flags uint) error

PutMulti stores a set of contiguous items with stride size under key. PutMulti panics if len(page) is not a multiple of stride. The cursor's database must be DupFixed and DupSort.

See mdb_cursor_put.

func (*Cursor) PutReserve 

func (c *Cursor) PutReserve(key []byte, n int, flags uint) ([]byte, error)

PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.

func (*Cursor) Renew 

func (c *Cursor) Renew(txn *Txn) error

Renew associates readonly cursor with txn.

See mdb_cursor_renew.

func (*Cursor) Txn 

func (cursor *Cursor) Txn() *Txn

Txn returns the cursor's transaction.

type DBI 

type DBI C.MDB_dbi

DBI is a handle for a database in an Env.

See MDB_dbi

type Env 

type Env struct {
	// contains filtered or unexported fields
}

Env is opaque structure for a database environment. A DB environment supports multiple databases, all residing in the same shared-memory map.

See MDB_env.

Example

This example shows how to use the Env type and open a database. 

package main

import (
	"fmt"
	"io/ioutil"
	"os"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// create a directory to hold the database
	path, _ := ioutil.TempDir("", "mdb_test")
	defer os.RemoveAll(path)

	// open the LMDB environment
	env, err := lmdb.NewEnv()
	if err != nil {
		panic(err)
	}
	env.SetMaxDBs(1)
	env.Open(path, 0, 0664)
	defer env.Close()

	err = env.Update(func(txn *lmdb.Txn) error {
		// open a database, creating it if necessary.
		db, err := txn.OpenDBI("exampledb", lmdb.Create)
		if err != nil {
			return err
		}

		// get statistics about the db. print the number of key-value pairs (it
		// should be empty).
		stat, err := txn.Stat(db)
		if err != nil {
			return err
		}
		fmt.Println(stat.Entries)

		// commit the transaction, writing an entry for the newly created
		// database.
		return nil
	})
	if err != nil {
		panic(err)
	}

	// .. open more transactions and use the database

}

Output:

0

func NewEnv 

func NewEnv() (*Env, error)

NewEnv allocates and initializes a new Env.

See mdb_env_create.

func (*Env) BeginTxn 

func (env *Env) BeginTxn(parent *Txn, flags uint) (*Txn, error)

BeginTxn is a low-level (potentially dangerous) method to initialize a new transaction on env. BeginTxn does not attempt to serialize operations on write transactions to the same OS thread and without care its use for write transactions can cause undefined results.

Instead of BeginTxn users should call the View, Update, RunTxn methods.

See mdb_txn_begin.

func (*Env) Close 

func (env *Env) Close() error

Close shuts down the environment and releases the memory map.

See mdb_env_close.

func (*Env) CloseDBI 

func (env *Env) CloseDBI(db DBI)

CloseDBI closes the database handle, db. Normally calling CloseDBI explicitly is not necessary.

It is the caller's responsibility to serialize calls to CloseDBI.

See mdb_dbi_close.

func (*Env) Copy 

func (env *Env) Copy(path string) error

Copy copies the data in env to an environment at path.

See mdb_env_copy.

func (*Env) CopyFlag 

func (env *Env) CopyFlag(path string, flags uint) error

CopyFlag copies the data in env to an environment at path created with flags.

See mdb_env_copy2.

func (*Env) Flags 

func (env *Env) Flags() (uint, error)

Flags returns the flags set in the environment.

See mdb_env_get_flags.

func (*Env) Info 

func (env *Env) Info() (*EnvInfo, error)

Info returns information about the environment.

See mdb_env_info.

func (*Env) MaxKeySize 

func (env *Env) MaxKeySize() int

MaxKeySize returns the maximum allowed length for a key.

See mdb_env_get_maxkeysize.

func (*Env) MaxReaders 

func (env *Env) MaxReaders() (int, error)

MaxReaders returns the maximum number of reader slots for the environment.

See mdb_env_get_maxreaders.

func (*Env) Open 

func (env *Env) Open(path string, flags uint, mode os.FileMode) error

Open an environment handle. If this function fails Close() must be called to discard the Env handle. Open passes flags|NoTLS to mdb_env_open.

See mdb_env_open.

func (*Env) Path 

func (env *Env) Path() (string, error)

Path returns the path argument passed to Open. Path returns a non-nil error if env.Open() was not previously called.

See mdb_env_get_path.

func (*Env) ReaderCheck 

func (env *Env) ReaderCheck() (int, error)

ReaderCheck clears stale entries from the reader lock table and returns the number of entries cleared.

See mdb_reader_check()

func (*Env) RunTxn 

func (env *Env) RunTxn(flags uint, fn TxnOp) error

Run creates a new Txn and calls fn with it as an argument. Run commits the transaction if fn returns nil otherwise the transaction is aborted. Because RunTxn terminates the transaction goroutines should not retain references to it or its data after fn returns.

RunTxn does not lock the thread of the calling goroutine. Unless the Readonly flag is passed the calling goroutine should ensure it is locked to its thread.

See mdb_txn_begin.

func (*Env) SetFlags 

func (env *Env) SetFlags(flags uint) error

SetFlags sets flags in the environment.

See mdb_env_set_flags.

func (*Env) SetMapSize 

func (env *Env) SetMapSize(size int64) error

SetMapSize sets the size of the environment memory map.

See mdb_env_set_mapsize.

Example

This example demonstrates how an application typically uses Env.SetMapSize. The call to Env.SetMapSize() is made before calling env.Open(). Any calls after calling Env.Open() must take special care to synchronize with other goroutines. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	env, err := lmdb.NewEnv()
	if err != nil {
		// ...
	}

	// set the memory map size (maximum database size) to 1GB.
	err = env.SetMapSize(1 << 30)
	if err != nil {
		// ...
	}

	err = env.Open("mydb", 0, 0644)
	if err != nil {
		// ...
	}
	// ...
}

Output:
Example (MapResized)

This example demonstrates how to handle a MapResized error, encountered after another process has called mdb_env_set_mapsize (Env.SetMapSize). Applications which don't expect another process to resize the mmap don't need to check for the MapResized error.

The example is simplified for clarity. Many real applications will need to synchronize calls to Env.SetMapSize using something like a sync.RWMutex to ensure there are no active readonly transactions (those opened successfully before MapResized was encountered). 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

// These values can be used as no-op placeholders in examples.
func doUpdate(txn *lmdb.Txn) error { return nil }

func main() {
retry:
	err := env.Update(doUpdate)
	if lmdb.IsMapResized(err) {
		// If concurrent read transactions are possible then a sync.RWMutex
		// must be used here to ensure they all terminate before calling
		// env.SetMapSize().
		err = env.SetMapSize(0)
		if err != nil {
			panic(err)
		}

		// retry the update. a goto is not necessary but it simplifies error
		// handling with minimal overhead.
		goto retry
	} else if err != nil {
		// ...
	}
	// ...
}

Output:

func (*Env) SetMaxDBs 

func (env *Env) SetMaxDBs(size int) error

SetMaxDBs sets the maximum number of named databases for the environment.

See mdb_env_set_maxdbs.

func (*Env) SetMaxReaders 

func (env *Env) SetMaxReaders(size int) error

SetMaxReaders sets the maximum number of reader slots in the environment.

See mdb_env_set_maxreaders.

func (*Env) Stat 

func (env *Env) Stat() (*Stat, error)

Stat returns statistics about the environment.

See mdb_env_stat.

func (*Env) Sync 

func (env *Env) Sync(force bool) error

Sync flushes buffers to disk. If force is true a synchronous flush occurs and ignores any NoSync or MapAsync flag on the environment.

See mdb_env_sync.

func (*Env) UnsetFlags 

func (env *Env) UnsetFlags(flags uint) error

UnsetFlags clears flags in the environment.

See mdb_env_set_flags.

func (*Env) Update 

func (env *Env) Update(fn TxnOp) error

Update calls fn with a writable transaction. Update commits the transaction if fn returns a nil error otherwise Update aborts the transaction and returns the error.

Update locks the calling goroutine to its thread and unlocks it after fn returns. The Txn must not be used from multiple goroutines, even with synchronization.

Any call to Commit, Abort, Reset or Renew on a Txn created by Update will panic.

func (*Env) UpdateLocked 

func (env *Env) UpdateLocked(fn TxnOp) error

UpdateLocked behaves like Update but does not lock the calling goroutine to its thread. UpdateLocked should be used if the calling goroutine is already locked to its thread for another purpose.

Any call to Commit, Abort, Reset or Renew on a Txn created by UpdateLocked will panic.

func (*Env) View 

func (env *Env) View(fn TxnOp) error

View creates a readonly transaction with a consistent view of the environment and passes it to fn. View terminates its transaction after fn returns. Any error encountered by View is returned.

Any call to Commit, Abort, Reset or Renew on a Txn created by View will panic.

type EnvInfo 

type EnvInfo struct {
	MapSize    int64 // Size of the data memory map
	LastPNO    int64 // ID of the last used page
	LastTxnID  int64 // ID of the last committed transaction
	MaxReaders uint  // maximum number of threads for the environment
	NumReaders uint  // maximum number of threads used in the environment
}

Information about the environment.

See MDB_envinfo.

type Errno 

type Errno C.int

Errno is an error type that represents the (unique) errno values defined by LMDB. Other errno values (such as EINVAL) are represented with type syscall.Errno.

Most often helper functions such as IsNotFound may be used instead of dealing with Errno values directly. 

lmdb.IsNotFound(err)
lmdb.IsErrno(err, lmdb.TxnFull)
lmdb.IsErrnoSys(err, syscall.EINVAL)
lmdb.IsErrnoFn(err, os.IsPermission)

const (
	KeyExist        Errno = C.MDB_KEYEXIST
	NotFound        Errno = C.MDB_NOTFOUND
	PageNotFound    Errno = C.MDB_PAGE_NOTFOUND
	Corrupted       Errno = C.MDB_CORRUPTED
	Panic           Errno = C.MDB_PANIC
	VersionMismatch Errno = C.MDB_VERSION_MISMATCH
	Invalid         Errno = C.MDB_INVALID
	MapFull         Errno = C.MDB_MAP_FULL
	DBsFull         Errno = C.MDB_DBS_FULL
	ReadersFull     Errno = C.MDB_READERS_FULL
	TLSFull         Errno = C.MDB_TLS_FULL
	TxnFull         Errno = C.MDB_TXN_FULL
	CursorFull      Errno = C.MDB_CURSOR_FULL
	PageFull        Errno = C.MDB_PAGE_FULL
	MapResized      Errno = C.MDB_MAP_RESIZED
	Incompatible    Errno = C.MDB_INCOMPATIBLE
	BadRSlot        Errno = C.MDB_BAD_RSLOT
	BadTxn          Errno = C.MDB_BAD_TXN
	BadValSize      Errno = C.MDB_BAD_VALSIZE
	BadDBI          Errno = C.MDB_BAD_DBI
)

func (Errno) Error 

func (e Errno) Error() string

type Multi 

type Multi struct {
	// contains filtered or unexported fields
}

Multi is a wrapper for a contiguous page of sorted, fixed-length values passed to Cursor.PutMulti or retrieved using Cursor.Get with the GetMultiple/NextMultiple flag.

Multi values are only useful in databases opened with DupSort|DupFixed.

func WrapMulti 

func WrapMulti(page []byte, stride int) *Multi

WrapMulti converts a page of contiguous values with stride size into a Multi. WrapMulti panics if len(page) is not a multiple of stride. 

_, val, _ := cursor.Get(nil, nil, lmdb.FirstDup)
_, page, _ := cursor.Get(nil, nil, lmdb.GetMultiple)
multi := lmdb.WrapMulti(page, len(val))

See mdb_cursor_get and MDB_GET_MULTIPLE.

func (*Multi) Len 

func (m *Multi) Len() int

Len returns the number of values in the Multi.

func (*Multi) Page 

func (m *Multi) Page() []byte

Page returns the Multi page data as a raw slice of bytes with length m.Size().

func (*Multi) Size 

func (m *Multi) Size() int

Size returns the total size of the Multi data and is equal to 

m.Len()*m.Stride()

func (*Multi) Stride 

func (m *Multi) Stride() int

Stride returns the length of an individual value in the m.

func (*Multi) Val 

func (m *Multi) Val(i int) []byte

Val returns the value at index i. Val panics if i is out of range.

func (*Multi) Vals 

func (m *Multi) Vals() [][]byte

Vals returns a slice containing the values in m. The returned slice has length m.Len() and each item has length m.Stride().

type OpError 

type OpError struct {
	Op    string
	Errno error
}

func (*OpError) Error 

func (err *OpError) Error() string

type Stat 

type Stat struct {
	PSize         uint   // Size of a database page. This is currently the same for all databases.
	Depth         uint   // Depth (height) of the B-tree
	BranchPages   uint64 // Number of internal (non-leaf) pages
	LeafPages     uint64 // Number of leaf pages
	OverflowPages uint64 // Number of overflow pages
	Entries       uint64 // Number of data items
}

Statistics for a database in the environment

See MDB_stat.

type Txn 

type Txn struct {
	// If RawRead is true []byte values retrieved from Get() calls on the Txn
	// and its cursors will point directly into the memory-mapped structure.
	// Such slices will be readonly and must only be referenced wthin the
	// transaction's lifetime.
	RawRead bool
	// contains filtered or unexported fields
}

Txn is a database transaction in an environment.

WARNING: A writable Txn is not threadsafe and may only be used in the goroutine that created it.

See MDB_txn.

Example

This example shows how to read and write data with a Txn. Errors are ignored for brevity. Real code should check and handle are errors which may require more modular code. 

package main

import (
	"fmt"
	"io/ioutil"
	"os"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// create a directory to hold the database
	path, _ := ioutil.TempDir("", "mdb_test")
	defer os.RemoveAll(path)

	// open the LMDB environment
	env, _ := lmdb.NewEnv()
	env.SetMaxDBs(1)
	env.Open(path, 0, 0664)
	defer env.Close()

	// open a database.
	var dbi lmdb.DBI
	err := env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("exampledb", lmdb.Create)
		// the transaction will be commited if the database was successfully
		// opened/created.
		return err
	})
	if err != nil {
		panic(err)
	}

	err = env.Update(func(txn *lmdb.Txn) (err error) {
		// it can be helpful to define closures that abstract the transaction
		// and short circuit after errors.
		put := func(k, v string) {
			if err == nil {
				err = txn.Put(dbi, []byte(k), []byte(v), 0)
			}
		}

		// use the closure above to insert into the database.
		put("key0", "val0")
		put("key1", "val1")
		put("key2", "val2")

		return err
	})
	if err != nil {
		panic(err)
	}

	err = env.View(func(txn *lmdb.Txn) error {
		// databases can be inspected inside transactions.  here the number of
		// entries (keys) are printed.
		stat, err := txn.Stat(dbi)
		if err != nil {
			return err
		}
		fmt.Println(stat.Entries)
		return nil
	})
	if err != nil {
		panic(err)
	}

	err = env.Update(func(txn *lmdb.Txn) error {
		// random access of a key
		bval, err := txn.Get(dbi, []byte("key1"))
		if err != nil {
			return err
		}
		fmt.Println(string(bval))
		return nil
	})
	if err != nil {
		panic(err)
	}

}

Output:

3
val1
Example (DupFixed)

This complete example demonstrates populating and iterating a database with the DupFixed|DupSort DBI flags. The use case is probably too trivial to warrant such optimization but it demonstrates the key points.

Note the importance of supplying both DupFixed and DupSort flags on database creation. 

package main

import (
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"os"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// Open an environment as normal. DupSort is applied at the database level.
	env, err := lmdb.NewEnv()
	if err != nil {
		log.Panic(err)
	}
	path, err := ioutil.TempDir("", "mdb_test")
	if err != nil {
		log.Panic(err)
	}
	defer os.RemoveAll(path)
	err = env.Open(path, 0, 0644)
	defer env.Close()
	if err != nil {
		log.Panic(err)
	}

	// open the database of friends' phone numbers.  in this limited world
	// phone nubers are all the same length.
	var phonedbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		phonedbi, err = txn.OpenRoot(lmdb.DupSort | lmdb.DupFixed)
		return
	})
	if err != nil {
		panic(err)
	}

	// load some static values into the phone database.  values are loaded in
	// bulk using the PutMulti method on Cursor.
	err = env.Update(func(txn *lmdb.Txn) error {
		cur, err := txn.OpenCursor(phonedbi)
		if err != nil {
			return fmt.Errorf("cursor: %v", err)
		}
		defer cur.Close()

		for _, entry := range []struct {
			name    string
			numbers []string
		}{
			{"alice", []string{"234-1234"}},
			{"bob", []string{"825-1234"}},
			{"carol", []string{"828-1234", "824-1234", "502-1234"}}, // values are not sorted
			{"bob", []string{"433-1234", "957-1234"}},               // sorted dup values may be interleaved with existing dups
			{"jenny", []string{"867-5309"}},
		} {
			// write the values into a contiguous chunk of memory.  it is
			// critical that the values have the same length so the page has an
			// even stride.
			stride := len(entry.numbers[0])
			pagelen := stride * len(entry.numbers)
			data := bytes.NewBuffer(make([]byte, 0, pagelen))
			for _, num := range entry.numbers {
				io.WriteString(data, num)
			}

			// write the values to the database.
			err = cur.PutMulti([]byte(entry.name), data.Bytes(), stride, 0)
			if err != nil {
				return err
			}
		}

		return nil
	})
	if err != nil {
		panic(err)
	}

	// grab the first page of phone numbers for each name and print them.
	err = env.View(func(txn *lmdb.Txn) error {
		cur, err := txn.OpenCursor(phonedbi)
		if err != nil {
			return fmt.Errorf("cursor: %v", err)
		}
		defer cur.Close()
		for {
			// move to the next key
			name, phoneFirst, err := cur.Get(nil, nil, lmdb.NextNoDup)
			if lmdb.IsNotFound(err) {
				break
			}
			if err != nil {
				return fmt.Errorf("nextnodup: %v", err)
			}

			// determine if multiple values should be printed and short circuit
			// if not.
			ndup, err := cur.Count()
			if err != nil {
				return fmt.Errorf("count: %v", err)
			}
			if ndup == 1 {
				fmt.Printf("%s %s\n", name, phoneFirst)
				continue
			}

			// get a page of records and split it into discrete values.  the
			// length of the first item is used to split the page of contiguous
			// values.
			_, page, err := cur.Get(nil, nil, lmdb.GetMultiple)
			if err != nil {
				return fmt.Errorf("getmultiple: %v", err)
			}

			// print the phone numbers for the person. the first number is
			// printed on the same line as the person's name. others numbers of
			// offset to the same depth as the primary number.
			m := lmdb.WrapMulti(page, len(phoneFirst))
			fmt.Printf("%s %s\n", name, m.Val(0))
			offsetRest := bytes.Repeat([]byte{' '}, len(name))
			for i, n := 1, m.Len(); i < n; i++ {
				fmt.Printf("%s %s\n", offsetRest, m.Val(i))
			}
		}

		return nil
	})
	if err != nil {
		panic(err)
	}

}

Output:

alice 234-1234
bob 433-1234
    825-1234
    957-1234
carol 502-1234
      824-1234
      828-1234
jenny 867-5309
Example (DupSort)

This complete example demonstrates populating and iterating a database with the DupSort DBI flags. 

package main

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"log"
	"os"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// Open an environment as normal. DupSort is applied at the database level.
	env, err := lmdb.NewEnv()
	if err != nil {
		log.Panic(err)
	}
	path, err := ioutil.TempDir("", "mdb_test")
	if err != nil {
		log.Panic(err)
	}
	defer os.RemoveAll(path)
	err = env.Open(path, 0, 0644)
	defer env.Close()
	if err != nil {
		log.Panic(err)
	}

	var phonedbi lmdb.DBI

	// open the database of friends' phone numbers.  a single person can have
	// multiple phone numbers.
	err = env.Update(func(txn *lmdb.Txn) error {
		dbi, err := txn.OpenRoot(lmdb.DupSort)
		if err != nil {
			return err
		}
		phonedbi = dbi
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		for _, entry := range []struct{ name, number string }{
			{"alice", "234-1234"},
			{"bob", "825-1234"},
			{"carol", "824-1234"},
			{"jenny", "867-5309"},
			{"carol", "828-1234"}, // a second value for the key
			{"carol", "502-1234"}, // will be retrieved in sorted order
		} {
			err = cur.Put([]byte(entry.name), []byte(entry.number), 0)
			if err != nil {
				return err
			}
		}
		return nil
	})
	if err != nil {
		panic(err)
	}

	// iterate the database and print the phone numbers for each name.
	// multiple phone numbers for the same name are printed aligned on separate
	// rows.
	env.View(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(phonedbi)
		if err != nil {
			return err
		}

		var nameprev, name, phone []byte
		for {
			name, phone, err = cur.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				// the database was exausted
				return nil
			} else if err != nil {
				return err
			}

			// print the name and phone number. offset with space instead of
			// printing the name if name is a duplicate.
			isdup := bytes.Equal(nameprev, name)
			nameprev = name
			firstcol := name
			if isdup {
				firstcol = bytes.Repeat([]byte{' '}, len(name))
			}
			fmt.Printf("%s %s\n", firstcol, phone)
		}
	})
	if err != nil {
		panic(err)
	}

}

Output:

alice 234-1234
bob 825-1234
carol 502-1234
      824-1234
      828-1234
jenny 867-5309

func (*Txn) Abort 

func (txn *Txn) Abort()

Abort discards pending writes in the transaction. A Txn cannot be used again after Abort is called.

See mdb_txn_abort.

func (*Txn) Commit 

func (txn *Txn) Commit() error

Commit commits all operations of the transaction to the database. A Txn cannot be used again after Commit is called.

See mdb_txn_commit.

func (*Txn) CreateDBI 

func (txn *Txn) CreateDBI(name string) (DBI, error)

CreateDBI is a shorthand for OpenDBI that passed the flag lmdb.Create.

func (*Txn) Del 

func (txn *Txn) Del(dbi DBI, key, val []byte) error

Del deletes an item from database dbi. Del ignores val unless dbi has the DupSort flag.

See mdb_del.

func (*Txn) Drop 

func (txn *Txn) Drop(dbi DBI, del bool) error

Drop empties the database if del is false. Drop deletes and closes the database if del is true.

See mdb_drop.

func (*Txn) Flags 

func (txn *Txn) Flags(dbi DBI) (uint, error)

Flags returns the database flags for handle dbi.

func (*Txn) Get 

func (txn *Txn) Get(dbi DBI, key []byte) ([]byte, error)

Get retrieves items from database dbi. If txn.RawRead is true the slice returned by Get references a readonly section of memory that must not be accessed after txn has terminated.

See mdb_get.

Example

This example shows how to properly handle data retrieved from the database and applies to Txn.Get() as well as Cursor.Get(). It is important to handle data retreival carefully to make sure the application does not retain pointers to memory pages which may be reclaimed by LMDB after the transaction terminates. Typically an application would define helper functions/methods to conveniently handle data safe retrieval. 

package main

import (
	"encoding/json"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values shouldn't actually be assigned to.  The are used as stand-ins
// for tests which do not act as tests.
var EnvEx *lmdb.Env
var DBIEx lmdb.DBI

func main() {
	// variables to hold data extracted from the database
	var point struct{ X, Y int }
	var str string
	var p1, p2 []byte

	// extract data from an example environment/database.  it is critical for application
	// code to handle errors  but that is omitted here to save space.
	EnvEx.View(func(txn *lmdb.Txn) (err error) {
		// OK
		// A []byte to string conversion will always copy the data
		v, _ := txn.Get(DBIEx, []byte("mykey"))
		str = string(v)

		// OK
		// If []byte is the desired data type then an explicit copy is required
		// for safe access after the transaction returns.
		v, _ = txn.Get(DBIEx, []byte("mykey"))
		p1 = make([]byte, len(v))
		copy(p1, v)

		// OK
		// The data does not need be copied because it is parsed while txn is
		// open.
		v, _ = txn.Get(DBIEx, []byte("mykey"))
		_ = json.Unmarshal(v, &point)

		// BAD
		// Assigning the result directly to p2 leaves its pointer volatile
		// after the transaction completes which can result in unpredictable
		// behavior.
		p2, _ = txn.Get(DBIEx, []byte("mykey"))

		return nil
	})
}

Output:

func (*Txn) OpenCursor 

func (txn *Txn) OpenCursor(dbi DBI) (*Cursor, error)

OpenCursor allocates and initializes a Cursor to database dbi.

See mdb_cursor_open.

func (*Txn) OpenDBI 

func (txn *Txn) OpenDBI(name string, flags uint) (DBI, error)

OpenDBI opens a named database in the environment. An error is returned if name is empty. The DBI returned by OpenDBI can be used in other transactions but not before Txn has terminated.

OpenDBI can only be called after env.SetMaxDBs() has been called to set the maximum number of named databases.

The C API uses null terminated strings for database names. A consequence is that names cannot contain null bytes themselves. OpenDBI does not check for null bytes in the name argument.

See mdb_dbi_open.

Example 
package main

import (
	"fmt"
	"io/ioutil"
	"os"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	dbpath, err := ioutil.TempDir("", "lmdb-test")
	if err != nil {
		panic(err)
	}
	defer os.RemoveAll(dbpath)

	env, err := lmdb.NewEnv()
	if err != nil {
		panic(err)
	}

	// when using named databases SetMaxDBs is required to be at least the
	// number of named databases needed.
	if err = env.SetMaxDBs(1); err != nil {
		panic(err)
	}

	err = env.Open(dbpath, 0, 0644)
	defer env.Close()
	if err != nil {
		panic(err)
	}

	var db1 lmdb.DBI
	var dbroot lmdb.DBI
	env.Update(func(txn *lmdb.Txn) (err error) {
		_, err = txn.OpenDBI("db0", 0) // ErrNotFound
		if err != nil {
			fmt.Println("db0", err)
		}
		db1, err = txn.OpenDBI("db1", lmdb.Create)
		if err != nil {
			fmt.Println("db1", err)
		}
		_, err = txn.OpenDBI("db2", lmdb.Create) // ErrDBsFull
		if err != nil {
			fmt.Println("db5", err)
		}
		dbroot, err = txn.OpenRoot(0) // does not count against maxdbs
		if err != nil {
			fmt.Println("root", err)
		}
		return nil
	})

	env.View(func(txn *lmdb.Txn) error {
		cursor, err := txn.OpenCursor(dbroot)
		if err != nil {
			return fmt.Errorf("cursor: %v", err)
		}

		fmt.Println("databases:")
		for {
			k, _, err := cursor.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return fmt.Errorf("root next: %v", err)
			}
			fmt.Printf("  %s\n", k)
		}
	})

}

Output:

db0 mdb_dbi_open: MDB_NOTFOUND: No matching key/data pair found
db5 mdb_dbi_open: MDB_DBS_FULL: Environment maxdbs limit reached
databases:
  db1

func (*Txn) OpenRoot 

func (txn *Txn) OpenRoot(flags uint) (DBI, error)

OpenRoot opens the root database. OpenRoot behaves similarly to OpenDBI but does not require env.SetMaxDBs() to be called beforehand. And, OpenRoot can be called without flags in a View transaction.

Example

Txn.OpenRoot does not need to be called with the lmdb.Create flag. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values shouldn't actually be assigned to.  The are used as stand-ins
// for tests which do not act as tests.
var EnvEx *lmdb.Env
var DBIEx lmdb.DBI

func main() {
	err := EnvEx.Update(func(txn *lmdb.Txn) (err error) {
		DBIEx, err = txn.OpenRoot(0)
		return err
	})
	if err != nil {
		panic(err)
	}
}

Output:
Example (View)

Txn.OpenRoot may be called without flags inside View transactions. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values shouldn't actually be assigned to.  The are used as stand-ins
// for tests which do not act as tests.
var EnvEx *lmdb.Env

func main() {
	err := EnvEx.View(func(txn *lmdb.Txn) (err error) {
		db, err := txn.OpenRoot(0)
		if err != nil {
			return err
		}
		cur, err := txn.OpenCursor(db)
		if err != nil {
			return err
		}
		for {
			k, v, err := cur.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				panic(err)
			}
			fmt.Printf("%s: %s", k, v)
		}
	})
	if err != nil {
		panic(err)
	}
}

Output:

func (*Txn) Put 

func (txn *Txn) Put(dbi DBI, key []byte, val []byte, flags uint) error

Put stores an item in database dbi.

See mdb_put.

func (*Txn) PutReserve 

func (txn *Txn) PutReserve(dbi DBI, key []byte, n int, flags uint) ([]byte, error)

PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.

Example

This example demonstrates the use of PutReserve to store a string value in the root database. This may be faster than Put alone for large values because a string to []byte conversion is not required. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values shouldn't actually be assigned to.  The are used as stand-ins
// for tests which do not act as tests.
var EnvEx *lmdb.Env

func main() {
	EnvEx.Update(func(txn *lmdb.Txn) (err error) {
		dbroot, err := txn.OpenRoot(0)
		if err != nil {
			return err
		}

		valstr := "value"
		p, err := txn.PutReserve(dbroot, []byte("key"), len(valstr), 0)
		if err != nil {
			return err
		}
		copy(p, valstr)

		return nil
	})
}

Output:

func (*Txn) Renew 

func (txn *Txn) Renew() error

Renew reuses a transaction that was previously reset by calling txn.Reset(). Renew panics if txn is managed by Update, View, etc.

See mdb_txn_renew.

func (*Txn) Reset 

func (txn *Txn) Reset()

Reset aborts the transaction clears internal state so the transaction may be reused by calling Renew. If txn is not going to be reused txn.Abort() must be called to release its slot in the lock table and free its memory. Reset panics if txn is managed by Update, View, etc.

See mdb_txn_reset.

func (*Txn) Stat 

func (txn *Txn) Stat(dbi DBI) (*Stat, error)

Stat returns a Stat describing the database dbi.

See mdb_stat.

func (*Txn) Sub 

func (txn *Txn) Sub(fn TxnOp) error

Sub executes fn in a subtransaction. Sub commits the subtransaction iff a nil error is returned by fn and otherwise aborts it. Sub returns any error it encounters.

Sub may only be called on an Update (a Txn created without the Readonly flag). Calling Sub on a View transaction will return an error.

Any call to Abort, Commit, Renew, or Reset on a Txn created by Sub will panic.

type TxnOp 

type TxnOp func(txn *Txn) error

TxnOp is an operation applied to a managed transaction. The Txn passed to a TxnOp is managed and the operation must not call Commit, Abort, Renew, or Reset on it.

IMPORTANT: TxnOps that write to the database (those passed to Update or BeginUpdate) must not use the Txn in another goroutine (passing it directly or otherwise through closure). Doing so has undefined results.

Notes

Bugs

  • MDB_INTEGERKEY and MDB_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.