lmdb

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 10, 2015 License: BSD-3-Clause Imports: 12 Imported by: 17

Documentation

Overview

Package lmdb provides bindings to the lmdb C API. The package bindings are fairly low level and are designed to provide a minimal interface that prevents misuse to a reasonable extent. When in doubt refer to the C documentation as a reference. 

http://symas.com/mdb/doc/group__mdb.html

Environment

An LMDB environment holds named databases (key-value stores). An environment is represented as one file on the filesystem (though often a corresponding lock file exists).

LMDB recommends setting an environment's size as large as possible at the time of creation. On filesystems that support sparse files this should not adversely affect disk usage. Resizing an environment is possible but must be handled with care when concurrent access is involved.

Databases

A database in an LMDB environment is an ordered key-value store that holds arbitrary binary data. Typically the keys are unique but duplicate keys may be allowed (DupSort), in which case the values for each duplicate key are ordered.

A single LMDB environment can have multiple named databases. But there is also a 'root' (unnamed) database that can be used to store data. Use caution storing data in the root database when named databases are in use. The root database serves as an index for named databases.

A database is referenced by an opaque handle known as its DBI which must be opened inside a transaction with the OpenDBI or OpenRoot methods. DBIs may be closed but it is not required. Typically, applications aquire handles for all their databases immediately after opening an environment and retain them for the lifetime of the process.

Transactions

View (readonly) transactions in LMDB operate on a snapshot of the database at the time the transaction began. The number of simultaneously active view transactions is bounded and configured when the environment is initialized.

Update (read-write) transactions are serialized in LMDB. Attempts to create update transactions block until a lock may be obtained. Update transactions can create subtransactions which may be rolled back independently from their parent.

The lmdb package supplies managed and unmanaged transactions. Managed transactions do not require explicit calling of Abort/Commit and are provided through the Env methods Update, View, and RunTxn. The BeginTxn method on Env creates an unmanaged transaction but its use is not advised in most applications.

Example

This example demonstrates a complete workflow for an lmdb environment. The Env is first created. After being configured the Env is mapped to memory. Once mapped, database handles are opened and normal database operations begin. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// create an environment and make sure it is eventually closed.
	env, err := lmdb.NewEnv()
	if err != nil {
		// ...
	}
	defer env.Close()

	// configure and open the environment.  most configuration must be done
	// before opening the environment.
	err = env.SetMaxDBs(1)
	if err != nil {
		// ..
	}
	err = env.SetMapSize(1 << 30)
	if err != nil {
		// ..
	}
	err = env.Open("/path/to/db/", 0, 0644)
	if err != nil {
		// ..
	}

	// open a database that can be used as long as the enviroment is mapped.
	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.CreateDBI("example")
		return err
	})
	if err != nil {
		// ...
	}

	// the database is now ready for use.  read the value for a key and print
	// it to standard output.
	err = env.View(func(txn *lmdb.Txn) (err error) {
		v, err := txn.Get(dbi, []byte("hello"))
		if err != nil {
			return err
		}
		fmt.Println(string(v))
		return nil
	})
	if err != nil {
		// ...
	}
}

Output:

Index

Examples

Constants

View Source 
const (
	First        = C.MDB_FIRST          // The first item.
	FirstDup     = C.MDB_FIRST_DUP      // The first value of current key (DupSort).
	GetBoth      = C.MDB_GET_BOTH       // Get the key as well as the value (DupSort).
	GetBothRange = C.MDB_GET_BOTH_RANGE // Get the key and the nearsest value (DupSort).
	GetCurrent   = C.MDB_GET_CURRENT    // Get the key and value at the current position.
	GetMultiple  = C.MDB_GET_MULTIPLE   // Get up to a page dup values for key at current position (DupFixed).
	Last         = C.MDB_LAST           // Last item.
	LastDup      = C.MDB_LAST_DUP       // Position at last value of current key (DupSort).
	Next         = C.MDB_NEXT           // Next value.
	NextDup      = C.MDB_NEXT_DUP       // Next value of the current key (DupSort).
	NextMultiple = C.MDB_NEXT_MULTIPLE  // Get key and up to a page of values from the next cursor position (DupFixed).
	NextNoDup    = C.MDB_NEXT_NODUP     // The first value of the next key (DupSort).
	Prev         = C.MDB_PREV           // The previous item.
	PrevDup      = C.MDB_PREV_DUP       // The previous item of the current key (DupSort).
	PrevNoDup    = C.MDB_PREV_NODUP     // The last data item of the previous key (DupSort).
	Set          = C.MDB_SET            // The specified key.
	SetKey       = C.MDB_SET_KEY        // Get key and data at the specified key.
	SetRange     = C.MDB_SET_RANGE      // The first key no less than the specified key.
)

These flags are used exclusively for Cursor.Get.

View Source 
const (
	Current     = C.MDB_CURRENT     // Replace the item at the current key position (Cursor only)
	NoDupData   = C.MDB_NODUPDATA   // Store the key-value pair only if key is not present (DupSort).
	NoOverwrite = C.MDB_NOOVERWRITE // Store a new key-value pair only if key is not present.
	Append      = C.MDB_APPEND      // Append an item to the database.
	AppendDup   = C.MDB_APPENDDUP   // Append an item to the database (DupSort).
)

The MDB_MULTIPLE and MDB_RESERVE flags are special and do not fit the calling pattern of other calls to Put. They are not exported because they require special methods, PutMultiple and PutReserve in which the flag is implied and does not need to be passed.

View Source 
const (
	FixedMap    = C.MDB_FIXEDMAP   // Danger zone. Map memory at a fixed address.
	NoSubdir    = C.MDB_NOSUBDIR   // Argument to Open is a file, not a directory.
	Readonly    = C.MDB_RDONLY     // Used in several functions to denote an object as readonly.
	WriteMap    = C.MDB_WRITEMAP   // Use a writable memory map.
	NoMetaSync  = C.MDB_NOMETASYNC // Don't fsync metapage after commit.
	NoSync      = C.MDB_NOSYNC     // Don't fsync after commit.
	MapAsync    = C.MDB_MAPASYNC   // Flush asynchronously when using the WriteMap flag.
	NoTLS       = C.MDB_NOTLS      // Danger zone. When unset reader locktable slots are tied to their thread.
	NoLock      = C.MDB_NOLOCK     // Danger zone. LMDB does not use any locks.
	NoReadahead = C.MDB_NORDAHEAD  // Disable readahead. Requires OS support.
	NoMemInit   = C.MDB_NOMEMINIT  // Disable LMDB memory initialization.
)

These flags are used exclusively for Env types and are set with Env.Open. Some flags may be set/unset later using Env.Set/.Unset methods. Others will produce syscall.EINVAL. Refer to the C documentation for detailed information.

View Source 
const (
	ReverseKey = C.MDB_REVERSEKEY // Use reverse string keys.
	DupSort    = C.MDB_DUPSORT    // Use sorted duplicates.
	DupFixed   = C.MDB_DUPFIXED   // Duplicate items have a fixed size (DupSort).
	ReverseDup = C.MDB_REVERSEDUP // Reverse duplicate values (DupSort).
	Create     = C.MDB_CREATE     // Create DB if not already existing.
)

This flags are used exclusively for Txn.OpenDBI and Txn.OpenRoot. The Create flag must always be supplied when opening a non-root DBI for the first time.

BUG(bmatsuo): MDB_INTEGERKEY and MDB_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.

View Source 
const (
	CopyCompact = C.MDB_CP_COMPACT // Perform compaction while copying
)

These flags are exclusively used in the Env.CopyFlags and Env.CopyFDFlags methods.

Variables

This section is empty.

Functions

func IsErrno 

func IsErrno(err error, errno Errno) bool

IsErrno returns true if err's errno is the given errno.

func IsErrnoFn 

func IsErrnoFn(err error, fn func(error) bool) bool

IsErrnoFn calls fn on the error underlying err and returns the result. If err is an *OpError then err.Errno is passed to fn. Otherwise err is passed directly to fn.

func IsErrnoSys 

func IsErrnoSys(err error, errno syscall.Errno) bool

IsErrnoSys returns true if err's errno is the given errno.

func IsMapFull 

func IsMapFull(err error) bool

IsMapFull returns true if the environment map size has been reached.

func IsMapResized 

func IsMapResized(err error) bool

IsMapResized returns true if the environment has grown too large for the current map after being resized by another process.

func IsNotExist 

func IsNotExist(err error) bool

IsNotExist returns true the path passed to the Env.Open method does not exist.

func IsNotFound 

func IsNotFound(err error) bool

IsNotFound returns true if the key requested in Txn.Get or Cursor.Get does not exist or if the Cursor reached the end of the database without locating a value (EOF).

func Version 

func Version() (major, minor, patch int, s string)

Version return the major, minor, and patch version numbers of the LMDB C library and a string representation of the version.

See mdb_version.

func VersionString 

func VersionString() string

VersionString returns a string representation of the LMDB C library version.

See mdb_version.

Types

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor operates on data inside a transaction and holds a position in the database.

See MDB_cursor.

func (*Cursor) Close 

func (c *Cursor) Close()

Close the cursor handle and clear the finalizer on c. Cursors belonging to write transactions are closed automatically when the transaction is terminated.

See mdb_cursor_close.

func (*Cursor) Count 

func (c *Cursor) Count() (uint64, error)

Count returns the number of duplicates for the current key.

See mdb_cursor_count.

Example

This example demonstrates how to iterate a database opened with the DupSort flag and get the number of values present for each distinct key. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		for {
			k, _, err := cur.Get(nil, nil, lmdb.NextNoDup)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return err
			}

			numdup, err := cur.Count()
			if err != nil {
				return err
			}
			fmt.Printf("%d values for key %q", numdup, k)
		}
	})
}

Output:

func (*Cursor) DBI 

func (c *Cursor) DBI() DBI

DBI returns the cursor's database handle. If c has been closed than an invalid DBI is returned.

func (*Cursor) Del 

func (c *Cursor) Del(flags uint) error

Del deletes the item referred to by the cursor from the database.

See mdb_cursor_del.

Example

This example demonstrates how to delete all elements in a database with a key less than a given value (an RFC3339 timestamp in this case). 

package main

import (
	"bytes"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	before := []byte("2014-05-06T03:04:02Z")
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		for {
			k, _, err := cur.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return err
			}

			if bytes.Compare(k, before) != -1 {
				return nil
			}

			err = cur.Del(0)
			if err != nil {
				return err
			}
		}
	})
}

Output:

func (*Cursor) Get 

func (c *Cursor) Get(setkey, setval []byte, op uint) (key, val []byte, err error)

Get retrieves items from the database. If c.Txn().RawRead is true the slices returned by Get reference readonly sections of memory that must not be accessed after the transaction has terminated.

Get ignores setval if setkey is empty.

See mdb_cursor_get.

Example

This simple example shows how to iterate a database. The Next flag may be used without an initial call passing the First flag. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	err = env.View(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		for {
			k, v, err := cur.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return err
			}

			fmt.Printf("%s %s\n", k, v)
		}
	})
}

Output:
Example (DupFixed)

This simple example shows how to iterate a database opened with the DupSort|DupSort flags. It is not necessary to use the GetMultiple flag before passing the NextMultiple flag. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	err = env.View(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		for {
			k, first, err := cur.Get(nil, nil, lmdb.NextNoDup)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return err
			}

			stride := len(first)

			for {
				_, v, err := cur.Get(nil, nil, lmdb.NextMultiple)
				if lmdb.IsNotFound(err) {
					break
				}
				if err != nil {
					return err
				}

				multi := lmdb.WrapMulti(v, stride)
				for i := 0; i < multi.Len(); i++ {
					fmt.Printf("%s %s\n", k, multi.Val(i))
				}
			}
		}
	})
}

Output:
Example (DupSort)

This example shows how duplicates can be processed using LMDB. It is possible to iterate all key-value pairs (including duplicate key values) by passing Next. But if special handling of duplicates is needed it may be beneficial to use NextNoDup or NextDup. 

package main

import (
	"log"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	err = env.View(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}

		for {
			k, v, err := cur.Get(nil, nil, lmdb.NextNoDup)
			if lmdb.IsNotFound(err) {
				// the database was exausted
				return nil
			} else if err != nil {
				return err
			}

			// process duplicates
			var dups [][]byte
			for {
				dups = append(dups, v)

				_, v, err = cur.Get(nil, nil, lmdb.NextDup)
				if lmdb.IsNotFound(err) {
					break
				} else if err != nil {
					return err
				}
			}

			log.Printf("%q %q", k, dups)
		}
	})
}

Output:
Example (Reverse)

This simple example shows how to iterate a database in reverse. As when passing the Next flag, the Prev flag may be used without an initial call using Last. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	err = env.View(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		for {
			k, v, err := cur.Get(nil, nil, lmdb.Prev)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return err
			}

			fmt.Printf("%s %s\n", k, v)
		}
	})
}

Output:

func (*Cursor) Put 

func (c *Cursor) Put(key, val []byte, flags uint) error

Put stores an item in the database.

See mdb_cursor_put.

func (*Cursor) PutMulti 

func (c *Cursor) PutMulti(key []byte, page []byte, stride int, flags uint) error

PutMulti stores a set of contiguous items with stride size under key. PutMulti panics if len(page) is not a multiple of stride. The cursor's database must be DupFixed and DupSort.

See mdb_cursor_put.

Example

This example shows how to write a page of contiguous, fixed-size values to a database opened with DupSort|DupFixed. It doesn't matter if the values are sorted. Values will be stored in sorted order. 

package main

import (
	"bytes"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var dbi lmdb.DBI

var err error

func main() {
	key := []byte("k")
	items := [][]byte{
		[]byte("v0"),
		[]byte("v2"),
		[]byte("v1"),
	}
	page := bytes.Join(items, nil)
	stride := 2

	err = env.Update(func(txn *lmdb.Txn) (err error) {
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		defer cur.Close()

		return cur.PutMulti(key, page, stride, 0)
	})
}

Output:

func (*Cursor) PutReserve 

func (c *Cursor) PutReserve(key []byte, n int, flags uint) ([]byte, error)

PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.

func (*Cursor) Renew 

func (c *Cursor) Renew(txn *Txn) error

Renew associates readonly cursor with txn.

See mdb_cursor_renew.

Example

This example shows a trivial case using Renew to service read requests on a database. Close must be called when the cursor will no longer be renewed. Before using Renew benchmark your application to understand its benefits. 

package main

import (
	"log"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	var cur *lmdb.Cursor
	err = env.View(func(txn *lmdb.Txn) (err error) {
		dbi, err := txn.OpenRoot(0)
		if err != nil {
			return err
		}
		cur, err = txn.OpenCursor(dbi)
		return err
	})
	if err != nil {
		panic(err)
	}

	keys := make(chan []byte)
	go func() {

		// Close must called when the cursor is no longer needed.
		defer cur.Close()

		for key := range keys {
			err := env.View(func(txn *lmdb.Txn) (err error) {
				err = cur.Renew(txn)
				if err != nil {
					return err
				}

				// retrieve the number of items in the database with the given
				// key (DupSort).
				count := uint64(0)
				_, _, err = cur.Get(key, nil, lmdb.SetKey)
				if lmdb.IsNotFound(err) {
					err = nil
				} else if err == nil {
					count, err = cur.Count()
				}
				if err != nil {
					return err
				}

				log.Printf("%d %q", count, key)

				return nil
			})
			if err != nil {
				panic(err)
			}
		}
	}()

	// ...
}

Output:

func (*Cursor) Txn 

func (c *Cursor) Txn() *Txn

Txn returns the cursor's transaction.

type DBI 

type DBI C.MDB_dbi

DBI is a handle for a database in an Env.

See MDB_dbi

type Env 

type Env struct {
	// contains filtered or unexported fields
}

Env is opaque structure for a database environment. A DB environment supports multiple databases, all residing in the same shared-memory map.

See MDB_env.

Example

This example shows the general workflow of LMDB. An environment is created and configured before being opened. After the environment is opened its databases are created and their handles are saved for use in future transactions. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	// open the LMDB environment and configure common options like its size and
	// maximum number of databases.
	env, err := lmdb.NewEnv()
	if err != nil {
		// ...
	}
	err = env.SetMapSize(100 * 1024 * 1024) // 100MB
	if err != nil {
		// ...
	}
	err = env.SetMaxDBs(1)
	if err != nil {
		// ...
	}

	// open the environment only after the it has been configured.  some
	// settings may only be called before the environment is opened where
	// others may have caveats.
	err = env.Open("mydb/", 0, 0664)
	if err != nil {
		// ...
	}
	defer env.Close()

	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		// open a database, creating it if necessary.  the database is stored
		// outside the transaction via closure and can be use after the
		// transaction is committed.
		dbi, err = txn.OpenDBI("exampledb", lmdb.Create)
		if err != nil {
			return err
		}

		// commit the transaction, writing an entry for the newly created
		// database if it was just created and allowing the dbi to be used in
		// future transactions.
		return nil
	})
	if err != nil {
		panic(err)
	}
}

Output:

func NewEnv 

func NewEnv() (*Env, error)

NewEnv allocates and initializes a new Env.

See mdb_env_create.

func (*Env) BeginTxn 

func (env *Env) BeginTxn(parent *Txn, flags uint) (*Txn, error)

BeginTxn is an unsafe, low-level method to initialize a new transaction on env. The Txn returned by BeginTxn is unmanaged and must be terminated by calling either its Abort or Commit methods to ensure that its resources are released.

A finalizer detects unreachable, live transactions and logs thems to standard error. The transactions are aborted, but their presence should be interpreted as an application error which should be patched so transactions are terminated explicitly. Unterminated transactions can adversly effect database performance and cause the database to grow until the map is full.

BeginTxn does not attempt to serialize write transaction operations to an OS thread and without care its use for write transactions can have undefined results.

Instead of BeginTxn users should call the View, Update, RunTxn methods.

See mdb_txn_begin.

func (*Env) Close 

func (env *Env) Close() error

Close shuts down the environment, releases the memory map, and clears the finalizer on env.

See mdb_env_close.

func (*Env) CloseDBI 

func (env *Env) CloseDBI(db DBI)

CloseDBI closes the database handle, db. Normally calling CloseDBI explicitly is not necessary.

It is the caller's responsibility to serialize calls to CloseDBI.

See mdb_dbi_close.

func (*Env) Copy 

func (env *Env) Copy(path string) error

Copy copies the data in env to an environment at path.

See mdb_env_copy.

Example

This example uses env.Copy to periodically create atomic backups of an environment. A real implementation would need to solve problems with failures, remote persistence, purging old backups, etc. But the core loop would have the same form. 

package main

import (
	"fmt"
	"os"
	"time"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error
var stop chan struct{}

func main() {
	go func(backup <-chan time.Time) {
		for {
			select {
			case <-backup:
				now := time.Now().UTC()
				backup := fmt.Sprintf("backup-%s", now.Format(time.RFC3339))
				os.Mkdir(backup, 0755)
				err = env.Copy(backup)
				if err != nil {
					// ...
					continue
				}
			case <-stop:
				return
			}
		}
	}(time.Tick(time.Hour))
}

Output:

func (*Env) CopyFD added in v1.1.1 

func (env *Env) CopyFD(fd uintptr) error

CopyFD copies env to the the file descriptor fd.

See mdb_env_copyfd.

func (*Env) CopyFDFlag added in v1.1.1 

func (env *Env) CopyFDFlag(fd uintptr, flags uint) error

CopyFDFlag copies env to the file descriptor fd, with options.

See mdb_env_copyfd2.

func (*Env) CopyFlag 

func (env *Env) CopyFlag(path string, flags uint) error

CopyFlag copies the data in env to an environment at path created with flags.

See mdb_env_copy2.

func (*Env) FD added in v1.2.0 

func (env *Env) FD() (uintptr, error)

FD returns the open file descriptor (or Windows file handle) for the given environment. An error is returned if the environment has not been successfully Opened (where C API just retruns an invalid handle).

See mdb_env_get_fd.

func (*Env) Flags 

func (env *Env) Flags() (uint, error)

Flags returns the flags set in the environment.

See mdb_env_get_flags.

func (*Env) Info 

func (env *Env) Info() (*EnvInfo, error)

Info returns information about the environment.

See mdb_env_info.

func (*Env) MaxKeySize 

func (env *Env) MaxKeySize() int

MaxKeySize returns the maximum allowed length for a key.

See mdb_env_get_maxkeysize.

func (*Env) MaxReaders 

func (env *Env) MaxReaders() (int, error)

MaxReaders returns the maximum number of reader slots for the environment.

See mdb_env_get_maxreaders.

func (*Env) Open 

func (env *Env) Open(path string, flags uint, mode os.FileMode) error

Open an environment handle. If this function fails Close() must be called to discard the Env handle. Open passes flags|NoTLS to mdb_env_open.

See mdb_env_open.

func (*Env) Path 

func (env *Env) Path() (string, error)

Path returns the path argument passed to Open. Path returns a non-nil error if env.Open() was not previously called.

See mdb_env_get_path.

func (*Env) ReaderCheck 

func (env *Env) ReaderCheck() (int, error)

ReaderCheck clears stale entries from the reader lock table and returns the number of entries cleared.

See mdb_reader_check()

func (*Env) ReaderList added in v1.1.1 

func (env *Env) ReaderList(fn func(string) error) error

ReaderList dumps the contents of the reader lock table as text. Readers start on the second line as space-delimited fields described by the first line.

See mdb_reader_list.

func (*Env) RunTxn 

func (env *Env) RunTxn(flags uint, fn TxnOp) error

RunTxn creates a new Txn and calls fn with it as an argument. Run commits the transaction if fn returns nil otherwise the transaction is aborted. Because RunTxn terminates the transaction goroutines should not retain references to it or its data after fn returns.

RunTxn does not lock the thread of the calling goroutine. Unless the Readonly flag is passed the calling goroutine should ensure it is locked to its thread.

See mdb_txn_begin.

func (*Env) SetFlags 

func (env *Env) SetFlags(flags uint) error

SetFlags sets flags in the environment.

See mdb_env_set_flags.

func (*Env) SetMapSize 

func (env *Env) SetMapSize(size int64) error

SetMapSize sets the size of the environment memory map.

See mdb_env_set_mapsize.

Example

This example demonstrates how an application typically uses Env.SetMapSize. The call to Env.SetMapSize() is made before calling env.Open(). Any calls after calling Env.Open() must take special care to synchronize with other goroutines. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

func main() {
	env, err := lmdb.NewEnv()
	if err != nil {
		// ...
	}

	// set the memory map size (maximum database size) to 1GB.
	err = env.SetMapSize(1 << 30)
	if err != nil {
		// ...
	}

	err = env.Open("mydb", 0, 0644)
	if err != nil {
		// ...
	}
	// ...
}

Output:
Example (MapResized)

This example demonstrates how to handle a MapResized error, encountered after another process has called mdb_env_set_mapsize (Env.SetMapSize). Applications which don't expect another process to resize the mmap don't need to check for the MapResized error.

The example is simplified for clarity. Many real applications will need to synchronize calls to Env.SetMapSize using something like a sync.RWMutex to ensure there are no active readonly transactions (those opened successfully before MapResized was encountered). 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

// These values can be used as no-op placeholders in examples.
func doUpdate(txn *lmdb.Txn) error { return nil }

func main() {
retry:
	err := env.Update(doUpdate)
	if lmdb.IsMapResized(err) {
		// If concurrent read transactions are possible then a sync.RWMutex
		// must be used here to ensure they all terminate before calling
		// env.SetMapSize().
		err = env.SetMapSize(0)
		if err != nil {
			panic(err)
		}

		// retry the update. a goto is not necessary but it simplifies error
		// handling with minimal overhead.
		goto retry
	} else if err != nil {
		// ...
	}
	// ...
}

Output:

func (*Env) SetMaxDBs 

func (env *Env) SetMaxDBs(size int) error

SetMaxDBs sets the maximum number of named databases for the environment.

See mdb_env_set_maxdbs.

func (*Env) SetMaxReaders 

func (env *Env) SetMaxReaders(size int) error

SetMaxReaders sets the maximum number of reader slots in the environment.

See mdb_env_set_maxreaders.

func (*Env) Stat 

func (env *Env) Stat() (*Stat, error)

Stat returns statistics about the environment.

See mdb_env_stat.

func (*Env) Sync 

func (env *Env) Sync(force bool) error

Sync flushes buffers to disk. If force is true a synchronous flush occurs and ignores any NoSync or MapAsync flag on the environment.

See mdb_env_sync.

func (*Env) UnsetFlags 

func (env *Env) UnsetFlags(flags uint) error

UnsetFlags clears flags in the environment.

See mdb_env_set_flags.

func (*Env) Update 

func (env *Env) Update(fn TxnOp) error

Update calls fn with a writable transaction. Update commits the transaction if fn returns a nil error otherwise Update aborts the transaction and returns the error.

Update locks the calling goroutine to its thread and unlocks it after fn returns. The Txn must not be used from multiple goroutines, even with synchronization.

Any call to Commit, Abort, Reset or Renew on a Txn created by Update will panic.

func (*Env) UpdateLocked 

func (env *Env) UpdateLocked(fn TxnOp) error

UpdateLocked behaves like Update but does not lock the calling goroutine to its thread. UpdateLocked should be used if the calling goroutine is already locked to its thread for another purpose.

Any call to Commit, Abort, Reset or Renew on a Txn created by UpdateLocked will panic.

func (*Env) View 

func (env *Env) View(fn TxnOp) error

View creates a readonly transaction with a consistent view of the environment and passes it to fn. View terminates its transaction after fn returns. Any error encountered by View is returned.

Any call to Commit, Abort, Reset or Renew on a Txn created by View will panic.

type EnvInfo 

type EnvInfo struct {
	MapSize    int64 // Size of the data memory map
	LastPNO    int64 // ID of the last used page
	LastTxnID  int64 // ID of the last committed transaction
	MaxReaders uint  // maximum number of threads for the environment
	NumReaders uint  // maximum number of threads used in the environment
}

EnvInfo contains information an environment.

See MDB_envinfo.

type Errno 

type Errno C.int

Errno is an error type that represents the (unique) errno values defined by LMDB. Other errno values (such as EINVAL) are represented with type syscall.Errno. On Windows, LMDB return codes are translated into portable syscall.Errno constants (e.g. syscall.EINVAL, syscall.EACCES, etc.).

Most often helper functions such as IsNotFound may be used instead of dealing with Errno values directly. 

lmdb.IsNotFound(err)
lmdb.IsErrno(err, lmdb.TxnFull)
lmdb.IsErrnoSys(err, syscall.EINVAL)
lmdb.IsErrnoFn(err, os.IsPermission)

const (
	KeyExist        Errno = C.MDB_KEYEXIST
	NotFound        Errno = C.MDB_NOTFOUND
	PageNotFound    Errno = C.MDB_PAGE_NOTFOUND
	Corrupted       Errno = C.MDB_CORRUPTED
	Panic           Errno = C.MDB_PANIC
	VersionMismatch Errno = C.MDB_VERSION_MISMATCH
	Invalid         Errno = C.MDB_INVALID
	MapFull         Errno = C.MDB_MAP_FULL
	DBsFull         Errno = C.MDB_DBS_FULL
	ReadersFull     Errno = C.MDB_READERS_FULL
	TLSFull         Errno = C.MDB_TLS_FULL
	TxnFull         Errno = C.MDB_TXN_FULL
	CursorFull      Errno = C.MDB_CURSOR_FULL
	PageFull        Errno = C.MDB_PAGE_FULL
	MapResized      Errno = C.MDB_MAP_RESIZED
	Incompatible    Errno = C.MDB_INCOMPATIBLE
	BadRSlot        Errno = C.MDB_BAD_RSLOT
	BadTxn          Errno = C.MDB_BAD_TXN
	BadValSize      Errno = C.MDB_BAD_VALSIZE
	BadDBI          Errno = C.MDB_BAD_DBI
)

The most common error codes do not need to be handled explicity. Errors can be checked through helper functions IsNotFound, IsMapFull, etc, Otherwise they should be checked using the IsErrno function instead of direct comparison because they will typically be wrapped with an OpError.

func (Errno) Error 

func (e Errno) Error() string

type Multi 

type Multi struct {
	// contains filtered or unexported fields
}

Multi is a wrapper for a contiguous page of sorted, fixed-length values passed to Cursor.PutMulti or retrieved using Cursor.Get with the GetMultiple/NextMultiple flag.

Multi values are only useful in databases opened with DupSort|DupFixed.

func WrapMulti 

func WrapMulti(page []byte, stride int) *Multi

WrapMulti converts a page of contiguous values with stride size into a Multi. WrapMulti panics if len(page) is not a multiple of stride. 

_, val, _ := cursor.Get(nil, nil, lmdb.FirstDup)
_, page, _ := cursor.Get(nil, nil, lmdb.GetMultiple)
multi := lmdb.WrapMulti(page, len(val))

See mdb_cursor_get and MDB_GET_MULTIPLE.

func (*Multi) Len 

func (m *Multi) Len() int

Len returns the number of values in the Multi.

func (*Multi) Page 

func (m *Multi) Page() []byte

Page returns the Multi page data as a raw slice of bytes with length m.Size().

func (*Multi) Size 

func (m *Multi) Size() int

Size returns the total size of the Multi data and is equal to 

m.Len()*m.Stride()

func (*Multi) Stride 

func (m *Multi) Stride() int

Stride returns the length of an individual value in the m.

func (*Multi) Val 

func (m *Multi) Val(i int) []byte

Val returns the value at index i. Val panics if i is out of range.

func (*Multi) Vals 

func (m *Multi) Vals() [][]byte

Vals returns a slice containing the values in m. The returned slice has length m.Len() and each item has length m.Stride().

type OpError 

type OpError struct {
	Op    string
	Errno error
}

OpError is an error returned by the C API. Not all errors returned by lmdb-go but typically they are. The Errno field type will either be Errno or syscall.Errno.

func (*OpError) Error 

func (err *OpError) Error() string

Error implements the error interface.

type Stat 

type Stat struct {
	PSize         uint   // Size of a database page. This is currently the same for all databases.
	Depth         uint   // Depth (height) of the B-tree
	BranchPages   uint64 // Number of internal (non-leaf) pages
	LeafPages     uint64 // Number of leaf pages
	OverflowPages uint64 // Number of overflow pages
	Entries       uint64 // Number of data items
}

Stat contains database status information.

See MDB_stat.

type Txn 

type Txn struct {
	// If RawRead is true []byte values retrieved from Get() calls on the Txn
	// and its cursors will point directly into the memory-mapped structure.
	// Such slices will be readonly and must only be referenced wthin the
	// transaction's lifetime.
	RawRead bool
	// contains filtered or unexported fields
}

Txn is a database transaction in an environment.

WARNING: A writable Txn is not threadsafe and may only be used in the goroutine that created it.

See MDB_txn.

Example

This example shows the basic operations used when creating and working with Txn types. 

package main

import (
	"fmt"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

func main() {
	// open a database.
	var dbi lmdb.DBI
	err := env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("exampledb", lmdb.Create)
		// the transaction will be commited if the database was successfully
		// opened/created.
		return err
	})
	if err != nil {
		// ...
	}

	err = env.Update(func(txn *lmdb.Txn) (err error) {
		return txn.Put(dbi, []byte("k"), []byte("v"), 0)
	})
	if err != nil {
		// ...
	}

	err = env.View(func(txn *lmdb.Txn) (err error) {
		v, err := txn.Get(dbi, []byte("k"))
		if err != nil {
			return err
		}
		fmt.Println(string(v))
		return nil
	})
	if err != nil {
		// ...
	}
}

Output:

func (*Txn) Abort 

func (txn *Txn) Abort()

Abort discards pending writes in the transaction and clears the finalizer on txn. A Txn cannot be used again after Abort is called.

See mdb_txn_abort.

func (*Txn) Commit 

func (txn *Txn) Commit() error

Commit persists all transaction operations to the database and clears the finalizer on txn. A Txn cannot be used again after Commit is called.

See mdb_txn_commit.

func (*Txn) CreateDBI 

func (txn *Txn) CreateDBI(name string) (DBI, error)

CreateDBI is a shorthand for OpenDBI that passed the flag lmdb.Create.

func (*Txn) Del 

func (txn *Txn) Del(dbi DBI, key, val []byte) error

Del deletes an item from database dbi. Del ignores val unless dbi has the DupSort flag.

See mdb_del.

func (*Txn) Drop 

func (txn *Txn) Drop(dbi DBI, del bool) error

Drop empties the database if del is false. Drop deletes and closes the database if del is true.

See mdb_drop.

func (*Txn) Flags 

func (txn *Txn) Flags(dbi DBI) (uint, error)

Flags returns the database flags for handle dbi.

func (*Txn) Get 

func (txn *Txn) Get(dbi DBI, key []byte) ([]byte, error)

Get retrieves items from database dbi. If txn.RawRead is true the slice returned by Get references a readonly section of memory that must not be accessed after txn has terminated.

See mdb_get.

Example

This example shows how to properly handle data retrieved from the database and applies to Txn.Get() as well as Cursor.Get(). It is important to handle data retreival carefully to make sure the application does not retain pointers to memory pages which may be reclaimed by LMDB after the transaction terminates. Typically an application would define helper functions/methods to conveniently handle data safe retrieval. 

package main

import (
	"encoding/json"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values shouldn't actually be assigned to.  The are used as stand-ins
// for tests which do not act as tests.
var EnvEx *lmdb.Env
var DBIEx lmdb.DBI

func main() {
	// variables to hold data extracted from the database
	var point struct{ X, Y int }
	var str string
	var p1, p2 []byte

	// extract data from an example environment/database.  it is critical for application
	// code to handle errors  but that is omitted here to save space.
	EnvEx.View(func(txn *lmdb.Txn) (err error) {
		// OK
		// A []byte to string conversion will always copy the data
		v, _ := txn.Get(DBIEx, []byte("mykey"))
		str = string(v)

		// OK
		// If []byte is the desired data type then an explicit copy is required
		// for safe access after the transaction returns.
		v, _ = txn.Get(DBIEx, []byte("mykey"))
		p1 = make([]byte, len(v))
		copy(p1, v)

		// OK
		// The data does not need be copied because it is parsed while txn is
		// open.
		v, _ = txn.Get(DBIEx, []byte("mykey"))
		_ = json.Unmarshal(v, &point)

		// BAD
		// Assigning the result directly to p2 leaves its pointer volatile
		// after the transaction completes which can result in unpredictable
		// behavior.
		p2, _ = txn.Get(DBIEx, []byte("mykey"))

		return nil
	})
}

Output:

func (*Txn) OpenCursor 

func (txn *Txn) OpenCursor(dbi DBI) (*Cursor, error)

OpenCursor allocates and initializes a Cursor to database dbi.

See mdb_cursor_open.

func (*Txn) OpenDBI 

func (txn *Txn) OpenDBI(name string, flags uint) (DBI, error)

OpenDBI opens a named database in the environment. An error is returned if name is empty. The DBI returned by OpenDBI can be used in other transactions but not before Txn has terminated.

OpenDBI can only be called after env.SetMaxDBs() has been called to set the maximum number of named databases.

The C API uses null terminated strings for database names. A consequence is that names cannot contain null bytes themselves. OpenDBI does not check for null bytes in the name argument.

See mdb_dbi_open.

Example

If the database being opened is known to exist then no flags need to be passed. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	// DBI handles can be saved after their opening transaction has committed
	// and may be reused as long as the environment remains open.
	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("dbfound", 0)
		return err
	})
	if err != nil {
		panic(err)
	}
}

Output:
Example (Create)

When Create is passed to Txn.OpenDBI() the database will be created if it didn't already exist. An error will be returned if the name is occupied by data written by Txn./Cursor.Put(). 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("dbnew", lmdb.Create)
		return err
	})
	if err != nil {
		panic(err)
	}
}

Output:
Example (DBsFull)

When the number of open named databases in an environment reaches the value specified by Env.SetMaxDBs() attempts to open additional databases will return an error with errno DBsFull. If an application needs to handle this case then the function IsError() can test an error for this condition. 

package main

import (
	"log"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("dbnotexist", 0)
		return err
	})
	log.Print(err) // mdb_dbi_open: MDB_DBS_FULL: Environment maxdbs limit reached
}

Output:
Example (NotFound)

When a non-existent database is opened without the Create flag the errno is NotFound. If an application needs to handle this case the function IsNotFound() will test an error for this condition. 

package main

import (
	"log"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenDBI("dbnotfound", 0)
		return err
	})
	log.Print(err) // mdb_dbi_open: MDB_NOTFOUND: No matching key/data pair found
}

Output:

func (*Txn) OpenRoot 

func (txn *Txn) OpenRoot(flags uint) (DBI, error)

OpenRoot opens the root database. OpenRoot behaves similarly to OpenDBI but does not require env.SetMaxDBs() to be called beforehand. And, OpenRoot can be called without flags in a View transaction.

Example

Txn.OpenRoot does not need to be called with the Create flag. And Txn.OpenRoot, unlike Txn.OpenDBI, will never produce the error DBsFull. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	var dbi lmdb.DBI
	err = env.Update(func(txn *lmdb.Txn) (err error) {
		dbi, err = txn.OpenRoot(0)
		return err
	})
	if err != nil {
		panic(err)
	}
}

Output:
Example (View)

Txn.OpenRoot may also be called without flags inside View transactions before being openend in an Update transaction. 

package main

import (
	"log"

	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values can only be used is code-only examples (no test output).
var env *lmdb.Env

var err error

func main() {
	err = env.View(func(txn *lmdb.Txn) (err error) {
		dbi, err := txn.OpenRoot(0)
		if err != nil {
			return err
		}
		cur, err := txn.OpenCursor(dbi)
		if err != nil {
			return err
		}
		for {
			k, v, err := cur.Get(nil, nil, lmdb.Next)
			if lmdb.IsNotFound(err) {
				return nil
			}
			if err != nil {
				return err
			}

			log.Printf("%s=%s\n", k, v)
			// ...
		}
	})
	if err != nil {
		// ...
	}
}

Output:

func (*Txn) Put 

func (txn *Txn) Put(dbi DBI, key []byte, val []byte, flags uint) error

Put stores an item in database dbi.

See mdb_put.

func (*Txn) PutReserve 

func (txn *Txn) PutReserve(dbi DBI, key []byte, n int, flags uint) ([]byte, error)

PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.

Example

This example demonstrates the use of PutReserve to store a string value in the root database. This may be faster than Put alone for large values because a string to []byte conversion is not required. 

package main

import (
	"github.com/bmatsuo/lmdb-go/lmdb"
)

// These values shouldn't actually be assigned to.  The are used as stand-ins
// for tests which do not act as tests.
var EnvEx *lmdb.Env

func main() {
	EnvEx.Update(func(txn *lmdb.Txn) (err error) {
		dbroot, err := txn.OpenRoot(0)
		if err != nil {
			return err
		}

		valstr := "value"
		p, err := txn.PutReserve(dbroot, []byte("key"), len(valstr), 0)
		if err != nil {
			return err
		}
		copy(p, valstr)

		return nil
	})
}

Output:

func (*Txn) Renew 

func (txn *Txn) Renew() error

Renew reuses a transaction that was previously reset by calling txn.Reset(). Renew panics if txn is managed by Update, View, etc.

See mdb_txn_renew.

func (*Txn) Reset 

func (txn *Txn) Reset()

Reset aborts the transaction clears internal state so the transaction may be reused by calling Renew. If txn is not going to be reused txn.Abort() must be called to release its slot in the lock table and free its memory. Reset panics if txn is managed by Update, View, etc.

See mdb_txn_reset.

func (*Txn) Stat 

func (txn *Txn) Stat(dbi DBI) (*Stat, error)

Stat returns a Stat describing the database dbi.

See mdb_stat.

func (*Txn) Sub 

func (txn *Txn) Sub(fn TxnOp) error

Sub executes fn in a subtransaction. Sub commits the subtransaction iff a nil error is returned by fn and otherwise aborts it. Sub returns any error it encounters.

Sub may only be called on an Update (a Txn created without the Readonly flag). Calling Sub on a View transaction will return an error.

Any call to Abort, Commit, Renew, or Reset on a Txn created by Sub will panic.

type TxnOp 

type TxnOp func(txn *Txn) error

TxnOp is an operation applied to a managed transaction. The Txn passed to a TxnOp is managed and the operation must not call Commit, Abort, Renew, or Reset on it.

IMPORTANT: TxnOps that write to the database (those passed to Update or BeginUpdate) must not use the Txn in another goroutine (passing it directly or otherwise through closure). Doing so has undefined results.

Notes

Bugs

  • MDB_INTEGERKEY and MDB_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.