lmdbsync

Documentation

Overview

Package lmdbsync provides advanced synchronization for LMDB environments at the cost of performance. The package provides a drop-in replacement for *lmdb.Env that can be used in situations where the database may be resized or where the flag lmdb.NoLock is used.

Bypassing an Env's methods to access the underlying lmdb.Env is not safe. The severity of such usage depends such behavior should be strictly avoided as it may produce undefined behavior from the LMDB C library.

Resizing the environment

The Env type synchronizes all calls to Env.SetMapSize so that it may, with some caveats, be safely called in the presence of concurrent transactions after an environment has been opened. All running transactions must complete before the method will be called on the underlying lmdb.Env.

If an open transaction depends on a call to Env.SetMapSize then the Env will deadlock and block all future transactions. When using a Handler to automatically call Env.SetMapSize this implies the restriction that transactions must terminate independently of the creation/termination of other transactions to avoid deadlock.

In the simplest example, a view transaction that attempts an update on the underlying Env will deadlock the environment if the map is full and a Handler attempts to resize the map so the update may be retried.

env.View(func(txn *lmdb.Txn) (err error) {
	v, err := txn.Get(db, key)
	if err != nil {
		return err
	}
	err = env.Update(func(txn *lmdb.Txn) (err error) { // deadlock on lmdb.MapFull!
		txn.Put(dbi, key, append(v, b...))
	})
	return err
}

The update should instead be prepared inside the view and then executed following its termination. This removes the implicit dependence of the view on calls to Env.SetMapSize().

var v []byte
env.View(func(txn *lmdb.Txn) (err error) {
	// RawRead isn't used because the value will be used outside the
	// transaction.
	v, err = txn.Get(db, key)
	if err != nil {
		return err
	}
	return nil
}
if err != nil {
	// ...
}
err = env.Update(func(txn *lmdb.Txn) (err error) { // no deadlock, even if env is resized!
	txn.Put(dbi, key, append(v, b...))
})

The developers of LMDB officially recommend against applications changing the memory map size for an open database. It requires careful synchronization by all processes accessing the database file. And, a large memory map will not affect disk usage on operating systems that support sparse files (e.g. Linux, not OS X).

See mdb_env_set_mapsize.

MapFull

The MapFullHandler function configures an Env to automatically call increase the map size with Env.SetMapSize and retry transactions when a lmdb.MapFull error prevents an update from being committed.

Because updates may need to execute multiple times in the presence of lmdb.MapFull it is important to make sure their TxnOp functions are idempotent and do not cause unwanted additive change to the program state.

See mdb_txn_commit and MDB_MAP_FULL.

MapResized

When multiple processes access and resize an environment it is not uncommon to encounter a MapResized error which prevents the TxnOp from being executed and requires a synchronized call to Env.SetMapSize before continuing normal operation.

The MapResizedHandler function configures an Env to automatically adopt a new map size when a lmdb.MapResized error is encountered and retry execution of the TxnOp.

See mdb_txn_begin and MDB_MAP_RESIZED.

NoLock

When the lmdb.NoLock flag is set on an environment Env handles all transaction synchronization using Go structures and is an experimental feature. It is unclear what benefits this provides.

Usage of lmdb.NoLock requires that update transactions acquire an exclusive lock on the environment. In such cases it is required that view transactions execute independently of update transactions, a requirement more strict than that from handling MapFull.

See mdb_env_open and MDB_NOLOCK.

Index

• Variables
• type DelayFunc
  • func ExponentialBackoff(base time.Duration, max time.Duration, factor float64) DelayFunc
• type Env
  • func NewEnv(env *lmdb.Env, h ...Handler) (*Env, error)
  • func (r *Env) BeginTxn(parent *lmdb.Txn, flags uint) (*lmdb.Txn, error)
  • func (r *Env) Open(path string, flags uint, mode os.FileMode) error
  • func (r *Env) RunTxn(flags uint, op lmdb.TxnOp) (err error)
  • func (r *Env) SetMapSize(size int64) error
  • func (r *Env) Update(op lmdb.TxnOp) error
  • func (r *Env) UpdateLocked(op lmdb.TxnOp) error
  • func (r *Env) View(op lmdb.TxnOp) error
  • func (r *Env) WithHandler(h Handler) TxnRunner
• type Handler
  • func MapFullHandler(fn MapFullFunc) Handler
  • func MapResizedHandler(maxRetry int, delay DelayFunc) Handler
• type HandlerChain
  • func (c HandlerChain) Append(h ...Handler) HandlerChain
  • func (c HandlerChain) HandleTxnErr(ctx context.Context, env *Env, err error) (context.Context, error)
• type MapFullFunc
• type TxnRunner

Variables

var ErrTxnRetry = errors.New("lmdbsync: retry failed txn")

ErrTxnRetry is returned by a Handler to have the Env retry the transaction.

var MapResizedDefaultDelay = ExponentialBackoff(time.Millisecond, 5*time.Millisecond, 2)

MapResizedDefaultDelay is the default DelayFunc when MapResizedHandler is passed a nil value.

var MapResizedDefaultRetry = 2

MapResizedDefaultRetry is the default number of attempts MapResizedHandler will make adopt a new map size when lmdb.MapResized is encountered repeatedly.

Types

type DelayFunc

type DelayFunc func(attempt int) time.Duration

DelayFunc takes as input the number of previous attempts and returns the delay before making another attempt.

func ExponentialBackoff

func ExponentialBackoff(base time.Duration, max time.Duration, factor float64) DelayFunc

ExponentialBackoff returns a function that delays each attempt by random number between 0 and the minimum of max and base*factor^attempt.

type Env

type Env struct {
	*lmdb.Env
	Handlers HandlerChain
	// contains filtered or unexported fields
}

Env wraps an *lmdb.Env, receiving all the same methods and proxying some to provide transaction management. Transactions run by an Env handle lmdb.MapResized error transparently through additional synchronization. Additionally, Env is safe to use on environments setting the lmdb.NoLock flag. When in NoLock mode write transactions block all read transactions from running (in addition to blocking other write transactions like a normal lmdb.Env would).

Env proxies several methods to provide synchronization required for safe operation in some scenarios. It is important not byprass proxies and call the methods directly on the underlying lmdb.Env or synchronization may be interfered with. Calling proxied methods directly on the lmdb.Env may result in poor transaction performance or unspecified behavior in from the C library.

func NewEnv

func NewEnv(env *lmdb.Env, h ...Handler) (*Env, error)

NewEnv returns an newly allocated Env that wraps env. If env is nil then lmdb.NewEnv() will be called to allocate an lmdb.Env.

func (*Env) BeginTxn

func (r *Env) BeginTxn(parent *lmdb.Txn, flags uint) (*lmdb.Txn, error)

BeginTxn overrides the r.Env.BeginTxn and always returns an error. An unmanaged transaction.

func (*Env) Open

func (r *Env) Open(path string, flags uint, mode os.FileMode) error

Open is a proxy for r.Env.Open() that detects the lmdb.NoLock flag to properly manage transaction synchronization.

func (*Env) RunTxn

func (r *Env) RunTxn(flags uint, op lmdb.TxnOp) (err error)

RunTxn is a proxy for r.Env.RunTxn().

If lmdb.NoLock is set on r.Env then RunTxn will block while other updates are in progress, regardless of flags.

func (*Env) SetMapSize

func (r *Env) SetMapSize(size int64) error

SetMapSize is a proxy for r.Env.SetMapSize() that blocks while concurrent transactions are in progress.

func (*Env) Update

func (r *Env) Update(op lmdb.TxnOp) error

Update is a proxy for r.Env.Update().

If lmdb.NoLock is set on r.Env then Update blocks until all other transactions have terminated and blocks all other transactions from running while in progress (including readonly transactions).

func (*Env) UpdateLocked

func (r *Env) UpdateLocked(op lmdb.TxnOp) error

UpdateLocked is a proxy for r.Env.UpdateLocked().

If lmdb.NoLock is set on r.Env then UpdateLocked blocks until all other transactions have terminated and blocks all other transactions from running while in progress (including readonly transactions).

func (*Env) View

func (r *Env) View(op lmdb.TxnOp) error

View is a proxy for r.Env.View().

If lmdb.NoLock is set on r.Env then View will block until any running update completes.

func (*Env) WithHandler

func (r *Env) WithHandler(h Handler) TxnRunner

WithHandler returns a TxnRunner than handles transaction errors r.Handlers chained with h.

type Handler

type Handler interface {
	HandleTxnErr(ctx context.Context, env *Env, err error) (context.Context, error)
}

Handler can intercept errors returned by a transaction and handle them in an application-specific way, including by resizing the environment and retrying the transaction by returning ErrTxnRetry.

func MapFullHandler

func MapFullHandler(fn MapFullFunc) Handler

MapFullHandler returns a Handler that retries updates which failed to commit due to lmdb.MapFull errors. When lmdb.MapFull is encountered fn is used to set a new new map size before opening a new transaction and executing the lmdb.TxnOp again.

When using MapFullHandler it is important that updates are idempotent. An Env.Update that encounters lmdb.MapFull may execute its lmdb.TxnOp function multiple times before successfully committing it (or aborting).

Open view transactions must not wait for updates to complete when using MapFullHandler or the environment will deadlock.

func MapResizedHandler

func MapResizedHandler(maxRetry int, delay DelayFunc) Handler

MapResizedHandler returns a Handler that transparently adopts map sizes set by external processes and retries any transactions that failed to start because of lmdb.MapResized.

If the database is growing too rapidly and maxRetry consecutive transactions fail due to lmdb.MapResized then the Handler returned by MapResizedHandler gives up and returns the lmdb.MapResized error to the caller. Delay will be called before each call to Env.SetMapSize to insert an optional delay.

Open transactions must not directly create new (non-child) transactions when using MapResizedHandler or the environment will deadlock.

type HandlerChain

type HandlerChain []Handler

HandlerChain is a Handler implementation that iteratively calls each handler in the underlying slice when handling an error.

func (HandlerChain) Append

func (c HandlerChain) Append(h ...Handler) HandlerChain

Append returns a new HandlerChain that will evaluate h in sequence after the Handlers already in C are evaluated. Append does not modify the storage undelying c.

func (HandlerChain) HandleTxnErr

func (c HandlerChain) HandleTxnErr(ctx context.Context, env *Env, err error) (context.Context, error)

HandleTxnErr implements the Handler interface. Each handler in c processes the context.Context and error returned by the previous handler.

type MapFullFunc

type MapFullFunc func(size int64) (int64, bool)

MapFullFunc is a function for resizing a memory map after it has become full. The function receives the current map size as its argument and returns a new map size. The new size will only be applied if the second return value is true.

type TxnRunner

type TxnRunner interface {
	RunTxn(flags uint, op lmdb.TxnOp) error
	View(op lmdb.TxnOp) error
	Update(op lmdb.TxnOp) error
	UpdateLocked(op lmdb.TxnOp) error
	WithHandler(h Handler) TxnRunner
}

TxnRunner is an interface for types that can run lmdb transactions. TxnRunner is satisfied by Env.