lmdbsync

Documentation

Overview

Package lmdbsync provides advanced synchronization for LMDB environments at the cost of performance. The package provides a drop-in replacement for *lmdb.Env that can be used in situations where the database may be resized or where the flag lmdb.NoLock is used.

Bypassing an Env's methods to access the underlying lmdb.Env is safe. The severity of such usage depends such behavior should be strictly avoided as it may produce undefined behavior from the LMDB C library.

Resizing the environment

The Env type synchronizes all calls to Env.SetMapSize so that it may be safely called in the presence of concurrent transactinos after an environment has been opened. All running transactions complete before the method is called on the underlying lmdb.Env.

However, appplications are recommended against attempting to change the memory map size for an open database. It requires careful synchronization by all processes accessing the database file. And, a large memory map will not affect disk usage on operating systems that support sparse files (e.g. Linux, not OS X).

See mdb_env_set_mapsize.

Multi-processing

The Env type intercepts any MapResized error returned from a transaction and transparently handles it, retrying the transaction after the new size has been adopted. All synchronization is handled so running transactions complete before SetMapSize is called on the underlying lmdb.Env.

See mdb_txn_begin and MDB_MAP_RESIZED.

NoLock

The lmdb.NoLock flag performs all transaction synchronization with Go structures and is an experimental feature. It is unclear what benefits this provides.

See mdb_env_open and MDB_NOLOCK.

MapFull

The Env type does no special handling of the MapFull error. If a call to Txn.Put() or Cursor.Put() returns lmdb.MapFull it is the application's prerogative to detect the error, call Env.SetMapSize, and retry the transaction as necessary.

See mdb_env_set_mapsize and MDB_MAP_FULL.

Index

• Variables
• type Bag
  • func Background() Bag
  • func BagWith(b Bag, key, value interface{}) Bag
• type Env
  • func BagEnv(b Bag) *Env
  • func NewEnv(env *lmdb.Env, h ...Handler) (*Env, error)
  • func (r *Env) Open(path string, flags uint, mode os.FileMode) error
  • func (r *Env) RunTxn(flags uint, op lmdb.TxnOp) (err error)
  • func (r *Env) SetFlags(flags uint) error
  • func (r *Env) SetMapSize(size int64) error
  • func (r *Env) UnsetFlags(flags uint) error
  • func (r *Env) Update(op lmdb.TxnOp) error
  • func (r *Env) UpdateLocked(op lmdb.TxnOp) error
  • func (r *Env) View(op lmdb.TxnOp) error
  • func (r *Env) WithHandler(h Handler) TxnRunner
• type Handler
  • func MapFullHandler(fn MapFullFunc) Handler
  • func MapResizedHandler(maxRetry int, repeatDelay func(retry int) time.Duration) Handler
• type HandlerChain
  • func (c HandlerChain) Append(h Handler) HandlerChain
  • func (c HandlerChain) HandleTxnErr(b Bag, err error) (Bag, error)
• type HandlerFunc
  • func (fn HandlerFunc) HandleTxnErr(c Bag, err error) (Bag, error)
• type MapFullFunc
• type TxnRunner

Variables

var DefaultDelayRepeatResize = time.Millisecond

If a transaction returns MapResize DefaultRetryResize times consequtively an Env will stop attempting to run it and return MapResize to the caller.

var DefaultRetryResize = 2

The default number of times to retry a transaction that is returning repeatedly MapResized. This signifies rapid database growth from another process or some bug/corruption in memory.

If DefaultRetryResize is less than zero the transaction will be retried indefinitely.

var RetryTxn = errors.New("lmdbsync: retry failed txn")

RetryTxn is returned by a Handler to have the Env retry the transaction.

Types

type Bag

type Bag interface {
	Value(key interface{}) interface{}
	// contains filtered or unexported methods
}

func Background

func Background() Bag

func BagWith

func BagWith(b Bag, key, value interface{}) Bag

type Env

type Env struct {
	*lmdb.Env
	Handlers HandlerChain
	// contains filtered or unexported fields
}

Env wraps an *lmdb.Env, receiving all the same methods and proxying some to provide transaction management. Transactions run by an Env handle lmdb.MapResized error transparently through additional synchronization. Additionally, Env is safe to use on environments setting the lmdb.NoLock flag. When in NoLock mode write transactions block all read transactions from running (in addition to blocking other write transactions like a normal lmdb.Env would).

Env proxies several methods to provide synchronization required for safe operation in some scenarios. It is important not byprass proxies and call the methods directly on the underlying lmdb.Env or synchronization may be interfered with. Calling proxied methods directly on the lmdb.Env may result in poor transaction performance or unspecified behavior in from the C library.

func BagEnv

func BagEnv(b Bag) *Env

func NewEnv

func NewEnv(env *lmdb.Env, h ...Handler) (*Env, error)

NewEnv returns an newly allocated Env that wraps env. If env is nil then lmdb.NewEnv() will be called to allocate an lmdb.Env.

func (*Env) Open

func (r *Env) Open(path string, flags uint, mode os.FileMode) error

Open is a proxy for r.Env.Open() that detects the lmdb.NoLock flag to properly manage transaction synchronization.

func (*Env) RunTxn

func (r *Env) RunTxn(flags uint, op lmdb.TxnOp) (err error)

RunTxn is a proxy for r.Env.RunTxn().

If lmdb.NoLock is set on r.Env then RunTxn will block while other updates are in progress, regardless of flags.

If RunTxn returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.

func (*Env) SetFlags

func (r *Env) SetFlags(flags uint) error

SetFlags is a proxy for r.Env.SetFlags() that detects the lmdb.NoLock flag to properly manage transaction synchronization.

func (*Env) SetMapSize

func (r *Env) SetMapSize(size int64) error

SetMapSize is a proxy for r.Env.SetMapSize() that blocks while concurrent transactions are in progress.

func (*Env) UnsetFlags

func (r *Env) UnsetFlags(flags uint) error

UnsetFlags is a proxy for r.Env.UnsetFlags() that detects the lmdb.NoLock flag to properly manage transaction synchronization.

func (*Env) Update

func (r *Env) Update(op lmdb.TxnOp) error

Update is a proxy for r.Env.RunTxn().

If lmdb.NoLock is set on r.Env then Update blocks until all other transactions have terminated and blocks all other transactions from running while in progress (including readonly transactions).

If Update returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.

func (*Env) UpdateLocked

func (r *Env) UpdateLocked(op lmdb.TxnOp) error

UpdateLocked is a proxy for r.Env.RunTxn().

If lmdb.NoLock is set on r.Env then UpdateLocked blocks until all other transactions have terminated and blocks all other transactions from running while in progress (including readonly transactions).

If UpdateLocked returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.

func (*Env) View

func (r *Env) View(op lmdb.TxnOp) error

View is a proxy for r.Env.RunTxn().

If lmdb.NoLock is set on r.Env then View will block until any running update completes.

If View returns MapResized it means another process(es) was writing too fast to the database and the calling process could not get a valid transaction handle.

func (*Env) WithHandler

func (r *Env) WithHandler(h Handler) TxnRunner

WithHandler returns a TxnRunner than handles transaction errors r.Handlers chained with h.

type Handler

type Handler interface {
	HandleTxnErr(c Bag, err error) (Bag, error)
}

func MapFullHandler

func MapFullHandler(fn MapFullFunc) Handler

MapFullHandler returns a Handler that retries Txns that failed due to MapFull errors by increasing the environment map size according to fn.

A lmdb.TxnOp which is handled by the returned Handler will execute multiple times in the occurrance of a MapFull error.

func MapResizedHandler

func MapResizedHandler(maxRetry int, repeatDelay func(retry int) time.Duration) Handler

MapResizedHandler returns a Handler than transparently retrie Txns that failed to start due to MapResized errors.

type HandlerChain

type HandlerChain []Handler

HandlerChain is a Handler implementation that iteratively calls each handler in the underlying slice when handling an error.

func (HandlerChain) Append

func (c HandlerChain) Append(h Handler) HandlerChain

func (HandlerChain) HandleTxnErr

func (c HandlerChain) HandleTxnErr(b Bag, err error) (Bag, error)

type HandlerFunc

type HandlerFunc func(c Bag, err error) (Bag, error)

func (HandlerFunc) HandleTxnErr

func (fn HandlerFunc) HandleTxnErr(c Bag, err error) (Bag, error)

type MapFullFunc

type MapFullFunc func(size int64) (int64, bool)

MapFullFunc is a function for resizing a memory map after it has become full. The function receives the current map size as its argument and returns a new map size. The new size will only be applied if the second return value is true.

type TxnRunner

type TxnRunner interface {
	RunTxn(flags uint, op lmdb.TxnOp) error
	View(op lmdb.TxnOp) error
	Update(op lmdb.TxnOp) error
	UpdateLocked(op lmdb.TxnOp) error
	WithHandler(h Handler) TxnRunner
}

TxnRunner is an interface for types that can run lmdb transactions. TxnRunner is satisfied by Env.