lmdbbs

package module
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 8, 2023 License: Apache-2.0, MIT Imports: 16 Imported by: 0

README

!!! WARNING: This package is not stable and may cause random crashes or data corruption in your application if used.

go-bs-lmdb

An LMDB-backed IPFS Blockstore.

License

Dual-licensed: MIT, Apache Software License v2, by way of the Permissive License Stack.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// DefaultInitialMmapSize is the default initial mmap size to be used if the
	// supplied value is zero or invalid. Unless modified, this value is 1GiB.
	DefaultInitialMmapSize = int64(1 << 30) // 1GiB.

	// DefaultMmapGrowthStepFactor is the default mmap growth step factor to be
	// used if the supplied value is zero or invalid. Unless modified, this
	// value is 1.5, which multiplies the mmap size by 1.5 every time we
	// encounter an MDB_MAP_FULL error.
	DefaultMmapGrowthStepFactor = 1.5 // 1.5x the mmap every time.

	// DefaultMmapGrowthStepMax is the default mmap growth maximum step to be
	// used if the supplied value is zero or invalid. Unless modified, this
	// value is 4GiB.
	DefaultMmapGrowthStepMax = int64(4 << 30) // maximum step size is 4GiB at a time.

	// DefaultMaxReaders is the default number of max readers if one is not
	// provided. By default it is 254, not 256, following the note from the LMDB
	// author that indicates that the original default was 126 because it fit
	// exactly into 8KiB along with a couple of mutexes.
	//
	// http://www.lmdb.tech/doc/group__readers.html#gadff1f7b4d4626610a8d616e0c6dbbea4
	DefaultMaxReaders = 254

	// DefaultRetryDelay is the default retry delay for reattempting transactions
	// that errored with MDB_READERS_FULL.
	DefaultRetryDelay = 10 * time.Millisecond

	// RetryJitter is the jitter to apply to the delay interval. Default: 20%.
	RetryJitter = 0.2
)

Functions

This section is empty.

Types

type Blockstore 

type Blockstore struct {
	// contains filtered or unexported fields
}

Blockstore is a wrapper around lmdb database and environment, containing some additional parameters to execute growing of mmaped section.

func Open 

func Open(opts *Options) (*Blockstore, error)

Open initiates lmdb environment, database and returns Blockstore

func (*Blockstore) AllKeysChan 

func (b *Blockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan starts a cursor to return all keys from the underlying MDB store. The cursor could be preempted at any time by a mmap grow operation. When that happens, the cursor yields, and the grow operation resumes it after the mmap expansion is completed.

Consistency is not guaranteed. That is, keys returned are not a snapshot taken when this method is called. The set of returned keys will vary with concurrent writes.

All errors happening while iterating (except for concurrent reader limit hit) are ignored, hence this method is to be avoided whenever possible.

func (*Blockstore) Close 

func (b *Blockstore) Close() error

Close closes the blockstore Note that DBs created with OpenDB should be closed before this function

func (*Blockstore) CloseDB added in v1.0.8 

func (b *Blockstore) CloseDB(db lmdb.DBI)

func (*Blockstore) DeleteBlock 

func (b *Blockstore) DeleteBlock(ctx context.Context, cid cid.Cid) error

DeleteBlock removes the block from the blockstore, given its cid. This is a no-op for cid that is absent in the Blockstore.

func (*Blockstore) DeleteBlocksIf added in v1.2.0 

func (b *Blockstore) DeleteBlocksIf(ctx context.Context, keys [][]byte, predicate func(txn *lmdb.Txn, key []byte) (bool, error)) error

DeleteMany removes blocks from the blockstore with the given cids. This is a no-op for cids that are absent in the Blockstore.

func (*Blockstore) Get 

func (b *Blockstore) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)

Get retrieves a block with the given cid When block is not found, ipld.ErrNotFound is returned Please note that bytes of the whole block are copied, if you want only to read some of the bytes of the block, use View

func (*Blockstore) GetData added in v1.0.8 

func (b *Blockstore) GetData(ctx context.Context, db lmdb.DBI, key []byte) ([]byte, error)

GetData retrieves arbitrary key-value pair from the blockstore 

This is useful to store some meta data in the storage

func (*Blockstore) GetSize 

func (b *Blockstore) GetSize(ctx context.Context, key cid.Cid) (int, error)

GetSize returns size of the block. When block is not found, ipld.ErrNotFound is returned

func (*Blockstore) Has 

func (b *Blockstore) Has(ctx context.Context, cid cid.Cid) (bool, error)

Has checks if the cid is present in the blockstore

func (*Blockstore) HashOnRead 

func (b *Blockstore) HashOnRead(enabled bool)

HashOnRead sets a variable which controls whether cid is checked to match the hash of the block

func (*Blockstore) OpenDB added in v1.0.8 

func (b *Blockstore) OpenDB(name string) (db lmdb.DBI, err error)

OpenDB opens a database in the same environment as the block storage

func (*Blockstore) Put 

func (b *Blockstore) Put(ctx context.Context, block blocks.Block) error

Put adds the block to the blockstore This is a no-op when block is already present in the Blockstore, no overwrite will take place.

func (*Blockstore) PutData added in v1.0.8 

func (b *Blockstore) PutData(ctx context.Context, db lmdb.DBI, key []byte, valueF func(prevVal []byte, exists bool) (newVal []byte, newExists bool, err error)) error

PutData saves arbitrary key-value pair in the blockstore 

	This is useful to store some meta data in the storage
 Any existing value will be overwritten

func (*Blockstore) PutMany 

func (b *Blockstore) PutMany(ctx context.Context, blocks []blocks.Block) error

PutMany adds the blocks to the blockstore This is a no-op for blocks that are already present in the Blockstore, no overwrites will take place.

func (*Blockstore) Stat 

func (b *Blockstore) Stat() (*lmdb.Stat, error)

Stat returns lmdb statistics

func (*Blockstore) View 

func (b *Blockstore) View(ctx context.Context, key cid.Cid, callback func([]byte) error) error

View retrieves bytes of a block with the given cid and calls the callback on it. When block is not found, ipld.ErrNotFound is returned Note that it is not safe to access bytes passed to the callback outside of the callback's call context.

type DataStorage added in v1.0.8 

type DataStorage interface {
	OpenDB(name string) (lmdb.DBI, error)
	GetData(ctx context.Context, db lmdb.DBI, key []byte) ([]byte, error)
	PutData(ctx context.Context, db lmdb.DBI, key []byte, valueF func(prevVal []byte, exists bool) (newVal []byte, newExists bool, err error)) error
	CloseDB(lmdb.DBI)
}

type Options 

type Options struct {
	// Path is the directory where the LMDB blockstore resides. If it doesn't
	// exist, it will be created.
	Path string

	// ReadOnly, if true, opens this blockstore in read-only mode.
	ReadOnly bool

	// NoSync disables flushing system buffers to disk immediately when
	// committing transactions.
	NoSync bool

	// +++ DB sizing fields. +++ //
	// InitialMmapSize is the initial mmap size passed to LMDB when
	// creating/opening the environment.
	InitialMmapSize int64

	// MmapGrowthStepFactor determines the next map size when a write fails. The
	// current size is multiplied by the factor, and rounded up to the next
	// value divisible by os.Getpagesize(), to obtain the new map size, which is
	// applied with mdb_env_set_mapsize.
	MmapGrowthStepFactor float64

	// MmapGrowthStepMax is the maximum step size by which we'll grow the mmap.
	MmapGrowthStepMax int64

	// MaxReaders is the maximum amount of concurrent reader slots that exist
	// in the lock table. By default 254.
	MaxReaders int

	// RetryDelay is a fixed delay to wait before a transaction that errored
	// with MDB_READERS_FULL will be reattempted. Contention due to incorrect
	// sizing of MaxReaders will thus lead to a system slowdown via
	// backpressure, instead of a straight out error.
	// Jittered by RetryJitter (default: +/-20%).
	RetryDelay time.Duration

	// NoLock: Don't do any locking. If concurrent access is anticipated, the caller must manage all concurrency
	//   itself. For proper operation the caller must enforce single-writer semantics, and must ensure that no readers
	//   are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that
	//   no readers may be active at all when a writer begins.
	NoLock bool

	// MaxDBs is the maximum amount of open databases that will be open withing the LMDB environment
	// Useful when environment is accessed directly to open databases directly within it
	MaxDBs int

	// CidToKeyMapper is a function which converts cid to a key in the LMDB
	CidToKeyMapper func(cid.Cid) []byte

	// CidToKeyMapper is a function which converts cid to a key in the LMDB
	// 	Returns nil when key doesn't correspond to []byte
	// 	Function must make a copy of input byte array
	KeyToCidMapper func([]byte) cid.Cid
}

Options for the Blockstore

func (*Options) CidToKey added in v1.0.8 

func (opts *Options) CidToKey(cid cid.Cid) []byte

func (*Options) KeyToCid added in v1.0.8 

func (opts *Options) KeyToCid(key []byte) cid.Cid

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.