database

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 20, 2023 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefaultBlockCacheSize is 0 MB.
	DefaultBlockCacheSize = 0

	// DefaultIndexCacheSize is 2 GB.
	DefaultIndexCacheSize = 2000 << 20

	// TinyIndexCacheSize is 10 MB.
	TinyIndexCacheSize = 10 << 20

	// DefaultMaxTableSize is 256 MB. The larger
	// this value is, the larger database transactions
	// storage can handle (~15% of the max table size
	// == max commit size).
	DefaultMaxTableSize = 256 << 20

	// DefaultLogValueSize is 64 MB.
	DefaultLogValueSize = 64 << 20

	// PerformanceMaxTableSize is 3072 MB. The larger
	// this value is, the larger database transactions
	// storage can handle (~15% of the max table size
	// == max commit size).
	PerformanceMaxTableSize = 3072 << 20

	// PerformanceLogValueSize is 256 MB.
	PerformanceLogValueSize = 256 << 20

	// AllInMemoryTableSize is 2048 MB.
	AllInMemoryTableSize = 2048 << 20

	// PerformanceLogValueSize is 512 MB.
	AllInMemoryLogValueSize = 512 << 20

	// DefaultCompressionMode is the default block
	// compression setting.
	DefaultCompressionMode = options.None
)

Variables

This section is empty.

Functions

func AllInMemoryBadgerOptions 

func AllInMemoryBadgerOptions(dir string) badger.Options

AllInMemoryBadgerOptions are performance geared BadgerDB options that use much more RAM than the default settings and PerformanceBadger settings

func BadgerTrain 

func BadgerTrain(
	ctx context.Context,
	namespace string,
	db string,
	output string,
	maxEntries int,
	compressorEntries []*encoder.CompressorEntry,
) (float64, float64, error)

BadgerTrain creates a zstd dictionary for a given BadgerDatabase DB namespace. Optionally, you can specify the maximum number of entries to load into storage (if -1 is provided, then all possible are loaded).

func DefaultBadgerOptions 

func DefaultBadgerOptions(dir string) badger.Options

DefaultBadgerOptions are the default options used to initialized a new BadgerDB. These settings override many of the default BadgerDB settings to restrict memory usage to ~6 GB. If constraining memory usage is not desired for your use case, you can provide your own BadgerDB settings with the configuration option WithCustomSettings.

There are many threads about optimizing memory usage in Badger (which can grow to many GBs if left untuned). Our own research indicates that each MB increase in MaxTableSize and/or ValueLogFileSize corresponds to a 10 MB increase in RAM usage (all other settings equal). Our primary concern is large database transaction size, so we configure MaxTableSize to be 4 times the size of ValueLogFileSize (if we skewed any further to MaxTableSize, we would quickly hit the default open file limit on many OSes).

func PerformanceBadgerOptions 

func PerformanceBadgerOptions(dir string) badger.Options

PerformanceBadgerOptions are performance geared BadgerDB options that use much more RAM than the default settings.

Types

type BadgerDatabase 

type BadgerDatabase struct {
	// contains filtered or unexported fields
}

BadgerDatabase is a wrapper around Badger DB that implements the Database interface.

func (*BadgerDatabase) Close 

func (b *BadgerDatabase) Close(ctx context.Context) error

Close closes the database to prevent corruption. The caller should defer this in main.

func (*BadgerDatabase) Encoder 

func (b *BadgerDatabase) Encoder() *encoder.Encoder

Encoder returns the BadgerDatabase encoder.

func (*BadgerDatabase) GetMetaData 

func (b *BadgerDatabase) GetMetaData() string

GetInfo returns customized metaData for db's metaData

func (*BadgerDatabase) ReadTransaction 

func (b *BadgerDatabase) ReadTransaction(
	ctx context.Context,
) Transaction

ReadTransaction creates a new read BadgerTransaction.

func (*BadgerDatabase) Transaction 

func (b *BadgerDatabase) Transaction(
	ctx context.Context,
) Transaction

Transaction creates a new exclusive write BadgerTransaction.

func (*BadgerDatabase) WriteTransaction 

func (b *BadgerDatabase) WriteTransaction(
	ctx context.Context,
	identifier string,
	priority bool,
) Transaction

WriteTransaction creates a new write BadgerTransaction for a particular identifier.

type BadgerOption 

type BadgerOption func(b *BadgerDatabase)

BadgerOption is used to overwrite default values in BadgerDatabase construction. Any Option not provided falls back to the default value.

func WithCompressorEntries 

func WithCompressorEntries(entries []*encoder.CompressorEntry) BadgerOption

WithCompressorEntries provides zstd dictionaries for given namespaces.

func WithCustomSettings 

func WithCustomSettings(settings badger.Options) BadgerOption

WithCustomSettings allows for overriding all default BadgerDB options with custom settings.

func WithIndexCacheSize 

func WithIndexCacheSize(size int64) BadgerOption

WithIndexCacheSize override the DefaultIndexCacheSize setting for the BadgerDB. The size here is in bytes. If you provide custom BadgerDB settings, do not use this config as it will be overridden by your custom settings.

func WithMetaData 

func WithMetaData(metaData string) BadgerOption

add a info map to BadgerDatabase

func WithTableSize 

func WithTableSize(size int64) BadgerOption

WithTableSize override the MaxTableSize setting for the BadgerDB. The size here is in GB.

func WithValueLogFileSize 

func WithValueLogFileSize(size int64) BadgerOption

WithTableSize override the ValueLogFileSize setting for the BadgerDB. The size here is in MB.

func WithWriterShards 

func WithWriterShards(shards int) BadgerOption

WithWriterShards overrides the default shards used in the writer utils.MutexMap. It is recommended to set this value to your write concurrency to prevent lock contention.

func WithoutCompression 

func WithoutCompression() BadgerOption

WithoutCompression disables zstd compression.

type BadgerTransaction 

type BadgerTransaction struct {
	// contains filtered or unexported fields
}

BadgerTransaction is a wrapper around a Badger DB transaction that implements the DatabaseTransaction interface.

func (*BadgerTransaction) Commit 

func (b *BadgerTransaction) Commit(context.Context) error

Commit attempts to commit and discard the transaction.

func (*BadgerTransaction) Delete 

func (b *BadgerTransaction) Delete(ctx context.Context, key []byte) error

Delete removes the key and its value within the transaction.

func (*BadgerTransaction) Discard 

func (b *BadgerTransaction) Discard(context.Context)

Discard discards an open transaction. All transactions must be either discarded or committed.

func (*BadgerTransaction) Get 

func (b *BadgerTransaction) Get(
	ctx context.Context,
	key []byte,
) (bool, []byte, error)

Get accesses the value of the key within a transaction. It is up to the caller to reclaim any memory returned.

func (*BadgerTransaction) Scan 

func (b *BadgerTransaction) Scan(
	ctx context.Context,
	prefix []byte,
	seekStart []byte,
	worker func([]byte, []byte) error,
	logEntries bool,
	reverse bool,
) (int, error)

Scan calls a worker for each item in a scan instead of reading all items into memory.

func (*BadgerTransaction) Set 

func (b *BadgerTransaction) Set(
	ctx context.Context,
	key []byte,
	value []byte,
	reclaimValue bool,
) error

Set changes the value of the key to the value within a transaction.

type CommitWorker 

type CommitWorker func(context.Context) error

CommitWorker is returned by a module to be called after changes have been committed. It is common to put logging activities in here (that shouldn't be printed until the block is committed).

type Database 

type Database interface {
	// Transaction acquires an exclusive write lock on the database.
	// This ensures all other calls to Transaction and WriteTransaction
	// will block until the returned DatabaseTransaction is committed or
	// discarded. This is useful for making changes across
	// multiple prefixes but incurs a large performance overhead.
	Transaction(context.Context) Transaction

	// ReadTransaction allows for consistent, read-only access
	// to the database. This does not acquire any lock
	// on the database.
	ReadTransaction(context.Context) Transaction

	// WriteTransaction acquires a granular write lock for a particular
	// identifier. All subsequent calls to WriteTransaction with the same
	// identifier will block until the DatabaseTransaction returned is either
	// committed or discarded.
	WriteTransaction(ctx context.Context, identifier string, priority bool) Transaction

	// Close shuts down the database.
	Close(context.Context) error

	// Encoder returns the *Encoder used to store/read data
	// in the database. This *Encoder often performs some
	// form of compression on data.
	Encoder() *encoder.Encoder

	// GetMetaData returns customized metaData for db's metaData
	GetMetaData() string
}

Database is an interface that provides transactional access to a KV store.

func NewBadgerDatabase 

func NewBadgerDatabase(
	ctx context.Context,
	dir string,
	storageOptions ...BadgerOption,
) (Database, error)

NewBadgerDatabase creates a new BadgerDatabase.

type Transaction 

type Transaction interface {
	Set(context.Context, []byte, []byte, bool) error
	Get(context.Context, []byte) (bool, []byte, error)
	Delete(context.Context, []byte) error

	Scan(
		context.Context,
		[]byte,
		[]byte,
		func([]byte, []byte) error,
		bool,
		bool,
	) (int, error)

	Commit(context.Context) error
	Discard(context.Context)
}

Transaction is an interface that provides access to a KV store within some transaction context provided by a Database.

When a Transaction is committed or discarded, all memory utilized is reclaimed. If you want to persist any data retrieved, make sure to make a copy!

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.