rdb

package
v0.0.0-...-10f18be Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2025 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultBatchNum = 10

DefaultBatchNum is the default number of Batches to allocate

View Source 
const DefaultBatchSize = 100000

DefaultBatchSize is the default allocation for Batch

View Source 
const Mb = 1 << 20

Mb is megabyte

View Source 
const NumberOfIterators int = 15

NumberOfIterators is empirically found number of iterators which make sense to keep in pool

Variables

View Source 
var (
	// ErrNXVal is a non-existing value error
	ErrNXVal = errors.New("attempt to delete non-existing value")
	// ErrNXKey is a non-existing key error
	ErrNXKey = errors.New("attempt to delete non-existing key")
)

Functions

func ApplyDiff 

func ApplyDiff(diffpath, dbpath string) error

ApplyDiff applies a diff from inputFileName into RDB database at destPath.

func Backup 

func Backup(dbPath, backupPath string) error

Backup creates new backup

func CleanRDBDir 

func CleanRDBDir(rdbDir string) error

CleanRDBDir removes all files from the directory - for instance, to clean up output directory before compilation

func Compile 

func Compile(in io.Reader, serial uint32, destPath string, opts CompilationOptions) (int, error)

Compile data from io.Reader into RDB database at destPath. useHardlinks allows to use hardlinks in Builder mode. Not supported by fbcode filesystem.

func CompileToRDB 

func CompileToRDB(inputFileName, destPath string, o CompilationOptions) (int, error)

CompileToRDB compiles inputFileName into RDB database at destPath. useHardlinks allows to use hardlinks in Builder mode. Not supported by fbcode filesystem.

func CompileToSpecificRDBVersion 

func CompileToSpecificRDBVersion(inputFileName, destPath string, o CompilationOptions) (int, error)

CompileToSpecificRDBVersion compiles inputFileName into RDB database at destPath and useHardlinks allows to use hardlinks in Builder mode. Not supported by fbcode filesystem. useV2KeySyntax specifies whether v2 keys syntax should be used

func DefaultOptions 

func DefaultOptions() *rocksdb.Options

DefaultOptions returns rocksdb Options initialized with DNSROCKS default values, including potential overrides from ENV variables.

func ReadNextChunk 

func ReadNextChunk(data []byte) (chunk []byte, leftover []byte, err error)

ReadNextChunk splits next chunk from data assuming the following format <4 byte length><chunk>[<<4 byte length><chunk>>...]

func Restore 

func Restore(dbPath, backupPath string) error

Restore restores data from backup

Types

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

Batch allows to batch changes (Add/Del) and then atomically execute them

func (*Batch) Add 

func (batch *Batch) Add(key, value []byte)

Add schedules addition of kv pair

func (*Batch) ApplyDiff 

func (batch *Batch) ApplyDiff(d *dbdiff.Entry)

func (*Batch) Del 

func (batch *Batch) Del(key, value []byte)

Del schedules removal of kv pair; key will be removed if this value is the only value stored for this key, otherwise the value will be removed from multivalue

func (*Batch) IsEmpty 

func (batch *Batch) IsEmpty() bool

IsEmpty returns true if the batch is empty

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder is specifically optimized for building a database from scratch. It has a number of optimizations that would not work on an already existing database, so it should not be used if the database already exist (the result will be undefined if so).

func NewBuilder 

func NewBuilder(path string, useHardlinks bool) (*Builder, error)

NewBuilder creates a new instance of Builder

func (*Builder) Execute 

func (b *Builder) Execute() error

Execute builds the database from accumulated dataset

func (*Builder) FreeBuilder 

func (b *Builder) FreeBuilder()

FreeBuilder closes the database

func (*Builder) ScheduleAdd 

func (b *Builder) ScheduleAdd(d dnsdata.MapRecord)

ScheduleAdd schedules addition of a multi-value pair of key and value we split values between NumCPU() buckets, and all values with the same key will belong to the same bucket thanks to hashing. Effectively, each bucket will contain a non-overlapping with other buckets set of sorted keys.

type CompilationOptions 

type CompilationOptions struct {
	NumCPU         int  // Parser and builder parallelism
	UseV2KeySyntax bool // specifies whether v2 keys syntax should be used
	// builder-related settings
	UseBuilder          bool // if we use RDB builder (mem hungry, fastest) or not
	BuilderUseHardlinks bool // if RDB builder can use hardlinks instead of copying sst files
	// batch-related settings
	BatchNumParallel int // When not using builder, how many batches can we backlog while parsing, affects mem consumption
	BatchSize        int // When not using builder, ize of RDB batches
}

CompilationOptions allows us to provide options to RDB compiler

type Context 

type Context struct {
	// contains filtered or unexported fields
}

Context is a structure holding the state between calls to DB

func NewContext 

func NewContext() *Context

NewContext creates a new structure holding state across FindNext calls

func (*Context) Reset 

func (ctx *Context) Reset()

Reset prepares the context for a new look-up cycle. May be called at the start or at the end of the cycle, upon the caller's discretion

type DBI 

type DBI interface {
	Put(writeOptions *rocksdb.WriteOptions, key, value []byte) error
	Get(readOptions *rocksdb.ReadOptions, key []byte) ([]byte, error)
	Delete(writeOptions *rocksdb.WriteOptions, key []byte) error
	NewBatch() *rocksdb.Batch
	GetMulti(readOptions *rocksdb.ReadOptions, keys [][]byte) ([][]byte, []error)
	ExecuteBatch(batch *rocksdb.Batch, writeOptions *rocksdb.WriteOptions) error
	IngestSSTFiles(fileNames []string, useHardlinks bool) error
	Flush() error
	CreateIterator(readOptions *rocksdb.ReadOptions) *rocksdb.Iterator
	CatchWithPrimary() error
	CloseDatabase()
	GetProperty(string) string
	GetOptions() *rocksdb.Options
	CompactRangeAll()
	WaitForCompact(options *rocksdb.WaitForCompactOptions) error
}

DBI is an interface abstracting RocksDB operations. Enables mocks.

type IteratorPool 

type IteratorPool struct {
	// contains filtered or unexported fields
}

IteratorPool allows RDB iterators reuse. Iterator creation is happen to be pretty costly operation

type RDB 

type RDB struct {
	// contains filtered or unexported fields
}

RDB is RocksDB-backed DNS database

func NewRDB 

func NewRDB(path string) (*RDB, error)

NewRDB creates an instance of RDB; path should be an existing path to the directory, the database will be opened or initialized

func NewReader 

func NewReader(path string) (*RDB, error)

NewReader creates a read-only instance of RDB; path should be an existing path to the directory, the database will be opened as secondary

func NewUpdater 

func NewUpdater(dbpath string) (*RDB, error)

NewUpdater opens an existing database for update. It returns an instance of RDB; dbpath should be an existing path to the directory containing a RocksDB database.

func (*RDB) Add 

func (rdb *RDB) Add(key, value []byte) error

Add inserts a multi-value pair of key and value

func (*RDB) ApplyDiff 

func (rdb *RDB) ApplyDiff(r io.Reader, serial uint32) error

func (*RDB) CatchWithPrimary 

func (rdb *RDB) CatchWithPrimary() error

CatchWithPrimary is the best effort catching up with primary database.

func (*RDB) Close 

func (rdb *RDB) Close() error

Close closes the database and frees up resources

func (*RDB) CreateBatch 

func (rdb *RDB) CreateBatch() *Batch

CreateBatch returns an empty batch

func (*RDB) Del 

func (rdb *RDB) Del(key, value []byte) error

Del deletes the key-value pair. If the value to be deleted is the last value for the associated key, it will delete the key as well. Attempts to delete non-existing key or non-existing value will cause an error

func (*RDB) ExecuteBatch 

func (rdb *RDB) ExecuteBatch(batch *Batch) error

ExecuteBatch will apply all operations from the batch. The same batch cannot be applied twice.

func (*RDB) Find 

func (rdb *RDB) Find(key []byte, context *Context) ([]byte, error)

Find returns the first data value for the given key as a byte slice. Find is the same as FindStart followed by FindNext.

func (*RDB) FindClosest 

func (rdb *RDB) FindClosest(key []byte, ctx *Context) ([]byte, []byte, error)

FindClosest is executed outside of context, and given the key, returns KV for either the exact key match, or for the largest key preceding the requested key. For instance, if key {1, 2, 3, 4} is requested, but such a key does not exist - it will return existing key {1, 2, 3, 3}.

func (*RDB) FindFirst 

func (rdb *RDB) FindFirst(keys [][]byte) ([]byte, int, error)

FindFirst is executed outside of any context, and returns the value of the first existing key, as well as key offset. If the key is not found or error happened, the key offset is -1.

For instance: if keys A and C do not exist and key B exists, being asked for {A, B, C} FindFirst will return the first value of key B. The order of keys on the input DOES matter. Will return nil/no error if nothing was found. If key has more than one value - will return the first one anyway.

func (*RDB) ForEach 

func (rdb *RDB) ForEach(key []byte, f func(value []byte) error, ctx *Context) (err error)

ForEach calls a function for each key match. The function takes a byte slice as a value and return an error. if error is not nil, the loop will stop.

func (*RDB) GetStats 

func (rdb *RDB) GetStats() map[string]int64

GetStats reports main memory stats from RocksDB. See https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB for details.

func (*RDB) IsV2KeySyntaxUsed 

func (rdb *RDB) IsV2KeySyntaxUsed() bool

IsV2KeySyntaxUsed returns value indicating whether v2 syntax is used for DB keys

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.