lotusdb

package module
v2.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 18, 2024 License: Apache-2.0 Imports: 26 Imported by: 1

README

Snipaste_2023-09-15_21-45-26.png

What is LotusDB

LotusDB is the most advanced key-value store written in Go, extremely fast, compatible with LSM tree and B+ tree, and optimization of badger and bbolt.

Key features:

  • Combine the advantages of LSM and B+ tree
  • Fast read/write performance
  • Much lower read and space amplification than typical LSM

Design Overview

Getting Started

package main

import (
	"github.com/lotusdblabs/lotusdb/v2"
)

func main() {
	// specify the options
	options := lotusdb.DefaultOptions
	options.DirPath = "/tmp/lotusdb_basic"

	// open a database
	db, err := lotusdb.Open(options)
	if err != nil {
		panic(err)
	}
	defer func() {
		_ = db.Close()
	}()

	// put a key
	err = db.Put([]byte("name"), []byte("lotusdb"), nil)
	if err != nil {
		panic(err)
	}

	// get a key
	val, err := db.Get([]byte("name"))
	if err != nil {
		panic(err)
	}
	println(string(val))

	// delete a key
	err = db.Delete([]byte("name"), nil)
	if err != nil {
		panic(err)
	}
}

see the examples for more details.

Community

Welcome to join the Slack channel and Discussions to connect with LotusDB team members and other users.

If you are a Chinese user, you are also welcome to join our WeChat group, scan the QR code and you will be invited:

Documentation

Index

Constants

View Source 
const (
	BptreeItr iterType = iota
	MemItr
)
View Source 
const (
	B  = 1
	KB = 1024 * B
	MB = 1024 * KB
	GB = 1024 * MB
)

Variables

View Source 
var (
	ErrKeyIsEmpty                    = errors.New("the key is empty")
	ErrKeyNotFound                   = errors.New("key not found in database")
	ErrDatabaseIsUsing               = errors.New("the database directory is used by another process")
	ErrReadOnlyBatch                 = errors.New("the batch is read only")
	ErrBatchCommitted                = errors.New("the batch is committed")
	ErrDBClosed                      = errors.New("the database is closed")
	ErrDBDirectoryISEmpty            = errors.New("the database directory path can not be empty")
	ErrWaitMemtableSpaceTimeOut      = errors.New("wait memtable space timeout, try again later")
	ErrDBIteratorUnsupportedTypeHASH = errors.New("hash index does not support iterator")
)
View Source 
var DefaultBatchOptions = BatchOptions{
	WriteOptions: DefaultWriteOptions,
	ReadOnly:     false,
}
View Source 
var DefaultOptions = Options{
	DirPath: tempDBDir(),

	MemtableSize: 64 * MB,

	MemtableNums: 15,
	BlockCache:   0,
	Sync:         false,
	BytesPerSync: 0,

	PartitionNum:     3,
	KeyHashFunction:  xxhash.Sum64,
	ValueLogFileSize: 1 * GB,
	IndexType:        BTree,

	CompactBatchCount: 10000,

	WaitMemSpaceTimeout: 100 * time.Millisecond,
}
View Source 
var DefaultWriteOptions = WriteOptions{
	Sync:       false,
	DisableWal: false,
}

Functions

func MatchKeyFunc 

func MatchKeyFunc(db *DB, key []byte, keyPos **KeyPosition, value *[]byte) func(slot diskhash.Slot) (bool, error)

MatchKeyFunc Set nil if do not need keyPos or value.

Types

type BPTree 

type BPTree struct {
	// contains filtered or unexported fields
}

BPTree is the BoltDB index implementation.

func (*BPTree) Close 

func (bt *BPTree) Close() error

Close releases all boltdb database resources. It will block waiting for any open transactions to finish before closing the database and returning.

func (*BPTree) DeleteBatch 

func (bt *BPTree) DeleteBatch(keys [][]byte, _ ...diskhash.MatchKeyFunc) error

DeleteBatch deletes the specified keys from the index.

func (*BPTree) Get 

func (bt *BPTree) Get(key []byte, _ ...diskhash.MatchKeyFunc) (*KeyPosition, error)

Get gets the position of the specified key.

func (*BPTree) PutBatch 

func (bt *BPTree) PutBatch(positions []*KeyPosition, _ ...diskhash.MatchKeyFunc) error

PutBatch puts the specified key positions into the index.

func (*BPTree) Sync 

func (bt *BPTree) Sync() error

Sync executes fdatasync() against the database file handle.

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

Batch is a batch operations of the database. If readonly is true, you can only get data from the batch by Get method. An error will be returned if you try to use Put or Delete method.

If readonly is false, you can use Put and Delete method to write data to the batch. The data will be written to the database when you call Commit method.

Batch is not a transaction, it does not guarantee isolation. But it can guarantee atomicity, consistency and durability(if the Sync options is true).

You must call Commit method to commit the batch, otherwise the DB will be locked.

func (*Batch) Commit 

func (b *Batch) Commit() error

Commit commits the batch, if the batch is readonly or empty, it will return directly.

It will iterate the pendingWrites and write the data to the database, then write a record to indicate the end of the batch to guarantee atomicity. Finally, it will write the index.

func (*Batch) Delete 

func (b *Batch) Delete(key []byte) error

Delete marks a key for deletion in the batch.

func (*Batch) Exist 

func (b *Batch) Exist(key []byte) (bool, error)

Exist checks if the key exists in the database.

func (*Batch) Get 

func (b *Batch) Get(key []byte) ([]byte, error)

Get retrieves the value associated with a given key from the batch.

func (*Batch) Put 

func (b *Batch) Put(key []byte, value []byte) error

Put adds a key-value pair to the batch for writing.

type BatchOptions 

type BatchOptions struct {
	// WriteOptions used in batch operation
	WriteOptions

	// ReadOnly specifies whether the batch is read only.
	ReadOnly bool
}

BatchOptions specifies the options for creating a batch.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB is the main structure of the LotusDB database. It contains all the information needed to operate the database.

DB is thread-safe, and can be used by multiple goroutines. But you can not open multiple DBs with the same directory path at the same time. ErrDatabaseIsUsing will be returned if you do so.

LotusDB is the most advanced key-value database written in Go. It combines the advantages of LSM tree and B+ tree, read and write are both very fast. It is also very memory efficient, and can store billions of key-value pairs in a single machine.

func Open 

func Open(options Options) (*DB, error)

Open a database with the specified options. If the database directory does not exist, it will be created automatically.

Multiple processes can not use the same database directory at the same time, otherwise it will return ErrDatabaseIsUsing.

It will first open the wal to rebuild the memtable, then open the index and value log. Return the DB object if succeeded, otherwise return the error.

func (*DB) Close 

func (db *DB) Close() error

Close the database, close all data files and release file lock. Set the closed flag to true. The DB instance cannot be used after closing.

func (*DB) Compact 

func (db *DB) Compact() error

Compact will iterate all values in vlog, and write the valid values to a new vlog file. Then replace the old vlog file with the new one, and delete the old one.

func (*DB) Delete 

func (db *DB) Delete(key []byte) error

Delete delete with defaultWriteOptions.

func (*DB) DeleteWithOptions 

func (db *DB) DeleteWithOptions(key []byte, options WriteOptions) error

DeleteWithOptions the specified key from the database. Actually, it will open a new batch and commit it. You can think the batch has only one Delete operation.

func (*DB) Exist 

func (db *DB) Exist(key []byte) (bool, error)

Exist checks if the specified key exists in the database. Actually, it will open a new batch and commit it. You can think the batch has only one Exist operation.

func (*DB) Get 

func (db *DB) Get(key []byte) ([]byte, error)

Get the value of the specified key from the database. Actually, it will open a new batch and commit it. You can think the batch has only one Get operation.

func (*DB) NewBatch 

func (db *DB) NewBatch(options BatchOptions) *Batch

NewBatch creates a new Batch instance.

func (*DB) NewIterator 

func (db *DB) NewIterator(options IteratorOptions) (*Iterator, error)

NewIterator returns a new iterator. The iterator will iterate all the keys in DB. It's the caller's responsibility to call Close when iterator is no longer used, otherwise resources will be leaked. The iterator is not goroutine-safe, you should not use the same iterator concurrently from multiple goroutines.

func (*DB) Put 

func (db *DB) Put(key []byte, value []byte) error

Put put with defaultWriteOptions.

func (*DB) PutWithOptions 

func (db *DB) PutWithOptions(key []byte, value []byte, options WriteOptions) error

PutWithOptions a key-value pair into the database. Actually, it will open a new batch and commit it. You can think the batch has only one Put operation.

func (*DB) Sync 

func (db *DB) Sync() error

Sync all data files to the underlying storage.

type HashTable 

type HashTable struct {
	// contains filtered or unexported fields
}

HashTable is the diskhash index implementation. see: https://github.com/rosedblabs/diskhash

func (*HashTable) Close 

func (ht *HashTable) Close() error

Close close index.

func (*HashTable) DeleteBatch 

func (ht *HashTable) DeleteBatch(keys [][]byte, matchKeyFunc ...diskhash.MatchKeyFunc) error

DeleteBatch delete batch records from index.

func (*HashTable) Get 

func (ht *HashTable) Get(key []byte, matchKeyFunc ...diskhash.MatchKeyFunc) (*KeyPosition, error)

Get chunk position by key.

func (*HashTable) PutBatch 

func (ht *HashTable) PutBatch(positions []*KeyPosition, matchKeyFunc ...diskhash.MatchKeyFunc) error

PutBatch put batch records to index.

func (*HashTable) Sync 

func (ht *HashTable) Sync() error

Sync sync index data to disk.

type Index 

type Index interface {
	// PutBatch put batch records to index
	PutBatch(keyPositions []*KeyPosition, matchKeyFunc ...diskhash.MatchKeyFunc) error

	// Get chunk position by key
	Get(key []byte, matchKeyFunc ...diskhash.MatchKeyFunc) (*KeyPosition, error)

	// DeleteBatch delete batch records from index
	DeleteBatch(keys [][]byte, matchKeyFunc ...diskhash.MatchKeyFunc) error

	// Sync sync index data to disk
	Sync() error

	// Close index
	Close() error
}

Index is the interface for index implementations. An index is a key-value store that maps keys to chunk positions. The index is used to find the chunk position of a key.

Currently, the only implementation is a BoltDB index. But you can implement your own index if you want.

type IndexType 

type IndexType int8
const (
	// BTree is the BoltDB index type.
	BTree IndexType = iota
	// Hash is the diskhash index type.
	// see: https://github.com/rosedblabs/diskhash
	Hash
)

type Iterator 

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator holds a heap and a set of iterators that implement the baseIterator interface.

func (*Iterator) Close 

func (mi *Iterator) Close() error

Close the iterator.

func (*Iterator) Key 

func (mi *Iterator) Key() []byte

Key get the current key.

func (*Iterator) Next 

func (mi *Iterator) Next()

Next moves the iterator to the next key.

func (*Iterator) Rewind 

func (mi *Iterator) Rewind()

Rewind seek the first key in the iterator.

func (*Iterator) Seek 

func (mi *Iterator) Seek(key []byte)

Seek move the iterator to the key which is greater(less when reverse is true) than or equal to the specified key.

func (*Iterator) Valid 

func (mi *Iterator) Valid() bool

Valid returns whether the iterator is exhausted.

func (*Iterator) Value 

func (mi *Iterator) Value() []byte

Value get the current value.

type IteratorOptions 

type IteratorOptions struct {
	// Prefix filters the keys by prefix.
	Prefix []byte

	// Reverse indicates whether the iterator is reversed.
	// false is forward, true is backward.
	Reverse bool
}

IteratorOptions is the options for the iterator.

type KeyPosition 

type KeyPosition struct {
	// contains filtered or unexported fields
}

KeyPosition is the position of the key in the value log.

type LogRecord 

type LogRecord struct {
	Key     []byte
	Value   []byte
	Type    LogRecordType
	BatchID uint64
}

LogRecord is the log record of the key/value pair. It contains the key, the value, the record type and the batch id It will be encoded to byte slice and written to the wal.

type LogRecordType 

type LogRecordType = byte

LogRecordType is the type of the log record. 

const (
	// LogRecordNormal is the normal log record type.
	LogRecordNormal LogRecordType = iota
	// LogRecordDeleted is the deleted log record type.
	LogRecordDeleted
	// LogRecordBatchFinished is the batch finished log record type.
	LogRecordBatchFinished
)

type Options 

type Options struct {
	// DirPath specifies the directory path where all the database files will be stored.
	DirPath string

	// MemtableSize represents the maximum size in bytes for a memtable.
	// It means that each memtable will occupy so much memory.
	// Default value is 64MB.
	MemtableSize uint32

	// MemtableNums represents maximum number of memtables to keep in memory before flushing.
	// Default value is 15.
	MemtableNums int

	// BlockCache specifies the size of the block cache in number of bytes.
	// A block cache is used to store recently accessed data blocks, improving read performance.
	// If BlockCache is set to 0, no block cache will be used.
	BlockCache uint32

	// Sync is whether to synchronize writes through os buffer cache and down onto the actual disk.
	// Setting sync is required for durability of a single write operation, but also results in slower writes.
	//
	// If false, and the machine crashes, then some recent writes may be lost.
	// Note that if it is just the process that crashes (machine does not) then no writes will be lost.
	//
	// In other words, Sync being false has the same semantics as a write
	// system call. Sync being true means write followed by fsync.
	Sync bool

	// BytesPerSync specifies the number of bytes to write before calling fsync.
	BytesPerSync uint32

	// PartitionNum specifies the number of partitions to use for the index and value log.
	PartitionNum int

	// KeyHashFunction specifies the hash function for sharding.
	// It is used to determine which partition a key belongs to.
	// Default value is xxhash.
	KeyHashFunction func([]byte) uint64

	// ValueLogFileSize size of a single value log file.
	// Default value is 1GB.
	ValueLogFileSize int64

	// indexType.
	// default value is bptree.
	IndexType IndexType

	// writing entries to disk after reading the specified number of entries.
	CompactBatchCount int

	// WaitMemSpaceTimeout specifies the timeout for waiting for space in the memtable.
	// When all memtables are full, it will be flushed to disk by the background goroutine.
	// But if the flush speed is slower than the write speed, there may be no space in the memtable.
	// So the write operation will wait for space in the memtable, and the timeout is specified by WaitMemSpaceTimeout.
	// If the timeout is exceeded, the write operation will fail, you can try again later.
	// Default value is 100ms.
	WaitMemSpaceTimeout time.Duration
}

type ValueLogRecord 

type ValueLogRecord struct {
	// contains filtered or unexported fields
}

ValueLogRecord is the record of the key/value pair in the value log.

type WriteOptions 

type WriteOptions struct {

	// Default value is false.
	Sync bool

	// DisableWal if true, writes will not first go to the write ahead log, and the write may get lost after a crash.
	// Setting true only if don`t care about the data loss.
	// Default value is false.
	DisableWal bool
}

WriteOptions set optional params for PutWithOptions and DeleteWithOptions. If you use Put and Delete (without options), that means to use the default values.

Source Files

View all Source files

Directories

Path Synopsis
examples
basic
batch
iterator

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.