bitalosdb

package module
v1.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 3, 2024 License: Apache-2.0 Imports: 40 Imported by: 1

README

bitalos

Bitalosdb is a high-performance KV storage engine. 中文版

Introduction

  • Bitalosdb is a high-performance KV storage engine (self-developed), which is based on the creative IO architecture and storage technology and focuses on solving the problem of LSM-Tree read and write amplification. As a substitute for Rocksdb, Bitalosdb can improve read and write performance greatly.

Team

Key Technology

  • Bithash (KV separation technology), significantly reduce write amplification. For bithash, time complexity of retrieval is O(1). GC can be completed independently, and value and index are decoupled.

  • Bitalostree (high-performance compression index technology), basically eliminate read amplification. If B+ Tree has a number of huge Pages, write amplification is a severe problem. With a creative index compression technology, Bitalostree eliminates B+ Tree write amplification, and improves the read performance.

  • Bitalostable (cold and hot data separation technology), stores cold data, which is calculated according to the data scale and access frequency. Storage engine writes cold data to Bitalostable when QPS becomes low. Improve data compression, reduce index memory consumption, and achieve more rational resource utilization. (open source stable edition has basic features, enterprise edition supports more comprehensive hot and cold separation).

Performance

  • This benchmark is based on Bitloasdb version (v5.0) and rocksdb stable version (v7.6.0).

Hardware

CPU:    Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
Memory: 384GB
Disk:   2*3.5TB NVMe SSD

Program

  • Benchmark thread number: 8

  • Program cpu cgroup: 8 core

  • Comparison standard: QPS on single-core (multi-core QPS / core number), single-core performance reflects cost advantage better.

Data

  • Key-value spec: key-size=32B、value-size=1KB

  • Comparison dimensions: Total data size(25/50/100GB) * IO ratio(100% random write, 100% random read, 50% random write + 50% random read, 30% random write + 70% random read)

Config

  • rocksdb
Memtable：256MB
WAL：enable
Cache：8GB
TargetFileSize：256M
L0CompactTrigger：8
L0StopWritesTrigger：24
  • bitalosdb
Memtable：256MB
WAL：enable
Cache：disable

Result

benchmark

Document

Technical architecture and documentation, refer to the official website: bitalos.zuoyebang.com

Documentation

Index

Constants

View Source 
const (
	InternalKeyKindDelete       = base.InternalKeyKindDelete
	InternalKeyKindSet          = base.InternalKeyKindSet
	InternalKeyKindLogData      = base.InternalKeyKindLogData
	InternalKeyKindPrefixDelete = base.InternalKeyKindPrefixDelete
	InternalKeyKindMax          = base.InternalKeyKindMax
	InternalKeySeqNumMax        = base.InternalKeySeqNumMax
)

Variables

View Source 
var (
	ErrNotFound = base.ErrNotFound
	ErrClosed   = errors.New("bitalosdb: closed")
)
View Source 
var DefaultComparer = base.DefaultComparer
View Source 
var DefaultLogger = base.DefaultLogger
View Source 
var ErrInvalidBatch = errors.New("bitalosdb: invalid batch")
View Source 
var IterAll = IterOptions{IsAll: true}
View Source 
var NoSync = &WriteOptions{Sync: false}
View Source 
var Sync = &WriteOptions{Sync: true}

Functions

This section is empty.

Types

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

func (*Batch) Close 

func (b *Batch) Close() error

func (*Batch) Commit 

func (b *Batch) Commit(o *WriteOptions) error

func (*Batch) Count 

func (b *Batch) Count() uint32

func (*Batch) Delete 

func (b *Batch) Delete(key []byte, opts *WriteOptions) error

func (*Batch) Empty 

func (b *Batch) Empty() bool

func (*Batch) PrefixDeleteKeySet added in v1.1.0 

func (b *Batch) PrefixDeleteKeySet(key []byte, opts *WriteOptions) error

func (*Batch) Reset 

func (b *Batch) Reset()

func (*Batch) Set 

func (b *Batch) Set(key, value []byte, opts *WriteOptions) error

func (*Batch) SetMultiValue 

func (b *Batch) SetMultiValue(key []byte, values ...[]byte) error

type BatchBitower added in v1.1.0 

type BatchBitower struct {
	// contains filtered or unexported fields
}

func (*BatchBitower) Close added in v1.1.0 

func (b *BatchBitower) Close() error

func (*BatchBitower) Commit added in v1.1.0 

func (b *BatchBitower) Commit(o *WriteOptions) error

func (*BatchBitower) Count added in v1.1.0 

func (b *BatchBitower) Count() uint32

func (*BatchBitower) Delete added in v1.1.0 

func (b *BatchBitower) Delete(key []byte, _ *WriteOptions) error

func (*BatchBitower) DeleteDeferred added in v1.1.0 

func (b *BatchBitower) DeleteDeferred(keyLen int) *DeferredBatchOp

func (*BatchBitower) Empty added in v1.1.0 

func (b *BatchBitower) Empty() bool

func (*BatchBitower) LogData added in v1.1.0 

func (b *BatchBitower) LogData(data []byte, _ *WriteOptions) error

func (*BatchBitower) PrefixDeleteKeySet added in v1.1.0 

func (b *BatchBitower) PrefixDeleteKeySet(key []byte, _ *WriteOptions) error

func (*BatchBitower) Reader added in v1.1.0 

func (b *BatchBitower) Reader() BatchBitowerReader

func (*BatchBitower) Repr added in v1.1.0 

func (b *BatchBitower) Repr() []byte

func (*BatchBitower) Reset added in v1.1.0 

func (b *BatchBitower) Reset()

func (*BatchBitower) SeqNum added in v1.1.0 

func (b *BatchBitower) SeqNum() uint64

func (*BatchBitower) Set added in v1.1.0 

func (b *BatchBitower) Set(key, value []byte, _ *WriteOptions) error

func (*BatchBitower) SetDeferred added in v1.1.0 

func (b *BatchBitower) SetDeferred(keyLen, valueLen int) *DeferredBatchOp

func (*BatchBitower) SetMultiValue added in v1.1.0 

func (b *BatchBitower) SetMultiValue(key []byte, values ...[]byte) error

func (*BatchBitower) SetRepr added in v1.1.0 

func (b *BatchBitower) SetRepr(data []byte) error

type BatchBitowerReader added in v1.1.0 

type BatchBitowerReader []byte

func ReadBatchBitower added in v1.1.0 

func ReadBatchBitower(repr []byte) (r BatchBitowerReader, count uint32)

func (*BatchBitowerReader) Next added in v1.1.0 

func (r *BatchBitowerReader) Next() (kind InternalKeyKind, ukey []byte, value []byte, ok bool)

type BitableOptions 

type BitableOptions = options.BitableOptions

type Bitower added in v1.1.0 

type Bitower struct {
	// contains filtered or unexported fields
}

func (*Bitower) AsyncFlush added in v1.1.0 

func (s *Bitower) AsyncFlush() (<-chan struct{}, error)

func (*Bitower) Close added in v1.1.0 

func (s *Bitower) Close() (err error)

func (*Bitower) Exist added in v1.1.0 

func (s *Bitower) Exist(key []byte) bool

func (*Bitower) Flush added in v1.1.0 

func (s *Bitower) Flush() error

func (*Bitower) Get added in v1.1.0 

func (s *Bitower) Get(key []byte) ([]byte, func(), error)

type CheckpointOption 

type CheckpointOption func(*checkpointOptions)

func WithFlushedWAL 

func WithFlushedWAL() CheckpointOption

type CompactEnv 

type CompactEnv struct {
	StartHour     int
	EndHour       int
	DeletePercent float64
	BitreeMaxSize int64
	Interval      int
}

type Compare 

type Compare = base.Compare

type Comparer 

type Comparer = base.Comparer

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func Open 

func Open(dirname string, opts *Options) (db *DB, err error)

func (*DB) Apply 

func (d *DB) Apply(batch *Batch, opts *WriteOptions) error

func (*DB) ApplyBitower added in v1.1.0 

func (d *DB) ApplyBitower(batch *BatchBitower, opts *WriteOptions) error

func (*DB) AsyncFlush 

func (d *DB) AsyncFlush() (<-chan struct{}, error)

func (*DB) CacheInfo 

func (d *DB) CacheInfo() string

func (*DB) CheckAndCompact 

func (d *DB) CheckAndCompact(jobId int)

func (*DB) CheckKeySize 

func (d *DB) CheckKeySize(key []byte) error

func (*DB) Checkpoint 

func (d *DB) Checkpoint(destDir string, opts ...CheckpointOption) (ckErr error)

func (*DB) Close 

func (d *DB) Close() (err error)

func (*DB) CompactBitree 

func (d *DB) CompactBitree(jobId int)

func (*DB) DebugInfo 

func (d *DB) DebugInfo() string

func (*DB) Delete 

func (d *DB) Delete(key []byte, opts *WriteOptions) error

func (*DB) Exist 

func (d *DB) Exist(key []byte) (bool, error)

func (*DB) Flush 

func (d *DB) Flush() error

func (*DB) Get 

func (d *DB) Get(key []byte) ([]byte, func(), error)

func (*DB) GetAutoCompact added in v1.1.0 

func (d *DB) GetAutoCompact() bool

func (*DB) Id 

func (d *DB) Id() int

func (*DB) IsClosed 

func (d *DB) IsClosed() bool

func (*DB) LockTask 

func (d *DB) LockTask()

func (*DB) LogData 

func (d *DB) LogData(data []byte, index int, opts *WriteOptions) error

func (*DB) MetricsInfo added in v1.1.0 

func (d *DB) MetricsInfo() MetricsInfo

func (*DB) NewBatch 

func (d *DB) NewBatch() *Batch

func (*DB) NewBatchBitower added in v1.1.0 

func (d *DB) NewBatchBitower() *BatchBitower

func (*DB) NewIter 

func (d *DB) NewIter(o *IterOptions) *Iterator

func (*DB) PrefixDeleteKeySet added in v1.1.0 

func (d *DB) PrefixDeleteKeySet(key []byte, opts *WriteOptions) error

func (*DB) Set 

func (d *DB) Set(key, value []byte, opts *WriteOptions) error

func (*DB) SetAutoCompact added in v1.1.0 

func (d *DB) SetAutoCompact(newVal bool)

func (*DB) SetCheckpointHighPriority 

func (d *DB) SetCheckpointHighPriority(v bool)

func (*DB) SetCheckpointLock 

func (d *DB) SetCheckpointLock(lock bool)

func (*DB) UnlockTask 

func (d *DB) UnlockTask()

type DeferredBatchOp 

type DeferredBatchOp struct {
	Key, Value []byte
	// contains filtered or unexported fields
}

func (DeferredBatchOp) Finish 

func (d DeferredBatchOp) Finish() error

type DiskSlowInfo 

type DiskSlowInfo struct {
	Path     string
	Duration time.Duration
}

func (DiskSlowInfo) SafeFormat 

func (i DiskSlowInfo) SafeFormat(w redact.SafePrinter, _ rune)

func (DiskSlowInfo) String 

func (i DiskSlowInfo) String() string

type Equal 

type Equal = base.Equal

type EventListener 

type EventListener struct {
	BackgroundError func(error)
	DiskSlow        func(DiskSlowInfo)
	FlushBegin      func(FlushInfo)
	FlushEnd        func(FlushInfo)
	WALCreated      func(WALCreateInfo)
	WALDeleted      func(WALDeleteInfo)
	WriteStallBegin func(WriteStallBeginInfo)
	WriteStallEnd   func()
}

func MakeLoggingEventListener 

func MakeLoggingEventListener(logger Logger) EventListener

func TeeEventListener 

func TeeEventListener(a, b EventListener) EventListener

func (*EventListener) EnsureDefaults 

func (l *EventListener) EnsureDefaults(logger Logger)

type FileNum 

type FileNum = base.FileNum

type FlushInfo 

type FlushInfo struct {
	Index    int
	Reason   string
	Input    int
	Iterated uint64
	Written  int64

	Duration time.Duration
	Done     bool
	Err      error
	// contains filtered or unexported fields
}

func (FlushInfo) SafeFormat 

func (i FlushInfo) SafeFormat(w redact.SafePrinter, _ rune)

func (FlushInfo) String 

func (i FlushInfo) String() string

type InternalKey 

type InternalKey = base.InternalKey

type InternalKeyKind 

type InternalKeyKind = base.InternalKeyKind

type IterOptions 

type IterOptions = options.IterOptions

type IterValidityState 

type IterValidityState int8
const (
	IterExhausted IterValidityState = iota
	IterValid
)

type Iterator 

type Iterator struct {
	// contains filtered or unexported fields
}

func (*Iterator) Close 

func (i *Iterator) Close() error

func (*Iterator) Error 

func (i *Iterator) Error() error

func (*Iterator) First 

func (i *Iterator) First() bool

func (*Iterator) Key 

func (i *Iterator) Key() []byte

func (*Iterator) Last 

func (i *Iterator) Last() bool

func (*Iterator) Metrics 

func (i *Iterator) Metrics() IteratorMetrics

func (*Iterator) Next 

func (i *Iterator) Next() bool

func (*Iterator) NextWithLimit 

func (i *Iterator) NextWithLimit() IterValidityState

func (*Iterator) Prev 

func (i *Iterator) Prev() bool

func (*Iterator) PrevWithLimit 

func (i *Iterator) PrevWithLimit() IterValidityState

func (*Iterator) ResetStats 

func (i *Iterator) ResetStats()

func (*Iterator) SeekGE 

func (i *Iterator) SeekGE(key []byte) bool

func (*Iterator) SeekGEWithLimit 

func (i *Iterator) SeekGEWithLimit(key []byte) IterValidityState

func (*Iterator) SeekLT 

func (i *Iterator) SeekLT(key []byte) bool

func (*Iterator) SeekLTWithLimit 

func (i *Iterator) SeekLTWithLimit(key []byte) IterValidityState

func (*Iterator) SeekPrefixGE 

func (i *Iterator) SeekPrefixGE(key []byte) bool

func (*Iterator) SetBounds 

func (i *Iterator) SetBounds(lower, upper []byte)

func (*Iterator) Stats 

func (i *Iterator) Stats() IteratorStats

func (*Iterator) Valid 

func (i *Iterator) Valid() bool

func (*Iterator) Value 

func (i *Iterator) Value() []byte

type IteratorMetrics 

type IteratorMetrics struct {
	ReadAmp int
}

type IteratorStats 

type IteratorStats struct {
	ForwardSeekCount [NumStatsKind]int
	ReverseSeekCount [NumStatsKind]int
	ForwardStepCount [NumStatsKind]int
	ReverseStepCount [NumStatsKind]int
}

func (*IteratorStats) SafeFormat 

func (stats *IteratorStats) SafeFormat(s redact.SafePrinter, verb rune)

func (*IteratorStats) String 

func (stats *IteratorStats) String() string

type IteratorStatsKind 

type IteratorStatsKind int8
const (
	InterfaceCall IteratorStatsKind = iota
	InternalIterCall
	NumStatsKind
)

type Logger 

type Logger = base.Logger

type MetricsInfo added in v1.1.0 

type MetricsInfo struct {
	FlushMemTime       int64 `json:"-"`
	BithashFileTotal   int   `json:"bithash_file_total"`
	BithashKeyTotal    int   `json:"bithash_key_total"`
	BithashDelKeyTotal int   `json:"bithash_del_key_total"`
}

func (MetricsInfo) String added in v1.1.0 

func (s MetricsInfo) String() string

type Options 

type Options struct {
	BytesPerSync                int
	Comparer                    *Comparer
	DisableWAL                  bool
	EventListener               EventListener
	MemTableSize                int
	MemTableStopWritesThreshold int
	WALBytesPerSync             int
	WALDir                      string
	WALMinSyncInterval          func() time.Duration
	FS                          vfs.FS
	Logger                      Logger
	Id                          int
	Verbose                     bool
	LogTag                      string
	DataType                    string
	CompressionType             int
	DeleteFileInternal          int
	UseBithash                  bool
	UseBitable                  bool
	BitableOpts                 *options.BitableOptions
	AutoCompact                 bool
	CompactInfo                 CompactEnv
	CacheSize                   int64
	CacheType                   int
	CacheShards                 int
	CacheHashSize               int
	UseMapIndex                 bool
	UsePrefixCompress           bool
	UseBlockCompress            bool
	BlockCacheSize              int64
	FlushReporter               func(int)
	KeyHashFunc                 func([]byte) int
	KvCheckExpireFunc           func(int, []byte, []byte) bool
	KvTimestampFunc             func([]byte, uint8) (bool, uint64)
	IOWriteLoadThresholdFunc    func() bool
	KeyPrefixDeleteFunc         func([]byte) uint64
	// contains filtered or unexported fields
}

func (*Options) Clone 

func (o *Options) Clone() *Options

func (*Options) EnsureDefaults 

func (o *Options) EnsureDefaults() *Options

type Reader 

type Reader interface {
	Get(key []byte) (value []byte, closer func(), err error)
	NewIter(o *IterOptions) *Iterator
	Close() error
}

type Split 

type Split = base.Split

type WALCreateInfo 

type WALCreateInfo struct {
	Index           int
	Path            string
	FileNum         FileNum
	RecycledFileNum FileNum
	Err             error
}

func (WALCreateInfo) SafeFormat 

func (i WALCreateInfo) SafeFormat(w redact.SafePrinter, _ rune)

func (WALCreateInfo) String 

func (i WALCreateInfo) String() string

type WALDeleteInfo 

type WALDeleteInfo struct {
	Index   int
	Path    string
	FileNum FileNum
	Err     error
}

func (WALDeleteInfo) SafeFormat 

func (i WALDeleteInfo) SafeFormat(w redact.SafePrinter, _ rune)

func (WALDeleteInfo) String 

func (i WALDeleteInfo) String() string

type WriteOptions 

type WriteOptions struct {
	Sync bool
}

func (*WriteOptions) GetSync 

func (o *WriteOptions) GetSync() bool

type WriteStallBeginInfo 

type WriteStallBeginInfo struct {
	Index  int
	Reason string
}

func (WriteStallBeginInfo) SafeFormat 

func (i WriteStallBeginInfo) SafeFormat(w redact.SafePrinter, _ rune)

func (WriteStallBeginInfo) String 

func (i WriteStallBeginInfo) String() string

type Writer 

type Writer interface {
	Apply(batch *Batch, o *WriteOptions) error
	ApplyBitower(batch *BatchBitower, o *WriteOptions) error
	Delete(key []byte, o *WriteOptions) error
	LogData(data []byte, index int, opts *WriteOptions) error
	Set(key, value []byte, o *WriteOptions) error
	PrefixDeleteKeySet(key []byte, o *WriteOptions) error
}

Source Files

View all Source files

Directories

Path Synopsis
bdb
internal
arenaskl
base
batchskl
bindex
bitask
bytepools
cache
cache/lfucache
cache/lfucache/internal/arenaskl
cache/lfucache/internal/base
cache/lrucache
compress
consts
crc
errorfs
fastrand
hash
humanize
invariants
list2
manifest
manual
mmap
options
rate
Package rate provides a rate limiter.
 Package rate provides a rate limiter.
rawalloc
record
sortedkv
statemachine
unsafe2
utils
vfs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.