raftpebbledb

Documentation

Index

• Variables
• func FirstError(err1 error, err2 error) error
• type LogDBCallback
• type Option
  • func WithConfig(config RaftLogRocksDBConfig) Option
  • func WithDbDirPath(dbDir string) Option
  • func WithFS(fs vfs.FS) Option
  • func WithLogDBCallback(cb LogDBCallback) Option
  • func WithLogger(logger pebble.Logger) Option
  • func WithPebbleOptions(opts *pebble.Options) Option
  • func WithWalDirPath(walDir string) Option
• type PebbleKVStore
  • func New(options ...Option) (*PebbleKVStore, error)
  • func (s *PebbleKVStore) Close() error
  • func (s *PebbleKVStore) DeleteRange(min, max uint64) (err error)
  • func (s *PebbleKVStore) FirstIndex() (first uint64, err error)
  • func (s *PebbleKVStore) Get(key []byte) (value []byte, err error)
  • func (s *PebbleKVStore) GetLog(index uint64, log *raft.Log) (err error)
  • func (s *PebbleKVStore) GetUint64(key []byte) (term uint64, err error)
  • func (s *PebbleKVStore) GetValue(key []byte, op func([]byte) error) (err error)
  • func (s *PebbleKVStore) LastIndex() (last uint64, err error)
  • func (s *PebbleKVStore) Set(key []byte, val []byte) (err error)
  • func (s *PebbleKVStore) SetUint64(key []byte, val uint64) error
  • func (s *PebbleKVStore) StoreLog(log *raft.Log) (err error)
  • func (s *PebbleKVStore) StoreLogs(logs []*raft.Log) (err error)
• type RaftLogRocksDBConfig
  • func GetDefaultRaftLogRocksDBConfig() RaftLogRocksDBConfig
  • func GetLargeMemRaftLogRocksDBConfig() RaftLogRocksDBConfig
  • func GetMediumMemRaftLogRocksDBConfig() RaftLogRocksDBConfig
  • func GetSmallMemRaftLogRocksDBConfig() RaftLogRocksDBConfig
  • func GetTinyMemRaftLogRocksDBConfig() RaftLogRocksDBConfig
  • func (cfg *RaftLogRocksDBConfig) IsEmpty() bool
  • func (cfg *RaftLogRocksDBConfig) MemorySizeMB() uint64

Variables

var (

	// ErrKeyNotFound is an error indicating a given key does not exist
	// for hashicorp raft vote meta stable get check, if err != nil && err.Error() != "not found"
	ErrKeyNotFound = errors.New("not found")
)

Functions

func FirstError

func FirstError(err1 error, err2 error) error

FirstError returns the first error.

Types

type LogDBCallback

type LogDBCallback func(busy bool)

LogDBCallback is a callback function called by the LogDB eg: do some metrics export

type Option

type Option interface {
	// contains filtered or unexported methods
}

func WithConfig

func WithConfig(config RaftLogRocksDBConfig) Option

func WithDbDirPath

func WithDbDirPath(dbDir string) Option

func WithFS

func WithFS(fs vfs.FS) Option

func WithLogDBCallback

func WithLogDBCallback(cb LogDBCallback) Option

func WithLogger

func WithLogger(logger pebble.Logger) Option

func WithPebbleOptions

func WithPebbleOptions(opts *pebble.Options) Option

func WithWalDirPath

func WithWalDirPath(walDir string) Option

type PebbleKVStore

type PebbleKVStore struct {
	// contains filtered or unexported fields
}

KV is a pebble based LogStore StableStore type.

func New

func New(options ...Option) (*PebbleKVStore, error)

New uses the supplied config to open the Pebble db and prepare it for using as a raft backend pebble kv store. level no compression for raft meta/log store

func (*PebbleKVStore) Close

func (s *PebbleKVStore) Close() error

Close the Raft log

func (*PebbleKVStore) DeleteRange

func (s *PebbleKVStore) DeleteRange(min, max uint64) (err error)

DeleteRange deletes logs within a given range inclusively.

func (*PebbleKVStore) FirstIndex

func (s *PebbleKVStore) FirstIndex() (first uint64, err error)

FirstIndex returns the first known index from the Raft log. use SeekPrefixGE,Reverse iteration (Prev) is not supported when an iterator is in prefix iteration mode. https://pkg.go.dev/github.com/cockroachdb/pebble#Iterator.SeekPrefixGE so use lowerBound,UpperBound for iter prefixLog notice: if not found return 0, nil

func (*PebbleKVStore) Get

func (s *PebbleKVStore) Get(key []byte) (value []byte, err error)

Get is used to retrieve a value from the k/v store by key notice: if key/val not found return ErrKeyNotFound

func (*PebbleKVStore) GetLog

func (s *PebbleKVStore) GetLog(index uint64, log *raft.Log) (err error)

GetLog gets a log entry from Pebble at a given index. notice: if index log not found return raft ErrLogNotFound

func (*PebbleKVStore) GetUint64

func (s *PebbleKVStore) GetUint64(key []byte) (term uint64, err error)

GetUint64 is like Get, but return uint64 values

func (*PebbleKVStore) GetValue

func (s *PebbleKVStore) GetValue(key []byte, op func([]byte) error) (err error)

GetValue defer closer.Close to do op safely when modify get value

func (*PebbleKVStore) LastIndex

func (s *PebbleKVStore) LastIndex() (last uint64, err error)

LastIndex returns the last known index from the Raft log. use SeekPrefixGE,Reverse iteration (Prev) is not supported when an iterator is in prefix iteration mode. https://pkg.go.dev/github.com/cockroachdb/pebble#Iterator.SeekPrefixGE so use lowerBound,UpperBound for iter prefixLog notice: if not found return 0, nil

func (*PebbleKVStore) Set

func (s *PebbleKVStore) Set(key []byte, val []byte) (err error)

Set is used to set a key/value set outside of the raft log.

func (*PebbleKVStore) SetUint64

func (s *PebbleKVStore) SetUint64(key []byte, val uint64) error

SetUint64 is like Set, but handles uint64 values

func (*PebbleKVStore) StoreLog

func (s *PebbleKVStore) StoreLog(log *raft.Log) (err error)

StoreLog stores a single raft log.

func (*PebbleKVStore) StoreLogs

func (s *PebbleKVStore) StoreLogs(logs []*raft.Log) (err error)

StoreLogs stores a set of raft logs.

type RaftLogRocksDBConfig

type RaftLogRocksDBConfig struct {
	Shards                             uint64
	KVKeepLogFileNum                   uint64
	KVMaxBackgroundCompactions         uint64
	KVMaxBackgroundFlushes             uint64
	KVLRUCacheSize                     uint64
	KVWriteBufferSize                  uint64
	KVMaxWriteBufferNumber             uint64
	KVLevel0FileNumCompactionTrigger   uint64
	KVLevel0SlowdownWritesTrigger      uint64
	KVLevel0StopWritesTrigger          uint64
	KVMaxBytesForLevelBase             uint64
	KVMaxBytesForLevelMultiplier       uint64
	KVTargetFileSizeBase               uint64
	KVTargetFileSizeMultiplier         uint64
	KVLevelCompactionDynamicLevelBytes uint64
	KVRecycleLogFileNum                uint64
	KVNumOfLevels                      uint64
	KVBlockSize                        uint64
	SaveBufferSize                     uint64
	MaxSaveBufferSize                  uint64
}

RaftLogRocksDBConfig pebble add sst block lru cache for read others more detail see rocksdb guid wiki

func GetDefaultRaftLogRocksDBConfig

func GetDefaultRaftLogRocksDBConfig() RaftLogRocksDBConfig

GetDefaultRaftLogRocksDBConfig returns the default configurations for the LogDB storage engine. The default LogDB configuration use up to 8GBytes memory.

func GetLargeMemRaftLogRocksDBConfig

func GetLargeMemRaftLogRocksDBConfig() RaftLogRocksDBConfig

GetLargeMemRaftLogRocksDBConfig returns a LogDB config aimed to keep memory size to be large for good I/O performance. It is the default setting used by the system. When using the returned config, LogDB takes up to 8GBytes memory.

func GetMediumMemRaftLogRocksDBConfig

func GetMediumMemRaftLogRocksDBConfig() RaftLogRocksDBConfig

GetMediumMemRaftLogRocksDBConfig returns a LogDB config aimed to keep memory size at medium level. When using the returned config, LogDB takes up to 4GBytes memory.

func GetSmallMemRaftLogRocksDBConfig

func GetSmallMemRaftLogRocksDBConfig() RaftLogRocksDBConfig

GetSmallMemRaftLogRocksDBConfig returns a LogDB config aimed to keep memory size at low level. When using the returned config, LogDB takes up to 1GBytes memory.

func GetTinyMemRaftLogRocksDBConfig

func GetTinyMemRaftLogRocksDBConfig() RaftLogRocksDBConfig

GetTinyMemRaftLogRocksDBConfig returns a LogDB config aimed for minimizing memory size. When using the returned config, LogDB takes up to 256MBytes memory.

func (*RaftLogRocksDBConfig) IsEmpty

func (cfg *RaftLogRocksDBConfig) IsEmpty() bool

IsEmpty returns a boolean value indicating whether the RaftLogRocksDBConfig instance is empty.

func (*RaftLogRocksDBConfig) MemorySizeMB

func (cfg *RaftLogRocksDBConfig) MemorySizeMB() uint64

MemorySizeMB returns the estimated upper bound memory size used by the LogDB storage engine. The returned value is in MBytes.