kvdb

README

StreamingFast Key/Value Store

This library contains different implementations for storing blocks and transactions in a key/value database. It is used by StreamingFast.

Usage

db, err := store.New(dsn)
if err != nil {
    return fmt.Errorf("failed setting up db: %w", err)
}

Backends

The following DSNs are provided by this package:

• Google Cloud BigTable: bigkv://project.instance/namespace-prefix?createTables=true This works very well, is fully managed, and scales horizontally with minimal effort.

• TiKV: tikv://pd0,pd1,pd2:2379?prefix=namespace_prefix This is useful for bare metal deployments, is self managed, and scales very well (with the hardware you throw at it)

• Badger: badger:///home/user/dfuse-data/component/my-badger.db This is useful for local development. It is a library (similar to RocksDB and LevelDB), and thus creates a database that cannot be shared.

• NetKV: netkv://localhost:6789?insecure=true This connects to a netkv server (which you can install with go install -v ./store/netkv/server/netkvserver from this repo), which in turn can serve a badger:// database. It allows for simple badger-based backend (single database, no replication, no scaling), but allow decoupling of StreamingFast processes

Beware that the TiKV backend does not support 0-length values. If your application uses 0-length values, use the WithEmptyValue option.

Contributing

Issues and PR in this repo related strictly to the kvdb library.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

License

Apache 2.0

Documentation

Index

• Variables
• func BlockNum(blockID string) uint32
• func BoolToByte(value bool) byte
• func ByteToBool(value []byte) bool
• func FromHexUint16(input string) (uint16, error)
• func FromHexUint64(input string) (uint64, error)
• func FromRevBlockNum64(input string) (uint64, error)
• func HexName(name uint64) string
• func HexRevBlockNum(blockNum uint32) string
• func HexRevBlockNum64(blockNum uint64) string
• func HexUint16(input uint16) string
• func HexUint32(input uint32) string
• func HexUint64(input uint64) string
• func IncreaseBlockIDSuffix(blockID string) string
• func MustProtoMarshal(obj proto.Message) []byte
• func ReversedBlockID(blockID string) string
• func ReversedUint16(input uint16) uint16
• func StringListToBytes(value []string, separator string) []byte
• func Uint64ToBytes(value uint64) []byte

Variables

var B = func(s string) []byte {
	out, err := hex.DecodeString(s)
	if err != nil {
		panic(err)
	}

	return out
}

B is a shortcut for (must) hex.DecodeString

var (
	ErrNotFound = errors.New("not found")
)

var H = hex.EncodeToString

H is a shortcut for hex.EncodeToString

Functions

func BlockNum

func BlockNum(blockID string) uint32

func BoolToByte

func BoolToByte(value bool) byte

func ByteToBool

func ByteToBool(value []byte) bool

func FromHexUint16

func FromHexUint16(input string) (uint16, error)

func FromHexUint64

func FromHexUint64(input string) (uint64, error)

func FromRevBlockNum64

func FromRevBlockNum64(input string) (uint64, error)

func HexName

func HexName(name uint64) string

func HexRevBlockNum

func HexRevBlockNum(blockNum uint32) string

func HexRevBlockNum64

func HexRevBlockNum64(blockNum uint64) string

func HexUint16

func HexUint16(input uint16) string

func HexUint32

func HexUint32(input uint32) string

func HexUint64

func HexUint64(input uint64) string

func IncreaseBlockIDSuffix

func IncreaseBlockIDSuffix(blockID string) string

increaseBlockIDSuffix increases the last bits of the blockID, to make it to the next block in an InfiniteRange search. WARN: if the last characters are: "ffffffff", then it will overflow, and return a *lower* key.

func MustProtoMarshal

func MustProtoMarshal(obj proto.Message) []byte

func ReversedBlockID

func ReversedBlockID(blockID string) string

func ReversedUint16

func ReversedUint16(input uint16) uint16

func StringListToBytes

func StringListToBytes(value []string, separator string) []byte

func Uint64ToBytes

func Uint64ToBytes(value uint64) []byte