database

README

General concept

Package database represents a database layer in Dusk Network. Current design should embody both backend-agnostic and storage-agnostic concepts.

Terminology

Storage: Underlying storage can be any kind of KV stores (BoltDB, LMDB, LevelDB, RocksDB, Redis), non-KV stores (SQLite) or even a custom ACID-complient flat-file store.

Backend/Driver: represents a DB implementation that uses one or multiple underlying storages to provide block chain database specifics with regard to particular purpose. It must implement Driver, DB and Tx intefaces.

Interfaces exposed to upper layers:

• database.Driver: DB connection initiator. Inspired by database/sql/driver
• database.DB: A thin layer on top of database.Tx providing a manageable Tx execution. Inspired by BoltDB
• database.Tx: A transaction layer tightly coupled with DUSK block chain specifics to fully benefit from underlying storage capabilities

Available Drivers

• /database/heavy driver is designed to provide efficient, robust and persistent DUSK block chain DB on top of syndtr/goleveldb/leveldb store (unofficial LevelDB porting). It must be Mainnet-complient.

Testing Drivers

• /database/testing implements a boilerplate method to verify if a registered driver does satisfy minimum database requirements. The package defines a set of unit tests that are executed only on registered drivers. It can serve also as a detailed and working database guideline.

Code example:

More code examples can be found in /database/heavy/database_test.go

// By changing Driver name, one can switch between different backends
// Each backend is bound to one or multiple underlying stores
readonly := false

// Retrieve
driver, _ := database.From(lite.DriverName)
db, err := driver.Open(path, protocol.DevNet, readonly)

if err != nil {
	...
}

defer db.Close()

blocks := make([]*block.Block, 100)
// Populate blocks here ...

// a managed read-write DB Tx to store all blocks via atomic update
err = db.Update(func(tx database.Tx) error {
	for _, block := range blocks {
		err := tx.StoreBlock(block)
		if err != nil {
			return err
		}
	}
	return nil
})

if err != nil {
	fmt.Printf("Transaction failed. No blocks stored")
	return
}

 // a managed read-only DB Tx to check if all blocks have been stored
_ = db.View(func(tx database.Tx) error {
	for _, block := range blocks {
		exists, err := tx.FetchBlockExists(block.Header.Hash)
		if err != nil {
			fmt.Printf(err.Error())
			return nil
		}

		if !exists {
			fmt.Printf("Block with Height %d was not found", block.Header.Height)
			return nil
		}
	}
	return nil
})

Additional features

Additional features that can be provided by a Driver:

• Read-only transactions/connections
• Validated storage connection
• Profiling: Collect transactions and connections statistics
• Traces: Log transactions data
• Alarming: Trigger an event in case of critical backend or storage failure
• Iterators/Cursors
• Safe Concurrency model: One read-write transaction, many read-only transactions
• Redundancy: blockchain data stored in a secondary in-memory or on-disk structure

Documentation

Index

• Variables
• func Drivers() []string
• func Register(driver Driver) error
• type DB
• type Driver
  • func From(name string) (Driver, error)
• type State
• type Transaction

Variables

var (

	// ErrKeyImageNotFound returned on a keyImage lookup
	ErrKeyImageNotFound = errors.New("database: keyImage not found")
	// ErrTxNotFound returned on a tx lookup by hash
	ErrTxNotFound = errors.New("database: transaction not found")
	// ErrBlockNotFound returned on a block lookup by hash or height
	ErrBlockNotFound = errors.New("database: block not found")
	// ErrStateNotFound returned on missing state db entry
	ErrStateNotFound = errors.New("database: state not found")
	// ErrOutputNotFound returned on output lookup during tx verification
	ErrOutputNotFound = errors.New("database: output not found")

	// AnyTxType is used as a filter value on FetchBlockTxByHash
	AnyTxType = transactions.TxType(math.MaxUint8)
)

Functions

func Drivers

func Drivers() []string

Drivers returns a sorted list of the names of the registered drivers.

func Register

func Register(driver Driver) error

Register makes a database driver available by the provided name. If Register is called twice with the same name or if driver is nil, it returns error.

Types

type DB

type DB interface {

	// View provides a managed execution of a read-only transaction
	View(fn func(t Transaction) error) error

	// Update provides a managed execution of a read-write atomic transaction.
	//
	// An atomic transaction is an indivisible and irreducible series of
	// database operations such that either all occur, or nothing occurs.
	//
	// Transaction commit will happen only if no error is returned by `fn`
	// and no panic is raised on `fn` execution.
	Update(fn func(t Transaction) error) error

	Close() error
}

DB is a thin layer on top of Transaction providing a manageable execution

type Driver

type Driver interface {
	// Open returns a new connection to a blockchain database. The path is a
	// string in a driver-specific format.
	Open(path string, network protocol.Magic, readonly bool) (DB, error)

	// Close terminates all DB connections and closes underlying storage
	Close() error

	// Name returns a unique identifier that can be used to register the driver
	Name() string
}

A Driver represents an application programming interface for accessing blockchain database management systems.

It is conceptually similar to ODBC for DBMS

func From

func From(name string) (Driver, error)

From returns a registered Driver by name

type State

type State struct {
	TipHash []byte
}

State represents a single db entry that provides chain metadata. This includes currently only chain tip hash but could be extended at later stage

type Transaction

type Transaction interface {
	FetchBlockHeader(hash []byte) (*block.Header, error)
	// Fetch all of the Txs that belong to a block with this header.hash
	FetchBlockTxs(hash []byte) ([]transactions.Transaction, error)
	// Fetch tx by txID. If succeeds, it returns tx data, tx index and
	// hash of the block it belongs to.
	FetchBlockTxByHash(txID []byte) (tx transactions.Transaction, txIndex uint32, blockHeaderHash []byte, err error)
	FetchBlockHashByHeight(height uint64) ([]byte, error)
	FetchBlockExists(hash []byte) (bool, error)
	// Fetch chain state information (chain tip hash)
	FetchState() (*State, error)

	// Check if an input keyImage is already stored. If succeeds, it returns
	// also txID the input belongs to
	FetchKeyImageExists(keyImage []byte) (exists bool, txID []byte, err error)

	// Read-write transactions
	// Store the next chain block in a append-only manner
	// Overwrites only if block with same hash already stored
	// Not to be called concurrently, as it updates chain tip
	StoreBlock(block *block.Block) error

	// FetchBlock will return a block, given a hash.
	FetchBlock(hash []byte) (*block.Block, error)

	// FetchCurrentHeight returns the height of the most recently stored
	// block in the database.
	FetchCurrentHeight() (uint64, error)

	// FetchDecoys will return a number of decoy public keys, to be used
	// when constructing a ring signature for a transaction.
	FetchDecoys(numDecoys int) []ristretto.Point

	// FetchOutputExists returns whether or not an output exists for the
	// given destination public key
	FetchOutputExists(destkey []byte) (bool, error)

	// FetchOutputUnlockHeight will return the unlock height for an output
	// given a destination public key.
	FetchOutputUnlockHeight(destkey []byte) (uint64, error)

	// StoreBidValues stores the D and K values passed by the caller in
	// the database, as well as the expiry height. It should be passed
	// the transaction locktime as a third argument, as the database
	// can infer the current height and consequently, the expiry height,
	// on its own.
	StoreBidValues([]byte, []byte, uint64) error

	// FetchBidValues retrieves the D and K values with the lowest
	// expiry height from the database.
	FetchBidValues() ([]byte, []byte, error)

	// FetchBlockHeightSince try to find height of a block generated around
	// sinceUnixTime starting the search from height (tip - offset)
	FetchBlockHeightSince(sinceUnixTime int64, offset uint64) (uint64, error)

	// ClearDatabase will remove all information from the database.
	ClearDatabase() error

	// Atomic storage
	Commit() error
	Rollback() error
	Close()
}

Transaction represents transaction layer. Transaction should provide basic transactions to fetch and store blockchain data.

To simplify code reading with database pkg we should use 'Tx' to refer to blockchain transaction and 'Transaction' to refer to database transaction