database

README

README

General concept

Package database represents a database layer in Dusk Network. Current design should embody both backend-agnostic and storage-agnostic concepts.

Terminology

Storage: Underlying storage can be any kind of KV stores (BoltDB, LMDB, LevelDB, RocksDB, Redis), non-KV stores (SQLite) or even a custom ACID-complient flat-file store.

Backend/Driver: represents a DB implementation that uses one or multiple underlying storages to provide block chain database specifics with regard to particular purpose. It must implement Driver, DB and Tx intefaces.

Interfaces exposed to upper layers:

• database.Driver: DB connection initiator. Inspired by database/sql/driver
• database.DB: A thin layer on top of database.Tx providing a manageable Tx execution. Inspired by BoltDB
• database.Tx: A transaction layer tightly coupled with DUSK block chain specifics to fully benefit from underlying storage capabilities

Available Drivers

• /database/heavy driver is designed to provide efficient, robust and persistent DUSK block chain DB on top of syndtr/goleveldb/leveldb store (unofficial LevelDB porting). It must be Mainnet-complient.

Testing Drivers

• /database/testing implements a boilerplate method to verify if a registered driver does satisfy minimum database requirements. The package defines a set of unit tests that are executed only on registered drivers. It can serve also as a detailed and working database guideline.

Code example:

More code examples can be found in /database/heavy/database_test.go

// By changing Driver name, one can switch between different backends
// Each backend is bound to one or multiple underlying stores
readonly := false

// Retrieve
driver, _ := database.From(lite.DriverName)
db, err := driver.Open(path, protocol.DevNet, readonly)

if err != nil {
    ...
}

defer db.Close()

blocks := make([]*block.Block, 100)
// Populate blocks here ...

// a managed read-write DB Tx to store all blocks via atomic update
err = db.Update(func(tx database.Tx) error {
    for _, block := range blocks {
        err := tx.StoreBlock(block)
        if err != nil {
            return err
        }
    }
    return nil
})

if err != nil {
    fmt.Printf("Transaction failed. No blocks stored")
    return
}

 // a managed read-only DB Tx to check if all blocks have been stored
_ = db.View(func(tx database.Tx) error {
    for _, block := range blocks {
        exists, err := tx.FetchBlockExists(block.Header.Hash)
        if err != nil {
            fmt.Printf(err.Error())
            return nil
        }

        if !exists {
            fmt.Printf("Block with Height %d was not found", block.Header.Height)
            return nil
        }
    }
    return nil
})

Additional features

Additional features that can be provided by a Driver:

• Read-only transactions/connections
• Validated storage connection
• Profiling: Collect transactions and connections statistics
• Traces: Log transactions data
• Alarming: Trigger an event in case of critical backend or storage failure
• Iterators/Cursors
• Safe Concurrency model: One read-write transaction, many read-only transactions
• Redundancy: blockchain data stored in a secondary in-memory or on-disk structure

Documentation

Index

• Variables
• func Drivers() []string
• func Register(driver Driver) error
• type DB
• type Driver
  • func From(name string) (Driver, error)
• type State
• type Transaction

Variables

var (

	// ErrKeyImageNotFound returned on a keyImage lookup.
	ErrKeyImageNotFound = errors.New("database: keyImage not found")
	// ErrTxNotFound returned on a tx lookup by hash.
	ErrTxNotFound = errors.New("database: transaction not found")
	// ErrBlockNotFound returned on a block lookup by hash or height.
	ErrBlockNotFound = errors.New("database: block not found")
	// ErrStateNotFound returned on missing state db entry.
	ErrStateNotFound = errors.New("database: state not found")
	// ErrOutputNotFound returned on output lookup during tx verification.
	ErrOutputNotFound = errors.New("database: output not found")

	// AnyTxType is used as a filter value on FetchBlockTxByHash.
	AnyTxType = transactions.TxType(math.MaxUint8)
)

Functions

func Drivers

func Drivers() []string

Drivers returns a sorted list of the names of the registered drivers.

func Register

func Register(driver Driver) error

Register makes a database driver available by the provided name. If Register is called twice with the same name or if driver is nil, it returns error.

Types

type DB

type DB interface {
	// View provides a managed execution of a read-only transaction.
	View(fn func(t Transaction) error) error

	// Update provides a managed execution of a read-write atomic transaction.
	//
	// An atomic transaction is an indivisible and irreducible series of
	// database operations such that either all occur, or nothing occurs.
	//
	// Transaction commit will happen only if no error is returned by `fn`
	// and no panic is raised on `fn` execution.
	Update(fn func(t Transaction) error) error

	Close() error
}

DB is a thin layer on top of Transaction providing a manageable execution.

type Driver

type Driver interface {
	// Open returns a new connection to a blockchain database. The path is a
	// string in a driver-specific format.
	Open(path string, network protocol.Magic, readonly bool) (DB, error)

	// Close terminates all DB connections and closes underlying storage.
	Close() error

	// Name returns a unique identifier that can be used to register the driver.
	Name() string
}

A Driver represents an application programming interface for accessing blockchain database management systems.

It is conceptually similar to ODBC for DBMS.

func From

func From(name string) (Driver, error)

From returns a registered Driver by name.

type State

type State struct {
	TipHash []byte
}

State represents a single db entry that provides chain metadata. This includes currently only chain tip hash but could be extended at later stage.

type Transaction

type Transaction interface {
	FetchBlockHeader(hash []byte) (*block.Header, error)
	// Fetch all of the Txs that belong to a block with this header.hash.
	FetchBlockTxs(hash []byte) ([]transactions.ContractCall, error)
	// Fetch tx by txID. If succeeds, it returns tx data, tx index and
	// hash of the block it belongs to.
	FetchBlockTxByHash(txID []byte) (tx transactions.ContractCall, txIndex uint32, blockHeaderHash []byte, err error)
	FetchBlockHashByHeight(height uint64) ([]byte, error)
	FetchBlockExists(hash []byte) (bool, error)
	// Fetch chain state information (chain tip hash).
	FetchState() (*State, error)

	// Check if an input keyImage is already stored. If succeeds, it returns
	// also txID the input belongs to.
	FetchKeyImageExists(keyImage []byte) (exists bool, txID []byte, err error)

	// Read-write transactions
	// Store the next chain block in a append-only manner
	// Overwrites only if block with same hash already stored
	// Not to be called concurrently, as it updates chain tip.
	StoreBlock(block *block.Block) error

	// FetchBlock will return a block, given a hash.
	FetchBlock(hash []byte) (*block.Block, error)

	// FetchCurrentHeight returns the height of the most recently stored
	// block in the database.
	FetchCurrentHeight() (uint64, error)

	// FetchOutputExists returns whether or not an output exists for the
	// given destination public key.
	FetchOutputExists(destkey []byte) (bool, error)

	// FetchOutputUnlockHeight will return the unlock height for an output
	// given a destination public key.
	FetchOutputUnlockHeight(destkey []byte) (uint64, error)

	// StoreBidValues stores the D and K values passed by the caller in
	// the database, as well as the expiry height. It should be passed
	// the transaction lock time as a third argument, as the database
	// can infer the current height and consequently, the expiry height,
	// on its own.
	// XXX the Unused value was erroneously marked as Seed.
	StoreBidValues(D []byte, K []byte, BidIndex uint64, ExpiryHeight uint64) error

	// FetchBidValues retrieves the D and K values with the lowest
	// expiry height from the database.
	// XXX the Unused value was erroneously marked as Seed.
	FetchBidValues() (D []byte, K []byte, BidIndex uint64, err error)

	// FetchBlockHeightSince try to find height of a block generated around
	// sinceUnixTime starting the search from height (tip - offset).
	FetchBlockHeightSince(sinceUnixTime int64, offset uint64) (uint64, error)

	// StoreCandidateMessage will...
	StoreCandidateMessage(cm block.Block) error

	FetchCandidateMessage(hash []byte) (block.Block, error)

	ClearCandidateMessages() error

	// ClearDatabase will remove all information from the database.
	ClearDatabase() error

	// Atomic storage.
	Commit() error
	Rollback() error
	Close()
}

Transaction represents transaction layer. Transaction should provide basic transactions to fetch and store blockchain data.

To simplify code reading with database pkg we should use 'Tx' to refer to blockchain transaction and 'Transaction' to refer to database transaction.