versiondb

package module
v0.0.0-...-a6f2824 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 13, 2023 License: Apache-2.0 Imports: 21 Imported by: 0

README

VersionDB

VersionDB is a solution for the size issue of IAVL database, aka. application.db, at current stage, it's only recommended for archive and non-validator nodes to try (validator nodes are recommended to do pruning anyway).

VersionDB stores multiple versions of on-chain state key-value pairs directly, without using a merklized tree structure like IAVL tree, both db size and query performance are much better than IAVL tree. The major lacking feature compared to IAVL tree is root hash and merkle proof generation, so we still need IAVL tree for those tasks.

Currently grpc query service don't need to support proof generation, so versiondb alone is enough to support grpc query service, there's already a --grpc-only flag for one to start a standalone grpc query service.

There could be different implementations for the idea of versiondb, the current implementation we delivered is based on rocksdb v7's experimental user-defined timestamp^1, it stores the data in a standalone rocksdb instance, it don't support other db backend yet, but the other databases in the node still support multiple backends as before.

After versiondb is enabled, there's no point to keep the full the archived IAVL tree anymore, it's recommended to prune the IAVL tree to keep only recent versions, for example versions within the unbonding period or even less.

Configuration

To enable versiondb, add versiondb to the list of store.streamers in app.toml like this:

[store]
streamers = ["versiondb"]

On startup, the node will create a StreamingService to subscribe to latest state changes in realtime and save them to versiondb, the db instance is placed at $NODE_HOME/data/versiondb directory, there's no way to customize the db path currently. It'll also switch grpc query service's backing store to versiondb from IAVL tree, you should migrate the legacy states in advance to make the transition smooth, otherwise, the grpc queries can't see the legacy versions.

If the versiondb is not empty and it's latest version doesn't match the IAVL db's last committed version, the startup will fail with error message "versiondb lastest version %d doesn't match iavl latest version %d", that's to avoid creating gaps in versiondb accidentally. When this error happens, you just need to update versiondb to the latest version in iavl tree manually, or restore IAVL db to the same version as versiondb (see ).

Migration

Since our chain is pretty big now, a lot of efforts have been put to make sure the transition process can finish in practical time. The migration process will try to parallelize the tasks as much as possible, and use significant ram, but there's flags for user to control the concurrency level and ram usage to make it runnable on different machine specs.

The legacy state migration process is done in two main steps:

  • Extract state change sets from existing archived IAVL tree.
  • Feed the change set files to versiondb.
Extract Change Sets
$ elysiumd changeset dump data --home /chain/.elysiumd

dump command will extract the change sets from the IAVL tree, and store each store in separate directories, it use the store list registered in current version of App by default, you can customize that with --stores parameter. The change set files are segmented into different block chunks and compressed with zlib level 6 by default, the chunk size defaults to 1m blocks, the result data directly looks like this:

data/acc/block-0.zz
data/acc/block-1000000.zz
data/acc/block-2000000.zz
...
data/authz/block-0.zz
data/authz/block-1000000.zz
data/authz/block-2000000.zz
...

The extraction is the slowest step, the test run on testnet archive node takes around 11 hours on a 8core ssd machine, but fortunately, the change set files can be verified pretty fast(a few minutes), so they can be share on CDN in a trustless manner, normal users should just download them from CDN and verify the correctness locally, should be much faster than extract by yourself.

For rocksdb backend, dump command opens the db in readonly mode, it can run on live node's db, but goleveldb backend don't support this feature yet.

Verify Change Sets
$ elysiumd changeset verify data
35b85a775ff51cbcc48537247eb786f98fc6a178531d48560126e00f545251be
{"version":"189","storeInfos":[{"name":"acc","commitId":{"version":"189" ...

verify command will replay all the change sets and rebuild the target IAVL tree and output the app hash and commit info of the target version (defaults to latest version in the change sets), then user can manually check the app hash against the block headers.

verify command takes several minutes and several gigabytes of ram to run, if ram usage is a problem, it can also run incrementally, you can export the snapshot for a middle version, then verify the remaining versions start from that snapshot:

$ elysiumd changeset verify data --save-snapshot snapshot --target-version 3000000
$ elysiumd changeset verify data --load-snapshot snapshot

The format of change set files are documented here.

Build VersionDB

To maximize the speed of initial data ingestion speed into rocksdb, we take advantage of the sst file writer feature to write out sst files first, then ingest them into final db, the sst files for each store can be written out in parallel. We also developed an external sorting algorithm to sort the data before writing the sst files, so the sst files don't have overlaps and can be ingested into the bottom-most level in final db.

$ elysiumd changeset build-versiondb-sst ./data ./sst
$ elysiumd changeset ingest-versiondb-sst /home/.elysiumd/data/versiondb sst/*.sst --move-files --maximum-version 189

User can control the peak ram usage by controlling the --concurrency and --sorter-chunk-size.

With default parameters it can finish in around 12minutes for testnet archive node on our test node (8cores, peak RSS 2G).

Restore IAVL Tree

When migrating an existing archive node to versiondb, it's recommended to rebuild the application.db from scratch to reclaim disk space faster, we provide a command to restore a single version of IAVL trees from memiavl snapshot.

$ # create memiavl snapshot
$ elysiumd changeset verify data --save-snapshot snapshot
$ # restore application.db
$ elysiumd changeset restore-app-db snapshot application.db

Then replace the whole application.db in the node with the newly generated one.

It only takes a few minutes to run on our testnet archive node, it only suppot generating rocksdb application.db right now, so please set app-db-backend="rocksdb" in app.toml.

Catch Up With IAVL Tree

If an non-empty versiondb lags behind from the current application.db, the node will refuse to startup, in this case user can either sync versiondb to catch up with application.db, or simply restore the application.db with the correct version of snapshot. To catch up, you can follow the similar procedure as migrating from genesis, just passing the block range in change set dump command.

Documentation

Index

Constants

View Source 
const StoreTypeVersionDB = 100

Variables

This section is empty.

Functions

func Run 

func Run(t *testing.T, storeCreator func() VersionStore)

func SetupTestDB 

func SetupTestDB(t *testing.T, store VersionStore)

Types

type MultiStore 

type MultiStore struct {
	// contains filtered or unexported fields
}

MultiStore wraps `VersionStore` to implement `MultiStore` interface.

func NewMultiStore 

func NewMultiStore(versionDB VersionStore, storeKeys []types.StoreKey) *MultiStore

NewMultiStore returns a new versiondb `MultiStore`.

func (*MultiStore) CacheMultiStore 

func (s *MultiStore) CacheMultiStore() sdk.CacheMultiStore

CacheMultiStore implements `MultiStore` interface

func (*MultiStore) CacheMultiStoreWithVersion 

func (s *MultiStore) CacheMultiStoreWithVersion(version int64) (sdk.CacheMultiStore, error)

CacheMultiStoreWithVersion implements `MultiStore` interface

func (*MultiStore) CacheWrap 

func (s *MultiStore) CacheWrap() types.CacheWrap

CacheWrap implements CacheWrapper/MultiStore/CommitStore.

func (*MultiStore) CacheWrapWithTrace 

func (s *MultiStore) CacheWrapWithTrace(_ io.Writer, _ types.TraceContext) types.CacheWrap

CacheWrapWithTrace implements the CacheWrapper interface.

func (*MultiStore) GetKVStore 

func (s *MultiStore) GetKVStore(storeKey types.StoreKey) sdk.KVStore

GetKVStore implements `MultiStore` interface

func (*MultiStore) GetStore 

func (s *MultiStore) GetStore(storeKey types.StoreKey) sdk.Store

GetStore implements `MultiStore` interface

func (*MultiStore) GetStoreType 

func (s *MultiStore) GetStoreType() types.StoreType

GetStoreType implements `MultiStore` interface.

func (*MultiStore) LatestVersion 

func (s *MultiStore) LatestVersion() int64

LatestVersion returns the latest version saved in versiondb

func (*MultiStore) MountMemoryStores 

func (s *MultiStore) MountMemoryStores(keys map[string]*types.MemoryStoreKey)

MountMemoryStores simlates the same behavior as sdk to support grpc query service.

func (*MultiStore) MountTransientStores 

func (s *MultiStore) MountTransientStores(keys map[string]*types.TransientStoreKey)

MountTransientStores simlates the same behavior as sdk to support grpc query service.

func (*MultiStore) SetTracer 

func (s *MultiStore) SetTracer(w io.Writer) types.MultiStore

SetTracer sets the tracer for the MultiStore that the underlying stores will utilize to trace operations. A MultiStore is returned.

func (*MultiStore) SetTracingContext 

func (s *MultiStore) SetTracingContext(tc types.TraceContext) types.MultiStore

SetTracingContext updates the tracing context for the MultiStore by merging the given context with the existing context by key. Any existing keys will be overwritten. It is implied that the caller should update the context when necessary between tracing operations. It returns a modified MultiStore.

func (*MultiStore) TracingEnabled 

func (s *MultiStore) TracingEnabled() bool

TracingEnabled returns if tracing is enabled for the MultiStore.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store Implements types.KVStore

func NewKVStore 

func NewKVStore(store VersionStore, storeKey types.StoreKey, version *int64) *Store

func (*Store) CacheWrap 

func (st *Store) CacheWrap() types.CacheWrap

Implements Store.

func (*Store) CacheWrapWithListeners 

func (st *Store) CacheWrapWithListeners(storeKey types.StoreKey, listeners []types.WriteListener) types.CacheWrap

CacheWrapWithListeners implements the CacheWrapper interface.

func (*Store) CacheWrapWithTrace 

func (st *Store) CacheWrapWithTrace(w io.Writer, tc types.TraceContext) types.CacheWrap

CacheWrapWithTrace implements the Store interface.

func (*Store) Delete 

func (st *Store) Delete(key []byte)

Implements types.KVStore.

func (*Store) Get 

func (st *Store) Get(key []byte) []byte

Implements types.KVStore.

func (*Store) GetStoreType 

func (st *Store) GetStoreType() types.StoreType

Implements Store.

func (*Store) Has 

func (st *Store) Has(key []byte) (exists bool)

Implements types.KVStore.

func (*Store) Iterator 

func (st *Store) Iterator(start, end []byte) types.Iterator

Implements types.KVStore.

func (*Store) ReverseIterator 

func (st *Store) ReverseIterator(start, end []byte) types.Iterator

Implements types.KVStore.

func (*Store) Set 

func (st *Store) Set(key, value []byte)

Implements types.KVStore.

type StreamingService 

type StreamingService struct {
	// contains filtered or unexported fields
}

StreamingService is a concrete implementation of StreamingService that accumulate the state changes in current block, writes the ordered changeset out to version storage.

func NewStreamingService 

func NewStreamingService(versionStore VersionStore, storeKeys []types.StoreKey) *StreamingService

NewStreamingService creates a new StreamingService for the provided writeDir, (optional) filePrefix, and storeKeys

func (*StreamingService) Close 

func (fss *StreamingService) Close() error

Close satisfies the io.Closer interface, which satisfies the baseapp.StreamingService interface

func (*StreamingService) ListenBeginBlock 

func (fss *StreamingService) ListenBeginBlock(ctx context.Context, req abci.RequestBeginBlock, res abci.ResponseBeginBlock) error

ListenBeginBlock satisfies the baseapp.ABCIListener interface It sets the currentBlockNumber.

func (*StreamingService) ListenCommit 

func (fss *StreamingService) ListenCommit(ctx context.Context, res abci.ResponseCommit) error

func (*StreamingService) ListenDeliverTx 

func (fss *StreamingService) ListenDeliverTx(ctx context.Context, req abci.RequestDeliverTx, res abci.ResponseDeliverTx) error

ListenDeliverTx satisfies the baseapp.ABCIListener interface

func (*StreamingService) ListenEndBlock 

func (fss *StreamingService) ListenEndBlock(ctx context.Context, req abci.RequestEndBlock, res abci.ResponseEndBlock) error

ListenEndBlock satisfies the baseapp.ABCIListener interface It merge the state caches of all the listeners together, and write out to the versionStore.

func (*StreamingService) Listeners 

func (fss *StreamingService) Listeners() map[types.StoreKey][]types.WriteListener

Listeners satisfies the baseapp.StreamingService interface

func (*StreamingService) Stream 

func (fss *StreamingService) Stream(wg *sync.WaitGroup) error

Stream satisfies the baseapp.StreamingService interface

type VersionStore 

type VersionStore interface {
	GetAtVersion(storeKey string, key []byte, version *int64) ([]byte, error)
	HasAtVersion(storeKey string, key []byte, version *int64) (bool, error)
	IteratorAtVersion(storeKey string, start, end []byte, version *int64) (types.Iterator, error)
	ReverseIteratorAtVersion(storeKey string, start, end []byte, version *int64) (types.Iterator, error)
	GetLatestVersion() (int64, error)

	// Persist the change set of a block,
	// the `changeSet` should be ordered by (storeKey, key),
	// the version should be latest version plus one.
	PutAtVersion(version int64, changeSet []types.StoreKVPair) error
}

VersionStore is a versioned storage of a flat key-value pairs. it don't need to support merkle proof, so could be implemented in a much more efficient way. `nil` version means the latest version.

Source Files

View all Source files

Directories

Path Synopsis
Package extsort implements external sorting algorithm, it has several differnet design choices compared with alternatives like https://github.com/lanrat/extsort:
 Package extsort implements external sorting algorithm, it has several differnet design choices compared with alternatives like https://github.com/lanrat/extsort:

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.