trxdb_loader

package
v0.1.0-beta4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 23, 2020 License: Apache-2.0 Imports: 15 Imported by: 0

README

KVDB Loader

This repository holds the code necessary to take the blockchain data and insert it into out database instance (BigTable).

Indexing Development

Here the instructions to easily try out your code modifications. First, ensure you have the gcloud BigTable Emulator installed through the gcloud CLI:

gcloud components list | grep "Cloud Bigtable Emulator"

# If not already installed
gcloud components install bigtable

Once installed, you will need to have three terminals open. The first one will run the actual BigTable Emulator.

The second one will open a port forward so a communication channel can exists with the block relayer to pull blocks from.

The third one will run the actual indexing process (your code) that connects to the BigTable Emulator instead of a real database instance.

In the first terminal, starts the BigTable Emulator:

gcloud beta emulators bigtable start

In the second terminal, open a port forward with the block relayer service you are interested with (use eos-dev1 namespace to start):

kc -n eos-dev1 port-forward svc/relayer 10101

In the third terminal, use the index_dev.sh script to automatically starts the indexing in development mode, filling your own local BigTable Emulator with data so you can inspect it.

./index_dev.sh

By default, the index_dev.sh scripts is configured to pull blocks from the eos-dev1 development test network.

You can easily override those parameters by specifying the extra_args environment variable, being at the bottom, it will override the default parameters.

For example, to index EOS Mainnet blocks, first kill your initial port forward and point it to EOS Mainnet block relayer:

kc -n eos-mainnet port-forward relayer 10101

Then in the third terminal, start the index_dev.sh script like this:

extra_args="-chain-id=aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906 \
-source-store=gs://example/blocks" ./index_dev.sh

As long as the BigTable Emulator loader instance is running, restarting the process will start back where it left previously, just like in production.

To start from scratch, simply kill the BigTable Emulator and start it back.

Specific Block Range

If you are hunting a specific thing that happen within a particular block or block range, the best is to start the indexer for this exact range. To do that, we will simply use extra_args environment parameters to override the processing mode to batch and pass the block range we are interested in.

extra_args="-processing-type=batch -start-block-num=10 -stop-block-num=20" ./index_dev.sh

This will start the loading process, in batch mode for block range [10, 20].

Inspect Data

The best way to check out if to inspect the data being inserted into BigTable Emulator. Open a fourth terminal (yeah sorry :$).

Note: You can also kill the ./index_dev.sh script and use the terminal there if you don't need the indexing to continue while testing your stuff.

In this terminal, initialize the environment so cbt connects to BigTable Emulator:

$(gcloud beta emulators bigtable env-init)

Once this is done, you can now use cbt to check some tables:

cbt -project=test -instance=dev read eos-test-v1-accounts prefix=a: count=1
cbt -project=test -instance=dev read eos-test-v1-blocks prefix=fffffffc count=1
cbt -project=test -instance=dev read eos-test-v1-trxs prefix=3 count=1
cbt -project=test -instance=dev read eos-test-v1-timeline prefix=bf: count=1

More queries (perform them on low-level instances):

# Count number of blocks
cbt -project=test -instance=dev read eos-test-v1-blocks columns=meta:header | grep "meta:header" | wc -l

# Count number of transaction keys
cbt -project=test -instance=dev read eos-test-v1-trxs | grep '\----------------------------------------' | wc -l

# Count number of irreversible transactions
cbt -project=test -instance=dev read eos-test-v1-trxs columns=meta:irreversible | grep "meta:irreversible" | wc -l
Troubleshooting

If you see the message:

2019/06/04 11:02:37 -creds flag unset, will use gcloud credential
2019/06/04 11:02:38 Getting list of tables: rpc error: code = InvalidArgument desc = When parsing 'projects/dev/instances/dev' : Invalid id for collection instances : Length should be between [6,33], but found 3 'dev'

When using cbt, it means you are trying to connect to a real Google BigTable instance and the terminal environment is not initialized correctly with $(gcloud beta emulators bigtable env-init).

Run $(gcloud beta emulators bigtable env-init) and retry the command.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BigtableLoader 

type BigtableLoader struct {
	*shutter.Shutter
	// contains filtered or unexported fields
}

func NewBigtableLoader 

func NewBigtableLoader(
	blockStreamAddr string,
	blocksStore dstore.Store,
	batchSize uint64,
	db trxdb.DBWriter,
	parallelFileDownloadCount int,
) *BigtableLoader

func (*BigtableLoader) BuildPipelineBatch 

func (l *BigtableLoader) BuildPipelineBatch(startBlockNum uint64, numBlocksBeforeStart uint64)

func (*BigtableLoader) BuildPipelineJob 

func (l *BigtableLoader) BuildPipelineJob(startBlockNum uint64, numBlocksBeforeStart uint64, job Job)

func (*BigtableLoader) BuildPipelineLive 

func (l *BigtableLoader) BuildPipelineLive(allowLiveOnEmptyTable bool) error

func (*BigtableLoader) BuildPipelinePatch 

func (l *BigtableLoader) BuildPipelinePatch(startBlockNum uint64, numBlocksBeforeStart uint64)

func (*BigtableLoader) DoFlush 

func (l *BigtableLoader) DoFlush(blockNum uint64) error

func (*BigtableLoader) FlushIfNeeded 

func (l *BigtableLoader) FlushIfNeeded(blockNum uint64, blockTime time.Time) error

func (*BigtableLoader) FullJob 

func (l *BigtableLoader) FullJob(blockNum uint64, block *pbcodec.Block, fObj *forkable.ForkableObject) (err error)

fullJob does all the database insertions needed to load the blockchain into our database.

func (*BigtableLoader) Healthy 

func (l *BigtableLoader) Healthy() bool

func (*BigtableLoader) InitLIB 

func (l *BigtableLoader) InitLIB(libID string)

func (*BigtableLoader) Launch 

func (l *BigtableLoader) Launch()

func (*BigtableLoader) PatchJob 

func (l *BigtableLoader) PatchJob(blockNum uint64, blk *pbcodec.Block, fObj *forkable.ForkableObject) (err error)

patchDatabase is a "scratch" pad to define patch code that can be applied on an ad-hoc basis. The idea is to leave this function empty when no patch needs to be applied.

When a patch is required, the suggested workflow is to develop the patch code in a side branch. When the code is ready, the "production" commit is tagged with the `patch-<tag>-<date>` where the tag is giving an overview of the patch and the date is the effective date (`<year>-<month>-<day>`): `patch-add-trx-meta-written-2019-06-30`. The branch is then deleted and the tag is pushed to the remote repository.

func (*BigtableLoader) ProcessBlock 

func (l *BigtableLoader) ProcessBlock(blk *bstream.Block, obj interface{}) (err error)

func (*BigtableLoader) ShouldPushLIBUpdates 

func (l *BigtableLoader) ShouldPushLIBUpdates(dposLIBNum uint64) bool

func (*BigtableLoader) ShowProgress 

func (l *BigtableLoader) ShowProgress(blockNum uint64)

func (*BigtableLoader) StopBeforeBlock 

func (l *BigtableLoader) StopBeforeBlock(blockNum uint64)

StopBeforeBlock indicates the stop block (exclusive), means that block num will not be inserted.

func (*BigtableLoader) UpdateIrreversibleData 

func (l *BigtableLoader) UpdateIrreversibleData(nowIrreversibleBlocks []*bstream.PreprocessedBlock) error

type Job 

type Job = func(blockNum uint64, blk *pbcodec.Block, fObj *forkable.ForkableObject) (err error)

type Loader 

type Loader interface {
	StopBeforeBlock(uint64)

	BuildPipelineLive(bool) error
	BuildPipelineBatch(startBlock uint64, beforeStart uint64)
	BuildPipelinePatch(startBlock uint64, beforeStart uint64)

	Launch()
	Healthy() bool

	// Shutter related
	Shutdown(err error)
	OnTerminated(f func(error))
	Terminated() <-chan struct{}
	Err() error
}

Source Files

View all Source files

Directories

Path Synopsis
app
trxdb-loader

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.