polkadothub-indexer

README

Description

Polkadothub Indexer project is responsible for fetching and indexing Polkadot data.

Internal dependencies:

This package connects via gRPC to a polkadothub-proxy which in turn connects to Polkadot node.

External Packages:

• polkadothub-proxy - Go proxy to Polkadot node
• indexing-engine - A backbone for indexing process
• gin - Http server
• gorm - ORM with PostgreSQL interface
• cron - Cron jobs runner
• zap - logging

Environmental variables:

• APP_ENV - application environment (development | production)
• PROXY_URL - url to polkadothub-proxy
• SERVER_ADDR - address to use for API
• SERVER_PORT - port to use for API
• FIRST_BLOCK_HEIGHT - height of first block in chain
• INDEX_WORKER_INTERVAL - index interval for worker
• SUMMARIZE_WORKER_INTERVAL - summary interval for worker
• PURGE_WORKER_INTERVAL - purge interval for worker
• DEFAULT_BATCH_SIZE - syncing batch size. Setting this value to 0 means no batch size
• DATABASE_DSN - PostgreSQL database URL
• DEBUG - turn on db debugging mode
• LOG_LEVEL - level of log
• LOG_OUTPUT - log output (ie. stdout or /tmp/logs.json)
• ROLLBAR_ACCESS_TOKEN - Rollbar access token for error reporting
• INDEXER_METRIC_ADDR - Prometheus server address for indexer metrics
• SERVER_METRIC_ADDR - Prometheus server address for server metrics
• PURGE_BLOCK_INTERVAL - Block sequence older than given interval will be purged
• PURGE_BLOCK_HOURLY_SUMMARY_INTERVAL - Block hourly summary records older than given interval will be purged
• PURGE_BLOCK_DAILY_SUMMARY_INTERVAL - Block daily summary records older than given interval will be purged
• PURGE_VALIDATOR_INTERVAL - Validator sequence older than given interval will be purged
• PURGE_VALIDATOR_HOURLY_SUMMARY_INTERVAL - Validator hourly summary records older than given interval will be purged
• PURGE_VALIDATOR_DAILY_SUMMARY_INTERVAL - Validator daily summary records older than given interval will be purged
• INDEXER_CONFIG_FILE - JSON file with targets and its task names

Available endpoints:

Method	Path	Description	Params
GET	/health	health endpoint	-
GET	/status	status of the application and chain	include_chain (bool, optional) - when true, returns chain status
GET	/block	return block by height	height (optional) - height [Default: 0 = last]
GET	/block_times/:limit	get last x block times	limit (required) - limit of blocks
GET	/blocks_summary	get block summary	interval (required) - time interval [hourly or daily] period (required) - summary period [ie. 24 hours]
GET	/transactions	get list of transactions	height (optional) - height [Default: 0 = last]
GET	/account/:stash_account	get account information for height	stash_account (required) - stash account height (optional) - height [Default: 0 = last]
GET	/account_details/:stash_account	get account details	stash_account (required) - stash account
GET	/rewards/:stash_account	get daily rewards for account	stash_account (required), start (optional) - the starting era [Default: 1 = first], end (optional) - the ending era (if unspecified, returns latest)(optional)
GET	/validators	get list of validators	height (optional) - height [Default: 0 = last]
GET	/validators/for_min_height/:height	get the list of validators for height greater than provided	height (required) - height [Default: 0 = last]
GET	/validator/:stash_account	get validator by address	stash_account (required) - validator's stash account sessions_limit (required) - number of last sessions to include eras_limit (required) - number of last eras to include
GET	/validators_summary	validator summary	interval (required) - time interval [hourly or daily] period (required) - summary period [ie. 24 hours] stash_account (optional) - validator's stash account
GET	/system_events	get system events for validator	after (optional) - height kind (optional) - system event kind [eg. "joined_set"]
GET	/apr	get daily calculated APRs (annualized percentage rates) for time range	account (required) - stash account start (required) - date in format 2006-01-02 end (required) - date in format 2006-01-02

Running app

Once you have created a database and specified all configuration options, you need to migrate the database. You can do that by running the command below:

polkadothub-indexer -config path/to/config.json -cmd=migrate

Start the data indexer:

polkadothub-indexer -config path/to/config.json -cmd=worker

Start the API server:

polkadothub-indexer -config path/to/config.json -cmd=server

IMPORTANT!!! Make sure that you have polkadothub-proxy running and connected to Polkadot node.

Running one-off commands

Start indexer:

polkadothub-indexer -config path/to/config.json -cmd=indexer_start

Create summary tables for sequences:

polkadothub-indexer -config path/to/config.json -cmd=indexer_summarize

Purge old data:

polkadothub-indexer -config path/to/config.json -cmd=indexer_purge

Running tests

To run tests with coverage you can use test Makefile target:

make test