About
Vulcanize DB is a set of tools that make it easier for developers to write application-specific indexes and caches for dapps built on Ethereum.
Dependencies
- Go 1.11+
- Postgres 11
- Ethereum Node
Project Setup
Using Vulcanize for the first time requires several steps be done in order to allow use of the software. The following instructions will offer a guide through the steps of the process:
- Fetching the project
- Installing dependencies
- Configuring shell environment
- Database setup
- Configuring synced Ethereum node integration
- Data syncing
Installation
In order to fetch the project codebase for local use or modification, install it to your GOPATH
via:
go get github.com/makerdao/vulcanizedb
go get gopkg.in/DataDog/dd-trace-go.v1/ddtrace
Once fetched, dependencies can be installed via go get
or (the preferred method) at specific versions via golang/dep
, the prototype golang pakcage manager. Installation instructions are here.
In order to install packages with dep
, ensure you are in the project directory now within your GOPATH
(default location is ~/go/src/github.com/makerdao/vulcanizedb/
) and run:
dep ensure
After dep
finishes, dependencies should be installed within your GOPATH
at the versions specified in Gopkg.toml
.
Lastly, ensure that GOPATH
is defined in your shell. If necessary, GOPATH
can be set in ~/.bashrc
or ~/.bash_profile
, depending upon your system. It can be additionally helpful to add $GOPATH/bin
to your shell's $PATH
.
Setting up the Database
-
Install Postgres
-
Create a superuser for yourself and make sure psql --list
works without prompting for a password.
-
createdb vulcanize_public
-
cd $GOPATH/src/github.com/makerdao/vulcanizedb
-
Run the migrations: make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432
- To rollback a single step:
make rollback NAME=vulcanize_public
- To rollback to a certain migration:
make rollback_to MIGRATION=n NAME=vulcanize_public
- To see status of migrations:
make migration_status NAME=vulcanize_public
- See below for configuring additional environments
Setting up GraphQL frontend
The database migrations sets up prerequisites for a GraphQL API, which can be generated with Postgraphile.
To present the API as specified here:
- Install Postgraphile:
npm install -g postgraphile
- Invoke
postgraphile --connection postgres://[superuser]:[password]@localhost:5432/vulcanize_public --schema api --disable-default-mutations --no-ignore-rbac --watch --enhance-graphiql
- The GraphiQL frontend can be found at
localhost:5000/graphiql
, and the GraphQL endpoint at /graphql
Configuration
-
To use a local Ethereum node, copy environments/public.toml.example
to
environments/public.toml
and update the ipcPath
and levelDbPath
.
-
See environments/infura.toml
to configure commands to run against infura, if a local node is unavailable.
-
Copy environments/local.toml.example
to environments/local.toml
to configure commands to run against a local node such as Ganache or ganache-cli.
Start syncing with postgres
Syncs VulcanizeDB with the configured Ethereum node, populating blocks, transactions, receipts, and logs.
This command is useful when you want to maintain a broad cache of what's happening on the blockchain.
- Start Ethereum node (if fast syncing your Ethereum node, wait for initial sync to finish)
- In a separate terminal start VulcanizeDB:
./vulcanizedb sync --config <config.toml> --starting-block-number <block-number>
Alternatively, sync from Geth's underlying LevelDB
Sync VulcanizeDB from the LevelDB underlying a Geth node.
- Assure node is not running, and that it has synced to the desired block height.
- Start vulcanize_db
./vulcanizedb coldImport --config <config.toml>
- Optional flags:
--starting-block-number <block number>
/-s <block number>
: block number to start syncing from
--ending-block-number <block number>
/-e <block number>
: block number to sync to
--all
/-a
: sync all missing blocks
Syncs VulcanizeDB with the configured Ethereum node, populating only block headers.
This command is useful when you want a minimal baseline from which to track targeted data on the blockchain (e.g. individual smart contract storage values).
- Start Ethereum node
- In a separate terminal start VulcanizeDB:
./vulcanizedb headerSync --config <config.toml> --starting-block-number <block-number>
Continuously syncs Maker event logs from the configured Ethereum node based on the populated block headers.
This includes logs related to auctions, multi-collateral dai, and price feeds.
This command requires that the headerSync
process is also being run so as to be able to sync in real time.
- Start Ethereum node (or plan to configure the commands to point to a remote IPC path).
- In a separate terminal run the headerSync command (see above).
- In another terminal window run the continuousLogSync command:
./vulcanizedb continuousLogSync --config <config.toml>
- An option
--transformers
flag may be passed to the command to specific which transformers to execute, this will default to all transformers if the flag is not passed.
./vulcanizedb continuousLogSync --config environments/private.toml --transformers="priceFeed"
- see the
buildTransformerInitializerMap
method in cmd/continuousLogSync.go
for available transformers
Backfills Maker event logs from the configured Ethereum node based on the populated block headers.
This includes logs related to auctions, multi-collateral dai, and price feeds.
This command requires that a header sync (see command above) has previously been run.
Since auction/mcd contracts have not yet been deployed, this command will need to be run a local blockchain at the moment. As such, a new environment file will need to be added. See environments/local.toml.example
.
- Start Ethereum node
- In a separate terminal run the backfill command:
./vulcanizedb backfillMakerLogs --config <config.toml>
Running the Tests
make test
will run the unit tests and skip the integration tests
make integrationtest
will run the just the integration tests
Deploying
- you will need to make sure you have ssh agent running and your ssh key added to it. instructions here
go get -u github.com/pressly/sup/cmd/sup
sup staging deploy