blockatlas

README

Block Atlas by Trust Wallet

BlockAtlas is a clean explorer API and transaction observer for cryptocurrencies.

BlockAtlas connects to nodes or explorer APIs of the supported coins and maps transaction data, account transaction history into a generic, easy to work with JSON format. It is in production use at the Trust Wallet app, the official cryptocurrency wallet of Binance. Also is in production at the BUTTON Wallet, Telegram based non-custodial wallet. The observer API watches the chain for new transactions and generates notifications by guids.

Supported Coins

Architecture

NOTE

Currently Block Atlas is under active development and is not well documented. If you still want to run it on your own or help to contribute, please pay attention that currently integration, nemwan, functional tests are not working locally without all endpoints. We are fixing that issue and soon you will be able to test all the stuff locally

Blockatlas allows to:

• Get information about transactions, tokens, staking details, collectibles, crypto domains for supported coins.
• Subscribe for price notifications via Rabbit MQ

Platform API is independent service and can work with the specific blockchain only (like Bitcoin, Ethereum, etc)

Notifications:

(Observer Subscriber Producer) - Create new blockatlas.SubscriptionEvent [Not implemented at Atlas, write it on your own]

(Observer Subscriber) - Get subscriptions from queue, set them to the DB

(Observer Parser) - Parse the block, convert block to the transactions batch, send to queue

(Observer Notifier) - Check each transaction for having the same address as stored at DB, if so - send tx data and id to the next queue

(Observer Notifier Consumer) - Notify the user [Not implemented at Atlas, write it on your own]

New Subscriptions --(Rabbit MQ)--> Subscriber --> DB
                                                   |
                      Parser  --(Rabbit MQ)--> Notifier --(Rabbit MQ)--> Notifier Consumer --> User

The whole flow is not available at Atlas repo. We will have integration tests with it. Also there will be examples of all instances soon.

Setup

Prerequisite

• Go Toolchain versions 1.14+

Depends on what type of Blockatlas service you would like to run will also be needed.

• Postgres to store user subscriptions and latest parsed block number
• Rabbit MQ to pass subscriptions and send transaction notifications

Quick Start

Get source code

Download source to GOPATH

go get -u github.com/trustwallet/blockatlas
cd $(go env GOPATH)/src/github.com/trustwallet/blockatlas

Build and run

Read configuration info

# Start Platform API server at port 8420 with the path to the config.yml ./
go build -o platform-api-bin cmd/platform_api/main.go && ./platform-api-bin -p 8420

# Start observer_parser with the path to the config.yml ./ 
go build -o observer_parser-bin cmd/observer_parser/main.go && ./observer_parser-bin

# Start observer_notifier with the path to the config.yml ./ 
go build -o observer_notifier-bin cmd/observer_notifier/main.go && ./observer_notifier-bin

# Start observer_subscriber with the path to the config.yml ./ 
go build -o observer_subscriber-bin cmd/observer_subscriber/main.go && ./observer_subscriber-bin

# Startp Swagger API server at port 8422 with the path to the config.yml ./ 
go build -o swagger-api-bin cmd/swagger-api/main.go && ./swagger-api-bin -p 8423

# Start Platform API server with mocked config, at port 8437 ./ 
go build -o platform-api-bin cmd/platform_api/main.go && ./platform-api-bin -p 8437 -c configmock.yml

make command

Build and start all services:

make go-build
make start

Build and start individual service:

make go-build-platform-api
make start

Docker

Build and run all services:

docker-compose build
docker-compose up

Build and run individual service:

docker-compose build swagger_api
docker-compose start swagger_api

Configuration

When any of Block Atlas services started they look up inside default configuration. Most coins offering public RPC/explorer APIs are enabled, thus Block Atlas can be started and used right away, no additional configuration needed. By default starting any of the services will enable all platforms

To run a specific service only by passing environmental variable, e.g: platfrom_api :

ATLAS_PLATFORM=ethereum go run cmd/platform_api/main.go

or change in config file

platform: ethereum

This way you can one platform per binary, for scalability and sustainability.

To enable use of private endpoint:

nimiq:
  api: http://localhost:8648

It works the same for observer_worker - you can run all observer at 1 binary or 30 coins per 30 binaries

Environment

The rest gets loaded from environment variables. Every config option is available under the ATLAS_ prefix. Nested keys are joined via _.

Example:

ATLAS_NIMIQ_API=http://localhost:8648

Tests

Unit tests

make test

Mocked tests

End-to-end tests with calls to external APIs has great value, but is not suitable for regular CI verification, as any external reasons could break the tests.

Therefore mocked API-level tests are used, whereby external APIs are replaced by mocks.

• External mocks are implemented using dyson, as javascript files. They generally return constant, pre-canned responses to the requests that occur during tests.
• Mocks are 'turned on' by corresponding API endpoints in the configmock.yml config file (localhost:3347).
• Tests invoke into blockatlas through public APIs only, and are executed using newman (Postman cli -- make newman-mocked).
• Product code, and even test code should not be aware whether it runs with mocks or the real external endpoints.
• See Makefile for targets with 'mock'; platform can be started locally with mocks using make start-platform-api-mock.
• When dyson is started (e.g. with make start-platform-api-mock), it outputs requests, which helps debugging
• The newman tests can be executed with unmocked external APIs as well, but verifications may fail, because some APIs return variable responses. Unmocked tests are not intended for regular CI execution, but as ad-hoc development tests.
• General steps for creating new mocked tests: replace endpoint to localhost:3347, observer incoming calls (dyson output), obtain real response from external API (with exact same parameters), enhance dsyon script to return the same output, verify that blockatlas provides correct output. Also, add verifications of results to the tests.

Docs

Swagger API docs provided at path /swagger/index.html

Updating Docs

• After creating a new route, add comments to your API source code, See Declarative Comments Format.

• Download Swag for Go by using:

  $ go get -u github.com/swaggo/swag/cmd/swag

• Run the Swag in your Go project root folder.

  $ swag init -g ./cmd/platform_api/main.go -o ./docs

Contributing

If you'd like to add support for a new blockchain, feel free to file a pull request. Note that most tokens that run on top of other chains are already supported and don't require code changes (e.g. ERC-20).

The best way to submit feedback and report bugs is to open a GitHub issue. Please be sure to include your operating system, version number, and steps to reproduce reported bugs.

More resources for developers are in CONTRIBUTING.md.