substreams

README

Streaming Data Engine based on SF Firehose Tech

DEVELOPER PREVIEW OF SUBSTREAMS

Think Fluvio for deterministic blockchain data.

The successor of https://github.com/streamingfast/sparkle, enabling greater composability, yet similar powers of parallelisation, and a much simpler model to work with.

Install

Get a release (soon).

From source:

git clone git@github.com:streamingfast/substreams
cd substreams
go install -v ./cmd/substreams

From source without checkout:

go install github.com/streamingfast/substreams/cmd/substreams@latest

Run remotely

Using StreamingFast's infrastructure

Dump that somewhere like .bashrc:

export STREAMINGFAST_KEY=server_YOUR_KEY_HERE  # Ask us on Discord for a key
function sftoken {
    export FIREHOSE_API_TOKEN=$(curl https://auth.dfuse.io/v1/auth/issue -s --data-binary '{"api_key":"'$STREAMINGFAST_KEY'"}' | jq -r .token)
	export SUBSTREAMS_API_TOKEN=$FIREHOSE_API_TOKEN
    echo Token set on FIREHOSE_API_TOKEN and SUBSTREAMS_API_TOKEN
}

Then in your shell, load a key in an env var with:

sftoken

Then, try to run the PancakeSwap Substreams (under pcs-rust/, we build instructions over there)

cd substreams-playground
substreams run -e bsc-dev.streamingfast.io:443 wasm_substreams_manifest.yaml pairs,block_to_pairs,volumes,totals,db_out -s 6810706 -t 6810711

Run locally

You can run the substreams service locally this way:

Get a recent release of the Ethereum Firehose, and install sfeth.

Alternatively, you can use this Docker image: ghcr.io/streamingfast/sf-ethereum:6aa70ca, known to work with version v0.0.5-beta of the substreams release herein.

Get some data (merged blocks) to play with locally (here on BSC mainnet):

# Downloads 2.6GB of data
sfeth tools download-from-firehose bsc-dev.streamingfast.io:443 6810000 6820000 ./localblocks
sfeth tools generate-irreversible-index ./localblocks ./localirr 6810000 6819700

Then run the firehose service locally in a terminal, reading blocks from your disk:

sfeth start firehose  --config-file=  --log-to-file=false  --common-blockstream-addr=  --common-blocks-store-url=./localdata --firehose-grpc-listen-addr=:9000* --substreams-enabled --substreams-rpc-endpoint=https://URL.POINTING.TO.A.BSC.ARCHIVE.NODE/if-you/want-to-use/eth_call/within/substreams

And then run the substreams command against your local deployment (checkout substreams-playground in the Run remotely section above):

substreams run -k -e localhost:9000 wasm_substreams_manifest.yaml pairs,block_to_pairs,db_out,volumes,totals -s 6810706 -t 6810711

Documentation

Index

• type BlockHook
• type PostJobHook
• type ReturnFunc

Types

type BlockHook

type BlockHook func(ctx context.Context, clock *pbsubstreams.Clock) error

type PostJobHook

type PostJobHook func(ctx context.Context, clock *pbsubstreams.Clock) error

type ReturnFunc

type ReturnFunc func(any *pbsubstreams.BlockScopedData) error