benchutil

README

Benchmarks for Prysm State Transition

This package contains the functionality needed for benchmarking Prysms state transitions, this includes its block processing (with and without caching) and epoch processing functions. There is also a benchmark for HashTreeRoot on a large beacon state.

Benchmark Configuration

The following configs are in config.go:

• ValidatorCount: Sets the amount of active validators to perform the benchmarks with. Default is 16384.
• AttestationsPerEpoch: Sets the amount of attestations per epoch for the benchmarks to perform with, this affects the amount of attestations in a full block and the amount of attestations per epoch in the state for the ProcessEpoch and HashTreeRoot benchmark. Default is 128.

Generating new SSZ files

Due to the sheer size of the benchmarking configurations (16384 validators), the files used for benchmarking are pregenerated so there's no wasted computations on generating a genesis state with 16384 validators. This should only be needed if there is a breaking spec change and the tests fail from SSZ issues.

To generate new files to use for benchmarking, run the below command in the root of Prysm.

bazel run //tools/benchmark-files-gen -- --output-dir $PRYSMPATH/shared/benchutil/benchmark_files/ --overwrite

Running the benchmarks

To run the ExecuteStateTransition benchmark:

bazel test //beacon-chain/core/state:go_default_test --test_filter=BenchmarkExecuteStateTransition_FullBlock --test_arg=-test.bench=BenchmarkExecuteStateTransition_FullBlock

To run the ExecuteStateTransition (with cache) benchmark:

bazel test //beacon-chain/core/state:go_default_test --test_filter=BenchmarkExecuteStateTransition_WithCache --test_arg=-test.bench=BenchmarkExecuteStateTransition_WithCache

To run the ProcessEpoch benchmark:

bazel test //beacon-chain/core/state:go_default_test --test_filter=BenchmarkProcessEpoch_2FullEpochs --test_arg=-test.bench=BenchmarkProcessEpoch_2FullEpochs

To run the HashTreeRoot benchmark:

bazel test //beacon-chain/core/state:go_default_test --test_filter=BenchmarkHashTreeRoot_FullState --test_arg=-test.bench=BenchmarkHashTreeRoot_FullState

To run the HashTreeRootState benchmark:

bazel test //beacon-chain/core/state:go_default_test --test_filter=BenchmarkHashTreeRootState_FullState --test_arg=-test.bench=BenchmarkHashTreeRootState_FullState

Extra flags needed to benchmark properly:

--nocache_test_results --test_arg=-test.v --test_timeout=2000 --test_arg=-test.cpuprofile=/tmp/cpu.profile --test_arg=-test.memprofile=/tmp/mem.profile --test_output=streamed

Current Results as of January 2020

BenchmarkExecuteStateTransition_FullBlock-4           20	  2031438030 ns/op
BenchmarkExecuteStateTransition_WithCache-4   	      20	  1857290454 ns/op
BenchmarkHashTreeRoot_FullState-4   	              50	   297655834 ns/op
BenchmarkHashTreeRootState_FullState-4                50           155535883 ns/op

Documentation

Overview

Package benchutil contains useful helpers for pregenerating filled data structures such as blocks/states for benchmarks.

Index

• Variables
• func PreGenFullBlock() (*ethpb.SignedBeaconBlock, error)
• func PreGenState1Epoch() (iface.BeaconState, error)
• func PreGenState2FullEpochs() (iface.BeaconState, error)
• func SetBenchmarkConfig()

Variables

var AttestationsPerEpoch = uint64(128)

AttestationsPerEpoch represents the requested amount attestations in an epoch. This affects the amount of attestations in a fully attested for block and the amount of attestations in the state per epoch, so a full 2 epochs should result in twice this amount of attestations in the state. Default is 128.

var BState1EpochFileName = fmt.Sprintf("bState1Epoch-%dAtts-%dVals.ssz", AttestationsPerEpoch, ValidatorCount)

BState1EpochFileName is the generated beacon state after 1 skipped epoch file name.

var BState2EpochFileName = fmt.Sprintf("bState2Epochs-%dAtts-%dVals.ssz", AttestationsPerEpoch, ValidatorCount)

BState2EpochFileName is the generated beacon state after 2 full epochs file name.

var FullBlockFileName = fmt.Sprintf("fullBlock-%dAtts-%dVals.ssz", AttestationsPerEpoch, ValidatorCount)

FullBlockFileName is the generated full block file name.

var GenesisFileName = fmt.Sprintf("bStateGenesis-%dAtts-%dVals.ssz", AttestationsPerEpoch, ValidatorCount)

GenesisFileName is the generated genesis beacon state file name.

var ValidatorCount = uint64(16384)

ValidatorCount is for declaring how many validators the benchmarks will be performed with. Default is 16384 or 524K ETH staked.

Functions

func PreGenFullBlock

func PreGenFullBlock() (*ethpb.SignedBeaconBlock, error)

PreGenFullBlock unmarshals the pre-generated signed beacon block containing an epochs worth of attestations and returns it.

func PreGenState1Epoch

func PreGenState1Epoch() (iface.BeaconState, error)

PreGenState1Epoch unmarshals the pre-generated beacon state after 1 epoch of block processing and returns it.

func PreGenState2FullEpochs

func PreGenState2FullEpochs() (iface.BeaconState, error)

PreGenState2FullEpochs unmarshals the pre-generated beacon state after 2 epoch of full block processing and returns it.

func SetBenchmarkConfig

func SetBenchmarkConfig()

SetBenchmarkConfig changes the beacon config to match the requested amount of attestations set to AttestationsPerEpoch.