beacon

package
v1.13.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2024 License: GPL-3.0 Imports: 40 Imported by: 0

README

Dora: beaconchain indexer

Welcome to the documentation for the indexer package of Dora the Beacon Chain Explorer. This package is responsible for indexing the beacon chain, including blocks, duties, and the current chain state. It consumes events from one or more pool.consensus.Client clients, updates database indexes, and provides functions to gain insights into the current network state.

Overview

The indexer package is divided into several components:

  • Initialization routine
  • Input processor
  • Caching subsystems
    • Block cache
    • Epoch cache
    • Fork cache
  • Processing routines
    • Finalization
    • Pruning
    • Synchronization

Initialization Routine

The initialization routine prefills all caching subsystems with the latest preserved state from the database. It uses the currently finalized epoch as a boundary for loading entities, so only objects between the current wallclock epoch and the current finalized epoch are restored. Entities older than inMemoryEpochs are directly pruned to save memory. This process runs before the indexer starts to receive events from the attached clients. For networks with large unfinality periods, this might take a few seconds to minutes.

Input Processor

The input processor consumes events from an underlying consensus pool client and feeds the caching subsystems with new data. It:

  • Attaches to "block" and "head" events.
  • Processes each received block, loading its block header and body.
  • Ensures the completeness of the block cache by backfilling parent blocks.
  • Dumps fully received blocks into the unfinalized blocks table for restoration after restart.
  • Tracks the latest head of the client and detects chain reorgs.

An instance of the input processor is created for every client attached to the indexer via indexer.AddClient.

Caching Subsystems

Block Cache

The block cache subsystem keeps an overview of all unfinalized blocks in the network. It:

  • Provides block accessors by various block properties.
  • Checks block relationships and calculates distances between blocks.
  • Might hold pruned blocks (without block bodies), but bodies can be restored on purpose.
Epoch Cache

The epoch cache subsystem maintains an overview of unfinalized epoch stats. It:

  • Holds duty and epoch statistics for each epoch.
  • Identifies epoch stats by the epoch's dependent root and number.
  • Might hold pruned epoch stats, keeping only proposer + sync committee duties and aggregations in memory
  • Can restores full epoch stats from the database, which is an expensive operation.
  • Can pre-calculate the next epoch stats, however this might not be accurate all the time.
Fork Cache

The fork cache subsystem keeps an overview of all unfinalized forks in the network. It:

  • Identifies forks by the base block root and leaf block root.
  • Numbers forks with an ascending internal counter.
  • Holds the fork detection system to identify current fork IDs and detect possible forks.
  • Uses forkId as a placeholder for unknown or canonical finalized forks.

Processing Routines

Finalization Routine

The finalization routine processes unfinalized blocks from the block cache after finalization. It:

  • Writes blocks, epoch aggregations, and child objects to the database.
  • Stores orphaned blocks in the orphaned blocks table.
  • Wipes processed blocks and epoch stats from caches.
Pruning Routine

The pruning routine pre-finalizes old unfinalized epochs and prunes memory-heavy data from cache. It:

  • Is controlled by the inMemoryEpochs setting.
  • Generates non-final epoch voting aggregations.
  • Writes blocks and child objects to the database.
  • Stores pruned epoch aggregations in cache and the unfinalized epochs table for restoration.
Synchronization Routine

The synchronization routine processes all missed historic data. It:

  • Ensures each epoch is properly indexed.
  • Loads canonical blocks and dependent states from a ready node.
  • Computes epoch aggregations and writes them, along with canonical blocks and child objects, to the database.
  • Is triggered by failed finalization or the initialization routine.

Documentation

Index

Constants

View Source 
const EtherGweiFactor = 1_000_000_000
View Source 
const FarFutureEpoch = phase0.Epoch(math.MaxUint64)

Variables

This section is empty.

Functions

func LoadBeaconBlock 

func LoadBeaconBlock(ctx context.Context, client *Client, root phase0.Root) (*spec.VersionedSignedBeaconBlock, error)

LoadBeaconBlock loads the block body from the RPC client.

func LoadBeaconHeader 

func LoadBeaconHeader(ctx context.Context, client *Client, root phase0.Root) (*phase0.SignedBeaconBlockHeader, error)

LoadBeaconHeader loads the block header from the client.

func LoadBeaconHeaderBySlot 

func LoadBeaconHeaderBySlot(ctx context.Context, client *Client, slot phase0.Slot) (*phase0.SignedBeaconBlockHeader, phase0.Root, bool, error)

LoadBeaconHeaderBySlot loads the block header with given slot number from the client.

func LoadBeaconState 

func LoadBeaconState(ctx context.Context, client *Client, root phase0.Root) (*spec.VersionedBeaconState, error)

LoadBeaconState loads the beacon state from the client.

Types

type Block 

type Block struct {
	Root phase0.Root
	Slot phase0.Slot
	// contains filtered or unexported fields
}

Block represents a beacon block.

func (*Block) AwaitBlock 

func (block *Block) AwaitBlock(ctx context.Context, timeout time.Duration) *spec.VersionedSignedBeaconBlock

AwaitBlock waits for the versioned signed beacon block of this block to be available.

func (*Block) AwaitHeader 

func (block *Block) AwaitHeader(ctx context.Context, timeout time.Duration) *phase0.SignedBeaconBlockHeader

AwaitHeader waits for the signed beacon block header of this block to be available.

func (*Block) EnsureBlock 

func (block *Block) EnsureBlock(loadBlock func() (*spec.VersionedSignedBeaconBlock, error)) (bool, error)

EnsureBlock ensures that the versioned signed beacon block of this block is available.

func (*Block) EnsureHeader 

func (block *Block) EnsureHeader(loadHeader func() (*phase0.SignedBeaconBlockHeader, error)) error

EnsureHeader ensures that the signed beacon block header of this block is available.

func (*Block) GetBlock 

func (block *Block) GetBlock() *spec.VersionedSignedBeaconBlock

GetBlock returns the versioned signed beacon block of this block.

func (*Block) GetBlockIndex 

func (block *Block) GetBlockIndex() *BlockBodyIndex

GetBlockIndex returns the block index of this block.

func (*Block) GetDbBlock 

func (block *Block) GetDbBlock(indexer *Indexer) *dbtypes.Slot

GetDbBlock returns the database representation of this block.

func (*Block) GetDbConsolidationRequests 

func (block *Block) GetDbConsolidationRequests(indexer *Indexer) []*dbtypes.ConsolidationRequest

GetDbConsolidationRequests returns the database representation of the consolidation requests in this block.

func (*Block) GetDbDeposits 

func (block *Block) GetDbDeposits(indexer *Indexer, depositIndex *uint64) []*dbtypes.Deposit

GetDbDeposits returns the database representation of the deposits in this block.

func (*Block) GetDbSlashings 

func (block *Block) GetDbSlashings(indexer *Indexer) []*dbtypes.Slashing

GetDbSlashings returns the database representation of the slashings in this block.

func (*Block) GetDbVoluntaryExits 

func (block *Block) GetDbVoluntaryExits(indexer *Indexer) []*dbtypes.VoluntaryExit

GetDbVoluntaryExits returns the database representation of the voluntary exits in this block.

func (*Block) GetDbWithdrawalRequests added in v1.12.0 

func (block *Block) GetDbWithdrawalRequests(indexer *Indexer) []*dbtypes.WithdrawalRequest

GetDbWithdrawalRequests returns the database representation of the withdrawal requests in this block.

func (*Block) GetForkId 

func (block *Block) GetForkId() ForkKey

GetForkId returns the fork ID of this block.

func (*Block) GetHeader 

func (block *Block) GetHeader() *phase0.SignedBeaconBlockHeader

GetHeader returns the signed beacon block header of this block.

func (*Block) GetParentRoot 

func (block *Block) GetParentRoot() *phase0.Root

GetParentRoot returns the parent root of this block.

func (*Block) GetSeenBy 

func (block *Block) GetSeenBy() []*Client

GetSeenBy returns a list of clients that have seen this block.

func (*Block) SetBlock 

func (block *Block) SetBlock(body *spec.VersionedSignedBeaconBlock)

SetBlock sets the versioned signed beacon block of this block.

func (*Block) SetHeader 

func (block *Block) SetHeader(header *phase0.SignedBeaconBlockHeader)

SetHeader sets the signed beacon block header of this block.

func (*Block) SetSeenBy 

func (block *Block) SetSeenBy(client *Client)

SetSeenBy sets the client that has seen this block.

type BlockBodyIndex 

type BlockBodyIndex struct {
	Graffiti           [32]byte
	ExecutionExtraData []byte
	ExecutionHash      phase0.Hash32
	ExecutionNumber    uint64
}

BlockBodyIndex holds important block propoerties that are used as index for cache lookups. this structure should be preserved after pruning, so the block is still identifiable.

type CacheDebugMapSize added in v1.13.0 

type CacheDebugMapSize struct {
	Length int
	Size   int64
}

type CacheDebugStats added in v1.13.0 

type CacheDebugStats struct {
	BlockCache struct {
		SlotMap      CacheDebugMapSize
		RootMap      CacheDebugMapSize
		ParentMap    CacheDebugMapSize
		ExecBlockMap CacheDebugMapSize
		BlockHeader  uint64
		BlockBodies  uint64
		BlockIndexes uint64
		BlockSize    uint64
	}
	EpochCache struct {
		StatsMap       CacheDebugMapSize
		StateMap       CacheDebugMapSize
		StatsFull      uint64
		StatsPrecalc   uint64
		StatsPruned    uint64
		StateLoaded    uint64
		VotesCacheLen  uint64
		VotesCacheHit  uint64
		VotesCacheMiss uint64
	}
	ForkCache struct {
		ForkMap           CacheDebugMapSize
		ParentIdCacheLen  uint64
		ParentIdCacheHit  uint64
		ParentIdCacheMiss uint64
	}
	ValidatorCache struct {
		Validators        uint64
		ValidatorDiffs    uint64
		ValidatorData     uint64
		ValidatorActivity uint64
		PubkeyMap         CacheDebugMapSize
	}
}

type ChainHead 

type ChainHead struct {
	HeadBlock             *Block      // The head block of the chain.
	AggregatedHeadVotes   phase0.Gwei // The aggregated votes of the last 2 epochs for the head block.
	PerEpochVotingPercent []float64   // The voting percentage in the last epochs (ascendeing order).
}

ChainHead represents a head block of the chain.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client represents a consensus pool client that should be used for indexing beacon blocks.

func (*Client) GetClient 

func (c *Client) GetClient() *consensus.Client

func (*Client) GetIndex 

func (c *Client) GetIndex() uint16

func (*Client) GetPriority 

func (c *Client) GetPriority() int

type EpochStats 

type EpochStats struct {
	// contains filtered or unexported fields
}

EpochStats holds the epoch-specific information based on the underlying dependent beacon state.

func (*EpochStats) GetDbEpoch 

func (es *EpochStats) GetDbEpoch(indexer *Indexer, headBlock *Block) *dbtypes.Epoch

GetDbEpoch returns the database Epoch representaion for the EpochStats.

func (*EpochStats) GetDependentRoot added in v1.12.1 

func (es *EpochStats) GetDependentRoot() phase0.Root

func (*EpochStats) GetEpoch 

func (es *EpochStats) GetEpoch() phase0.Epoch

func (*EpochStats) GetEpochVotes 

func (es *EpochStats) GetEpochVotes(indexer *Indexer, headBlock *Block) *EpochVotes

GetEpochVotes aggregates & returns the EpochVotes for the EpochStats.

func (*EpochStats) GetOrLoadValues 

func (es *EpochStats) GetOrLoadValues(indexer *Indexer, withPrecalc bool, keepInCache bool) *EpochStatsValues

GetOrLoadValues returns the EpochStats values, loading them from the database if necessary.

func (*EpochStats) GetValues 

func (es *EpochStats) GetValues(withPrecalc bool) *EpochStatsValues

GetValues returns the EpochStats values.

type EpochStatsPacked 

type EpochStatsPacked struct {
	ActiveValidators    []EpochStatsPackedValidator
	SyncCommitteeDuties []phase0.ValidatorIndex
	RandaoMix           phase0.Hash32
	NextRandaoMix       phase0.Hash32
	TotalBalance        phase0.Gwei
	ActiveBalance       phase0.Gwei
	FirstDepositIndex   uint64
}

EpochStatsPacked holds the packed values for the epoch-specific information.

type EpochStatsPackedValidator 

type EpochStatsPackedValidator struct {
	ValidatorIndexOffset uint32 // offset to the previous index in the list (this is smaller than storing the full validator index)
	EffectiveBalanceEth  uint16 // effective balance in full ETH
}

EpochStatsPackedValidator holds the packed values for an active validator.

type EpochStatsValues 

type EpochStatsValues struct {
	RandaoMix           phase0.Hash32
	NextRandaoMix       phase0.Hash32
	ActiveIndices       []phase0.ValidatorIndex
	EffectiveBalances   []uint16
	ProposerDuties      []phase0.ValidatorIndex
	AttesterDuties      [][][]duties.ActiveIndiceIndex
	SyncCommitteeDuties []phase0.ValidatorIndex
	ActiveValidators    uint64
	TotalBalance        phase0.Gwei
	ActiveBalance       phase0.Gwei
	EffectiveBalance    phase0.Gwei
	FirstDepositIndex   uint64
}

EpochStatsValues holds the values for the epoch-specific information.

func (*EpochStatsValues) GetEffectiveBalance 

func (v *EpochStatsValues) GetEffectiveBalance(index duties.ActiveIndiceIndex) phase0.Gwei

GetEffectiveBalance returns the effective balance for the given active validator indice.

type EpochVotes 

type EpochVotes struct {
	CurrentEpoch struct {
		TargetVoteAmount phase0.Gwei
		HeadVoteAmount   phase0.Gwei
		TotalVoteAmount  phase0.Gwei
	}
	NextEpoch struct {
		TargetVoteAmount phase0.Gwei
		HeadVoteAmount   phase0.Gwei
		TotalVoteAmount  phase0.Gwei
	}
	TargetVotePercent float64
	HeadVotePercent   float64
	TotalVotePercent  float64
	AmountIsCount     bool
}

EpochVotes represents the aggregated votes for an epoch.

type Fork 

type Fork struct {
	// contains filtered or unexported fields
}

Fork represents a fork in the beacon chain.

func (*Fork) GetBase 

func (fork *Fork) GetBase() (phase0.Slot, phase0.Root)

func (*Fork) GetLeaf 

func (fork *Fork) GetLeaf() (phase0.Slot, phase0.Root)

func (*Fork) GetParent 

func (fork *Fork) GetParent() ForkKey

type ForkHead 

type ForkHead struct {
	ForkId ForkKey
	Fork   *Fork
	Block  *Block
}

ForkHead represents a fork head with its ID, fork, and block.

type ForkKey 

type ForkKey uint64

ForkKey represents a key used for indexing forks.

type Indexer 

type Indexer struct {
	// contains filtered or unexported fields
}

Indexer is responsible for indexing the ethereum beacon chain.

func NewIndexer 

func NewIndexer(logger logrus.FieldLogger, consensusPool *consensus.Pool) *Indexer

NewIndexer creates a new instance of the Indexer.

func (*Indexer) AddClient 

func (indexer *Indexer) AddClient(index uint16, client *consensus.Client, priority int, archive bool, skipValidators bool) *Client

AddClient adds a new consensus pool client to the indexer.

func (*Indexer) GetActivityHistoryLength added in v1.13.0 

func (indexer *Indexer) GetActivityHistoryLength() uint16

func (*Indexer) GetAllClients 

func (indexer *Indexer) GetAllClients() []*Client

GetAllClients returns a slice of all clients in the indexer.

func (*Indexer) GetBlockByParentRoot 

func (indexer *Indexer) GetBlockByParentRoot(blockRoot phase0.Root) []*Block

GetBlockByParentRoot returns a slice of blocks with the given parent root.

func (*Indexer) GetBlockByRoot 

func (indexer *Indexer) GetBlockByRoot(blockRoot phase0.Root) *Block

GetBlockByRoot returns the block with the given block root.

func (*Indexer) GetBlockByStateRoot 

func (indexer *Indexer) GetBlockByStateRoot(stateRoot phase0.Root) *Block

GetBlockByStateRoot returns the block with the given state root.

func (*Indexer) GetBlockCacheState 

func (indexer *Indexer) GetBlockCacheState() (finalizedEpoch phase0.Epoch, prunedEpoch phase0.Epoch)

GetBlockCacheState returns the state of the block cache, including the last finalized epoch and the last pruned epoch. this represents the internal cache state and might be behind the actual finalization checkpoint.

func (*Indexer) GetBlockDistance 

func (indexer *Indexer) GetBlockDistance(baseRoot phase0.Root, headRoot phase0.Root) (bool, uint64)

GetBlockDistance returns whether the base root is in the canonical chain defined by the head root and the distance between both blocks.

func (*Indexer) GetBlocksByExecutionBlockHash 

func (indexer *Indexer) GetBlocksByExecutionBlockHash(blockHash phase0.Hash32) []*Block

GetBlocksByExecutionBlockHash returns a slice of blocks with the given execution block hash.

func (*Indexer) GetBlocksByExecutionBlockNumber 

func (indexer *Indexer) GetBlocksByExecutionBlockNumber(blockNumber uint64) []*Block

GetBlocksByExecutionBlockNumber returns a slice of blocks with the given execution block number.

func (*Indexer) GetBlocksBySlot 

func (indexer *Indexer) GetBlocksBySlot(slot phase0.Slot) []*Block

GetBlocksBySlot returns a slice of blocks with the given slot.

func (*Indexer) GetCacheDebugStats added in v1.13.0 

func (indexer *Indexer) GetCacheDebugStats() *CacheDebugStats

func (*Indexer) GetCanonicalHead 

func (indexer *Indexer) GetCanonicalHead(overrideForkId *ForkKey) *Block

GetCanonicalHead returns the canonical head block of the chain.

func (*Indexer) GetChainHeads 

func (indexer *Indexer) GetChainHeads() []*ChainHead

GetChainHeads returns the chain heads sorted by voting percentages.

func (*Indexer) GetEpochStats 

func (indexer *Indexer) GetEpochStats(epoch phase0.Epoch, overrideForkId *ForkKey) *EpochStats

GetEpochStats returns the epoch stats for the given epoch and optional fork ID override.

func (*Indexer) GetEpochValidator added in v1.13.0 

func (indexer *Indexer) GetEpochValidator(validatorIndex phase0.ValidatorIndex, epoch phase0.Epoch, overrideForkId *ForkKey, withBalances bool) *v1.Validator

GetEpochValidator returns the full validator set for a given epoch, including balances and validator status. If an overrideForkId is provided, the validator set for the fork is returned.

func (*Indexer) GetEpochValidatorSet added in v1.13.0 

func (indexer *Indexer) GetEpochValidatorSet(epoch phase0.Epoch, overrideForkId *ForkKey, withBalances bool) []*v1.Validator

GetEpochValidatorSet returns the full validator set for a given epoch, including balances and validator status. If an overrideForkId is provided, the validator set for the fork is returned.

func (*Indexer) GetForkHeads 

func (indexer *Indexer) GetForkHeads() []*ForkHead

GetForkHeads returns a slice of fork heads in the indexer.

func (*Indexer) GetOrphanedBlockByRoot 

func (indexer *Indexer) GetOrphanedBlockByRoot(blockRoot phase0.Root) (*Block, error)

GetOrphanedBlockByRoot returns the orphaned block with the given block root.

func (*Indexer) GetParentForkIds added in v1.12.1 

func (indexer *Indexer) GetParentForkIds(forkId ForkKey) []ForkKey

GetParentForkIds returns the parent fork ids of the given fork.

func (*Indexer) GetReadyClient 

func (indexer *Indexer) GetReadyClient(preferArchive bool) *Client

GetReadyClient returns a single client that is on the finalized chain and preference for archive clients.

func (*Indexer) GetReadyClientByBlockRoot 

func (indexer *Indexer) GetReadyClientByBlockRoot(blockRoot phase0.Root, preferArchive bool) *Client

GetReadyClientByBlockRoot returns a single client that is ready for requests for the chain including the block root and preference for archive clients.

func (*Indexer) GetReadyClients 

func (indexer *Indexer) GetReadyClients(preferArchive bool) []*Client

GetReadyClients returns a slice of clients that are on the finalized chain and preference for archive clients.

func (*Indexer) GetReadyClientsByBlockRoot 

func (indexer *Indexer) GetReadyClientsByBlockRoot(blockRoot phase0.Root, preferArchive bool) []*Client

GetReadyClientsByBlockRoot returns a slice of clients that are ready for requests for the chain including the block root and preference for archive clients.

func (*Indexer) GetReadyClientsByCheckpoint 

func (indexer *Indexer) GetReadyClientsByCheckpoint(finalizedRoot phase0.Root, preferArchive bool) []*Client

GetReadyClientsByCheckpoint returns a slice of clients that are ready for processing based on the finalized root and preference for archive clients.

func (*Indexer) GetSynchronizerState added in v1.12.1 

func (indexer *Indexer) GetSynchronizerState() (running bool, syncHead phase0.Epoch)

GetSynchronizerState returns the state of the synchronizer, including whether it is running and the current epoch.

func (*Indexer) GetValidatorActivity added in v1.13.0 

func (indexer *Indexer) GetValidatorActivity(validatorIndex phase0.ValidatorIndex) ([]ValidatorActivity, phase0.Epoch)

GetValidatorActivity returns the validator activity for a given validator index.

func (*Indexer) GetValidatorByIndex added in v1.13.0 

func (indexer *Indexer) GetValidatorByIndex(index phase0.ValidatorIndex, overrideForkId *ForkKey) *phase0.Validator

GetValidatorByIndex returns the validator by index for a given forkId.

func (*Indexer) GetValidatorIndexByPubkey added in v1.13.0 

func (indexer *Indexer) GetValidatorIndexByPubkey(pubkey phase0.BLSPubKey) (phase0.ValidatorIndex, bool)

GetValidatorIndexByPubkey returns the validator index for a given pubkey.

func (*Indexer) GetValidatorSet added in v1.13.0 

func (indexer *Indexer) GetValidatorSet(overrideForkId *ForkKey) []*phase0.Validator

GetValidatorSet returns the most recent basic validator set, excluding balances and validator status. If an overrideForkId is provided, the validator set for the fork is returned.

func (*Indexer) IsCanonicalBlock 

func (indexer *Indexer) IsCanonicalBlock(block *Block, overrideForkId *ForkKey) bool

func (*Indexer) StartIndexer 

func (indexer *Indexer) StartIndexer()

StartIndexer starts the indexing process.

type ValidatorActivity added in v1.13.0 

type ValidatorActivity struct {
	VoteBlock *Block // the block where the vote was included
	VoteDelay uint16 // the inclusion delay of the vote in slots
}

ValidatorActivity represents a validator's activity in an epoch. entry size: 18 bytes (10 bytes data + 8 bytes pointer) max. entries per validator: 3-8 (inMemoryEpochs) total memory consumption:

  • 10k active validators: min: 10000 * 18 * 3 = 540kB = 0.54MB max: 10000 * 18 * 8 = 1440kB = 1.44MB
  • 100k active validators: min: 100000 * 18 * 3 = 5400kB = 5.4MB max: 100000 * 18 * 8 = 14400kB = 14.4MB
  • 1M active validators: min: 1000000 * 18 * 3 = 54000kB = 54MB max: 1000000 * 18 * 8 = 144000kB = 144MB

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.