backfill

Documentation

Overview

Package backfill is used to get logs from previous blocks

Index

• Constants
• func BlockTimesInRange(ctx context.Context, backend ScribeBackend, startHeight uint64, ...) (*immutable.Map[uint64, uint64], error)
• func GetLogsInRange(ctx context.Context, backend ScribeBackend, startHeight uint64, ...) (*immutable.List[*[]types.Log], error)
• func LogEvent(level logLevel, msg string, logData LogData)
• type ChainBackfiller
  • func NewChainBackfiller(chainID uint32, eventDB db.EventDB, client []ScribeBackend, ...) (*ChainBackfiller, error)
  • func (c ChainBackfiller) Backfill(ctx context.Context, onlyOneBlock *uint64) error
• type ContractBackfiller
  • func NewContractBackfiller(chainConfig config.ChainConfig, address string, eventDB db.EventDB, ...) (*ContractBackfiller, error)
  • func (c *ContractBackfiller) Backfill(ctx context.Context, givenStart uint64, endHeight uint64) error
• type LogData
• type LogFilterer
• type LogInfo
• type RangeFilter
  • func NewRangeFilter(address ethCommon.Address, backend ScribeBackend, ...) *RangeFilter
  • func (f *RangeFilter) Done() bool
  • func (f *RangeFilter) Drain(ctx context.Context) (filteredLogs []types.Log, err error)
  • func (f *RangeFilter) FilterLogs(ctx context.Context, chunk *util.Chunk) (*LogInfo, error)
  • func (f *RangeFilter) GetLogChan() chan *LogInfo
  • func (f *RangeFilter) Start(ctx context.Context) error
• type ScribeBackend
  • func DialBackend(ctx context.Context, url string) (ScribeBackend, error)
• type ScribeBackfiller
  • func NewScribeBackfiller(eventDB db.EventDB, clientsMap map[uint32][]ScribeBackend, ...) (*ScribeBackfiller, error)
  • func (s ScribeBackfiller) Backfill(ctx context.Context) error

Constants

const (
	// InfoLevel prints a log at the info level.
	InfoLevel logLevel = iota
	// WarnLevel prints a log at the warn level.
	WarnLevel
	// ErrorLevel prints a log at the error level.
	ErrorLevel
)

Functions

func BlockTimesInRange

func BlockTimesInRange(ctx context.Context, backend ScribeBackend, startHeight uint64, endHeight uint64) (*immutable.Map[uint64, uint64], error)

BlockTimesInRange gets all block times in a range with a single batch request in successful cases an immutable map is returned of [height->time], otherwise an error is returned.

func GetLogsInRange

func GetLogsInRange(ctx context.Context, backend ScribeBackend, startHeight uint64, endHeight uint64, subChunkSize uint64, contractAddress common.Address, expectedChainID uint64) (*immutable.List[*[]types.Log], error)

GetLogsInRange gets all logs in a range with a single batch request in successful cases an immutable list is returned, otherwise an error is returned.

func LogEvent

func LogEvent(level logLevel, msg string, logData LogData)

LogEvent formats and logs an event.

Types

type ChainBackfiller

type ChainBackfiller struct {
	// contains filtered or unexported fields
}

ChainBackfiller is a backfiller that fetches logs for a chain. It aggregates logs from a slice of ContractBackfillers.

func NewChainBackfiller

func NewChainBackfiller(chainID uint32, eventDB db.EventDB, client []ScribeBackend, chainConfig config.ChainConfig) (*ChainBackfiller, error)

NewChainBackfiller creates a new backfiller for a chain. This is done by passing through all the function parameters into the ChainBackfiller struct, as well as iterating through all the contracts in the chain config and creating ContractBackfillers for each contract.

func (ChainBackfiller) Backfill

func (c ChainBackfiller) Backfill(ctx context.Context, onlyOneBlock *uint64) error

Backfill iterates over each contract backfiller and calls Backfill concurrently on each one. If `onlyOneBlock` is true, the backfiller will only backfill the block at `currentBlock`.

type ContractBackfiller

type ContractBackfiller struct {
	// contains filtered or unexported fields
}

ContractBackfiller is a backfiller that fetches logs for a specific contract.

func NewContractBackfiller

func NewContractBackfiller(chainConfig config.ChainConfig, address string, eventDB db.EventDB, client []ScribeBackend) (*ContractBackfiller, error)

NewContractBackfiller creates a new backfiller for a contract.

func (*ContractBackfiller) Backfill

func (c *ContractBackfiller) Backfill(ctx context.Context, givenStart uint64, endHeight uint64) error

Backfill retrieves logs, receipts, and transactions for a contract from a given range and does so in the following manner. 1. Get logs for the contract in chunks of batch requests. 2. Iterate through each log's Tx Hash and performs the following

• Get the receipt for each log and store it and all of its logs.
• Get the transaction for each log and store it.

type LogData

type LogData map[string]interface{}

LogData holds all the data passed to LogEvent to be logged.

type LogFilterer

type LogFilterer interface {
	// FilterLogs executes a log filter operation, blocking during execution and
	// returning all the results in one batch.
	//
	// TODO(karalabe): Deprecate when the subscription one can return past data too.
	FilterLogs(ctx context.Context, query ethereum.FilterQuery) ([]types.Log, error)
}

LogFilterer is the interface for filtering logs.

type LogInfo

type LogInfo struct {
	// contains filtered or unexported fields
}

LogInfo is the log info.

type RangeFilter

type RangeFilter struct {
	// contains filtered or unexported fields
}

RangeFilter pre-fetches filter logs into a channel in deterministic order.

func NewRangeFilter

func NewRangeFilter(address ethCommon.Address, backend ScribeBackend, startBlock, endBlock *big.Int, chunkSize int, reverse bool, subChunkSize int, chainID uint32) *RangeFilter

NewRangeFilter creates a new filtering interface for a range of blocks. If reverse is not set, block heights are filtered from start->end.

func (*RangeFilter) Done

func (f *RangeFilter) Done() bool

Done returns a bool indicating whether the filtering operation is done.

func (*RangeFilter) Drain

func (f *RangeFilter) Drain(ctx context.Context) (filteredLogs []types.Log, err error)

Drain fetches all logs and concatenated them into a single slice.

func (*RangeFilter) FilterLogs

func (f *RangeFilter) FilterLogs(ctx context.Context, chunk *util.Chunk) (*LogInfo, error)

FilterLogs safely calls FilterLogs with the filtering implementing a backoff in the case of rate limiting and respecting context cancellation.

nolint:cyclop

func (*RangeFilter) GetLogChan

func (f *RangeFilter) GetLogChan() chan *LogInfo

GetLogChan returns a log chan with the logs filtered ahead to bufferSize. Iteration oder is only guaranteed with up to one consumer.

func (*RangeFilter) Start

func (f *RangeFilter) Start(ctx context.Context) error

Start starts the filtering process. If the context is canceled, logs will stop being filtered. This should be run on an independent goroutine.

type ScribeBackend

type ScribeBackend interface {
	// ChainID gets the chain id from the rpc server.
	ChainID(ctx context.Context) (*big.Int, error)
	// BlockByNumber retrieves a block from the database by number, caching it
	// (associated with its hash) if found.
	BlockByNumber(ctx context.Context, number *big.Int) (*types.Block, error)
	// TransactionByHash checks the pool of pending transactions in addition to the
	// blockchain. The isPending return value indicates whether the transaction has been
	// mined yet. Note that the transaction may not be part of the canonical chain even if
	// it's not pending.
	TransactionByHash(ctx context.Context, txHash common.Hash) (tx *types.Transaction, isPending bool, err error)
	// TransactionReceipt returns the receipt of a mined transaction. Note that the
	// transaction may not be included in the current canonical chain even if a receipt
	// exists.
	TransactionReceipt(ctx context.Context, txHash common.Hash) (*types.Receipt, error)
	// BlockNumber gets the latest block number.
	BlockNumber(ctx context.Context) (uint64, error)
	// FilterLogs executes a log filter operation, blocking during execution and
	// returning all the results in one batch.
	//
	// TODO(karalabe): Deprecate when the subscription one can return past data too.
	FilterLogs(ctx context.Context, query ethereum.FilterQuery) ([]types.Log, error)
	// HeaderByNumber returns the block header with the given block number.
	HeaderByNumber(ctx context.Context, number *big.Int) (*types.Header, error)
	// Batch batches multiple
	Batch(ctx context.Context, calls ...w3types.Caller) error
}

ScribeBackend is the set of functions that the scribe needs from a client.

func DialBackend

func DialBackend(ctx context.Context, url string) (ScribeBackend, error)

DialBackend returns a scribe backend.

type ScribeBackfiller

type ScribeBackfiller struct {

	// ChainBackfillers is a mapping of chain IDs -> chain backfillers.
	ChainBackfillers map[uint32]*ChainBackfiller
	// contains filtered or unexported fields
}

ScribeBackfiller is a backfiller that aggregates all backfilling from ChainBackfillers.

func NewScribeBackfiller

func NewScribeBackfiller(eventDB db.EventDB, clientsMap map[uint32][]ScribeBackend, config config.Config) (*ScribeBackfiller, error)

NewScribeBackfiller creates a new backfiller for the scribe.

func (ScribeBackfiller) Backfill

func (s ScribeBackfiller) Backfill(ctx context.Context) error

Backfill iterates over each chain backfiller and calls Backfill concurrently on each one.