indexer

README

Indexer Framework

Defining an Indexer

Indexer implementations should be registered with the indexer.Register function with a unique type name. Indexers take the configuration options defined by indexer.Config which defines a common set of configuration options as well as indexer-specific options under the config sub-key. Indexers do not need to manage the common filtering options specified in Config - the indexer manager will manage these for the indexer. Indexer implementations just need to return a correct InitResult response.

Integrating the Indexer Manager

The indexer manager should be used for managing all indexers and should be integrated directly with applications wishing to support indexing. The StartManager function is used to start the manager. The configuration options for the manager and all indexer targets should be passed as the ManagerOptions.Config field and should match the json structure of ManagerConfig. An example configuration section in app.toml might look like this:

[indexer.target.postgres]
type = "postgres"
config.database_url = "postgres://user:password@localhost:5432/dbname"

Documentation

Index

• func Register(indexerType string, initFunc InitFunc)
• func StartManager(opts ManagerOptions) (appdata.Listener, error)
• type Config
• type InitFunc
• type InitParams
• type InitResult
• type ManagerConfig
• type ManagerOptions

Functions

func Register

func Register(indexerType string, initFunc InitFunc)

Register registers an indexer type with the given initialization function.

func StartManager

func StartManager(opts ManagerOptions) (appdata.Listener, error)

StartManager starts the indexer manager with the given options. The state machine should write all relevant app data to the returned listener.

Types

type Config

type Config struct {
	// Type is the name of the indexer type as registered with Register.
	Type string `json:"type"`

	// Config are the indexer specific config options specified by the user.
	Config map[string]interface{} `json:"config"`

	// ExcludeState specifies that the indexer will not receive state updates.
	ExcludeState bool `json:"exclude_state"`

	// ExcludeEvents specifies that the indexer will not receive events.
	ExcludeEvents bool `json:"exclude_events"`

	// ExcludeTxs specifies that the indexer will not receive transaction's.
	ExcludeTxs bool `json:"exclude_txs"`

	// ExcludeBlockHeaders specifies that the indexer will not receive block headers,
	// although it will still receive StartBlock and Commit callbacks, just without
	// the header data.
	ExcludeBlockHeaders bool `json:"exclude_block_headers"`

	// IncludeModules specifies a list of modules whose state the indexer will
	// receive state updates for.
	// Only one of include or exclude modules should be specified.
	IncludeModules []string `json:"include_modules"`

	// ExcludeModules specifies a list of modules whose state the indexer will not
	// receive state updates for.
	// Only one of include or exclude modules should be specified.
	ExcludeModules []string `json:"exclude_modules"`
}

Config species the configuration passed to an indexer initialization function. It includes both common configuration options related to include or excluding parts of the data stream as well as indexer specific options under the config subsection.

NOTE: it is an error for an indexer to change its common options, such as adding or removing indexed modules, after the indexer has been initialized because this could result in an inconsistent state.

type InitFunc

type InitFunc = func(InitParams) (InitResult, error)

type InitParams

type InitParams struct {
	// Config is the indexer config.
	Config Config

	// Context is the context that the indexer should use to listen for a shutdown signal via Context.Done(). Other
	// parameters may also be passed through context from the app if necessary. It is expected to be non-nil.
	Context context.Context

	// Logger is a logger the indexer can use to write log messages. It may be nil if the indexer does not need
	// to write logs.
	Logger logutil.Logger

	// AddressCodec is the address codec that the indexer can use to encode and decode addresses. It is
	// expected to be non-nil.
	AddressCodec addressutil.AddressCodec
}

InitParams is the input to the indexer initialization function.

type InitResult

type InitResult struct {
	// Listener is the indexer's app data listener.
	Listener appdata.Listener

	// View is a view of indexed data that indexers can provide. It is optional and may be nil.
	// If it is provided it can be used for automated testing and other purposes.
	// At indexer start-up, the block number returned by the view will be used to determine the
	// starting block for the indexer. If the block number is 0, the indexer manager will attempt
	// to perform a catch-up sync of state. Historical events will not be replayed, but an accurate
	// representation of the current state at the height at which indexing began can be reproduced.
	// If the block number is non-zero but does not match the current chain height, a runtime error
	// will occur because this is an unsafe condition that indicates lost data.
	View view.AppData
}

InitResult is the indexer initialization result and includes the indexer's listener implementation.

type ManagerConfig

type ManagerConfig struct {
	// Target is a map of named indexer targets to their configuration.
	Target map[string]Config
}

ManagerConfig is the configuration of the indexer manager and contains the configuration for each indexer target.

type ManagerOptions

type ManagerOptions struct {
	// Config is the user configuration for all indexing. It should generally be an instance of map[string]interface{}
	// and match the json structure of ManagerConfig. The manager will attempt to convert it to ManagerConfig.
	Config interface{}

	// Resolver is the decoder resolver that will be used to decode the data. It is required.
	Resolver decoding.DecoderResolver

	// SyncSource is a representation of the current state of key-value data to be used in a catch-up sync.
	// Catch-up syncs will be performed at initialization when necessary. SyncSource is optional but if
	// it is omitted, indexers will only be able to start indexing state from genesis.
	SyncSource decoding.SyncSource

	// Logger is the logger that indexers can use to write logs. It is optional.
	Logger logutil.Logger

	// Context is the context that indexers should use for shutdown signals via Context.Done(). It can also
	// be used to pass down other parameters to indexers if necessary. If it is omitted, context.Background
	// will be used.
	Context context.Context

	// AddressCodec is the address codec that indexers can use to encode and decode addresses. It should always be
	// provided, but if it is omitted, the indexer manager will use a default codec which encodes and decodes addresses
	// as hex strings.
	AddressCodec addressutil.AddressCodec
}

ManagerOptions are the options for starting the indexer manager.