engine

Documentation

Overview

Package engine provides a reference implementation of the provider.Interface in order to advertise the availability of a list of multihashes to indexer nodes such as "storetheindex". See: https://github.com/filecoin-project/storetheindex

The advertisements are published as a chan of diffs that signal the list of multihashes that are added or removed represented as an IPLD DAG. Walking the chain of advertisements would then provide the latest state of the total multihashes provided by the engine. The list of multihashes are paginated as a collection of interlinked chunks. For the complete advertisement IPLD schema, see:

• https://github.com/filecoin-project/storetheindex/blob/main/api/v0/ingest/schema/schema.ipldsch

The engine internally uses "go-legs" to sync the IPLD DAG of advertisements. See: https://github.com/filecoin-project/go-legs

Index

• type Engine
  • func New(ingestCfg config.Ingest, privKey crypto.PrivKey, dt dt.Manager, h host.Host, ...) (*Engine, error)
  • func NewFromConfig(cfg config.Config, dt dt.Manager, host host.Host, ds datastore.Batching, ...) (*Engine, error)
  • func (e *Engine) GetAdv(_ context.Context, adCid cid.Cid) (schema.Advertisement, error)
  • func (e *Engine) GetLatestAdv(ctx context.Context) (cid.Cid, schema.Advertisement, error)
  • func (e *Engine) NotifyPut(ctx context.Context, contextID []byte, metadata stiapi.Metadata) (cid.Cid, error)
  • func (e *Engine) NotifyRemove(ctx context.Context, contextID []byte) (cid.Cid, error)
  • func (e *Engine) Publish(ctx context.Context, adv schema.Advertisement) (cid.Cid, error)
  • func (e *Engine) PublishLatest(ctx context.Context) error
  • func (e *Engine) PublishLocal(ctx context.Context, adv schema.Advertisement) (cid.Cid, error)
  • func (e *Engine) RegisterCallback(cb provider.Callback)
  • func (e *Engine) Shutdown() error
  • func (e *Engine) Start(ctx context.Context) error
• type Option
  • func WithExtraGossipData(extraData []byte) Option

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is an implementation of the core reference provider interface

func New

func New(ingestCfg config.Ingest, privKey crypto.PrivKey, dt dt.Manager, h host.Host, ds datastore.Batching, retAddrs []string, options ...Option) (*Engine, error)

New creates a new index provider Engine as the default implementation of provider.Interface. It provides the ability to advertise the availability of a list of multihashes associated to a context ID as a chain of linked advertisements as defined by the indexer node protocol implemented by "storetheindex". Engine internally uses "go-legs", a protocol for propagating and synchronizing changes an IPLD DAG, to publish advertisements. See:

• https://github.com/filecoin-project/storetheindex
• https://github.com/filecoin-project/go-legs

Published advertisements are signed using the given private key. The retAddrs corresponds to the endpoints at which the data block associated to the advertised multihashes can be retrieved. Note that if no retAddrs is specified the listen addresses of the given libp2p host are used.

The engine also provides the ability to generate advertisements via Engine.NotifyPut and Engine.NotifyRemove as long as a provider.Callback is registered. See: provider.Callback, Engine.RegisterCallback.

The engine must be started via Engine.Start before use and discarded via Engine.Shutdown when no longer needed. See: Engine.Start, Engine.Shutdown.

func NewFromConfig

func NewFromConfig(cfg config.Config, dt dt.Manager, host host.Host, ds datastore.Batching, options ...Option) (*Engine, error)

NewFromConfig instantiates a new engine by using the given config.Config. The instantiation gets the private key via config.Identity, and uses RetrievalMultiaddrs in config.ProviderServer as retrieval addresses in advertisements.

See: engine.New .

func (*Engine) GetAdv

func (e *Engine) GetAdv(_ context.Context, adCid cid.Cid) (schema.Advertisement, error)

GetAdv gets the advertisement associated to the given cid c. The context is not used.

func (*Engine) GetLatestAdv

func (e *Engine) GetLatestAdv(ctx context.Context) (cid.Cid, schema.Advertisement, error)

GetLatestAdv gets the latest advertisement by the provider. If there are not previously published advertisements, then cid.Undef is returned as the advertisement CID.

func (*Engine) NotifyPut

func (e *Engine) NotifyPut(ctx context.Context, contextID []byte, metadata stiapi.Metadata) (cid.Cid, error)

NotifyPut publishes an advertisement that signals the list of multihashes associated to the given contextID is available by this provider with the given metadata. A provider.Callback is required, and is used to look up the list of multihashes associated to a context ID.

Note that prior to calling this function a provider.Callback must be registered.

See: Engine.RegisterCallback, Engine.Publish.

func (*Engine) NotifyRemove

func (e *Engine) NotifyRemove(ctx context.Context, contextID []byte) (cid.Cid, error)

NotifyRemove publishes an advertisement that signals the list of multihashes associated to the given contextID is no longer available by this provider.

Note that prior to calling this function a provider.Callback must be registered.

See: Engine.RegisterCallback, Engine.Publish.

func (*Engine) Publish

func (e *Engine) Publish(ctx context.Context, adv schema.Advertisement) (cid.Cid, error)

Publish stores the given advertisement locally via Engine.PublishLocal first, then publishes a message onto the gossipsub to signal the change in the latest advertisement by the provider to indexer nodes.

The publication mechanism uses legs.Publisher internally. See: https://github.com/filecoin-project/go-legs

func (*Engine) PublishLatest

func (e *Engine) PublishLatest(ctx context.Context) error

PublishLatest re-publishes the latest existing advertisement to pubsub.

func (*Engine) PublishLocal

func (e *Engine) PublishLocal(ctx context.Context, adv schema.Advertisement) (cid.Cid, error)

PublishLocal stores the advertisement in the local link system and marks it locally as the latest advertisement.

The context is used for storing internal mapping information onto the datastore.

See: Engine.Publish.

func (*Engine) RegisterCallback

func (e *Engine) RegisterCallback(cb provider.Callback)

RegisterCallback registers a provider.Callback that is used to look up the list of multihashes associated to a context ID. At least one such callback must be registered before calls to Engine.NotifyPut and Engine.NotifyRemove.

Note that successive calls to this function will replace the previous callback. Only a single callback is supported.

See: provider.Interface

func (*Engine) Shutdown

func (e *Engine) Shutdown() error

Shutdown shuts down the engine and discards all resources opened by the engine. The engine is no longer usable after the call to this function.

func (*Engine) Start

func (e *Engine) Start(ctx context.Context) error

Start starts the engine by instantiating the internal storage and joins the configured gossipsub topic used for publishing advertisements.

The context is used to instantiate the internal LRU cache storage.

See: Engine.Shutdown, lrustore.New, dtsync.NewPublisherFromExisting.

type Option

type Option func(*engineConfig) error

func WithExtraGossipData

func WithExtraGossipData(extraData []byte) Option

WithExtraGossipData supplies extra data to include in the pubsub announcement.