ingest

Documentation

Index

• type Ingester
  • func NewIngester(cfg config.Ingest, h host.Host, idxr indexer.Interface, reg *registry.Registry, ...) (*Ingester, error)
  • func (ing *Ingester) Close() error
  • func (ing *Ingester) GetLatestSync(peerID peer.ID) (cid.Cid, error)
  • func (ing *Ingester) Sync(ctx context.Context, peerID peer.ID, peerAddr multiaddr.Multiaddr, depth int64, ...) (<-chan cid.Cid, error)

Types

type Ingester

type Ingester struct {
	// contains filtered or unexported fields
}

Ingester is a type that uses go-legs for the ingestion protocol.

func NewIngester

func NewIngester(cfg config.Ingest, h host.Host, idxr indexer.Interface, reg *registry.Registry, ds datastore.Batching) (*Ingester, error)

NewIngester creates a new Ingester that uses a go-legs Subscriber to handle communication with providers.

func (*Ingester) Close

func (ing *Ingester) Close() error

func (*Ingester) GetLatestSync

func (ing *Ingester) GetLatestSync(peerID peer.ID) (cid.Cid, error)

Get the latest CID synced for the peer.

func (*Ingester) Sync

func (ing *Ingester) Sync(ctx context.Context, peerID peer.ID, peerAddr multiaddr.Multiaddr, depth int64, resync bool) (<-chan cid.Cid, error)

Sync syncs advertisements, up to the the latest advertisement, from a publisher. A channel is returned that gives the caller the option to wait for Sync to complete. The channel returns the final CID that was synced by the call to Sync.

Sync works by first fetching each advertisement from the specidief peer starting at the most recent and traversing to the advertisement last seen by the indexer, or until the advertisement depth limit is reached. Then the entries in each advertisement are synced and the multihashes in each entry are indexed.

The selector used to sync the advertisement is controlled by the following parameters: depth, and resync.

The depth argument specifies the recursion depth limit to use during sync. Its value may be one of: -1 for no-limit, 0 for same value as config.Ingest, or larger than 0 for an explicit limit.

The resync argument specifies whether to stop the traversal at the latest known advertisement that is already synced. If set to true, the traversal will continue until either there are no more advertisements left or the recursion depth limit is reached.

The reference to the latest synced advertisement returned by GetLatestSync is only updated if the given depth is zero and resync is set to false. Otherwise, a custom selector with the given depth limit and stop link is constructed and used for traversal. See legs.Subscriber.Sync.

The Context argument controls the lifetime of the sync. Canceling it cancels the sync and causes the multihash channel to close without any data.