provider

README

Indexer Reference Provider 📢

A reference implementation of index data provider

This repo provides a reference data provider implementation that can be used to advertise content to indexer nodes and serve retreival requests over graphsync.

Current status 🚧

This implementation is under active development.

Install

Prerequisite:

• Go 1.16+

To use the provider as a Go library, execute:

go get github.com/filecoin-project/index-provider

To install the latest runnable version of the provider service, execute:

go install github.com/filecoin-project/index-provider/cmd/provider@latest

Running Provider Service

To run a provider service it must first be initialized. To initialize the provider, execute:

provider init

Initialization generates a default configuration for the provider instance along with a randomly generated identity keypair. The configuration is stored at user home under .index-provider/config in JSON format. The root configuration path can be overridden by setting the PROVIDER_PATH environment variable

Once initialized, start the service daemon by executing:

provider daemon

The running daemon allows advertisement of new content to the indexer nodes and retrieval of content over graphsync. Additionally, it starts an admin HTTP server that enables administrative operations using the provider CLI tool. By default, the admin server is bound to http://localhost:3102.

provider CLI

The provider CLI can be used to interact with a running daemon via the admin server to perform a range of administrative operations. For example, the provider CLI can be used to import a CAR file and advertise its content to the indexer nodes by executing:

provider import car -l http://localhost:3102 -i <path-to-car-file>

For full usage, execute provider. Usage:

NAME:
   provider - Indexer Reference Provider Implementation

USAGE:
   provider [global options] command [command options] [arguments...]

VERSION:
   v0.0.0+unknown

COMMANDS:
   daemon     Starts a reference provider
   find       Query an indexer for indexed content
   index      Push a single content index into an indexer
   init       Initialize reference provider config file and identity
   connect    Connects to an indexer through its multiaddr
   import, i  Imports sources of multihashes to the index provider.
   register   Register provider information with an indexer that trusts the provider
   help, h    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

License

SPDX-License-Identifier: Apache-2.0 OR MIT

Documentation

Overview

Package provider represents a reference implementation of an index provider.

Index

• Variables
• type Callback
• type Interface
• type MultihashIterator
  • func CarMultihashIterator(idx carindex.IterableIndex) (MultihashIterator, error)

Variables

var (
	// ErrNoCallback is thrown when no callback has been defined.
	ErrNoCallback = errors.New("no callback is registered")

	// ErrContextIDNotFound signals that no item is associated to the given context ID.
	ErrContextIDNotFound = errors.New("context ID not found")

	// ErrAlreadyAdvertised indicates that an advertisement for identical
	// content was already published.
	ErrAlreadyAdvertised = errors.New("advertisement already published")
)

Types

type Callback

type Callback func(ctx context.Context, contextID []byte) (MultihashIterator, error)

Callback is used by provider to look up a list of multihashes associated to a context ID. The callback must be deterministic: it must produce the same list of multihashes in the same order for the same context ID.

See: Interface.NotifyPut, Interface.NotifyRemove, MultihashIterator.

type Interface

type Interface interface {
	// PublishLocal appends adv to the locally stored advertisement chain and
	// returns the corresponding CID to it.  This function does not publish the
	// changes to the advertisement chain onto gossip pubsub channel.  Use
	// Publish instead if indexer nodes must be made aware of the appended
	// advertisement.
	//
	// See: Publish.
	PublishLocal(context.Context, schema.Advertisement) (cid.Cid, error)

	// Publish appends adv to the locally stored advertisement chain, and
	// publishes the new advertisement onto gossip pubsub.  The CID returned
	// represents the ID of the advertisement appended to the chain.
	Publish(context.Context, schema.Advertisement) (cid.Cid, error)

	// RegisterCallback registers the callback used by the provider to look up
	// a list of multihashes by context ID.  Only a single callback is
	// supported; repeated calls to this function will replace the previous
	// callback.
	RegisterCallback(Callback)

	// NotifyPut sginals to the provider that the list of multihashes looked up
	// by the given contextID are available.  The given contextID is then used to look up
	// the list of multihashes via Callback.  An advertisement is then
	// generated, appended to the chain of advertisements and published onto
	// the gossip pubsub channel.
	// Therefore, a Callback must be registered prior to using this function.
	// ErrNoCallback is returned if no such callback is registered.
	//
	// The metadata is data that provides hints about how to retrieve data and
	// is protocol dependant.  The metadata must at least specify a protocol
	// ID, but its data is optional.
	//
	// If both the contextID and metadata are the same as a previous call to
	// NotifyPut, then ErrAlreadyAdvertised is returned.
	//
	// This function returns the ID of the advertisement published.
	NotifyPut(ctx context.Context, contextID []byte, metadata stiapi.Metadata) (cid.Cid, error)

	// NotifyRemove sginals to the provider that the multihashes that
	// corresponded to the given contextID are no longer available.  An advertisement
	// is then generated, appended to the chain of advertisements and published
	// onto the gossip pubsub channel.
	// The given contextID must have previously been put via NotifyPut.
	// If not found ErrContextIDNotFound is returned.
	//
	// This function returns the ID of the advertisement published.
	NotifyRemove(ctx context.Context, contextID []byte) (cid.Cid, error)

	// GetAdv gets the advertisement that corresponds to the given cid.
	GetAdv(context.Context, cid.Cid) (schema.Advertisement, error)

	// GetLatestAdv gets the latest advertisement on this provider's
	// advertisement chain and the CID to which it corresponds.
	GetLatestAdv(context.Context) (cid.Cid, schema.Advertisement, error)

	// Shutdown shuts down this provider, and blocks until all resources
	// occupied by it are discared.  After calling this function the provider
	// is no longer available for use.
	Shutdown() error
}

Interface represents an index provider that manages the advertisement of multihashes to indexer nodes.

type MultihashIterator

type MultihashIterator interface {
	// Next returns the next multihash in the list of mulitihashes.  The
	// iterator fails fast: errors that occur during iteration are returned
	// immediately.  This function returns a zero multihash and io.EOF when
	// there are no more elements to return.
	Next() (multihash.Multihash, error)
}

MultihashIterator iterates over a list of multihashes.

See: CarMultihashIterator.

func CarMultihashIterator

func CarMultihashIterator(idx carindex.IterableIndex) (MultihashIterator, error)

CarMultihashIterator constructs a new MultihashIterator from a CAR index.

This iterator supplies multihashes in deterministic order of their corresponding CAR offset. The order is maintained consistently regardless of the underlying IterableIndex implementation.