graphsync

README

go-graphsync

An implementation of the graphsync protocol in go!

Table of Contents

• Background
• Install
• Usage
• Architecture
• Contribute
• License

Background

GraphSync is a protocol for synchronizing IPLD graphs among peers. It allows a host to make a single request to a remote peer for all of the results of traversing an IPLD selector on the remote peer's local IPLD graph.

go-graphsync provides an implementation of the Graphsync protocol in go.

Go-IPLD-Prime

go-graphsync relies on go-ipld-prime to traverse IPLD Selectors in an IPLD graph. go-ipld-prime implements the IPLD specification in go and is an alternative to older implementations such as go-ipld-format and go-ipld-cbor. In order to use go-graphsync, some understanding and use of go-ipld-prime concepts is necessary.

If your existing library (i.e. go-ipfs or go-filecoin) uses these other older libraries, you can largely use go-graphsync without switching to go-ipld-prime across your codebase, but it will require some translations

Install

go-graphsync requires Go >= 1.11 and can be installed using Go modules

Usage

Initializing a GraphSync Exchange

import (
  graphsync "github.com/maurycy/go-graphsync/impl"
  gsnet "github.com/maurycy/go-graphsync/network"
  gsbridge "github.com/maurycy/go-graphsync/ipldbridge"
  ipld "github.com/ipld/go-ipld-prime"
)

var ctx context.Context
var host libp2p.Host
var loader ipld.Loader
var storer ipld.Storer

network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
exchange := graphsync.New(ctx, network, ipldBridge, loader, storer)

Parameter Notes:

1. context is just the parent context for all of GraphSync
2. network is a network abstraction provided to Graphsync on top of libp2p. This allows graphsync to be tested without the actual network
3. ipldBridge is an IPLD abstraction provided to Graphsync on top of go-ipld-prime. This makes the graphsync library testable in isolation
4. loader is used to load blocks from content ids from the local block store. It's used when RESPONDING to requests from other clients. It should conform to the IPLD loader interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
5. storer is used to store incoming blocks to the local block store. It's used when REQUESTING a graphsync query, to store blocks locally once they are validated as part of the correct response. It should conform to the IPLD storer interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go

Using GraphSync With An IPFS BlockStore

GraphSync provides two convenience functions in the storeutil package for integrating with BlockStore's from IPFS.

import (
  graphsync "github.com/maurycy/go-graphsync/impl"
  gsnet "github.com/maurycy/go-graphsync/network"
  gsbridge "github.com/maurycy/go-graphsync/ipldbridge"
  gsbridge "github.com/maurycy/go-graphsync/storeutil"
  ipld "github.com/ipld/go-ipld-prime"
  blockstore "github.com/ipfs/go-ipfs-blockstore"
)

var ctx context.Context
var host libp2p.Host
var bs blockstore.Blockstore
var storer ipld.Storer

network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
loader := storeutil.LoaderForBlockstore(bs)
storer := storeutil.StorerForBlockstore(bs)

exchange := graphsync.New(ctx, network, ipldBridge, loader, storer)

Write A Loader An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your link loading function looks like this:

type BlockStore interface {
	Get(lnk cid.Cid) (blocks.Block, error)
}

or, more generally:

type Cid2BlockFn func (lnk cid.Cid) (blocks.Block, error)

in go-ipld-prime, the signature for a link loader is as follows:

type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)

go-ipld-prime intentionally keeps its interfaces as abstract as possible to limit dependencies on other ipfs/filecoin specific packages. An IPLD Link is an abstraction for a CID, and IPLD expects io.Reader's rather than an actual block. IPLD provides a cidLink package for working with Links that use CIDs as the underlying data, and it's safe to assume that's the type in use if your code deals only with CIDs. A conversion would look something like this:

import (
   ipld "github.com/ipld/go-ipld-prime"
   cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func LoaderFromCid2BlockFn(cid2BlockFn Cid2BlockFn) ipld.Loader {
	return func(lnk ipld.Link, lnkCtx ipld.LinkContext) (io.Reader, error) {
		asCidLink, ok := lnk.(cidlink.Link)
		if !ok {
			return nil, fmt.Errorf("Unsupported Link Type")
		}
		block, err := cid2BlockFn(asCidLink.Cid)
		if err != nil {
			return nil, err
		}
		return bytes.NewReader(block.RawData()), nil
	}
}

Write A Storer From An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your storage function looks like this:

type BlockStore interface {
	Put(blocks.Block) error
}

or, more generally:

type BlockStoreFn func (blocks.Block) (error)

in go-ipld-prime, the signature for a link storer is a bit different:

type StoreCommitter func(Link) error
type Storer func(lnkCtx LinkContext) (io.Writer, StoreCommitter, error)

go-ipld-prime stores in two parts to support streaming -- the storer is called and returns an IO.Writer and a function to commit changes when finished. Here's how you can write a storer from a traditional block storing signature.

import (
	blocks "github.com/ipfs/go-block-format"
  ipld "github.com/ipld/go-ipld-prime"
  cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func StorerFromBlockStoreFn(blockStoreFn BlockStoreFn) ipld.Storer {
	return func(lnkCtx ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) {
		var buffer bytes.Buffer
		committer := func(lnk ipld.Link) error {
			asCidLink, ok := lnk.(cidlink.Link)
			if !ok {
				return fmt.Errorf("Unsupported Link Type")
			}
			block := blocks.NewBlockWithCid(buffer.Bytes(), asCidLink.Cid)
			return blockStoreFn(block)
		}
		return &buffer, committer, nil
	}
}

Calling Graphsync

var exchange graphsync.GraphSync
var ctx context.Context
var p peer.ID
var selector ipld.Node
var rootLink ipld.Link

var responseProgress <-chan graphsync.ResponseProgress
var errors <-chan error

responseProgress, errors = exchange.Request(ctx context.Context, p peer.ID, root ipld.Link, selector ipld.Node)

Paramater Notes:

1. ctx is the context for this request. To cancel an in progress request, cancel the context.
2. p is the peer you will send this request to
3. link is an IPLD Link, i.e. a CID (cidLink.Link{Cid})
4. selector is an IPLD selector node. Recommend using selector builders from go-ipld-prime to construct these

Response Type

type ResponseProgress struct {
  Node      ipld.Node // a node which matched the graphsync query
  Path      ipld.Path // the path of that node relative to the traversal start
	LastBlock struct {  // LastBlock stores the Path and Link of the last block edge we had to load.
		ipld.Path
		ipld.Link
	}
}

The above provides both immediate and relevant metadata for matching nodes in a traversal, and is very similar to the information provided by a local IPLD selector traversal in go-ipld-prime

Contribute

PRs are welcome!

Before doing anything heavy, checkout the Graphsync Architecture

Small note: If editing the Readme, please conform to the standard-readme specification.

License

This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2019. Protocol Labs, Inc.

Documentation

Index

• Constants
• Variables
• type ExtensionData
• type ExtensionName
• type GraphExchange
• type OnRequestReceivedHook
• type OnResponseReceivedHook
• type Priority
• type RequestData
• type RequestID
• type RequestReceivedHookActions
• type ResponseData
• type ResponseProgress
• type ResponseStatusCode

Constants

const (

	// ExtensionMetadata provides response metadata for a Graphsync request and is
	// documented at
	// https://github.com/ipld/specs/blob/master/block-layer/graphsync/known_extensions.md
	ExtensionMetadata = ExtensionName("graphsync/response-metadata")

	// ExtensionDoNotSendCIDs tells the responding peer not to send certain blocks if they
	// are encountered in a traversal and is documented at
	// https://github.com/ipld/specs/blob/master/block-layer/graphsync/known_extensions.md
	ExtensionDoNotSendCIDs = ExtensionName("graphsync/do-not-send-cids")

	// RequestAcknowledged means the request was received and is being worked on.
	RequestAcknowledged = ResponseStatusCode(10)
	// AdditionalPeers means additional peers were found that may be able
	// to satisfy the request and contained in the extra block of the response.
	AdditionalPeers = ResponseStatusCode(11)
	// NotEnoughGas means fulfilling this request requires payment.
	NotEnoughGas = ResponseStatusCode(12)
	// OtherProtocol means a different type of response than GraphSync is
	// contained in extra.
	OtherProtocol = ResponseStatusCode(13)
	// PartialResponse may include blocks and metadata about the in progress response
	// in extra.
	PartialResponse = ResponseStatusCode(14)

	// RequestCompletedFull means the entire fulfillment of the GraphSync request
	// was sent back.
	RequestCompletedFull = ResponseStatusCode(20)
	// RequestCompletedPartial means the response is completed, and part of the
	// GraphSync request was sent back, but not the complete request.
	RequestCompletedPartial = ResponseStatusCode(21)

	// RequestRejected means the node did not accept the incoming request.
	RequestRejected = ResponseStatusCode(30)
	// RequestFailedBusy means the node is too busy, try again later. Backoff may
	// be contained in extra.
	RequestFailedBusy = ResponseStatusCode(31)
	// RequestFailedUnknown means the request failed for an unspecified reason. May
	// contain data about why in extra.
	RequestFailedUnknown = ResponseStatusCode(32)
	// RequestFailedLegal means the request failed for legal reasons.
	RequestFailedLegal = ResponseStatusCode(33)
	// RequestFailedContentNotFound means the respondent does not have the content.
	RequestFailedContentNotFound = ResponseStatusCode(34)
)

Variables

var (
	// ErrExtensionAlreadyRegistered means a user extension can be registered only once
	ErrExtensionAlreadyRegistered = errors.New("extension already registered")
)

Types

type ExtensionData

type ExtensionData struct {
	Name ExtensionName
	Data []byte
}

ExtensionData is a name/data pair for a graphsync extension

type ExtensionName

type ExtensionName string

ExtensionName is a name for a GraphSync extension

type GraphExchange

type GraphExchange interface {
	// Request initiates a new GraphSync request to the given peer using the given selector spec.
	Request(ctx context.Context, p peer.ID, root ipld.Link, selector ipld.Node, extensions ...ExtensionData) (<-chan ResponseProgress, <-chan error)

	// RegisterRequestReceivedHook adds a hook that runs when a request is received
	// If overrideDefaultValidation is set to true, then if the hook does not error,
	// it is considered to have "validated" the request -- and that validation supersedes
	// the normal validation of requests Graphsync does (i.e. all selectors can be accepted)
	RegisterRequestReceivedHook(hook OnRequestReceivedHook) error

	// RegisterResponseReceivedHook adds a hook that runs when a response is received
	RegisterResponseReceivedHook(OnResponseReceivedHook) error
}

GraphExchange is a protocol that can exchange IPLD graphs based on a selector

type OnRequestReceivedHook

type OnRequestReceivedHook func(p peer.ID, request RequestData, hookActions RequestReceivedHookActions)

OnRequestReceivedHook is a hook that runs each time a request is received. It receives the peer that sent the request and all data about the request. It should return: extensionData - any extension data to add to the outgoing response err - error - if not nil, halt request and return RequestRejected with the responseData

type OnResponseReceivedHook

type OnResponseReceivedHook func(p peer.ID, responseData ResponseData) error

OnResponseReceivedHook is a hook that runs each time a response is received. It receives the peer that sent the response and all data about the response. If it returns an error processing is halted and the original request is cancelled.

type Priority

type Priority int32

Priority a priority for a GraphSync request.

type RequestData

type RequestData interface {
	// ID Returns the request ID for this Request
	ID() RequestID

	// Root returns the CID to the root block of this request
	Root() cid.Cid

	// Selector returns the byte representation of the selector for this request
	Selector() []byte

	// Priority returns the priority of this request
	Priority() Priority

	// Extension returns the content for an extension on a response, or errors
	// if extension is not present
	Extension(name ExtensionName) ([]byte, bool)

	// IsCancel returns true if this particular request is being cancelled
	IsCancel() bool
}

RequestData describes a received graphsync request.

type RequestID

type RequestID int32

RequestID is a unique identifier for a GraphSync request.

type RequestReceivedHookActions

type RequestReceivedHookActions interface {
	SendExtensionData(ExtensionData)
	TerminateWithError(error)
	ValidateRequest()
}

RequestReceivedHookActions are actions that a request hook can take to change behavior for the response

type ResponseData

type ResponseData interface {
	// RequestID returns the request ID for this response
	RequestID() RequestID

	// Status returns the status for a response
	Status() ResponseStatusCode

	// Extension returns the content for an extension on a response, or errors
	// if extension is not present
	Extension(name ExtensionName) ([]byte, bool)
}

ResponseData describes a received Graphsync response

type ResponseProgress

type ResponseProgress struct {
	Node      ipld.Node // a node which matched the graphsync query
	Path      ipld.Path // the path of that node relative to the traversal start
	LastBlock struct {
		Path ipld.Path
		Link ipld.Link
	}
}

ResponseProgress is the fundamental unit of responses making progress in Graphsync.

type ResponseStatusCode

type ResponseStatusCode int32

ResponseStatusCode is a status returned for a GraphSync Request.