The highest tagged major version is
README
¶
go-data-transfer
A go module to perform data transfers over ipfs/go-graphsync
Description
This module encapsulates protocols for exchanging piece data between storage clients and miners, both when consummating a storage deal and when retrieving the piece later.
Table of Contents
Background
Please see the design documentation for this module for a high-level overview and and explanation of the terms and concepts.
Usage
Requires go 1.13
Install the module in your package or app with go get "github.com/filecoin-project/go-data-transfer/datatransfer"
Initialize a data transfer module
-
Set up imports. You need, minimally, the following imports:
package mypackage import ( gsimpl "github.com/ipfs/go-graphsync/impl" "github.com/filecoin-project/go-data-transfer/datatransfer" "github.com/libp2p/go-libp2p-core/host" ) -
Provide or create a libp2p host.Host
-
Provide or create a go-graphsync GraphExchange
-
Create a new instance of GraphsyncDataTransfer
func NewGraphsyncDatatransfer(h host.Host, gs graphsync.GraphExchange) { dt := datatransfer.NewGraphSyncDataTransfer(h, gs) } -
If needed, build out your voucher struct and its validator.
A push or pull request must include a voucher. The voucher's type must have been registered with the node receiving the request before it's sent, otherwise the request will be rejected.
datatransfer.Voucher and datatransfer.Validator are the interfaces used for validation of graphsync datatransfer messages. Voucher types plus a Validator for them must be registered with the peer to whom requests will be sent.
Example Toy Voucher and Validator
type myVoucher struct {
data string
}
func (v *myVoucher) ToBytes() ([]byte, error) {
return []byte(v.data), nil
}
func (v *myVoucher) FromBytes(data []byte) error {
v.data = string(data)
return nil
}
func (v *myVoucher) Type() string {
return "FakeDTType"
}
type myValidator struct {
ctx context.Context
validationsReceived chan receivedValidation
}
func (vl *myValidator) ValidatePush(
sender peer.ID,
voucher datatransfer.Voucher,
baseCid cid.Cid,
selector ipld.Node) error {
v := voucher.(*myVoucher)
if v.data == "" || v.data != "validpush" {
return errors.New("invalid")
}
return nil
}
func (vl *myValidator) ValidatePull(
receiver peer.ID,
voucher datatransfer.Voucher,
baseCid cid.Cid,
selector ipld.Node) error {
v := voucher.(*myVoucher)
if v.data == "" || v.data != "validpull" {
return errors.New("invalid")
}
return nil
}
Please see go-data-transfer/blob/master/types.go for more detail.
Register a validator
Before sending push or pull requests, you must register a datatransfer.Voucher
by its reflect.Type and dataTransfer.RequestValidator for vouchers that
must be sent with the request. Using the trivial examples above:
func NewGraphsyncDatatransfer(h host.Host, gs graphsync.GraphExchange) {
dt := datatransfer.NewGraphSyncDataTransfer(h, gs)
vouch := &myVoucher{}
mv := &myValidator{}
dt.RegisterVoucherType(reflect.TypeOf(vouch), mv)
}
For more detail, please see the unit tests.
Open a Push or Pull Request
For a push or pull request, provide a context, a datatransfer.Voucher, a host recipient peer.ID, a baseCID cid.CID and a selector ipld.Node. These
calls return a datatransfer.ChannelID and any error:
channelID, err := dtm.OpenPullDataChannel(ctx, recipient, voucher, baseCid, selector)
// OR
channelID, err := dtm.OpenPushDataChannel(ctx, recipient, voucher, baseCid, selector)
Subscribe to Events
The module allows the consumer to be notified when a graphsync Request is sent or a datatransfer push or pull request response is received:
func ToySubscriberFunc (event Event, channelState ChannelState) {
if event.Code == datatransfer.Error {
// log error, flail about helplessly
return
}
//
if channelState.Recipient() == our.PeerID && channelState.Received() > 0 {
// log some stuff, update some state somewhere, send data to a channel, etc.
}
}
dtm := SetupDataTransferManager(ctx, h, gs, baseCid, snode)
unsubFunc := dtm.SubscribeToEvents(ToySubscriberFunc)
// . . . later, when you don't need to know about events any more:
unsubFunc()
Contributing
PRs are welcome! Please first read the design docs and look over the current code. PRs against master require approval of at least two maintainers. For the rest, please see our CONTRIBUTING guide.
License
This repository is dual-licensed under Apache 2.0 and MIT terms.
Copyright 2019. Protocol Labs, Inc.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var EmptyChannelState = ChannelState{}
Functions ¶
This section is empty.
Types ¶
type Channel ¶
type Channel struct {
// contains filtered or unexported fields
}
Channel represents all the parameters for a single data transfer
func NewChannel ¶
func NewChannel(transferID TransferID, baseCid cid.Cid, selector ipld.Node, voucher Voucher, sender peer.ID, recipient peer.ID, totalSize uint64) Channel
NewChannel makes a new channel
func (Channel) BaseCID ¶
func (c Channel) BaseCID() cid.Cid
BaseCID returns the CID that is at the root of this data transfer
func (Channel) Selector ¶
func (c Channel) Selector() ipld.Node
Selector returns the IPLD selector for this data transfer (represented as an IPLD node)
func (Channel) TransferID ¶
func (c Channel) TransferID() TransferID
TransferID returns the transfer id for this channel
type ChannelID ¶
type ChannelID struct {
Initiator peer.ID
ID TransferID
}
ChannelID is a unique identifier for a channel, distinct by both the other party's peer ID + the transfer ID
type ChannelState ¶
type ChannelState struct {
Channel
// contains filtered or unexported fields
}
ChannelState is immutable channel data plus mutable state
func (ChannelState) Received ¶
func (c ChannelState) Received() uint64
Received returns the number of bytes received
func (ChannelState) Sent ¶
func (c ChannelState) Sent() uint64
Sent returns the number of bytes sent
type Event ¶
type Event struct {
Code EventCode // What type of event it is
Message string // Any clarifying information about the event
Timestamp time.Time // when the event happened
}
Event is a struct containing information about a data transfer event
type EventCode ¶
type EventCode int
EventCode is a name for an event that occurs on a data transfer channel
const ( // Open is an event occurs when a channel is first opened Open EventCode = iota // Progress is an event that gets emitted every time more data is transferred Progress // Error is an event that emits when an error occurs in a data transfer Error // Complete is emitted when a data transfer is complete Complete )
type Manager ¶
type Manager interface {
// RegisterVoucherType registers a validator for the given voucher type
// will error if voucher type does not implement voucher
// or if there is a voucher type registered with an identical identifier
RegisterVoucherType(voucherType reflect.Type, validator RequestValidator) error
// open a data transfer that will send data to the recipient peer and
// transfer parts of the piece that match the selector
OpenPushDataChannel(ctx context.Context, to peer.ID, voucher Voucher, baseCid cid.Cid, selector ipld.Node) (ChannelID, error)
// open a data transfer that will request data from the sending peer and
// transfer parts of the piece that match the selector
OpenPullDataChannel(ctx context.Context, to peer.ID, voucher Voucher, baseCid cid.Cid, selector ipld.Node) (ChannelID, error)
// close an open channel (effectively a cancel)
CloseDataTransferChannel(x ChannelID)
// get status of a transfer
TransferChannelStatus(x ChannelID) Status
// get notified when certain types of events happen
SubscribeToEvents(subscriber Subscriber) Unsubscribe
// get all in progress transfers
InProgressChannels() map[ChannelID]ChannelState
}
Manager is the core interface presented by all implementations of of the data transfer sub system
type RequestValidator ¶
type RequestValidator interface {
// ValidatePush validates a push request received from the peer that will send data
ValidatePush(
sender peer.ID,
voucher Voucher,
baseCid cid.Cid,
selector ipld.Node) error
// ValidatePull validates a pull request received from the peer that will receive data
ValidatePull(
receiver peer.ID,
voucher Voucher,
baseCid cid.Cid,
selector ipld.Node) error
}
RequestValidator is an interface implemented by the client of the data transfer module to validate requests
type Subscriber ¶
type Subscriber func(event Event, channelState ChannelState)
Subscriber is a callback that is called when events are emitted
type TransferID ¶
type TransferID uint64
TransferID is an identifier for a data transfer, shared between request/responder and unique to the requester
type Unsubscribe ¶
type Unsubscribe func()
Unsubscribe is a function that gets called to unsubscribe from data transfer events
type Voucher ¶
type Voucher interface {
// ToBytes converts the Voucher to raw bytes
ToBytes() ([]byte, error)
// FromBytes reads a Voucher from raw bytes
FromBytes([]byte) error
// Type is a unique string identifier for this voucher type
Type() string
}
Voucher is used to validate a data transfer request against the underlying storage or retrieval deal that precipitated it. The only requirement is a voucher can read and write from bytes, and has a string identifier type
Source Files
Directories