Documentation ¶
Overview ¶
Package retrievalmarket implements the Filecoin retrieval protocol.
An overview of the retrieval protocol can be found in the Filecoin specification:
https://filecoin-project.github.io/specs/#systems__filecoin_markets__retrieval_market
The following architectural components provide a brief overview of the design of the retrieval market module:
Public Interfaces And Node Dependencies ¶
While retrieval deals primarily happen off-chain, there are some chain operations that must be performed by a Filecoin node implementation. The module is intended to separate the primarily off-chain retrieval deal flow from the on-chain operations related primarily to payment channels, the mechanism for getting paid for retrieval deals.
As such for both the client and the provider in the retrieval market, the module defines a top level public interface which it provides an implementation for, and a node interface that must be implemented by the Filecoin node itself, and provided as a dependency. These node interfaces provide a universal way to talk to potentially multiple different Filecoin node implementations, and can be implemented as using HTTP or other interprocess communication to talk to a node implementation running in a different process.
The top level interfaces this package implements are RetrievalClient & RetrievalProvider. The dependencies the Filecoin node is expected to implement are RetrievalClientNode & RetrievalProviderNode. Further documentation of exactly what those dependencies should do can be found in the readme.
Finite State Machines ¶
While retrieval deals in general should be fairly fast, making a retrieval deal is still an asynchronous process. As documented in the Filecoin spec, the basic architecture of the Filecoin retrieval protocol is incremental payments. Because neither client nor provider trust each other, we bootstrap trust by essentially paying in small increments as we receive data. The client only sends payment when it verifies data and the provider only sends more data when it receives payment. Not surprisingly, many things can go wrong along the way. To manage this back and forth asynchronous process, we use finite state machines that update deal state when discrete events occur. State updates always persist state to disk. This means we have a permanent record of exactly what's going on with deals at any time, and we can ideally survive our Filecoin processes shutting down and restarting.
The following diagrams visualize the statemachine flows for the client and the provider:
Client FSM - https://raw.githubusercontent.com/filecoin-project/go-fil-markets/master/docs/retrievalclient.mmd.svg
Provider FSM - https://raw.githubusercontent.com/filecoin-project/go-fil-markets/master/docs/retrievalprovider.mmd.svg
Identifying Retrieval Providers ¶
The RetrievalClient provides two functions to locate a provider from which to retrieve data.
`FindProviders` returns a list of retrieval peers who may have the data your looking for. FindProviders delegates its work to an implementation of the PeerResolver interface.
`Query` queries a specific retrieval provider to find out definitively if they have the requested data and if so, the parameters they will accept for a retrieval deal.
Deal Flow ¶
The primary mechanism for initiating storage deals is the `Retrieve` method on the RetrievalClient.
When `Retrieve` is called, it allocates a new DealID from its stored counter, constructs a DealProposal, sends the deal proposal to the provider, initiates tracking of deal state and hands the deal to the Client FSM, and returns the DealID which constitutes the identifier for that deal.
The Retrieval provider receives the deal in `HandleDealStream`. `HandleDealStream` initiates tracking of deal state on the Provider side and hands the deal to the Provider FSM, which handles the rest of deal flow.
From this point forward, deal negotiation is completely asynchronous and runs in the FSMs.
A user of the modules can monitor deal progress through `SubscribeToEvents` methods on RetrievalClient and RetrievalProvider, or by simply calling `ListDeals` to get all deal statuses.
The FSMs implement every remaining step in deal negotiation. Importantly, the RetrievalProvider delegates unsealing sectors back to the node via the `UnsealSector` method (the node itself likely delegates management of sectors and sealing to an implementation of the Storage Mining subsystem of the Filecoin spec). Sectors are unsealed on an as needed basis using the `PieceStore` to locate sectors that contain data related to the deal.
Major Dependencies ¶
Other libraries in go-fil-markets:
https://github.com/filecoin-project/go-fil-markets/tree/master/piecestore - used to locate data for deals in sectors https://github.com/filecoin-project/go-fil-markets/tree/master/shared - types and utility functions shared with storagemarket package
Other Filecoin Repos:
https://github.com/filecoin-project/go-data-transfer - for transferring data, via go-graphsync https://github.com/filecoin-project/go-statemachine - a finite state machine that tracks deal state https://github.com/filecoin-project/go-storedcounter - for generating and persisting unique deal IDs https://github.com/filecoin-project/specs-actors - the Filecoin actors
IPFS Project Repos:
https://github.com/ipfs/go-graphsync - used by go-data-transfer https://github.com/ipfs/go-datastore - for persisting statemachine state for deals https://github.com/ipfs/go-ipfs-blockstore - for storing and retrieving block data for deals
Other Repos:
https://github.com/libp2p/go-libp2p) the network over which retrieval deal data is exchanged. https://github.com/hannahhoward/go-pubsub - for pub/sub notifications external to the statemachine
Root package ¶
This top level package defines top level enumerations and interfaces. The primary implementation lives in the `impl` directory
Index ¶
- Constants
- Variables
- func DecodeNode(defnode *cbg.Deferred) (ipld.Node, error)
- func IsTerminalError(status DealStatus) bool
- func IsTerminalStatus(status DealStatus) bool
- func IsTerminalSuccess(status DealStatus) bool
- func NewShortfallError(shortfall abi.TokenAmount) error
- type Ask
- type AskStore
- type ChannelAvailableFunds
- type ClientDealState
- type ClientEvent
- type ClientSubscriber
- type DealID
- type DealPayment
- type DealProposal
- type DealResponse
- type DealStatus
- type Params
- type PaymentInfo
- type PeerResolver
- type ProviderDealIdentifier
- type ProviderDealState
- type ProviderEvent
- type ProviderSubscriber
- type Query
- type QueryItemStatus
- type QueryParams
- type QueryResponse
- type QueryResponseStatus
- type RetrievalClient
- type RetrievalClientNode
- type RetrievalPeer
- type RetrievalProvider
- type RetrievalProviderNode
- type ShortfallError
- type Unsubscribe
Constants ¶
const QueryProtocolID = "/fil/retrieval/qry/0.0.1"
QueryProtocolID is the protocol for querying information about retrieval deal parameters
Variables ¶
var ( // ErrNotFound means a piece was not found during retrieval ErrNotFound = errors.New("not found") // ErrVerification means a retrieval contained a block response that did not verify ErrVerification = errors.New("Error when verify data") )
var ClientEvents = map[ClientEvent]string{ ClientEventOpen: "ClientEventOpen", ClientEventPaymentChannelErrored: "ClientEventPaymentChannelErrored", ClientEventDealProposed: "ClientEventDealProposed", ClientEventAllocateLaneErrored: "ClientEventAllocateLaneErrored", ClientEventPaymentChannelCreateInitiated: "ClientEventPaymentChannelCreateInitiated", ClientEventPaymentChannelReady: "ClientEventPaymentChannelReady", ClientEventPaymentChannelAddingFunds: "ClientEventPaymentChannelAddingFunds", ClientEventPaymentChannelAddFundsErrored: "ClientEventPaymentChannelAddFundsErrored", ClientEventWriteDealProposalErrored: "ClientEventWriteDealProposalErrored", ClientEventDealRejected: "ClientEventDealRejected", ClientEventDealNotFound: "ClientEventDealNotFound", ClientEventDealAccepted: "ClientEventDealAccepted", ClientEventProviderCancelled: "ClientEventProviderCancelled", ClientEventUnknownResponseReceived: "ClientEventUnknownResponseReceived", ClientEventLastPaymentRequested: "ClientEventLastPaymentRequested", ClientEventAllBlocksReceived: "ClientEventAllBlocksReceived", ClientEventPaymentRequested: "ClientEventPaymentRequested", ClientEventUnsealPaymentRequested: "ClientEventUnsealPaymentRequested", ClientEventBlocksReceived: "ClientEventBlocksReceived", ClientEventSendFunds: "ClientEventSendFunds", ClientEventFundsExpended: "ClientEventFundsExpended", ClientEventBadPaymentRequested: "ClientEventBadPaymentRequested", ClientEventCreateVoucherFailed: "ClientEventCreateVoucherFailed", ClientEventWriteDealPaymentErrored: "ClientEventWriteDealPaymentErrored", ClientEventPaymentSent: "ClientEventPaymentSent", ClientEventDataTransferError: "ClientEventDataTransferError", ClientEventComplete: "ClientEventComplete", ClientEventCancelComplete: "ClientEventCancelComplete", ClientEventEarlyTermination: "ClientEventEarlyTermination", ClientEventCompleteVerified: "ClientEventCompleteVerified", ClientEventLaneAllocated: "ClientEventLaneAllocated", ClientEventVoucherShortfall: "ClientEventVoucherShortfall", ClientEventRecheckFunds: "ClientEventRecheckFunds", ClientEventCancel: "ClientEventCancel", }
ClientEvents is a human readable map of client event name -> event description
var DealPaymentUndefined = DealPayment{}
DealPaymentUndefined is an undefined deal payment
var DealProposalUndefined = DealProposal{}
DealProposalUndefined is an undefined deal proposal
var DealResponseUndefined = DealResponse{}
DealResponseUndefined is an undefined deal response
var DealStatuses = map[DealStatus]string{ DealStatusNew: "DealStatusNew", DealStatusUnsealing: "DealStatusUnsealing", DealStatusUnsealed: "DealStatusUnsealed", DealStatusWaitForAcceptance: "DealStatusWaitForAcceptance", DealStatusPaymentChannelCreating: "DealStatusPaymentChannelCreating", DealStatusPaymentChannelAddingFunds: "DealStatusPaymentChannelAddingFunds", DealStatusAccepted: "DealStatusAccepted", DealStatusFundsNeededUnseal: "DealStatusFundsNeededUnseal", DealStatusFailing: "DealStatusFailing", DealStatusRejected: "DealStatusRejected", DealStatusFundsNeeded: "DealStatusFundsNeeded", DealStatusSendFunds: "DealStatusSendFunds", DealStatusSendFundsLastPayment: "DealStatusSendFundsLastPayment", DealStatusOngoing: "DealStatusOngoing", DealStatusFundsNeededLastPayment: "DealStatusFundsNeededLastPayment", DealStatusCompleted: "DealStatusCompleted", DealStatusDealNotFound: "DealStatusDealNotFound", DealStatusErrored: "DealStatusErrored", DealStatusBlocksComplete: "DealStatusBlocksComplete", DealStatusFinalizing: "DealStatusFinalizing", DealStatusCompleting: "DealStatusCompleting", DealStatusCheckComplete: "DealStatusCheckComplete", DealStatusCheckFunds: "DealStatusCheckFunds", DealStatusInsufficientFunds: "DealStatusInsufficientFunds", DealStatusPaymentChannelAllocatingLane: "DealStatusPaymentChannelAllocatingLane", DealStatusCancelling: "DealStatusCancelling", DealStatusCancelled: "DealStatusCancelled", }
DealStatuses maps deal status to a human readable representation
var DefaultPaymentInterval = uint64(1 << 20)
DefaultPaymentInterval is the baseline interval, set to 1Mb if the miner does not explicitly set it otherwise
var DefaultPaymentIntervalIncrease = uint64(1 << 20)
DefaultPaymentIntervalIncrease is the amount interval increases on each payment, set to to 1Mb if the miner does not explicitly set it otherwise
var DefaultPricePerByte = abi.NewTokenAmount(2)
DefaultPricePerByte is the charge per byte retrieved if the miner does not specifically set it
var DefaultUnsealPrice = abi.NewTokenAmount(0)
DefaultUnsealPrice is the default charge to unseal a sector for retrieval
var ProviderEvents = map[ProviderEvent]string{ ProviderEventOpen: "ProviderEventOpen", ProviderEventDealNotFound: "ProviderEventDealNotFound", ProviderEventDealRejected: "ProviderEventDealRejected", ProviderEventDealAccepted: "ProviderEventDealAccepted", ProviderEventBlockSent: "ProviderEventBlockSent", ProviderEventBlocksCompleted: "ProviderEventBlocksCompleted", ProviderEventPaymentRequested: "ProviderEventPaymentRequested", ProviderEventSaveVoucherFailed: "ProviderEventSaveVoucherFailed", ProviderEventPartialPaymentReceived: "ProviderEventPartialPaymentReceived", ProviderEventPaymentReceived: "ProviderEventPaymentReceived", ProviderEventComplete: "ProviderEventComplete", ProviderEventUnsealError: "ProviderEventUnsealError", ProviderEventUnsealComplete: "ProviderEventUnsealComplete", ProviderEventDataTransferError: "ProviderEventDataTransferError", ProviderEventCancelComplete: "ProviderEventCancelComplete", ProviderEventCleanupComplete: "ProviderEventCleanupComplete", ProviderEventMultiStoreError: "ProviderEventMultiStoreError", ProviderEventClientCancelled: "ProviderEventClientCancelled", }
ProviderEvents is a human readable map of provider event name -> event description
var QueryResponseUndefined = QueryResponse{}
QueryResponseUndefined is an empty QueryResponse
var QueryUndefined = Query{}
QueryUndefined is a query with no values
Functions ¶
func DecodeNode ¶
DecodeNode validates and computes a decoded ipld.Node selector from the provided cbor-encoded selector
func IsTerminalError ¶
func IsTerminalError(status DealStatus) bool
IsTerminalError returns true if this status indicates processing of this deal is complete with an error
func IsTerminalStatus ¶
func IsTerminalStatus(status DealStatus) bool
IsTerminalStatus returns true if this status indicates processing of a deal is complete (either success or error)
func IsTerminalSuccess ¶
func IsTerminalSuccess(status DealStatus) bool
IsTerminalSuccess returns true if this status indicates processing of this deal is complete with a success
func NewShortfallError ¶ added in v0.6.0
func NewShortfallError(shortfall abi.TokenAmount) error
NewShortfallError returns a new error indicating a shortfall of funds
Types ¶
type Ask ¶ added in v0.5.3
type Ask struct { PricePerByte abi.TokenAmount UnsealPrice abi.TokenAmount PaymentInterval uint64 PaymentIntervalIncrease uint64 }
type AskStore ¶ added in v0.6.2
AskStore is an interface which provides access to a persisted retrieval Ask
type ChannelAvailableFunds ¶ added in v0.6.0
type ChannelAvailableFunds struct { // ConfirmedAmt is the amount of funds that have been confirmed on-chain // for the channel ConfirmedAmt abi.TokenAmount // PendingAmt is the amount of funds that are pending confirmation on-chain PendingAmt abi.TokenAmount // PendingWaitSentinel can be used with PaychGetWaitReady to wait for // confirmation of pending funds PendingWaitSentinel *cid.Cid // QueuedAmt is the amount that is queued up behind a pending request QueuedAmt abi.TokenAmount // VoucherRedeemedAmt is the amount that is redeemed by vouchers on-chain // and in the local datastore VoucherReedeemedAmt abi.TokenAmount }
ChannelAvailableFunds provides information about funds in a channel
type ClientDealState ¶
type ClientDealState struct { DealProposal StoreID *multistore.StoreID ChannelID datatransfer.ChannelID LastPaymentRequested bool AllBlocksReceived bool TotalFunds abi.TokenAmount ClientWallet address.Address MinerWallet address.Address PaymentInfo *PaymentInfo Status DealStatus Sender peer.ID TotalReceived uint64 Message string BytesPaidFor uint64 CurrentInterval uint64 PaymentRequested abi.TokenAmount FundsSpent abi.TokenAmount UnsealFundsPaid abi.TokenAmount WaitMsgCID *cid.Cid // the CID of any message the client deal is waiting for VoucherShortfall abi.TokenAmount }
ClientDealState is the current state of a deal from the point of view of a retrieval client
func (*ClientDealState) MarshalCBOR ¶
func (t *ClientDealState) MarshalCBOR(w io.Writer) error
func (*ClientDealState) UnmarshalCBOR ¶
func (t *ClientDealState) UnmarshalCBOR(r io.Reader) error
type ClientEvent ¶
type ClientEvent uint64
ClientEvent is an event that occurs in a deal lifecycle on the client
const ( // ClientEventOpen indicates a deal was initiated ClientEventOpen ClientEvent = iota // ClientEventWriteDealProposalErrored means a network error writing a deal proposal ClientEventWriteDealProposalErrored // ClientEventDealProposed means a deal was successfully sent to a miner ClientEventDealProposed // ClientEventDealRejected means a deal was rejected by the provider ClientEventDealRejected // ClientEventDealNotFound means a provider could not find a piece for a deal ClientEventDealNotFound // ClientEventDealAccepted means a provider accepted a deal ClientEventDealAccepted // ClientEventProviderCancelled means a provider has sent a message to cancel a deal ClientEventProviderCancelled // ClientEventUnknownResponseReceived means a client received a response it doesn't // understand from the provider ClientEventUnknownResponseReceived // ClientEventPaymentChannelErrored means there was a failure creating a payment channel ClientEventPaymentChannelErrored // ClientEventAllocateLaneErrored means there was a failure creating a lane in a payment channel ClientEventAllocateLaneErrored // ClientEventPaymentChannelCreateInitiated means we are waiting for a message to // create a payment channel to appear on chain ClientEventPaymentChannelCreateInitiated // ClientEventPaymentChannelReady means the newly created payment channel is ready for the // deal to resume ClientEventPaymentChannelReady // ClientEventPaymentChannelAddingFunds mean we are waiting for funds to be // added to a payment channel ClientEventPaymentChannelAddingFunds // ClientEventPaymentChannelAddFundsErrored means that adding funds to the payment channel // failed ClientEventPaymentChannelAddFundsErrored // ClientEventLastPaymentRequested indicates the provider requested a final payment ClientEventLastPaymentRequested // ClientEventAllBlocksReceived indicates the provider has sent all blocks ClientEventAllBlocksReceived // ClientEventPaymentRequested indicates the provider requested a payment ClientEventPaymentRequested // ClientEventUnsealPaymentRequested indicates the provider requested a payment for unsealing the sector ClientEventUnsealPaymentRequested // ClientEventBlocksReceived indicates the provider has sent blocks ClientEventBlocksReceived // ClientEventSendFunds emits when we reach the threshold to send the next payment ClientEventSendFunds // ClientEventFundsExpended indicates a deal has run out of funds in the payment channel // forcing the client to add more funds to continue the deal ClientEventFundsExpended // when totalFunds is expended // ClientEventBadPaymentRequested indicates the provider asked for funds // in a way that does not match the terms of the deal ClientEventBadPaymentRequested // ClientEventCreateVoucherFailed indicates an error happened creating a payment voucher ClientEventCreateVoucherFailed // ClientEventWriteDealPaymentErrored indicates a network error trying to write a payment ClientEventWriteDealPaymentErrored // ClientEventPaymentSent indicates a payment was sent to the provider ClientEventPaymentSent // ClientEventComplete indicates a deal has completed ClientEventComplete // ClientEventDataTransferError emits when something go wrong at the data transfer level ClientEventDataTransferError // ClientEventCancelComplete happens when a deal cancellation is transmitted to the provider ClientEventCancelComplete // ClientEventEarlyTermination indications a provider send a deal complete without sending all data ClientEventEarlyTermination // ClientEventCompleteVerified means that a provider completed without requesting a final payment but // we verified we received all data ClientEventCompleteVerified // ClientEventLaneAllocated is called when a lane is allocated ClientEventLaneAllocated // ClientEventVoucherShortfall means we tried to create a voucher but did not have enough funds in channel // to create it ClientEventVoucherShortfall // ClientEventRecheckFunds runs when an external caller indicates there may be new funds in a payment channel ClientEventRecheckFunds // ClientEventCancel runs when a user cancels a deal ClientEventCancel )
type ClientSubscriber ¶
type ClientSubscriber func(event ClientEvent, state ClientDealState)
ClientSubscriber is a callback that is registered to listen for retrieval events
type DealPayment ¶
type DealPayment struct { ID DealID PaymentChannel address.Address PaymentVoucher *paych.SignedVoucher }
DealPayment is a payment for an in progress retrieval deal
func (*DealPayment) MarshalCBOR ¶
func (t *DealPayment) MarshalCBOR(w io.Writer) error
func (*DealPayment) Type ¶ added in v0.5.0
func (dr *DealPayment) Type() datatransfer.TypeIdentifier
Type method makes DealPayment usable as a voucher
func (*DealPayment) UnmarshalCBOR ¶
func (t *DealPayment) UnmarshalCBOR(r io.Reader) error
type DealProposal ¶
DealProposal is a proposal for a new retrieval deal
func (*DealProposal) MarshalCBOR ¶
func (t *DealProposal) MarshalCBOR(w io.Writer) error
func (*DealProposal) Type ¶ added in v0.5.0
func (dp *DealProposal) Type() datatransfer.TypeIdentifier
Type method makes DealProposal usable as a voucher
func (*DealProposal) UnmarshalCBOR ¶
func (t *DealProposal) UnmarshalCBOR(r io.Reader) error
type DealResponse ¶
type DealResponse struct { Status DealStatus ID DealID // payment required to proceed PaymentOwed abi.TokenAmount Message string }
DealResponse is a response to a retrieval deal proposal
func (*DealResponse) MarshalCBOR ¶
func (t *DealResponse) MarshalCBOR(w io.Writer) error
func (*DealResponse) Type ¶ added in v0.5.0
func (dr *DealResponse) Type() datatransfer.TypeIdentifier
Type method makes DealResponse usable as a voucher result
func (*DealResponse) UnmarshalCBOR ¶
func (t *DealResponse) UnmarshalCBOR(r io.Reader) error
type DealStatus ¶
type DealStatus uint64
DealStatus is the status of a retrieval deal returned by a provider in a DealResponse
const ( // DealStatusNew is a deal that nothing has happened with yet DealStatusNew DealStatus = iota // DealStatusUnsealing means the provider is unsealing data DealStatusUnsealing // DealStatusUnsealed means the provider has finished unsealing data DealStatusUnsealed // DealStatusWaitForAcceptance means we're waiting to hear back if the provider accepted our deal DealStatusWaitForAcceptance // DealStatusPaymentChannelCreating is the status set while waiting for the // payment channel creation to complete DealStatusPaymentChannelCreating // DealStatusPaymentChannelAddingFunds is the status when we are waiting for funds // to finish being sent to the payment channel DealStatusPaymentChannelAddingFunds // DealStatusAccepted means a deal has been accepted by a provider // and its is ready to proceed with retrieval DealStatusAccepted // DealStatusFundsNeededUnseal means a deal has been accepted by a provider // and payment is needed to unseal the data DealStatusFundsNeededUnseal // DealStatusFailing indicates something went wrong during a retrieval, // and we are cleaning up before termianting with an error DealStatusFailing // DealStatusRejected indicates the provider rejected a client's deal proposal // for some reason DealStatusRejected // DealStatusFundsNeeded indicates the provider needs a payment voucher to // continue processing the deal DealStatusFundsNeeded // DealStatusSendFunds indicats the client is now going to send funds because we reached the threshold of the last payment DealStatusSendFunds // DealStatusSendFundsLastPayment indicats the client is now going to send final funds because // we reached the threshold of the final payment DealStatusSendFundsLastPayment // DealStatusOngoing indicates the provider is continuing to process a deal DealStatusOngoing // DealStatusFundsNeededLastPayment indicates the provider needs a payment voucher // in order to complete a deal DealStatusFundsNeededLastPayment // DealStatusCompleted indicates a deal is complete DealStatusCompleted // DealStatusDealNotFound indicates an update was received for a deal that could // not be identified DealStatusDealNotFound // DealStatusErrored indicates a deal has terminated in an error DealStatusErrored // DealStatusBlocksComplete indicates that all blocks have been processed for the piece DealStatusBlocksComplete // DealStatusFinalizing means the last payment has been received and // we are just confirming the deal is complete DealStatusFinalizing // DealStatusCompleting is just an inbetween state to perform final cleanup of // complete deals DealStatusCompleting // DealStatusCheckComplete is used for when the provided completes without a last payment // requested cycle, to verify we have received all blocks DealStatusCheckComplete // DealStatusCheckFunds means we are looking at the state of funding for the channel to determine // if more money is incoming DealStatusCheckFunds // DealStatusInsufficientFunds indicates we have depleted funds for the retrieval payment channel // - we can resume after funds are added DealStatusInsufficientFunds // DealStatusPaymentChannelAllocatingLane is the status when we are making a lane for this channel DealStatusPaymentChannelAllocatingLane // DealStatusCancelling means we are cancelling an inprogress deal DealStatusCancelling // DealStatusCancelled means a deal has been cancelled DealStatusCancelled )
type Params ¶
type Params struct { Selector *cbg.Deferred // V1 PieceCID *cid.Cid PricePerByte abi.TokenAmount PaymentInterval uint64 // when to request payment PaymentIntervalIncrease uint64 UnsealPrice abi.TokenAmount }
Params are the parameters requested for a retrieval deal proposal
func NewParamsV0 ¶
func NewParamsV0(pricePerByte abi.TokenAmount, paymentInterval uint64, paymentIntervalIncrease uint64) Params
NewParamsV0 generates parameters for a retrieval deal, which is always a whole piece deal
func NewParamsV1 ¶
func NewParamsV1(pricePerByte abi.TokenAmount, paymentInterval uint64, paymentIntervalIncrease uint64, sel ipld.Node, pieceCid *cid.Cid, unsealPrice abi.TokenAmount) (Params, error)
NewParamsV1 generates parameters for a retrieval deal, including a selector
func (Params) SelectorSpecified ¶ added in v0.5.1
type PaymentInfo ¶
type PaymentInfo struct { PayCh address.Address Lane uint64 }
PaymentInfo is the payment channel and lane for a deal, once it is setup
func (*PaymentInfo) MarshalCBOR ¶
func (t *PaymentInfo) MarshalCBOR(w io.Writer) error
func (*PaymentInfo) UnmarshalCBOR ¶
func (t *PaymentInfo) UnmarshalCBOR(r io.Reader) error
type PeerResolver ¶
type PeerResolver interface {
GetPeers(payloadCID cid.Cid) ([]RetrievalPeer, error) // TODO: channel
}
PeerResolver is an interface for looking up providers that may have a piece
type ProviderDealIdentifier ¶
ProviderDealIdentifier is a value that uniquely identifies a deal
func (ProviderDealIdentifier) String ¶
func (p ProviderDealIdentifier) String() string
type ProviderDealState ¶
type ProviderDealState struct { DealProposal StoreID multistore.StoreID ChannelID datatransfer.ChannelID PieceInfo *piecestore.PieceInfo Status DealStatus Receiver peer.ID TotalSent uint64 FundsReceived abi.TokenAmount Message string CurrentInterval uint64 }
ProviderDealState is the current state of a deal from the point of view of a retrieval provider
func (ProviderDealState) Identifier ¶
func (pds ProviderDealState) Identifier() ProviderDealIdentifier
Identifier provides a unique id for this provider deal
func (*ProviderDealState) MarshalCBOR ¶
func (t *ProviderDealState) MarshalCBOR(w io.Writer) error
func (*ProviderDealState) UnmarshalCBOR ¶
func (t *ProviderDealState) UnmarshalCBOR(r io.Reader) error
type ProviderEvent ¶
type ProviderEvent uint64
ProviderEvent is an event that occurs in a deal lifecycle on the provider
const ( // ProviderEventOpen indicates a new deal was received from a client ProviderEventOpen ProviderEvent = iota // ProviderEventDealNotFound happens when the provider cannot find the piece for the // deal proposed by the client ProviderEventDealNotFound // ProviderEventDealRejected happens when a provider rejects a deal proposed // by the client ProviderEventDealRejected // ProviderEventDealAccepted happens when a provider accepts a deal ProviderEventDealAccepted // ProviderEventBlockSent happens when the provider reads another block // in the piece ProviderEventBlockSent // ProviderEventBlocksCompleted happens when the provider reads the last block // in the piece ProviderEventBlocksCompleted // ProviderEventPaymentRequested happens when a provider asks for payment from // a client for blocks sent ProviderEventPaymentRequested // ProviderEventSaveVoucherFailed happens when an attempt to save a payment // voucher fails ProviderEventSaveVoucherFailed // ProviderEventPartialPaymentReceived happens when a provider receives and processes // a payment that is less than what was requested to proceed with the deal ProviderEventPartialPaymentReceived // ProviderEventPaymentReceived happens when a provider receives a payment // and resumes processing a deal ProviderEventPaymentReceived // ProviderEventComplete indicates a retrieval deal was completed for a client ProviderEventComplete // ProviderEventUnsealError emits when something wrong occurs while unsealing data ProviderEventUnsealError // ProviderEventUnsealComplete emits when the unsealing process is done ProviderEventUnsealComplete // ProviderEventDataTransferError emits when something go wrong at the data transfer level ProviderEventDataTransferError // ProviderEventCancelComplete happens when a deal cancellation is transmitted to the provider ProviderEventCancelComplete // ProviderEventCleanupComplete happens when a deal is finished cleaning up and enters a complete state ProviderEventCleanupComplete // ProviderEventMultiStoreError occurs when an error happens attempting to operate on the multistore ProviderEventMultiStoreError // ProviderEventClientCancelled happens when the provider gets a cancel message from the client's data transfer ProviderEventClientCancelled )
type ProviderSubscriber ¶
type ProviderSubscriber func(event ProviderEvent, state ProviderDealState)
ProviderSubscriber is a callback that is registered to listen for retrieval events on a provider
type Query ¶
type Query struct { PayloadCID cid.Cid // V0 QueryParams // V1 }
Query is a query to a given provider to determine information about a piece they may have available for retrieval
func NewQueryV0 ¶
func NewQueryV0(payloadCID cid.Cid) Query
NewQueryV0 creates a V0 query (which only specifies a payload)
func NewQueryV1 ¶ added in v0.4.0
func NewQueryV1(payloadCID cid.Cid, pieceCID *cid.Cid) Query
NewQueryV1 creates a V1 query (which has an optional pieceCID)
type QueryItemStatus ¶
type QueryItemStatus uint64
QueryItemStatus (V1) indicates whether the requested part of a piece (payload or selector) is available for retrieval
const ( // QueryItemAvailable indicates requested part of the piece is available to be // served QueryItemAvailable QueryItemStatus = iota // item or it cannot be served QueryItemUnavailable // QueryItemUnknown indicates the provider cannot determine if the given item // is part of the requested piece (for example, if the piece is sealed and the // miner does not maintain a payload CID index) QueryItemUnknown )
type QueryParams ¶
type QueryParams struct {
PieceCID *cid.Cid // optional, query if miner has this cid in this piece. some miners may not be able to respond.
}
QueryParams - V1 - indicate what specific information about a piece that a retrieval client is interested in, as well as specific parameters the client is seeking for the retrieval deal
func (*QueryParams) MarshalCBOR ¶
func (t *QueryParams) MarshalCBOR(w io.Writer) error
func (*QueryParams) UnmarshalCBOR ¶
func (t *QueryParams) UnmarshalCBOR(r io.Reader) error
type QueryResponse ¶
type QueryResponse struct { Status QueryResponseStatus PieceCIDFound QueryItemStatus // V1 - if a PieceCID was requested, the result Size uint64 // Total size of piece in bytes PaymentAddress address.Address // address to send funds to -- may be different than miner addr MinPricePerByte abi.TokenAmount MaxPaymentInterval uint64 MaxPaymentIntervalIncrease uint64 Message string UnsealPrice abi.TokenAmount }
QueryResponse is a miners response to a given retrieval query
func (*QueryResponse) MarshalCBOR ¶
func (t *QueryResponse) MarshalCBOR(w io.Writer) error
func (QueryResponse) PieceRetrievalPrice ¶
func (qr QueryResponse) PieceRetrievalPrice() abi.TokenAmount
PieceRetrievalPrice is the total price to retrieve the piece (size * MinPricePerByte + UnsealedPrice)
func (*QueryResponse) UnmarshalCBOR ¶
func (t *QueryResponse) UnmarshalCBOR(r io.Reader) error
type QueryResponseStatus ¶
type QueryResponseStatus uint64
QueryResponseStatus indicates whether a queried piece is available
const ( // QueryResponseAvailable indicates a provider has a piece and is prepared to // return it QueryResponseAvailable QueryResponseStatus = iota // serve the queried piece to the client QueryResponseUnavailable // QueryResponseError indicates something went wrong generating a query response QueryResponseError )
type RetrievalClient ¶
type RetrievalClient interface { // Find Providers finds retrieval providers who may be storing a given piece FindProviders(payloadCID cid.Cid) []RetrievalPeer // Query asks a provider for information about a piece it is storing Query( ctx context.Context, p RetrievalPeer, payloadCID cid.Cid, params QueryParams, ) (QueryResponse, error) // Retrieve retrieves all or part of a piece with the given retrieval parameters Retrieve( ctx context.Context, payloadCID cid.Cid, params Params, totalFunds abi.TokenAmount, p RetrievalPeer, clientWallet address.Address, minerWallet address.Address, storeID *multistore.StoreID, ) (DealID, error) // SubscribeToEvents listens for events that happen related to client retrievals SubscribeToEvents(subscriber ClientSubscriber) Unsubscribe // TryRestartInsufficientFunds attempts to restart any deals stuck in the insufficient funds state // after funds are added to a given payment channel TryRestartInsufficientFunds(paymentChannel address.Address) error // CancelDeal attempts to cancel an inprogress deal CancelDeal(id DealID) error // GetDeal returns a given deal by deal ID, if it exists GetDeal(dealID DealID) (ClientDealState, error) // ListDeals returns all deals ListDeals() (map[DealID]ClientDealState, error) }
RetrievalClient is a client interface for making retrieval deals
type RetrievalClientNode ¶
type RetrievalClientNode interface { GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error) // GetOrCreatePaymentChannel sets up a new payment channel if one does not exist // between a client and a miner and ensures the client has the given amount of funds available in the channel GetOrCreatePaymentChannel(ctx context.Context, clientAddress, minerAddress address.Address, clientFundsAvailable abi.TokenAmount, tok shared.TipSetToken) (address.Address, cid.Cid, error) // CheckAvailableFunds returns the amount of current and incoming funds in a channel CheckAvailableFunds(ctx context.Context, paymentChannel address.Address) (ChannelAvailableFunds, error) // Allocate late creates a lane within a payment channel so that calls to // CreatePaymentVoucher will automatically make vouchers only for the difference // in total AllocateLane(ctx context.Context, paymentChannel address.Address) (uint64, error) // CreatePaymentVoucher creates a new payment voucher in the given lane for a // given payment channel so that all the payment vouchers in the lane add up // to the given amount (so the payment voucher will be for the difference) CreatePaymentVoucher(ctx context.Context, paymentChannel address.Address, amount abi.TokenAmount, lane uint64, tok shared.TipSetToken) (*paych.SignedVoucher, error) // WaitForPaymentChannelReady just waits for the payment channel's pending operations to complete WaitForPaymentChannelReady(ctx context.Context, waitSentinel cid.Cid) (address.Address, error) // GetKnownAddresses gets any on known multiaddrs for a given address, so we can add to the peer store GetKnownAddresses(ctx context.Context, p RetrievalPeer, tok shared.TipSetToken) ([]ma.Multiaddr, error) }
RetrievalClientNode are the node dependencies for a RetrievalClient
type RetrievalPeer ¶
RetrievalPeer is a provider address/peer.ID pair (everything needed to make deals for with a miner)
func (*RetrievalPeer) MarshalCBOR ¶ added in v0.4.0
func (t *RetrievalPeer) MarshalCBOR(w io.Writer) error
func (*RetrievalPeer) UnmarshalCBOR ¶ added in v0.4.0
func (t *RetrievalPeer) UnmarshalCBOR(r io.Reader) error
type RetrievalProvider ¶
type RetrievalProvider interface { // Start begins listening for deals on the given host Start() error // Stop stops handling incoming requests Stop() error // SetAsk sets the retrieval payment parameters that this miner will accept SetAsk(ask *Ask) // GetAsk returns the retrieval providers pricing information GetAsk() *Ask // SubscribeToEvents listens for events that happen related to client retrievals SubscribeToEvents(subscriber ProviderSubscriber) Unsubscribe ListDeals() map[ProviderDealIdentifier]ProviderDealState }
RetrievalProvider is an interface by which a provider configures their retrieval operations and monitors deals received and process
type RetrievalProviderNode ¶
type RetrievalProviderNode interface { GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error) // returns the worker address associated with a miner GetMinerWorkerAddress(ctx context.Context, miner address.Address, tok shared.TipSetToken) (address.Address, error) UnsealSector(ctx context.Context, sectorID abi.SectorNumber, offset abi.UnpaddedPieceSize, length abi.UnpaddedPieceSize) (io.ReadCloser, error) SavePaymentVoucher(ctx context.Context, paymentChannel address.Address, voucher *paych.SignedVoucher, proof []byte, expectedAmount abi.TokenAmount, tok shared.TipSetToken) (abi.TokenAmount, error) }
RetrievalProviderNode are the node depedencies for a RetrevalProvider
type ShortfallError ¶ added in v0.6.0
type ShortfallError struct {
// contains filtered or unexported fields
}
ShortfallErorr is an error that indicates a short fall of funds
func (ShortfallError) Error ¶ added in v0.6.0
func (se ShortfallError) Error() string
func (ShortfallError) Shortfall ¶ added in v0.6.0
func (se ShortfallError) Shortfall() abi.TokenAmount
Shortfall returns the numerical value of the shortfall
type Unsubscribe ¶
type Unsubscribe func()
Unsubscribe is a function that unsubscribes a subscriber for either the client or the provider
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
Package retrievalimpl provides the primary implementation of retrieval market top level interfaces interfaces This package provides a production implementation of `RetrievalClient` and `RetrievalProvider`.
|
Package retrievalimpl provides the primary implementation of retrieval market top level interfaces interfaces This package provides a production implementation of `RetrievalClient` and `RetrievalProvider`. |
clientstates
Package clientstates contains state machine logic relating to the `RetrievalClient`.
|
Package clientstates contains state machine logic relating to the `RetrievalClient`. |
dtutils
Package dtutils provides event listeners for the client and provider to listen for events on the data transfer module and dispatch FSM events based on them
|
Package dtutils provides event listeners for the client and provider to listen for events on the data transfer module and dispatch FSM events based on them |
providerstates
Package providerstates contains state machine logic relating to the `RetrievalProvider`.
|
Package providerstates contains state machine logic relating to the `RetrievalProvider`. |
testnodes
Package testnodes contains stubbed implementations of the RetrievalProviderNode and RetrievalClientNode interface to simulate communications with a filecoin node
|
Package testnodes contains stubbed implementations of the RetrievalProviderNode and RetrievalClientNode interface to simulate communications with a filecoin node |
Package network providers an abstraction over a libp2p host for managing retrieval's Libp2p protocols: network.go - defines the interfaces that must be implemented to serve as a retrieval network deal-stream.go - implements the `RetrievalDealStream` interface, a data stream for retrieval deal traffic only query-stream.go - implements the `RetrievalQueryStream` interface, a data stream for retrieval query traffic only libp2p_impl.go - provides the production implementation of the `RetrievalMarketNetwork` interface.
|
Package network providers an abstraction over a libp2p host for managing retrieval's Libp2p protocols: network.go - defines the interfaces that must be implemented to serve as a retrieval network deal-stream.go - implements the `RetrievalDealStream` interface, a data stream for retrieval deal traffic only query-stream.go - implements the `RetrievalQueryStream` interface, a data stream for retrieval query traffic only libp2p_impl.go - provides the production implementation of the `RetrievalMarketNetwork` interface. |
Package testing provides test implementations of retieval market interfaces
|
Package testing provides test implementations of retieval market interfaces |