ipld

package
v0.3.0-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 25, 2022 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MaxSquareSize is currently the maximum size supported for unerasured data in rsmt2d.ExtendedDataSquare.
	MaxSquareSize = consts.MaxSquareSize
	// NamespaceSize is a system-wide size for NMT namespaces.
	NamespaceSize = consts.NamespaceSize
	// ShareSize is a system-wide size of a share, including both data and namespace ID
	ShareSize = consts.ShareSize
)

Variables

View Source 
var DefaultRSMT2DCodec = consts.DefaultCodec

DefaultRSMT2DCodec sets the default rsmt2d.Codec for shares.

View Source 
var NumConcurrentSquares = 8

NumConcurrentSquares limits the amount of squares that are fetched concurrently/simultaneously.

View Source 
var NumWorkersLimit = MaxSquareSize * MaxSquareSize / 2 * NumConcurrentSquares

NumWorkersLimit sets global limit for workers spawned by GetShares. GetShares could be called MaxSquareSize(128) times per data square each spawning up to 128/2 goroutines and altogether this is 8192. Considering there can be N blocks fetched at the same time, e.g. during catching up data from the past, we multiply this number by the amount of allowed concurrent data square fetches(NumConcurrentSquares).

NOTE: This value only limits amount of simultaneously running workers that are spawned as the load increases and are killed, once the load declines. TODO(@Wondertan): This assumes we have parallelized DASer implemented. 

Sync the values once it is shipped.

TODO(@Wondertan): Allow configuration of values without global state.

View Source 
var RetrieveQuadrantTimeout = time.Minute * 5

RetrieveQuadrantTimeout limits the time for retrieval of a quadrant so that Retriever can retry another quadrant.

Functions

func AddShares added in v0.3.0 

func AddShares(ctx context.Context, shares []Share, adder ipld.NodeAdder) (*rsmt2d.ExtendedDataSquare, error)

AddShares erasures and extends shares to IPLD DAG using the provided ipld.NodeAdder.

func EqualEDS 

func EqualEDS(a *rsmt2d.ExtendedDataSquare, b *rsmt2d.ExtendedDataSquare) bool

EqualEDS check whether two given EDSes are equal. TODO(Wondertan): Move to rsmt2d TODO(Wondertan): Propose use of int by default instead of uint for the sake convenience and Golang practices

func GetLeaf 

func GetLeaf(ctx context.Context, dag ipld.NodeGetter, root cid.Cid, leaf, total int) (ipld.Node, error)

GetLeaf fetches and returns the raw leaf. It walks down the IPLD NMT tree until it finds the requested one.

func GetLeavesByNamespace 

func GetLeavesByNamespace(
	ctx context.Context,
	dag ipld.NodeGetter,
	root cid.Cid,
	nID namespace.ID,
) ([]ipld.Node, error)

GetLeavesByNamespace returns all the leaves from the given root with the given namespace.ID. If nothing is found it returns both data and err as nil.

func GetProof added in v0.3.0 

func GetProof(
	ctx context.Context,
	dag ipld.NodeGetter,
	root cid.Cid,
	proof []cid.Cid,
	leaf, total int,
) ([]cid.Cid, error)

GetProof fetches and returns the leaf's Merkle Proof. It walks down the IPLD NMT tree until it reaches the leaf and returns collected proof

func GetShares added in v0.3.0 

func GetShares(ctx context.Context, dag format.NodeGetter, root cid.Cid, shares int, put func(int, Share))

GetShares gets shares from either local storage, or, if not found, requests them from immediate/connected peers. It puts them into the given 'put' func, does not return any error, and returns/unblocks only on success (got all shares) or on context cancellation.

It works concurrently by spawning workers in the pool which do one basic thing - block until data is fetched, s. t. share processing is never sequential, and thus we request *all* the shares available without waiting for others to finish. It is the required property to maximize data availability. As a side effect, we get concurrent tree traversal reducing time to data time.

GetShares relies on the fact that the underlying data structure is a binary tree, so it's not suitable for anything else besides that. Parts on the implementation that rely on this property are explicitly tagged with (bin-tree-feat).

func ImportShares added in v0.3.0 

func ImportShares(ctx context.Context, shares [][]byte, na ipld.NodeAdder) (*rsmt2d.ExtendedDataSquare, error)

ImportShares imports flattend chunks of data into Extended Data square and saves it in IPLD DAG

func ProtoToProof added in v0.3.0 

func ProtoToProof(protoProof *pb.MerkleProof) nmt.Proof

func RandEDS 

func RandEDS(t *testing.T, size int) *rsmt2d.ExtendedDataSquare

RandEDS generates EDS filled with the random data with the given size for original square.

func SanityCheckNID added in v0.3.0 

func SanityCheckNID(nID []byte) error

func ShareData added in v0.3.0 

func ShareData(s Share) []byte

ShareData gets data from the share.

func ShareID added in v0.3.0 

func ShareID(s Share) namespace.ID

ShareID gets the namespace ID from the share.

Types

type ErrByzantine added in v0.3.0 

type ErrByzantine struct {
	Index  uint8
	Shares []*ShareWithProof
	// TODO(@vgokivs): Change to enum type and rename to Axis after
	// updating rsmt2d
	IsRow bool
}

ErrByzantine is a thrown when recovered data square is not correct (merkle proofs do not match parity erasure-coding data).

It is converted from rsmt2d.ByzantineRow/Col + Merkle Proof for each share.

func NewErrByzantine added in v0.3.0 

func NewErrByzantine(
	ctx context.Context,
	dag format.NodeGetter,
	dah *da.DataAvailabilityHeader,
	errByz *rsmt2d.ErrByzantineData,
) *ErrByzantine

NewErrByzantine creates new ErrByzantine from rsmt2d error. If error happens during proof collection, it terminates the process with os.Exit(1). TODO(@Wondertan): Migrate to ErrByzantineData in the newest rsmt2d

func (*ErrByzantine) Error added in v0.3.0 

func (e *ErrByzantine) Error() string

type NmtNodeAdder 

type NmtNodeAdder struct {
	// contains filtered or unexported fields
}

NmtNodeAdder adds ipld.Nodes to the underlying ipld.Batch if it is inserted into a nmt tree.

func NewNmtNodeAdder 

func NewNmtNodeAdder(ctx context.Context, add ipld.NodeAdder) *NmtNodeAdder

NewNmtNodeAdder returns a new NmtNodeAdder with the provided context and batch. Note that the context provided should have a timeout It is not thread-safe.

func (*NmtNodeAdder) Commit 

func (n *NmtNodeAdder) Commit() error

Commit checks for errors happened during Visit and if absent commits data to inner Batch.

func (*NmtNodeAdder) NodeAdder 

func (n *NmtNodeAdder) NodeAdder() ipld.NodeAdder

NodeAdder returns the ipld.NodeAdder originally provided to the NmtNodeAdder.

func (*NmtNodeAdder) Visit 

func (n *NmtNodeAdder) Visit(hash []byte, children ...[]byte)

Visit is a NodeVisitor that can be used during the creation of a new NMT to create and add ipld.Nodes to the Batch while computing the root of the NMT.

type Retriever added in v0.3.0 

type Retriever struct {
	// contains filtered or unexported fields
}

Retriever retrieves rsmt2d.ExtendedDataSquares from the IPLD network. Instead of requesting data 'share by share' it requests data by quadrants minimizing bandwidth usage in the happy cases. 

---- ----

| 0 | 1 | 

---- ----

| 2 | 3 | 

---- ----

Retriever randomly picks one of the data square quadrants and tries to request them one by one until it is able to reconstruct the whole square.

func NewRetriever added in v0.3.0 

func NewRetriever(dag format.DAGService) *Retriever

NewRetriever creates a new instance of the Retriever over IPLD Service and rmst2d.Codec

func (*Retriever) Retrieve added in v0.3.0 

func (r *Retriever) Retrieve(ctx context.Context, dah *da.DataAvailabilityHeader) (*rsmt2d.ExtendedDataSquare, error)

Retrieve retrieves all the data committed to DataAvailabilityHeader. If not available locally, it aims to request from the network only 1/4 of the data and reconstructs other 3/4 parts using the rsmt2d.Codec. It steadily tries to request other 3/4 data if 1/4 is not found within RetrieveQuadrantTimeout or unrecoverable.

type Share 

type Share = []byte

Share contains the raw share data without the corresponding namespace. NOTE: Alias for the byte is chosen to keep maximal compatibility, especially with rsmt2d. Ideally, we should define reusable type elsewhere and make everyone(Core, rsmt2d, ipld) to rely on it.

func ExtractEDS added in v0.3.0 

func ExtractEDS(eds *rsmt2d.ExtendedDataSquare) []Share

ExtractEDS takes an EDS and extracts all shares from it in a flattened slice(row by row).

func ExtractODS added in v0.3.0 

func ExtractODS(eds *rsmt2d.ExtendedDataSquare) []Share

ExtractODS returns the original shares of the given ExtendedDataSquare. This is a helper function for circumstances where AddShares must be used after the EDS has already been generated.

func GetShare added in v0.3.0 

func GetShare(
	ctx context.Context,
	dag ipld.NodeGetter,
	rootCid cid.Cid,
	leafIndex int,
	totalLeafs int,
) (Share, error)

GetShare fetches and returns the data for leaf `leafIndex` of root `rootCid`.

func GetSharesByNamespace added in v0.3.0 

func GetSharesByNamespace(
	ctx context.Context,
	dag ipld.NodeGetter,
	root cid.Cid,
	nID namespace.ID,
) ([]Share, error)

GetSharesByNamespace returns all the shares from the given root with the given namespace.ID.

func RandShares added in v0.3.0 

func RandShares(t *testing.T, total int) []Share

RandShares generate 'total' amount of shares filled with random data.

type ShareWithProof added in v0.3.0 

type ShareWithProof struct {
	// Share is a full data including namespace
	Share
	// Proof is a Merkle Proof of current share
	Proof *nmt.Proof
}

ShareWithProof contains data with corresponding Merkle Proof

func GetProofsForShares added in v0.3.0 

func GetProofsForShares(
	ctx context.Context,
	dag ipld.NodeGetter,
	root cid.Cid,
	shares [][]byte,
) ([]*ShareWithProof, error)

GetProofsForShares fetches Merkle proofs for the given shares and returns the result as an array of ShareWithProof.

func NewShareWithProof added in v0.3.0 

func NewShareWithProof(index int, share Share, pathToLeaf []cid.Cid) *ShareWithProof

NewShareWithProof takes the given leaf and its path, starting from the tree root, and computes the nmt.Proof for it.

func ProtoToShare added in v0.3.0 

func ProtoToShare(protoShares []*pb.Share) []*ShareWithProof

func (*ShareWithProof) ShareWithProofToProto added in v0.3.0 

func (s *ShareWithProof) ShareWithProofToProto() *pb.Share

func (*ShareWithProof) Validate added in v0.3.0 

func (s *ShareWithProof) Validate(root cid.Cid) bool

Validate validates inclusion of the share under the given root CID.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.