ipld

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2022 License: Apache-2.0 Imports: 32 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MaxSquareSize is currently the maximum size supported for unerasured data in rsmt2d.ExtendedDataSquare.
	MaxSquareSize = consts.MaxSquareSize
	// NamespaceSize is a system-wide size for NMT namespaces.
	NamespaceSize = consts.NamespaceSize
	// ShareSize is a system-wide size of a share, including both data and namespace ID
	ShareSize = consts.ShareSize
)

Variables

View Source 
var DefaultRSMT2DCodec = consts.DefaultCodec

DefaultRSMT2DCodec sets the default rsmt2d.Codec for shares.

View Source 
var NumConcurrentSquares = 8

NumConcurrentSquares limits the amount of squares that are fetched concurrently/simultaneously.

View Source 
var NumWorkersLimit = MaxSquareSize * MaxSquareSize / 2 * NumConcurrentSquares

NumWorkersLimit sets global limit for workers spawned by GetShares. GetShares could be called MaxSquareSize(128) times per data square each spawning up to 128/2 goroutines and altogether this is 8192. Considering there can be N blocks fetched at the same time, e.g. during catching up data from the past, we multiply this number by the amount of allowed concurrent data square fetches(NumConcurrentSquares).

NOTE: This value only limits amount of simultaneously running workers that are spawned as the load increases and are killed, once the load declines.

TODO(@Wondertan): This assumes we have parallelized DASer implemented. Sync the values once it is shipped. TODO(@Wondertan): Allow configuration of values without global state.

View Source 
var RetrieveQuadrantTimeout = blockTime / numQuadrants * 2

RetrieveQuadrantTimeout defines how much time Retriever waits before starting to retrieve another quadrant.

NOTE: - The whole data square must be retrieved in less than block time. - We have 4 quadrants from two sources(rows, cols) which equals to 8 in total.

Functions

func AddShares added in v0.3.0 

func AddShares(
	ctx context.Context,
	shares []Share,
	adder blockservice.BlockService,
) (*rsmt2d.ExtendedDataSquare, error)

AddShares erasures and extends shares to blockservice.BlockService using the provided ipld.NodeAdder.

func EqualEDS 

func EqualEDS(a *rsmt2d.ExtendedDataSquare, b *rsmt2d.ExtendedDataSquare) bool

EqualEDS check whether two given EDSes are equal. TODO(Wondertan): Move to rsmt2d TODO(Wondertan): Propose use of int by default instead of uint for the sake convenience and Golang practices

func GetLeaf 

func GetLeaf(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid, leaf, total int) (ipld.Node, error)

GetLeaf fetches and returns the raw leaf. It walks down the IPLD NMT tree until it finds the requested one.

func GetProof added in v0.3.0 

func GetProof(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	proof []cid.Cid,
	leaf, total int,
) ([]cid.Cid, error)

GetProof fetches and returns the leaf's Merkle Proof. It walks down the IPLD NMT tree until it reaches the leaf and returns collected proof

func GetShares added in v0.3.0 

func GetShares(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid, shares int, put func(int, Share))

GetShares gets shares from either local storage, or, if not found, requests them from immediate/connected peers. It puts them into the given 'put' func, does not return any error, and returns/unblocks only on success (got all shares) or on context cancellation.

It works concurrently by spawning workers in the pool which do one basic thing - block until data is fetched, s. t. share processing is never sequential, and thus we request *all* the shares available without waiting for others to finish. It is the required property to maximize data availability. As a side effect, we get concurrent tree traversal reducing time to data time.

GetShares relies on the fact that the underlying data structure is a binary tree, so it's not suitable for anything else besides that. Parts on the implementation that rely on this property are explicitly tagged with (bin-tree-feat).

func ImportShares added in v0.3.0 

func ImportShares(
	ctx context.Context,
	shares [][]byte,
	adder blockservice.BlockService) (*rsmt2d.ExtendedDataSquare, error)

ImportShares imports flattend chunks of data into Extended Data square and saves it in blockservice.BlockService

func ProtoToProof added in v0.3.0 

func ProtoToProof(protoProof *pb.MerkleProof) nmt.Proof

func RandEDS 

func RandEDS(t *testing.T, size int) *rsmt2d.ExtendedDataSquare

RandEDS generates EDS filled with the random data with the given size for original square.

func SanityCheckNID added in v0.3.0 

func SanityCheckNID(nID []byte) error

func ShareData added in v0.3.0 

func ShareData(s Share) []byte

ShareData gets data from the share.

func ShareID added in v0.3.0 

func ShareID(s Share) namespace.ID

ShareID gets the namespace ID from the share.

Types

type ErrByzantine added in v0.3.0 

type ErrByzantine struct {
	Index  uint32
	Shares []*ShareWithProof
	Axis   rsmt2d.Axis
}

ErrByzantine is a thrown when recovered data square is not correct (merkle proofs do not match parity erasure-coding data).

It is converted from rsmt2d.ByzantineRow/Col + Merkle Proof for each share.

func NewErrByzantine added in v0.3.0 

func NewErrByzantine(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	dah *da.DataAvailabilityHeader,
	errByz *rsmt2d.ErrByzantineData,
) *ErrByzantine

NewErrByzantine creates new ErrByzantine from rsmt2d error. If error happens during proof collection, it terminates the process with os.Exit(1).

func (*ErrByzantine) Error added in v0.3.0 

func (e *ErrByzantine) Error() string

type NmtNodeAdder 

type NmtNodeAdder struct {
	// contains filtered or unexported fields
}

NmtNodeAdder adds ipld.Nodes to the underlying ipld.Batch if it is inserted into a nmt tree.

func NewNmtNodeAdder 

func NewNmtNodeAdder(ctx context.Context, bs blockservice.BlockService, opts ...ipld.BatchOption) *NmtNodeAdder

NewNmtNodeAdder returns a new NmtNodeAdder with the provided context and batch. Note that the context provided should have a timeout It is not thread-safe.

func (*NmtNodeAdder) Commit 

func (n *NmtNodeAdder) Commit() error

Commit checks for errors happened during Visit and if absent commits data to inner Batch.

func (*NmtNodeAdder) Visit 

func (n *NmtNodeAdder) Visit(hash []byte, children ...[]byte)

Visit is a NodeVisitor that can be used during the creation of a new NMT to create and add ipld.Nodes to the Batch while computing the root of the NMT.

type Retriever added in v0.3.0 

type Retriever struct {
	// contains filtered or unexported fields
}

Retriever retrieves rsmt2d.ExtendedDataSquares from the IPLD network. Instead of requesting data 'share by share' it requests data by quadrants minimizing bandwidth usage in the happy cases. 

 ---- ----
| 0  | 1  |
 ---- ----
| 2  | 3  |
 ---- ----

Retriever randomly picks one of the data square quadrants and tries to request them one by one until it is able to reconstruct the whole square.

func NewRetriever added in v0.3.0 

func NewRetriever(bServ blockservice.BlockService) *Retriever

NewRetriever creates a new instance of the Retriever over IPLD Service and rmst2d.Codec

func (*Retriever) Retrieve added in v0.3.0 

func (r *Retriever) Retrieve(ctx context.Context, dah *da.DataAvailabilityHeader) (*rsmt2d.ExtendedDataSquare, error)

Retrieve retrieves all the data committed to DataAvailabilityHeader.

If not available locally, it aims to request from the network only one quadrant (1/4) of the data square and reconstructs the other three quadrants (3/4). If the requested quadrant is not available within RetrieveQuadrantTimeout, it starts requesting another quadrant until either the data is reconstructed, context is canceled or ErrByzantine is generated.

type Share 

type Share = []byte

Share contains the raw share data without the corresponding namespace. NOTE: Alias for the byte is chosen to keep maximal compatibility, especially with rsmt2d. Ideally, we should define reusable type elsewhere and make everyone(Core, rsmt2d, ipld) to rely on it.

func ExtractEDS added in v0.3.0 

func ExtractEDS(eds *rsmt2d.ExtendedDataSquare) []Share

ExtractEDS takes an EDS and extracts all shares from it in a flattened slice(row by row).

func ExtractODS added in v0.3.0 

func ExtractODS(eds *rsmt2d.ExtendedDataSquare) []Share

ExtractODS returns the original shares of the given ExtendedDataSquare. This is a helper function for circumstances where AddShares must be used after the EDS has already been generated.

func GetShare added in v0.3.0 

func GetShare(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	rootCid cid.Cid,
	leafIndex int,
	totalLeafs int,
) (Share, error)

GetShare fetches and returns the data for leaf `leafIndex` of root `rootCid`.

func GetSharesByNamespace added in v0.3.0 

func GetSharesByNamespace(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	nID namespace.ID,
	maxShares int,
) ([]Share, error)

GetSharesByNamespace walks the tree of a given root and returns its shares within the given namespace.ID. If a share could not be retrieved, err is not nil, and the returned array contains nil shares in place of the shares it was unable to retrieve.

func RandShares added in v0.3.0 

func RandShares(t *testing.T, total int) []Share

RandShares generate 'total' amount of shares filled with random data.

type ShareWithProof added in v0.3.0 

type ShareWithProof struct {
	// Share is a full data including namespace
	Share
	// Proof is a Merkle Proof of current share
	Proof *nmt.Proof
}

ShareWithProof contains data with corresponding Merkle Proof

func GetProofsForShares added in v0.3.0 

func GetProofsForShares(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	shares [][]byte,
) ([]*ShareWithProof, error)

GetProofsForShares fetches Merkle proofs for the given shares and returns the result as an array of ShareWithProof.

func NewShareWithProof added in v0.3.0 

func NewShareWithProof(index int, share Share, pathToLeaf []cid.Cid) *ShareWithProof

NewShareWithProof takes the given leaf and its path, starting from the tree root, and computes the nmt.Proof for it.

func ProtoToShare added in v0.3.0 

func ProtoToShare(protoShares []*pb.Share) []*ShareWithProof

func (*ShareWithProof) ShareWithProofToProto added in v0.3.0 

func (s *ShareWithProof) ShareWithProofToProto() *pb.Share

func (*ShareWithProof) Validate added in v0.3.0 

func (s *ShareWithProof) Validate(root cid.Cid) bool

Validate validates inclusion of the share under the given root CID.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.