ipld

Documentation

Index

• Constants
• Variables
• func BatchSize(squareSize int) int
• func CidFromNamespacedSha256(namespacedHash []byte) (cid.Cid, error)
• func GetLeaf(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid, ...) (ipld.Node, error)
• func GetLeaves(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid, ...)
• func GetLeavesByNamespace(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid, ...) ([]ipld.Node, error)
• func GetNode(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid) (ipld.Node, error)
• func GetProof(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid, ...) ([]cid.Cid, error)
• func MaxSizeBatchOption(size int) ipld.BatchOption
• func MustCidFromNamespacedSha256(hash []byte) cid.Cid
• func NamespacedSha256FromCID(cid cid.Cid) []byte
• func NewNMTLeafNode(id cid.Cid, data []byte) ipld.Node
• func NewNMTNode(id cid.Cid, l, r []byte) ipld.Node
• func RandNamespacedCID(t *testing.T) cid.Cid
• func Translate(dah *da.DataAvailabilityHeader, row, col int) (cid.Cid, int)
• type NmtNodeAdder
  • func NewNmtNodeAdder(ctx context.Context, bs blockservice.BlockService, opts ...ipld.BatchOption) *NmtNodeAdder
  • func (n *NmtNodeAdder) Commit() error
  • func (n *NmtNodeAdder) Visit(hash []byte, children ...[]byte)

Constants

const (

	// MaxSquareSize is currently the maximum size supported for unerasured data in rsmt2d.ExtendedDataSquare.
	MaxSquareSize = appconsts.MaxSquareSize

	// NamespaceSize is a system-wide size for NMT namespaces.
	NamespaceSize = appconsts.NamespaceSize
)

Variables

var NumConcurrentSquares = 8

NumConcurrentSquares limits the amount of squares that are fetched concurrently/simultaneously.

var NumWorkersLimit = MaxSquareSize * MaxSquareSize / 2 * NumConcurrentSquares

NumWorkersLimit sets global limit for workers spawned by GetShares. GetShares could be called MaxSquareSize(128) times per data square each spawning up to 128/2 goroutines and altogether this is 8192. Considering there can be N blocks fetched at the same time, e.g. during catching up data from the past, we multiply this number by the amount of allowed concurrent data square fetches(NumConcurrentSquares).

NOTE: This value only limits amount of simultaneously running workers that are spawned as the load increases and are killed, once the load declines.

TODO(@Wondertan): This assumes we have parallelized DASer implemented. Sync the values once it is shipped. TODO(@Wondertan): Allow configuration of values without global state.

Functions

func BatchSize

func BatchSize(squareSize int) int

BatchSize calculates the amount of nodes that are generated from block of 'squareSizes' to be batched in one write.

func CidFromNamespacedSha256

func CidFromNamespacedSha256(namespacedHash []byte) (cid.Cid, error)

CidFromNamespacedSha256 uses a hash from an nmt tree to create a CID

func GetLeaf

func GetLeaf(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	leaf, total int,
) (ipld.Node, error)

GetLeaf fetches and returns the raw leaf. It walks down the IPLD NMT tree until it finds the requested one.

func GetLeaves

func GetLeaves(ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	maxShares int,
	put func(int, ipld.Node),
)

GetLeaves gets leaves from either local storage, or, if not found, requests them from immediate/connected peers. It puts them into the slice under index of node position in the tree (bin-tree-feat). Does not return any error, and returns/unblocks only on success (got all shares) or on context cancellation.

It works concurrently by spawning workers in the pool which do one basic thing - block until data is fetched, s. t. share processing is never sequential, and thus we request *all* the shares available without waiting for others to finish. It is the required property to maximize data availability. As a side effect, we get concurrent tree traversal reducing time to data time.

GetLeaves relies on the fact that the underlying data structure is a binary tree, so it's not suitable for anything else besides that. Parts on the implementation that rely on this property are explicitly tagged with (bin-tree-feat).

func GetLeavesByNamespace

func GetLeavesByNamespace(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	nID namespace.ID,
	maxShares int,
) ([]ipld.Node, error)

GetLeavesByNamespace returns as many leaves from the given root with the given namespace.ID as it can retrieve. If no shares are found, it returns both data and error as nil. A non-nil error means that only partial data is returned, because at least one share retrieval failed The following implementation is based on `GetShares`.

func GetNode

func GetNode(ctx context.Context, bGetter blockservice.BlockGetter, root cid.Cid) (ipld.Node, error)

func GetProof

func GetProof(
	ctx context.Context,
	bGetter blockservice.BlockGetter,
	root cid.Cid,
	proof []cid.Cid,
	leaf, total int,
) ([]cid.Cid, error)

GetProof fetches and returns the leaf's Merkle Proof. It walks down the IPLD NMT tree until it reaches the leaf and returns collected proof

func MaxSizeBatchOption

func MaxSizeBatchOption(size int) ipld.BatchOption

MaxSizeBatchOption sets the maximum amount of buffered data before writing blocks.

func MustCidFromNamespacedSha256

func MustCidFromNamespacedSha256(hash []byte) cid.Cid

MustCidFromNamespacedSha256 is a wrapper around cidFromNamespacedSha256 that panics in case of an error. Use with care and only in places where no error should occur.

func NamespacedSha256FromCID

func NamespacedSha256FromCID(cid cid.Cid) []byte

NamespacedSha256FromCID derives the Namespaced hash from the given CID.

func NewNMTLeafNode

func NewNMTLeafNode(id cid.Cid, data []byte) ipld.Node

func NewNMTNode

func NewNMTNode(id cid.Cid, l, r []byte) ipld.Node

func RandNamespacedCID

func RandNamespacedCID(t *testing.T) cid.Cid

func Translate

func Translate(dah *da.DataAvailabilityHeader, row, col int) (cid.Cid, int)

Translate transforms square coordinates into IPLD NMT tree path to a leaf node. It also adds randomization to evenly spread fetching from Rows and Columns.

Types

type NmtNodeAdder

type NmtNodeAdder struct {
	// contains filtered or unexported fields
}

NmtNodeAdder adds ipld.Nodes to the underlying ipld.Batch if it is inserted into a nmt tree.

func NewNmtNodeAdder

func NewNmtNodeAdder(ctx context.Context, bs blockservice.BlockService, opts ...ipld.BatchOption) *NmtNodeAdder

NewNmtNodeAdder returns a new NmtNodeAdder with the provided context and batch. Note that the context provided should have a timeout It is not thread-safe.

func (*NmtNodeAdder) Commit

func (n *NmtNodeAdder) Commit() error

Commit checks for errors happened during Visit and if absent commits data to inner Batch.

func (*NmtNodeAdder) Visit

func (n *NmtNodeAdder) Visit(hash []byte, children ...[]byte)

Visit is a NodeVisitor that can be used during the creation of a new NMT to create and add ipld.Nodes to the Batch while computing the root of the NMT.