sharding

Documentation

Overview

Package sharding implements a sharding ClusterDAGService places content in different shards while it's being added, creating a final Cluster DAG and pinning it.

Index

• Constants
• func CborDataToNode(raw []byte, format string) (ipld.Node, error)
• func VerifyShards(t *testing.T, rootCid *cid.Cid, pins MockPinStore, ipfs MockBlockStore, ...) (map[string]struct{}, error)
• type DAGService
  • func New(rpc *rpc.Client, opts api.PinOptions, out chan<- *api.AddedOutput) *DAGService
  • func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error
  • func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error
  • func (dgs *DAGService) Finalize(ctx context.Context, dataRoot *cid.Cid) (*cid.Cid, error)
• type MockBlockStore
• type MockPinStore

Constants

const MaxLinks = 5984

MaxLinks is the max number of links that, when serialized fit into a block

Functions

func CborDataToNode

func CborDataToNode(raw []byte, format string) (ipld.Node, error)

CborDataToNode parses cbor data into a clusterDAG node while making a few checks

func VerifyShards

func VerifyShards(t *testing.T, rootCid *cid.Cid, pins MockPinStore, ipfs MockBlockStore, expectedShards int) (map[string]struct{}, error)

VerifyShards checks that a sharded CID has been correctly formed and stored. This is a helper function for testing. It returns a map with all the blocks from all shards.

Types

type DAGService

type DAGService struct {
	adder.BaseDAGService
	// contains filtered or unexported fields
}

DAGService is an implementation of a ClusterDAGService which shards content while adding among several IPFS Cluster peers, creating a Cluster DAG to track and pin that content selectively in the IPFS daemons allocated to it.

func New

func New(rpc *rpc.Client, opts api.PinOptions, out chan<- *api.AddedOutput) *DAGService

New returns a new ClusterDAGService, which uses the given rpc client to perform Allocate, IPFSBlockPut and Pin requests to other cluster components.

func (*DAGService) Add

func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error

Add puts the given node in its corresponding shard and sends it to the destination peers.

func (*DAGService) AddMany

func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error

AddMany calls Add for every given node.

func (*DAGService) Finalize

func (dgs *DAGService) Finalize(ctx context.Context, dataRoot *cid.Cid) (*cid.Cid, error)

Finalize finishes sharding, creates the cluster DAG and pins it along with the meta pin for the root node of the content.

type MockBlockStore

type MockBlockStore interface {
	// Gets a block
	BlockGet(*cid.Cid) ([]byte, error)
}

MockBlockStore is used in VerifyShards

type MockPinStore

type MockPinStore interface {
	// Gets a pin
	PinGet(*cid.Cid) (api.Pin, error)
}

MockPinStore is used in VerifyShards