sharding

Documentation

Overview

Package sharding implements a sharding ClusterDAGService places content in different shards while it's being added, creating a final Cluster DAG and pinning it.

Index

• Constants
• func CborDataToNode(raw []byte, format string) (ipld.Node, error)
• func VerifyShards(t *testing.T, rootCid cid.Cid, pins MockPinStore, ipfs MockBlockStore, ...) (map[string]struct{}, error)
• type DAGService
  • func New(rpc *rpc.Client, opts api.PinOptions, out chan<- *api.AddedOutput) *DAGService
  • func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error
  • func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error
  • func (dgs *DAGService) Finalize(ctx context.Context, dataRoot cid.Cid) (cid.Cid, error)
• type MockBlockStore
• type MockPinStore

Constants

const MaxLinks = 5984

MaxLinks is the max number of links that, when serialized fit into a block

Functions

func CborDataToNode

func CborDataToNode(raw []byte, format string) (ipld.Node, error)

CborDataToNode parses cbor data into a clusterDAG node while making a few checks

func VerifyShards

func VerifyShards(t *testing.T, rootCid cid.Cid, pins MockPinStore, ipfs MockBlockStore, expectedShards int) (map[string]struct{}, error)

VerifyShards checks that a sharded CID has been correctly formed and stored. This is a helper function for testing. It returns a map with all the blocks from all shards.

Types

type DAGService

type DAGService struct {
	adder.BaseDAGService
	// contains filtered or unexported fields
}

DAGService is an implementation of a ClusterDAGService which shards content while adding among several IPFS Cluster peers, creating a Cluster DAG to track and pin that content selectively in the IPFS daemons allocated to it.

func New

func New(rpc *rpc.Client, opts api.PinOptions, out chan<- *api.AddedOutput) *DAGService

New returns a new ClusterDAGService, which uses the given rpc client to perform Allocate, IPFSBlockPut and Pin requests to other cluster components.

func (*DAGService) Add

func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error

Add puts the given node in its corresponding shard and sends it to the destination peers.

func (*DAGService) AddMany

func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error

AddMany calls Add for every given node.

func (*DAGService) Finalize

func (dgs *DAGService) Finalize(ctx context.Context, dataRoot cid.Cid) (cid.Cid, error)

Finalize finishes sharding, creates the cluster DAG and pins it along with the meta pin for the root node of the content.

type MockBlockStore

type MockBlockStore interface {
	// Gets a block
	BlockGet(context.Context, cid.Cid) ([]byte, error)
}

MockBlockStore is used in VerifyShards

type MockPinStore

type MockPinStore interface {
	// Gets a pin
	PinGet(context.Context, cid.Cid) (*api.Pin, error)
}

MockPinStore is used in VerifyShards