sharding

Documentation

Overview

Package sharding implements a sharding ClusterDAGService places content in different shards while it's being added, creating a final Cluster DAG and pinning it.

Index

• Constants
• func CborDataToNode(raw []byte, format string) (ipld.Node, error)
• func VerifyShards(t *testing.T, rootCid api.Cid, pins MockPinStore, ipfs MockBlockStore, ...) (map[string]struct{}, error)
• type DAGService
  • func New(ctx context.Context, rpc *rpc.Client, opts api.AddParams, ...) *DAGService
  • func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error
  • func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error
  • func (dgs *DAGService) Allocations() []peer.ID
  • func (dgs *DAGService) Close() error
  • func (dgs *DAGService) Finalize(ctx context.Context, dataRoot api.Cid) (api.Cid, error)
• type MockBlockStore
• type MockPinStore

Constants

const MaxLinks = 5984

MaxLinks is the max number of links that, when serialized fit into a block

Functions

func CborDataToNode

func CborDataToNode(raw []byte, format string) (ipld.Node, error)

CborDataToNode parses cbor data into a clusterDAG node while making a few checks

func VerifyShards

func VerifyShards(t *testing.T, rootCid api.Cid, pins MockPinStore, ipfs MockBlockStore, expectedShards int) (map[string]struct{}, error)

VerifyShards checks that a sharded CID has been correctly formed and stored. This is a helper function for testing. It returns a map with all the blocks from all shards.

Types

type DAGService

type DAGService struct {
	adder.BaseDAGService
	// contains filtered or unexported fields
}

DAGService is an implementation of a ClusterDAGService which shards content while adding among several IPFS Cluster peers, creating a Cluster DAG to track and pin that content selectively in the IPFS daemons allocated to it.

func New

func New(ctx context.Context, rpc *rpc.Client, opts api.AddParams, out chan<- api.AddedOutput) *DAGService

New returns a new ClusterDAGService, which uses the given rpc client to perform Allocate, IPFSStream and Pin requests to other cluster components.

func (*DAGService) Add

func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error

Add puts the given node in its corresponding shard and sends it to the destination peers.

func (*DAGService) AddMany

func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error

AddMany calls Add for every given node.

func (*DAGService) Allocations

func (dgs *DAGService) Allocations() []peer.ID

Allocations returns the current allocations for the current shard.

func (*DAGService) Close

func (dgs *DAGService) Close() error

Close performs cleanup and should be called when the DAGService is not going to be used anymore.

func (*DAGService) Finalize

func (dgs *DAGService) Finalize(ctx context.Context, dataRoot api.Cid) (api.Cid, error)

Finalize finishes sharding, creates the cluster DAG and pins it along with the meta pin for the root node of the content.

type MockBlockStore

type MockBlockStore interface {
	// Gets a block
	BlockGet(context.Context, api.Cid) ([]byte, error)
}

MockBlockStore is used in VerifyShards

type MockPinStore

type MockPinStore interface {
	// Gets a pin
	PinGet(context.Context, api.Cid) (api.Pin, error)
}

MockPinStore is used in VerifyShards