helpers

package
v0.4.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 10, 2018 License: MIT Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var BlockSizeLimit = 1048576 // 1 MB

BlockSizeLimit specifies the maximum size an imported block can have.

View Source 
var DefaultLinksPerBlock = roughLinkBlockSize / roughLinkSize

DefaultLinksPerBlock governs how the importer decides how many links there will be per block. This calculation is based on expected distributions of:

  • the expected distribution of block sizes
  • the expected distribution of link sizes
  • desired access speed

For now, we use: 

var roughLinkBlockSize = 1 << 13 // 8KB
var roughLinkSize = 288          // sha256 + framing + name
var DefaultLinksPerBlock = (roughLinkBlockSize / roughLinkSize)

See calc_test.go

View Source 
var ErrSizeLimitExceeded = fmt.Errorf("object size limit exceeded")

ErrSizeLimitExceeded signals that a block is larger than BlockSizeLimit.

Functions

This section is empty.

Types

type DagBuilderHelper 

type DagBuilderHelper struct {
	// contains filtered or unexported fields
}

DagBuilderHelper wraps together a bunch of objects needed to efficiently create unixfs dag trees

func (*DagBuilderHelper) Add 

func (db *DagBuilderHelper) Add(node *UnixfsNode) (ipld.Node, error)

Add sends a node to the DAGService, and returns it.

func (*DagBuilderHelper) Close added in v0.3.6 

func (db *DagBuilderHelper) Close() error

Close has the DAGService perform a batch Commit operation. It should be called at the end of the building process to make sure all data is persisted.

func (*DagBuilderHelper) Done 

func (db *DagBuilderHelper) Done() bool

Done returns whether or not we're done consuming the incoming data.

func (*DagBuilderHelper) FillNodeLayer 

func (db *DagBuilderHelper) FillNodeLayer(node *UnixfsNode) error

FillNodeLayer will add datanodes as children to the give node until at most db.indirSize nodes are added.

func (*DagBuilderHelper) GetDagServ added in v0.3.2 

func (db *DagBuilderHelper) GetDagServ() ipld.DAGService

GetDagServ returns the dagservice object this Helper is using

func (*DagBuilderHelper) GetNextDataNode added in v0.4.5 

func (db *DagBuilderHelper) GetNextDataNode() (*UnixfsNode, error)

GetNextDataNode builds a UnixFsNode with the data obtained from the Splitter, given the constraints (BlockSizeLimit, RawLeaves) specified when creating the DagBuilderHelper. 

func (db *DagBuilderHelper) Maxlinks() int

Maxlinks returns the configured maximum number for links for nodes built with this helper.

func (*DagBuilderHelper) NewUnixfsNode added in v0.4.9 

func (db *DagBuilderHelper) NewUnixfsNode() *UnixfsNode

NewUnixfsNode creates a new Unixfs node to represent a file.

func (*DagBuilderHelper) Next 

func (db *DagBuilderHelper) Next() ([]byte, error)

Next returns the next chunk of data to be inserted into the dag if it returns nil, that signifies that the stream is at an end, and that the current building operation should finish.

func (*DagBuilderHelper) SetPosInfo added in v0.4.5 

func (db *DagBuilderHelper) SetPosInfo(node *UnixfsNode, offset uint64)

SetPosInfo sets the offset information of a node using the fullpath and stat from the DagBuilderHelper.

type DagBuilderParams 

type DagBuilderParams struct {
	// Maximum number of links per intermediate node
	Maxlinks int

	// RawLeaves signifies that the importer should use raw ipld nodes as leaves
	// instead of using the unixfs TRaw type
	RawLeaves bool

	// CID Prefix to use if set
	Prefix *cid.Prefix

	// DAGService to write blocks to (required)
	Dagserv ipld.DAGService

	// NoCopy signals to the chunker that it should track fileinfo for
	// filestore adds
	NoCopy bool
}

DagBuilderParams wraps configuration options to create a DagBuilderHelper from a chunker.Splitter.

func (*DagBuilderParams) New 

func (dbp *DagBuilderParams) New(spl chunker.Splitter) *DagBuilderHelper

New generates a new DagBuilderHelper from the given params and a given chunker.Splitter as data source.

type UnixfsNode 

type UnixfsNode struct {
	// contains filtered or unexported fields
}

UnixfsNode is a struct created to aid in the generation of unixfs DAG trees

func NewUnixfsNodeFromDag added in v0.3.2 

func NewUnixfsNodeFromDag(nd *dag.ProtoNode) (*UnixfsNode, error)

NewUnixfsNodeFromDag reconstructs a Unixfs node from a given dag node

func (*UnixfsNode) AddChild 

func (n *UnixfsNode) AddChild(child *UnixfsNode, db *DagBuilderHelper) error

AddChild adds the given UnixfsNode as a child of the receiver. The passed in DagBuilderHelper is used to store the child node an pin it locally so it doesnt get lost.

func (*UnixfsNode) FileSize added in v0.4.5 

func (n *UnixfsNode) FileSize() uint64

FileSize returns the total file size of this tree (including children) In the case of raw nodes, it returns the length of the raw data.

func (*UnixfsNode) GetChild added in v0.3.2 

func (n *UnixfsNode) GetChild(ctx context.Context, i int, ds ipld.DAGService) (*UnixfsNode, error)

GetChild gets the ith child of this node from the given DAGService.

func (*UnixfsNode) GetDagNode 

func (n *UnixfsNode) GetDagNode() (ipld.Node, error)

GetDagNode fills out the proper formatting for the unixfs node inside of a DAG node and returns the dag node.

func (*UnixfsNode) NumChildren 

func (n *UnixfsNode) NumChildren() int

NumChildren returns the number of children referenced by this UnixfsNode.

func (*UnixfsNode) RemoveChild added in v0.3.2 

func (n *UnixfsNode) RemoveChild(index int, dbh *DagBuilderHelper)

RemoveChild deletes the child node at the given index.

func (*UnixfsNode) Set added in v0.4.5 

func (n *UnixfsNode) Set(other *UnixfsNode)

Set replaces the current UnixfsNode with another one. It performs a shallow copy.

func (*UnixfsNode) SetData added in v0.3.2 

func (n *UnixfsNode) SetData(data []byte)

SetData stores data in this node.

func (*UnixfsNode) SetPosInfo added in v0.4.5 

func (n *UnixfsNode) SetPosInfo(offset uint64, fullPath string, stat os.FileInfo)

SetPosInfo sets information about the offset of the data of this node in a filesystem file.

func (*UnixfsNode) SetPrefix added in v0.4.9 

func (n *UnixfsNode) SetPrefix(prefix *cid.Prefix)

SetPrefix sets the CID Prefix

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.