Documentation ¶
Index ¶
- Variables
- type DagBuilderHelper
- func (db *DagBuilderHelper) Add(node ipld.Node) error
- func (db *DagBuilderHelper) Done() bool
- func (db *DagBuilderHelper) FillNodeLayer(node *FSNodeOverDag) error
- func (db *DagBuilderHelper) GetCidBuilder() cid.Builder
- func (db *DagBuilderHelper) GetDagServ() ipld.DAGService
- func (db *DagBuilderHelper) Maxlinks() int
- func (db *DagBuilderHelper) NewFSNFromDag(nd *dag.ProtoNode) (*FSNodeOverDag, error)
- func (db *DagBuilderHelper) NewFSNodeOverDag(fsNodeType pb.Data_DataType) *FSNodeOverDag
- func (db *DagBuilderHelper) NewLeafDataNode(fsNodeType pb.Data_DataType) (node ipld.Node, dataSize uint64, err error)
- func (db *DagBuilderHelper) NewLeafNode(data []byte, fsNodeType pb.Data_DataType) (ipld.Node, error)
- func (db *DagBuilderHelper) Next() ([]byte, error)
- func (db *DagBuilderHelper) ProcessFileStore(node ipld.Node, dataSize uint64) ipld.Node
- type DagBuilderParams
- type FSNodeOverDag
- func (n *FSNodeOverDag) AddChild(child ipld.Node, fileSize uint64, db *DagBuilderHelper) error
- func (n *FSNodeOverDag) Commit() (ipld.Node, error)
- func (n *FSNodeOverDag) FileSize() uint64
- func (n *FSNodeOverDag) GetChild(ctx context.Context, i int, ds ipld.DAGService) (*FSNodeOverDag, error)
- func (n *FSNodeOverDag) GetDagNode() (ipld.Node, error)
- func (n *FSNodeOverDag) NumChildren() int
- func (n *FSNodeOverDag) RemoveChild(index int, dbh *DagBuilderHelper)
- func (n *FSNodeOverDag) SetFileData(fileData []byte)
Constants ¶
This section is empty.
Variables ¶
var BlockSizeLimit = 1048576 // 1 MB
BlockSizeLimit specifies the maximum size an imported block can have.
var DefaultLinksPerBlock = roughLinkBlockSize / roughLinkSize
DefaultLinksPerBlock governs how the importer decides how many links there will be per block. This calculation is based on expected distributions of:
- the expected distribution of block sizes
- the expected distribution of link sizes
- desired access speed
For now, we use:
var roughLinkBlockSize = 1 << 13 // 8KB var roughLinkSize = 34 + 8 + 5 // sha256 multihash + size + no name // + protobuf framing var DefaultLinksPerBlock = (roughLinkBlockSize / roughLinkSize) = ( 8192 / 47 ) = (approximately) 174
var ErrMissingFsRef = errors.New("missing file path or URL, can't create filestore reference")
var ErrSizeLimitExceeded = errors.New("object size limit exceeded")
ErrSizeLimitExceeded signals that a block is larger than BlockSizeLimit.
Functions ¶
This section is empty.
Types ¶
type DagBuilderHelper ¶
type DagBuilderHelper struct {
// contains filtered or unexported fields
}
DagBuilderHelper wraps together a bunch of objects needed to efficiently create unixfs dag trees
func (*DagBuilderHelper) Add ¶
func (db *DagBuilderHelper) Add(node ipld.Node) error
Add inserts the given node in the DAGService.
func (*DagBuilderHelper) Done ¶
func (db *DagBuilderHelper) Done() bool
Done returns whether or not we're done consuming the incoming data.
func (*DagBuilderHelper) FillNodeLayer ¶
func (db *DagBuilderHelper) FillNodeLayer(node *FSNodeOverDag) error
FillNodeLayer will add datanodes as children to the give node until it is full in this layer or no more data. NOTE: This function creates raw data nodes so it only works for the `trickle.Layout`.
func (*DagBuilderHelper) GetCidBuilder ¶
func (db *DagBuilderHelper) GetCidBuilder() cid.Builder
GetCidBuilder returns the internal `cid.CidBuilder` set in the builder.
func (*DagBuilderHelper) GetDagServ ¶
func (db *DagBuilderHelper) GetDagServ() ipld.DAGService
GetDagServ returns the dagservice object this Helper is using
func (*DagBuilderHelper) Maxlinks ¶
func (db *DagBuilderHelper) Maxlinks() int
Maxlinks returns the configured maximum number for links for nodes built with this helper.
func (*DagBuilderHelper) NewFSNFromDag ¶
func (db *DagBuilderHelper) NewFSNFromDag(nd *dag.ProtoNode) (*FSNodeOverDag, error)
NewFSNFromDag reconstructs a FSNodeOverDag node from a given dag node
func (*DagBuilderHelper) NewFSNodeOverDag ¶
func (db *DagBuilderHelper) NewFSNodeOverDag(fsNodeType pb.Data_DataType) *FSNodeOverDag
NewFSNodeOverDag creates a new `dag.ProtoNode` and `ft.FSNode` decoupled from one onther (and will continue in that way until `Commit` is called), with `fsNodeType` specifying the type of the UnixFS layer node (either `File` or `Raw`).
func (*DagBuilderHelper) NewLeafDataNode ¶
func (db *DagBuilderHelper) NewLeafDataNode(fsNodeType pb.Data_DataType) (node ipld.Node, dataSize uint64, err error)
NewLeafDataNode builds the `node` with the data obtained from the Splitter with the given constraints (BlockSizeLimit, RawLeaves) specified when creating the DagBuilderHelper. It returns `ipld.Node` with the `dataSize` (that will be used to keep track of the DAG file size). The size of the data is computed here because after that it will be hidden by `NewLeafNode` inside a generic `ipld.Node` representation.
func (*DagBuilderHelper) NewLeafNode ¶
func (db *DagBuilderHelper) NewLeafNode(data []byte, fsNodeType pb.Data_DataType) (ipld.Node, error)
NewLeafNode creates a leaf node filled with data. If rawLeaves is defined then a raw leaf will be returned. Otherwise, it will create and return `FSNodeOverDag` with `fsNodeType`.
func (*DagBuilderHelper) Next ¶
func (db *DagBuilderHelper) Next() ([]byte, error)
Next returns the next chunk of data to be inserted into the dag if it returns nil, that signifies that the stream is at an end, and that the current building operation should finish.
func (*DagBuilderHelper) ProcessFileStore ¶
ProcessFileStore generates, if Filestore is being used, the `FilestoreNode` representation of the `ipld.Node` that contains the file data. If Filestore is not being used just return the same node to continue with its addition to the DAG.
The `db.offset` is updated at this point (instead of when `NewLeafDataNode` is called, both work in tandem but the offset is more related to this function).
type DagBuilderParams ¶
type DagBuilderParams struct { // Maximum number of links per intermediate node Maxlinks int // RawLeaves signifies that the importer should use raw ipld nodes as leaves // instead of using the unixfs TRaw type RawLeaves bool // CID Builder to use if set CidBuilder cid.Builder // DAGService to write blocks to (required) Dagserv ipld.DAGService // NoCopy signals to the chunker that it should track fileinfo for // filestore adds NoCopy bool }
DagBuilderParams wraps configuration options to create a DagBuilderHelper from a chunker.Splitter.
func (*DagBuilderParams) New ¶
func (dbp *DagBuilderParams) New(spl chunker.Splitter) (*DagBuilderHelper, error)
New generates a new DagBuilderHelper from the given params and a given chunker.Splitter as data source.
type FSNodeOverDag ¶
type FSNodeOverDag struct {
// contains filtered or unexported fields
}
FSNodeOverDag encapsulates an `unixfs.FSNode` that will be stored in a `dag.ProtoNode`. Instead of just having a single `ipld.Node` that would need to be constantly (un)packed to access and modify its internal `FSNode` in the process of creating a UnixFS DAG, this structure stores an `FSNode` cache to manipulate it (add child nodes) directly , and only when the node has reached its final (immutable) state (signaled by calling `Commit()`) is it committed to a single (indivisible) `ipld.Node`.
It is used mainly for internal (non-leaf) nodes, and for some representations of data leaf nodes (that don't use raw nodes or Filestore).
It aims to replace the `UnixfsNode` structure which encapsulated too many possible node state combinations.
TODO: Revisit the name.
func NewFSNFromDag ¶
func NewFSNFromDag(nd *dag.ProtoNode) (*FSNodeOverDag, error)
NewFSNFromDag reconstructs a FSNodeOverDag node from a given dag node
func (*FSNodeOverDag) AddChild ¶
func (n *FSNodeOverDag) AddChild(child ipld.Node, fileSize uint64, db *DagBuilderHelper) error
AddChild adds a `child` `ipld.Node` to both node layers. The `dag.ProtoNode` creates a link to the child node while the `ft.FSNode` stores its file size (that is, not the size of the node but the size of the file data that it is storing at the UnixFS layer). The child is also stored in the `DAGService`.
func (*FSNodeOverDag) Commit ¶
func (n *FSNodeOverDag) Commit() (ipld.Node, error)
Commit unifies (resolves) the cache nodes into a single `ipld.Node` that represents them: the `ft.FSNode` is encoded inside the `dag.ProtoNode`.
TODO: Make it read-only after committing, allow to commit only once.
func (*FSNodeOverDag) FileSize ¶
func (n *FSNodeOverDag) FileSize() uint64
FileSize returns the `Filesize` attribute from the underlying representation of the `ft.FSNode`.
func (*FSNodeOverDag) GetChild ¶
func (n *FSNodeOverDag) GetChild(ctx context.Context, i int, ds ipld.DAGService) (*FSNodeOverDag, error)
GetChild gets the ith child of this node from the given DAGService.
func (*FSNodeOverDag) GetDagNode ¶
func (n *FSNodeOverDag) GetDagNode() (ipld.Node, error)
GetDagNode fills out the proper formatting for the FSNodeOverDag node inside of a DAG node and returns the dag node. TODO: Check if we have committed (passed the UnixFS information to the DAG layer) before returning this.
func (*FSNodeOverDag) NumChildren ¶
func (n *FSNodeOverDag) NumChildren() int
NumChildren returns the number of children of the `ft.FSNode`.
func (*FSNodeOverDag) RemoveChild ¶
func (n *FSNodeOverDag) RemoveChild(index int, dbh *DagBuilderHelper)
RemoveChild deletes the child node at the given index.
func (*FSNodeOverDag) SetFileData ¶
func (n *FSNodeOverDag) SetFileData(fileData []byte)
SetFileData stores the `fileData` in the `ft.FSNode`. It should be used only when `FSNodeOverDag` represents a leaf node (internal nodes don't carry data, just file sizes).