unixfsnode

package module
v1.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2023 License: Apache-2.0, MIT Imports: 18 Imported by: 53

README

go-unixfsnode

This is an IPLD ADL that provides string based pathing for protobuf nodes. The top level node behaves like a map where LookupByString returns the Hash property on the Link in the protobufs list of Links whos Name property matches the key. This should enable selector traversals that work based of paths.

Note that while it works internally with go-codec-dagpb, the Reify method (used to get a UnixFSNode from a DagPB node should actually work successfully with go-ipld-prime-proto nodes)

Usage

The primary interaction with this package is to register an ADL on a link system. This is done with via a helper method.

AddUnixFSReificationToLinkSystem(lsys *ipld.LinkSystem)

For link systems which have UnixFS reification registered, two ADLs will be available to the InterpretAs selector: 'unixfs' and 'unixfs-preload'. The different between these two ADLs is that the preload variant will access all blocks within a UnixFS Object (file or directory) when that object is accessed by a selector traversal. The non-preload variant in contrast will only access the subset of blocks strictly needed for the traversal. In practice, this means the subset of a sharded directory needed to access a specific file, or the sub-range of a file directly accessed by a range selector.

License

Apache-2.0/MIT © Protocol Labs

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ExploreAllRecursivelySelector = specBuilder(func(ssb builder.SelectorSpecBuilder) builder.SelectorSpec {
	return ssb.ExploreRecursive(
		selector.RecursionLimitNone(),
		ssb.ExploreAll(ssb.ExploreRecursiveEdge()),
	)
})

ExploreAllRecursivelySelector is a selector that will explore all nodes. It is the same selector as selectorparse.CommonSelector_ExploreAllRecursively but it is precompiled for use with UnixFSPathSelectorBuilder().

View Source 
var MatchUnixFSEntitySelector = specBuilder(func(ssb builder.SelectorSpecBuilder) builder.SelectorSpec {
	return ssb.ExploreInterpretAs("unixfs", ssb.ExploreUnion(ssb.Matcher(),
		ssb.ExploreRecursive(
			selector.RecursionLimitDepth(1),
			ssb.ExploreAll(ssb.ExploreRecursiveEdge()),
		),
	))
})

MatchUnixFSEntitySelector is a selector that will match a single node and its direct children.

For UnixFS files, this will match the file and its blocks.

For UnixFS directories, and will iterate through the list of child links but will not iterate _into_ the child directories.

View Source 
var MatchUnixFSPreloadSelector = specBuilder(func(ssb builder.SelectorSpecBuilder) builder.SelectorSpec {
	return ssb.ExploreInterpretAs("unixfs-preload", ssb.Matcher())
})

MatchUnixFSPreloadSelector is a selector that will match a single node, similar to selectorparse.CommonSelector_MatchPoint, but uses the "unixfs-preload" ADL to load sharded files and directories as a single node. Can be used to shallow load an entire UnixFS directory listing, sharded or not, but not its contents. MatchUnixfsPreloadSelector is precompiled for use with UnixFSPathSelectorBuilder().

NOTE: This selector may be deprecated in a future release. Users should instead use MatchUnixFSEntitySelector instead, which is intended to have the same effect but doesn't use the "unixfs-preload" ADL.

View Source 
var MatchUnixFSSelector = specBuilder(func(ssb builder.SelectorSpecBuilder) builder.SelectorSpec {
	return ssb.ExploreInterpretAs("unixfs", ssb.Matcher())
})

MatchUnixFSSelector is a selector that will match a single node, similar to selectorparse.CommonSelector_MatchPoint, but uses the "unixfs" ADL to load as UnixFS data. Unlike MatchUnixFSPreloadSelector, this selector will not preload all blocks in sharded directories or files. Use MatchUnixFSPreloadSelector where the blocks that constitute the full UnixFS resource being selected are important to load.

Functions

func AddUnixFSReificationToLinkSystem 

func AddUnixFSReificationToLinkSystem(lsys *ipld.LinkSystem)

AddUnixFSReificationToLinkSystem will add both unixfs and unixfs-preload reifiers to a LinkSystem. This is primarily useful for traversals that use an interpretAs clause, such as Match* selectors in this package.

func BytesConsumingMatcher added in v1.7.2 

func BytesConsumingMatcher(p traversal.Progress, n datamodel.Node) error

BytesConsumingMatcher is a traversal.WalkMatching matcher function that consumes the bytes of a LargeBytesNode where one is matched. Use this in conjunction with the Match* selectors in this package to ensure that all blocks of sharded files are loaded during a traversal, or that the subset of blocks required to fulful a range selector are loaded.

func Reify 

func Reify(lnkCtx ipld.LinkContext, maybePBNodeRoot ipld.Node, lsys *ipld.LinkSystem) (ipld.Node, error)

Reify looks at an ipld Node and tries to interpret it as a UnixFSNode if successful, it returns the UnixFSNode

func UnixFSPathSelector 

func UnixFSPathSelector(path string) datamodel.Node

UnixFSPathSelector creates a selector for IPLD path to a UnixFS resource if UnixFS reification is setup on a LinkSystem being used for traversal.

Use UnixFSPathSelectorBuilder for more control over the selector, this function is the same as calling 

UnixFSPathSelectorBuilder(path, MatchUnixFSSelector, false)

func UnixFSPathSelectorBuilder added in v1.6.0 

func UnixFSPathSelectorBuilder(path string, targetSelector builder.SelectorSpec, matchPath bool) ipld.Node

UnixFSPathSelectorBuilder creates a selector for IPLD path to a UnixFS resource if UnixFS reification is setup on a LinkSystem being used for traversal.

The path is interpreted according to github.com/ipld/go-ipld-prime/datamodel/Path rules, i.e.

  • leading and trailing slashes are ignored
  • redundant slashes are ignored
  • the segment `..` is a field named `..`, same with `.`

targetSelector is the selector to apply to the final node in the path. Use ExploreAllRecursivelySelector to explore (i.e. load the blocks) all of the content from the terminus of the path. Use MatchUnixFSPreloadSelector to match the terminus of the path, but preload all blocks in sharded files and directories. Use MatchUnixFSSelector to match the terminus of the path, but not preload any blocks if the terminus is sharded. Or any other custom SelectorSpec can be supplied.

If matchPath is false, the selector will explore, not match, so it's useful for traversals where block loads are important, not where the matcher visitor callback is important. if matchPath is true, the selector will match the nodes along the path while exploring them.

Types

type PathedPBNode 

type PathedPBNode = *_PathedPBNode

func (PathedPBNode) AsBool 

func (n PathedPBNode) AsBool() (bool, error)

func (PathedPBNode) AsBytes 

func (n PathedPBNode) AsBytes() ([]byte, error)

func (PathedPBNode) AsFloat 

func (n PathedPBNode) AsFloat() (float64, error)

func (PathedPBNode) AsInt 

func (n PathedPBNode) AsInt() (int64, error)
func (n PathedPBNode) AsLink() (ipld.Link, error)

func (PathedPBNode) AsString 

func (n PathedPBNode) AsString() (string, error)

func (PathedPBNode) FieldData 

func (n PathedPBNode) FieldData() dagpb.MaybeBytes
func (n PathedPBNode) FieldLinks() dagpb.PBLinks

func (PathedPBNode) IsAbsent 

func (n PathedPBNode) IsAbsent() bool

func (PathedPBNode) IsNull 

func (n PathedPBNode) IsNull() bool

func (PathedPBNode) Iterator 

func (n PathedPBNode) Iterator() *iter.UnixFSDir__Itr

func (PathedPBNode) Kind 

func (n PathedPBNode) Kind() ipld.Kind

func (PathedPBNode) Length 

func (n PathedPBNode) Length() int64

Length returns the length of a list, or the number of entries in a map, or -1 if the node is not of list nor map kind.

func (PathedPBNode) ListIterator 

func (n PathedPBNode) ListIterator() ipld.ListIterator

ListIterator returns an iterator which yields key-value pairs traversing the node. If the node kind is anything other than a list, nil will be returned.

The iterator will yield every entry in the list; that is, it can be expected that itr.Next will be called node.Length times before itr.Done becomes true.

func (PathedPBNode) Lookup 

func (n PathedPBNode) Lookup(key dagpb.String) dagpb.Link

func (PathedPBNode) LookupByIndex 

func (n PathedPBNode) LookupByIndex(idx int64) (ipld.Node, error)

func (PathedPBNode) LookupByNode 

func (n PathedPBNode) LookupByNode(key ipld.Node) (ipld.Node, error)

func (PathedPBNode) LookupBySegment 

func (n PathedPBNode) LookupBySegment(seg ipld.PathSegment) (ipld.Node, error)

func (PathedPBNode) LookupByString 

func (n PathedPBNode) LookupByString(key string) (ipld.Node, error)

LookupByString looks for the key in the list of links with a matching name

func (PathedPBNode) MapIterator 

func (n PathedPBNode) MapIterator() ipld.MapIterator

func (PathedPBNode) Prototype 

func (n PathedPBNode) Prototype() ipld.NodePrototype

func (PathedPBNode) Representation 

func (n PathedPBNode) Representation() ipld.Node

func (PathedPBNode) Substrate 

func (n PathedPBNode) Substrate() ipld.Node

Substrate returns the underlying PBNode -- note: only the substrate will encode successfully to protobuf if writing

func (PathedPBNode) Type 

func (PathedPBNode) Type() schema.Type

satisfy schema.TypedNode

Source Files

View all Source files

Directories

Path Synopsis
Package data provides tools for working with the UnixFS data structure that is encoded in the "Data" field of the larger a DagPB encoded IPLD node.
 Package data provides tools for working with the UnixFS data structure that is encoded in the "Data" field of the larger a DagPB encoded IPLD node.
builder
builder/quick
Package quickbuilder is designed as a replacement for the existing ipfs-files constructor for a simple way to generate synthetic directory trees.
 Package quickbuilder is designed as a replacement for the existing ipfs-files constructor for a simple way to generate synthetic directory trees.
gen
Package test provides ADL testing of the ipld specification around * traversal making use of match subsets * largeByteNode readers
 Package test provides ADL testing of the ipld specification around * traversal making use of match subsets * largeByteNode readers
Package testutil provides utilities for writing tests that require nontrivial UnixFS data of various forms
 Package testutil provides utilities for writing tests that require nontrivial UnixFS data of various forms
namegen

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.