unixfs

package module
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2024 License: MIT Imports: 9 Imported by: 11

README

go-unixfs

go-unixfs implements unix-like filesystem utilities on top of an ipld merkledag

Table of Contents

Package Directory

This package contains many subpackages, each of which can be very large on its own.

Top Level

The top level unixfs package defines the unixfs format datastructures, and some helper methods around it.

importers

The importer subpackage is what you'll use when you want to turn a normal file into a unixfs file.

io

The io subpackage provides helpers for reading files and manipulating directories. The DagReader takes a reference to a unixfs file and returns a file handle that can be read from and seeked through. The Directory interface allows you to easily read items in a directory, add items to a directory, and do lookups.

mod

The mod subpackage implements a DagModifier type that can be used to write to an existing unixfs file, or create a new one. The logic for this is significantly more complicated than for the dagreader, so its a separate type. (TODO: maybe it still belongs in the io subpackage though?)

hamt

The hamt subpackage implements a CHAMP hamt that is used in unixfs directory sharding.

archive

The archive subpackage implements a tar importer and exporter. The objects created here are not officially unixfs, but in the future, this may be integrated more directly.

test

The test subpackage provides several utilities to make testing unixfs related things easier.

Install

go get github.com/bittorrent/go-unixfs

Contribute

PRs are welcome!

Small note: If editing the Readme, please conform to the standard-readme specification.

License

MIT © Bittorrent

Documentation

Overview

Package unixfs implements a data format for files in the IPFS filesystem It is not the only format in ipfs, but it is the one that the filesystem assumes

Index

Constants

View Source 
const (
	TRaw       = pb.Data_Raw
	TFile      = pb.Data_File
	TDirectory = pb.Data_Directory
	TMetadata  = pb.Data_Metadata
	TSymlink   = pb.Data_Symlink
	THAMTShard = pb.Data_HAMTShard
	TTokenMeta = pb.Data_TokenMeta
)

Shorthands for protobuffer types

Variables

View Source 
var (
	ErrMalformedFileFormat  = errors.New("malformed data in file format")
	ErrUnrecognizedType     = errors.New("unrecognized node type")
	ErrNotMetadataRoot      = errors.New("expected token metadata protobuf dag node")
	ErrUnexpectedLinks      = errors.New("expected more than two links under the given dag node")
	ErrMetadataAccessDenied = errors.New("Token metadata can not be accessed by default. Use --meta option.")
)

Common errors

Functions

func BytesForMetadata 

func BytesForMetadata(m *Metadata) ([]byte, error)

BytesForMetadata wraps the given Metadata as a profobuf message of Data type, setting the DataType to Metadata. The wrapped bytes are itself the result of calling m.Bytes().

func DataSize 

func DataSize(data []byte) (uint64, error)

DataSize returns the size of the contents in protobuf wrapped slice. For raw data it simply provides the length of it. For Data_Files, it will return the associated filesize. Note that Data_Directories will return an error.

func EmptyDirNode 

func EmptyDirNode() *dag.ProtoNode

EmptyDirNode creates an empty folder Protonode.

func EmptyFileNode 

func EmptyFileNode() *dag.ProtoNode

EmptyFileNode creates an empty file Protonode.

func FilePBData 

func FilePBData(data []byte, totalsize uint64) []byte

FilePBData creates a protobuf File with the given byte slice and returns the marshaled protobuf bytes representing it.

func FolderPBData 

func FolderPBData() []byte

FolderPBData returns Bytes that represent a Directory.

func FromBytes 

func FromBytes(data []byte) (*pb.Data, error)

FromBytes unmarshals a byte slice as protobuf Data. Deprecated: Use `FSNodeFromBytes` instead to avoid direct manipulation of `pb.Data`.

func GetFSType 

func GetFSType(n *dag.ProtoNode) (pb.Data_DataType, error)

func GetMetaSubdagRoot 

func GetMetaSubdagRoot(ctx context.Context, n ipld.Node, serv ipld.NodeGetter) (ipld.Node, error)

GetMetaSubdagRoot returns metadata subDag root if the given 'nd' is the dummy root of a DAG with metadata subDag.

func HAMTShardData 

func HAMTShardData(data []byte, fanout uint64, hashType uint64) ([]byte, error)

HAMTShardData return a `Data_HAMTShard` protobuf message

func ModePermsToUnixPerms added in v0.8.0 

func ModePermsToUnixPerms(fileMode os.FileMode) uint32

ModePermsToUnixPerms converts the permission bits of an os.FileMode to unix style mode permissions.

func ReadUnixFSNodeData 

func ReadUnixFSNodeData(node ipld.Node) (data []byte, err error)

ReadUnixFSNodeData extracts the UnixFS data from an IPLD node. Raw nodes are (also) processed because they are used as leaf nodes containing (only) UnixFS data.

func SymlinkData 

func SymlinkData(path string) ([]byte, error)

SymlinkData returns a `Data_Symlink` protobuf message for the path you specify.

func UnixPermsToModePerms added in v0.8.0 

func UnixPermsToModePerms(unixPerms uint32) os.FileMode

UnixPermsToModePerms converts unix style mode permissions to os.FileMode permissions, as it only operates on permission bits it does not set the underlying type (fs.ModeDir, fs.ModeSymlink, etc.) in the returned os.FileMode.

func UnwrapData 

func UnwrapData(data []byte) ([]byte, error)

UnwrapData unmarshals a protobuf messages and returns the contents.

func WrapData 

func WrapData(b []byte) []byte

WrapData marshals raw bytes into a `Data_Raw` type protobuf message.

Types

type DagMetaNodes 

type DagMetaNodes struct {
	DataNode ipld.Node
	MetaNode ipld.Node
}

func GetChildrenForDagWithMeta 

func GetChildrenForDagWithMeta(ctx context.Context, nd ipld.Node, ds ipld.DAGService) (*DagMetaNodes, error)

GetChildrenForDagWithMeta returns a struct containing both the meta root node and the data root node, if the given 'nd' is top of the DAG with token metadata. Return `nil, nil` if 'nd' is no such node and there is no error.

type FSNode 

type FSNode struct {
	// contains filtered or unexported fields
}

An FSNode represents a filesystem object using the UnixFS specification.

The `NewFSNode` constructor should be used instead of just calling `new(FSNode)` to guarantee that the required (`Type` and `Filesize`) fields in the `format` structure are initialized before marshaling (in `GetBytes()`).

func ExtractFSNode 

func ExtractFSNode(node ipld.Node) (*FSNode, error)

ExtractFsNode extracts the `unixfs.FSNode` from the `ipld.Node` (assuming this was implemented by a `mdag.ProtoNode`).

func FSNodeFromBytes 

func FSNodeFromBytes(b []byte) (*FSNode, error)

FSNodeFromBytes unmarshal a protobuf message onto an FSNode.

func NewFSNode 

func NewFSNode(dataType pb.Data_DataType) *FSNode

NewFSNode creates a new FSNode structure with the given `dataType`.

It initializes the (required) `Type` field (that doesn't have a `Set()` accessor so it must be specified at creation), otherwise the `Marshal()` method in `GetBytes()` would fail (`required field "Type" not set`).

It also initializes the `Filesize` pointer field to ensure its value is never nil before marshaling, this is not a required field but it is done to be backwards compatible with previous `go-ipfs` versions hash. (If it wasn't initialized there could be cases where `Filesize` could have been left at nil, when the `FSNode` was created but no data or child nodes were set to adjust it, as is the case in `NewLeaf()`.)

func (*FSNode) AddBlockSize 

func (n *FSNode) AddBlockSize(s uint64)

AddBlockSize adds the size of the next child block of this node

func (*FSNode) BlockSize 

func (n *FSNode) BlockSize(i int) uint64

BlockSize returns the block size indexed by `i`. TODO: Evaluate if this function should be bounds checking.

func (*FSNode) BlockSizes 

func (n *FSNode) BlockSizes() []uint64

BlockSizes gets blocksizes of format

func (*FSNode) Data 

func (n *FSNode) Data() []byte

Data retrieves the `Data` field from the internal `format`.

func (*FSNode) Fanout 

func (n *FSNode) Fanout() uint64

Fanout gets fanout of format

func (*FSNode) FileSize 

func (n *FSNode) FileSize() uint64

FileSize returns the size of the file.

func (*FSNode) GetBytes 

func (n *FSNode) GetBytes() ([]byte, error)

GetBytes marshals this node as a protobuf message.

func (*FSNode) HashType 

func (n *FSNode) HashType() uint64

HashType gets hash type of format

func (*FSNode) IsDir 

func (n *FSNode) IsDir() bool

IsDir checks whether the node represents a directory

func (*FSNode) Metadata 

func (n *FSNode) Metadata() []byte

Data retrieves the `Data` field from the internal `format`.

func (*FSNode) ModTime added in v0.8.0 

func (n *FSNode) ModTime() time.Time

ModTime returns the stored last modified timestamp if available.

func (*FSNode) Mode added in v0.8.0 

func (n *FSNode) Mode() (m os.FileMode)

Mode returns the optionally stored file permissions

func (*FSNode) NumChildren 

func (n *FSNode) NumChildren() int

NumChildren returns the number of child blocks of this node

func (*FSNode) RemoveAllBlockSizes 

func (n *FSNode) RemoveAllBlockSizes()

RemoveAllBlockSizes removes all the child block sizes of this node.

func (*FSNode) RemoveBlockSize 

func (n *FSNode) RemoveBlockSize(i int)

RemoveBlockSize removes the given child block's size.

func (*FSNode) SetData 

func (n *FSNode) SetData(newData []byte)

SetData sets the `Data` field from the internal `format` updating its `Filesize`.

func (*FSNode) SetModTime added in v0.8.0 

func (n *FSNode) SetModTime(ts time.Time)

SetModTime stores the given last modified timestamp, otherwise nullifies stored timestamp.

func (*FSNode) SetMode added in v0.8.0 

func (n *FSNode) SetMode(m os.FileMode)

SetMode stores the given mode permissions, or nullifies stored permissions if none were provided and there are no extended bits set.

func (*FSNode) SetModeFromUnixPermissions added in v0.8.0 

func (n *FSNode) SetModeFromUnixPermissions(unixPerms uint32)

SetModeFromUnixPermissions stores the given unix permissions, or nullifies stored permissions if none were provided and there are no extended bits set.

func (*FSNode) Type 

func (n *FSNode) Type() pb.Data_DataType

Type retrieves the `Type` field from the internal `format`.

func (*FSNode) UpdateFilesize 

func (n *FSNode) UpdateFilesize(filesizeDiff int64)

UpdateFilesize updates the `Filesize` field from the internal `format` by a signed difference (`filesizeDiff`). TODO: Add assert to check for `Filesize` > 0?

type LinkResult 

type LinkResult struct {
	Link *ipld.Link
	Err  error
}

A LinkResult for any parallel enumeration of links TODO: Should this live in go-ipld-format?

type Metadata 

type Metadata struct {
	MimeType string
	Size     uint64
}

Metadata is used to store additional FSNode information.

func MetadataFromBytes 

func MetadataFromBytes(b []byte) (*Metadata, error)

MetadataFromBytes Unmarshals a protobuf Data message into Metadata. The provided slice should have been encoded with BytesForMetadata().

func (*Metadata) Bytes 

func (m *Metadata) Bytes() ([]byte, error)

Bytes marshals Metadata as a protobuf message of Metadata type.

Source Files

View all Source files

Directories

Path Synopsis
Package hamt implements a Hash Array Mapped Trie over ipfs merkledag nodes.
 Package hamt implements a Hash Array Mapped Trie over ipfs merkledag nodes.
Package importer implements utilities used to create IPFS DAGs from files and readers.
 Package importer implements utilities used to create IPFS DAGs from files and readers.
balanced
Package balanced provides methods to build balanced DAGs, which are generalistic DAGs in which all leaves (nodes representing chunks of data) are at the same distance from the root.
 Package balanced provides methods to build balanced DAGs, which are generalistic DAGs in which all leaves (nodes representing chunks of data) are at the same distance from the root.
helpers
trickle
Package trickle allows to build trickle DAGs.
 Package trickle allows to build trickle DAGs.
Package io implements convenience objects for working with the ipfs unixfs data format.
 Package io implements convenience objects for working with the ipfs unixfs data format.
Package mod provides DAG modification utilities to, for example, insert additional nodes in a unixfs DAG or truncate them.
 Package mod provides DAG modification utilities to, for example, insert additional nodes in a unixfs DAG or truncate them.
private
linksize

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.