unixfs

README

go-unixfs

go-unixfs implements unix-like filesystem utilities on top of an ipld merkledag

Lead Maintainer

Steven Allen

Table of Contents

• Directory
• Install
• Contribute
• License

Package Directory

This package contains many subpackages, each of which can be very large on its own.

Top Level

The top level unixfs package defines the unixfs format datastructures, and some helper methods around it.

importers

The importer subpackage is what you'll use when you want to turn a normal file into a unixfs file.

io

The io subpackage provides helpers for reading files and manipulating directories. The DagReader takes a reference to a unixfs file and returns a file handle that can be read from and seeked through. The Directory interface allows you to easily read items in a directory, add items to a directory, and do lookups.

mod

The mod subpackage implements a DagModifier type that can be used to write to an existing unixfs file, or create a new one. The logic for this is significantly more complicated than for the dagreader, so its a separate type. (TODO: maybe it still belongs in the io subpackage though?)

hamt

The hamt subpackage implements a CHAMP hamt that is used in unixfs directory sharding.

archive

The archive subpackage implements a tar importer and exporter. The objects created here are not officially unixfs, but in the future, this may be integrated more directly.

test

The test subpackage provides several utilities to make testing unixfs related things easier.

Install

go get github.com/ipfs/go-unixfs

Contribute

PRs are welcome!

Small note: If editing the Readme, please conform to the standard-readme specification.

License

MIT © Juan Batiz-Benet

Documentation

Overview

Package unixfs implements a data format for files in the IPFS filesystem It is not the only format in ipfs, but it is the one that the filesystem assumes

Index

• Constants
• Variables
• func BytesForMetadata(m *Metadata) ([]byte, error)
• func DataSize(data []byte) (uint64, error)
• func EmptyDirNode() *dag.ProtoNode
• func EmptyFileNode() *dag.ProtoNode
• func FilePBData(data []byte, totalsize uint64) []byte
• func FolderPBData() []byte
• func FromBytes(data []byte) (*pb.Data, error)
• func HAMTShardData(data []byte, fanout uint64, hashType uint64) ([]byte, error)
• func ReadUnixFSNodeData(node ipld.Node) (data []byte, err error)
• func SymlinkData(path string) ([]byte, error)
• func UnwrapData(data []byte) ([]byte, error)
• func WrapData(b []byte) []byte
• type FSNode
  • func ExtractFSNode(node ipld.Node) (*FSNode, error)
  • func FSNodeFromBytes(b []byte) (*FSNode, error)
  • func NewFSNode(dataType pb.Data_DataType) *FSNode
  • func (n *FSNode) AddBlockSize(s uint64)
  • func (n *FSNode) BlockSize(i int) uint64
  • func (n *FSNode) BlockSizes() []uint64
  • func (n *FSNode) Data() []byte
  • func (n *FSNode) Fanout() uint64
  • func (n *FSNode) FileSize() uint64
  • func (n *FSNode) GetBytes() ([]byte, error)
  • func (n *FSNode) HashType() uint64
  • func (n *FSNode) IsDir() bool
  • func (n *FSNode) NumChildren() int
  • func (n *FSNode) RemoveAllBlockSizes()
  • func (n *FSNode) RemoveBlockSize(i int)
  • func (n *FSNode) SetData(newData []byte)
  • func (n *FSNode) Type() pb.Data_DataType
  • func (n *FSNode) UpdateFilesize(filesizeDiff int64)
• type LinkResult
• type Metadata
  • func MetadataFromBytes(b []byte) (*Metadata, error)
  • func (m *Metadata) Bytes() ([]byte, error)

Constants

const (
	TRaw       = pb.Data_Raw
	TFile      = pb.Data_File
	TDirectory = pb.Data_Directory
	TMetadata  = pb.Data_Metadata
	TSymlink   = pb.Data_Symlink
	THAMTShard = pb.Data_HAMTShard
)

Shorthands for protobuffer types

Variables

var (
	ErrMalformedFileFormat = errors.New("malformed data in file format")
	ErrUnrecognizedType    = errors.New("unrecognized node type")
)

Common errors

Functions

func BytesForMetadata

func BytesForMetadata(m *Metadata) ([]byte, error)

BytesForMetadata wraps the given Metadata as a profobuf message of Data type, setting the DataType to Metadata. The wrapped bytes are itself the result of calling m.Bytes().

func DataSize

func DataSize(data []byte) (uint64, error)

DataSize returns the size of the contents in protobuf wrapped slice. For raw data it simply provides the length of it. For Data_Files, it will return the associated filesize. Note that Data_Directories will return an error.

func EmptyDirNode

func EmptyDirNode() *dag.ProtoNode

EmptyDirNode creates an empty folder Protonode.

func EmptyFileNode

func EmptyFileNode() *dag.ProtoNode

EmptyFileNode creates an empty file Protonode.

func FilePBData

func FilePBData(data []byte, totalsize uint64) []byte

FilePBData creates a protobuf File with the given byte slice and returns the marshaled protobuf bytes representing it.

func FolderPBData

func FolderPBData() []byte

FolderPBData returns Bytes that represent a Directory.

func FromBytes

func FromBytes(data []byte) (*pb.Data, error)

FromBytes unmarshals a byte slice as protobuf Data. Deprecated: Use `FSNodeFromBytes` instead to avoid direct manipulation of `pb.Data`.

func HAMTShardData

func HAMTShardData(data []byte, fanout uint64, hashType uint64) ([]byte, error)

HAMTShardData return a `Data_HAMTShard` protobuf message

func ReadUnixFSNodeData

func ReadUnixFSNodeData(node ipld.Node) (data []byte, err error)

ReadUnixFSNodeData extracts the UnixFS data from an IPLD node. Raw nodes are (also) processed because they are used as leaf nodes containing (only) UnixFS data.

func SymlinkData

func SymlinkData(path string) ([]byte, error)

SymlinkData returns a `Data_Symlink` protobuf message for the path you specify.

func UnwrapData

func UnwrapData(data []byte) ([]byte, error)

UnwrapData unmarshals a protobuf messages and returns the contents.

func WrapData

func WrapData(b []byte) []byte

WrapData marshals raw bytes into a `Data_Raw` type protobuf message.

Types

type FSNode

type FSNode struct {
	// contains filtered or unexported fields
}

An FSNode represents a filesystem object using the UnixFS specification.

The `NewFSNode` constructor should be used instead of just calling `new(FSNode)` to guarantee that the required (`Type` and `Filesize`) fields in the `format` structure are initialized before marshaling (in `GetBytes()`).

func ExtractFSNode

func ExtractFSNode(node ipld.Node) (*FSNode, error)

Extract the `unixfs.FSNode` from the `ipld.Node` (assuming this was implemented by a `mdag.ProtoNode`).

func FSNodeFromBytes

func FSNodeFromBytes(b []byte) (*FSNode, error)

FSNodeFromBytes unmarshal a protobuf message onto an FSNode.

func NewFSNode

func NewFSNode(dataType pb.Data_DataType) *FSNode

NewFSNode creates a new FSNode structure with the given `dataType`.

It initializes the (required) `Type` field (that doesn't have a `Set()` accessor so it must be specified at creation), otherwise the `Marshal()` method in `GetBytes()` would fail (`required field "Type" not set`).

It also initializes the `Filesize` pointer field to ensure its value is never nil before marshaling, this is not a required field but it is done to be backwards compatible with previous `go-ipfs` versions hash. (If it wasn't initialized there could be cases where `Filesize` could have been left at nil, when the `FSNode` was created but no data or child nodes were set to adjust it, as is the case in `NewLeaf()`.)

func (*FSNode) AddBlockSize

func (n *FSNode) AddBlockSize(s uint64)

AddBlockSize adds the size of the next child block of this node

func (*FSNode) BlockSize

func (n *FSNode) BlockSize(i int) uint64

BlockSize returns the block size indexed by `i`. TODO: Evaluate if this function should be bounds checking.

func (*FSNode) BlockSizes

func (n *FSNode) BlockSizes() []uint64

BlockSizes gets blocksizes of format

func (*FSNode) Data

func (n *FSNode) Data() []byte

Data retrieves the `Data` field from the internal `format`.

func (*FSNode) Fanout

func (n *FSNode) Fanout() uint64

Fanout gets fanout of format

func (*FSNode) FileSize

func (n *FSNode) FileSize() uint64

FileSize returns the size of the file.

func (*FSNode) GetBytes

func (n *FSNode) GetBytes() ([]byte, error)

GetBytes marshals this node as a protobuf message.

func (*FSNode) HashType

func (n *FSNode) HashType() uint64

HashType gets hash type of format

func (*FSNode) IsDir

func (n *FSNode) IsDir() bool

IsDir checks whether the node represents a directory

func (*FSNode) NumChildren

func (n *FSNode) NumChildren() int

NumChildren returns the number of child blocks of this node

func (*FSNode) RemoveAllBlockSizes

func (n *FSNode) RemoveAllBlockSizes()

RemoveAllBlockSizes removes all the child block sizes of this node.

func (*FSNode) RemoveBlockSize

func (n *FSNode) RemoveBlockSize(i int)

RemoveBlockSize removes the given child block's size.

func (*FSNode) SetData

func (n *FSNode) SetData(newData []byte)

SetData sets the `Data` field from the internal `format` updating its `Filesize`.

func (*FSNode) Type

func (n *FSNode) Type() pb.Data_DataType

Type retrieves the `Type` field from the internal `format`.

func (*FSNode) UpdateFilesize

func (n *FSNode) UpdateFilesize(filesizeDiff int64)

UpdateFilesize updates the `Filesize` field from the internal `format` by a signed difference (`filesizeDiff`). TODO: Add assert to check for `Filesize` > 0?

type LinkResult

type LinkResult struct {
	Link *ipld.Link
	Err  error
}

A LinkResult for any parallel enumeration of links TODO: Should this live in go-ipld-format?

type Metadata

type Metadata struct {
	MimeType string
	Size     uint64
}

Metadata is used to store additional FSNode information.

func MetadataFromBytes

func MetadataFromBytes(b []byte) (*Metadata, error)

MetadataFromBytes Unmarshals a protobuf Data message into Metadata. The provided slice should have been encoded with BytesForMetadata().

func (*Metadata) Bytes

func (m *Metadata) Bytes() ([]byte, error)

Bytes marshals Metadata as a protobuf message of Metadata type.