pack

Documentation

Index

• Constants
• Variables
• func AssembleFileFromLinks(links []format.Link) ([]blocks.Block, *merkledag.ProtoNode, error)
• func GenerateCarHeader(root cid.Cid) ([]byte, error)
• func GetBlockStreamFromFileRange(ctx context.Context, handler datasource.ReadHandler, fileRange model.FileRange, ...) (<-chan BlockResult, fs.Object, error)
• func GetCommp(calc *commp.Calc, targetPieceSize uint64) (cid.Cid, uint64, error)
• func GetMultiWriter(outDir string) (io.WriteCloser, *commp.Calc, string, error)
• func IsSameEntry(ctx context.Context, file model.File, object fs.Object) (bool, string)
• func Min(i int, i2 int) int
• func WriteCarBlock(writer io.Writer, block blocks.Block) (int64, error)
• func WriteCarHeader(writer io.Writer, root cid.Cid) ([]byte, error)
• type BlockResult
• type CarResult
• type Result
  • func AssembleCar(ctx context.Context, handler datasource.ReadHandler, dataset model.Dataset, ...) (*Result, error)
• type WriteCloser

Constants

const ChunkSize int64 = 1 << 20

const NumLinkPerNode = 1024

Variables

var EmptyFileCid = cid.NewCidV1(cid.Raw, util.Hash([]byte("")))

var ErrFileModified = errors.New("file has been modified")

Functions

func AssembleFileFromLinks

func AssembleFileFromLinks(links []format.Link) ([]blocks.Block, *merkledag.ProtoNode, error)

AssembleFileFromLinks constructs a MerkleDAG from a list of links. It organizes the links into a tree structure where each internal node can have up to NumLinkPerNode children. This function assembles the DAG and returns the blocks that make up the DAG and the root node of the DAG.

Parameters:

• links: An array of format.Link objects representing the links to the content that will be part of the MerkleDAG.

Returns:

• []blocks.Block: A slice of Block objects representing the blocks of the constructed MerkleDAG. Each block contains a part of the overall data.
• *merkledag.ProtoNode: The root node of the created MerkleDAG. This node provides a starting point to navigate through the rest of the DAG.
• error: An error that can occur during the MerkleDAG creation process, or nil if the operation was successful.

func GenerateCarHeader

func GenerateCarHeader(root cid.Cid) ([]byte, error)

GenerateCarHeader generates the CAR (Content Addressable aRchive) format header based on the given root CID (Content Identifier).

Parameters:

• root: The root CID of the MerkleDag that the CAR file represents.

Returns:

• []byte: The byte representation of the CAR header.
• error: An error that can occur during the header generation process, or nil if successful.

func GetBlockStreamFromFileRange

func GetBlockStreamFromFileRange(ctx context.Context,
	handler datasource.ReadHandler,
	fileRange model.FileRange,
	encryptor encryption.Encryptor) (<-chan BlockResult, fs.Object, error)

GetBlockStreamFromFileRange reads a file (or a part of a file) identified by fileRange from a specified data source. Optionally, it applies encryption to the file's content. It then streams the resulting data blocks to the caller through a Go channel.

Parameters:

• ctx: A context.Context used to control the lifecycle of the function.
• handler: A datasource.ReadHandler interface implementation that is capable of reading files from a data source.
• fileRange: A model.FileRange struct that specifies the file to read and the range of bytes to read.
• encryptor: An encryption.Encryptor interface implementation that is capable of encrypting the stream. If this is nil, the file's content is streamed without encryption.

Returns: - A channel that the caller can range over to receive data blocks from the file. - An fs.Object representing the metadata of the file being read. - An error if any error occurs while processing.

Note:

• If encryption is requested (i.e., encryptor is not nil), partial reads (i.e., reading a subrange of the file) are not supported and the function will return an error in this case.
• The function is designed to be used concurrently, as it runs a goroutine to read blocks from the file.

func GetCommp

func GetCommp(calc *commp.Calc, targetPieceSize uint64) (cid.Cid, uint64, error)

GetCommp calculates the data commitment (CommP) and the piece size based on the provided commp.Calc instance and target piece size. It ensures that the calculated piece size matches the target piece size specified. If necessary, it pads the data to meet the target piece size.

Parameters:

• calc: A pointer to a commp.Calc instance, which has been used to write data and will be used to calculate the piece commitment for that data.
• targetPieceSize: The desired size of the piece, specified in bytes.

Returns:

• cid.Cid: A CID (Content Identifier) representing the data commitment (CommP).
• uint64: The size of the piece, in bytes, after potential padding.
• error: An error indicating issues during the piece commitment calculation, padding, or CID conversion, or nil if the operation was successful.

func GetMultiWriter

func GetMultiWriter(outDir string) (io.WriteCloser, *commp.Calc, string, error)

GetMultiWriter constructs a writer that writes to multiple outputs (i.e., a MultiWriter) based on the specified output directory. This function is designed to handle the use case where you want to calculate a piece commitment (via a commp.Calc instance) while writing data, and potentially write that data to a file at the same time.

Parameters:

• outDir: A string representing the path of the output directory. If this string is empty, the function will return a writer that only writes to the commp.Calc instance.

Returns:

• io.WriteCloser: An interface that groups the basic Write and Close methods. The Write method is implemented by an io.MultiWriter that writes to both a commp.Calc instance and an os.File (if an output directory was specified). The Close method is implemented by the os.File, or a nopCloser if no output directory was specified.
• *commp.Calc: A pointer to a commp.Calc instance. This is used to calculate the piece commitment of the data that is written.
• string: The path of the file that the data is being written to. This is an empty string if no output directory was specified.
• error: An error that can occur when creating the file at the specified path, or nil if the operation was successful.

func IsSameEntry

func IsSameEntry(ctx context.Context, file model.File, object fs.Object) (bool, string)

IsSameEntry checks if a given model.File and a given fs.Object represent the same entry, based on their size, hash, and last modification time.

Parameters:

• ctx: Context that allows for asynchronous task cancellation.
• file: A model.File instance representing a file in the model.
• object: An fs.Object instance representing a file or object in the filesystem.

Returns:

• bool: A boolean indicating whether the given model.File and fs.Object are considered the same entry.
• string: A string providing details in case of a mismatch.

The function performs the following checks:

1. Compares the sizes of 'file' and 'object'. If there is a mismatch, it returns false along with a formatted string that shows the mismatched sizes.
2. Retrieves the last modified time of 'object'.
3. Identifies a supported hash type for the storage backend of 'object' and computes its hash value.
4. The hash computation is skipped if the storage backend is a local file system or does not support any hash types.
5. If both 'file' and 'object' have non-empty hash values and these values do not match, it returns false along with a formatted string that shows the mismatched hash values.
6. Compares the last modified times of 'file' and 'object' at the nanosecond precision. If there is a mismatch, it returns false along with a formatted string that shows the mismatched times.
7. If all the checks pass (sizes, hashes, and last modified times match), the function returns true, indicating that 'file' and 'object' are considered to be the same entry.

Note: - In certain cases (e.g., failures during fetch), the last modified time might not be reliable. - For local file systems, hash computation is skipped to avoid inefficient operations.

func Min

func Min(i int, i2 int) int

func WriteCarBlock

func WriteCarBlock(writer io.Writer, block blocks.Block) (int64, error)

WriteCarBlock writes a block in CAR (Content Addressable aRchive) format to a given io.Writer.

Parameters:

• writer: An io.Writer to which the CAR-formatted block will be written.
• block: A blocks.Block instance representing the data to be written.

Returns:

• int64: The number of bytes written to the writer.
• error: An error that can occur during the write process, or nil if the write was successful.

func WriteCarHeader

func WriteCarHeader(writer io.Writer, root cid.Cid) ([]byte, error)

WriteCarHeader writes the CAR (Content Addressable aRchive) format header to a given io.Writer.

Parameters:

• writer: An io.Writer to which the CAR header will be written.
• root: The root CID (Content Identifier) for the CAR file.

Returns:

• []byte: The byte representation of the CAR header.
• error: An error that can occur during the write process, or nil if the write was successful.

Types

type BlockResult

type BlockResult struct {
	// Offset is the offset of the block in the potentially encrypted stream
	Offset int64
	// Raw is the block data which is potentially encrypted
	Raw []byte
	// CID is the CID of the block
	CID   cid.Cid
	Error error
}

type CarResult

type CarResult struct {
	CarFilePath string
	CarFileSize int64
	PieceCID    cid.Cid
	PieceSize   int64
	RootCID     cid.Cid
	Header      []byte
	CarBlocks   []model.CarBlock
}

type Result

type Result struct {
	FileRangeCIDs map[uint64]cid.Cid
	Objects       map[uint64]fs.Object
	CarResults    []CarResult
}

func AssembleCar

func AssembleCar(
	ctx context.Context,
	handler datasource.ReadHandler,
	dataset model.Dataset,
	fileRanges []model.FileRange,
	outDir string,
	pieceSize int64,
) (*Result, error)

AssembleCar assembles a Content Addressable aRchive (CAR) file from a list of data blocks. It writes these blocks to the specified output directory, and it attempts to ensure that the size of the CAR file matches a given target piece size. The function can handle assembling multiple CAR files if the blocks don't fit into a single CAR file of the target size.

Parameters: - ctx: Context used to handle cancellation and deadlines. - handler: The ReadHandler interface which handles reading data from a data source. - dataset: Information about the dataset being processed. - fileRanges: List of data blocks that need to be included in the CAR file. - outDir: Directory where the CAR file(s) will be written. - pieceSize: Target size of each CAR file, in bytes.

Returns:

• *Result: A structure that contains results of the CAR file creation, including CIDs of the files, file paths, and sizes.
• error: An error that occurred during the CAR assembly process, or nil if the operation was successful.

type WriteCloser

type WriteCloser struct {
	io.Writer
	io.Closer
}