car

package
v0.10.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2023 License: Apache-2.0, MIT, Apache-2.0, + 1 more Imports: 26 Imported by: 0

Documentation

Overview

Package car allows inspecting and reading CAR files, described at https://ipld.io/specs/transport/car/. The entire library is geared towards the CARv2 spec, but many of the APIs consuming CAR files also accept CARv1.

The blockstore sub-package contains an implementation of the go-ipfs-blockstore interface.

Index

Examples

Constants

View Source 
const (
	// PragmaSize is the size of the CARv2 pragma in bytes.
	PragmaSize = 11
	// HeaderSize is the fixed size of CARv2 header in number of bytes.
	HeaderSize = 40
	// CharacteristicsSize is the fixed size of Characteristics bitfield within CARv2 header in number of bytes.
	CharacteristicsSize = 16
)
View Source 
const DefaultMaxAllowedHeaderSize = carv1.DefaultMaxAllowedHeaderSize

DefaultMaxAllowedHeaderSize specifies the default maximum size that a CARv1 decode (including within a CARv2 container) will allow a header to be without erroring. This is to prevent OOM errors where a header prefix includes a too-large size specifier. Currently set to 32 MiB.

View Source 
const DefaultMaxAllowedSectionSize = carv1.DefaultMaxAllowedSectionSize

DefaultMaxAllowedHeaderSize specifies the default maximum size that a CARv1 decode (including within a CARv2 container) will allow a section to be without erroring. This is to prevent OOM errors where a section prefix includes a too-large size specifier. Typically IPLD blocks should be under 2 MiB (ideally under 1 MiB), so unless atypical data is expected, this should not be a large value. Currently set to 8 MiB.

View Source 
const DefaultMaxIndexCidSize = 2 << 10 // 2 KiB

DefaultMaxIndexCidSize specifies the maximum size in byptes accepted as a section CID by CARv2 index.

Variables

View Source 
var ErrAlreadyV1 = errors.New("already a CARv1")

ErrAlreadyV1 signals that the given payload is already in CARv1 format.

View Source 
var ErrOffsetImpossible = fmt.Errorf("car-error-offsetimpossible")

ErrOffsetImpossible is returned when specified paddings or offsets of either a wrapped carv1 or index cannot be satisfied based on the data being written.

View Source 
var ErrSizeMismatch = fmt.Errorf("car-error-sizemismatch")

ErrSizeMismatch is returned when a written traversal realizes the written header size does not match the actual number of car bytes written.

View Source 
var Pragma = []byte{
	0x0a,
	0xa1,
	0x67,
	0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e,
	0x02,
}

The pragma of a CARv2, containing the version number. This is a valid CARv1 header, with version number of 2 and no root CIDs.

Functions

func AttachIndex 

func AttachIndex(path string, idx index.Index, offset uint64) error

AttachIndex attaches a given index to an existing CARv2 file at given path and offset.

func ExtractV1File 

func ExtractV1File(srcPath, dstPath string, opts ...Option) (err error)

ExtractV1File takes a CARv2 file and extracts its CARv1 data payload, unmodified. The resulting CARv1 file will not include any data payload padding that may be present in the CARv2 srcPath. If srcPath represents a CARv1 ErrAlreadyV1 error is returned. The srcPath is assumed to exist, and the destination path is created if not exist. Note that the destination path might still be created even if an error occurred. If srcPath and dstPath are the same, then the dstPath is converted, in-place, to CARv1.

This function aims to extract the CARv1 payload as efficiently as possible. The method is best-effort and depends on your operating system; for example, it should use copy_file_range on recent Linux versions. This API should be preferred over copying directly via Reader.DataReader, as it should allow for better performance while always being at least as efficient.

func GenerateIndex 

func GenerateIndex(v1r io.Reader, opts ...Option) (index.Index, error)

GenerateIndex generates index for the given car payload reader. The index can be stored in serialized format using index.WriteTo.

Note, the index is re-generated every time even if the payload is in CARv2 format and already has an index. To read existing index when available see ReadOrGenerateIndex. See: LoadIndex.

func GenerateIndexFromFile 

func GenerateIndexFromFile(path string, opts ...Option) (index.Index, error)

GenerateIndexFromFile walks a CAR file at the give path and generates an index of cid->byte offset. The index can be stored using index.WriteTo. Both CARv1 and CARv2 formats are accepted.

Note, the index is re-generated every time even if the given CAR file is in CARv2 format and already has an index. To read existing index when available see ReadOrGenerateIndex.

See: GenerateIndex.

func LoadIndex 

func LoadIndex(idx index.Index, r io.Reader, opts ...Option) error

LoadIndex populates idx with index records generated from r. The r may be in CARv1 or CARv2 format.

If the StoreIdentityCIDs option is set when calling LoadIndex, identity CIDs will be included in the index. By default this option is off, and identity CIDs will not be included in the index.

Note, the index is re-generated every time even if r is in CARv2 format and already has an index. To read existing index when available see ReadOrGenerateIndex.

func ReadOrGenerateIndex 

func ReadOrGenerateIndex(rs io.ReadSeeker, opts ...Option) (index.Index, error)

ReadOrGenerateIndex accepts both CARv1 and CARv2 formats, and reads or generates an index for it. When the given reader is in CARv1 format an index is always generated. For a payload in CARv2 format, an index is only generated if Header.HasIndex returns false. An error is returned for all other formats, i.e. pragma with versions other than 1 or 2.

Note, the returned index lives entirely in memory and will not depend on the given reader to fulfill index lookup.

func ReadVersion 

func ReadVersion(r io.Reader, opts ...Option) (uint64, error)

ReadVersion reads the version from the pragma. This function accepts both CARv1 and CARv2 payloads.

func ReplaceRootsInFile 

func ReplaceRootsInFile(path string, roots []cid.Cid, opts ...Option) (err error)

ReplaceRootsInFile replaces the root CIDs in CAR file at given path with the given roots. This function accepts both CARv1 and CARv2 files.

Note that the roots are only replaced if their total serialized size exactly matches the total serialized size of existing roots in CAR file.

func TraverseToFile 

func TraverseToFile(ctx context.Context, ls *ipld.LinkSystem, root cid.Cid, selector ipld.Node, destination string, opts ...Option) error

TraverseToFile writes a car file matching a given root and selector to the path at `destination` using one read of each block.

func TraverseV1 

func TraverseV1(ctx context.Context, ls *ipld.LinkSystem, root cid.Cid, selector ipld.Node, writer io.Writer, opts ...Option) (uint64, error)

TraverseV1 walks through the proposed dag traversal and writes a carv1 to the provided io.Writer

func WrapV1 

func WrapV1(src io.ReadSeeker, dst io.Writer, opts ...Option) error

WrapV1 takes a CARv1 file and wraps it as a CARv2 file with an index. The resulting CARv2 file's inner CARv1 payload is left unmodified, and does not use any padding before the innner CARv1 or index.

func WrapV1File 

func WrapV1File(srcPath, dstPath string) error

WrapV1File is a wrapper around WrapV1 that takes filesystem paths. The source path is assumed to exist, and the destination path is overwritten. Note that the destination path might still be created even if an error occurred.

Example 
package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"os"
	"path/filepath"

	carv2 "github.com/ipfs/boxo/ipld/car/v2"
	"github.com/ipfs/boxo/ipld/car/v2/blockstore"
)

func main() {
	// We have a sample CARv1 file.
	// Wrap it as-is in a CARv2, with an index.
	// Writing the result to testdata allows reusing that file in other tests,
	// and also helps ensure that the result is deterministic.
	src := "testdata/sample-v1.car"
	tdir, err := os.MkdirTemp(os.TempDir(), "example-*")
	if err != nil {
		panic(err)
	}
	dst := filepath.Join(tdir, "wrapped-v2.car")
	if err := carv2.WrapV1File(src, dst); err != nil {
		panic(err)
	}

	// Open our new CARv2 file and show some info about it.
	cr, err := carv2.OpenReader(dst)
	if err != nil {
		panic(err)
	}
	defer func() {
		if err := cr.Close(); err != nil {
			panic(err)
		}
	}()

	roots, err := cr.Roots()
	if err != nil {
		panic(err)
	}
	fmt.Println("Roots:", roots)
	fmt.Println("Has index:", cr.Header.HasIndex())

	// Verify that the CARv1 remains exactly the same.
	orig, err := os.ReadFile(src)
	if err != nil {
		panic(err)
	}
	dr, err := cr.DataReader()
	if err != nil {
		panic(err)
	}
	inner, err := io.ReadAll(dr)
	if err != nil {
		panic(err)
	}
	fmt.Println("Inner CARv1 is exactly the same:", bytes.Equal(orig, inner))

	// Verify that the CARv2 works well with its index.
	bs, err := blockstore.OpenReadOnly(dst)
	if err != nil {
		panic(err)
	}
	fmt.Println(bs.Get(context.TODO(), roots[0]))

}

Output:

Roots: [bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy]
Has index: true
Inner CARv1 is exactly the same: true
[Block bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy] <nil>

Types

type BlockMetadata 

type BlockMetadata struct {
	cid.Cid
	Offset uint64
	Size   uint64
}

type BlockReader 

type BlockReader struct {
	// The detected version of the CAR payload.
	Version uint64
	// The roots of the CAR payload. May be empty.
	Roots []cid.Cid
	// contains filtered or unexported fields
}

BlockReader facilitates iteration over CAR blocks for both CARv1 and CARv2. See NewBlockReader

func NewBlockReader 

func NewBlockReader(r io.Reader, opts ...Option) (*BlockReader, error)

NewBlockReader instantiates a new BlockReader facilitating iteration over blocks in CARv1 or CARv2 payload. Upon instantiation, the version is automatically detected and exposed via BlockReader.Version. The root CIDs of the CAR payload are exposed via BlockReader.Roots

See BlockReader.Next

Example

ExampleNewBlockReader instantiates a new BlockReader for a CARv1 file and its wrapped CARv2 version. For each file, it prints the version, the root CIDs and the first five block CIDs. Note, the roots and first five block CIDs are identical in both files since both represent the same root CIDs and data blocks. 

package main

import (
	"fmt"
	"io"
	"os"

	carv2 "github.com/ipfs/boxo/ipld/car/v2"
)

func main() {
	for _, path := range []string{
		"testdata/sample-v1.car",
		"testdata/sample-wrapped-v2.car",
	} {
		fmt.Println("File:", path)
		f, err := os.Open(path)
		if err != nil {
			panic(err)
		}
		br, err := carv2.NewBlockReader(f)
		if err != nil {
			panic(err)
		}
		defer func() {
			if err := f.Close(); err != nil {
				panic(err)
			}
		}()
		fmt.Println("Version:", br.Version)
		fmt.Println("Roots:", br.Roots)
		fmt.Println("First 5 block CIDs:")
		for i := 0; i < 5; i++ {
			bl, err := br.Next()
			if err == io.EOF {
				break
			}
			if err != nil {
				panic(err)
			}
			fmt.Printf("\t%v\n", bl.Cid())
		}
	}
}

Output:

File: testdata/sample-v1.car
Version: 1
Roots: [bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy]
First 5 block CIDs:
	bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy
	bafy2bzaceaycv7jhaegckatnncu5yugzkrnzeqsppzegufr35lroxxnsnpspu
	bafy2bzaceb62wdepofqu34afqhbcn4a7jziwblt2ih5hhqqm6zitd3qpzhdp4
	bafy2bzaceb3utcspm5jqcdqpih3ztbaztv7yunzkiyfq7up7xmokpxemwgu5u
	bafy2bzacedjwekyjresrwjqj4n2r5bnuuu3klncgjo2r3slsp6wgqb37sz4ck
File: testdata/sample-wrapped-v2.car
Version: 2
Roots: [bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy]
First 5 block CIDs:
	bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy
	bafy2bzaceaycv7jhaegckatnncu5yugzkrnzeqsppzegufr35lroxxnsnpspu
	bafy2bzaceb62wdepofqu34afqhbcn4a7jziwblt2ih5hhqqm6zitd3qpzhdp4
	bafy2bzaceb3utcspm5jqcdqpih3ztbaztv7yunzkiyfq7up7xmokpxemwgu5u
	bafy2bzacedjwekyjresrwjqj4n2r5bnuuu3klncgjo2r3slsp6wgqb37sz4ck

func (*BlockReader) Next 

func (br *BlockReader) Next() (blocks.Block, error)

Next iterates over blocks in the underlying CAR payload with an io.EOF error indicating the end is reached. Note, this function is forward-only; once the end has been reached it will always return io.EOF.

When the payload represents a CARv1 the BlockReader.Next simply iterates over blocks until it reaches the end of the underlying io.Reader stream.

As for CARv2 payload, the underlying io.Reader is read only up to the end of the last block. Note, in a case where ZeroLengthSectionAsEOF Option is enabled, io.EOF is returned immediately upon encountering a zero-length section without reading any further bytes from the underlying io.Reader.

func (*BlockReader) SkipNext 

func (br *BlockReader) SkipNext() (*BlockMetadata, error)

SkipNext jumps over the next block, returning metadata about what it is (the CID, offset, and size). Like Next it will return an io.EOF once it has reached the end.

If the underlying reader used by the BlockReader is actually a ReadSeeker, this method will attempt to seek over the underlying data rather than reading it into memory.

type Characteristics 

type Characteristics struct {
	Hi uint64
	Lo uint64
}

Characteristics is a bitfield placeholder for capturing the characteristics of a CARv2 such as order and determinism.

func (*Characteristics) IsFullyIndexed 

func (c *Characteristics) IsFullyIndexed() bool

IsFullyIndexed specifies whether the index of CARv2 represents a catalog of all CID segments. See StoreIdentityCIDs

func (*Characteristics) ReadFrom 

func (c *Characteristics) ReadFrom(r io.Reader) (int64, error)

func (*Characteristics) SetFullyIndexed 

func (c *Characteristics) SetFullyIndexed(b bool)

SetFullyIndexed sets whether of CARv2 represents a catalog of all CID segments.

func (Characteristics) WriteTo 

func (c Characteristics) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes this characteristics to the given w.

type ErrCidTooLarge 

type ErrCidTooLarge struct {
	MaxSize     uint64
	CurrentSize uint64
}

ErrCidTooLarge signals that a CID is too large to include in CARv2 index. See: MaxIndexCidSize.

func (*ErrCidTooLarge) Error 

func (e *ErrCidTooLarge) Error() string
type Header struct {
	// 128-bit characteristics of this CARv2 file, such as order, deduplication, etc. Reserved for future use.
	Characteristics Characteristics
	// The byte-offset from the beginning of the CARv2 to the first byte of the CARv1 data payload.
	DataOffset uint64
	// The byte-length of the CARv1 data payload.
	DataSize uint64
	// The byte-offset from the beginning of the CARv2 to the first byte of the index payload. This value may be 0 to indicate the absence of index data.
	IndexOffset uint64
}

Header represents the CARv2 header/pragma.

func NewHeader 

func NewHeader(dataSize uint64) Header

NewHeader instantiates a new CARv2 header, given the data size.

func (Header) HasIndex 

func (h Header) HasIndex() bool

HasIndex indicates whether the index is present.

func (*Header) ReadFrom 

func (h *Header) ReadFrom(r io.Reader) (int64, error)

ReadFrom populates fields of this header from the given r.

func (Header) WithDataPadding 

func (h Header) WithDataPadding(padding uint64) Header

WithDataPadding sets the data payload byte-offset from the beginning of the file for this header and returns the header for convenient chained calls. The Data offset is calculated as the sum of PragmaSize, HeaderSize and the given padding. The call to this function also shifts the Header.IndexOffset forward by the given padding.

func (Header) WithDataSize 

func (h Header) WithDataSize(size uint64) Header

func (Header) WithIndexPadding 

func (h Header) WithIndexPadding(padding uint64) Header

WithIndexPadding sets the index offset from the beginning of the file for this header and returns the header for convenient chained calls. The index offset is calculated as the sum of PragmaSize, HeaderSize, Header.DataSize, and the given padding.

func (Header) WriteTo 

func (h Header) WriteTo(w io.Writer) (n int64, err error)

WriteTo serializes this header as bytes and writes them using the given io.Writer.

type Option 

type Option func(*Options)

Option describes an option which affects behavior when interacting with CAR files.

func AllowDuplicatePuts 

func AllowDuplicatePuts(allow bool) Option

AllowDuplicatePuts is a write option which makes a CAR interface (blockstore or storage) not deduplicate blocks in Put and PutMany. The default is to deduplicate, which matches the current semantics of go-ipfs-blockstore v1.

Note that this option only affects the storage interfaces (blockstore or storage), and is ignored by the root go-car/v2 package.

func MaxAllowedHeaderSize 

func MaxAllowedHeaderSize(max uint64) Option

MaxAllowedHeaderSize overrides the default maximum size (of 32 MiB) that a CARv1 decode (including within a CARv2 container) will allow a header to be without erroring.

func MaxAllowedSectionSize 

func MaxAllowedSectionSize(max uint64) Option

MaxAllowedSectionSize overrides the default maximum size (of 8 MiB) that a CARv1 decode (including within a CARv2 container) will allow a header to be without erroring. Typically IPLD blocks should be under 2 MiB (ideally under 1 MiB), so unless atypical data is expected, this should not be a large value.

func MaxIndexCidSize 

func MaxIndexCidSize(s uint64) Option

MaxIndexCidSize specifies the maximum allowed size for indexed CIDs in bytes. Indexing a CID with larger than the allowed size results in ErrCidTooLarge error. 

func MaxTraversalLinks(MaxTraversalLinks uint64) Option

MaxTraversalLinks changes the allowed number of links a selector traversal can execute before failing.

Note that setting this option may cause an error to be returned from selector execution when building a SelectiveCar.

func StoreIdentityCIDs 

func StoreIdentityCIDs(b bool) Option

StoreIdentityCIDs sets whether to persist sections that are referenced by CIDs with multihash.IDENTITY digest. When writing CAR files with this option, Characteristics.IsFullyIndexed will be set.

By default, the blockstore interface will always return true for Has() called with identity CIDs, but when this option is turned on, it will defer to the index.

When creating an index (or loading a CARv1 as a blockstore), when this option is on, identity CIDs will be included in the index.

This option is disabled by default.

func UseDataPadding 

func UseDataPadding(p uint64) Option

UseDataPadding sets the padding to be added between CARv2 header and its data payload on Finalize.

func UseIndexCodec 

func UseIndexCodec(c multicodec.Code) Option

UseIndexCodec sets the codec used for index generation.

func UseIndexPadding 

func UseIndexPadding(p uint64) Option

UseIndexPadding sets the padding between data payload and its index on Finalize.

func UseWholeCIDs 

func UseWholeCIDs(enable bool) Option

UseWholeCIDs is a read option which makes a CAR storage interface (blockstore or storage) identify blocks by whole CIDs, and not just their multihashes. The default is to use multihashes, which matches the current semantics of go-ipfs-blockstore v1.

Enabling this option affects a number of methods, including read-only ones:

• Get, Has, and HasSize will only return a block only if the entire CID is present in the CAR file.

• AllKeysChan will return the original whole CIDs, instead of with their multicodec set to "raw" to just provide multihashes.

• If AllowDuplicatePuts isn't set, Put and PutMany will deduplicate by the whole CID, allowing different CIDs with equal multihashes.

Note that this option only affects the storage interfaces (blockstore or storage), and is ignored by the root go-car/v2 package.

func WithTraversalPrototypeChooser 

func WithTraversalPrototypeChooser(t traversal.LinkTargetNodePrototypeChooser) Option

WithTraversalPrototypeChooser specifies the prototype chooser that should be used when performing traversals in writes from a linksystem.

func WithTrustedCAR 

func WithTrustedCAR(t bool) Option

WithTrustedCAR specifies whether CIDs match the block data as they are read from the CAR files.

func WithoutIndex 

func WithoutIndex() Option

WithoutIndex flags that no index should be included in generation.

func WriteAsCarV1 

func WriteAsCarV1(asCarV1 bool) Option

WriteAsCarV1 is a write option which makes a CAR interface (blockstore or storage) write the output as a CARv1 only, with no CARv2 header or index. Indexing is used internally during write but is discarded upon finalization.

Note that this option only affects the storage interfaces (blockstore or storage), and is ignored by the root go-car/v2 package.

func ZeroLengthSectionAsEOF 

func ZeroLengthSectionAsEOF(enable bool) Option

ZeroLengthSectionAsEOF sets whether to allow the CARv1 decoder to treat a zero-length section as the end of the input CAR file. For example, this can be useful to allow "null padding" after a CARv1 without knowing where the padding begins.

type Options 

type Options struct {
	DataPadding            uint64
	IndexPadding           uint64
	IndexCodec             multicodec.Code
	ZeroLengthSectionAsEOF bool
	MaxIndexCidSize        uint64
	StoreIdentityCIDs      bool

	BlockstoreAllowDuplicatePuts bool
	BlockstoreUseWholeCIDs       bool
	MaxTraversalLinks            uint64
	WriteAsCarV1                 bool
	TraversalPrototypeChooser    traversal.LinkTargetNodePrototypeChooser
	TrustedCAR                   bool

	MaxAllowedHeaderSize  uint64
	MaxAllowedSectionSize uint64
}

Options holds the configured options after applying a number of Option funcs.

This type should not be used directly by end users; it's only exposed as a side effect of Option.

func ApplyOptions 

func ApplyOptions(opt ...Option) Options

ApplyOptions applies given opts and returns the resulting Options. This function should not be used directly by end users; it's only exposed as a side effect of Option.

type ReadOption 

type ReadOption = Option

ReadOption hints that an API wants options related only to reading CAR files.

type ReadWriteOption 

type ReadWriteOption = Option

ReadWriteOption is either a ReadOption or a WriteOption. Deprecated: use Option instead.

type Reader 

type Reader struct {
	Header  Header
	Version uint64
	// contains filtered or unexported fields
}

Reader represents a reader of CARv2.

func NewReader 

func NewReader(r io.ReaderAt, opts ...Option) (*Reader, error)

NewReader constructs a new reader that reads either CARv1 or CARv2 from the given r. Upon instantiation, the reader inspects the payload and provides appropriate read operations for both CARv1 and CARv2.

Note that any other version other than 1 or 2 will result in an error. The caller may use Reader.Version to get the actual version r represents. In the case where r represents a CARv1 Reader.Header will not be populated and is left as zero-valued.

func OpenReader 

func OpenReader(path string, opts ...Option) (*Reader, error)

OpenReader is a wrapper for NewReader which opens the file at path.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the underlying reader if it was opened by OpenReader.

func (*Reader) DataReader 

func (r *Reader) DataReader() (SectionReader, error)

DataReader provides a reader containing the data payload in CARv1 format.

func (*Reader) IndexReader 

func (r *Reader) IndexReader() (io.Reader, error)

IndexReader provides an io.Reader containing the index for the data payload if the index is present. Otherwise, returns nil. Note, this function will always return nil if the backing payload represents a CARv1.

func (*Reader) Inspect 

func (r *Reader) Inspect(validateBlockHash bool) (Stats, error)

Inspect does a quick scan of a CAR, performing basic validation of the format and returning a Stats object that provides a high-level description of the contents of the CAR. Inspect works for CARv1 and CARv2 contents. A CARv1 will return an uninitialized Header value.

If validateBlockHash is true, all block data in the payload will be hashed and compared to the CID for that block and an error will return if there is a mismatch. If false, block data will be skipped over and not checked. Performing a full block hash validation is similar to using a BlockReader and calling Next over all blocks.

Inspect will perform a basic check of a CARv2 index, where present, but this does not guarantee that the index is correct. Attempting to read index data from untrusted sources is not recommended. If required, further validation of an index can be performed by loading the index and performing a ForEach() and sanity checking that the offsets are within the data payload section of the CAR. However, re-generation of index data in this case is the recommended course of action.

Beyond the checks performed by Inspect, a valid / good CAR is somewhat use-case dependent. Factors to consider include:

  • Bad indexes, including incorrect offsets, duplicate entries, or other faulty data. Indexes should be re-generated, regardless, if you need to use them and have any reason to not trust the source.

  • Blocks use codecs that your system doesn't have access to—which may mean you can't traverse a DAG or use the contained data. Stats.CodecCounts contains a list of codecs found in the CAR so this can be checked.

  • CIDs use multihashes that your system doesn't have access to—which will mean you can't validate block hashes are correct (using validateBlockHash in this case will result in a failure). Stats.MhTypeCounts contains a list of multihashes found in the CAR so this can be checked.

  • The presence of IDENTITY CIDs, which may not be supported (or desired) by the consumer of the CAR. Stats.CodecCounts can determine the presence of IDENTITY CIDs.

  • Roots: the number of roots, duplicates, and whether they are related to the blocks contained within the CAR. Stats contains a list of Roots and a RootsPresent bool so further checks can be performed.

  • DAG completeness is not checked. Any properties relating to the DAG, or DAGs contained within a CAR are the responsibility of the user to check.

func (*Reader) Roots 

func (r *Reader) Roots() ([]cid.Cid, error)

Roots returns the root CIDs. The root CIDs are extracted lazily from the data payload header.

type SectionReader 

type SectionReader interface {
	io.Reader
	io.Seeker
	io.ReaderAt
}

SectionReader implements both io.ReadSeeker and io.ReaderAt. It is the interface version of io.SectionReader, but note that the implementation is not guaranteed to be an io.SectionReader.

type Stats 

type Stats struct {
	Version        uint64
	Header         Header
	Roots          []cid.Cid
	RootsPresent   bool
	BlockCount     uint64
	CodecCounts    map[multicodec.Code]uint64
	MhTypeCounts   map[multicodec.Code]uint64
	AvgCidLength   uint64
	MaxCidLength   uint64
	MinCidLength   uint64
	AvgBlockLength uint64
	MaxBlockLength uint64
	MinBlockLength uint64
	IndexCodec     multicodec.Code
}

Stats is returned by an Inspect() call

type WriteOption 

type WriteOption = Option

WriteOption hints that an API wants options related only to reading CAR files.

type Writer 

type Writer interface {
	io.WriterTo
}

Writer is an interface allowing writing a car prepared by PrepareTraversal

func NewSelectiveWriter 

func NewSelectiveWriter(ctx context.Context, ls *ipld.LinkSystem, root cid.Cid, selector ipld.Node, opts ...Option) (Writer, error)

NewSelectiveWriter walks through the proposed dag traversal to learn its total size in order to be able to stream out a car to a writer in the expected traversal order in one go.

Source Files

View all Source files

Directories

Path Synopsis
Package blockstore implements the IPFS blockstore interface backed by a CAR file.
 Package blockstore implements the IPFS blockstore interface backed by a CAR file.
package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset.
 package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset.
internal
carv1
Forked from CARv1 to avoid dependency to ipld-prime 0.9.0 due to outstanding upgrades in filecoin.
 Forked from CARv1 to avoid dependency to ipld-prime 0.9.0 due to outstanding upgrades in filecoin.
carv1/util
errsort
io
loader
store
Package storage provides a CAR abstraction for the github.com/ipld/go-ipld-prime/storage interfaces in the form of a StorageCar.
 Package storage provides a CAR abstraction for the github.com/ipld/go-ipld-prime/storage interfaces in the form of a StorageCar.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.