car

package module
v2.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 30, 2021 License: Apache-2.0, MIT Imports: 14 Imported by: 0

Documentation

Overview

Package car allows inspecting and reading CAR files, described at https://ld.io/specs/transport/car/. The entire library is geared towards the CARv2 spec, but many of the APIs consuming CAR files also accept CARv1.

The blockstore sub-package contains an implementation of the go-dms3-blockstore interface.

Index

Examples

Constants

View Source 
const (
	// PragmaSize is the size of the CARv2 pragma in bytes.
	PragmaSize = 11
	// HeaderSize is the fixed size of CARv2 header in number of bytes.
	HeaderSize = 40
	// CharacteristicsSize is the fixed size of Characteristics bitfield within CARv2 header in number of bytes.
	CharacteristicsSize = 16
)

Variables

View Source 
var ErrAlreadyV1 = errors.New("already a CARv1")

ErrAlreadyV1 signals that the given payload is already in CARv1 format.

View Source 
var Pragma = []byte{
	0x0a,
	0xa1,
	0x67,
	0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e,
	0x02,
}

The pragma of a CARv2, containing the version number. This is a valid CARv1 header, with version number of 2 and no root CIDs.

Functions

func AttachIndex 

func AttachIndex(path string, idx index.Index, offset uint64) error

AttachIndex attaches a given index to an existing CARv2 file at given path and offset.

func ExtractV1File 

func ExtractV1File(srcPath, dstPath string) (err error)

ExtractV1File takes a CARv2 file and extracts its CARv1 data payload, unmodified. The resulting CARv1 file will not include any data payload padding that may be present in the CARv2 srcPath. If srcPath represents a CARv1 ErrAlreadyV1 error is returned. The srcPath is assumed to exist, and the destination path is created if not exist. Note that the destination path might still be created even if an error occurred. If srcPath and dstPath are the same, then the dstPath is converted, in-place, to CARv1.

This function aims to extract the CARv1 payload as efficiently as possible. The method is best-effort and depends on your operating system; for example, it should use copy_file_range on recent Linux versions. This API should be preferred over copying directly via Reader.DataReader, as it should allow for better performance while always being at least as efficient.

func GenerateIndex 

func GenerateIndex(v1r io.Reader, opts ...ReadOption) (index.Index, error)

GenerateIndex generates index for a given car in v1 format. The index can be stored using index.Save into a file or serialized using index.WriteTo.

func GenerateIndexFromFile 

func GenerateIndexFromFile(path string) (index.Index, error)

GenerateIndexFromFile walks a car v1 file at the give path and generates an index of cid->byte offset. The index can be stored using index.Save into a file or serialized using index.WriteTo.

func ReadOrGenerateIndex 

func ReadOrGenerateIndex(rs io.ReadSeeker, opts ...ReadOption) (index.Index, error)

ReadOrGenerateIndex accepts both CARv1 and CARv2 formats, and reads or generates an index for it. When the given reader is in CARv1 format an index is always generated. For a payload in CARv2 format, an index is only generated if Header.HasIndex returns false. An error is returned for all other formats, i.e. pragma with versions other than 1 or 2.

Note, the returned index lives entirely in memory and will not depend on the given reader to fulfill index lookup.

func ReadVersion 

func ReadVersion(r io.Reader) (uint64, error)

ReadVersion reads the version from the pragma. This function accepts both CARv1 and CARv2 payloads.

func WrapV1 

func WrapV1(src io.ReadSeeker, dst io.Writer) error

WrapV1 takes a CARv1 file and wraps it as a CARv2 file with an index. The resulting CARv2 file's inner CARv1 payload is left unmodified, and does not use any padding before the innner CARv1 or index.

func WrapV1File 

func WrapV1File(srcPath, dstPath string) error

WrapV1File is a wrapper around WrapV1 that takes filesystem paths. The source path is assumed to exist, and the destination path is overwritten. Note that the destination path might still be created even if an error occurred.

Example 
package main

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"os"
	"path/filepath"

	carv2 "gitlab.dms3.io/ld/go-car/v2"
	"gitlab.dms3.io/ld/go-car/v2/blockstore"
)

func main() {
	// We have a sample CARv1 file.
	// Wrap it as-is in a CARv2, with an index.
	// Writing the result to testdata allows reusing that file in other tests,
	// and also helps ensure that the result is deterministic.
	src := "testdata/sample-v1.car"
	tdir, err := ioutil.TempDir(os.TempDir(), "example-*")
	if err != nil {
		panic(err)
	}
	dst := filepath.Join(tdir, "wrapped-v2.car")
	if err := carv2.WrapV1File(src, dst); err != nil {
		panic(err)
	}

	// Open our new CARv2 file and show some info about it.
	cr, err := carv2.OpenReader(dst)
	if err != nil {
		panic(err)
	}
	defer func() {
		if err := cr.Close(); err != nil {
			panic(err)
		}
	}()

	roots, err := cr.Roots()
	if err != nil {
		panic(err)
	}
	fmt.Println("Roots:", roots)
	fmt.Println("Has index:", cr.Header.HasIndex())

	// Verify that the CARv1 remains exactly the same.
	orig, err := ioutil.ReadFile(src)
	if err != nil {
		panic(err)
	}
	inner, err := ioutil.ReadAll(cr.DataReader())
	if err != nil {
		panic(err)
	}
	fmt.Println("Inner CARv1 is exactly the same:", bytes.Equal(orig, inner))

	// Verify that the CARv2 works well with its index.
	bs, err := blockstore.OpenReadOnly(dst)
	if err != nil {
		panic(err)
	}
	fmt.Println(bs.Get(roots[0]))

}

Output:

Roots: [bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy]
Has index: true
Inner CARv1 is exactly the same: true
[Block bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy] <nil>

Types

type BlockReader 

type BlockReader struct {
	// The detected version of the CAR payload.
	Version uint64
	// The roots of the CAR payload. May be empty.
	Roots []cid.Cid
	// contains filtered or unexported fields
}

BlockReader facilitates iteration over CAR blocks for both CARv1 and CARv2. See NewBlockReader

func NewBlockReader 

func NewBlockReader(r io.Reader, opts ...ReadOption) (*BlockReader, error)

NewBlockReader instantiates a new BlockReader facilitating iteration over blocks in CARv1 or CARv2 payload. Upon instantiation, the version is automatically detected and exposed via BlockReader.Version. The root CIDs of the CAR payload are exposed via BlockReader.Roots

See BlockReader.Next

Example

ExampleNewBlockReader instantiates a new BlockReader for a CARv1 file and its wrapped CARv2 version. For each file, it prints the version, the root CIDs and the first five block CIDs. Note, the roots and first five block CIDs are identical in both files since both represent the same root CIDs and data blocks. 

package main

import (
	"fmt"
	"io"
	"os"

	carv2 "gitlab.dms3.io/ld/go-car/v2"
)

func main() {
	for _, path := range []string{
		"testdata/sample-v1.car",
		"testdata/sample-wrapped-v2.car",
	} {
		fmt.Println("File:", path)
		f, err := os.Open(path)
		if err != nil {
			panic(err)
		}
		br, err := carv2.NewBlockReader(f)
		if err != nil {
			panic(err)
		}
		defer func() {
			if err := f.Close(); err != nil {
				panic(err)
			}
		}()
		fmt.Println("Version:", br.Version)
		fmt.Println("Roots:", br.Roots)
		fmt.Println("First 5 block CIDs:")
		for i := 0; i < 5; i++ {
			bl, err := br.Next()
			if err == io.EOF {
				break
			}
			if err != nil {
				panic(err)
			}
			fmt.Printf("\t%v\n", bl.Cid())
		}
	}
}

Output:

File: testdata/sample-v1.car
Version: 1
Roots: [bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy]
First 5 block CIDs:
	bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy
	bafy2bzaceaycv7jhaegckatnncu5yugzkrnzeqsppzegufr35lroxxnsnpspu
	bafy2bzaceb62wdepofqu34afqhbcn4a7jziwblt2ih5hhqqm6zitd3qpzhdp4
	bafy2bzaceb3utcspm5jqcdqpih3ztbaztv7yunzkiyfq7up7xmokpxemwgu5u
	bafy2bzacedjwekyjresrwjqj4n2r5bnuuu3klncgjo2r3slsp6wgqb37sz4ck
File: testdata/sample-wrapped-v2.car
Version: 2
Roots: [bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy]
First 5 block CIDs:
	bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy
	bafy2bzaceaycv7jhaegckatnncu5yugzkrnzeqsppzegufr35lroxxnsnpspu
	bafy2bzaceb62wdepofqu34afqhbcn4a7jziwblt2ih5hhqqm6zitd3qpzhdp4
	bafy2bzaceb3utcspm5jqcdqpih3ztbaztv7yunzkiyfq7up7xmokpxemwgu5u
	bafy2bzacedjwekyjresrwjqj4n2r5bnuuu3klncgjo2r3slsp6wgqb37sz4ck

func (*BlockReader) Next 

func (br *BlockReader) Next() (blocks.Block, error)

Next iterates over blocks in the underlying CAR payload with an io.EOF error indicating the end is reached. Note, this function is forward-only; once the end has been reached it will always return io.EOF.

When the payload represents a CARv1 the BlockReader.Next simply iterates over blocks until it reaches the end of the underlying io.Reader stream.

As for CARv2 payload, the underlying io.Reader is read only up to the end of the last block. Note, in a case where ReadOption.ZeroLengthSectionAsEOF is enabled, io.EOF is returned immediately upon encountering a zero-length section without reading any further bytes from the underlying io.Reader.

type Characteristics 

type Characteristics struct {
	Hi uint64
	Lo uint64
}

Characteristics is a bitfield placeholder for capturing the characteristics of a CARv2 such as order and determinism.

func (*Characteristics) ReadFrom 

func (c *Characteristics) ReadFrom(r io.Reader) (int64, error)

func (Characteristics) WriteTo 

func (c Characteristics) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes this characteristics to the given w. 

type Header struct {
	// 128-bit characteristics of this CARv2 file, such as order, deduplication, etc. Reserved for future use.
	Characteristics Characteristics
	// The byte-offset from the beginning of the CARv2 to the first byte of the CARv1 data payload.
	DataOffset uint64
	// The byte-length of the CARv1 data payload.
	DataSize uint64
	// The byte-offset from the beginning of the CARv2 to the first byte of the index payload. This value may be 0 to indicate the absence of index data.
	IndexOffset uint64
}

Header represents the CARv2 header/pragma.

func NewHeader 

func NewHeader(dataSize uint64) Header

NewHeader instantiates a new CARv2 header, given the data size.

func (Header) HasIndex 

func (h Header) HasIndex() bool

HasIndex indicates whether the index is present.

func (*Header) ReadFrom 

func (h *Header) ReadFrom(r io.Reader) (int64, error)

ReadFrom populates fields of this header from the given r.

func (Header) WithDataPadding 

func (h Header) WithDataPadding(padding uint64) Header

WithDataPadding sets the data payload byte-offset from the beginning of the file for this header and returns the header for convenient chained calls. The Data offset is calculated as the sum of PragmaSize, HeaderSize and the given padding. The call to this function also shifts the Header.IndexOffset forward by the given padding.

func (Header) WithDataSize 

func (h Header) WithDataSize(size uint64) Header

func (Header) WithIndexPadding 

func (h Header) WithIndexPadding(padding uint64) Header

WithIndexPadding sets the index offset from the beginning of the file for this header and returns the header for convenient chained calls. The index offset is calculated as the sum of PragmaSize, HeaderSize, Header.DataSize, and the given padding.

func (Header) WriteTo 

func (h Header) WriteTo(w io.Writer) (n int64, err error)

WriteTo serializes this header as bytes and writes them using the given io.Writer.

type ReadOption 

type ReadOption func(*ReadOptions)

ReadOption describes an option which affects behavior when parsing CAR files.

func ZeroLengthSectionAsEOF 

func ZeroLengthSectionAsEOF(enable bool) ReadOption

ZeroLengthSectionAsEOF is a read option which allows a CARv1 decoder to treat a zero-length section as the end of the input CAR file. For example, this can be useful to allow "null padding" after a CARv1 without knowing where the padding begins.

type ReadOptions 

type ReadOptions struct {
	ZeroLengthSectionAsEOF bool

	BlockstoreUseWholeCIDs bool
}

ReadOptions holds the configured options after applying a number of ReadOption funcs.

This type should not be used directly by end users; it's only exposed as a side effect of ReadOption.

type ReadWriteOption 

type ReadWriteOption interface {
	// contains filtered or unexported methods
}

ReadWriteOption is either a ReadOption or a WriteOption.

type Reader 

type Reader struct {
	Header  Header
	Version uint64
	// contains filtered or unexported fields
}

Reader represents a reader of CARv2.

func NewReader 

func NewReader(r io.ReaderAt, opts ...ReadOption) (*Reader, error)

NewReader constructs a new reader that reads either CARv1 or CARv2 from the given r. Upon instantiation, the reader inspects the payload and provides appropriate read operations for both CARv1 and CARv2.

Note that any other version other than 1 or 2 will result in an error. The caller may use Reader.Version to get the actual version r represents. In the case where r represents a CARv1 Reader.Header will not be populated and is left as zero-valued.

func OpenReader 

func OpenReader(path string, opts ...ReadOption) (*Reader, error)

OpenReader is a wrapper for NewReader which opens the file at path.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the underlying reader if it was opened by OpenReader.

func (*Reader) DataReader 

func (r *Reader) DataReader() SectionReader

DataReader provides a reader containing the data payload in CARv1 format.

func (*Reader) IndexReader 

func (r *Reader) IndexReader() io.Reader

IndexReader provides an io.Reader containing the index for the data payload if the index is present. Otherwise, returns nil. Note, this function will always return nil if the backing payload represents a CARv1.

func (*Reader) Roots 

func (r *Reader) Roots() ([]cid.Cid, error)

Roots returns the root CIDs. The root CIDs are extracted lazily from the data payload header.

type SectionReader 

type SectionReader interface {
	io.Reader
	io.Seeker
	io.ReaderAt
}

SectionReader implements both io.ReadSeeker and io.ReaderAt. It is the interface version of io.SectionReader, but note that the implementation is not guaranteed to be an io.SectionReader.

type WriteOption 

type WriteOption func(*WriteOptions)

WriteOption describes an option which affects behavior when encoding CAR files.

func UseDataPadding 

func UseDataPadding(p uint64) WriteOption

UseDataPadding is a write option which sets the padding to be added between CARv2 header and its data payload on Finalize.

func UseIndexPadding 

func UseIndexPadding(p uint64) WriteOption

UseIndexPadding is a write option which sets the padding between data payload and its index on Finalize.

type WriteOptions 

type WriteOptions struct {
	DataPadding  uint64
	IndexPadding uint64

	BlockstoreAllowDuplicatePuts bool
}

WriteOptions holds the configured options after applying a number of WriteOption funcs.

This type should not be used directly by end users; it's only exposed as a side effect of WriteOption.

Source Files

View all Source files

Directories

Path Synopsis
package blockstore implements the DMS3 blockstore interface backed by a CAR file.
 package blockstore implements the DMS3 blockstore interface backed by a CAR file.
package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset.
 package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset.
internal
carv1
Forked from CARv1 to avoid dependency to ld-prime 0.9.0 due to outstanding upgrades in filecoin.
 Forked from CARv1 to avoid dependency to ld-prime 0.9.0 due to outstanding upgrades in filecoin.
carv1/util
io

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.