car

README

go-car (go!)

Work with car (Content addressed ARchive) files!

This is a Golang implementation of the CAR specifications, both CARv1 and CARv2.

As a format, there are two major module versions:

• go-car/v2 is geared towards reading and writing CARv2 files, and also supports consuming CARv1 files and using CAR files as an IPFS blockstore.
• go-car, in the root directory, only supports reading and writing CARv1 files.

Most users should use v2, especially for new software, since the v2 API transparently supports both CAR formats.

Unstable

This package will be refactored and thinned out soon. You can find an implementation that supports indexes and go-ipld-prime there at ipld/go-car.

Usage / Installation

This repository provides a car binary that can be used for creating, extracting, and working with car files.

To install the latest version of car, run:

go install github.com/boxo/ipld/car/cmd/car@latest

More information about this binary is available in cmd/car

Features

CARv2 features:

• Generate index from an existing CARv1 file
• Wrap CARv1 files into a CARv2 with automatic index generation.
• Random-access to blocks in a CAR file given their CID via Read-Only blockstore API, with transparent support for both CARv1 and CARv2
• Write CARv2 files via Read-Write blockstore API, with support for appending blocks to an existing CARv2 file, and resumption from a partially written CARv2 files.
• Individual access to inner CARv1 data payload and index of a CARv2 file via the Reader API.

API Documentation

See docs on pkg.go.dev.

Examples

Here is a shortlist of other examples from the documentation

• Wrap an existing CARv1 file into an indexed CARv2 file
• Open read-only blockstore from a CAR file
• Open read-write blockstore from a CAR file
• Read the index from an existing CARv2 file
• Extract the index from a CARv2 file and store it as a separate file

Contribute

PRs are welcome!

When editing the Readme, please conform to the standard-readme specification.

License

Apache-2.0/MIT © Protocol Labs

Documentation

Index

• func DefaultWalkFunc(nd format.Node) ([]*format.Link, error)
• func HeaderSize(h *CarHeader) (uint64, error)
• func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, ...) error
• func WriteCarWithWalker(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, ...) error
• func WriteHeader(h *CarHeader, w io.Writer) error
• type Block
• type CarHeader
  • func LoadCar(ctx context.Context, s Store, r io.Reader) (*CarHeader, error)
  • func ReadHeader(br *bufio.Reader) (*CarHeader, error)
• type CarReader
  • func NewCarReader(r io.Reader) (*CarReader, error)
  • func (cr *CarReader) Next() (blocks.Block, error)
• type Dag
• type OnCarHeaderFunc
• type OnNewCarBlockFunc
• type Option
  • func MaxTraversalLinks(MaxTraversalLinks uint64) Option
  • func TraverseLinksOnlyOnce() Option
• type ReadStore
• type SelectiveCar
  • func NewSelectiveCar(ctx context.Context, store ReadStore, dags []Dag, opts ...Option) SelectiveCar
  • func (sc SelectiveCar) Prepare(userOnNewCarBlocks ...OnNewCarBlockFunc) (SelectiveCarPrepared, error)
  • func (sc SelectiveCar) Write(w io.Writer, userOnNewCarBlocks ...OnNewCarBlockFunc) error
• type SelectiveCarPrepared
  • func (sc SelectiveCarPrepared) Cids() []cid.Cid
  • func (sc SelectiveCarPrepared) Dump(ctx context.Context, w io.Writer) error
  • func (sc SelectiveCarPrepared) Header() CarHeader
  • func (sc SelectiveCarPrepared) Size() uint64
• type Store
• type WalkFunc

Functions

func DefaultWalkFunc

func DefaultWalkFunc(nd format.Node) ([]*format.Link, error)

func HeaderSize

func HeaderSize(h *CarHeader) (uint64, error)

func WriteCar

func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, options ...merkledag.WalkOption) error

func WriteCarWithWalker

func WriteCarWithWalker(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, walk WalkFunc, options ...merkledag.WalkOption) error

func WriteHeader

func WriteHeader(h *CarHeader, w io.Writer) error

Types

type Block

type Block struct {
	BlockCID cid.Cid
	Data     []byte
	Offset   uint64
	Size     uint64
}

Block is all information and metadata about a block that is part of a car file

type CarHeader

type CarHeader struct {
	Roots   []cid.Cid
	Version uint64
}

func LoadCar

func LoadCar(ctx context.Context, s Store, r io.Reader) (*CarHeader, error)

func ReadHeader

func ReadHeader(br *bufio.Reader) (*CarHeader, error)

type CarReader

type CarReader struct {
	Header *CarHeader
	// contains filtered or unexported fields
}

func NewCarReader

func NewCarReader(r io.Reader) (*CarReader, error)

func (*CarReader) Next

func (cr *CarReader) Next() (blocks.Block, error)

type Dag

type Dag struct {
	Root     cid.Cid
	Selector ipld.Node
}

Dag is a root/selector combo to put into a car

type OnCarHeaderFunc

type OnCarHeaderFunc func(CarHeader) error

OnCarHeaderFunc is called during traversal when the header is created

type OnNewCarBlockFunc

type OnNewCarBlockFunc func(Block) error

OnNewCarBlockFunc is called during traveral when a new unique block is encountered

type Option

type Option func(*options)

Option describes an option which affects behavior when interacting with the interface.

func MaxTraversalLinks

func MaxTraversalLinks(MaxTraversalLinks uint64) Option

MaxTraversalLinks changes the allowed number of links a selector traversal can execute before failing.

Note that setting this option may cause an error to be returned from selector execution when building a SelectiveCar.

func TraverseLinksOnlyOnce

func TraverseLinksOnlyOnce() Option

TraverseLinksOnlyOnce prevents the traversal engine from repeatedly visiting the same links more than once.

This can be an efficient strategy for an exhaustive selector where it's known that repeat visits won't impact the completeness of execution. However it should be used with caution with most other selectors as repeat visits of links for different reasons during selector execution can be valid and necessary to perform full traversal.

type ReadStore

type ReadStore interface {
	Get(context.Context, cid.Cid) (blocks.Block, error)
}

type SelectiveCar

type SelectiveCar struct {
	// contains filtered or unexported fields
}

SelectiveCar is a car file based on root + selector combos instead of just a single root and complete dag walk

func NewSelectiveCar

func NewSelectiveCar(ctx context.Context, store ReadStore, dags []Dag, opts ...Option) SelectiveCar

NewSelectiveCar creates a new SelectiveCar for the given car file based a block store and set of root+selector pairs

func (SelectiveCar) Prepare

func (sc SelectiveCar) Prepare(userOnNewCarBlocks ...OnNewCarBlockFunc) (SelectiveCarPrepared, error)

Prepare traverse a car file and collects data on what is about to be written, but does not actually write the file

func (SelectiveCar) Write

func (sc SelectiveCar) Write(w io.Writer, userOnNewCarBlocks ...OnNewCarBlockFunc) error

type SelectiveCarPrepared

type SelectiveCarPrepared struct {
	SelectiveCar
	// contains filtered or unexported fields
}

SelectiveCarPrepared is a SelectiveCar that has already been traversed, such that it can be written quicker with Dump. It also contains metadata already collection about the Car file like size and number of blocks that go into it

func (SelectiveCarPrepared) Cids

func (sc SelectiveCarPrepared) Cids() []cid.Cid

Cids returns the list of unique block cids that will be written to the car file

func (SelectiveCarPrepared) Dump

func (sc SelectiveCarPrepared) Dump(ctx context.Context, w io.Writer) error

Dump writes the car file as quickly as possible based on information already collected

func (SelectiveCarPrepared) Header

func (sc SelectiveCarPrepared) Header() CarHeader

Header returns the header for the car file that will be written

func (SelectiveCarPrepared) Size

func (sc SelectiveCarPrepared) Size() uint64

Size returns the total size in bytes of the car file that will be written

type Store

type Store interface {
	Put(context.Context, blocks.Block) error
}

type WalkFunc

type WalkFunc func(format.Node) ([]*format.Link, error)