carbites

README

carbites

Chunking for CAR files. Split a single CAR into multiple CARs.

Install

go get github.com/alanshaw/go-carbites

Usage

Carbites supports 2 different strategies:

1. Simple (default) - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID. The first CAR output has roots in the header, subsequent CARs have an empty root CID bafkqaaa as recommended.
2. Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR has the same root CID, but contains a different portion of the DAG. Every CAR file has the same root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.

package main

import (
	"github.com/alanshaw/go-carbites"
	"github.com/ipld/go-car"
)

func main() {
    out := make(chan io.Reader)

    go func() {
        var i int
        for {
            select {
            case r := <-out:
                b, _ := ioutil.ReadAll(r)
                ioutil.WriteFile(fmt.Sprintf("chunk-%d.car", i), b, 0644)
                i++
            }
        }
    }()

    car, _ := car.NewCarReader(reader)
    targetSize := 1000 // 1kb chunks
    strategy := carbites.Simple // also carbites.TreeWalk
    err := carbites.Split(context.Background(), car, targetSize, strategy, out)
}

API

pkg.go.dev Reference

Contribute

Feel free to dive in! Open an issue or submit PRs.

License

Dual-licensed under MIT + Apache 2.0

Documentation

Index

• func Split(ctx context.Context, in io.Reader, targetSize int, s Strategy, ...) error
• func SplitSimple(ctx context.Context, in io.Reader, targetSize int, out chan io.Reader) error
• func SplitTreewalk(ctx context.Context, r io.Reader, targetSize int, out chan io.Reader) error
• func SplitTreewalkWithBlockReader(ctx context.Context, root cid.Cid, br CarBlockReader, targetSize int, ...) error
• type CarBlockReader
• type Strategy

Functions

func Split

func Split(ctx context.Context, in io.Reader, targetSize int, s Strategy, out chan io.Reader) error

Split a CAR file and create multiple smaller CAR files.

func SplitSimple

func SplitSimple(ctx context.Context, in io.Reader, targetSize int, out chan io.Reader) error

Split a CAR file and create multiple smaller CAR files using the "simple" strategy.

func SplitTreewalk

func SplitTreewalk(ctx context.Context, r io.Reader, targetSize int, out chan io.Reader) error

Split a CAR file and create multiple smaller CAR files using the "treewalk" strategy.

func SplitTreewalkWithBlockReader

func SplitTreewalkWithBlockReader(ctx context.Context, root cid.Cid, br CarBlockReader, targetSize int, out chan io.Reader) error

Split a CAR file and create multiple smaller CAR files using the "treewalk" strategy and passing in the root CID of the CAR and a block reader populated with the blocks from the CAR.

Types

type CarBlockReader

type CarBlockReader interface {
	Get(cid.Cid) (blocks.Block, error)
}

type Strategy

type Strategy int

Strategy describes how CAR files should be split.

const (
	// Simple is fast but naive, only the first CAR output has a root CID,
	// subsequent CARs have a placeholder "empty" CID.
	Simple Strategy = iota
	// TreeWalk walks the DAG to pack sub-graphs into each CAR file that is
	// output. Every CAR has the same root CID, but contains a different portion
	// of the DAG.
	TreeWalk
)