carbites

README

carbites

Chunking for CAR files. Split a single CAR into multiple CARs.

Install

go get github.com/alanshaw/go-carbites

Usage

Carbites supports 2 different strategies:

1. Simple (default) - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID.
2. Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR has the same root CID, but contains a different portion of the DAG.

Simple

The first CAR output has roots in the header, subsequent CARs have an empty root CID bafkqaaa as recommended.

Treewalk

Every CAR file has the same root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.

Example

package main

import (
	"github.com/alanshaw/go-carbites"
	"github.com/ipld/go-car"
)

func main() {
    out := make(chan io.Reader)

    go func() {
        var i int
        for {
            select {
            case r := <-out:
                b, _ := ioutil.ReadAll(r)
                ioutil.WriteFile(fmt.Sprintf("chunk-%d.car", i), b, 0644)
                i++
            }
        }
    }()

    car, _ := car.NewCarReader(reader)
    targetSize := 1000 // 1kb chunks
    strategy := carbites.Simple // also carbites.TreeWalk
    err := carbites.Split(context.Background(), car, targetSize, strategy, out)
}

API

pkg.go.dev Reference

Contribute

Feel free to dive in! Open an issue or submit PRs.

License

Dual-licensed under MIT + Apache 2.0

Documentation

Index

• func Split(ctx context.Context, r *car.CarReader, targetSize int, s Strategy, ...) error
• type Strategy

Functions

func Split

func Split(ctx context.Context, r *car.CarReader, targetSize int, s Strategy, out chan io.Reader) error

Split a CAR file and create multiple smaller CAR files.

Types

type Strategy

type Strategy int

Strategy describes how CAR files should be split.

const (
	// Simple is fast but naive, only the first CAR output has a root CID,
	// subsequent CARs have a placeholder "empty" CID.
	Simple Strategy = iota
	// TreeWalk walks the DAG to pack sub-graphs into each CAR file that is
	// output. Every CAR has the same root CID, but contains a different portion
	// of the DAG.
	TreeWalk
)