Documentation ¶
Overview ¶
Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.
Index ¶
Constants ¶
const ( DefaultReedSolomonDataShards = 10 DefaultReedSolomonParityShards = 20 DefaultReedSolomonShardSize = DefaultBlockSize )
const DefaultBlockSize int64 = 1024 * 256
DefaultBlockSize is the chunk size that splitters produce (or aim to).
Variables ¶
var ( ErrRabinMin = errors.New("rabin min must be greater than 16") ErrSize = errors.New("chunker size must be greater than 0") )
var IpfsRabinPoly = chunker.Pol(17437180132763653)
IpfsRabinPoly is the irreducible polynomial of degree 53 used by for Rabin.
Functions ¶
Types ¶
type MultiSplitter ¶ added in v0.2.0
A MultiSplitter encapsulates multiple splitters useful for concurrent reading of chunks and also specialized dag building schemas. Each MultiSplitter also provides Splitter-compatible interface to read sequentially (the Splitter-default way).
type Rabin ¶
type Rabin struct {
// contains filtered or unexported fields
}
Rabin implements the Splitter interface and splits content with Rabin fingerprints.
func NewRabinMinMax ¶
NewRabinMinMax returns a new Rabin splitter which uses the given min, average and max block sizes.
type Splitter ¶
A Splitter reads bytes from a Reader and creates "chunks" (byte slices) that can be used to build DAG nodes.
func DefaultSplitter ¶
DefaultSplitter returns a SizeSplitter with the DefaultBlockSize.
func FromString ¶
FromString returns a Splitter depending on the given string: it supports "default" (""), "size-{size}", "rabin", "rabin-{blocksize}", "rabin-{min}-{avg}-{max}", "reed-solomon", and "reed-solomon-{#data}-{#parity}-{size}".
type SplitterGen ¶
SplitterGen is a splitter generator, given a reader.
func SizeSplitterGen ¶
func SizeSplitterGen(size int64) SplitterGen
SizeSplitterGen returns a SplitterGen function which will create a splitter with the given size when called.