Documentation ¶
Overview ¶
Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.
Index ¶
- Constants
- Variables
- func Chan(s Splitter) (<-chan []byte, <-chan error)
- func IsReedSolomon(chunker string) bool
- func NewReedSolomonSplitter(r io.Reader, numData, numParity, size uint64) (*reedSolomonSplitter, error)
- type MetaSplitter
- type MultiSplitter
- type Rabin
- type RsMetaMap
- type Splitter
- type SplitterGen
Constants ¶
const ( PrefixForDefault = "default" PrefixForSize = "size-" PrefixForRabin = "rabin" PrefixForReedSolomon = "reed-solomon" )
const ( DefaultReedSolomonDataShards = 10 DefaultReedSolomonParityShards = 20 DefaultReedSolomonShardSize = DefaultBlockSize )
const DefaultBlockSize int64 = 1024 * 256
DefaultBlockSize is the chunk size that splitters produce (or aim to).
Variables ¶
var ( ErrRabinMin = errors.New("rabin min must be greater than 16") ErrSize = errors.New("chunker size must be greater than 0") )
var IpfsRabinPoly = chunker.Pol(17437180132763653)
IpfsRabinPoly is the irreducible polynomial of degree 53 used by for Rabin.
Functions ¶
func Chan ¶
Chan returns a channel that receives each of the chunks produced by a splitter, along with another one for errors.
func IsReedSolomon ¶ added in v0.2.6
Types ¶
type MetaSplitter ¶ added in v0.2.2
type MetaSplitter struct {
// contains filtered or unexported fields
}
func (*MetaSplitter) ChunkSize ¶ added in v0.2.2
func (ms *MetaSplitter) ChunkSize() uint64
ChunkSize returns the chunk size of this Splitter.
func (*MetaSplitter) MetaData ¶ added in v0.2.3
func (ms *MetaSplitter) MetaData() interface{}
MetaData returns metadata object from this chunker (none).
func (*MetaSplitter) NextBytes ¶ added in v0.2.2
func (ms *MetaSplitter) NextBytes() ([]byte, error)
NextBytes produces a new chunk.
func (*MetaSplitter) Reader ¶ added in v0.2.2
func (ms *MetaSplitter) Reader() io.Reader
Reader returns the io.Reader associated to this Splitter.
func (*MetaSplitter) SetIsDir ¶ added in v0.2.6
func (rss *MetaSplitter) SetIsDir(v bool)
type MultiSplitter ¶ added in v0.2.0
A MultiSplitter encapsulates multiple splitters useful for concurrent reading of chunks and also specialized dag building schemas. Each MultiSplitter also provides Splitter-compatible interface to read sequentially (the Splitter-default way).
type Rabin ¶
type Rabin struct {
// contains filtered or unexported fields
}
Rabin implements the Splitter interface and splits content with Rabin fingerprints.
func NewRabinMinMax ¶
NewRabinMinMax returns a new Rabin splitter which uses the given min, average and max block sizes.
func (*Rabin) MetaData ¶ added in v0.2.3
func (r *Rabin) MetaData() interface{}
MetaData returns metadata object from this chunker (none).
type RsMetaMap ¶ added in v0.2.3
func GetRsMetaMapFromString ¶ added in v0.2.6
type Splitter ¶
type Splitter interface { Reader() io.Reader NextBytes() ([]byte, error) ChunkSize() uint64 MetaData() interface{} SetIsDir(bool) }
A Splitter reads bytes from a Reader and creates "chunks" (byte slices) that can be used to build DAG nodes.
func DefaultSplitter ¶
DefaultSplitter returns a SizeSplitter with the DefaultBlockSize.
func FromString ¶
FromString returns a Splitter depending on the given string: it supports "default" (""), "size-{size}", "rabin", "rabin-{blocksize}", "rabin-{min}-{avg}-{max}", "reed-solomon", and "reed-solomon-{#data}-{#parity}-{size}".
type SplitterGen ¶
SplitterGen is a splitter generator, given a reader.
func MetaSplitterGen ¶ added in v0.2.4
func MetaSplitterGen(size int64) SplitterGen
func SizeSplitterGen ¶
func SizeSplitterGen(size int64) SplitterGen
SizeSplitterGen returns a SplitterGen function which will create a splitter with the given size when called.