ultracdc

README

go-ultracdc

go-ultracdc is a Golang package implementing the ultraCDC chunking algorithm.

    chunker, err := fastcdc.NewChunker(rd, nil)
    if err != nil {
        log.Fatal(err)
    }

    offset := 0
    for {
        chunk, err := chunker.Next()
        if err != nil && err != io.EOF {
            log.Fatal(err)
        }

        chunkLen := len(chunk)
        fmt.Println(offset, chunkLen)

        if err == io.EOF {
            // no more chunks to read
            break
        }
        offset += chunkLen
    }

Reference

• Zhou, Wang, Xia, Zhang "UltraCDC:A Fast and Stable Content-Defined Chunking Algorithm for Deduplication-based Backup Storage Systems" (2022)

Documentation

Index

• type Chunker
  • func NewChunker(reader io.Reader, options *ChunkerOpts) (*Chunker, error)
  • func (chunker *Chunker) Next() ([]byte, error)
• type ChunkerOpts
  • func NewChunkerOptions() *ChunkerOpts

Types

type Chunker

type Chunker struct {
	NormalSize uint32
	MinSize    uint32
	MaxSize    uint32
	// contains filtered or unexported fields
}

func NewChunker

func NewChunker(reader io.Reader, options *ChunkerOpts) (*Chunker, error)

func (*Chunker) Next

func (chunker *Chunker) Next() ([]byte, error)

type ChunkerOpts

type ChunkerOpts struct {
	MinSize uint32
	MaxSize uint32
}

func NewChunkerOptions

func NewChunkerOptions() *ChunkerOpts