chunker

README

chunker

What does this project do?

This is a universal wrapper for application-level to use chunking in Go. The original chunker module is designed for restic backup program. To be honest, it is really hard to use it RIGHT AWAY. You need to take time to find it out how does it work and how to apply it into your code. It's not worth that.

So I make this. You can find the chunker module from here: restic/chunker.

What is the default setting then?

Current setting for chunking are as follows:

• minimum chunk size : 2KB
• maximum chunk size : 16KB
• average chunk size : 8KB

You can change this setting via changing bits:

• 2KB is 1 << 11
• 16KB is 1 << 14
• 8KB is 1 << 13

Change as you want. Corresponding code is:

NewWithBoundaries(rd io.Reader, minBits, maxBits, averageBits int);

If you doesn't care about the default setting, just use:

chnker := chunker.New(reader)

Usage/Example

You can use chunker as bellow:

package main

import (
    "fmt"
    chunk "github.com/Gabirel/chunker"
    "io"
    "os"
)

func main() {
    fmt.Println("Starting to open file...")

    filename := "test.txt"
    reader, err := os.Open(filename)
    if err != nil {
        fmt.Println("Read error: ", err)
        _ = reader.Close()
        return
    }

    chunker := chunk.New(reader)

    var totalLength uint = 0
    var averageLength uint = 0
    var chunkNum uint = 0

    for {
        chunkInfo, err := chunker.Chunking()
        if err == io.EOF {
            fmt.Println("\nEOF. Exiting...")
            fmt.Println("Total length = ", totalLength)
            fmt.Println("Chunk Number = ", chunkNum)
            if chunkNum != 0 {
                averageLength = totalLength / chunkNum
                fmt.Println("Average length = ", averageLength)
            }
            break
        }

        if chunkInfo == nil {
            panic("chunkInfo is nil!")
        }
        fmt.Printf("%d:\t%x\n", chunkInfo.Chunk.Length, chunkInfo.Digest)
        totalLength += chunkInfo.Chunk.Length
        chunkNum += 1
    }
}

Documentation

Index

• type ChunkInfo
• type Chunking
  • func New(rd io.Reader) *Chunking
  • func NewWithBoundaries(rd io.Reader, minBits, maxBits, averageBits int) *Chunking
  • func (c *Chunking) Chunking() (*ChunkInfo, error)

Types

type ChunkInfo

type ChunkInfo struct {
	Chunk  *chunker.Chunk
	Digest []byte
}

ChunkInfo stores one chunk information and its digest

type Chunking

type Chunking struct {
	// contains filtered or unexported fields
}

func New

func New(rd io.Reader) *Chunking

New returns a new Chunking min chunk size : 2KB max chunk size : 16KB average chunk size : 8KB

func NewWithBoundaries

func NewWithBoundaries(rd io.Reader, minBits, maxBits, averageBits int) *Chunking

NewWithBoundaries returns a new Chunking this function can set min, max and average chunk size

func (*Chunking) Chunking

func (c *Chunking) Chunking() (*ChunkInfo, error)