chunker

Documentation

Overview

Package chunker provides a way to chunk an input into uploadable-size byte slices.

Index

• Constants
• Variables
• type Chunk
• type Chunker
  • func New(ue *uploadinfo.Entry, compressed bool, chunkSize int) (*Chunker, error)
  • func (c *Chunker) ChunkSize() int
  • func (c *Chunker) FullData() ([]byte, error)
  • func (c *Chunker) HasNext() bool
  • func (c *Chunker) Next() (*Chunk, error)
  • func (c *Chunker) Offset() int64
  • func (c *Chunker) Reset() error
  • func (c *Chunker) String() string

Constants

const DefaultChunkSize = 1024 * 1024

DefaultChunkSize is the default chunk size for ByteStream.Write RPCs.

Variables

var ErrEOF = errors.New("ErrEOF")

ErrEOF is returned when Next is called when HasNext is false.

var IOBufferSize = 10 * 1024 * 1024

IOBufferSize regulates how many bytes at a time the Chunker will read from a file source.

Types

type Chunk

type Chunk struct {
	Offset int64
	Data   []byte
}

Chunk is a piece of a byte[] blob suitable for being uploaded.

type Chunker

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker can be used to chunk an input into uploadable-size byte slices. A single Chunker is NOT thread-safe; it should be used by a single uploader thread.

func New

func New(ue *uploadinfo.Entry, compressed bool, chunkSize int) (*Chunker, error)

New creates a new chunker from an uploadinfo.Entry. If compressed, the data will of the Entry will be compressed on the fly.

func (*Chunker) ChunkSize

func (c *Chunker) ChunkSize() int

ChunkSize returns the maximum size of each chunk.

func (*Chunker) FullData

func (c *Chunker) FullData() ([]byte, error)

FullData returns the overall (non-chunked) underlying data. The Chunker is Reset. It is supposed to be used for batch uploading small inputs.

func (*Chunker) HasNext

func (c *Chunker) HasNext() bool

HasNext returns whether a subsequent call to Next will return a valid chunk. Always true for a newly created Chunker.

func (*Chunker) Next

func (c *Chunker) Next() (*Chunk, error)

Next returns the next chunk of data or error. ErrEOF is returned if and only if HasNext is false. Chunk.Data will be empty if and only if the full underlying data is empty (in which case it will be the only chunk returned). Chunk.Digest will only be filled for the first chunk.

func (*Chunker) Offset

func (c *Chunker) Offset() int64

Offset returns the current Chunker offset.

func (*Chunker) Reset

func (c *Chunker) Reset() error

Reset the Chunker state to when it was newly constructed. Useful for upload retries. TODO(olaola): implement Seek(offset) when we have resumable uploads.

func (*Chunker) String

func (c *Chunker) String() string

String returns an identifiable representation of the Chunker.