compressio

Documentation

Overview

Package compressio provides parallel compression and decompression, as well as optional SHA-256 hashing.

The stream format is defined as follows.

/------------------------------------------------------\ | chunk size (4-bytes) | +------------------------------------------------------+ | (optional) hash (32-bytes) | +------------------------------------------------------+ | compressed data size (4-bytes) | +------------------------------------------------------+ | compressed data | +------------------------------------------------------+ | (optional) hash (32-bytes) | +------------------------------------------------------+ | compressed data size (4-bytes) | +------------------------------------------------------+ | ...... | \------------------------------------------------------/

where each subsequent hash is calculated from the following items in order

compressed data
compressed data size
previous hash

so the stream integrity cannot be compromised by switching and mixing compressed chunks.

Index

• Variables
• type Reader
  • func NewReader(in io.Reader, key []byte) (*Reader, error)
  • func (r *Reader) Read(p []byte) (int, error)
  • func (r *Reader) ReadByte() (byte, error)
• type Writer
  • func NewWriter(out io.Writer, key []byte, chunkSize uint32, level int) (*Writer, error)
  • func (w *Writer) Close() error
  • func (w *Writer) Write(p []byte) (int, error)
  • func (w *Writer) WriteByte(b byte) error

Variables

var ErrHashMismatch = errors.New("hash mismatch")

ErrHashMismatch is returned if the hash does not match.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is a compressed reader.

func NewReader

func NewReader(in io.Reader, key []byte) (*Reader, error)

NewReader returns a new compressed reader. If key is non-nil, the data stream is assumed to contain expected hash values, which will be compared against hash values computed from the compressed bytes. See package comments for details.

func (*Reader) Read

func (r *Reader) Read(p []byte) (int, error)

Read implements io.Reader.Read.

func (*Reader) ReadByte

func (r *Reader) ReadByte() (byte, error)

ReadByte implements wire.Reader.ReadByte.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is a compressed writer.

func NewWriter

func NewWriter(out io.Writer, key []byte, chunkSize uint32, level int) (*Writer, error)

NewWriter returns a new compressed writer. If key is non-nil, hash values are generated and written out for compressed bytes. See package comments for details.

The recommended chunkSize is on the order of 1M. Extra memory may be buffered (in the form of read-ahead, or buffered writes), and is limited to O(chunkSize * [1+GOMAXPROCS]).

func (*Writer) Close

func (w *Writer) Close() error

Close implements io.Closer.Close.

func (*Writer) Write

func (w *Writer) Write(p []byte) (int, error)

Write implements io.Writer.Write.

func (*Writer) WriteByte

func (w *Writer) WriteByte(b byte) error

WriteByte implements wire.Writer.WriteByte.

Note that this implementation is necessary on the object itself, as an interface-based dispatch cannot tell whether the array backing the slice escapes, therefore the all bytes written will generate an escape.