meta

Documentation

Overview

Package meta implements the XFLATE meta encoding scheme.

The XFLATE meta encoding is a method of encoding arbitrary data into one or more RFC 1951 compliant DEFLATE blocks. This encoding has the special property that when the blocks are decoded by a RFC 1951 compliant decompressor, they produce absolutely no output. However, when decoded with the XFLATE meta decoder, it losslessly produces the original input.

The meta encoding works by encoding arbitrary data into the Huffman tree definition of dynamic DEFLATE blocks. These blocks have an empty data section and produce no output. Due to the Huffman definition overhead, the encoded output is usually larger than the input. However, for most input datasets, this encoding scheme is able to achieve an efficiency of at least 50%.

Index

• Constants
• func ReverseSearch(data []byte) int
• type FinalMode
• type Reader
  • func NewReader(rd io.Reader) *Reader
  • func (mr *Reader) Close() error
  • func (mr *Reader) Read(buf []byte) (int, error)
  • func (mr *Reader) Reset(rd io.Reader)
• type Writer
  • func NewWriter(wr io.Writer) *Writer
  • func (mw *Writer) Close() error
  • func (mw *Writer) Reset(wr io.Writer)
  • func (mw *Writer) Write(buf []byte) (int, error)

Constants

const (
	// MinRawBytes and MaxRawBytes are the theoretical minimum and maximum
	// number of bytes a block can encode.
	MinRawBytes = 0
	MaxRawBytes = 31

	// MinEncBytes and MaxEncBytes are the theoretical minimum and maximum
	// number of bytes an encoded block will occupy.
	MinEncBytes = 12
	MaxEncBytes = 64

	// EnsureRawBytes is the maximum number of bytes that a single block is
	// ensured to encode (computed using brute force).
	EnsureRawBytes = 22
)

These are some constants regarding the theoretical and practical limits for the meta encoding of a single block.

Functions

func ReverseSearch

func ReverseSearch(data []byte) int

ReverseSearch searches for a meta header in reverse. This returns the last index where the header was found. If not found, it returns -1.

Types

type FinalMode

type FinalMode int

FinalMode controls or indicates which final bits are set in the last block in the meta stream. In the meta encoding, there are 2 final bits:

Stream: This is the final bit from DEFLATE (as defined in RFC 1951).
and indicates that the entire compression stream has come to an end.
This bit indicate absolute termination of the stream.

Meta: This final bit indicates that the current sequence of meta blocks has
terminated. The decoded data from those blocks form a meta substream.
This bit is used as a form of message framing for the meta encoding format.

It invalid for the stream final bit to be set, while the meta final bit is not set. All other combinations are legal.

const (
	// FinalNil has neither the stream nor meta final bits set.
	FinalNil FinalMode = iota

	// FinalMeta has the final bit set, but not stream final bit.
	FinalMeta

	// FinalStream has both the meta and stream final bits set.
	FinalStream
)

type Reader

type Reader struct {
	InputOffset  int64 // Total number of bytes read from underlying io.Reader
	OutputOffset int64 // Total number of bytes emitted from Read
	NumBlocks    int64 // Number of blocks decoded

	// FinalMode indicates which final bits (if any) were set.
	// This will be valid after a call to Close or upon hitting io.EOF.
	FinalMode FinalMode
	// contains filtered or unexported fields
}

A Reader is an io.Reader that can read XFLATE's meta encoding. The zero value of Reader is valid once Reset is called.

func NewReader

func NewReader(rd io.Reader) *Reader

NewReader creates a new Reader reading from the given reader. If rd does not also implement compress.ByteReader or compress.BufferedReader, then the decoder may read more data than necessary from rd.

func (*Reader) Close

func (mr *Reader) Close() error

Close ends the meta stream. The FinalMode encountered becomes valid after calling Close.

func (*Reader) Read

func (mr *Reader) Read(buf []byte) (int, error)

Read reads the decoded meta data from the underlying io.Reader. This returns io.EOF either when a meta block with final bits set is found or when io.EOF is hit in the underlying reader.

func (*Reader) Reset

func (mr *Reader) Reset(rd io.Reader)

Reset discards the Reader's state and makes it equivalent to the result of a call to NewReader, but reading from rd instead.

This is used to reduce memory allocations.

type Writer

type Writer struct {
	InputOffset  int64 // Total number of bytes issued to Write
	OutputOffset int64 // Total number of bytes written to underlying io.Writer
	NumBlocks    int64 // Number of blocks encoded

	// FinalMode determines which final bits (if any) to set.
	// This must be set prior to a call to Close.
	FinalMode FinalMode
	// contains filtered or unexported fields
}

A Writer is an io.Writer that can write XFLATE's meta encoding. The zero value of Writer is valid once Reset is called.

func NewWriter

func NewWriter(wr io.Writer) *Writer

NewWriter creates a new Writer writing to the given writer. It is the caller's responsibility to call Close to complete the meta stream.

func (*Writer) Close

func (mw *Writer) Close() error

Close ends the meta stream and flushes all buffered data. The desired FinalMode must be set prior to calling Close.

func (*Writer) Reset

func (mw *Writer) Reset(wr io.Writer)

Reset discards the Writer's state and makes it equivalent to the result of a call to NewWriter, but writes to wr instead.

This is used to reduce memory allocations.

func (*Writer) Write

func (mw *Writer) Write(buf []byte) (int, error)

Write writes the encoded form of buf to the underlying io.Writer. The Writer may buffer the input in order to produce larger meta blocks.