huff0

Documentation

Overview

Package huff0 provides fast huffman encoding as used in zstd.

See README.md at https://github.com/zhangdapeng520/zdpgo_fasthttp/compress/tree/master/huff0 for details.

Index

• Constants
• Variables
• func Compress1X(in []byte, s *Scratch) (out []byte, reUsed bool, err error)
• func Compress4X(in []byte, s *Scratch) (out []byte, reUsed bool, err error)
• func EstimateSizes(in []byte, s *Scratch) (tableSz, dataSz, reuseSz int, err error)
• type Decoder
  • func (d *Decoder) Decompress1X(dst, src []byte) ([]byte, error)
  • func (d *Decoder) Decompress4X(dst, src []byte) ([]byte, error)
• type ReusePolicy
• type Scratch
  • func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error)
  • func (s *Scratch) Decoder() *Decoder
  • func (s *Scratch) Decompress1X(in []byte) (out []byte, err error)
  • func (s *Scratch) Decompress4X(in []byte, dstSize int) (out []byte, err error)
  • func (s *Scratch) TransferCTable(src *Scratch)

Constants

const (

	// BlockSizeMax is maximum input size for a single block uncompressed.
	BlockSizeMax = 1<<18 - 1
)

Variables

var (
	// ErrIncompressible is returned when input is judged to be too hard to compress.
	ErrIncompressible = errors.New("input is not compressible")

	// ErrUseRLE is returned from the compressor when the input is a single byte value repeated.
	ErrUseRLE = errors.New("input is single value repeated")

	// ErrTooBig is return if input is too large for a single block.
	ErrTooBig = errors.New("input too big")

	// ErrMaxDecodedSizeExceeded is return if input is too large for a single block.
	ErrMaxDecodedSizeExceeded = errors.New("maximum output size exceeded")
)

Functions

func Compress1X

func Compress1X(in []byte, s *Scratch) (out []byte, reUsed bool, err error)

Compress1X will compress the input. The output can be decoded using Decompress1X. Supply a Scratch object. The scratch object contains state about re-use, So when sharing across independent encodes, be sure to set the re-use policy.

func Compress4X

func Compress4X(in []byte, s *Scratch) (out []byte, reUsed bool, err error)

Compress4X will compress the input. The input is split into 4 independent blocks and compressed similar to Compress1X. The output can be decoded using Decompress4X. Supply a Scratch object. The scratch object contains state about re-use, So when sharing across independent encodes, be sure to set the re-use policy.

func EstimateSizes

func EstimateSizes(in []byte, s *Scratch) (tableSz, dataSz, reuseSz int, err error)

EstimateSizes will estimate the data sizes

Types

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder provides stateless decoding.

func (*Decoder) Decompress1X

func (d *Decoder) Decompress1X(dst, src []byte) ([]byte, error)

Decompress1X will decompress a 1X encoded stream. The cap of the output buffer will be the maximum decompressed size. The length of the supplied input must match the end of a block exactly.

func (*Decoder) Decompress4X

func (d *Decoder) Decompress4X(dst, src []byte) ([]byte, error)

Decompress4X will decompress a 4X encoded stream. The length of the supplied input must match the end of a block exactly. The *capacity* of the dst slice must match the destination size of the uncompressed data exactly.

type ReusePolicy

type ReusePolicy uint8

const (
	// ReusePolicyAllow will allow reuse if it produces smaller output.
	ReusePolicyAllow ReusePolicy = iota

	// ReusePolicyPrefer will re-use aggressively if possible.
	// This will not check if a new table will produce smaller output,
	// except if the current table is impossible to use or
	// compressed output is bigger than input.
	ReusePolicyPrefer

	// ReusePolicyNone will disable re-use of tables.
	// This is slightly faster than ReusePolicyAllow but may produce larger output.
	ReusePolicyNone

	// ReusePolicyMust must allow reuse and produce smaller output.
	ReusePolicyMust
)

type Scratch

type Scratch struct {

	// Out is output buffer.
	// If the scratch is re-used before the caller is done processing the output,
	// set this field to nil.
	// Otherwise the output buffer will be re-used for next Compression/Decompression step
	// and allocation will be avoided.
	Out []byte

	// OutTable will contain the table data only, if a new table has been generated.
	// Slice of the returned data.
	OutTable []byte

	// OutData will contain the compressed data.
	// Slice of the returned data.
	OutData []byte

	// MaxDecodedSize will set the maximum allowed output size.
	// This value will automatically be set to BlockSizeMax if not set.
	// Decoders will return ErrMaxDecodedSizeExceeded is this limit is exceeded.
	MaxDecodedSize int

	// MaxSymbolValue will override the maximum symbol value of the next block.
	MaxSymbolValue uint8

	// TableLog will attempt to override the tablelog for the next block.
	// Must be <= 11 and >= 5.
	TableLog uint8

	// Reuse will specify the reuse policy
	Reuse ReusePolicy

	// WantLogLess allows to specify a log 2 reduction that should at least be achieved,
	// otherwise the block will be returned as incompressible.
	// The reduction should then at least be (input size >> WantLogLess)
	// If WantLogLess == 0 any improvement will do.
	WantLogLess uint8
	// contains filtered or unexported fields
}

func ReadTable

func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error)

ReadTable will read a table from the input. The size of the input may be larger than the table definition. Any content remaining after the table definition will be returned. If no Scratch is provided a new one is allocated. The returned Scratch can be used for encoding or decoding input using this table.

func (*Scratch) Decoder

func (s *Scratch) Decoder() *Decoder

Decoder will return a stateless decoder that can be used by multiple decompressors concurrently. Before this is called, the table must be initialized with ReadTable. The Decoder is still linked to the scratch buffer so that cannot be reused. However, it is safe to discard the scratch.

func (*Scratch) Decompress1X

func (s *Scratch) Decompress1X(in []byte) (out []byte, err error)

Decompress1X will decompress a 1X encoded stream. The length of the supplied input must match the end of a block exactly. Before this is called, the table must be initialized with ReadTable unless the encoder re-used the table. deprecated: Use the stateless Decoder() to get a concurrent version.

func (*Scratch) Decompress4X

func (s *Scratch) Decompress4X(in []byte, dstSize int) (out []byte, err error)

Decompress4X will decompress a 4X encoded stream. Before this is called, the table must be initialized with ReadTable unless the encoder re-used the table. The length of the supplied input must match the end of a block exactly. The destination size of the uncompressed data must be known and provided. deprecated: Use the stateless Decoder() to get a concurrent version.

func (*Scratch) TransferCTable

func (s *Scratch) TransferCTable(src *Scratch)

TransferCTable will transfer the previously used compression table.