gozlib

README

gozlib

gozlib is a GC-friendly compression package with a low memory allocation footprint. It provides various interfaces into the zlib library, using cgo for interoperability. See https://github.com/madler/zlib for details about zlib.

Requirements

Using this package requires cgo and a gnu compiler (clang or gcc), as well as the development version of zlib installed. By default, it expect the zlib header and so files to be in the standard include and library path. If not, you can override it by setting the appropriate paths in the environment variables CGO_CFLAGS and CGO_LDFLAGS.

Implementation and usage

Internally gozlib utilizes dxpool as an off-heap memory pool to maximize memory usage. At this moment, allocated memory is never returned to the system so gozlib is best used when gzip operations are frequent and constant.

This pool is also available for use in the Go code as a way to allocate and reuse byte slices. See NativeSlicePool for details.

Compression and uncompression components

gozlib supports 3 different mechanisms for compressing and uncompressing data, each ideal to different use cases.

1. Single step, in memory using GoGZipCompressBuffer/GoUncompressBuffer
2. Event based with GoGZipCompressStream/GoUncompressStream
3. Stream based, implementing io.Reader/io.Writer created through NewGoZLibCompressor and NewGoZLibUncompressor. The returned object can be used as a drop in replacement to the standard library gzip implementation (or anything compatible with the io interfaces).

Single step and event based possible through stateless functions while the stream based option keeps states through the returned object.

Like the standard library gzip implementation, it's possible to flush and reset gozlib's compressor and uncompressor so that they can be pooled and reused.

See the documentation and test files for usage examples and details.

Remember that the stream based, stateful use of gozlib require Close() to be invoked to avoid memory leaks.

Performance

While gozlib is designed to be GC friendly in that it keeps heap allocations to a minimum, managed and unmanaged.

For memory intensive applications, like web services that have to support http compression, an overall performance gain can be obtained just by reducing the frequency and duration of GC runs and pauses.

It's important to note that there is a penalty calling cgo and while the underlying zlib implementation can be faster, benchmarking and carefully measuring performance is necessary to decided on the best approach for each use case.

General case benchmarks show that gozlib performs better in terms of speed and memory utilization. See results below.

HTTP request compression benchmark results

gozlib
  Requests/sec:	259.1624

stdlib gzip
  Requests/sec:	139.3972

Response time distribution

GC Pause time, as described in the runtime documentation

PauseTotalNs is the cumulative nanoseconds in GC stop-the-world pauses since the program started. During a stop-the-world pause, all goroutines are paused and only the garbage collector can run.

Cumulative number of GC runs

Documentation

Overview

Using this package requires cgo and a gnu compiler (clang or gcc), as well as the development version of zlib installed By default, it expect the zlib header and so files to be in the standard include and library path. If not, you can override it by setting the appropriate paths in the environment variables CGO_CFLAGS and CGO_LDFLAGS Internally gozlib utilizes an off-heap memory pool to maximize memory usage. At this moment, allocated memory is never returned to the system so gozlib is best used when gzip operations are frequent and constant. This pool is also available for use in the Go code as a way to allocate and reuse byte slices. See NativeSlicePool for details

Index

• Variables
• func Flush(compressor io.WriteCloser) error
• func GoGZipCompressBuffer(level CompressionLevel, input []byte, output []byte) (uint64, error)
• func GoGZipCompressStream(level CompressionLevel, inputBufferSize uint32, outputBufferSize uint32, ...) (uint64, error)
• func GoStreamDataInputHandler(ptr unsafe.Pointer, buffer unsafe.Pointer, bufferLength uint32) uint32
• func GoStreamDataOutputHandler(ptr unsafe.Pointer, buffer unsafe.Pointer, bufferLength uint32) uint32
• func GoUncompressBuffer(input []byte, output []byte) (uint64, error)
• func GoUncompressStream(inputBufferSize uint32, outputBufferSize uint32, ...) (uint64, error)
• func NewGoGZipCompressor(output io.Writer, level CompressionLevel, bufferSize uint32) (io.WriteCloser, error)
• func NewGoZLibUncompressor(input io.Reader, bufferSize uint32) (io.ReadCloser, error)
• func ResetCompressor(output io.Writer, compressor io.WriteCloser)
• func ResetUncompressor(input io.Reader, uncompressor io.ReadCloser)
• type CompressionLevel
• type DataStreamEventHandler
• type NativeSlicePool
  • func NewNativeSlicePool() *NativeSlicePool
  • func (nsp *NativeSlicePool) Acquire(size int) []byte
  • func (nsp *NativeSlicePool) Free()
  • func (nsp *NativeSlicePool) Return(slice []byte)
• type TransformMode

Variables

var (
	// transformer
	TransformerUncompressionError  = errors.New("error uncompressing data")
	TransformerInitializationError = errors.New("error initializing transformer")
	TransformerCompressionError    = errors.New("error compressing data")

	// streaming
	StreamCompressError   = errors.New("error streaming compressed data")
	StreamUncompressError = errors.New("error streaming uncompressed data")

	// buffer to buffer
	OutputBufferTooSmallError = errors.New("output buffer too small")
	BufferCompressError       = errors.New("error compressing buffer")
	BufferUncompressError     = errors.New("error uncompressing buffer")
)

Functions

func Flush

func Flush(compressor io.WriteCloser) error

Flush is a helper function to flush a compressor given an interface

func GoGZipCompressBuffer

func GoGZipCompressBuffer(level CompressionLevel, input []byte, output []byte) (uint64, error)

GoGZipCompressBuffer compresses data in gzip format, reading from input and writing to a pre allocated output buffer. If the output is too small to contain the compressed data, an error is returned

func GoGZipCompressStream

func GoGZipCompressStream(level CompressionLevel, inputBufferSize uint32, outputBufferSize uint32, inputReader DataStreamEventHandler, outputWriter DataStreamEventHandler) (uint64, error)

GoGZipCompressStream compresses a stream of data The compression level can be CompressionLevelBestCompression or CompressionLevelBestSpeed `inputReader` is a function used to read uncompressed data `outputWriter` is a function that takes the compressed data `inputBufferSize` and `outputBufferSize` are the sizes of the internal work buffers. For best performance, use large enough power of 2 sizes The function returns the number of bytes written to the output stream and an error, if any.

func GoStreamDataInputHandler

func GoStreamDataInputHandler(ptr unsafe.Pointer, buffer unsafe.Pointer, bufferLength uint32) uint32

func GoStreamDataOutputHandler

func GoStreamDataOutputHandler(ptr unsafe.Pointer, buffer unsafe.Pointer, bufferLength uint32) uint32

func GoUncompressBuffer

func GoUncompressBuffer(input []byte, output []byte) (uint64, error)

GoUncompressBuffer uncompresses zipb or an input buffer writing to a pre allocated output if the output is too small to contain the compressed data, an error is returned

func GoUncompressStream

func GoUncompressStream(inputBufferSize uint32, outputBufferSize uint32, inputReader DataStreamEventHandler, outputWriter DataStreamEventHandler) (uint64, error)

GoUncompressStream uncompresses a stream of data in gzip or standard zlib format `inputReader` is a function used to read compressed data `outputWriter` is a function that takes the uncompressed data `inputBufferSize` and `outputBufferSize` are the sizes of the internal work buffers. For best performance, use large enough power of 2 sizes The function returns the number of bytes written to the output stream and an error, if any.

func NewGoGZipCompressor

func NewGoGZipCompressor(output io.Writer, level CompressionLevel, bufferSize uint32) (io.WriteCloser, error)

NewGoGZipCompressor creates a new gzip compressor The compressor writes compressed data to the provided output Writer. The level parameter specifies the compression level. It can be set to CompressionLevelBestCompression or CompressionLevelBestSpeed The bufferSize parameter specifies the size of the buffer used by the compressor. For best performance, set it to a size that's power 2, large enough for the expected input. Returns an io.WriteCloser for writing compressed data and an error, if any.

func NewGoZLibUncompressor

func NewGoZLibUncompressor(input io.Reader, bufferSize uint32) (io.ReadCloser, error)

NewGoZLibUncompressor creates a new uncompressor that supports zlib or gzip inputs The input parameter is the io.Reader providing the compressed data to be uncompressed, and the bufferSize parameter is the size of the buffer to use in the internal compression transformer. For best performance, set it to a size that's power 2, large enough for the expected input.

func ResetCompressor

func ResetCompressor(output io.Writer, compressor io.WriteCloser)

ResetCompressor is a helper function that can be used when pooling compressors The compressor will use the given output to write data to

func ResetUncompressor

func ResetUncompressor(input io.Reader, uncompressor io.ReadCloser)

ResetUncompressor is a helper function that can be used when pooling uncompressors the uncompressor will use the given input to read data from

Types

type CompressionLevel

type CompressionLevel int

const (
	CompressionLevelBestCompression CompressionLevel = C.Z_BEST_COMPRESSION
	CompressionLevelBestSpeed       CompressionLevel = C.Z_BEST_SPEED
)

type DataStreamEventHandler

type DataStreamEventHandler func(data []byte) uint32

type NativeSlicePool

type NativeSlicePool struct {
	// contains filtered or unexported fields
}

NativeSlicePool is a byte slice pool manager where memory allocated for each slice is allocated off-heap The pool allows for slices of various types to be allocated and returned but given the way memory is internally tracked slices of sizes that are power of 2 provide an optimal memory utilization.

func NewNativeSlicePool

func NewNativeSlicePool() *NativeSlicePool

NewNativeSlicePool creates a new slice pool. Manually call NewNativeSlicePool.Free() to release the resouces allocated by the returned NewNativeSlicePool.

func (*NativeSlicePool) Acquire

func (nsp *NativeSlicePool) Acquire(size int) []byte

Acquire acquires a new byte array. For optimal memory utilization use sizes that are power of 2 The maximum size of a slice is limited to 4Mb and the returned slice cannot have its capacity changed. The returned slice is not zeroed out and it has length zero but capacity equals to size

func (*NativeSlicePool) Free

func (nsp *NativeSlicePool) Free()

Free releases the resources allocated by this pool It must be invoked once the pool is not in use anymore to avoid resource leaks

func (*NativeSlicePool) Return

func (nsp *NativeSlicePool) Return(slice []byte)

Return returns the slice to the pool.

type TransformMode

type TransformMode int

const (
	TransformModeZLib       TransformMode = 0
	TransformModeGZip       TransformMode = 1
	TransformModeUncompress TransformMode = 2
)