zlib

README

zlib for go

- For whole buffered data (i.e. it fits into memory) use https://github.com/4kills/go-libdeflate !

4kills/go-libdeflate is much faster (at least 3 times) and completely compatible with zlib!
With that said, if you need to stream large data from disk, you may continue with this library.

---

This ultra fast Go zlib library wraps the original zlib library written in C by Jean-loup Gailly and Mark Adler using cgo.

It offers considerable performance benefits compared to the standard Go zlib library, as the benchmarks show.

This library is designed to be completely and easily interchangeable with the Go standard zlib library. You won't have to rewrite or modify a single line of code! Checking if this library works for you is as easy as changing imports!

This library also offers fast convenience methods that can be used as a clean, alternative interface to that provided by the Go standard library. (See usage).

Table of Contents

• Features
• Installation
  • Install cgo
  • Install pkg-config and zlib
  • Download
  • Import
• Usage
  • Compress
  • Decompress
• Notes
• Benchmarks
  • Compression
  • Decompression
• License
• Links

Features

• zlib compression / decompression
• A variety of different compression strategies and compression levels to choose from
• Seamless interchangeability with the Go standard zlib library
• Alternative, super fast convenience methods for compression / decompression
• Benchmarks with comparisons to the Go standard zlib library
• Custom, user-defined dictionaries
• More customizable memory management
• Support streaming of data to compress/decompress data.
• Out-of-the-box support for amd64 Linux, Windows, MacOS
• Support for most common architecture/os combinations (see Installation for a particular OS and Architecture)

Installation

For the library to work, you need cgo, zlib (which is used by this library under the hood), and pkg-config (linker):

Install cgo

TL;DR: Get cgo working.

In order to use this library with your Go source code, you must be able to use the Go tool cgo, which, in turn, requires a GCC compiler.

If you are on Linux, there is a good chance you already have GCC installed, otherwise just get it with your favorite package manager.

If you are on MacOS, Xcode - for instance - supplies the required tools.

If you are on Windows, you will need to install GCC. I can recommend tdm-gcc which is based off of MinGW. Please note that cgo requires the 64-bit version (as stated here).

For any other the procedure should be about the same. Just google.

Install pkg-config and zlib

This SDK uses zlib under the hood. For the SDK to work, you need to install zlib on your system which is super easy! Additionally we require pkg-config which facilitates linking zlib with this (cgo) SDK. How exactly you install these two packages depends on your operating system.

MacOS (HomeBrew):

brew install zlib
brew install pkg-config

Linux:

Use the package manager available on your distro to install the required packages.

Windows (MinGW/WSL2):

Here, you can either use WSL2 or MinGW and from there install the required packages.

Download

To get the most recent stable version of this library just type:

$ go get github.com/4kills/go-zlib

You may also use Go modules (available since Go 1.11) to get the version of a specific branch or tag if you want to try out or use experimental features. However, beware that these versions are not necessarily guaranteed to be stable or thoroughly tested.

Import

This library is designed in a way to make it easy to swap it out for the Go standard zlib library. Therefore, you should only need to change imports and not a single line of your written code.

Just remove:

import compress/zlib

and use instead:

import "github.com/4kills/go-zlib"

If there are any problems with your existing code after this step, please let me know.

Usage

This library can be used exactly like the go standard zlib library but it also adds additional methods to make your life easier.

Compress

Like with the standard library:

var b bytes.Buffer              // use any writer
w := zlib.NewWriter(&b)         // create a new zlib.Writer, compressing to b
w.Write([]byte("uncompressed")) // put in any data as []byte  
w.Close()                       // don't forget to close this

Alternatively:

w := zlib.NewWriter(nil)                           // requires no writer if WriteBuffer is used
defer w.Close()                                    // always close when you are done with it
c, _ := w.WriteBuffer([]byte("uncompressed"), nil) // compresses input & returns compressed []byte 

Decompress

Like with the standard library:

b := bytes.NewBuffer(compressed) // reader with compressed data
r, err := zlib.NewReader(&b)     // create a new zlib.Reader, decompressing from b 
defer r.Close()                  // don't forget to close this either
io.Copy(os.Stdout, r)            // read all the decompressed data and write it somewhere
// or:
// r.Read(someBuffer)            // or use read yourself

Alternatively:

r := zlib.NewReader(nil)                 // requires no reader if ReadBuffer is used
defer r.Close()                          // always close or bad things will happen
_, dc, _ := r.ReadBuffer(compressed, nil) // decompresses input & returns decompressed []byte 

Notes

• Do NOT use the same Reader / Writer across multiple threads simultaneously. You can do that if you sync the read/write operations, but you could also create as many readers/writers as you like - for each thread one, so to speak. This library is generally considered thread-safe.

• Always Close() your Reader / Writer when you are done with it - especially if you create a new reader/writer for each decompression/compression you undertake (which is generally discouraged anyway). As the C-part of this library is not subject to the Go garbage collector, the memory allocated by it must be released manually (by a call to Close()) to avoid memory leakage.

• HuffmanOnly does NOT work as with the standard library. If you want to use HuffmanOnly, refer to the NewWriterLevelStrategy() constructor function. However, your existing code won't break by leaving HuffmanOnly as argument to NewWriterLevel(), it will just use the default compression strategy and compression level 2.

• Memory Usage: Compressing requires ~256 KiB of additional memory during execution, while Decompressing requires ~39 KiB of additional memory during execution. So if you have 8 simultaneous WriteBytes working from 8 Writers across 8 threads, your memory footprint from that alone will be about ~2MiByte.

• You are strongly encouraged to use the same Reader / Writer for multiple Decompressions / Compressions as it is not required nor beneficial in any way, shape or form to create a new one every time. The contrary is true: It is more performant to reuse a reader/writer. Of course, if you use the same reader/writer multiple times, you do not need to close them until you are completely done with them (perhaps only at the very end of your program).

• A Reader can be created with an empty underlying reader, unlike with the standard library. I decided to diverge from the standard behavior there, because I thought it was too cumbersome.

Benchmarks

These benchmarks were conducted with "real-life-type data" to ensure that these tests are most representative for an actual use case in a practical production environment. As the zlib standard has been traditionally used for compressing smaller chunks of data, I have decided to follow suite by opting for Minecraft client-server communication packets, as they represent the optimal use case for this library.

To that end, I have recorded 930 individual Minecraft packets, totalling 11,445,993 bytes in umcompressed data and 1,564,159 bytes in compressed data. These packets represent actual client-server communication and were recorded using this software.

The benchmarks were executed on different hardware and operating systems, including AMD and Intel processors, as well as all the supported operating systems (Windows, Linux, MacOS). All the benchmarked functions/methods were executed hundreds of times, and the numbers you are about to see are the averages over all these executions.

These benchmarks compare this library (blue) to the Go standard library (yellow) and show that this library performs better in all cases.

• (A note regarding testing on your machine)

  Please note that you will need an Internet connection for some benchmarks to function. This is because these benchmarks will download the mc packets from here and temporarily store them in memory for the duration of the benchmark tests, so this repository won't have to include the data in order save space on your machine and to make it a lightweight library.

Compression

This chart shows how long it took for the methods of this library (blue) and the standard library (yellow) to compress all of the 930 packets (~11.5 MB) on different systems in nanoseconds. Note that the two rightmost data points were tested on exactly the same hardware in a dual-boot setup and that Linux seems to generally perform better than Windows.

This chart shows the time it took for this library's Write (blue) to compress the data in nanoseconds, as well as the time it took for the standard library's Write (WriteStd, yellow) to compress the data in nanoseconds. The vertical axis shows percentages relative to the time needed by the standard library, thus you can see how much faster this library is.

For example: This library only needed ~88% of the time required by the standard library to compress the packets on an Intel Core i5-6600K on Windows. That makes the standard library ~13.6% slower than this library.

Decompression

This chart shows how long it took for the methods of this library (blue) and the standard library (yellow) to decompress all of the 930 packets (~1.5 MB) on different systems in nanoseconds. Note that the two rightmost data points were tested on exactly the same hardware in a dual-boot setup and that Linux seems to generally perform better than Windows.

This chart shows the time it took for this library's Read (blue) to decompress the data in nanoseconds, as well as the time it took for the standard library's Read (ReadStd, Yellow) to decompress the data in nanoseconds. The vertical axis shows percentages relative to the time needed by the standard library, thus you can see how much faster this library is.

For example: This library only needed a whopping ~57% of the time required by the standard library to decompress the packets on an Intel Core i5-6600K on Windows. That makes the standard library a substantial ~75.4% slower than this library.

License

  Copyright (c) 1995-2017 Jean-loup Gailly and Mark Adler
  Copyright (c) 2020 Dominik Ochs

This software is provided 'as-is', without any express or implied
warranty.  In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

  1. The origin of this software must not be misrepresented; you must not
     claim that you wrote the original software. If you use this software
     in a product, an acknowledgment in the product documentation would be
     appreciated but is not required.
 
  2. Altered source versions must be plainly marked as such, and must not be
     misrepresented as being the original software.
  
  3. This notice may not be removed or altered from any source distribution.

Links

• Original zlib by Jean-loup Gailly and Mark Adler:
  • github
  • website
• Go standard zlib by the Go Authors:
  • github

Documentation

Index

• Constants
• type Reader
  • func NewReader(r io.Reader) (*Reader, error)
  • func NewReaderDict(r io.Reader, dict []byte) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) Read(p []byte) (int, error)
  • func (r *Reader) ReadBuffer(compressed, out []byte) (n int, decompressed []byte, err error)
  • func (r *Reader) Reset(reader io.Reader, dict []byte) error
• type Resetter
• type Writer
  • func NewWriter(w io.Writer) *Writer
  • func NewWriterLevel(w io.Writer, level int) (*Writer, error)
  • func NewWriterLevelDict(w io.Writer, level int, dict []byte) (*Writer, error)
  • func NewWriterLevelStrategy(w io.Writer, level, strategy int) (*Writer, error)
  • func NewWriterLevelStrategyDict(w io.Writer, level, strategy int, dict []byte) (*Writer, error)
  • func (zw *Writer) Close() error
  • func (zw *Writer) Flush() error
  • func (zw *Writer) Reset(w io.Writer)
  • func (zw *Writer) Write(p []byte) (int, error)
  • func (zw *Writer) WriteBuffer(in, out []byte) ([]byte, error)

Constants

const (

	//NoCompression does not compress given input
	NoCompression = 0
	//BestSpeed is fastest but with lowest compression
	BestSpeed = 1
	//BestCompression is slowest but with best compression
	BestCompression = 9
	//DefaultCompression is a compromise between BestSpeed and BestCompression.
	//The level might change if algorithms change.
	DefaultCompression = -1

	// Filtered is more effective for small (but not all too many) randomly distributed values.
	// It forces more Huffman encoding. Use it for filtered data. It's between Default and Huffman only.
	Filtered = 1
	//HuffmanOnly only uses Huffman encoding to compress the given data
	HuffmanOnly = 2
	//RLE (run-length encoding) limits match distance to one, thereby being almost as fast as HuffmanOnly
	// but giving better compression for PNG data.
	RLE = 3
	//Fixed disallows dynamic Huffman codes, thereby making it a simpler decoder
	Fixed = 4
	// DefaultStrategy is the default compression strategy that should be used for most appliances
	DefaultStrategy = 0
)

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader decompresses data from an underlying io.Reader or via the ReadBuffer method, which should be preferred

func NewReader

func NewReader(r io.Reader) (*Reader, error)

NewReader returns a new reader, reading from r. It decompresses read data. r may be nil if you only plan on using ReadBuffer

func NewReaderDict

func NewReaderDict(r io.Reader, dict []byte) (*Reader, error)

NewReaderDict does exactly like NewReader as of NOW. This will change once custom dicionaries are implemented. This function has been added for compatibility with the std lib.

func (*Reader) Close

func (r *Reader) Close() error

Close closes the Reader by closing and freeing the underlying zlib stream. You should not forget to call this after being done with the writer.

func (*Reader) Read

func (r *Reader) Read(p []byte) (int, error)

Read reads compressed data from the underlying Reader into the provided buffer p. To reuse the reader after an EOF condition, you have to Reset it. Please consider using ReadBuffer for whole-buffered data instead, as it is faster and generally easier to use.

func (*Reader) ReadBuffer

func (r *Reader) ReadBuffer(compressed, out []byte) (n int, decompressed []byte, err error)

ReadBuffer takes compressed data p, decompresses it to out in one go and returns out sliced accordingly. This method is generally faster than Read if you know the output size beforehand. If you don't, you can still try to use that method (provide out == nil) but that might take longer than Read. The method also returns the number n of bytes that were processed from the compressed slice. If n < len(compressed) and err == nil then only the first n compressed bytes were in a suitable zlib format and as such decompressed. ReadBuffer resets the reader for new decompression.

func (*Reader) Reset

func (r *Reader) Reset(reader io.Reader, dict []byte) error

Reset resets the Reader to the state of being initialized with zlib.NewX(..), but with the new underlying reader instead. It allows for reuse of the same reader. AS OF NOW dict IS NOT USED. It's just there to implement the Resetter interface to allow for easy interchangeability with the std lib. Just pass nil.

type Resetter

type Resetter interface {
	// Reset resets the Reader to the state of being initialized with zlib.NewX(..),
	// but with the new underlying reader and dict instead. It allows for reuse of the same reader.
	Reset(r io.Reader, dict []byte) error
}

Resetter resets the zlib.Reader returned by NewReader by assigning a new underyling reader, discarding any buffered data from the previous reader. This interface is mainly for compatibility with the std lib

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer compresses and writes given data to an underlying io.Writer

func NewWriter

func NewWriter(w io.Writer) *Writer

NewWriter returns a new Writer with the underlying io.Writer to compress to. w may be nil if you only plan on using WriteBuffer. Panics if the underlying c stream cannot be allocated which would indicate a severe error not only for this library but also for the rest of your code.

func NewWriterLevel

func NewWriterLevel(w io.Writer, level int) (*Writer, error)

NewWriterLevel performs like NewWriter but you may also specify the compression level. w may be nil if you only plan on using WriteBuffer.

func NewWriterLevelDict

func NewWriterLevelDict(w io.Writer, level int, dict []byte) (*Writer, error)

NewWriterLevelDict does exactly like NewWriterLevel as of NOW. This will change once custom dicionaries are implemented. This function has been added for compatibility with the std lib.

func NewWriterLevelStrategy

func NewWriterLevelStrategy(w io.Writer, level, strategy int) (*Writer, error)

NewWriterLevelStrategy performs like NewWriter but you may also specify the compression level and strategy. w may be nil if you only plan on using WriteBuffer.

func NewWriterLevelStrategyDict

func NewWriterLevelStrategyDict(w io.Writer, level, strategy int, dict []byte) (*Writer, error)

NewWriterLevelStrategyDict does exactly like NewWriterLevelStrategy as of NOW. This will change once custom dicionaries are implemented. This function has been added mainly for completeness' sake.

func (*Writer) Close

func (zw *Writer) Close() error

Close closes the writer by flushing any unwritten data to the underlying writer. You should not forget to call this after being done with the writer.

func (*Writer) Flush

func (zw *Writer) Flush() error

Flush writes compressed buffered data to the underlying writer.

func (*Writer) Reset

func (zw *Writer) Reset(w io.Writer)

Reset flushes the buffered data to the current underyling writer, resets the Writer to the state of being initialized with zlib.NewX(..), but with the new underlying writer instead. This will panic if the writer has already been closed, writer could not be reset or could not write to current underlying writer.

func (*Writer) Write

func (zw *Writer) Write(p []byte) (int, error)

Write compresses the given data p and writes it to the underlying io.Writer. The data is not necessarily written to the underlying writer, if no Flush is called. It returns the number of *uncompressed* bytes written to the underlying io.Writer in case of err = nil, or the number of *compressed* bytes in case of err != nil. Please consider using WriteBuffer as it might be more convenient for your use case.

func (*Writer) WriteBuffer

func (zw *Writer) WriteBuffer(in, out []byte) ([]byte, error)

WriteBuffer takes uncompressed data in, compresses it to out and returns out sliced accordingly. In most cases (if the compressed data is smaller than the uncompressed) an out buffer of size len(in) should be sufficient. If you pass nil for out, this function will try to allocate a fitting buffer. Use this for whole-buffered, in-memory data.