README
¶
pgzip
Go parallel gzip compression/decompression. This is a fully gzip compatible drop in replacement for "compress/gzip".
This will split compression into blocks that are compressed in parallel. This can be useful for compressing big amounts of data. The output is a standard gzip file.
The gzip decompression is modified so it decompresses ahead of the current reader. This means that reads will be non-blocking if the decompressor can keep ahead of your code reading from it. CRC calculation also takes place in a separate goroutine.
You should only use this if you are (de)compressing big amounts of data, say more than 1MB at the time, otherwise you will not see any benefit, and it will likely be faster to use the internal gzip library or this package.
It is important to note that this library creates and reads standard gzip files. You do not have to match the compressor/decompressor to get the described speedups, and the gzip files are fully compatible with other gzip readers/writers.
A golang variant of this is bgzf, which has the same feature, as well as seeking in the resulting file. The only drawback is a slightly bigger overhead compared to this and pure gzip. See a comparison below.
Installation
go get github.com/klauspost/pgzip/...
You might need to get/update the dependencies:
go get -u github.com/klauspost/compress
go get -u github.com/klauspost/crc32
Usage
To use as a replacement for gzip, exchange
import "compress/gzip"
with
import gzip "github.com/klauspost/pgzip".
Changes
- Oct 6, 2016: Fixed an issue if the destination writer returned an error.
- Oct 6, 2016: Better buffer reuse, should now generate less garbage.
- Oct 6, 2016: Output does not change based on write sizes.
- Dec 8, 2015: Decoder now supports the io.WriterTo interface, giving a speedup and less GC pressure.
- Oct 9, 2015: Reduced allocations by ~35 by using sync.Pool. ~15% overall speedup.
Changes in github.com/klauspost/compress are also carried over, so see that for more changes.
Compression
The simplest way to use this is to simply do the same as you would when using compress/gzip.
To change the block size, use the added (*pgzip.Writer).SetConcurrency(blockSize, blocks int) function. With this you can control the approximate size of your blocks, as well as how many you want to be processing in parallel. Default values for this is SetConcurrency(250000, 16), meaning blocks are split at 250000 bytes and up to 16 blocks can be processing at once before the writer blocks.
Example:
var b bytes.Buffer
w := gzip.NewWriter(&b)
w.SetConcurrency(100000, 10)
w.Write([]byte("hello, world\n"))
w.Close()
To get any performance gains, you should at least be compressing more than 1 megabyte of data at the time.
You should at least have a block size of 100k and at least a number of blocks that match the number of cores your would like to utilize, but about twice the number of blocks would be the best.
Another side effect of this is, that it is likely to speed up your other code, since writes to the compressor only blocks if the compressor is already compressing the number of blocks you have specified. This also means you don't have worry about buffering input to the compressor.
Decompression
Decompression works similar to compression. That means that you simply call pgzip the same way as you would call compress/gzip.
The only difference is that if you want to specify your own readahead, you have to use pgzip.NewReaderN(r io.Reader, blockSize, blocks int) to get a reader with your custom blocksizes. The blockSize is the size of each block decoded, and blocks is the maximum number of blocks that is decoded ahead.
Performance
Compression
See my blog post in Benchmarks of Golang Gzip.
Compression cost is usually about 0.2% with default settings with a block size of 250k.
Example with GOMAXPROC set to 8 (quad core with 8 hyperthreads)
Content is Matt Mahoneys 10GB corpus. Compression level 6.
| Compressor | MB/sec | speedup | size | size overhead (lower=better) |
|---|---|---|---|---|
| gzip (golang) | 7.21MB/s | 1.0x | 4786608902 | 0% |
| gzip (klauspost) | 10.98MB/s | 1.52x | 4781331645 | -0.11% |
| pgzip (klauspost) | 50.76MB/s | 7.04x | 4784121440 | -0.052% |
| bgzf (biogo) | 38.65MB/s | 5.36x | 4924899484 | 2.889% |
| pargzip (builder) | 32.00MB/s | 4.44x | 4791226567 | 0.096% |
pgzip also contains a linear time compression mode, that will allow compression at ~150MB per core per second, independent of the content.
See the complete sheet for different content types and compression settings.
Decompression
The decompression speedup is there because it allows you to do other work while the decompression is taking place.
In the example above, the numbers are as follows on a 4 CPU machine:
| Decompressor | Time | Speedup |
|---|---|---|
| gzip (golang) | 1m28.85s | 0% |
| pgzip (golang) | 43.48s | 104% |
But wait, since gzip decompression is inherently singlethreaded (aside from CRC calculation) how can it be more than 100% faster? Because pgzip due to its design also acts as a buffer. When using ubuffered gzip, you are also waiting for io when you are decompressing. If the gzip decoder can keep up, it will always have data ready for your reader, and you will not be waiting for input to the gzip decompressor to complete.
This is pretty much an optimal situation for pgzip, but it reflects most common usecases for CPU intensive gzip usage.
I haven't included bgzf in this comparision, since it only can decompress files created by a compatible encoder, and therefore cannot be considered a generic gzip decompressor. But if you are able to compress your files with a bgzf compatible program, you can expect it to scale beyond 100%.
#License This contains large portions of code from the go repository - see GO_LICENSE for more information. The changes are released under MIT License. See LICENSE for more information.
Documentation
¶
Overview ¶
Package pgzip implements reading and writing of gzip format compressed files, as specified in RFC 1952.
This is a drop in replacement for "compress/gzip". This will split compression into blocks that are compressed in parallel. This can be useful for compressing big amounts of data. The gzip decompression has not been modified, but remains in the package, so you can use it as a complete replacement for "compress/gzip".
See more at https://github.com/klauspost/pgzip
Index ¶
Constants ¶
const ( NoCompression = flate.NoCompression BestSpeed = flate.BestSpeed BestCompression = flate.BestCompression DefaultCompression = flate.DefaultCompression ConstantCompression = flate.ConstantCompression HuffmanOnly = flate.HuffmanOnly )
These constants are copied from the flate package, so that code that imports "compress/gzip" does not also have to import "compress/flate".
Variables ¶
var ( // ErrChecksum is returned when reading GZIP data that has an invalid checksum. ErrChecksum = errors.New("gzip: invalid checksum") // ErrHeader is returned when reading GZIP data that has an invalid header. ErrHeader = errors.New("gzip: invalid header") )
Functions ¶
This section is empty.
Types ¶
type Header ¶
type Header struct {
Comment string // comment
Extra []byte // "extra data"
ModTime time.Time // modification time
Name string // file name
OS byte // operating system type
}
The gzip file stores a header giving metadata about the compressed file. That header is exposed as the fields of the Writer and Reader structs.
type Reader ¶
type Reader struct {
Header
// contains filtered or unexported fields
}
A Reader is an io.Reader that can be read to retrieve uncompressed data from a gzip-format compressed file.
In general, a gzip file can be a concatenation of gzip files, each with its own header. Reads from the Reader return the concatenation of the uncompressed data of each. Only the first header is recorded in the Reader fields.
Gzip files store a length and checksum of the uncompressed data. The Reader will return a ErrChecksum when Read reaches the end of the uncompressed data if it does not have the expected length or checksum. Clients should treat data returned by Read as tentative until they receive the io.EOF marking the end of the data.
func NewReader ¶
NewReader creates a new Reader reading the given reader. The implementation buffers input and may read more data than necessary from r. It is the caller's responsibility to call Close on the Reader when done.
func NewReaderN ¶
NewReaderN creates a new Reader reading the given reader. The implementation buffers input and may read more data than necessary from r. It is the caller's responsibility to call Close on the Reader when done.
With this you can control the approximate size of your blocks, as well as how many blocks you want to have prefetched.
Default values for this is blockSize = 250000, blocks = 16, meaning up to 16 blocks of maximum 250000 bytes will be prefetched.
func (*Reader) Multistream ¶
Multistream controls whether the reader supports multistream files.
If enabled (the default), the Reader expects the input to be a sequence of individually gzipped data streams, each with its own header and trailer, ending at EOF. The effect is that the concatenation of a sequence of gzipped files is treated as equivalent to the gzip of the concatenation of the sequence. This is standard behavior for gzip readers.
Calling Multistream(false) disables this behavior; disabling the behavior can be useful when reading file formats that distinguish individual gzip data streams or mix gzip data streams with other data streams. In this mode, when the Reader reaches the end of the data stream, Read returns io.EOF. If the underlying reader implements io.ByteReader, it will be left positioned just after the gzip stream. To start the next stream, call z.Reset(r) followed by z.Multistream(false). If there is no next stream, z.Reset(r) will return io.EOF.
type Writer ¶
type Writer struct {
Header
// contains filtered or unexported fields
}
A Writer is an io.WriteCloser. Writes to a Writer are compressed and written to w.
func NewWriter ¶
NewWriter returns a new Writer. Writes to the returned writer are compressed and written to w.
It is the caller's responsibility to call Close on the WriteCloser when done. Writes may be buffered and not flushed until Close.
Callers that wish to set the fields in Writer.Header must do so before the first call to Write or Close. The Comment and Name header fields are UTF-8 strings in Go, but the underlying format requires NUL-terminated ISO 8859-1 (Latin-1). NUL or non-Latin-1 runes in those strings will lead to an error on Write.
func NewWriterLevel ¶
NewWriterLevel is like NewWriter but specifies the compression level instead of assuming DefaultCompression.
The compression level can be DefaultCompression, NoCompression, or any integer value between BestSpeed and BestCompression inclusive. The error returned will be nil if the level is valid.
func (*Writer) Close ¶
Close closes the Writer, flushing any unwritten data to the underlying io.Writer, but does not close the underlying io.Writer.
func (*Writer) Flush ¶
Flush flushes any pending compressed data to the underlying writer.
It is useful mainly in compressed network protocols, to ensure that a remote reader has enough data to reconstruct a packet. Flush does not return until the data has been written. If the underlying writer returns an error, Flush returns that error.
In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH.
func (*Writer) Reset ¶
Reset discards the Writer z's state and makes it equivalent to the result of its original state from NewWriter or NewWriterLevel, but writing to w instead. This permits reusing a Writer rather than allocating a new one.
func (*Writer) SetConcurrency ¶
Use SetConcurrency to finetune the concurrency level if needed.
With this you can control the approximate size of your blocks, as well as how many you want to be processing in parallel.
Default values for this is SetConcurrency(250000, 16), meaning blocks are split at 250000 bytes and up to 16 blocks can be processing at once before the writer blocks.
func (*Writer) UncompressedSize ¶
UncompressedSize will return the number of bytes written. pgzip only, not a function in the official gzip package.
func (*Writer) Write ¶
Write writes a compressed form of p to the underlying io.Writer. The compressed bytes are not necessarily flushed to output until the Writer is closed or Flush() is called.
The function will return quickly, if there are unused buffers. The sent slice (p) is copied, and the caller is free to re-use the buffer when the function returns.
Errors that occur during compression will be reported later, and a nil error does not signify that the compression succeeded (since it is most likely still running) That means that the call that returns an error may not be the call that caused it. Only Flush and Close functions are guaranteed to return any errors up to that point.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
compress
|
|
|
flate
Package flate implements the DEFLATE compressed data format, described in RFC 1951.
|
Package flate implements the DEFLATE compressed data format, described in RFC 1951. |
|
gzip
Package gzip implements reading and writing of gzip format compressed files, as specified in RFC 1952.
|
Package gzip implements reading and writing of gzip format compressed files, as specified in RFC 1952. |
|
snappy
Package snappy implements the snappy block-based compression format.
|
Package snappy implements the snappy block-based compression format. |
|
zip
Package zip provides support for reading and writing ZIP archives.
|
Package zip provides support for reading and writing ZIP archives. |
|
zlib
Package zlib implements reading and writing of zlib format compressed data, as specified in RFC 1950.
|
Package zlib implements reading and writing of zlib format compressed data, as specified in RFC 1950. |
|
Package cpuid provides information about the CPU running the current program.
|
Package cpuid provides information about the CPU running the current program. |
|
Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32, checksum.
|
Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32, checksum. |