pbzip2

README

go-pbzip2

A go library to use pbzip2 for faster bzip2 operations than the stdlib. Supports both compression and decompression.

Benchmarks

For small amounts of bzip2 data, using the standard bzip2 library is probably faster. Once you start decompressing data beyond 1MB, pbzip2 will be a lot faster with roughly linear performance to the number of cores. These benchmarks are run with %d random bytes compressed with pbzip2. Machine was an i7-8550 processor with 4 cores and 8 threads.

goos: linux
goarch: amd64
pkg: github.com/d4l3k/go-pbzip2
BenchmarkPBZip2Read/1B-8                    5000           3509845 ns/op
BenchmarkPBZip2Read/10B-8                   5000           3306304 ns/op
BenchmarkPBZip2Read/100B-8                  5000           3315698 ns/op
BenchmarkPBZip2Read/1000B-8                 5000           3663066 ns/op
BenchmarkPBZip2Read/10000B-8                3000           4580623 ns/op
BenchmarkPBZip2Read/100000B-8               2000          11377153 ns/op
BenchmarkPBZip2Read/1000000B-8               200          66438579 ns/op
BenchmarkPBZip2Read/10000000B-8               50         329762754 ns/op
BenchmarkPBZip2Read/100000000B-8               5        2416747246 ns/op
BenchmarkBZip2Read/1B-8                    30000            388585 ns/op
BenchmarkBZip2Read/10B-8                   30000            401892 ns/op
BenchmarkBZip2Read/100B-8                  30000            492336 ns/op
BenchmarkBZip2Read/1000B-8                 20000            743518 ns/op
BenchmarkBZip2Read/10000B-8                10000           1981048 ns/op
BenchmarkBZip2Read/100000B-8                2000           9123679 ns/op
BenchmarkBZip2Read/1000000B-8                200          76655666 ns/op
BenchmarkBZip2Read/10000000B-8                20         745520450 ns/op
BenchmarkBZip2Read/100000000B-8                2        8047164946 ns/op

Usage

Reader

// If pbzip2 is not present on the system, stdlib bzip2.Reader is used instead.
func NewReader(r io.Reader) (io.ReadCloser, error) { ... }

Writer

// NewWriter will use the default Level of 9
func NewWriter(w io.Writer) (io.WriteCloser, error) { ... }

type WriterConfig struct {
    Level int
}

func NewWriterConfig(w io.Writer, conf *WriterConfig) (io.WriteCloser, error) { ... }

Documentation

Overview

Package pbzip2 is a wrapper around the pbzip2 command. If it's present on the system, this library will use it for higher performance when decompressing bzip2 data.

Index

• Constants
• func NewReader(r io.Reader) (io.ReadCloser, error)
• func NewWriter(w io.Writer) (io.WriteCloser, error)
• func NewWriterConfig(w io.Writer, conf *WriterConfig) (io.WriteCloser, error)
• type WriterConfig

Constants

const (
	// Command is the pbzip2 command
	Command = "pbzip2"

	// BestSpeed represents the smallest block size (100k)
	BestSpeed = 1
	// BestCompression represents the largest block size (900k)
	BestCompression = 9
	// DefaultCompression represents the default block size (900k)
	DefaultCompression = 9
)

Functions

func NewReader

func NewReader(r io.Reader) (io.ReadCloser, error)

NewReader creates a new pbzip2 reader. This will print a warning if pbzip2 is not present on the system and return a stdlib bzip2.Reader instead.

func NewWriter

func NewWriter(w io.Writer) (io.WriteCloser, error)

NewWriter creates a new pbzip2 writer. This will return an error if pbzip2 is not present on the system.

func NewWriterConfig

func NewWriterConfig(w io.Writer, conf *WriterConfig) (io.WriteCloser, error)

NewWriterConfig creates a new pbzip2 writer with configuration options. This will return an error if pbzip2 is not present on the system.

Types

type WriterConfig

type WriterConfig struct {
	Level int
}

WriterConfig stores configuration options for a pbzip2 writer