msgpackzip

README

msgpackzip

Usage

You can test compressing a message pack encoded input file by running:

go run ./cli/main.go <inputfile>

Documentation

Index

• Variables
• func Compress(input []byte) (output []byte, err error)
• func CompressWithWhitelist(input []byte, wl ValueWhitelist) (output []byte, err error)
• func Inflate(input []byte) (output []byte, err error)
• type BinaryMapKey
• type Frequency
  • func ReportValuesFrequencies(input []byte) (ret []Frequency, err error)
• type ValueWhitelist
  • func NewValueWhitelist() *ValueWhitelist
  • func (v *ValueWhitelist) AddBinary(b []byte)
  • func (v *ValueWhitelist) AddString(s string)
• type Version

Variables

var ErrBinaryTooBig = errors.New("binary allocation is too big")

var ErrContainerTooBig = errors.New("container allocation is too big")

var ErrExtTooBig = errors.New("extenal data type too big")

var ErrIntTooBig = errors.New("Cannot handle ints largers than int64 max")

var ErrLenTooBig = errors.New("Lenghts bigger than 0x8000000 are too big")

var ErrMaxDepth = errors.New("input exceeded maximum allowed depth")

var ErrStringTooBig = errors.New("string allocation is too big")

Functions

func Compress

func Compress(input []byte) (output []byte, err error)

Compress the given msgpack encoding, compressing only static map keys, and not compressing any values.

func CompressWithWhitelist

func CompressWithWhitelist(input []byte, wl ValueWhitelist) (output []byte, err error)

CompressWithWhitelist takes as input a msgpack encoded payload, and also a whitelist of values that it's OK to compress. It then compresses all map keys and values in the given whitelist, and returns a compression, or an error on error.

func Inflate

func Inflate(input []byte) (output []byte, err error)

Types

type BinaryMapKey

type BinaryMapKey string

BinaryMapKey is a wrapper around a []byte vector of binary data so that it can be stored as an interface{} and differentiated from proper strings.

type Frequency

type Frequency struct {
	Key  interface{}
	Freq int
}

Frequency is a tuple, with a `Key interface{}` that can be an int64, a string, or a BinaryMapKey (which is a wrapper around a binary buffer). The `Freq` field is a count for how many times the `Key` shows up in the encoded msgpack object.

func ReportValuesFrequencies

func ReportValuesFrequencies(input []byte) (ret []Frequency, err error)

ReportValuesFrequencies takes as input a msgpack encoding, and reports which values are the most fequent in the encoding. It returns a list of Frequency objects, sorted from most frequent to least frequent.

type ValueWhitelist

type ValueWhitelist struct {
	// contains filtered or unexported fields
}

ValueWhitelist can be used to specify which values can be compressed. Values are either strings or binary []byte arrays.

func NewValueWhitelist

func NewValueWhitelist() *ValueWhitelist

NewValueWhitelist makes an empty value white list, initialzed with empty lists.

func (*ValueWhitelist) AddBinary

func (v *ValueWhitelist) AddBinary(b []byte)

AddBinary adds a binary buffer to the value whitelist

func (*ValueWhitelist) AddString

func (v *ValueWhitelist) AddString(s string)

AddString adds a string to the value whitelist

type Version

type Version int