xerial

package
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2024 License: Apache-2.0, BSD-3-Clause, MIT, + 2 more Imports: 4 Imported by: 0

README

go-xerial-snappy

Xerial-compatible Snappy framing support for golang.

Packages using Xerial for snappy encoding use a framing format incompatible with basically everything else in existence.

Apps that use this format include Apache Kafka (see https://github.com/dpkp/kafka-python/issues/126#issuecomment-35478921 for details).

Fork

Forked from github.com/eapache/go-xerial-snappy.

Changes:

  • Uses S2 for better/faster compression and decompression.
  • Fixes 0-length roundtrips.
  • Adds DecodeCapped, which allows decompression with capped output size.
  • DecodeInto will decode directly into destination if there is space enough.
  • Encode will now encode directly into 'dst' if it has space enough.
  • Fixes short snappy buffers returning ErrMalformed.
  • Renames EncodeStream to Encode.
  • Adds EncodeBetter for better than default compression at ~half the speed.

Comparison (before/after):

BenchmarkSnappyStreamEncode-32    	  959010	      1170 ns/op	 875.15 MB/s	    1280 B/op	       1 allocs/op
BenchmarkSnappyStreamEncode-32    	 1000000	      1107 ns/op	 925.04 MB/s	       0 B/op	       0 allocs/op
--> Output size: 913 -> 856 bytes

BenchmarkSnappyStreamEncodeBetter-32    	  477739	      2506 ns/op	 408.62 MB/s	       0 B/op	       0 allocs/op
--> Output size: 835 bytes

BenchmarkSnappyStreamEncodeMassive-32  	     100    	  10596963 ns/op	 966.31 MB/s	   40977 B/op	       1 allocs/op
BenchmarkSnappyStreamEncodeMassive-32  	     100       	  10220236 ns/op	1001.93 MB/s	       0 B/op	       0 allocs/op
--> Output size: 2365547 -> 2256991 bytes

BenchmarkSnappyStreamEncodeBetterMassive-32    	      69	  16983314 ns/op	 602.94 MB/s	       0 B/op	       0 allocs/op
--> Output size: 2011997 bytes

BenchmarkSnappyStreamDecodeInto-32    	 1887378	       639.5 ns/op	1673.19 MB/s	    1088 B/op	       3 allocs/op
BenchmarkSnappyStreamDecodeInto-32    	 2707915	       436.2 ns/op	2452.99 MB/s	       0 B/op	       0 allocs/op

BenchmarkSnappyStreamDecodeIntoMassive-32    	     267	   4559594 ns/op	2245.81 MB/s	   71120 B/op	       1 allocs/op
BenchmarkSnappyStreamDecodeIntoMassive-32    	     282	   4285844 ns/op	2389.26 MB/s	       0 B/op	       0 allocs/op

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrDstTooSmall = errors.New("destination buffer too small")
View Source
var (

	// ErrMalformed is returned by the decoder when the xerial framing
	// is malformed
	ErrMalformed = errors.New("malformed xerial framing")
)

Functions

func Decode

func Decode(src []byte) ([]byte, error)

Decode decodes snappy data whether it is traditional unframed or includes the xerial framing format.

func DecodeCapped

func DecodeCapped(dst, src []byte) ([]byte, error)

DecodeCapped decodes snappy data whether it is traditional unframed or includes the xerial framing format into the specified `dst`. It is assumed that the entirety of `dst` including all capacity is available for use by this function. If `dst` is nil *or* insufficiently large to hold the decoded `src`, ErrDstTooSmall is returned.

func DecodeInto

func DecodeInto(dst, src []byte) ([]byte, error)

DecodeInto decodes snappy data whether it is traditional unframed or includes the xerial framing format into the specified `dst`. It is assumed that the entirety of `dst` including all capacity is available for use by this function. If `dst` is nil *or* insufficiently large to hold the decoded `src`, new space will be allocated. To never allocate bigger destination, use DecodeCapped.

func Encode

func Encode(dst, src []byte) []byte

Encode *appends* to the specified 'dst' the compressed 'src' in xerial framing format. If 'dst' does not have enough capacity, then a new slice will be allocated. If 'dst' has non-zero length, then if *must* have been built using this function.

func EncodeBetter

func EncodeBetter(dst, src []byte) []byte

EncodeBetter *appends* to the specified 'dst' the compressed 'src' in xerial framing format. If 'dst' does not have enough capacity, then a new slice will be allocated. If 'dst' has non-zero length, then if *must* have been built using this function.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL