Documentation ¶
Overview ¶
Package lz4 implements reading and writing lz4 compressed data.
The package supports both the LZ4 stream format, as specified in http://fastcompression.blogspot.fr/2013/04/lz4-streaming-format-final.html, and the LZ4 block format, defined at http://fastcompression.blogspot.fr/2011/05/lz4-explained.html.
See https://github.com/lz4/lz4 for the reference C implementation.
Example ¶
package main import ( "io" "os" "strings" "github.com/carmel/lz4" ) func main() { // Compress and uncompress an input string. s := "hello world" r := strings.NewReader(s) // The pipe will uncompress the data from the writer. pr, pw := io.Pipe() zw := lz4.NewWriter(pw) zr := lz4.NewReader(pr) go func() { // Compress the input string. _, _ = io.Copy(zw, r) _ = zw.Close() // Make sure the writer is closed _ = pw.Close() // Terminate the pipe }() _, _ = io.Copy(os.Stdout, zr) }
Output: hello world
Index ¶
- Constants
- Variables
- func CompressBlock(src, dst []byte, _ []int) (int, error)
- func CompressBlockBound(n int) int
- func CompressBlockHC(src, dst []byte, depth CompressionLevel, _, _ []int) (int, error)
- func UncompressBlock(src, dst []byte) (int, error)
- type BlockSize
- type CompressionLevel
- type Compressor
- type CompressorHC
- type Option
- func BlockChecksumOption(flag bool) Option
- func BlockSizeOption(size BlockSize) Option
- func ChecksumOption(flag bool) Option
- func CompressionLevelOption(level CompressionLevel) Option
- func ConcurrencyOption(n int) Option
- func LegacyOption(legacy bool) Option
- func OnBlockDoneOption(handler func(size int)) Option
- func SizeOption(size uint64) Option
- type Reader
- type Writer
Examples ¶
Constants ¶
const ( // ErrInvalidSourceShortBuffer is returned by UncompressBlock or CompressBLock when a compressed // block is corrupted or the destination buffer is not large enough for the uncompressed data. ErrInvalidSourceShortBuffer = lz4errors.ErrInvalidSourceShortBuffer // ErrInvalidFrame is returned when reading an invalid LZ4 archive. ErrInvalidFrame = lz4errors.ErrInvalidFrame // ErrInternalUnhandledState is an internal error. ErrInternalUnhandledState = lz4errors.ErrInternalUnhandledState // ErrInvalidHeaderChecksum is returned when reading a frame. ErrInvalidHeaderChecksum = lz4errors.ErrInvalidHeaderChecksum // ErrInvalidBlockChecksum is returned when reading a frame. ErrInvalidBlockChecksum = lz4errors.ErrInvalidBlockChecksum // ErrInvalidFrameChecksum is returned when reading a frame. ErrInvalidFrameChecksum = lz4errors.ErrInvalidFrameChecksum // ErrOptionInvalidCompressionLevel is returned when the supplied compression level is invalid. ErrOptionInvalidCompressionLevel = lz4errors.ErrOptionInvalidCompressionLevel // ErrOptionClosedOrError is returned when an option is applied to a closed or in error object. ErrOptionClosedOrError = lz4errors.ErrOptionClosedOrError // ErrOptionInvalidBlockSize is returned when ErrOptionInvalidBlockSize = lz4errors.ErrOptionInvalidBlockSize // ErrOptionNotApplicable is returned when trying to apply an option to an object not supporting it. ErrOptionNotApplicable = lz4errors.ErrOptionNotApplicable // ErrWriterNotClosed is returned when attempting to reset an unclosed writer. ErrWriterNotClosed = lz4errors.ErrWriterNotClosed )
Variables ¶
var ( DefaultBlockSizeOption = BlockSizeOption(Block4Mb) DefaultChecksumOption = ChecksumOption(true) DefaultConcurrency = ConcurrencyOption(1) )
Default options.
Functions ¶
func CompressBlock ¶
CompressBlock is equivalent to Compressor.CompressBlock. The final argument is ignored and should be set to nil.
This function is deprecated. Use a Compressor instead.
Example ¶
package main import ( "fmt" "strings" "github.com/carmel/lz4" ) func main() { s := "hello world" data := []byte(strings.Repeat(s, 100)) buf := make([]byte, len(data)) var c lz4.Compressor n, err := c.CompressBlock(data, buf) if err != nil { fmt.Println(err) } if n >= len(data) { fmt.Printf("`%s` is not compressible", s) } buf = buf[:n] // compressed data // Allocate a very large buffer for decompression. out := make([]byte, 10*len(data)) n, err = lz4.UncompressBlock(buf, out) if err != nil { fmt.Println(err) } out = out[:n] // uncompressed data fmt.Println(string(out[:len(s)])) }
Output: hello world
func CompressBlockBound ¶
CompressBlockBound returns the maximum size of a given buffer of size n, when not compressible.
func CompressBlockHC ¶
func CompressBlockHC(src, dst []byte, depth CompressionLevel, _, _ []int) (int, error)
CompressBlockHC is equivalent to CompressorHC.CompressBlock. The final two arguments are ignored and should be set to nil.
This function is deprecated. Use a CompressorHC instead.
func UncompressBlock ¶
UncompressBlock uncompresses the source buffer into the destination one, and returns the uncompressed size.
The destination buffer must be sized appropriately.
An error is returned if the source data is invalid or the destination buffer is too small.
Types ¶
type BlockSize ¶
type BlockSize uint32
BlockSizeIndex defines the size of the blocks to be compressed.
type CompressionLevel ¶
type CompressionLevel uint32
CompressionLevel defines the level of compression to use. The higher the better, but slower, compression.
const ( Fast CompressionLevel = 0 Level1 CompressionLevel = 1 << (8 + iota) Level2 Level3 Level4 Level5 Level6 Level7 Level8 Level9 )
func (CompressionLevel) String ¶
func (i CompressionLevel) String() string
type Compressor ¶
type Compressor struct {
// contains filtered or unexported fields
}
A Compressor compresses data into the LZ4 block format. It uses a fast compression algorithm.
A Compressor is not safe for concurrent use by multiple goroutines.
Use a Writer to compress into the LZ4 stream format.
func (*Compressor) CompressBlock ¶
func (c *Compressor) CompressBlock(src, dst []byte) (int, error)
CompressBlock compresses the source buffer src into the destination dst.
If compression is successful, the first return value is the size of the compressed data, which is always >0.
If dst has length at least CompressBlockBound(len(src)), compression always succeeds. Otherwise, the first return value is zero. The error return is non-nil if the compressed data does not fit in dst, but it might fit in a larger buffer that is still smaller than CompressBlockBound(len(src)). The return value (0, nil) means the data is likely incompressible and a buffer of length CompressBlockBound(len(src)) should be passed in.
type CompressorHC ¶
type CompressorHC struct { // Level is the maximum search depth for compression. // Values <= 0 mean no maximum. Level CompressionLevel // contains filtered or unexported fields }
A CompressorHC compresses data into the LZ4 block format. Its compression ratio is potentially better than that of a Compressor, but it is also slower and requires more memory.
A Compressor is not safe for concurrent use by multiple goroutines.
Use a Writer to compress into the LZ4 stream format.
func (*CompressorHC) CompressBlock ¶
func (c *CompressorHC) CompressBlock(src, dst []byte) (int, error)
CompressBlock compresses the source buffer src into the destination dst.
If compression is successful, the first return value is the size of the compressed data, which is always >0.
If dst has length at least CompressBlockBound(len(src)), compression always succeeds. Otherwise, the first return value is zero. The error return is non-nil if the compressed data does not fit in dst, but it might fit in a larger buffer that is still smaller than CompressBlockBound(len(src)). The return value (0, nil) means the data is likely incompressible and a buffer of length CompressBlockBound(len(src)) should be passed in.
type Option ¶
type Option func(applier) error
Option defines the parameters to setup an LZ4 Writer or Reader.
func BlockChecksumOption ¶
BlockChecksumOption enables or disables block checksum (default=false).
func BlockSizeOption ¶
BlockSizeOption defines the maximum size of compressed blocks (default=Block4Mb).
func ChecksumOption ¶
ChecksumOption enables/disables all blocks or content checksum (default=true).
func CompressionLevelOption ¶
func CompressionLevelOption(level CompressionLevel) Option
CompressionLevelOption defines the compression level (default=Fast).
func ConcurrencyOption ¶
ConcurrencyOption sets the number of go routines used for compression. If n <= 0, then the output of runtime.GOMAXPROCS(0) is used.
func LegacyOption ¶
LegacyOption provides support for writing LZ4 frames in the legacy format.
See https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#legacy-frame.
NB. compressed Linux kernel images use a tweaked LZ4 legacy format where the compressed stream is followed by the original (uncompressed) size of the kernel (https://events.static.linuxfound.org/sites/events/files/lcjpcojp13_klee.pdf). This is also supported as a special case.
func OnBlockDoneOption ¶
OnBlockDoneOption is triggered when a block has been processed. For a Writer, it is when is has been compressed, for a Reader, it is when it has been uncompressed.
func SizeOption ¶
SizeOption sets the size of the original uncompressed data (default=0). It is useful to know the size of the whole uncompressed data stream.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader allows reading an LZ4 stream.
func (*Reader) Reset ¶
Reset clears the state of the Reader r such that it is equivalent to its initial state from NewReader, but instead writing to writer. No access to reader is performed.
w.Close must be called before Reset.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer allows writing an LZ4 stream.
func (*Writer) Close ¶
Close closes the Writer, flushing any unwritten data to the underlying io.Writer, but does not close the underlying io.Writer.
func (*Writer) ReadFrom ¶
ReadFrom efficiently reads from r and compressed into the Writer destination.
func (*Writer) Reset ¶
Reset clears the state of the Writer w such that it is equivalent to its initial state from NewWriter, but instead writing to writer. Reset keeps the previous options unless overwritten by the supplied ones. No access to writer is performed.
w.Close must be called before Reset or pending data may be dropped.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
internal
|
|
lz4block
Package lz4block provides LZ4 BlockSize types and pools of buffers.
|
Package lz4block provides LZ4 BlockSize types and pools of buffers. |
lz4stream
Package lz4stream provides the types that support reading and writing LZ4 data streams.
|
Package lz4stream provides the types that support reading and writing LZ4 data streams. |
xxh32
Package xxh32 implements the very fast XXH hashing algorithm (32 bits version).
|
Package xxh32 implements the very fast XXH hashing algorithm (32 bits version). |