Documentation ¶
Overview ¶
Package zstd provides decompression of zstandard files.
For advanced usage and examples, go to the README: https://github.com/klauspost/compress/tree/master/zstd#zstd
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrSnappyCorrupt reports that the input is invalid. ErrSnappyCorrupt = errors.New("snappy: corrupt input") // ErrSnappyTooLarge reports that the uncompressed length is too large. ErrSnappyTooLarge = errors.New("snappy: decoded block is too large") // ErrSnappyUnsupported reports that the input isn't supported. ErrSnappyUnsupported = errors.New("snappy: unsupported input") )
var ( // ErrReservedBlockType is returned when a reserved block type is found. // Typically this indicates wrong or corrupted input. ErrReservedBlockType = errors.New("invalid input: reserved block type encountered") // ErrCompressedSizeTooBig is returned when a block is bigger than allowed. // Typically this indicates wrong or corrupted input. ErrCompressedSizeTooBig = errors.New("invalid input: compressed size too big") // ErrBlockTooSmall is returned when a block is too small to be decoded. // Typically returned on invalid input. ErrBlockTooSmall = errors.New("block too small") // ErrMagicMismatch is returned when a "magic" number isn't what is expected. // Typically this indicates wrong or corrupted input. ErrMagicMismatch = errors.New("invalid input: magic number mismatch") // ErrWindowSizeExceeded is returned when a reference exceeds the valid window size. // Typically this indicates wrong or corrupted input. ErrWindowSizeExceeded = errors.New("window size exceeded") // ErrWindowSizeTooSmall is returned when no window size is specified. // Typically this indicates wrong or corrupted input. ErrWindowSizeTooSmall = errors.New("invalid input: window size was too small") // ErrDecoderSizeExceeded is returned if decompressed size exceeds the configured limit. ErrDecoderSizeExceeded = errors.New("decompressed size exceeds configured limit") // ErrUnknownDictionary is returned if the dictionary ID is unknown. // For the time being dictionaries are not supported. ErrUnknownDictionary = errors.New("unknown dictionary") // ErrCRCMismatch is returned if CRC mismatches. ErrCRCMismatch = errors.New("CRC check failed") // ErrDecoderClosed will be returned if the Decoder was used after // Close has been called. ErrDecoderClosed = errors.New("decoder used after Close") )
Functions ¶
This section is empty.
Types ¶
type DOption ¶
type DOption func(*decoderOptions) error
DOption is an option for creating a decoder.
func WithDecoderConcurrency ¶
WithDecoderConcurrency will set the concurrency, meaning the maximum number of decoders to run concurrently. The value supplied must be at least 1. By default this will be set to GOMAXPROCS.
func WithDecoderLowmem ¶
WithDecoderLowmem will set whether to use a lower amount of memory, but possibly have to allocate more while running.
func WithDecoderMaxMemory ¶
WithDecoderMaxMemory allows to set a maximum decoded size for in-memory (non-streaming) operations. Maxmimum and default is 1 << 63 bytes.
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder provides decoding of zstandard streams. The decoder has been designed to operate without allocations after a warmup. This means that you should store the decoder for best performance. To re-use a stream decoder, use the Reset(r io.Reader) error to switch to another stream. A decoder can safely be re-used even if the previous stream failed. To release the resources, you must call the Close() function on a decoder.
func NewReader ¶
NewReader creates a new decoder. A nil Reader can be provided in which case Reset can be used to start a decode.
A Decoder can be used in two modes:
1) As a stream, or 2) For stateless decoding using DecodeAll or DecodeBuffer.
Only a single stream can be decoded concurrently, but the same decoder can run multiple concurrent stateless decodes. It is even possible to use stateless decodes while a stream is being decoded.
The Reset function can be used to initiate a new stream, which is will considerably reduce the allocations normally caused by NewReader.
func (*Decoder) Close ¶
func (d *Decoder) Close()
Close will release all resources. It is NOT possible to reuse the decoder after this.
func (*Decoder) DecodeAll ¶
DecodeAll allows stateless decoding of a blob of bytes. Output will be appended to dst, so if the destination size is known you can pre-allocate the destination slice to avoid allocations. DecodeAll can be used concurrently. The Decoder concurrency limits will be respected.
func (*Decoder) Read ¶
Read bytes from the decompressed stream into p. Returns the number of bytes written and any error that occurred. When the stream is done, io.EOF will be returned.
type EOption ¶ added in v1.6.0
type EOption func(*encoderOptions) error
DOption is an option for creating a encoder.
func WithEncoderCRC ¶ added in v1.6.0
WithEncoderCRC will add CRC value to output. Output will be 4 bytes larger.
func WithEncoderConcurrency ¶ added in v1.6.0
WithEncoderConcurrency will set the concurrency, meaning the maximum number of decoders to run concurrently. The value supplied must be at least 1. By default this will be set to GOMAXPROCS.
func WithSingleSegment ¶ added in v1.6.0
WithSingleSegment will set the "single segment" flag when EncodeAll is used. If this flag is set, data must be regenerated within a single continuous memory segment. In this case, Window_Descriptor byte is skipped, but Frame_Content_Size is necessarily present. As a consequence, the decoder must allocate a memory segment of size equal or larger than size of your content. In order to preserve the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame which requests a memory size beyond decoder's authorized range. For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation, each decoder is free to support higher or lower limits, depending on local limitations. This setting has no effect on streamed encodes.
type Encoder ¶ added in v1.6.0
type Encoder struct {
// contains filtered or unexported fields
}
Encoder provides encoding to Zstandard. An Encoder can be used for either compressing a stream via the io.WriteCloser interface supported by the Encoder or as multiple independent tasks via the EncodeAll function. Smaller encodes are encouraged to use the EncodeAll function. Use NewWriter to create a new instance.
func NewWriter ¶ added in v1.6.0
NewWriter will create a new Zstandard encoder. If the encoder will be used for encoding blocks a nil writer can be used.
func (*Encoder) Close ¶ added in v1.6.0
Close will flush the final output and close the stream. The function will block until everything has been written.
func (*Encoder) EncodeAll ¶ added in v1.6.0
EncodeAll will encode all input in src and append it to dst. This function can be called concurrently, but each call will only run on a single goroutine. If empty input is given, nothing is returned. Encoded blocks can be concatenated and the result will be the combined input stream. Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or DecodeAll.
func (*Encoder) Flush ¶ added in v1.6.0
Flush will send the currently written data to output and block until everything has been written. This should only be used on rare occasions where pushing the currently queued data is critical.
func (*Encoder) ReadFrom ¶ added in v1.6.0
ReadFrom reads data from r until EOF or error. The return value n is the number of bytes read. Any error except io.EOF encountered during the read is also returned.
The Copy function uses ReaderFrom if available.
func (*Encoder) Reset ¶ added in v1.6.0
Reset will re-initialize the writer and new writes will encode to the supplied writer as a new, independent stream.
type SnappyConverter ¶ added in v1.6.0
type SnappyConverter struct {
// contains filtered or unexported fields
}
SnappyConverter can read SnappyConverter-compressed streams and convert them to zstd. Conversion is done by converting the stream directly from Snappy without intermediate full decoding. Therefore the compression ratio is much less than what can be done by a full decompression and compression, and a faulty Snappy stream may lead to a faulty Zstandard stream without any errors being generated. No CRC value is being generated and not all CRC values of the Snappy stream are checked. However, it provides really fast recompression of Snappy streams. The converter can be reused to avoid allocations, even after errors.
Source Files ¶
- bitreader.go
- bitwriter.go
- blockdec.go
- blockenc.go
- blocktype_string.go
- bytebuf.go
- bytereader.go
- decoder.go
- decoder_options.go
- enc_fast.go
- enc_params.go
- encoder.go
- encoder_options.go
- framedec.go
- frameenc.go
- fse_decoder.go
- fse_encoder.go
- fse_predefined.go
- hash.go
- history.go
- seqdec.go
- seqenc.go
- snappy.go
- zstd.go