Documentation
¶
Overview ¶
Package s2 implements the S2 compression format.
S2 is an extension of Snappy. Similar to Snappy S2 is aimed for high throughput, which is why it features concurrent compression for bigger payloads.
Decoding is compatible with Snappy compressed content, but content compressed with S2 cannot be decompressed by Snappy.
For more information on Snappy/S2 differences see README in: https://github.com/zhangdapeng520/zdpgo_fasthttp/compress/tree/master/s2
There are actually two S2 formats: block and stream. They are related, but different: trying to decompress block-compressed data as a S2 stream will fail, and vice versa. The block format is the Decode and Encode functions and the stream format is the Reader and Writer types.
A "better" compression option is available. This will trade some compression speed
The block format, the more common case, is used when the complete size (the number of bytes) of the original data is known upfront, at the time compression starts. The stream format, also known as the framing format, is for when that isn't always true.
Blocks to not offer much data protection, so it is up to you to add data validation of decompressed blocks.
Streams perform CRC validation of the decompressed data. Stream compression will also be performed on multiple CPU cores concurrently significantly improving throughput.
Index ¶
- Constants
- Variables
- func ConcatBlocks(dst []byte, blocks ...[]byte) ([]byte, error)
- func Decode(dst, src []byte) ([]byte, error)
- func DecodedLen(src []byte) (int, error)
- func Encode(dst, src []byte) []byte
- func EncodeBest(dst, src []byte) []byte
- func EncodeBetter(dst, src []byte) []byte
- func EncodeSnappy(dst, src []byte) []byte
- func EncodeSnappyBest(dst, src []byte) []byte
- func EncodeSnappyBetter(dst, src []byte) []byte
- func IndexStream(r io.Reader) ([]byte, error)
- func MaxEncodedLen(srcLen int) int
- type ErrCantSeek
- type Index
- type ReadSeeker
- type Reader
- func (r *Reader) Read(p []byte) (int, error)
- func (r *Reader) ReadByte() (byte, error)
- func (r *Reader) ReadSeeker(random bool, index []byte) (*ReadSeeker, error)
- func (r *Reader) Reset(reader io.Reader)
- func (r *Reader) Skip(n int64) error
- func (r *Reader) SkippableCB(id uint8, fn func(r io.Reader) error) error
- type ReaderOption
- type Writer
- func (w *Writer) AddSkippableBlock(id uint8, data []byte) (err error)
- func (w *Writer) Close() error
- func (w *Writer) CloseIndex() ([]byte, error)
- func (w *Writer) EncodeBuffer(buf []byte) (err error)
- func (w *Writer) Flush() error
- func (w *Writer) ReadFrom(r io.Reader) (n int64, err error)
- func (w *Writer) Reset(writer io.Writer)
- func (w *Writer) Write(p []byte) (nRet int, errRet error)
- type WriterOption
- func WriterAddIndex() WriterOption
- func WriterBestCompression() WriterOption
- func WriterBetterCompression() WriterOption
- func WriterBlockSize(n int) WriterOption
- func WriterConcurrency(n int) WriterOption
- func WriterFlushOnWrite() WriterOption
- func WriterPadding(n int) WriterOption
- func WriterPaddingSrc(reader io.Reader) WriterOption
- func WriterSnappyCompat() WriterOption
- func WriterUncompressed() WriterOption
Constants ¶
const ( S2IndexHeader = "s2idx\x00" S2IndexTrailer = "\x00xdi2s" )
const (
ChunkTypeIndex = 0x99
)
const MaxBlockSize = math.MaxUint32 - binary.MaxVarintLen32 - 5
MaxBlockSize is the maximum value where MaxEncodedLen will return a valid block size. Blocks this big are highly discouraged, though.
Variables ¶
var ( // ErrCorrupt reports that the input is invalid. ErrCorrupt = errors.New("s2: corrupt input") // ErrCRC reports that the input failed CRC validation (streams only) ErrCRC = errors.New("s2: corrupt input, crc mismatch") // ErrTooLarge reports that the uncompressed length is too large. ErrTooLarge = errors.New("s2: decoded block is too large") // ErrUnsupported reports that the input isn't supported. ErrUnsupported = errors.New("s2: unsupported input") )
Functions ¶
func ConcatBlocks ¶
ConcatBlocks will concatenate the supplied blocks and append them to the supplied destination. If the destination is nil or too small, a new will be allocated. The blocks are not validated, so garbage in = garbage out. dst may not overlap block data. Any data in dst is preserved as is, so it will not be considered a block.
func Decode ¶
Decode returns the decoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire decoded block. Otherwise, a newly allocated slice will be returned.
The dst and src must not overlap. It is valid to pass a nil dst.
func DecodedLen ¶
DecodedLen returns the length of the decoded block.
func Encode ¶
Encode returns the encoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire encoded block. Otherwise, a newly allocated slice will be returned.
The dst and src must not overlap. It is valid to pass a nil dst.
The blocks will require the same amount of memory to decode as encoding, and does not make for concurrent decoding. Also note that blocks do not contain CRC information, so corruption may be undetected.
If you need to encode larger amounts of data, consider using the streaming interface which gives all of these features.
func EncodeBest ¶
EncodeBest returns the encoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire encoded block. Otherwise, a newly allocated slice will be returned.
EncodeBest compresses as good as reasonably possible but with a big speed decrease.
The dst and src must not overlap. It is valid to pass a nil dst.
The blocks will require the same amount of memory to decode as encoding, and does not make for concurrent decoding. Also note that blocks do not contain CRC information, so corruption may be undetected.
If you need to encode larger amounts of data, consider using the streaming interface which gives all of these features.
func EncodeBetter ¶
EncodeBetter returns the encoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire encoded block. Otherwise, a newly allocated slice will be returned.
EncodeBetter compresses better than Encode but typically with a 10-40% speed decrease on both compression and decompression.
The dst and src must not overlap. It is valid to pass a nil dst.
The blocks will require the same amount of memory to decode as encoding, and does not make for concurrent decoding. Also note that blocks do not contain CRC information, so corruption may be undetected.
If you need to encode larger amounts of data, consider using the streaming interface which gives all of these features.
func EncodeSnappy ¶
EncodeSnappy returns the encoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire encoded block. Otherwise, a newly allocated slice will be returned.
The output is Snappy compatible and will likely decompress faster.
The dst and src must not overlap. It is valid to pass a nil dst.
The blocks will require the same amount of memory to decode as encoding, and does not make for concurrent decoding. Also note that blocks do not contain CRC information, so corruption may be undetected.
If you need to encode larger amounts of data, consider using the streaming interface which gives all of these features.
func EncodeSnappyBest ¶
EncodeSnappyBest returns the encoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire encoded block. Otherwise, a newly allocated slice will be returned.
The output is Snappy compatible and will likely decompress faster.
The dst and src must not overlap. It is valid to pass a nil dst.
The blocks will require the same amount of memory to decode as encoding, and does not make for concurrent decoding. Also note that blocks do not contain CRC information, so corruption may be undetected.
If you need to encode larger amounts of data, consider using the streaming interface which gives all of these features.
func EncodeSnappyBetter ¶
EncodeSnappyBetter returns the encoded form of src. The returned slice may be a sub- slice of dst if dst was large enough to hold the entire encoded block. Otherwise, a newly allocated slice will be returned.
The output is Snappy compatible and will likely decompress faster.
The dst and src must not overlap. It is valid to pass a nil dst.
The blocks will require the same amount of memory to decode as encoding, and does not make for concurrent decoding. Also note that blocks do not contain CRC information, so corruption may be undetected.
If you need to encode larger amounts of data, consider using the streaming interface which gives all of these features.
func IndexStream ¶
IndexStream will return an index for a stream. The stream structure will be checked, but data within blocks is not verified. The returned index can either be appended to the end of the stream or stored separately.
func MaxEncodedLen ¶
MaxEncodedLen returns the maximum length of a snappy block, given its uncompressed length.
It will return a negative value if srcLen is too large to encode. 32 bit platforms will have lower thresholds for rejecting big content.
Types ¶
type ErrCantSeek ¶
type ErrCantSeek struct {
Reason string
}
ErrCantSeek is returned if the stream cannot be seeked.
type Index ¶
type Index struct {
TotalUncompressed int64 // Total Uncompressed size if known. Will be -1 if unknown.
TotalCompressed int64 // Total Compressed size if known. Will be -1 if unknown.
// contains filtered or unexported fields
}
Index represents an S2/Snappy index.
func (*Index) Find ¶
Find the offset at or before the wanted (uncompressed) offset. If offset is 0 or positive it is the offset from the beginning of the file. If the uncompressed size is known, the offset must be within the file. If an offset outside the file is requested io.ErrUnexpectedEOF is returned. If the offset is negative, it is interpreted as the distance from the end of the file, where -1 represents the last byte. If offset from the end of the file is requested, but size is unknown, ErrUnsupported will be returned.
func (*Index) Load ¶
Load a binary index. A zero value Index can be used or a previous one can be reused.
func (*Index) LoadStream ¶
func (i *Index) LoadStream(rs io.ReadSeeker) error
LoadStream will load an index from the end of the supplied stream. ErrUnsupported will be returned if the signature cannot be found. ErrCorrupt will be returned if unexpected values are found. io.ErrUnexpectedEOF is returned if there are too few bytes. IO errors are returned as-is.
type ReadSeeker ¶
type ReadSeeker struct {
*Reader
}
ReadSeeker provides random or forward seeking in compressed content. See Reader.ReadSeeker
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is an io.Reader that can read Snappy-compressed bytes.
func NewReader ¶
func NewReader(r io.Reader, opts ...ReaderOption) *Reader
NewReader returns a new Reader that decompresses from r, using the framing format described at https://github.com/google/snappy/blob/master/framing_format.txt with S2 changes.
func (*Reader) ReadSeeker ¶
func (r *Reader) ReadSeeker(random bool, index []byte) (*ReadSeeker, error)
ReadSeeker will return an io.ReadSeeker compatible version of the reader. If 'random' is specified the returned io.Seeker can be used for random seeking, otherwise only forward seeking is supported. Enabling random seeking requires the original input to support the io.Seeker interface. A custom index can be specified which will be used if supplied. When using a custom index, it will not be read from the input stream. The returned ReadSeeker contains a shallow reference to the existing Reader, meaning changes performed to one is reflected in the other.
func (*Reader) Reset ¶
Reset discards any buffered data, resets all state, and switches the Snappy reader to read from r. This permits reusing a Reader rather than allocating a new one.
func (*Reader) Skip ¶
Skip will skip n bytes forward in the decompressed output. For larger skips this consumes less CPU and is faster than reading output and discarding it. CRC is not checked on skipped blocks. io.ErrUnexpectedEOF is returned if the stream ends before all bytes have been skipped. If a decoding error is encountered subsequent calls to Read will also fail.
func (*Reader) SkippableCB ¶
SkippableCB will register a callback for chunks with the specified ID. ID must be a Reserved skippable chunks ID, 0x80-0xfe (inclusive). For each chunk with the ID, the callback is called with the content. Any returned non-nil error will abort decompression. Only one callback per ID is supported, latest sent will be used. Sending a nil function will disable previous callbacks.
type ReaderOption ¶
ReaderOption is an option for creating a decoder.
func ReaderAllocBlock ¶
func ReaderAllocBlock(blockSize int) ReaderOption
ReaderAllocBlock allows to control upfront stream allocations and not allocate for frames bigger than this initially. If frames bigger than this is seen a bigger buffer will be allocated.
Default is 1MB, which is default output size.
func ReaderIgnoreStreamIdentifier ¶
func ReaderIgnoreStreamIdentifier() ReaderOption
ReaderIgnoreStreamIdentifier will make the reader skip the expected stream identifier at the beginning of the stream. This can be used when serving a stream that has been forwarded to a specific point.
func ReaderMaxBlockSize ¶
func ReaderMaxBlockSize(blockSize int) ReaderOption
ReaderMaxBlockSize allows to control allocations if the stream has been compressed with a smaller WriterBlockSize, or with the default 1MB. Blocks must be this size or smaller to decompress, otherwise the decoder will return ErrUnsupported.
For streams compressed with Snappy this can safely be set to 64KB (64 << 10).
Default is the maximum limit of 4MB.
func ReaderSkippableCB ¶
func ReaderSkippableCB(id uint8, fn func(r io.Reader) error) ReaderOption
ReaderSkippableCB will register a callback for chuncks with the specified ID. ID must be a Reserved skippable chunks ID, 0x80-0xfd (inclusive). For each chunk with the ID, the callback is called with the content. Any returned non-nil error will abort decompression. Only one callback per ID is supported, latest sent will be used.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is an io.Writer that can write Snappy-compressed bytes.
func NewWriter ¶
func NewWriter(w io.Writer, opts ...WriterOption) *Writer
NewWriter returns a new Writer that compresses to w, using the framing format described at https://github.com/google/snappy/blob/master/framing_format.txt
Users must call Close to guarantee all data has been forwarded to the underlying io.Writer and that resources are released. They may also call Flush zero or more times before calling Close.
func (*Writer) AddSkippableBlock ¶
AddSkippableBlock will add a skippable block to the stream. The ID must be 0x80-0xfe (inclusive). Length of the skippable block must be <= 16777215 bytes.
func (*Writer) Close ¶
Close calls Flush and then closes the Writer. Calling Close multiple times is ok, but calling CloseIndex after this will make it not return the index.
func (*Writer) CloseIndex ¶
CloseIndex calls Close and returns an index on first call. This is not required if you are only adding index to a stream.
func (*Writer) EncodeBuffer ¶
EncodeBuffer will add a buffer to the stream. This is the fastest way to encode a stream, but the input buffer cannot be written to by the caller until Flush or Close has been called when concurrency != 1.
If you cannot control that, use the regular Write function.
Note that input is not buffered. This means that each write will result in discrete blocks being created. For buffered writes, use the regular Write function.
func (*Writer) Flush ¶
Flush flushes the Writer to its underlying io.Writer. This does not apply padding.
func (*Writer) ReadFrom ¶
ReadFrom implements the io.ReaderFrom interface. Using this is typically more efficient since it avoids a memory copy. ReadFrom reads data from r until EOF or error. The return value n is the number of bytes read. Any error except io.EOF encountered during the read is also returned.
type WriterOption ¶
WriterOption is an option for creating a encoder.
func WriterAddIndex ¶
func WriterAddIndex() WriterOption
WriterAddIndex will append an index to the end of a stream when it is closed.
func WriterBestCompression ¶
func WriterBestCompression() WriterOption
WriterBestCompression will enable better compression. EncodeBetter compresses better than Encode but typically with a big speed decrease on compression.
func WriterBetterCompression ¶
func WriterBetterCompression() WriterOption
WriterBetterCompression will enable better compression. EncodeBetter compresses better than Encode but typically with a 10-40% speed decrease on both compression and decompression.
func WriterBlockSize ¶
func WriterBlockSize(n int) WriterOption
WriterBlockSize allows to override the default block size. Blocks will be this size or smaller. Minimum size is 4KB and and maximum size is 4MB.
Bigger blocks may give bigger throughput on systems with many cores, and will increase compression slightly, but it will limit the possible concurrency for smaller payloads for both encoding and decoding. Default block size is 1MB.
When writing Snappy compatible output using WriterSnappyCompat, the maximum block size is 64KB.
func WriterConcurrency ¶
func WriterConcurrency(n int) WriterOption
WriterConcurrency will set the concurrency, meaning the maximum number of decoders to run concurrently. The value supplied must be at least 1. By default this will be set to GOMAXPROCS.
func WriterFlushOnWrite ¶
func WriterFlushOnWrite() WriterOption
WriterFlushOnWrite will compress blocks on each call to the Write function.
This is quite inefficient as blocks size will depend on the write size.
Use WriterConcurrency(1) to also make sure that output is flushed. When Write calls return, otherwise they will be written when compression is done.
func WriterPadding ¶
func WriterPadding(n int) WriterOption
WriterPadding will add padding to all output so the size will be a multiple of n. This can be used to obfuscate the exact output size or make blocks of a certain size. The contents will be a skippable frame, so it will be invisible by the decoder. n must be > 0 and <= 4MB. The padded area will be filled with data from crypto/rand.Reader. The padding will be applied whenever Close is called on the writer.
func WriterPaddingSrc ¶
func WriterPaddingSrc(reader io.Reader) WriterOption
WriterPaddingSrc will get random data for padding from the supplied source. By default crypto/rand is used.
func WriterSnappyCompat ¶
func WriterSnappyCompat() WriterOption
WriterSnappyCompat will write snappy compatible output. The output can be decompressed using either snappy or s2. If block size is more than 64KB it is set to that.
func WriterUncompressed ¶
func WriterUncompressed() WriterOption
WriterUncompressed will bypass compression. The stream will be written as uncompressed blocks only. If concurrency is > 1 CRC and output will still be done async.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
internal/filepathx
Package filepathx adds double-star globbing support to the Glob function from the core path/filepath package.
|
Package filepathx adds double-star globbing support to the Glob function from the core path/filepath package. |
|
internal/readahead
Package readahead will do asynchronous read-ahead from an input io.Reader and make the data available as an io.Reader.
|
Package readahead will do asynchronous read-ahead from an input io.Reader and make the data available as an io.Reader. |