snappystream

README

go-snappystream

a Go package for framed snappy streams.

This package wraps snappy-go and supplies a Reader and Writer for the snappy framed stream format.

Documentation

Overview

snappystream wraps snappy-go and supplies a Reader and Writer for the snappy framed stream format:

https://snappy.googlecode.com/svn/trunk/framing_format.txt

Index

• Constants
• func NewReader(r io.Reader, verifyChecksum bool) io.Reader
• func NewWriter(w io.Writer) io.Writer
• type BufferedWriter
  • func NewBufferedWriter(w io.Writer) *BufferedWriter
  • func (w *BufferedWriter) Close() error
  • func (w *BufferedWriter) Flush() error
  • func (w *BufferedWriter) ReadFrom(r io.Reader) (int64, error)
  • func (w *BufferedWriter) Write(p []byte) (int, error)

Constants

const ContentEncoding = "x-snappy-framed"

ContentEncoding is the appropriate HTTP Content-Encoding header value for requests containing a snappy framed entity body.

const Ext = ".sz"

Ext is the file extension for files whose content is a snappy framed stream.

const MaxBlockSize = 65536

MaxBlockSize is the maximum number of decoded bytes allowed to be represented in a snappy framed block (sections 4.2 and 4.3).

const MediaType = "application/x-snappy-framed"

MediaType is the MIME type used to represent snappy framed content.

const SkipVerifyChecksum = false

const VerifyChecksum = true

Functions

func NewReader

func NewReader(r io.Reader, verifyChecksum bool) io.Reader

NewReader returns an io.Reader interface to the snappy framed stream format.

It transparently handles reading the stream identifier (but does not proxy this to the caller), decompresses blocks, and (optionally) validates checksums.

Internally, three buffers are maintained. The first two are for reading off the wrapped io.Reader and for holding the decompressed block (both are grown automatically and re-used and will never exceed the largest block size, 65536). The last buffer contains the *unread* decompressed bytes (and can grow indefinitely).

The second param determines whether or not the reader will verify block checksums and can be enabled/disabled with the constants VerifyChecksum and SkipVerifyChecksum

For each Read, the returned length will be up to the lesser of len(b) or 65536 decompressed bytes, regardless of the length of *compressed* bytes read from the wrapped io.Reader.

func NewWriter

func NewWriter(w io.Writer) io.Writer

NewWriter returns an io.Writer that writes its input to an underlying io.Writer encoded as a snappy framed stream. A stream identifier block is written to w preceding the first data block. The returned writer will never emit a block with length in bytes greater than MaxBlockSize+4 nor one containing more than MaxBlockSize bytes of (uncompressed) data.

For each Write, the returned length will only ever be len(p) or 0, regardless of the length of *compressed* bytes written to the wrapped io.Writer. If the returned length is 0 then error will be non-nil. If len(p) exceeds 65536, the slice will be automatically chunked into smaller blocks which are all emitted before the call returns.

Types

type BufferedWriter

type BufferedWriter struct {
	// contains filtered or unexported fields
}

BufferedWriter is an io.WriteCloser with behavior similar to writers returned by NewWriter but it buffers written data, maximizing block size (to improve the output compression ratio) at the cost of speed. Benefits over NewWriter are most noticible when individual writes are small and when streams are long.

Failure to call a BufferedWriter's Close or Flush methods after it is done being written to will likely result in missing data frames which will be undetectable in the decoding process.

NOTE: BufferedWriter cannot be instantiated via struct literal and must use NewBufferedWriter (i.e. its zero value is not usable).

func NewBufferedWriter

func NewBufferedWriter(w io.Writer) *BufferedWriter

NewBufferedWriter allocates and returns a BufferedWriter with an internal buffer of MaxBlockSize bytes. If an error occurs writing a block to w, all future writes will fail with the same error. After all data has been written, the client should call the Flush method to guarantee all data has been forwarded to the underlying io.Writer.

func (*BufferedWriter) Close

func (w *BufferedWriter) Close() error

Close flushes w's internal buffer and tears down internal data structures. After a successful call to Close method calls on w return an error. Close makes no attempt to close the underlying writer.

func (*BufferedWriter) Flush

func (w *BufferedWriter) Flush() error

Flush encodes and writes a block with the contents of w's internal buffer to the underlying writer even if the buffer does not contain a full block of data (MaxBlockSize bytes).

func (*BufferedWriter) ReadFrom

func (w *BufferedWriter) ReadFrom(r io.Reader) (int64, error)

ReadFrom implements the io.ReaderFrom interface used by io.Copy. It encodes data read from r as a snappy framed stream that is written to the underlying writer. ReadFrom returns the number number of bytes read, along with any error encountered (other than io.EOF).

func (*BufferedWriter) Write

func (w *BufferedWriter) Write(p []byte) (int, error)

Write buffers p internally, encoding and writing a block to the underlying buffer if the buffer grows beyond MaxBlockSize bytes. The returned int will be 0 if there was an error and len(p) otherwise.