recordio

Documentation

Overview

Package recordio implements a basic RecordIO reader and writer.

Each RecordIO frame begins with a Uvarint ( http://golang.org/pkg/encoding/binary/#Uvarint) containing the size of the frame, followed by that many bytes of frame data.

The frame protocol does not handle data integrity; that is left to the outer protocol or medium which uses the frame.

Index

• Variables
• func FrameHeaderSize(s int64) int
• func Split(data []byte) (records [][]byte, err error)
• func WriteFrame(w io.Writer, frame []byte) (int, error)
• type Reader
  • func NewReader(r io.Reader, maxSize int64) Reader
• type Writer
  • func NewWriter(w io.Writer) Writer

Variables

var ErrFrameTooLarge = fmt.Errorf("frame: frame size exceeds maximum")

ErrFrameTooLarge is an error that is returned if a frame that is larger than the maximum allowed size (not including the frame header) is read.

Functions

func FrameHeaderSize

func FrameHeaderSize(s int64) int

FrameHeaderSize calculates the size of the RecordIO frame header for a given amount of data.

A RecordIO frame header is a Uvarint containing the length. Uvarint values contain 7 bits of size data and 1 continuation bit (see encoding/binary).

func Split

func Split(data []byte) (records [][]byte, err error)

Split splits the supplied buffer into its component records.

This method implements zero-copy segmentation, so the individual records are slices of the original data set.

func WriteFrame

func WriteFrame(w io.Writer, frame []byte) (int, error)

WriteFrame writes a single frame to an io.Writer.

Types

type Reader

type Reader interface {
	// ReadFrame reads the next frame, returning the frame's size and an io.Reader
	// for that frame's data. The io.Reader is restricted such that it cannot read
	// past the frame.
	//
	// The frame must be fully read before another Reader call can be made.
	// Failure to do so will cause the Reader to become unsynchronized.
	ReadFrame() (int64, io.Reader, error)

	// ReadFrame returns the contents of the next frame. If there are no more
	// frames available, ReadFrame will return io.EOF.
	ReadFrameAll() ([]byte, error)
}

Reader reads individual frames from a frame-formatted input Reader.

func NewReader

func NewReader(r io.Reader, maxSize int64) Reader

NewReader creates a new Reader which reads frame data from the supplied Reader instance.

If the Reader instance is also an io.ByteReader, its ReadByte method will be used directly.

type Writer

type Writer interface {
	io.Writer

	// Flush writes the buffered frame
	Flush() error

	// Reset clears the writer state and attaches it to a new inner Writer
	// instance.
	Reset(io.Writer)
}

Writer implements the io.Writer interface. Data written to the Writer is translated into a series of frames. Each frame is spearated by a call to Flush.

Frame boundaries are created by calling Flush. Flush will always write a frame, even if the frame's data size is zero.

Data written over consecutive Write calls belongs to the same frame. It is buffered until a frame boundary is created via Flush().

func NewWriter

func NewWriter(w io.Writer) Writer

NewWriter creates a new Writer instance that data as frames to an underlying io.Writer.