recordio

README

RecordIO

Read/Write variable size record bytes to / from io.Reader / io.Writer, default to use snappy compression.

Interface and implementation aims for minimize alloc and memory copy.

Documentation

Overview

Package recordio implements simple var length record format

Index

• Variables
• type Reader
  • func (r *Reader) Next(buf []byte) ([]byte, error)
• type Writer
  • func (w *Writer) Write(b []byte) (int, error)

Variables

var (
	// ErrIncompleteHeader indicates cannot read or write complete header
	ErrIncompleteHeader = errors.New("incomplete header")
	// ErrChecksumFailed indicated CRC checksum failed
	ErrChecksumFailed = errors.New("checksum failed")
	// ErrRecordTooLarge returned when reading a record larger than allowed, this often indicates data corrupted
	ErrRecordTooLarge = errors.New("record to large")
)

Types

type Reader

type Reader struct {
	IO            io.Reader
	MaxRecordSize int
}

Reader reads records from io.Reader

func (*Reader) Next

func (r *Reader) Next(buf []byte) ([]byte, error)

Next reads next record from reader, if size of next record smaller than len(buf), memory of buf will be used in the returning slice, otherwise, new memory will be allocated. when there's no next record, io.EOF will be returned.

type Writer

type Writer struct {
	IO io.Writer
}

Writer writes record to a basic io.Writer with headers

func (*Writer) Write

func (w *Writer) Write(b []byte) (int, error)