delimited

Documentation

Overview

Package delimited implements a reader and writer for simple streams of length-delimited byte records. Each record is written as a varint-encoded length in bytes, followed immediately by the record itself.

A stream consists of a sequence of such records packed consecutively without additional padding. There are no checksums or compression.

Index

• func Copy(wr Writer, rd Reader) error
• type Reader
  • func NewReader(r io.Reader) Reader
• type UniqReader
  • func NewUniqReader(r Reader, maxSize int) (*UniqReader, error)
  • func (u *UniqReader) Next() ([]byte, error)
  • func (u *UniqReader) NextProto(pb proto.Message) error
  • func (u *UniqReader) Skipped() uint64
• type Writer
  • func NewWriter(w io.Writer) Writer

Functions

func Copy

func Copy(wr Writer, rd Reader) error

Copy writes each record read from rd to wr until rd returns io.EOF or an error occurs.

Types

type Reader

type Reader interface {
	// Next returns the next length-delimited record from the input, or io.EOF if
	// there are no more records available.  Returns io.ErrUnexpectedEOF if a
	// short record is found, with a length of n but fewer than n bytes of data.
	// Because there is no resynchronization mechanism, it is generally not
	// possible to recover from a short record in this format.
	//
	// The slice returned is valid only until a subsequent call to Next.
	Next() ([]byte, error)

	// NextProto reads a record using Next and decodes it into the given
	// proto.Message.
	NextProto(pb proto.Message) error
}

Reader consumes length-delimited records from a byte source.

Usage:

rd := delimited.NewReader(r)
for {
  rec, err := rd.Next()
  if err == io.EOF {
    break
  } else if err != nil {
    log.Fatal(err)
  }
  doStuffWith(rec)
}

func NewReader

func NewReader(r io.Reader) Reader

NewReader constructs a new delimited Reader for the records in r.

type UniqReader

type UniqReader struct {
	// contains filtered or unexported fields
}

UniqReader implements the Reader interface. Duplicate records are removed by hashing each and checking against a set of known record hashes. This is a quick-and-dirty method of removing duplicates; it will not be perfect.

func NewUniqReader

func NewUniqReader(r Reader, maxSize int) (*UniqReader, error)

NewUniqReader returns a UniqReader over the given Reader. maxSize is the maximum byte size of the cache of known record hashes.

func (*UniqReader) Next

func (u *UniqReader) Next() ([]byte, error)

Next implements part of the Reader interface.

func (*UniqReader) NextProto

func (u *UniqReader) NextProto(pb proto.Message) error

NextProto implements part of the Reader interface.

func (*UniqReader) Skipped

func (u *UniqReader) Skipped() uint64

Skipped returns the number of skipped records.

type Writer

type Writer interface {
	io.Writer

	// Put writes the specified record to the writer.  It equivalent to Write, but
	// discards the number of bytes written.
	Put(record []byte) error

	// PutProto encodes and writes the specified proto.Message to the writer.
	PutProto(msg proto.Message) error
}

A Writer outputs delimited records to an io.Writer.

Basic usage:

wr := delimited.NewWriter(w)
for record := range records {
   if err := wr.Put(record); err != nil {
     log.Fatal(err)
   }
}

func NewWriter

func NewWriter(w io.Writer) Writer

NewWriter constructs a new delimited Writer that writes records to w.