delimited

package
v0.0.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 10, 2015 License: Apache-2.0 Imports: 6 Imported by: 1

Documentation

Overview

Package delimited implements a reader and writer for simple streams of length-delimited byte records. Each record is written as a varint-encoded length in bytes, followed immediately by the record itself.

A stream consists of a sequence of such records packed consecutively without additional padding. There are no checksums or compression.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Copy added in v0.0.9

func Copy(wr Writer, rd Reader) error

Copy writes each record read from rd to wr until rd returns io.EOF or an error occurs.

Types

type Reader

type Reader interface {
	// Next returns the next length-delimited record from the input, or io.EOF if
	// there are no more records available.  Returns io.ErrUnexpectedEOF if a
	// short record is found, with a length of n but fewer than n bytes of data.
	// Because there is no resynchronization mechanism, it is generally not
	// possible to recover from a short record in this format.
	//
	// The slice returned is valid only until a subsequent call to Next.
	Next() ([]byte, error)

	// NextProto reads a record using Next and decodes it into the given
	// proto.Message.
	NextProto(pb proto.Message) error
}

Reader consumes length-delimited records from a byte source.

Usage:

rd := delimited.NewReader(r)
for {
  rec, err := rd.Next()
  if err == io.EOF {
    break
  } else if err != nil {
    log.Fatal(err)
  }
  doStuffWith(rec)
}

func NewReader

func NewReader(r io.Reader) Reader

NewReader constructs a new delimited Reader for the records in r.

type UniqReader added in v0.0.9

type UniqReader struct {
	// contains filtered or unexported fields
}

UniqReader implements the Reader interface. Duplicate records are removed by hashing each and checking against a set of known record hashes. This is a quick-and-dirty method of removing duplicates; it will not be perfect.

func NewUniqReader added in v0.0.9

func NewUniqReader(r Reader, maxSize int) (*UniqReader, error)

NewUniqReader returns a UniqReader over the given Reader. maxSize is the maximum byte size of the cache of known record hashes.

func (*UniqReader) Next added in v0.0.9

func (u *UniqReader) Next() ([]byte, error)

Next implements part of the Reader interface.

func (*UniqReader) NextProto added in v0.0.9

func (u *UniqReader) NextProto(pb proto.Message) error

NextProto implements part of the Reader interface.

func (*UniqReader) Skipped added in v0.0.9

func (u *UniqReader) Skipped() uint64

Skipped returns the number of skipped records.

type Writer

type Writer interface {
	io.Writer

	// Put writes the specified record to the writer.  It equivalent to Write, but
	// discards the number of bytes written.
	Put(record []byte) error

	// PutProto encodes and writes the specified proto.Message to the writer.
	PutProto(msg proto.Message) error
}

A Writer outputs delimited records to an io.Writer.

Basic usage:

wr := delimited.NewWriter(w)
for record := range records {
   if err := wr.Put(record); err != nil {
     log.Fatal(err)
   }
}

func NewWriter

func NewWriter(w io.Writer) Writer

NewWriter constructs a new delimited Writer that writes records to w.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL