Documentation ¶
Overview ¶
Package delimited implements a reader and writer for simple streams of length-delimited byte records. Each record is written as a varint-encoded length in bytes, followed immediately by the record itself.
A stream consists of a sequence of such records packed consecutively without additional padding. There are no checksums or compression.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Reader ¶
type Reader interface { // Next returns the next length-delimited record from the input, or io.EOF if // there are no more records available. Returns io.ErrUnexpectedEOF if a // short record is found, with a length of n but fewer than n bytes of data. // Because there is no resynchronization mechanism, it is generally not // possible to recover from a short record in this format. // // The slice returned is valid only until a subsequent call to Next. Next() ([]byte, error) // NextProto reads a record using Next and decodes it into the given // proto.Message. NextProto(pb proto.Message) error }
Reader consumes length-delimited records from a byte source.
Usage:
rd := delimited.NewReader(r) for { rec, err := rd.Next() if err == io.EOF { break } else if err != nil { log.Fatal(err) } doStuffWith(rec) }
type UniqReader ¶ added in v0.0.9
type UniqReader struct {
// contains filtered or unexported fields
}
UniqReader implements the Reader interface. Duplicate records are removed by hashing each and checking against a set of known record hashes. This is a quick-and-dirty method of removing duplicates; it will not be perfect.
func NewUniqReader ¶ added in v0.0.9
func NewUniqReader(r Reader, maxSize int) (*UniqReader, error)
NewUniqReader returns a UniqReader over the given Reader. maxSize is the maximum byte size of the cache of known record hashes.
func (*UniqReader) Next ¶ added in v0.0.9
func (u *UniqReader) Next() ([]byte, error)
Next implements part of the Reader interface.
func (*UniqReader) NextProto ¶ added in v0.0.9
func (u *UniqReader) NextProto(pb proto.Message) error
NextProto implements part of the Reader interface.
func (*UniqReader) Skipped ¶ added in v0.0.9
func (u *UniqReader) Skipped() uint64
Skipped returns the number of skipped records.
type Writer ¶
type Writer interface { io.Writer // Put writes the specified record to the writer. It equivalent to Write, but // discards the number of bytes written. Put(record []byte) error // PutProto encodes and writes the specified proto.Message to the writer. PutProto(msg proto.Message) error }
A Writer outputs delimited records to an io.Writer.
Basic usage:
wr := delimited.NewWriter(w) for record := range records { if err := wr.Put(record); err != nil { log.Fatal(err) } }