pubsubprotocol

Documentation

Overview

Package pubsubprotocol implements the LogDog pubsub wire protocol. This protocol wraps messages that are published to Cloud Pub/Sub for LogDog consumption.

The protocol begins with a set of header bytes to identify and parameterize the remaining data, followed by the data itself.

Note that the Pub/Sub layer is assumed to provide both a total length (so no need to length-prefix) and integrity (so no need to checksum).

Header

The header is a fixed four bytes which positively identify the message as a Butler Pub/Sub message and describe the remaining data. Variant parameters can use different magic numbers to identify different parameters.

Two magic numbers are currently defined:

• 0x10 0x6d 0x06 0x00 (LogDog protocol ButlerLogBundle Raw)
• 0x10 0x6d 0x06 0x62 (LogDog protocol ButlerLogBundle GZip compressed)

Data

The data component is described by the header, and consists of all data in the Pub/Sub message past the last header byte.

Index

• Constants
• type Reader
  • func (r *Reader) Read(ir io.Reader) error
• type Writer
  • func (w *Writer) Write(iw io.Writer, b *logpb.ButlerLogBundle) error
  • func (w *Writer) WriteWith(fw recordio.Writer, b *logpb.ButlerLogBundle) error

Constants

const (
	// DefaultCompressThreshold is the byte size threshold for compressing message
	// data. Messages whose byte count is less than or equal to this threshold
	// will not be compressed.
	//
	// This is the value used by Akamai for its compression threshold:
	// "The reasons 860 bytes is the minimum size for compression is twofold:
	// (1) The overhead of compressing an object under 860 bytes outweighs
	// performance gain. (2) Objects under 860 bytes can be transmitted via a
	// single packet anyway, so there isn't a compelling reason to compress them."
	DefaultCompressThreshold = 860
)

Types

type Reader

type Reader struct {

	// Metadata is the unpacked ButlerMetadata. It is populated when the
	// metadata has been read.
	Metadata *logpb.ButlerMetadata

	// Bundle is the unpacked ButlerLogBundle. It is populated when the
	// protocol data has been read and the Metadata indicates a ButlerLogBundle
	// type.
	Bundle *logpb.ButlerLogBundle
	// contains filtered or unexported fields
}

Reader is a protocol reader instance.

func (*Reader) Read

func (r *Reader) Read(ir io.Reader) error

type Writer

type Writer struct {

	// ProtoVersion is the protocol version string to use. If empty, the current
	// ProtoVersion will be used.
	ProtoVersion string

	// Compress, if true, allows the Writer to choose to compress data when
	// applicable.
	Compress bool

	// CompressThreshold is the minimum size that data must be in order to
	CompressThreshold int
	// contains filtered or unexported fields
}

Writer writes Butler messages that the Reader can read.

func (*Writer) Write

func (w *Writer) Write(iw io.Writer, b *logpb.ButlerLogBundle) error

WriteWith writes a ButlerLogBundle to the supplied Writer.

func (*Writer) WriteWith

func (w *Writer) WriteWith(fw recordio.Writer, b *logpb.ButlerLogBundle) error

WriteWith writes a ButlerLogBundle to the supplied recordio.Writer.