binary

Documentation

Overview

Package binary provides encoding and decoding support for a simple, deterministic binary encoding scheme.

Fundamentally, this scheme is simple: it encodes an object as a set of fields where each field is tagged with an identifier. A field identifier must be an integer between 1 and 31, inclusive.

A field is encoded as it's identifier, encoded as a byte, followed by its data. An object is simply a sequence of fields. Fields can be omitted but they cannot be encoded out of order. Attempting to encode or decode field 2 followed by field 1 results in an error. A field may be repeated arbitrarily. If an object would otherwise be empty - if all fields are omitted - EmptyObject is written.

This package supports directly encoding a few types:

• A boolean is written as a byte, either 1 (true) or 0 (false).
• An integer is written as a varint, signed or unsigned.
• A float is written as a big-endian IEEE 754 64-bit float.
• A byte slice or string is written as the length as an unsigned varint followed by data.
• A hash (a fixed-length 32-byte value) is written directly without modification.

Nested objects are encoded then written as a byte slice - the length as an unsigned varint followed by data. Values that cannot be represented as one of the above types must implement their own encoding and are written as a byte slice.

Minimizing memory use

Use Pool and type-specific methods such as Encoder.EncodeInt to minimize memory use during encoding and decoding.

Type-specific methods require the io.Writer or io.Reader to implement additional methods to limit memory use.

• Zero-allocation encoding and decoding of booleans and integers requires io.ByteWriter and io.ByteReader.
• Zero-allocation encoding and decoding of strings and byte slices requires io.ByteWriter and io.ByteReader.
• Zero-allocation encoding of strings also requires io.StringWriter.
• When encoding if the writer does not also implement LenWriter, an internal writer will be used and other methods of the writer will be masked.
• A Pool is used to manage buffers used for encoding nested objects. Reusing a Pool across multiple encoder invocations will reduce allocations.

Index

• Constants
• Variables
• func Marshal(v any) ([]byte, error)
• func Unmarshal(b []byte, v any) error
• type Decoder
  • func NewDecoder(rd io.Reader, opts ...Option) *Decoder
  • func (d *Decoder) Decode(v any) error
  • func (d *Decoder) DecodeBool() (bool, error)
  • func (d *Decoder) DecodeBytes() ([]byte, error)
  • func (d *Decoder) DecodeFloat() (float64, error)
  • func (d *Decoder) DecodeHash() ([32]byte, error)
  • func (d *Decoder) DecodeInt() (int64, error)
  • func (d *Decoder) DecodeString() (string, error)
  • func (d *Decoder) DecodeUint() (uint64, error)
  • func (d *Decoder) DecodeValue(v encoding.BinaryUnmarshaler) error
  • func (d *Decoder) DecodeValueFrom(v unmarshallerV1From) error
  • func (d *Decoder) DecodeValueV2(v Unmarshaller) error
  • func (d *Decoder) EndObject() error
  • func (d *Decoder) Field() (uint, error)
  • func (d *Decoder) InField() bool
  • func (d *Decoder) NoField() error
  • func (d *Decoder) Reset(rd io.Reader, opts ...Option)
  • func (d *Decoder) StartObject() error
• type Encoder
  • func NewEncoder(w io.Writer, opts ...Option) *Encoder
  • func (e *Encoder) Context() *traverse.Context[reflect.Value]
  • func (e *Encoder) Done() error
  • func (e *Encoder) Encode(v any) error
  • func (e *Encoder) EncodeBool(v bool) error
  • func (e *Encoder) EncodeBytes(v []byte) error
  • func (e *Encoder) EncodeField(n uint, v any) error
  • func (e *Encoder) EncodeFloat(v float64) error
  • func (e *Encoder) EncodeHash(v [32]byte) error
  • func (e *Encoder) EncodeInt(v int64) error
  • func (e *Encoder) EncodeString(v string) error
  • func (e *Encoder) EncodeUint(v uint64) error
  • func (e *Encoder) EncodeValue(v encoding.BinaryMarshaler) error
  • func (e *Encoder) EncodeValueV2(v Marshaller) error
  • func (e *Encoder) EndObject() error
  • func (e *Encoder) Field(n uint) error
  • func (e *Encoder) InField() bool
  • func (e *Encoder) NoField() error
  • func (e *Encoder) RepeatLastField() error
  • func (e *Encoder) Reset(w io.Writer, opts ...Option)
  • func (e *Encoder) StartObject() error
• type LenWriter
• type Marshaller
• type Option
  • func IgnoreFieldOrder() Option
  • func LeaveTrailing() Option
  • func WithBufferPool(pool *Pool[*bytes.Buffer]) Option
  • func WithContext(ctx *traverse.Context[reflect.Value]) Option
• type Pool
  • func NewPointerPool[T any]() *Pool[*T]
  • func NewPool[T any](new func() T) *Pool[T]
  • func (p *Pool[T]) Get() T
  • func (p *Pool[T]) Put(v T)
• type Unmarshaller

Constants

const EmptyObject = 0x80

EmptyObject is written when an object would otherwise be empty.

const MaxFieldID = 31

MaxFieldID is the maximum field identifier.

Variables

var ErrFieldsOutOfOrder = errors.New("fields are out of order")

ErrFieldsOutOfOrder is returned when an out of order field is encountered.

var ErrInvalidFieldNumber = errors.New("field number is invalid")

ErrInvalidFieldNumber is returned when an invalid field number is encountered.

Functions

func Marshal

func Marshal(v any) ([]byte, error)

Marshal binary-encodes the value.

func Unmarshal

func Unmarshal(b []byte, v any) error

Unmarshal decodes a binary-encoded value.

Types

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder decodes a binary-encoded object.

func NewDecoder

func NewDecoder(rd io.Reader, opts ...Option) *Decoder

NewDecoder creates a new decoder for the reader.

func (*Decoder) Decode

func (d *Decoder) Decode(v any) error

Decode decodes a value. The value must be a pointer to one of the supported types or a type that implements Unmarshaller or encoding.BinaryUnmarshaler.

Decode accepts reflect.Value and decodes into the underlying value.

func (*Decoder) DecodeBool

func (d *Decoder) DecodeBool() (bool, error)

DecodeBool reads a byte and interprets it as a boolean. DecodeBool returns an error if the value is not 0 or 1.

func (*Decoder) DecodeBytes

func (d *Decoder) DecodeBytes() ([]byte, error)

DecodeBytes reads a length-prefixed byte slice.

func (*Decoder) DecodeFloat

func (d *Decoder) DecodeFloat() (float64, error)

DecodeFloat reads a big-endian, 64-bit IEEE 754 floating point number.

func (*Decoder) DecodeHash

func (d *Decoder) DecodeHash() ([32]byte, error)

DecodeHash reads a hash.

func (*Decoder) DecodeInt

func (d *Decoder) DecodeInt() (int64, error)

DecodeInt reads a signed varint.

func (*Decoder) DecodeString

func (d *Decoder) DecodeString() (string, error)

DecodeString reads a length-prefixed string.

func (*Decoder) DecodeUint

func (d *Decoder) DecodeUint() (uint64, error)

DecodeUint reads an unsigned varint.

func (*Decoder) DecodeValue

func (d *Decoder) DecodeValue(v encoding.BinaryUnmarshaler) error

DecodeValue reads a length-prefixed byte slice and calls encoding.BinaryUnmarshaler.UnmarshalBinary.

func (*Decoder) DecodeValueFrom

func (d *Decoder) DecodeValueFrom(v unmarshallerV1From) error

DecodeValue reads a length-prefixed value unmarshalled with UnmarshalBinaryFrom.

func (*Decoder) DecodeValueV2

func (d *Decoder) DecodeValueV2(v Unmarshaller) error

DecodeValue calls [Unmarshaller.UnmarshalBinaryV2].

func (*Decoder) EndObject

func (d *Decoder) EndObject() error

EndObject marks the end of an object.

func (*Decoder) Field

func (d *Decoder) Field() (uint, error)

Field read a field identifier. If there are no remaining fields, Field returns io.EOF.

func (*Decoder) InField

func (d *Decoder) InField() bool

InField returns true if the decoder just read a field ID.

This is a hack and somewhat exposes the internal state of the decoder, but I can't think of a cleaner way of handling arrays (given that changing the encoding isn't an option).

func (*Decoder) NoField

func (d *Decoder) NoField() error

NoField indicates that the caller will decode a value that is *not* prefixed with a field identifier.

func (*Decoder) Reset

func (d *Decoder) Reset(rd io.Reader, opts ...Option)

Reset is equivalent to NewDecoder. Using Reset for repeatedly decoding objects reduces memory pressure.

func (*Decoder) StartObject

func (d *Decoder) StartObject() error

StartObject marks the start of an object.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder binary-encodes an object.

func NewEncoder

func NewEncoder(w io.Writer, opts ...Option) *Encoder

NewEncoder creates a new encoder for the writer.

func (*Encoder) Context

func (e *Encoder) Context() *traverse.Context[reflect.Value]

Context returns the traverse.Context associated with the encoder, creating a new one if needed. Context is provided for the caller, it is not used by the encoder.

func (*Encoder) Done

func (e *Encoder) Done() error

Done returns an error if the object was not completed.

func (*Encoder) Encode

func (e *Encoder) Encode(v any) error

Encode encodes a value. The value must be one of the supported types or a type that implements Marshaller or encoding.BinaryMarshaler.

Encode accepts reflect.Value and encodes the underlying value.

func (*Encoder) EncodeBool

func (e *Encoder) EncodeBool(v bool) error

EncodeBool writes a single byte, 0 (false) or 1 (true).

func (*Encoder) EncodeBytes

func (e *Encoder) EncodeBytes(v []byte) error

EncodeBytes writes the length (unsigned varint) followed by the data.

func (*Encoder) EncodeField

func (e *Encoder) EncodeField(n uint, v any) error

EncodeField calls Encoder.Field and Encoder.Encode.

func (*Encoder) EncodeFloat

func (e *Encoder) EncodeFloat(v float64) error

EncodeFloat writes a big-endian, 64-bit IEEE 754 floating point number.

func (*Encoder) EncodeHash

func (e *Encoder) EncodeHash(v [32]byte) error

EncodeHash writes the hash as bytes.

func (*Encoder) EncodeInt

func (e *Encoder) EncodeInt(v int64) error

EncodeInt writes a signed varint.

func (*Encoder) EncodeString

func (e *Encoder) EncodeString(v string) error

EncodeString writes the length (unsigned varint) followed by the data.

func (*Encoder) EncodeUint

func (e *Encoder) EncodeUint(v uint64) error

EncodeUint writes an unsigned varint.

func (*Encoder) EncodeValue

func (e *Encoder) EncodeValue(v encoding.BinaryMarshaler) error

EncodeValue calls encoding.BinaryMarshaler.MarshalBinary and writes it as a length-prefixed byte slice.

func (*Encoder) EncodeValueV2

func (e *Encoder) EncodeValueV2(v Marshaller) error

EncodeValueV2 calls [Marshaller.MarshalBinaryV2].

func (*Encoder) EndObject

func (e *Encoder) EndObject() error

EndObject marks the end of an object. If the object would otherwise be empty - if no fields were written - EndObject writes EmptyObject. If the object was nested, EndObject writes the length of the object then copies contents of the intermediate buffer to the output.

func (*Encoder) Field

func (e *Encoder) Field(n uint) error

Field writes a field identifier, verifying that it is a valid field number and is not out of sequence.

func (*Encoder) InField

func (e *Encoder) InField() bool

InField returns true if the decoder just read a field ID.

This is a hack and somewhat exposes the internal state of the decoder, but I can't think of a cleaner way of handling arrays (given that changing the encoding isn't an option).

func (*Encoder) NoField

func (e *Encoder) NoField() error

func (*Encoder) RepeatLastField

func (e *Encoder) RepeatLastField() error

RepeatLastField number repeats the last call to Encoder.Field. If Encoder.Encode has not been called since the last call to Encoder.Field, RepeatLastField is a no-op.

func (*Encoder) Reset

func (e *Encoder) Reset(w io.Writer, opts ...Option)

Reset is equivalent to NewEncoder. Using Reset for repeatedly encoding objects reduces memory pressure.

func (*Encoder) StartObject

func (e *Encoder) StartObject() error

StartObject marks the start of an object. StartObject does not write anything. If the object is nested, StartObject configures the encoder to write the nested object to a buffer retrieved from a Pool.

type LenWriter

type LenWriter interface {
	io.Writer
	Len() int
}

LenWriter is an io.Writer with a Len method.

type Marshaller

type Marshaller interface {
	MarshalBinaryV2(*Encoder) error
}

Marshaller is the interface implemented by types that can marshal a binary description of themselves.

type Option

type Option func(o *options)

Option is an option for constructing an Encoder or Decoder.

func IgnoreFieldOrder

func IgnoreFieldOrder() Option

IgnoreFieldOrder disables enforcement of field ordering.

func LeaveTrailing

func LeaveTrailing() Option

LeaveTrailing instructs the Decoder to leave trailing data untouched. Not applicable to Encoder.

func WithBufferPool

func WithBufferPool(pool *Pool[*bytes.Buffer]) Option

WithBufferPool sets the buffer pool of an Encoder. This can be used to reduce allocations. Not applicable to Decoder.

func WithContext

func WithContext(ctx *traverse.Context[reflect.Value]) Option

WithContext sets the context of an Encoder. Not applicable to Decoder. This can be used to preserve the context across nested invocations.

type Pool

type Pool[T any] sync.Pool

func NewPointerPool

func NewPointerPool[T any]() *Pool[*T]

func NewPool

func NewPool[T any](new func() T) *Pool[T]

func (*Pool[T]) Get

func (p *Pool[T]) Get() T

func (*Pool[T]) Put

func (p *Pool[T]) Put(v T)

type Unmarshaller

type Unmarshaller interface {
	UnmarshalBinaryV2(*Decoder) error
}

Unmarshaller is the interface implemented by types that can unmarshal a binary description of themselves.