protobuf_go_lite

README

protobuf-go-lite

protobuf-go-lite is a stripped-down version of the protobuf-go code generator modified to work without reflection and merged with vtprotobuf to provide modular features with static code generation for marshal/unmarshal, size, clone, and equal. It bundles a fork of protoc-gen-go-json for JSON.

Static code generation without reflection is more efficient at runtime and results in smaller code binaries. It also provides better support for tinygo which has limited reflection support.

protobuf-go-lite does not support fieldmasks and extensions.

Ecosystem

Lightweight Protobuf 3 RPCs are implemented in StaRPC for Go and TypeScript.

protoc-gen-doc is recommended for generating documentation.

protobuf-es-lite is recommended for lightweight TypeScript protobufs.

Protobuf

protocol buffers are a cross-platform cross-language message serialization format. Protobuf is a language for specifying the schema for structured data. This schema is compiled into language specific bindings. This project provides both a tool to generate Go code for the protocol buffer language, and also the runtime implementation to handle serialization of messages in Go.

See the protocol buffer developer guide for more information about protocol buffers themselves.

Example

See the protobuf-project template for an example of how to use this package and vtprotobuf together with protowrap to generate protobufs for your project.

This package is available at github.com/aperturerobotics/protobuf-go-lite.

Package index

Summary of the packages provided by this module:

• compiler/protogen: Package protogen provides support for writing protoc plugins.
• cmd/protoc-gen-go-lite: The protoc-gen-go-lite binary is a protoc plugin to generate a Go protocol buffer package.

Usage

1. Install protoc-gen-go-lite:

   go install github.com/aperturerobotics/protobuf-go-lite/cmd/protoc-gen-go-lite@latest
   

2. Update your protoc generator to use the new plug-in.

   for name in $(PROTO_SRC_NAMES); do \
       protoc \
         --plugin protoc-gen-go-lite="${GOBIN}/protoc-gen-go-lite"
         --go-lite_out=.  \
         --go-lite_opt=features=marshal+unmarshal+size+equal+clone \
       proto/$${name}.proto; \
   done
   

protobuf-go-lite replaces protoc-gen-go and protoc-gen-go-vtprotobuf and should not be used with those generators.

Check out the template for a quick start!

Available features

The following additional features from vtprotobuf can be enabled:

• size: generates a func (p *YourProto) SizeVT() int helper that behaves identically to calling proto.Size(p) on the message, except the size calculation is fully unrolled and does not use reflection. This helper function can be used directly, and it'll also be used by the marshal codegen to ensure the destination buffer is properly sized before ProtoBuf objects are marshalled to it.

• equal: generates the following helper methods

  • func (this *YourProto) EqualVT(that *YourProto) bool: this function behaves almost identically to calling proto.Equal(this, that) on messages, except the equality calculation is fully unrolled and does not use reflection. This helper function can be used directly.

  • func (this *YourProto) EqualMessageVT(thatMsg any) bool: this function behaves like the above this.EqualVT(that), but allows comparing against arbitrary proto messages. If thatMsg is not of type *YourProto, false is returned. The uniform signature provided by this method allows accessing this method via type assertions even if the message type is not known at compile time. This allows implementing a generic func EqualVT(proto.Message, proto.Message) bool without reflection.

• marshal: generates the following helper methods

  • func (p *YourProto) MarshalVT() ([]byte, error): this function behaves identically to calling proto.Marshal(p), except the actual marshalling has been fully unrolled and does not use reflection or allocate memory. This function simply allocates a properly sized buffer by calling SizeVT on the message and then uses MarshalToSizedBufferVT to marshal to it.

  • func (p *YourProto) MarshalToVT(data []byte) (int, error): this function can be used to marshal a message to an existing buffer. The buffer must be large enough to hold the marshalled message, otherwise this function will panic. It returns the number of bytes marshalled. This function is useful e.g. when using memory pooling to re-use serialization buffers.

  • func (p *YourProto) MarshalToSizedBufferVT(data []byte) (int, error): this function behaves like MarshalTo but expects that the input buffer has the exact size required to hold the message, otherwise it will panic.

• marshal_strict: generates the following helper methods

  • func (p *YourProto) MarshalVTStrict() ([]byte, error): this function behaves like MarshalVT, except fields are marshalled in a strict order by field's numbers they were declared in .proto file.

  • func (p *YourProto) MarshalToVTStrict(data []byte) (int, error): this function behaves like MarshalToVT, except fields are marshalled in a strict order by field's numbers they were declared in .proto file.

  • func (p *YourProto) MarshalToSizedBufferVTStrict(data []byte) (int, error): this function behaves like MarshalToSizedBufferVT, except fields are marshalled in a strict order by field's numbers they were declared in .proto file.

• unmarshal: generates a func (p *YourProto) UnmarshalVT(data []byte) that behaves similarly to calling proto.Unmarshal(data, p) on the message, except the unmarshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible. If the receiver p is not fully zeroed-out, the unmarshal call will actually behave like proto.Merge(data, p). This is because the proto.Unmarshal in the ProtoBuf API is implemented by resetting the destination message and then calling proto.Merge on it. To ensure proper Unmarshal semantics, ensure you've called proto.Reset on your message before calling UnmarshalVT, or that your message has been newly allocated.

• unmarshal_unsafe generates a func (p *YourProto) UnmarshalVTUnsafe(data []byte) that behaves like UnmarshalVT, except it unsafely casts slices of data to bytes and string fields instead of copying them to newly allocated arrays, so that it performs less allocations. Data received from the wire has to be left untouched for the lifetime of the message. Otherwise, the message's bytes and string fields can be corrupted.

• clone: generates the following helper methods

  • func (p *YourProto) CloneVT() *YourProto: this function behaves similarly to calling proto.Clone(p) on the message, except the cloning is performed by unrolled codegen without using reflection. If the receiver p is nil a typed nil is returned.

  • func (p *YourProto) CloneMessageVT() any: this function behaves like the above p.CloneVT(), but provides a uniform signature in order to be accessible via type assertions even if the type is not known at compile time. This allows implementing a generic func CloneMessageVT() any without reflection. If the receiver p is nil, a typed nil pointer of the message type will be returned inside a any interface.

• json: generates the following helper methods

  • func (p *YourProto) UnmarshalJSON(data []byte) error behaves similarly to calling protojson.Unmarshal(data, p) on the message, except the unmarshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible (with json-iterator/go). If the receiver p is not fully zeroed-out, the unmarshal call will actually behave like proto.Merge(data, p). To ensure proper Unmarshal semantics, ensure you've called proto.Reset on your message before calling UnmarshalJSON, or that your message has been newly allocated.

  • func (p *YourProto) UnmarshalJSONValue(val *fastjson.Value) error unmarshals a *fastjson.Value.

  • func (p *YourProto) MarshalJSON() ([]byte, error) behaves similarly to calling protojson.Marshal(p) on the message, except the marshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible (with json-iterator/go).

  • Adding a //protobuf-go-lite:disable-json comment before a message or enum will disable the json marshaler / unmarshaler.

License

BSD-3

Documentation

Index

• Variables
• func AppendVarint(b []byte, v uint64) []byte
• func CloneVTSlice[S ~[]E, E CloneVT[E]](s S) S
• func CompareComparable[T comparable]() func(t1, t2 T) bool
• func CompareEqualVT[T EqualVT[T]]() func(t1, t2 T) bool
• func ConsumeVarint(b []byte) (v uint64, n int)
• func EncodeVarint(dAtA []byte, offset int, v uint64) int
• func IsEqualVT[T EqualVT[T]](t1, t2 T) bool
• func IsEqualVTSlice[S ~[]E, E EqualVT[E]](s1, s2 S) bool
• func SizeOfVarint(x uint64) (n int)
• func SizeOfZigzag(x uint64) (n int)
• func Skip(dAtA []byte) (n int, err error)
• type CloneMessage
• type CloneVT
• type EqualVT
• type Message

Variables

var (
	// ErrInvalidLength is returned when decoding a negative length.
	ErrInvalidLength = errors.New("proto: negative length found during unmarshaling")
	// ErrIntOverflow is returned when decoding a varint representation of an integer that overflows 64 bits.
	ErrIntOverflow = errors.New("proto: integer overflow")
	// ErrUnexpectedEndOfGroup is returned when decoding a group end without a corresponding group start.
	ErrUnexpectedEndOfGroup = errors.New("proto: unexpected end of group")
)

Functions

func AppendVarint

func AppendVarint(b []byte, v uint64) []byte

AppendVarint appends v to b as a varint-encoded uint64.

func CloneVTSlice

func CloneVTSlice[S ~[]E, E CloneVT[E]](s S) S

CloneVTSlice clones a slice of CloneVT messages.

func CompareComparable

func CompareComparable[T comparable]() func(t1, t2 T) bool

CompareComparable returns a compare function to compare two comparable types.

func CompareEqualVT

func CompareEqualVT[T EqualVT[T]]() func(t1, t2 T) bool

CompareEqualVT returns a compare function to compare two VTProtobuf messages.

func ConsumeVarint

func ConsumeVarint(b []byte) (v uint64, n int)

ConsumeVarint parses b as a varint-encoded uint64, reporting its length. This returns -1 upon any error, -1 for parse error and -2 for overflow.

func EncodeVarint

func EncodeVarint(dAtA []byte, offset int, v uint64) int

EncodeVarint encodes a uint64 into a varint-encoded byte slice and returns the offset of the encoded value. The provided offset is the offset after the last byte of the encoded value.

func IsEqualVT

func IsEqualVT[T EqualVT[T]](t1, t2 T) bool

IsEqualVT checks if two EqualVT objects are equal.

func IsEqualVTSlice

func IsEqualVTSlice[S ~[]E, E EqualVT[E]](s1, s2 S) bool

IsEqualVTSlice checks if two slices of EqualVT messages are equal.

func SizeOfVarint

func SizeOfVarint(x uint64) (n int)

SizeOfVarint returns the size of the varint-encoded value.

func SizeOfZigzag

func SizeOfZigzag(x uint64) (n int)

SizeOfZigzag returns the size of the zigzag-encoded value.

func Skip

func Skip(dAtA []byte) (n int, err error)

Skip the first record of the byte slice and return the offset of the next record.

Types

type CloneMessage

type CloneMessage interface {
	// Message extends the base message type.
	Message
	// CloneMessageVT clones the object.
	CloneMessageVT() CloneMessage
}

CloneMessage is a message with a CloneMessage function.

type CloneVT

type CloneVT[T comparable] interface {
	comparable
	// CloneMessage is the non-generic clone interface.
	CloneMessage
	// CloneVT clones the object.
	CloneVT() T
}

CloneVT is a message with a CloneVT function (VTProtobuf).

type EqualVT

type EqualVT[T comparable] interface {
	comparable
	// EqualVT compares against the other message for equality.
	EqualVT(other T) bool
}

EqualVT is a message with a EqualVT function (VTProtobuf).

type Message

type Message interface {
	// SizeVT returns the size of the message when marshaled.
	SizeVT() int
	// MarshalToSizedBufferVT marshals to a buffer that already is SizeVT bytes long.
	MarshalToSizedBufferVT(dAtA []byte) (int, error)
	// MarshalVT marshals the message with vtprotobuf.
	MarshalVT() ([]byte, error)
	// UnmarshalVT unmarshals the message object with vtprotobuf.
	UnmarshalVT(data []byte) error
	// Reset resets the message.
	Reset()
}

Message is the base vtprotobuf message marshal/unmarshal interface.