picobuf

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 31, 2023 License: MIT Imports: 4 Imported by: 10

README

Picobuf

Picobuf is a minimal replacement for google.golang.org/protobuf with focusing on small binary size.

It does not support the whole Protocol Buffers feature set and features are added on as the need arises.

Currently the implementation is still in beta; the API can change.

How?

The usage is quite similar to the official implementation.

Ensure that you have installed the protoc protobuf compiler.

Second, install the picobuf tool with:

go install storj.io/picobuf/protoc-gen-pico@latest

Finally, add the go generate line that generates your picobuf encoding:

//go:generate protoc  -I. --pico_out=paths=source_relative:. example.proto

The example.proto looks like a usual proto file:

syntax = "proto3";
option go_package = "example.test/pb";

package example;

message Person {
  string  name = 1;
  Address address = 2;
}

message Address {
  string street = 1;
}

picobuf also supports some non-standard features:

syntax = "proto3";
option go_package = "example.test/pb";

package example;

message Timestamp {
  int64 seconds = 1;
  int32 nanos = 2;
}

message Person {
  string name = 1;

  Timestamp created_at = 5 [
    // "always_present" makes the field be a non-pointer.
    // This can help with allocation overhead. Of course, this will then lose the presence information.
    (pico.field).always_present = true,
    // "custom_type" allows to serialize the field into a different type, that was not generated by picobuf.
    (pico.field).custom_type = "time.Time",
    // "custom_serialize" allows to use a different type for serialization.
    // For example, we cannot add functions to `time.Time` so it needs custom serialization.
    (pico.field).custom_serialize = "storj.io/picobuf/picoconv.Timestamp"
  ];
}

Take a look at picoconv.Timestamp on how to implement custom serialization.

Why?

This library came out of a concern of creating smaller binaries. The official library, at the time of writing, relies on calling methods via reflection which switches the compiler into conservative mode. In conservative mode a lot of deadcode elimination doesn't work and hence the binaries get bloated.

Similarly, picobuf has optimizations and custom type support; which allows to use a different data representation from the official implementation.

Should I use it?

As a rule-of-thumb, you are probably better off using the official implementation. There are plenty of features this implementation doesn't support and in most cases the official implementation is completely fine.

Here are a few cases where you might consider using picobuf:

  • You need a small binary, for example:
    • WASM module
    • C library
    • embedded systems (TinyGo)
  • Custom serialization
    • this allows to avoid some allocations
  • Custom types
    • this allows to avoid some annoying conversions across the codebase

Note, you only get the binary size benefit when you do not have the official protobuf included in other ways. Or any other library that makes the compiler go into conservative mode (e.g. text/template or html/template).

Similarly, there are contraindications when it doesn't make sense to use it:

  • you need features such as:
    • any messages
    • reflection
    • json
    • descriptors

Benchmark

sizebench contains tests for the binary size.

go1.19.5
goos: linux
goarch: amd64
// A basic "fmt.Println" executable size.
ExeBase               1     1814.766 KB
// A basic "fmt.Println" executable size, but contains code
// that switches the compiler into conservative mode.
ExeBaseConserv        1     2549.444 KB
// Picobuf generated binary size (non-conservative).
ExePico/Sml           1     2000.987 KB      186.221 KB-Δ
// Picobuf generated binary size (conservative).
ExePicoConserv/Sml    1     2700.553 KB      151.109 KB-Δ
// Protobuf generated binary size (conservative).
ExeProto/Sml          1     4978.815 KB     2429.371 KB-Δ

The message definitions are in msg-sml.proto.

As can be seen, the protobuf binary is ~3MB bigger than picobuf in non-conservative mode; and it's ~2.3MB bigger than picobuf in conservative mode.

Documentation

Overview

Package picobuf is a light replacement for a subset of protobuf.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Marshal 

func Marshal(msg Message) ([]byte, error)

Marshal encodes msg as bytes.

func MarshalBuffer 

func MarshalBuffer(msg Message, buffer []byte) ([]byte, error)

MarshalBuffer encodes msg as bytes with buffer.

func Unmarshal 

func Unmarshal(data []byte, msg Message) error

Unmarshal decodes msg as bytes.

Types

type CustomType 

type CustomType interface {
	PicoEncode(*Encoder, FieldNumber)
	PicoDecode(*Decoder, FieldNumber)
}

CustomType defines methods that are used for custom encode or decode behaviors.

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder implements decoding of protobuf messages.

func NewDecoder 

func NewDecoder(data []byte) *Decoder

NewDecoder returns a new Decoder.

func (*Decoder) Bool 

func (dec *Decoder) Bool(field FieldNumber, v *bool)

Bool decodes bool protobuf type.

func (*Decoder) Bytes 

func (dec *Decoder) Bytes(field FieldNumber, v *[]byte)

Bytes decodes bytes protobuf type.

func (*Decoder) Double 

func (dec *Decoder) Double(field FieldNumber, v *float64)

Double decodes double protobuf type.

func (*Decoder) Err 

func (dec *Decoder) Err() error

Err returns error that occurred during decoding.

func (*Decoder) Fail 

func (dec *Decoder) Fail(field FieldNumber, msg string)

Fail fails the decoding process.

func (*Decoder) Fixed32 

func (dec *Decoder) Fixed32(field FieldNumber, v *uint32)

Fixed32 decodes fixed32 protobuf type.

func (*Decoder) Fixed64 

func (dec *Decoder) Fixed64(field FieldNumber, v *uint64)

Fixed64 decodes fixed64 protobuf type.

func (*Decoder) Float 

func (dec *Decoder) Float(field FieldNumber, v *float32)

Float decodes float protobuf type.

func (*Decoder) Int32 

func (dec *Decoder) Int32(field FieldNumber, v *int32)

Int32 decodes int32 protobuf type.

func (*Decoder) Int64 

func (dec *Decoder) Int64(field FieldNumber, v *int64)

Int64 decodes int64 protobuf type.

func (*Decoder) Loop 

func (dec *Decoder) Loop(fn func(*Decoder))

Loop loops fields until all messages have been processed.

func (*Decoder) Message 

func (dec *Decoder) Message(field FieldNumber, fn func(*Decoder))

Message decodes a message.

func (*Decoder) PendingField 

func (dec *Decoder) PendingField() FieldNumber

PendingField returns the next field number in the stream.

func (*Decoder) PresentMessage 

func (dec *Decoder) PresentMessage(field FieldNumber, fn func(*Decoder))

PresentMessage decodes an always present message.

func (*Decoder) RepeatedBool 

func (dec *Decoder) RepeatedBool(field FieldNumber, v *[]bool)

RepeatedBool decodes repeated bool protobuf type.

func (*Decoder) RepeatedBytes 

func (dec *Decoder) RepeatedBytes(field FieldNumber, v *[][]byte)

RepeatedBytes decodes repeated bytes protobuf type.

func (*Decoder) RepeatedDouble 

func (dec *Decoder) RepeatedDouble(field FieldNumber, v *[]float64)

RepeatedDouble decodes repeated double protobuf type.

func (*Decoder) RepeatedEnum 

func (dec *Decoder) RepeatedEnum(field FieldNumber, add func(x int32))

RepeatedEnum decodes a repeated enumeration.

func (*Decoder) RepeatedFixed32 

func (dec *Decoder) RepeatedFixed32(field FieldNumber, v *[]uint32)

RepeatedFixed32 decodes repeated fixed32 protobuf type.

func (*Decoder) RepeatedFixed64 

func (dec *Decoder) RepeatedFixed64(field FieldNumber, v *[]uint64)

RepeatedFixed64 decodes repeated fixed64 protobuf type.

func (*Decoder) RepeatedFloat 

func (dec *Decoder) RepeatedFloat(field FieldNumber, v *[]float32)

RepeatedFloat decodes repeated float protobuf type.

func (*Decoder) RepeatedInt32 

func (dec *Decoder) RepeatedInt32(field FieldNumber, v *[]int32)

RepeatedInt32 decodes repeated int32 protobuf type.

func (*Decoder) RepeatedInt64 

func (dec *Decoder) RepeatedInt64(field FieldNumber, v *[]int64)

RepeatedInt64 decodes repeated int64 protobuf type.

func (*Decoder) RepeatedMessage 

func (dec *Decoder) RepeatedMessage(field FieldNumber, fn func(c *Decoder))

RepeatedMessage decodes a message.

func (*Decoder) RepeatedSfixed32 

func (dec *Decoder) RepeatedSfixed32(field FieldNumber, v *[]int32)

RepeatedSfixed32 decodes repeated sfixed32 protobuf type.

func (*Decoder) RepeatedSfixed64 

func (dec *Decoder) RepeatedSfixed64(field FieldNumber, v *[]int64)

RepeatedSfixed64 decodes repeated sfixed64 protobuf type.

func (*Decoder) RepeatedSint32 

func (dec *Decoder) RepeatedSint32(field FieldNumber, v *[]int32)

RepeatedSint32 decodes repeated sint32 protobuf type.

func (*Decoder) RepeatedSint64 

func (dec *Decoder) RepeatedSint64(field FieldNumber, v *[]int64)

RepeatedSint64 decodes repeated sint64 protobuf type.

func (*Decoder) RepeatedString 

func (dec *Decoder) RepeatedString(field FieldNumber, v *[]string)

RepeatedString decodes repeated string protobuf type.

func (*Decoder) RepeatedUint32 

func (dec *Decoder) RepeatedUint32(field FieldNumber, v *[]uint32)

RepeatedUint32 decodes repeated uint32 protobuf type.

func (*Decoder) RepeatedUint64 

func (dec *Decoder) RepeatedUint64(field FieldNumber, v *[]uint64)

RepeatedUint64 decodes repeated uint64 protobuf type.

func (*Decoder) Sfixed32 

func (dec *Decoder) Sfixed32(field FieldNumber, v *int32)

Sfixed32 decodes sfixed32 protobuf type.

func (*Decoder) Sfixed64 

func (dec *Decoder) Sfixed64(field FieldNumber, v *int64)

Sfixed64 decodes sfixed64 protobuf type.

func (*Decoder) Sint32 

func (dec *Decoder) Sint32(field FieldNumber, v *int32)

Sint32 decodes sint32 protobuf type.

func (*Decoder) Sint64 

func (dec *Decoder) Sint64(field FieldNumber, v *int64)

Sint64 decodes sint64 protobuf type.

func (*Decoder) String 

func (dec *Decoder) String(field FieldNumber, v *string)

String decodes string protobuf type.

func (*Decoder) Uint32 

func (dec *Decoder) Uint32(field FieldNumber, v *uint32)

Uint32 decodes uint32 protobuf type.

func (*Decoder) Uint64 

func (dec *Decoder) Uint64(field FieldNumber, v *uint64)

Uint64 decodes uint64 protobuf type.

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder implements encoding of protobuf format.

func NewEncoder 

func NewEncoder() *Encoder

NewEncoder creates a new Encoder.

func NewEncoderBuffer 

func NewEncoderBuffer(buffer []byte) *Encoder

NewEncoderBuffer creates a new encoder using a preallocated buffer.

func (*Encoder) AlwaysAnyBytes 

func (enc *Encoder) AlwaysAnyBytes(field FieldNumber, fn func()) bool

AlwaysAnyBytes encodes field as Bytes and handles encoding the length.

func (*Encoder) AlwaysMessage 

func (enc *Encoder) AlwaysMessage(field FieldNumber, fn func(enc *Encoder) bool)

AlwaysMessage encodes an message always.

func (*Encoder) Bool 

func (enc *Encoder) Bool(field FieldNumber, v *bool)

Bool encodes non-default bool protobuf type.

func (*Encoder) Buffer 

func (enc *Encoder) Buffer() []byte

Buffer returns the encoded internal buffer.

func (*Encoder) Bytes 

func (enc *Encoder) Bytes(field FieldNumber, v *[]byte)

Bytes encodes non-default bytes protobuf type.

func (*Encoder) Double 

func (enc *Encoder) Double(field FieldNumber, v *float64)

Double encodes non-default double protobuf type.

func (*Encoder) Fixed32 

func (enc *Encoder) Fixed32(field FieldNumber, v *uint32)

Fixed32 encodes non-default fixed32 protobuf type.

func (*Encoder) Fixed64 

func (enc *Encoder) Fixed64(field FieldNumber, v *uint64)

Fixed64 encodes non-default fixed64 protobuf type.

func (*Encoder) Float 

func (enc *Encoder) Float(field FieldNumber, v *float32)

Float encodes non-default float protobuf type.

func (*Encoder) Int32 

func (enc *Encoder) Int32(field FieldNumber, v *int32)

Int32 encodes non-default int32 protobuf type.

func (*Encoder) Int64 

func (enc *Encoder) Int64(field FieldNumber, v *int64)

Int64 encodes non-default int64 protobuf type.

func (*Encoder) Message 

func (enc *Encoder) Message(field FieldNumber, fn func(enc *Encoder) bool)

Message decodes a message.

func (*Encoder) PresentMessage 

func (enc *Encoder) PresentMessage(field FieldNumber, fn func(enc *Encoder) bool)

PresentMessage encodes an always present message.

func (*Encoder) RepeatedBool 

func (enc *Encoder) RepeatedBool(field FieldNumber, v *[]bool)

RepeatedBool encodes non-empty repeated bool protobuf type.

func (*Encoder) RepeatedBytes 

func (enc *Encoder) RepeatedBytes(field FieldNumber, v *[][]byte)

RepeatedBytes encodes non-empty repeated bytes protobuf type.

func (*Encoder) RepeatedDouble 

func (enc *Encoder) RepeatedDouble(field FieldNumber, v *[]float64)

RepeatedDouble encodes non-empty repeated double protobuf type.

func (*Encoder) RepeatedEnum 

func (enc *Encoder) RepeatedEnum(field FieldNumber, n int, fn func(index uint) int32)

RepeatedEnum encodes a repeated enumeration.

func (*Encoder) RepeatedFixed32 

func (enc *Encoder) RepeatedFixed32(field FieldNumber, v *[]uint32)

RepeatedFixed32 encodes non-empty repeated fixed32 protobuf type.

func (*Encoder) RepeatedFixed64 

func (enc *Encoder) RepeatedFixed64(field FieldNumber, v *[]uint64)

RepeatedFixed64 encodes non-empty repeated fixed64 protobuf type.

func (*Encoder) RepeatedFloat 

func (enc *Encoder) RepeatedFloat(field FieldNumber, v *[]float32)

RepeatedFloat encodes non-empty repeated float protobuf type.

func (*Encoder) RepeatedInt32 

func (enc *Encoder) RepeatedInt32(field FieldNumber, v *[]int32)

RepeatedInt32 encodes non-empty repeated int32 protobuf type.

func (*Encoder) RepeatedInt64 

func (enc *Encoder) RepeatedInt64(field FieldNumber, v *[]int64)

RepeatedInt64 encodes non-empty repeated int64 protobuf type.

func (*Encoder) RepeatedSfixed32 

func (enc *Encoder) RepeatedSfixed32(field FieldNumber, v *[]int32)

RepeatedSfixed32 encodes non-empty repeated sfixed32 protobuf type.

func (*Encoder) RepeatedSfixed64 

func (enc *Encoder) RepeatedSfixed64(field FieldNumber, v *[]int64)

RepeatedSfixed64 encodes non-empty repeated sfixed64 protobuf type.

func (*Encoder) RepeatedSint32 

func (enc *Encoder) RepeatedSint32(field FieldNumber, v *[]int32)

RepeatedSint32 encodes non-empty repeated sint32 protobuf type.

func (*Encoder) RepeatedSint64 

func (enc *Encoder) RepeatedSint64(field FieldNumber, v *[]int64)

RepeatedSint64 encodes non-empty repeated sint64 protobuf type.

func (*Encoder) RepeatedString 

func (enc *Encoder) RepeatedString(field FieldNumber, v *[]string)

RepeatedString encodes non-empty repeated string protobuf type.

func (*Encoder) RepeatedUint32 

func (enc *Encoder) RepeatedUint32(field FieldNumber, v *[]uint32)

RepeatedUint32 encodes non-empty repeated uint32 protobuf type.

func (*Encoder) RepeatedUint64 

func (enc *Encoder) RepeatedUint64(field FieldNumber, v *[]uint64)

RepeatedUint64 encodes non-empty repeated uint64 protobuf type.

func (*Encoder) Sfixed32 

func (enc *Encoder) Sfixed32(field FieldNumber, v *int32)

Sfixed32 encodes non-default sfixed32 protobuf type.

func (*Encoder) Sfixed64 

func (enc *Encoder) Sfixed64(field FieldNumber, v *int64)

Sfixed64 encodes non-default sfixed64 protobuf type.

func (*Encoder) Sint32 

func (enc *Encoder) Sint32(field FieldNumber, v *int32)

Sint32 encodes non-default sint32 protobuf type.

func (*Encoder) Sint64 

func (enc *Encoder) Sint64(field FieldNumber, v *int64)

Sint64 encodes non-default sint64 protobuf type.

func (*Encoder) String 

func (enc *Encoder) String(field FieldNumber, v *string)

String encodes non-default string protobuf type.

func (*Encoder) Uint32 

func (enc *Encoder) Uint32(field FieldNumber, v *uint32)

Uint32 encodes non-default uint32 protobuf type.

func (*Encoder) Uint64 

func (enc *Encoder) Uint64(field FieldNumber, v *uint64)

Uint64 encodes non-default uint64 protobuf type.

type FieldNumber 

type FieldNumber protowire.Number

FieldNumber corresponds to a protobuf field number.

func (FieldNumber) IsValid 

func (field FieldNumber) IsValid() bool

IsValid returns whether the field number is in correct range.

type Message 

type Message interface {
	Encode(*Encoder) bool
	Decode(*Decoder)
}

Message is an interface that all generated messages implement.

Source Files

View all Source files

Directories

Path Synopsis
internal
bitset
dependencytest
generatecoder
picotest
Package picotest holds generated types for testing.
 Package picotest holds generated types for testing.
picotest/pic
protocompat
Package protocompat holds compatibility tests with protobuf.
 Package protocompat holds compatibility tests with protobuf.
protocompat/pico
protocompat/prot
sizebench
sizebench/base-fmt
sizebench/basz
sizebench/basz-fmt
sizebench/pico
sizebench/pico-one
sizebench/pico-sml
sizebench/pico-two
sizebench/pico/one
sizebench/pico/sml
sizebench/pico/two
sizebench/picz-one
sizebench/picz-sml
sizebench/picz-two
sizebench/prot
sizebench/prot-one
sizebench/prot-sml
sizebench/prot-two
sizebench/prot/one
sizebench/prot/sml
sizebench/prot/two

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.