protobuf

README

ProtoBuf

https://stackoverflow.com/questions/41348512/protobuf-unmarshal-unknown

Embedded message wire type

Currently Protocol Buffers is an inherently flawed protocol, in that it is not self describing. This is evidenced by the fact that "type 2" wire type [1], can be any of these values:

1. string
2. byte slice
3. embedded message

Its pretty straight forward to check for string or byte slice [2]. Also, not all input is a valid message. That gives you this result:

valid string	valid message	result
no	no	byte slice
no	yes	message or byte slice
yes	no	string
yes	yes	message or string

Ideally the protocol would have an extra wire type, for example "type 6", to designate embedded messages. But without that, its not possible in every case to know the structure of the input, unless you have a schema. Other protocols such as JSON are self describing, nothing is stopping ProtoBuf from being self describing as well. Granted, you lose the field names, but even just having the field numbers and values, with a defined structure as read from the input, would be quite useful. Quoting from the spec:

in other words, the last three bits of the number store the wire type

So that gives you this currently:

000 Varint
001 64-bit
010 Length-delimited
011 Start group
100 End group
101 32-bit
110
111

So you have at least two extra types that could be added. Even a single extra type for embedded messages would essentially make ProtoBuf self describing.

1. https://developers.google.com/protocol-buffers/docs/encoding
2. https://github.com/89z/format/blob/v1.34.0/format.go#L17-L35

Why not use generic add?

If you use generic add with method interface, it would allow recurive slices. We could solve this by also adding type interface, but its probably simpler to just not use generics for this.

Why not implement a decoder?

I tried, but Unmarshal is way faster:

Benchmark_Decode-12       367295 ns/op         1481091 B/op       2588 allocs/op
Benchmark_Unmarshal-12     86033 ns/op           80245 B/op       1104 allocs/op

Documentation

Index

• type Bytes
• type Encoder
• type Fixed32
• type Fixed64
• type Message
  • func Unmarshal(buf []byte) (Message, error)
  • func (m Message) Add(num Number, value Message) error
  • func (m Message) Add_Fixed32(num Number, value uint32) error
  • func (m Message) Add_Fixed64(num Number, value uint64) error
  • func (m Message) Add_String(num Number, value string) error
  • func (m Message) Add_Varint(num Number, value uint64) error
  • func (m Message) Get(num Number) Message
  • func (m Message) Get_Bytes(num Number) ([]byte, error)
  • func (m Message) Get_Fixed64(num Number) (uint64, error)
  • func (m Message) Get_Messages(num Number) []Message
  • func (m Message) Get_String(num Number) (string, error)
  • func (m Message) Get_Varint(num Number) (uint64, error)
  • func (m Message) GoString() string
  • func (m Message) Marshal() []byte
• type Number
• type Raw
  • func (m Raw) GoString() string
• type Slice
• type String
• type Varint

Types

type Bytes

type Bytes []byte

type Encoder

type Encoder interface {
	// contains filtered or unexported methods
}

type Fixed32

type Fixed32 uint32

type Fixed64

type Fixed64 uint64

type Message

type Message map[Number]Encoder

func Unmarshal

func Unmarshal(buf []byte) (Message, error)

func (Message) Add

func (m Message) Add(num Number, value Message) error

func (Message) Add_Fixed32

func (m Message) Add_Fixed32(num Number, value uint32) error

func (Message) Add_Fixed64

func (m Message) Add_Fixed64(num Number, value uint64) error

func (Message) Add_String

func (m Message) Add_String(num Number, value string) error

func (Message) Add_Varint

func (m Message) Add_Varint(num Number, value uint64) error

func (Message) Get

func (m Message) Get(num Number) Message

func (Message) Get_Bytes

func (m Message) Get_Bytes(num Number) ([]byte, error)

func (Message) Get_Fixed64

func (m Message) Get_Fixed64(num Number) (uint64, error)

func (Message) Get_Messages

func (m Message) Get_Messages(num Number) []Message

func (Message) Get_String

func (m Message) Get_String(num Number) (string, error)

func (Message) Get_Varint

func (m Message) Get_Varint(num Number) (uint64, error)

func (Message) GoString

func (m Message) GoString() string

If you need indent, just use this with `go fmt`.

func (Message) Marshal

func (m Message) Marshal() []byte

type Number

type Number = protowire.Number

type Raw

type Raw struct {
	Bytes   []byte
	String  string
	Message Message
}

func (Raw) GoString

func (m Raw) GoString() string

type Slice

type Slice[T Encoder] []T

type String

type String string

type Varint

type Varint uint64