Documentation
¶
Overview ¶
The marshalto plugin generates a Marshal and MarshalTo method for each message. The `Marshal() ([]byte, error)` method results in the fact that the message implements the Marshaler interface. This allows proto.Marshal to be faster by calling the generated Marshal method rather than using reflect to Marshal the struct.
If is enabled by the following extensions:
- marshaler
- marshaler_all
Or the following extensions:
- unsafe_marshaler
- unsafe_marshaler_all
That is if you want to use the unsafe package in your generated code. The speed up using the unsafe package is not very significant.
The generation of marshalling tests are enabled using one of the following extensions:
- testgen
- testgen_all
And benchmarks given it is enabled using one of the following extensions:
- benchgen
- benchgen_all
Let us look at:
github.com/gogo/protobuf/test/example/example.proto
Btw all the output can be seen at:
github.com/gogo/protobuf/test/example/*
The following message:
option (gogoproto.marshaler_all) = true;
message B {
option (gogoproto.description) = true;
optional A A = 1 [(gogoproto.nullable) = false, (gogoproto.embed) = true];
repeated bytes G = 2 [(gogoproto.customtype) = "github.com/gogo/protobuf/test/custom.Uint128", (gogoproto.nullable) = false];
}
given to the marshalto plugin, will generate the following code:
func (m *B) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalToSizedBuffer(dAtA[:size])
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
func (m *B) MarshalTo(dAtA []byte) (int, error) {
size := m.Size()
return m.MarshalToSizedBuffer(dAtA[:size])
}
func (m *B) MarshalToSizedBuffer(dAtA []byte) (int, error) {
i := len(dAtA)
_ = i
var l int
_ = l
if m.XXX_unrecognized != nil {
i -= len(m.XXX_unrecognized)
copy(dAtA[i:], m.XXX_unrecognized)
}
if len(m.G) > 0 {
for iNdEx := len(m.G) - 1; iNdEx >= 0; iNdEx-- {
{
size := m.G[iNdEx].Size()
i -= size
if _, err := m.G[iNdEx].MarshalTo(dAtA[i:]); err != nil {
return 0, err
}
i = encodeVarintExample(dAtA, i, uint64(size))
}
i--
dAtA[i] = 0x12
}
}
{
size, err := m.A.MarshalToSizedBuffer(dAtA[:i])
if err != nil {
return 0, err
}
i -= size
i = encodeVarintExample(dAtA, i, uint64(size))
}
i--
dAtA[i] = 0xa
return len(dAtA) - i, nil
}
As shown above Marshal calculates the size of the not yet marshalled message and allocates the appropriate buffer. This is followed by calling the MarshalToSizedBuffer method which requires a preallocated buffer, and marshals backwards. The MarshalTo method allows a user to rather preallocated a reusable buffer.
The Size method is generated using the size plugin and the gogoproto.sizer, gogoproto.sizer_all extensions. The user can also using the generated Size method to check that his reusable buffer is still big enough.
The generated tests and benchmarks will keep you safe and show that this is really a significant speed improvement.
An additional message-level option `stable_marshaler` (and the file-level option `stable_marshaler_all`) exists which causes the generated marshalling code to behave deterministically. Today, this only changes the serialization of maps; they are serialized in sort order.
The size plugin generates a Size or ProtoSize method for each message. This is useful with the MarshalTo method generated by the marshalto plugin and the gogoproto.marshaler and gogoproto.marshaler_all extensions.
It is enabled by the following extensions:
- sizer
- sizer_all
- protosizer
- protosizer_all
The size plugin also generates a test given it is enabled using one of the following extensions:
- testgen
- testgen_all
And a benchmark given it is enabled using one of the following extensions:
- benchgen
- benchgen_all
Let us look at:
github.com/gogo/protobuf/test/example/example.proto
Btw all the output can be seen at:
github.com/gogo/protobuf/test/example/*
The following message:
option (gogoproto.sizer_all) = true;
message B {
option (gogoproto.description) = true;
optional A A = 1 [(gogoproto.nullable) = false, (gogoproto.embed) = true];
repeated bytes G = 2 [(gogoproto.customtype) = "github.com/gogo/protobuf/test/custom.Uint128", (gogoproto.nullable) = false];
}
given to the size plugin, will generate the following code:
func (m *B) XSize() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = m.A.XSize()
n += 1 + l + sovExample(uint64(l))
if len(m.G) > 0 {
for _, e := range m.G {
l = e.XSize()
n += 1 + l + sovExample(uint64(l))
}
}
if m.XXX_unrecognized != nil {
n += len(m.XXX_unrecognized)
}
return n
}
and the following test code:
func TestBSize(t *testing5.T) {
popr := math_rand5.New(math_rand5.NewSource(time5.Now().UnixNano()))
p := NewPopulatedB(popr, true)
dAtA, err := github_com_gogo_protobuf_proto2.Marshal(p)
if err != nil {
panic(err)
}
size := g.XSize()
if len(dAtA) != size {
t.Fatalf("size %v != marshalled size %v", size, len(dAtA))
}
}
func BenchmarkBSize(b *testing5.B) {
popr := math_rand5.New(math_rand5.NewSource(616))
total := 0
pops := make([]*B, 1000)
for i := 0; i < 1000; i++ {
pops[i] = NewPopulatedB(popr, false)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
total += pops[i%1000].Size()
}
b.SetBytes(int64(total / b.N))
}
The sovExample function is a size of varint function for the example.pb.go file.
The unmarshal plugin generates a Unmarshal method for each message. The `Unmarshal([]byte) error` method results in the fact that the message implements the Unmarshaler interface. The allows proto.Unmarshal to be faster by calling the generated Unmarshal method rather than using reflect.
If is enabled by the following extensions:
- unmarshaler
- unmarshaler_all
Or the following extensions:
- unsafe_unmarshaler
- unsafe_unmarshaler_all
That is if you want to use the unsafe package in your generated code. The speed up using the unsafe package is not very significant.
The generation of unmarshalling tests are enabled using one of the following extensions:
- testgen
- testgen_all
And benchmarks given it is enabled using one of the following extensions:
- benchgen
- benchgen_all
Let us look at:
github.com/gogo/protobuf/test/example/example.proto
Btw all the output can be seen at:
github.com/gogo/protobuf/test/example/*
The following message:
option (gogoproto.unmarshaler_all) = true;
message B {
option (gogoproto.description) = true;
optional A A = 1 [(gogoproto.nullable) = false, (gogoproto.embed) = true];
repeated bytes G = 2 [(gogoproto.customtype) = "github.com/gogo/protobuf/test/custom.Uint128", (gogoproto.nullable) = false];
}
given to the unmarshal plugin, will generate the following code:
func (m *B) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
var wire uint64
for shift := uint(0); ; shift += 7 {
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
switch fieldNum {
case 1:
if wireType != 2 {
return proto.ErrWrongType
}
var msglen int
for shift := uint(0); ; shift += 7 {
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
msglen |= (int(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
postIndex := iNdEx + msglen
if postIndex > l {
return io.ErrUnexpectedEOF
}
if err := m.A.Unmarshal(dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
case 2:
if wireType != 2 {
return proto.ErrWrongType
}
var byteLen int
for shift := uint(0); ; shift += 7 {
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
byteLen |= (int(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
postIndex := iNdEx + byteLen
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.G = append(m.G, github_com_gogo_protobuf_test_custom.Uint128{})
if err := m.G[len(m.G)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
default:
var sizeOfWire int
for {
sizeOfWire++
wire >>= 7
if wire == 0 {
break
}
}
iNdEx -= sizeOfWire
skippy, err := skip(dAtA[iNdEx:])
if err != nil {
return err
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...)
iNdEx += skippy
}
}
return nil
}
Remember when using this code to call proto.Unmarshal. This will call m.Reset and invoke the generated Unmarshal method for you. If you call m.Unmarshal without m.Reset you could be merging protocol buffers.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var TagString = "gen"
Source Files