protobuf

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 11, 2018 License: GPL-2.0 Imports: 15 Imported by: 0

README

Reflection-based Protocol Buffers

Package protobuf implements Protocol Buffers reflectively using Go types to define message formats. This approach provides convenience similar to Gob encoding, but with a widely-used and language-neutral wire format. For detailed API documentation see http://godoc.org/github.com/DeDiS/protobuf. For general information on Protocol buffers see http://protobuf.googlecode.com.

Features

  • Reflection-based encoding and decoding to/from protocol buffer wire format.
  • Use Go struct field tags to control protobuf fields (ID, optional/required, names).
  • Generate .proto files from Go structures.
  • Encode time.Time as an sfixed64 UnixNano.
  • Support for enums.

Details

In contrast with goprotobuf, this package does not require users to write or compile .proto files; you just define the message formats you want as Go struct types. Consider for example this example message format definition from the Protocol Buffers overview:

message Person {
  required string name = 1;
  required int32  id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string    number = 1;
    optional PhoneType type = 2;
  }

  repeated PhoneNumber phone = 4;
}

The following Go type and const definitions express exactly the same format, for purposes of encoding and decoding with this protobuf package:

type Person struct {
  Name  string
  Id    int32
  Email *string
  Phone []PhoneNumber
}

type PhoneType uint32
const (
  MobilePhone PhoneType = iota
  HomePhone
  WorkPhone
)

type PhoneNumber struct {
  Number string
  Type *PhoneType
}

To encode a message, you simply call the Encode() function with a pointer to the struct you wish to encode, and Encode() returns a []byte slice containing the protobuf-encoded struct:

person := Person{...}
buf := Encode(&person)
output.Write(buf)

To decode an encoded message, simply call Decode() on the byte-slice:

err := Decode(buf,&person)
if err != nil {
  panic("Decode failed: "+err.Error())
}

If you want to interoperate with code in other languages using the same message formats, you may of course still end up writing .proto files for the code in those other languages.

However, defining message formats with native Go types enables these types to be tailored to the code using them without affecting wire compatibility, such as by attaching useful methods to these struct types.

Translation Rules

The translation between a Go struct definition and a basic Protocol Buffers message format definition is straightforward; the rules are as follows.

Go field tags with the key "protobuf" may be used to control naming, IDs, and optional/required state. The options are comma-separated like so:

type Tags struct {
  Field1 string `protobuf:"10,req,field_1"`
  Field2 int32  `protobuf:"20,opt,field_2"`
}

A message definition in a .proto file translates to a Go struct, whose fields are implicitly assigned consecutive numbers starting from 1.

type Padded struct {
  Field1 string                  // = 1
  Field2 int32  `protobuf:"3"`   // = 3
}

A 'required' protobuf field translates to a plain field of a corresponding type in the Go struct. The following table summarizes the correspondence between .proto definition types and Go field types:

Protobuf Go
bool bool
enum Enum
int32 uint32
int64 uint64
uint32 uint32
uint64 uint64
sint32 int32
sint64 int64
fixed32 Ufixed32
fixed64 Ufixed64
sfixed32 Sfixed32
sfixed64 Sfixed64
float float32
double float64
string string
bytes []byte
message struct

An 'optional' protobuf field is expressed as a pointer field in Go. Encode() will transmit the field only if the pointer is non-nil. Decode() will instantiate the pointed-to type and fill in the pointer if the field is present in the message being decoded, leaving the pointer unmodified (usually nil) if the field is not present.

A 'repeated' protobuf field translates to a slice field in Go. Slices of primitive bool, integer, and float types are encoded and decoded in packed format, as if the [packed=true] option was declared for the field in the .proto file.

For flexibility and convenience, struct fields may have interface types, which this package interprets as having dynamic types to be bound at runtime. Encode() follows the interface's implicit pointer and uses reflection to determine the referred-to object's actual type for encoding Decode() takes an optional map of interface types to constructor functions, which it uses to instantiate concrete types for interfaces while decoding. Furthermore, if the instantiated types support the Encoding interface, Encode() and Decode() will invoke the methods of that interface, allowing objects to implement their own custom encoding/decoding methods.

This package does not try to support all possible protobuf formats. It currently does not support nonzero default value declarations for enums, the legacy unpacked formats for repeated numeric fields, messages with extremely sparse field numbering, or other more exotic features like extensions or oneof. If you need to interoperate with existing protobuf code using these features, then you should probably use goprotobuf, at least for those particular message formats. Many of these limitations could be fixed by creative use of struct tag metadata (see https://golang.org/ref/spec#Struct_types).

Another downside of this reflective approach to protobuf implementation is that reflective code is generally less efficient than statically generated code, as gogoprotobuf produces for example. If we decide we want the convenience of format definitions in Go with the runtime performance of static code generation, we could in principle achieve that by adding a "Go-format" message format compiler frontend to goprotobuf or gogoprotobuf - but we leave this as an exercise for the reader.

Generating .proto files

.proto files can be generated from Go structs using the GenerateProtobufDefinition() function. The following:

types := []interface{}{
  Person{},
  PhoneNumber{},
}
enums := EnumMap{
  "MobilePhone": MobilePhone,
  "HomePhone": HomePhone,
  "WorkPhone": WorkPhone,
}
GenerateProtobufDefinition(w, types, enums, nil)

Will generate:

message Person {
  required string name = 1;
  required sint32 id = 2;
  optional string email = 3;
  repeated PhoneNumber phone = 4;
}

message PhoneNumber {
  required string number = 1;
  optional uint32 type = 2;
}

Note: It can be quite tedious to manually synchronise the type and enum maps with the types in your package. I've found pkgreflect very useful for automating this.

Documentation

Overview

Package protobuf implements Protocol Buffers reflectively using Go types to define message formats. This approach provides convenience similar to Gob encoding, but with a widely-used and language-neutral wire format. For general information on Protocol buffers see http://protobuf.googlecode.com.

In contrast with goprotobuf, this package does not require users to write or compile .proto files; you just define the message formats you want as Go struct types. Consider for example this example message format definition from the Protocol Buffers overview: 

message Person {
  required string name = 1;
  required int32  id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string    number = 1;
    optional PhoneType type = 2;
  }

  repeated PhoneNumber phone = 4;
}

The following Go type and const definitions express exactly the same format, for purposes of encoding and decoding with this protobuf package: 

type Person struct {
	Name  string
	Id    int32
	Email *string
	Phone []PhoneNumber
}

type PhoneType uint32
const (
	MOBILE PhoneType = iota
	HOME
	WORK
)

type PhoneNumber struct {
	Number string
	Type *PhoneType
}

To encode a message, you simply call the Encode() function with a pointer to the struct you wish to encode, and Encode() returns a []byte slice containing the protobuf-encoded struct: 

person := Person{...}
buf := Encode(&person)
output.Write(buf)

To decode an encoded message, simply call Decode() on the byte-slice: 

err := Decode(buf,&person,nil)
if err != nil {
	panic("Decode failed: "+err.Error())
}

If you want to interoperate with code in other languages using the same message formats, you may of course still end up writing .proto files for the code in those other languages. However, defining message formats with native Go types enables these types to be tailored to the code using them without affecting wire compatibility, such as by attaching useful methods to these struct types. The translation between a Go struct definition and a basic Protocol Buffers message format definition is straightforward; the rules are as follows.

A message definition in a .proto file translates to a Go struct, whose fields are implicitly assigned consecutive numbers starting from 1. If you need to leave gaps in the field number sequence (e.g., to delete an obsolete field without breaking wire compatibility), then you can skip that field number using a blank Go field, like this: 

type Padded struct {
	Field1 string		// = 1
	_ struct{}		// = 2 (unused field number)
	Field2 int32		// = 3
}

A 'required' protobuf field translates to a plain field of a corresponding type in the Go struct. The following table summarizes the correspondence between .proto definition types and Go field types: 

Protobuf		Go
--------		--
bool			bool
enum			Enum
int32			uint32
int64			uint64
uint32			uint32
uint64			uint64
sint32			int32
sint64			int64
fixed32			Ufixed32
fixed64			Ufixed64
sfixed32		Sfixed32
sfixed64		Sfixed64
float			float32
double			float64
string			string
bytes			[]byte
message			struct

An 'optional' protobuf field is expressed as a pointer field in Go. Encode() will transmit the field only if the pointer is non-nil. Decode() will instantiate the pointed-to type and fill in the pointer if the field is present in the message being decoded, leaving the pointer unmodified (usually nil) if the field is not present.

A 'repeated' protobuf field translates to a slice field in Go. Slices of primitive bool, integer, and float types are encoded and decoded in packed format, as if the [packed=true] option was declared for the field in the .proto file.

For flexibility and convenience, struct fields may have interface types, which this package interprets as having dynamic types to be bound at runtime. Encode() follows the interface's implicit pointer and uses reflection to determine the referred-to object's actual type for encoding Decode() takes an optional map of interface types to constructor functions, which it uses to instantiate concrete types for interfaces while decoding. Furthermore, if the instantiated types support the Encoding interface, Encode() and Decode() will invoke the methods of that interface, allowing objects to implement their own custom encoding/decoding methods.

This package does not try to support all possible protobuf formats. It currently does not support nonzero default value declarations for enums, the legacy unpacked formats for repeated numeric fields, messages with extremely sparse field numbering, or other more exotic features like extensions or oneof. If you need to interoperate with existing protobuf code using these features, then you should probably use goprotobuf, at least for those particular message formats.

Another downside of this reflective approach to protobuf implementation is that reflective code is generally less efficient than statically generated code, as gogoprotobuf produces for example. If we decide we want the convenience of format definitions in Go with the runtime performance of static code generation, we could in principle achieve that by adding a "Go-format" message format compiler frontend to goprotobuf or gogoprotobuf - but we leave this as an exercise for the reader.

Example (Protobuf)

This example defines, encodes, and decodes a Person message format equivalent to the example used in the Protocol Buffers overview. 

package main

import (
	"encoding/hex"
	"fmt"
)

// Go-based protobuf definition for the example Person message format
type Person struct {
	Name  string        // = 1, required
	Id    int32         // = 2, required
	Email *string       // = 3, optional
	Phone []PhoneNumber // = 4, repeated
}

type PhoneType uint32 // protobuf enums are uint32
const (
	MOBILE PhoneType = iota // = 0
	HOME                    // = 1
	WORK                    // = 2
)

type PhoneNumber struct {
	Number string     // = 1, required
	Type   *PhoneType // = 2, optional
}

// This example defines, encodes, and decodes a Person message format
// equivalent to the example used in the Protocol Buffers overview.
func main() {

	// Create a Person record
	email := "alice@somewhere"
	ptype := WORK
	person := Person{"Alice", 123, &email,
		[]PhoneNumber{PhoneNumber{"111-222-3333", nil},
			PhoneNumber{"444-555-6666", &ptype}}}

	// Encode it
	buf, err := Encode(&person)
	if err != nil {
		panic("Encode failed: " + err.Error())
	}
	fmt.Print(hex.Dump(buf))

	// Decode it
	person2 := Person{}
	if err := Decode(buf, &person2); err != nil {
		panic("Decode failed")
	}

}

Output:

00000000  0a 05 41 6c 69 63 65 10  f6 01 1a 0f 61 6c 69 63  |..Alice.....alic|
00000010  65 40 73 6f 6d 65 77 68  65 72 65 22 0e 0a 0c 31  |e@somewhere"...1|
00000020  31 31 2d 32 32 32 2d 33  33 33 33 22 10 0a 0c 34  |11-222-3333"...4|
00000030  34 34 2d 35 35 35 2d 36  36 36 36 10 02           |44-555-6666..|

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Decode 

func Decode(buf []byte, structPtr interface{}) error

Decode a protocol buffer into a Go struct. The caller must pass a pointer to the struct to decode into.

Decode() currently does not explicitly check that all 'required' fields are actually present in the input buffer being decoded. If required fields are missing, then the corresponding fields will be left unmodified, meaning they will take on their default Go zero values if Decode() is passed a fresh struct.

func DecodeWithConstructors 

func DecodeWithConstructors(buf []byte, structPtr interface{}, cons Constructors) (err error)

DecodeWithConstructors is like Decode, but you can pass a map of constructors with which to instantiate interface types.

func Encode 

func Encode(structPtr interface{}) (bytes []byte, err error)

Encode a Go struct into protocol buffer format. The caller must pass a pointer to the struct to encode.

Example (Test1)

This example encodes the simple message defined at the start of the Protocol Buffers encoding specification: https://developers.google.com/protocol-buffers/docs/encoding 

package main

import (
	"encoding/hex"
	"fmt"
)

type Test1 struct {
	A uint32
}

// This example encodes the simple message defined at the start of
// the Protocol Buffers encoding specification:
// https://developers.google.com/protocol-buffers/docs/encoding
func main() {

	t := Test1{150}
	buf, _ := Encode(&t)
	fmt.Print(hex.Dump(buf))

}

Output:

00000000  08 96 01                                          |...|
Example (Test2)

This example encodes the Test2 message illustrating strings in the Protocol Buffers encoding specification. 

package main

import (
	"encoding/hex"
	"fmt"
)

type Test2 struct {
	_ interface{} // = 1
	B string      // = 2
}

// This example encodes the Test2 message illustrating strings
// in the Protocol Buffers encoding specification.
func main() {

	t := Test2{B: "testing"}
	buf, _ := Encode(&t)
	fmt.Print(hex.Dump(buf))

}

Output:

00000000  12 07 74 65 73 74 69 6e  67                       |..testing|
Example (Test3)

This example encodes the Test3 message illustrating embedded messages in the Protocol Buffers encoding specification. 

package main

import (
	"encoding/hex"
	"fmt"
)

type Test3 struct {
	_ interface{} // = 1
	_ interface{} // = 2
	C Test1       // = 3
}

// This example encodes the Test3 message illustrating embedded messages
// in the Protocol Buffers encoding specification.
func main() {

	t := Test3{C: Test1{150}}
	buf, _ := Encode(&t)
	fmt.Print(hex.Dump(buf))

}

Output:

00000000  1a 03 08 96 01                                    |.....|

func GenerateProtobufDefinition 

func GenerateProtobufDefinition(w io.Writer, types []interface{}, enumMap EnumMap, renamer GeneratorNamer) (err error)

GenerateProtobufDefinition generates a .proto file from a list of structs via reflection. fieldNamer is a function that maps ProtoField types to generated protobuf field names.

Types

type Constructors 

type Constructors map[reflect.Type]func() interface{}

Constructors represents a map defining how to instantiate any interface types that Decode() might encounter while reading and decoding structured data. The keys are reflect.Type values denoting interface types. The corresponding values are functions expected to instantiate, and initialize as necessary, an appropriate concrete object type supporting that interface. A caller could use this capability to support dynamic instantiation of objects of the concrete type appropriate for a given abstract type.

func (*Constructors) String 

func (c *Constructors) String() string

String returns an easy way to visualize what you have in your constructors.

type DefaultGeneratorNamer 

type DefaultGeneratorNamer struct{}

DefaultGeneratorNamer renames symbols when mapping from Go to .proto files.

The rules are: - Field names are mapped from SomeFieldName to some_field_name. - Type names are not modified. - Constants are mapped form SomeConstantName to SOME_CONSTANT_NAME.

func (*DefaultGeneratorNamer) ConstName 

func (d *DefaultGeneratorNamer) ConstName(name string) string

func (*DefaultGeneratorNamer) FieldName 

func (d *DefaultGeneratorNamer) FieldName(f ProtoField) string

func (*DefaultGeneratorNamer) TypeName 

func (d *DefaultGeneratorNamer) TypeName(name string) string

type Enum 

type Enum uint32

Protobufs enums are transmitted as unsigned varints; using this type alias is optional but recommended to ensure they get the correct type.

type EnumMap 

type EnumMap map[string]interface{}

type GeneratorNamer 

type GeneratorNamer interface {
	FieldName(ProtoField) string
	TypeName(name string) string
	ConstName(name string) string
}

type ProtoField 

type ProtoField struct {
	ID     int64
	Prefix TagPrefix
	Name   string // If non-empty, tag-defined field name.
	Index  []int
	Field  reflect.StructField
}

ProtoField contains cached reflected metadata for struct fields.

func ProtoFields 

func ProtoFields(t reflect.Type) []*ProtoField

func (*ProtoField) Required 

func (p *ProtoField) Required() bool

type Sfixed32 

type Sfixed32 int32

Message fields declared to have exactly this type will be transmitted as fixed-size 32-bit signed integers.

type Sfixed64 

type Sfixed64 int64

Message fields declared to have exactly this type will be transmitted as fixed-size 64-bit signed integers.

type TagPrefix 

type TagPrefix int
const (
	TagNone TagPrefix = iota
	TagOptional
	TagRequired
)

Possible tag options.

func ParseTag 

func ParseTag(field reflect.StructField) (id int, opt TagPrefix, name string)

type Ufixed32 

type Ufixed32 uint32

Message fields declared to have exactly this type will be transmitted as fixed-size 32-bit unsigned integers.

type Ufixed64 

type Ufixed64 uint64

Message fields declared to have exactly this type will be transmitted as fixed-size 64-bit unsigned integers.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.