wire

package
v0.1.10-beta Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 29, 2019 License: GPL-3.0, Apache-2.0 Imports: 14 Imported by: 0

README

Wire encoding for Golang

This software implements Go bindings for the Wire encoding protocol. The goal of the Wire encoding protocol is to be a simple language-agnostic encoding protocol for rapid prototyping of blockchain applications.

This package also includes a compatible (and slower) JSON codec.

Supported types

Primary types: uvarint, varint, byte, uint[8,16,32,64], int[8,16,32,64], string, and time types are supported

Arrays: Arrays can hold items of any arbitrary type. For example, byte-arrays and byte-array-arrays are supported.

Structs: Struct fields are encoded by value (without the key name) in the order that they are declared in the struct. In this way it is similar to Apache Avro.

Interfaces: Interfaces are like union types where the value can be any non-interface type. The actual value is preceded by a single "type byte" that shows which concrete is encoded.

Pointers: Pointers are like optional fields. The first byte is 0x00 to denote a null pointer (e.g. no value), otherwise it is 0x01.

Unsupported types

Maps: Maps are not supported because for most languages, key orders are nondeterministic. If you need to encode/decode maps of arbitrary key-value pairs, encode an array of {key,value} structs instead.

Floating points: Floating point number types are discouraged because of reasons. If you need to use them, use the field tag wire:"unsafe".

Enums: Enum types are not supported in all languages, and they're simple enough to model as integers anyways.

A struct example

Struct types can be automatically encoded with reflection. Unlike json-encoding, no field name or type information is encoded. Field values are simply encoded in order.

package main

import (
  "bytes"
  "fmt"
  "math"
  "github.com/CyberMiles/travis/sdk/go-wire"
)

type Foo struct {
  MyString       string
  MyUint32       uint32
  myPrivateBytes []byte
}

func main() {

  foo := Foo{"my string", math.MaxUint32, []byte("my private bytes")}

  buf, n, err := new(bytes.Buffer), int(0), error(nil)
  wire.WriteBinary(foo, buf, &n, &err)

  fmt.Printf("%X\n", buf.Bytes())
}

The above example prints:

01096D7920737472696E67FFFFFFFF, where

0109                            is the varint encoding of the length of string "my string"
    6D7920737472696E67          is the bytes of string "my string"
                      FFFFFFFF  is the bytes for math.MaxUint32, a uint32

Note that the unexported "myPrivateBytes" isn't encoded.

An interface example

Here's an example with interfaces.

package main

import (
  "bytes"
  "fmt"
  "github.com/CyberMiles/travis/sdk/go-wire"
)

type Animal interface{}
type Dog struct{ Name string }
type Cat struct{ Name string }
type Cow struct{ Name string }

var _ = wire.RegisterInterface(
  struct{ Animal }{},
  wire.ConcreteType{Dog{}, 0x01}, // type-byte of 0x01 for Dogs
  wire.ConcreteType{Cat{}, 0x02}, // type-byte of 0x02 for Cats
  wire.ConcreteType{Cow{}, 0x03}, // type-byte of 0x03 for Cows
)

func main() {

  animals := []Animal{
    Dog{"Snoopy"},
    Cow{"Daisy"},
  }

  buf, n, err := new(bytes.Buffer), int(0), error(nil)
  wire.WriteBinary(animals, buf, &n, &err)

  fmt.Printf("%X\n", buf.Bytes())
}

The above example prints:

0102010106536E6F6F70790301054461697379, where

0102                                    is the varint encoding of the length of the array
    01                                  is the type-byte for a Dog
      0106                              is the varint encoding of the length of the Dog's name
          536E6F6F7079                  is the Dog's name "Snoopy"
                      03                is the type-byte for a Cow
                        0105            is the varint encoding of the length of the Cow's name
                            4461697379  is the Cow's name "Daisy"
A pointer example

Here's an example with pointers (and interfaces too).

package main

import (
	"bytes"
	"fmt"
	"github.com/CyberMiles/travis/sdk/go-wire"
)

type Animal interface{}
type Dog struct{ Name string }
type Cat struct{ Name string }
type Cow struct{ Name string }

var _ = wire.RegisterInterface(
	struct{ Animal }{},
	wire.ConcreteType{Dog{}, 0x01},  // type-byte of 0x01 for Dogs
	wire.ConcreteType{&Dog{}, 0x02}, // type-byte of 0x02 for Dog pointers
)

type MyStruct struct {
	Field1 Animal
	Field2 *Dog
	Field3 *Dog
}

func main() {

	myStruct := MyStruct{
		Field1: &Dog{"Snoopy"},
		Field2: &Dog{"Smappy"},
		Field3: (*Dog)(nil),
	}

	buf, n, err := new(bytes.Buffer), int(0), error(nil)
	wire.WriteBinary(myStruct, buf, &n, &err)

	fmt.Printf("%X\n", buf.Bytes())
}

The above example prints:

020106536E6F6F7079010106536D6170707900, where

02                                      is the type-byte for a Dog pointer for Field1
  0106                                  is the varint encoding of the length of the Dog's name
      536E6F6F7079                      is the Dog's name "Snoopy"
                  01                    is a byte indicating a non-null pointer for Field2
                    0106                is the varint encoding of the length of the Dog's name
                        536D61707079    is the Dog's name "Smappy"
                                    00  is a byte indicating a null pointer for Field3

Notice that in Field1, that the value is non-null is implied in the type-byte of 0x02. While Golang lets you have nil-pointers as interface values, this is a Golang-specific feature that is absent in other OOP languages such as Java. So, Go-Wire does not support nil-pointers for interface values. The following example would return an error:

myStruct := MyStruct{
  Field1: (*Dog)(nil),    // Error!
  Field2: &Dog{"Smappy"}, // Ok!
  Field3: (*Dog)(nil),    // Ok!
}

buf, n, err := new(bytes.Buffer), int(0), error(nil)
wire.WriteBinary(myStruct, buf, &n, &err)
fmt.Println(err)

// Unexpected nil-pointer of type main.Dog for registered interface Animal.
// For compatibility with other languages, nil-pointer interface values are forbidden.

Documentation

Overview

go-wire is our custom codec package for serializing and deserializing data and structures as binary and JSON blobs.

In order to get started with go-wire we need to:

1) Choose the receiving structure for deserializing. It MUST be an interface{} and during registration it MUST be wrapped as a struct for example

struct { Receiver }{}

2) Decide the IDs for the respective types that we'll be dealing with. We shall call these the concrete types.

3) Register the receiving structure as well as each of the concrete types. Typically do this in the init function so that it gets run before other functions are invoked

  func init() {
    wire.RegisterInterface(
	struct { Receiver }{},
	wire.ConcreteType{&bcMessage{}, 0x01},
	wire.ConcreteType{&bcResponse{}, 0x02},
	wire.ConcreteType{&bcStatus{}, 0x03},
    )
  }

  type bcMessage struct {
    Content string
    Height int
  }

  type bcResponse struct {
    Status int
    Message string
  }

  type bcResponse struct {
    Status int
    Message string
  }

Encoding to binary is performed by invoking wire.WriteBinary. You'll need to provide the data to be encoded/serialized as well as where to store it and storage for the number of bytes written as well as any error encountered

var n int
var err error
buf := new(bytes.Buffer)
bm := &bcMessage{Message: "Tendermint", Height: 100}
wire.WriteBinary(bm, buf, &n, &err)

Decoding from binary is performed by invoking wire.ReadBinary. The data being decoded has to be retrieved from the decoding receiver that we previously defined i.e. Receiver for example

recv := wire.ReadBinary(struct{ Receiver }{}, buf, 0, &n, &err).(struct{ Receiver }).Receiver
decoded := recv.(*bcMessage)
fmt.Printf("Decoded: %#v\n", decoded)

Note that in the decoding example we used

struct { Receiver }{} --> .(struct{ Receiver }).Receiver

to receive the value. That correlates with the type that we registered in wire.RegisterInterface

Index

Constants

View Source
const (
	RFC3339Millis = "2006-01-02T15:04:05.000Z" // forced microseconds
)
View Source
const (
	ReadSliceChunkSize = 1024
)
View Source
const Version = "0.7.3"

Variables

View Source
var (
	ErrBinaryReadOverflow                  = errors.New("Error: binary read overflow")
	ErrBinaryReadInvalidLength             = errors.New("Error: binary read invalid length")
	ErrBinaryReadInvalidTimeNegative       = errors.New("Error: binary read invalid time - negative")
	ErrBinaryReadInvalidTimeSubMillisecond = errors.New("Error: binary read invalid time - sub millisecond")
	ErrBinaryWriteOverflow                 = errors.New("Error: binary write overflow")
)

Functions

func BinaryBytes

func BinaryBytes(o interface{}) []byte

func BinaryCompare

func BinaryCompare(a, b interface{}) int

NOTE: does not care about the type, only the binary representation.

func BinaryEqual

func BinaryEqual(a, b interface{}) bool

NOTE: does not care about the type, only the binary representation.

func BinaryRipemd160

func BinaryRipemd160(o interface{}) []byte

NOTE: The default hash function is Ripemd160.

func BinarySha256

func BinarySha256(o interface{}) []byte

NOTE: only use this if you need 32 bytes.

func ByteSliceSize

func ByteSliceSize(bz []byte) int

Returns the total encoded size of a byteslice

func GetBool

func GetBool(buf []byte) (bool, error)

func GetByteSlice

func GetByteSlice(buf []byte) (bz []byte, n int, err error)

func GetInt16

func GetInt16(buf []byte) int16

func GetInt32

func GetInt32(buf []byte) int32

func GetInt64

func GetInt64(buf []byte) int64

func GetString

func GetString(buf []byte) (s string, n int, err error)

func GetTypeFromStructDeclaration

func GetTypeFromStructDeclaration(o interface{}) reflect.Type

e.g. If o is struct{Foo}{}, return is the Foo reflection type.

func GetUint16

func GetUint16(buf []byte) uint16

func GetUint32

func GetUint32(buf []byte) uint32

func GetUint64

func GetUint64(buf []byte) uint64

func GetUvarint

func GetUvarint(buf []byte) (i uint, n int, err error)

func GetVarint

func GetVarint(buf []byte) (i int, n int, err error)

func JSONBytes

func JSONBytes(o interface{}) []byte

func JSONBytesPretty

func JSONBytesPretty(o interface{}) []byte

NOTE: inefficient

func MarshalBinary

func MarshalBinary(o interface{}) ([]byte, error)

func MarshalJSON

func MarshalJSON(o interface{}) ([]byte, error)

func PutBool

func PutBool(buf []byte, b bool)

func PutByteSlice

func PutByteSlice(buf []byte, bz []byte) (n int, err error)

func PutInt16

func PutInt16(buf []byte, i int16)

func PutInt32

func PutInt32(buf []byte, i int32)

func PutInt64

func PutInt64(buf []byte, i int64)

func PutString

func PutString(buf []byte, s string) (n int, err error)

func PutUint16

func PutUint16(buf []byte, i uint16)

func PutUint32

func PutUint32(buf []byte, i uint32)

func PutUint64

func PutUint64(buf []byte, i uint64)

func PutUvarint

func PutUvarint(buf []byte, i uint) (n int, err error)

func PutVarint

func PutVarint(buf []byte, i int) (n int, err error)

func ReadBinary

func ReadBinary(o interface{}, r io.Reader, lmt int, n *int, err *error) (res interface{})

func ReadBinaryBytes

func ReadBinaryBytes(d []byte, ptr interface{}) error

ptr: a pointer to the object to be filled

func ReadBinaryPtr

func ReadBinaryPtr(o interface{}, r io.Reader, lmt int, n *int, err *error) (res interface{})

func ReadBinaryPtrLengthPrefixed

func ReadBinaryPtrLengthPrefixed(o interface{}, r io.Reader, lmt int, n *int, err *error) (res interface{})

func ReadBool

func ReadBool(r io.Reader, n *int, err *error) bool

func ReadByte

func ReadByte(r io.Reader, n *int, err *error) byte

func ReadByteSlice

func ReadByteSlice(r io.Reader, lmt int, n *int, err *error) []byte

func ReadByteSlices

func ReadByteSlices(r io.Reader, lmt int, n *int, err *error) [][]byte

func ReadFloat32

func ReadFloat32(r io.Reader, n *int, err *error) float32

func ReadFloat64

func ReadFloat64(r io.Reader, n *int, err *error) float64

func ReadFull

func ReadFull(buf []byte, r io.Reader, n *int, err *error)

Read len(buf) from r Increment n and set err accordingly.

func ReadInt16

func ReadInt16(r io.Reader, n *int, err *error) int16

func ReadInt32

func ReadInt32(r io.Reader, n *int, err *error) int32

func ReadInt64

func ReadInt64(r io.Reader, n *int, err *error) int64

func ReadInt8

func ReadInt8(r io.Reader, n *int, err *error) int8

func ReadJSON

func ReadJSON(o interface{}, bytes []byte, err *error) interface{}

func ReadJSONBytes

func ReadJSONBytes(d []byte, ptr interface{}) (err error)

ptr: a pointer to the object to be filled

func ReadJSONObject

func ReadJSONObject(o interface{}, object interface{}, err *error) interface{}

o is the ultimate destination, object is the result of json unmarshal

func ReadJSONObjectPtr

func ReadJSONObjectPtr(o interface{}, object interface{}, err *error) interface{}

func ReadJSONPtr

func ReadJSONPtr(o interface{}, bytes []byte, err *error) interface{}

func ReadString

func ReadString(r io.Reader, lmt int, n *int, err *error) string

func ReadTime

func ReadTime(r io.Reader, n *int, err *error) time.Time

ReadTime reads an Int64 from the Reader, interprets it as the number of nanoseconds since January 1, 1970 UTC, and returns the corresponding time. If the Int64 read is less than zero, or not a multiple of a million, it sets the error and returns the default time.

func ReadUint16

func ReadUint16(r io.Reader, n *int, err *error) uint16

func ReadUint16s

func ReadUint16s(r io.Reader, n *int, err *error) []uint16

func ReadUint32

func ReadUint32(r io.Reader, n *int, err *error) uint32

func ReadUint64

func ReadUint64(r io.Reader, n *int, err *error) uint64

func ReadUint8

func ReadUint8(r io.Reader, n *int, err *error) uint8

func ReadUvarint

func ReadUvarint(r io.Reader, n *int, err *error) uint

func ReadVarint

func ReadVarint(r io.Reader, n *int, err *error) int

func UnmarshalBinary

func UnmarshalBinary(bz []byte, ptr interface{}) error

func UnmarshalJSON

func UnmarshalJSON(bz []byte, ptr interface{}) (err error)

func UvarintSize

func UvarintSize(i uint64) int

func WriteBinary

func WriteBinary(o interface{}, w io.Writer, n *int, err *error)

WriteBinary is the binary encoder. Its arguments are the subject to be encoded, the writer that'll receive the encoded bytes, as well as a receiver to store the bytes written and any error encountered.

func WriteBinaryLengthPrefixed

func WriteBinaryLengthPrefixed(o interface{}, w io.Writer, n *int, err *error)

func WriteBool

func WriteBool(b bool, w io.Writer, n *int, err *error)

func WriteByte

func WriteByte(b byte, w io.Writer, n *int, err *error)

func WriteByteSlice

func WriteByteSlice(bz []byte, w io.Writer, n *int, err *error)

func WriteByteSlices

func WriteByteSlices(bzz [][]byte, w io.Writer, n *int, err *error)

func WriteFloat32

func WriteFloat32(f float32, w io.Writer, n *int, err *error)

func WriteFloat64

func WriteFloat64(f float64, w io.Writer, n *int, err *error)

func WriteInt16

func WriteInt16(i int16, w io.Writer, n *int, err *error)

func WriteInt32

func WriteInt32(i int32, w io.Writer, n *int, err *error)

func WriteInt64

func WriteInt64(i int64, w io.Writer, n *int, err *error)

func WriteInt8

func WriteInt8(i int8, w io.Writer, n *int, err *error)

func WriteJSON

func WriteJSON(o interface{}, w io.Writer, n *int, err *error)

func WriteString

func WriteString(s string, w io.Writer, n *int, err *error)

func WriteTime

func WriteTime(t time.Time, w io.Writer, n *int, err *error)

WriteTime writes the number of nanoseconds, with millisecond resolution, since January 1, 1970 UTC, to the Writer as an Int64. Milliseconds are used to ease compatibility with Javascript, which does not support finer resolution. NOTE: panics if the given time is less than January 1, 1970 UTC

func WriteTo

func WriteTo(bz []byte, w io.Writer, n *int, err *error)

Write all of bz to w Increment n and set err accordingly.

func WriteUint16

func WriteUint16(i uint16, w io.Writer, n *int, err *error)

func WriteUint16s

func WriteUint16s(iz []uint16, w io.Writer, n *int, err *error)

func WriteUint32

func WriteUint32(i uint32, w io.Writer, n *int, err *error)

func WriteUint64

func WriteUint64(i uint64, w io.Writer, n *int, err *error)

func WriteUint8

func WriteUint8(i uint8, w io.Writer, n *int, err *error)

func WriteUvarint

func WriteUvarint(i uint, w io.Writer, n *int, err *error)

func WriteVarint

func WriteVarint(i int, w io.Writer, n *int, err *error)

Types

type ConcreteType

type ConcreteType struct {
	O    interface{}
	Byte byte
}

For use with the RegisterInterface declaration

type Options

type Options struct {
	JSONName      string      // (JSON) Corresponding JSON field name. (override with `json=""`)
	JSONOmitEmpty bool        // (JSON) Omit field if value is empty
	Varint        bool        // (Binary) Use length-prefixed encoding for (u)int64
	Unsafe        bool        // (JSON/Binary) Explicitly enable support for floats or maps
	ZeroValue     interface{} // Prototype zero object
}

type StructFieldInfo

type StructFieldInfo struct {
	Index   int          // Struct field index
	Type    reflect.Type // Struct field type
	Options              // Encoding options
}

type TypeInfo

type TypeInfo struct {
	Type reflect.Type // The type

	// If Type is kind reflect.Interface, is registered
	IsRegisteredInterface bool
	ByteToType            map[byte]reflect.Type
	TypeToByte            map[reflect.Type]byte

	// If Type is kind reflect.Struct
	Fields []StructFieldInfo
	Unwrap bool // if struct has only one field and its an anonymous interface
}

func GetTypeInfo

func GetTypeInfo(rt reflect.Type) *TypeInfo

func MakeTypeInfo

func MakeTypeInfo(rt reflect.Type) *TypeInfo

func RegisterInterface

func RegisterInterface(o interface{}, ctypes ...ConcreteType) *TypeInfo

This function should be used to register the receiving interface that will be used to decode an underlying concrete type. The interface MUST be embedded in a struct, and the interface MUST be the only field and it MUST be exported. For example:

RegisterInterface(struct{ Animal }{}, ConcreteType{&foo, 0x01})

Directories

Path Synopsis
Data is designed to provide a standard interface and helper functions to easily allow serialization and deserialization of your data structures in both binary and json representations.
Data is designed to provide a standard interface and helper functions to easily allow serialization and deserialization of your data structures in both binary and json representations.
base58
Package base58 provides base58-check encoding.
Package base58 provides base58-check encoding.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL