wire

package
v0.1.3-beta-hotfix2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 3, 2018 License: GPL-3.0, Apache-2.0 Imports: 14 Imported by: 0

README

Wire encoding for Golang

This software implements Go bindings for the Wire encoding protocol. The goal of the Wire encoding protocol is to be a simple language-agnostic encoding protocol for rapid prototyping of blockchain applications.

This package also includes a compatible (and slower) JSON codec.

Supported types

Primary types: uvarint, varint, byte, uint[8,16,32,64], int[8,16,32,64], string, and time types are supported

Arrays: Arrays can hold items of any arbitrary type. For example, byte-arrays and byte-array-arrays are supported.

Structs: Struct fields are encoded by value (without the key name) in the order that they are declared in the struct. In this way it is similar to Apache Avro.

Interfaces: Interfaces are like union types where the value can be any non-interface type. The actual value is preceded by a single "type byte" that shows which concrete is encoded.

Pointers: Pointers are like optional fields. The first byte is 0x00 to denote a null pointer (e.g. no value), otherwise it is 0x01.

Unsupported types

Maps: Maps are not supported because for most languages, key orders are nondeterministic. If you need to encode/decode maps of arbitrary key-value pairs, encode an array of {key,value} structs instead.

Floating points: Floating point number types are discouraged because of reasons. If you need to use them, use the field tag wire:"unsafe".

Enums: Enum types are not supported in all languages, and they're simple enough to model as integers anyways.

A struct example

Struct types can be automatically encoded with reflection. Unlike json-encoding, no field name or type information is encoded. Field values are simply encoded in order.

package main

import (
  "bytes"
  "fmt"
  "math"
  "github.com/CyberMiles/travis/sdk/go-wire"
)

type Foo struct {
  MyString       string
  MyUint32       uint32
  myPrivateBytes []byte
}

func main() {

  foo := Foo{"my string", math.MaxUint32, []byte("my private bytes")}

  buf, n, err := new(bytes.Buffer), int(0), error(nil)
  wire.WriteBinary(foo, buf, &n, &err)

  fmt.Printf("%X\n", buf.Bytes())
}

The above example prints:

01096D7920737472696E67FFFFFFFF, where

0109                            is the varint encoding of the length of string "my string"
    6D7920737472696E67          is the bytes of string "my string"
                      FFFFFFFF  is the bytes for math.MaxUint32, a uint32

Note that the unexported "myPrivateBytes" isn't encoded.

An interface example

Here's an example with interfaces.

package main

import (
  "bytes"
  "fmt"
  "github.com/CyberMiles/travis/sdk/go-wire"
)

type Animal interface{}
type Dog struct{ Name string }
type Cat struct{ Name string }
type Cow struct{ Name string }

var _ = wire.RegisterInterface(
  struct{ Animal }{},
  wire.ConcreteType{Dog{}, 0x01}, // type-byte of 0x01 for Dogs
  wire.ConcreteType{Cat{}, 0x02}, // type-byte of 0x02 for Cats
  wire.ConcreteType{Cow{}, 0x03}, // type-byte of 0x03 for Cows
)

func main() {

  animals := []Animal{
    Dog{"Snoopy"},
    Cow{"Daisy"},
  }

  buf, n, err := new(bytes.Buffer), int(0), error(nil)
  wire.WriteBinary(animals, buf, &n, &err)

  fmt.Printf("%X\n", buf.Bytes())
}

The above example prints:

0102010106536E6F6F70790301054461697379, where

0102                                    is the varint encoding of the length of the array
    01                                  is the type-byte for a Dog
      0106                              is the varint encoding of the length of the Dog's name
          536E6F6F7079                  is the Dog's name "Snoopy"
                      03                is the type-byte for a Cow
                        0105            is the varint encoding of the length of the Cow's name
                            4461697379  is the Cow's name "Daisy"
A pointer example

Here's an example with pointers (and interfaces too).

package main

import (
	"bytes"
	"fmt"
	"github.com/CyberMiles/travis/sdk/go-wire"
)

type Animal interface{}
type Dog struct{ Name string }
type Cat struct{ Name string }
type Cow struct{ Name string }

var _ = wire.RegisterInterface(
	struct{ Animal }{},
	wire.ConcreteType{Dog{}, 0x01},  // type-byte of 0x01 for Dogs
	wire.ConcreteType{&Dog{}, 0x02}, // type-byte of 0x02 for Dog pointers
)

type MyStruct struct {
	Field1 Animal
	Field2 *Dog
	Field3 *Dog
}

func main() {

	myStruct := MyStruct{
		Field1: &Dog{"Snoopy"},
		Field2: &Dog{"Smappy"},
		Field3: (*Dog)(nil),
	}

	buf, n, err := new(bytes.Buffer), int(0), error(nil)
	wire.WriteBinary(myStruct, buf, &n, &err)

	fmt.Printf("%X\n", buf.Bytes())
}

The above example prints:

020106536E6F6F7079010106536D6170707900, where

02                                      is the type-byte for a Dog pointer for Field1
  0106                                  is the varint encoding of the length of the Dog's name
      536E6F6F7079                      is the Dog's name "Snoopy"
                  01                    is a byte indicating a non-null pointer for Field2
                    0106                is the varint encoding of the length of the Dog's name
                        536D61707079    is the Dog's name "Smappy"
                                    00  is a byte indicating a null pointer for Field3

Notice that in Field1, that the value is non-null is implied in the type-byte of 0x02. While Golang lets you have nil-pointers as interface values, this is a Golang-specific feature that is absent in other OOP languages such as Java. So, Go-Wire does not support nil-pointers for interface values. The following example would return an error:

myStruct := MyStruct{
  Field1: (*Dog)(nil),    // Error!
  Field2: &Dog{"Smappy"}, // Ok!
  Field3: (*Dog)(nil),    // Ok!
}

buf, n, err := new(bytes.Buffer), int(0), error(nil)
wire.WriteBinary(myStruct, buf, &n, &err)
fmt.Println(err)

// Unexpected nil-pointer of type main.Dog for registered interface Animal.
// For compatibility with other languages, nil-pointer interface values are forbidden.

Documentation

Overview

go-wire is our custom codec package for serializing and deserializing data and structures as binary and JSON blobs.

In order to get started with go-wire we need to:

1) Choose the receiving structure for deserializing. It MUST be an interface{} and during registration it MUST be wrapped as a struct for example 

struct { Receiver }{}

2) Decide the IDs for the respective types that we'll be dealing with. We shall call these the concrete types.

3) Register the receiving structure as well as each of the concrete types. Typically do this in the init function so that it gets run before other functions are invoked 

  func init() {
    wire.RegisterInterface(
	struct { Receiver }{},
	wire.ConcreteType{&bcMessage{}, 0x01},
	wire.ConcreteType{&bcResponse{}, 0x02},
	wire.ConcreteType{&bcStatus{}, 0x03},
    )
  }

  type bcMessage struct {
    Content string
    Height int
  }

  type bcResponse struct {
    Status int
    Message string
  }

  type bcResponse struct {
    Status int
    Message string
  }

Encoding to binary is performed by invoking wire.WriteBinary. You'll need to provide the data to be encoded/serialized as well as where to store it and storage for the number of bytes written as well as any error encountered 

var n int
var err error
buf := new(bytes.Buffer)
bm := &bcMessage{Message: "Tendermint", Height: 100}
wire.WriteBinary(bm, buf, &n, &err)

Decoding from binary is performed by invoking wire.ReadBinary. The data being decoded has to be retrieved from the decoding receiver that we previously defined i.e. Receiver for example 

recv := wire.ReadBinary(struct{ Receiver }{}, buf, 0, &n, &err).(struct{ Receiver }).Receiver
decoded := recv.(*bcMessage)
fmt.Printf("Decoded: %#v\n", decoded)

Note that in the decoding example we used 

struct { Receiver }{} --> .(struct{ Receiver }).Receiver

to receive the value. That correlates with the type that we registered in wire.RegisterInterface

Index

Constants

View Source 
const (
	RFC3339Millis = "2006-01-02T15:04:05.000Z" // forced microseconds
)
View Source 
const (
	ReadSliceChunkSize = 1024
)
View Source 
const Version = "0.7.3"

Variables

View Source 
var (
	ErrBinaryReadOverflow                  = errors.New("Error: binary read overflow")
	ErrBinaryReadInvalidLength             = errors.New("Error: binary read invalid length")
	ErrBinaryReadInvalidTimeNegative       = errors.New("Error: binary read invalid time - negative")
	ErrBinaryReadInvalidTimeSubMillisecond = errors.New("Error: binary read invalid time - sub millisecond")
	ErrBinaryWriteOverflow                 = errors.New("Error: binary write overflow")
)

Functions

func BinaryBytes 

func BinaryBytes(o interface{}) []byte

func BinaryCompare 

func BinaryCompare(a, b interface{}) int

NOTE: does not care about the type, only the binary representation.

func BinaryEqual 

func BinaryEqual(a, b interface{}) bool

NOTE: does not care about the type, only the binary representation.

func BinaryRipemd160 

func BinaryRipemd160(o interface{}) []byte

NOTE: The default hash function is Ripemd160.

func BinarySha256 

func BinarySha256(o interface{}) []byte

NOTE: only use this if you need 32 bytes.

func ByteSliceSize 

func ByteSliceSize(bz []byte) int

Returns the total encoded size of a byteslice

func GetBool 

func GetBool(buf []byte) (bool, error)

func GetByteSlice 

func GetByteSlice(buf []byte) (bz []byte, n int, err error)

func GetInt16 

func GetInt16(buf []byte) int16

func GetInt32 

func GetInt32(buf []byte) int32

func GetInt64 

func GetInt64(buf []byte) int64

func GetString 

func GetString(buf []byte) (s string, n int, err error)

func GetTypeFromStructDeclaration 

func GetTypeFromStructDeclaration(o interface{}) reflect.Type

e.g. If o is struct{Foo}{}, return is the Foo reflection type.

func GetUint16 

func GetUint16(buf []byte) uint16

func GetUint32 

func GetUint32(buf []byte) uint32

func GetUint64 

func GetUint64(buf []byte) uint64

func GetUvarint 

func GetUvarint(buf []byte) (i uint, n int, err error)

func GetVarint 

func GetVarint(buf []byte) (i int, n int, err error)

func JSONBytes 

func JSONBytes(o interface{}) []byte

func JSONBytesPretty 

func JSONBytesPretty(o interface{}) []byte

NOTE: inefficient

func MarshalBinary 

func MarshalBinary(o interface{}) ([]byte, error)

func MarshalJSON 

func MarshalJSON(o interface{}) ([]byte, error)

func PutBool 

func PutBool(buf []byte, b bool)

func PutByteSlice 

func PutByteSlice(buf []byte, bz []byte) (n int, err error)

func PutInt16 

func PutInt16(buf []byte, i int16)

func PutInt32 

func PutInt32(buf []byte, i int32)

func PutInt64 

func PutInt64(buf []byte, i int64)

func PutString 

func PutString(buf []byte, s string) (n int, err error)

func PutUint16 

func PutUint16(buf []byte, i uint16)

func PutUint32 

func PutUint32(buf []byte, i uint32)

func PutUint64 

func PutUint64(buf []byte, i uint64)

func PutUvarint 

func PutUvarint(buf []byte, i uint) (n int, err error)

func PutVarint 

func PutVarint(buf []byte, i int) (n int, err error)

func ReadBinary 

func ReadBinary(o interface{}, r io.Reader, lmt int, n *int, err *error) (res interface{})

func ReadBinaryBytes 

func ReadBinaryBytes(d []byte, ptr interface{}) error

ptr: a pointer to the object to be filled

func ReadBinaryPtr 

func ReadBinaryPtr(o interface{}, r io.Reader, lmt int, n *int, err *error) (res interface{})

func ReadBinaryPtrLengthPrefixed 

func ReadBinaryPtrLengthPrefixed(o interface{}, r io.Reader, lmt int, n *int, err *error) (res interface{})

func ReadBool 

func ReadBool(r io.Reader, n *int, err *error) bool

func ReadByte 

func ReadByte(r io.Reader, n *int, err *error) byte

func ReadByteSlice 

func ReadByteSlice(r io.Reader, lmt int, n *int, err *error) []byte

func ReadByteSlices 

func ReadByteSlices(r io.Reader, lmt int, n *int, err *error) [][]byte

func ReadFloat32 

func ReadFloat32(r io.Reader, n *int, err *error) float32

func ReadFloat64 

func ReadFloat64(r io.Reader, n *int, err *error) float64

func ReadFull 

func ReadFull(buf []byte, r io.Reader, n *int, err *error)

Read len(buf) from r Increment n and set err accordingly.

func ReadInt16 

func ReadInt16(r io.Reader, n *int, err *error) int16

func ReadInt32 

func ReadInt32(r io.Reader, n *int, err *error) int32

func ReadInt64 

func ReadInt64(r io.Reader, n *int, err *error) int64

func ReadInt8 

func ReadInt8(r io.Reader, n *int, err *error) int8

func ReadJSON 

func ReadJSON(o interface{}, bytes []byte, err *error) interface{}

func ReadJSONBytes 

func ReadJSONBytes(d []byte, ptr interface{}) (err error)

ptr: a pointer to the object to be filled

func ReadJSONObject 

func ReadJSONObject(o interface{}, object interface{}, err *error) interface{}

o is the ultimate destination, object is the result of json unmarshal

func ReadJSONObjectPtr 

func ReadJSONObjectPtr(o interface{}, object interface{}, err *error) interface{}

func ReadJSONPtr 

func ReadJSONPtr(o interface{}, bytes []byte, err *error) interface{}

func ReadString 

func ReadString(r io.Reader, lmt int, n *int, err *error) string

func ReadTime 

func ReadTime(r io.Reader, n *int, err *error) time.Time

ReadTime reads an Int64 from the Reader, interprets it as the number of nanoseconds since January 1, 1970 UTC, and returns the corresponding time. If the Int64 read is less than zero, or not a multiple of a million, it sets the error and returns the default time.

func ReadUint16 

func ReadUint16(r io.Reader, n *int, err *error) uint16

func ReadUint16s 

func ReadUint16s(r io.Reader, n *int, err *error) []uint16

func ReadUint32 

func ReadUint32(r io.Reader, n *int, err *error) uint32

func ReadUint64 

func ReadUint64(r io.Reader, n *int, err *error) uint64

func ReadUint8 

func ReadUint8(r io.Reader, n *int, err *error) uint8

func ReadUvarint 

func ReadUvarint(r io.Reader, n *int, err *error) uint

func ReadVarint 

func ReadVarint(r io.Reader, n *int, err *error) int

func UnmarshalBinary 

func UnmarshalBinary(bz []byte, ptr interface{}) error

func UnmarshalJSON 

func UnmarshalJSON(bz []byte, ptr interface{}) (err error)

func UvarintSize 

func UvarintSize(i uint64) int

func WriteBinary 

func WriteBinary(o interface{}, w io.Writer, n *int, err *error)

WriteBinary is the binary encoder. Its arguments are the subject to be encoded, the writer that'll receive the encoded bytes, as well as a receiver to store the bytes written and any error encountered.

func WriteBinaryLengthPrefixed 

func WriteBinaryLengthPrefixed(o interface{}, w io.Writer, n *int, err *error)

func WriteBool 

func WriteBool(b bool, w io.Writer, n *int, err *error)

func WriteByte 

func WriteByte(b byte, w io.Writer, n *int, err *error)

func WriteByteSlice 

func WriteByteSlice(bz []byte, w io.Writer, n *int, err *error)

func WriteByteSlices 

func WriteByteSlices(bzz [][]byte, w io.Writer, n *int, err *error)

func WriteFloat32 

func WriteFloat32(f float32, w io.Writer, n *int, err *error)

func WriteFloat64 

func WriteFloat64(f float64, w io.Writer, n *int, err *error)

func WriteInt16 

func WriteInt16(i int16, w io.Writer, n *int, err *error)

func WriteInt32 

func WriteInt32(i int32, w io.Writer, n *int, err *error)

func WriteInt64 

func WriteInt64(i int64, w io.Writer, n *int, err *error)

func WriteInt8 

func WriteInt8(i int8, w io.Writer, n *int, err *error)

func WriteJSON 

func WriteJSON(o interface{}, w io.Writer, n *int, err *error)

func WriteString 

func WriteString(s string, w io.Writer, n *int, err *error)

func WriteTime 

func WriteTime(t time.Time, w io.Writer, n *int, err *error)

WriteTime writes the number of nanoseconds, with millisecond resolution, since January 1, 1970 UTC, to the Writer as an Int64. Milliseconds are used to ease compatibility with Javascript, which does not support finer resolution. NOTE: panics if the given time is less than January 1, 1970 UTC

func WriteTo 

func WriteTo(bz []byte, w io.Writer, n *int, err *error)

Write all of bz to w Increment n and set err accordingly.

func WriteUint16 

func WriteUint16(i uint16, w io.Writer, n *int, err *error)

func WriteUint16s 

func WriteUint16s(iz []uint16, w io.Writer, n *int, err *error)

func WriteUint32 

func WriteUint32(i uint32, w io.Writer, n *int, err *error)

func WriteUint64 

func WriteUint64(i uint64, w io.Writer, n *int, err *error)

func WriteUint8 

func WriteUint8(i uint8, w io.Writer, n *int, err *error)

func WriteUvarint 

func WriteUvarint(i uint, w io.Writer, n *int, err *error)

func WriteVarint 

func WriteVarint(i int, w io.Writer, n *int, err *error)

Types

type ConcreteType 

type ConcreteType struct {
	O    interface{}
	Byte byte
}

For use with the RegisterInterface declaration

type Options 

type Options struct {
	JSONName      string      // (JSON) Corresponding JSON field name. (override with `json=""`)
	JSONOmitEmpty bool        // (JSON) Omit field if value is empty
	Varint        bool        // (Binary) Use length-prefixed encoding for (u)int64
	Unsafe        bool        // (JSON/Binary) Explicitly enable support for floats or maps
	ZeroValue     interface{} // Prototype zero object
}

type StructFieldInfo 

type StructFieldInfo struct {
	Index   int          // Struct field index
	Type    reflect.Type // Struct field type
	Options              // Encoding options
}

type TypeInfo 

type TypeInfo struct {
	Type reflect.Type // The type

	// If Type is kind reflect.Interface, is registered
	IsRegisteredInterface bool
	ByteToType            map[byte]reflect.Type
	TypeToByte            map[reflect.Type]byte

	// If Type is kind reflect.Struct
	Fields []StructFieldInfo
	Unwrap bool // if struct has only one field and its an anonymous interface
}

func GetTypeInfo 

func GetTypeInfo(rt reflect.Type) *TypeInfo

func MakeTypeInfo 

func MakeTypeInfo(rt reflect.Type) *TypeInfo

func RegisterInterface 

func RegisterInterface(o interface{}, ctypes ...ConcreteType) *TypeInfo

This function should be used to register the receiving interface that will be used to decode an underlying concrete type. The interface MUST be embedded in a struct, and the interface MUST be the only field and it MUST be exported. For example: 

RegisterInterface(struct{ Animal }{}, ConcreteType{&foo, 0x01})

Source Files

View all Source files

Directories

Path Synopsis
Data is designed to provide a standard interface and helper functions to easily allow serialization and deserialization of your data structures in both binary and json representations.
 Data is designed to provide a standard interface and helper functions to easily allow serialization and deserialization of your data structures in both binary and json representations.
base58
Package base58 provides base58-check encoding.
 Package base58 provides base58-check encoding.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.