codec

package
v0.0.0-...-2a696aa Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2016 License: Apache-2.0, MIT Imports: 25 Imported by: 0

README

Codec

High Performance, Feature-Rich Idiomatic Go codec/encoding library for binc, msgpack, cbor, json.

Supported Serialization formats are:

To install:

go get github.com/ugorji/go/codec

This package understands the unsafe tag, to allow using unsafe semantics:

  • When decoding into a struct, you need to read the field name as a string so you can find the struct field it is mapped to. Using unsafe will bypass the allocation and copying overhead of []byte->string conversion.

To use it, you must pass the unsafe tag during install:

go install -tags=unsafe github.com/ugorji/go/codec

Online documentation: http://godoc.org/github.com/ugorji/go/codec
Detailed Usage/How-to Primer: http://ugorji.net/blog/go-codec-primer

The idiomatic Go support is as seen in other encoding packages in the standard library (ie json, xml, gob, etc).

Rich Feature Set includes:

  • Simple but extremely powerful and feature-rich API
  • Very High Performance. Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
  • Multiple conversions: Package coerces types where appropriate e.g. decode an int in the stream into a float, etc.
  • Corner Cases: Overflows, nil maps/slices, nil values in streams are handled correctly
  • Standard field renaming via tags
  • Support for omitting empty fields during an encoding
  • Encoding from any value and decoding into pointer to any value (struct, slice, map, primitives, pointers, interface{}, etc)
  • Extensions to support efficient encoding/decoding of any named types
  • Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
  • Decoding without a schema (into a interface{}). Includes Options to configure what specific map or slice type to use when decoding an encoded list or map into a nil interface{}
  • Encode a struct as an array, and decode struct from an array in the data stream
  • Comprehensive support for anonymous fields
  • Fast (no-reflection) encoding/decoding of common maps and slices
  • Code-generation for faster performance.
  • Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
  • Support indefinite-length formats to enable true streaming (for formats which support it e.g. json, cbor)
  • Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes. This mostly applies to maps, where iteration order is non-deterministic.
  • NIL in data stream decoded as zero value
  • Never silently skip data when decoding. User decides whether to return an error or silently skip data when keys or indexes in the data stream do not map to fields in the struct.
  • Encode/Decode from/to chan types (for iterative streaming support)
  • Drop-in replacement for encoding/json. json: key in struct tag supported.
  • Provides a RPC Server and Client Codec for net/rpc communication protocol.
  • Handle unique idiosynchracies of codecs e.g.

Extension Support

Users can register a function to handle the encoding or decoding of their custom types.

There are no restrictions on what the custom type can be. Some examples:

type BisSet   []int
type BitSet64 uint64
type UUID     string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }

As an illustration, MyStructWithUnexportedFields would normally be encoded as an empty map because it has no exported fields, while UUID would be encoded as a string. However, with extension support, you can encode any of these however you like.

RPC

RPC Client and Server Codecs are implemented, so the codecs can be used with the standard net/rpc package.

Usage

Typical usage model:

// create and configure Handle
var (
  bh codec.BincHandle
  mh codec.MsgpackHandle
  ch codec.CborHandle
)

mh.MapType = reflect.TypeOf(map[string]interface{}(nil))

// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)

// create and use decoder/encoder
var (
  r io.Reader
  w io.Writer
  b []byte
  h = &bh // or mh to use msgpack
)

dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v)

enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)

//RPC Server
go func() {
    for {
        conn, err := listener.Accept()
        rpcCodec := codec.GoRpc.ServerCodec(conn, h)
        //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
        rpc.ServeCodec(rpcCodec)
    }
}()

//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)

Documentation

Overview

High Performance, Feature-Rich Idiomatic Go codec/encoding library for binc, msgpack, cbor, json.

Supported Serialization formats are:

To install: 

go get github.com/ugorji/go/codec

This package understands the 'unsafe' tag, to allow using unsafe semantics:

  • When decoding into a struct, you need to read the field name as a string so you can find the struct field it is mapped to. Using `unsafe` will bypass the allocation and copying overhead of []byte->string conversion.

To install using unsafe, pass the 'unsafe' tag: 

go get -tags=unsafe github.com/ugorji/go/codec

For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer .

The idiomatic Go support is as seen in other encoding packages in the standard library (ie json, xml, gob, etc).

Rich Feature Set includes:

  • Simple but extremely powerful and feature-rich API
  • Very High Performance. Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
  • Multiple conversions: Package coerces types where appropriate e.g. decode an int in the stream into a float, etc.
  • Corner Cases: Overflows, nil maps/slices, nil values in streams are handled correctly
  • Standard field renaming via tags
  • Support for omitting empty fields during an encoding
  • Encoding from any value and decoding into pointer to any value (struct, slice, map, primitives, pointers, interface{}, etc)
  • Extensions to support efficient encoding/decoding of any named types
  • Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
  • Decoding without a schema (into a interface{}). Includes Options to configure what specific map or slice type to use when decoding an encoded list or map into a nil interface{}
  • Encode a struct as an array, and decode struct from an array in the data stream
  • Comprehensive support for anonymous fields
  • Fast (no-reflection) encoding/decoding of common maps and slices
  • Code-generation for faster performance.
  • Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
  • Support indefinite-length formats to enable true streaming (for formats which support it e.g. json, cbor)
  • Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes. This mostly applies to maps, where iteration order is non-deterministic.
  • NIL in data stream decoded as zero value
  • Never silently skip data when decoding. User decides whether to return an error or silently skip data when keys or indexes in the data stream do not map to fields in the struct.
  • Detect and error when encoding a cyclic reference (instead of stack overflow shutdown)
  • Encode/Decode from/to chan types (for iterative streaming support)
  • Drop-in replacement for encoding/json. `json:` key in struct tag supported.
  • Provides a RPC Server and Client Codec for net/rpc communication protocol.
  • Handle unique idiosynchracies of codecs e.g.
  • For messagepack, configure how ambiguities in handling raw bytes are resolved
  • For messagepack, provide rpc server/client codec to support msgpack-rpc protocol defined at: https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md

Extension Support

Users can register a function to handle the encoding or decoding of their custom types.

There are no restrictions on what the custom type can be. Some examples: 

type BisSet   []int
type BitSet64 uint64
type UUID     string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }

As an illustration, MyStructWithUnexportedFields would normally be encoded as an empty map because it has no exported fields, while UUID would be encoded as a string. However, with extension support, you can encode any of these however you like.

RPC

RPC Client and Server Codecs are implemented, so the codecs can be used with the standard net/rpc package.

Usage

The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification.

The Encoder and Decoder are NOT safe for concurrent use.

Consequently, the usage model is basically:

  • Create and initialize the Handle before any use. Once created, DO NOT modify it.
  • Multiple Encoders or Decoders can now use the Handle concurrently. They only read information off the Handle (never write).
  • However, each Encoder or Decoder MUST not be used concurrently
  • To re-use an Encoder/Decoder, call Reset(...) on it first. This allows you use state maintained on the Encoder/Decoder.

Sample usage model: 

// create and configure Handle
var (
  bh codec.BincHandle
  mh codec.MsgpackHandle
  ch codec.CborHandle
)

mh.MapType = reflect.TypeOf(map[string]interface{}(nil))

// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)

// create and use decoder/encoder
var (
  r io.Reader
  w io.Writer
  b []byte
  h = &bh // or mh to use msgpack
)

dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v)

enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)

//RPC Server
go func() {
    for {
        conn, err := listener.Accept()
        rpcCodec := codec.GoRpc.ServerCodec(conn, h)
        //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
        rpc.ServeCodec(rpcCodec)
    }
}()

//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)

MSGPACK

Msgpack-c implementation powers the c, c++, python, ruby, etc libraries. We need to maintain compatibility with it and how it encodes integer values without caring about the type.

For compatibility with behaviour of msgpack-c reference implementation:

  • Go intX (>0) and uintX IS ENCODED AS msgpack +ve fixnum, unsigned
  • Go intX (<0) IS ENCODED AS msgpack -ve fixnum, signed

Index

Constants

View Source 
const (
	CborStreamBytes  byte = 0x5f
	CborStreamString      = 0x7f
	CborStreamArray       = 0x9f
	CborStreamMap         = 0xbf
	CborStreamBreak       = 0xff
)
View Source 
const (
	// AsSymbolDefault is default.
	// Currently, this means only encode struct field names as symbols.
	// The default is subject to change.
	AsSymbolDefault AsSymbolFlag = iota

	// AsSymbolAll means encode anything which could be a symbol as a symbol.
	AsSymbolAll = 0xfe

	// AsSymbolNone means do not encode anything as a symbol.
	AsSymbolNone = 1 << iota

	// AsSymbolMapStringKeys means encode keys in map[string]XXX as symbols.
	AsSymbolMapStringKeysFlag

	// AsSymbolStructFieldName means encode struct field names as symbols.
	AsSymbolStructFieldNameFlag
)
View Source 
const GenVersion = 5

GenVersion is the current version of codecgen.

NOTE: Increment this value each time codecgen changes fundamentally. Fundamental changes are:

  • helper methods change (signature change, new ones added, some removed, etc)
  • codecgen command line changes

v1: Initial Version v2: v3: Changes for Kubernetes: 

changes in signature of some unpublished helper methods and codecgen cmdline arguments.

v4: Removed separator support from (en|de)cDriver, and refactored codec(gen) v5: changes to support faster json decoding. Let encoder/decoder maintain state of collections.

Variables

View Source 
var GoRpc goRpc

GoRpc implements Rpc using the communication protocol defined in net/rpc package. Its methods (ServerCodec and ClientCodec) return values that implement RpcCodecBuffered.

View Source 
var MsgpackSpecRpc msgpackSpecRpc

MsgpackSpecRpc implements Rpc using the communication protocol defined in the msgpack spec at https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md . Its methods (ServerCodec and ClientCodec) return values that implement RpcCodecBuffered.

Functions

func Gen 

func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeInfos, typ ...reflect.Type)

Gen will write a complete go file containing Selfer implementations for each type passed. All the types must be in the same package.

Library users: *DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.*

func GenHelperDecoder 

func GenHelperDecoder(d *Decoder) (genHelperDecoder, decDriver)

GenHelperDecoder is exported so that it can be used externally by codecgen. Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.

func GenHelperEncoder 

func GenHelperEncoder(e *Encoder) (genHelperEncoder, encDriver)

GenHelperEncoder is exported so that it can be used externally by codecgen. Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.

func NoopHandle 

func NoopHandle(slen int) *noopHandle

NoopHandle returns a no-op handle. It basically does nothing. It is only useful for benchmarking, as it gives an idea of the overhead from the codec framework.

LIBRARY USERS: *** DO NOT USE ***

Types

type AsSymbolFlag 

type AsSymbolFlag uint8

AsSymbolFlag defines what should be encoded as symbols.

type BasicHandle 

type BasicHandle struct {
	// TypeInfos is used to get the type info for any type.
	//
	// If not configured, the default TypeInfos is used, which uses struct tag keys: codec, json
	TypeInfos *TypeInfos

	EncodeOptions
	DecodeOptions
	// contains filtered or unexported fields
}

WARNING: DO NOT USE DIRECTLY. EXPORTED FOR GODOC BENEFIT. WILL BE REMOVED.

BasicHandle encapsulates the common options and extension functions.

func (*BasicHandle) AddExt 

func (o *BasicHandle) AddExt(
	rt reflect.Type, tag byte,
	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

AddExt registes an encode and decode function for a reflect.Type. AddExt internally calls SetExt. To deregister an Ext, call AddExt with nil encfn and/or nil decfn.

func (*BasicHandle) SetExt 

func (o *BasicHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

Note that the type must be a named type, and specifically not a pointer or Interface. An error is returned if that is not honored.

To Deregister an ext, call SetExt with nil Ext

type BincHandle 

type BincHandle struct {
	BasicHandle
	// contains filtered or unexported fields
}

BincHandle is a Handle for the Binc Schema-Free Encoding Format defined at https://github.com/ugorji/binc .

BincHandle currently supports all Binc features with the following EXCEPTIONS:

  • only integers up to 64 bits of precision are supported. big integers are unsupported.
  • Only IEEE 754 binary32 and binary64 floats are supported (ie Go float32 and float64 types). extended precision and decimal IEEE 754 floats are unsupported.
  • Only UTF-8 strings supported. Unicode_Other Binc types (UTF16, UTF32) are currently unsupported.

Note that these EXCEPTIONS are temporary and full support is possible and may happen soon.

func (*BincHandle) AddExt 

func (o *BincHandle) AddExt(
	rt reflect.Type, tag byte,
	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

AddExt registes an encode and decode function for a reflect.Type. AddExt internally calls SetExt. To deregister an Ext, call AddExt with nil encfn and/or nil decfn.

func (*BincHandle) SetBytesExt 

func (h *BincHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error)

func (*BincHandle) SetExt 

func (o *BincHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

Note that the type must be a named type, and specifically not a pointer or Interface. An error is returned if that is not honored.

To Deregister an ext, call SetExt with nil Ext

type BytesExt 

type BytesExt interface {
	// WriteExt converts a value to a []byte.
	//
	// Note: v *may* be a pointer to the extension type, if the extension type was a struct or array.
	WriteExt(v interface{}) []byte

	// ReadExt updates a value from a []byte.
	ReadExt(dst interface{}, src []byte)
}

BytesExt handles custom (de)serialization of types to/from []byte. It is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types.

type CborHandle 

type CborHandle struct {
	BasicHandle
	// contains filtered or unexported fields
}

CborHandle is a Handle for the CBOR encoding format, defined at http://tools.ietf.org/html/rfc7049 and documented further at http://cbor.io .

CBOR is comprehensively supported, including support for:

  • indefinite-length arrays/maps/bytes/strings
  • (extension) tags in range 0..0xffff (0 .. 65535)
  • half, single and double-precision floats
  • all numbers (1, 2, 4 and 8-byte signed and unsigned integers)
  • nil, true, false, ...
  • arrays and maps, bytes and text strings

None of the optional extensions (with tags) defined in the spec are supported out-of-the-box. Users can implement them as needed (using SetExt), including spec-documented ones:

  • timestamp, BigNum, BigFloat, Decimals, Encoded Text (e.g. URL, regexp, base64, MIME Message), etc.

To encode with indefinite lengths (streaming), users will use (Must)Encode methods of *Encoder, along with writing CborStreamXXX constants.

For example, to encode "one-byte" as an indefinite length string: 

var buf bytes.Buffer
e := NewEncoder(&buf, new(CborHandle))
buf.WriteByte(CborStreamString)
e.MustEncode("one-")
e.MustEncode("byte")
buf.WriteByte(CborStreamBreak)
encodedBytes := buf.Bytes()
var vv interface{}
NewDecoderBytes(buf.Bytes(), new(CborHandle)).MustDecode(&vv)
// Now, vv contains the same string "one-byte"

func (*CborHandle) AddExt 

func (o *CborHandle) AddExt(
	rt reflect.Type, tag byte,
	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

AddExt registes an encode and decode function for a reflect.Type. AddExt internally calls SetExt. To deregister an Ext, call AddExt with nil encfn and/or nil decfn.

func (*CborHandle) SetExt 

func (o *CborHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

Note that the type must be a named type, and specifically not a pointer or Interface. An error is returned if that is not honored.

To Deregister an ext, call SetExt with nil Ext

func (*CborHandle) SetInterfaceExt 

func (h *CborHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error)

type DecodeOptions 

type DecodeOptions struct {
	// MapType specifies type to use during schema-less decoding of a map in the stream.
	// If nil, we use map[interface{}]interface{}
	MapType reflect.Type

	// SliceType specifies type to use during schema-less decoding of an array in the stream.
	// If nil, we use []interface{}
	SliceType reflect.Type

	// MaxInitLen defines the initial length that we "make" a collection (slice, chan or map) with.
	// If 0 or negative, we default to a sensible value based on the size of an element in the collection.
	//
	// For example, when decoding, a stream may say that it has MAX_UINT elements.
	// We should not auto-matically provision a slice of that length, to prevent Out-Of-Memory crash.
	// Instead, we provision up to MaxInitLen, fill that up, and start appending after that.
	MaxInitLen int

	// If ErrorIfNoField, return an error when decoding a map
	// from a codec stream into a struct, and no matching struct field is found.
	ErrorIfNoField bool

	// If ErrorIfNoArrayExpand, return an error when decoding a slice/array that cannot be expanded.
	// For example, the stream contains an array of 8 items, but you are decoding into a [4]T array,
	// or you are decoding into a slice of length 4 which is non-addressable (and so cannot be set).
	ErrorIfNoArrayExpand bool

	// If SignedInteger, use the int64 during schema-less decoding of unsigned values (not uint64).
	SignedInteger bool

	// MapValueReset controls how we decode into a map value.
	//
	// By default, we MAY retrieve the mapping for a key, and then decode into that.
	// However, especially with big maps, that retrieval may be expensive and unnecessary
	// if the stream already contains all that is necessary to recreate the value.
	//
	// If true, we will never retrieve the previous mapping,
	// but rather decode into a new value and set that in the map.
	//
	// If false, we will retrieve the previous mapping if necessary e.g.
	// the previous mapping is a pointer, or is a struct or array with pre-set state,
	// or is an interface.
	MapValueReset bool

	// InterfaceReset controls how we decode into an interface.
	//
	// By default, when we see a field that is an interface{...},
	// or a map with interface{...} value, we will attempt decoding into the
	// "contained" value.
	//
	// However, this prevents us from reading a string into an interface{}
	// that formerly contained a number.
	//
	// If true, we will decode into a new "blank" value, and set that in the interface.
	// If false, we will decode into whatever is contained in the interface.
	InterfaceReset bool

	// InternString controls interning of strings during decoding.
	//
	// Some handles, e.g. json, typically will read map keys as strings.
	// If the set of keys are finite, it may help reduce allocation to
	// look them up from a map (than to allocate them afresh).
	//
	// Note: Handles will be smart when using the intern functionality.
	// So everything will not be interned.
	InternString bool
}

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

A Decoder reads and decodes an object from an input stream in the codec format.

func NewDecoder 

func NewDecoder(r io.Reader, h Handle) *Decoder

NewDecoder returns a Decoder for decoding a stream of bytes from an io.Reader.

For efficiency, Users are encouraged to pass in a memory buffered reader (eg bufio.Reader, bytes.Buffer).

func NewDecoderBytes 

func NewDecoderBytes(in []byte, h Handle) *Decoder

NewDecoderBytes returns a Decoder which efficiently decodes directly from a byte slice with zero copying.

func (*Decoder) Decode 

func (d *Decoder) Decode(v interface{}) (err error)

Decode decodes the stream from reader and stores the result in the value pointed to by v. v cannot be a nil pointer. v can also be a reflect.Value of a pointer.

Note that a pointer to a nil interface is not a nil pointer. If you do not know what type of stream it is, pass in a pointer to a nil interface. We will decode and store a value in that nil interface.

Sample usages: 

// Decoding into a non-nil typed value
var f float32
err = codec.NewDecoder(r, handle).Decode(&f)

// Decoding into nil interface
var v interface{}
dec := codec.NewDecoder(r, handle)
err = dec.Decode(&v)

When decoding into a nil interface{}, we will decode into an appropriate value based on the contents of the stream:

  • Numbers are decoded as float64, int64 or uint64.
  • Other values are decoded appropriately depending on the type: bool, string, []byte, time.Time, etc
  • Extensions are decoded as RawExt (if no ext function registered for the tag)

Configurations exist on the Handle to override defaults (e.g. for MapType, SliceType and how to decode raw bytes).

When decoding into a non-nil interface{} value, the mode of encoding is based on the type of the value. When a value is seen:

  • If an extension is registered for it, call that extension function
  • If it implements BinaryUnmarshaler, call its UnmarshalBinary(data []byte) error
  • Else decode it based on its reflect.Kind

There are some special rules when decoding into containers (slice/array/map/struct). Decode will typically use the stream contents to UPDATE the container.

  • A map can be decoded from a stream map, by updating matching keys.
  • A slice can be decoded from a stream array, by updating the first n elements, where n is length of the stream.
  • A slice can be decoded from a stream map, by decoding as if it contains a sequence of key-value pairs.
  • A struct can be decoded from a stream map, by updating matching fields.
  • A struct can be decoded from a stream array, by updating fields as they occur in the struct (by index).

When decoding a stream map or array with length of 0 into a nil map or slice, we reset the destination map or slice to a zero-length value.

However, when decoding a stream nil, we reset the destination container to its "zero" value (e.g. nil for slice/map, etc).

func (*Decoder) MustDecode 

func (d *Decoder) MustDecode(v interface{})

MustDecode is like Decode, but panics if unable to Decode. This provides insight to the code location that triggered the error.

func (*Decoder) Reset 

func (d *Decoder) Reset(r io.Reader)

func (*Decoder) ResetBytes 

func (d *Decoder) ResetBytes(in []byte)

type EncodeOptions 

type EncodeOptions struct {
	// Encode a struct as an array, and not as a map
	StructToArray bool

	// Canonical representation means that encoding a value will always result in the same
	// sequence of bytes.
	//
	// This only affects maps, as the iteration order for maps is random.
	//
	// The implementation MAY use the natural sort order for the map keys if possible:
	//
	//     - If there is a natural sort order (ie for number, bool, string or []byte keys),
	//       then the map keys are first sorted in natural order and then written
	//       with corresponding map values to the strema.
	//     - If there is no natural sort order, then the map keys will first be
	//       encoded into []byte, and then sorted,
	//       before writing the sorted keys and the corresponding map values to the stream.
	//
	Canonical bool

	// CheckCircularRef controls whether we check for circular references
	// and error fast during an encode.
	//
	// If enabled, an error is received if a pointer to a struct
	// references itself either directly or through one of its fields (iteratively).
	//
	// This is opt-in, as there may be a performance hit to checking circular references.
	CheckCircularRef bool

	// AsSymbols defines what should be encoded as symbols.
	//
	// Encoding as symbols can reduce the encoded size significantly.
	//
	// However, during decoding, each string to be encoded as a symbol must
	// be checked to see if it has been seen before. Consequently, encoding time
	// will increase if using symbols, because string comparisons has a clear cost.
	//
	// Sample values:
	//   AsSymbolNone
	//   AsSymbolAll
	//   AsSymbolMapStringKeys
	//   AsSymbolMapStringKeysFlag | AsSymbolStructFieldNameFlag
	AsSymbols AsSymbolFlag
}

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

An Encoder writes an object to an output stream in the codec format.

func NewEncoder 

func NewEncoder(w io.Writer, h Handle) *Encoder

NewEncoder returns an Encoder for encoding into an io.Writer.

For efficiency, Users are encouraged to pass in a memory buffered writer (eg bufio.Writer, bytes.Buffer).

func NewEncoderBytes 

func NewEncoderBytes(out *[]byte, h Handle) *Encoder

NewEncoderBytes returns an encoder for encoding directly and efficiently into a byte slice, using zero-copying to temporary slices.

It will potentially replace the output byte slice pointed to. After encoding, the out parameter contains the encoded contents.

func (*Encoder) Encode 

func (e *Encoder) Encode(v interface{}) (err error)

Encode writes an object into a stream.

Encoding can be configured via the struct tag for the fields. The "codec" key in struct field's tag value is the key name, followed by an optional comma and options. Note that the "json" key is used in the absence of the "codec" key.

To set an option on all fields (e.g. omitempty on all fields), you can create a field called _struct, and set flags on it.

Struct values "usually" encode as maps. Each exported struct field is encoded unless:

  • the field's tag is "-", OR
  • the field is empty (empty or the zero value) and its tag specifies the "omitempty" option.

When encoding as a map, the first string in the tag (before the comma) is the map key string to use when encoding.

However, struct values may encode as arrays. This happens when:

  • StructToArray Encode option is set, OR
  • the tag on the _struct field sets the "toarray" option

Values with types that implement MapBySlice are encoded as stream maps.

The empty values (for omitempty option) are false, 0, any nil pointer or interface value, and any array, slice, map, or string of length zero.

Anonymous fields are encoded inline except:

  • the struct tag specifies a replacement name (first value)
  • the field is of an interface type

Examples: 

// NOTE: 'json:' can be used as struct tag key, in place 'codec:' below.
type MyStruct struct {
    _struct bool    `codec:",omitempty"`   //set omitempty for every field
    Field1 string   `codec:"-"`            //skip this field
    Field2 int      `codec:"myName"`       //Use key "myName" in encode stream
    Field3 int32    `codec:",omitempty"`   //use key "Field3". Omit if empty.
    Field4 bool     `codec:"f4,omitempty"` //use key "f4". Omit if empty.
    io.Reader                              //use key "Reader".
    MyStruct        `codec:"my1"           //use key "my1".
    MyStruct                               //inline it
    ...
}

type MyStruct struct {
    _struct bool    `codec:",omitempty,toarray"`   //set omitempty for every field
                                                   //and encode struct as an array
}

The mode of encoding is based on the type of the value. When a value is seen:

  • If a Selfer, call its CodecEncodeSelf method
  • If an extension is registered for it, call that extension function
  • If it implements encoding.(Binary|Text|JSON)Marshaler, call its Marshal(Binary|Text|JSON) method
  • Else encode it based on its reflect.Kind

Note that struct field names and keys in map[string]XXX will be treated as symbols. Some formats support symbols (e.g. binc) and will properly encode the string only once in the stream, and use a tag to refer to it thereafter.

func (*Encoder) MustEncode 

func (e *Encoder) MustEncode(v interface{})

MustEncode is like Encode, but panics if unable to Encode. This provides insight to the code location that triggered the error.

func (*Encoder) Reset 

func (e *Encoder) Reset(w io.Writer)

Reset the Encoder with a new output stream.

This accomodates using the state of the Encoder, where it has "cached" information about sub-engines.

func (*Encoder) ResetBytes 

func (e *Encoder) ResetBytes(out *[]byte)

type Ext 

type Ext interface {
	BytesExt
	InterfaceExt
}

Ext handles custom (de)serialization of custom types / extensions.

type Handle 

type Handle interface {
	// contains filtered or unexported methods
}

Handle is the interface for a specific encoding format.

Typically, a Handle is pre-configured before first time use, and not modified while in use. Such a pre-configured Handle is safe for concurrent access.

type InterfaceExt 

type InterfaceExt interface {
	// ConvertExt converts a value into a simpler interface for easy encoding e.g. convert time.Time to int64.
	//
	// Note: v *may* be a pointer to the extension type, if the extension type was a struct or array.
	ConvertExt(v interface{}) interface{}

	// UpdateExt updates a value from a simpler interface for easy decoding e.g. convert int64 to time.Time.
	UpdateExt(dst interface{}, src interface{})
}

InterfaceExt handles custom (de)serialization of types to/from another interface{} value. The Encoder or Decoder will then handle the further (de)serialization of that known type.

It is used by codecs (e.g. cbor, json) which use the format to do custom serialization of the types.

type JsonHandle 

type JsonHandle struct {
	BasicHandle
	// RawBytesExt, if configured, is used to encode and decode raw bytes in a custom way.
	// If not configured, raw bytes are encoded to/from base64 text.
	RawBytesExt InterfaceExt

	// Indent indicates how a value is encoded.
	//   - If positive, indent by that number of spaces.
	//   - If negative, indent by that number of tabs.
	Indent int8

	// IntegerAsString controls how integers (signed and unsigned) are encoded.
	//
	// Per the JSON Spec, JSON numbers are 64-bit floating point numbers.
	// Consequently, integers > 2^53 cannot be represented as a JSON number without losing precision.
	// This can be mitigated by configuring how to encode integers.
	//
	// IntegerAsString interpretes the following values:
	//   - if 'L', then encode integers > 2^53 as a json string.
	//   - if 'A', then encode all integers as a json string
	//             containing the exact integer representation as a decimal.
	//   - else    encode all integers as a json number (default)
	IntegerAsString uint8
	// contains filtered or unexported fields
}

JsonHandle is a handle for JSON encoding format.

Json is comprehensively supported:

  • decodes numbers into interface{} as int, uint or float64
  • configurable way to encode/decode []byte . by default, encodes and decodes []byte using base64 Std Encoding
  • UTF-8 support for encoding and decoding

It has better performance than the json library in the standard library, by leveraging the performance improvements of the codec library and minimizing allocations.

In addition, it doesn't read more bytes than necessary during a decode, which allows reading multiple values from a stream containing json and non-json content. For example, a user can read a json value, then a cbor value, then a msgpack value, all from the same stream in sequence.

func (*JsonHandle) AddExt 

func (o *JsonHandle) AddExt(
	rt reflect.Type, tag byte,
	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

AddExt registes an encode and decode function for a reflect.Type. AddExt internally calls SetExt. To deregister an Ext, call AddExt with nil encfn and/or nil decfn.

func (*JsonHandle) SetExt 

func (o *JsonHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

Note that the type must be a named type, and specifically not a pointer or Interface. An error is returned if that is not honored.

To Deregister an ext, call SetExt with nil Ext

func (*JsonHandle) SetInterfaceExt 

func (h *JsonHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error)

type MapBySlice 

type MapBySlice interface {
	MapBySlice()
}

MapBySlice represents a slice which should be encoded as a map in the stream. The slice contains a sequence of key-value pairs. This affords storing a map in a specific sequence in the stream.

The support of MapBySlice affords the following:

  • A slice type which implements MapBySlice will be encoded as a map
  • A slice can be decoded from a map in the stream

type MsgpackHandle 

type MsgpackHandle struct {
	BasicHandle

	// RawToString controls how raw bytes are decoded into a nil interface{}.
	RawToString bool

	// WriteExt flag supports encoding configured extensions with extension tags.
	// It also controls whether other elements of the new spec are encoded (ie Str8).
	//
	// With WriteExt=false, configured extensions are serialized as raw bytes
	// and Str8 is not encoded.
	//
	// A stream can still be decoded into a typed value, provided an appropriate value
	// is provided, but the type cannot be inferred from the stream. If no appropriate
	// type is provided (e.g. decoding into a nil interface{}), you get back
	// a []byte or string based on the setting of RawToString.
	WriteExt bool
	// contains filtered or unexported fields
}

MsgpackHandle is a Handle for the Msgpack Schema-Free Encoding Format.

func (*MsgpackHandle) AddExt 

func (o *MsgpackHandle) AddExt(
	rt reflect.Type, tag byte,
	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

AddExt registes an encode and decode function for a reflect.Type. AddExt internally calls SetExt. To deregister an Ext, call AddExt with nil encfn and/or nil decfn.

func (*MsgpackHandle) SetBytesExt 

func (h *MsgpackHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error)

func (*MsgpackHandle) SetExt 

func (o *MsgpackHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

Note that the type must be a named type, and specifically not a pointer or Interface. An error is returned if that is not honored.

To Deregister an ext, call SetExt with nil Ext

type MsgpackSpecRpcMultiArgs 

type MsgpackSpecRpcMultiArgs []interface{}

MsgpackSpecRpcMultiArgs is a special type which signifies to the MsgpackSpecRpcCodec that the backend RPC service takes multiple arguments, which have been arranged in sequence in the slice.

The Codec then passes it AS-IS to the rpc service (without wrapping it in an array of 1 element).

type RawExt 

type RawExt struct {
	Tag uint64
	// Data is the []byte which represents the raw ext. If Data is nil, ext is exposed in Value.
	// Data is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types
	Data []byte
	// Value represents the extension, if Data is nil.
	// Value is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
	Value interface{}
}

RawExt represents raw unprocessed extension data. Some codecs will decode extension data as a *RawExt if there is no registered extension for the tag.

Only one of Data or Value is nil. If Data is nil, then the content of the RawExt is in the Value.

type Rpc 

type Rpc interface {
	ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec
	ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec
}

Rpc provides a rpc Server or Client Codec for rpc communication.

type RpcCodecBuffered 

type RpcCodecBuffered interface {
	BufferedReader() *bufio.Reader
	BufferedWriter() *bufio.Writer
}

RpcCodecBuffered allows access to the underlying bufio.Reader/Writer used by the rpc connection. It accomodates use-cases where the connection should be used by rpc and non-rpc functions, e.g. streaming a file after sending an rpc response.

type Selfer 

type Selfer interface {
	CodecEncodeSelf(*Encoder)
	CodecDecodeSelf(*Decoder)
}

Selfer defines methods by which a value can encode or decode itself.

Any type which implements Selfer will be able to encode or decode itself. Consequently, during (en|de)code, this takes precedence over (text|binary)(M|Unm)arshal or extension support.

type SimpleHandle 

type SimpleHandle struct {
	BasicHandle
	// contains filtered or unexported fields
}

SimpleHandle is a Handle for a very simple encoding format.

simple is a simplistic codec similar to binc, but not as compact.

  • Encoding of a value is always preceeded by the descriptor byte (bd)
  • True, false, nil are encoded fully in 1 byte (the descriptor)
  • Integers (intXXX, uintXXX) are encoded in 1, 2, 4 or 8 bytes (plus a descriptor byte). There are positive (uintXXX and intXXX >= 0) and negative (intXXX < 0) integers.
  • Floats are encoded in 4 or 8 bytes (plus a descriptor byte)
  • Lenght of containers (strings, bytes, array, map, extensions) are encoded in 0, 1, 2, 4 or 8 bytes. Zero-length containers have no length encoded. For others, the number of bytes is given by pow(2, bd%3)
  • maps are encoded as [bd] [length] [[key][value]]...
  • arrays are encoded as [bd] [length] [value]...
  • extensions are encoded as [bd] [length] [tag] [byte]...
  • strings/bytearrays are encoded as [bd] [length] [byte]...

The full spec will be published soon.

func (*SimpleHandle) AddExt 

func (o *SimpleHandle) AddExt(
	rt reflect.Type, tag byte,
	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

AddExt registes an encode and decode function for a reflect.Type. AddExt internally calls SetExt. To deregister an Ext, call AddExt with nil encfn and/or nil decfn.

func (*SimpleHandle) SetBytesExt 

func (h *SimpleHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error)

func (*SimpleHandle) SetExt 

func (o *SimpleHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error)

DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.

Note that the type must be a named type, and specifically not a pointer or Interface. An error is returned if that is not honored.

To Deregister an ext, call SetExt with nil Ext

type TypeInfos 

type TypeInfos struct {
	// contains filtered or unexported fields
}

TypeInfos caches typeInfo for each type on first inspection.

It is configured with a set of tag keys, which are used to get configuration for the type.

func NewTypeInfos 

func NewTypeInfos(tags []string) *TypeInfos

NewTypeInfos creates a TypeInfos given a set of struct tags keys.

This allows users customize the struct tag keys which contain configuration of their types.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.