tt

README

tt

TinyTranscoder is a library that can encode/decode map[interface{}]interface{} where the interface is any of the supported types or a Transmitter. tt is faster than GOB but does support less types and does use a significant more bytes to transmis the data

Benchmarks

Data is are benchmarks that use Data, map benchmarks just convert the data to a map[interface{}]interface{}.

r5 3600 make bench

BenchmarkV3               303894              3921 ns/op            1337 B/op         33 allocs/op
BenchmarkV3Decode         401946              2827 ns/op             778 B/op         28 allocs/op
BenchmarkV3Encode        1208140              1049 ns/op             558 B/op          5 allocs/op
BenchmarkV3int64         1657701               730 ns/op             572 B/op         11 allocs/op
BenchmarkV2               450891              2442 ns/op            1615 B/op         26 allocs/op
BenchmarkV2Decode         914095              1216 ns/op             776 B/op         22 allocs/op
BenchmarkV2Encode         962227              1174 ns/op             791 B/op          4 allocs/op
BenchmarkGobMap            99562             11806 ns/op            1916 B/op         68 allocs/op
BenchmarkGobMapDecode     183130              6511 ns/op            1260 B/op         48 allocs/op
BenchmarkGobMapEncode     235747              4983 ns/op             656 B/op         20 allocs/op

notes

• all tests are run using GOMAXPROCS=1, this is because on zen running on multiple threads will cause horrible cache-invalidation. A single alloc/op would cause the GC to run at some point, this would kick the benching to a diferent core. The reason I decided to run using GOMAXPROCS=1 is because this doesnt have a big impact on Intel cpus, and any real world application would be generating garbage anyways, so eleminitin the GC from running should be part of the benchmark. Another reason coul be: real world applications would so something else in between runs causing cache-invalidation anyways.

v3 vs v2

As seen in the benchmarks v3 is significantly slower than v2, this has a couple of reasons:

• v3 supports all kinds of maps and structs meaning we have to use reflect for a lot of things. We have some fast-paths for commonly used maps but this doesnt negate all performance loss.
• v3 supports encoding/decoding from all kinds of readerswriters, the added indirection adds a significant amount of overhead.
• v3 uses the google varints, this means that we have removed any size limitations but it does mean slower encoding/decoding. see #limitations in ttv2

other diferences are:

• map[interaface{}]interface{} and map[string]interface{} get converted to Data in v2 but not in v3. In v3 all maps are converted into map[interface{}]interface{} if no other type is provided, much like json and gob.

limitations in ttv2

ttv2 only supports up to 255 child nodes per node ttv2 only supports up to 2^32 number of nodes(each submap, array, and value is a node) text is up to 2^32 bytes long, same for the key

Documentation

Index

• Variables
• func Decodev1(b []byte, d *Data) (err error)
• func Decodev2(b []byte, d *Data) (err error)
• func Decodev3(buf v3.Reader, e interface{}) error
• func Encodev2(d Data, values *bytes.Buffer)
• func Encodev3(d interface{}, out io.Writer) error
• func RegisterTransmitter(tr Transmitter) error
• type Data
  • func (d *Data) GobDecode(data []byte) error
  • func (d Data) GobEncode() ([]byte, error)
  • func (d Data) Size() (int, error)
• type Key
• type Transmitter
• type V3Decoder
  • func NewV3Decoder(in v3.Reader, init bool) *V3Decoder
  • func (dec *V3Decoder) Decode(e interface{}) error
  • func (dec *V3Decoder) Init() error
  • func (dec *V3Decoder) SetAllocLimmit(limit uint64)
• type V3Encoder
  • func NewV3Encoder(out io.Writer, isStream bool) *V3Encoder
  • func (enc *V3Encoder) Encode(d interface{}) error

Variables

var (
	//ErrCodeUsed is for when the code for the transmitter is already used
	ErrCodeUsed = errors.New("code already used")

	//ErrInvalidInput is used for when the input it invalid
	ErrInvalidInput = errors.New("invalid input")
)

Functions

func Decodev1

func Decodev1(b []byte, d *Data) (err error)

Decodev1 decodes a ttv1 encoded byte-slice into tt.Data

func Decodev2

func Decodev2(b []byte, d *Data) (err error)

Decodev2 decodes a ttv2 encoded byte-slice into tt.Data

func Decodev3

func Decodev3(buf v3.Reader, e interface{}) error

Decodev3 decodes a ttv3 encoded byte-slice into tt.Data

func Encodev2

func Encodev2(d Data, values *bytes.Buffer)

Encodev2 encodes a tt.Data into a bytebuffer using ttv2

func Encodev3

func Encodev3(d interface{}, out io.Writer) error

Encodev3 encodes an `interface{}“ into a bytebuffer using ttv3

func RegisterTransmitter

func RegisterTransmitter(tr Transmitter) error

RegisterTransmitter registers a new transmitter

Types

type Data

type Data map[interface{}]interface{}

Data is the type

func (*Data) GobDecode

func (d *Data) GobDecode(data []byte) error

GobDecode decodes the data back into a map[interface{}]interface{}

func (Data) GobEncode

func (d Data) GobEncode() ([]byte, error)

GobEncode encodes the data of a map[interface{}]interface{}

func (Data) Size

func (d Data) Size() (int, error)

Size gives an accurate size value for the final size of buffer needed by encoding

type Key

type Key struct {
	Value []byte //the key to the data of this object in its final form
	Vtype byte   //the type of the data of this object in its final form
}

Key is the key used for storing the key of a Value

type Transmitter

type Transmitter interface {
	Encode() ([]byte, error)
	Decode([]byte) (interface{}, error)
	GetCode() byte
}

Transmitter transmits the data

type V3Decoder

type V3Decoder struct {
	sync.Mutex
	// contains filtered or unexported fields
}

V3Decoder is the decoder used to decode a ttv3 data stream

func NewV3Decoder

func NewV3Decoder(in v3.Reader, init bool) *V3Decoder

NewV3Decoder creates aa new V3Decoder to decode a ttv3 data stream. The init flag specifies wether it should initialize the decoder. Initializing the decoder blocks until at least the first 2 bytes are read.

func (*V3Decoder) Decode

func (dec *V3Decoder) Decode(e interface{}) error

Decode decodes a one ttv3 encoded value from a stream. Note that a stream of one value is the same as one value just with the stream bit set

func (*V3Decoder) Init

func (dec *V3Decoder) Init() error

Init initizes the decoder, initizlizing blocks until at least the first 2 bytes are read.

func (*V3Decoder) SetAllocLimmit

func (dec *V3Decoder) SetAllocLimmit(limit uint64)

SetAllocLimmit sets the limmit of allocations. This does not induce a global limmit in tt but only for individual allocations

type V3Encoder

type V3Encoder struct {
	sync.Mutex
	// contains filtered or unexported fields
}

V3Encoder is the encoder used to encode a ttv3 data stream

func NewV3Encoder

func NewV3Encoder(out io.Writer, isStream bool) *V3Encoder

NewV3Encoder creates a new encoder to encode a ttv3 data stream

func (*V3Encoder) Encode

func (enc *V3Encoder) Encode(d interface{}) error

Encode encodes an `interface{}` into a bytebuffer using ttv3