structform

README

Go-Structform - Structured Formatters

go-structform provides capabilities for serializing, desirializing, and transcoding structured data efficiently and generically.

The top-level package provides the common layer by which serializers and deserializers interact with each other. Serializers convert the input into a stream of events by calling methods on the Visitor interfaces. The Deserializers implement the Visitor interface to convert the stream of events into the target structure.

A type implementing the Visitor interface, but forwards calls to another visitor is called a Transformer or Filter. Transformers/Filters can manipulate the data stream or create/collect errors if some wanted validation fails.

Examples

Transcode a stream of JSON objects into a stream of CBOR objects:

func TranscodeJSON2CBOR(out io.Writer, in io.Reader) (int64, error) {
	bytesCount, err := json.ParseReader(in, cborl.NewVisitor(out))
	return bytesCount, err
}

Parse a stream of JSON objects:

	var in io.Reader = ...
	visitor, _ := gotype.NewUnfolder(nil)
	dec := json.NewDecoder(in, 2048, visitor)
	for {
		var to interface{}
		visitor.SetTarget(&to)
		if err := dec.Next(); err != nil {
			return err
		}

		// process `to`
	}

Encode a channel of go map[string]interface{} to a stream of cbor objects:

	var out io.Writer = ...
	var objects chan map[string]interface{} = ...

	visitor := cborl.NewVisitor(out)
	it, _ := gotype.NewIterator(visitor)
	for _, obj := range objects {
		if err := it.Fold(obj); err != nil {
			return err
		}
	}

	return nil

Convert between go types:

	var to map[string]int
	visitor, _ := gotype.NewUnfolder(&to)
	it, _ := gotype.NewIterator(visitor)

	st := struct {
		A int    `struct:"a"`
		B string `struct:",omit"`
	}{A: 1, B: "hello"}
	if err := it.Fold(st); err != nil {
		return err
	}

	// to == map[string]int{"a": 1}

Use custom folder and unfolder for existing go types:

type durationUnfolder struct {
	gotype.BaseUnfoldState // Use BaseUnfoldState to create errors for methods not implemented
	to                     *time.Duration
}

func foldDuration(in *time.Duration, v structform.ExtVisitor) error {
	return v.OnString(in.String())
}

func unfoldDuration(to *time.Duration) gotype.UnfoldState {
	return &durationUnfolder{to: to}
}

type durationUnfolder struct {
	gotype.BaseUnfoldState // Use BaseUnfoldState to create errors for methods not implemented
	to                     *time.Duration
}

func (u *durationUnfolder) OnString(_ gotype.UnfoldCtx, str string) error {
	d, err := time.ParseDuration(str)
	if err == nil {
		*u.to = d
	}
	return err
}


...


	visitor, _ := gotype.NewUnfolder(nil, gotype.Unfolders(
		unfoldDuration,
	))
	it, _ := gotype.NewIterator(visitor, gotype.Folders(
		foldDuration,
	))

	// store duration in temporary value
	var tmp interface{}
	visitor.SetTarget(&tmp)
	it.Fold(5 * time.Minute)

	// restore duration from temporary value
	var dur time.Duration
	visitor.SetTarget(&dur)
	it.Fold(tmp)

	// dur == 5 * time.Minute

Data Model

The data model describes by which data types Serializers and Deserializers interact. In this sense the data model provides a simplified type system, that supports a subset of serializable go types (for example channels or function pointers can not be serialized).

Go-structform provides a simplified, common data model via the Visitor interface.

Types

• primitives (See ValueVisitor interface): Bool, Byte, String, Int, Int8/16/32/64, Uint, Uint8/16/32/64, Float32, Float64, untyped Nil
• compound: objects (ObjectVisitor), arrays (ArrayVisitor)

Extended Data Model

The extended data model provides support for similar types to the Visitor interface, but provides a number of optimizations, allowing users to pass a set of common go values directly without having to serialize and deserialize those values.

The extended data model is provided by the ExtVisitor interface.

All features in the Extended Data Model can be seamlessly converted to the Data Model provided by the `Visitor interface.

Deserializers supporting the Extended Data Model must always implement the common Data Model as is required by the Visitor interface.

Serializers wanting to interact with the Extended Data Model should still accept the Visitor interface only and use EnsureExtVisitor in order to create an ExtVisitor. EnsureExtVisitor wraps the Visitor if necessarry, allowing the Visitor to implement only a subset of features in the Extended Data Model.

• extended primitives: The Visitor adds support for []byte values as strings. The value must be consumed immediately or copied, as the buffer is not guaranteed to be stable.
• slices: The visitor adds support for slice types, for each supported primitive type. For example []int8/16/32/64 can be passed as is.
• map: The visitor adds support for map[string]T types, for each supported primitive type. For example map[string]int8/16/32/64.

Data Formats

• JSON: the json package provides a JSON parser and JSON serializer. The serializer implements a subset of ExtVisitor.
• UBJSON: the ubjson packages provides a parser and serializer for Universal Binary JSON.
• CBOR: the cborl package supports a compatible subset of CBOR (for example object keys must be strings).
• Go Types: the gotype package provides a Folder to convert go values into a stream of events and an Unfolder to apply a stream of events to go values.

Documentation

Index

• type ArrayValueVisitor
• type ArrayVisitor
• type BaseType
  • func (i BaseType) String() string
• type ExtVisitor
  • func EnsureExtVisitor(v Visitor) ExtVisitor
• type ObjectValueVisitor
• type ObjectVisitor
• type StringRefVisitor
  • func MakeStringRefVisitor(v Visitor) StringRefVisitor
• type ValueVisitor
• type Visitor

Types

type ArrayValueVisitor

type ArrayValueVisitor interface {
	OnBoolArray([]bool) error

	OnStringArray([]string) error

	// int
	OnInt8Array([]int8) error
	OnInt16Array([]int16) error
	OnInt32Array([]int32) error
	OnInt64Array([]int64) error
	OnIntArray([]int) error

	// uint
	OnBytes([]byte) error
	OnUint8Array([]uint8) error
	OnUint16Array([]uint16) error
	OnUint32Array([]uint32) error
	OnUint64Array([]uint64) error
	OnUintArray([]uint) error

	// float
	OnFloat32Array([]float32) error
	OnFloat64Array([]float64) error
}

ArrayValueVisitor passes arrays with known type. Implementation of ArrayValueVisitor is optional.

type ArrayVisitor

type ArrayVisitor interface {
	// OnArrayStart is called whan a new array is going to be reported.
	//
	// The `len` argument should report the length if known. `len` MUST BE -1 if
	// the length of the array is not known.  The BaseType should indicate the
	// element type of the array. If the element type is unknown or can be any
	// type (e.g. interface{}), AnyType must be used.
	OnArrayStart(len int, baseType BaseType) error

	// OnArrayFinished indicates that there are no more elements in the array.
	OnArrayFinished() error
}

ArrayVisitor defines the support for arrays/slices in the Data Model.

type BaseType

type BaseType uint8

const (
	AnyType BaseType = iota
	ByteType
	StringType
	BoolType
	ZeroType
	IntType
	Int8Type
	Int16Type
	Int32Type
	Int64Type
	UintType
	Uint8Type
	Uint16Type
	Uint32Type
	Uint64Type
	Float32Type
	Float64Type
)

func (BaseType) String

func (i BaseType) String() string

type ExtVisitor

type ExtVisitor interface {
	Visitor
	ArrayValueVisitor
	ObjectValueVisitor
	StringRefVisitor
}

ExtVisitor interface defines the Extended Data Model. Usage and implementation of the Extended Data Model is optional, but can speed up operations.

func EnsureExtVisitor

func EnsureExtVisitor(v Visitor) ExtVisitor

EnsureExtVisitor converts a Visitor into an ExtVisitor. If v already implements ExtVisitor, it is directly implemented. If v only implements a subset of ExtVisitor, then conversions for the missing interfaces will be created.

type ObjectValueVisitor

type ObjectValueVisitor interface {
	OnBoolObject(map[string]bool) error

	OnStringObject(map[string]string) error

	// int
	OnInt8Object(map[string]int8) error
	OnInt16Object(map[string]int16) error
	OnInt32Object(map[string]int32) error
	OnInt64Object(map[string]int64) error
	OnIntObject(map[string]int) error

	// uint
	OnUint8Object(map[string]uint8) error
	OnUint16Object(map[string]uint16) error
	OnUint32Object(map[string]uint32) error
	OnUint64Object(map[string]uint64) error
	OnUintObject(map[string]uint) error

	// float
	OnFloat32Object(map[string]float32) error
	OnFloat64Object(map[string]float64) error
}

ObjectValueVisitor passes map[string]T. Implementation of ObjectValueVisitor is optional.

type ObjectVisitor

type ObjectVisitor interface {
	// OnObjectStart is called when a new object (key-value pairs) is going to be reported.
	// A call to OnKey or OnObjectFinished must follow directly.
	OnObjectStart(len int, baseType BaseType) error

	// OnArrayFinished indicates that there are no more key value pairs to report.
	OnObjectFinished() error

	// OnKey adds a new key to the object. A value must directly follow a call to OnKey.
	OnKey(s string) error
}

ObjectVisitor iterates all fields in a dictionary like structure.

type StringRefVisitor

type StringRefVisitor interface {
	OnStringRef(s []byte) error
	OnKeyRef(s []byte) error
}

StringRefVisitor handles strings by reference into a byte string. The reference must be processed immediately, as the string passed might get modified after the callback returns.

func MakeStringRefVisitor

func MakeStringRefVisitor(v Visitor) StringRefVisitor

type ValueVisitor

type ValueVisitor interface {
	// untyped nil value
	OnNil() error

	OnBool(b bool) error

	OnString(s string) error

	// int
	OnInt8(i int8) error
	OnInt16(i int16) error
	OnInt32(i int32) error
	OnInt64(i int64) error
	OnInt(i int) error

	// uint
	OnByte(b byte) error
	OnUint8(u uint8) error
	OnUint16(u uint16) error
	OnUint32(u uint32) error
	OnUint64(u uint64) error
	OnUint(u uint) error

	// float
	OnFloat32(f float32) error
	OnFloat64(f float64) error
}

ValueVisitor defines the set of supported primitive types in the Data Model.

type Visitor

type Visitor interface {
	ObjectVisitor
	ArrayVisitor
	ValueVisitor
}

Visitor interface for accepting events. The Vistor defined the common Data Model all serializers should accept and all deserializers must implement.