decoder

README

Decoder

Dynamic decoders based on inspector framework and vector packages, like jsonvector.

Basics

One of the major the problems we ran into was a necessity to convert tons of different response formats from external services into our internal response format. Also, there was a requirement to connect new services at runtime and provide a possibility to decode their responses without rebooting the application.

This package provides a possibility to describe decoding rules in Go-like meta-language and add new decoders or edit existing at runtime.

Decoders is a context-based and all variables should be preliminarily registered in special context before decoding.

Syntax

Recommend reading inspector first.

Each decoder consists of name and body. Name uses only as index and may contain any symbols. There is only one requirement - name should be unique.

Decoder's body is a multi-line string (that calls ruleset) and each line (rule) has the following format

<destination path> = [<source path>|<callback>]

Both <source path> and <destination path> describes a source/destination fields accessed using a dot, example:

dstObj.Name = user.FullName

As mentioned above, both dstObj and user variables should be preliminarily registered in the context like this

ctx := decoder.AcquireCtx()
ctx.Set("dstObj", dst, DstInspector{})
ctx.Set("user", user, UserInspector{})

In this example user is a Go struct, but you may use as source raw response, example in JSON

import _ "github.com/koykov/decoder_vector"
...
jsonResponse = []byte(`{"a":"foo"}`)
ctx := decoder.AcquireCtx()
ctx.SetStatic("raw", jsonResponse)
// in decoder body:
// ctx.response = vector::parseJSON(raw)
// dstObj.Name = response.a

In this way, decoder provides a possibility to describe where source data should be taken and where it should be came.

Enough easy, isn't it?

Modifiers

Sometimes, isn't enough just specify source data address, but need to modify it before assigning to destination.

Especially for that cases was added support of source modifiers. The syntax:

<destination path> = <source path>|<modifier name>(<arg0>, <arg1>, ...)

Example

dstObj.Balance = user.Finance.Rest|default(0)

Default is a built-in modifier, but you may register your own modifiers using modifiers registry:

func modMyCustomMod(ctx *Ctx, buf *any, val any, args []any) error {
    // ...
}

decoder.RegisterModFn("myCustomMod", "customMod", modMyCustomMod)

Modifier arguments:

• ctx is a storage of variables/buffers you may use.
• buf is a type-free buffer that receives result of modifier's work. Please note the type *any is an alloc-free trick.
• val if a value of variable from left side of modifier separator (|).
• args array of modifier arguments, specified in rule.

See mod.go for details and mod_builtin.go for more example of modifiers.

Extensions

Decoders may be extended by including modules to the project. Currently supported modules:

• decoder_vector provide support of vector parsers.
• decoder_i18n allows legacy features in the project.

To enable necessary module just import it to the project, eg:

import (
	_ "https://github.com/koykov/decoder_vector"
)

and vector's features will be available inside templates.

Feel free to develop your own extensions. Strongly recommend to register new modifiers using namespaces, like this.

Documentation

Index

• Constants
• Variables
• func AssignVectorNode(dst, src any, _ inspector.AccumulativeBuffer) (ok bool)
• func Decode(id string, ctx *Ctx) error
• func DecodeRuleset(ruleset Ruleset, ctx *Ctx) (err error)
• func Docgen(format DocgenFormat) ([]byte, error)
• func RegisterDecoder(id string, rules Ruleset)
• func RegisterPool(key string, pool Pool) error
• func ReleaseCtx(ctx *Ctx)
• func WriteDocgen(w io.Writer, format DocgenFormat) error
• type CallbackFn
  • func GetCallbackFn(name string) CallbackFn
• type CallbackFnTuple
  • func RegisterCallbackFn(name, alias string, cb CallbackFn) *CallbackFnTuple
  • func RegisterCallbackFnNS(namespace, name, alias string, cb CallbackFn) *CallbackFnTuple
  • func (t *CallbackFnTuple) WithDescription(desc string) *docgen
  • func (t *CallbackFnTuple) WithExample(example string) *docgen
  • func (t *CallbackFnTuple) WithNote(note string) *docgen
  • func (t *CallbackFnTuple) WithParam(param, desc string) *docgen
  • func (t *CallbackFnTuple) WithType(typ string) *docgen
• type Ctx
  • func AcquireCtx() *Ctx
  • func NewCtx() *Ctx
  • func (ctx *Ctx) AcquireBytes() []byte
  • func (ctx *Ctx) AcquireFrom(pool string) (any, error)
  • func (ctx *Ctx) Bufferize(p []byte) []byte
  • func (ctx *Ctx) BufferizeString(s string) string
  • func (ctx *Ctx) Get(path string) any
  • func (ctx *Ctx) ReleaseBytes(p []byte)
  • func (ctx *Ctx) Reset()
  • func (ctx *Ctx) Set(key string, val any, ins inspector.Inspector)
  • func (ctx *Ctx) SetStatic(key string, val any)
  • func (ctx *Ctx) SetVector(key string, vec vector.Interface)
  • func (ctx *Ctx) SetVectorNode(key string, node *vector.Node) error
• type CtxPool
  • func (p *CtxPool) Get() *Ctx
  • func (p *CtxPool) Put(ctx *Ctx)
• type Decoder
• type DocgenFormat
• type GetterFn
  • func GetGetterFn(name string) GetterFn
• type GetterFnTuple
  • func RegisterGetterFn(name, alias string, cb GetterFn) *GetterFnTuple
  • func RegisterGetterFnNS(namespace, name, alias string, cb GetterFn) *GetterFnTuple
  • func (t *GetterFnTuple) WithDescription(desc string) *docgen
  • func (t *GetterFnTuple) WithExample(example string) *docgen
  • func (t *GetterFnTuple) WithNote(note string) *docgen
  • func (t *GetterFnTuple) WithParam(param, desc string) *docgen
  • func (t *GetterFnTuple) WithType(typ string) *docgen
• type ModFn
  • func GetModFn(name string) ModFn
• type ModFnTuple
  • func RegisterModFn(name, alias string, mod ModFn) *ModFnTuple
  • func RegisterModFnNS(namespace, name, alias string, mod ModFn) *ModFnTuple
  • func (t *ModFnTuple) WithDescription(desc string) *docgen
  • func (t *ModFnTuple) WithExample(example string) *docgen
  • func (t *ModFnTuple) WithNote(note string) *docgen
  • func (t *ModFnTuple) WithParam(param, desc string) *docgen
  • func (t *ModFnTuple) WithType(typ string) *docgen
• type Pool
• type Ruleset
  • func Parse(src []byte) (ruleset Ruleset, err error)
  • func ParseFile(fileName string) (rules Ruleset, err error)
  • func (r *Ruleset) HumanReadable() []byte

Constants

const (
	DocgenFormatMarkdown DocgenFormat = "markdown"
	DocgenFormatHTML                  = "html"
	DocgenFormatJSON                  = "json"
)

Variables

var (
	ErrDecoderNotFound = errors.New("decoder not found")
	ErrEmptyNode       = errors.New("provided node is empty")
	ErrModNoArgs       = errors.New("empty arguments list")
	ErrModPoorArgs     = errors.New("arguments list in modifier is too small")
	ErrCbPoorArgs      = errors.New("arguments list in callback is too small")
	ErrGetterPoorArgs  = errors.New("arguments list in getter callback is too small")

	ErrUnknownPool = errors.New("unknown pool")
)

Functions

func AssignVectorNode

func AssignVectorNode(dst, src any, _ inspector.AccumulativeBuffer) (ok bool)

AssignVectorNode implements assign callback to convert vector.Node to destination with arbitrary type.

func Decode

func Decode(id string, ctx *Ctx) error

Decode applies decoder rules using given id.

ctx should contain all variables mentioned in the decoder's body.

func DecodeRuleset

func DecodeRuleset(ruleset Ruleset, ctx *Ctx) (err error)

DecodeRuleset applies decoder ruleset without using id.

func Docgen

func Docgen(format DocgenFormat) ([]byte, error)

func RegisterDecoder

func RegisterDecoder(id string, rules Ruleset)

RegisterDecoder registers decoder ruleset in the registry.

func RegisterPool

func RegisterPool(key string, pool Pool) error

RegisterPool adds new internal pool to the registry by given key.

func ReleaseCtx

func ReleaseCtx(ctx *Ctx)

ReleaseCtx puts object back to default pool.

func WriteDocgen

func WriteDocgen(w io.Writer, format DocgenFormat) error

Types

type CallbackFn

type CallbackFn func(ctx *Ctx, args []any) error

CallbackFn represents the signature of callback function.

args contains list of all arguments you passed in decoder rule.

func GetCallbackFn

func GetCallbackFn(name string) CallbackFn

GetCallbackFn returns callback function from the registry.

type CallbackFnTuple

type CallbackFnTuple struct {
	// contains filtered or unexported fields
}

func RegisterCallbackFn

func RegisterCallbackFn(name, alias string, cb CallbackFn) *CallbackFnTuple

RegisterCallbackFn registers new callback to the registry.

func RegisterCallbackFnNS

func RegisterCallbackFnNS(namespace, name, alias string, cb CallbackFn) *CallbackFnTuple

RegisterCallbackFnNS registers new callback in given namespace.

func (*CallbackFnTuple) WithDescription

func (t *CallbackFnTuple) WithDescription(desc string) *docgen

func (*CallbackFnTuple) WithExample

func (t *CallbackFnTuple) WithExample(example string) *docgen

func (*CallbackFnTuple) WithNote

func (t *CallbackFnTuple) WithNote(note string) *docgen

func (*CallbackFnTuple) WithParam

func (t *CallbackFnTuple) WithParam(param, desc string) *docgen

func (*CallbackFnTuple) WithType

func (t *CallbackFnTuple) WithType(typ string) *docgen

type Ctx

type Ctx struct {

	// External buffers to use in modifier and condition helpers.
	BufAcc bytebuf.Accumulative
	// todo remove as unused later
	Buf, Buf1, Buf2 bytebuf.Chain

	Err error
	// contains filtered or unexported fields
}

Ctx represents decoder context object.

Contains list of variables that can be used as source or destination.

func AcquireCtx

func AcquireCtx() *Ctx

AcquireCtx returns object from the default context pool.

func NewCtx

func NewCtx() *Ctx

NewCtx makes new context object.

func (*Ctx) AcquireBytes

func (ctx *Ctx) AcquireBytes() []byte

AcquireBytes returns accumulative buffer.

func (*Ctx) AcquireFrom

func (ctx *Ctx) AcquireFrom(pool string) (any, error)

AcquireFrom receives new variable from given pool and register it to return batch after finish template processing.

func (*Ctx) Bufferize

func (ctx *Ctx) Bufferize(p []byte) []byte

func (*Ctx) BufferizeString

func (ctx *Ctx) BufferizeString(s string) string

func (*Ctx) Get

func (ctx *Ctx) Get(path string) any

Get arbitrary value from the context by path.

See Ctx.get(). Path syntax: <ctxVrName>[.<Field>[.<NestedField0>[....<NestedFieldN>]]] Examples: * user.Bio.Birthday * staticVar

func (*Ctx) ReleaseBytes

func (ctx *Ctx) ReleaseBytes(p []byte)

ReleaseBytes updates accumulative buffer with p.

func (*Ctx) Reset

func (ctx *Ctx) Reset()

Reset the context.

Made to use together with pools.

func (*Ctx) Set

func (ctx *Ctx) Set(key string, val any, ins inspector.Inspector)

Set the variable to context. Inspector ins should be corresponded to variable val.

func (*Ctx) SetStatic

func (ctx *Ctx) SetStatic(key string, val any)

SetStatic registers static variable in context.

func (*Ctx) SetVector

func (ctx *Ctx) SetVector(key string, vec vector.Interface)

SetVector directly register vector in context.

func (*Ctx) SetVectorNode

func (ctx *Ctx) SetVectorNode(key string, node *vector.Node) error

SetVectorNode directly registers vector's node in context under given key.

type CtxPool

type CtxPool struct {
	// contains filtered or unexported fields
}

CtxPool represents context pool.

var (
	// CP is a default instance of context pool.
	// You may use it directly as decoder.CP.Get()/Put() or using functions AcquireCtx()/ReleaseCtx().
	CP CtxPool
)

func (*CtxPool) Get

func (p *CtxPool) Get() *Ctx

Get context object from the pool or make new object if pool is empty.

func (*CtxPool) Put

func (p *CtxPool) Put(ctx *Ctx)

Put the object to the pool.

type Decoder

type Decoder struct {
	Id string
	// contains filtered or unexported fields
}

Decoder represents main decoder object. Decoder contains only parsed ruleset. All temporary and intermediate data should be store in context logic to make using of decoders thread-safe.

type DocgenFormat

type DocgenFormat string

type GetterFn

type GetterFn func(ctx *Ctx, buf *any, args []any) error

GetterFn represents signature of getter callback function.

args contains list of all arguments you passed in decoder rule.

func GetGetterFn

func GetGetterFn(name string) GetterFn

GetGetterFn returns getter callback function from the registry.

type GetterFnTuple

type GetterFnTuple struct {
	// contains filtered or unexported fields
}

func RegisterGetterFn

func RegisterGetterFn(name, alias string, cb GetterFn) *GetterFnTuple

RegisterGetterFn registers new getter callback to the registry.

func RegisterGetterFnNS

func RegisterGetterFnNS(namespace, name, alias string, cb GetterFn) *GetterFnTuple

RegisterGetterFnNS registers new getter callback in given namespace.

func (*GetterFnTuple) WithDescription

func (t *GetterFnTuple) WithDescription(desc string) *docgen

func (*GetterFnTuple) WithExample

func (t *GetterFnTuple) WithExample(example string) *docgen

func (*GetterFnTuple) WithNote

func (t *GetterFnTuple) WithNote(note string) *docgen

func (*GetterFnTuple) WithParam

func (t *GetterFnTuple) WithParam(param, desc string) *docgen

func (*GetterFnTuple) WithType

func (t *GetterFnTuple) WithType(typ string) *docgen

type ModFn

type ModFn func(ctx *Ctx, buf *any, val any, args []any) error

ModFn represents signature of the modifier functions.

Arguments description: * ctx provides access to additional variables and various buffers to reduce allocations. * buf is a storage for final result after finishing modifier work. * val is a left side variable that preceded to call of modifier func, example: {%= val|mod(...) %} * args is a list of all arguments listed on modifier call.

func GetModFn

func GetModFn(name string) ModFn

GetModFn returns modifier from the registry.

type ModFnTuple

type ModFnTuple struct {
	// contains filtered or unexported fields
}

func RegisterModFn

func RegisterModFn(name, alias string, mod ModFn) *ModFnTuple

RegisterModFn registers new modifier function.

func RegisterModFnNS

func RegisterModFnNS(namespace, name, alias string, mod ModFn) *ModFnTuple

RegisterModFnNS registers new modifier function in given namespace.

func (*ModFnTuple) WithDescription

func (t *ModFnTuple) WithDescription(desc string) *docgen

func (*ModFnTuple) WithExample

func (t *ModFnTuple) WithExample(example string) *docgen

func (*ModFnTuple) WithNote

func (t *ModFnTuple) WithNote(note string) *docgen

func (*ModFnTuple) WithParam

func (t *ModFnTuple) WithParam(param, desc string) *docgen

func (*ModFnTuple) WithType

func (t *ModFnTuple) WithType(typ string) *docgen

type Pool

type Pool interface {
	Get() any
	Put(any)
	// Reset cleanups data before putting to the pool.
	Reset(any)
}

Pool represents internal pool. In addition to native sync.Pool requires Reset() method.

type Ruleset

type Ruleset []rule

Ruleset represents list of rules.

func Parse

func Parse(src []byte) (ruleset Ruleset, err error)

Parse parses the decoder rules.

func ParseFile

func ParseFile(fileName string) (rules Ruleset, err error)

ParseFile parses the file.

func (*Ruleset) HumanReadable

func (r *Ruleset) HumanReadable() []byte

HumanReadable builds human-readable view of the rules list.