olrgen

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 22, 2023 License: MIT Imports: 25 Imported by: 0

README

olrgen

Go Reference

Operation Log Record processing GENerator framework for Go.

Table of Contents

Installation.

go get github.com/sirkon/olrgen

What it is about.

Imagine we have some KV storage. This means we have a snapshot and an operations log with operations records:

Operation:
    : Create(key string, value []byte)
    | Update(key string, value []byte)
    | Delete(key string)
    ;

Go's first choice to abstract these operation will be:

// LogOperationsRecorder to write operations into an operation log.
type LogOperationsRecorder interface{
    Create(key string, value []byte) error
    Update(key string, value []byte) error
    Delete(key string) error
}

// LogOperationDispatcher to dispatch operations retrieved from an operation log
type LogOperationDispatcher interface{
    Create(key string, value []byte) error
    Update(key string, value []byte) error
    Delete(key string) error
}

And here we have:

  • LogOperationsRecorder encodes parameters of methods calls into a binary form and save encoded data into a "physical" log.
  • LogOperationDispatcher methods are being called by a dispatcher that decodes an operation retrieved from a physical log.

Where:

  • LogOperationsRecorder implementation seems to be easy enough target for a code generation.

  • Likewise, a dispatcher is a reverse for LogOperationsRecorder and is also easy enough for a codegen. It is

    func logRecordDispatch(disp LogOperationDispatcher, rec []byte) error {
    ...
}

  • Although LogOperationDispatcher is an exact match for the LogRecorder as an interface, but it is an actual business logic, which is to be written by a user.

So, this code generator is about rendering a code for a LogOperationRecorder's t7y8uio]; \;'l implementation and a dispatching.

How the final utility is expected to work.

  1. Write an interface A.
  2. Write a type B having two methods a generated code will rely on to encode events:
    • allocateBuffer(n int) []byte method returning an empty slice with capacity ≥ n.
    • May be a writeBuffer(buf []byte) <returnTuple> method to write encoded events back. This method defines returns of encoding methods, they will have the same return tuple as writeBuffer or will be just []byte otherwise.
  3. Write a dispatcher type C what implements A. This type will be used to handle decoded events.
  4. Run utility pointing A, B, may be C (this is optional) and a dispatching function name.
    • Methods to encode events in <c>_generated.go file, here <c>.go is the file where C is defined.
    • Dispatch function <name>(h <B>|<A>, data []byte) error in the same <c>_generated.go file.
    • When C is set the dispatching function will use C directly instead of using generic interface A.

Arguments of the LogRecorder interface are having their own types. Some are supported out of the box, so as types satisfying certain predefined interface. And you can define your own codegen steps for certain types too. This kind of customization is a reason why this thing is a framework rather than a ready to use utility.

List of types subported out of the box.
type
bool
int8
int16
int32
int64
uint8
uint16
uint32
uint64
intypes.VI16
intypes.VI32
intypes.VI64
intypes.VU16
intypes.VU32
intypes.VU64
float32
float64
[N]byte
[]byte
string

Here:

  • intypes.VIX and intypes.VUX are defined in the sirkon/intypes package and their sole purpose is to represent int16..64 and uint16..64 with uleb128 encoding applied rather than a regular little endian encoding. I mean, if your LogRecorder interface will have, say, intypes.VU64 argument type in one of its methods, the argument type will be replaced to uint64 in both recorder and handler implementations.

These types are called builtins.

Auto-supported types.

type Encoder interface{
    Len() int
	// Encode must append to the dst slice and returns
	// the resulted slice.
    Encode([]byte) []byte
}

type Decoder interface{
	// Decode returns the rest of the data after it ends
	// its job.
    Decode([]byte) ([]byte, error)
}

any type that:

  • Satisfies the first interface.
  • The type itself or a pointer of the type satisfies the second interface.

Will be handled automatically. Beware though, values of this type must be usable at their zero state.

Custom types.

You may define custom encoding and decoding for your own types.

You need to implement Handler interface and register a handler factory for them using either CustomHandler option (HandleByName or NewHandler).

LogRecorder code generation details.

The general scheme of data encoding is:

graph TD
    size[Compute an output size of a data]
    allc[Allocate a buffer to keep the encoding data]
    encd[Encode data into the buffer]
    stor[Save buffer]
    
    size --> allc %% Relies on user defined method.
    allc --> encd %% Relies on generated code. 
    encd --> stor %% Relies on user defined method.

Usage example.

There's an example of the framework usage in the example folder.

IMPORTANT: Binary compatibility details.

We start from a recorder interface which has a set of (public) methods M1, M2,…, Mn, each having its own set of arguments:

type XXXRecorder interface{
	M1(…)
	…
	Mn(…)
}

Each one is getting an encoder. And we have a dispatching procedure, which gets encoded data, decode it and decides what method to call then. It requires some kind of method reference to be a part of the encoding. The generator does this that way:

Method number (uint32 kind) Arguments encoding

Where method number is its index in the list of methods.

What does it mean? It means we MUST NOT change a method order in any way: no reordering, no insertion, only appends are allowed.

Arguments are encoded in the order they come in their methods. This means everything what have been said for methods is applied to them. A little notice on appended arguments though: their encoders should tolerate empty buffer – this means you can't add arguments with builtin types, they are not like this.

Conclusion:

  • You cannot reorder anything generally.
  • You can only append methods to the end and append arguments with custom handler which tolerates empty buffer on decoding.
  • You can rename arguments and methods whatever you like, because only positional information matters for both encoding and decoding.

Imagine we have Op method in our recorder and want to replace it with updated version. The best approach will be to rename Op -> DeprecatedOp and append a new Op method. This will do the trick.

You can even remove DeprecatedOp arguments altogether at some point, once you are sure there are no records of the deprecated Op in your logs anymore. But don't remove the method or make it private nevertheless, cause the order.

TODO

  • Support types definitions where an underlying type is one of the builtins.
  • Provide support for pointers over numeric, boolean and string types + nil []byte values.
  • Provide auto-support for struct types where all fields are supported.

Documentation

Overview

Package olrgen is a code generation framework for operations log encoding.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ReturnError added in v0.1.0 

func ReturnError() *er.RType

ReturnError returns a handler to simplify building error messages with structured context in a way that closely replicates the API of github.com/sirkon/errors error processing package.

Example usage: 

olrgen.ReturnError().Wrap("err", "$decode").Int("count-$0", 14).Rend(r, countIndex)

The code rendered will look like – countIndex, say, is equal to 25: 

return errors.Wrap(err, "decode BranchName.argName(argType)").Int("count-25", 14)

There is a set of predefined values in the code rendering context, besides $decode. Here is the full list (2023-05-15):

  • $decode -> "decode $branch.$dst($dstType)"
  • $recordTooSmall -> "record buffer is too small"
  • $malformedUvarint -> "malformed uvarint sequence"
  • $malformedVarint -> "malformed varint sequence"

func Run 

func Run(
	appName string,
	hnlrs *TypesHandlers,
) error

Run is the entry point of a final utility.

func SetStructuredErrorsPkgPath added in v0.3.0 

func SetStructuredErrorsPkgPath(p string)

SetStructuredErrorsPkgPath sets custom errors package.

Types

type CustomTypeHandler 

type CustomTypeHandler func(handlerPlaceholder, *TypesHandlers) error

CustomTypeHandler custom option type. Cannot be defined outside of this package and only implemented by TypeHandlerByName and NewTypeHandler functions.

func NewTypeHandler 

func NewTypeHandler(handler func(p types.Type) TypeHandler) CustomTypeHandler

NewTypeHandler adds custom handler.

func TypeHandlerByName 

func TypeHandlerByName(name string, handler func() TypeHandler) CustomTypeHandler

TypeHandlerByName adds this custom handler for the given type.

type Go 

type Go = gogh.GoRenderer[*renderer.Imports]

Go makes Go renderer public.

type L added in v0.1.0 

type L = er.L

L does the opposite job to Q for structured values: When you want the builder key to be shown as a variable you use olrgen.L: 

olrgen.ReturnError()...Str(olrgen.L("key"), olrgen.Q("value"))...

This will be rendered as 

return errors...Str(key, "value")

type LoggerType 

type LoggerType interface {
	Pos(pos token.Pos, err error)
	Error(err error)
}

LoggerType error logging abstraction to log with text position.

func Logger 

func Logger(fset *token.FileSet) LoggerType

Logger returns a standard LoggerType implementation that seems to be sufficient for most needs.

type Package 

type Package = gogh.Package[*renderer.Imports]

Package makes Package public.

type Project 

type Project = gogh.Module[*renderer.Imports]

Project makes Project public.

type Q added in v0.1.0 

type Q = er.Q

Q is to be used for structured values to be string. Like 

orlgen.ReturnError()...Str("key", olrgen.Q("value"))...

To be rendered like 

return errors...Str("key", "value")

It will be 

return errors...Str("key", value)

Without olrgen.Q

type TypeHandler 

type TypeHandler interface {
	// Name returns a Go type name this handler is for. Meaning
	// it will be int8 for int8 handler, etc.
	Name(r *Go) string

	// Pre renders can be used to render additional stuff at the top
	// of the encoding function.
	// Take a look at the example:
	//
	// Let we have a recorder interface
	//
	//    type LogRecorder interface{
	//        …
	//        Append(sid types.Index, data []byte) error
	//        …
	//    }
	//
	// This means Append method for an implementation of LogRecorder
	// must be generated. And it looks like
	//
	//     func (x *LogRecorderImpl) Append(sid types.Index, data []byte) error {
	//         {% code generated by Pre calls by handlers of each argument's type %}
	//         …
	//     }
	//
	// A handler can put everything it wants in that area, including
	// local variables, additional checks, whatever.
	//
	// It can be used to store computed lengths for []byte and string
	// types handlers for instance. Kinda meaningless for them as
	// it len(x) is fast to compute, but some types can be tougher to
	// deal with, so this can be pretty useful for them.
	Pre(r *Go, src string)

	// Len returns a length of buffer required to keep an encoded value.
	// It is not possible to have a meaningful value for every type.
	// In particular, []byte and string required buffer lentghs are
	// only known at the runtime. Implementation MUST return a negative
	// value in this case.
	Len() int

	// LenExpr gives an expression to compute required buffer length
	// to encode a type being handled. Must be be equal to
	//     strconv.Itoa(x.Len())
	// In case of fixed length, and it is checked.
	LenExpr(r *Go, src string) string

	// Encoding generates a code that encodes a value of $src into $dst.
	// $src and $dst values are also available in the scope of r under
	// these exact names, i.e. you can reference them like this:
	//
	//     $dst = binary.LittleEndian.AppendUint32($dst, $src)
	//
	// And there's $dstType scope variable too containing $dst type.
	// The $dst slice is guaranteed to have enough capacity to store
	// all encoded data, and it always comes empty, so every encoding
	// must follow Append "protocol".
	//
	// A code is rendered within a context where it (a code generated)
	// may return an error. An example of how the end result will look
	// like:
	//
	//     func (x *LogRecorderImpl) Append(sid types.Index, data []byte) error {
	//         dataLen := varsize.Len(data) + len(data)
	//
	//         dst := x.allocateBuffer(4 + 16 + dataLen) // returns empty slice with enough capacity
	//
	//         // Autogenerated data to put Append method code.
	//         dst = binary.LittleEndian.AppendUint32(dst, uint32(logRecorderCodeAppend))
	//
	//         // Autogenerated code to encode Index type of the sid parameter.
	//         dst = dst[:len(dst)+16]
	//         types.IndexEncode(dst, sid)
	//
	//         // Autogenerated code to encode []byte type of the data parameter.
	//         dst = binary.AppendUvarint(dst, uint64(len(data)))
	//         dst = append(dst, data...)
	//
	//         if _, err := x.writeBuffer(dst); err != nil {
	//             return errors.Wrap(err, "write encoded data") // Forcing my errors pkg LMAO.
	//         }
	//
	//         return nil
	//     }
	//
	// It is guaranteed err name is not taken by anything generated - err
	// parameter name is prohibited and reserved for its common usage.
	// It is not guaranteed 100% though because a user can slap
	// something like
	//     err := "Hello World!"
	// in his custom encoding code.
	// Take a look at [Auto] handler to see how to deal with err
	// within an encoding function scope.
	//
	// [Auto]: ./internal/handlers/handler_auto.go
	Encoding(r *Go, dst, src string)

	// Decoding generates a code that decodes a data from $src and stores
	// it into $dst.
	//  - The semantics is exactly the same as for the Encoding, except
	//    it is $src that is []byte now and $dst is a variable of a
	//    respective type.
	//  - It is $srcType instead of the $dstType this time for obvious
	//    reasons.
	//  - Same single return value – an error – context for a generated
	//    code.
	//
	// Another sort of compliance is needed in this case though:
	//  - We encode reading data at the head of the $src.
	//  - When encoding ends we cut amount of bytes we read from the $dst
	//    to encode. We will cut 4 bytes for encoded int32 value for example.
	//  - Ulike the Encoding() we may return true or false, in case if we cut
	//    the head ourselves or want the autogen to make this for us.
	//    The general advice is to cut manually for a variable length encoding
	//    and let the autogen do this for the fixed sized ones.
	//
	// And here is the context similar to one a code generated will work within:
	//
	//    type LogRecorder interface{
	//        Append(sid Index, data []byte) error
	//        Delete(sid Index) error
	//    }
	//    …
	//
	//    func DispatchLog(h LogHandler, rec []byte) error {
	//        if len(data) < 4 {
	//            return errors.New("extract branch code: record buffer is too small")
	//        }
	//
	//        switch v := logRecorderCode(binary.LittleEndian.Uint32(rec)); v {
	//        case logRecorderCodeAppend:
	//            if err := dispatchLogDecodeAppend(h, rec[4:]); err != nil {
	//                return errors.Wrap(err, "dispatch Append branch")
	//            }
	//        case logRecorderCodeDelete:
	//            if err := dispatchLogDecodeDelete(h, rec[4:]); err != nil {
	//                return errors.Wrap(err, "dispatch Delete branch")
	//            }
	//        default:
	//            return errors.Newf("unknown branch code %d", v)
	//        }
	//    }
	//
	//    func dispatchLogDecodeAppend(h LogHandler, rec []byte) error {
	//        // decoding code here spawning sid and data variables
	//
	//        var sid Index
	//        // sid decoding in here
	//        ...
	//        if err := h.Append(sid, data); err != nil {
	//            return errors.New("call Append operation handler").Stg("sid", sid).Int("data-len", len(data))
	//        }
	//
	//        return nil
	//    }
	//
	// Decoding method body has the same err guarantees as for the Encoding,
	// that is err is not taken by anything generated by this library handlers.
	// Remember, $dst is always declared as
	//     var $dst $dstType
	// Before the code produced by a Decoding call, you just use this variable.
	Decoding(r *Go, dst, src string) bool
}

TypeHandler an abstraction for a codegen support of a type.

type TypesHandlers 

type TypesHandlers struct {
	// contains filtered or unexported fields
}

TypesHandlers this returns type handlers.

func NewTypesHandlers 

func NewTypesHandlers(handlers ...CustomTypeHandler) (*TypesHandlers, error)

NewTypesHandlers constructor. User can additional handlers via TypeHandlerByName and NewTypeHandler.

func (*TypesHandlers) Handler 

func (h *TypesHandlers) Handler(arg *types.Var) TypeHandler

Handler returns a handler for the given type by its name.

Source Files

View all Source files

Directories

Path Synopsis
internal/example
Package example.
 Package example.
internal
er
Package er provides helpers to simplify building errors returning.
 Package er provides helpers to simplify building errors returning.
generator
Package generator makes code generation.
 Package generator makes code generation.
handlers
Package handlers for builtin types.
 Package handlers for builtin types.
logger
Package logger provides loggers for testing and for the actual job.
 Package logger provides loggers for testing and for the actual job.
renderer
Package renderer provides a renderer with a custom reporter.
 Package renderer provides a renderer with a custom reporter.
tdetect
Package tdetect provides types detections means.
 Package tdetect provides types detections means.
tnmatchers
Package tnmatchers means Type Name MATCHERS.
 Package tnmatchers means Type Name MATCHERS.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.