ranger

package
v0.0.0-...-9742f5a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2020 License: Apache-2.0 Imports: 10 Imported by: 0

README

Stable byte serialisation a-la protobuf

Code Design

Note: this is aspirational.

The config file describes the desired native language types and their serialised behaviours.

This is parsed into a data structure that provides an abstraction that can render a native language type and the required public (de)serialisation code for it.

For golang this is comprised of:

  • The struct definition
  • The Marshal method
  • The MarshalTo method
  • The Unmarshal Method
  • The UnMarshalFrom method
  • The Size method

Each type is expected to self delimit - no exterior containers are supplied. If a given type isn't of well defined size, then the serialisation code for that type should include its own length prefix tag.

Additionally though, to allow for clean and safe deserialisation, each type also creates additional helper methods:

  • MinimumSize - the smallest size that this type is permitted to be deserialised into. This is useful as it allows rolling up the requirements for structs and the like.

The abstract data structure has methods on it to emit the above methods, but also, to help write out those methods, it can be asked to write out valid default value - this is distinct from a zero value, because pointer types will be given an instance of the type they point at.

  • DefaultValueFor

On tests

Tests generate and then run generated tests on the generated code. This code is generated in ./ranger/testdata/pkg. If you wish to re-run the tests without a generation step (to test code you hacked up), run:

$ go test -race -v ./ranger/testdata/pkg -count 1

To fuzz:

$ make fuzz FUZZ=ranger/testdata/pkg

After running the first generation (AKA, run this first):

$ go test -race -v ./ranger

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrBadInterface = errors.New("bad interface type")

ErrBadInterface is for when interface types (enums) are unaccounted for.

View Source 
var ErrLeftOverBytes = errors.New("leftover bytes during unmarshal")

ErrLeftOverBytes is from when unmarhsals leave dangling content

View Source 
var ErrLengthMismatch = errors.New("lengths do not match")

ErrLengthMismatch is when two lengths should match but do not.

View Source 
var ErrMarshalLength = errors.New("marshal length does not match generated content")

ErrMarshalLength is returned when the marshal length does not match the generated content.

View Source 
var ErrShortRead = errors.New("short read during unmarshal")

ErrShortRead indicates to users when an unmarshaling error has occurred because we ran out of bytes.

View Source 
var ErrShortWrite = errors.New("short write during marshal")

ErrShortWrite indicates to users when a marshal error has occurred because we ran out of bytes.

View Source 
var ErrTooLarge = errors.New("value is too large for type")

ErrTooLarge is for when values exceeed their input boundaries

View Source 
var ErrTooMany = errors.New("too many items during (un)marshal")

ErrTooMany is for when a length constraint was exceeded.

Functions

func UvarintSize 

func UvarintSize(i uint64) int

UvarintSize calculates the size for a binary.Uvarint

Types

type ConfigFormat 

type ConfigFormat struct {
	// Package is the name of the package we're generating into.
	Package string `yaml:"package"`
	// Types is the listing of types. Each definition consists of a Type name as
	// map key and properties as values.
	Types map[string]*ConfigType `yaml:"types"`

	// Comment adds a comment to the package declaration.
	Comment string `yaml:"comment"`
	// contains filtered or unexported fields
}

ConfigFormat defines the format of the configuration file used to generate code -- this is the top level.

func Parse 

func Parse(content []byte) (*ConfigFormat, error)

Parse parses the content and returns a ConfigFormat; if an error is returned ConfigFormat is invalid.

func ParseFile 

func ParseFile(filename string) (*ConfigFormat, error)

ParseFile parses the content in a file specified by filename and returns a ConfigFormat; if an error is returned ConfigFormat is invalid.

func (*ConfigFormat) DefaultValueFor 

func (cf *ConfigFormat) DefaultValueFor(value *ConfigTypeDefinition) string

DefaultValueFor creates a default of a type. This differs very slightly from a zero value in that a pointer type is initialised to a default value of that type rather than to nil.

func (*ConfigFormat) Execute 

func (cf *ConfigFormat) Execute(name string, data interface{}) ([]byte, error)

Execute a named template with some data and return the string it renders.

func (*ConfigFormat) ExecuteString 

func (cf *ConfigFormat) ExecuteString(name string, data interface{}) (string, error)

func (*ConfigFormat) GenerateCode 

func (cf *ConfigFormat) GenerateCode() ([]byte, error)

GenerateCode generates the source code.

func (*ConfigFormat) GenerateFuzz 

func (cf *ConfigFormat) GenerateFuzz() ([]byte, error)

GenerateFuzz generates the fuzzer code.

func (*ConfigFormat) GenerateTest 

func (cf *ConfigFormat) GenerateTest() ([]byte, error)

GenerateTest generates the test code.

func (*ConfigFormat) GetType 

func (cf *ConfigFormat) GetType(name string) (Type, error)

GetType looks up a specific type in both the user defined types and the built in native type definitions

type ConfigInterface 

type ConfigInterface struct {
	// Output method. This is a method which is called on the member to determine
	// the type information for embedding into the marshal.
	Output string `yaml:"output"`
	// Input type. This is a type we can unmarshal out of a byte stream to
	// determine what type in cases to use.
	Input string `yaml:"input"`
	// Cases is a list of cases; this is a list of one-key maps that correspond
	// comparisons in case statement, and result in a type marshal of the value.
	// Types specified in this manner must still conform to ranger.Marshaler.
	Cases []map[string]string `yaml:"cases"`
}

ConfigInterface defines a polymorphic type to marshal. It requires an input type, an output method for returning the type, and a series of cases for what member corresponds to what type.

type ConfigMatch 

type ConfigMatch struct {
	// LengthOfField indicates the field must match the Member field specified in the value.
	LengthOfField string `yaml:"length_of_field"`
}

ConfigMatch specifies matching rules for validations. Matching means in this context, that two fields must match each other in some way.

type ConfigRequire 

type ConfigRequire struct {
	// MaxLength means the field must have a length no longer than this.
	MaxLength uint64 `yaml:"max_length"`
	// Length means the field must be exactly this length.
	Length uint64 `yaml:"length"`
}

ConfigRequire is hard requirements for validations.

type ConfigType 

type ConfigType struct {
	// Fields is a list of fields in serialisation order. This is ordered because byte stability in the
	// serialisation format is a requirement.
	Fields []*ConfigTypeDefinition `yaml:"fields"`
	// Interface is a polymorphic type definition; see ConfigInterface for more. A type defined with an interface
	// configuration must have no fields defined.
	Interface *ConfigInterface `yaml:"interface,omitempty"`

	// Comment is a field to add a comment to the type's declaration.
	Comment string `yaml:"comment"`

	TypeName string `yaml:"-"` // populated by parse
	// contains filtered or unexported fields
}

ConfigType is the type definition wrapper; used for specifying member fields as well as any struct-level operations to apply such as validations or filtering.

func (*ConfigType) ConfigFormat 

func (typ *ConfigType) ConfigFormat() *ConfigFormat

func (*ConfigType) ConstantSize 

func (typ *ConfigType) ConstantSize(instance TypeInstance) (bool, error)

func (*ConfigType) HasLen 

func (ct *ConfigType) HasLen(instance TypeInstance) (bool, error)

Does the type support len()

func (*ConfigType) InterfaceAdapter 

func (typ *ConfigType) InterfaceAdapter(instance TypeInstance) TypeInstance

InterfaceAdapter creates an interface adapter for TypeInstance

func (*ConfigType) IsInterface 

func (ct *ConfigType) IsInterface() bool

func (*ConfigType) MinimumSize 

func (typ *ConfigType) MinimumSize(instance TypeInstance) (uint64, error)

MinimumSize returns the minimum serialized size of the type. This is the sum of the minimum size of the serialized fields of the type.

func (*ConfigType) Name 

func (ct *ConfigType) Name() string

The name of the type

func (*ConfigType) PointerType 

func (ct *ConfigType) PointerType(instance TypeInstance) bool

PointerType returns whether the type instance is a value or a pointer to value

func (*ConfigType) Read 

func (ct *ConfigType) Read(instance TypeInstance) (string, error)

Read returns code to deserialise an instance of the type

func (*ConfigType) SetConfigFormat 

func (typ *ConfigType) SetConfigFormat(cf *ConfigFormat)

SetConfigFormat provides the type with a reference to the root of the structure for rendering templates etc

func (*ConfigType) Type 

func (ct *ConfigType) Type(instance TypeInstance) (string, error)

func (*ConfigType) Write 

func (ct *ConfigType) Write(instance TypeInstance) (string, error)

Write returns code to serialise an instance of the

func (*ConfigType) WriteSize 

func (ct *ConfigType) WriteSize(instance TypeInstance) (string, error)

WriteSize returns code to caculate the size of an instance of the type when serialized

type ConfigTypeDefinition 

type ConfigTypeDefinition struct {
	// FieldName is the name of the field in the containing type. While the yaml file can represent multiple fields with
	// the same name this is a semantic error and will produce bad code that won't compile.
	FieldName string `yaml:"name"`
	// StructureType is the type of data structure; this may be "struct" or
	// "array". This determines how the field is marshaled by wrapping length
	// headers over array sections for ease of reading.
	StructureType string `yaml:"structure_type"`
	// ValueType is the type that we'll use for the actual marshaling. The
	// ValueType must conform to ranger.Marshaler or be a built in type (uint,
	// string, etc) that we support marshaling to/from natively.
	ValueType string `yaml:"value_type"`
	// Static ensures that the value is of a fixed size for integer types.
	Static bool `yaml:"static"`
	// Match is a list of matching rules for validations.
	Match ConfigMatch `yaml:"match,omitempty"`
	// Require is a list of requirements for validations.
	Require ConfigRequire `yaml:"require,omitempty"`
	// ItemRequire defines length/maxlength etc for the elements of an array.
	// This is a compromise vs a full type system overhaul that will probably be
	// Ranger v2.
	ItemRequire *ConfigRequire `yaml:"item_require,omitempty"`
	// Marshal if false, will not marshal in or out.
	Marshal *bool `yaml:"marshal,omitempty"`
	// Embedded defines this field as embedded in the struct.
	Embedded bool `yaml:"embedded"`

	// Comment is a field to add a comment to the field's declaration.
	Comment string `yaml:"comment"`

	TypeName string `yaml:"-"` // populated by parse
	// contains filtered or unexported fields
}

ConfigTypeDefinition is the definition of an individual type member.

func (*ConfigTypeDefinition) ConfigFormat 

func (field *ConfigTypeDefinition) ConfigFormat() *ConfigFormat

ConfigFormat returns the config format for the occasional cases where we need global context

func (*ConfigTypeDefinition) FieldInstance 

func (field *ConfigTypeDefinition) FieldInstance() TypeInstance

FieldInstance returns a TypeInstance implementation adapted to the field in structure form.

func (*ConfigTypeDefinition) GetInterface 

func (field *ConfigTypeDefinition) GetInterface() *ConfigInterface

func (*ConfigTypeDefinition) GetType 

func (field *ConfigTypeDefinition) GetType() (Type, error)

GetType returns the type of this field. For array fields this is the type of the items of the array.

func (*ConfigTypeDefinition) Initializer 

func (field *ConfigTypeDefinition) Initializer() (string, error)

Initializer returns the value to assign for unmarshaling into

func (*ConfigTypeDefinition) IsBytesType 

func (value *ConfigTypeDefinition) IsBytesType() bool

func (*ConfigTypeDefinition) IsInterface 

func (field *ConfigTypeDefinition) IsInterface() bool

func (*ConfigTypeDefinition) IsNativeType 

func (value *ConfigTypeDefinition) IsNativeType() bool

func (*ConfigTypeDefinition) ItemInstance 

func (field *ConfigTypeDefinition) ItemInstance() TypeInstance

ItemInstance returns a TypeInstance implementation adapted to the field in array item form.

func (*ConfigTypeDefinition) MaybeItemInstance 

func (field *ConfigTypeDefinition) MaybeItemInstance() TypeInstance

MaybeItemInstance returns ItemInstance if the field is an array, FieldInstance otherwise.

func (*ConfigTypeDefinition) NeedsInitializer 

func (field *ConfigTypeDefinition) NeedsInitializer() (bool, error)

NeedsInitializer returns true if the zero value isn't usable for unmarshaling into.

func (*ConfigTypeDefinition) QualName 

func (field *ConfigTypeDefinition) QualName() string

QualName returns the qualified name of the field. For instance Transaction.Version

func (*ConfigTypeDefinition) SetConfigFormat 

func (field *ConfigTypeDefinition) SetConfigFormat(cf *ConfigFormat)

SetConfigFormat provides a reference to the ConfigFormat instance for lookups of referenced type definitions

func (*ConfigTypeDefinition) SymbolName 

func (field *ConfigTypeDefinition) SymbolName() string

type FieldInstance 

type FieldInstance struct {
	// contains filtered or unexported fields
}

func (*FieldInstance) ConfigFormat 

func (instance *FieldInstance) ConfigFormat() *ConfigFormat

func (*FieldInstance) GetLength 

func (instance *FieldInstance) GetLength() (uint64, error)

func (*FieldInstance) GetMaxLength 

func (instance *FieldInstance) GetMaxLength() (uint64, error)

func (*FieldInstance) HasLen 

func (instance *FieldInstance) HasLen() (bool, error)

func (*FieldInstance) IsPointer 

func (instance *FieldInstance) IsPointer() bool

func (*FieldInstance) QualName 

func (instance *FieldInstance) QualName() string

func (*FieldInstance) ReadSymbolName 

func (instance *FieldInstance) ReadSymbolName() string

func (*FieldInstance) Static 

func (instance *FieldInstance) Static() bool

func (*FieldInstance) WriteSymbolName 

func (instance *FieldInstance) WriteSymbolName() string

type InputInstanceAdapter 

type InputInstanceAdapter struct {
	// contains filtered or unexported fields
}

func (*InputInstanceAdapter) ConfigFormat 

func (instance *InputInstanceAdapter) ConfigFormat() *ConfigFormat

func (*InputInstanceAdapter) GetLength 

func (instance *InputInstanceAdapter) GetLength() (uint64, error)

func (*InputInstanceAdapter) GetMaxLength 

func (instance *InputInstanceAdapter) GetMaxLength() (uint64, error)

func (*InputInstanceAdapter) HasLen 

func (instance *InputInstanceAdapter) HasLen() (bool, error)

func (*InputInstanceAdapter) IsPointer 

func (instance *InputInstanceAdapter) IsPointer() bool

func (*InputInstanceAdapter) QualName 

func (instance *InputInstanceAdapter) QualName() string

func (*InputInstanceAdapter) ReadSymbolName 

func (instance *InputInstanceAdapter) ReadSymbolName() string

func (*InputInstanceAdapter) Static 

func (instance *InputInstanceAdapter) Static() bool

func (*InputInstanceAdapter) WriteSymbolName 

func (instance *InputInstanceAdapter) WriteSymbolName() string

type Integral 

type Integral struct {
	// contains filtered or unexported fields
}

Integral represents integers that can be static or variable length.

func (*Integral) ConstantSize 

func (typ *Integral) ConstantSize(instance TypeInstance) (bool, error)

func (*Integral) HasLen 

func (typ *Integral) HasLen(instance TypeInstance) (bool, error)

func (*Integral) MinimumSize 

func (typ *Integral) MinimumSize(instance TypeInstance) (uint64, error)

func (*Integral) Name 

func (typ *Integral) Name() string

func (*Integral) PointerType 

func (typ *Integral) PointerType(instance TypeInstance) bool

func (*Integral) Read 

func (typ *Integral) Read(instance TypeInstance) (string, error)

func (*Integral) Type 

func (typ *Integral) Type(instance TypeInstance) (string, error)

func (*Integral) Write 

func (typ *Integral) Write(instance TypeInstance) (string, error)

func (*Integral) WriteSize 

func (typ *Integral) WriteSize(instance TypeInstance) (string, error)

type ItemInstance 

type ItemInstance struct {
	// contains filtered or unexported fields
}

ItemInstance adapts a field for use in arrays

func (*ItemInstance) ConfigFormat 

func (instance *ItemInstance) ConfigFormat() *ConfigFormat

func (*ItemInstance) GetLength 

func (instance *ItemInstance) GetLength() (uint64, error)

func (*ItemInstance) GetMaxLength 

func (instance *ItemInstance) GetMaxLength() (uint64, error)

func (*ItemInstance) HasLen 

func (instance *ItemInstance) HasLen() (bool, error)

The schema cannot specify arrays of arrays, so this is always false unless the contained type supports len.

func (*ItemInstance) IsPointer 

func (instance *ItemInstance) IsPointer() bool

We only support pointers to structs in arrays

func (*ItemInstance) QualName 

func (instance *ItemInstance) QualName() string

func (*ItemInstance) ReadSymbolName 

func (instance *ItemInstance) ReadSymbolName() string

func (*ItemInstance) Static 

func (instance *ItemInstance) Static() bool

func (*ItemInstance) WriteSymbolName 

func (instance *ItemInstance) WriteSymbolName() string

type Marshaler 

type Marshaler interface {
	// Allocate storage
	Marshal() ([]byte, error)
	// Marshal into preallocated storage and returned consumed bytes.
	MarshalTo(data []byte) (int, error)
	// Unmarshal the first document in data
	Unmarshal(data []byte) error
	// Unmarshal the first document in data returning consumed bytes
	UnmarshalFrom(data []byte) (int, error)
	// Return the size that the struct will need to Marshal successfully. If the
	// struct cannot be marshalled successfully for some reason, will return a
	// sufficient number of bytes (perhaps even 0) to allow marshalTo to reason
	// the point where that marshalling error will occur, and will itself not
	// panic. MarshalTo will then proceed to attempt to marshal into this space
	// and the marshaling will fail. The failure should still be due to the
	// underlying reason, not due to a shortness of space returned from Size.
	Size() int
}

Marshaler is an interface to define how marshaling happens in Ranger. All specified types must be either a supported, native type such as uint or string, or implement this interface by either doing it by hand or through rangergen.

This is deliberately compatible with the way that GRPC/gogoproto generated code wants to call things, so that generated types can be embedded as foreign types in protobufs.

type Strings 

type Strings struct {
	// contains filtered or unexported fields
}

func (*Strings) ConstantSize 

func (typ *Strings) ConstantSize(instance TypeInstance) (bool, error)

func (*Strings) HasLen 

func (typ *Strings) HasLen(instance TypeInstance) (bool, error)

func (*Strings) MinimumSize 

func (typ *Strings) MinimumSize(instance TypeInstance) (uint64, error)

func (*Strings) Name 

func (typ *Strings) Name() string

func (*Strings) PointerType 

func (typ *Strings) PointerType(instance TypeInstance) bool

func (*Strings) Read 

func (typ *Strings) Read(instance TypeInstance) (string, error)

func (*Strings) Type 

func (typ *Strings) Type(instance TypeInstance) (string, error)

func (*Strings) Write 

func (typ *Strings) Write(instance TypeInstance) (string, error)

func (*Strings) WriteSize 

func (typ *Strings) WriteSize(instance TypeInstance) (string, error)

type Type 

type Type interface {
	// ConstantSize returns true if the type is always the same size
	ConstantSize(TypeInstance) (bool, error)
	// Does the type support len()
	HasLen(TypeInstance) (bool, error)
	// MinimumSize returns the minimum serialized size of the type.
	MinimumSize(TypeInstance) (uint64, error)
	// The name of the type
	Name() string
	// PointerType returns whether the type instance is a value or a pointer to
	// value
	PointerType(TypeInstance) bool
	// Read returns code to deserialise an instance of the type
	Read(TypeInstance) (string, error)
	// Type returns the go code to describe the type - e.g. [32]byte.
	Type(TypeInstance) (string, error)
	// WriteSize returns code to caculate the size of an instance of the type when serialized
	WriteSize(TypeInstance) (string, error)
	// Write returns code to serialise an instance of the
	Write(TypeInstance) (string, error)
}

A Type is a type for which we can emit (de)serialisation code. Built in types cover the basic native types. Users can compose these into custom types.

type TypeInstance 

type TypeInstance interface {
	// Get at the ConfigFormat for e.g. running templates
	ConfigFormat() *ConfigFormat
	// What if any Length is configured for 'instance'.
	// NB: this should move to being part of the type.
	GetLength() (uint64, error)
	// What max length is configured for 'instance'.
	GetMaxLength() (uint64, error)
	// Does the type support len()
	HasLen() (bool, error)
	// Is this a Reference
	IsPointer() bool
	// Whats the fully qualified name (for human errors)
	QualName() string
	// How is this instance address in code when reading in
	ReadSymbolName() string
	// How is this instance addressed in code when writing out
	WriteSymbolName() string
	// Is this instance statically sized? (Separate to static array lengths at
	// least for now)
	Static() bool
}

TypeInstance is an instance of a type e.g. Type == struct definition. 

TypeInstance == usage - plain, in an array, or as a pointer.

type UInt8 

type UInt8 struct{}

A static sized UInt8

func (*UInt8) ConstantSize 

func (typ *UInt8) ConstantSize(instance TypeInstance) (bool, error)

func (*UInt8) HasLen 

func (typ *UInt8) HasLen(instance TypeInstance) (bool, error)

func (*UInt8) MinimumSize 

func (typ *UInt8) MinimumSize(instance TypeInstance) (uint64, error)

func (*UInt8) Name 

func (typ *UInt8) Name() string

func (*UInt8) PointerType 

func (typ *UInt8) PointerType(instance TypeInstance) bool

func (*UInt8) Read 

func (typ *UInt8) Read(instance TypeInstance) (string, error)

func (*UInt8) Type 

func (typ *UInt8) Type(instance TypeInstance) (string, error)

func (*UInt8) Write 

func (typ *UInt8) Write(instance TypeInstance) (string, error)

func (*UInt8) WriteSize 

func (typ *UInt8) WriteSize(instance TypeInstance) (string, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.