colfer

README

Colfer

install

go install github.com/frbrno/colfer/cmd/colf

Colfer is a binary serialization format optimized for speed and size.

The project's compiler colf(1) generates source code from schema definitions to marshal and unmarshall data structures.

This is free and unencumbered software released into the public domain. The format is inspired by Protocol Buffers.

Language Support

• C, ISO/IEC 9899:2011 compliant a.k.a. C11, C++ compatible
• Go, a.k.a. golang
• Java, Android compatible
• JavaScript, a.k.a. ECMAScript, NodeJS compatible
• 🚧 Gergely Bódi realised a functional Dart port.
• 🚧 Karthik Kumar Viswanathan has a Python alternative under construction.

Features

• Simple and straightforward in use
• No dependencies other than the core library
• Both faster and smaller than the competition
• Robust against malicious input
• Maximum of 127 fields per data structure
• No support for enumerations
• Framed; suitable for concatenation/streaming

TODO's

• Rust and Python support
• Protocol revision

Use

Download a prebuilt compiler or run go get -u github.com/frbrno/colfer/cmd/colf to make one yourself. Homebrew users can also brew install colfer.

The command prints its own manual when invoked without arguments.

NAME
	colf — compile Colfer schemas

SYNOPSIS
	colf [-h]
	colf [-vf] [-b directory] [-p package] \
		[-s expression] [-l expression] C [file ...]
	colf [-vf] [-b directory] [-p package] [-t files] \
		[-s expression] [-l expression] Go [file ...]
	colf [-vf] [-b directory] [-p package] [-t files] \
		[-x class] [-i interfaces] [-c file] \
		[-s expression] [-l expression] Java [file ...]
	colf [-vf] [-b directory] [-p package] \
		[-s expression] [-l expression] JavaScript [file ...]

DESCRIPTION
	The output is source code for either C, Go, Java or JavaScript.

	For each operand that names a file of a type other than
	directory, colf reads the content as schema input. For each
	named directory, colf reads all files with a .colf extension
	within that directory. If no operands are given, the contents of
	the current directory are used.

	A package definition may be spread over several schema files.
	The directory hierarchy of the input is not relevant to the
	generated code.

OPTIONS
  -b directory
    	Use a base directory for the generated code. (default ".")
  -c file
    	Insert a code snippet from a file.
  -f	Normalize the format of all schema input on the fly.
  -h	Prints the manual to standard error.
  -i interfaces
    	Make all generated classes implement one or more interfaces.
    	Use commas as a list separator.
  -l expression
    	Set the default upper limit for the number of elements in a
    	list. The expression is applied to the target language under
    	the name ColferListMax. (default "64 * 1024")
  -p package
    	Compile to a package prefix.
  -s expression
    	Set the default upper limit for serial byte sizes. The
    	expression is applied to the target language under the name
    	ColferSizeMax. (default "16 * 1024 * 1024")
  -t files
    	Supply custom tags with one or more files. Use commas as a list
    	separator. See the TAGS section for details.
  -v	Enable verbose reporting to standard error.
  -x class
    	Make all generated classes extend a super class.

TAGS
	Tags, a.k.a. annotations, are source code additions for structs
	and/or fields. Input for the compiler can be specified with the
	-t option. The data format is line-oriented.

		<line> :≡ <qual> <space> <code> ;
		<qual> :≡ <package> '.' <dest> ;
		<dest> :≡ <struct> | <struct> '.' <field> ;

	Lines starting with a '#' are ignored (as comments). Java output
	can take multiple tag lines for the same struct or field. Each
	code line is applied in order of appearance.

EXIT STATUS
	The command exits 0 on success, 1 on error and 2 when invoked
	without arguments.

EXAMPLES
	Compile ./io.colf with compact limits as C:

		colf -b src -s 2048 -l 96 C io.colf

	Compile ./*.colf with a common parent as Java:

		colf -p com.example.model -x com.example.io.IOBean Java

BUGS
	Report bugs at <https://github.com/frbrno/colfer/issues>.

	Text validation is not part of the marshalling and unmarshalling
	process. C and Go just pass any malformed UTF-8 characters. Java
	and JavaScript replace unmappable content with the '?' character
	(ASCII 63).

SEE ALSO
	protoc(1), flatc(1)

It is recommended to commit the generated source code into the respective version control to preserve build consistency and minimise the need for compiler installations. Alternatively, you may use the Maven plugin.

<plugin>
	<groupId>net.quies.colfer</groupId>
	<artifactId>colfer-maven-plugin</artifactId>
	<version>1.11.2</version>
	<configuration>
		<packagePrefix>com/example</packagePrefix>
	</configuration>
</plugin>

Schema

Data structures are defined in .colf files. The format is quite self-explanatory.

// Package demo offers a demonstration.
// These comment lines will end up in the generated code.
package demo

// Course is the grounds where the game of golf is played.
type course struct {
	ID    uint64
	name  text
	holes []hole
	image binary
	tags  []text
}

type hole struct {
	// Lat is the latitude of the cup.
	lat float64
	// Lon is the longitude of the cup.
	lon float64
	// Par is the difficulty index.
	par uint8
	// Water marks the presence of water.
	water bool
	// Sand marks the presence of sand.
	sand bool
}

See what the generated code looks like in C, Go, Java or JavaScript.

The following table shows how Colfer data types are applied per language.

Colfer	C	Go	Java	JavaScript
bool	char	bool	boolean	Boolean
uint8	uint8_t	uint8	byte †	Number
uint16	uint16_t	uint16	short †	Number
uint32	uint32_t	uint32	int †	Number
uint64	uint64_t	uint64	long †	Number ‡
int32	int32_t	int32	int	Number
int64	int64_t	int64	long	Number ‡
float32	float	float32	float	Number
float64	double	float64	double	Number
timestamp	timespec	time.Time ††	time.Instant	Date + Number
text	const char* + size_t	string	String	String
binary	uint8_t* + size_t	[]byte	byte[]	Uint8Array
list	* + size_t	slice	array	Array

• † signed representation of unsigned data, i.e. may overflow to negative.
• ‡ range limited to [1 - 2⁵³, 2⁵³ - 1]
• †† timezone not preserved

Lists may contain floating points, text, binaries or data structures.

Security

Colfer is suited for untrusted data sources such as network I/O or bulk streams. Marshalling and unmarshalling comes with built-in size protection to ensure predictable memory consumption. The format prevents memory bombs by design.

The marshaller may not produce malformed output, regardless of the data input. In no event may the unmarshaller read outside the boundaries of a serial. Fuzz testing did not reveal any volnurabilities yet. Computing power is welcome.

Compatibility

Name changes do not affect the serialization format. Deprecated fields should be renamed to clearly discourage their use. For backwards compatibility new fields must be added to the end of colfer structs. Thus the number of fields can be seen as the schema version.

Performance

Colfer aims to be the fastest and the smallest format without compromising on reliability. See the benchmark wiki for a comparison. Suboptimal performance is treated like a bug.

Documentation

Overview

Package colfer provides schema definitions.

Index

• func FormatFile(path string) (changed bool, err error)
• func GenerateC(basedir string, packages Packages) error
• func GenerateECMA(basedir string, packages Packages) error
• func GenerateGo(basedir string, packages Packages) error
• func GenerateJava(basedir string, packages Packages) error
• type Field
  • func (f *Field) DocText(indent string) string
  • func (f *Field) String() string
• type Package
  • func (p *Package) DocText(ident string) string
  • func (p *Package) HasFloat() bool
  • func (p *Package) HasList() bool
  • func (p *Package) HasTimestamp() bool
  • func (p *Package) Refs() Packages
  • func (p *Package) SchemaFileList() string
• type Packages
  • func ParseFiles(paths ...string) (Packages, error)
  • func (p Packages) ApplyTagFile(path string, options TagOptions) error
  • func (p Packages) FieldsByQName() map[string]*Field
  • func (p Packages) HasTimestamp() bool
  • func (p Packages) Len() int
  • func (p Packages) Less(i, j int) bool
  • func (p Packages) StructsByQName() map[string]*Struct
  • func (p Packages) Swap(i, j int)
• type Struct
  • func (t *Struct) DocText(indent string) string
  • func (t *Struct) HasBinary() bool
  • func (t *Struct) HasBinaryList() bool
  • func (t *Struct) HasFloat() bool
  • func (t *Struct) HasList() bool
  • func (t *Struct) HasText() bool
  • func (t *Struct) HasTimestamp() bool
  • func (t *Struct) String() string
• type TagAllow
• type TagOptions

Functions

func FormatFile

func FormatFile(path string) (changed bool, err error)

FormatFile normalizes the structure. The content is expected to be syntactically correct.

func GenerateC

func GenerateC(basedir string, packages Packages) error

GenerateC writes the code into file "Colfer.h" and "Colfer.c".

func GenerateECMA

func GenerateECMA(basedir string, packages Packages) error

GenerateECMA writes the code into file "Colfer.js".

func GenerateGo

func GenerateGo(basedir string, packages Packages) error

GenerateGo writes the code into file "Colfer.go".

func GenerateJava

func GenerateJava(basedir string, packages Packages) error

GenerateJava writes the code into the respective ".java" files.

Types

type Field

type Field struct {
	// Struct is the parent.
	Struct *Struct
	// Index is the Struct.Fields position.
	Index int
	// Name is the identification token.
	Name string
	// NameNative is the language specific Name.
	NameNative string
	// Docs are the documentation texts.
	Docs []string
	// Type is the datatype.
	Type string
	// TypeNative is the language specific Type.
	TypeNative string
	// TypeRef is the Colfer data structure reference.
	TypeRef *Struct
	// TypeList flags whether the datatype is a list.
	TypeList bool
	// TagAdd has optional source code additions.
	TagAdd []string
}

Field is a Struct member definition.

func (*Field) DocText

func (f *Field) DocText(indent string) string

DocText returns the documentation lines prefixed with ident.

func (*Field) String

func (f *Field) String() string

String returns the qualified name.

type Package

type Package struct {
	// Name is the identification token.
	Name string
	// NameNative is the language specific Name.
	NameNative string
	// Docs are the documentation texts.
	Docs []string
	// Structs are the type definitions.
	Structs []*Struct
	// SchemaFiles are the source filenames.
	SchemaFiles []string
	// SizeMax is the uper limit expression.
	SizeMax string
	// ListMax is the uper limit expression.
	ListMax string
	// SuperClass is the fully qualified path.
	SuperClass string
	// SuperClassNative is the language specific SuperClass.
	SuperClassNative string
	// Interfaces are the fully qualified paths.
	Interfaces []string
	// InterfaceNatives are the language specific Interfaces.
	InterfaceNatives []string
	// CodeSnippet is helpful in book-keeping functionality.
	CodeSnippet string
}

Package is a named definition bundle.

func (*Package) DocText

func (p *Package) DocText(ident string) string

DocText returns the documentation lines prefixed with ident.

func (*Package) HasFloat

func (p *Package) HasFloat() bool

HasFloat returns whether p has one or more floating point fields.

func (*Package) HasList

func (p *Package) HasList() bool

HasList returns whether p has one or more list fields.

func (*Package) HasTimestamp

func (p *Package) HasTimestamp() bool

HasTimestamp returns whether p has one or more timestamp fields.

func (*Package) Refs

func (p *Package) Refs() Packages

Refs returns all direct references sorted by name.

func (*Package) SchemaFileList

func (p *Package) SchemaFileList() string

SchemaFileList returns a listing text.

type Packages

type Packages []*Package

func ParseFiles

func ParseFiles(paths ...string) (Packages, error)

ParseFiles returns the schema definitions.

func (Packages) ApplyTagFile

func (p Packages) ApplyTagFile(path string, options TagOptions) error

func (Packages) FieldsByQName

func (p Packages) FieldsByQName() map[string]*Field

FieldsByQName maps each Field to its respective qualified name (as in <package>.<type>.<field>).

func (Packages) HasTimestamp

func (p Packages) HasTimestamp() bool

HasTimestamp returns whether any of the packages has one or more timestamp fields.

func (Packages) Len

func (p Packages) Len() int

func (Packages) Less

func (p Packages) Less(i, j int) bool

func (Packages) StructsByQName

func (p Packages) StructsByQName() map[string]*Struct

StructsByQName maps each Struct to its respective qualified name (as in <package>.<type>).

func (Packages) Swap

func (p Packages) Swap(i, j int)

type Struct

type Struct struct {
	Pkg *Package
	// Name is the identification token.
	Name string
	// NameNative is the language specific Name.
	NameNative string
	// Docs are the documentation texts.
	Docs []string
	// Fields are the elements in order of appearance.
	Fields []*Field
	// SchemaFile is the source filename.
	SchemaFile string
	// TagAdd has optional source code additions.
	TagAdd []string
}

Struct is a data structure definition.

func (*Struct) DocText

func (t *Struct) DocText(indent string) string

DocText returns the documentation lines prefixed with ident.

func (*Struct) HasBinary

func (t *Struct) HasBinary() bool

HasBinary returns whether s has one or more binary fields.

func (*Struct) HasBinaryList

func (t *Struct) HasBinaryList() bool

HasBinaryList returns whether s has one or more binary list fields.

func (*Struct) HasFloat

func (t *Struct) HasFloat() bool

HasFloat returns whether s has one or more floating point fields.

func (*Struct) HasList

func (t *Struct) HasList() bool

HasList returns whether s has one or more list fields.

func (*Struct) HasText

func (t *Struct) HasText() bool

HasText returns whether s has one or more text fields.

func (*Struct) HasTimestamp

func (t *Struct) HasTimestamp() bool

HasTimestamp returns whether s has one or more timestamp fields.

func (*Struct) String

func (t *Struct) String() string

String returns the qualified name.

type TagAllow

type TagAllow int

TagAllow defines tag options.

const (
	TagNone   TagAllow = iota // not allowed
	TagSingle                 // zero or one
	TagMulti                  // any number
)

type TagOptions

type TagOptions struct {
	StructAllow TagAllow
	FieldAllow  TagAllow
}