goetf

README

GoETF

[!WARNING] The module is not yet at version 1.0.0 so it's not possible to ensure that there will be no breaking changes.

Go encoding module for ETF (External Term Format).

Why GoETF?

The external term format is mainly used in Erlang's distribution system. Occasionally, it's necessary to encode and decode this particular binary format for communication between different APIs. This format offers the advantage of being faster and more lightweight compared to traditional JSON.

Getting started

Requirements

• go v1.22+

Installation

go get -u github.com/nicolito128/goetf

Encoding example

package main

import (
	"fmt"

	"github.com/nicolito128/goetf"
)

func main() {
	phrase := "Hello, world!"

	data, err := goetf.Marshal(phrase)
	if err != nil {
		panic(err)
	}

	fmt.Println("Encoded:", data)
}

Decoding example

package main

import (
	"fmt"

	"github.com/nicolito128/goetf"
)

func main() {
	var out int

    bin := []byte{131, 98, 0, 0, 1, 1}
	if err := goetf.Unmarshal(bin, &out); err != nil {
		panic(err)
	}

	fmt.Println("Out:", out)
}

For both examples, use the go run command, like:

go run example_file.go

Examples

• Visit the examples folder for more sample codes.

References

• Erlang Runtime System Application (ERTS) - External Term Format

Documentation

Overview

GoETF is a module capable of encoding and decoding byte slices of the ETF type.

The external term format is mainly used in Erlang's distribution system. Occasionally, it's necessary to encode and decode this particular binary format for communication between different APIs.

Start by importing the module:

import "github.com/nicolito128/goetf"

You can use the Marshal function to encode values:

func main() {
	phrase := "Hello, world!"

	data, err := goetf.Marshal(phrase)
	if err != nil {
		panic(err)
	}

	fmt.Println("Encoded:", data)
}

Or use Unmarshal to decode the value:

func main() {
	data := []byte{...}

	var out string
	if err := goetf.Unmarshal(data, &out); err != nil {
		panic(err)
	}

	fmt.Println("Output:", out)
}

Alternatively, you can use the NewEncoder or NewDecoder functions to create your owns.

Index

• Constants
• func IsValidEtt(b byte) bool
• func Marshal(v any, opts ...EncoderOpt) ([]byte, error)
• func TagString(ett ExternalTagType) string
• func Unmarshal(data []byte, v any, opts ...DecoderOpt) error
• type Alias
• type Atom
• type BitString
• type Decoder
  • func NewDecoder(r io.Reader, opts ...DecoderOpt) *Decoder
  • func (d *Decoder) Decode(v any) error
• type DecoderConfig
  • func DefaultDecoderConfig() *DecoderConfig
• type DecoderOpt
  • func WithCacheSize(size int) DecoderOpt
• type Encoder
  • func NewEncoder(w io.Writer, opts ...EncoderOpt) *Encoder
  • func (e *Encoder) Encode(v any) error
  • func (e *Encoder) ReadAll() ([]byte, error)
• type EncoderConfig
  • func DefaultEncoderConfig() *EncoderConfig
• type EncoderOpt
  • func WithStringOverAtom(b bool) EncoderOpt
• type Export
• type ExternalTagType
• type Function
• type List
• type ListImproper
• type Map
• type Marshaler
• type Pid
  • func (p Pid) String() string
• type Port
• type Ref
  • func (r Ref) String() string
• type SizeType
• type String
• type Term
• type Tuple
• type Unmarshaler

Constants

const (
	// Erlang external term format version
	Version = byte(131)
	// Erlang distribution header
	DistHeader = byte(68)
)

const DefaultCacheSize = 1024 * 1024

Functions

func IsValidEtt

func IsValidEtt(b byte) bool

IsValidEtt validates whether the byte argument is an external format flag.

func Marshal

func Marshal(v any, opts ...EncoderOpt) ([]byte, error)

Marshal returns the ETF encoding of v.

func TagString

func TagString(ett ExternalTagType) string

TagString returns the string representation for an external format tag.

func Unmarshal

func Unmarshal(data []byte, v any, opts ...DecoderOpt) error

Unmarshal parses the ETF-encoded data and stores the result in the value pointed to by v.

Types

type Alias

type Alias = Ref

Alias type.

type Atom

type Atom = string

Atom type. An atom is a literal, a constant with name.

Ref: https://www.erlang.org/doc/system/data_types.html#atom

type BitString

type BitString = string

BitString type. A bit string value encodes as a binary (Erlang type: <<...>>)

Ref: https://www.erlang.org/doc/system/data_types.html#bit-strings-and-binaries

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

A Decoder reads and decodes ETF values from an input stream buffer.

func NewDecoder

func NewDecoder(r io.Reader, opts ...DecoderOpt) *Decoder

NewDecoder returns a new *Decoder that reads from r.

The decoder uses its own buffering.

func (*Decoder) Decode

func (d *Decoder) Decode(v any) error

Decode reads the next ETF-encoded data from its buffer and stores it in the value pointed to by v.

type DecoderConfig

type DecoderConfig struct {
	CacheSize int
}

func DefaultDecoderConfig

func DefaultDecoderConfig() *DecoderConfig

DefaultDecoderConfig creates a new default decoder configuration.

type DecoderOpt

type DecoderOpt func(*DecoderConfig)

func WithCacheSize

func WithCacheSize(size int) DecoderOpt

WithCacheSize tells the decoder to an specific size for the internal cache.

CacheSize default value is 1048576 (1024*1024).

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

An Encoder writes ETF values to an output stream.

func NewEncoder

func NewEncoder(w io.Writer, opts ...EncoderOpt) *Encoder

NewEncoder returns a new encoder that writes to w.

func (*Encoder) Encode

func (e *Encoder) Encode(v any) error

Encode writes the ETF encoding of v to the stream.

func (*Encoder) ReadAll

func (e *Encoder) ReadAll() ([]byte, error)

ReadAll reads the entire encoder buffer, then returns a byte slice and an error.

type EncoderConfig

type EncoderConfig struct {
	// Encode data as string over atom type
	StringOverAtom bool
}

An EncoderConfig struct to handle encoding.

func DefaultEncoderConfig

func DefaultEncoderConfig() *EncoderConfig

DefaultEncoderConfig creates a new default encoder configuration.

type EncoderOpt

type EncoderOpt func(*EncoderConfig)

func WithStringOverAtom

func WithStringOverAtom(b bool) EncoderOpt

WithStringOverAtom tells the encoder to always encode strings as ETF String.

StringOverAtom default value is false.

type Export

type Export struct {
	Module   Atom
	Function Atom
	Arity    int
}

Export type.

type ExternalTagType

type ExternalTagType = byte

const (
	EttAtomCacheRef ExternalTagType = 82

	EttAtomUTF8      ExternalTagType = (118)
	EttSmallAtomUTF8 ExternalTagType = (119)

	EttSmallInteger ExternalTagType = (97)
	EttInteger      ExternalTagType = (98)
	EttSmallBig     ExternalTagType = (110)
	EttLargeBig     ExternalTagType = (111)

	EttNewFloat ExternalTagType = (70)
	EttFloat    ExternalTagType = (99)

	EttNewPort ExternalTagType = (89)
	EttPort    ExternalTagType = (102) // since OTP 23, only when BIG_CREATION flag is set

	EttV4Port ExternalTagType = (120)

	EttSmallTuple ExternalTagType = (104)
	EttLargeTuple ExternalTagType = (105)

	EttMap ExternalTagType = (116) // 116 Arity Pairs | K1,V1,K2,V2,...

	EttNil ExternalTagType = (106) // Empyu list: []

	EttListImproper ExternalTagType = (18)  // to be able to encode improper lists like [a|b]
	EttString       ExternalTagType = (107) // used for lists with integers in the range 0..255
	EttList         ExternalTagType = (108)

	EttBitBinary ExternalTagType = (77)
	EttBinary    ExternalTagType = (109)

	EttNewPid         ExternalTagType = (88) // since OTP 23, only when BIG_CREATION flag is set
	EttNewerReference ExternalTagType = (90) // since OTP 21, only when BIG_CREATION flag is set
	EttPid            ExternalTagType = (103)
	EttNewReference   ExternalTagType = (114)

	EttNewFun ExternalTagType = (112)
	EttExport ExternalTagType = (113)
	EttFun    ExternalTagType = (117) // legacy

	EttLocal ExternalTagType = (121) // OTP 26.0

	EttAtom      ExternalTagType = (100) // deprecated
	EttRef       ExternalTagType = (101) // deprecated
	EttSmallAtom ExternalTagType = (115) // deprecated
)

Erlang external term tags.

type Function

type Function struct {
	Arity  byte
	Unique [16]byte
	Index  uint32
	//	Free      uint32
	Module    Atom
	OldIndex  uint32
	OldUnique uint32
	Pid       Pid
	FreeVars  []Term
}

Function type.

Ref: https://www.erlang.org/doc/system/data_types.html#fun

type List

type List = []Term

List type. A list is a compound data type with a variable number of terms.

Ref: https://www.erlang.org/doc/system/data_types.html#list

type ListImproper

type ListImproper = []Term

ListImproper as a workaround for the Erlang's improper list [a|b]. Intended to be used to interact with Erlang.

type Map

type Map = map[Term]Term

Map type. A map is a compound data type with a variable number of key-value associations.

Ref: https://www.erlang.org/doc/system/data_types.html#map

type Marshaler

type Marshaler interface {
	MarshalETF(data []byte, src any) (err error)
}

Marshaler is the interface implemented by types that can marshal themselves into valid ETF.

type Pid

type Pid struct {
	Node     Atom
	ID       uint64
	Serial   uint32
	Creation uint32
}

Pid type.

Ref: https://www.erlang.org/doc/system/data_types.html#pid

func (Pid) String

func (p Pid) String() string

type Port

type Port struct {
	Node     Atom
	ID       uint32
	Creation uint32
}

Port type.

Ref: https://www.erlang.org/doc/system/data_types.html#port-identifier

type Ref

type Ref struct {
	Node     Atom
	Creation uint32
	ID       [5]uint32
}

Ref type.

Link: https://www.erlang.org/doc/system/data_types.html#reference

func (Ref) String

func (r Ref) String() string

type SizeType

type SizeType = int

SizeType refers to the length in bytes associated with an fixed external term type.

const (
	SizeNil SizeType = 0

	SizeSmallInteger    SizeType = 1
	SizeAtomCacheRef    SizeType = 1
	SizeSmallTupleArity SizeType = 1
	SizeSmallBigN       SizeType = 1
	SizeSmallBigSign    SizeType = 1
	SizeLargeBigSign    SizeType = 1
	SizeBitBinaryBits   SizeType = 1
	SizeSmallAtom       SizeType = 1
	SizeSmallAtomUTF8   SizeType = 1

	SizeAtom         SizeType = 2
	SizeAtomUTF8     SizeType = 2
	SizeStringLength SizeType = 2

	SizeLargeBigN       SizeType = 4
	SizeInteger         SizeType = 4
	SizeMapArity        SizeType = 4
	SizeBinaryLen       SizeType = 4
	SizeListLength      SizeType = 4
	SizeLargeTupleArity SizeType = 4
	SizeBitBinaryLen    SizeType = 4

	SizeNewFloat SizeType = 8

	SizeFloat SizeType = 31
)

ETF fixed type sizes.

type String

type String = string

String type. Strings are a shorthan for a character list (Erlang type: [$e, $t, $f]).

Ref: https://www.erlang.org/doc/system/data_types.html#string

type Term

type Term = any

Term is a piece of data of any data type.

Ref: https://www.erlang.org/doc/system/data_types.html#terms

type Tuple

type Tuple = []Term

Tuple type. A tuple is a compound data type with a fixed number of terms.

Ref: https://www.erlang.org/doc/system/data_types.html#tuple

type Unmarshaler

type Unmarshaler interface {
	UnmarshalETF(data []byte, dst any) (err error)
}

Unmarshaler is the interface implemented by types that can unmarshal a ETF description of themselves. The input can be assumed to be a valid encoding of a ETF value. UnmarshalETF must copy the ETF data if it wishes to retain the data after returning.