scale

README

go-scale Codec

Go implementation of the SCALE (Simple Concatenated Aggregate Little-Endian) data format for types used in the Parity Substrate framework.

SCALE is a light-weight format which allows encoding (and decoding) which makes it highly suitable for resource-constrained execution environments like blockchain runtimes and low-power, low-memory devices.

It is important to note that the encoding context (knowledge of how the types and data structures look) needs to be known separately at both encoding and decoding ends. The encoded data does not include this contextual information.

This codec attempts to translate the primitive Go types to the associated types in SCALE. It also introduces a few custom types to implement Rust primitives that have no direct translation to a Go primitive type.

Translating From SCALE to Go

When translating from SCALE to native Go data, go-scale returns primitive Go data values for corresponding SCALE data values. The table below shows how go-scale translates SCALE types to Go.

Primitives

SCALE/Rust	Go
i8	int8
u8	uint8
i16	int16
u16	uint16
i32	int32
u32	uint32
i64	int64
u64	uint64
i128	*big.Int
u128	*scale.Uint128
bytes	[]byte
string	string
enum	scale.VaryingDataType
struct	struct

Structs

When decoding SCALE data, knowledge of the structure of the destination data type is required to decode. Structs are encoded as a SCALE Tuple, where each struct field is encoded in the sequence of the fields.

Struct Tags

go-scale uses a scale struct tag to modify the order of the field values during encoding. This is also used when decoding attributes back to the original type. This essentially allows you to modify struct field ordering but preserve the encoding/decoding ordering.

See the usage example.

Option

For all Option<T> a pointer to the underlying type is used in go-scale. In the None case the pointer value is nil.

SCALE/Rust	Go
Option<i8>	*int8
Option<u8>	*uint8
Option<i16>	*int16
Option<u16>	*uint16
Option<i32>	*int32
Option<u32>	*uint32
Option<i64>	*int64
Option<u64>	*uint64
Option<i128>	**big.Int
Option<u128>	**scale.Uint128
Option<bytes>	*[]byte
Option<string>	*string
Option<enum>	*scale.VaryingDataType
Option<struct>	*struct
None	nil

Compact Encoding

SCALE uses a compact encoding for variable width unsigned integers.

SCALE/Rust	Go
Compact<u8>	uint
Compact<u16>	uint
Compact<u32>	uint
Compact<u64>	uint
Compact<u128>	*big.Int

Usage

Basic Example

Basic example which encodes and decodes a uint.

import (
	"fmt"
	"github.com/ChainSafe/gossamer/pkg/scale"
)

func ExampleBasic() {
	// compact length encoded uint
	var ui uint = 999
	bytes, err := scale.Marshal(ui)
	if err != nil {
		panic(err)
	}

	var unmarshaled uint
	err = scale.Unmarshal(bytes, &unmarshaled)
	if err != nil {
		panic(err)
	}

	// 999
	fmt.Printf("%d", unmarshaled)
}

Struct Tag Example

Use the scale struct tag for struct fields to conform to specific encoding sequence of struct field values. A struct tag of "-" will be omitted from encoding and decoding.

import (
	"fmt"
	"github.com/ChainSafe/gossamer/pkg/scale"
)

func ExampleStruct() {
	type MyStruct struct {
		Baz bool      `scale:"3"`
		Bar int32     `scale:"2"`
		Foo []byte    `scale:"1"`
		Ignored int64 `scale:"-"`
	}
	var ms = MyStruct{
		Baz: true,
		Bar: 999,
		Foo: []byte{1, 2},
		Ignored: 999
	}
	bytes, err := scale.Marshal(ms)
	if err != nil {
		panic(err)
	}

	var unmarshaled MyStruct
	err = scale.Unmarshal(bytes, &unmarshaled)
	if err != nil {
		panic(err)
	}

	// {Baz:true Bar:999 Foo:[1 2] Ignored:0}
	fmt.Printf("%+v", unmarshaled)
}

Result

A Result is custom type analogous to a rust result. A Result needs to be constructed using the NewResult constructor. The two parameters accepted are the expected types that are associated to the Ok, and Err cases.

// Rust
Result<i32, i32> = Ok(10)

// go-scale
result := scale.NewResult(int32(0), int32(0)
result.Set(scale.Ok, 10)

import (
	"fmt"
	"github.com/ChainSafe/gossamer/pkg/scale"
)

func ExampleResult() {
	// pass in zero or non-zero values of the types for Ok and Err cases
	res := scale.NewResult(bool(false), string(""))

	// set the OK case with a value of true, any values for OK that are not bool will return an error
	err := res.Set(scale.OK, true)
	if err != nil {
		panic(err)
	}

	bytes, err := scale.Marshal(res)
	if err != nil {
		panic(err)
	}

	// [0x00, 0x01]
	fmt.Printf("%v\n", bytes)

	res1 := scale.NewResult(bool(false), string(""))

	err = scale.Unmarshal(bytes, &res1)
	if err != nil {
		panic(err)
	}

	// res1 should be Set with OK mode and value of true
	ok, err := res1.Unwrap()
	if err != nil {
		panic(err)
	}

	switch ok := ok.(type) {
	case bool:
		if !ok {
			panic(fmt.Errorf("unexpected ok value: %v", ok))
		}
	default:
		panic(fmt.Errorf("unexpected type: %T", ok))
	}
}

Varying Data Type

A VaryingDataType is analogous to a Rust enum. A VaryingDataType is an interface that needs to be implemented. From the Polkadot spec there are values associated to a VaryingDataType, which is analagous to a rust enum variant. Each value has an associated index integer value which is used to determine which value type go-scale should decode to. The following interface needs to be implemented for go-scale to be able to marshal from or unmarshal into.

type EncodeVaryingDataType interface {
	IndexValue() (index uint, value any, err error)
	Value() (value any, err error)
	ValueAt(index uint) (value any, err error)
}

type VaryingDataType interface {
	EncodeVaryingDataType
	SetValue(value any) (err error)
}

Example implementation of VaryingDataType:

import (
	"fmt"
	"reflect"

	"github.com/ChainSafe/gossamer/pkg/scale"
)

type MyStruct struct {
	Baz bool
	Bar uint32
	Foo []byte
}

type MyOtherStruct struct {
	Foo string
	Bar uint64
	Baz uint
}

type MyInt16 int16

type MyVaryingDataType struct {
	inner any
}

type MyVaryingDataTypeValues interface {
	MyStruct | MyOtherStruct | MyInt16
}

func setMyVaryingDataType[Value MyVaryingDataTypeValues](mvdt *MyVaryingDataType, value Value) {
	mvdt.inner = value
}

func (mvdt *MyVaryingDataType) SetValue(value any) (err error) {
	switch value := value.(type) {
	case MyStruct:
		setMyVaryingDataType(mvdt, value)
		return
	case MyOtherStruct:
		setMyVaryingDataType(mvdt, value)
		return
	case MyInt16:
		setMyVaryingDataType(mvdt, value)
		return
	default:
		return fmt.Errorf("unsupported type")
	}
}

func (mvdt MyVaryingDataType) IndexValue() (index uint, value any, err error) {
	switch mvdt.inner.(type) {
	case MyStruct:
		return 1, mvdt.inner, nil
	case MyOtherStruct:
		return 2, mvdt.inner, nil
	case MyInt16:
		return 3, mvdt.inner, nil
	}
	return 0, nil, scale.ErrUnsupportedVaryingDataTypeValue
}

func (mvdt MyVaryingDataType) Value() (value any, err error) {
	_, value, err = mvdt.IndexValue()
	return
}

func (mvdt MyVaryingDataType) ValueAt(index uint) (value any, err error) {
	switch index {
	case 1:
		return MyStruct{}, nil
	case 2:
		return MyOtherStruct{}, nil
	case 3:
		return MyInt16(0), nil
	}
	return nil, scale.ErrUnknownVaryingDataTypeValue
}

func ExampleVaryingDataType() {
	vdt := MyVaryingDataType{}

	err := vdt.SetValue(MyStruct{
		Baz: true,
		Bar: 999,
		Foo: []byte{1, 2},
	})
	if err != nil {
		panic(err)
	}

	bytes, err := scale.Marshal(vdt)
	if err != nil {
		panic(err)
	}

	dst := MyVaryingDataType{}

	err = scale.Unmarshal(bytes, &dst)
	if err != nil {
		panic(err)
	}
	
	fmt.Println(reflect.DeepEqual(vdt, dst))
	// Output: true
}

Documentation

Index

• Variables
• func Marshal(v interface{}) (b []byte, err error)
• func MustMarshal(v interface{}) (b []byte)
• func Unmarshal(data []byte, dst interface{}) (err error)
• type Decoder
  • func NewDecoder(r io.Reader) (d *Decoder)
  • func (d *Decoder) Decode(dst interface{}) (err error)
  • func (ds *Decoder) ReadByte() (byte, error)
• type EncodeVaryingDataType
• type Encoder
  • func NewEncoder(writer io.Writer) (encoder *Encoder)
  • func (e *Encoder) Encode(value interface{}) (err error)
• type Marshaler
• type Result
  • func NewResult(okIn, errIn interface{}) (res Result)
  • func (r *Result) IsSet() bool
  • func (r *Result) Set(mode ResultMode, in interface{}) (err error)
  • func (r *Result) Unwrap() (ok interface{}, err error)
• type ResultMode
• type Uint128
  • func MustNewUint128(in interface{}, order ...binary.ByteOrder) (u *Uint128)
  • func NewUint128(in interface{}, order ...binary.ByteOrder) (u *Uint128, err error)
  • func (u *Uint128) Bytes(order ...binary.ByteOrder) (b []byte)
  • func (u *Uint128) Compare(other *Uint128) int
  • func (u Uint128) MarshalJSON() ([]byte, error)
  • func (u *Uint128) String() string
  • func (u *Uint128) UnmarshalJSON(data []byte) error
• type Unmarshaler
• type UnsetResult
• type VaryingDataType
• type WrappedErr
  • func (r WrappedErr) Error() string

Examples

• Result
• VaryingDataType

Variables

var (
	ErrU16OutOfRange            = errors.New("uint16 out of range")
	ErrU32OutOfRange            = errors.New("uint32 out of range")
	ErrU64OutOfRange            = errors.New("uint64 out of range")
	ErrU64NotSupported          = errors.New("uint64 is not supported")
	ErrCompactUintPrefixUnknown = errors.New("unknown prefix for compact uint")
)

var (
	ErrUnsupportedDestination = errors.New("must be a non-nil pointer to a destination")

	ErrUnsupportedType   = errors.New("unsupported type")
	ErrUnsupportedResult = errors.New("unsupported result")

	ErrUnknownVaryingDataTypeValue = errors.New("unable to find VaryingDataTypeValue with index")

	ErrResultNotSet                    = errors.New("result not set")
	ErrResultAlreadySet                = errors.New("result already has an assigned value")
	ErrUnsupportedVaryingDataTypeValue = errors.New("unsupported VaryingDataTypeValue")
	ErrMustProvideVaryingDataTypeValue = errors.New("must provide at least one VaryingDataTypeValue")

	ErrVaryingDataTypeNotSet      = errors.New("varying data type not set")
	ErrUnsupportedCustomPrimitive = errors.New("unsupported type for custom primitive")
	ErrInvalidScaleIndex          = errors.New("invalid scale index")
)

var MaxUint128 = &Uint128{
	Upper: ^uint64(0),
	Lower: ^uint64(0),
}

MaxUint128 is the maximum uint128 value

Functions

func Marshal

func Marshal(v interface{}) (b []byte, err error)

Marshal takes in an interface{} and attempts to marshal into []byte

func MustMarshal

func MustMarshal(v interface{}) (b []byte)

MustMarshal runs Marshal and panics on error.

func Unmarshal

func Unmarshal(data []byte, dst interface{}) (err error)

Unmarshal takes data and a destination pointer to unmarshal the data to.

Types

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder is used to decode from an io.Reader

func NewDecoder

func NewDecoder(r io.Reader) (d *Decoder)

NewDecoder is constructor for Decoder

func (*Decoder) Decode

func (d *Decoder) Decode(dst interface{}) (err error)

Decode accepts a pointer to a destination and decodes into supplied destination

func (*Decoder) ReadByte

func (ds *Decoder) ReadByte() (byte, error)

type EncodeVaryingDataType

type EncodeVaryingDataType interface {
	IndexValue() (index uint, value any, err error)
	Value() (value any, err error)
	ValueAt(index uint) (value any, err error)
}

EncodeVaryingDataType is used in VaryingDataType. It contains the methods required for encoding.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder scale encodes to a given io.Writer.

func NewEncoder

func NewEncoder(writer io.Writer) (encoder *Encoder)

NewEncoder creates a new encoder with the given writer.

func (*Encoder) Encode

func (e *Encoder) Encode(value interface{}) (err error)

Encode scale encodes value to the encoder writer.

type Marshaler

type Marshaler interface {
	MarshalSCALE() ([]byte, error)
}

Marshaler is the interface for custom SCALE marshalling for a given type

type Result

type Result struct {
	// contains filtered or unexported fields
}

Result encapsulates an Ok or an Err case

Example

package main

import (
	"fmt"

	"github.com/ChainSafe/gossamer/pkg/scale"
)

func main() {
	// pass in zero or non-zero values of the types for Ok and Err cases
	res := scale.NewResult(bool(false), string(""))

	// set the OK case with a value of true, any values for OK that are not bool will return an error
	err := res.Set(scale.OK, true)
	if err != nil {
		panic(err)
	}

	bytes, err := scale.Marshal(res)
	if err != nil {
		panic(err)
	}

	// [0x00, 0x01]
	fmt.Printf("%v\n", bytes)

	res1 := scale.NewResult(bool(false), string(""))

	err = scale.Unmarshal(bytes, &res1)
	if err != nil {
		panic(err)
	}

	// res1 should be Set with OK mode and value of true
	ok, err := res1.Unwrap()
	if err != nil {
		panic(err)
	}

	switch ok := ok.(type) {
	case bool:
		if !ok {
			panic(fmt.Errorf("unexpected ok value: %v", ok))
		}
	default:
		panic(fmt.Errorf("unexpected type: %T", ok))
	}
}

func NewResult

func NewResult(okIn, errIn interface{}) (res Result)

NewResult is constructor for Result. Use nil to represent empty tuple () in Rust.

func (*Result) IsSet

func (r *Result) IsSet() bool

IsSet returns whether the Result is set with an Ok or Err value.

func (*Result) Set

func (r *Result) Set(mode ResultMode, in interface{}) (err error)

Set takes in a mode (OK/Err) and the associated interface and sets the Result value

func (*Result) Unwrap

func (r *Result) Unwrap() (ok interface{}, err error)

Unwrap returns the result in go standard wrapping the Err case in a ResultErr

type ResultMode

type ResultMode int

ResultMode is the mode the Result is set to

const (
	// Unset ResultMode is zero value mode
	Unset ResultMode = iota
	// OK case
	OK
	// Err case
	Err
)

type Uint128

type Uint128 struct {
	Upper uint64
	Lower uint64
}

Uint128 represents an unsigned 128 bit integer

func MustNewUint128

func MustNewUint128(in interface{}, order ...binary.ByteOrder) (u *Uint128)

MustNewUint128 will panic if NewUint128 returns an error

func NewUint128

func NewUint128(in interface{}, order ...binary.ByteOrder) (u *Uint128, err error)

NewUint128 is constructor for Uint128 that accepts an option binary.ByteOrder option is only used when inputted interface{} is of type []byte by default binary.LittleEndian is used for []byte since this is SCALE

func (*Uint128) Bytes

func (u *Uint128) Bytes(order ...binary.ByteOrder) (b []byte)

Bytes returns the Uint128 in little endian format by default. A variadic parameter order can be used to specify the binary.ByteOrder used

func (*Uint128) Compare

func (u *Uint128) Compare(other *Uint128) int

Compare returns 1 if the receiver is greater than other, 0 if they are equal, and -1 otherwise.

func (Uint128) MarshalJSON

func (u Uint128) MarshalJSON() ([]byte, error)

MarshalJSON converts Uint128 to []byte.

func (*Uint128) String

func (u *Uint128) String() string

String returns the string format from the Uint128 value

func (*Uint128) UnmarshalJSON

func (u *Uint128) UnmarshalJSON(data []byte) error

UnmarshalJSON converts data to Uint128.

type Unmarshaler

type Unmarshaler interface {
	UnmarshalSCALE(io.Reader) error
}

Unmarshaler is the interface for custom SCALE unmarshalling for a given type

type UnsetResult

type UnsetResult error

UnsetResult is error when Result is unset with a value.

type VaryingDataType

type VaryingDataType interface {
	EncodeVaryingDataType
	SetValue(value any) (err error)
}

VaryingDataType is analogous to a rust enum. Name is taken from polkadot spec.

Example

package main

import (
	"fmt"
	"reflect"

	"github.com/ChainSafe/gossamer/pkg/scale"
)

type MyStruct struct {
	Baz bool
	Bar uint32
	Foo []byte
}

type MyOtherStruct struct {
	Foo string
	Bar uint64
	Baz uint
}

type MyInt16 int16

type MyVaryingDataType struct {
	inner any
}

type MyVaryingDataTypeValues interface {
	MyStruct | MyOtherStruct | MyInt16
}

func setMyVaryingDataType[Value MyVaryingDataTypeValues](mvdt *MyVaryingDataType, value Value) {
	mvdt.inner = value
}

func (mvdt *MyVaryingDataType) SetValue(value any) (err error) {
	switch value := value.(type) {
	case MyStruct:
		setMyVaryingDataType(mvdt, value)
		return
	case MyOtherStruct:
		setMyVaryingDataType(mvdt, value)
		return
	case MyInt16:
		setMyVaryingDataType(mvdt, value)
		return
	default:
		return fmt.Errorf("unsupported type")
	}
}

func (mvdt MyVaryingDataType) IndexValue() (index uint, value any, err error) {
	switch mvdt.inner.(type) {
	case MyStruct:
		return 0, mvdt.inner, nil
	case MyOtherStruct:
		return 1, mvdt.inner, nil
	case MyInt16:
		return 2, mvdt.inner, nil
	}
	return 0, nil, scale.ErrUnsupportedVaryingDataTypeValue
}

func (mvdt MyVaryingDataType) Value() (value any, err error) {
	_, value, err = mvdt.IndexValue()
	return
}

func (mvdt MyVaryingDataType) ValueAt(index uint) (value any, err error) {
	switch index {
	case 0:
		return MyStruct{}, nil
	case 1:
		return MyOtherStruct{}, nil
	case 2:
		return MyInt16(0), nil
	}
	return nil, scale.ErrUnknownVaryingDataTypeValue
}

func main() {
	vdt := MyVaryingDataType{}

	err := vdt.SetValue(MyStruct{
		Baz: true,
		Bar: 999,
		Foo: []byte{1, 2},
	})
	if err != nil {
		panic(err)
	}

	bytes, err := scale.Marshal(vdt)
	if err != nil {
		panic(err)
	}

	dst := MyVaryingDataType{}

	err = scale.Unmarshal(bytes, &dst)
	if err != nil {
		panic(err)
	}

	fmt.Println(reflect.DeepEqual(vdt, dst))
}

Output:

true

type WrappedErr

type WrappedErr struct {
	Err interface{}
}

WrappedErr is returned by Result.Unwrap(). The underlying Err value is wrapped and stored in Err attribute

func (WrappedErr) Error

func (r WrappedErr) Error() string

Error fulfils the error interface