ssz

package module
v0.1.2-patch Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2023 License: MIT Imports: 17 Imported by: 3

README

FastSSZ

The FastSSZ project in this reposity is a combination of two things: a high performant low level library to work with SSZ encodings (root of this project) and the (sszgen) code generator that generates the SSZ encodings for Go structs using the SSZ library. By combining both, this library achieves peak Go native performance and zero memory allocation. The repository uses as test the official Ethereum SSZ tests (spectests) for the Consensus Spec data structures.

If you are only looking for the Consensus data structures and types with the SSZ support, it is recommended to use go-eth-consensus instead, since it is already integrated with other parts of the Consensus stack, like the Beacon http API.

Clone:

$ git clone git@github.com:ferranbt/fastssz.git

Download the eth2.0 spec tests

$ make get-spec-tests

Regenerate the test spec encodings:

$ make build-spec-tests

Generate encodings for a specific package:

$ go run sszgen/*.go --path ./ethereumapis/eth/v1alpha1 [--objs BeaconBlock,Eth1Data]

Optionally, you can specify the objs you want to generate. Otherwise, it will generate encodings for all structs in the package. Note that if a struct does not have 'ssz' tags when required (i.e size of arrays), the generator will fail.

By default, it generates a file with the prefix '_encoding.go' for each file that contains a generated struct. Optionally, you can combine all the outputs in a single file with the 'output' flag.

$ go run sszgen/*.go --path ./ethereumapis/eth/v1alpha1 --output ./ethereumapis/eth/v1alpha1/encoding.go

Test the spectests:

$ go test -v ./spectests/... -run TestSpec

Run the fuzzer:

$ FUZZ_TESTS=True go test -v ./spectests/... -run TestFuzz

To install the generator run:

$ go get github.com/ferranbt/fastssz/sszgen

Benchmark (BeaconBlock):

$ go test -v ./spectests/... -run=XXX -bench=.
goos: linux
goarch: amd64
pkg: github.com/ferranbt/fastssz/spectests
cpu: AMD Ryzen 5 2400G with Radeon Vega Graphics
BenchmarkMarshal_Fast
BenchmarkMarshal_Fast-8             	  291054	      4088 ns/op	    8192 B/op	       1 allocs/op
BenchmarkMarshal_SuperFast
BenchmarkMarshal_SuperFast-8        	  798883	      1354 ns/op	       0 B/op	       0 allocs/op
BenchmarkUnMarshal_Fast
BenchmarkUnMarshal_Fast-8           	   64065	     17614 ns/op	   11900 B/op	     210 allocs/op
BenchmarkHashTreeRoot_Fast
BenchmarkHashTreeRoot_Fast-8        	   25863	     45932 ns/op	       0 B/op	       0 allocs/op
BenchmarkHashTreeRoot_SuperFast
BenchmarkHashTreeRoot_SuperFast-8   	   54078	     21999 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/ferranbt/fastssz/spectests	5.501s

Package reference

To reference a struct from another package use the '--include' flag to point to that package.

Example:

$ go run sszgen/*.go --path ./example2
$ go run sszgen/*.go --path ./example
[ERR]: could not find struct with name 'Checkpoint'
$ go run sszgen/*.go --path ./example --include ./example2

There are some caveats required to use this functionality.

  • If multiple input paths import the same package, all of them need to import it with the same alias if any.
  • If the folder of the package is not the same as the name of the package, any input file that imports this package needs to do it with an alias.

Fast HashTreeRoot

Fastssz integrates with Prysm gohashtree library to do high performance and concurrent Sha256 hashing. It achieves a 2x performance improvement with respect to the normal sequential hashing. As of now, this feature is not yet enabled by default since it does not use the gohashtree main branch. You can track the updates on this issue.

In order to use this feature, enable manually the hash function in the Hasher like in the benchmark example.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrOffset                = fmt.Errorf("incorrect offset")
	ErrSize                  = fmt.Errorf("incorrect size")
	ErrBytesLength           = fmt.Errorf("bytes array does not have the correct length")
	ErrVectorLength          = fmt.Errorf("vector does not have the correct length")
	ErrListTooBig            = fmt.Errorf("list length is higher than max value")
	ErrEmptyBitlist          = fmt.Errorf("bitlist is empty")
	ErrInvalidVariableOffset = fmt.Errorf("invalid ssz encoding. first variable element offset indexes into fixed value data")
)
View Source
var (
	// ErrIncorrectByteSize means that the byte size is incorrect
	ErrIncorrectByteSize = fmt.Errorf("incorrect byte size")

	// ErrIncorrectListSize means that the size of the list is incorrect
	ErrIncorrectListSize = fmt.Errorf("incorrect list size")
)

Functions

func CalculateLimit

func CalculateLimit(maxCapacity, numItems, size uint64) uint64

func DecodeDynamicLength

func DecodeDynamicLength(buf []byte, maxSize int) (int, error)

DecodeDynamicLength decodes the length from the dynamic input

func DivideInt

func DivideInt(a, b int) (int, bool)

DivideInt divides the int fully

func DivideInt2

func DivideInt2(a, b, max int) (int, error)

func ErrBytesLengthFn

func ErrBytesLengthFn(name string, found, expected int) error

func ErrListTooBigFn

func ErrListTooBigFn(name string, found, max int) error

func ErrVectorLengthFn

func ErrVectorLengthFn(name string, found, expected int) error

func ExtendUint16

func ExtendUint16(b []uint16, needLen int) []uint16

ExtendUint16 extends a uint16 buffer to a given size

func ExtendUint64

func ExtendUint64(b []uint64, needLen int) []uint64

ExtendUint64 extends a uint64 buffer to a given size

func ExtendUint8

func ExtendUint8(b []uint8, needLen int) []uint8

ExtendUint16 extends a uint16 buffer to a given size

func HashWithDefaultHasher

func HashWithDefaultHasher(v HashRoot) ([32]byte, error)

HashWithDefaultHasher hashes a HashRoot object with a Hasher from the default HasherPool

func MarshalBool

func MarshalBool(dst []byte, b bool) []byte

MarshalBool marshals a boolean to dst

func MarshalSSZ

func MarshalSSZ(m Marshaler) ([]byte, error)

MarshalSSZ marshals an object

func MarshalTime

func MarshalTime(dst []byte, t time.Time) []byte

MarshalTime marshals a time to dst

func MarshalUint16

func MarshalUint16(dst []byte, i uint16) []byte

MarshalUint16 marshals a little endian uint16 to dst

func MarshalUint32

func MarshalUint32(dst []byte, i uint32) []byte

MarshalUint32 marshals a little endian uint32 to dst

func MarshalUint64

func MarshalUint64(dst []byte, i uint64) []byte

MarshalUint64 marshals a little endian uint64 to dst

func MarshalUint8

func MarshalUint8(dst []byte, i uint8) []byte

MarshalUint8 marshals a little endian uint8 to dst

func ReadOffset

func ReadOffset(buf []byte) uint64

ReadOffset reads an offset from buf

func UnmarshalBool

func UnmarshalBool(src []byte) bool

UnmarshalBool unmarshals a boolean from the src input

func UnmarshalDynamic

func UnmarshalDynamic(src []byte, length int, f func(indx int, b []byte) error) error

UnmarshalDynamic unmarshals the dynamic items from the input

func UnmarshalSSZTest

func UnmarshalSSZTest(content []byte, result interface{}) error

func UnmarshalTime

func UnmarshalTime(src []byte) time.Time

UnmarshalTime unmarshals a time.Time from the src input

func UnmarshallUint16

func UnmarshallUint16(src []byte) uint16

UnmarshallUint16 unmarshals a little endian uint16 from the src input

func UnmarshallUint32

func UnmarshallUint32(src []byte) uint32

UnmarshallUint32 unmarshals a little endian uint32 from the src input

func UnmarshallUint64

func UnmarshallUint64(src []byte) uint64

UnmarshallUint64 unmarshals a little endian uint64 from the src input

func UnmarshallUint8

func UnmarshallUint8(src []byte) uint8

UnmarshallUint8 unmarshals a little endian uint8 from the src input

func ValidateBitlist

func ValidateBitlist(buf []byte, bitLimit uint64) error

ValidateBitlist validates that the bitlist is correct

func VerifyMultiproof

func VerifyMultiproof(root []byte, proof [][]byte, leaves [][]byte, indices []int) (bool, error)

VerifyMultiproof verifies a proof for multiple leaves against the given root.

func VerifyProof

func VerifyProof(root []byte, proof *Proof) (bool, error)

VerifyProof verifies a single merkle branch. It's more efficient than VerifyMultiproof for proving one leaf.

func WriteOffset

func WriteOffset(dst []byte, i int) []byte

WriteOffset writes an offset to dst

Types

type CompressedMultiproof

type CompressedMultiproof struct {
	Indices    []int
	Leaves     [][]byte
	Hashes     [][]byte
	ZeroLevels []int // Stores the level for every omitted zero hash in the proof
}

CompressedMultiproof represents a compressed merkle proof of several leaves. Compression is achieved by omitting zero hashes (and their hashes). `ZeroLevels` contains information which helps the verifier fill in those hashes.

func (*CompressedMultiproof) Decompress

func (c *CompressedMultiproof) Decompress() *Multiproof

Decompress returns a new multiproof, filling in the omitted zero hashes. See `CompressedMultiProof` for more info.

type HashFn

type HashFn func(dst []byte, input []byte) error

func NativeHashWrapper

func NativeHashWrapper(hashFn hash.Hash) HashFn

type HashRoot

type HashRoot interface {
	GetTree() (*Node, error)
	HashTreeRoot() ([32]byte, error)
	HashTreeRootWith(hh HashWalker) error
}

type HashWalker

type HashWalker interface {
	// Intended for testing purposes to know the latest hash generated during merkleize
	Hash() []byte
	AppendUint8(i uint8)
	AppendUint64(i uint64)
	AppendBytes32(b []byte)
	PutUint64(i uint64)
	PutUint32(i uint32)
	PutUint16(i uint16)
	PutUint8(i uint8)
	FillUpTo32()
	Append(i []byte)
	PutBitlist(bb []byte, maxSize uint64)
	PutBool(b bool)
	PutBytes(b []byte)
	Index() int
	Merkleize(indx int)
	MerkleizeWithMixin(indx int, num, limit uint64)
}

type Hasher

type Hasher struct {
	// contains filtered or unexported fields
}

Hasher is a utility tool to hash SSZ structs

func NewHasher

func NewHasher() *Hasher

NewHasher creates a new Hasher object with sha256 hash

func NewHasherWithHash

func NewHasherWithHash(hh hash.Hash) *Hasher

NewHasherWithHash creates a new Hasher object with a custom hash.Hash function

func NewHasherWithHashFn

func NewHasherWithHashFn(hh HashFn) *Hasher

NewHasherWithHashFn creates a new Hasher object with a custom HashFn function

func (*Hasher) Append

func (h *Hasher) Append(i []byte)

func (*Hasher) AppendBytes32

func (h *Hasher) AppendBytes32(b []byte)

func (*Hasher) AppendUint64

func (h *Hasher) AppendUint64(i uint64)

func (*Hasher) AppendUint8

func (h *Hasher) AppendUint8(i uint8)

func (*Hasher) FillUpTo32

func (h *Hasher) FillUpTo32()

func (*Hasher) Hash

func (h *Hasher) Hash() []byte

func (*Hasher) HashRoot

func (h *Hasher) HashRoot() (res [32]byte, err error)

HashRoot creates the hash final hash root

func (*Hasher) Index

func (h *Hasher) Index() int

Index marks the current buffer index

func (*Hasher) Merkleize

func (h *Hasher) Merkleize(indx int)

Merkleize is used to merkleize the last group of the hasher

func (*Hasher) MerkleizeWithMixin

func (h *Hasher) MerkleizeWithMixin(indx int, num, limit uint64)

MerkleizeWithMixin is used to merkleize the last group of the hasher

func (*Hasher) PutBitlist

func (h *Hasher) PutBitlist(bb []byte, maxSize uint64)

PutBitlist appends a ssz bitlist

func (*Hasher) PutBool

func (h *Hasher) PutBool(b bool)

PutBool appends a boolean

func (*Hasher) PutBytes

func (h *Hasher) PutBytes(b []byte)

PutBytes appends bytes

func (*Hasher) PutRootVector

func (h *Hasher) PutRootVector(b [][]byte, maxCapacity ...uint64) error

PutRootVector appends an array of roots

func (*Hasher) PutUint16

func (h *Hasher) PutUint16(i uint16)

PutUint16 appends a uint16 in 32 bytes

func (*Hasher) PutUint32

func (h *Hasher) PutUint32(i uint32)

PutUint32 appends a uint32 in 32 bytes

func (*Hasher) PutUint64

func (h *Hasher) PutUint64(i uint64)

PutUint64 appends a uint64 in 32 bytes

func (*Hasher) PutUint64Array

func (h *Hasher) PutUint64Array(b []uint64, maxCapacity ...uint64)

PutUint64Array appends an array of uint64

func (*Hasher) PutUint8

func (h *Hasher) PutUint8(i uint8)

PutUint16 appends a uint16 in 32 bytes

func (*Hasher) Reset

func (h *Hasher) Reset()

Reset resets the Hasher obj

type HasherPool

type HasherPool struct {
	// contains filtered or unexported fields
}

HasherPool may be used for pooling Hashers for similarly typed SSZs.

var DefaultHasherPool HasherPool

DefaultHasherPool is a default hasher pool

func (*HasherPool) Get

func (hh *HasherPool) Get() *Hasher

Get acquires a Hasher from the pool.

func (*HasherPool) Put

func (hh *HasherPool) Put(h *Hasher)

Put releases the Hasher to the pool.

type Marshaler

type Marshaler interface {
	MarshalSSZTo(dst []byte) ([]byte, error)
	MarshalSSZ() ([]byte, error)
	SizeSSZ() int
}

Marshaler is the interface implemented by types that can marshal themselves into valid SZZ.

type Multiproof

type Multiproof struct {
	Indices []int
	Leaves  [][]byte
	Hashes  [][]byte
}

Multiproof represents a merkle proof of several leaves.

func (*Multiproof) Compress

func (p *Multiproof) Compress() *CompressedMultiproof

Compress returns a new proof with zero hashes omitted. See `CompressedMultiproof` for more info.

type Node

type Node struct {
	// contains filtered or unexported fields
}

Node represents a node in the tree backing of a SSZ object.

func EmptyLeaf

func EmptyLeaf() *Node

func LeafFromBool

func LeafFromBool(b bool) *Node

func LeafFromBytes

func LeafFromBytes(b []byte) *Node

func LeafFromUint16

func LeafFromUint16(i uint16) *Node

func LeafFromUint32

func LeafFromUint32(i uint32) *Node

func LeafFromUint64

func LeafFromUint64(i uint64) *Node

func LeafFromUint8

func LeafFromUint8(i uint8) *Node

func LeavesFromUint64

func LeavesFromUint64(items []uint64) []*Node

func NewNodeWithLR

func NewNodeWithLR(left, right *Node) *Node

NewNodeWithLR initializes a branch node.

func NewNodeWithValue

func NewNodeWithValue(value []byte) *Node

NewNodeWithValue initializes a leaf node.

func ProofTree

func ProofTree(v HashRoot) (*Node, error)

ProofTree hashes a HashRoot object with a Hasher from the default HasherPool

func TreeFromChunks

func TreeFromChunks(chunks [][]byte) (*Node, error)

TreeFromChunks constructs a tree from leaf values. The number of leaves should be a power of 2.

func TreeFromNodes

func TreeFromNodes(leaves []*Node) (*Node, error)

TreeFromNodes constructs a tree from leaf nodes. This is useful for merging subtrees. The number of leaves should be a power of 2.

func TreeFromNodesWithMixin

func TreeFromNodesWithMixin(leaves []*Node, num, limit int) (*Node, error)

func (*Node) Get

func (n *Node) Get(index int) (*Node, error)

Get fetches a node with the given general index.

func (*Node) Hash

func (n *Node) Hash() []byte

Hash returns the hash of the subtree with the given Node as its root. If root has no children, it returns root's value (not its hash).

func (*Node) Prove

func (n *Node) Prove(index int) (*Proof, error)

Prove returns a list of sibling values and hashes needed to compute the root hash for a given general index.

func (*Node) ProveMulti

func (n *Node) ProveMulti(indices []int) (*Multiproof, error)

func (*Node) Show

func (n *Node) Show(maxDepth int)

type Proof

type Proof struct {
	Index  int
	Leaf   []byte
	Hashes [][]byte
}

Proof represents a merkle proof against a general index.

type Unmarshaler

type Unmarshaler interface {
	UnmarshalSSZ(buf []byte) error
}

Unmarshaler is the interface implemented by types that can unmarshal a SSZ description of themselves

type Wrapper

type Wrapper struct {
	// contains filtered or unexported fields
}

func (*Wrapper) AddBytes

func (w *Wrapper) AddBytes(b []byte)

func (*Wrapper) AddEmpty

func (w *Wrapper) AddEmpty()

func (*Wrapper) AddNode

func (w *Wrapper) AddNode(n *Node)

func (*Wrapper) AddUint16

func (w *Wrapper) AddUint16(i uint16)

func (*Wrapper) AddUint32

func (w *Wrapper) AddUint32(i uint32)

func (*Wrapper) AddUint64

func (w *Wrapper) AddUint64(i uint64)

func (*Wrapper) AddUint8

func (w *Wrapper) AddUint8(i uint8)

func (*Wrapper) Append

func (w *Wrapper) Append(i []byte)

func (*Wrapper) AppendBytes32

func (w *Wrapper) AppendBytes32(b []byte)

func (*Wrapper) AppendUint64

func (w *Wrapper) AppendUint64(i uint64)

func (*Wrapper) AppendUint8

func (w *Wrapper) AppendUint8(i uint8)

func (*Wrapper) Commit

func (w *Wrapper) Commit(i int)

func (*Wrapper) CommitWithMixin

func (w *Wrapper) CommitWithMixin(i, num, limit int)

func (*Wrapper) FillUpTo32

func (w *Wrapper) FillUpTo32()

func (*Wrapper) Hash

func (w *Wrapper) Hash() []byte

func (*Wrapper) Index

func (w *Wrapper) Index() int

func (*Wrapper) Merkleize

func (w *Wrapper) Merkleize(indx int)

func (*Wrapper) MerkleizeWithMixin

func (w *Wrapper) MerkleizeWithMixin(indx int, num, limit uint64)

func (*Wrapper) Node

func (w *Wrapper) Node() *Node

func (*Wrapper) PutBitlist

func (w *Wrapper) PutBitlist(bb []byte, maxSize uint64)

func (*Wrapper) PutBool

func (w *Wrapper) PutBool(b bool)

func (*Wrapper) PutBytes

func (w *Wrapper) PutBytes(b []byte)

func (*Wrapper) PutUint16

func (w *Wrapper) PutUint16(i uint16)

func (*Wrapper) PutUint32

func (w *Wrapper) PutUint32(i uint32)

func (*Wrapper) PutUint64

func (w *Wrapper) PutUint64(i uint64)

func (*Wrapper) PutUint8

func (w *Wrapper) PutUint8(i uint8)

Directories

Path Synopsis
Code generated by fastssz.
Code generated by fastssz.
testcases
Code generated by fastssz.
Code generated by fastssz.
testcases/other
Code generated by fastssz.
Code generated by fastssz.
Code generated by fastssz.
Code generated by fastssz.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL