collection

README

Collection

The collection node is responsible for accepting transactions from users, packaging them into collections to ease the load on consensus nodes, and storing transaction texts for the duration they are needed by the network.

This document provides a high-level overview of the collection node. Each section includes links to the appropriate package, which may contain more detailed documentation.

Table of Contents

• Terminology
• Processes
  • Transaction Lifecycle
  • Collection Lifecycle
• Engines
  • Ingest
  • Proposal
  • Synchronization
  • Provider
• Storage
  • Cluster State
    • Collection Builder
    • Collection Finalizer

Terminology

• Collection - a set of transactions proposed by a cluster of collection nodes.
• Guaranteed Collection - a collection that a quorum of nodes in the cluster has committed to storing.
• Collection Guarantee - an attestation to a collection that has been guaranteed. Concretely, this is a hash over the collection and signatures from a qualified majority of cluster members. (Sometimes simply referred to as guarantee.)
• Cluster - a group of collection nodes that work together to create collections. Each cluster is responsible for a different subset of transactions.

Processes

Transaction Lifecycle

1. Transactions are received by a collection node (typically via an Access Node).
2. Transactions are propagated to collection nodes in the responsible cluster.
3. Transactions are introduced into the memory pool.
4. Transactions are included in a collection proposal.
5. Transactions are removed from the memory pool when the collection is guaranteed.

Collection Lifecycle

1. A collection is proposed by a cluster member.
2. The collection is finalized by the core consensus algorithm.
3. A guarantee for the finalized collection is submitted to consensus nodes.
4. The guarantee is included in a block.
5. The block is propagated to execution nodes.
6. Execution nodes request the full collection from collection nodes.

Engines

Ingest

The ingest engine is responsible for accepting, validating, and storing new transactions. Once a transaction has been ingested, it can be included in a new collection via the proposal engine.

In general, collection nodes cannot fully ensure that a transaction is valid. Consequently, the validation performed at this stage is best-effort.

Proposal

The proposal engine is responsible for handling the consensus process for the cluster. It runs an instance of HotStuff within the cluster. This results in each cluster building a secondary blockchain, where each block represents a proposed collection.

Synchronization

The synchronization engine is responsible for keeping the node in sync with its cluster. It periodically polls cluster members for their latest finalized collection, and handles requesting ranges of finalized collections when the node is behind. It also handles requesting specific collections when the proposal engine receives a new proposal for which the parent is not known.

Provider

The provider engine is responsible for responding to requests for resources stored by the collection node, typically full collections or individual transactions. In particular, execution nodes and verification nodes request collections in order to execute the constituent transactions.

Storage

Cluster State

The cluster state implements a storage layer for the blockchain produced by the proposal engine. The design mirrors the Protocol State, which is the storage layer for the main Flow blockchain produced by consensus nodes.

Since the core HotStuff logic is only aware of blocks, we sometimes use the term "cluster block" to refer to a block within a cluster's auxiliary blockchain. Each cluster block contains a single proposed collection.

Collection Builder

The collection builder implements the logic for building new, valid collections to propose to the cluster.

Valid collections contain transactions that have passed basic validation, are not expired, and do not contain any transactions that already exist in a parent collection.

Collection Finalizer

The collection finalizer marks a collection as finalized (aka guaranteed). This is invoked by the core HotStuff logic when it determines that a collection has been finalized by the cluster.

When a collection is finalized, the corresponding collection guarantee is sent to consensus nodes for inclusion in a block.