mir

README

Mir - The Distributed Protocol Implementation Framework

Mir is a framework for implementing, debugging, and analyzing distributed protocols. It has the form of a library that provides abstractions representing different components of a distributed system and an engine orchestrating their interaction.

Mir aims to be general enough to enable implementing distributed protocols in a way that is agnostic to network transport, storage, and cryptographic primitive implementation. All these (and other) usual components of a distributed protocol implementation are encapsulated in abstractions with well-defined interfaces. While Mir provides some implementations of those abstractions to be used directly "out of the box", the consumer is free to provide their own implementations.

The first intended use of Mir is as a scalable and efficient consensus layer in Filecoin subnets and, potentially, as a Byzantine fault-tolerant ordering service in Hyperledger Fabric. However, Mir hopes to be a building block of a next generation of distributed systems, being used by many applications.

Nodes, Modules, and Events

Mir is a framework for implementing distributed protocols (also referred to as distributed algorithms) meant to run on a distributed system. The basic unit of a distributed system is a node. Each node locally executes (its portion of) a protocol, sending and receiving messages to and from other nodes over a communication network.

Mir models a node of such a distributed system and presents the consumer (the programmer using Mir) with a Node abstraction. The main task of a Node is to process events.

A node contains one or multiple Modules that implement the logic for event processing. Each module independently consumes, processes, and outputs events. This approach bears resemblance to the actor model, where events exchanged between modules correspond to messages exchanged between actors.

The Node implements an event loop where all events created by modules are stored in a buffer and distributed to their corresponding target modules for processing. For example, when the networking module receives a protocol message over the network, it generates a MessageReceived event (containing the received message) that the node implementation routes to the protocol module, which processes the message, potentially outputting SendMessage events that the Node implementation routes back to the networking module.

The architecture described above enables a powerful debugging approach. All Events in the event loop can, in debug mode, be recorded, inspected, or even modified and replayed to the Node using a debugging interface.

In practice, when instantiating a Node, the consumer of Mir provides implementations of these modules to Mir. For example, instantiating a node of a state machine replication system might look as follows:

    // Example Node instantiation
    node, err := mir.NewNode(
		/* some more technical arguments ... */
		&modules.Modules{
			// ... 
			"app":      NewChatApp(),
			"protocol": TOBProtocol,
			"net":      grpcNetworking,
			"crypto":   ecdsaCrypto,
		},
		eventInterceptor,
		writeAheadLog,
	)

Here the consumer provides modules for networking (implements sending and receiving messages over the network), the protocol logic (using some total-order broadcast protocol), the application (implementing the logic of the replicated app), and a cryptographic module (able to produce and verify digital signatures using ECDSA). the eventInterceptor implements recording of the events passed between the modules for analysis and debugging purposes. The writeAheadLog is a special module that enables the node to recover from crashes. For more details, see the Documentation.

The programmer working with Mir is free to provide own implementations of these modules, but Mir already comes bundled with several module implementations out of the box.

Relation to the Mir-BFT protocol

Mir-BFT is a scalable atomic broadcast protocol. The Mir framework initially started as an implementation of that protocol - thus the related name - but has since been made completely independent of Mir-BFT. Even the implementation of the Mir-BFT protocol itself has been abandoned and replaced by its successor, ISS, which is intended to be the first protocol implemented within Mir. However, since Mir is designed to be modular and versatile, ISS is just one (the first) of the protocols implemented in Mir.

Current Status

This library is in development. This document describes what the library should become rather than what it currently is. This document itself is more than likely to still change. You are more than welcome to contribute to accelerating the development of the Mir library as an open-source contributor. Have a look at the Contributions section if you want to help out!

Compiling and running tests

Assuming Go version 1.18 or higher is installed, the tests can be run by executing go test ./... in the root folder of the repository. The dependencies should be downloaded and installed automatically. Some of the dependencies may require gcc installed. On Ubuntu Linux, it can be done by invoking sudo apt install gcc.

If the sources have been updated, it is possible that some of the generated source files need to be updated as well. More specifically, the Mir library relies on Protocol Buffers and gomock. The protoc compiler and the corresponding Go plugin need to be installed as well as mockgen. On Ubuntu Linux, those can be installed using

sudo snap install --classic protobuf
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install github.com/golang/mock/mockgen@v1.6.0

Make sure (by configuring your shell interpreter) that the directory with Go binaries (usually ~/go/bin by default) is included in the PATH environment variable. On a default Ubuntu Linux system, for example, this can be achieved by running

echo 'PATH=$PATH:~/go/bin' >> ~/.profile

and restarting the terminal.

Once the dependencies are ready, the generated sources can be updated by executing go generate ./... in the root directory.

Documentation

For a description of the design and inner workings of the library, see Mir Library Overview. We also keep a log of Architecture Decision Records (ADRs).

For a small demo application, see /samples/chat-demo

For an automated deployment of Mir on a set of remote machines, see the remote deployment instructions.

Getting started

To get started using (and contributing to) Mir, in addition to this README, we recommend the following:

1. Watch the first introductory video
2. Read the Mir Library Overview
3. Watch the second introductory video. (Very low-level coding, this is not how Mir coding works in real life - it is for developers to understand how Mir internally works. Realistic coding videos will follow soon.)
4. Check out the chat-demo sample application to learn how to use Mir for state machine replication.
5. To see an example of using a DSL module (allowing to write pseudocode-like code for the protocol logic), look at the implementation of Byzantine Consistent Broadcast (BCB) being used in the corresponding sample application. Original pseudocode can also be found in these lecture notes (Algorithm 4 (Echo broadcast [Rei94])).
6. A more complex example of DSL code is the implementation of the SMR availability layer (concretely the multisigcollector).

To learn about the first complex system being built on Mir, have a look at Trantor, a complex SMR syste being implemented using Mir.

Contributing

Contributions are more than welcome!

If you want to contribute, have a look at the open issues. If you have any questions (specific or general), do not hesitate to drop an email to the active maintainer(s) or write a message to the developers in the public Slack channel #mir-dev of the Filecoin project.

Active maintainer(s)

• Matej Pavlovic (matopavlovic@gmail.com)

License

The Mir library source code is made available under the Apache License, version 2.0 (Apache-2.0), located in the LICENSE file.

Acknowledgments

This project is a continuation of the development started under the name mirbft as a Hyperledger Lab.

Documentation

Index

• Variables
• type Node
  • func NewNode(id t.NodeID, config *NodeConfig, m modules.Modules, ...) (*Node, error)
  • func (n *Node) Debug(ctx context.Context, eventsOut chan *events.EventList) error
  • func (n *Node) InjectEvents(ctx context.Context, events *events.EventList) error
  • func (n *Node) Run(ctx context.Context) error
  • func (n *Node) Stop()
• type NodeConfig
  • func DefaultNodeConfig() *NodeConfig
  • func (c *NodeConfig) Validate() error
  • func (c *NodeConfig) WithLogger(logger logging.Logger) *NodeConfig
• type Stopwatch
  • func (s *Stopwatch) Read() time.Duration
  • func (s *Stopwatch) Reset() time.Duration
  • func (s *Stopwatch) Start()
  • func (s *Stopwatch) Stop()

Variables

var ErrStopped = fmt.Errorf("stopped at caller request")

Types

type Node

type Node struct {
	ID     t.NodeID    // Protocol-level node ID
	Config *NodeConfig // Node-level (protocol-independent) configuration, like buffer sizes, logging, ...
	// contains filtered or unexported fields
}

Node is the local instance of Mir and the application's interface to the mir library.

func NewNode

func NewNode(
	id t.NodeID,
	config *NodeConfig,
	m modules.Modules,
	interceptor eventlog.Interceptor,
) (*Node, error)

NewNode creates a new node with ID id. The config parameter specifies Node-level (protocol-independent) configuration, like buffer sizes, logging, ... The modules parameter must contain initialized, ready-to-use modules that the new Node will use.

func (*Node) Debug

func (n *Node) Debug(ctx context.Context, eventsOut chan *events.EventList) error

Debug runs the Node in debug mode. The Node will ony process events submitted through the Step method. All internally generated events will be ignored and, if the eventsOut argument is not nil, written to eventsOut instead. Note that if the caller supplies such a channel, the caller is expected to read from it. Otherwise, the Node's execution might block while writing to the channel.

func (*Node) InjectEvents

func (n *Node) InjectEvents(ctx context.Context, events *events.EventList) error

InjectEvents inserts a list of Events in the Node.

func (*Node) Run

func (n *Node) Run(ctx context.Context) error

Run starts the Node. It first adds an Init event to the work items, giving the modules the possibility to perform any initialization. Run then launches the processing of incoming messages, and internal events. The node stops when the ctx is canceled. The function call is blocking and only returns when the node stops.

func (*Node) Stop

func (n *Node) Stop()

Stop stops the Node and returns only after the node has stopped, i.e., after a call to Run or Debug returns. If neither Run nor Debug has been called, Stop blocks until either Run or Debug is called by another goroutine and returns.

type NodeConfig

type NodeConfig struct {
	// Logger provides the logging functions.
	Logger logging.Logger

	// MaxEventBatchSize is the maximum number of events that can be dispatched to a module in a single batch.
	MaxEventBatchSize int

	// PauseInputThreshold is the number of events in the node's internal event buffer that triggers the disabling
	// of further external input (i.e. events emitted by active modules). Events emitted by passive modules are
	// not affected. The processing of external events is resumed when the number of events in the buffer drops
	// below the ResumeInputThreshold.
	PauseInputThreshold int

	// ResumeInputThreshold is the number of events in the node's internal event buffer that triggers the enabling
	// of external input (i.e. events emitted by active modules). Events emitted by passive modules are not affected.
	// When the external input is disabled and the number of events in the buffer drops below ResumeInputThreshold,
	// external input events can be added to the event buffer again.
	ResumeInputThreshold int

	// If not zero, the Node will emit a log entry (level Debug) every StatsLogInterval
	// containing statistics about event processing.
	StatsLogInterval time.Duration
}

The NodeConfig struct represents configuration parameters of the node that are independent of the protocol the Node is executing. NodeConfig only contains protocol-independent parameters. Protocol-specific parameters should be specified when instantiating the protocol implementation as one of the Node's modules.

func DefaultNodeConfig

func DefaultNodeConfig() *NodeConfig

DefaultNodeConfig returns the default node configuration. It can be used as a base for creating more specific configurations when instantiating a Node.

func (*NodeConfig) Validate

func (c *NodeConfig) Validate() error

func (*NodeConfig) WithLogger

func (c *NodeConfig) WithLogger(logger logging.Logger) *NodeConfig

type Stopwatch

type Stopwatch struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func (*Stopwatch) Read

func (s *Stopwatch) Read() time.Duration

func (*Stopwatch) Reset

func (s *Stopwatch) Reset() time.Duration

func (*Stopwatch) Start

func (s *Stopwatch) Start()

func (*Stopwatch) Stop

func (s *Stopwatch) Stop()