network

README

Avalanche Networking

Table of Contents

• Overview
• Peers
  • Lifecycle
    • Bootstrapping
    • Connecting
      • Peer Handshake
    • Connected
      • PeerList Gossip
        • Messages
        • Gossip

Overview

Avalanche is a decentralized p2p (peer-to-peer) network of nodes that work together to run the Avalanche blockchain protocol.

The network package implements the networking layer of the protocol which allows a node to discover, connect to, and communicate with other peers.

Peers

Peers are defined as members of the network that communicate with one another to participate in the Avalanche protocol.

Peers communicate by enqueuing messages between one another. Each peer on either side of the connection asynchronously reads and writes messages to and from the remote peer. Messages include both application-level messages used to support the Avalanche protocol, as well as networking-level messages used to implement the peer-to-peer communication layer.

sequenceDiagram
    loop 
        Peer-1->>Peer-2: Write outbound messages
        Peer-2->>Peer-1: Read incoming messages
    end
    loop
        Peer-2->>Peer-1: Write outbound messages
        Peer-1->>Peer-2: Read incoming messages
    end

Lifecycle

Bootstrapping

When starting an Avalanche node, a node needs to be able to initiate some process that eventually allows itself to become a participating member of the network. In traditional web2 systems, it's common to use a web service by hitting the service's DNS and being routed to an available server behind a load balancer. In decentralized p2p systems however, connecting to a node is more complex as no single entity owns the network. Avalanche consensus requires a node to repeatedly sample peers in the network, so each node needs some way of discovering and connecting to every other peer to participate in the protocol.

In Avalanche, nodes connect to an initial set of bootstrapper nodes known as beacons (this is user-configurable). Once connected to a set of beacons, a node is able to discover other nodes in the network. Over time, a node eventually discovers other peers in the network through PeerList messages it receives through:

• The handshake initiated between two peers when attempting to connect to a peer (see Connecting).
• Periodic PeerList gossip messages that every peer sends to the peers it's connected to (see Connected).

Connecting

Peer Handshake

Upon connection to any peer, a handshake is performed between the node attempting to establish the outbound connection to the peer and the peer receiving the inbound connection.

When attempting to establish the connection, the first message that the node attempting to connect to the peer in the network is a Version message describing compatibility of the candidate node with the peer. As an example, nodes that are attempting to connect with an incompatible version of AvalancheGo or a significantly skewed local clock are rejected by the peer.

sequenceDiagram
Note over Node,Peer: Initiate Handshake
Note left of Node: I want to connect to you!
Note over Node,Peer: Version message
Node->>Peer: AvalancheGo v1.0.0
Note right of Peer: My version v1.9.4 is incompatible with your version v1.0.0.
Peer-xNode: Connection dropped
Note over Node,Peer: Handshake Failed

If the Version message is successfully received and the peer decides that it wants a connection with this node, it replies with a PeerList message that contains metadata about other peers that allows a node to connect to them. Upon reception of a PeerList message, a node will attempt to connect to any peers that the node is not already connected to to allow the node to discover more peers in the network.

sequenceDiagram
Note over Node,Peer: Initiate Handshake
Note left of Node: I want to connect to you!
Note over Node,Peer: Version message
Node->>Peer: AvalancheGo v1.9.4
Note right of Peer: LGTM!
Note over Node,Peer: PeerList message
Peer->>Node: Peer-X, Peer-Y, Peer-Z
Note over Node,Peer: Handshake Complete
Node->>Peer: ACK Peer-X, Peer-Y, Peer-Z

Once the node attempting to join the network receives this PeerList message, the handshake is complete and the node is now connected to the peer. The node attempts to connect to the new peers discovered in the PeerList message. Each connection results in another peer handshake, which results in the node incrementally discovering more and more peers in the network as more and more PeerList messages are exchanged.

Connected

Some peers aren't discovered through the PeerList messages exchanged through peer handshakes. This can happen if a peer is either not randomly sampled, or if a new peer joins the network after the node has already connected to the network.

sequenceDiagram
Node ->> Peer-1: Version - v1.9.5
Peer-1 ->> Node: PeerList - Peer-2
Node ->> Peer-1: ACK - Peer-2
Note left of Node: Node is connected to Peer-1 and now tries to connect to Peer-2.
Node ->> Peer-2: Version - v1.9.5
Peer-2 ->> Node: PeerList - Peer-1
Node ->> Peer-2: ACK - Peer-1
Note left of Node: Peer-3 was never sampled, so we haven't connected yet!
Node --> Peer-3: No connection

To guarantee that a node can discover all peers, each node periodically gossips a sample of the peers it knows about to other peers.

PeerList Gossip

Messages

A PeerList is the message that is used to communicate the presence of peers in the network. Each PeerList message contains networking-level metadata about the peer that provides the necessary information to connect to it, alongside the corresponding transaction id that added that peer to the validator set. Transaction ids are unique hashes that only add a single validator, so it is guaranteed that there is a 1:1 mapping between a validator and its associated transaction id.

PeerListAck messages are sent in response to PeerList messages to allow a peer to confirm which peers it will actually attempt to connect to. Because nodes only gossip peers they believe another peer doesn't already know about to optimize bandwidth, PeerListAck messages are important to confirm that a peer will attempt to connect to someone. Without this, a node might gossip a peer to another peer and assume a connection between the two is being established, and not re-gossip the peer in future gossip cycles. If the connection was never actually wanted by the peer being gossiped to due to a transient reason, that peer would never be able to re-discover the gossiped peer and could be isolated from a subset of the network.

Once a PeerListAck message is received from a peer, the node that sent the original PeerList message marks the corresponding acknowledged validators as already having been transmitted to the peer, so that it's excluded from subsequent iterations of PeerList gossip.

Gossip

Handshake messages provide a node with some knowledge of peers in the network, but offers no guarantee that learning about a subset of peers from each peer the node connects with will result in the node learning about every peer in the network.

In order to provide a probabilistic guarantee that all peers in the network will eventually learn of one another, each node periodically gossips a sample of the peers that they're aware of to a sample of the peers that they're connected to. Over time, this probabilistically guarantees that every peer will eventually learn of every other peer.

To optimize bandwidth usage, each node tracks which peers are guaranteed to know of which peers. A node learns this information by tracking both inbound and outbound PeerList gossip.

• Inbound
  • If a node ever receives PeerList from a peer, that peer must have known about the peers in that PeerList message in order to have gossiped them.
• Outbound
  • If a node sends a PeerList to a peer and the peer replies with an PeerListAck message, then all peers in the PeerListAck must be known by the peer.

To efficiently track which peers know of which peers, the peers that each peer is aware of is represented in a bit set. A peer is represented by either a 0 if it isn't known by the peer yet, or a 1 if it is known by the peer.

An node follows the following steps for every cycle of PeerList gossip:

1. Get a sample of peers in the network that the node is connected to
2. For each peer:
   1. Figure out which peers the node hasn't gossiped to them yet.
   2. Take a random sample of these unknown peers.
   3. Send a message describing these peers to the peer.

sequenceDiagram
Note left of Node: Initialize gossip bit set for Peer-123
Note left of Node: Peer-123: [0, 0, 0]
Node->>Peer-123: PeerList - Peer-1
Peer-123->>Node: PeerListAck - Peer-1
Note left of Node: Peer-123: [1, 0, 0]
Node->>Peer-123: PeerList - Peer-3
Peer-123->>Node: PeerListAck - Peer-3
Note left of Node: Peer-123: [1, 0, 1]
Node->>Peer-123: PeerList - Peer-2
Peer-123->>Node: PeerListAck - Peer-2
Note left of Node: Peer-123: [1, 1, 1]
Note left of Node: No more gossip left to send to Peer-123!

Because network state is generally expected to be stable (i.e nodes are not continuously flickering online/offline), as more and more gossip messages are exchanged nodes eventually realize that the peers that they are connected to have learned about every other peer.

A node eventually stops gossiping peers when there's no more new peers to gossip about. PeerList gossip only resumes once:

1. a new peer joins
2. a peer disconnects and reconnects
3. a new validator joins the network
4. a validator's IP is updated

Documentation

Index

• Constants
• type Config
• type DelayConfig
• type HealthConfig
• type Network
  • func NewNetwork(config *Config, msgCreator message.Creator, ...) (Network, error)
  • func NewTestNetwork(log logging.Logger, networkID uint32, currentValidators validators.Set, ...) (Network, error)
• type PeerListGossipConfig
• type ThrottlerConfig
• type TimeoutConfig
• type UptimeResult

Examples

• NewTestNetwork

Constants

const (
	ConnectedPeersKey           = "connectedPeers"
	TimeSinceLastMsgReceivedKey = "timeSinceLastMsgReceived"
	TimeSinceLastMsgSentKey     = "timeSinceLastMsgSent"
	SendFailRateKey             = "sendFailRate"
)

Types

type Config

type Config struct {
	HealthConfig         `json:"healthConfig"`
	PeerListGossipConfig `json:"peerListGossipConfig"`
	TimeoutConfig        `json:"timeoutConfigs"`
	DelayConfig          `json:"delayConfig"`
	ThrottlerConfig      ThrottlerConfig `json:"throttlerConfig"`

	ProxyEnabled           bool          `json:"proxyEnabled"`
	ProxyReadHeaderTimeout time.Duration `json:"proxyReadHeaderTimeout"`

	DialerConfig dialer.Config `json:"dialerConfig"`
	TLSConfig    *tls.Config   `json:"-"`

	TLSKeyLogFile string `json:"tlsKeyLogFile"`

	Namespace          string            `json:"namespace"`
	MyNodeID           ids.NodeID        `json:"myNodeID"`
	MyIPPort           ips.DynamicIPPort `json:"myIP"`
	NetworkID          uint32            `json:"networkID"`
	MaxClockDifference time.Duration     `json:"maxClockDifference"`
	PingFrequency      time.Duration     `json:"pingFrequency"`
	AllowPrivateIPs    bool              `json:"allowPrivateIPs"`

	// The compression type to use when compressing outbound messages.
	// Assumes all peers support this compression type.
	CompressionType compression.Type `json:"compressionType"`

	// TLSKey is this node's TLS key that is used to sign IPs.
	TLSKey crypto.Signer `json:"-"`

	// TrackedSubnets of the node.
	TrackedSubnets set.Set[ids.ID] `json:"-"`
	Beacons        validators.Set  `json:"-"`

	// Validators are the current validators in the Avalanche network
	Validators validators.Manager `json:"-"`

	UptimeCalculator uptime.Calculator `json:"-"`

	// UptimeMetricFreq marks how frequently this node will recalculate the
	// observed average uptime metrics.
	UptimeMetricFreq time.Duration `json:"uptimeMetricFreq"`

	// UptimeRequirement is the fraction of time a validator must be online and
	// responsive for us to vote that they should receive a staking reward.
	UptimeRequirement float64 `json:"-"`

	// RequireValidatorToConnect require that all connections must have at least
	// one validator between the 2 peers. This can be useful to enable if the
	// node wants to connect to the minimum number of nodes without impacting
	// the network negatively.
	RequireValidatorToConnect bool `json:"requireValidatorToConnect"`

	// MaximumInboundMessageTimeout is the maximum deadline duration in a
	// message. Messages sent by clients setting values higher than this value
	// will be reset to this value.
	MaximumInboundMessageTimeout time.Duration `json:"maximumInboundMessageTimeout"`

	// Size, in bytes, of the buffer that we read peer messages into
	// (there is one buffer per peer)
	PeerReadBufferSize int `json:"peerReadBufferSize"`

	// Size, in bytes, of the buffer that we write peer messages into
	// (there is one buffer per peer)
	PeerWriteBufferSize int `json:"peerWriteBufferSize"`

	// Tracks the CPU/disk usage caused by processing messages of each peer.
	ResourceTracker tracker.ResourceTracker `json:"-"`

	// Specifies how much CPU usage each peer can cause before
	// we rate-limit them.
	CPUTargeter tracker.Targeter `json:"-"`

	// Specifies how much disk usage each peer can cause before
	// we rate-limit them.
	DiskTargeter tracker.Targeter `json:"-"`

	// Tracks which validators have been sent to which peers
	GossipTracker peer.GossipTracker `json:"-"`
}

type DelayConfig

type DelayConfig struct {
	// InitialReconnectDelay is the minimum amount of time the node will delay a
	// reconnection to a peer. This value is used to start the exponential
	// backoff.
	InitialReconnectDelay time.Duration `json:"initialReconnectDelay"`

	// MaxReconnectDelay is the maximum amount of time the node will delay a
	// reconnection to a peer.
	MaxReconnectDelay time.Duration `json:"maxReconnectDelay"`
}

type HealthConfig

type HealthConfig struct {
	// Marks if the health check should be enabled
	Enabled bool `json:"-"`

	// MinConnectedPeers is the minimum number of peers that the network should
	// be connected to be considered healthy.
	MinConnectedPeers uint `json:"minConnectedPeers"`

	// MaxTimeSinceMsgReceived is the maximum amount of time since the network
	// last received a message to be considered healthy.
	MaxTimeSinceMsgReceived time.Duration `json:"maxTimeSinceMsgReceived"`

	// MaxTimeSinceMsgSent is the maximum amount of time since the network last
	// sent a message to be considered healthy.
	MaxTimeSinceMsgSent time.Duration `json:"maxTimeSinceMsgSent"`

	// MaxPortionSendQueueBytesFull is the maximum percentage of the pending
	// send byte queue that should be used for the network to be considered
	// healthy. Should be in (0,1].
	MaxPortionSendQueueBytesFull float64 `json:"maxPortionSendQueueBytesFull"`

	// MaxSendFailRate is the maximum percentage of send attempts that should be
	// failing for the network to be considered healthy. This does not include
	// send attempts that were not made due to benching. Should be in [0,1].
	MaxSendFailRate float64 `json:"maxSendFailRate"`

	// SendFailRateHalflife is the halflife of the averager used to calculate
	// the send fail rate percentage. Should be > 0. Larger values mean that the
	// fail rate is affected less by recently dropped messages.
	SendFailRateHalflife time.Duration `json:"sendFailRateHalflife"`
}

HealthConfig describes parameters for network layer health checks.

type Network

type Network interface {
	// All consensus messages can be sent through this interface. Thread safety
	// must be managed internally in the network.
	sender.ExternalSender

	// Has a health check
	health.Checker

	peer.Network

	// StartClose this network and all existing connections it has. Calling
	// StartClose multiple times is handled gracefully.
	StartClose()

	// Should only be called once, will run until either a fatal error occurs,
	// or the network is closed.
	Dispatch() error

	// WantsConnection returns true if this node is willing to attempt to
	// connect to the provided nodeID. If the node is attempting to connect to
	// the minimum number of peers, then it should only connect if the peer is a
	// validator or beacon.
	WantsConnection(ids.NodeID) bool

	// Attempt to connect to this IP. The network will never stop attempting to
	// connect to this ID.
	ManuallyTrack(nodeID ids.NodeID, ip ips.IPPort)

	// PeerInfo returns information about peers. If [nodeIDs] is empty, returns
	// info about all peers that have finished the handshake. Otherwise, returns
	// info about the peers in [nodeIDs] that have finished the handshake.
	PeerInfo(nodeIDs []ids.NodeID) []peer.Info

	// NodeUptime returns given node's [subnetID] UptimeResults in the view of
	// this node's peer validators.
	NodeUptime(subnetID ids.ID) (UptimeResult, error)
}

Network defines the functionality of the networking library.

func NewNetwork

func NewNetwork(
	config *Config,
	msgCreator message.Creator,
	metricsRegisterer prometheus.Registerer,
	log logging.Logger,
	listener net.Listener,
	dialer dialer.Dialer,
	router router.ExternalHandler,
) (Network, error)

NewNetwork returns a new Network implementation with the provided parameters.

func NewTestNetwork

func NewTestNetwork(
	log logging.Logger,
	networkID uint32,
	currentValidators validators.Set,
	trackedSubnets set.Set[ids.ID],
	router router.ExternalHandler,
) (Network, error)

Example

package main

import (
	"context"
	"os"
	"time"

	"go.uber.org/zap"

	"github.com/ava-labs/avalanchego/genesis"
	"github.com/ava-labs/avalanchego/ids"
	"github.com/ava-labs/avalanchego/message"
	"github.com/ava-labs/avalanchego/snow/networking/router"
	"github.com/ava-labs/avalanchego/snow/validators"
	"github.com/ava-labs/avalanchego/utils/constants"
	"github.com/ava-labs/avalanchego/utils/ips"
	"github.com/ava-labs/avalanchego/utils/logging"
	"github.com/ava-labs/avalanchego/utils/set"
	"github.com/ava-labs/avalanchego/version"
)

var _ router.ExternalHandler = (*testExternalHandler)(nil)

// Note: all of the external handler's methods are called on peer goroutines. It
// is possible for multiple concurrent calls to happen with different NodeIDs.
// However, a given NodeID will only be performing one call at a time.
type testExternalHandler struct {
	log logging.Logger
}

// Note: HandleInbound will be called with raw P2P messages, the networking
// implementation does not implicitly register timeouts, so this handler is only
// called by messages explicitly sent by the peer. If timeouts are required,
// that must be handled by the user of this utility.
func (t *testExternalHandler) HandleInbound(_ context.Context, message message.InboundMessage) {
	t.log.Info(
		"receiving message",
		zap.Stringer("op", message.Op()),
	)
}

func (t *testExternalHandler) Connected(nodeID ids.NodeID, version *version.Application, subnetID ids.ID) {
	t.log.Info(
		"connected",
		zap.Stringer("nodeID", nodeID),
		zap.Stringer("version", version),
		zap.Stringer("subnetID", subnetID),
	)
}

func (t *testExternalHandler) Disconnected(nodeID ids.NodeID) {
	t.log.Info(
		"disconnected",
		zap.Stringer("nodeID", nodeID),
	)
}

type testAggressiveValidatorSet struct {
	validators.Set
}

func (*testAggressiveValidatorSet) Contains(ids.NodeID) bool {
	return true
}

func main() {
	log := logging.NewLogger(
		"networking",
		logging.NewWrappedCore(
			logging.Info,
			os.Stdout,
			logging.Colors.ConsoleEncoder(),
		),
	)

	// Needs to be periodically updated by the caller to have the latest
	// validator set
	validators := &testAggressiveValidatorSet{
		Set: validators.NewSet(),
	}

	// If we want to be able to communicate with non-primary network subnets, we
	// should register them here.
	trackedSubnets := set.Set[ids.ID]{}

	// Messages and connections are handled by the external handler.
	handler := &testExternalHandler{
		log: log,
	}

	network, err := NewTestNetwork(
		log,
		constants.FujiID,
		validators,
		trackedSubnets,
		handler,
	)
	if err != nil {
		log.Fatal(
			"failed to create test network",
			zap.Error(err),
		)
		return
	}

	// We need to initially connect to some nodes in the network before peer
	// gossip will enable connecting to all the remaining nodes in the network.
	bootstrappers := genesis.SampleBootstrappers(constants.FujiID, 5)
	for _, bootstrapper := range bootstrappers {
		network.ManuallyTrack(bootstrapper.ID, ips.IPPort(bootstrapper.IP))
	}

	// Typically network.StartClose() should be called based on receiving a
	// SIGINT or SIGTERM. For the example, we close the network after 15s.
	go log.RecoverAndPanic(func() {
		time.Sleep(15 * time.Second)
		network.StartClose()
	})

	// network.Send(...) and network.Gossip(...) can be used here to send
	// messages to peers.

	// Calling network.Dispatch() will block until a fatal error occurs or
	// network.StartClose() is called.
	err = network.Dispatch()
	log.Info(
		"network exited",
		zap.Error(err),
	)
}

type PeerListGossipConfig

type PeerListGossipConfig struct {
	// PeerListNumValidatorIPs is the number of validator IPs to gossip in every
	// gossip event.
	PeerListNumValidatorIPs uint32 `json:"peerListNumValidatorIPs"`

	// PeerListValidatorGossipSize is the number of validators to gossip the IPs
	// to in every IP gossip event.
	PeerListValidatorGossipSize uint32 `json:"peerListValidatorGossipSize"`

	// PeerListNonValidatorGossipSize is the number of non-validators to gossip
	// the IPs to in every IP gossip event.
	PeerListNonValidatorGossipSize uint32 `json:"peerListNonValidatorGossipSize"`

	// PeerListPeersGossipSize is the number of peers to gossip
	// the IPs to in every IP gossip event.
	PeerListPeersGossipSize uint32 `json:"peerListPeersGossipSize"`

	// PeerListGossipFreq is the frequency that this node will attempt to gossip
	// signed IPs to its peers.
	PeerListGossipFreq time.Duration `json:"peerListGossipFreq"`
}

type ThrottlerConfig

type ThrottlerConfig struct {
	InboundConnUpgradeThrottlerConfig throttling.InboundConnUpgradeThrottlerConfig `json:"inboundConnUpgradeThrottlerConfig"`
	InboundMsgThrottlerConfig         throttling.InboundMsgThrottlerConfig         `json:"inboundMsgThrottlerConfig"`
	OutboundMsgThrottlerConfig        throttling.MsgByteThrottlerConfig            `json:"outboundMsgThrottlerConfig"`
	MaxInboundConnsPerSec             float64                                      `json:"maxInboundConnsPerSec"`
}

type TimeoutConfig

type TimeoutConfig struct {
	// PingPongTimeout is the maximum amount of time to wait for a Pong response
	// from a peer we sent a Ping to.
	PingPongTimeout time.Duration `json:"pingPongTimeout"`

	// ReadHandshakeTimeout is the maximum amount of time to wait for the peer's
	// connection upgrade to finish before starting the p2p handshake.
	ReadHandshakeTimeout time.Duration `json:"readHandshakeTimeout"`
}

type UptimeResult

type UptimeResult struct {
	// RewardingStakePercentage shows what percent of network stake thinks we're
	// above the uptime requirement.
	RewardingStakePercentage float64

	// WeightedAveragePercentage is the average perceived uptime of this node,
	// weighted by stake.
	// Note that this is different from RewardingStakePercentage, which shows
	// the percent of the network stake that thinks this node is above the
	// uptime requirement. WeightedAveragePercentage is weighted by uptime.
	// i.e If uptime requirement is 85 and a peer reports 40 percent it will be
	// counted (40*weight) in WeightedAveragePercentage but not in
	// RewardingStakePercentage since 40 < 85
	WeightedAveragePercentage float64
}