network

README

Avalanche Networking

Table of Contents

• Overview
• Peers
  • Lifecycle
    • Bootstrapping
    • Connecting
      • Peer Handshake
    • Connected
      • PeerList Gossip
        • Messages
        • Gossip

Overview

Avalanche is a decentralized p2p (peer-to-peer) network of nodes that work together to run the Avalanche blockchain protocol.

The network package implements the networking layer of the protocol which allows a node to discover, connect to, and communicate with other peers.

Peers

Peers are defined as members of the network that communicate with one another to participate in the Avalanche protocol.

Peers communicate by enqueuing messages between one another. Each peer on either side of the connection asynchronously reads and writes messages to and from the remote peer. Messages include both application-level messages used to support the Avalanche protocol, as well as networking-level messages used to implement the peer-to-peer communication layer.

sequenceDiagram
    loop 
        Peer-1->>Peer-2: Write outbound messages
        Peer-2->>Peer-1: Read incoming messages
    end
    loop
        Peer-2->>Peer-1: Write outbound messages
        Peer-1->>Peer-2: Read incoming messages
    end

Lifecycle

Bootstrapping

When starting an Avalanche node, a node needs to be able to initiate some process that eventually allows itself to become a participating member of the network. In traditional web2 systems, it's common to use a web service by hitting the service's DNS and being routed to an available server behind a load balancer. In decentralized p2p systems however, connecting to a node is more complex as no single entity owns the network. Avalanche consensus requires a node to repeatedly sample peers in the network, so each node needs some way of discovering and connecting to every other peer to participate in the protocol.

In Avalanche, nodes connect to an initial set of bootstrapper nodes known as beacons (this is user-configurable). Once connected to a set of beacons, a node is able to discover other nodes in the network. Over time, a node eventually discovers other peers in the network through PeerList messages it receives through:

• The handshake initiated between two peers when attempting to connect to a peer (see Connecting).
• Responses to periodically sent GetPeerList messages requesting a PeerList of unknown peers (see Connected).

Connecting

Peer Handshake

Upon connection to any peer, a handshake is performed between the node attempting to establish the outbound connection to the peer and the peer receiving the inbound connection.

When attempting to establish the connection, the first message that the node attempting to connect to the peer in the network is a Handshake message describing compatibility of the candidate node with the peer. As an example, nodes that are attempting to connect with an incompatible version of AvalancheGo or a significantly skewed local clock are rejected by the peer.

sequenceDiagram
Note over Node,Peer: Initiate Handshake
Note left of Node: I want to connect to you!
Note over Node,Peer: Handshake message
Node->>Peer: AvalancheGo v1.0.0
Note right of Peer: My version v1.9.4 is incompatible with your version v1.0.0.
Peer-xNode: Connection dropped
Note over Node,Peer: Handshake Failed

If the Handshake message is successfully received and the peer decides that it wants a connection with this node, it replies with a PeerList message that contains metadata about other peers that allows a node to connect to them. Upon reception of a PeerList message, a node will attempt to connect to any peers that the node is not already connected to to allow the node to discover more peers in the network.

sequenceDiagram
Note over Node,Peer: Initiate Handshake
Note left of Node: I want to connect to you!
Note over Node,Peer: Handshake message
Node->>Peer: AvalancheGo v1.9.4
Note right of Peer: LGTM!
Note over Node,Peer: PeerList message
Peer->>Node: Peer-X, Peer-Y, Peer-Z
Note over Node,Peer: Handshake Complete

Once the node attempting to join the network receives this PeerList message, the handshake is complete and the node is now connected to the peer. The node attempts to connect to the new peers discovered in the PeerList message. Each connection results in another peer handshake, which results in the node incrementally discovering more and more peers in the network as more and more PeerList messages are exchanged.

Connected

Some peers aren't discovered through the PeerList messages exchanged through peer handshakes. This can happen if a peer is either not randomly sampled, or if a new peer joins the network after the node has already connected to the network.

sequenceDiagram
Node ->> Peer-1: Handshake - v1.9.5
Peer-1 ->> Node: PeerList - Peer-2
Note left of Node: Node is connected to Peer-1 and now tries to connect to Peer-2.
Node ->> Peer-2: Handshake - v1.9.5
Peer-2 ->> Node: PeerList - Peer-1
Note left of Node: Peer-3 was never sampled, so we haven't connected yet!
Node --> Peer-3: No connection

To guarantee that a node can discover all peers, each node periodically sends a GetPeerList message to a random peer.

PeerList Gossip

Messages

A GetPeerList message requests that the peer sends a PeerList message. GetPeerList messages contain a bloom filter of already known peers to reduce useless bandwidth on PeerList messages. The bloom filter reduces bandwidth by enabling the PeerList message to only include peers that aren't already known.

A PeerList is the message that is used to communicate the presence of peers in the network. Each PeerList message contains signed networking-level metadata about a peer that provides the necessary information to connect to it.

Once peer metadata is received, the node will add that data to its bloom filter to prevent learning about it again.

Gossip

Handshake messages provide a node with some knowledge of peers in the network, but offers no guarantee that learning about a subset of peers from each peer the node connects with will result in the node learning about every peer in the network.

To provide an eventual guarantee that all peers learn of one another, each node periodically requests peers from a random peer.

To optimize bandwidth, each node tracks the most recent IPs of validators. The validator's nodeID and timestamp are inserted into a bloom filter which is used to select only necessary IPs to gossip.

As the number of entries increases in the bloom filter, the probability of a false positive increases. False positives can cause recent IPs not to be gossiped when they otherwise should be, slowing down the rate of PeerList gossip. To prevent the bloom filter from having too many false positives, a new bloom filter is periodically generated and the number of entries a validator is allowed to have in the bloom filter is capped. Generating the new bloom filter both removes stale entries and modifies the hash functions to avoid persistent hash collisions.

A node follows the following steps for of PeerList gossip:

sequenceDiagram
Note left of Node: Initialize bloom filter
Note left of Node: Bloom: [0, 0, 0]
Node->>Peer-123: GetPeerList [0, 0, 0]
Note right of Peer-123: Any peers can be sent.
Peer-123->>Node: PeerList - Peer-1
Note left of Node: Bloom: [1, 0, 0]
Node->>Peer-123: GetPeerList [1, 0, 0]
Note right of Peer-123: Either Peer-2 or Peer-3 can be sent.
Peer-123->>Node: PeerList - Peer-3
Note left of Node: Bloom: [1, 0, 1]
Node->>Peer-123: GetPeerList [1, 0, 1]
Note right of Peer-123: Only Peer-2 can be sent.
Peer-123->>Node: PeerList - Peer-2
Note left of Node: Bloom: [1, 1, 1]
Node->>Peer-123: GetPeerList [1, 1, 1]
Note right of Peer-123: There are no more peers left to send!

Documentation

Index

• Constants
• type Config
• type DelayConfig
• type HealthConfig
• type Network
  • func NewNetwork(config *Config, msgCreator message.Creator, ...) (Network, error)
  • func NewTestNetwork(log logging.Logger, networkID uint32, currentValidators validators.Manager, ...) (Network, error)
• type PeerListGossipConfig
• type ThrottlerConfig
• type TimeoutConfig
• type UptimeResult

Examples

• NewTestNetwork

Constants

const (
	ConnectedPeersKey           = "connectedPeers"
	TimeSinceLastMsgReceivedKey = "timeSinceLastMsgReceived"
	TimeSinceLastMsgSentKey     = "timeSinceLastMsgSent"
	SendFailRateKey             = "sendFailRate"
)

Types

type Config

type Config struct {
	HealthConfig         `json:"healthConfig"`
	PeerListGossipConfig `json:"peerListGossipConfig"`
	TimeoutConfig        `json:"timeoutConfigs"`
	DelayConfig          `json:"delayConfig"`
	ThrottlerConfig      ThrottlerConfig `json:"throttlerConfig"`

	ProxyEnabled           bool          `json:"proxyEnabled"`
	ProxyReadHeaderTimeout time.Duration `json:"proxyReadHeaderTimeout"`

	DialerConfig dialer.Config `json:"dialerConfig"`
	TLSConfig    *tls.Config   `json:"-"`

	TLSKeyLogFile string `json:"tlsKeyLogFile"`

	Namespace          string            `json:"namespace"`
	MyNodeID           ids.NodeID        `json:"myNodeID"`
	MyIPPort           ips.DynamicIPPort `json:"myIP"`
	NetworkID          uint32            `json:"networkID"`
	MaxClockDifference time.Duration     `json:"maxClockDifference"`
	PingFrequency      time.Duration     `json:"pingFrequency"`
	AllowPrivateIPs    bool              `json:"allowPrivateIPs"`

	SupportedACPs set.Set[uint32] `json:"supportedACPs"`
	ObjectedACPs  set.Set[uint32] `json:"objectedACPs"`

	// The compression type to use when compressing outbound messages.
	// Assumes all peers support this compression type.
	CompressionType compression.Type `json:"compressionType"`

	// TLSKey is this node's TLS key that is used to sign IPs.
	TLSKey crypto.Signer `json:"-"`

	// TrackedSubnets of the node.
	TrackedSubnets set.Set[ids.ID]    `json:"-"`
	Beacons        validators.Manager `json:"-"`

	// Validators are the current validators in the Avalanche network
	Validators validators.Manager `json:"-"`

	UptimeCalculator uptime.Calculator `json:"-"`

	// UptimeMetricFreq marks how frequently this node will recalculate the
	// observed average uptime metrics.
	UptimeMetricFreq time.Duration `json:"uptimeMetricFreq"`

	// UptimeRequirement is the fraction of time a validator must be online and
	// responsive for us to vote that they should receive a staking reward.
	UptimeRequirement float64 `json:"-"`

	// RequireValidatorToConnect require that all connections must have at least
	// one validator between the 2 peers. This can be useful to enable if the
	// node wants to connect to the minimum number of nodes without impacting
	// the network negatively.
	RequireValidatorToConnect bool `json:"requireValidatorToConnect"`

	// MaximumInboundMessageTimeout is the maximum deadline duration in a
	// message. Messages sent by clients setting values higher than this value
	// will be reset to this value.
	MaximumInboundMessageTimeout time.Duration `json:"maximumInboundMessageTimeout"`

	// Size, in bytes, of the buffer that we read peer messages into
	// (there is one buffer per peer)
	PeerReadBufferSize int `json:"peerReadBufferSize"`

	// Size, in bytes, of the buffer that we write peer messages into
	// (there is one buffer per peer)
	PeerWriteBufferSize int `json:"peerWriteBufferSize"`

	// Tracks the CPU/disk usage caused by processing messages of each peer.
	ResourceTracker tracker.ResourceTracker `json:"-"`

	// Specifies how much CPU usage each peer can cause before
	// we rate-limit them.
	CPUTargeter tracker.Targeter `json:"-"`

	// Specifies how much disk usage each peer can cause before
	// we rate-limit them.
	DiskTargeter tracker.Targeter `json:"-"`
}

type DelayConfig

type DelayConfig struct {
	// InitialReconnectDelay is the minimum amount of time the node will delay a
	// reconnection to a peer. This value is used to start the exponential
	// backoff.
	InitialReconnectDelay time.Duration `json:"initialReconnectDelay"`

	// MaxReconnectDelay is the maximum amount of time the node will delay a
	// reconnection to a peer.
	MaxReconnectDelay time.Duration `json:"maxReconnectDelay"`
}

type HealthConfig

type HealthConfig struct {
	// Marks if the health check should be enabled
	Enabled bool `json:"-"`

	// MinConnectedPeers is the minimum number of peers that the network should
	// be connected to be considered healthy.
	MinConnectedPeers uint `json:"minConnectedPeers"`

	// MaxTimeSinceMsgReceived is the maximum amount of time since the network
	// last received a message to be considered healthy.
	MaxTimeSinceMsgReceived time.Duration `json:"maxTimeSinceMsgReceived"`

	// MaxTimeSinceMsgSent is the maximum amount of time since the network last
	// sent a message to be considered healthy.
	MaxTimeSinceMsgSent time.Duration `json:"maxTimeSinceMsgSent"`

	// MaxPortionSendQueueBytesFull is the maximum percentage of the pending
	// send byte queue that should be used for the network to be considered
	// healthy. Should be in (0,1].
	MaxPortionSendQueueBytesFull float64 `json:"maxPortionSendQueueBytesFull"`

	// MaxSendFailRate is the maximum percentage of send attempts that should be
	// failing for the network to be considered healthy. This does not include
	// send attempts that were not made due to benching. Should be in [0,1].
	MaxSendFailRate float64 `json:"maxSendFailRate"`

	// SendFailRateHalflife is the halflife of the averager used to calculate
	// the send fail rate percentage. Should be > 0. Larger values mean that the
	// fail rate is affected less by recently dropped messages.
	SendFailRateHalflife time.Duration `json:"sendFailRateHalflife"`
}

HealthConfig describes parameters for network layer health checks.

type Network

type Network interface {
	// All consensus messages can be sent through this interface. Thread safety
	// must be managed internally in the network.
	sender.ExternalSender

	// Has a health check
	health.Checker

	peer.Network

	// StartClose this network and all existing connections it has. Calling
	// StartClose multiple times is handled gracefully.
	StartClose()

	// Should only be called once, will run until either a fatal error occurs,
	// or the network is closed.
	Dispatch() error

	// Attempt to connect to this IP. The network will never stop attempting to
	// connect to this ID.
	ManuallyTrack(nodeID ids.NodeID, ip ips.IPPort)

	// PeerInfo returns information about peers. If [nodeIDs] is empty, returns
	// info about all peers that have finished the handshake. Otherwise, returns
	// info about the peers in [nodeIDs] that have finished the handshake.
	PeerInfo(nodeIDs []ids.NodeID) []peer.Info

	// NodeUptime returns given node's [subnetID] UptimeResults in the view of
	// this node's peer validators.
	NodeUptime(subnetID ids.ID) (UptimeResult, error)
}

Network defines the functionality of the networking library.

func NewNetwork

func NewNetwork(
	config *Config,
	msgCreator message.Creator,
	metricsRegisterer prometheus.Registerer,
	log logging.Logger,
	listener net.Listener,
	dialer dialer.Dialer,
	router router.ExternalHandler,
) (Network, error)

NewNetwork returns a new Network implementation with the provided parameters.

func NewTestNetwork

func NewTestNetwork(
	log logging.Logger,
	networkID uint32,
	currentValidators validators.Manager,
	trackedSubnets set.Set[ids.ID],
	router router.ExternalHandler,
) (Network, error)

Example

package main

import (
	"context"
	"os"
	"time"

	"go.uber.org/zap"

	"github.com/ava-labs/avalanchego/genesis"
	"github.com/ava-labs/avalanchego/ids"
	"github.com/ava-labs/avalanchego/message"
	"github.com/ava-labs/avalanchego/snow/networking/router"
	"github.com/ava-labs/avalanchego/snow/validators"
	"github.com/ava-labs/avalanchego/utils/constants"
	"github.com/ava-labs/avalanchego/utils/ips"
	"github.com/ava-labs/avalanchego/utils/logging"
	"github.com/ava-labs/avalanchego/utils/set"
	"github.com/ava-labs/avalanchego/version"
)

var _ router.ExternalHandler = (*testExternalHandler)(nil)

// Note: all of the external handler's methods are called on peer goroutines. It
// is possible for multiple concurrent calls to happen with different NodeIDs.
// However, a given NodeID will only be performing one call at a time.
type testExternalHandler struct {
	log logging.Logger
}

// Note: HandleInbound will be called with raw P2P messages, the networking
// implementation does not implicitly register timeouts, so this handler is only
// called by messages explicitly sent by the peer. If timeouts are required,
// that must be handled by the user of this utility.
func (t *testExternalHandler) HandleInbound(_ context.Context, message message.InboundMessage) {
	t.log.Info(
		"receiving message",
		zap.Stringer("op", message.Op()),
	)
}

func (t *testExternalHandler) Connected(nodeID ids.NodeID, version *version.Application, subnetID ids.ID) {
	t.log.Info(
		"connected",
		zap.Stringer("nodeID", nodeID),
		zap.Stringer("version", version),
		zap.Stringer("subnetID", subnetID),
	)
}

func (t *testExternalHandler) Disconnected(nodeID ids.NodeID) {
	t.log.Info(
		"disconnected",
		zap.Stringer("nodeID", nodeID),
	)
}

type testAggressiveValidatorManager struct {
	validators.Manager
}

func (*testAggressiveValidatorManager) Contains(ids.ID, ids.NodeID) bool {
	return true
}

func main() {
	log := logging.NewLogger(
		"networking",
		logging.NewWrappedCore(
			logging.Info,
			os.Stdout,
			logging.Colors.ConsoleEncoder(),
		),
	)

	// Needs to be periodically updated by the caller to have the latest
	// validator set
	validators := &testAggressiveValidatorManager{
		Manager: validators.NewManager(),
	}

	// If we want to be able to communicate with non-primary network subnets, we
	// should register them here.
	trackedSubnets := set.Set[ids.ID]{}

	// Messages and connections are handled by the external handler.
	handler := &testExternalHandler{
		log: log,
	}

	network, err := NewTestNetwork(
		log,
		constants.FujiID,
		validators,
		trackedSubnets,
		handler,
	)
	if err != nil {
		log.Fatal(
			"failed to create test network",
			zap.Error(err),
		)
		return
	}

	// We need to initially connect to some nodes in the network before peer
	// gossip will enable connecting to all the remaining nodes in the network.
	bootstrappers := genesis.SampleBootstrappers(constants.FujiID, 5)
	for _, bootstrapper := range bootstrappers {
		network.ManuallyTrack(bootstrapper.ID, ips.IPPort(bootstrapper.IP))
	}

	// Typically network.StartClose() should be called based on receiving a
	// SIGINT or SIGTERM. For the example, we close the network after 15s.
	go log.RecoverAndPanic(func() {
		time.Sleep(15 * time.Second)
		network.StartClose()
	})

	// network.Send(...) and network.Gossip(...) can be used here to send
	// messages to peers.

	// Calling network.Dispatch() will block until a fatal error occurs or
	// network.StartClose() is called.
	err = network.Dispatch()
	log.Info(
		"network exited",
		zap.Error(err),
	)
}

type PeerListGossipConfig

type PeerListGossipConfig struct {
	// PeerListNumValidatorIPs is the number of validator IPs to gossip in every
	// gossip event.
	PeerListNumValidatorIPs uint32 `json:"peerListNumValidatorIPs"`

	// PeerListValidatorGossipSize is the number of validators to gossip the IPs
	// to in every IP gossip event.
	PeerListValidatorGossipSize uint32 `json:"peerListValidatorGossipSize"`

	// PeerListNonValidatorGossipSize is the number of non-validators to gossip
	// the IPs to in every IP gossip event.
	PeerListNonValidatorGossipSize uint32 `json:"peerListNonValidatorGossipSize"`

	// PeerListPeersGossipSize is the number of peers to gossip
	// the IPs to in every IP gossip event.
	PeerListPeersGossipSize uint32 `json:"peerListPeersGossipSize"`

	// PeerListGossipFreq is the frequency that this node will attempt to gossip
	// signed IPs to its peers.
	PeerListGossipFreq time.Duration `json:"peerListGossipFreq"`

	// PeerListPullGossipFreq is the frequency that this node will attempt to
	// request signed IPs from its peers.
	PeerListPullGossipFreq time.Duration `json:"peerListPullGossipFreq"`

	// PeerListBloomResetFreq is how frequently this node will recalculate the
	// IP tracker's bloom filter.
	PeerListBloomResetFreq time.Duration `json:"peerListBloomResetFreq"`
}

type ThrottlerConfig

type ThrottlerConfig struct {
	InboundConnUpgradeThrottlerConfig throttling.InboundConnUpgradeThrottlerConfig `json:"inboundConnUpgradeThrottlerConfig"`
	InboundMsgThrottlerConfig         throttling.InboundMsgThrottlerConfig         `json:"inboundMsgThrottlerConfig"`
	OutboundMsgThrottlerConfig        throttling.MsgByteThrottlerConfig            `json:"outboundMsgThrottlerConfig"`
	MaxInboundConnsPerSec             float64                                      `json:"maxInboundConnsPerSec"`
}

type TimeoutConfig

type TimeoutConfig struct {
	// PingPongTimeout is the maximum amount of time to wait for a Pong response
	// from a peer we sent a Ping to.
	PingPongTimeout time.Duration `json:"pingPongTimeout"`

	// ReadHandshakeTimeout is the maximum amount of time to wait for the peer's
	// connection upgrade to finish before starting the p2p handshake.
	ReadHandshakeTimeout time.Duration `json:"readHandshakeTimeout"`
}

type UptimeResult

type UptimeResult struct {
	// RewardingStakePercentage shows what percent of network stake thinks we're
	// above the uptime requirement.
	RewardingStakePercentage float64

	// WeightedAveragePercentage is the average perceived uptime of this node,
	// weighted by stake.
	// Note that this is different from RewardingStakePercentage, which shows
	// the percent of the network stake that thinks this node is above the
	// uptime requirement. WeightedAveragePercentage is weighted by uptime.
	// i.e If uptime requirement is 85 and a peer reports 40 percent it will be
	// counted (40*weight) in WeightedAveragePercentage but not in
	// RewardingStakePercentage since 40 < 85
	WeightedAveragePercentage float64
}