network

README

Avalanche Networking

Table of Contents

• Overview
• Peers
  • Message Handling
  • Peer Handshake
  • Ping-Pong Messages
• Peer Discovery
  • Inbound Connections
  • Outbound Connections
    • IP Authentication
  • Bootstrapping
  • PeerList Gossip
    • Bloom Filter
    • GetPeerList
    • PeerList

Overview

Avalanche is a decentralized p2p (peer-to-peer) network of nodes that work together to run the Avalanche blockchain protocol.

The network package implements the networking layer of the protocol which allows a node to discover, connect to, and communicate with other peers.

All connections are authenticated using TLS. However, there is no reliance on any certificate authorities. The network package identifies peers by the public key in the leaf certificate.

Peers

Peers are defined as members of the network that communicate with one another to participate in the Avalanche protocol.

Peers communicate by enqueuing messages between one another. Each peer on either side of the connection asynchronously reads and writes messages to and from the remote peer. Messages include both application-level messages used to support the Avalanche protocol, as well as networking-level messages used to implement the peer-to-peer communication layer.

sequenceDiagram
    actor Alice
    actor Bob
    loop 
        Alice->>Bob: Write outbound messages
        Bob->>Alice: Read incoming messages
    end
    loop
        Bob->>Alice: Write outbound messages
        Alice->>Bob: Read incoming messages
    end

Message Handling

All messages are prefixed with their length. Reading a message first reads the 4-byte message length from the connection. The rate-limiting logic then waits until there is sufficient capacity to read these bytes from the connection.

A peer will then read the full message and attempt to parse it into either a networking message or an application message. If the message is malformed the connection is not dropped. The peer will simply continue to the next sent message.

Peer Handshake

Upon connection to a new peer, a handshake is performed between the node attempting to establish the outbound connection to the peer and the peer receiving the inbound connection.

When attempting to establish the connection, the first message that the node sends is a Handshake message describing the compatibility of the nodes. If the Handshake message is successfully received and the peer decides that it wants a connection with this node, it replies with a PeerList message that contains metadata about other peers that allows a node to connect to them. See Peerlist Gossip.

As an example, nodes that are attempting to connect with an incompatible version of CryftGo or a significantly skewed local clock are rejected.

sequenceDiagram
    actor Alice
    actor Bob
    Note over Alice,Bob: Connection Created
    par
        Alice->>Bob: CryftGo v1.0.0
    and
        Bob->>Alice: CryftGo v1.11.4
    end
    Note right of Bob: v1.0.0 is incompatible with v1.11.4.
    Note left of Alice: v1.11.4 could be compatible with v1.0.0!
    par
        Bob-->>Alice: Disconnect
    and
        Alice-XBob: Peerlist
    end
    Note over Alice,Bob: Handshake Failed

Nodes that mutually desire the connection will both respond with PeerList messages and complete the handshake.

sequenceDiagram
    actor Alice
    actor Bob
    Note over Alice,Bob: Connection Created
    par
        Alice->>Bob: CryftGo v1.11.0
    and
        Bob->>Alice: CryftGo v1.11.4
    end
    Note right of Bob: v1.11.0 is compatible with v1.11.4!
    Note left of Alice: v1.11.4 could be compatible with v1.11.0!
    par
        Bob->>Alice: Peerlist
    and
        Alice->>Bob: Peerlist
    end
    Note over Alice,Bob: Handshake Complete

Ping-Pong Messages

Peers periodically send Ping messages containing perceived uptime information. This information can be used to monitor how the node is considered to be performing by the network. It is expected for a node to reply to a Ping message with a Pong message.

sequenceDiagram
    actor Alice
    actor Bob
    Note left of Alice: Send Ping
    Alice->>Bob: I think your uptime is 95%
    Note right of Bob: Send Pong
    Bob->>Alice: ACK

Peer Discovery

When starting an Avalanche node, a node needs to be able to initiate some process that eventually allows itself to become a participating member of the network. In traditional web2 systems, it's common to use a web service by hitting the service's DNS and being routed to an available server behind a load balancer. In decentralized p2p systems, however, connecting to a node is more complex as no single entity owns the network. Avalanche consensus requires a node to repeatedly sample peers in the network, so each node needs some way of discovering and connecting to every other peer to participate in the protocol.

Inbound Connections

It is expected for Avalanche nodes to allow inbound connections. If a validator does not allow inbound connections, its observed uptime may be reduced.

Outbound Connections

Avalanche nodes that have identified the IP:Port pair of a node they want to connect to will initiate outbound connections to this IP:Port pair. If the connection is not able to complete the Peer Handshake, the connection will be re-attempted with an Exponential Backoff.

A node should initiate outbound connections to an IP:Port pair that is believed to belong to a node that is not connected and meets at least one of the following conditions:

• The node is in the initial bootstrapper set.
• The node is in the default bootstrapper set.
• The node in the current Primary Network validator set.

IP Authentication

To ensure that outbound connections are being made to the correct IP:Port pair of a node, all IP:Port pairs sent by the network are signed by the node that is claiming ownership of the pair. To prevent replays of these messages, the signature is over the Timestamp in addition to the IP:Port pair.

The Timestamp guarantees that nodes provided an IP:Port pair can track the most up-to-date IP:Port pair of a peer.

Bootstrapping

In Avalanche, nodes connect to an initial set (this is user-configurable) of bootstrap nodes.

PeerList Gossip

Once connected to an initial set of peers, a node can use these connections to discover additional peers.

Peers are discovered by receiving PeerList messages during the Peer Handshake. These messages quickly provide a node with knowledge of peers in the network. However, they offer no guarantee that the node will connect to and maintain connections with every peer in the network.

To provide an eventual guarantee that all peers learn of one another, nodes periodically send a GetPeerList message to a randomly selected validator with the node's current Bloom Filter and Salt.

Bloom Filter

A Bloom Filter is used to track which nodes are known.

The parameterization of the Bloom Filter is based on the number of desired peers.

Entries in the Bloom Filter are determined by a locally calculated Salt along with the NodeID and Timestamp of the most recently known IP:Port. The Salt is added to prevent griefing attacks where malicious nodes intentionally generate hash collisions with other virtuous nodes to reduce their connectivity.

The Bloom Filter is reconstructed if there are more entries than expected to avoid increasing the false positive probability. It is also reconstructed periodically. When reconstructing the Bloom Filter, a new Salt is generated.

To prevent a malicious node from arbitrarily filling this Bloom Filter, only 2 entries are added to the Bloom Filter per node. If a node's IP:Port pair changes once, it will immediately be added to the Bloom Filter. If a node's IP:Port pair changes more than once, it will only be added to the Bloom Filter after the Bloom Filter is reconstructed.

GetPeerList

A GetPeerList message contains the Bloom Filter of the currently known peers along with the Salt that was used to add entries to the Bloom Filter. Upon receipt of a GetPeerList message, a node is expected to respond with a PeerList message.

PeerList

PeerList messages are expected to contain IP:Port pairs that satisfy all of the following constraints:

• The Bloom Filter sent when requesting the PeerList message does not contain the node claiming the IP:Port pair.
• The node claiming the IP:Port pair is currently connected.
• The IP:Port pair the node shared during the Handshake message is the node's most recently known IP:Port pair.
• The node claiming the IP:Port pair is either in the default bootstrapper set or is a current Primary Network validator.

Example PeerList Gossip

The following diagram shows an example of Alice repeatedly learning about new peers from Bob.

sequenceDiagram
    actor Alice
    actor Bob
    Note left of Alice: Initialize Bloom Filter
    Note left of Alice: Bloom: [0, 0, 0]
    Alice->>Bob: GetPeerList [0, 0, 0]
    Note right of Bob: Any peers can be sent.
    Bob->>Alice: PeerList - Peer-1
    Note left of Alice: Bloom: [1, 0, 0]
    Alice->>Bob: GetPeerList [1, 0, 0]
    Note right of Bob: Either Peer-2 or Peer-3 can be sent.
    Bob->>Alice: PeerList - Peer-3
    Note left of Alice: Bloom: [1, 0, 1]
    Alice->>Bob: GetPeerList [1, 0, 1]
    Note right of Bob: Only Peer-2 can be sent.
    Bob->>Alice: PeerList - Peer-2
    Note left of Alice: Bloom: [1, 1, 1]
    Alice->>Bob: GetPeerList [1, 1, 1]
    Note right of Bob: There are no more peers left to send!
    Bob->>Alice: PeerList - Empty

Documentation

Index

• Constants
• type Config
• type DelayConfig
• type HealthConfig
• type Network
  • func NewNetwork(config *Config, msgCreator message.Creator, ...) (Network, error)
  • func NewTestNetwork(log logging.Logger, networkID uint32, currentValidators validators.Manager, ...) (Network, error)
• type PeerListGossipConfig
• type ThrottlerConfig
• type TimeoutConfig
• type UptimeResult

Examples

• NewTestNetwork

Constants

const (
	ConnectedPeersKey           = "connectedPeers"
	TimeSinceLastMsgReceivedKey = "timeSinceLastMsgReceived"
	TimeSinceLastMsgSentKey     = "timeSinceLastMsgSent"
	SendFailRateKey             = "sendFailRate"
)

Types

type Config

type Config struct {
	HealthConfig         `json:"healthConfig"`
	PeerListGossipConfig `json:"peerListGossipConfig"`
	TimeoutConfig        `json:"timeoutConfigs"`
	DelayConfig          `json:"delayConfig"`
	ThrottlerConfig      ThrottlerConfig `json:"throttlerConfig"`

	ProxyEnabled           bool          `json:"proxyEnabled"`
	ProxyReadHeaderTimeout time.Duration `json:"proxyReadHeaderTimeout"`

	DialerConfig dialer.Config `json:"dialerConfig"`
	TLSConfig    *tls.Config   `json:"-"`

	TLSKeyLogFile string `json:"tlsKeyLogFile"`

	Namespace          string            `json:"namespace"`
	MyNodeID           ids.NodeID        `json:"myNodeID"`
	MyIPPort           ips.DynamicIPPort `json:"myIP"`
	NetworkID          uint32            `json:"networkID"`
	MaxClockDifference time.Duration     `json:"maxClockDifference"`
	PingFrequency      time.Duration     `json:"pingFrequency"`
	AllowPrivateIPs    bool              `json:"allowPrivateIPs"`

	SupportedACPs set.Set[uint32] `json:"supportedACPs"`
	ObjectedACPs  set.Set[uint32] `json:"objectedACPs"`

	// The compression type to use when compressing outbound messages.
	// Assumes all peers support this compression type.
	CompressionType compression.Type `json:"compressionType"`

	// TLSKey is this node's TLS key that is used to sign IPs.
	TLSKey crypto.Signer `json:"-"`
	// BLSKey is this node's BLS key that is used to sign IPs.
	BLSKey *bls.SecretKey `json:"-"`

	// TrackedSubnets of the node.
	TrackedSubnets set.Set[ids.ID]    `json:"-"`
	Beacons        validators.Manager `json:"-"`

	// Validators are the current validators in the Avalanche network
	Validators validators.Manager `json:"-"`

	UptimeCalculator uptime.Calculator `json:"-"`

	// UptimeMetricFreq marks how frequently this node will recalculate the
	// observed average uptime metrics.
	UptimeMetricFreq time.Duration `json:"uptimeMetricFreq"`

	// UptimeRequirement is the fraction of time a validator must be online and
	// responsive for us to vote that they should receive a staking reward.
	UptimeRequirement float64 `json:"-"`

	// RequireValidatorToConnect require that all connections must have at least
	// one validator between the 2 peers. This can be useful to enable if the
	// node wants to connect to the minimum number of nodes without impacting
	// the network negatively.
	RequireValidatorToConnect bool `json:"requireValidatorToConnect"`

	// MaximumInboundMessageTimeout is the maximum deadline duration in a
	// message. Messages sent by clients setting values higher than this value
	// will be reset to this value.
	MaximumInboundMessageTimeout time.Duration `json:"maximumInboundMessageTimeout"`

	// Size, in bytes, of the buffer that we read peer messages into
	// (there is one buffer per peer)
	PeerReadBufferSize int `json:"peerReadBufferSize"`

	// Size, in bytes, of the buffer that we write peer messages into
	// (there is one buffer per peer)
	PeerWriteBufferSize int `json:"peerWriteBufferSize"`

	// Tracks the CPU/disk usage caused by processing messages of each peer.
	ResourceTracker tracker.ResourceTracker `json:"-"`

	// Specifies how much CPU usage each peer can cause before
	// we rate-limit them.
	CPUTargeter tracker.Targeter `json:"-"`

	// Specifies how much disk usage each peer can cause before
	// we rate-limit them.
	DiskTargeter tracker.Targeter `json:"-"`
}

type DelayConfig

type DelayConfig struct {
	// InitialReconnectDelay is the minimum amount of time the node will delay a
	// reconnection to a peer. This value is used to start the exponential
	// backoff.
	InitialReconnectDelay time.Duration `json:"initialReconnectDelay"`

	// MaxReconnectDelay is the maximum amount of time the node will delay a
	// reconnection to a peer.
	MaxReconnectDelay time.Duration `json:"maxReconnectDelay"`
}

type HealthConfig

type HealthConfig struct {
	// Marks if the health check should be enabled
	Enabled bool `json:"-"`

	// MinConnectedPeers is the minimum number of peers that the network should
	// be connected to be considered healthy.
	MinConnectedPeers uint `json:"minConnectedPeers"`

	// MaxTimeSinceMsgReceived is the maximum amount of time since the network
	// last received a message to be considered healthy.
	MaxTimeSinceMsgReceived time.Duration `json:"maxTimeSinceMsgReceived"`

	// MaxTimeSinceMsgSent is the maximum amount of time since the network last
	// sent a message to be considered healthy.
	MaxTimeSinceMsgSent time.Duration `json:"maxTimeSinceMsgSent"`

	// MaxPortionSendQueueBytesFull is the maximum percentage of the pending
	// send byte queue that should be used for the network to be considered
	// healthy. Should be in (0,1].
	MaxPortionSendQueueBytesFull float64 `json:"maxPortionSendQueueBytesFull"`

	// MaxSendFailRate is the maximum percentage of send attempts that should be
	// failing for the network to be considered healthy. This does not include
	// send attempts that were not made due to benching. Should be in [0,1].
	MaxSendFailRate float64 `json:"maxSendFailRate"`

	// SendFailRateHalflife is the halflife of the averager used to calculate
	// the send fail rate percentage. Should be > 0. Larger values mean that the
	// fail rate is affected less by recently dropped messages.
	SendFailRateHalflife time.Duration `json:"sendFailRateHalflife"`
}

HealthConfig describes parameters for network layer health checks.

type Network

type Network interface {
	// All consensus messages can be sent through this interface. Thread safety
	// must be managed internally in the network.
	sender.ExternalSender

	// Has a health check
	health.Checker

	peer.Network

	// StartClose this network and all existing connections it has. Calling
	// StartClose multiple times is handled gracefully.
	StartClose()

	// Should only be called once, will run until either a fatal error occurs,
	// or the network is closed.
	Dispatch() error

	// Attempt to connect to this IP. The network will never stop attempting to
	// connect to this ID.
	ManuallyTrack(nodeID ids.NodeID, ip ips.IPPort)

	// PeerInfo returns information about peers. If [nodeIDs] is empty, returns
	// info about all peers that have finished the handshake. Otherwise, returns
	// info about the peers in [nodeIDs] that have finished the handshake.
	PeerInfo(nodeIDs []ids.NodeID) []peer.Info

	// NodeUptime returns given node's [subnetID] UptimeResults in the view of
	// this node's peer validators.
	NodeUptime(subnetID ids.ID) (UptimeResult, error)
}

Network defines the functionality of the networking library.

func NewNetwork

func NewNetwork(
	config *Config,
	msgCreator message.Creator,
	metricsRegisterer prometheus.Registerer,
	log logging.Logger,
	listener net.Listener,
	dialer dialer.Dialer,
	router router.ExternalHandler,
) (Network, error)

NewNetwork returns a new Network implementation with the provided parameters.

func NewTestNetwork

func NewTestNetwork(
	log logging.Logger,
	networkID uint32,
	currentValidators validators.Manager,
	trackedSubnets set.Set[ids.ID],
	router router.ExternalHandler,
) (Network, error)

Example

package main

import (
	"context"
	"os"
	"time"

	"go.uber.org/zap"

	"github.com/cryft-labs/cryftgo/genesis"
	"github.com/cryft-labs/cryftgo/ids"
	"github.com/cryft-labs/cryftgo/message"
	"github.com/cryft-labs/cryftgo/snow/networking/router"
	"github.com/cryft-labs/cryftgo/snow/validators"
	"github.com/cryft-labs/cryftgo/utils/constants"
	"github.com/cryft-labs/cryftgo/utils/ips"
	"github.com/cryft-labs/cryftgo/utils/logging"
	"github.com/cryft-labs/cryftgo/utils/set"
	"github.com/cryft-labs/cryftgo/version"
)

var _ router.ExternalHandler = (*testExternalHandler)(nil)

// Note: all of the external handler's methods are called on peer goroutines. It
// is possible for multiple concurrent calls to happen with different NodeIDs.
// However, a given NodeID will only be performing one call at a time.
type testExternalHandler struct {
	log logging.Logger
}

// Note: HandleInbound will be called with raw P2P messages, the networking
// implementation does not implicitly register timeouts, so this handler is only
// called by messages explicitly sent by the peer. If timeouts are required,
// that must be handled by the user of this utility.
func (t *testExternalHandler) HandleInbound(_ context.Context, message message.InboundMessage) {
	t.log.Info(
		"receiving message",
		zap.Stringer("op", message.Op()),
	)
}

func (t *testExternalHandler) Connected(nodeID ids.NodeID, version *version.Application, subnetID ids.ID) {
	t.log.Info(
		"connected",
		zap.Stringer("nodeID", nodeID),
		zap.Stringer("version", version),
		zap.Stringer("subnetID", subnetID),
	)
}

func (t *testExternalHandler) Disconnected(nodeID ids.NodeID) {
	t.log.Info(
		"disconnected",
		zap.Stringer("nodeID", nodeID),
	)
}

type testAggressiveValidatorManager struct {
	validators.Manager
}

func (*testAggressiveValidatorManager) Contains(ids.ID, ids.NodeID) bool {
	return true
}

func main() {
	log := logging.NewLogger(
		"networking",
		logging.NewWrappedCore(
			logging.Info,
			os.Stdout,
			logging.Colors.ConsoleEncoder(),
		),
	)

	// Needs to be periodically updated by the caller to have the latest
	// validator set
	validators := &testAggressiveValidatorManager{
		Manager: validators.NewManager(),
	}

	// If we want to be able to communicate with non-primary network subnets, we
	// should register them here.
	trackedSubnets := set.Set[ids.ID]{}

	// Messages and connections are handled by the external handler.
	handler := &testExternalHandler{
		log: log,
	}

	network, err := NewTestNetwork(
		log,
		constants.FujiID,
		validators,
		trackedSubnets,
		handler,
	)
	if err != nil {
		log.Fatal(
			"failed to create test network",
			zap.Error(err),
		)
		return
	}

	// We need to initially connect to some nodes in the network before peer
	// gossip will enable connecting to all the remaining nodes in the network.
	bootstrappers := genesis.SampleBootstrappers(constants.FujiID, 5)
	for _, bootstrapper := range bootstrappers {
		network.ManuallyTrack(bootstrapper.ID, ips.IPPort(bootstrapper.IP))
	}

	// Typically network.StartClose() should be called based on receiving a
	// SIGINT or SIGTERM. For the example, we close the network after 15s.
	go log.RecoverAndPanic(func() {
		time.Sleep(15 * time.Second)
		network.StartClose()
	})

	// network.Send(...) and network.Gossip(...) can be used here to send
	// messages to peers.

	// Calling network.Dispatch() will block until a fatal error occurs or
	// network.StartClose() is called.
	err = network.Dispatch()
	log.Info(
		"network exited",
		zap.Error(err),
	)
}

type PeerListGossipConfig

type PeerListGossipConfig struct {
	// PeerListNumValidatorIPs is the number of validator IPs to gossip in every
	// gossip event.
	PeerListNumValidatorIPs uint32 `json:"peerListNumValidatorIPs"`

	// PeerListPullGossipFreq is the frequency that this node will attempt to
	// request signed IPs from its peers.
	PeerListPullGossipFreq time.Duration `json:"peerListPullGossipFreq"`

	// PeerListBloomResetFreq is how frequently this node will recalculate the
	// IP tracker's bloom filter.
	PeerListBloomResetFreq time.Duration `json:"peerListBloomResetFreq"`
}

type ThrottlerConfig

type ThrottlerConfig struct {
	InboundConnUpgradeThrottlerConfig throttling.InboundConnUpgradeThrottlerConfig `json:"inboundConnUpgradeThrottlerConfig"`
	InboundMsgThrottlerConfig         throttling.InboundMsgThrottlerConfig         `json:"inboundMsgThrottlerConfig"`
	OutboundMsgThrottlerConfig        throttling.MsgByteThrottlerConfig            `json:"outboundMsgThrottlerConfig"`
	MaxInboundConnsPerSec             float64                                      `json:"maxInboundConnsPerSec"`
}

type TimeoutConfig

type TimeoutConfig struct {
	// PingPongTimeout is the maximum amount of time to wait for a Pong response
	// from a peer we sent a Ping to.
	PingPongTimeout time.Duration `json:"pingPongTimeout"`

	// ReadHandshakeTimeout is the maximum amount of time to wait for the peer's
	// connection upgrade to finish before starting the p2p handshake.
	ReadHandshakeTimeout time.Duration `json:"readHandshakeTimeout"`
}

type UptimeResult

type UptimeResult struct {
	// RewardingStakePercentage shows what percent of network stake thinks we're
	// above the uptime requirement.
	RewardingStakePercentage float64

	// WeightedAveragePercentage is the average perceived uptime of this node,
	// weighted by stake.
	// Note that this is different from RewardingStakePercentage, which shows
	// the percent of the network stake that thinks this node is above the
	// uptime requirement. WeightedAveragePercentage is weighted by uptime.
	// i.e If uptime requirement is 85 and a peer reports 40 percent it will be
	// counted (40*weight) in WeightedAveragePercentage but not in
	// RewardingStakePercentage since 40 < 85
	WeightedAveragePercentage float64
}