ragep2p

Documentation

Overview

ragep2p is a simple p2p networking library that provides best-effort, self-healing, message-oriented, authenticated, encrypted bidirectional communication streams between peers.

Concepts

ragep2p peers are identified by their PeerID. PeerIDs are public keys. PeerIDs' string representation is compatible with libp2p to ease migration from libp2p to ragep2p.

ragep2p provides Host and Stream abstractions.

A Host allows users to establish Streams with other peers identified by their PeerID. The host will transparently handle peer discovery, secure connection (re)establishment, multiplexing streams over the connection and rate limiting.

A Stream allows users to send binary messages to another peer. Messages are delivered on a best-effort basis. If the underlying connection to the other peer drops or isn't fast enough or the other peer has not opened a corresponding stream or ..., messages may get dropped. Streams are self-healing: users don't need to close and re-open streams even if the underlying connection drops or the other peer becomes unavailable. We guarantee that messages that are delivered are delivered in FIFO order and without modifications.

Peer discovery

ragep2p will handle peer discovery (i.e. associating network addresses like 1.2.3.4:1337 with PeerIDs) automatically. Upon construction of a Host, a Discoverer is passed in, which is then used by the Host for this purpose.

If multiple network addresses are discovered for a PeerID, ragep2p will try sequentially dialing all of them until a connection is successfully established.

Thread Safety

All public functions on Host and Stream are thread-safe.

It is safe to double-Close(), though all but the first Close() may return an error.

Allocations

We allocate a buffer for each message received. In principle, this could allo an adversary to force a recipient to run out of memory. To defend against this, we put limits on the length of messages and rate limit messages, thereby also limiting adversarially-controlled allocations.

Security

Note: Users don't need to care about the details of how these security measures work, only what properties they provide.

ragep2p's security model assumes that all Streams on the local Host behave honestly and cooperatively. Since many Streams are multiplexed over a single connection, a single "bad" Stream could completely exhaust the entire connection preventing other Streams from delivering messages as well. Other network participants, however, are not assumed to behave honestly and we attempt to defend against fingerprinting, impersonation, MitM, resource exhaustion, tarpitting, etc.

ragep2p uses the Ed25519 signature algorithm and sha2 hash function.

ragep2p attempts to prevent fingerprinting of ragep2p nodes. ragep2p will not respond on a connection until it has received a valid knock message constructed over the PeerID of the connection initiator and the PeerID of the connection receiver. (knocks carry a signature for authentication, though it's important to note that by their uni-directional nature a knock does not constitute a proper handshake and can be replayed.)

ragep2p connections are authenticated and encrypted using mutual TLS 1.3, using the crypto/tls package from Go's standard library. TLS is used with ephemeral certificates using the keypair corresponding to the Host's PeerID.

ragep2p tries to defend against resource exchaustion attacks. In particular, we enforce maximum Stream counts per peer, maximum lengths for various messages, apply rate limiting at the tcp connection level as well as at the individual Stream level, and have a constant bound on the number of buffered messages per Stream.

ragep2p defends against tarpitting, i.e. other peers that intentionally read/write from the underlying connection slowly. Host.NewStream(), Stream.Close(), Stream.SendMessage(), and Stream.Receive() return immediately and do any potential resulting communication asynchronously in the background. Host.Close() terminates after at most a few seconds.

Metrics

ragep2p exposes prometheus metrics. Their names are prefixed with "ragep2p_". The audience for these metrics are operators of software using ragep2p and developers building on top of ragep2p.

Suggested Health Checks

The following example health checks in PromQL enable operators to monitor connectivity to remote peers and health of connections with them. We suggest monitoring *all* of these metrics. These examples only serve as a starting point, operators are encouraged to adjust thresholds empirically and come up with more sophisticated queries.

Is my ragep2p host receiving data from the remote peer? If not, there is a connectivity problem.

rate(ragep2p_peer_conn_read_processed_bytes_total[5m]) > 0

Is my ragep2p host sending data to the remote peer? If not, there is a connectivity problem.

rate(ragep2p_peer_conn_written_bytes_total[5m]) > 0

Is connection churn reasonable (e.g. less than one new connection every 3m)? If not, this may point to infrastructure or ISP problems.

rate(ragep2p_peer_conn_established_total[10m]) < 1/(3 * 60)

Is my ragep2p host not skipping any data received from the remote peer? if not, this may point to bugs in the software running on top of ragep2p.

rate(ragep2p_peer_conn_read_skipped_bytes_total[5m]) == 0

Is my ragep2p host receiving inbound dials at all? If not, this may point to a misconfiguration in my infrastructure.

rate(ragep2p_host_inbound_dials_total[48h]) > 0

Index

• Constants
• type Discoverer
• type Host
  • func NewHost(config HostConfig, secretKey ed25519.PrivateKey, listenAddresses []string, ...) (*Host, error)
  • func (ho *Host) Close() error
  • func (ho *Host) ID() types.PeerID
  • func (ho *Host) NewStream(other types.PeerID, streamName string, outgoingBufferSize int, ...) (*Stream, error)
  • func (ho *Host) Start() error
• type HostConfig
• type Stream
  • func (st *Stream) Close() error
  • func (st *Stream) Name() string
  • func (st *Stream) Other() types.PeerID
  • func (st *Stream) ReceiveMessages() <-chan []byte
  • func (st *Stream) SendMessage(data []byte)
• type TokenBucketParams

Constants

const MaxMessageLength = 1024 * 1024 * 1024 // 1 GiB. This must be smaller than INT32_MAX

Maximum length of messages sent with ragep2p

const MaxStreamNameLength = 256

Maximum stream name length

const MaxStreamsPerPeer = 2_000

Maximum number of streams with another peer that can be opened on a host

Types

type Discoverer

type Discoverer interface {
	Start(host *Host, privKey ed25519.PrivateKey, logger loghelper.LoggerWithContext) error
	Close() error
	FindPeer(peer types.PeerID) ([]types.Address, error)
}

Discoverer is responsible for discovering the addresses of peers on the network.

type Host

type Host struct {
	// contains filtered or unexported fields
}

A Host allows users to establish Streams with other peers identified by their PeerID. The host will transparently handle peer discovery, secure connection (re)establishment, multiplexing streams over the connection and rate limiting.

func NewHost

func NewHost(
	config HostConfig,
	secretKey ed25519.PrivateKey,
	listenAddresses []string,
	discoverer Discoverer,
	logger commontypes.Logger,
	metricsRegisterer prometheus.Registerer,
) (*Host, error)

NewHost creates a new Host with the provided config, Ed25519 secret key, network listen address. A Discoverer is also provided to NewHost for discovering addresses of peers.

func (*Host) Close

func (ho *Host) Close() error

Close stops listening on the network interface(s) and closes all active streams.

func (*Host) ID

func (ho *Host) ID() types.PeerID

func (*Host) NewStream

func (ho *Host) NewStream(
	other types.PeerID,
	streamName string,
	outgoingBufferSize int,
	incomingBufferSize int,
	maxMessageLength int,
	messagesLimit TokenBucketParams,
	bytesLimit TokenBucketParams,
) (*Stream, error)

NewStream creates a new bidirectional stream with peer other for streamName. It is parameterized with a maxMessageLength, the maximum size of a message in bytes and two parameters for rate limiting.

func (*Host) Start

func (ho *Host) Start() error

Start listening on the network interfaces and dialling peers.

type HostConfig

type HostConfig struct {
	// DurationBetweenDials is the minimum duration between two dials. It is
	// not the exact duration because of jitter.
	DurationBetweenDials time.Duration
}

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

Stream is an over-the-network channel between two peers. Two peers may share multiple disjoint streams with different names. Streams are persistent and agnostic to the state of the connection. They completely abstract the underlying connection. Messages are delivered on a best effort basis.

func (*Stream) Close

func (st *Stream) Close() error

Close the stream. This closes any channel returned by ReceiveMessages earlier. After close the stream cannot be reopened. If the stream is needed in the future it should be created again through NewStream. After close, any messages passed to SendMessage will be dropped.

func (*Stream) Name

func (st *Stream) Name() string

Name returns the name of the stream.

func (*Stream) Other

func (st *Stream) Other() types.PeerID

Other returns the peer ID of the stream counterparty.

func (*Stream) ReceiveMessages

func (st *Stream) ReceiveMessages() <-chan []byte

Best effort receiving of messages. The returned channel will be closed when the stream is closed. Note that this function may return the same channel across invocations.

func (*Stream) SendMessage

func (st *Stream) SendMessage(data []byte)

Best effort sending of messages. May fail without returning an error.

type TokenBucketParams

type TokenBucketParams struct {
	Rate     float64
	Capacity uint32
}

TokenBucketParams contains the two parameters for a token bucket rate limiter.