fastafpacket

README

fast_afpacket

fast_afpacket is a Go library for sending and receiving packets using AF_PACKET sockets. It's built around mdlayher/socket. The library also fully supports linux socket timestamp options, setting the correct ioctls and socket options automatically, and includes APIs for retrieving RX/TX timestamps back from the socket.

Requirements

Because this library deals with interacting with raw packets and directly with network interfaces you need the following:

• libpcap-dev installed for raw packet encoding/decoding
• sudo to run with root privileges so you program can access the network interfaces

Example App

For convenience there is an example service which acts as a client and server to send packets to other instances of itself that is running.

To set up the network in which to get the example working we use Vagrant to create compete Linux virtual machines and specifically configured network interfaces and their associated MAC and IPv4/6 addresses and ARP table entries.

Running

1. Install Vagrant for your operating system.

2. From the root folder where the Vagrantfile is located run vagrant up. You should now have 2 virtual machines running:

>  vagrant up
Bringing machine 'fast-afpacket-101' up with 'virtualbox' provider...
Bringing machine 'fast-afpacket-102' up with 'virtualbox' provider...
...
==> fast-afpacket-101: Machine 'fast-afpacket-101' has a post `vagrant up` message. This is a message
==> fast-afpacket-101: from the creator of the Vagrantfile, and not from Vagrant itself:
==> fast-afpacket-101:
==> fast-afpacket-101: Vanilla Debian box. See https://app.vagrantup.com/debian for help and bug reports

==> fast-afpacket-102: Machine 'fast-afpacket-102' has a post `vagrant up` message. This is a message
==> fast-afpacket-102: from the creator of the Vagrantfile, and not from Vagrant itself:
==> fast-afpacket-102:
==> fast-afpacket-102: Vanilla Debian box. See https://app.vagrantup.com/debian for help and bug reports

>  vagrant status
Current machine states:

fast-afpacket-101          running (virtualbox)
fast-afpacket-102          running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

1. Open up a second terminal window in the root of the repo.

2. In the first terminal window SSH into the first virtual machine

>  vagrant ssh fast-afpacket-101
...
Last login: Wed May  4 17:12:12 2022 from 10.0.2.2
vagrant@fast-afpacket-101:~$

1. In the second terminal window SSH into the second virtual machine

>  vagrant ssh fast-afpacket-102
...
Last login: Wed May  4 17:12:12 2022 from 10.0.2.2
vagrant@fast-afpacket-102:~$

1. In each of the terminal windows change into the shared folder containing the repo.

vagrant@fast-afpacket-101:~$ cd /vagrant/
vagrant@fast-afpacket-101:/vagrant$

1. In the first terminal where you're connected fast-afpacket-101 start the example app to start it sending packets to the second instance which we will start in the next step.

vagrant@fast-afpacket-101:/vagrant$ ./server-1.sh

1. In the second terminal where you're connected fast-afpacket-102 start the example app to start it sending packets to the first instance.

vagrant@fast-afpacket-102:/vagrant$ ./server-2.sh

1. Watching each terminal you will see timestamps being printed for each packet being send and received.

fast-afpacket-101

INFO[0013] TX Recvmsg                                    hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=13 software="2022-05-09T23:16:42.507654453Z" software_ns=1652138202507654453
INFO[0013] RX Recvmsg                                    delay="-423.796µs" hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=33 software="2022-05-09T23:16:42.855011458Z" software_ns=1652138202855011458 userspace="2022-05-09T23:16:42.855435254Z" userspace_ns=1652138202855435254
INFO[0014] TX Recvmsg                                    hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=14 software="2022-05-09T23:16:43.50814104Z" software_ns=1652138203508141040
INFO[0014] RX Recvmsg                                    delay="-393.518µs" hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=34 software="2022-05-09T23:16:43.85478227Z" software_ns=1652138203854782270 userspace="2022-05-09T23:16:43.855175788Z" userspace_ns=165213820385517578

fast-afpacket-102

INFO[0034] TX Recvmsg                                    hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=34 software="2022-05-09T23:16:43.853715657Z" software_ns=1652138203853715657
INFO[0034] RX Recvmsg                                    delay="-324.866µs" hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=15 software="2022-05-09T23:16:44.508531621Z" software_ns=1652138204508531621 userspace="2022-05-09T23:16:44.508856487Z" userspace_ns=1652138204508856487
INFO[0035] TX Recvmsg                                    hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=35 software="2022-05-09T23:16:44.853216023Z" software_ns=1652138204853216023
INFO[0035] RX Recvmsg                                    delay="-448.397µs" hardware="0001-01-01T00:00:00Z" hardware_ns=-6795364578871345152 probe=16 software="2022-05-09T23:16:45.507733046Z" software_ns=1652138205507733046 userspace="2022-05-09T23:16:45.508181443Z" userspace_ns=1652138205508181443

Note: Because we are using virtual machines to run the example you will not see any hardware timestamps. You will only see software timestamps which is the kernel setting the timestamp on the packet. In order to get real hardware timestamps you will need to use the library on bare metal hardware with NICs which support hardware timestamping.

Authors

fast_afpacket was designed and authored by Blain Smith and Joe williams at Subspace for use in our high resolution network telemetry system.

Documentation

Index

• Constants
• type Addr
  • func (a *Addr) Network() string
  • func (a *Addr) String() string
• type Config
• type Conn
  • func Listen(iface *net.Interface, socketType int, socketProtocol int, config *Config) (*Conn, error)
  • func (c *Conn) Close() error
  • func (c *Conn) LocalAddr() net.Addr
  • func (c *Conn) ReadFrom(b []byte) (int, net.Addr, error)
  • func (c *Conn) RecvRxTimestamps(b []byte) (int, net.Addr, SocketTimestamps, error)
  • func (c *Conn) RecvTxTimestamps(b []byte) (int, net.Addr, SocketTimestamps, error)
  • func (c *Conn) SetBPF(filter []bpf.RawInstruction) error
  • func (c *Conn) SetDeadline(t time.Time) error
  • func (c *Conn) SetReadDeadline(t time.Time) error
  • func (c *Conn) SetWriteDeadline(t time.Time) error
  • func (c *Conn) Stats() (*Stats, error)
  • func (c *Conn) WriteTo(b []byte, addr net.Addr) (int, error)
• type SocketTimestamps
  • func ParseSocketTimestamps(msg unix.SocketControlMessage) (SocketTimestamps, error)
• type Stats

Constants

const (
	// SocketOptTimestamping is an alias for unix.SO_TIMESTAMPING
	SocketOptTimestamping = 0x25

	// MsgErrQueue is an alias for unix.MSG_ERRQUEUE
	MsgErrQueue = 0x2000
)

Types

type Addr

type Addr struct {
	HardwareAddr net.HardwareAddr
}

Addr is the Layer 1 address

func (*Addr) Network

func (a *Addr) Network() string

Network returns the network name

func (*Addr) String

func (a *Addr) String() string

String returns the Layer 1 string address

type Config

type Config struct {
	// DualConn creates separate send and recv connections to minimize contention
	// on a single file descriptor. Enable this option if you're sending packets
	// at a high rate. Calling methods on Conn will transparently use the correct
	// underlying socket.
	DualConn bool

	// Filter will be applied to the socket before bind is called to ensure
	// no packets will come through before the connection is opened.
	Filter []bpf.RawInstruction
}

Config

type Conn

type Conn struct {
	// contains filtered or unexported fields
}

Conn implements net.PacketConn and uses AF_PACKET under the hood

func Listen

func Listen(iface *net.Interface, socketType int, socketProtocol int, config *Config) (*Conn, error)

func (*Conn) Close

func (c *Conn) Close() error

Close implements net.PacketConn Close

func (*Conn) LocalAddr

func (c *Conn) LocalAddr() net.Addr

LocalAddr implements net.PacketConn LocalAddr

func (*Conn) ReadFrom

func (c *Conn) ReadFrom(b []byte) (int, net.Addr, error)

ReadFrom implements net.PacketConn ReadFrom

func (*Conn) RecvRxTimestamps

func (c *Conn) RecvRxTimestamps(b []byte) (int, net.Addr, SocketTimestamps, error)

RecvRxTimestamps blocks and listens on the underlying socket's Rx queue for incoming packets while also returning the Rx timestamps for that packet. This can be used instead of ReadFrom to get SocketTimestamps returned along with the packet data.

func (*Conn) RecvTxTimestamps

func (c *Conn) RecvTxTimestamps(b []byte) (int, net.Addr, SocketTimestamps, error)

RecvTxTimestamps blocks and listens on the underlying socket's error queue for outgoing Tx timestamp information and returns the original packet plus the timestamp information.

func (*Conn) SetBPF

func (c *Conn) SetBPF(filter []bpf.RawInstruction) error

SetBPF applies the BPF program filter to the socket

func (*Conn) SetDeadline

func (c *Conn) SetDeadline(t time.Time) error

SetDeadline implements net.PacketConn SetDeadline

func (*Conn) SetReadDeadline

func (c *Conn) SetReadDeadline(t time.Time) error

SetReadDeadline implements net.PacketConn SetReadDeadline

func (*Conn) SetWriteDeadline

func (c *Conn) SetWriteDeadline(t time.Time) error

SetWriteDeadline implements net.PacketConn Close

func (*Conn) Stats

func (c *Conn) Stats() (*Stats, error)

Stats reads statistics from the kernel for the connection. Calling this resets the kernel counters to 0 so you must keep running totals in the calling code.

func (*Conn) WriteTo

func (c *Conn) WriteTo(b []byte, addr net.Addr) (int, error)

WriteTo implements net.PacketConn WriteTo

type SocketTimestamps

type SocketTimestamps struct {
	Software time.Time
	Hardware time.Time
}

SocketTimestamps represet the timestamps generated by the NIC (Hardware) and the Kernel (Software) that is parse from the control message.

func ParseSocketTimestamps

func ParseSocketTimestamps(msg unix.SocketControlMessage) (SocketTimestamps, error)

ParseSocketTimestamps parses the timestamp information from the control message it will also convert Unix epoch times to Go's zero value time to be able to use .IsZero() on timestamps that have a default value or are not available. https://www.kernel.org/doc/html/v5.14/networking/timestamping.html#scm-timestamping-records

type Stats

type Stats struct {
	// Packets are the total number of packets received
	Packets uint32

	// Drops are the number of packets dropped
	Drops uint32

	// FreezeQueueCount is the total number of times that a receive queue is
	// frozen. May be zero if the kernel is not new enough to support TPACKET_V3
	FreezeQueueCount uint32
}

Stats are the statistics obtained from the kernel.