xvs

package module
v0.1.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2024 License: GPL-2.0 Imports: 19 Imported by: 1

README

XDP Virtual Server

An XDP/eBPF load balancer and Go API for Linux.

This code is originally from the VC5 load balancer, and has been split out to be developed seperately.

The eBPF object file is committed to this repository which means that it can be used as a standard Go module without having to build the binary as a seperate step. libbpf is still required but this will be linked in the standard CGO manner by setting the CGO_CFLAGS and CGO_LDFLAGS environment variables to the location of the library (see the Makefile for an example of how to do this).

The code implements an IPv4 layer 2 Direct Server Return load balancer. Backend servers need to be on the same VLAN as the load balancer. Multiple VLANs/interfaces are supported.

SUpport for layer 3 DSR and IPv6 is planned.

NOTICE

I've reworked the library, prompted by issues getting the eBPF code to run under the verifier on Ubuntu 22.04. The counters for /20 prefixes all return zero currently, but I think that the code is a lot better than previously.

Now works on Raspberry Pi - although devices with constrained memory might have have issues loading in the default size flow state tables, so you have have to rebuild the eBPF object file (see the raspberrypi target in the Makefile for dependencies).

My wi-fi load balancer:

cmd/balancer wlan0 192.168.0.16 192.168.101.1 192.168.0.10 192.168.0.11

Documentation

https://pkg.go.dev/github.com/davidcoles/xvs

The API is based on the Cloudflare IPVS library (Go reference).

Sample application

A simple application in the cmd/ directory will balance traffic to a VIP (TCP port 80 by default, can be changed with flags) to a number of backend servers on the same IP subnet.

Compile/run with:

  • make example
  • cmd/balancer ens192 10.1.2.3 192.168.101.1 10.1.2.10 10.1.2.11 10.1.2.12

Replace ens192 with your ethernet interface name, 10.1.2.3 with the address of the machine you are running the program on, 192.168.101.1 with the VIP you want to use and 10.1.2.10-12 with any number of real server addresses.

On a seperate client machine on the same subnet you should add a static route for the VIP, eg.:

  • ip r add 192.168.101.1 via 10.1.2.3

You should then be able to contact the service:

  • curl http://192.168.101.1/

No healthchecking is done, so you'll have to make sure that a webserver is running on the real servers and that the VIP has been configured on the loopback address on them (ip a add 192.168.101.1 dev lo).

A more complete example with health check and BGP route health injection is currently available at VC5.

Performance

This has mostly been tested using Icecast backend servers with clients pulling a mix of low and high bitrate streams (48kbps - 192kbps).

A VMWare guest (4 core, 8GB) using the XDP generic driver was able to support 100K concurrent clients, 380Mbps/700Kpps through the load balancer and 8Gbps of traffic from the backends directly to the clients. Going above 700Kpps cause connections to be dropped, regardless of the number of cores or memory assigned to the VM, so I suspect that there is a limit on the number of interrupts that the VM is able to handle per second.

On a single (non-virtualised) Intel Xeon Gold 6314U CPU (2.30GHz 32 physical cores, with hyperthreading enabled for 64 logical cores) and an Intel 10G 4P X710-T4L-t ethernet card, I was able to run 700K streams at 2Gbps/3.8Mpps ingress traffic and 46.5Gbps egress. The server was more than 90% idle. Unfortunately I did not have the resources available to create more clients/servers. I realised that I carried this out when the server's profile was set to performance per-watt. Using the performance mode the CPU usage is barely 2% and latencey is less than 250 nanoseconds.

On a Raspberry Pi (B+) ... don't get your hopes up!

Documentation

Index

Constants

View Source 
const (
	F_NO_SHARE_FLOWS    = bpf.F_NO_SHARE_FLOWS
	F_NO_TRACK_FLOWS    = bpf.F_NO_TRACK_FLOWS
	F_NO_ESTIMATE_CONNS = bpf.F_NO_ESTIMATE_CONNS
	F_NO_STORE_STATS    = bpf.F_NO_STORE_STATS
)
View Source 
const PREFIXES = 1048576
View Source 
const VETH = bpf.VETH_ID

Variables

This section is empty.

Functions

This section is empty.

Types

type Client 

type Client struct {
	NAT        bool
	Native     bool
	Interfaces []string
	Address    netip.Addr
	VLANs      map[uint16]net.IPNet
	Debug      Debug

	Redirect bool // obsolete
	Share    bool // obsolete
	Logger   any  // obsolete
	// contains filtered or unexported fields
}

func (*Client) Block added in v0.1.13 

func (c *Client) Block(b [PREFIXES]bool)

func (*Client) CreateDestination 

func (c *Client) CreateDestination(s Service, d Destination) error

func (*Client) CreateService 

func (c *Client) CreateService(s Service) error

func (*Client) Destinations 

func (c *Client) Destinations(s Service) (destinations []DestinationExtended, e error)

func (*Client) Flags added in v0.1.11 

func (c *Client) Flags(f uint8)

func (*Client) Info 

func (c *Client) Info() (i Info)

func (*Client) NATAddress 

func (c *Client) NATAddress(vip, rip netip.Addr) (r netip.Addr, _ bool)

func (*Client) Namespace 

func (c *Client) Namespace() string

func (*Client) NamespaceAddress 

func (c *Client) NamespaceAddress() string

func (*Client) Prefixes 

func (c *Client) Prefixes() [PREFIXES]uint64

func (*Client) ReadFlow added in v0.1.3 

func (c *Client) ReadFlow() []byte

func (*Client) RemoveDestination 

func (c *Client) RemoveDestination(s Service, d Destination) error

func (*Client) RemoveService 

func (c *Client) RemoveService(s Service) error

func (*Client) Service 

func (c *Client) Service(s Service) (se ServiceExtended, e error)

func (*Client) Services 

func (c *Client) Services() (services []ServiceExtended, e error)

func (*Client) SetService added in v0.1.2 

func (c *Client) SetService(s Service, dst ...Destination) error

func (*Client) Start 

func (c *Client) Start() error

func (*Client) UpdateVLANs 

func (c *Client) UpdateVLANs(vlans map[uint16]net.IPNet)

func (*Client) WriteFlow added in v0.1.3 

func (c *Client) WriteFlow(fs []byte)

Make a flow returned from ReadFlow() known to the local eBPF program.

type Debug added in v0.1.11 

type Debug interface {
	NAT(tag map[netip.Addr]int16, arp map[netip.Addr][6]byte, vrn map[[2]netip.Addr]netip.Addr, nat map[netip.Addr]string, out []netip.Addr, in []string)
	Redirects(vlans map[uint16]string)
	Backend(vip netip.Addr, port uint16, protocol uint8, backends []byte, took time.Duration)
}

type Destination 

type Destination struct {
	Address netip.Addr // Destinationserver IP address
	Weight  uint8      // Not full implemented; 0 - don't use, non-zero enables the destination
	// contains filtered or unexported fields
}

type DestinationExtended 

type DestinationExtended struct {
	Destination Destination
	MAC         MAC
	Stats       Stats
}

type Info 

type Info struct {
	Packets   uint64
	Octets    uint64
	Flows     uint64
	Latency   uint64
	Dropped   uint64
	Blocked   uint64
	NotQueued uint64
}

type MAC 

type MAC = mac

type Protocol 

type Protocol = uint8
const (
	TCP Protocol = 0x06
	UDP Protocol = 0x11
)

type Service 

type Service struct {
	Address  netip.Addr // The Virtual IP address of the service
	Port     uint16     // Layer 4 port number
	Protocol Protocol   // IP protocol number; TCP (6), and UDP (17) are current supported
	Sticky   bool       // Only use source and destination IP addresses when determining backend
}

type ServiceExtended 

type ServiceExtended struct {
	Service Service
	Stats   Stats
}

type Stats 

type Stats struct {
	Packets uint64
	Octets  uint64
	Flows   uint64
	Current uint64
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.