ipvs

README

How to use IPVS

This document shows how to use kube-proxy ipvs mode.

What is IPVS

IPVS (IP Virtual Server) implements transport-layer load balancing, usually called Layer 4 LAN switching, as part of Linux kernel.

IPVS runs on a host and acts as a load balancer in front of a cluster of real servers. IPVS can direct requests for TCP and UDP-based services to the real servers, and make services of real servers appear as irtual services on a single IP address.

How to use

Load IPVS kernel modules

Currently the IPVS kernel module can't be loaded automatically, so first we should use the following command to load IPVS kernel modules manually.

modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4

After that, use lsmod | grep ip_vs to make sure kernel modules are loaded.

Run kube-proxy in ipvs mode

First, run cluster locally.

By default kube-proxy will run in iptables mode, with configuration file /tmp/kube-proxy.yaml. so we need to change the configuration file and restart it. Here is a yaml file for reference.

apiVersion: componentconfig/v1alpha1
kind: KubeProxyConfiguration
clientConnection:
  kubeconfig: /var/run/kubernetes/kube-proxy.kubeconfig
hostnameOverride: 127.0.0.1
mode: ipvs
featureGates: AllAlpha=true
ipvs:
  minSyncPeriod: 10s
  syncPeriod: 60s

Test

Use ipvsadm tool to test whether the kube-proxy start succeed. By default we may get result like:

# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.1:443 rr persistent 10800
  -> 10.229.43.2:6443             Masq    1      0          0         
TCP  10.0.0.10:53 rr      
UDP  10.0.0.10:53 rr

Documentation

Index

• Constants
• func CanUseIPVSProxier() (bool, error)
• func CleanupLeftovers(execer utilexec.Interface, ipvs utilipvs.Interface, ipt utiliptables.Interface) (encounteredError bool)
• type IPGetter
• type NetLinkHandle
  • func NewNetLinkHandle() NetLinkHandle
• type Proxier
  • func NewProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ...) (*Proxier, error)
  • func (proxier *Proxier) OnEndpointsAdd(endpoints *api.Endpoints)
  • func (proxier *Proxier) OnEndpointsDelete(endpoints *api.Endpoints)
  • func (proxier *Proxier) OnEndpointsSynced()
  • func (proxier *Proxier) OnEndpointsUpdate(oldEndpoints, endpoints *api.Endpoints)
  • func (proxier *Proxier) OnServiceAdd(service *api.Service)
  • func (proxier *Proxier) OnServiceDelete(service *api.Service)
  • func (proxier *Proxier) OnServiceSynced()
  • func (proxier *Proxier) OnServiceUpdate(oldService, service *api.Service)
  • func (proxier *Proxier) Sync()
  • func (proxier *Proxier) SyncLoop()

Constants

const (

	// KubeMarkMasqChain is the mark-for-masquerade chain
	KubeMarkMasqChain utiliptables.Chain = "KUBE-MARK-MASQ"

	// KubeMarkDropChain is the mark-for-drop chain
	KubeMarkDropChain utiliptables.Chain = "KUBE-MARK-DROP"
)

const (
	// DefaultScheduler is the default ipvs scheduler algorithm - round robin.
	DefaultScheduler = "rr"
	// DefaultDummyDevice is the default dummy interface where ipvs service address will bind to it.
	DefaultDummyDevice = "kube-ipvs0"
)

Functions

func CanUseIPVSProxier

func CanUseIPVSProxier() (bool, error)

CanUseIPVSProxier returns true if we can use the ipvs Proxier. This is determined by checking if all the required kernel modules are loaded. It may return an error if it fails to get the kernel modules information without error, in which case it will also return false.

func CleanupLeftovers

func CleanupLeftovers(execer utilexec.Interface, ipvs utilipvs.Interface, ipt utiliptables.Interface) (encounteredError bool)

CleanupLeftovers clean up all ipvs and iptables rules created by ipvs Proxier.

Types

type IPGetter

type IPGetter interface {
	NodeIPs() ([]net.IP, error)
}

IPGetter helps get node network interface IP

type NetLinkHandle

type NetLinkHandle interface {
	// EnsureAddressBind checks if address is bound to the interface and, if not, binds it. If the address is already bound, return true.
	EnsureAddressBind(address, devName string) (exist bool, err error)
	// UnbindAddress unbind address from the interface
	UnbindAddress(address, devName string) error
}

NetLinkHandle for revoke netlink interface

func NewNetLinkHandle

func NewNetLinkHandle() NetLinkHandle

NewNetLinkHandle will crate a new netlinkHandle

type Proxier

type Proxier struct {
	// contains filtered or unexported fields
}

Proxier is an ipvs based proxy for connections between a localhost:lport and services that provide the actual backends.

func NewProxier

func NewProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface,
	sysctl utilsysctl.Interface,
	exec utilexec.Interface,
	syncPeriod time.Duration,
	minSyncPeriod time.Duration,
	masqueradeAll bool,
	masqueradeBit int,
	clusterCIDR string,
	hostname string,
	nodeIP net.IP,
	recorder record.EventRecorder,
	healthzServer healthcheck.HealthzUpdater,
	scheduler string,
) (*Proxier, error)

NewProxier returns a new Proxier given an iptables and ipvs Interface instance. Because of the iptables and ipvs logic, it is assumed that there is only a single Proxier active on a machine. An error will be returned if it fails to update or acquire the initial lock. Once a proxier is created, it will keep iptables and ipvs rules up to date in the background and will not terminate if a particular iptables or ipvs call fails.

func (*Proxier) OnEndpointsAdd

func (proxier *Proxier) OnEndpointsAdd(endpoints *api.Endpoints)

OnEndpointsAdd is called whenever creation of new endpoints object is observed.

func (*Proxier) OnEndpointsDelete

func (proxier *Proxier) OnEndpointsDelete(endpoints *api.Endpoints)

OnEndpointsDelete is called whenever deletion of an existing endpoints object is observed.

func (*Proxier) OnEndpointsSynced

func (proxier *Proxier) OnEndpointsSynced()

OnEndpointsSynced is called once all the initial event handlers were called and the state is fully propagated to local cache.

func (*Proxier) OnEndpointsUpdate

func (proxier *Proxier) OnEndpointsUpdate(oldEndpoints, endpoints *api.Endpoints)

OnEndpointsUpdate is called whenever modification of an existing endpoints object is observed.

func (*Proxier) OnServiceAdd

func (proxier *Proxier) OnServiceAdd(service *api.Service)

OnServiceAdd is called whenever creation of new service object is observed.

func (*Proxier) OnServiceDelete

func (proxier *Proxier) OnServiceDelete(service *api.Service)

OnServiceDelete is called whenever deletion of an existing service object is observed.

func (*Proxier) OnServiceSynced

func (proxier *Proxier) OnServiceSynced()

OnServiceSynced is called once all the initial even handlers were called and the state is fully propagated to local cache.

func (*Proxier) OnServiceUpdate

func (proxier *Proxier) OnServiceUpdate(oldService, service *api.Service)

OnServiceUpdate is called whenever modification of an existing service object is observed.

func (*Proxier) Sync

func (proxier *Proxier) Sync()

Sync is called to synchronize the proxier state to iptables and ipvs as soon as possible.

func (*Proxier) SyncLoop

func (proxier *Proxier) SyncLoop()

SyncLoop runs periodic work. This is expected to run as a goroutine or as the main loop of the app. It does not return.