README ¶
How to use IPVS
This document shows how to use kube-proxy ipvs mode.
What is IPVS
IPVS (IP Virtual Server) implements transport-layer load balancing, usually called Layer 4 LAN switching, as part of Linux kernel.
IPVS runs on a host and acts as a load balancer in front of a cluster of real servers. IPVS can direct requests for TCP and UDP-based services to the real servers, and make services of real servers appear as irtual services on a single IP address.
How to use
Load IPVS kernel modules
Currently the IPVS kernel module can't be loaded automatically, so first we should use the following command to load IPVS kernel modules manually.
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
After that, use lsmod | grep ip_vs
to make sure kernel modules are loaded.
Run kube-proxy in ipvs mode
Local UP Cluster
Kube-proxy will run in iptables mode by default in a local-up cluster.
Users should export the env KUBEPROXY_MODE=ipvs
to specify the ipvs mode before deploying the cluster if want to run kube-proxy in ipvs mode.
Cluster Created by Kubeadm
Kube-proxy will run in iptables mode by default in a cluster deployed by kubeadm.
Since IPVS mode is still feature-gated, users should add the flag --feature-gates=SupportIPVSProxyMode=true
in kubeadm init
command
kubeadm init --feature-gates=SupportIPVSProxyMode=true
to specify the ipvs mode before deploying the cluster if want to run kube-proxy in ipvs mode.
If you are using kubeadm with a configuration file, you can specify the ipvs mode adding SupportIPVSProxyMode: true
below the featureGates
field.
Then the configuration file is similar to:
kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
...
featureGates:
SupportIPVSProxyMode: true
...
Test
Use ipvsadm
tool to test whether the kube-proxy start succeed. By default we may get result like:
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.1:443 rr persistent 10800
-> 10.229.43.2:6443 Masq 1 0 0
TCP 10.0.0.10:53 rr
UDP 10.0.0.10:53 rr
Documentation ¶
Index ¶
- Constants
- func CanUseIPVSProxier(ipsetver IPSetVersioner) (bool, error)
- func CleanupLeftovers(ipvs utilipvs.Interface, ipt utiliptables.Interface, ipset utilipset.Interface, ...) (encounteredError bool)
- type IPGetter
- type IPSet
- type IPSetVersioner
- type NetLinkHandle
- type Proxier
- func (proxier *Proxier) OnEndpointsAdd(endpoints *api.Endpoints)
- func (proxier *Proxier) OnEndpointsDelete(endpoints *api.Endpoints)
- func (proxier *Proxier) OnEndpointsSynced()
- func (proxier *Proxier) OnEndpointsUpdate(oldEndpoints, endpoints *api.Endpoints)
- func (proxier *Proxier) OnServiceAdd(service *api.Service)
- func (proxier *Proxier) OnServiceDelete(service *api.Service)
- func (proxier *Proxier) OnServiceSynced()
- func (proxier *Proxier) OnServiceUpdate(oldService, service *api.Service)
- func (proxier *Proxier) Sync()
- func (proxier *Proxier) SyncLoop()
Constants ¶
const ( // MinIPSetCheckVersion is the min ipset version we need. IPv6 is supported in ipset 6.x MinIPSetCheckVersion = "6.0" // KubeLoopBackIPSet is used to store endpoints dst ip:port, source ip for solving hairpin purpose. KubeLoopBackIPSet = "KUBE-LOOP-BACK" // KubeClusterIPSet is used to store service cluster ip + port for masquerade purpose. KubeClusterIPSet = "KUBE-CLUSTER-IP" // KubeExternalIPSet is used to store service external ip + port for masquerade and filter purpose. KubeExternalIPSet = "KUBE-EXTERNAL-IP" // KubeLoadBalancerSet is used to store service load balancer ingress ip + port, it is the service lb portal. KubeLoadBalancerSet = "KUBE-LOAD-BALANCER" // KubeLoadBalancerMasqSet is used to store service load balancer ingress ip + port for masquerade purpose. KubeLoadBalancerMasqSet = "KUBE-LOAD-BALANCER-MASQ" // KubeLoadBalancerSourceIPSet is used to store service load balancer ingress ip + port + source IP for packet filter purpose. KubeLoadBalancerSourceIPSet = "KUBE-LOAD-BALANCER-SOURCE-IP" // KubeLoadBalancerSourceCIDRSet is used to store service load balancer ingress ip + port + source cidr for packet filter purpose. KubeLoadBalancerSourceCIDRSet = "KUBE-LOAD-BALANCER-SOURCE-CIDR" // KubeNodePortSetTCP is used to store nodeport TCP port for masquerade purpose. KubeNodePortSetTCP = "KUBE-NODE-PORT-TCP" // KubeNodePortSetUDP is used to store nodeport UDP port for masquerade purpose. KubeNodePortSetUDP = "KUBE-NODE-PORT-UDP" )
const ( // KubeServiceIPSetsChain is the services access IP chain KubeServiceIPSetsChain utiliptables.Chain = "KUBE-SVC-IPSETS" // KubeFireWallChain is the kubernetes firewall chain. KubeFireWallChain utiliptables.Chain = "KUBE-FIRE-WALL" // KubeMarkMasqChain is the mark-for-masquerade chain KubeMarkMasqChain utiliptables.Chain = "KUBE-MARK-MASQ" // KubeMarkDropChain is the mark-for-drop chain KubeMarkDropChain utiliptables.Chain = "KUBE-MARK-DROP" // DefaultScheduler is the default ipvs scheduler algorithm - round robin. DefaultScheduler = "rr" // DefaultDummyDevice is the default dummy interface where ipvs service address will bind to it. DefaultDummyDevice = "kube-ipvs0" )
Variables ¶
This section is empty.
Functions ¶
func CanUseIPVSProxier ¶
func CanUseIPVSProxier(ipsetver IPSetVersioner) (bool, error)
CanUseIPVSProxier returns true if we can use the ipvs Proxier. This is determined by checking if all the required kernel modules can be loaded. It may return an error if it fails to get the kernel modules information without error, in which case it will also return false.
func CleanupLeftovers ¶
func CleanupLeftovers(ipvs utilipvs.Interface, ipt utiliptables.Interface, ipset utilipset.Interface, cleanupIPVS bool) (encounteredError bool)
CleanupLeftovers clean up all ipvs and iptables rules created by ipvs Proxier.
Types ¶
type IPSetVersioner ¶ added in v1.9.0
IPSetVersioner can query the current ipset version.
type NetLinkHandle ¶ added in v1.9.0
type NetLinkHandle interface { // EnsureAddressBind checks if address is bound to the interface and, if not, binds it. If the address is already bound, return true. EnsureAddressBind(address, devName string) (exist bool, err error) // UnbindAddress unbind address from the interface UnbindAddress(address, devName string) error // EnsureDummyDevice checks if dummy device is exist and, if not, create one. If the dummy device is already exist, return true. EnsureDummyDevice(devName string) (exist bool, err error) // DeleteDummyDevice deletes the given dummy device by name. DeleteDummyDevice(devName string) error }
NetLinkHandle for revoke netlink interface
func NewNetLinkHandle ¶ added in v1.9.0
func NewNetLinkHandle() NetLinkHandle
NewNetLinkHandle will crate a new netlinkHandle
type Proxier ¶
type Proxier struct {
// contains filtered or unexported fields
}
Proxier is an ipvs based proxy for connections between a localhost:lport and services that provide the actual backends.
func NewProxier ¶
func NewProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset utilipset.Interface, sysctl utilsysctl.Interface, exec utilexec.Interface, syncPeriod time.Duration, minSyncPeriod time.Duration, masqueradeAll bool, masqueradeBit int, clusterCIDR string, hostname string, nodeIP net.IP, recorder record.EventRecorder, healthzServer healthcheck.HealthzUpdater, scheduler string, ) (*Proxier, error)
NewProxier returns a new Proxier given an iptables and ipvs Interface instance. Because of the iptables and ipvs logic, it is assumed that there is only a single Proxier active on a machine. An error will be returned if it fails to update or acquire the initial lock. Once a proxier is created, it will keep iptables and ipvs rules up to date in the background and will not terminate if a particular iptables or ipvs call fails.
func (*Proxier) OnEndpointsAdd ¶
OnEndpointsAdd is called whenever creation of new endpoints object is observed.
func (*Proxier) OnEndpointsDelete ¶
OnEndpointsDelete is called whenever deletion of an existing endpoints object is observed.
func (*Proxier) OnEndpointsSynced ¶
func (proxier *Proxier) OnEndpointsSynced()
OnEndpointsSynced is called once all the initial event handlers were called and the state is fully propagated to local cache.
func (*Proxier) OnEndpointsUpdate ¶
OnEndpointsUpdate is called whenever modification of an existing endpoints object is observed.
func (*Proxier) OnServiceAdd ¶
OnServiceAdd is called whenever creation of new service object is observed.
func (*Proxier) OnServiceDelete ¶
OnServiceDelete is called whenever deletion of an existing service object is observed.
func (*Proxier) OnServiceSynced ¶
func (proxier *Proxier) OnServiceSynced()
OnServiceSynced is called once all the initial even handlers were called and the state is fully propagated to local cache.
func (*Proxier) OnServiceUpdate ¶
OnServiceUpdate is called whenever modification of an existing service object is observed.