e2e

README

End-to-End (E2E) Tests

E2E tests cover multi-node K3s configuration and administration: bringup, update, teardown etc. across a wide range of operating systems. E2E tests are run nightly as part of K3s quality assurance (QA).

Framework

End-to-end tests utilize Ginkgo and Gomega like the integration tests, but rely on Vagrant to provide the underlying cluster configuration.

Currently tested operating systems are:

• Ubuntu 20.04
• Leap 15.3 (stand-in for SLE-Server)
• MicroOS (stand-in for SLE-Micro)

Format

All E2E tests should be placed under tests/e2e/<TEST_NAME>.
All E2E test functions should be named: Test_E2E<TEST_NAME>.
A E2E test consists of two parts:

1. Vagrantfile: a vagrant file which describes and configures the VMs upon which the cluster and test will run
2. <TEST_NAME>.go: A go test file which calls vagrant up and controls the actual testing of the cluster

See the validate cluster test as an example.

Setup

To run the E2E tests, you must first install the following:

• Vagrant
• Libvirt
• Vagrant plugins

Vagrant

Download the latest version (currently 2.2.19) of Vagrant from the website. Do not use built-in packages, they often old or do not include the required ruby library extensions necessary to get certain plugins working.

Libvirt

Follow the OS specific guides to install libvirt/qemu on your host:

• openSUSE
• ubuntu 20.04
• ubuntu 22.04:

  sudo apt install ruby-libvirt qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev libguestfs-tools
  

• debian
• fedora

Vagrant plugins

Install the necessary vagrant plugins with the following command:

vagrant plugin install vagrant-libvirt vagrant-scp vagrant-k3s vagrant-reload

Kubectl

For linux

   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

If it does not work, or you are on a different system, check the official tutorial

Running

Generally, E2E tests are run as a nightly Jenkins job for QA. They can still be run locally but additional setup may be required. By default, all E2E tests are designed with libvirt as the underlying VM provider. Instructions for installing libvirt and its associated vagrant plugin, vagrant-libvirt can be found here. VirtualBox is also supported as a backup VM provider.

Once setup is complete, all E2E tests can be run with:

go test -timeout=15m ./tests/e2e/... -run E2E

Tests can be run individually with:

go test -timeout=15m ./tests/e2e/validatecluster/... -run E2E
#or
go test -timeout=15m ./tests/e2e/... -run E2EClusterValidation

Additionally, to generate junit reporting for the tests, the Ginkgo CLI is used. Installation instructions can be found here.

To run the all E2E tests and generate JUnit testing reports:

ginkgo --junit-report=result.xml ./tests/e2e/...

Note: The go test default timeout is 10 minutes, thus the -timeout flag should be used. The ginkgo default timeout is 1 hour, no timeout flag is needed.

Debugging

In the event of a test failure, the cluster and VMs are retained in their broken state. Startup logs are retained in vagrant.log.
To see a list of nodes: vagrant status
To ssh into a node: vagrant ssh <NODE>
Once you are done/ready to restart the test, use vagrant destroy -f to remove the broken cluster.

Documentation

Index

• func CountOfStringInSlice(str string, pods []Pod) int
• func CreateCluster(nodeOS string, serverCount, agentCount int) ([]string, []string, error)
• func CreateLocalCluster(nodeOS string, serverCount, agentCount int) ([]string, []string, error)
• func DeployWorkload(workload, kubeconfig string, hardened bool) (string, error)
• func DestroyCluster() error
• func FetchClusterIP(kubeconfig string, servicename string, dualStack bool) (string, error)
• func FetchIngressIP(kubeconfig string) ([]string, error)
• func FetchNodeExternalIP(nodename string) (string, error)
• func GenKubeConfigFile(serverName string) (string, error)
• func GenReport(specReport ginkgo.SpecReport)
• func GetCoverageReport(nodeNames []string) error
• func GetJournalLogs(node string) (string, error)
• func GetVagrantLog(cErr error) string
• func RestartCluster(nodeNames []string) error
• func RunCmdOnNode(cmd string, nodename string) (string, error)
• func RunCommand(cmd string) (string, error)
• func StartCluster(nodeNames []string) error
• func StopCluster(nodeNames []string) error
• func UpgradeCluster(nodeNames []string, local bool) error
• type Node
  • func ParseNodes(kubeConfig string, print bool) ([]Node, error)
  • func (n Node) String() string
• type NodeError
  • func (ne *NodeError) Error() string
  • func (ne *NodeError) Unwrap() error
• type ObjIP
  • func GetNodeIPs(kubeConfigFile string) ([]ObjIP, error)
  • func GetObjIPs(cmd string) ([]ObjIP, error)
  • func GetPodIPs(kubeConfigFile string) ([]ObjIP, error)
• type Pod
  • func ParsePods(kubeConfig string, print bool) ([]Pod, error)

Functions

func CountOfStringInSlice

func CountOfStringInSlice(str string, pods []Pod) int

func CreateCluster

func CreateCluster(nodeOS string, serverCount, agentCount int) ([]string, []string, error)

func CreateLocalCluster

func CreateLocalCluster(nodeOS string, serverCount, agentCount int) ([]string, []string, error)

CreateLocalCluster creates a cluster using the locally built k3s binary. The vagrant-scp plugin must be installed for this function to work. The binary is deployed as an airgapped install of k3s on the VMs. This is intended only for local testing purposes when writing a new E2E test.

func DeployWorkload

func DeployWorkload(workload, kubeconfig string, hardened bool) (string, error)

func DestroyCluster

func DestroyCluster() error

func FetchClusterIP

func FetchClusterIP(kubeconfig string, servicename string, dualStack bool) (string, error)

func FetchIngressIP

func FetchIngressIP(kubeconfig string) ([]string, error)

func FetchNodeExternalIP

func FetchNodeExternalIP(nodename string) (string, error)

func GenKubeConfigFile

func GenKubeConfigFile(serverName string) (string, error)

func GenReport

func GenReport(specReport ginkgo.SpecReport)

func GetCoverageReport

func GetCoverageReport(nodeNames []string) error

func GetJournalLogs

func GetJournalLogs(node string) (string, error)

func GetVagrantLog

func GetVagrantLog(cErr error) string

GetVagrantLog returns the logs of on vagrant commands that initialize the nodes and provision K3s on each node. It also attempts to fetch the systemctl logs of K3s on nodes where the k3s.service failed.

func RestartCluster

func RestartCluster(nodeNames []string) error

RestartCluster restarts the k3s service on each node given

func RunCmdOnNode

func RunCmdOnNode(cmd string, nodename string) (string, error)

RunCmdOnNode executes a command from within the given node as sudo

func RunCommand

func RunCommand(cmd string) (string, error)

func StartCluster

func StartCluster(nodeNames []string) error

StartCluster starts the k3s service on each node given

func StopCluster

func StopCluster(nodeNames []string) error

StopCluster starts the k3s service on each node given

func UpgradeCluster

func UpgradeCluster(nodeNames []string, local bool) error

Types

type Node

type Node struct {
	Name       string
	Status     string
	Roles      string
	InternalIP string
	ExternalIP string
}

func ParseNodes

func ParseNodes(kubeConfig string, print bool) ([]Node, error)

func (Node) String

func (n Node) String() string

type NodeError

type NodeError struct {
	Node string
	Cmd  string
	Err  error
}

func (*NodeError) Error

func (ne *NodeError) Error() string

func (*NodeError) Unwrap

func (ne *NodeError) Unwrap() error

type ObjIP

type ObjIP struct {
	Name string
	IPv4 string
	IPv6 string
}

func GetNodeIPs

func GetNodeIPs(kubeConfigFile string) ([]ObjIP, error)

getNodeIPs returns the IPs of all nodes

func GetObjIPs

func GetObjIPs(cmd string) ([]ObjIP, error)

getObjIPs executes a command to collect IPs

func GetPodIPs

func GetPodIPs(kubeConfigFile string) ([]ObjIP, error)

getPodIPs returns the IPs of all pods

type Pod

type Pod struct {
	NameSpace string
	Name      string
	Ready     string
	Status    string
	Restarts  string
	IP        string
	Node      string
}

func ParsePods

func ParsePods(kubeConfig string, print bool) ([]Pod, error)