clustertest

README

clustertest

A test framework for helping with E2E testing of Giant Swarm clusters.

Installation

go get github.com/giantswarm/clustertest

Features

• Kubernetes client for interacting with the Management Cluster
• Kubernetes client for interacting with the Workload Clusters
• Wrapper types around Apps and their respective values ConfigMaps
• Wrapper types around Cluster apps (and their default-apps)
• Management (creation and deletion) or Organization resources
• Wait and polling helpers
• Override App versions using environment variables. (See application documentation for details)

Supported Environment Variables

The following environment variables can be set to control or override different behaviour:

Env Var	Description
E2E_KUBECONFIG	Points to the file containing the kubeconfig to use for the Management Clusters
E2E_WC_NAME	The name of an existing Workload Cluster to load from the Management Cluster instead of creating a new one. Must be used with E2E_WC_NAMESPACE
E2E_WC_NAMESPACE	The namespace an existing Workload Cluster is in which will be loaded instead of creating a new one. Must be used with E2E_WC_NAME
E2E_WC_KEEP	This environment variable is used to indicate that the workload cluster should not be deleted at the end of a test run. Note: not used within this codebase but exposed for use by other tooling.
E2E_OVERRIDE_VERSIONS	Sets the version of Apps to use instead of installing the latest released. Example format: E2E_OVERRIDE_VERSIONS="cluster-aws=0.38.0-5f4372ac697fce58d524830a985ede2082d7f461"
E2E_RELEASE_VERSION	The base Release version to use when creating the Workload Cluster. Must be used with E2E_RELEASE_COMMIT
E2E_RELEASE_COMMIT	The git commit from the releases repo that contains the Release version to use when creating the Workload Cluster. Must be used with E2E_RELEASE_VERSION
E2E_RELEASE_PRE_UPGRADE	Intended to be used in E2E tests to indicate what Release version to make use of before performing an upgade to a newer Release. Note: not used within this codebase but exposed for use by tests.

All of these can be found in ./pkg/env/const.go.

Documentation

Documentation can be found at: pkg.go.dev/github.com/giantswarm/clustertest.

Example Usage

ctx := context.Background()

framework, err := clustertest.New("capa_standard")
if err != nil {
  panic(err)
}

cluster := application.NewClusterApp(utils.GenerateRandomName("t"), application.ProviderAWS).
  WithOrg(organization.NewRandomOrg()).
  WithAppValuesFile(
    path.Clean("./test_data/cluster_values.yaml"),
    path.Clean("./test_data/default-apps_values.yaml"),
  )

client, err := framework.ApplyCluster(ctx, cluster)

// Run tests...

err = framework.DeleteCluster(ctx, cluster)

Documentation

Overview

Example using an existing Workload Cluster

ctx := context.Background()

framework, err := clustertest.New("context_name")
if err != nil {
	panic(err)
}

// The E2E_WC_NAME and E2E_WC_NAMESPACE env vars must be exported
cluster, err := framework.LoadCluster()
if err != nil {
	panic(err)
}
if cluster == nil {
	// Handle the case where the env vars aren't provided
}

// Run tests...
// No need to clean up as the user is responsible for the cluster

Example Using Ginkgo

func TestCAPA(t *testing.T) {
	var err error
	ctx := context.Background()

	framework, err = clustertest.New("context_name")
	if err != nil {
		panic(err)
	}
	logger.LogWriter = GinkgoWriter

	cluster = application.NewClusterApp(utils.GenerateRandomName("t"), application.ProviderAWS).
		WithOrg(organization.NewRandomOrg()).
		WithAppVersions("", ""). // If not set, the latest is fetched
		WithAppValuesFile(path.Clean("./test_data/cluster_values.yaml"), path.Clean("./test_data/default-apps_values.yaml"))

	BeforeSuite(func() {
		client, err := framework.ApplyCluster(ctx, cluster)
		Expect(err).To(BeNil())

		Eventually(
			wait.AreNumNodesReady(ctx, client, 3, &cr.MatchingLabels{"node-role.kubernetes.io/control-plane": ""}),
			20*time.Minute,
			30*time.Second,
		).Should(BeTrue())
	})

	AfterSuite(func() {
		err := framework.DeleteCluster(ctx, cluster)
		Expect(err).To(BeNil())
	})

	RegisterFailHandler(Fail)
	RunSpecs(t, "CAPA Suite")
}

Index

• type Framework
  • func New(contextName string) (*Framework, error)
  • func (f *Framework) ApplyBuiltCluster(ctx context.Context, builtCluster *application.BuiltCluster) (*client.Client, error)
  • func (f *Framework) ApplyCluster(ctx context.Context, cluster *application.Cluster) (*client.Client, error)
  • func (f *Framework) CreateOrg(ctx context.Context, org *organization.Org) error
  • func (f *Framework) DeleteCluster(ctx context.Context, cluster *application.Cluster) error
  • func (f *Framework) DeleteOrg(ctx context.Context, org *organization.Org) error
  • func (f *Framework) GetApp(ctx context.Context, name, namespace string) (*applicationv1alpha1.App, error)
  • func (f *Framework) GetAppAndValues(ctx context.Context, name, namespace string) (*applicationv1alpha1.App, *corev1.ConfigMap, error)
  • func (f *Framework) GetConfigMap(ctx context.Context, name, namespace string) (*corev1.ConfigMap, error)
  • func (f *Framework) GetExpectedControlPlaneReplicas(ctx context.Context, clusterName string, clusterNamespace string) (int32, error)
  • func (f *Framework) GetKubeadmControlPlane(ctx context.Context, clusterName string, clusterNamespace string) (*kubeadm.KubeadmControlPlane, error)
  • func (f *Framework) GetMachinePools(ctx context.Context, clusterName string, clusterNamespace string) ([]capiexp.MachinePool, error)
  • func (f *Framework) LoadCluster() (*application.Cluster, error)
  • func (f *Framework) MC() *client.Client
  • func (f *Framework) WC(clusterName string) (*client.Client, error)
  • func (f *Framework) WaitForClusterReady(ctx context.Context, clusterName string, namespace string) (*client.Client, error)
  • func (f *Framework) WaitForControlPlane(ctx context.Context, c *client.Client, expectedNodes int) error

Types

type Framework

type Framework struct {
	// contains filtered or unexported fields
}

Framework is the overall framework for testing of clusters

func New

func New(contextName string) (*Framework, error)

New initializes a new Framework instance using the provided context from the kubeconfig found in the env var `E2E_KUBECONFIG`

func (*Framework) ApplyBuiltCluster

func (f *Framework) ApplyBuiltCluster(ctx context.Context, builtCluster *application.BuiltCluster) (*client.Client, error)

ApplyBuiltCluster takes a pre-built Cluster object, applies it to the MC in the correct order and then waits for a valid Kubeconfig to be available

A timeout can be provided via the given `ctx` value by using `context.WithTimeout()`

Example:

timeoutCtx, cancelTimeout := context.WithTimeout(context.Background(), 20*time.Minute)
defer cancelTimeout()

cluster := application.NewClusterApp(utils.GenerateRandomName("t"), application.ProviderAWS)
builtCluster, _ := cluster.Build()
client, err := framework.ApplyBuiltCluster(timeoutCtx, builtCluster)

func (*Framework) ApplyCluster

func (f *Framework) ApplyCluster(ctx context.Context, cluster *application.Cluster) (*client.Client, error)

ApplyCluster takes a Cluster object, builds it, then applies it to the MC in the correct order and then waits for a valid Kubeconfig to be available

A timeout can be provided via the given `ctx` value by using `context.WithTimeout()`

Example:

timeoutCtx, cancelTimeout := context.WithTimeout(context.Background(), 20*time.Minute)
defer cancelTimeout()

cluster := application.NewClusterApp(utils.GenerateRandomName("t"), application.ProviderAWS)

client, err := framework.ApplyCluster(timeoutCtx, cluster)

func (*Framework) CreateOrg

func (f *Framework) CreateOrg(ctx context.Context, org *organization.Org) error

CreateOrg create a new Organization in the MC (which then triggers the creation of the org namespace)

func (*Framework) DeleteCluster

func (f *Framework) DeleteCluster(ctx context.Context, cluster *application.Cluster) error

DeleteCluster removes the Cluster app from the MC

func (*Framework) DeleteOrg

func (f *Framework) DeleteOrg(ctx context.Context, org *organization.Org) error

DeleteOrg deletes an Organization from the MC, waiting for all Clusters in the org namespace to be deleted first

func (*Framework) GetApp

func (f *Framework) GetApp(ctx context.Context, name, namespace string) (*applicationv1alpha1.App, error)

GetApp gets the App resource from the cluster

func (*Framework) GetAppAndValues

func (f *Framework) GetAppAndValues(ctx context.Context, name, namespace string) (*applicationv1alpha1.App, *corev1.ConfigMap, error)

GetAppAndValues will return the specified App CR and uservalues ConfigMap from the Management Cluster

func (*Framework) GetConfigMap

func (f *Framework) GetConfigMap(ctx context.Context, name, namespace string) (*corev1.ConfigMap, error)

GetConfigMap gets a ConfigMap from the cluster

func (*Framework) GetExpectedControlPlaneReplicas

func (f *Framework) GetExpectedControlPlaneReplicas(ctx context.Context, clusterName string, clusterNamespace string) (int32, error)

GetExpectedControlPlaneReplicas returns the number of control plane node expected according to the clusters KubeadmControlPlane resource

func (*Framework) GetKubeadmControlPlane

func (f *Framework) GetKubeadmControlPlane(ctx context.Context, clusterName string, clusterNamespace string) (*kubeadm.KubeadmControlPlane, error)

GetKubeadmControlPlane returns the KubeadmControlPlane resource. If we don't find the `KubeadmControlPlane` we assume it's a managed control plane cluster and expect nil pointer to be returned without error.

func (*Framework) GetMachinePools

func (f *Framework) GetMachinePools(ctx context.Context, clusterName string, clusterNamespace string) ([]capiexp.MachinePool, error)

GetMachinePools returns the MachinePool resources. If we don't find the `MachinePools` we assume that the provider is not using MachinePools, so nil pointer is returned without error.

func (*Framework) LoadCluster

func (f *Framework) LoadCluster() (*application.Cluster, error)

LoadCluster will construct a Cluster struct using a Workload Cluster's cluster and default-apps App CRs on the targeted Management Cluster. The name and namespace where the cluster are installed need to be provided with the E2E_WC_NAME and E2E_WC_NAMESPACE env vars.

If one of the env vars are not set, a nil Cluster and nil error will be returned.

Example:

cluster, err := framework.LoadCluster()
if err != nil {
	// handle error
}
if cluster == nil {
	// handle cluster not provided
}

func (*Framework) MC

func (f *Framework) MC() *client.Client

MC returns an initialized client for the Management Cluster

func (*Framework) WC

func (f *Framework) WC(clusterName string) (*client.Client, error)

WC returns an initialized client for the Workload Cluster matching the given name. If no Workload Cluster is found matching the given name an error is returned.

func (*Framework) WaitForClusterReady

func (f *Framework) WaitForClusterReady(ctx context.Context, clusterName string, namespace string) (*client.Client, error)

WaitForClusterReady watches for a Kubeconfig secret to be created on the MC and then waits until that cluster's api-server response successfully

A timeout can be provided via the given `ctx` value by using `context.WithTimeout()`

Example:

timeoutCtx, cancelTimeout := context.WithTimeout(context.Background(), 20*time.Minute)
defer cancelTimeout()

wcClient, err := framework.WaitForClusterReady(timeoutCtx, "test-cluster", "default")

func (*Framework) WaitForControlPlane

func (f *Framework) WaitForControlPlane(ctx context.Context, c *client.Client, expectedNodes int) error

WaitForControlPlane polls the provided cluster and waits until the provided number of Control Plane nodes are reporting as ready

Example:

timeoutCtx, cancelTimeout := context.WithTimeout(context.Background(), 20*time.Minute)
defer cancelTimeout()

err := framework.WaitForControlPlane(timeoutCtx, wcClient, 3)