e2e

README

End-to-End Testing Framework

Testify

We rely on testify to provide the structure for our end-to-end testing. This allows us to decouple where tests are defined, from where they are run.

TestCluster

A TestCluster is the structure that manages tests running against a single Kubernetes Cluster.

Its sole responsibility is to create TestInstallations.

TestInstallation

A TestInstallation is the structure that manages a group of tests that run against an installation of Gloo Gateway, within a Kubernetes Cluster.

We try to define a single TestInstallation per file in a TestCluster. This way, it is easy to identify what behaviors are expected for that installation.

Features

We define all tests in the features package. This is done for a variety of reasons:

1. We group the tests by feature, so it's easy to identify which behaviors we assert for a given feature.
2. We can invoke that same test against different TestInstallations. This means we can test a feature against a variety of installation values, or even against OSS and Enterprise installations.

Many examples of testing features may be found in the features package. The general pattern for adding a new feature should be to create a directory for the feature under features/, write manifest files for the resources the tests will need into features/my_feature/testdata/, define Go objects for them in a file called features/my_feature/types.go, and finally define the test suite in features/my_feature/suite.go. There are occasions where multiple suites will need to be created under a single feature. See Suites for more info on this case.

Test Suites

A Test Suite is a subset of the Feature concept. A single Feature has at minimum one Test Suite, and can have many. Each Test Suite should have its own appropriately named .go file from which is exported an appropriately named function which satisfies the signature NewSuiteFunc found in suite.go.

These test suites are registered by a name and this func in Tests to be run against various TestInstallations.

Tests

This package holds the entry point for each of our TestInstallation.

See Load balancing tests for more information about how these tests are run in CI.

Each *_test.go file contains a specific test installation and exists within the tests_test package. In order for tests to be imported and run from other repos, each *_test.go file has a corresponding *_test.go file which exists in the tests package. This is done because _test packages cannot be imported.

In order to add a feature suite to be run in a given test installation, it must be added to the exported function in the corresponding *_tests.go file. e.g. In order to add a feature suite to be run with the test installation defined in istio_test.go, we have to register it by adding it to IstioTests() in istio_tests.go following the existing paradigm.

Adding Tests to CI

When writing new tests, they should be added to the the Kubernetes Tests that run on all PRs if they are not already covered by an existing regex. This way we ensure parity between PR runs and nightlies. Additionally they should be added to the OSS Tests which run in Enterprise that verify that the feature works in Enterprise as well. When adding it to the list, ensure that the tests are load balanced to allow quick iteration on PRs and update the date and the duration of corresponding test. The only exception to this is the Upgrade tests that are not run on the main branch but all LTS branches.

Environment Variables

Some tests may require environment variables to be set. Some required env vars are:

• Istio features: Require ISTIO_VERSION to be set. The tests running in CI use ISTIO_VERSION="${ISTIO_VERSION:-1.19.9}" to default to a specific version of Istio.

Debugging

Refer to the Debugging guide for more information on how to debug tests.

Thanks

Inspiration

This framework was inspired by the following projects:

• Kubernetes Gateway API
• Gloo Platform

Areas of Improvement

Help Wanted: This framework is not feature complete, and we welcome any improvements to it.

Below are a set of known areas of improvement. The goal is to provide a starting point for developers looking to contribute. There are likely other improvements that are not currently captured, so please add/remove entries to this list as you see fit:

• Debug Improvements: On test failure, we should emit a report about the entire state of the cluster. This should be a CLI utility as well.
• Curl assertion: We need a re-usable way to execute Curl requests against a Pod, and assert properties of the response.
• Improved install action(s): We rely on the SoloTestHelper currently, and it would be nice if we relied directly on Helm or Glooctl.
• Cluster provisioning: We rely on the setup-kind script to provision a cluster. We should make this more flexible by providing a configurable, declarative way to do this.
• Istio action: We need a way to perform Istio actions against a cluster.
• Argo action: We need an easy utility to perform ArgoCD commands against a cluster.

Documentation

Index

• Variables
• func ManifestPath(pathParts ...string) string
• func MustTestHelper(ctx context.Context, installation *TestInstallation) *helper.SoloTestHelper
• func ProfilePath(path string) string
• type GeneratedFiles
  • func MustGeneratedFiles(tmpDirId, clusterId string) GeneratedFiles
• type NewSuiteFunc
• type SuiteRunner
  • func NewSuiteRunner(ordered bool) SuiteRunner
• type TestInstallation
  • func CreateTestInstallation(t *testing.T, glooGatewayContext *gloogateway.Context) *TestInstallation
  • func CreateTestInstallationForCluster(t *testing.T, runtimeContext testruntime.Context, ...) *TestInstallation
  • func (i *TestInstallation) AddIstioctl(ctx context.Context) error
  • func (i *TestInstallation) CreateIstioBugReport(ctx context.Context)
  • func (i *TestInstallation) InstallGlooGateway(ctx context.Context, installFn func(ctx context.Context) error)
  • func (i *TestInstallation) InstallGlooGatewayWithTestHelper(ctx context.Context, testHelper *helper.SoloTestHelper, timeout time.Duration)
  • func (i *TestInstallation) InstallMinimalIstio(ctx context.Context) error
  • func (i *TestInstallation) InstallRevisionedIstio(ctx context.Context, rev, profile string) error
  • func (i *TestInstallation) PreFailHandler(ctx context.Context)
  • func (i *TestInstallation) String() string
  • func (i *TestInstallation) UninstallGlooGateway(ctx context.Context, uninstallFn func(ctx context.Context) error)
  • func (i *TestInstallation) UninstallGlooGatewayWithTestHelper(ctx context.Context, testHelper *helper.SoloTestHelper)
  • func (i *TestInstallation) UninstallIstio() error

Variables

var (
	EdgeGatewayProfilePath = ProfilePath("edge-gateway.yaml")

	KubernetesGatewayProfilePath = ProfilePath("kubernetes-gateway.yaml")

	FullGatewayProfilePath = ProfilePath("full-gateway.yaml")

	CommonRecommendationManifest = ManifestPath("common-recommendations.yaml")

	// EmptyValuesManifestPath returns the path to a manifest with no values
	// We prefer to have our tests be explicit and require defining a values file. However, some tests
	// rely entirely on the values provided by the "profile". In those cases, the test supplies this reference
	EmptyValuesManifestPath = ManifestPath("empty-values.yaml")
)

Functions

func ManifestPath

func ManifestPath(pathParts ...string) string

ManifestPath returns the absolute path to a manifest file. These are all stored in the tests/manifests directory

func MustTestHelper

func MustTestHelper(ctx context.Context, installation *TestInstallation) *helper.SoloTestHelper

MustTestHelper returns the SoloTestHelper used for e2e tests The SoloTestHelper is a wrapper around `glooctl` and we should eventually phase it out in favor of using the exact tool that users rely on

func ProfilePath

func ProfilePath(path string) string

ProfilePath returns the absolute path to a profile file. These are all stored in the tests/manifests/profiles directory

Types

type GeneratedFiles

type GeneratedFiles struct {
	// TempDir is the directory where any temporary files should be created
	// Tests may create files for any number of reasons:
	// - A: When a test renders objects in a file, and then uses this file to create and delete values
	// - B: When a test invokes a command that produces a file as a side effect (glooctl, for example)
	// Files in this directory are an implementation detail of the test itself.
	// As a result, it is the callers responsibility to clean up the TempDir when the tests complete
	TempDir string

	// FailureDir is the directory where any assets that are produced on failure will be created
	FailureDir string
}

GeneratedFiles is a collection of files that are generated during the execution of a set of tests

func MustGeneratedFiles

func MustGeneratedFiles(tmpDirId, clusterId string) GeneratedFiles

MustGeneratedFiles returns GeneratedFiles, or panics if there was an error generating the directories

type NewSuiteFunc

type NewSuiteFunc func(ctx context.Context, testInstallation *TestInstallation) suite.TestingSuite

type SuiteRunner

type SuiteRunner interface {
	Run(ctx context.Context, t *testing.T, testInstallation *TestInstallation)
	Register(name string, newSuite NewSuiteFunc)
}

A SuiteRunner is an interface that allows E2E tests to simply Register tests in one location and execute them with Run.

func NewSuiteRunner

func NewSuiteRunner(ordered bool) SuiteRunner

NewSuiteRunner returns an implementation of TestRunner that will execute tests as specified in the ordered parameter.

NOTE: it should be strongly preferred to use unordered tests. Only pass true to this function if there is a clear need for the tests to be ordered, and specify in a comment near the call to NewSuiteRunner why the tests need to be ordered.

type TestInstallation

type TestInstallation struct {
	fmt.Stringer

	// RuntimeContext contains the set of properties that are defined at runtime by whoever is invoking tests
	RuntimeContext testruntime.Context

	// ClusterContext contains the metadata about the Kubernetes Cluster that is used for this TestCluster
	ClusterContext *cluster.Context

	// Metadata contains the properties used to install Gloo Gateway
	Metadata *gloogateway.Context

	// ResourceClients is a set of clients that can manipulate resources owned by Gloo Gateway
	ResourceClients gloogateway.ResourceClients

	// Actions is the entity that creates actions that can be executed by the Operator
	Actions *actions.Provider

	// Assertions is the entity that creates assertions that can be executed by the Operator
	Assertions *assertions.Provider

	// GeneratedFiles is the collection of directories and files that this test installation _may_ create
	GeneratedFiles GeneratedFiles

	// IstioctlBinary is the path to the istioctl binary that can be used to interact with Istio
	IstioctlBinary string
}

TestInstallation is the structure around a set of tests that validate behavior for an installation of Gloo Gateway.

func CreateTestInstallation

func CreateTestInstallation(
	t *testing.T,
	glooGatewayContext *gloogateway.Context,
) *TestInstallation

CreateTestInstallation is the simplest way to construct a TestInstallation in Gloo Gateway OSS It is syntactic sugar on top of CreateTestInstallationForCluster

func CreateTestInstallationForCluster

func CreateTestInstallationForCluster(
	t *testing.T,
	runtimeContext testruntime.Context,
	clusterContext *cluster.Context,
	glooGatewayContext *gloogateway.Context,
) *TestInstallation

CreateTestInstallationForCluster is the standard way to construct a TestInstallation It accepts context objects from 3 relevant sources:

runtime - These are properties that are supplied at runtime and will impact how tests are executed
cluster - These are properties that are used to connect to the Kubernetes cluster
glooGateway - These are properties that are relevant to how Gloo Gateway will be configured

func (*TestInstallation) AddIstioctl

func (i *TestInstallation) AddIstioctl(ctx context.Context) error

func (*TestInstallation) CreateIstioBugReport

func (i *TestInstallation) CreateIstioBugReport(ctx context.Context)

func (*TestInstallation) InstallGlooGateway

func (i *TestInstallation) InstallGlooGateway(ctx context.Context, installFn func(ctx context.Context) error)

func (*TestInstallation) InstallGlooGatewayWithTestHelper

func (i *TestInstallation) InstallGlooGatewayWithTestHelper(ctx context.Context, testHelper *helper.SoloTestHelper, timeout time.Duration)

InstallGlooGatewayWithTestHelper is the common way to install Gloo Gateway. However, it relies on a SoloTestHelper which is an artifact of the legacy e2e tests that we hope to deprecate

func (*TestInstallation) InstallMinimalIstio

func (i *TestInstallation) InstallMinimalIstio(ctx context.Context) error

func (*TestInstallation) InstallRevisionedIstio

func (i *TestInstallation) InstallRevisionedIstio(ctx context.Context, rev, profile string) error

func (*TestInstallation) PreFailHandler

func (i *TestInstallation) PreFailHandler(ctx context.Context)

PreFailHandler is the function that is invoked if a test in the given TestInstallation fails

func (*TestInstallation) String

func (i *TestInstallation) String() string

func (*TestInstallation) UninstallGlooGateway

func (i *TestInstallation) UninstallGlooGateway(ctx context.Context, uninstallFn func(ctx context.Context) error)

func (*TestInstallation) UninstallGlooGatewayWithTestHelper

func (i *TestInstallation) UninstallGlooGatewayWithTestHelper(ctx context.Context, testHelper *helper.SoloTestHelper)

UninstallGlooGatewayWithTestHelper is the common way to uninstall Gloo Gateway. However, it relies on a SoloTestHelper which is an artifact of the legacy e2e tests that we hope to deprecate

func (*TestInstallation) UninstallIstio

func (i *TestInstallation) UninstallIstio() error