kube2e

README

Kubernetes End-to-End tests

This directory contains tests that install each of the 3 Gloo Edge flavors (gateway, ingress, and knative) in a Kubernetes cluster, and run end-to-end tests against them.

Note: All commands should be run from the root directory of the Gloo repository

Background

Kubernetes may be relied on for scheduling, persistence or security. These tests validate that Gloo Edge can successfully operate within a Kubernetes cluster.

How do the tests work?

1. Install Gloo Edge in Kubernetes cluster using Helm
2. Apply Gloo resources using Kubernetes resource clients
3. Execute requests against the Envoy proxy and confirm the expected response. This validates that the Gloo resources have been picked up by the controllers, were been translated correctly into Envoy configuration, the configuration was sent to the Envoy proxy, and the proxy behaves appropriately.

CI

These tests are run by a GitHub action as part of our CI pipeline.

If a test fails, you can retry it from a browser window. If you do this, please make sure to comment on the Pull Request with a link to the failed logs for debugging purposes.

Local Development

Setup

For these tests to run, we require the following conditions:

• Gloo Edge Helm chart archive be present in the _test folder,
• glooctl be built in the_output folder
• kind cluster set up and loaded with the images to be installed by the helm chart

(Option A) - Use the CI Install Script (preferred)

ci/deploy-to-kind-cluster.sh gets run in CI to setup the test environment for the above requirements. It accepts a number of environment variables, to control the creation of a kind cluster and deployment of Gloo resources to that kind cluster.

Name	Default	Description
CLUSTER_NAME	kind	The name of the cluster that will be generated
CLUSTER_NODE_VERSION	v1.22.4	The version of the Node Docker image to use for booting the cluster
VERSION	0.0.0-kind	The version used to tag Gloo images that are deployed to the cluster
KUBE2E_TESTS	gateway	Name of the test suite to be run. Options: 'gateway', 'gloo', 'ingress', 'helm', 'gloomtls', 'glooctl'

Example:

CLUSTER_NAME=solo-test-cluster CLUSTER_NODE_VERSION=v1.22.4 VERSION=v1.0.0-solo-test ci/deploy-to-kind-cluster.sh

(Option B) - Manually Run Make Targets

The CI install script executes a series of make targets.

Create a kind cluster: kind create cluster --name kind
Build the helm chart: VERSION=0.0.0-kind make build-test-chart
Build glooctl: make glooctl
Load the images into the cluster: CLUSTER_NAME=kind VERSION=0.0.0-kind make push-kind-images

Verify Your Setup

Before running your tests, it's worthwhile to verify that a cluster was created, and the proper images have been loaded.

kubectl get nodes
docker exec -ti <nodename> crictl images

You should see the list of images in the cluster, including the ones you just uploaded

Common Setup Errors

Error: validation: chart.metadata.version "solo" is invalid
In newer versions of helm (>3.5), the version used to build the helm chart (ie the VERSION env variable), needs to respect semantic versioning. This error implies that the version provided does not.

Prepend a valid semver to avoid the error. (ie kind can become 0.0.0-kind)

Run Tests

To run the regression tests, your kubeconfig file must point to a running Kubernetes cluster. kubectl config current-context should run kind-<CLUSTER_NAME>

(Option A) - Use the Make Target (preferred)

Use the same command that CI relies on:

KUBE2E_TESTS=<test-to-run> make run-ci-regression-tests

(Option B) - Use Ginkgo Directly

The make target just runs ginkgo with a set of useful flags. If you want to control the flags that are provided, you can run:

KUBE2E_TESTS=<test-to-run> ginkgo -r <other-flags>

Test Environment Variables

The below table contains the environment variables that can be used to configure the test execution.

Name	Default	Description
KUBE2E_TESTS	gateway	Name of the test suite to be run. Options: 'gateway', 'gloo', 'ingress', 'helm', 'gloomtls', 'glooctl'
DEBUG	0	Set to 1 for debug log output
WAIT_ON_FAIL	0	Set to 1 to prevent Ginkgo from cleaning up the Gloo Edge installation in case of failure. Useful to exec into inspect resources created by the test. A command to resume the test run (and thus clean up resources) will be logged to the output.
TEAR_DOWN	false	Set to true to uninstall Gloo after the test suite completes

Common Test Errors

getting Helm chart version: expected a single entry with name [gloo], found: 5
The test helm charts are written to the _test directory, with the index.yaml file containing references to all available charts. The tests require that this file contain only 1 entry. Delete the other entries manually, or run make clean to delete this folder entirely, and then re-build the test helm chart.

Documentation

Index

• Constants
• func EventuallyLogLevel(port int, logLevel zapcore.Level)
• func EventuallyMetricsBecomeConsistent(offset int, metricsPort int)
• func EventuallyReachesConsistentState(installNamespace string)
• func ExpectEqualProtoMessages(g Gomega, a, b proto.Message, optionalDescription ...interface{})
• func GetHelmValuesOverrideFile() (filename string, cleanup func())
• func GetHttpEchoImage() string
• func GetSimpleTestRunnerHttpResponse() string
• func GlooctlCheckEventuallyHealthy(offset int, testHelper *helper.SoloTestHelper, timeoutInterval string)
• func MustKubeClient() kubernetes.Interface
• func UpdateAlwaysAcceptSetting(ctx context.Context, alwaysAccept bool, installNamespace string)
• func UpdateDisableTransformationValidationSetting(ctx context.Context, shouldDisable bool, installNamespace string)
• func UpdateReplaceInvalidRoutes(ctx context.Context, replaceInvalidRoutes bool, installNamespace string)
• func UpdateRestEdsSetting(ctx context.Context, enableRestEds bool, installNamespace string)
• func UpdateSettings(ctx context.Context, updateSettings func(settings *v1.Settings), ...)
• func UpdateSettingsWithPropagationDelay(updateSettings func(settings *v1.Settings), waitForSettingsToPropagate func(), ...)
• type ResourceClientSet
  • func NewKubeResourceClientSet(ctx context.Context, cfg *rest.Config) (*ResourceClientSet, error)
  • func (r ResourceClientSet) DeleteSnapshot(ctx context.Context, snapshot *gloosnapshot.ApiSnapshot) error
  • func (r ResourceClientSet) WriteSnapshot(ctx context.Context, snapshot *gloosnapshot.ApiSnapshot) error

Constants

const SimpleTestRunnerHttpResponse = `` /* 851-byte string literal not displayed */

const SimpleTestRunnerHttpResponseArm = `` /* 807-byte string literal not displayed */

Functions

func EventuallyLogLevel

func EventuallyLogLevel(port int, logLevel zapcore.Level)

EventuallyLogLevel ensures that we can query the endpoint responsible for getting the current log level of a gloo component, and updating the log level dynamically

func EventuallyMetricsBecomeConsistent

func EventuallyMetricsBecomeConsistent(offset int, metricsPort int)

func EventuallyReachesConsistentState

func EventuallyReachesConsistentState(installNamespace string)

func ExpectEqualProtoMessages

func ExpectEqualProtoMessages(g Gomega, a, b proto.Message, optionalDescription ...interface{})

Copied from: https://github.com/solo-io/go-utils/blob/176c4c008b4d7cde836269c7a817f657b6981236/testutils/assertions.go#L20

func GetHelmValuesOverrideFile

func GetHelmValuesOverrideFile() (filename string, cleanup func())

func GetHttpEchoImage

func GetHttpEchoImage() string

func GetSimpleTestRunnerHttpResponse

func GetSimpleTestRunnerHttpResponse() string

https://github.com/solo-io/gloo/issues/4043#issuecomment-772706604 We should move tests away from using the testrunner, and instead depend on EphemeralContainers. The default response changed in later kube versions, which caused this value to change. Ideally the test utilities used by Gloo are maintained in the Gloo repo, so I opted to move this constant here. This response is given by the testrunner when the SimpleServer is started

func GlooctlCheckEventuallyHealthy

func GlooctlCheckEventuallyHealthy(offset int, testHelper *helper.SoloTestHelper, timeoutInterval string)

Check that everything is OK by running `glooctl check`

func MustKubeClient

func MustKubeClient() kubernetes.Interface

func UpdateAlwaysAcceptSetting

func UpdateAlwaysAcceptSetting(ctx context.Context, alwaysAccept bool, installNamespace string)

enable/disable strict validation

func UpdateDisableTransformationValidationSetting

func UpdateDisableTransformationValidationSetting(ctx context.Context, shouldDisable bool, installNamespace string)

func UpdateReplaceInvalidRoutes

func UpdateReplaceInvalidRoutes(ctx context.Context, replaceInvalidRoutes bool, installNamespace string)

func UpdateRestEdsSetting

func UpdateRestEdsSetting(ctx context.Context, enableRestEds bool, installNamespace string)

func UpdateSettings

func UpdateSettings(ctx context.Context, updateSettings func(settings *v1.Settings), installNamespace string)

func UpdateSettingsWithPropagationDelay

func UpdateSettingsWithPropagationDelay(updateSettings func(settings *v1.Settings), waitForSettingsToPropagate func(), ctx context.Context, installNamespace string)

Types

type ResourceClientSet

type ResourceClientSet struct {
	GatewayClient           gatewayv1.GatewayClient
	HttpGatewayClient       gatewayv1.MatchableHttpGatewayClient
	VirtualServiceClient    gatewayv1.VirtualServiceClient
	RouteTableClient        gatewayv1.RouteTableClient
	VirtualHostOptionClient gatewayv1.VirtualHostOptionClient
	RouteOptionClient       gatewayv1.RouteOptionClient
	UpstreamGroupClient     gloov1.UpstreamGroupClient
	UpstreamClient          gloov1.UpstreamClient
	ProxyClient             gloov1.ProxyClient
	ServiceClient           skkube.ServiceClient
}

ResourceClientSet is a set of ResourceClients

func NewKubeResourceClientSet

func NewKubeResourceClientSet(ctx context.Context, cfg *rest.Config) (*ResourceClientSet, error)

func (ResourceClientSet) DeleteSnapshot

func (r ResourceClientSet) DeleteSnapshot(ctx context.Context, snapshot *gloosnapshot.ApiSnapshot) error

func (ResourceClientSet) WriteSnapshot

func (r ResourceClientSet) WriteSnapshot(ctx context.Context, snapshot *gloosnapshot.ApiSnapshot) error