README
¶
Kubernetes End-to-End tests
This directory contains tests that install each of the 3 Gloo Edge flavors (gateway
, ingress
, and knative
) in a Kubernetes cluster, and run
end-to-end tests against them.
Note: All commands should be run from the root directory of the Gloo repository
Background
Kubernetes may be relied on for scheduling, persistence or security. These tests validate that Gloo Edge can successfully operate within a Kubernetes cluster.
How do the tests work?
- Install Gloo Edge in Kubernetes cluster using Helm
- Apply Gloo resources using Kubernetes resource clients
- Execute requests against the Envoy proxy and confirm the expected response. This validates that the Gloo resources have been picked up by the controllers, were been translated correctly into Envoy configuration, the configuration was sent to the Envoy proxy, and the proxy behaves appropriately.
CI
These tests are run by a GitHub action as part of our CI pipeline.
If a test fails, you can retry it from a browser window. If you do this, please make sure to comment on the Pull Request with a link to the failed logs for debugging purposes.
Local Development
Setup
For these tests to run, we require the following conditions:
- Gloo Edge Helm chart archive be present in the
_test
folder, glooctl
be built in the_output
folder- kind cluster set up and loaded with the images to be installed by the helm chart
Use the CI Install Script
ci/deploy-to-kind-cluster.sh gets run in CI to setup the test environment for the above requirements. It accepts a number of environment variables, to control the creation of a kind cluster and deployment of Gloo resources to that kind cluster.
Name | Default | Description |
---|---|---|
CLUSTER_NAME | kind | The name of the cluster that will be generated |
CLUSTER_NODE_VERSION | v1.22.4 | The version of the Node Docker image to use for booting the cluster |
VERSION | 1.0.0-ci | The version used to tag Gloo images that are deployed to the cluster |
KUBE2E_TESTS | gateway | Name of the test suite to be run. Options: 'gateway', 'gloo', 'ingress', 'helm', 'gloomtls', 'glooctl', 'upgrade' |
SKIP_DOCKER | false | Skip building docker images (used when testing a release version) |
Example:
CLUSTER_NAME=solo-test-cluster CLUSTER_NODE_VERSION=v1.22.4 VERSION=v1.0.0-solo-test ci/deploy-to-kind-cluster.sh
Verify Your Setup
Before running your tests, it's worthwhile to verify that a cluster was created, and the proper images have been loaded.
CLUSTER_NAME=solo-test-cluster make kind-list-images
You should see the list of images in the cluster, including the ones you just uploaded
Common Setup Errors
Error: validation: chart.metadata.version "solo" is invalid
In newer versions of helm (>3.5), the version used to build the helm chart (ie the VERSION env variable), needs to respect semantic versioning. This error implies that the version provided does not.
Prepend a valid semver to avoid the error. (ie kind
can become 0.0.0-kind1
)
Run Tests
To run the regression tests, your kubeconfig file must point to a running Kubernetes cluster.
kubectl config current-context
should run kind-<CLUSTER_NAME>
Use the Make Target
Use the same command that CI relies on:
KUBE2E_TESTS=<test-to-run> make run-ci-regression-tests
Test Environment Variables
The below table contains the environment variables that can be used to configure the test execution.
Name | Default | Description |
---|---|---|
KUBE2E_TESTS | gateway | Name of the test suite to be run. Options: 'gateway', 'gloo', 'ingress', 'helm', 'gloomtls', 'glooctl', 'upgrade' |
DEBUG | 0 | Set to 1 for debug log output |
WAIT_ON_FAIL | 0 | Set to 1 to prevent Ginkgo from cleaning up the Gloo Edge installation in case of failure. Useful to exec into inspect resources created by the test. A command to resume the test run (and thus clean up resources) will be logged to the output. |
TEAR_DOWN | false | Set to true to uninstall Gloo after the test suite completes |
Common Test Errors
getting Helm chart version: expected a single entry with name [gloo], found: 5
The test helm charts are written to the _test
directory, with the index.yaml
file containing references to all available charts. The tests require that this file contain only 1 entry. Delete the other entries manually, or run make clean
to delete this folder entirely, and then re-build the test helm chart.
Documentation
¶
Index ¶
- Constants
- func EventuallyLogLevel(port int, logLevel zapcore.Level)
- func EventuallyMetricsBecomeConsistent(offset int, metricsPort int)
- func EventuallyReachesConsistentState(installNamespace string)
- func ExpectEqualProtoMessages(g Gomega, a, b proto.Message, optionalDescription ...interface{})
- func GetHelmValuesOverrideFile() (filename string, cleanup func())
- func GetHttpEchoImage() string
- func GetSimpleTestRunnerHttpResponse() string
- func GetTestHelper(ctx context.Context, namespace string) (*helper.SoloTestHelper, error)
- func GetTestReleasedVersion(ctx context.Context, repoName string) string
- func GlooctlCheckEventuallyHealthy(offset int, testHelper *helper.SoloTestHelper, timeoutInterval string)
- func MustKubeClient() kubernetes.Interface
- func ToFile(content string) string
- func UpdateAlwaysAcceptSetting(ctx context.Context, alwaysAccept bool, installNamespace string)
- func UpdateDisableTransformationValidationSetting(ctx context.Context, shouldDisable bool, installNamespace string)
- func UpdateReplaceInvalidRoutes(ctx context.Context, replaceInvalidRoutes bool, installNamespace string)
- func UpdateRestEdsSetting(ctx context.Context, enableRestEds bool, installNamespace string)
- func UpdateSettings(ctx context.Context, updateSettings func(settings *v1.Settings), ...)
- func UpdateSettingsWithPropagationDelay(updateSettings func(settings *v1.Settings), waitForSettingsToPropagate func(), ...)
- type KubeResourceClientSet
- func (k KubeResourceClientSet) ArtifactClient() gloov1.ArtifactClient
- func (k KubeResourceClientSet) GatewayClient() gatewayv1.GatewayClient
- func (k KubeResourceClientSet) HttpGatewayClient() gatewayv1.MatchableHttpGatewayClient
- func (k KubeResourceClientSet) KubeClients() *kubernetes.Clientset
- func (k KubeResourceClientSet) ProxyClient() gloov1.ProxyClient
- func (k KubeResourceClientSet) RateLimitConfigClient() externalrl.RateLimitConfigClient
- func (k KubeResourceClientSet) RouteOptionClient() gatewayv1.RouteOptionClient
- func (k KubeResourceClientSet) RouteTableClient() gatewayv1.RouteTableClient
- func (k KubeResourceClientSet) SecretClient() gloov1.SecretClient
- func (k KubeResourceClientSet) ServiceClient() skkube.ServiceClient
- func (k KubeResourceClientSet) SettingsClient() gloov1.SettingsClient
- func (k KubeResourceClientSet) UpstreamClient() gloov1.UpstreamClient
- func (k KubeResourceClientSet) UpstreamGroupClient() gloov1.UpstreamGroupClient
- func (k KubeResourceClientSet) VirtualHostOptionClient() gatewayv1.VirtualHostOptionClient
- func (k KubeResourceClientSet) VirtualServiceClient() gatewayv1.VirtualServiceClient
Constants ¶
const SimpleTestRunnerHttpResponse = `` /* 851-byte string literal not displayed */
const SimpleTestRunnerHttpResponseArm = `` /* 807-byte string literal not displayed */
Variables ¶
This section is empty.
Functions ¶
func EventuallyLogLevel ¶ added in v1.9.21
EventuallyLogLevel ensures that we can query the endpoint responsible for getting the current log level of a gloo component, and updating the log level dynamically
func EventuallyMetricsBecomeConsistent ¶ added in v1.9.25
func EventuallyReachesConsistentState ¶ added in v1.5.0
func EventuallyReachesConsistentState(installNamespace string)
func ExpectEqualProtoMessages ¶ added in v1.9.13
Copied from: https://github.com/solo-io/go-utils/blob/176c4c008b4d7cde836269c7a817f657b6981236/testutils/assertions.go#L20
func GetHelmValuesOverrideFile ¶ added in v1.3.16
func GetHelmValuesOverrideFile() (filename string, cleanup func())
func GetHttpEchoImage ¶ added in v1.9.25
func GetHttpEchoImage() string
func GetSimpleTestRunnerHttpResponse ¶ added in v1.9.25
func GetSimpleTestRunnerHttpResponse() string
https://github.com/solo-io/gloo/issues/4043#issuecomment-772706604 We should move tests away from using the testrunner, and instead depend on EphemeralContainers. The default response changed in later kube versions, which caused this value to change. Ideally the test utilities used by Gloo are maintained in the Gloo repo, so I opted to move this constant here. This response is given by the testrunner when the SimpleServer is started
func GetTestHelper ¶ added in v1.12.48
func GetTestReleasedVersion ¶ added in v1.12.48
For nightly runs, we want to install a released version rathher than using a locally built chart To do this, set the environment variable RELEASED_VERSION with either a version name or "LATEST" to get the last release
func GlooctlCheckEventuallyHealthy ¶ added in v1.3.16
func GlooctlCheckEventuallyHealthy(offset int, testHelper *helper.SoloTestHelper, timeoutInterval string)
GlooctlCheckEventuallyHealthy will run up until proved timeoutInterval or until gloo is reported as healthy
func MustKubeClient ¶ added in v0.11.2
func MustKubeClient() kubernetes.Interface
func UpdateAlwaysAcceptSetting ¶ added in v1.5.0
enable/disable strict validation
func UpdateDisableTransformationValidationSetting ¶ added in v1.8.3
func UpdateReplaceInvalidRoutes ¶ added in v1.8.3
func UpdateRestEdsSetting ¶ added in v1.7.0
func UpdateSettings ¶ added in v1.5.0
Types ¶
type KubeResourceClientSet ¶ added in v1.12.21
type KubeResourceClientSet struct {
// contains filtered or unexported fields
}
KubeResourceClientSet is a set of ResourceClients
func NewKubeResourceClientSet ¶ added in v1.9.25
func (KubeResourceClientSet) ArtifactClient ¶ added in v1.12.21
func (k KubeResourceClientSet) ArtifactClient() gloov1.ArtifactClient
func (KubeResourceClientSet) GatewayClient ¶ added in v1.12.21
func (k KubeResourceClientSet) GatewayClient() gatewayv1.GatewayClient
func (KubeResourceClientSet) HttpGatewayClient ¶ added in v1.12.21
func (k KubeResourceClientSet) HttpGatewayClient() gatewayv1.MatchableHttpGatewayClient
func (KubeResourceClientSet) KubeClients ¶ added in v1.12.21
func (k KubeResourceClientSet) KubeClients() *kubernetes.Clientset
func (KubeResourceClientSet) ProxyClient ¶ added in v1.12.21
func (k KubeResourceClientSet) ProxyClient() gloov1.ProxyClient
func (KubeResourceClientSet) RateLimitConfigClient ¶ added in v1.12.21
func (k KubeResourceClientSet) RateLimitConfigClient() externalrl.RateLimitConfigClient
func (KubeResourceClientSet) RouteOptionClient ¶ added in v1.12.21
func (k KubeResourceClientSet) RouteOptionClient() gatewayv1.RouteOptionClient
func (KubeResourceClientSet) RouteTableClient ¶ added in v1.12.21
func (k KubeResourceClientSet) RouteTableClient() gatewayv1.RouteTableClient
func (KubeResourceClientSet) SecretClient ¶ added in v1.12.21
func (k KubeResourceClientSet) SecretClient() gloov1.SecretClient
func (KubeResourceClientSet) ServiceClient ¶ added in v1.12.21
func (k KubeResourceClientSet) ServiceClient() skkube.ServiceClient
func (KubeResourceClientSet) SettingsClient ¶ added in v1.12.21
func (k KubeResourceClientSet) SettingsClient() gloov1.SettingsClient
func (KubeResourceClientSet) UpstreamClient ¶ added in v1.12.21
func (k KubeResourceClientSet) UpstreamClient() gloov1.UpstreamClient
func (KubeResourceClientSet) UpstreamGroupClient ¶ added in v1.12.21
func (k KubeResourceClientSet) UpstreamGroupClient() gloov1.UpstreamGroupClient
func (KubeResourceClientSet) VirtualHostOptionClient ¶ added in v1.12.21
func (k KubeResourceClientSet) VirtualHostOptionClient() gatewayv1.VirtualHostOptionClient
func (KubeResourceClientSet) VirtualServiceClient ¶ added in v1.12.21
func (k KubeResourceClientSet) VirtualServiceClient() gatewayv1.VirtualServiceClient