test

README

Test

This directory contains tests and testing docs for Knative Serving:

• Unit tests currently reside in the codebase alongside the code they test
• End-to-end tests, of which there are two types:
  • Conformance tests in /test/conformance
  • Other end-to-end tests in /test/e2e
• Performance tests reside in /test/performance

The conformance tests are a subset of the end to end test with more strict requirements around what can be tested.

If you want to add more tests, see adding_tests.md.

Presubmit tests

presubmit-tests.sh is the entry point for both the end-to-end tests and the conformance tests

This script, and consequently, the e2e and conformance tests will be run before every code submission. You can run these tests manually with:

test/presubmit-tests.sh

Note that to run presubmit-tests.sh or e2e-tests.sh scripts, you'll need kubernetes kubetest installed:

go get -u k8s.io/test-infra/kubetest

Running unit tests

To run all unit tests:

go test ./...

By default go test will not run the e2e tests, which need -tags=e2e to be enabled.

Running end to end tests

To run the e2e tests, you need to have a running environment that meets the e2e test environment requirements, and you need to specify the build tag e2e.

go test -v -tags=e2e -count=1 ./test/e2e

Running conformance tests

To run the conformance tests, you need to have a running environment that meets the e2e test environment requirements, and you need to specify the build tag e2e.

go test -v -tags=e2e -count=1 ./test/conformance/...

Running conformance as a project admin

It is possible to run the conformance tests by a user with reduced privileges, e.g. project admin. The environment needs to meet similar requirements as in running conformance tests but the test resources can be limited to minimum and so the user can install them. Running the conformance tests then consists of these steps:

1. The cluster admin creates the cluster scope resources

   kubectl apply -f test/config/cluster-resources.yaml
   

2. The project admin installs minimum test resources:

   kubectl apply -f test/config/test-resources.yaml
   

3. The project admin then runs the conformance test suite using the --disable-logstream flag:

   go test -v -tags=e2e -count=1 ./test/conformance/... \
     -disable-logstream \
     -kubeconfig=$PROJECT_ADMIN_KUBECONFIG
   

The tests can be run in arbitrary test namespaces. If you modify the namespaces of the resources above you must pass the updated values to the Golang test suite:

go test -tags=e2e -count=1 ./test/conformance/... \
  -disable-logstream \
  -kubeconfig=$PROJECT_ADMIN_KUBECONFIG \
  -test-namespace=serving-tests \
  -alt-test-namespace=serving-tests-alt \
  -tls-test-namespace=tls

Running performance tests

Each performance test case in Knative serving is a benchmark, to run these benchmarks, please follow dev.md.

As of now, only Googlers can run these benchmarks due to one issue of Mako - the benchmarking tool we use. Details can be found in the issue report.

Running a single test case

To run one e2e test case, e.g. TestAutoscaleUpDownUp, use the -run flag with go test:

go test -v -tags=e2e -count=1 ./test/e2e -run ^TestAutoscaleUpDownUp$

Running tests in short mode

Running tests in short mode excludes some large-scale E2E tests and saves time/resources required for running the test suite. To run the tests in short mode, use the -short flag with go test

go test -v -tags=e2e -count=1 -short ./test/e2e

To get a better idea where the flag is used, search for testing.Short() throughout the test source code.

Environment requirements

These tests require:

1. A running Knative Serving cluster.

2. The knative-testing resources:

   ko apply -f test/config
   

3. A docker repo containing the test images

Common Flags

• By default the e2e tests against the current cluster in ~/.kube/config using the environment specified in your environment variables.
• Since these tests are fairly slow, running them with logging enabled is recommended (-v).
• Using -count=1 is the idiomatic way to disable test caching

You can use test flags to control the environment your tests run against, i.e. override your environment variables:

go test -v -tags=e2e -count=1 ./test/conformance/... --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
go test -v -tags=e2e -count=1 ./test/e2e --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo

Test images

Building the test images

Note: this is only required when you run conformance/e2e tests locally with go test commands.

The upload-test-images.sh script can be used to build and push the test images used by the conformance and e2e tests. The script expects your environment to be setup as described in DEVELOPMENT.md.

To run the script for all end to end test images:

./test/upload-test-images.sh

A docker tag may be passed as an optional parameter. This can be useful on Minikube in tandem with the --tag flag:

PLATFORM environment variable is optional. If it is specified, test images will be built for specific hardware architecture, according to its value (for instance,linux/arm64).

eval $(minikube docker-env)
./test/upload-test-images.sh any-old-tag

Adding new test images

New test images should be placed in ./test/test_images.

Flags

These flags are useful for running against an existing cluster, making use of your existing environment setup.

Tests importing knative.dev/serving/test recognize these flags:

• All flags added by knative/pkg/test such as:
  • --dockerrepo
  • --tag
  • --ingressendpoint
• --resolvabledomain
• --https

Overriding docker repo

The --dockerrepo argument lets you specify the docker repo from which images used by your tests should be pulled. This will default to the value of your KO_DOCKER_REPO environment variable if not specified.

go test -v -tags=e2e -count=1 ./test/conformance/... --dockerrepo gcr.myhappyproject
go test -v -tags=e2e -count=1 ./test/e2e --dockerrepo gcr.myhappyproject

Using a docker tag

The default docker tag used for the test images is latest, which can be problematic on Minikube. To avoid having to configure a remote container registry to support the Always pull policy for latest tags, you can have the tests use a specific tag:

go test -v -tags=e2e -count=1 ./test/conformance/... --tag any-old-tag
go test -v -tags=e2e -count=1 ./test/e2e --tag any-old-tag

Of course, this implies that you tagged the images when you uploaded them.

Using a custom ingress endpoint

Some environments (like minikube) do not support a Loadbalancer to make Knative services externally available. These environments usually rely on rewriting the Loadbalancer to a NodePort. The external address of such a NodePort is usually not easily obtained within the cluster automatically, but can be provided from the outside through the --ingressendpoint flag. For a minikube setup for example, you'd want to run tests against the default ingressgateway (port number 31380) running on the minikube node:

go test -v -tags=e2e -count=1 ./test/conformance/... --ingressendpoint "$(minikube ip):31380"
go test -v -tags=e2e -count=1 ./test/e2e --ingressendpoint "$(minikube ip):31380"

Using a resolvable domain

If you set up your cluster using the getting started docs, Routes created in the test will use the domain example.com, unless the route has label app=prod in which case they will use the domain prod-domain.com. Since these domains will not be resolvable to deployments in your test cluster, in order to make a request against the endpoint, the test use the IP assigned to the service istio-ingressgateway in the namespace istio-system and spoof the Host in the header.

If you have configured your cluster to use a resolvable domain, you can use the --resolvabledomain flag to indicate that the test should make requests directly against Route.Status.Domain and does not need to spoof the Host.

Overriding the gateway used for spoofing

If you are using an ingress provider other than Istio, and have not set up a resolvable domain (above), you will also need to set the GATEWAY_OVERRIDE and GATEWAY_NAMESPACE_OVERRIDE environment variables to allow the gateway to be discovered for spoofing. For example, for kourier you would do:

export GATEWAY_OVERRIDE=kourier
export GATEWAY_NAMESPACE_OVERRIDE=kourier-system

Using https

You can use the --https flag to have all tests run with https.

Documentation

Index

• Constants
• Variables
• func AddRootCAtoTransport(ctx context.Context, logf logging.FormatLogger, clients *Clients, https bool) spoof.TransportOption
• func AddTestAnnotation(t testing.TB, m metav1.ObjectMeta)
• func AssertProberDefault(t testing.TB, p Prober)
• func AssertProberSLO(t testing.TB, p Prober, slo float64)
• func CheckSLO(slo float64, name string, p Prober) error
• func CleanupOnInterrupt(cleanup func())
• func EnsureCleanup(t *testing.T, cleanup func())
• func EnsureTearDown(t *testing.T, clients *Clients, names *ResourceNames)
• func ListenAndServeGracefully(addr string, handler func(w http.ResponseWriter, r *http.Request))
• func ListenAndServeGracefullyWithHandler(addr string, handler http.Handler)
• func PemDataFromSecret(ctx context.Context, logf logging.FormatLogger, clients *Clients, ...) []byte
• func TLSClientConfig(ctx context.Context, logf logging.FormatLogger, clients *Clients) *tls.Config
• func TearDown(clients *Clients, names *ResourceNames)
• type Clients
  • func NewClients(cfg *rest.Config, namespace string) (*Clients, error)
  • func Setup(t testing.TB, namespace ...string) *Clients
• type NetworkingClients
• type Prober
  • func RunRouteProber(logf logging.FormatLogger, clients *Clients, url *url.URL, ...) Prober
• type ProberManager
  • func NewProberManager(logf logging.FormatLogger, clients *Clients, minProbes int64, ...) ProberManager
• type ResourceNames
• type ServingAlphaClients
• type ServingBetaClients
• type ServingClients
  • func (clients *ServingClients) Delete(routes, configs, services []string) []error
• type ServingEnvironmentFlags

Constants

const (
	// Test image names
	Autoscale           = "autoscale"
	EmptyDir            = "emptydir"
	Failing             = "failing"
	GRPCPing            = "grpc-ping"
	HelloHTTP2          = "hellohttp2"
	HelloVolume         = "hellovolume"
	HelloWorld          = "helloworld"
	HTTPProxy           = "httpproxy"
	InvalidHelloWorld   = "invalidhelloworld" // Not a real image
	PizzaPlanet1        = "pizzaplanetv1"
	PizzaPlanet2        = "pizzaplanetv2"
	Protocols           = "protocols"
	Readiness           = "readiness"
	Runtime             = "runtime"
	ServingContainer    = "servingcontainer"
	SidecarContainer    = "sidecarcontainer"
	SingleThreadedImage = "singlethreaded"
	Timeout             = "timeout"
	WorkingDir          = "workingdir"

	// Constants for test image output.
	PizzaPlanetText1 = "What a spaceport!"
	PizzaPlanetText2 = "Re-energize yourself with a slice of pepperoni!"
	HelloWorldText   = "Hello World! How about some tasty noodles?"
	HelloHTTP2Text   = "Hello, New World! How about donuts and coffee?"
	EmptyDirText     = "From file in empty dir!"

	MultiContainerResponse = "Yay!! multi-container works"

	ConcurrentRequests = 200
	// We expect to see 100% of requests succeed for traffic sent directly to revisions.
	// This might be a bad assumption.
	MinDirectPercentage = 1
	// We expect to see at least 25% of either response since we're routing 50/50.
	// The CDF of the binomial distribution tells us this will flake roughly
	// 1 time out of 10^12 (roughly the number of galaxies in the observable universe).
	MinSplitPercentage = 0.25
)

Constants for test images located in test/test_images.

const (

	// ConformanceConfigMap is the name of the configmap to propagate env variables from
	ConformanceConfigMap = "conformance-test-configmap"

	// ConformanceSecret is the name of the secret to propagate env variables from
	ConformanceSecret = "conformance-test-secret"

	// EnvKey is the configmap/secret key which contains test value
	EnvKey = "testKey"

	// EnvValue is the configmap/secret test value to match env variable with
	EnvValue = "testValue"
)

const (
	// PollInterval is how frequently e2e tests will poll for updates.
	PollInterval = 1 * time.Second
	// PollTimeout is how long e2e tests will wait for resource updates when polling.
	PollTimeout = 10 * time.Minute

	// HelloVolumePath is the path to the test volume.
	HelloVolumePath = "/hello/world"
)

Variables

var AppendRandomString = helpers.AppendRandomString

AppendRandomString will generate a random string that begins with prefix. This is useful if you want to make sure that your tests can run at the same time against the same environment without conflicting. This method will seed rand with the current time when called for the first time.

var ObjectNameForTest = helpers.ObjectNameForTest

ObjectNameForTest generates a random object name based on the test name.

var ServingFlags = initializeServingFlags()

ServingFlags holds the flags or defaults for knative/serving settings in the user's environment.

Functions

func AddRootCAtoTransport

func AddRootCAtoTransport(ctx context.Context, logf logging.FormatLogger, clients *Clients, https bool) spoof.TransportOption

AddRootCAtoTransport returns TransportOption when HTTPS option is true. Otherwise it returns plain spoof.TransportOption.

func AddTestAnnotation

func AddTestAnnotation(t testing.TB, m metav1.ObjectMeta)

AddTestAnnotation adds the knative-e2e-test label to the resource.

func AssertProberDefault

func AssertProberDefault(t testing.TB, p Prober)

AssertProberDefault is a helper for stopping the Prober and checking its SLI against the default SLO, which requires perfect responses. This takes `testing.T` so that it may be used in `defer`.

func AssertProberSLO

func AssertProberSLO(t testing.TB, p Prober, slo float64)

AssertProberSLO is a helper for stopping the Prober and checking its SLI against the given SLO.

func CheckSLO

func CheckSLO(slo float64, name string, p Prober) error

CheckSLO compares the SLI of the given prober against the SLO, erroring if too low.

func CleanupOnInterrupt

func CleanupOnInterrupt(cleanup func())

CleanupOnInterrupt stores cleanup functions to execute if an interrupt signal is caught

func EnsureCleanup

func EnsureCleanup(t *testing.T, cleanup func())

EnsureCleanup will run the provided cleanup function when the test ends, either via t.Cleanup or on interrupt via CleanupOnInterrupt.

func EnsureTearDown

func EnsureTearDown(t *testing.T, clients *Clients, names *ResourceNames)

EnsureTearDown will delete created names when the test ends, either via t.Cleanup, or on interrupt via CleanupOnInterrupt.

func ListenAndServeGracefully

func ListenAndServeGracefully(addr string, handler func(w http.ResponseWriter, r *http.Request))

ListenAndServeGracefully calls into ListenAndServeGracefullyWithPattern by passing handler to handle requests for "/"

func ListenAndServeGracefullyWithHandler

func ListenAndServeGracefullyWithHandler(addr string, handler http.Handler)

ListenAndServeGracefullyWithHandler creates an HTTP server, listens on the defined address and handles incoming requests with the given handler. It blocks until SIGTERM is received and the underlying server has shutdown gracefully.

func PemDataFromSecret

func PemDataFromSecret(ctx context.Context, logf logging.FormatLogger, clients *Clients, ns, secretName string) []byte

PemDataFromSecret gets pem data from secret.

func TLSClientConfig

func TLSClientConfig(ctx context.Context, logf logging.FormatLogger, clients *Clients) *tls.Config

func TearDown

func TearDown(clients *Clients, names *ResourceNames)

TearDown will delete created names using clients.

Types

type Clients

type Clients struct {
	KubeClient         kubernetes.Interface
	ServingAlphaClient *ServingAlphaClients
	ServingBetaClient  *ServingBetaClients
	ServingClient      *ServingClients
	NetworkingClient   *NetworkingClients
	Dynamic            dynamic.Interface
	Apiextensions      *apiextensionsv1.ApiextensionsV1Client
}

Clients holds instances of interfaces for making requests to Knative Serving.

func NewClients

func NewClients(cfg *rest.Config, namespace string) (*Clients, error)

NewClients instantiates and returns several clientsets required for making request to the Knative Serving cluster specified by the rest Config. Clients can make requests within namespace.

func Setup

func Setup(t testing.TB, namespace ...string) *Clients

Setup creates client to run Knative Service requests

type NetworkingClients

type NetworkingClients struct {
	ServerlessServices networkingv1alpha1.ServerlessServiceInterface
	Ingresses          networkingv1alpha1.IngressInterface
	Certificates       networkingv1alpha1.CertificateInterface
}

NetworkingClients holds instances of interfaces for making requests to Knative networking clients.

type Prober

type Prober interface {
	// SLI returns the "service level indicator" for the prober, which is the observed
	// success rate of the probes.  This will panic if the prober has not been stopped.
	SLI() (total int64, failures int64)

	// Stop terminates the prober, returning any observed errors.
	// Implementations may choose to put additional requirements on
	// the prober, which may cause this to block (e.g. a minimum number
	// of probes to achieve a population suitable for SLI measurement).
	Stop() error
}

Prober is the interface for a prober, which checks the result of the probes when stopped.

func RunRouteProber

func RunRouteProber(logf logging.FormatLogger, clients *Clients, url *url.URL, opts ...spoof.TransportOption) Prober

RunRouteProber starts a single Prober of the given domain.

type ProberManager

type ProberManager interface {
	// The ProberManager should expose a way to collectively reason about spawned
	// probes as a sort of aggregating Prober.
	Prober

	// Spawn creates a new Prober
	Spawn(url *url.URL) Prober

	// Foreach iterates over the probers spawned by this ProberManager.
	Foreach(func(url *url.URL, p Prober))
}

ProberManager is the interface for spawning probers, and checking their results.

func NewProberManager

func NewProberManager(logf logging.FormatLogger, clients *Clients, minProbes int64, opts ...spoof.TransportOption) ProberManager

NewProberManager creates a new manager for probes.

type ResourceNames

type ResourceNames struct {
	Config        string
	Route         string
	Revision      string
	Service       string
	TrafficTarget string
	URL           *url.URL
	Image         string
	Sidecars      []string
}

ResourceNames holds names of various resources.

type ServingAlphaClients

type ServingAlphaClients struct {
	DomainMappings servingv1alpha1.DomainMappingInterface
}

ServingAlphaClients holds instances of interfaces for making requests to knative serving clients.

type ServingBetaClients

type ServingBetaClients struct {
	DomainMappings servingv1beta1.DomainMappingInterface
}

ServingBetaClients holds instances of interfaces for making requests to knative serving clients.

type ServingClients

type ServingClients struct {
	Routes    servingv1.RouteInterface
	Configs   servingv1.ConfigurationInterface
	Revisions servingv1.RevisionInterface
	Services  servingv1.ServiceInterface
}

ServingClients holds instances of interfaces for making requests to knative serving clients.

func (*ServingClients) Delete

func (clients *ServingClients) Delete(routes, configs, services []string) []error

Delete will delete all Routes and Configs with the named routes and configs, if clients have been successfully initialized.

type ServingEnvironmentFlags

type ServingEnvironmentFlags struct {
	ResolvableDomain         bool   // Resolve Route controller's `domainSuffix`
	CustomDomain             string // Indicates the `domainSuffix` for custom domain test.
	HTTPS                    bool   // Indicates where the test service will be created with https
	Buckets                  int    // The number of reconciler buckets configured.
	Replicas                 int    // The number of controlplane replicas being run.
	EnableAlphaFeatures      bool   // Indicates whether we run tests for alpha features
	EnableBetaFeatures       bool   // Indicates whether we run tests for beta features
	DisableLogStream         bool   // Indicates whether log streaming is disabled
	DisableOptionalAPI       bool   // Indicates whether to skip conformance tests against optional API
	TestNamespace            string // Default namespace for Serving E2E/Conformance tests
	AltTestNamespace         string // Alternative namespace for running cross-namespace tests in
	TLSTestNamespace         string // Namespace for Serving TLS tests
	ExceedingMemoryLimitSize int    // Memory size used to trigger a non-200 response when the service is set with 300MB memory limit.
}

ServingEnvironmentFlags holds the e2e flags needed only by the serving repo.