test

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 14, 2019 License: Apache-2.0 Imports: 44 Imported by: 0

README

Test

This directory contains tests and testing docs for Knative Serving:

The conformance tests are a subset of the end to end test with more strict requirements around what can be tested.

If you want to add more tests, see adding_tests.md.

Presubmit tests

presubmit-tests.sh is the entry point for both the end-to-end tests and the conformance tests

This script, and consequently, the e2e and conformance tests will be run before every code submission. You can run these tests manually with:

test/presubmit-tests.sh

Note that to run presubmit-tests.sh or e2e-tests.sh scripts, you'll need kubernetes kubetest installed:

go get -u k8s.io/test-infra/kubetest

Running unit tests

To run all unit tests:

go test ./...

By default go test will not run the e2e tests, which need -tags=e2e to be enabled.

Running end to end tests

To run the e2e tests and the conformance tests, you need to have a running environment that meets the e2e test environment requirements, and you need to specify the build tag e2e.

go test -v -tags=e2e -count=1 ./test/conformance
go test -v -tags=e2e -count=1 ./test/e2e

Running performance tests

To run the performance tests, you need to have a running environment that meets the test environment requirements, and you need to specify the build tag performance.

go test -v -tags=performance -count=1 ./test/performance
Running a single test case

To run one e2e test case, e.g. TestAutoscaleUpDownUp, use the -run flag with go test:

go test -v -tags=e2e -count=1 ./test/e2e -run ^TestAutoscaleUpDownUp$
Running tests in short mode

Running tests in short mode excludes some large-scale E2E tests and saves time/resources required for running the test suite. To run the tests in short mode, use the -short flag with go test

go test -v -tags=e2e -count=1 -short ./test/e2e

To get a better idea where the flag is used, search for testing.Short() throughout the test source code.

Environment requirements

These tests require:

  1. A running Knative Serving cluster.

  2. The knative-testing resources:

    ko apply -f test/config
    
  3. A docker repo containing the test images

Common Flags

You can use test flags to control the environment your tests run against, i.e. override your environment variables:

go test -v -tags=e2e -count=1 ./test/conformance --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
go test -v -tags=e2e -count=1 ./test/e2e --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo

Test images

Building the test images

Note: this is only required when you run conformance/e2e tests locally with go test commands.

The upload-test-images.sh script can be used to build and push the test images used by the conformance and e2e tests. The script expects your environment to be setup as described in DEVELOPMENT.md.

To run the script for all end to end test images:

./test/upload-test-images.sh

A docker tag may be passed as an optional parameter. This can be useful on Minikube in tandem with the --tag flag:

eval $(minikube docker-env)
./test/upload-test-images.sh any-old-tag
Adding new test images

New test images should be placed in ./test/test_images.

Flags

These flags are useful for running against an existing cluster, making use of your existing environment setup.

Tests importing github.com/knative/serving/test recognize these flags:

Overridding docker repo

The --dockerrepo argument lets you specify the docker repo from which images used by your tests should be pulled. This will default to the value of your KO_DOCKER_REPO environment variable if not specified.

go test -v -tags=e2e -count=1 ./test/conformance --dockerrepo gcr.myhappyproject
go test -v -tags=e2e -count=1 ./test/e2e --dockerrepo gcr.myhappyproject
Using a docker tag

The default docker tag used for the test images is latest, which can be problematic on Minikube. To avoid having to configure a remote container registry to support the Always pull policy for latest tags, you can have the tests use a specific tag:

go test -v -tags=e2e -count=1 ./test/conformance --tag any-old-tag
go test -v -tags=e2e -count=1 ./test/e2e --tag any-old-tag

Of course, this implies that you tagged the images when you uploaded them.

Using a custom ingress endpoint

Some environments (like minikube) do not support a Loadbalancer to make Knative services externally available. These environments usually rely on rewriting the Loadbalancer to a NodePort. The external address of such a NodePort is usually not easily obtained within the cluster automatically, but can be provided from the outside through the --ingressendpoint flag. For a minikube setup for example, you'd want to run tests against the default ingressgateway (port number 31380) running on the minikube node:

go test -v -tags=e2e -count=1 ./test/conformance --ingressendpoint "$(minikube ip):31380"
go test -v -tags=e2e -count=1 ./test/e2e --ingressendpoint "$(minikube ip):31380"
Using a resolvable domain

If you set up your cluster using the getting started docs, Routes created in the test will use the domain example.com, unless the route has label app=prod in which case they will use the domain prod-domain.com. Since these domains will not be resolvable to deployments in your test cluster, in order to make a request against the endpoint, the test use the IP assigned to the service istio-ingressgateway in the namespace istio-system and spoof the Host in the header.

If you have configured your cluster to use a resolvable domain, you can use the --resolvabledomain flag to indicate that the test should make requests directly against Route.Status.Domain and does not need to spoof the Host.

Documentation

Index

Constants

View Source
const (
	// ServingNamespace is the default namespace for serving e2e tests
	ServingNamespace = "serving-tests"

	// AlternativeServingNamespace is a different namepace to run cross-
	// namespace tests in.
	AlternativeServingNamespace = "serving-tests-alt"

	// E2EMetricExporter is the name for the metrics exporter logger
	E2EMetricExporter = "e2e-metrics"

	// ConformanceConfigMap is the name of the configmap to propagate env variables from
	ConformanceConfigMap = "conformance-test-configmap"
	// ConformanceSecret is the name of the secret to propagate env variables from
	ConformanceSecret = "conformance-test-secret"
	// EnvKey is the configmap/secret key which contains test value
	EnvKey = "testKey"
	// EnvValue is the configmap/secret test value to match env variable with
	EnvValue = "testValue"
)
View Source
const HelloVolumePath = "/hello/world"

HelloVolumePath is the path to the test volume.

Variables

View Source
var AppendRandomString = helpers.AppendRandomString

AppendRandomString will generate a random string that begins with prefix. This is useful if you want to make sure that your tests can run at the same time against the same environment without conflicting. This method will seed rand with the current time when called for the first time.

View Source
var ServingFlags = initializeServingFlags()

ServingFlags holds the flags or defaults for knative/serving settings in the user's environment.

Functions

func AllRouteTrafficAtRevision

func AllRouteTrafficAtRevision(names ResourceNames) func(r *v1alpha1.Route) (bool, error)

AllRouteTrafficAtRevision will check the revision that route r is routing traffic to and return true if 100% of the traffic is routing to revisionName.

func AssertProberDefault added in v0.4.0

func AssertProberDefault(t *testing.T, p Prober)

AssertProberDefault is a helper for stopping the Prober and checking its SLI against the default SLO, which requires perfect responses. This takes `testing.T` so that it may be used in `defer`.

func CheckConfigurationState

func CheckConfigurationState(client *ServingClients, name string, inState func(r *v1alpha1.Configuration) (bool, error)) error

CheckConfigurationState verifies the status of the Configuration called name from client is in a particular state by calling `inState` and expecting `true`. This is the non-polling variety of WaitForConfigurationState

func CheckRevisionState

func CheckRevisionState(client *ServingClients, name string, inState func(r *v1alpha1.Revision) (bool, error)) error

CheckRevisionState verifies the status of the Revision called name from client is in a particular state by calling `inState` and expecting `true`. This is the non-polling variety of WaitForRevisionState

func CheckRouteState

func CheckRouteState(client *ServingClients, name string, inState func(r *v1alpha1.Route) (bool, error)) error

CheckRouteState verifies the status of the Route called name from client is in a particular state by calling `inState` and expecting `true`. This is the non-polling variety of WaitForRouteState

func CheckSLO added in v0.4.0

func CheckSLO(slo float64, name string, p Prober) error

CheckSLO compares the SLI of the given prober against the SLO, erroring if too low.

func CheckServiceState

func CheckServiceState(client *ServingClients, name string, inState func(s *v1alpha1.Service) (bool, error)) error

CheckServiceState verifies the status of the Service called name from client is in a particular state by calling `inState` and expecting `true`. This is the non-polling variety of WaitForServiceState.

func CleanupOnInterrupt

func CleanupOnInterrupt(cleanup func())

CleanupOnInterrupt will execute the function cleanup if an interrupt signal is caught

func Configuration

func Configuration(names ResourceNames, options *Options, fopt ...v1alpha1testing.ConfigOption) *v1alpha1.Configuration

Configuration returns a Configuration object in namespace with the name names.Config that uses the image specified by names.Image

func ConfigurationHasCreatedRevision added in v0.2.0

func ConfigurationHasCreatedRevision(c *v1alpha1.Configuration) (bool, error)

ConfigurationHasCreatedRevision returns whether the Configuration has created a Revision.

func ConfigurationSpec added in v0.3.0

func ConfigurationSpec(imagePath string, options *Options) *v1alpha1.ConfigurationSpec

ConfigurationSpec returns the spec of a configuration to be used throughout different CRD helpers.

func ConfigurationWithBuild added in v0.2.0

func ConfigurationWithBuild(names ResourceNames, build *v1alpha1.RawExtension) *v1alpha1.Configuration

ConfigurationWithBuild returns a Configuration object in the `namespace` with the name `names.Config` that uses the provided Build spec `build` and image specified by `names.Image`.

func CreateConfiguration added in v0.2.0

func CreateConfiguration(t *testing.T, clients *Clients, names ResourceNames, options *Options, fopt ...rtesting.ConfigOption) (*v1alpha1.Configuration, error)

CreateConfiguration create a configuration resource in namespace with the name names.Config that uses the image specified by names.Image.

func CreateLatestService added in v0.2.0

func CreateLatestService(t *testing.T, clients *Clients, names ResourceNames, options *Options, fopt ...rtesting.ServiceOption) (*v1alpha1.Service, error)

CreateLatestService creates a service in namespace with the name names.Service and names.Image

func CreateLatestServiceLegacy added in v0.6.0

func CreateLatestServiceLegacy(t *testing.T, clients *Clients, names ResourceNames, options *Options, fopt ...rtesting.ServiceOption) (*v1alpha1.Service, error)

CreateLatestServiceLegacy creates a service in namespace with the name names.Service and names.Image

func CreateRoute added in v0.2.0

func CreateRoute(t *testing.T, clients *Clients, names ResourceNames, fopt ...rtesting.RouteOption) (*v1alpha1.Route, error)

CreateRoute creates a route in the given namespace using the route name in names

func DeploymentScaledToZeroFunc added in v0.4.0

func DeploymentScaledToZeroFunc(d *appsv1.Deployment) (bool, error)

DeploymentScaledToZeroFunc returns a func that evaluates if a deployment has scaled to 0 pods.

func GetConfigMap added in v0.2.0

func GetConfigMap(client *pkgTest.KubeClient) k8styped.ConfigMapInterface

GetConfigMap gets the knative serving config map.

func IsConfigRevisionCreationFailed added in v0.2.0

func IsConfigRevisionCreationFailed(c *v1alpha1.Configuration) (bool, error)

IsConfigRevisionCreationFailed will check the status conditions of the configuration and return true if the configuration's revision failed to create.

func IsRevisionAtExpectedGeneration added in v0.2.0

func IsRevisionAtExpectedGeneration(expectedGeneration string) func(r *v1alpha1.Revision) (bool, error)

IsRevisionAtExpectedGeneration returns a function that will check if the annotations on the revision include an annotation for the generation and that the annotation is set to the expected value.

func IsRevisionBuildFailed added in v0.2.0

func IsRevisionBuildFailed(r *v1alpha1.Revision) (bool, error)

IsRevisionBuildFailed will check the status conditions of the revision and return true if the revision's build failed.

func IsRevisionReady

func IsRevisionReady(r *v1alpha1.Revision) (bool, error)

IsRevisionReady will check the status conditions of the revision and return true if the revision is ready to serve traffic. It will return false if the status indicates a state other than deploying or being ready. It will also return false if the type of the condition is unexpected.

func IsRouteReady

func IsRouteReady(r *v1alpha1.Route) (bool, error)

IsRouteReady will check the status conditions of the route and return true if the route is ready.

func IsServiceReady

func IsServiceReady(s *v1alpha1.Service) (bool, error)

IsServiceReady will check the status conditions of the service and return true if the service is ready. This means that its configurations and routes have all reported ready.

func LatestService

func LatestService(names ResourceNames, options *Options, fopt ...v1alpha1testing.ServiceOption) *v1alpha1.Service

LatestService returns a Service object in namespace with the name names.Service that uses the image specified by names.Image.

func LatestServiceLegacy added in v0.6.0

func LatestServiceLegacy(names ResourceNames, options *Options, fopt ...v1alpha1testing.ServiceOption) *v1alpha1.Service

LatestServiceLegacy returns a DeprecatedRunLatest Service object in namespace with the name names.Service that uses the image specified by names.Image.

func LegacyConfigurationSpec added in v0.6.0

func LegacyConfigurationSpec(imagePath string, options *Options) *v1alpha1.ConfigurationSpec

LegacyConfigurationSpec returns the spec of a configuration to be used throughout different CRD helpers.

func ListenAndServeGracefully added in v0.3.0

func ListenAndServeGracefully(addr string, handler func(w http.ResponseWriter, r *http.Request))

ListenAndServeGracefully calls into ListenAndServeGracefullyWithPattern by passing handler to handle requests for "/"

func ListenAndServeGracefullyWithPattern added in v0.3.0

func ListenAndServeGracefullyWithPattern(addr string, handlers map[string]func(w http.ResponseWriter, r *http.Request))

ListenAndServeGracefullyWithPattern creates an HTTP server, listens on the defined address and handles incoming requests specified on pattern(path) with the given handlers. It blocks until SIGTERM is received and the underlying server has shutdown gracefully.

func LogResourceObject added in v0.2.0

func LogResourceObject(t *testing.T, value ResourceObjects)

LogResourceObject logs the resource object with the resource name and value

func ObjectNameForTest added in v0.4.0

func ObjectNameForTest(t *testing.T) string

ObjectNameForTest generates a random object name based on the test name.

func PatchConfigImage added in v0.3.0

func PatchConfigImage(clients *Clients, cfg *v1alpha1.Configuration, imagePath string) (*v1alpha1.Configuration, error)

PatchConfigImage patches the existing config passed in with a new imagePath. Returns the latest Configuration object

func PatchService added in v0.3.0

func PatchService(t *testing.T, clients *Clients, curSvc *v1alpha1.Service, desiredSvc *v1alpha1.Service) (*v1alpha1.Service, error)

PatchService creates and applies a patch from the diff between curSvc and desiredSvc. Returns the latest service object.

func PatchServiceImage added in v0.3.0

func PatchServiceImage(t *testing.T, clients *Clients, svc *v1alpha1.Service, imagePath string) (*v1alpha1.Service, error)

PatchServiceImage patches the existing service passed in with a new imagePath. Returns the latest service object

func PatchServiceTemplateMetadata added in v0.6.0

func PatchServiceTemplateMetadata(t *testing.T, clients *Clients, svc *v1alpha1.Service, metadata metav1.ObjectMeta) (*v1alpha1.Service, error)

PatchServiceTemplateMetadata patches an existing service by adding metadata to the service's RevisionTemplateSpec.

func RetryingRouteInconsistency added in v0.5.0

func RetryingRouteInconsistency(innerCheck spoof.ResponseChecker) spoof.ResponseChecker

RetryingRouteInconsistency retries common requests seen when creating a new route - 404 until the route is propagated to the proxy

func Route

Route returns a Route object in namespace using the route and configuration names in names.

func SubServiceNameForTest added in v0.4.0

func SubServiceNameForTest(t *testing.T, subsvc string) string

SubServiceNameForTest generates a random service name based on the test name and the given subservice name.

func TODO_RouteTrafficToRevisionWithInClusterDNS

func TODO_RouteTrafficToRevisionWithInClusterDNS(r *v1alpha1.Route) (bool, error)

RouteTrafficToRevisionWithInClusterDNS will check the revision that route r is routing traffic using in cluster DNS and return true if the revision received the request.

func TODO_ServiceTrafficToRevisionWithInClusterDNS

func TODO_ServiceTrafficToRevisionWithInClusterDNS(s *v1alpha1.Service) (bool, error)

TODO_ServiceTrafficToRevisionWithInClusterDNS will check the revision that route r is routing traffic using in cluster DNS and return true if the revision received the request.

func TearDown added in v0.4.0

func TearDown(clients *Clients, names ResourceNames)

TearDown will delete created names using clients.

func UpdateServiceRouteSpec added in v0.6.0

func UpdateServiceRouteSpec(t *testing.T, clients *Clients, names ResourceNames, rs v1alpha1.RouteSpec) (*v1alpha1.Service, error)

UpdateServiceRouteSpec updates a service to use the route name in names.

func WaitForConfigLatestRevision added in v0.3.0

func WaitForConfigLatestRevision(clients *Clients, names ResourceNames) (string, error)

WaitForConfigLatestRevision takes a revision in through names and compares it to the current state of LatestCreatedRevisionName in Configuration. Once an update is detected in the LatestCreatedRevisionName, the function waits for the created revision to be set in LatestReadyRevisionName before returning the name of the revision.

func WaitForConfigurationState

func WaitForConfigurationState(client *ServingClients, name string, inState func(c *v1alpha1.Configuration) (bool, error), desc string) error

WaitForConfigurationState polls the status of the Configuration called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForRevisionState

func WaitForRevisionState(client *ServingClients, name string, inState func(r *v1alpha1.Revision) (bool, error), desc string) error

WaitForRevisionState polls the status of the Revision called name from client every `interval` until `inState` returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForRouteState

func WaitForRouteState(client *ServingClients, name string, inState func(r *v1alpha1.Route) (bool, error), desc string) error

WaitForRouteState polls the status of the Route called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForServiceLatestRevision added in v0.3.0

func WaitForServiceLatestRevision(clients *Clients, names ResourceNames) (string, error)

WaitForServiceLatestRevision takes a revision in through names and compares it to the current state of LatestCreatedRevisionName in Service. Once an update is detected in the LatestCreatedRevisionName, the function waits for the created revision to be set in LatestReadyRevisionName before returning the name of the revision.

func WaitForServiceState

func WaitForServiceState(client *ServingClients, name string, inState func(s *v1alpha1.Service) (bool, error), desc string) error

WaitForServiceState polls the status of the Service called name from client every `interval` until `inState` returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

Types

type BuildClient added in v0.2.0

type BuildClient struct {
	TestBuilds testbuildtyped.BuildInterface
}

BuildClient holds instances of interfaces for making requests to build client.

type Clients

type Clients struct {
	KubeClient    *test.KubeClient
	ServingClient *ServingClients
	BuildClient   *BuildClient
	Dynamic       dynamic.Interface
}

Clients holds instances of interfaces for making requests to Knative Serving.

func NewClients

func NewClients(configPath string, clusterName string, namespace string) (*Clients, error)

NewClients instantiates and returns several clientsets required for making request to the Knative Serving cluster specified by the combination of clusterName and configPath. Clients can make requests within namespace.

type Options added in v0.2.0

type Options struct {
	EnvVars                []corev1.EnvVar
	ContainerPorts         []corev1.ContainerPort
	ContainerConcurrency   int
	RevisionTimeoutSeconds int64
	ContainerResources     corev1.ResourceRequirements
	ReadinessProbe         *corev1.Probe
	SecurityContext        *corev1.SecurityContext
}

Options are test setup parameters.

type Prober added in v0.4.0

type Prober interface {
	// SLI returns the "service level indicator" for the prober, which is the observed
	// success rate of the probes.  This will panic if the prober has not been stopped.
	SLI() (total int64, failures int64)

	// Stop terminates the prober, returning any observed errors.
	// Implementations may choose to put additional requirements on
	// the prober, which may cause this to block (e.g. a minimum number
	// of probes to achieve a population suitable for SLI measurement).
	Stop() error
}

Prober is the interface for a prober, which checks the result of the probes when stopped.

func RunRouteProber added in v0.2.0

func RunRouteProber(t *testing.T, clients *Clients, domain string) Prober

RunRouteProber starts a single Prober of the given domain.

type ProberManager added in v0.4.0

type ProberManager interface {
	// The ProberManager should expose a way to collectively reason about spawned
	// probes as a sort of aggregating Prober.
	Prober

	// Spawn creates a new Prober
	Spawn(domain string) Prober

	// Foreach iterates over the probers spawned by this ProberManager.
	Foreach(func(domain string, p Prober))
}

ProberManager is the interface for spawning probers, and checking their results.

func NewProberManager added in v0.4.0

func NewProberManager(t *testing.T, clients *Clients, minProbes int64) ProberManager

NewProberManager creates a new manager for probes.

type ResourceNames

type ResourceNames struct {
	Config        string
	Route         string
	Revision      string
	Service       string
	TrafficTarget string
	Domain        string
	Image         string
}

ResourceNames holds names of various resources.

type ResourceObjects added in v0.2.0

type ResourceObjects struct {
	Route    *v1alpha1.Route
	Config   *v1alpha1.Configuration
	Service  *v1alpha1.Service
	Revision *v1alpha1.Revision
}

ResourceObjects holds types of the resource objects.

func CreateRunLatestServiceReady added in v0.3.0

func CreateRunLatestServiceReady(t *testing.T, clients *Clients, names *ResourceNames, options *Options, fopt ...rtesting.ServiceOption) (*ResourceObjects, error)

CreateRunLatestServiceReady creates a new Service in state 'Ready'. This function expects Service and Image name passed in through 'names'. Names is updated with the Route and Configuration created by the Service and ResourceObjects is returned with the Service, Route, and Configuration objects. Returns error if the service does not come up correctly.

func GetResourceObjects added in v0.4.0

func GetResourceObjects(clients *Clients, names ResourceNames) (*ResourceObjects, error)

GetResourceObjects obtains the services resources from the k8s API server.

type ServingClients added in v0.2.0

ServingClients holds instances of interfaces for making requests to knative serving clients

func (*ServingClients) Delete added in v0.2.0

func (clients *ServingClients) Delete(routes []string, configs []string, services []string) error

Delete will delete all Routes and Configs with the names routes and configs, if clients has been successfully initialized.

type ServingEnvironmentFlags added in v0.2.0

type ServingEnvironmentFlags struct {
	ResolvableDomain bool // Resolve Route controller's `domainSuffix`
}

ServingEnvironmentFlags holds the e2e flags needed only by the serving repo.

Directories

Path Synopsis
apicoverage
+k8s:deepcopy-gen=package
+k8s:deepcopy-gen=package
testing/v1alpha1
+k8s:deepcopy-gen=package +groupName=testing.internal.knative.dev
+k8s:deepcopy-gen=package +groupName=testing.internal.knative.dev
client
clientset/versioned
This package has the automatically generated clientset.
This package has the automatically generated clientset.
clientset/versioned/fake
This package has the automatically generated fake clientset.
This package has the automatically generated fake clientset.
clientset/versioned/scheme
This package contains the scheme of the automatically generated clientset.
This package contains the scheme of the automatically generated clientset.
clientset/versioned/typed/testing/v1alpha1
This package has the automatically generated typed clients.
This package has the automatically generated typed clients.
clientset/versioned/typed/testing/v1alpha1/fake
Package fake has the automatically generated clients.
Package fake has the automatically generated clients.
reconciler
test_images
singlethreaded
The singlethreaded program
The singlethreaded program

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL