test

README

Tests

To run tests:

# Land the latest codes
ko apply -f ./config/

# Unit tests
go test ./...

# Integration tests (against your current kube cluster)
go test -v -count=1 -tags=e2e -timeout=20m ./test

#conformance tests  (against your current kube cluster)
go test -v -count=1 -tags=conformance -timeout=10m ./test

Unit tests

Unit tests live side by side with the code they are testing and can be run with:

go test ./...

By default go test will not run the end to end tests, which need -tags=e2e to be enabled.

Unit testing Controllers

Kubernetes client-go provides a number of fake clients and objects for unit testing. The ones we are using are:

1. Fake Kubernetes client: Provides a fake REST interface to interact with Kubernetes API
2. Fake pipeline client : Provides a fake REST PipelineClient Interface to interact with Pipeline CRDs.

You can create a fake PipelineClient for the Controller under test like this:

import (
    fakepipelineclientset "github.com/tektoncd/pipeline/pkg/client/clientset/versioned/fake
)
pipelineClient := fakepipelineclientset.NewSimpleClientset()

This pipelineClient is initialized with no runtime objects. You can also initialize the client with Kubernetes objects and can interact with them using the pipelineClient.Pipeline()

import (
    v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

obj := &v1alpha1.PipelineRun {
    ObjectMeta: metav1.ObjectMeta {
        Name:      "name",
        Namespace: "namespace",
    },
    Spec: v1alpha1.PipelineRunSpec {
        PipelineRef: v1alpha1.PipelineRef {
            Name:       "test-pipeline",
            APIVersion: "a1",
        },
    }
}
pipelineClient := fakepipelineclientset.NewSimpleClientset(obj)
objs := pipelineClient.Pipeline().PipelineRuns("namespace").List(v1.ListOptions{})
// You can verify if List was called in your test like this
action :=  pipelineClient.Actions()[0]
if action.GetVerb() != "list" {
    t.Errorf("expected list to be called, found %s", action.GetVerb())
}

To test the Controller of CRD (CustomResourceDefinitions), you need to add the CRD to the informers so that the listers can get the access.

For example, the following code will test PipelineRun

pipelineClient := fakepipelineclientset.NewSimpleClientset()
sharedInfomer := informers.NewSharedInformerFactory(pipelineClient, 0)
pipelineRunsInformer := sharedInfomer.Pipeline().V1alpha1().PipelineRuns()

obj := &v1alpha1.PipelineRun {
    ObjectMeta: metav1.ObjectMeta {
        Name:      "name",
        Namespace: "namespace",
    },
    Spec: v1alpha1.PipelineRunSpec {
        PipelineRef: v1alpha1.PipelineRef {
            Name:       "test-pipeline",
            APIVersion: "a1",
        },
    }
}
pipelineRunsInformer.Informer().GetIndexer().Add(obj)

End to end tests

Setup

Environment variables used by end to end tests:

• KO_DOCKER_REPO - Set this to an image registry your tests can push images to

• GCP_SERVICE_ACCOUNT_KEY_PATH - Tests that need to interact with GCS buckets will use the json credentials at this path to authenticate with GCS.

• SYSTEM_NAMESPACE - Set this to your Tekton deployment namespace like tekton-pipelines. Without this setting, the E2E test will use knative-testing as default namespace.

• In Kaniko e2e test, setting GCP_SERVICE_ACCOUNT_KEY_PATH as the path of the GCP service account JSON key which has permissions to push to the registry specified in KO_DOCKER_REPO will enable Kaniko to use those credentials when pushing an image.

• In GCS taskrun test, GCP service account JSON key file at path GCP_SERVICE_ACCOUNT_KEY_PATH, if present, is used to generate Kubernetes secret to access GCS bucket.

• In Storage artifact bucket test, the GCP_SERVICE_ACCOUNT_KEY_PATH JSON key is used to create/delete a bucket which will be used for output to input linking by the PipelineRun controller.

To create a service account usable in the e2e tests:

PROJECT_ID=your-gcp-project
ACCOUNT_NAME=service-account-name
# gcloud configure project
gcloud config set project $PROJECT_ID

# create the service account
gcloud iam service-accounts create $ACCOUNT_NAME --display-name $ACCOUNT_NAME
EMAIL=$(gcloud iam service-accounts list | grep $ACCOUNT_NAME | awk '{print $2}')

# add the storage.admin policy to the account so it can push containers
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:$EMAIL --role roles/storage.admin

# create the JSON key
gcloud iam service-accounts keys create config.json --iam-account $EMAIL

export GCP_SERVICE_ACCOUNT_KEY_PATH="$PWD/config.json"

export SYSTEM_NAMESPACE=tekton-pipelines

Running

End to end tests live in this directory. To run these tests, you must provide go with -tags=e2e. By default the tests run against your current kubeconfig context, but you can change that and other settings with the flags:

go test -v -count=1 -tags=e2e -timeout=20m ./test
go test -v -count=1 -tags=e2e -timeout=20m ./test --kubeconfig ~/special/kubeconfig --cluster myspecialcluster

If tests are applied to the cluster with hardware architecture different to the base one (for instance go test starts on amd64 architecture and --kubeconfig points to s390x Kubernetes cluster), use TEST_RUNTIME_ARCH environment variable to specify the target hardware architecture(amd64, s390x, ppc64le, arm, arm64, etc)

You can also use all of flags defined in knative/pkg/test.

Flags

• By default the e2e tests against the current cluster in ~/.kube/config using the environment specified in your environment variables.
• Since these tests are fairly slow, running them with logging enabled is recommended (-v).
• Using --logverbose to see the verbose log output from test as well as from k8s libraries.
• Using -count=1 is the idiomatic way to disable test caching.
• The end to end tests take a long time to run so a value like -timeout=20m can be useful depending on what you're running
• TestKanikoTaskRun requires containers to run with root user. Using -skipRootUserTests=true skips it.

You can use test flags to control the environment your tests run against, i.e. override your environment variables:

go test -v -tags=e2e -count=1 ./test --kubeconfig ~/special/kubeconfig --cluster myspecialcluster

Tests importing github.com/tektoncd/pipeline/test recognize the flags added by knative/pkg/test.

Tests are run in a new random namespace prefixed with the word arendelle-. Unless you set the TEST_KEEP_NAMESPACES environment variable they will get automatically cleaned up after running the test.

Running specific test cases

To run all the test cases with their names starting with the same letters, e.g. TestTaskRun, use the -run flag with go test:

go test -v -tags=e2e -count=1 ./test -run ^TestTaskRun

Running YAML tests

To run the YAML e2e tests, run the following command:

go test -v -count=1 -tags=examples -timeout=20m ./test/

To limit parallelism of tests, use -parallel=n where n is the number of tests to run in parallel.

Running upgrade tests

There are two scenarios in upgrade tests. One is to install the previous release, upgrade to the current release, and validate whether the Tekton pipeline works. The other is to install the previous release, create the pipelines and tasks, upgrade to the current release, and validate whether the Tekton pipeline works.

To run the upgrade tests, run the following command:

./test/e2e-tests-upgrade.sh

Adding integration tests

In the test dir you will find several libraries in the test package you can use in your tests.

This library exists partially in this directory and partially in knative/pkg/test.

The libs in this dir can:

• init_test.go initializes anything needed globally be the tests
• Get access to client objects
• Generate random names
• Poll Pipeline resources

All integration tests must be marked with the e2e build constraint so that go test ./... can be used to run only the unit tests, i.e.:

// +build e2e

Get access to client objects

To initialize client objects use the command line flags which describe the environment:

func setup(t *testing.T) *test.Clients {
    clients, err := test.NewClients(kubeconfig, cluster, namespaceName)
    if err != nil {
        t.Fatalf("Couldn't initialize clients: %v", err)
    }
    return clients
}

The Clients struct contains initialized clients for accessing:

• Kubernetes objects
• Pipelines

For example, to create a Pipeline:

_, err = clients.PipelineClient.Pipelines.Create(test.Route(namespaceName, pipelineName))

And you can use the client to clean up resources created by your test (e.g. in your test cleanup):

func tearDown(clients *test.Clients) {
    if clients != nil {
        clients.Delete([]string{routeName}, []string{configName})
    }
}

See clients.go.

Generate random names

You can use the function GenerateName() to append a random string for crds or anything else, so that your tests can use unique names each time they run.

import "github.com/tektoncd/pipeline/pkg/names"

namespace := names.SimpleNameGenerator.GenerateName("arendelle")

Poll Pipeline resources

After creating Pipeline resources or making changes to them, you will need to wait for the system to realize those changes. You can use polling methods to check the resources reach the desired state.

The WaitFor* functions use the Kubernetes wait package. For polling they use PollImmediate behind the scene. And the callback function is ConditionFunc, which returns a bool to indicate if the function should stop, and an error to indicate if there was an error.

For example, you can poll a TaskRun until having a Status.Condition:

err = WaitForTaskRunState(c, hwTaskRunName, func(tr *v1alpha1.TaskRun) (bool, error) {
    if len(tr.Status.Conditions) > 0 {
        return true, nil
    }
    return false, nil
}, "TaskRunHasCondition")

Metrics will be emitted for these Wait methods tracking how long test poll for.

Conformance tests

Conformance tests live in this directory. These tests are used to check API specs of Pipelines. To run these tests, you must provide go with -tags=conformance. By default, the tests run against your current kubeconfig context, but you can change that and other settings with the flags like the end to end tests:

go test -v -count=1 -tags=conformace -timeout=10m ./test
go test -v -count=1 -tags=conformace -timeout=10m ./test --kubeconfig ~/special/kubeconfig --cluster myspecialcluster

Flags that could be set in conformance tests are exactly the same as flags in end to end tests. Just note that the build tags should be -tags=conformance.

Presubmit tests

presubmit-tests.sh is the entry point for all tests run on presubmit by Prow.

You can run this locally with:

test/presubmit-tests.sh
test/presubmit-tests.sh --build-tests
test/presubmit-tests.sh --unit-tests

Prow is configured in the knative config.yaml in tektoncd/plumbing via the sections for tektoncd/pipeline.

Running presubmit integration tests

The presubmit integration tests entrypoint will run:

• The integration tests
• A test of our example CRDs

When run using Prow, integration tests will try to get a new cluster using boskos and these hardcoded GKE projects, which only the tektoncd/plumbing OWNERS have access to.

If you would like to run the integration tests against your cluster, you can use the current context in your kubeconfig, provide KO_DOCKER_REPO (as specified in the DEVELOPMENT.md), use e2e-tests.sh directly and provide the --run-tests argument:

export KO_DOCKER_REPO=gcr.io/my_docker_repo
test/e2e-tests.sh --run-tests

Or you can set $PROJECT_ID to a GCP project and rely on kubetest to setup a cluster for you:

export PROJECT_ID=my_gcp_project
test/presubmit-tests.sh --integration-tests

Documentation

Overview

Get access to client objects

To initialize client objects you can use the setup function. It returns a clients struct that contains initialized clients for accessing:

• Kubernetes objects
• Pipelines (https://github.com/tektoncd/pipeline#pipeline)

For example, to create a Pipeline

_, err = clients.PipelineClient.Pipelines.Create(test.Pipeline(namespaceName, pipelineName))

And you can use the client to clean up resources created by your test

func tearDown(clients *test.Clients) {
    if clients != nil {
        clients.Delete([]string{routeName}, []string{configName})
    }
}

Package test holds the project's end-to-end tests (e2e).

Poll Pipeline resources

After creating Pipeline resources or making changes to them, you will need to wait for the system to realize those changes. You can use polling methods to check the resources reach the desired state.

The WaitFor* functions use the kubernetes wait package (https://godoc.org/k8s.io/apimachinery/pkg/util/wait). To poll they use PollImmediate (https://godoc.org/k8s.io/apimachinery/pkg/util/wait#PollImmediate) and the return values of the function you provide behave the same as ConditionFunc (https://godoc.org/k8s.io/apimachinery/pkg/util/wait#ConditionFunc): a boolean to indicate if the function should stop or continue polling, and an error to indicate if there has been an error.

For example, you can poll a TaskRun object to wait for it to have a Status.Condition:

err = WaitForTaskRunState(c, hwTaskRunName, func(tr *v1alpha1.TaskRun) (bool, error) {
	if len(tr.Status.Conditions) > 0 {
		return true, nil
	}
	return false, nil
}, "TaskRunHasCondition")

Index

• func AddToInformer(t *testing.T, store cache.Store) func(ktesting.Action) (bool, runtime.Object, error)
• func CollectPodLogs(ctx context.Context, c *clients, podName, namespace string, ...)
• func CreateImage(ref string, objs ...runtime.Object) (string, error)
• func PrependResourceVersionReactor(f *ktesting.Fake)
• func SeedTestData(t *testing.T, ctx context.Context, d Data) (Clients, Informers)
• func WaitForDeploymentState(ctx context.Context, c *clients, name string, namespace string, ...) error
• func WaitForPipelineRunState(ctx context.Context, c *clients, name string, polltimeout time.Duration, ...) error
• func WaitForPodState(ctx context.Context, c *clients, name string, namespace string, ...) error
• func WaitForServiceExternalIPState(ctx context.Context, c *clients, namespace, name string, ...) error
• func WaitForTaskRunState(ctx context.Context, c *clients, name string, inState ConditionAccessorFn, ...) error
• type Assets
• type Clients
• type ConditionAccessorFn
  • func Chain(fns ...ConditionAccessorFn) ConditionAccessorFn
  • func Failed(name string) ConditionAccessorFn
  • func FailedWithMessage(message, name string) ConditionAccessorFn
  • func FailedWithReason(reason, name string) ConditionAccessorFn
  • func PipelineRunFailed(name string) ConditionAccessorFn
  • func PipelineRunPending(name string) ConditionAccessorFn
  • func PipelineRunSucceed(name string) ConditionAccessorFn
  • func Running(name string) ConditionAccessorFn
  • func Succeed(name string) ConditionAccessorFn
  • func TaskRunFailed(name string) ConditionAccessorFn
  • func TaskRunSucceed(name string) ConditionAccessorFn
• type Data
• type Informers
• type ResourceVersionReactor
  • func (r *ResourceVersionReactor) Handles(action ktesting.Action) bool
  • func (r *ResourceVersionReactor) React(action ktesting.Action) (handled bool, ret runtime.Object, err error)

Functions

func AddToInformer

func AddToInformer(t *testing.T, store cache.Store) func(ktesting.Action) (bool, runtime.Object, error)

func CollectPodLogs

func CollectPodLogs(ctx context.Context, c *clients, podName, namespace string, logf logging.FormatLogger)

CollectPodLogs will get the logs for all containers in a Pod

func CreateImage

func CreateImage(ref string, objs ...runtime.Object) (string, error)

CreateImage will push a new OCI image artifact with the provided raw data object as a layer and return the full image reference with a digest to fetch the image. Key must be specified as [lowercase kind]/[object name]. The image ref with a digest is returned.

func PrependResourceVersionReactor

func PrependResourceVersionReactor(f *ktesting.Fake)

PrependResourceVersionReactor will instrument a client-go testing Fake with a reactor that simulates resourceVersion changes on mutations. This does not work with patches.

func SeedTestData

func SeedTestData(t *testing.T, ctx context.Context, d Data) (Clients, Informers)

SeedTestData returns Clients and Informers populated with the given Data. nolint: golint

func WaitForDeploymentState

func WaitForDeploymentState(ctx context.Context, c *clients, name string, namespace string, inState func(d *appsv1.Deployment) (bool, error), desc string) error

WaitForDeploymentState polls the status of the Deployment called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForPipelineRunState

func WaitForPipelineRunState(ctx context.Context, c *clients, name string, polltimeout time.Duration, inState ConditionAccessorFn, desc string) error

WaitForPipelineRunState polls the status of the PipelineRun called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForPodState

func WaitForPodState(ctx context.Context, c *clients, name string, namespace string, inState func(r *corev1.Pod) (bool, error), desc string) error

WaitForPodState polls the status of the Pod called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForServiceExternalIPState

func WaitForServiceExternalIPState(ctx context.Context, c *clients, namespace, name string, inState func(s *corev1.Service) (bool, error), desc string) error

WaitForServiceExternalIPState polls the status of the a k8s Service called name from client every interval until an external ip is assigned indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForTaskRunState

func WaitForTaskRunState(ctx context.Context, c *clients, name string, inState ConditionAccessorFn, desc string) error

WaitForTaskRunState polls the status of the TaskRun called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

Types

type Assets

type Assets struct {
	Logger     *zap.SugaredLogger
	Controller *controller.Impl
	Clients    Clients
	Informers  Informers
	Recorder   *record.FakeRecorder
	Ctx        context.Context
}

Assets holds references to the controller, logs, clients, and informers.

type Clients

type Clients struct {
	Pipeline    *fakepipelineclientset.Clientset
	Resource    *fakeresourceclientset.Clientset
	Kube        *fakekubeclientset.Clientset
	CloudEvents cloudeventclient.CEClient
}

Clients holds references to clients which are useful for reconciler tests.

type ConditionAccessorFn

type ConditionAccessorFn func(ca apis.ConditionAccessor) (bool, error)

ConditionAccessorFn is a condition function used polling functions

func Chain

func Chain(fns ...ConditionAccessorFn) ConditionAccessorFn

Chain allows multiple ConditionAccessorFns to be chained together, checking the condition of each in order.

func Failed

func Failed(name string) ConditionAccessorFn

Failed provides a poll condition function that checks if the ConditionAccessor resource has failed or not.

func FailedWithMessage

func FailedWithMessage(message, name string) ConditionAccessorFn

FailedWithMessage provides a poll function that checks if the ConditionAccessor resource has failed with the TimeoudOut reason

func FailedWithReason

func FailedWithReason(reason, name string) ConditionAccessorFn

FailedWithReason provides a poll function that checks if the ConditionAccessor resource has failed with the TimeoudOut reason

func PipelineRunFailed

func PipelineRunFailed(name string) ConditionAccessorFn

PipelineRunFailed provides a poll condition function that checks if the PipelineRun has failed.

func PipelineRunPending

func PipelineRunPending(name string) ConditionAccessorFn

PipelineRunPending provides a poll condition function that checks if the PipelineRun has been marked pending by the Tekton controller.

func PipelineRunSucceed

func PipelineRunSucceed(name string) ConditionAccessorFn

PipelineRunSucceed provides a poll condition function that checks if the PipelineRun has successfully completed.

func Running

func Running(name string) ConditionAccessorFn

Running provides a poll condition function that checks if the ConditionAccessor resource is currently running.

func Succeed

func Succeed(name string) ConditionAccessorFn

Succeed provides a poll condition function that checks if the ConditionAccessor resource has successfully completed or not.

func TaskRunFailed

func TaskRunFailed(name string) ConditionAccessorFn

TaskRunFailed provides a poll condition function that checks if the TaskRun has failed.

func TaskRunSucceed

func TaskRunSucceed(name string) ConditionAccessorFn

TaskRunSucceed provides a poll condition function that checks if the TaskRun has successfully completed.

type Data

type Data struct {
	PipelineRuns      []*v1beta1.PipelineRun
	Pipelines         []*v1beta1.Pipeline
	TaskRuns          []*v1beta1.TaskRun
	Tasks             []*v1beta1.Task
	ClusterTasks      []*v1beta1.ClusterTask
	PipelineResources []*v1alpha1.PipelineResource
	Conditions        []*v1alpha1.Condition
	Runs              []*v1alpha1.Run
	Pods              []*corev1.Pod
	Namespaces        []*corev1.Namespace
	ConfigMaps        []*corev1.ConfigMap
	ServiceAccounts   []*corev1.ServiceAccount
}

Data represents the desired state of the system (i.e. existing resources) to seed controllers with.

type Informers

type Informers struct {
	PipelineRun      informersv1beta1.PipelineRunInformer
	Pipeline         informersv1beta1.PipelineInformer
	TaskRun          informersv1beta1.TaskRunInformer
	Run              informersv1alpha1.RunInformer
	Task             informersv1beta1.TaskInformer
	ClusterTask      informersv1beta1.ClusterTaskInformer
	PipelineResource resourceinformersv1alpha1.PipelineResourceInformer
	Condition        informersv1alpha1.ConditionInformer
	Pod              coreinformers.PodInformer
	ConfigMap        coreinformers.ConfigMapInformer
	ServiceAccount   coreinformers.ServiceAccountInformer
}

Informers holds references to informers which are useful for reconciler tests.

type ResourceVersionReactor

type ResourceVersionReactor struct {
	// contains filtered or unexported fields
}

func (*ResourceVersionReactor) Handles

func (r *ResourceVersionReactor) Handles(action ktesting.Action) bool

func (*ResourceVersionReactor) React

func (r *ResourceVersionReactor) React(action ktesting.Action) (handled bool, ret runtime.Object, err error)

React is noop-function