test

README

Tests

To run tests:

# Land the latest codes
ko apply -f ./config/

# Unit tests
go test ./...

# Integration tests (against your current kube cluster)
go test -v -count=1 -tags=e2e ./test

Unit tests

Unit tests live side by side with the code they are testing and can be run with:

go test ./...

By default go test will not run the end to end tests, which need -tags=e2e to be enabled.

Unit testing Controllers

Kubernetes client-go provides a number of fake clients and objects for unit testing. The ones we are using are:

1. Fake Kubernetes client: Provides a fake REST interface to interact with Kubernetes API
2. Fake pipeline client : Provides a fake REST PipelineClient Interface to interact with Pipeline CRDs.

You can create a fake PipelineClient for the Controller under test like this:

import (
    fakepipelineclientset "github.com/knative/build-pipeline/pkg/client/clientset/versioned/fake
)
pipelineClient := fakepipelineclientset.NewSimpleClientset()

This pipelineClient is initialized with no runtime objects. You can also initialize the client with Kubernetes objects and can interact with them using the pipelineClient.Pipeline()

import (
    v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

obj := &v1alpha1.PipelineRun {
    ObjectMeta: metav1.ObjectMeta {
        Name:      "name",
        Namespace: "namespace",
    },
    Spec: v1alpha1.PipelineRunSpec {
        PipelineRef: v1alpha1.PipelineRef {
            Name:       "test-pipeline",
            APIVersion: "a1",
        },
    }
}
pipelineClient := fakepipelineclientset.NewSimpleClientset(obj)
objs := pipelineClient.Pipeline().PipelineRuns("namespace").List(v1.ListOptions{})
// You can verify if List was called in your test like this
action :=  pipelineClient.Actions()[0]
if action.GetVerb() != "list" {
    t.Errorf("expected list to be called, found %s", action.GetVerb())
}

To test the Controller of CRD (CustomResourceDefinitions), you need to add the CRD to the informers so that the listers can get the access.

For example, the following code will test PipelineRun

pipelineClient := fakepipelineclientset.NewSimpleClientset()
sharedInfomer := informers.NewSharedInformerFactory(pipelineClient, 0)
pipelineRunsInformer := sharedInfomer.Pipeline().V1alpha1().PipelineRuns()

obj := &v1alpha1.PipelineRun {
    ObjectMeta: metav1.ObjectMeta {
        Name:      "name",
        Namespace: "namespace",
    },
    Spec: v1alpha1.PipelineRunSpec {
        PipelineRef: v1alpha1.PipelineRef {
            Name:       "test-pipeline",
            APIVersion: "a1",
        },
    }
}
pipelineRunsInformer.Informer().GetIndexer().Add(obj)

End to end tests

Setup

Besides the environment variable KO_DOCKER_REPO, you may also need the permissions inside the TaskRun to run the Kaniko e2e test and GCS taskrun test.

• In Kaniko e2e test, setting GCP_SERVICE_ACCOUNT_KEY_PATH as the path of the GCP service account JSON key which has permissions to push to the registry specified in KO_DOCKER_REPO will enable Kaniko to use those credentials when pushing an image.
• In GCS taskrun test, GCP service account JSON key file at path GCP_SERVICE_ACCOUNT_KEY_PATH is used to generate Kubernetes secret to access GCS bucket. This e2e test requires valid service account configuration json but it does not require any role binding.
• In Storage artifact bucket, setting the GCP_SERVICE_ACCOUNT_KEY_PATH as the path of the GCP service account JSON key which has permissions to create/delete a bucket.

To reduce e2e test setup developers can use the same environment variable for both Kaniko e2e test and GCS taskrun test. To create a service account usable in the e2e tests:

PROJECT_ID=your-gcp-project
ACCOUNT_NAME=service-account-name
# gcloud configure project
gcloud config set project $PROJECT_ID

# create the service account
gcloud iam service-accounts create $ACCOUNT_NAME --display-name $ACCOUNT_NAME
EMAIL=$(gcloud iam service-accounts list | grep $ACCOUNT_NAME | awk '{print $2}')

# add the storage.admin policy to the account so it can push containers
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:$EMAIL --role roles/storage.admin

# create the JSON key
gcloud iam service-accounts keys create config.json --iam-account $EMAIL

export GCP_SERVICE_ACCOUNT_KEY_PATH="$PWD/config.json"

Running

Integration tests live in this directory. To run these tests, you must provide go with -tags=e2e. By default the tests run against your current kubeconfig context, but you can change that and other settings with the flags:

go test -v -count=1 -tags=e2e ./test
go test -v -tags=e2e -count=1 ./test --kubeconfig ~/special/kubeconfig --cluster myspecialcluster

You can also use all of flags defined in knative/pkg/test.

Flags

• By default the e2e tests against the current cluster in ~/.kube/config using the environment specified in your environment variables.
• Since these tests are fairly slow, running them with logging enabled is recommended (-v).
• Using --logverbose to see the verbose log output from test as well as from k8s libraries.
• Using -count=1 is the idiomatic way to disable test caching.

You can use test flags to control the environment your tests run against, i.e. override your environment variables:

go test -v -tags=e2e -count=1 ./test --kubeconfig ~/special/kubeconfig --cluster myspecialcluster

Tests importing github.com/knative/build-pipline/test recognize the flags added by knative/pkg/test.

One test case

To run one e2e test case, e.g. TestTaskRun, use the -run flag with go test:

go test -v -tags=e2e -count=1 ./test -run ^TestTaskRun$

Running YAML tests

To run the YAML e2e tests, run the following command:

./test/e2e-tests-yaml.sh

Adding integration tests

In the test dir you will find several libraries in the test package you can use in your tests.

This library exists partially in this directory and partially in knative/pkg/test.

The libs in this dir can:

• init_test.go initializes anything needed globally be the tests
• Get access to client objects
• Generate random names
• Poll Pipeline resources

All integration tests must be marked with the e2e build constraint so that go test ./... can be used to run only the unit tests, i.e.:

// +build e2e

Create build-pipeline objects

To create build-pipeline objects (e.g. Task, Pipeline, …), you can use the builder package to reduce noise:

func MyTest(t *testing.T){
    // Pipeline
    pipeline := tb.Pipeline("tomatoes", "namespace",
        tb.PipelineSpec(tb.PipelineTask("foo", "banana")),
    )
    // … and PipelineRun
    pipelineRun := tb.PipelineRun("pear", "namespace",
        tb.PipelineRunSpec("tomatoes", tb.PipelineRunServiceAccount("inexistent")),
    )
    // And do something with them
    // […]
    if _, err := c.PipelineClient.Create(pipeline); err != nil {
        t.Fatalf("Failed to create Pipeline `%s`: %s", "tomatoes", err)
    }
    if _, err := c.PipelineRunClient.Create(pipelineRun); err != nil {
        t.Fatalf("Failed to create PipelineRun `%s`: %s", "pear", err)
    }
}

Get access to client objects

To initialize client objects use the command line flags which describe the environment:

func setup(t *testing.T) *test.Clients {
    clients, err := test.NewClients(kubeconfig, cluster, namespaceName)
    if err != nil {
        t.Fatalf("Couldn't initialize clients: %v", err)
    }
    return clients
}

The Clients struct contains initialized clients for accessing:

• Kubernetes objects
• Pipelines

For example, to create a Pipeline:

_, err = clients.PipelineClient.Pipelines.Create(test.Route(namespaceName, pipelineName))

And you can use the client to clean up resources created by your test (e.g. in your test cleanup):

func tearDown(clients *test.Clients) {
    if clients != nil {
        clients.Delete([]string{routeName}, []string{configName})
    }
}

See clients.go.

Generate random names

You can use the function GenerateName() to append a random string for crds or anything else, so that your tests can use unique names each time they run.

import "github.com/knative/build-pipeline/pkg/names"

namespace := names.SimpleNameGenerator.GenerateName("arendelle")

Poll Pipeline resources

After creating Pipeline resources or making changes to them, you will need to wait for the system to realize those changes. You can use polling methods to check the resources reach the desired state.

The WaitFor* functions use the Kubernetes wait package. For polling they use PollImmediate behind the scene. And the callback function is ConditionFunc, which returns a bool to indicate if the function should stop, and an error to indicate if there was an error.

For example, you can poll a TaskRun until having a Status.Condition:

err = WaitForTaskRunState(c, hwTaskRunName, func(tr *v1alpha1.TaskRun) (bool, error) {
    if len(tr.Status.Conditions) > 0 {
        return true, nil
    }
    return false, nil
}, "TaskRunHasCondition")

Metrics will be emitted for these Wait methods tracking how long test poll for.

Presubmit tests

presubmit-tests.sh is the entry point for all tests run on presubmit by Prow.

You can run this locally with:

test/presubmit-tests.sh
test/presubmit-tests.sh --build-tests
test/presubmit-tests.sh --unit-tests

Prow is configured in the knative config.yaml in knative/test-infra via the sections for knative/build-pipeline.

Running presubmit integration tests

The presubmit integration tests entrypoint will run:

• The integration tests
• A test of our example CRDs

When run using Prow, integration tests will try to get a new cluster using boskos and these hardcoded GKE projects, which only the knative/test-infra OWNERS have access to.

If you would like to run the integration tests against your cluster, you can use the current context in your kubeconfig, provide KO_DOCKER_REPO (as specified in the DEVELOPMENT.md), use e2e-tests.sh directly and provide the --run-tests argument:

export KO_DOCKER_REPO=gcr.io/my_docker_repo
test/e2e-tests.sh --run-tests

Or you can set $PROJECT_ID to a GCP project and rely on kubetest to setup a cluster for you:

export PROJECT_ID=my_gcp_project
test/presubmit-tests.sh --integration-tests

Documentation

Overview

Package test holds the project's test helpers and end-to-end tests (e2e).

Create Pipeline resources

To create build-pipeline objects (e.g. Task, Pipeline, …), you can use the builder (./builder) package to reduce noise:

func MyTest(t *testing.T){
	// Pipeline
	pipeline := tb.Pipeline("tomatoes", "namespace",
		tb.PipelineSpec(tb.PipelineTask("foo", "banana")),
	)
 	// … and PipelineRun
	pipelineRun := tb.PipelineRun("pear", "namespace",
		tb.PipelineRunSpec("tomatoes", tb.PipelineRunServiceAccount("inexistent")),
	)
	// And do something with them
	// […]
	if _, err := c.PipelineClient.Create(pipeline); err != nil {
		t.Fatalf("Failed to create Pipeline `%s`: %s", "tomatoes", err)
	}
	if _, err := c.PipelineRunClient.Create(pipelineRun); err != nil {
		t.Fatalf("Failed to create PipelineRun `%s`: %s", "pear", err)
	}
}

Get access to client objects

To initialize client objects you can use the setup function. It returns a clients struct that contains initialized clients for accessing:

• Kubernetes objects
• Pipelines (https://github.com/knative/build-pipeline#pipeline)

For example, to create a Pipeline

_, err = clients.PipelineClient.Pipelines.Create(test.Pipeline(namespaceName, pipelineName))

And you can use the client to clean up resources created by your test

func tearDown(clients *test.Clients) {
    if clients != nil {
        clients.Delete([]string{routeName}, []string{configName})
    }
}

Poll Pipeline resources

After creating Pipeline resources or making changes to them, you will need to wait for the system to realize those changes. You can use polling methods to check the resources reach the desired state.

The WaitFor* functions use the kubernetes wait package (https://godoc.org/k8s.io/apimachinery/pkg/util/wait). To poll they use PollImmediate (https://godoc.org/k8s.io/apimachinery/pkg/util/wait#PollImmediate) and the return values of the function you provide behave the same as ConditionFunc (https://godoc.org/k8s.io/apimachinery/pkg/util/wait#ConditionFunc): a boolean to indicate if the function should stop or continue polling, and an error to indicate if there has been an error.

For example, you can poll a TaskRun object to wait for it to have a Status.Condition:

err = WaitForTaskRunState(c, hwTaskRunName, func(tr *v1alpha1.TaskRun) (bool, error) {
	if len(tr.Status.Conditions) > 0 {
		return true, nil
	}
	return false, nil
}, "TaskRunHasCondition")

Index

• func CollectBuildLogs(c *clients, podName, namespace string, logger *logging.BaseLogger)
• func GetLogMessages(logs *observer.ObservedLogs) []string
• func SeedTestData(d Data) (Clients, Informers)
• func WaitForPipelineRunState(c *clients, name string, polltimeout time.Duration, inState PipelineRunStateFn, ...) error
• func WaitForPodState(c *clients, name string, namespace string, ...) error
• func WaitForServiceExternalIPState(c *clients, namespace, name string, ...) error
• func WaitForTaskRunState(c *clients, name string, inState TaskRunStateFn, desc string) error
• type Clients
• type Data
• type Informers
• type PipelineRunStateFn
  • func PipelineRunFailed(name string) PipelineRunStateFn
  • func PipelineRunSucceed(name string) PipelineRunStateFn
• type TaskRunStateFn
  • func TaskRunFailed(name string) TaskRunStateFn
  • func TaskRunSucceed(name string) TaskRunStateFn
• type TestAssets

Examples

• WaitForPipelineRunState
• WaitForTaskRunState

Functions

func CollectBuildLogs

func CollectBuildLogs(c *clients, podName, namespace string, logger *logging.BaseLogger)

CollectBuildLogs will get the build logs for a task run

func GetLogMessages

func GetLogMessages(logs *observer.ObservedLogs) []string

GetLogMessages returns a list of all string logs in logs.

func SeedTestData

func SeedTestData(d Data) (Clients, Informers)

SeedTestData returns Clients and Informers populated with the given Data.

func WaitForPipelineRunState

func WaitForPipelineRunState(c *clients, name string, polltimeout time.Duration, inState PipelineRunStateFn, desc string) error

WaitForPipelineRunState polls the status of the PipelineRun called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

Example

// […] setup the test, get clients
if err := WaitForPipelineRunState(c, "pipelineRunName", 1*time.Minute, func(pr *v1alpha1.PipelineRun) (bool, error) {
	if len(pr.Status.Conditions) > 0 {
		return true, nil
	}
	return false, nil
}, "PipelineRunHasCondition"); err != nil {
	t.Errorf("Error waiting for PipelineRun pipelineRunName to finish: %s", err)
}

func WaitForPodState

func WaitForPodState(c *clients, name string, namespace string, inState func(r *corev1.Pod) (bool, error), desc string) error

WaitForPodState polls the status of the Pod called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForServiceExternalIPState

func WaitForServiceExternalIPState(c *clients, namespace, name string, inState func(s *corev1.Service) (bool, error), desc string) error

WaitForServiceExternalIPState polls the status of the a k8s Service called name from client every interval until an external ip is assigned indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

func WaitForTaskRunState

func WaitForTaskRunState(c *clients, name string, inState TaskRunStateFn, desc string) error

WaitForTaskRunState polls the status of the TaskRun called name from client every interval until inState returns `true` indicating it is done, returns an error or timeout. desc will be used to name the metric that is emitted to track how long it took for name to get into the state checked by inState.

Example

// […] setup the test, get clients
if err := WaitForTaskRunState(c, "taskRunName", func(tr *v1alpha1.TaskRun) (bool, error) {
	if len(tr.Status.Conditions) > 0 {
		return true, nil
	}
	return false, nil
}, "TaskRunHasCondition"); err != nil {
	t.Errorf("Error waiting for TaskRun taskRunName to finish: %s", err)
}

Types

type Clients

type Clients struct {
	Pipeline *fakepipelineclientset.Clientset
	Kube     *fakekubeclientset.Clientset
}

Clients holds references to clients which are useful for reconciler tests.

type Data

type Data struct {
	PipelineRuns      []*v1alpha1.PipelineRun
	Pipelines         []*v1alpha1.Pipeline
	TaskRuns          []*v1alpha1.TaskRun
	Tasks             []*v1alpha1.Task
	ClusterTasks      []*v1alpha1.ClusterTask
	PipelineResources []*v1alpha1.PipelineResource
	Pods              []*corev1.Pod
}

Data represents the desired state of the system (i.e. existing resources) to seed controllers with.

type Informers

type Informers struct {
	PipelineRun      informersv1alpha1.PipelineRunInformer
	Pipeline         informersv1alpha1.PipelineInformer
	TaskRun          informersv1alpha1.TaskRunInformer
	Task             informersv1alpha1.TaskInformer
	ClusterTask      informersv1alpha1.ClusterTaskInformer
	PipelineResource informersv1alpha1.PipelineResourceInformer
	Pod              coreinformers.PodInformer
}

Informers holds references to informers which are useful for reconciler tests.

type PipelineRunStateFn

type PipelineRunStateFn func(pr *v1alpha1.PipelineRun) (bool, error)

PipelineRunStateFn is a condition function on TaskRun used polling functions

func PipelineRunFailed

func PipelineRunFailed(name string) PipelineRunStateFn

PipelineRunFailed provides a poll condition function that checks if the PipelineRun has failed.

func PipelineRunSucceed

func PipelineRunSucceed(name string) PipelineRunStateFn

PipelineRunSucceed provides a poll condition function that checks if the PipelineRun has successfully completed.

type TaskRunStateFn

type TaskRunStateFn func(r *v1alpha1.TaskRun) (bool, error)

TaskRunStateFn is a condition function on TaskRun used polling functions

func TaskRunFailed

func TaskRunFailed(name string) TaskRunStateFn

TaskRunFailed provides a poll condition function that checks if the TaskRun has failed.

func TaskRunSucceed

func TaskRunSucceed(name string) TaskRunStateFn

TaskRunSucceed provides a poll condition function that checks if the TaskRun has successfully completed.

type TestAssets

type TestAssets struct {
	Controller *controller.Impl
	Logs       *observer.ObservedLogs
	Clients    Clients
	Informers  Informers
}

TestAssets holds references to the controller, logs, clients, and informers.