e2e/

README

End-to-End Testing

This directory contains Istio end-to-end tests and associated test framework.

E2E-bookInfoTest and E2E-smokeTest (both with auth enabled) is automatically triggered in Before-Merge stage for every PR. The full suite of E2E tests are run in After-Merge stage only. Their results can be found in https://prow.istio.io/ and https://k8s-testgrid.appspot.com/istio

Running E2E tests in PR stage

E2E-suites can be manually triggered in PR stage with specific command:

/test e2e-suite-rbac-no_auth

/test e2e-suite-rbac-auth

/test e2e-cluster_wide-auth

NOTE: Please do "/test e2e-*" only after "prow/istio-presubmit" completes, as the e2e tests need the artifacts generated by istio-presubmit.

Running E2E tests on your own kubernets cluster

NOTE: the e2e tests might not run on a Mac because istioctl-osx, needed for test execution, is only built for release builds and not for normal presubmits jobs, see examples for solutions.

• Step 1: Create a kubernetes cluster
• Step 2: Get cluster credentials
• Step 3: Create Clusterrolebinding
• Step 4: Export test script variables
• Step 5: Run
• Examples

Step 1: Create and setup a kubernetes cluster

E2E tests require a Kubernetes cluster. You can create one using the Google Container Engine using the following command:

gcloud container clusters \
  create ${CLUSTER_NAME} \
  --zone ${ZONE} \
  --project ${PROJECT_NAME} \
  --cluster-version ${CLUSTER_VERSION} \
  --machine-type ${MACHINE_TYPE} \
  --num-nodes ${NUM_NODES} \
  --enable-kubernetes-alpha \
  --no-enable-legacy-authorization

• CLUSTER_NAME: Whatever suits your fancy, 'istio-e2e' is a good choice.
• ZONE: 'us-central-f' is a good value to use.
• PROJECT_NAME: is the name of the GCP project that will house the cluster. You get a project by visiting GCP.
• CLUSTER_VERSION: 1.7.3 or later.
• MACHINE_TYPE: Use 'n1-standard-4'
• NUM_NODES: Use 3.
• no-enable-legacy-authorization: Optional, needed if you want to test RBAC.

You must set your default compute service account to include: roles/container.admin (Kubernetes Engine Admin) Editor (on by default)

To set this, navigate to the IAM section of the Cloud Console and find your default GCE/GKE service account in the following form: projectNumber-compute@developer.gserviceaccount.com: by default it should just have the Editor role. Then in the Roles drop-down list for that account, find the Kubernetes Engine group and select the role Kubernetes Engine Admin. The Roles listing for your account will change to Multiple.

Step 2: Get cluster credentials

gcloud container clusters get-credentials ${CLUSTER_NAME} --zone ${ZONE} --project ${PROJECT_NAME}

Step 3: Create Clusterrolebinding

kubectl create clusterrolebinding myname-cluster-admin-binding    --clusterrole=cluster-admin    --user="<user_email>"

• user_email should be the one you use to log in gcloud command. You can do gcloud info to find out current user info.

Step 4: Export test script variables

Option 1: Build your own images.

# Customize .istiorc.mk (at the top of the istio.io/istio source tree) with your HUB and optional TAG
# it allows you to customize Makefile rules. For example:
cat .istiorc.mk
HUB=costinm
TAG=mybranch

# Build images on the local docker.
make docker

# Push images to docker registry
# If you use minikube and its docker environment, images will be  available in minikube for use,
# you can skip this step.
make push

# the hub/tag set in your .istiorc.mk will be used by the test.

Option 2: Already committed changes to istio/istio master branch NOTE: SHA used as TAG is one that is already committed on istio/istio. You can pick any SHA you want.

export HUB="gcr.io/istio-testing"
export TAG="d0142e1afe41c18917018e2fa85ab37254f7e0ca"

Option 3: Testing local changes

If you want to test on uncommitted changes to master istio:

• Create a PR with your change.

• This will trigger istio-presubmit.sh. At the end of this script, it creates docker images for mixer, pilot, ca, with your changes and upload them to container registry. See the logs of this istio-presubmit.sh and at the end there must be a SHA which you need to copy and set it as a GIT_SHA. Example from a log: the SHA to copy is marked in bold

  I1207 04:42:40.881] 0077bb73e0b9d2841f8c299f15305193e42dae0d: digest: sha256:6f72528d475be56e8392bc3b833b94a815a1fbab8a70cd058b92982e61364021 size: 528

• Then set the export variables again

export HUB="gcr.io/istio-testing"
export TAG="<sha copied from the logs of istio-presubmit.sh>"

Step 5: Run

From the repo checkout root directory

make e2e_all E2E_ARGS="--skip_cleanup"

Each test has its own directory and can be run independently as a go_test target. The script has a number of options:

• --skip_cleanup - to skip cleanup steps
• --namespace <namespace> : If you don't specify namespace, a random namespace is generated for each test.
• --verbose <debug level noise from proxies>
• --istioctl <local istioctl path>: Use local istioctl binary (i.e. ${GOPATH}/out/linux_amd64/release/istioctl).
• --istioctl_url <remote istioctl url>: If local path is not defined, download istioctl from a remote location.
• --use_local_cluster: If running on minikube, this should be set to true.
• --auth_enable - if you want to include auth
• --cluster_wide - if you want to run the cluster wide installation and tests
• --use_automatic_injection - if you want to do transparent sidecar injection
• --use_galley_config_validator - if you want to enable automatic configuration validation
• --mixer_hub <mixer image hub>
• --mixer_tag <mixer image tag>
• --pilot_hub <pilot image hub>
• --pilot_tag <pilot image tag>
• --proxy_hub <proxy image hub>
• --proxy_tag <proxy image tag>
• --ca_hub <CA image hub>
• --ca_tag <CA image tag>
• --galley_hub <galley image hub>
• --galley_tag <galley image tag>

Examples

• Running on Mac:

  Although istioctl-osx currently is not built during presubmit/postsubmit, only in release process. You can build your own istioctl from source or download a release version from release page (although it's not the latest one), and then use --istioctl instead of --istioctl_url to specify the local path.

  make e2e_all E2E_ARGS="--skip_cleanup"
  

• Running single test file (bookinfo, mixer, simple) and also skip cleanup.

  make e2e_bookinfo E2E_ARGS="--skip_cleanup"
  

Please see golang testing options for more information.

go test --help

• For simple test specific, you can run test multiple time against the same environement setup by skip_setup:

  # First time you want to run: deploy in namespace e2e and leave it running:
  make e2e_simple E2E_ARGS="--skip_cleanup --namespace=e2e -istioctl ~/istioctl-osx --auth_enable"
  # Subsequent runs if only the TestSimpleIngress (for instance) changes:
  make e2e_simple E2E_ARGS="--skip_setup --skip_cleanup --namespace=e2e -istioctl ~/istioctl-osx --auth_enable --test.run TestSimpleIngress"
  

• Running pilot e2e tests locally on minikube:

go test -v ./tests/e2e/tests/pilot \
  --use_local_cluster \
  --istioctl="${GOPATH}/out/linux_amd64/release/istioctl"

demo_test.go

demo_test.go is a sample test. It's based on the shell script version of demo test. It has four test cases: default routing, version routing, fault delay and version migration. Each test case applies specific rules for itself and clean them up after finishing.

You can build and run this or any single test manually with the same options when testing specific version of master, mixer or istioctl

Writing tests

Follow the sample of demo_test.go

1. Create a new commonConfig for framework and add app used for this test in setTestConfig(). Each test file has a testConfig handling framework and test configuration. testConfig is a cleanable structure which has Setup and Teardown. Setup will run before all tests and Teardown is going to clean up after all tests.
2. Framework would handle all setting up: install and setup istio, deploy app.
3. Setup test-specific environment, like generate rule files from templates and apply routing rules. These could be done in testConfig.Setup() and would be executed by cleanup register right after framework setup.
4. Write a test. Test case name should start with 'Test' and using 't *testing.T' to log test failures. There is no guarantee for running order

Cluster in same local network

In order to talk to istio ingress, we use the ingress IP by default. If your cluster is on the same local network and cannot provide external IP (for example, minikube), use the --use-local-cluster flag. In that case, the framework will not create a LoadBalancer and talk directly to the Pod running istio-ingress.

Debugging

The requests from the containers deployed in tests are performed by client program. For example, to perform a call from a deployed test container to https://console.bluemix.net/, run:

kubectl exec -it <test pod> -n <test namespace> -c app -- client -url https://console.bluemix.net/

To see its usage run:

kubectl exec -it <test pod> -n <test namespace> -c app -- client -h

Test framework notes

The E2E test framework defines and creates structures and processes for creating cleanable test environments: install and setup istio modules and clean up afterward.

Writing new tests doesn't require knowledge of the framework.

• framework.go: Cleanable is a interface defined with setup() and teardown(). While initialization, framework calls setup() from all registered cleanable structures and calls teardown() while framework cleanup. The cleanable register works like a stack, first setup, last teardown.

• kubernetes.go: KubeInfo handles interactions between tests and kubectl, installs istioctl and apply istio module. Module yaml files are in store at install/kubernetes/templates and will finally use all-in-one yaml istio.yaml

• appManager.go: gather apps required for test into a array and deploy them while setup()