e2e/

README

E2E Tests

E2E tests are run in build-test workflow on pull_request event.

e2e_test.go uses Ginkgo to build a test suite from inventory.go and runs each test using testim/client.go or playwright/playwright.go

Tests are parallelized using Gingko's test focus. Each workflow definition in .github/workflows/build-test.yaml must define a test-focus parameter that matches the Test Name property defined in inventory.go. Each e2e test workflow skips all tests but what is defined in test-focus. Please colocate new playwright tests under the playwright comment with the other pw tests.

New tests should be written with playwright.

Playwright

Playwright is the preferred testing framework for new tests moving forward. See playwright's documentation for more information.

Development environment

To install dependencies run:

cd e2e/playwright
npm ci 
npx playwright install --with-deps

Install the playwright extension in vscode if you've not already done so:

code --install-extension ms-playwright.playwright

Adding a new test

To add a new test that you've already added in the kots-tests-app repo - do the following:

• Update .github/workflows/build-test.yaml to include the new test. You can copy an existing pw entry like validate-change-channel and update the test-focus, kots-namespace, and any other parameters needed for the test.
• Add the test to e2e/inventory.go , making sure the naming matches your kots-test-app and conforms to the naming convention of the other tests in the file:

func NewChangeChannel() Test {
	return Test{
		ID:          "change-channel",
		Name:        "Change Channel",
		Namespace:   "change-channel",
		AppSlug:     "change-channel",
		UpstreamURI: "change-channel/automated",
	}
}

• Add a new inventory test entry to e2e/e2e_test.go to ensure it actually runs:

Entry(nil, inventory.NewChangeChannel()),

• Create a new test directory in e2e/playwright/tests matching your test ID, with the corresponding test file:

$ tree e2e/playwright/tests/change-channel
e2e/playwright/tests/change-channel 
├── license.yaml  // a test specific license if needed
└── test.spec.ts  // the actual test file

• See e2e/playwright/tests/shared for test utility functions that can be used in your test for things like logging in or uploading a license.

testim

Testim is our legacy testing framework. It is being phased out in favor of playwright.

Development environment

To install dependencies run:

make kots
make -C e2e deps
npm install -g @testim/testim-cli

Set the testim access token:

export TESTIM_ACCESS_TOKEN=<my-testim-access-token>

The entire suite can be run with the command:

make e2e

To run an individual test run:

make e2e \
    FOCUS="Change License"

To build and run with ttl.sh images run:

make all-ttl.sh
make e2e \
    KOTSADM_IMAGE_REGISTRY=ttl.sh \
    KOTSADM_IMAGE_NAMESPACE=$USER \
    KOTSADM_IMAGE_TAG=24h

To run using a specific testim branch:

make e2e \
    TESTIM_BRANCH=$BRANCH_NAME

To run against the okteto dev environment run:

Note when using an existing cluster you must focus the suite on a single test

okteto context use https://replicated.okteto.dev
make e2e \
    FOCUS="Change License" \
    EXISTING_KUBECONFIG=${KUBECONFIG:-$HOME/.kube/config}

To skip cluster teardown in order to debug issues:

Note the namespace may be specific to the test

$ make e2e \
    FOCUS="Change License" \
    SKIP_TEARDOWN=1
...
    To set kubecontext run:
      export KUBECONFIG="$(k3d kubeconfig merge kots-e2e3629427925)"
    To delete cluster run:
      k3d cluster delete kots-e2e3629427925
$ export KUBECONFIG="$(k3d kubeconfig merge kots-e2e3629427925)"
$ kubectl -n smoke-test port-forward svc/kotsadm 3000 --address=0.0.0.0
Forwarding from 0.0.0.0:3000 -> 3000

Requirements

Currently, the admin console helm chart will not install on an M1 Macbook because of it's node affinity rules

1. Docker