upgrade

README

Upgrade Tests

In order to get coverage for the upgrade process from an operator’s perspective, we need an additional suite of tests that perform a complete knative upgrade. Running these tests on every commit will ensure that we don’t introduce any non-upgradeable changes, so every commit should be releasable.

This is inspired by kubernetes upgrade testing.

These tests are a pretty big hammer in that they cover more than just version changes, but it’s one of the only ways to make sure we don’t accidentally make breaking changes for now.

Flow

We’d like to validate that the upgrade doesn’t break any resources (they still respond to requests) and doesn't break our installation (we can still update resources).

At a high level, we want to do this:

1. Install the latest knative release.
2. Create some resources.
3. Install knative at HEAD.
4. Test those resources, verify that we didn’t break anything.

To achieve that, we just have three separate build tags:

1. Install the latest release from GitHub.
2. Run the preupgrade tests in this directory.
3. Install at HEAD (ko apply -f config/).
4. Run the postupgrade tests in this directory.
5. Install the latest release from GitHub.
6. Run the postdowngrade tests in this directory.

Tests

Service test

This was stolen from the conformance tests but stripped down to check fewer things.

preupgrade

Create a RunLatest Service pointing to image1, ensure it responds correctly.

postupgrade

Ensure the Service still responds correctly after upgrading. Update it to point to image2, ensure it responds correctly.

postdowngrade

Ensure the Service still responds correctly after downgrading. Update it to point back to image1, ensure it responds correctly.

Probe test

In order to verify that we don't have data-plane unavailability during our control-plane outages (when we're upgrading the knative/serving installation), we run a prober test that continually sends requests to a service during the entire upgrade process. When the upgrade completes, we make sure that none of those requests failed.