conformance

README

This directory contains the regression test for controlling the list of all conformance tests.

If you add or remove a conformance test, this test will fail and you will need to update the golden list of tests stored in testdata/. Changes to that file require review by sig-architecture.

To update the list, run hack/update-conformance-yaml.sh

Add the changed file to your PR, then send for review.

Documentation

Overview

This stand-alone package is utilized for dynamically generating/maintaining a list of conformance tests. It utilizes a two step approach:

• The test binary is built
• The test binary is run in dry mode with a custom ginkgo reporter dumping out types.SpecSummary objects which contain full test names and file/code information.
• The SpecSummary information is parsed to get file/line info on Conformance tests and then we use a simplified AST parser to grab the comments above the test.

Due to the complicated nature of how tests can be declared/wrapped in various contexts, this approach is much simpler to maintain than a pure-AST parser and allows us to easily capture the full test names/locations of the tests using the pre-existing ginkgo logic.