test

README

Please Tests

Reporting

Tests are reported in much the same output format as the Maven Surefire Plugin. This makes it easy to consume the results in popular CI tools with a minimum of effort.

After executing plz test against one or more targets, the test results are written to plz-out/log/test_results.xml.

We no longer support the idea of "expected failures" as people who write tests they expect to fail will find they are first against the wall when the revolution comes.

Executing

Executing plz test against a target produces a set of results that are collated into one core.TestSuite with the package and name matching the target's label (this is a slight departure from the normal package/classname decomposition as it does not particularly translate well to non-Java languages).

If any tests fail, and the target is marked as flaky, it will be run again (up to flaky times). Tests can also be explicitly run multiple times by passing --num-runs n on the command line. We no longer accept a default of 0 as running a test 0 times is (a) not useful and (b) not what actually happened.

Each executed test during this target becomes a core.TestExecution under a core.TestCase in this test suite - so that multiple executions of a test are put together. We ignore multiple successes or skips as they are not interesting. In the case of a flaky test where some runs succeed and some fail, we store at most one success but also all of the failures and errors.

The time of the testsuite is the total time it took to execute all of the tests (including any re-runs for flakes).

Once all of the targets have been executed, these core.TestSuites are collected into one core.TestSuites object. This is then rendered as XML to the output file mentioned above.

Documentation

Index

• func AddOriginalTargetsToCoverage(state *core.BuildState, includeAllFiles bool)
• func CopySurefireXmlFilesToDir(state *core.BuildState, surefireDir string)
• func CountCoverage(lines []core.LineCoverage) (int, int)
• func LoadPreviousFailures(filename string) ([]core.BuildLabel, []string)
• func RemoveFilesFromCoverage(coverage core.TestCoverage, extensions []string)
• func RemoveTestOutputs(target *core.BuildTarget) error
• func Test(tid int, state *core.BuildState, label core.BuildLabel)
• func WriteCoverageToFileOrDie(coverage core.TestCoverage, filename string)
• func WriteResultsToFileOrDie(graph *core.BuildGraph, filename string)
• func WriteXMLCoverageToFileOrDie(sources []core.BuildLabel, coverage core.TestCoverage, filename string)

Functions

func AddOriginalTargetsToCoverage

func AddOriginalTargetsToCoverage(state *core.BuildState, includeAllFiles bool)

AddOriginalTargetsToCoverage adds empty coverage entries for any files covered by the original query that we haven't discovered through tests to the overall report. The coverage reports only contain information about files that were covered during tests, so it's important that we identify anything with zero coverage here. This is made trickier by attempting to reconcile coverage targets from languages like Java that don't preserve the original file structure, which requires a slightly fuzzy match.

func CopySurefireXmlFilesToDir

func CopySurefireXmlFilesToDir(state *core.BuildState, surefireDir string)

CopySurefireXmlFilesToDir copies all the XML test results files into the given directory.

func CountCoverage

func CountCoverage(lines []core.LineCoverage) (int, int)

CountCoverage counts the number of lines covered and the total number coverable in a single file.

func LoadPreviousFailures

func LoadPreviousFailures(filename string) ([]core.BuildLabel, []string)

LoadPreviousFailures loads any failed tests from the given results file. It returns the set of targets that should be run and any arguments for them.

func RemoveFilesFromCoverage

func RemoveFilesFromCoverage(coverage core.TestCoverage, extensions []string)

RemoveFilesFromCoverage removes any files with extensions matching the given set from coverage.

func RemoveTestOutputs

func RemoveTestOutputs(target *core.BuildTarget) error

RemoveTestOutputs removes any cached test or coverage result files for a target.

func Test

func Test(tid int, state *core.BuildState, label core.BuildLabel)

Test runs the tests for a single target.

func WriteCoverageToFileOrDie

func WriteCoverageToFileOrDie(coverage core.TestCoverage, filename string)

WriteCoverageToFileOrDie writes the collected coverage data to a file in JSON format. Dies on failure.

func WriteResultsToFileOrDie

func WriteResultsToFileOrDie(graph *core.BuildGraph, filename string)

WriteResultsToFileOrDie writes test results out to a file in xUnit format. Dies on any errors.

func WriteXMLCoverageToFileOrDie

func WriteXMLCoverageToFileOrDie(sources []core.BuildLabel, coverage core.TestCoverage, filename string)

WriteXMLCoverageToFileOrDie writes the collected coverage data to a file in XML format. Dies on failure.