README
¶
Please Tests
Reporting
Tests are reported in much the same output format as the Maven Surefire Plugin. This makes it easy to consume the results in popular CI tools with a minimum of effort.
After executing plz test against one or more targets, the test results are
written to plz-out/log/test_results.xml.
We no longer support the idea of "expected failures" as people who write tests they expect to fail will find they are first against the wall when the revolution comes.
Executing
Executing plz test against a target produces a set of results that are
collated into one core.TestSuite with the package and name matching the
target's label (this is a slight departure from the normal package/classname
decomposition as it does not particularly translate well to non-Java languages).
If any tests fail, and the target is marked as flaky, it will be run again
(up to flaky times). Tests can also be explicitly run multiple times by
passing --num-runs n on the command line. We no longer accept a default of 0
as running a test 0 times is (a) not useful and (b) not what actually happened.
Each executed test during this target becomes a core.TestExecution under a
core.TestCase in this test suite - so that multiple executions of a test
are put together. We ignore multiple successes or skips as they are not interesting.
In the case of a flaky test where some runs succeed and some fail, we store at
most one success but also all of the failures and errors.
The time of the testsuite is the total time it took to execute all of the
tests (including any re-runs for flakes).
Once all of the targets have been executed, these core.TestSuites are
collected into one core.TestSuites object. This is then rendered as XML
to the output file mentioned above.
Documentation
¶
Index ¶
- func AddOriginalTargetsToCoverage(state *core.BuildState, includeAllFiles bool)
- func CopySurefireXmlFilesToDir(state *core.BuildState, surefireDir string)
- func CountCoverage(lines []core.LineCoverage) (int, int)
- func LoadPreviousFailures(filename string) ([]core.BuildLabel, []string)
- func RemoveFilesFromCoverage(coverage core.TestCoverage, extensions []string)
- func RemoveTestOutputs(target *core.BuildTarget) error
- func Test(tid int, state *core.BuildState, label core.BuildLabel)
- func WriteCoverageToFileOrDie(coverage core.TestCoverage, filename string)
- func WriteResultsToFileOrDie(graph *core.BuildGraph, filename string)
- func WriteXMLCoverageToFileOrDie(sources []core.BuildLabel, coverage core.TestCoverage, filename string)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AddOriginalTargetsToCoverage ¶
func AddOriginalTargetsToCoverage(state *core.BuildState, includeAllFiles bool)
AddOriginalTargetsToCoverage adds empty coverage entries for any files covered by the original query that we haven't discovered through tests to the overall report. The coverage reports only contain information about files that were covered during tests, so it's important that we identify anything with zero coverage here. This is made trickier by attempting to reconcile coverage targets from languages like Java that don't preserve the original file structure, which requires a slightly fuzzy match.
func CopySurefireXmlFilesToDir ¶
func CopySurefireXmlFilesToDir(state *core.BuildState, surefireDir string)
CopySurefireXmlFilesToDir copies all the XML test results files into the given directory.
func CountCoverage ¶
func CountCoverage(lines []core.LineCoverage) (int, int)
CountCoverage counts the number of lines covered and the total number coverable in a single file.
func LoadPreviousFailures ¶
func LoadPreviousFailures(filename string) ([]core.BuildLabel, []string)
LoadPreviousFailures loads any failed tests from the given results file. It returns the set of targets that should be run and any arguments for them.
func RemoveFilesFromCoverage ¶
func RemoveFilesFromCoverage(coverage core.TestCoverage, extensions []string)
RemoveFilesFromCoverage removes any files with extensions matching the given set from coverage.
func RemoveTestOutputs ¶
func RemoveTestOutputs(target *core.BuildTarget) error
RemoveTestOutputs removes any cached test or coverage result files for a target.
func Test ¶
func Test(tid int, state *core.BuildState, label core.BuildLabel)
Test runs the tests for a single target.
func WriteCoverageToFileOrDie ¶
func WriteCoverageToFileOrDie(coverage core.TestCoverage, filename string)
WriteCoverageToFileOrDie writes the collected coverage data to a file in JSON format. Dies on failure.
func WriteResultsToFileOrDie ¶
func WriteResultsToFileOrDie(graph *core.BuildGraph, filename string)
WriteResultsToFileOrDie writes test results out to a file in xUnit format. Dies on any errors.
func WriteXMLCoverageToFileOrDie ¶
func WriteXMLCoverageToFileOrDie(sources []core.BuildLabel, coverage core.TestCoverage, filename string)
WriteXMLCoverageToFileOrDie writes the collected coverage data to a file in XML format. Dies on failure.
Types ¶
This section is empty.
Source Files