gotestdox

README

gotestdox is a command-line tool for formatting Go test results as readable documentation, as recommended in my book The Power of Go: Tests.

Here's how to install it:

go install github.com/bitfield/gotestdox/cmd/gotestdox@latest

In any Go project, run:

gotestdox ./...

What does it do?

gotestdox runs your tests and reports the results, but it formats their names in a special way. It converts test names WrittenInCamelCase into ordinary sentences.

For example, suppose we have some tests named like this:

TestValidIsTrueForValidInputs
TestValidIsFalseForInvalidInputs

We can transform them into readably-spaced sentences that express the desired behaviour, by running gotestdox:

gotestdox

This will run the tests, and print:

 ✔ Valid is true for valid inputs (0.00s)
 ✔ Valid is false for invalid inputs (0.00s)

Why?

I read a blog post by Dan North, which says:

My first “Aha!” moment occurred as I was being shown a deceptively simple utility called agiledox, written by my colleague, Chris Stevenson. It takes a JUnit test class and prints out the method names as plain sentences.

The word “test” is stripped from both the class name and the method names, and the camel-case method name is converted into regular text. That’s all it does, but its effect is amazing.

Developers discovered it could do at least some of their documentation for them, so they started to write test methods that were real sentences.
—Dan North, Introducing BDD

How?

The original testdox tool (part of agiledox) was very simple, as Dan describes: it just turned a camel-case JUnit test name like testFailsForDuplicateCustomers into a space-separated sentence like fails for duplicate customers.

And that's what I find neat about it: it's so simple that it hardly seems like it could be of any value, but it is. I've already used the idea to improve a lot of my test names.

There are implementations of testdox for various languages other than Java: for example, PHP, Python, and .NET. I haven't found one for Go, so here it is.

gotestdox reads the JSON output generated by the go test -json command. This is easier than trying to parse Go source code, for example, and also gives us pass/fail information for the tests. It ignores all events except pass/fail events for individual tests (including subtests).

Getting fancy

Some more advanced ways to use gotestdox:

Exit status

If there are any test failures, gotestdox will print the output messages from the offending test and report status 1 on exit.

Colour

gotestdox indicates a passing test with a ✔ (check mark emoji), and a failing test with an x. These are displayed as green and red respectively, using the color library, which automagically detects if it's talking to a colour-capable terminal.

If not (for example, when you redirect output to a file), or if the NO_COLOR environment variable is set to any value, colour output will be disabled.

Test flags and arguments

gotestdox, with no arguments, will run the command go test -json and process its output.

Any arguments you supply will be passed on to go test. For example:

gotestdox -run ParseJSON

will run the command:

go test -json -run ParseJSON

You can supply a list of packages to test, or any other arguments or flags understood by go test. However, gotestdox only prints events about tests (ignoring benchmarks and examples).

Since fuzz test cases are autogenerated and don't tend to have useful names, these are not included in gotestdox output unless they are failing.

Multiple packages

To test all the packages in the current tree, run:

gotestdox ./...

Each package's test results will be prefixed by the fully-qualified name of the package. For example:

github.com/octocat/mymodule/api:
 ✔ NewServer errors on invalid config options (0.00s)
 ✔ NewServer returns a correctly configured server (0.00s)

github.com/octocat/mymodule/util:
 x LeftPad adds the correct number of leading spaces (0.00s)
    util_test.go:133: want "  dummy", got " dummy"

Multi-word function names

There's an ambiguity about test names involving functions whose names contain more than one word. For example, suppose we're testing a function HandleInput, and we write a test like this:

TestHandleInputClosesInputAfterReading

Unless we do something, this will be rendered as:

 ✔ Handle input closes input after reading

To let us give gotestdox a hint about this, there's one extra transformation rule: the first underscore marks the end of the function name. So we can name our test like this:

TestHandleInput_ClosesInputAfterReading

and this becomes:

 ✔ HandleInput closes input after reading

I think this is an acceptable compromise: the gotestdox output is much more readable, while the extra underscore in the test name doesn't seriously interfere with its readability.

The intent is not to perfectly render all sensible test names as sentences, in any case, but to do something useful with them, primarily to encourage developers to write test names that are informative descriptions of the unit's behaviour, and thus (as a side effect) read well when formatted by gotestdox.

In other words, gotestdox is not the thing. It's the thing that gets us to the thing, the end goal being meaningful test names (I like the term literate test names).

Filtering standard input

If you want to run go test -json yourself, for example as part of a shell pipeline, and pipe its output into gotestdox, you can do that too:

go test -json | gotestdox

In this case, any flags or arguments to gotestdox will be ignored, and it won't run the tests; instead, it will act purely as a text filter. However, just as when it runs the tests itself, it will report exit status 1 if there are any test failures.

As a package

See pkg.go.dev/github.com/bitfield/gotestdox for the full documentation on using gotestdox as a package in your own programs.

So what?

Why should you care, then? What's interesting about gotestdox, or any testdox-like tool, I find, is the way its output makes you think about your tests, how you name them, and what they do.

As Dan says in his blog post, turning test names into sentences is a very simple idea, but it has a powerful effect. Test names should be sentences.

Test names should be sentences

I don't know about you, but I've wasted a lot of time and energy over the years trying to choose good names for tests. I didn't really have a way to evaluate whether the name I chose was good or not. Now I do!

In fact, I wrote a whole blog post about it:

• Test names should be sentences

It might be interesting to show your gotestdox output to users, customers, or business folks, and see if it makes sense to them. If so, you're on the right lines. And it's quite likely to generate some interesting conversations (“Is that really what it does? But that's not what we asked for!”)

It seems that I'm not the only one who finds this idea useful. I hear that gotestdox is already being used in some fairly major Go projects and companies, helping their developers to get more value out of their existing tests, and encouraging them to think in interesting new ways about what tests are really for. How nice!

Links

• Bitfield Consulting
• Test names should be sentences
• The Power of Go: Tests

Gopher image by MariaLetta

Documentation

Index

• Constants
• Variables
• func Main() int
• func Prettify(input string) string
• type Event
  • func ParseJSON(line string) (Event, error)
  • func (e Event) IsFuzzFail() bool
  • func (e Event) IsOutput() bool
  • func (e Event) IsPackageResult() bool
  • func (e Event) IsTestResult() bool
  • func (e Event) String() string
• type TestDoxer
  • func NewTestDoxer() *TestDoxer
  • func (td *TestDoxer) ExecGoTest(userArgs []string)
  • func (td *TestDoxer) Filter()

Examples

• Event.IsTestResult (False)
• Event.IsTestResult (True)
• Event.String
• ParseJSON
• Prettify
• Prettify (UnderscoreHint)
• TestDoxer.Filter

Constants

const (
	ActionPass = "pass"
	ActionFail = "fail"
)

const Usage = `` /* 532-byte string literal not displayed */

Variables

var DebugWriter io.Writer = os.Stderr

DebugWriter identifies the stream to which debug information should be printed, if desired. By default it is os.Stderr.

Functions

func Main

func Main() int

Main runs the command-line interface for gotestdox. The exit status for the binary is 0 if the tests passed, or 1 if the tests failed, or there was some error.

func Prettify

func Prettify(input string) string

Prettify takes a string input representing the name of a Go test, and attempts to turn it into a readable sentence, by replacing camel-case transitions and underscores with spaces.

input is expected to be a valid Go test name, as produced by 'go test -json'. For example, input might be the string:

TestFoo/has_well-formed_output

Here, the parent test is TestFoo, and this data is about a subtest whose name is 'has well-formed output'. Go's testing package replaces spaces in subtest names with underscores, and unprintable characters with the equivalent Go literal.

Prettify does its best to reverse this transformation, yielding (something close to) the original subtest name. For example:

Foo has well-formed output

Multiword function names

Because Go function names are often in camel-case, there's an ambiguity in parsing a test name like this:

TestHandleInputClosesInputAfterReading

We can see that this is about a function named HandleInput, but Prettify has no way of knowing that. Without this information, it would produce:

Handle input closes input after reading

To give it a hint, we can add an underscore after the name of the function:

TestHandleInput_ClosesInputAfterReading

This will be interpreted as marking the end of a multiword function name:

HandleInput closes input after reading

Debugging

If the GOTESTDOX_DEBUG environment variable is set, Prettify will output (copious) debug information to the DebugWriter stream, elaborating on its decisions.

Example

package main

import (
	"fmt"

	"github.com/bitfield/gotestdox"
)

func main() {
	input := "TestFoo/has_well-formed_output"
	fmt.Println(gotestdox.Prettify(input))
}

Output:

Foo has well-formed output

Example (UnderscoreHint)

package main

import (
	"fmt"

	"github.com/bitfield/gotestdox"
)

func main() {
	input := "TestHandleInput_ClosesInputAfterReading"
	fmt.Println(gotestdox.Prettify(input))
}

Output:

HandleInput closes input after reading

Types

type Event

type Event struct {
	Action   string
	Package  string
	Test     string
	Sentence string
	Output   string
	Elapsed  float64
}

Event represents a Go test event as recorded by the 'go test -json' command. It does not attempt to unmarshal all the data, only those fields it needs to know about. It is based on the (unexported) 'event' struct used by Go's cmd/internal/test2json package.

func ParseJSON

func ParseJSON(line string) (Event, error)

ParseJSON takes a string representing a single JSON test record as emitted by 'go test -json', and attempts to parse it into an Event, returning any parsing error encountered.

Example

package main

import (
	"fmt"
	"log"

	"github.com/bitfield/gotestdox"
)

func main() {
	input := `{"Action":"pass","Package":"demo","Test":"TestItWorks","Output":"","Elapsed":0.2}`
	event, err := gotestdox.ParseJSON(input)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%#v\n", event)
}

Output:

gotestdox.Event{Action:"pass", Package:"demo", Test:"TestItWorks", Sentence:"", Output:"", Elapsed:0.2}

func (Event) IsFuzzFail

func (e Event) IsFuzzFail() bool

func (Event) IsOutput

func (e Event) IsOutput() bool

IsOutput determines whether or not the event is a test output (for example from testing.T.Error), excluding status messages automatically generated by 'go test' such as "--- FAIL: ..." or "=== RUN / PAUSE / CONT".

func (Event) IsPackageResult

func (e Event) IsPackageResult() bool

IsPackageResult determines whether or not the test event is a package pass or fail event. That is, whether it indicates the passing or failing of a package as a whole, rather than some individual test within the package.

func (Event) IsTestResult

func (e Event) IsTestResult() bool

IsTestResult determines whether or not the test event is one that we are interested in (namely, a pass or fail event on a test). Events on non-tests (for example, examples) are ignored, and all events on tests other than pass or fail events (for example, run or pause events) are also ignored.

Example (False)

package main

import (
	"fmt"

	"github.com/bitfield/gotestdox"
)

func main() {
	event := gotestdox.Event{
		Action: "fail",
		Test:   "ExampleEventsShouldBeIgnored",
	}
	fmt.Println(event.IsTestResult())
}

Output:

false

Example (True)

package main

import (
	"fmt"

	"github.com/bitfield/gotestdox"
)

func main() {
	event := gotestdox.Event{
		Action: "pass",
		Test:   "TestItWorks",
	}
	fmt.Println(event.IsTestResult())
}

Output:

true

func (Event) String

func (e Event) String() string

String formats a test Event for display. The prettified test name will be prefixed by a ✔ if the test passed, or an x if it failed.

The sentence generated by Prettify from the name of the test will be shown, followed by the elapsed time in parentheses, to 2 decimal places.

Colour

If the program is attached to an interactive terminal, as determined by github.com/mattn/go-isatty, and the NO_COLOR environment variable is not set, check marks will be shown in green and x's in red.

Example

package main

import (
	"fmt"

	"github.com/bitfield/gotestdox"
	"github.com/fatih/color"
)

func main() {
	event := gotestdox.Event{
		Action:   "pass",
		Sentence: "It works",
	}
	color.NoColor = true
	fmt.Println(event.String())
}

Output:

✔ It works (0.00s)

type TestDoxer

type TestDoxer struct {
	Stdin          io.Reader
	Stdout, Stderr io.Writer
	OK             bool
}

TestDoxer holds the state and config associated with a particular invocation of 'go test'.

func NewTestDoxer

func NewTestDoxer() *TestDoxer

NewTestDoxer returns a *TestDoxer configured with the default I/O streams: os.Stdin, os.Stdout, and os.Stderr.

func (*TestDoxer) ExecGoTest

func (td *TestDoxer) ExecGoTest(userArgs []string)

ExecGoTest runs the 'go test -json' command, with any extra args supplied by the user, and consumes its output. Any errors are reported to td's Stderr stream, including the full command line that was run. If all tests passed, td.OK will be true. If there was a test failure, or 'go test' returned some error, then td.OK will be false.

func (*TestDoxer) Filter

func (td *TestDoxer) Filter()

Filter reads from td's Stdin stream, line by line, processing JSON records emitted by 'go test -json'.

For each Go package it sees records about, it will print the full name of the package to td.Stdout, followed by a line giving the pass/fail status and the prettified name of each test, sorted alphabetically.

If all tests passed, td.OK will be true at the end. If not, or if there was a parsing error, it will be false. Errors will be reported to td.Stderr.

Example

package main

import (
	"strings"

	"github.com/bitfield/gotestdox"
	"github.com/fatih/color"
)

func main() {
	input := `{"Action":"pass","Package":"demo","Test":"TestItWorks"}
	{"Action":"pass","Package":"demo","Elapsed":0}`
	td := gotestdox.NewTestDoxer()
	td.Stdin = strings.NewReader(input)
	color.NoColor = true
	td.Filter()
}

Output:

demo:
 ✔ It works (0.00s)