harness

Documentation

Overview

Package harness provides a reusable test framework akin to the standard "testing" Go package. For now there is no automated code generation component like the "go test" command but that may be a future extension. Test functions must be of type `func(*harness.H)` and registered directly with a test Suite struct which can then be launched via the Run method.

Within these functions, use the Error, Fail or related methods to signal failure.

Tests may be skipped if not applicable with a call to the Skip method of *H:

func NeedsSomeData(h *harness.H) {
    if os.Getenv("SOME_DATA") == "" {
        h.Skip("skipping test due to missing SOME_DATA")
    }
    ...
}

Subtests

The Run method of H allow defining subtests, without having to define separate functions for each. This enables uses like table-driven and hierarchical tests. It also provides a way to share common setup and tear-down code:

func Foo(h *harness.H) {
    // <setup code>
    h.Run("A=1", func(h *harness.H) { ... })
    h.Run("A=2", func(h *harness.H) { ... })
    h.Run("B=1", func(h *harness.H) { ... })
    // <tear-down code>
}

Each subtest has a unique name: the combination of the name of the top-level test and the sequence of names passed to Run, separated by slashes, with an optional trailing sequence number for disambiguation.

The argument to the -harness.run command-line flag is an unanchored regular expression that matches the test's name. For tests with multiple slash-separated elements, such as subtests, the argument is itself slash-separated, with expressions matching each name element in turn. Because it is unanchored, an empty expression matches any string. For example, using "matching" to mean "whose name contains":

go run foo.go -harness.run ''      # Run all tests.
go run foo.go -harness.run Foo     # Run top-level tests matching "Foo", such as "TestFooBar".
go run foo.go -harness.run Foo/A=  # For top-level tests matching "Foo", run subtests matching "A=".
go run foo.go -harness.run /A=1    # For all top-level tests, run subtests matching "A=1".

Subtests can also be used to control parallelism. A parent test will only complete once all of its subtests complete. In this example, all tests are run in parallel with each other, and only with each other, regardless of other top-level tests that may be defined:

func GroupedParallel(h *harness.H) {
    for _, tc := range tests {
        tc := tc // capture range variable
        h.Run(tc.Name, func(h *harness.H) {
            h.Parallel()
            ...
        })
    }
}

Run does not return until parallel subtests have completed, providing a way to clean up after a group of parallel tests:

func TeardownParallel(h *harness.H) {
    // This Run will not return until the parallel tests finish.
    h.Run("group", func(h *harness.H) {
        h.Run("Test1", parallelTest1)
        h.Run("Test2", parallelTest2)
        h.Run("Test3", parallelTest3)
    })
    // <tear-down code>
}

Suite

Individual tests are grouped into a test suite in order to execute them. TODO: this part of the API deviates from the "testing" package and is TBD.

A simple implementation of a test suite:

func SomeTest(h *harness.H) {
	h.Skip("TODO")
}

func main() {
	suite := harness.NewSuite(Options{}, Tests{
		"SomeTest": SomeTest,
	})
	if err := suite.Run(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		fmt.Println("FAIL")
		os.Exit(1)
	}
	fmt.Println("PASS")
}

Index

• Constants
• Variables
• func CleanOutputDir(path string) (string, error)
• type H
  • func (c *H) Context() context.Context
  • func (c *H) Error(args ...interface{})
  • func (c *H) Errorf(format string, args ...interface{})
  • func (c *H) Fail()
  • func (c *H) FailNow()
  • func (c *H) Failed() bool
  • func (c *H) Fatal(args ...interface{})
  • func (c *H) Fatalf(format string, args ...interface{})
  • func (c *H) GetNonExclusiveTestStarted() bool
  • func (c *H) Log(args ...interface{})
  • func (c *H) Logf(format string, args ...interface{})
  • func (c *H) Name() string
  • func (c *H) NonExclusiveTestStarted()
  • func (h *H) OutputDir() string
  • func (t *H) Parallel()
  • func (t *H) Release()
  • func (t *H) Run(name string, f func(t *H)) bool
  • func (t *H) RunTimeout(name string, f func(t *H), timeout time.Duration) bool
  • func (t *H) RunWithExecTimeoutCheck(f func(), errMsg string)
  • func (c *H) SetSubtests(subtests []string)
  • func (c *H) Skip(args ...interface{})
  • func (c *H) SkipNow()
  • func (c *H) Skipf(format string, args ...interface{})
  • func (c *H) Skipped() bool
  • func (t *H) StartExecTimer()
  • func (t *H) StopExecTimer()
  • func (c *H) Subtests() []string
  • func (h *H) TempDir(prefix string) string
  • func (h *H) TempFile(prefix string) *os.File
  • func (c *H) TimedOut() bool
  • func (h *H) Verbose() bool
  • func (c *H) WarningOnFailure()
• type HarnessTest
• type Options
  • func (o *Options) FlagSet(prefix string, errorHandling flag.ErrorHandling) *flag.FlagSet
• type Suite
  • func NewSuite(opts Options, tests Tests) *Suite
  • func (s *Suite) Run() (err error)
• type Test
• type Tests
  • func (ts *Tests) Add(name string, test Test, timeout time.Duration)
  • func (ts Tests) List() []string

Constants

const DefaultTimeoutFlag = 0

Variables

var (
	SuiteEmpty  = errors.New("harness: no tests to run")
	SuiteFailed = errors.New("harness: test suite failed")
)

Functions

func CleanOutputDir

func CleanOutputDir(path string) (string, error)

CleanOutputDir creates/empties an output directory and returns the cleaned path. If the path already exists it must be named similar to `_foo_temp` or contain `.harness_temp` to indicate removal is safe; we don't want users wrecking things by accident with `--output-dir /tmp`

Types

type H

type H struct {
	// contains filtered or unexported fields
}

H is a type passed to Test functions to manage test state and support formatted test logs. Logs are accumulated during execution and dumped to standard output when done.

A test ends when its Test function returns or calls any of the methods FailNow, Fatal, Fatalf, SkipNow, Skip, or Skipf. Those methods, as well as the Parallel method, must be called only from the goroutine running the Test function.

The other reporting methods, such as the variations of Log and Error, may be called simultaneously from multiple goroutines.

func (*H) Context

func (c *H) Context() context.Context

Context returns the context for the current test. The context is cancelled when the test finishes. A goroutine started during a test can wait for the context's Done channel to become readable as a signal that the test is over, so that the goroutine can exit.

func (*H) Error

func (c *H) Error(args ...interface{})

Error is equivalent to Log followed by Fail.

func (*H) Errorf

func (c *H) Errorf(format string, args ...interface{})

Errorf is equivalent to Logf followed by Fail.

func (*H) Fail

func (c *H) Fail()

Fail marks the function as having failed but continues execution.

func (*H) FailNow

func (c *H) FailNow()

FailNow marks the function as having failed and stops its execution. Execution will continue at the next test. FailNow must be called from the goroutine running the test function, not from other goroutines created during the test. Calling FailNow does not stop those other goroutines.

func (*H) Failed

func (c *H) Failed() bool

Failed reports whether the function has failed.

func (*H) Fatal

func (c *H) Fatal(args ...interface{})

Fatal is equivalent to Log followed by FailNow.

func (*H) Fatalf

func (c *H) Fatalf(format string, args ...interface{})

Fatalf is equivalent to Logf followed by FailNow.

func (*H) GetNonExclusiveTestStarted

func (c *H) GetNonExclusiveTestStarted() bool

func (*H) Log

func (c *H) Log(args ...interface{})

Log formats its arguments using default formatting, analogous to Println, and records the text in the error log. The text will be printed only if the test fails or the -harness.v flag is set.

func (*H) Logf

func (c *H) Logf(format string, args ...interface{})

Logf formats its arguments according to the format, analogous to Printf, and records the text in the error log. A final newline is added if not provided. The text will be printed only if the test fails or the -harness.v flag is set.

func (*H) Name

func (c *H) Name() string

Name returns the name of the running test or benchmark.

func (*H) NonExclusiveTestStarted

func (c *H) NonExclusiveTestStarted()

func (*H) OutputDir

func (h *H) OutputDir() string

OutputDir returns the path to a directory for storing data used by the current test. Only test frameworks should care about this. Individual tests should normally use H.TempDir or H.TempFile

func (*H) Parallel

func (t *H) Parallel()

Parallel signals that this test is to be run in parallel with (and only with) other parallel tests.

func (*H) Release

func (t *H) Release()

This functionn is robust to being called multiple times. Previously t.suite.release() was only called by tRunner, so tRunner ensured that a test is released only once. Since we are now exposing the release mechanism outside the package level, it is important to introduce idempotence to avoid any corrupted test queue states that may result from one test being released multiple times.

func (*H) Run

func (t *H) Run(name string, f func(t *H)) bool

Run runs f as a subtest of t called name. It reports whether f succeeded. Run will block until all its parallel subtests have completed.

func (*H) RunTimeout

func (t *H) RunTimeout(name string, f func(t *H), timeout time.Duration) bool

func (*H) RunWithExecTimeoutCheck

func (t *H) RunWithExecTimeoutCheck(f func(), errMsg string)

func (*H) SetSubtests

func (c *H) SetSubtests(subtests []string)

func (*H) Skip

func (c *H) Skip(args ...interface{})

Skip is equivalent to Log followed by SkipNow.

func (*H) SkipNow

func (c *H) SkipNow()

SkipNow marks the test as having been skipped and stops its execution. If a test fails (see Error, Errorf, Fail) and is then skipped, it is still considered to have failed. Execution will continue at the next test. See also FailNow. SkipNow must be called from the goroutine running the test, not from other goroutines created during the test. Calling SkipNow does not stop those other goroutines.

func (*H) Skipf

func (c *H) Skipf(format string, args ...interface{})

Skipf is equivalent to Logf followed by SkipNow.

func (*H) Skipped

func (c *H) Skipped() bool

Skipped reports whether the test was skipped.

func (*H) StartExecTimer

func (t *H) StartExecTimer()

func (*H) StopExecTimer

func (t *H) StopExecTimer()

func (*H) Subtests

func (c *H) Subtests() []string

Subtests returns the list of subtests

func (*H) TempDir

func (h *H) TempDir(prefix string) string

TempDir creates a new directory under OutputDir. No cleanup is required.

func (*H) TempFile

func (h *H) TempFile(prefix string) *os.File

TempFile creates a new file under Outputdir. No cleanup is required.

func (*H) TimedOut

func (c *H) TimedOut() bool

Reports if the test was stopped because a timeout was reached.

func (*H) Verbose

func (h *H) Verbose() bool

Verbose reports whether the Suite's Verbose option is set.

func (*H) WarningOnFailure

func (c *H) WarningOnFailure()

type HarnessTest

type HarnessTest struct {
	// contains filtered or unexported fields
}

type Options

type Options struct {
	// The temporary directory in which to write profile files, logs, etc.
	OutputDir string

	// Report as tests are run; default is silent for success.
	Verbose bool

	// Run only tests matching a regexp.
	Match string

	// Enable memory profiling.
	MemProfile     bool
	MemProfileRate int

	// Enable CPU profiling.
	CpuProfile bool

	// Enable goroutine block profiling.
	BlockProfile     bool
	BlockProfileRate int

	// Enable execution trace.
	ExecutionTrace bool

	// Panic Suite execution after a timeout (0 means unlimited).
	Timeout time.Duration

	// Limit number of tests to run in parallel (0 means GOMAXPROCS).
	Parallel int

	// Sharding splits tests across runners
	Sharding string

	Reporters reporters.Reporters
}

Options

func (*Options) FlagSet

func (o *Options) FlagSet(prefix string, errorHandling flag.ErrorHandling) *flag.FlagSet

FlagSet can be used to setup options via command line flags. An optional prefix can be prepended to each flag. Defaults can be specified prior to calling FlagSet.

type Suite

type Suite struct {
	// contains filtered or unexported fields
}

Suite is a type passed to a TestMain function to run the actual tests. Suite manages the execution of a set of test functions.

func NewSuite

func NewSuite(opts Options, tests Tests) *Suite

NewSuite creates a new test suite. All parameters in Options cannot be modified once given to Suite.

func (*Suite) Run

func (s *Suite) Run() (err error)

Run runs the tests. Returns SuiteFailed for any test failure.

type Test

type Test func(*H)

Test is a single test function.

type Tests

type Tests map[string]*HarnessTest

Tests is a set of test functions and timeouts that can be given to a Suite.

func (*Tests) Add

func (ts *Tests) Add(name string, test Test, timeout time.Duration)

Add inserts the given Test into the set, initializing Tests if needed. If a test with the given name already exists Add will panic.

func (Tests) List

func (ts Tests) List() []string

List returns a sorted list of test names.