testreporter

Documentation

Overview

Package testreporter contains a Reporter for collecting detailed test reports.

While you could probably get at all of the exposed information in reports by examining the output of "go test", the generated report is intended to collect the information in one machine-readable file.

Proper use of the reporter requires some slight changes to how you normally write tests. If you do any of these "the normal way", the tests will still operate fine; you will just miss some detail in the external report.

First, the reporter instance must be initialized and Closed. The cmd/interchaintest package does this in a MainTest function, similar to this:

func TestMain(m *testing.M) {
  f, _ := os.Create("/tmp/report.json")
  reporter := testreporter.NewReporter(f)
  code := m.Run()
  _ = reporter.Close()
  os.Exit(code)
}

Next, every test that needs to be tracked must call TrackTest. If you omit the call to TrackTest, then the test's start and end time, and skip/fail status, will not be reported.

var reporter *testreporter.Reporter // Initialized somehow.

func TestFoo(t *testing.T) {
  reporter.TrackTest(t)
  // Normal test usage continues...
}

Calling TrackTest tracks the test's start and finish time, including whether the test was skipped or failed.

Parallel tests should not call t.Parallel directly, but instead should use TrackParallel. This will track the time the test paused waiting for parallel execution and when parallel execution resumes. If you omit the call to TrackParallel, then at worst you have a misleading test duration.

func TestFooParallel(t *testing.T) {
  reporter.TrackTest(t)
  reporter.TrackParallel(t)
  // Normal test usage continues...
}

If a test needs to be skipped, the TrackSkip method will track the skip reason. Like the other Track methods, calling t.Skip directly will still cause the test to be skipped, and the reporter will note that the test was skipped, but the reporter would not track the specific skip reason.

func TestFooSkip(t *testing.T) {
  if someReason() {
    reporter.TrackSkip(t, "skipping due to %s", whySkipped())
  }
}

Lastly, and perhaps most importantly, the reporter is designed to integrate with testify's require and assert packages. Plain "go test" runs simply have a stream of log lines and a failure/skip state. But if you connect the reporter with a require or assert instance, any failed assertions are stored as error messages in the report.

func TestBar(t *testing.T) {
  reporter.TrackTest(t)
  req := require.New(reporter.TestifyT(t))
  t.Log("About to test Bar()") // Goes to "go test" output, but not included in report.

  // If this fails, the report includes a "TestErrorMessage" entry in the report.
  req.NoError(Bar(), "failure executing Bar()")
}

If you use a plain require.NoError(t, err) call, the report will note that the test failed, but the report will not include the error line.

Index

• type BeginSuiteMessage
• type BeginTestMessage
• type ContinueTestMessage
• type FinishSuiteMessage
• type FinishTestMessage
• type LabelSet
• type Message
• type PauseTestMessage
• type RelayerExecMessage
• type RelayerExecReporter
  • func (r *RelayerExecReporter) TrackRelayerExec(containerName string, command []string, stdout, stderr string, exitCode int, ...)
• type Reporter
  • func NewNopReporter() *Reporter
  • func NewReporter(w io.WriteCloser) *Reporter
  • func (r *Reporter) Close() error
  • func (r *Reporter) RelayerExecReporter(t T) *RelayerExecReporter
  • func (r *Reporter) TestifyT(t TestifyT) *TestifyReporter
  • func (r *Reporter) TrackParallel(t T)
  • func (r *Reporter) TrackParameters(t T, relayerLabels []label.Relayer, chainLabels []label.Chain)
  • func (r *Reporter) TrackSkip(t T, format string, args ...any)
  • func (r *Reporter) TrackTest(t T, labels ...label.Test)
• type T
• type TestErrorMessage
• type TestSkipMessage
• type TestifyReporter
  • func (r *TestifyReporter) Errorf(format string, args ...any)
  • func (r *TestifyReporter) FailNow()
• type TestifyT
• type WrappedMessage
  • func JSONMessage(m Message) WrappedMessage
  • func (m *WrappedMessage) UnmarshalJSON(b []byte) error

Types

type BeginSuiteMessage

type BeginSuiteMessage struct {
	StartedAt time.Time
}

BeginSuiteMessage indicates when the Reporter was initialized, which should correlate with the beginning of a TestMain function, or an init function in a normal test suite.

type BeginTestMessage

type BeginTestMessage struct {
	Name      string
	StartedAt time.Time
	Labels    LabelSet
}

BeginTestMessage indicates the beginning of a single test. If the test uses t.Parallel (via (*Reporter).TrackParallel), the reporter will also track a PauseTestMessage and a ContinueTestMessage.

type ContinueTestMessage

type ContinueTestMessage struct {
	Name string
	When time.Time
}

ContinueTestMessage indicates that a test has resumed execution after a call to t.Parallel.

type FinishSuiteMessage

type FinishSuiteMessage struct {
	FinishedAt time.Time
}

FinishSuiteMessage indicates the time the test suite has finished.

type FinishTestMessage

type FinishTestMessage struct {
	Name       string
	FinishedAt time.Time

	Failed, Skipped bool
}

FinishTestMessage is tracked at the end of a single test.

type LabelSet

type LabelSet struct {
	Relayer []label.Relayer `json:",omitempty"`
	Chain   []label.Chain   `json:",omitempty"`

	Test []label.Test `json:",omitempty"`
}

LabelSet is the set of labels that can be associated with a test.

type Message

type Message interface {
	// contains filtered or unexported methods
}

Message is the sentinel interface to all testreporter messages.

type PauseTestMessage

type PauseTestMessage struct {
	Name string
	When time.Time
}

PauseTestMessage indicates that a test is entering parallel mode and waiting for its turn to continue execution.

type RelayerExecMessage

type RelayerExecMessage struct {
	Name string // Test name, but "Name" for consistency.

	StartedAt, FinishedAt time.Time

	ContainerName string `json:",omitempty"`

	Command []string

	Stdout, Stderr string

	ExitCode int

	Error string `json:",omitempty"`
}

RelayerExecMessage is the result of executing a relayer command. This message is populated through the RelayerExecReporter type, which is returned by the Reporter's RelayerExecReporter method.

type RelayerExecReporter

type RelayerExecReporter struct {
	// contains filtered or unexported fields
}

RelayerExecReporter provides one method that satisfies the ibc.RelayerExecReporter interface. Instances of RelayerExecReporter must be retrieved through (*Reporter).RelayerExecReporter.

func (*RelayerExecReporter) TrackRelayerExec

func (r *RelayerExecReporter) TrackRelayerExec(
	containerName string,
	command []string,
	stdout, stderr string,
	exitCode int,
	startedAt, finishedAt time.Time,
	err error,
)

TrackRelayerExec tracks the execution of an individual relayer command.

type Reporter

type Reporter struct {
	// contains filtered or unexported fields
}

func NewNopReporter

func NewNopReporter() *Reporter

NewNopReporter returns a reporter that does not write anywhere.

func NewReporter

func NewReporter(w io.WriteCloser) *Reporter

func (*Reporter) Close

func (r *Reporter) Close() error

Close closes the reporter and blocks until its results are flushed to the underlying writer.

func (*Reporter) RelayerExecReporter

func (r *Reporter) RelayerExecReporter(t T) *RelayerExecReporter

RelayerExecReporter returns a RelayerExecReporter associated with t.

func (*Reporter) TestifyT

func (r *Reporter) TestifyT(t TestifyT) *TestifyReporter

TestifyT returns a TestifyReporter which will track logged errors in test. Typically you will use this with the New method on the require or assert package:

req := require.New(reporter.TestifyT(t))
// ...
req.NoError(err, "failed to foo the bar")

func (*Reporter) TrackParallel

func (r *Reporter) TrackParallel(t T)

TrackParallel tracks when the pause begins for a parallel test and when it continues to resume.

func (*Reporter) TrackParameters

func (r *Reporter) TrackParameters(t T, relayerLabels []label.Relayer, chainLabels []label.Chain)

TrackParameters is intended to be called from the outermost layer of tests. It tracks the test run including labels indicative of what relayers and chains are used.

func (*Reporter) TrackSkip

func (r *Reporter) TrackSkip(t T, format string, args ...any)

TrackSkip records a the reason for a test being skipped, and calls t.Skip.

func (*Reporter) TrackTest

func (r *Reporter) TrackTest(t T, labels ...label.Test)

TrackTest tracks execution of a subtest using the supplied labels.

type T

type T interface {
	Name() string
	Cleanup(func())

	Skip(...any)

	Parallel()

	Failed() bool
	Skipped() bool
}

T is a subset of testing.TB, representing only the methods required by the reporter.

type TestErrorMessage

type TestErrorMessage struct {
	Name    string
	When    time.Time
	Message string
}

TestErrorMessage is tracked when a Reporter's TestifyT().Errorf method is called. This is the intended usage of a Reporter with require:

req := require.New(rep.TestifyT(t))
req.NoError(foo())

If req.NoError fails, then rep will track a TestErrorMessage.

type TestSkipMessage

type TestSkipMessage struct {
	Name    string
	When    time.Time
	Message string
}

TestSkipMessage is tracked when a Reporter's TrackSkip method is called. This allows the report to track the reason a test was skipped.

type TestifyReporter

type TestifyReporter struct {
	// contains filtered or unexported fields
}

TestifyReporter wraps a Reporter to satisfy the testify/require.TestingT interface. This allows the reporter to track logged errors.

func (*TestifyReporter) Errorf

func (r *TestifyReporter) Errorf(format string, args ...any)

Errorf records the error message in r's Reporter and then passes through to r's underlying TestifyT.

func (*TestifyReporter) FailNow

func (r *TestifyReporter) FailNow()

FailNow passes through to r's TestifyT. It does not need to log another message because r's Reporter should be tracking the test already.

type TestifyT

type TestifyT interface {
	Name() string

	Errorf(format string, args ...any)
	FailNow()
}

TestifyT is a superset of the testify/require.TestingT interface.

type WrappedMessage

type WrappedMessage struct {
	Type string
	Message
}

WrappedMessage wraps a Message with an outer Type field so that decoders can determine the underlying message's type.

func JSONMessage

func JSONMessage(m Message) WrappedMessage

JSONMessage produces a WrappedMessage so that a stream of Message values can be distinguishedu by a top-level "Type" key.

func (*WrappedMessage) UnmarshalJSON

func (m *WrappedMessage) UnmarshalJSON(b []byte) error