comparison

Documentation

Overview

Package comparison contains symbols for making your own comparison.Func implementations for use with go.chromium.org/luci/common/testing/truth.

Please see go.chromium.org/luci/common/testing/truth/should for a set of ready-made comparison functions.

Implementing comparisons

Comparison implementations will be used in the context of go.chromium.org/luci/common/testing/truth/assert and go.chromium.org/luci/common/testing/truth/check like:

assert.That(t, actualValue, comparisonName(comparisonArguments))

if check.That(t, actualValue, comparisonName(comparisonArguments)) {
   // code if comparison passed
}

With this in mind, try to pick comparisonName so that it reads well in this context. The default set of comparison implementations have the form "should.XXX", e.g. "should.Equal" or "should.NotBeNil" which work well here. If you are implementing comparisons in your own tests, you can make a similar Failure by dropping the ".", e.g. "shouldBeComplete", but you can also use other natural syntax like "isConsistentWith(databaseKey)", etc.

Typically, comparisons require additional data to produce a [*Failure] for a given value. Implementations which take additional data typically look like:

func isConsistentWith(databaseKey *concreteType) comparison.Func[expectedType] {
  return func(actualValue expectedType) *comparison.Failure {
     // compare databaseKey with actualValue here
  }
}

Functions returning comparison.Func may be generic, if necessary, but when writing them in package-specific contexts, generic type arguments are usually not needed. In general, if you can make your function accept a narrow, specific, type, it will be better. If you feel like you really need to make a generic comparison or function returning a comparison, please discuss directly contributing it to go.chromium.org/luci/common/testing/truth/should.

When the truth library uses a comparison with a 'loosely' variant (e.g. truth.AssertLoosely, or assert.Loosely), it will attempt to losslessly convert the actual value to the T type of the comparison.Func. The high-level takeaway is that it implements simple conversions which preserve the actual data, but may adjust the type. These conversion rules ONLY look at the types, not the value, so e.g. 'int64(1) -> float32' will not work, even though float32(1) is storable exactly, because in general, not all int64 fit into float32.

See go.chromium.org/luci/common/data/convert.LosslesslyTo for how this conversion works in more detail.

With that in mind, if you don't want any of this conversion to take place (e.g. customStringType -> string, int8 -> int64, etc.), you can use "any" for T in the comparison (in which case your comparison will need to do work to pull supported types out itself via type switches or reflect) OR you can use the `truth.Assert`, `truth.Check`, `assert.That`, or `check.That` functions to ensure that the actual value exactly matches the comparison.Func[T] type at compile-time.

Crafting *Failure

After your Func makes its evaluation of the actual data, if it fails, it needs to return a [*Failure], which is a proto message.

As a proto, you can construct [*Failure] however you like, but most comparison implementations will use [NewFailureBuilder] to construct a [*Failure] using a 'fluent' interface.

The cardinal rule to follow is that [*Failure] should contain exactly the information which allows the reader understand why the comparison failed. If you have Findings which are possibly redundant, you can mark them with the 'Warn' level to hide them from the test output until the user passes `go test -v`.

For example, in a comparison named 'shouldEqual', including all of the following would be redundant, even though they are all individually helpful:

Check shouldEqual[string] FAILED
  Because: "actual" does not equal "hello"
  Actual: "actual"
  Expected: "hello"
  Diff: \
        string(
      -  "actual",
      +  "hello",
        )

(4) alone would be enough, but if necessary, you could have (2) and (3) marked with a warning-level level, in case the diff can be hard to interpret.

A *Failure has a couple different pieces:

• The name of the comparison (or the function which generated the comparison), its type arguments (if any), and any particularly helpful expected arguments (e.g. the expected length of should.HaveLength).
• Zero or more "Findings".

The name of the *Failure is always required, and will be rendered like:

resultName FAILED

For some comparisons (like should.BeTrue or should.NotBeEmpty, etc.) this is enough context for the user to figure out what happened.

However, most comparisons will need to give additional context to the failure, which is where findings come in.

Findings are named data items in sequence. These will be rendered under the name like:

resultName FAILED
  ValueName: value contents

Finding values are a list of lines - in the event that the value contains multiple lines, you'll see it rendered like:

resultName FAILED
  ValueName: \
    This is a very long
    value with
    multiple lines.

By convention, this package defines a couple different common Finding labels w/ helper functions which are useful in many, but not all, comparisons.

• "Because" - This Finding should have a descriptive explanation of why the comparison failed.
• "Actual"/"Expected" - These reflect the actual or expected value of the assertion back to the reader of the assertion failure. Sometimes this is useful (e.g. should.AlmostEqual[float32] reflects the actual value back). However, sometimes this is not useful, e.g. when the value is a gigantic struct.
• "Diff" - This is the difference between Actual and Expected, usually computed with cmp.Diff (or, alternately, as a unified diff). These Findings should be marked with a diff type hint, so that they can have syntax coloring applied to them when rendered in a terminal.

As the implementor of the comparison, it is up to you to decide what Findings are the best way to explain to the test failure reader what happened and, potentially why. Sometimes none of these finding labels will be the right way to communicate that, and that's OK :).

Index

• func SetComparison(f *failure.Summary, comparisonName string, typeArgs ...any)
• type Func
  • func (cmp Func[T]) WithLineContext(skipFrames ...int) Func[T]
• type RenderCLI
  • func (r RenderCLI) Comparison(prefix string, c *failure.Comparison) string
  • func (r RenderCLI) Finding(prefix string, f *failure.Finding) string
  • func (r RenderCLI) Stack(prefix string, s *failure.Stack) string
  • func (r RenderCLI) Summary(prefix string, f *failure.Summary) string
• type SummaryBuilder
  • func NewSummaryBuilder(comparisonName string, typeArgs ...any) *SummaryBuilder
  • func (sb *SummaryBuilder) Actual(actual any) *SummaryBuilder
  • func (sb *SummaryBuilder) AddCmpDiff(diff string) *SummaryBuilder
  • func (sb *SummaryBuilder) AddComparisonArgs(args ...any) *SummaryBuilder
  • func (sb *SummaryBuilder) AddFindingf(name, format string, args ...any) *SummaryBuilder
  • func (sb *SummaryBuilder) Because(format string, args ...any) *SummaryBuilder
  • func (sb *SummaryBuilder) Expected(Expected any) *SummaryBuilder
  • func (sb *SummaryBuilder) GetFailure() *failure.Summary
  • func (sb *SummaryBuilder) RenameFinding(oldname, newname string) *SummaryBuilder
  • func (sb *SummaryBuilder) SmartCmpDiff(actual, expected any, extraCmpOpts ...cmp.Option) *SummaryBuilder
  • func (sb *SummaryBuilder) WarnIfLong() *SummaryBuilder

Functions

func SetComparison

func SetComparison(f *failure.Summary, comparisonName string, typeArgs ...any)

SetComparison sets the Comparison field of `failure.Summary`, formating `typeArgs` into strings with `fmt.Sprintf("%T")`.

Example:

actual := 123
SetComparison(f, "should.Equal", &actual)

Will SummaryBuilder for a comparison `should.Equal[*int]`.

Types

type Func

type Func[T any] func(T) *failure.Summary

Func takes in a value-to-be-compared and returns a failure.Summary if the value does not meet the expectation of this comparison.Func.

Example:

func BeTrue(value bool) *failure.Summary {
  if !value {
    return comparison.NewSummaryBuilder("should.BeTrue").Summary
  }
  return nil
}

In this example, BeTrue is a comparison.Func.

func (Func[T]) WithLineContext

func (cmp Func[T]) WithLineContext(skipFrames ...int) Func[T]

WithLineContext returns a transformed Func to add an "at" SourceContext with one frame containing the filename and line number of the frame calling WithLineContext, plus skipFrames[0] (if provided).

Example:

check.That(t, something, should.Equal(100).WithLineContext())

You usually will not need this, but it's very useful when writing a helper function for tests (e.g. using t.Helper()) to let you add the location of the specific assert inside of the helper function along side the 'top most' frame location, as computed directly by the Go testing library.

Example:

func TestThing(t *testing.T) {
  myHelper := func(actual, expected myType) {
    t.Helper()  // makes Go 'testing' package skip this to find original call.

    // We add WithLineContext to these comparisons so that if they fail, we
    // will see the file:line within this helper function.
    check.That(t, actual.field1, should.Equal(expected.field1).WithLineContext())
    check.That(t, actual.field2, should.Equal(expected.field2).WithLineContext())
 }
 // ...
 myHelper(a, expected)
}

In this example, the test will output something like:

--- FAIL: FakeTestName (0.00s)
    example_test.go:XX: Check should.Equal[int] FAILED
       (at example_test.go:YY)
       Actual: 10
       Expected: 20

Where XX is the line of the myHelper call, and YY is the line of the actual should.Equal check inside of the helper function.

type RenderCLI

type RenderCLI struct {
	// If true, will render all Verbose findings.
	//
	// Otherwise this will print an omission message which describes how long the
	// omitted value is and to pass `-v` to the test to see them.
	Verbose bool

	// If true, will add ANSI color codes to Findings with appropriate types
	// (currently just simple +/- per-line colorization for unified and cmp.Diff
	// Findings).
	Colorize bool

	// If true, will not truncate filenames in any source_context stacks.
	FullFilenames bool
}

func (RenderCLI) Comparison

func (r RenderCLI) Comparison(prefix string, c *failure.Comparison) string

Comparison pretty-prints a failure.Comparison.

If `c` is nil, or is missing the Name field, this will use the name "UNKNOWN COMPARISON", which means that this function never returns an empty string.

func (RenderCLI) Finding

func (r RenderCLI) Finding(prefix string, f *failure.Finding) string

Finding renders a Finding to a set of output lines which would be suitable for display as CLI output (e.g. to be logged with testing.T.Log calls).

func (RenderCLI) Stack

func (r RenderCLI) Stack(prefix string, s *failure.Stack) string

Stack pretty-prints a failure.Stack.

If r.FullFilenames=false this will truncate filenames to just the base path, like `go test` does by default.

func (RenderCLI) Summary

func (r RenderCLI) Summary(prefix string, f *failure.Summary) string

Summary pretty-prints the result as a list of lines for display via the `go test` CLI output.

If verbose is true, will render all verbose Findings. If colorize is true, will attempt to add ANSI coloring (currently just very basic per-line colors for diffs).

type SummaryBuilder

type SummaryBuilder struct {
	*failure.Summary
}

A SummaryBuilder builds a failure.Summary and has a fluent interface.

func NewSummaryBuilder

func NewSummaryBuilder(comparisonName string, typeArgs ...any) *SummaryBuilder

NewSummaryBuilder makes a new SummaryBuilder, filling in the for the given comparisonName and exemplar type arguments.

For example:

actual := 123
NewSummaryBuilder("should.Equal", &actual)

Will make a new SummaryBuilder for a comparison `should.Equal[*int]`.

func (*SummaryBuilder) Actual

func (sb *SummaryBuilder) Actual(actual any) *SummaryBuilder

Actual adds a new finding "Actual" to the failure.Summary.

`actual` will be rendered with fmt.Sprintf("%#v").

func (*SummaryBuilder) AddCmpDiff

func (sb *SummaryBuilder) AddCmpDiff(diff string) *SummaryBuilder

AddCmpDiff adds a 'Diff' finding which is type hinted to be the output of cmp.Diff.

The diff is split into multiple lines, but is otherwise untouched.

func (*SummaryBuilder) AddComparisonArgs

func (sb *SummaryBuilder) AddComparisonArgs(args ...any) *SummaryBuilder

AddComparisonArgs adds new arguments to the failure.Summary.ComparisonFunc formatted with %v.

func (*SummaryBuilder) AddFindingf

func (sb *SummaryBuilder) AddFindingf(name, format string, args ...any) *SummaryBuilder

AddFindingf adds a new single-line Finding to this failure.Summary with the given `name`.

If `args` is empty, then `format` will be used as the Finding value verbatim. Otherwise the value will be formatted as `fmt.Sprintf(format, args...)`.

The finding will have the type "FindingTypeHint_Text".

func (*SummaryBuilder) Because

func (sb *SummaryBuilder) Because(format string, args ...any) *SummaryBuilder

Because adds a new finding "Because" to the failure.Summary with AddFormattedFinding.

func (*SummaryBuilder) Expected

func (sb *SummaryBuilder) Expected(Expected any) *SummaryBuilder

Expected adds a new finding "Expected" to the failure.Summary.

`Expected` will be rendered with fmt.Sprintf("%#v").

func (*SummaryBuilder) GetFailure

func (sb *SummaryBuilder) GetFailure() *failure.Summary

GetFailure returns sb.Summary if it contains any Findings, otherwise returns nil.

This is useful if you build your comparison with a series of conditional findings.

func (*SummaryBuilder) RenameFinding

func (sb *SummaryBuilder) RenameFinding(oldname, newname string) *SummaryBuilder

RenameFinding finds the first Finding with the name `oldname` and renames it to `newname`.

Does nothing if `oldname` is not one of the current Findings.

func (*SummaryBuilder) SmartCmpDiff

func (sb *SummaryBuilder) SmartCmpDiff(actual, expected any, extraCmpOpts ...cmp.Option) *SummaryBuilder

SmartCmpDiff does a couple things:

• It adds "Actual" and "Expected" findings. If they have long renderings, they will be marked as Level=Warn.
• If either text representation is long, or they are identical, this will also add a Diff, using cmp.Diff and the provided Options.

"Long" is defined as a Value with multiple lines or which has > 30 characters in one line.

The default cmp.Options include a Transformer to handle protobufs. If you want to extend the default Options see `go.chromium.org/luci/common/testing/registry`.

func (*SummaryBuilder) WarnIfLong

func (sb *SummaryBuilder) WarnIfLong() *SummaryBuilder

WarnIfLong marks the previously-added Finding with Level 'Warn' if it has a long value.

A long value is defined as:

• More than one line OR
• A line exceeding 30 characters in length.

No-op if there are no findings in the failure yet.