README
¶
testcase
The testcase
package provides tooling to apply BDD testing conventions.
Features
- lightweight, it has no dependency, you just import testcase, no clutter
- supports classing flattened testing style, table testing style, and nested testing style
- nested testing style
- allows the usage of an immutable testing subject
- nesting visualize code complexity
- testing context building becomes DRY
- immutable testing subject construction forces disciplined testing conventions
- supports also the flattened nesting where testing context branches can to a top-level function
- nested testing style
- DDD based testing composition
- testing components can be defined and reused across the whole project
- unit tests can be converted into full-fledged E2E tests for CI/CD pipelines
- dependency injection during testing becomes a breeze to do
- reusable testing components increase maintainability aspects
- reusable testing components decrease required ramp-up time for writing new tests that depend on existing components
- safe parallel testing
- variables stored per test execution, which prevents leakage and race conditions across different tests
- repeatable pseudo-random fixture generation for test input
- repeatable pseudo-random test order shuffling
- prevents implicit test dependency on ordering
- ensures that tests can be added and removed freely without the fear of breaking other tests in the same coverage.
- flaky tests which depend on test execution order can be noticed at development time
Guide
testcase
is a powerful TDD Tooling that requires discipline and understanding of the fundamentals of testing.
If you are looking for a guide that helps streamline your knowledge on the topics,
then please consider reading the below-listed articles.
- why to test?
- nested testing style With testcase.Specs
- indirection and abstraction and the use in testing
- testing doubles, when to use what
- TDD, Role Interface and Contracts [WIP-90%]
- design system with DRY TDD for long term maintainability wins [WIP-1%]
Official API Documentation
If you already use the framework, and you just won't pick an example, you can go directly to the API documentation that is kept in godoc format.
Getting Started / Example
Examples kept in godoc format. Every exported functionality aims to have examples provided in the official documentation.
A Basic examples:
func Test_withTestcaseT(t *testing.T) {
tc := testcase.NewT(t, nil)
tc.Must.True(true)
_ = tc.Random.String()
}
func Test_withTestcaseSpec(t *testing.T) {
s := testcase.NewSpec(t)
s.Test("", func(t *testcase.T) {
t.Must.True(true)
_ = tc.Random.String()
})
}
An example with scoped test variables:
package main
import (
"testing"
"github.com/adamluzsi/testcase"
)
func TestMyFunc(t *testing.T) {
s := testcase.NewSpec(t)
s.NoSideEffect()
var (
input = testcase.Let[string](s, nil)
)
act := func(t *testcase.T) string {
return MyFunc(input.Get(t))
}
s.When("input is all lowercase", func(s *testcase.Spec) {
// testing scopes without the input being defined
// will warn the dev about the missing specification.
input.LetValue(s, "all lowercase")
s.Then("it is expected to ...", func(t *testcase.T) {
t.Must.Equal("all lowercase", act(t))
})
})
s.When("input is all upcase", func(s *testcase.Spec) {
input.LetValue(s, "ALL UPCASE")
s.Then("it is expected to ...", func(t *testcase.T) {
t.Must.Equal("all upcase", act(t))
})
})
}
Modules
- httpspec
- spec module helps you create HTTP API Specs.
- fixtures
- fixtures module helps you create random input values for testing
Summary
DRY
testcase
provides a way to express common Arrange, Act sections for the Asserts with a DRY principle in mind.
- First, you can define your Act section with a method under test as the subject of your test specification
- The Act section invokes the method under test with the arranged parameters.
- Then you can build the context of the Act by Arranging the inputs later with humanly explained reasons
- The Arrange section initializes objects and sets the value of the data that is passed to the method under test.
- And lastly, you can define the test expected outcome in an Assert section.
- The Assert section verifies that the action of the method under test behaves as expected.
Then adding test edge case to the testing suite becomes easier, as it will have a concrete place where it must be placed.
And if during the creation of the specification, an edge case turns out to be YAGNI, it can be noted, so visually it will be easier to see what edge case is not specified for the given subject.
The value it gives is that to build a test for a certain edge case, the required mental model size to express the context becomes smaller, as you only have to focus on one Arrange at a time, until you fully build the bigger picture.
It also implicitly visualize the required mental model of your production code by the nesting. You can read more on that in the nesting section.
Modularization
On top of the DRY convention, any time you need to Arrange a common scenario about your projects domain event, you can modularize these setup blocks in a helper function.
This helps the readability of the test while keeping the need for mocks to the minimum as possible for a given test. As a side effect, integration tests can become low-hanging fruit for the project.
e.g.:
package mypkg_test
import (
"testing"
"my/project/mypkg"
"github.com/adamluzsi/testcase"
. "my/project/testing/pkg"
)
func TestMyTypeMyFunc(t *testing.T) {
s := testcase.NewSpec(t)
myUser := GivenWeHaveUser(s)
// .. other givens
myType := testcase.Let(s, func(t *testcase.T) *mypkg.MyType {
return &mypkg.MyType{}
})
s.Describe(`.MyFunc`, func(s *testcase.Spec) {
act := func(t *testcase.T) { myType.Get(t).MyFunc(myUser.Get(t)) }
s.Then(`edge case description`, func(t *testcase.T) {
act(t)
})
})
}
Stability
- The package is considered stable.
- The package use rolling release conventions.
- No breaking change is planned to the package exported API.
- The package used in production development.
- The package API is only extended if the practical use case proves its necessity.
Case Study About testcase
Package Origin
Reference Project
Documentation
¶
Overview ¶
Package testcase is an opinionated testing framework.
Repository + README: https://github.com/adamluzsi/testcase
Guide: https://github.com/adamluzsi/testcase/blob/master/docs/README.md
Index ¶
- Constants
- Variables
- func Append[V any](t *T, v Var[V], x ...interface{})
- func Race(fn1, fn2 func(), more ...func())
- func RetryCount(times int) assert.RetryStrategy
- func RunOpenSuite[TBS anyTBOrSpec](tb TBS, contracts ...OpenSuite)
- func RunSuite[TBS anyTBOrSpec](tb TBS, contracts ...Suite)
- func Sandbox(fn func()) sandbox.RunOutcome
- func SetEnv(tb testing.TB, key, value string)
- func TableTest[TBS anyTBOrSpec, TC sBlock | tBlock | any, Act tBlock | sBlock | func(*T, TC)](tbs TBS, tcs map[string]TC, act Act)
- func UnsetEnv(tb testing.TB, key string)
- type Eventually
- type OpenSuite
- type OpenSuiteAdapter
- type RetryStrategy
- type RetryStrategyFunc
- type Spec
- func (spec *Spec) After(afterBlock tBlock)
- func (spec *Spec) AfterAll(blk func(tb testing.TB))
- func (spec *Spec) And(desc string, testContextBlock sBlock, opts ...SpecOption)
- func (spec *Spec) Around(block hookBlock)
- func (spec *Spec) AroundAll(blk func(tb testing.TB) func())
- func (spec *Spec) Before(beforeBlock tBlock)
- func (spec *Spec) BeforeAll(blk func(tb testing.TB))
- func (spec *Spec) Context(desc string, testContextBlock sBlock, opts ...SpecOption)
- func (spec *Spec) Describe(subjectTopic string, specification sBlock, opts ...SpecOption)
- func (spec *Spec) Finish()
- func (spec *Spec) HasSideEffect()
- func (spec *Spec) Let(varName string, blk VarInit[any]) Var[any]
- func (spec *Spec) LetValue(varName string, value any) Var[any]
- func (spec *Spec) NoSideEffect()
- func (spec *Spec) Parallel()
- func (spec *Spec) Sequential()
- func (spec *Spec) SkipBenchmark()
- func (spec *Spec) Tag(tags ...string)
- func (spec *Spec) Test(desc string, test tBlock, opts ...SpecOption)
- func (spec *Spec) Then(desc string, test tBlock, opts ...SpecOption)
- func (spec *Spec) When(desc string, testContextBlock sBlock, opts ...SpecOption)
- type SpecOption
- type StubTB
- type Suite
- type T
- func (t *T) Cleanup(fn func())
- func (t *T) Defer(fn interface{}, args ...interface{})
- func (t *T) Eventually(blk func(it assert.It), retryOpts ...interface{})
- func (t *T) HasTag(tag string) bool
- func (t *T) LogPretty(vs ...any)
- func (t *T) SetEnv(key, value string)
- func (t *T) Setenv(key, value string)
- func (t *T) SkipUntil(year int, month time.Month, day int)
- func (t *T) UnsetEnv(key string)
- type TBRunner
- type Var
- func (v Var[V]) Bind(s *Spec) Var[V]
- func (v Var[V]) EagerLoading(s *Spec) Var[V]
- func (v Var[V]) Get(t *T) V
- func (v Var[V]) Let(s *Spec, blk VarInit[V]) Var[V]
- func (v Var[V]) LetValue(s *Spec, value V) Var[V]
- func (v Var[V]) PreviousValue(t *T) V
- func (v Var[V]) Set(t *T, value V)
- func (v Var[V]) Super(t *T) V
- type VarInit
- type VarInitFunc
- type Waiter
Examples ¶
- Package (AssertEventually)
- Package (AssertEventuallyAsContextOption)
- Package (AssertEventuallyByCount)
- Package (AssertEventuallyByCustomRetryStrategy)
- Package (AssertEventuallyByTimeout)
- Package (AssertEventuallyCount)
- Package (AssertWaiterWait)
- Package (AssertWaiterWhile)
- Package (ClockAfter)
- Package (ClockSleep)
- Package (ClockTimeNow_withTimecop)
- Package (ClockTimeNow_withTravelByDate)
- Package (ClockTimeNow_withTravelByDuration)
- Package (FaultInject)
- Append
- Flaky (RetryNTimes)
- Flaky (RetryUntilTimeout)
- Group
- NewT
- Race
- Sandbox
- SetEnv
- SkipBenchmark
- Spec
- Spec (WhenProjectUseSharedSpecificationHelpers)
- Spec (WithBenchmark)
- Spec.After
- Spec.AfterAll
- Spec.And
- Spec.Around
- Spec.AroundAll
- Spec.Before
- Spec.BeforeAll
- Spec.Context
- Spec.Describe
- Spec.HasSideEffect
- Spec.Let
- Spec.Let (EagerLoading)
- Spec.Let (SqlDB)
- Spec.Let (TestingDouble)
- Spec.Let (UsageWithinNestedScope)
- Spec.LetValue
- Spec.LetValue (UsageWithinNestedScope)
- Spec.NoSideEffect
- Spec.Parallel
- Spec.Parallel (ScopedWithContext)
- Spec.Sequential
- Spec.Sequential (GlobalVar)
- Spec.Sequential (ScopedWithContext)
- Spec.SkipBenchmark
- Spec.SkipBenchmark (ScopedWithContext)
- Spec.Tag
- Spec.Test
- Spec.Then
- Spec.When
- StubTB (TestingATestHelper)
- T (Must)
- T (Random)
- T (Should)
- T.Defer
- T.Defer (WithArgs)
- T.Eventually
- T.HasTag
- T.LogPretty
- T.SetEnv
- T.SkipUntil
- T.UnsetEnv
- TableTest (ActAsCommonSpecContext)
- TableTest (ClassicInlined)
- TableTest (ClassicStructured)
- TableTest (WithSpecBlock)
- TableTest (WithTestBlock)
- UnsetEnv
- Var
- Var (Before)
- Var (Init)
- Var (OnLet)
- Var.Bind
- Var.EagerLoading
- Var.Get
- Var.Let
- Var.Let (EagerLoading)
- Var.Let (ValueDefinedAtTestingContextScope)
- Var.LetValue
- Var.LetValue (EagerLoading)
- Var.Set
- Var.Super
Constants ¶
const ( OrderingAsDefined testOrderingMod = `defined` OrderingAsRandom testOrderingMod = `random` )
const EnvKeyOrdering = `TESTCASE_ORDERING`
EnvKeyOrdering is the environment variable key that will be checked for testCase determine what order of execution should be used between test cases in a testing group. The default sorting behavior is pseudo random based on an the seed.
Mods: - defined: execute testCase in the order which they are being defined - random: pseudo random based ordering between tests.
const EnvKeySeed = `TESTCASE_SEED`
EnvKeySeed is the environment variable key that will be checked for a pseudo random seed, which will be used to randomize the order of executions between test cases.
Variables ¶
var DefaultEventually = assert.Eventually{RetryStrategy: assert.Waiter{Timeout: 3 * time.Second}}
Functions ¶
func Append ¶ added in v0.39.0
Append will append a value[T] to a current value of Var[[]T]. Append only possible if the value type of Var is a slice type of T.
func Race ¶ added in v0.40.0
func Race(fn1, fn2 func(), more ...func())
Race is a test helper that allows you to create a race situation easily. Race will execute each provided anonymous lambda function in a different goroutine, and make sure they are scheduled at the same time.
This is useful when you work on a component that requires thread-safety. By using the Race helper, you can write an example use of your component, and run the testing suite with `go test -race`. The race detector then should be able to notice issues with your implementation.
func RetryCount ¶ added in v0.26.0
func RetryCount(times int) assert.RetryStrategy
RetryCount is moved from this package.
DEPRECATED: use assert.RetryCount instead
func RunOpenSuite ¶ added in v0.79.0
func RunOpenSuite[TBS anyTBOrSpec](tb TBS, contracts ...OpenSuite)
func RunSuite ¶ added in v0.79.0
func RunSuite[TBS anyTBOrSpec](tb TBS, contracts ...Suite)
RunSuite is a helper function that makes execution one or many Suite easy. By using RunSuite, you don't have to distinguish between testing or benchmark execution mod. It supports *testing.T, *testing.B, *testcase.T, *testcase.Spec and CustomTB test runners.
func SetEnv ¶ added in v0.25.0
SetEnv will set the os environment variable for the current program to a given value, and prepares a cleanup function to restore the original state of the environment variable.
Spec using this helper should be flagged with Spec.HasSideEffect or Spec.Sequential.
func TableTest ¶ added in v0.104.0
func TableTest[TBS anyTBOrSpec, TC sBlock | tBlock | any, Act tBlock | sBlock | func(*T, TC)]( tbs TBS, tcs map[string]TC, act Act, )
TableTest allows you to make table tests, without the need to use a boilerplate. It optionally allows to use a Spec instead of a testing.TB, and then the table tests will inherit the Spec context. It guards against mistakes such as using for+t.Run+t.Parallel without variable shadowing. TableTest allows a variety of use, please check examples for further information on that.
Types ¶
type Eventually ¶ added in v0.75.0
type Eventually = assert.Eventually
Eventually
DEPRECATED: use assert.Eventually instead
type OpenSuite ¶ added in v0.79.0
type OpenSuite interface { // Test is the function that assert expected behavioral requirements from a supplier implementation. // These behavioral assumptions made by the Consumer in order to simplify and stabilise its own code complexity. // Every time a Consumer makes an assumption about the behavior of the role interface supplier, // it should be clearly defined it with tests under this functionality. Test(*testing.T) // Benchmark will help with what to measure. // When you define a role interface contract, you should clearly know what performance aspects important for your Consumer. // Those aspects should be expressed in a form of Benchmark, // so different supplier implementations can be easily A/B tested from this aspect as well. Benchmark(*testing.B) }
OpenSuite is a testcase independent testing suite interface standard.
type OpenSuiteAdapter ¶ added in v0.79.0
type OpenSuiteAdapter struct{ OpenSuite }
func (OpenSuiteAdapter) Spec ¶ added in v0.79.0
func (c OpenSuiteAdapter) Spec(s *Spec)
type RetryStrategy ¶ added in v0.25.0
type RetryStrategy = assert.RetryStrategy
RetryStrategy
DEPRECATED: use assert.RetryStrategy instead
type RetryStrategyFunc ¶ added in v0.26.0
type RetryStrategyFunc = assert.RetryStrategyFunc
RetryStrategyFunc
DEPRECATED: use assert.RetryStrategyFunc instead
type Spec ¶
type Spec struct {
// contains filtered or unexported fields
}
Spec provides you a struct that makes building nested test spec easy with the core T#Context function.
spec structure is a simple wrapping around the testing.T#Context. It doesn't use any global singleton cache object or anything like that. It doesn't force you to use global vars.
It uses the same idiom as the core go testing pkg also provide you. You can use the same way as the core testing pkg
go run ./... -v -run "the/name/of/the/test/it/print/orderingOutput/in/case/of/failure"
It allows you to do spec preparation for each test in a way, that it will be safe for use with testing.T#Parallel.
func NewSpec ¶
func NewSpec(tb testing.TB, opts ...SpecOption) *Spec
NewSpec create new Spec struct that is ready for usage.
func (*Spec) After ¶
func (spec *Spec) After(afterBlock tBlock)
After give you the ability to run a block after each test case. This is ideal for running cleanups. The received *testing.T object is the same as the Then block *testing.T object This hook applied to this scope and anything that is nested from here. All setup block is stackable.
DEPRECATED: use Spec.Before with T.Cleanup or Spec.Before with T.Defer instead
func (*Spec) AfterAll ¶ added in v0.52.0
AfterAll give you the ability to create a hook that runs only once after all the test cases already ran.
DEPRECATED: use Spec.BeforeAll with testing.TB#Cleanup
func (*Spec) And ¶
func (spec *Spec) And(desc string, testContextBlock sBlock, opts ...SpecOption)
And is an alias for testcase#Spec.Context And is used to represent additional requirement for reaching a certain testing runtime contexts.
func (*Spec) Around ¶
func (spec *Spec) Around(block hookBlock)
Around give you the ability to create "Before" setup for each test case, with the additional ability that the returned function will be deferred to run after the Then block is done. This is ideal for setting up mocks, and then return the assertion request calls in the return func. This hook applied to this scope and anything that is nested from here. All setup block is stackable.
DEPRECATED: use Spec.Before with T.Cleanup or Spec.Before with T.Defer instead
func (*Spec) AroundAll ¶ added in v0.52.0
AroundAll give you the ability to create a hook that first run before all test, then the returned lambda will run after the test cases.
DEPRECATED: use Spec.BeforeAll with testing.TB#Cleanup
func (*Spec) Before ¶
func (spec *Spec) Before(beforeBlock tBlock)
Before give you the ability to run a block before each test case. This is ideal for doing clean ahead before each test case. The received *testing.T object is the same as the Test block *testing.T object This hook applied to this scope and anything that is nested from here. All setup block is stackable.
func (*Spec) BeforeAll ¶ added in v0.52.0
BeforeAll give you the ability to create a hook that runs only once before the test cases.
func (*Spec) Context ¶
func (spec *Spec) Context(desc string, testContextBlock sBlock, opts ...SpecOption)
Context allow you to create a sub specification for a given spec. In the sub-specification it is expected to add more contextual information to the test in a form of hook of variable setting. With Context you can set your custom test description, without any forced prefix like describe/when/and.
It is basically piggybacking the testing#T.Context and create new subspec in that nested testing#T.Context scope. It is used to add more description spec for the given subject. It is highly advised to always use When + Before/Around together, in which you should setup exactly what you wrote in the When description input. You can Context as many When/And within each other, as you want to achieve the most concrete edge case you want to test.
To verify easily your state-machine, you can count the `if`s in your implementation, and check that each `if` has 2 `When` block to represent the two possible path.
func (*Spec) Describe ¶
func (spec *Spec) Describe(subjectTopic string, specification sBlock, opts ...SpecOption)
Describe creates a new spec scope, where you usually describe a subject.
By convention it is highly advised to create a variable `subject` with function that share the return signature of the method you test on a structure, and take *testcase.variables as the only input value. If your method require input values, you should strictly set those values within a `When`/`And` scope. This ensures you have to think trough the possible state-machines paths that are based on the input values.
For functions where 2 value is returned, and the second one is an error, in order to avoid repetitive test cases in the `Then` I often define a `onSuccess` variable, with a function that takes `testcase#variables` as well and test error return value there with `testcase#variables.T()`.
func (*Spec) Finish ¶ added in v0.29.0
func (spec *Spec) Finish()
Finish executes all unfinished test and mark them finished. Finish can be used when it is important to run the test before the Spec's testing#TB.Cleanup would execute.
Such case can be when a resource leaked inside a testing scope and resource closed with a deferred function, but the spec is still not ran.
func (*Spec) HasSideEffect ¶ added in v0.9.0
func (spec *Spec) HasSideEffect()
HasSideEffect means that after this call things defined that has software side effect during runtime. This suggest on its own that execution should be sequential in order to avoid retry tests.
HasSideEffect and NoSideEffect can be used together to describe a given piece of specification properties. Using them at the same location makes little sense, it was intended to be used in spec helper package where setup function handles what resource should be used in the spec variables. This allows flexibility for the developers to use side effect free variant for local development that has quick feedback loop, and replace them with the production implementation during CI/CD pipeline which less time critical.
func (*Spec) Let ¶
Let is a method to provide backward compatibility with the existing testing suite. Due to how Go type parameters work, methods are not allowed to have type parameters, thus Let has moved to be a pkg-level function in the package.
DEPRECATED: use testcase.Let instead testcase#Spec.Let.
func (*Spec) LetValue ¶ added in v0.2.1
LetValue is a method to provide backward compatibility with the existing testing suite. Due to how Go type parameters work, methods are not allowed to have type parameters, thus LetValue has moved to be a pkg-level function in the package.
DEPRECATED: use testcase.LetValue instead testcase#Spec.LetValue.
func (*Spec) NoSideEffect ¶ added in v0.2.1
func (spec *Spec) NoSideEffect()
NoSideEffect gives a hint to the reader of the current test that during the test execution, no side effect outside from the test specification scope is expected to be observable. It is important to note that this flag primary meant to represent the side effect possibility to the outside of the current testing specification, and not about the test specification's subject.
It is safe to state that if the subject of the test specification has no side effect, then the test specification must have no side effect as well.
If the subject of the test specification do side effect on an input value, then the test specification must have no side effect, as long Let memorization is used.
If the subject of the test specification does mutation on global variables such as OS Variable states for the current process, then it is likely, that even if the changes by the mutation is restored as part of the test specification, the test specification has side effects that would affect other test specification results, and, as such, must be executed sequentially.
func (*Spec) Parallel ¶
func (spec *Spec) Parallel()
Parallel allows you to set list test case for the spec where this is being called, and below to nested contexts, to be executed in parallel (concurrently). Keep in mind that you can call Parallel even from nested specs to apply Parallel testing for that spec and below. This is useful when your test suite has no side effects at list. Using values from *vars when Parallel is safe. It is a shortcut for executing *testing.T#Parallel() for each test
func (*Spec) Sequential ¶ added in v0.9.0
func (spec *Spec) Sequential()
Sequential allows you to set list test case for the spec where this is being called, and below to nested contexts, to be executed sequentially. It will negate any testcase.Spec#Parallel call effect. This is useful when you want to create a spec helper package and there you want to manage if you want to use components side effects or not.
func (*Spec) SkipBenchmark ¶ added in v0.23.0
func (spec *Spec) SkipBenchmark()
SkipBenchmark will flag the current Spec / Context to be skipped during Benchmark mode execution. If you wish to skip only a certain test, not the whole Spec / Context, use the SkipBenchmark SpecOption instead.
func (*Spec) Tag ¶ added in v0.10.0
Tag allow you to mark tests in the current and below specification scope with tags. This can be used to provide additional documentation about the nature of the testing scope. This later might be used as well to filter your test in your CI/CD pipeline to build separate testing stages like integration, e2e and so on.
To select or exclude tests with certain tags, you can provide a comma separated list to the following environment variables:
- TESTCASE_TAG_INCLUDE to filter down to test with a certain tag
- TESTCASE_TAG_EXCLUDE to exclude certain test from the overall testing scope.
They can be combined as well.
example usage:
TESTCASE_TAG_INCLUDE='E2E' go test ./... TESTCASE_TAG_EXCLUDE='E2E' go test ./... TESTCASE_TAG_INCLUDE='E2E' TESTCASE_TAG_EXCLUDE='list,of,excluded,tags' go test ./...
func (*Spec) Test ¶
func (spec *Spec) Test(desc string, test tBlock, opts ...SpecOption)
Test creates a test case block where you receive the fully configured `testcase#T` object. Hook contents that meant to run before the test edge cases will run before the function the Test receives, and hook contents that meant to run after the test edge cases will run after the function is done. After hooks are deferred after the received function block, so even in case of panic, it will still be executed.
It should not contain anything that modify the test subject input. It should focuses only on asserting the result of the subject.
func (*Spec) Then ¶
func (spec *Spec) Then(desc string, test tBlock, opts ...SpecOption)
Then is an alias for Test
type SpecOption ¶ added in v0.28.0
type SpecOption interface {
// contains filtered or unexported methods
}
func Flaky ¶ added in v0.23.0
func Flaky(CountOrTimeout interface{}) SpecOption
Flaky will mark the spec/testCase as unstable. Flaky testCase execution is tolerant towards failing assertion and these tests will be rerun in case of a failure. A Wait Timeout for a successful flaky testCase must be provided.
The primary use-case is that when a team focus on shipping orderingOutput the value, and time is short till deadlines. These flaky tests prevent CI/CD pipelines often turned off in the heat of the moment to let pass the latest changes. The motivation behind is to gain time for the team to revisit these tests after the release and then learn from it. At the same time, they intend to fix it as well. These tests, however often forgotten, and while they are not the greatest assets of the CI pipeline, they often still serve essential value.
As a Least wrong solution, instead of skipping these tests, you can mark them as flaky, so in a later time, finding these flaky tests in the project should be easy. When you flag a testCase as flaky, you must provide a timeout value that will define a testing time window where the testCase can be rerun multiple times by the framework. If the testCase can't run successfully within this time-window, the testCase will fail. This failure potentially means that the underlying functionality is broken, and the committer should reevaluate the changes in the last commit.
While this functionality might help in tough times, it is advised to pair the usage with a scheduled monthly CI pipeline job. The Job should check the testing code base for the flaky flag.
func Group ¶ added in v0.23.5
func Group(name string) SpecOption
Group creates a testing group in the specification. During testCase execution, a group will be bundled together, and parallel tests will run concurrently within the the testing group.
func RetryStrategyForEventually ¶ added in v0.59.0
func RetryStrategyForEventually(strategy assert.RetryStrategy) SpecOption
type Suite ¶ added in v0.79.0
type Suite interface { // Spec defines the tests on the received *Spec object. Spec(s *Spec) }
Suite meant to represent a testing suite. A test Suite is a collection of test cases. In a test suite, the test cases are organized in a logical order. A Suite is a great tool to define interface testing suites (contracts).
type T ¶
type T struct { // TB is the interface common to T and B. testing.TB // Random is a random generator that uses the Spec seed. // // When a test fails with random input from Random generator, // the failed test scenario can be recreated simply by providing the same TESTCASE_SEED // as you can read from the console output of the failed test. Random *random.Random // It provides asserters to make assertion easier. // Must Interface will use FailNow on a failed assertion. // This will make test exit early on. // Should Interface's will allow to continue the test scenario, // but mark test failed on a failed assertion. assert.It // contains filtered or unexported fields }
T embeds both testcase vars, and testing#T functionality. This leave place open for extension and but define a stable foundation for the hooks and testCase edge case function signatures
Works as a drop in replacement for packages where they depend on one of the function of testing#T
func (*T) Defer ¶ added in v0.2.1
func (t *T) Defer(fn interface{}, args ...interface{})
Defer function defers the execution of a function until the current test case returns. Deferred functions are guaranteed to run, regardless of panics during the test case execution. Deferred function calls are pushed onto a testcase runtime stack. When an function passed to the Defer function, it will be executed as a deferred call in last-in-first-orderingOutput order.
It is advised to use this inside a testcase.Spec#Let memorization function when spec variable defined that has finalizer requirements. This allow the specification to ensure the object finalizer requirements to be met, without using an testcase.Spec#After where the memorized function would be executed always, regardless of its actual need.
In a practical example, this means that if you have common vars defined with testcase.Spec#Let memorization, which needs to be Closed for example, after the test case already run. Ensuring such objects Close call in an after block would cause an initialization of the memorized object list the time, even in tests where this is not needed.
e.g.:
- mock initialization with mock controller, where the mock controller #Finish function must be executed after each testCase suite.
- sql.DB / sql.Tx
- basically anything that has the io.Closer interface
func (*T) Eventually ¶ added in v0.59.0
Eventually helper allows you to write expectations to results that will only be eventually true. A common scenario where using Eventually will benefit you is testing concurrent operations. Due to the nature of async operations, one might need to wait and observe the system with multiple tries before the outcome can be seen. Eventually will attempt to assert multiple times with the assertion function block, until the expectations in the function body yield no testing failure. Calling multiple times the assertion function block content should be a safe and repeatable operation. For more, read the documentation of Eventually and Eventually.Assert. In case Spec doesn't have a configuration for how to retry Eventually, the DefaultEventually will be used.
func (*T) LogPretty ¶ added in v0.115.0
LogPretty will Log out values in pretty print format (pp.Format).
func (*T) SetEnv ¶ added in v0.105.0
SetEnv will set the os environment variable for the current program to a given value, and prepares a cleanup function to restore the original state of the environment variable.
This cannot be used in parallel tests.
func (*T) Setenv ¶ added in v0.105.0
Setenv calls os.Setenv(key, value) and uses Cleanup to restore the environment variable to its original value after the test.
This cannot be used in parallel tests.
type TBRunner ¶ added in v0.37.0
type TBRunner interface { testing.TB // Run runs blk as a subtest of TBRunner called group. It runs blk in a separate goroutine // and blocks until blk returns or calls t.parallel to become a parallel testCase. // Run reports whether blk succeeded (or at least did not fail before calling t.parallel). // // Run may be called simultaneously from multiple goroutines, but list such calls // must return before the outer testCase function for t returns. Run(name string, blk func(tb testing.TB)) bool }
TBRunner defines the interface you need to implement if you want to create a custom TB that is compatible with Spec. To implement TBRunner correctly please use contracts.TB
import ( "github.com/adamluzsi/testcase/contracts" "testing" ) func TestMyTestRunner(t *testing.T) { contracts.TB{Subject: func(tb testing.TB) testcase.TBRunner { return MyTestRunner{TB: tb} }}.Test(t) }
type Var ¶ added in v0.16.0
type Var[V any] struct { // ID is the testCase spec variable group from where the cached value can be accessed later on. // ID is Mandatory when you create a variable, else the empty string will be used as the variable group. ID string // Init is an optional constructor definition that will be used when Var is bonded to a *Spec without constructor function passed to the Let function. // The goal of this field to initialize a variable that can be reused across different testing suites by bounding the Var to a given testing suite. // // Please use #Get if you wish to access a testCase runtime across cached variable value. // The value returned by this is not subject to any #Before and #Around hook that might mutate the variable value during the testCase runtime. // Init function doesn't cache the value in the testCase runtime spec but literally just meant to initialize a value for the Var in a given test case. // Please use it with caution. Init VarInit[V] // Before is a hook that will be executed once during the lifetime of tests that uses the Var. // If the Var is not bound to the Spec at Spec.Context level, the Before Hook will be executed at Var.Get. Before func(t *T, v Var[V]) // OnLet is an optional Var hook that is executed when the variable being bind to Spec context. // This hook is ideal to set up tags on the Spec, call Spec.Sequential // or ensure binding of further dependencies that this variable requires. // // In case OnLet is provided, the Var must be explicitly set to a Spec with a Let call // else accessing the Var value will panic and warn about this. OnLet func(s *Spec, v Var[V]) }
Var is a testCase helper structure, that allows easy way to access testCase runtime variables. In the future it will be updated to use Go2 type parameters.
Var allows creating testCase variables in a modular way. By modular, imagine that you can have commonly used values initialized and then access it from the testCase runtime spec. This approach allows an easy dependency injection maintenance at project level for your testing suite. It also allows you to have parallel testCase execution where you don't expect side effect from your subject.
e.g.: HTTP JSON API testCase and GraphQL testCase both use the business rule instances. Or multiple business rules use the same storage dependency.
The last use-case it allows is to define dependencies for your testCase subject before actually assigning values to it. Then you can focus on building up the testing spec and assign values to the variables at the right testing subcontext. With variables, it is easy to forget to assign a value to a variable or forgot to clean up the value of the previous run and then scratch the head during debugging. If you forgot to set a value to the variable in testcase, it warns you that this value is not yet defined to the current testing scope.
func Let ¶ added in v0.66.0
Let define a memoized helper method. Let creates lazily-evaluated test execution bound variables. Let variables don't exist until called into existence by the actual tests, so you won't waste time loading them for examples that don't use them. They're also memoized, so they're useful for encapsulating database objects, due to the cost of making a database request. The value will be cached across list use within the same test execution but not across different test cases. You can eager load a value defined in let by referencing to it in a Before hook. Let is threadsafe, the parallel running test will receive they own test variable instance.
Defining a value in a spec Context will ensure that the scope and it's nested scopes of the current scope will have access to the value. It cannot leak its value outside from the current scope. Calling Let in a nested/sub scope will apply the new value for that value to that scope and below.
It will panic if it is used after a When/And/Then scope definition, because those scopes would have no clue about the later defined variable. In order to keep the specification reading mental model requirement low, it is intentionally not implemented to handle such case. Defining test vars always expected in the beginning of a specification scope, mainly for readability reasons.
vars strictly belong to a given `Describe`/`When`/`And` scope, and configured before any hook would be applied, therefore hooks always receive the most latest version from the `Let` vars, regardless in which scope the hook that use the variable is define.
Let can enhance readability when used sparingly in any given example group, but that can quickly degrade with heavy overuse.
func LetValue ¶ added in v0.66.0
LetValue is a shorthand for defining immutable vars with Let under the hood. So the function blocks can be skipped, which makes tests more readable.
func (Var[V]) Bind ¶ added in v0.56.0
Bind is a syntax sugar shorthand for Var.Let(*Spec, nil), where skipping providing a block meant to be explicitly expressed.
func (Var[V]) EagerLoading ¶ added in v0.16.0
EagerLoading allows the variable to be loaded before the action and assertion block is reached. This can be useful when you want to have a variable that cause side effect on your system. Like it should be present in some sort of attached resource/storage.
For example, you may persist the value in a storage as part of the initialization block, and then when the testCase/then block is reached, the entity is already present in the resource.
func (Var[V]) Get ¶ added in v0.16.0
Get returns the current cached value of the given Variable Get is a thread safe operation. When Go2 released, it will replace type casting
func (Var[V]) PreviousValue ¶ added in v0.103.1
PreviousValue is a syntax sugar for Var.Super to access the previous declaration's value.
func (Var[V]) Set ¶ added in v0.16.0
Set sets a value to a given variable during testCase runtime Set is a thread safe operation.
func (Var[V]) Super ¶ added in v0.103.0
Super will return the inherited Super value of your Var. This means that if you declared Var in an outer Spec.Context, or your Var has an Var.Init field, then Var.Super will return its content. This allows you to incrementally extend with values the inherited value until you reach your testing scope. This also allows you to wrap your Super value with a Spy or Stub wrapping layer, and pry the interactions with the object while using the original value as a base.
type VarInitFunc ¶ added in v0.84.0
VarInitFunc is a backward compatibility type for VarInit.
DEPRECATED: use VarInit type instead.
Source Files
¶
Directories
¶
Path | Synopsis |
---|---|
docs
|
|
Package let contains Common Testcase Variable Let declarations for testing purpose.
|
Package let contains Common Testcase Variable Let declarations for testing purpose. |