Documentation
¶
Overview ¶
Package framework implements a model for developing end-to-end test suites. The model includes a top level Framework which TestSuites can be added to. TestSuites include conditions under which the suite will run and a list of TestCase implementations to run. TestCases can be implemented with methods that run before/after each and all tests.
Writing Tests ¶
Tests follow a similar patterns as go tests. They are functions that must start with 'Test' and instead of a *testing.T argument, a *framework.F is passed and they must have a receiver that implements the TestCase interface. A crude example as follows:
// foo_test.go
type MyTestCase struct {
framework.TC
}
func (tc *MyTestCase) TestMyFoo(f *framework.F) {
f.T().Log("bar")
}
func TestCalledFromGoTest(t *testing.T){
framework.New().AddSuites(&framework.TestSuite{
Component: "foo",
Cases: []framework.TestCase{
new(MyTestCase),
},
}).Run(t)
}
Test cases should embed the TC struct which satisfies the TestCase interface. Optionally a TestCase can also implement the Name() function which returns a string to name the test case. By default the name is the name of the struct type, which in the above example would be "MyTestCase"
Test cases may also optionally implement additional interfaces to define setup and teardown logic:
BeforeEachTest AfterEachTest BeforeAllTests AfterAllTests
The test case struct allows you to setup and teardown state in the struct that can be consumed by the tests. For example:
type ComplexNomadTC struct {
framework.TC
jobID string
}
func (tc *ComplexNomadTC) BeforeEach(f *framework.F){
// Do some complex job setup with a unique prefix string
jobID, err := doSomeComplexSetup(tc.Nomad(), f.ID())
f.NoError(err)
f.Set("jobID", jobID)
}
func (tc *ComplexNomadTC) TestSomeScenario(f *framework.F){
jobID := f.Value("jobID").(string)
doTestThingWithJob(f, tc.Nomad(), jobID)
}
func (tc *ComplexNomadTC) TestOtherScenario(f *framework.F){
jobID := f.Value("jobID").(string)
doOtherTestThingWithJob(f, tc.Nomad(), jobID)
}
func (tc *ComplexNomadTC) AfterEach(f *framework.F){
jobID := f.Value("jobID").(string)
_, _, err := tc.Nomad().Jobs().Deregister(jobID, true, nil)
f.NoError(err)
}
As demonstrated in the previous example, TC also exposes functions that return configured api clients including Nomad, Consul and Vault. If Consul or Vault are not provisioned their respective getter functions will return nil.
Testify Integration ¶
Test cases expose a T() function to fetch the current *testing.T context. While this means the author is able to most other testing libraries, github.com/stretch/testify is recommended and integrated into the framework. The TC struct also embeds testify assertions that are preconfigured with the current testing context. Additionally TC comes with a Require() method that yields a testify Require if that flavor is desired.
func (tc *MyTestCase) TestWithTestify() {
err := someErrFunc()
tc.NoError(err)
// Or tc.Require().NoError(err)
}
Parallelism ¶
The test framework honors go test's parallel feature under certain conditions. A TestSuite can be created with the Parallel field set to true to enable parallel execution of the test cases of the suite. Tests within a test case will be executed sequentially unless f.T().Parallel() is called. Note that if multiple tests are to be executed in parallel, access to TC is not syncronized. The *framework.F offers a way to store state between before/after each method if desired.
func (tc *MyTestCase) BeforeEach(f *framework.F){
jobID, _ := doSomeComplexSetup(tc.Nomad(), f.ID())
f.Set("jobID", jobID)
}
func (tc *MyTestCase) TestParallel(f *framework.F){
f.T().Parallel()
jobID := f.Value("jobID").(string)
}
Since test cases have the potential to work with a shared Nomad cluster in parallel any resources created or destroyed must be prefixed with a unique identifier for each test case. The framework.F struct exposes an ID() function that will return a string that is unique with in a test. Therefore, multiple tests with in the case can reliably create unique IDs between tests and setup/teardown. The string returned is 8 alpha numeric characters.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AfterAllTests ¶
type AfterAllTests interface {
AfterAll(*F)
}
AfterAllTests is used to define a method to be called after the execution of all tests.
type AfterEachTest ¶
type AfterEachTest interface {
AfterEach(*F)
}
AfterEachTest is used to define a method to be called after each test.
type BeforeAllTests ¶
type BeforeAllTests interface {
BeforeAll(*F)
}
BeforeAllTests is used to define a method to be called before the execution of all tests.
type BeforeEachTest ¶
type BeforeEachTest interface {
BeforeEach(*F)
}
BeforeEachTest is used to define a method to be called before each test.
type ClusterInfo ¶
type ClusterInfo struct {
ID string
Name string
NomadClient *napi.Client
ConsulClient *capi.Client
VaultClient *vapi.Client
}
ClusterInfo is a handle to a provisioned cluster, along with clients a test run can use to connect to the cluster.
type Constraints ¶
type Constraints struct {
Provider string // Cloud provider ex. 'aws', 'azure', 'gcp'
OS string // Operating system ex. 'windows', 'linux'
Arch string // CPU architecture ex. 'amd64', 'arm64'
Environment string // Environment name ex. 'simple'
Tags []string // Generic tags that must all exist in the environment
}
Constraints that must be satisfied for a TestSuite to run
type Environment ¶
type Environment struct {
Name string
Provider string
OS string
Arch string
Tags map[string]struct{}
}
Environment contains information about the test target environment, used to constrain the set of tests run. See the environment flags above.
type F ¶
type F struct {
*require.Assertions
// contains filtered or unexported fields
}
F is the framework context that is passed to each test. It is used to access the *testing.T context as well as testify helpers
func (*F) Assert ¶
func (f *F) Assert() *assert.Assertions
Assert fetches an assert flavor of testify assertions https://godoc.org/github.com/stretchr/testify/assert
type Framework ¶
type Framework struct {
// contains filtered or unexported fields
}
type Provisioner ¶
type Provisioner interface {
// SetupTestRun is called at the start of the entire test run.
SetupTestRun(t *testing.T, opts SetupOptions) (*ClusterInfo, error)
// SetupTestSuite is called at the start of each TestSuite.
// TODO: no current provisioner implementation uses this, but we
// could use it to provide each TestSuite with an entirely separate
// Nomad cluster.
SetupTestSuite(t *testing.T, opts SetupOptions) (*ClusterInfo, error)
// SetupTestCase is called at the start of each TestCase in every TestSuite.
SetupTestCase(t *testing.T, opts SetupOptions) (*ClusterInfo, error)
// TearDownTestCase is called after each TestCase in every TestSuite.
TearDownTestCase(t *testing.T, clusterID string) error
// TearDownTestSuite is called after every TestSuite.
TearDownTestSuite(t *testing.T, clusterID string) error
// TearDownTestRun is called at the end of the entire test run.
TearDownTestRun(t *testing.T, clusterID string) error
}
Provisioner interface is used by the test framework to provision API clients for a Nomad cluster, with the possibility of extending to provision standalone clusters for each test case in the future.
The Setup* methods are hooks that get run at the appropriate stage. They return a ClusterInfo handle that helps TestCases isolate test state if they use the ClusterInfo.ID as part of job IDs.
The TearDown* methods are hooks to clean up provisioned cluster state that isn't covered by the test case's implementation of AfterEachTest.
var DefaultProvisioner Provisioner = new(singleClusterProvisioner)
DefaultProvisioner is a Provisioner that doesn't deploy a Nomad cluster (because that's handled by Terraform elsewhere), but build clients from environment variables.
type SetupOptions ¶
type SetupOptions struct {
Name string
ExpectConsul bool // If true, fails if a Consul client can't be configured
ExpectVault bool // If true, fails if a Vault client can't be configured
}
SetupOptions defines options to be given to the Provisioner when calling Setup* methods.
type TC ¶
type TC struct {
// contains filtered or unexported fields
}
TC is the base test case which should be embedded in TestCase implementations.
type TestCase ¶
type TestCase interface {
Name() string
// contains filtered or unexported methods
}
TestCase is the interface which an E2E test case implements. It is not meant to be implemented directly, instead the struct should embed the 'framework.TC' struct
type TestSuite ¶
type TestSuite struct {
Component string // Name of the component/system/feature tested
CanRunLocal bool // Flags if the cases are safe to run on a local nomad cluster
Cases []TestCase // Cases to run
Constraints Constraints // Environment constraints to follow
Parallel bool // If true, will run test cases in parallel
Slow bool // Slow test suites don't run by default
// API Clients
Consul bool
Vault bool
}
TestSuite defines a set of test cases and under what conditions to run them
Source Files