framework

package
v0.10.0-beta1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 6, 2019 License: MPL-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package framework implements a model for developing end-to-end test suites. The model includes a top level Framework which TestSuites can be added to. TestSuites include conditions under which the suite will run and a list of TestCase implementations to run. TestCases can be implemented with methods that run before/after each and all tests.

Writing Tests

Tests follow a similar patterns as go tests. They are functions that must start with 'Test' and instead of a *testing.T argument, a *framework.F is passed and they must have a receiver that implements the TestCase interface. A crude example as follows:

// foo_test.go
type MyTestCase struct {
	framework.TC
}

func (tc *MyTestCase) TestMyFoo(f *framework.F) {
	f.T().Log("bar")
}

func TestCalledFromGoTest(t *testing.T){
	framework.New().AddSuites(&framework.TestSuite{
		Component:   "foo",
		Cases: []framework.TestCase{
			new(MyTestCase),
		},
	}).Run(t)
}

Test cases should embed the TC struct which satisfies the TestCase interface. Optionally a TestCase can also implement the Name() function which returns a string to name the test case. By default the name is the name of the struct type, which in the above example would be "MyTestCase"

Test cases may also optionally implement additional interfaces to define setup and teardown logic:

BeforeEachTest
AfterEachTest
BeforeAllTests
AfterAllTests

The test case struct allows you to setup and teardown state in the struct that can be consumed by the tests. For example:

type ComplexNomadTC struct {
	framework.TC
	jobID string
}

func (tc *ComplexNomadTC) BeforeEach(f *framework.F){
	// Do some complex job setup with a unique prefix string
	jobID, err := doSomeComplexSetup(tc.Nomad(), f.ID())
	f.NoError(err)
	f.Set("jobID", jobID)
}

func (tc *ComplexNomadTC) TestSomeScenario(f *framework.F){
	jobID := f.Value("jobID").(string)
	doTestThingWithJob(f, tc.Nomad(), jobID)
}

func (tc *ComplexNomadTC) TestOtherScenario(f *framework.F){
	jobID := f.Value("jobID").(string)
	doOtherTestThingWithJob(f, tc.Nomad(), jobID)
}

func (tc *ComplexNomadTC) AfterEach(f *framework.F){
	jobID := f.Value("jobID").(string)
	_, _, err := tc.Nomad().Jobs().Deregister(jobID, true, nil)
	f.NoError(err)
}

As demonstrated in the previous example, TC also exposes functions that return configured api clients including Nomad, Consul and Vault. If Consul or Vault are not provisioned their respective getter functions will return nil.

Testify Integration

Test cases expose a T() function to fetch the current *testing.T context. While this means the author is able to most other testing libraries, github.com/stretch/testify is recommended and integrated into the framework. The TC struct also embeds testify assertions that are preconfigured with the current testing context. Additionally TC comes with a Require() method that yields a testify Require if that flavor is desired.

func (tc *MyTestCase) TestWithTestify() {
	err := someErrFunc()
	tc.NoError(err)
	// Or tc.Require().NoError(err)
}

Parallelism

The test framework honors go test's parallel feature under certain conditions. A TestSuite can be created with the Parallel field set to true to enable parallel execution of the test cases of the suite. Tests within a test case will be executed sequentially unless f.T().Parallel() is called. Note that if multiple tests are to be executed in parallel, access to TC is not syncronized. The *framework.F offers a way to store state between before/after each method if desired.

func (tc *MyTestCase) BeforeEach(f *framework.F){
	jobID, _ := doSomeComplexSetup(tc.Nomad(), f.ID())
	f.Set("jobID", jobID)
}

func (tc *MyTestCase) TestParallel(f *framework.F){
	f.T().Parallel()
	jobID := f.Value("jobID").(string)
}

Since test cases have the potential to work with a shared Nomad cluster in parallel any resources created or destroyed must be prefixed with a unique identifier for each test case. The framework.F struct exposes an ID() function that will return a string that is unique with in a test. Therefore, multiple tests with in the case can reliably create unique IDs between tests and setup/teardown. The string returned is 8 alpha numeric characters.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run

func Run(t *testing.T)

Run starts the package scoped Framework, running each TestSuite

Types

type AfterAllTests

type AfterAllTests interface {
	AfterAll(*F)
}

AfterAllTests is used to define a method to be called after the execution of all tests.

type AfterEachTest

type AfterEachTest interface {
	AfterEach(*F)
}

AfterEachTest is used to degine a method to be called after each test.

type BeforeAllTests

type BeforeAllTests interface {
	BeforeAll(*F)
}

BeforeAllTests is used to define a method to be called before the execution of all tests.

type BeforeEachTest

type BeforeEachTest interface {
	BeforeEach(*F)
}

BeforeEachTest is used to define a method to be called before each test.

type ClusterInfo

type ClusterInfo struct {
	ID           string
	Name         string
	NomadClient  *napi.Client
	ConsulClient *capi.Client
	VaultClient  *vapi.Client
}

type Constraints

type Constraints struct {
	Provider    string   // Cloud provider ex. 'aws', 'azure', 'gcp'
	OS          string   // Operating system ex. 'windows', 'linux'
	Arch        string   // CPU architecture ex. 'amd64', 'arm64'
	Environment string   // Environment name ex. 'simple'
	Tags        []string // Generic tags that must all exist in the environment
}

Constraints that must be satisfied for a TestSuite to run

type Environment

type Environment struct {
	Name     string
	Provider string
	OS       string
	Arch     string
	Tags     map[string]struct{}
}

Environment includes information about the target environment the test framework is targeting. During 'go run', these fields are populated by the following flags:

-env <string>		"name of the environment executing against"
-env.provider <string>	"cloud provider for which the environment is executing against"
-env.os <string>		"operating system of environment"
-env.arch <string>	"cpu architecture of environment"
-env.tags <string>	"comma delimited list of environment tags"

These flags are not needed when executing locally

type F

type F struct {
	*require.Assertions
	// contains filtered or unexported fields
}

F is the framework context that is passed to each test. It is used to access the *testing.T context as well as testify helpers

func (*F) Assert

func (f *F) Assert() *assert.Assertions

Assert fetches an assert flavor of testify assertions https://godoc.org/github.com/stretchr/testify/assert

func (*F) ID

func (f *F) ID() string

ID returns the current context ID

func (*F) Set

func (f *F) Set(key, val interface{})

Set is used to set arbitrary key/values to pass between before/after and test methods

func (*F) T

func (f *F) T() *testing.T

T returns the *testing.T context

func (*F) Value

func (f *F) Value(key interface{}) interface{}

Value retrives values set by the F.Set method

type Framework

type Framework struct {
	// contains filtered or unexported fields
}

func AddSuites

func AddSuites(s ...*TestSuite) *Framework

AddSuites adds a set of test suites to the package scoped Framework

func New

func New() *Framework

New creates a Framework

func (*Framework) AddSuites

func (f *Framework) AddSuites(s ...*TestSuite) *Framework

AddSuites adds a set of test suites to a Framework

func (*Framework) Run

func (f *Framework) Run(t *testing.T)

Run starts the test framework, running each TestSuite

type Provisioner

type Provisioner interface {
	ProvisionCluster(opts ProvisionerOptions) (*ClusterInfo, error)
	DestroyCluster(clusterID string) error
}

Provisioner interface is used by the test framework to provision a Nomad cluster for each TestCase

var DefaultProvisioner Provisioner = new(singleClusterProvisioner)

DefaultProvisioner is a noop provisioner that builds clients from environment variables according to the respective client configuration

type ProvisionerOptions

type ProvisionerOptions struct {
	Name         string
	ExpectConsul bool // If true, fails if a Consul client can't be configured
	ExpectVault  bool // If true, fails if a Vault client can't be configured
}

ProvisionerOptions defines options to be given to the Provisioner when calling ProvisionCluster

type TC

type TC struct {
	// contains filtered or unexported fields
}

TC is the base test case which should be embedded in TestCase implementations.

func (*TC) Consul

func (tc *TC) Consul() *capi.Client

Consul returns a configured consul api client

func (*TC) Name

func (tc *TC) Name() string

Name returns the name of the test case which is set to the name of the implementing type.

func (*TC) Nomad

func (tc *TC) Nomad() *api.Client

Nomad returns a configured nomad api client

type TestCase

type TestCase interface {
	Name() string
	// contains filtered or unexported methods
}

TestCase is the interface which an E2E test case implements. It is not meant to be implemented directly, instead the struct should embed the 'framework.TC' struct

type TestSuite

type TestSuite struct {
	Component string // Name of the component/system/feature tested

	CanRunLocal bool        // Flags if the cases are safe to run on a local nomad cluster
	Cases       []TestCase  // Cases to run
	Constraints Constraints // Environment constraints to follow
	Parallel    bool        // If true, will run test cases in parallel
	Slow        bool        // Slow test suites don't run by default

	// API Clients
	Consul bool
	Vault  bool
}

TestSuite defines a set of test cases and under what conditions to run them

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL