e2e

package
v0.58.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2024 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package e2e provides the API to manage environments and organize E2E tests. Three major concepts are used to write E2E tests:

  • e2e.Provisioner: A provisioner is a component that provide compute resources (usually Cloud resources). Most common is Pulumi through `test-infra-definitions`.
  • e2e.BaseSuite: A TestSuite is a collection of tests that share the ~same environment.
  • Environment: An environment is a collection of resources (virtual machine, agent, etc). An environment is filled by provisioners.

See usage examples in the [examples] package.

Provisioners

Three provisioners are available:

Pulumi Provisioner can be typed or untyped:

  • Typed provisioners are provisioners that are typed with the environment they provision and the `Run` function must be defined in `datadog-agent` inline.
  • Untyped provisioners are provisioners that are not typed with the environment they provision and the `Run` function can come from anywhere.
  • e2e.StaticProvisioner: A provisioner that uses static resources from a JSON file. The static provisioner is Untyped.

# Impact of Typed vs Untyped provisioners Typed provisioners are more convenient to use as they are typed with the environment they provision, however they do require a close mapping between the RunFunc and the environment. With a Typed provisioner, the `component.Export()` function is used to match an Environment field with a Pulumi resource.

An Untyped provisioner is more flexible as it does not require a close mapping between the RunFunc and the environment. It allows to get resources from anywhere in the same environment. However it means that the environment needs to be annotated with the `import` tag to match the resource key. See for instance the examples/suite_serial_kube_test.go file.

Out-of-the-box environments and provisioners

Check the [environments] package for a list of out-of-the-box environments, for instance [environments.VM]. Check the `environments/<cloud>` for a list of out-of-the-box provisioners, for instance environments/aws/vm.

The BaseSuite test suite

The e2e.BaseSuite test suite is a testify Suite that wraps environment and provisioners. It allows to easily write tests that share the same environment without having to re-implement boilerplate code. Check all the e2e.SuiteOption to customize the behavior of the BaseSuite.

Note: By default, the BaseSuite test suite will delete the environment when the test suite finishes (whether it's successful or not). During development, it's highly recommended to use the [params.WithDevMode] option to prevent the environment from being deleted. [params.WithDevMode] is automatically enabled when the `E2E_DEV_MODE` environment variable is set to `true`.

Organizing your tests

The execution order for tests in testify Suite is IMPLEMENTATION SPECIFIC UNLIKE REGULAR GO TESTS. Use subtests for ordered tests and environments update.

Having a single environment

In the simple case, there is a single environment and each test checks one specific thing.

import (
	"testing"

	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/e2e"
	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments"
)

type singleEnvSuite struct {
	e2e.BaseSuite[environments.VM]
}

func TestSingleEnvSuite(t *testing.T) {
	e2e.Run(t, &singleEnvSuite{}, e2e.WithProvisioner(awshost.Provisioner()))
}

func (suite *singleEnvSuite) Test1() {
	// Check feature 1
}

func (suite *singleEnvSuite) Test2() {
	// Check feature 2
}

func (suite *singleEnvSuite) Test3() {
	// Check feature 3
}

Having different environments

You may sometime have different environments but several tests for each on them. You can use e2e.Suite.UpdateEnv to do that. Using `UpdateEnv` between groups of Subtests. Note that between `TestLogDebug` and `TestLogInfo`, the environment is reverted to the original one.

import (
	"testing"

	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/e2e"
	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments"
	awsvm "github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments/aws/vm"
	"github.com/DataDog/test-infra-definitions/components/datadog/agentparams"
)

type subTestSuite struct {
	e2e.Suite[environments.VM]
}

func TestSubTestSuite(t *testing.T) {
	e2e.Run(t, &singleEnvSuite{}, e2e.WithProvisioner(awshost.Provisioner()))
}

func (suite *subTestSuite) TestLogDebug() {
	// First group of subsets
	suite.T().Run("MySubTest1", func(t *testing.T) {
		// Sub test 1
	})
	suite.T().Run("MySubTest2", func(t *testing.T) {
		// Sub test 2
	})

	v.UpdateEnv(awshost.Provisioner(awshost.WithAgentOptions(agentparams.WithAgentConfig("log_level: debug"))))

	// Second group of subsets
	suite.T().Run("MySubTest3", func(t *testing.T) {
		// Sub test 3
	})
}

func (suite *subTestSuite) TestLogInfo() {
	// First group of subsets
	suite.T().Run("MySubTest1", func(t *testing.T) {
		// Sub test 1
	})
	suite.T().Run("MySubTest2", func(t *testing.T) {
		// Sub test 2
	})

	v.UpdateEnv(awshost.Provisioner(awshost.WithAgentOptions(agentparams.WithAgentConfig("log_level: info"))))

	// Second group of subsets
	suite.T().Run("MySubTest3", func(t *testing.T) {
		// Sub test 3
	})
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run

func Run[Env any, T Suite[Env]](t *testing.T, s T, options ...SuiteOption)

Run is a helper function to run a test suite. Unfortunately, we cannot use `s Suite[Env]` as Go is not able to match it with a struct However it's able to verify the same constraint on T

Types

type BaseSuite

type BaseSuite[Env any] struct {
	suite.Suite
	// contains filtered or unexported fields
}

BaseSuite is a generic test suite that wraps testify.Suite

func (*BaseSuite[Env]) AfterTest

func (bs *BaseSuite[Env]) AfterTest(suiteName, testName string)

AfterTest is executed right after the test finishes and receives the suite and test names as input. This function is called by testify Suite.

If you override AfterTest in your custom test suite type, the function must call [test.BaseSuite.AfterTest].

func (*BaseSuite[Env]) BeforeTest

func (bs *BaseSuite[Env]) BeforeTest(string, string)

BeforeTest is executed right before the test starts and receives the suite and test names as input. This function is called by testify Suite.

If you override BeforeTest in your custom test suite type, the function must call [test.BaseSuite.BeforeTest].

func (*BaseSuite[Env]) Env

func (bs *BaseSuite[Env]) Env() *Env

Env returns the current environment

func (*BaseSuite[Env]) IsDevMode

func (bs *BaseSuite[Env]) IsDevMode() bool

IsDevMode returns true if the test suite is running in dev mode. WARNING: IsDevMode should not be used. It's a recipe to get tests working locally but failing in CI.

func (*BaseSuite[Env]) SetupSuite

func (bs *BaseSuite[Env]) SetupSuite()

SetupSuite run before all the tests in the suite have been run. This function is called by testify Suite.

If you override SetupSuite in your custom test suite type, the function must call e2e.BaseSuite.SetupSuite.

func (*BaseSuite[Env]) TearDownSuite

func (bs *BaseSuite[Env]) TearDownSuite()

TearDownSuite run after all the tests in the suite have been run. This function is called by testify Suite.

If you override TearDownSuite in your custom test suite type, the function must call e2e.BaseSuite.TearDownSuite.

func (*BaseSuite[Env]) UpdateEnv

func (bs *BaseSuite[Env]) UpdateEnv(newProvisioners ...Provisioner)

UpdateEnv updates the environment with new provisioners.

type Context

type Context interface {
	T() *testing.T
}

Context defines an interface that allows to get information about current test context

type Diagnosable added in v0.56.0

type Diagnosable interface {
	Diagnose(ctx context.Context, stackName string) (string, error)
}

Diagnosable defines the interface for a diagnosable provider.

type FileProvisioner

type FileProvisioner struct {
	// contains filtered or unexported fields
}

FileProvisioner is a provisioner that reads JSON files from a filesystem.

func NewFileProvisioner

func NewFileProvisioner(id string, fs fs.FS) *FileProvisioner

NewFileProvisioner returns a new FileProvisioner.

func (*FileProvisioner) Destroy

Destroy is a no-op for the FileProvisioner.

func (*FileProvisioner) ID

func (fp *FileProvisioner) ID() string

ID returns the ID of the provisioner.

func (*FileProvisioner) Provision

Provision reads JSON files from the filesystem and returns them as raw resources.

type Initializable

type Initializable interface {
	Init(Context) error
}

Initializable defines the interface for an object that needs to be initialized

type Provisioner

type Provisioner interface {
	ID() string
	Destroy(context.Context, string, io.Writer) error
}

Provisioner defines the interface for a provisioner.

type ProvisionerMap

type ProvisionerMap map[string]Provisioner

ProvisionerMap is a map of provisioners.

type PulumiEnvRunFunc

type PulumiEnvRunFunc[Env any] func(ctx *pulumi.Context, env *Env) error

PulumiEnvRunFunc is a function that runs a Pulumi program with a given environment.

type PulumiProvisioner

type PulumiProvisioner[Env any] struct {
	// contains filtered or unexported fields
}

PulumiProvisioner is a provisioner based on Pulumi with binding to an environment.

func NewTypedPulumiProvisioner

func NewTypedPulumiProvisioner[Env any](id string, runFunc PulumiEnvRunFunc[Env], configMap runner.ConfigMap) *PulumiProvisioner[Env]

NewTypedPulumiProvisioner returns a new PulumiProvisioner.

func NewUntypedPulumiProvisioner

func NewUntypedPulumiProvisioner(id string, runFunc pulumi.RunFunc, configMap runner.ConfigMap) *PulumiProvisioner[any]

NewUntypedPulumiProvisioner returns a new PulumiProvisioner without env binding.

func (*PulumiProvisioner[Env]) Destroy

func (pp *PulumiProvisioner[Env]) Destroy(ctx context.Context, stackName string, logger io.Writer) error

Destroy deletes the Pulumi stack.

func (*PulumiProvisioner[Env]) Diagnose added in v0.56.0

func (pp *PulumiProvisioner[Env]) Diagnose(ctx context.Context, stackName string) (string, error)

Diagnose runs the diagnose function if it is set diagnoseFunc

func (*PulumiProvisioner[Env]) ID

func (pp *PulumiProvisioner[Env]) ID() string

ID returns the ID of the provisioner.

func (*PulumiProvisioner[Env]) Provision

func (pp *PulumiProvisioner[Env]) Provision(ctx context.Context, stackName string, logger io.Writer) (RawResources, error)

Provision runs the Pulumi program and returns the raw resources.

func (*PulumiProvisioner[Env]) ProvisionEnv

func (pp *PulumiProvisioner[Env]) ProvisionEnv(ctx context.Context, stackName string, logger io.Writer, env *Env) (RawResources, error)

ProvisionEnv runs the Pulumi program with a given environment and returns the raw resources.

func (*PulumiProvisioner[Env]) SetDiagnoseFunc added in v0.56.0

func (pp *PulumiProvisioner[Env]) SetDiagnoseFunc(diagnoseFunc func(ctx context.Context, stackName string) (string, error))

SetDiagnoseFunc sets the diagnose function.

type RawResources

type RawResources map[string][]byte

RawResources is the common types returned by provisioners

func (RawResources) Merge

func (rr RawResources) Merge(in RawResources)

Merge merges two RawResources maps

type Suite

type Suite[Env any] interface {
	suite.TestingSuite

	UpdateEnv(...Provisioner)
	Env() *Env
	// contains filtered or unexported methods
}

Suite is a generic inteface used internally, only implemented by BaseSuite

type SuiteOption

type SuiteOption = func(*suiteParams)

SuiteOption is an optional function parameter type for e2e options

func WithDevMode

func WithDevMode() SuiteOption

WithDevMode enables dev mode. Dev mode doesn't destroy the environment when the test is finished which can be useful when writing a new E2E test.

func WithProvisioner

func WithProvisioner(provisioner Provisioner) SuiteOption

WithProvisioner adds a provisioner to the suite

func WithPulumiProvisioner

func WithPulumiProvisioner[Env any](runFunc PulumiEnvRunFunc[Env], configMap runner.ConfigMap) SuiteOption

WithPulumiProvisioner adds a typed Pulumi provisioner to the suite

func WithSkipDeleteOnFailure

func WithSkipDeleteOnFailure() SuiteOption

WithSkipDeleteOnFailure doesn't destroy the environment when a test fails.

func WithStackName

func WithStackName(stackName string) SuiteOption

WithStackName overrides the default stack name. This function is useful only when using Run.

func WithUntypedPulumiProvisioner

func WithUntypedPulumiProvisioner(runFunc pulumi.RunFunc, configMap runner.ConfigMap) SuiteOption

WithUntypedPulumiProvisioner adds an untyped Pulumi provisioner to the suite

type TypedProvisioner

type TypedProvisioner[Env any] interface {
	Provisioner
	ProvisionEnv(context.Context, string, io.Writer, *Env) (RawResources, error)
}

TypedProvisioner defines the interface for a provisioner with env binding

type UntypedProvisioner

type UntypedProvisioner interface {
	Provisioner
	Provision(context.Context, string, io.Writer) (RawResources, error)
}

UntypedProvisioner defines the interface for a provisioner without env binding

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL