e2e

package
v0.62.0-rc.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 7, 2025 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package e2e provides the API to manage environments and organize E2E tests. Three major concepts are used to write E2E tests:

  • e2e.Provisioner: A provisioner is a component that provide compute resources (usually Cloud resources). Most common is Pulumi through `test-infra-definitions`.
  • e2e.BaseSuite: A TestSuite is a collection of tests that share the ~same environment.
  • Environment: An environment is a collection of resources (virtual machine, agent, etc). An environment is filled by provisioners.

See usage examples in the [examples] package.

Provisioners

Three provisioners are available:

Pulumi Provisioner can be typed or untyped:

  • Typed provisioners are provisioners that are typed with the environment they provision and the `Run` function must be defined in `datadog-agent` inline.
  • Untyped provisioners are provisioners that are not typed with the environment they provision and the `Run` function can come from anywhere.
  • e2e.StaticProvisioner: A provisioner that uses static resources from a JSON file. The static provisioner is Untyped.

# Impact of Typed vs Untyped provisioners Typed provisioners are more convenient to use as they are typed with the environment they provision, however they do require a close mapping between the RunFunc and the environment. With a Typed provisioner, the `component.Export()` function is used to match an Environment field with a Pulumi resource.

An Untyped provisioner is more flexible as it does not require a close mapping between the RunFunc and the environment. It allows to get resources from anywhere in the same environment. However it means that the environment needs to be annotated with the `import` tag to match the resource key. See for instance the examples/suite_serial_kube_test.go file.

Out-of-the-box environments and provisioners

Check the [environments] package for a list of out-of-the-box environments, for instance [environments.VM]. Check the `environments/<cloud>` for a list of out-of-the-box provisioners, for instance environments/aws/vm.

The BaseSuite test suite

The e2e.BaseSuite test suite is a testify Suite that wraps environment and provisioners. It allows to easily write tests that share the same environment without having to re-implement boilerplate code. Check all the e2e.SuiteOption to customize the behavior of the BaseSuite.

Note: By default, the BaseSuite test suite will delete the environment when the test suite finishes (whether it's successful or not). During development, it's highly recommended to use the [params.WithDevMode] option to prevent the environment from being deleted. [params.WithDevMode] is automatically enabled when the `E2E_DEV_MODE` environment variable is set to `true`.

Organizing your tests

The execution order for tests in testify Suite is IMPLEMENTATION SPECIFIC UNLIKE REGULAR GO TESTS. Use subtests for ordered tests and environments update.

Having a single environment

In the simple case, there is a single environment and each test checks one specific thing. 

import (
	"testing"

	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/e2e"
	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments"
)

type singleEnvSuite struct {
	e2e.BaseSuite[environments.VM]
}

func TestSingleEnvSuite(t *testing.T) {
	e2e.Run(t, &singleEnvSuite{}, e2e.WithProvisioner(awshost.Provisioner()))
}

func (suite *singleEnvSuite) Test1() {
	// Check feature 1
}

func (suite *singleEnvSuite) Test2() {
	// Check feature 2
}

func (suite *singleEnvSuite) Test3() {
	// Check feature 3
}

Having different environments

You may sometime have different environments but several tests for each on them. You can use e2e.Suite.UpdateEnv to do that. Using `UpdateEnv` between groups of Subtests. Note that between `TestLogDebug` and `TestLogInfo`, the environment is reverted to the original one. 

import (
	"testing"

	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/e2e"
	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments"
	awsvm "github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments/aws/vm"
	"github.com/DataDog/test-infra-definitions/components/datadog/agentparams"
)

type subTestSuite struct {
	e2e.Suite[environments.VM]
}

func TestSubTestSuite(t *testing.T) {
	e2e.Run(t, &singleEnvSuite{}, e2e.WithProvisioner(awshost.Provisioner()))
}

func (suite *subTestSuite) TestLogDebug() {
	// First group of subsets
	suite.T().Run("MySubTest1", func(t *testing.T) {
		// Sub test 1
	})
	suite.T().Run("MySubTest2", func(t *testing.T) {
		// Sub test 2
	})

	v.UpdateEnv(awshost.Provisioner(awshost.WithAgentOptions(agentparams.WithAgentConfig("log_level: debug"))))

	// Second group of subsets
	suite.T().Run("MySubTest3", func(t *testing.T) {
		// Sub test 3
	})
}

func (suite *subTestSuite) TestLogInfo() {
	// First group of subsets
	suite.T().Run("MySubTest1", func(t *testing.T) {
		// Sub test 1
	})
	suite.T().Run("MySubTest2", func(t *testing.T) {
		// Sub test 2
	})

	v.UpdateEnv(awshost.Provisioner(awshost.WithAgentOptions(agentparams.WithAgentConfig("log_level: info"))))

	// Second group of subsets
	suite.T().Run("MySubTest3", func(t *testing.T) {
		// Sub test 3
	})
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateRootOutputDir added in v0.60.0 

func CreateRootOutputDir() (string, error)

CreateRootOutputDir creates and returns a directory for tests to store output files and artifacts. A timestamp is included in the path to distinguish between multiple runs, and os.MkdirTemp() is used to avoid name collisions between parallel runs.

A new directory is created on each call to this function, it is recommended to save this result and use it for all tests in a run. For example see BaseSuite.GetRootOutputDir().

See runner.GetProfile().GetOutputDir() for the root output directory selection logic.

See CreateTestOutputDir and BaseSuite.CreateTestOutputDir for a function that returns a subdirectory for a specific test.

func CreateTestOutputDir added in v0.60.0 

func CreateTestOutputDir(root string, t *testing.T) (string, error)

CreateTestOutputDir creates a directory for a specific test that can be used to store output files and artifacts. The test name is used in the directory name, and invalid characters are replaced with underscores.

Example:

  • test name: TestInstallSuite/TestInstall/install_version=7.50.0
  • output directory: <root>/TestInstallSuite/TestInstall/install_version_7_50_0

func Run 

func Run[Env any, T Suite[Env]](t *testing.T, s T, options ...SuiteOption)

Run is a helper function to run a test suite. Unfortunately, we cannot use `s Suite[Env]` as Go is not able to match it with a struct However it's able to verify the same constraint on T

Types

type BaseSuite 

type BaseSuite[Env any] struct {
	suite.Suite
	// contains filtered or unexported fields
}

BaseSuite is a generic test suite that wraps testify.Suite

func (*BaseSuite[Env]) AfterTest 

func (bs *BaseSuite[Env]) AfterTest(suiteName, testName string)

AfterTest is executed right after the test finishes and receives the suite and test names as input. This function is called by testify Suite.

If you override AfterTest in your custom test suite type, the function must call [test.BaseSuite.AfterTest].

func (*BaseSuite[Env]) BeforeTest 

func (bs *BaseSuite[Env]) BeforeTest(string, string)

BeforeTest is executed right before the test starts and receives the suite and test names as input. This function is called by testify Suite.

If you override BeforeTest in your custom test suite type, the function must call [test.BaseSuite.BeforeTest].

func (*BaseSuite[Env]) CreateTestOutputDir added in v0.60.0 

func (bs *BaseSuite[Env]) CreateTestOutputDir() (string, error)

CreateTestOutputDir returns an output directory for the current test.

See also CreateTestOutputDir()

func (*BaseSuite[Env]) DatadogClient added in v0.60.0 

func (bs *BaseSuite[Env]) DatadogClient() *datadog.Client

DatadogClient returns a Datadog client that can be used to send telemtry info to dddev during e2e tests

func (*BaseSuite[Env]) EndTime added in v0.60.0 

func (bs *BaseSuite[Env]) EndTime() time.Time

EndTime returns the time when test suite ended

func (*BaseSuite[Env]) Env 

func (bs *BaseSuite[Env]) Env() *Env

Env returns the current environment

func (*BaseSuite[Env]) GetRootOutputDir added in v0.60.0 

func (bs *BaseSuite[Env]) GetRootOutputDir() (string, error)

GetRootOutputDir returns the root output directory for tests to store output files and artifacts. The directory is created on the first call to this function and reused in future calls.

See BaseSuite.CreateTestOutputDir() for a function that returns a directory for the current test.

See CreateRootOutputDir() for details on the root directory creation.

func (*BaseSuite[Env]) IsDevMode 

func (bs *BaseSuite[Env]) IsDevMode() bool

IsDevMode returns true if the test suite is running in dev mode. WARNING: IsDevMode should not be used. It's a recipe to get tests working locally but failing in CI.

func (*BaseSuite[Env]) SetupSuite 

func (bs *BaseSuite[Env]) SetupSuite()

SetupSuite run before all the tests in the suite have been run. This function is called by testify Suite.

If you override SetupSuite in your custom test suite type, the function must call e2e.BaseSuite.SetupSuite.

func (*BaseSuite[Env]) StartTime added in v0.60.0 

func (bs *BaseSuite[Env]) StartTime() time.Time

StartTime returns the time when test suite started

func (*BaseSuite[Env]) TearDownSuite 

func (bs *BaseSuite[Env]) TearDownSuite()

TearDownSuite run after all the tests in the suite have been run. This function is called by testify Suite.

If you override TearDownSuite in your custom test suite type, the function must call e2e.BaseSuite.TearDownSuite.

func (*BaseSuite[Env]) UpdateEnv 

func (bs *BaseSuite[Env]) UpdateEnv(newProvisioners ...provisioners.Provisioner)

UpdateEnv updates the environment with new provisioners.

type Suite 

type Suite[Env any] interface {
	suite.TestingSuite

	UpdateEnv(...provisioners.Provisioner)
	Env() *Env
	// contains filtered or unexported methods
}

Suite is a generic inteface used internally, only implemented by BaseSuite

type SuiteOption 

type SuiteOption = func(*suiteParams)

SuiteOption is an optional function parameter type for e2e options

func WithDevMode 

func WithDevMode() SuiteOption

WithDevMode enables dev mode. Dev mode doesn't destroy the environment when the test is finished which can be useful when writing a new E2E test.

func WithProvisioner 

func WithProvisioner(provisioner provisioners.Provisioner) SuiteOption

WithProvisioner adds a provisioner to the suite

func WithPulumiProvisioner 

func WithPulumiProvisioner[Env any](runFunc provisioners.PulumiEnvRunFunc[Env], configMap runner.ConfigMap) SuiteOption

WithPulumiProvisioner adds a typed Pulumi provisioner to the suite

func WithSkipDeleteOnFailure 

func WithSkipDeleteOnFailure() SuiteOption

WithSkipDeleteOnFailure doesn't destroy the environment when a test fails.

func WithStackName 

func WithStackName(stackName string) SuiteOption

WithStackName overrides the default stack name. This function is useful only when using Run.

func WithUntypedPulumiProvisioner 

func WithUntypedPulumiProvisioner(runFunc pulumi.RunFunc, configMap runner.ConfigMap) SuiteOption

WithUntypedPulumiProvisioner adds an untyped Pulumi provisioner to the suite

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.