e2e

package
v0.60.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2024 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package e2e provides the API to manage environments and organize E2E tests. Three major concepts are used to write E2E tests:

  • e2e.Provisioner: A provisioner is a component that provide compute resources (usually Cloud resources). Most common is Pulumi through `test-infra-definitions`.
  • e2e.BaseSuite: A TestSuite is a collection of tests that share the ~same environment.
  • Environment: An environment is a collection of resources (virtual machine, agent, etc). An environment is filled by provisioners.

See usage examples in the [examples] package.

Provisioners

Three provisioners are available:

Pulumi Provisioner can be typed or untyped:

  • Typed provisioners are provisioners that are typed with the environment they provision and the `Run` function must be defined in `datadog-agent` inline.
  • Untyped provisioners are provisioners that are not typed with the environment they provision and the `Run` function can come from anywhere.
  • e2e.StaticProvisioner: A provisioner that uses static resources from a JSON file. The static provisioner is Untyped.

# Impact of Typed vs Untyped provisioners Typed provisioners are more convenient to use as they are typed with the environment they provision, however they do require a close mapping between the RunFunc and the environment. With a Typed provisioner, the `component.Export()` function is used to match an Environment field with a Pulumi resource.

An Untyped provisioner is more flexible as it does not require a close mapping between the RunFunc and the environment. It allows to get resources from anywhere in the same environment. However it means that the environment needs to be annotated with the `import` tag to match the resource key. See for instance the examples/suite_serial_kube_test.go file.

Out-of-the-box environments and provisioners

Check the [environments] package for a list of out-of-the-box environments, for instance [environments.VM]. Check the `environments/<cloud>` for a list of out-of-the-box provisioners, for instance environments/aws/vm.

The BaseSuite test suite

The e2e.BaseSuite test suite is a testify Suite that wraps environment and provisioners. It allows to easily write tests that share the same environment without having to re-implement boilerplate code. Check all the e2e.SuiteOption to customize the behavior of the BaseSuite.

Note: By default, the BaseSuite test suite will delete the environment when the test suite finishes (whether it's successful or not). During development, it's highly recommended to use the [params.WithDevMode] option to prevent the environment from being deleted. [params.WithDevMode] is automatically enabled when the `E2E_DEV_MODE` environment variable is set to `true`.

Organizing your tests

The execution order for tests in testify Suite is IMPLEMENTATION SPECIFIC UNLIKE REGULAR GO TESTS. Use subtests for ordered tests and environments update.

Having a single environment

In the simple case, there is a single environment and each test checks one specific thing. 

import (
	"testing"

	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/e2e"
	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments"
)

type singleEnvSuite struct {
	e2e.BaseSuite[environments.VM]
}

func TestSingleEnvSuite(t *testing.T) {
	e2e.Run(t, &singleEnvSuite{}, e2e.WithProvisioner(awshost.Provisioner()))
}

func (suite *singleEnvSuite) Test1() {
	// Check feature 1
}

func (suite *singleEnvSuite) Test2() {
	// Check feature 2
}

func (suite *singleEnvSuite) Test3() {
	// Check feature 3
}

Having different environments

You may sometime have different environments but several tests for each on them. You can use e2e.Suite.UpdateEnv to do that. Using `UpdateEnv` between groups of Subtests. Note that between `TestLogDebug` and `TestLogInfo`, the environment is reverted to the original one. 

import (
	"testing"

	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/e2e"
	"github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments"
	awsvm "github.com/DataDog/datadog-agent/test/new-e2e/pkg/environments/aws/vm"
	"github.com/DataDog/test-infra-definitions/components/datadog/agentparams"
)

type subTestSuite struct {
	e2e.Suite[environments.VM]
}

func TestSubTestSuite(t *testing.T) {
	e2e.Run(t, &singleEnvSuite{}, e2e.WithProvisioner(awshost.Provisioner()))
}

func (suite *subTestSuite) TestLogDebug() {
	// First group of subsets
	suite.T().Run("MySubTest1", func(t *testing.T) {
		// Sub test 1
	})
	suite.T().Run("MySubTest2", func(t *testing.T) {
		// Sub test 2
	})

	v.UpdateEnv(awshost.Provisioner(awshost.WithAgentOptions(agentparams.WithAgentConfig("log_level: debug"))))

	// Second group of subsets
	suite.T().Run("MySubTest3", func(t *testing.T) {
		// Sub test 3
	})
}

func (suite *subTestSuite) TestLogInfo() {
	// First group of subsets
	suite.T().Run("MySubTest1", func(t *testing.T) {
		// Sub test 1
	})
	suite.T().Run("MySubTest2", func(t *testing.T) {
		// Sub test 2
	})

	v.UpdateEnv(awshost.Provisioner(awshost.WithAgentOptions(agentparams.WithAgentConfig("log_level: info"))))

	// Second group of subsets
	suite.T().Run("MySubTest3", func(t *testing.T) {
		// Sub test 3
	})
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateRootOutputDir 

func CreateRootOutputDir() (string, error)

CreateRootOutputDir creates and returns a directory for tests to store output files and artifacts. A timestamp is included in the path to distinguish between multiple runs, and os.MkdirTemp() is used to avoid name collisions between parallel runs.

A new directory is created on each call to this function, it is recommended to save this result and use it for all tests in a run. For example see BaseSuite.GetRootOutputDir().

See runner.GetProfile().GetOutputDir() for the root output directory selection logic.

See CreateTestOutputDir and BaseSuite.CreateTestOutputDir for a function that returns a subdirectory for a specific test.

func CreateTestOutputDir 

func CreateTestOutputDir(root string, t *testing.T) (string, error)

CreateTestOutputDir creates a directory for a specific test that can be used to store output files and artifacts. The test name is used in the directory name, and invalid characters are replaced with underscores.

Example:

  • test name: TestInstallSuite/TestInstall/install_version=7.50.0
  • output directory: <root>/TestInstallSuite/TestInstall/install_version_7_50_0

func Run 

func Run[Env any, T Suite[Env]](t *testing.T, s T, options ...SuiteOption)

Run is a helper function to run a test suite. Unfortunately, we cannot use `s Suite[Env]` as Go is not able to match it with a struct However it's able to verify the same constraint on T

Types

type BaseSuite 

type BaseSuite[Env any] struct {
	suite.Suite
	// contains filtered or unexported fields
}

BaseSuite is a generic test suite that wraps testify.Suite

func (*BaseSuite[Env]) AfterTest 

func (bs *BaseSuite[Env]) AfterTest(suiteName, testName string)

AfterTest is executed right after the test finishes and receives the suite and test names as input. This function is called by testify Suite.

If you override AfterTest in your custom test suite type, the function must call [test.BaseSuite.AfterTest].

func (*BaseSuite[Env]) BeforeTest 

func (bs *BaseSuite[Env]) BeforeTest(string, string)

BeforeTest is executed right before the test starts and receives the suite and test names as input. This function is called by testify Suite.

If you override BeforeTest in your custom test suite type, the function must call [test.BaseSuite.BeforeTest].

func (*BaseSuite[Env]) CreateTestOutputDir 

func (bs *BaseSuite[Env]) CreateTestOutputDir() (string, error)

CreateTestOutputDir returns an output directory for the current test.

See also CreateTestOutputDir()

func (*BaseSuite[Env]) DatadogClient 

func (bs *BaseSuite[Env]) DatadogClient() *datadog.Client

DatadogClient returns a Datadog client that can be used to send telemtry info to dddev during e2e tests

func (*BaseSuite[Env]) EndTime 

func (bs *BaseSuite[Env]) EndTime() time.Time

EndTime returns the time when test suite ended

func (*BaseSuite[Env]) Env 

func (bs *BaseSuite[Env]) Env() *Env

Env returns the current environment

func (*BaseSuite[Env]) GetRootOutputDir 

func (bs *BaseSuite[Env]) GetRootOutputDir() (string, error)

GetRootOutputDir returns the root output directory for tests to store output files and artifacts. The directory is created on the first call to this function and reused in future calls.

See BaseSuite.CreateTestOutputDir() for a function that returns a directory for the current test.

See CreateRootOutputDir() for details on the root directory creation.

func (*BaseSuite[Env]) IsDevMode 

func (bs *BaseSuite[Env]) IsDevMode() bool

IsDevMode returns true if the test suite is running in dev mode. WARNING: IsDevMode should not be used. It's a recipe to get tests working locally but failing in CI.

func (*BaseSuite[Env]) SetupSuite 

func (bs *BaseSuite[Env]) SetupSuite()

SetupSuite run before all the tests in the suite have been run. This function is called by testify Suite.

If you override SetupSuite in your custom test suite type, the function must call e2e.BaseSuite.SetupSuite.

func (*BaseSuite[Env]) StartTime 

func (bs *BaseSuite[Env]) StartTime() time.Time

StartTime returns the time when test suite started

func (*BaseSuite[Env]) TearDownSuite 

func (bs *BaseSuite[Env]) TearDownSuite()

TearDownSuite run after all the tests in the suite have been run. This function is called by testify Suite.

If you override TearDownSuite in your custom test suite type, the function must call e2e.BaseSuite.TearDownSuite.

func (*BaseSuite[Env]) UpdateEnv 

func (bs *BaseSuite[Env]) UpdateEnv(newProvisioners ...Provisioner)

UpdateEnv updates the environment with new provisioners.

type Context 

type Context interface {
	T() *testing.T
}

Context defines an interface that allows to get information about current test context

type Diagnosable added in v0.56.0 

type Diagnosable interface {
	Diagnose(ctx context.Context, stackName string) (string, error)
}

Diagnosable defines the interface for a diagnosable provider.

type FileProvisioner 

type FileProvisioner struct {
	// contains filtered or unexported fields
}

FileProvisioner is a provisioner that reads JSON files from a filesystem.

func NewFileProvisioner 

func NewFileProvisioner(id string, fs fs.FS) *FileProvisioner

NewFileProvisioner returns a new FileProvisioner.

func (*FileProvisioner) Destroy 

func (fp *FileProvisioner) Destroy(context.Context, string, io.Writer) error

Destroy is a no-op for the FileProvisioner.

func (*FileProvisioner) ID 

func (fp *FileProvisioner) ID() string

ID returns the ID of the provisioner.

func (*FileProvisioner) Provision 

func (fp *FileProvisioner) Provision(context.Context, string, io.Writer) (RawResources, error)

Provision reads JSON files from the filesystem and returns them as raw resources.

type Initializable 

type Initializable interface {
	Init(Context) error
}

Initializable defines the interface for an object that needs to be initialized

type Provisioner 

type Provisioner interface {
	ID() string
	Destroy(context.Context, string, io.Writer) error
}

Provisioner defines the interface for a provisioner.

type ProvisionerMap 

type ProvisionerMap map[string]Provisioner

ProvisionerMap is a map of provisioners.

type PulumiEnvRunFunc 

type PulumiEnvRunFunc[Env any] func(ctx *pulumi.Context, env *Env) error

PulumiEnvRunFunc is a function that runs a Pulumi program with a given environment.

type PulumiProvisioner 

type PulumiProvisioner[Env any] struct {
	// contains filtered or unexported fields
}

PulumiProvisioner is a provisioner based on Pulumi with binding to an environment.

func NewTypedPulumiProvisioner 

func NewTypedPulumiProvisioner[Env any](id string, runFunc PulumiEnvRunFunc[Env], configMap runner.ConfigMap) *PulumiProvisioner[Env]

NewTypedPulumiProvisioner returns a new PulumiProvisioner.

func NewUntypedPulumiProvisioner 

func NewUntypedPulumiProvisioner(id string, runFunc pulumi.RunFunc, configMap runner.ConfigMap) *PulumiProvisioner[any]

NewUntypedPulumiProvisioner returns a new PulumiProvisioner without env binding.

func (*PulumiProvisioner[Env]) Destroy 

func (pp *PulumiProvisioner[Env]) Destroy(ctx context.Context, stackName string, logger io.Writer) error

Destroy deletes the Pulumi stack.

func (*PulumiProvisioner[Env]) Diagnose added in v0.56.0 

func (pp *PulumiProvisioner[Env]) Diagnose(ctx context.Context, stackName string) (string, error)

Diagnose runs the diagnose function if it is set diagnoseFunc

func (*PulumiProvisioner[Env]) ID 

func (pp *PulumiProvisioner[Env]) ID() string

ID returns the ID of the provisioner.

func (*PulumiProvisioner[Env]) Provision 

func (pp *PulumiProvisioner[Env]) Provision(ctx context.Context, stackName string, logger io.Writer) (RawResources, error)

Provision runs the Pulumi program and returns the raw resources.

func (*PulumiProvisioner[Env]) ProvisionEnv 

func (pp *PulumiProvisioner[Env]) ProvisionEnv(ctx context.Context, stackName string, logger io.Writer, env *Env) (RawResources, error)

ProvisionEnv runs the Pulumi program with a given environment and returns the raw resources.

func (*PulumiProvisioner[Env]) SetDiagnoseFunc added in v0.56.0 

func (pp *PulumiProvisioner[Env]) SetDiagnoseFunc(diagnoseFunc func(ctx context.Context, stackName string) (string, error))

SetDiagnoseFunc sets the diagnose function.

type RawResources 

type RawResources map[string][]byte

RawResources is the common types returned by provisioners

func (RawResources) Merge 

func (rr RawResources) Merge(in RawResources)

Merge merges two RawResources maps

type Suite 

type Suite[Env any] interface {
	suite.TestingSuite

	UpdateEnv(...Provisioner)
	Env() *Env
	// contains filtered or unexported methods
}

Suite is a generic inteface used internally, only implemented by BaseSuite

type SuiteOption 

type SuiteOption = func(*suiteParams)

SuiteOption is an optional function parameter type for e2e options

func WithDevMode 

func WithDevMode() SuiteOption

WithDevMode enables dev mode. Dev mode doesn't destroy the environment when the test is finished which can be useful when writing a new E2E test.

func WithProvisioner 

func WithProvisioner(provisioner Provisioner) SuiteOption

WithProvisioner adds a provisioner to the suite

func WithPulumiProvisioner 

func WithPulumiProvisioner[Env any](runFunc PulumiEnvRunFunc[Env], configMap runner.ConfigMap) SuiteOption

WithPulumiProvisioner adds a typed Pulumi provisioner to the suite

func WithSkipDeleteOnFailure 

func WithSkipDeleteOnFailure() SuiteOption

WithSkipDeleteOnFailure doesn't destroy the environment when a test fails.

func WithStackName 

func WithStackName(stackName string) SuiteOption

WithStackName overrides the default stack name. This function is useful only when using Run.

func WithUntypedPulumiProvisioner 

func WithUntypedPulumiProvisioner(runFunc pulumi.RunFunc, configMap runner.ConfigMap) SuiteOption

WithUntypedPulumiProvisioner adds an untyped Pulumi provisioner to the suite

type TypedProvisioner 

type TypedProvisioner[Env any] interface {
	Provisioner
	ProvisionEnv(context.Context, string, io.Writer, *Env) (RawResources, error)
}

TypedProvisioner defines the interface for a provisioner with env binding

type UntypedProvisioner 

type UntypedProvisioner interface {
	Provisioner
	Provision(context.Context, string, io.Writer) (RawResources, error)
}

UntypedProvisioner defines the interface for a provisioner without env binding

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.