providertest

package module
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 3, 2024 License: Apache-2.0 Imports: 33 Imported by: 0

README

Provider Testing

Incubating facilities for testing Pulumi providers

NOTE: The libraries in this repo are used internally by the Pulumi providers team, but are still evolving; you should expect incomplete documentation and breaking changes.

Test Modes

End To End (e2e)

Purpose: Prove provider behaviour for a resource is correct.

This test mode does not use the SDKs, but executes the YAML program directly. There are two types of e2e test: quick and full:

  1. Quick will not provision real resources with the cloud provider.
  2. Full will test the whole lifecycle of resources.
SDK

Purpose: Ensure parity of each supported language's SDK behaviour.

SDK test are therefore split out per-language. Internally, this uses pulumi convert to automatically create the language-specific programs before executing them.

VerifyUpgrade

Purpose: Verifies that upgrading the provider does not generate any unexpected replacements.

What these tests specifically try to verify is that the provider binary release candidate under test will not generate any surprises for users attempting to upgrade to it from the baseline released version.

There are currently several UpgradeTestMode variations tests can run under, with different speed/accuracy trade-offs.

Example Usage

func TestSimple(t *testing.T) {
  test := NewProviderTest("test/simple", // Point to a directory containing a Pulumi YAML program
    WithProvider(StartLocalProvider), // Provider can be started and attached in-process
    WithUpdateStep(UpdateStepDir("../simple-2"))) // Multi-step tests are supported
  test.Run(t)
}

When calling .Run(), a suite of nested tests are run:

  • TestSimple/e2e
  • TestSimple/sdk-csharp
  • TestSimple/sdk-go
  • TestSimple/sdk-python
  • TestSimple/sdk-typescript

If you want to run just one of these test modes directly locally, then you can temporarily replace .Run(t) with:

test.RunE2e(t, true /*runFullTest*/)
test.RunSdk(t, "nodejs" /*language*/)
Upgrade Tests

Set these extra options to enable upgrade tests:

func TestSimple(t *testing.T) {
  test := NewProviderTest("test/simple",
    WithProviderName("gcp"),
    WithBaselineVersion("6.67.0"),
    WithResourceProviderServer(...))
  test.Run(t)
}

These nested tests are added:

  • TestSimple/upgrade-snapshot
  • TestSimple/upgrade-preview-only
  • TestSimple/upgrade-quick
  • TestSimple/upgrade-full

Note that upgrade-snapshot is a utility job rather than a test. go test --provider-snapshot runs this job to exercise the baseline version of the provider and record its behavior under testdata. The resulting recorded snapshot files are currently expected to be checked into the repo. They are used to inform upgrade-quick and upgrade-preview-only tests. When updating the baseline version, snapshots need to be recorded anew on the new version.

Fixing failing tests

  • If the tests fail by flagging unwanted resource updates or replacements that are actually acceptable, configure a custom DiffValidation setting with more relaxed asserts.

  • If the tests flag legitimate upgrade issues, fixes are necessarily specific to the provider and resource being tested.

  • Remember to re-record the test snapshots when making changes to the example program or the baseline provider dependency.

Controlling Test Mode

Which subtests are run, and in which mode (quick/full), are controlled by custom go test CLI flags. These can be set in makefiles or CI scripts as required.

To run all sub-tests:

go test -provider-e2e -provider-sdk-all ./...

By default, if no modes are explicitly set, only the fast end-to-end (e2e) sub-test is executed.

Environment Variables

As a temporary control method, test mode can also be enabled via the environment variable PULUMI_PROVIDER_TEST_MODE. Multiple options can be specified separated by commas:

PULUMI_PROVIDER_TEST_MODE=e2e,sdk-python

[!NOTE] The environment variables should not be used in make files or CI configuration. Prefer using CLI flags for this.

Reference
Option CLI flag Environment Description
Skip E2E -provider-skip-e2e skip-e2e Skip e2e provider tests
Full E2E -provider-e2e e2e Enable full e2e provider tests, otherwise uses quick mode by default.
All SDK -provider-sdk-all sdk-all Enable all SDK tests
C# SDK -provider-sdk-csharp sdk-csharp Enable C# SDK tests
Python SDK -provider-sdk-python sdk-python Enable Python SDK tests
Go SDK -provider-sdk-go sdk-go Enable Go SDK tests
Typescript SDK -provider-sdk-typescript sdk-typescript Enable TypeScript SDK tests
Snapshot -provider-snapshot snapshot Create snapshots for use with quick e2e tests

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetProviderAttachEnv 

func GetProviderAttachEnv(runningProviders []*ProviderAttach) string

func GetUpgradeCacheDir added in v0.1.1 

func GetUpgradeCacheDir(programName, baselineVersion string, cacheDirTemplatePath ...string) string

GetUpgradeCacheDir returns the cache directory for a provider upgrade test. If no cacheDirTemplatePath is provided, the default cache directory is used.

func PreviewProviderUpgrade added in v0.0.6 

func PreviewProviderUpgrade(t pulumitest.PT, pulumiTest *pulumitest.PulumiTest, providerName string, baselineVersion string, opts ...optproviderupgrade.PreviewProviderUpgradeOpt) auto.PreviewResult

PreviewProviderUpgrade captures the state of a stack from a baseline provider configuration, then previews the stack with the current provider configuration. Uses a default cache directory of "testdata/recorded/TestProviderUpgrade/{programName}/{baselineVersion}".

func RandomString 

func RandomString() string

func ReportUpgradeCoverage 

func ReportUpgradeCoverage(t *testing.T)

This is a temporary helper method to assess upgrade resource coverage until better methods for tracking coverage are built. Run with -test.v to see the data logged. This finds all recorded GRPC states and traverses them to find the union of all resources used. It does not take into account if the corresponding tests are skipped or passing.

Types

type ConvertedProgram 

type ConvertedProgram struct {
	Language    string
	Dir         string
	UpdateSteps []UpdateStep
}

func (*ConvertedProgram) NewStack 

func (convertedProgram *ConvertedProgram) NewStack(t *testing.T, ctx context.Context) *auto.Stack

func (*ConvertedProgram) RestorePackages 

func (convertedProgram *ConvertedProgram) RestorePackages(t *testing.T, ctx context.Context)

type Diff added in v0.0.3 

type Diff struct {
	URN        resource.URN
	HasChanges bool

	// Non-empty Replaces indicates that the plan is a resource replacement and not a simple
	// in-place update.
	Replaces []string

	Diffs               []string
	DeleteBeforeReplace bool

	// May only be populated if there's a change.
	Olds map[string]any
	// May only be populated if there's a change.
	News map[string]any
}

The structure is mapped directly from Pulumi gRPC DiffResponse structure in the provider protocol and is currently unstable / subject to change.

https://github.com/pulumi/pulumi/blob/master/proto/pulumi/provider.proto#L225

Need to verify if this is representative of the actual Pulumi plans, since it only considers the decisions made by the provider, not the engine. For example, unclear if replaceOnChanges option https://www.pulumi.com/docs/concepts/options/replaceonchanges/ would surface here.

Even with the above caveats, it is reasonable to rely on this for the purposes of testing the provider itself.

type DiffValidation added in v0.0.3 

type DiffValidation = func(*testing.T, Diffs)

func NoChanges added in v0.0.3 

func NoChanges() DiffValidation

func NoReplacements added in v0.0.3 

func NoReplacements() DiffValidation

type Diffs added in v0.0.3 

type Diffs []Diff

type E2eOption 

type E2eOption func(*integration.ProgramTestOptions)

type Option 

type Option func(*ProviderTest)

func WithBaselineVersion 

func WithBaselineVersion(v string) Option

func WithConfig 

func WithConfig(key, value string) Option

func WithDiffValidation added in v0.0.3 

func WithDiffValidation(valid DiffValidation) Option

func WithE2eOptions 

func WithE2eOptions(opts ...E2eOption) Option

func WithExtraBaselineDependencies added in v0.0.2 

func WithExtraBaselineDependencies(deps map[string]string) Option

When testing upgrades, this option specifies additional baseline dependency versions. For example, when testing pulumi-eks, WithBaselineVersion("1.0.4") will define the baseline version of eks provider itself, where WithExtraBaselineDependencies(map[string]string{"aws": "5.42.0"}) will pin the aws dependency.

func WithProvider 

func WithProvider(start StartProvider) Option

WithProvider adds a provider to be started and attached for the test run.

func WithProviderName 

func WithProviderName(name string) Option

func WithResourceProviderServer 

func WithResourceProviderServer(s pulumirpc.ResourceProviderServer) Option

TODO[pulumi/providertest#9] make this redundant.

func WithSkipSdk 

func WithSkipSdk(language string, reasonArgs ...any) Option

func WithSkippedUpgradeTestMode 

func WithSkippedUpgradeTestMode(m UpgradeTestMode, reason string) Option

func WithUpdateStep 

func WithUpdateStep(opts ...UpdateStepOption) Option

WithUpdateStep adds a step to the test will be applied before then executing an update.

type ProviderAttach 

type ProviderAttach struct {
	// Name of the provider e.g. "aws"
	Name string
	// Port the provider is listening on
	Port int
}

func StartProviders 

func StartProviders(ctx context.Context, providerStartups ...StartProvider) ([]*ProviderAttach, error)

type ProviderTest 

type ProviderTest struct {
	// contains filtered or unexported fields
}

func NewProviderTest 

func NewProviderTest(dir string, opts ...Option) *ProviderTest

NewProviderTest creates a new provider test with the initial directory to be tested.

func (*ProviderTest) Configure 

func (pt *ProviderTest) Configure(opts ...Option) *ProviderTest

func (*ProviderTest) ConvertProgram 

func (pt *ProviderTest) ConvertProgram(t *testing.T, language string) (*ConvertedProgram, error)

func (*ProviderTest) GetConfig 

func (pt *ProviderTest) GetConfig() auto.ConfigMap

func (*ProviderTest) Run 

func (pt *ProviderTest) Run(t *testing.T)

Run starts executing the configured tests

func (*ProviderTest) RunE2e 

func (pt *ProviderTest) RunE2e(t *testing.T, runFullTest bool, options ...E2eOption)

func (*ProviderTest) RunSdk 

func (pt *ProviderTest) RunSdk(t *testing.T, language string)

func (*ProviderTest) VerifyUpgrade 

func (pt *ProviderTest) VerifyUpgrade(t *testing.T, mode UpgradeTestMode)

Verifies that upgrading the provider does not generate any unexpected replacements.

Specifically check that for a given Pulumi program located in dir, users can run pulumi up on a baseline provider version, then upgrade the provider to the new version under test, run pulumi up again and observe an empty diff.

func (*ProviderTest) VerifyUpgradeSnapshot 

func (pt *ProviderTest) VerifyUpgradeSnapshot(t *testing.T)

type StartProvider 

type StartProvider func(ctx context.Context) (*ProviderAttach, error)

StartProvider is a function that starts a provider and returns the name and port it is listening on. When the test is complete, the context will be cancelled and the provider should exit.

type UpdateStep 

type UpdateStep struct {
	// contains filtered or unexported fields
}

type UpdateStepOption 

type UpdateStepOption func(*UpdateStep)

func UpdateStepClean 

func UpdateStepClean() UpdateStepOption

UpdateStepClean will remove files from the directory which were removed in this step compared to the previous step's directory.

func UpdateStepDir 

func UpdateStepDir(dir string) UpdateStepOption

UpdateStepDir fetches files from the dir before performing the update. If dir is a relative path, it will be resolved relative to the original test directory.

type UpgradeTestMode 

type UpgradeTestMode int

Enumerates various available modes to test provider upgrades. The modes differ in speed vs precision tradeoffs. 

const (
	// The least precise but fastest mode. Tests are performed in-memory using pre-recorded
	// snapshots of baseline provider behavior. No cloud credentials are required, no
	// subprocesses are launched, fully debuggable.
	UpgradeTestMode_Quick UpgradeTestMode = iota

	// The medium precision/speed mode. Imports Pulumi statefile recorded on a baseline version,
	// and performs pulumi preview, asserting that the preview results in an empty diff. Cloud
	// credentials are required, but no infra is actually provisioned, making it quicker to
	// verify slow-to-provision resources such as databases.
	UpgradeTestMode_PreviewOnly

	// Full fidelity, slow mode. No pre-recorded snapshots are required. Do a complete pulumi up
	// on the baseline version, followed by a complete pulumi up on the version under test, and
	// assert that there are no observable updates or replacements.
	UpgradeTestMode_Full
)

func UpgradeTestModes 

func UpgradeTestModes() []UpgradeTestMode

func (UpgradeTestMode) String 

func (m UpgradeTestMode) String() string

Source Files

View all Source files

Directories

Path Synopsis
assertpreview
assertrefresh
assertup
changesummary
optnewstack
optrun
opttest
sanitize
Replay tests allow to quickly add regression tests that exercise one or a small number of gRPC methods.
 Replay tests allow to quickly add regression tests that exercise one or a small number of gRPC methods.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.