crosstests

README

cross-tests

This package provides cross-testing for Terraform Plugin SDKv2 based Terraform providers, bridged into Pulumi with the Pulumi Terraform bridge. For PF based cross-tests, see pf/.../cross-tests.

Cross tests compare and contrast the behavior of a given Terraform provider under Terraform CLI against the behavior of the same provider bridged to Pulumi and used under Pulumi CLI. Cross-tests work by generating a full in-memory Terraform provider and HCL program and running it with the Terraform CLI, then bridging the Terraform provider (also in memory) and generating an equivalent Pulumi YAML program, and comparing the result.

    +--------------------+                      +---------------------+
    | Terraform Provider |--------------------->| Operation(tfInputs) |
    +--------------------+                      +---------------------+
              |                                                        \
              |                                                         \
              |                                                          \
              |                                                      +---------------------+
              | shimv2.NewProvider                                   |  Witness Operation  |
              |                                                      +---------------------+
              |                                                          /
              |                                                         /
              V                                                        /
    +--------------------+                      +---------------------+
    |   Pulumi Provider  |--------------------->| Operation(puInputs) |
    +--------------------+                      +---------------------+

To make debugging easier, execution faster and avoid an extra build step, these tests start both Pulumi and Terraform providers in-process and have the CLIs attach to these in-process providers:

  graph LR
      PC[pulumi up]       -- Drives --> BP[BridgedProvider]
      TC[terraform apply] -- Drives --> TP[TFProvider]
      Test                -- Runs   --> PC
      Test                -- Runs   --> TC
      Test                -- Hosts  --> TP
      Test                -- Hosts  --> BP

The exact sequence of operations and asserts depends on the use case, for example cross-testing Diff convergence would exercise state transition by imitating a change in resource inputs and comparing generated plans.

Currently, SDKv2 cross-testing supports:

• Create
• Configure

Cross-tests can be written either by specify both the Terraform value and the Pulumi value or by specifying only the Terraform value and letting the framework infer an equivalent Pulumi value.

// A cross-test that infers the Pulumi value.
crosstests.Create(t,
	map[string]*schema.Schema{
		"f0": {	Optional: true, Type: schema.TypeBool },
	},
	cty.ObjectVal(map[string]cty.Value{
       	"f0": cty.BoolVal(true),
    }),
)

// A cross-test that manually specifies the Pulumi value.
crosstests.Create(t,
	map[string]*schema.Schema{
		"f0": {	Optional: true, Type: schema.TypeBool },
	},
	cty.ObjectVal(map[string]cty.Value{
       	"f0": cty.BoolVal(true),
    }),
	crosstests.CreatePulumiConfig(resource.PropertyMap{
       	"f0": resource.NewProperty(true),
    }),
)

Both traditional test cases and property-based test cases powered by rapid could be built on top of cross-test.

---

Initial cross tests were developed inside the cross-test folder against APIs internal to cross-tests. This was vital to development of cross-tests. Currently we have multiple provider tests written in the cross-tests folder. New tests should live next to the functionality that they are testing, and consume only the exported API. As time permits, existing tests written using cross-tests against internal APIs will be moved outside of the /cross-tests folder and written in terms of public cross-tests APIs.

Documentation

Overview

Adapters for converting morally equivalent typed representations of TF values for integrating with all the libraries cross-testing is using.

Compares the effect of transitioning between two randomly sampled resource configurations.

Define pretty-printers to make test output easier to interpret.

Driver code for running tests against an in-process bridged provider under Pulumi CLI.

Rapid generators for schema and value spaces.

Helper code to drive Terraform CLI to run tests against an in-process provider.

Helper code to emit Terraform HCL code to drive the Terraform CLI.

Index

• func Configure(t T, provider map[string]*schema.Schema, tfConfig cty.Value, ...)
• func ConvertResourceValue(t require.TestingT, properties resource.PropertyMap) map[string]any
• func Create(t T, resourceSchema map[string]*schema.Schema, tfConfig cty.Value, ...)
• func Diff(t T, resource *schema.Resource, config1, config2 map[string]cty.Value, ...) crosstestsimpl.DiffResult
• func FailNotEqual(t T, name string, tfVal, pulVal any)
• func MakeConfigure(provider map[string]*schema.Schema, tfConfig cty.Value, ...) func(*testing.T)
• func MakeCreate(resource map[string]*schema.Schema, tfConfig cty.Value, ...) func(t *testing.T)
• type ConfigureOption
  • func ConfigureProviderInfo(info map[string]*info.Schema) ConfigureOption
  • func ConfigurePulumiConfig(config resource.PropertyMap) ConfigureOption
• type CreateOption
  • func CreatePulumiConfig(config resource.PropertyMap) CreateOption
  • func CreateResourceInfo(info info.Resource) CreateOption
  • func CreateStateUpgrader(schemaVersion int, upgraders []schema.StateUpgrader) CreateOption
  • func CreateTimeout(timeouts *schema.ResourceTimeout) CreateOption
• type DiffOption
  • func DiffDeleteBeforeReplace(deleteBeforeReplace bool) DiffOption
• type T
• type TfResDriver

Functions

func Configure

func Configure(
	t T, provider map[string]*schema.Schema, tfConfig cty.Value,
	options ...ConfigureOption,
)

Configure validates that a Terraform provider witnesses the same input when: - invoked directly with HCL on tfConfig - bridged and invoked via Pulumi YAML on puConfig

Create only applies to resources defined with github.com/hashicorp/terraform-plugin-sdk/v2. For cross-tests on Plugin Framework based resources, see github.com/pulumi/pulumi-terraform-bridge/pkg/pf/tests/internal/cross-tests.

func ConvertResourceValue

func ConvertResourceValue(t require.TestingT, properties resource.PropertyMap) map[string]any

func Create

func Create(
	t T, resourceSchema map[string]*schema.Schema, tfConfig cty.Value,
	options ...CreateOption,
)

Create validates that a Terraform provider witnesses the same input when: - invoked directly with HCL on tfConfig - bridged and invoked via Pulumi YAML on puConfig

Create only applies to resources defined with github.com/hashicorp/terraform-plugin-sdk/v2. For cross-tests on Plugin Framework based resources, see github.com/pulumi/pulumi-terraform-bridge/pkg/pf/tests/internal/cross-tests.

Create *does not* verify the outputs of the resource, only that the provider witnessed the same inputs.

func Diff

func Diff(
	t T, resource *schema.Resource, config1, config2 map[string]cty.Value, opts ...DiffOption,
) crosstestsimpl.DiffResult

func FailNotEqual

func FailNotEqual(t T, name string, tfVal, pulVal any)

func MakeConfigure

func MakeConfigure(
	provider map[string]*schema.Schema, tfConfig cty.Value,
	options ...ConfigureOption,
) func(*testing.T)

func MakeCreate

func MakeCreate(
	resource map[string]*schema.Schema, tfConfig cty.Value,
	options ...CreateOption,
) func(t *testing.T)

MakeCreate is a helper function for calling Create in testing.T.Run subcases.

Types

type ConfigureOption

type ConfigureOption func(*configureOpts)

An option that can be used to customize Configure.

func ConfigureProviderInfo

func ConfigureProviderInfo(info map[string]*info.Schema) ConfigureOption

CreateResourceInfo specifies an info.Resource to apply to the resource under test.

func ConfigurePulumiConfig

func ConfigurePulumiConfig(config resource.PropertyMap) ConfigureOption

ConfigurePulumiConfig specifies an explicit pulumi value for the configure call.

type CreateOption

type CreateOption func(*createOpts)

An option that can be used to customize Create.

func CreatePulumiConfig

func CreatePulumiConfig(config resource.PropertyMap) CreateOption

CreatePulumiConfig specifies an explicit config value in Pulumi's value space.

func CreateResourceInfo

func CreateResourceInfo(info info.Resource) CreateOption

CreateResourceInfo specifies an info.Resource to apply to the resource under test.

func CreateStateUpgrader

func CreateStateUpgrader(schemaVersion int, upgraders []schema.StateUpgrader) CreateOption

CreateStateUpgrader specifies a schema version and list of state upgrader for Create.

func CreateTimeout

func CreateTimeout(timeouts *schema.ResourceTimeout) CreateOption

CreateTimeout specifies a timeout option for Create.

type DiffOption

type DiffOption func(*diffOpts)

An option that can be used to customize Diff.

func DiffDeleteBeforeReplace

func DiffDeleteBeforeReplace(deleteBeforeReplace bool) DiffOption

DiffDeleteBeforeReplace specifies whether to delete the resource before replacing it.

type T

type T = crosstestsimpl.T

type TfResDriver

type TfResDriver struct {
	// contains filtered or unexported fields
}