replay

Documentation

Overview

Replay tests allow to quickly add regression tests that exercise one or a small number of gRPC methods.

How this works: once a problem reproduces, run Pulumi with PULUMI_DEBUG_GRPC="$PWD/logs.json" flag, find the offending method record under "logs.json" and turn it into a test using Replay from this package.

Note: this package used to be exposed under pulumi/pulumi-terraform-bridge/testing Go module but is moving here as it is not specific to bridged providers.

Index

• func AssertJSONMatchesPattern(t *testing.T, expectedPattern json.RawMessage, actual json.RawMessage)
• func Replay(t *testing.T, server pulumirpc.ResourceProviderServer, jsonLog string)
• func ReplayFile(t *testing.T, server pulumirpc.ResourceProviderServer, traceFile string)
• func ReplaySequence(t *testing.T, server pulumirpc.ResourceProviderServer, jsonLog string)

Functions

func AssertJSONMatchesPattern

func AssertJSONMatchesPattern(
	t *testing.T,
	expectedPattern json.RawMessage,
	actual json.RawMessage,
)

Assert that a given JSON document structurally matches a pattern.

The pattern language supports the following constructs:

"*" matches anything.

{"\\": x} matches only JSON documents strictly equal to x. This pattern essentially escapes the sub-tree, for example use {"\\": "*"} to match only the literal string "*".

func Replay

func Replay(t *testing.T, server pulumirpc.ResourceProviderServer, jsonLog string)

Replay executes a request from a provider operation log against an in-memory resource provider server and asserts that the server's response matches the logged response.

The jsonLog parameter is a verbatim JSON string such as this one:

{
  "method": "/pulumirpc.ResourceProvider/Create",
  "request": {
    "urn": "urn:pulumi:dev::repro-pulumi-random::random:index/randomString:RandomString::s",
    "properties": {
      "length": 1
    }
  },
  "response": {
    "id": "*",
    "properties": {
      "__meta": "{\"schema_version\":\"2\"}",
      "id": "*",
      "result": "*",
      "length": 1,
      "lower": true,
      "minLower": 0,
      "minNumeric": 0,
      "minSpecial": 0,
      "minUpper": 0,
      "number": true,
      "numeric": true,
      "special": true,
      "upper": true
    }
  }
}

The format is the JSON encoding of the gRPC protocol used by Pulumi ResourceProvider service.

https://github.com/pulumi/pulumi/blob/master/proto/pulumi/provider.proto#L27

Conveniently, the format matches what Pulumi CLI emits when invoked with PULUMI_DEBUG_GPRC:

PULUMI_DEBUG_GPRC=$PWD/log.json pulumi up

This allows quickly turning fragments of the program execution trace into test cases.

Instead of direct JSON equality, Replay uses AssertJSONMatchesPattern to compare the actual and expected responses. This allows patterns such as "*". In the above example, the random provider will generate new strings with every invocation and they would fail a strict equality check. Using "*" allows the test to succeed while ignoring the randomness.

Beware possible side-effects: although Replay executes in-memory without actual gRPC sockets, replaying against an actual resource provider will side-effect. For example, replaying Create calls against pulumi-aws provider may try to create resorces in AWS. This is not an issue with side-effect-free providers such as pulumi-random, or for methods that do not involve cloud interaction such as Diff.

Replay does not assume that the provider is a bridged provider and can be generally useful.

func ReplayFile

func ReplayFile(t *testing.T, server pulumirpc.ResourceProviderServer, traceFile string)

ReplayFile executes ReplaySequence on all pulumirpc.ResourceProvider events found in the file produced with PULUMI_DEBUG_GPRC. For example:

PULUMI_DEBUG_GPRC=testdata/log.json pulumi up

This produces the testdata/log.json file, which can then be used for Replay-style testing:

ReplayFile(t, server, "testdata/log.json")

func ReplaySequence

func ReplaySequence(t *testing.T, server pulumirpc.ResourceProviderServer, jsonLog string)

ReplaySequence is exactly like Replay, but expects jsonLog to encode a sequence of events `[e1, e2, e3]`, and will call Replay on each of those events in the given order.