runner

README

This package is an end-to-end test runner for kpt fn commands. It can also be used to test kpt KRM functions.

How Does It Work

After importing this package, you can use runner.ScanTestCases to recursively scan the test cases in a given directory. For each test case, call runner.NewRunner to create a test runner and then call Run to run the test. For example please see e2e/fn_test.go

All test cases should be independent and should be able to run in fully parallel. They should be self-contained so the runner doesn't need any manual configuration.

The configurations for a test case is located in the .expected directory in the test case directory. The runner will implicitly use following files in .expected:

• config.yaml: This is the file which contains the configurations for the test case. It can have following fields:
  • testType: eval or render. It controls which kpt fn command will be used to run the test. Default: render.
  • exitCode: The expected exit code for the command. Default: 0.
  • skip: Runner will skip the test if skip is set to true. Default: false.
  • sequential: This test case should be run sequentially. Default: false.
  • runtimes: If the current runtime doesn't match any of the desired runtimes here, the test case will be skipped. Valid values are docker and podman. If unspecified, it will match any runtime.
  • imagePullPolicy: The image pull policy to be used. It can be set to one of Always, IfNotPresent and Never. Default value is inherited from the CLI flag.
  • notIdempotent: The functions and commands should be idempotent, but in some cases it's not doable. The runner will not run the command twice to check idempotent if this is set to true. Default: false.
  • debug: Debug means will the debug behavior be enabled. Default: false. Debug behaviors:
    • Keep the temporary directory used to run the test cases after test.
  • stdOut: The expected standard output from running the command. Default: "".
  • stdErr: The expected standard error from running the command. Default: "".
  • StdErrRegEx: A regular expression that is expected to match the standard error. Default: "".
  • disableOutputTruncate: Control should error output be truncated. Default: false.
  • Configurations only apply to eval tests:
    • execPath: A path to the executable file that will be run as function. Mutually exclusive with Image. The path should be separated by slash '/'
    • image: The image name for the function. Either image or execPath must be provided.
    • args: A map contains the arguments that will be passed into function. Args will be passed as 'key=value' format after the '--' in command.
    • network: Control the accessibility of network from the function container. Default: false.
    • includeMetaResources: Enable including meta resources, like Kptfile, in the function input. Default: false.
    • fnConfig: The path to the function config file. The path should be separated by slash '/'.
• diff.patch: The expected git diff output after running the command. Default: "".
• results.yaml: The expected result file after running the command. Default: "".
• setup.sh: A bash script which will be run before the command if it exists.
• exec.sh: A bash script which will be run if it exists and will replace the command (kpt fn eval or kpt fn render) that will be run according to testType in configurations. All configurations that used to control command behavior, like disableOutputTruncate and args, will be ignored.
• teardown.sh: A bash script which will be run after the command and result comparison if it exists.

Expected Output

The runner will compare the actual values of:

• stdout
• stderr
• Exit Code
• Diff
• Results file

Documentation

Index

• Constants
• type EvalTestCaseConfig
• type Runner
  • func NewRunner(t *testing.T, testCase TestCase, c string) (*Runner, error)
  • func (r *Runner) IsFnResultExpected() bool
  • func (r *Runner) IsOutOfPlace() bool
  • func (r *Runner) Run() error
  • func (r *Runner) Skip() bool
• type TestCase
• type TestCaseConfig
  • func (c *TestCaseConfig) RunCount() int
• type TestCases
  • func ScanTestCases(path string) (*TestCases, error)

Constants

const (
	CommandFnEval   string = "eval"
	CommandFnRender string = "render"
)

Types

type EvalTestCaseConfig

type EvalTestCaseConfig struct {
	// ExecPath is a path to the executable file that will be run as function
	// Mutually exclusive with Image.
	// The path should be separated by slash '/'
	ExecPath string `json:"execPath,omitempty" yaml:"execPath,omitempty"`

	// Image is the image name for the function
	Image string `json:"image,omitempty" yaml:"image,omitempty"`
	// Args are the arguments that will be passed into function.
	// Args will be passed as 'key=value' format after the '--' in command.
	Args map[string]string `json:"args,omitempty" yaml:"args,omitempty"`
	// Network indicates is network accessible from the function container. Default: false
	Network bool `json:"network,omitempty" yaml:"network,omitempty"`
	// IncludeMetaResources enables including meta resources, like Kptfile,
	// in the function input. Default: false
	IncludeMetaResources bool `json:"includeMetaResources,omitempty" yaml:"includeMetaResources,omitempty"`
	// FnConfig is the path to the function config file.
	// The path should be separated by slash '/'
	FnConfig string `json:"fnConfig,omitempty" yaml:"fnConfig,omitempty"`
	// contains filtered or unexported fields
}

EvalTestCaseConfig contains the config only for imperative function run

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner runs an e2e test

func NewRunner

func NewRunner(t *testing.T, testCase TestCase, c string) (*Runner, error)

NewRunner returns a new runner for pkg

func (*Runner) IsFnResultExpected

func (r *Runner) IsFnResultExpected() bool

IsFnResultExpected determines if function results are expected for this testcase.

func (*Runner) IsOutOfPlace

func (r *Runner) IsOutOfPlace() bool

IsOutOfPlace determines if command output is saved in a different directory (out-of-place).

func (*Runner) Run

func (r *Runner) Run() error

Run runs the test.

func (*Runner) Skip

func (r *Runner) Skip() bool

type TestCase

type TestCase struct {
	Path   string
	Config TestCaseConfig
}

TestCase contains the information needed to run a test. Each test case run by this driver is described by a `TestCase`.

type TestCaseConfig

type TestCaseConfig struct {
	// ExitCode is the expected exit code from the kpt commands. Default: 0
	ExitCode int `json:"exitCode,omitempty" yaml:"exitCode,omitempty"`

	// StdErr is the expected standard error output and should be checked
	// when a nonzero exit code is expected. Default: ""
	StdErr string `json:"stdErr,omitempty" yaml:"stdErr,omitempty"`
	// StdErrRegEx is the regular expression to match standard error output and should be checked
	// when a nonzero exit code is expected. Default: ""
	StdErrRegEx string `json:"stdErrRegEx,omitempty" yaml:"stdErrRegEx,omitempty"`

	// StdOut is the expected standard output from running the command.
	// Default: ""
	StdOut string `json:"stdOut,omitempty" yaml:"stdOut,omitempty"`

	// Sequential means should this test case be run sequentially. Default: false
	Sequential bool `json:"sequential,omitempty" yaml:"sequential,omitempty"`

	// ImagePullPolicy controls the image pulling behavior. It can be set to one
	// of `Always`, `IfNotPresent` and `Never`. If unspecified, the default will
	// be the same as the CLI flag.
	ImagePullPolicy string `json:"imagePullPolicy,omitempty" yaml:"imagePullPolicy,omitempty"`

	// Runtimes controls if a test case should be skipped. If the current runtime doesn't match
	// any of the desired runtimes here, the test case will be skipped. Valid values are `docker`
	// and `podman`. If unspecified, it will match any runtime.
	Runtimes []string `json:"runtimes,omitempty" yaml:"runtimes,omitempty"`

	// AllowExec determines if `fn render` needs to be invoked with `--allow-exec` flag
	AllowExec bool `json:"allowExec,omitempty" yaml:"allowExec,omitempty"`

	// AllowExec determines if `fn render` needs to be invoked with `--allow-network` flag
	AllowNetwork bool `json:"allowNetwork,omitempty" yaml:"allowNetwork,omitempty"`

	// AllowWasm determines if `fn render` needs to be invoked with `--allow-alpha-wasm` flag
	AllowWasm bool `json:"allowWasm,omitempty" yaml:"allowWasm,omitempty"`

	// Skip means should this test case be skipped. Default: false
	Skip bool `json:"skip,omitempty" yaml:"skip,omitempty"`

	// Debug means will the debug behavior be enabled. Default: false
	// Debug behavior:
	//  1. Keep the temporary directory used to run the test cases
	//    after test.
	Debug bool `json:"debug,omitempty" yaml:"debug,omitempty"`

	// TestType is the type of the test case. Possible value: ['render', 'eval']
	// Default: 'render'
	TestType string `json:"testType,omitempty" yaml:"testType,omitempty"`

	// DisableOutputTruncate indicates should error output be truncated
	DisableOutputTruncate bool `json:"disableOutputTruncate,omitempty" yaml:"disableOutputTruncate,omitempty"`

	// EvalConfig is the configs for eval tests
	EvalConfig *EvalTestCaseConfig `json:",inline" yaml:",inline"`

	// Environment variables to be set for the test case.
	Env map[string]string `json:"env,omitempty" yaml:"env,omitempty"`
}

TestCaseConfig contains the config information for the test case

func (*TestCaseConfig) RunCount

func (c *TestCaseConfig) RunCount() int

type TestCases

type TestCases []TestCase

TestCases contains a list of TestCase.

func ScanTestCases

func ScanTestCases(path string) (*TestCases, error)

ScanTestCases will recursively scan the directory `path` and return a list of TestConfig found