e2e

README

Cifuzz E2E tests

Cifuzz supports a growing list of behaviors. Multiplied by the number of supported operating systems, environment setups etc. it's a lot of combinations.

• Command
• Arguments and their combinations
• Operating system
• Environment and its setup. E.g., CIFUZZ_PRERELEASE=true, tooling flags or config files and their state.
• Tooling versions. E.g., Gradle versions we rely on. Or Jazzer version.
• State of SaaS Auth
• [...]

This end-to-end matrix test is not meant to replace our Integration tests! Our Integration tests cover complex scenarios in detail. The E2E matrix is a way for us to have a high-level, but wide test coverage for the CLI.

Note Tests are currently not blocking PRs or Releases.

They run after every release and on a nightly schedule.

• Nightly tests are using make install and code that's available in the repository.
• The release tests are run once, on Release publish, using the latest curl | sh installation method, to simulate the user installing it.

Sample

When I call cifuzz bundle --help Then the exit code is 0 And there is expected help output

func TestHelpArgsWithSubcommands(t *testing.T) {
	e2e.RunTest(t, &e2e.Test{
		Command: "bundle",
		Args:    []string{"--help"},
		Assert: func(t *testing.T, output e2e.CommandOutput) {
			assert.EqualValues(t, 0, output.ExitCode)
			assert.Contains(t, output.Stdout, "This command bundles all runtime artifacts")
		},
	})
}

See tests related to cifuzz help command in the /help folder for more examples.

Documentation

Index

• func RunTests(t *testing.T, testCases []TestCase)
• type Assertion
• type CIUser
• type CommandOutput
  • func (co *CommandOutput) ErrorContains(expected string) *CommandOutput
  • func (co *CommandOutput) Failed() *CommandOutput
  • func (co *CommandOutput) NoError() *CommandOutput
  • func (co *CommandOutput) NoOutput() *CommandOutput
  • func (co *CommandOutput) OutputContains(expected string) *CommandOutput
  • func (co *CommandOutput) OutputNotContains(expected string) *CommandOutput
  • func (co *CommandOutput) Success() *CommandOutput
• type TestCase

Functions

func RunTests

func RunTests(t *testing.T, testCases []TestCase)

RunTests Runs all test cases generated from the input combinations

Types

type Assertion

type Assertion func(*testing.T, CommandOutput)

type CIUser

type CIUser string

const (
	AnonymousCIUser    CIUser = "anonymous"
	LoggedInCIUser     CIUser = "loggedIn"
	InvalidTokenCIUser CIUser = "invalidToken"
)

type CommandOutput

type CommandOutput struct {
	ExitCode int
	Stdout   string
	Stderr   string
	Stdall   string // Combined stdout and stderr output for simpler assertions
	Workdir  fs.FS  // Expose files from the test folder
	// contains filtered or unexported fields
}

func (*CommandOutput) ErrorContains

func (co *CommandOutput) ErrorContains(expected string) *CommandOutput

func (*CommandOutput) Failed

func (co *CommandOutput) Failed() *CommandOutput

func (*CommandOutput) NoError

func (co *CommandOutput) NoError() *CommandOutput

func (*CommandOutput) NoOutput

func (co *CommandOutput) NoOutput() *CommandOutput

func (*CommandOutput) OutputContains

func (co *CommandOutput) OutputContains(expected string) *CommandOutput

func (*CommandOutput) OutputNotContains

func (co *CommandOutput) OutputNotContains(expected string) *CommandOutput

func (*CommandOutput) Success

func (co *CommandOutput) Success() *CommandOutput

type TestCase

type TestCase struct {
	Description  string
	Command      string
	Environment  []string
	Args         []string
	SampleFolder []string
	CIUser       CIUser
	// Os        []OSRuntime // When we will have tests depending on the OS
	// ToolsRequired []string # TODO: depending on the tools we will want to test
	Assert Assertion
}