README ¶
E2E tests
The purpose of this docs is to describe KICS' E2E test suite
Getting Started
There are several ways to execute the E2E tests.
TLDR
This steps will build the kics and then run the E2E using the built binary (placed by default under ${PWD}/bin/kics
)
make test-e2e
Step by Step
These steps will build the kics and then run the E2E using the built binary.
go build -o ./bin/kics cmd/console/main.go
If you want to provide a version:
go build -o ./bin/kics -ldflags "-X github.com/Checkmarx/kics/internal/constants.Version=$(git rev-parse --short HEAD) cmd/console/main.go
and then:
E2E_KICS_BINARY=./bin/kics go test "github.com/Checkmarx/kics/e2e" -parallel 12 -v
Test Structure
Test case main structure
var tests = []struct {
name string // name of the test case
args args // args structure of the test
wantStatus int // expected output status code
removeFiles []string // path for removing files created during the test
validation Validation // custom validation function for the test case (usually regex)
}
Test case args structure
type cmdArgs []string
type args struct {
args []cmdArgs // args to pass to kics binary
expectedOut []string // path to file with expected output
expectedPayload []string // path to file with expected payload
expectedResult []string // path to file with expected result
expectedLog LogValidation // custom validation structure
}
type LogValidation struct {
logFile string // file name of the generated log
validationFunc Validation // custom validation function to verify the log content
}
Each test can use more or less keys within its structure, depending on the complexity of the test.
A single test can contain verification based on:
-
status code only (required for all tests)
-
status code + custom regex verification
-
status code + CLI output (file check)
-
status code + payload content (file check)
-
status code + result content (file check)
-
status code + log content (file check with validation function)
-
status code + result content (file check) + payload content (file check)
-
status code + result content (file check) + payload content (file check) + log content (file check with validation function)
-
status code + CLI output + payload content (file check) + result content (file check) + custom regex verification ....
Note: All Tests are running in parallel. To avoid errors when creating new E2E tests which contain flags that create files (payload, results, logs...) you should create this output file in a folder called "output" and the name of the generated file must contain the ID of the test case. This will guarantee that files generated by tests will not have the same name, avoiding read/write/remove errors when running several tests in parallel. The Example below (3 and 4) demonstrate E2E tests with file creation (payload/logs)
Test Scenarios
Example 1: A test case that executes a command and checks only the status code output
This type of test is simple and does not require the creation of files.
// E2E-CLI-020 - KICS scan with --exclude-queries flag
// should not run queries that was provided in this flag.
{
name: "E2E-CLI-020",
args: args{
args: []cmdArgs{
[]string{"scan", "--exclude-queries", "15ffbacc-fa42-4f6f-a57d-2feac7365caa", "-s",
"-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"},
},
},
wantStatus: []int{0},
}
Example 2: A test case that executes a command and checks:
- Status code output
- Custom validation function (regex)
The custom validation function must perform a validation based on the CLI output generated by the test. The function must return true if the validation is correct and false otherwise.
// E2E-CLI-006 - KICS generate-id should exhibit
// a valid UUID in the CLI and return exit code 0
{
name: "E2E-CLI-006",
args: args{
args: []cmdArgs{
[]string{"generate-id"},
},
},
wantStatus: 0,
// When the validation function is used,
// the expectedOut function can be suppressed
// The validator function allows the creation of customized functions to
// perform actions on the output generated by the test.
validation: func(outputText string) bool {
// Usually, a validation check for elements present in the output using regex
uuidRegex := "[a-f0-9]{8}-[a-f0-9]{4}-4{1}[a-f0-9]{3}-[89ab]{1}[a-f0-9]{3}-[a-f0-9]{12}"
match, _ := regexp.MatchString(uuidRegex, outputText)
// After the custom check, the function should return
// a boolean indicating if the test has passed or failed.
return match
},
},
}
Example 3: A test case that executes a command and checks:
- Status code output
- Kics CLI output
- Generated payload file content
The Tests that check CLI output, payload or results, need a comparison file for each output you want to compare (except for the status code).
The example below contains 2 files for comparing the outputs of CLI and the payload file: "E2E_CLI_005" and "E2E_CLI_005_PAYLOAD.json". Files used for comparing outputs must have the same name as the test and must be added to the "fixture" folder.
In addition, it is necessary to remove the files that will be generated during the test by adding them to removeFiles (required only for test cases that generates files from results/payloads/logs). Files created by Kics during testing should always be created in the 'outputs' folder
// E2E-CLI-005 - KICS scan with -- payload-path flag should create a file with the
// passed name containing the payload of the files scanned
{
name: "E2E-CLI-005",
args: args{
args: []cmdArgs{
[]string{"scan", "--silent", "-q", "../assets/queries", "-p", "fixtures/samples/terraform.tf",
"--payload-path", "output/E2E_CLI_005_PAYLOAD.json"},
},
expectedOut: []string{
"E2E_CLI_005",
},
expectedPayload: []string{
"E2E_CLI_005_PAYLOAD.json",
},
},
wantStatus: []int{50},
removeFiles: []string{"E2E_CLI_005_PAYLOAD.json"},
}
Example 4: A test case that executes a command and checks:
- Status code output
- Generated log file content
In contrast to what is expected for results and payload, log files don't have a file to compare, instead KICS validates logs like it validates CLI's output. In this case, the validation function will receive the log file data as its input and you can use any regex or do text validations in the log file content.
The example below generates an output file called: "E2E_CLI_038_LOG" and creates a validation function expecting 3 matches in the generated log file content.
In addition, it is necessary to remove files generated during the test, adding them to removeFiles
(required only for test cases that generates files from results/payloads/logs).
// E2E-CLI-038 - KICS scan command with --log-path
// should generate and save a log file for the scan
{
name: "E2E-CLI-038",
args: args{
args: []cmdArgs{
[]string{"scan", "--log-path", "output/E2E_CLI_038_LOG",
"-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"},
},
expectedLog: LogValidation{
logFile: "E2E_CLI_038_LOG",
validationFunc: func(logText string) bool {
match1, _ := regexp.MatchString("Scanning with Keeping Infrastructure as Code Secure", logText)
match2, _ := regexp.MatchString(`Files scanned: \d+`, logText)
match3, _ := regexp.MatchString(`Queries loaded: \d+`, logText)
return match1 && match2 && match3
},
},
},
wantStatus: []int{40},
removeFiles: []string{"E2E_CLI_038_LOG"},
}
E2E tests are skiped in short mode:
func Test_E2E_CLI(t *testing.T) {
kicsPath := getKICSBinaryPath("")
if testing.Short() {
t.Skip("skipping E2E tests in short mode.")
}
//...
}
Test Functions (utils.go)
getKICSBinaryPath: This function gets the path of the kics executable.
runCommand: This function is intended to run kics with the input arguments sent by the test
readFixture & readFile: These functions reads a file (from its folder and name). The folders used are always: fixtures or output.
prepareExpected: This function prepares the file to be compared, avoiding errors in reading and formatting.
checkLine & checkJSONLog: These functions are used to check the Kics output generated in the CLI and compare it with the expectedOutput file.
fileCheck & setFields: TThese functions read and compare the expected files with the files provided during the test. This function compares files from expectedPayload and expectedResult.