venom

package module
v1.2.0-beta.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 1, 2023 License: Apache-2.0 Imports: 39 Imported by: 38

README

🐍 Venom

Venom is a CLI (Command Line Interface) that aim to create, manage and run your integration tests with efficiency.

GitHub release GoDoc Go Report Card Discussions Contribute with Gitpod

Table of content

Overview

Venom allows you to handle integration tests the same way you code your application. With Venom, testcases will be managed as code: the readability of the tests means that the tests are part of the code reviews. Thanks to that, write and execute testsuites become easier for developers and teams.

Concretely, you have to write testsuite in a YAML file. Venom run executors (scripts, HTTP Request, web, IMAP, etc.) and apply assertions. It can also generate xUnit result files.

Venom Demonstration

Installing

Install from binaries

You can find latest binary release from: https://github.com/ovh/venom/releases/latest/.

Example for Linux:

$ curl https://github.com/ovh/venom/releases/download/v1.0.1/venom.linux-amd64 -L -o /usr/local/bin/venom && chmod +x /usr/local/bin/venom
$ venom -h

Updating

You can update to the latest version with venom update command:

$ venom update

The venom update command will download the latest version and replace the current binary:

Url to update venom: https://github.com/ovh/venom/releases/download/v1.0.1/venom.darwin-amd64
Getting latest release from: https://github.com/ovh/venom/releases/download/v1.0.1/venom.darwin-amd64 ...
Update done.

Check the new version with venom version command:

$ venom version
Version venom: v1.0.1

Docker image

Instead of installing (and updating) Venom locally, Venom can be started as a Docker image with following commands.

Considering your testsuites are in ./tests directory in your current directory and your test library is under ./tests/lib, the results will be available under the results directory.

$ mkdir -p results
$ docker run --mount type=bind,source=$(pwd)/tests,target=/workdir/tests --mount type=bind,source=$(pwd)/results,target=/workdir/results ovhcom/venom:latest

Please refer to https://hub.docker.com/r/ovhcom/venom/tags to get the available image tags.

CLI Usage

venom CLI is composed of several commands:

$ venom -h
Venom - RUN Integration Tests

Usage:
  venom [command]

Available Commands:
  help        Help about any command
  run         Run Tests
  update      Update venom to the latest release version: venom update
  version     Display Version of venom: venom version

Flags:
  -h, --help   help for venom

Use "venom [command] --help" for more information about a command.

You can see the help of a command with venom [command] -h:

$ venom run -h

run integration tests

Usage:
  venom run [flags]

Examples:
  Run all testsuites containing in files ending with *.yml or *.yaml: venom run
  Run a single testsuite: venom run mytestfile.yml
  Run a single testsuite and export the result in JSON format in test/ folder: venom run mytestfile.yml --format=json --output-dir=test
  Run a single testsuite and export the result in XML and HTML formats in test/ folder: venom run mytestfile.yml --format=xml --output-dir=test --html-report
  Run a single testsuite and specify a variable: venom run mytestfile.yml --var="foo=bar"
  Run a single testsuite and load all variables from a file: venom run mytestfile.yml --var-from-file variables.yaml
  Run all testsuites containing in files ending with *.yml or *.yaml with verbosity: VENOM_VERBOSE=2 venom run

  Notice that variables initialized with -var-from-file argument can be overrided with -var argument

  More info: https://github.com/ovh/venom

Flags:
      --format string           --format:json, tap, xml, yaml (default "xml")
  -h, --help                    help for run
      --html-report             Generate HTML Report
      --lib-dir string          Lib Directory: can contain user executors. example:/etc/venom/lib:$HOME/venom.d/lib
      --output-dir string       Output Directory: create tests results file inside this directory
      --stop-on-failure         Stop running Test Suite on first Test Case failure
      --var stringArray         --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file strings   --var-from-file filename.yaml --var-from-file filename2.yaml: yaml, must contains a dictionary
  -v, --verbose count           verbose. -v (INFO level in venom.log file), -vv to very verbose (DEBUG level) and -vvv to very verbose with CPU Profiling

Run test suites in a specific order

  • venom run 01_foo.yml 02_foo.yml will run 01 before 02.
  • venom run 02_foo.yml 01_foo.yml will run 02 before 01.

If you want to sort many testsuite files, you can use standard commands, example:

venom run `find . -type f -name "*.yml"|sort`

Globstar support

The venom CLI supports globstar:

$ venom run ./foo/b*/**/z*.yml

Variables

To specify individual variables on the command line, use the --var option when running the venom run commands:

$ venom run --var="foo=bar"
$ venom run --var='foo_list=["biz","buz"]'
$ venom run --var='foo={"biz":"bar","biz":"barr"}'

The --var option can be used many times in a single command.

Variable Definitions Files

To set a lot of variables, it is more convenient to specify their values in a variable definitions file. This file is a YAML dictionary. You have to specify that file on the command line with --var-from-file:

venom run --var-from-file variables.yaml
Environment Variables

As a fallback for the other ways of defining variables, venom tool searches the environment of its own process for environment variables named VENOM_VAR_ followed by the name of a declared variable.

$ export VENOM_VAR_foo=bar
$ venom run *.yml

You can also define the environment variable and run your testsuite in one line:

$ VENOM_VAR_foo=bar venom run *.yml

Arguments

You can define arguments on the command line using the flag name.

Flags are listed in the result of help command.

List of available flags for venom run command:

Flags:
      --format string           --format:json, tap, xml, yaml (default "xml")
  -h, --help                    help for run
      --html-report             Generate HTML Report
      --lib-dir string          Lib Directory: can contain user executors. example:/etc/venom/lib:$HOME/venom.d/lib
      --output-dir string       Output Directory: create tests results file inside this directory
      --stop-on-failure         Stop running Test Suite on first Test Case failure
      --var stringArray         --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file strings   --var-from-file filename.yaml --var-from-file filename2.yaml: yaml, must contains a dictionary
  -v, --verbose count           verbose. -vv to very verbose and -vvv to very verbose with CPU Profiling
Define arguments with environment variables

You can also define the arguments with environment variables:

# is the same as
VENOM_FORMAT=json venom run my-test-suite.yml

# is equivalent to
venom run my-test-suite.yml --format=json

Flags and their equivalent with environment variables usage:

  • --format="json" flag is equivalent to VENOM_FORMAT="json" environment variable
  • --lib-dir="/etc/venom/lib:$HOME/venom.d/lib" flag is equivalent to VENOM_LIB_DIR="/etc/venom/lib" environment variable
  • --output-dir="test-results" flag is equivalent to VENOM_OUTPUT_DIR="test-results" environment variable
  • --stop-on-failure flag is equivalent to VENOM_STOP_ON_FAILURE=true environment variable
  • --var foo=bar flag is equivalent to VENOM_VAR_foo='bar' environment variable
  • --var-from-file fileA.yml fileB.yml flag is equivalent to VENOM_VAR_FROM_FILE="fileA.yml fileB.yml" environment variable
  • -v flag is equivalent to VENOM_VERBOSE=1 environment variable
  • -vv flag is equivalent to VENOM_VERBOSE=2 environment variable

It is possible to set NO_COLOR=1 environment variable to disable colors from output.

Use a configuration file

You can define the Venom settings using a configuration file .venomrc. This configuration file should be placed in the current directory or in the home directory.

variables: 
  - foo=bar
variables_files:
  - my_var_file.yaml
stop_on_failure: true
format: xml
output_dir: output
lib_dir: lib
verbosity: 3

Please note that the command line flags overrides the configuration file. The configuration file overrides the environment variables.

Concepts

TestSuites

A test suite is a collection of test cases that are intended to be used to test a software program to show that it has a specified set of behaviors. A test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

In venom the testcases are executed sequentially within a testsuite. Each testcase is an ordered set of steps. Each step is based on an executor that enable some specific kind of behavior.

In venom a testsuite is written in one YAML file respecting the following structure:


name: Title of TestSuite
testcases:
- name: TestCase with default value, exec cmd. Check if exit code != 1
  steps:
  - script: echo 'foo'
    type: exec

- name: Title of First TestCase
  steps:
  - script: echo 'foo'
    assertions:
    - result.code ShouldEqual 0
  - script: echo 'bar'
    assertions:
    - result.systemout ShouldNotContainSubstring foo
    - result.timeseconds ShouldBeLessThan 1

- name: GET http testcase, with 5 seconds timeout
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    timeout: 5
    assertions:
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1

- name: Test with retries and delay in seconds between each try
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    retry: 3
    retry_if: # (optional, lets you early break unrecoverable errors)
    - result.statuscode ShouldNotEqual 403
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 200

Executors

User defined executors

You can define an executor with a single YAML file. This is a good way to abstract technical or functional behaviors and reuse them in complex testsuites.

Example:

file lib/customA.yml:

executor: hello
input:
  myarg: {}
steps:
- script: echo "{\"hello\":\"{{.input.myarg}}\"}"
  assertions:
  - result.code ShouldEqual 0
  vars:
    hello:
      from: result.systemoutjson.hello
    all:
      from: result.systemoutjson
output:
  display:
    hello: "{{.hello}}"
  all: "{{.all}}"

file testsuite.yml:

name: testsuite with a user executor
testcases:
- name: testA
  steps:
  - type: hello
    myarg: World
    assertions:
    - result.display.hello ShouldContainSubstring World
    - result.alljson.hello ShouldContainSubstring World

Notice the variable alljson. All variables declared in output are automatically converted in a json format with the suffix json. In the example above, two implicit variables are available: displayjson.hello and alljson.

Venom will load user's executors from the directory lib/ relative to the testsuite path. You add executors source path using the flag --lib-dir. Note that all folders listed with --lib-dir will be scanned recursively to find .yml files as user executors.

The user defined executors work with templating, you can check the templating result in venom.log. In this file, if you see an error as error converting YAML to JSON: yaml: line 14: found unexpected end of stream, you probably need to adjust indentation with the templating function indent.

Example:

name: testsuite with a user executor multilines
testcases:
- name: test
  steps:
  - type: multilines
    script: |
      # test multilines
      echo "5"
    assertions:
    - result.alljson ShouldEqual 5

using this executor:

executor: multilines
input:
  script: "echo 'foo'"
steps:
- type: exec
  script: {{ .input.script | nindent 4 }}
  assertions:
  - result.code ShouldEqual 0
  vars:
    all:
      from: result.systemoutjson
output:
  all: '{{.all}}'

# lib/*.yml files will be loaded as executors.
$ venom run testsuite.yml 

# executors will be loaded from /etc/venom/lib, $HOME/venom.d/lib and lib/ directory relative to testsuite.yml file.
$ venom run --lib-dir=/etc/venom/lib:$HOME/venom.d/lib testsuite.yml

Variables

Testsuite variables

You can define variables at the testsuite level.

name: myTestSuite
vars:
  foo: foo
  biz:
    bar: bar
  aString: '{"foo": "bar"}'

testcases:
- name: first-test-case
  steps:
  - type: exec
    script: echo '{{.foo}} {{.biz.bar}}'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual "foo bar"

- name: foobar
  steps:
  - script: echo '{{.aString}}'
    info: value of aString is {{.aString}}
    assertions:
    - result.systemoutjson.foo ShouldEqual bar
...

Each user variable used in testsuite must be declared in this section. You can override its value at runtime in a number of ways:

  • Individually, with the --var command line option.
  • In variable definitions files, either specified on the command line --var-from-file.
  • As environment variables.
Variable helpers

Available helpers and some examples:

  • abbrev
  • abbrevboth
  • trunc
  • trim
  • upper: {{.myvar | upper}}
  • lower: {{.myvar | lower}}
  • title
  • untitle
  • substr
  • repeat
  • trimall
  • trimAll
  • trimSuffix
  • trimPrefix
  • nospace
  • initials
  • randAlphaNum
  • randAlpha
  • randASCII
  • randNumeric
  • swapcase
  • shuffle
  • snakecase
  • camelcase
  • quote
  • squote
  • indent
  • nindent
  • replace: {{.myvar | replace "_" "."}}
  • plural
  • default: {{.myvar | default ""}}
  • empty
  • coalesce
  • toJSON
  • toPrettyJSON
  • b64enc
  • b64dec {{.result.bodyjson | b64enc}}
  • escape: replace ‘_‘, ‘/’, ‘.’ by ‘-’

More examples are available here

Use outputs from a test step as input of another test step

To be able to reuse a property from a teststep in a following testcase or step, you have to extract the variable, as the following example.

After the first step execution, venom extracts a value using a regular expression foo with a ([a-z]+) here from the content of the result.systemout property returned by the executor. Then this variable can be reused in another test, with the name testA.myvariable with testA corresponding to the name of the testcase. A default value could also be supplied if the variable can't be extracted from the output, which can commonly happen when parsing json output.

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo 'foo with a bar here'
    vars:
      myvariable:
        from: result.systemout
        regex: foo with a ([a-z]+) here
        default: "somevalue"

- name: testB
  steps:
  - type: exec
    script: echo {{.testA.myvariable}}
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldContainSubstring bar

Builtin venom variables

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo '{{.venom.testsuite}} {{.venom.testsuite.filename}} {{.venom.testcase}} {{.venom.teststep.number}} {{.venom.datetime}} {{.venom.timestamp}}'
    # will display something as: MyTestSuite MyTestSuiteWithVenomBuiltinVar.yml testA 0 2018-08-05T21:38:24+02:00 1533497904

Builtin variables:

  • {{.venom.datetime}}
  • {{.venom.executable}}
  • {{.venom.libdir}}
  • {{.venom.outputdir}}
  • {{.venom.testcase}}
  • {{.venom.teststep.number}}
  • {{.venom.testsuite.name}}
  • {{.venom.testsuite.filename}}
  • {{.venom.testsuite.filepath}}
  • {{.venom.testsuite.shortName}}
  • {{.venom.testsuite.workdir}}
  • {{.venom.testsuite}}
  • {{.venom.timestamp}}

Assertions

Keywords
Must keywords

All the above assertions keywords also have a Must counterpart which can be used to create a required passing assertion and prevent test cases (and custom executors) to run remaining steps.

Example:

- steps:
  - type: exec
    script: exit 1
    assertions:
      - result.code MustEqual 0
  # Remaining steps in this context will not be executed
Using logical operators

While assertions use and operator implicitly, it is possible to use other logical operators to perform complex assertions.

Supported operators are and, or and xor.

- name: Assertions operators
  steps:
  - script: echo 1
    assertions:
      - or:
        - result.systemoutjson ShouldEqual 1 
        - result.systemoutjson ShouldEqual 2
      # Nested operators
      - or:
        - result.systemoutjson ShouldBeGreaterThanOrEqualTo 1
        - result.systemoutjson ShouldBeLessThanOrEqualTo 1
        - or:
          - result.systemoutjson ShouldEqual 1

More examples are available in tests/assertions_operators.yml.

Write and run your first test suite

To understand how Venom is working, let's create and run a first testsuite together.

The first assertions that we will do, in this testsuite, are to check whether the site we want to test (a public REST API, OVHcloud API for example):

  • is accessible (respond with a 200 status code)
  • responds in less than 5 seconds
  • returns a valid response (which is in JSON format)

First, create your testsuite in a file called testsuite.yml for example. Open it in your favorite editor or IDE and fill it with this content:

name: APIIntegrationTest

vars:
  url: https://eu.api.ovh.com/

testcases:
- name: GET http testcase, with 5 seconds timeout
  steps:
  - type: http
    method: GET
    url: {{.url}}/1.0/
    timeout: 5
    assertions:
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1
    - result.bodyjson ShouldContainKey apis
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing

Then, run your testsuite with the following command:

$ venom run 

 • APIIntegrationTest (testsuite.yml)
 	• GET-http-testcase-with-5-seconds-timeout SUCCESS

You wrote and executed your first testsuite with the HTTP executor! :)

Export tests report

You can export your testsuite results as a report in several available formats: xUnit (XML), JSON, YAML, TAP.

You can specify the output directory with the --output-dir flag and the format with the --format flag (XML by default):

$ venom run --format=xml --output-dir="."

# html export
$ venom run --output-dir="." --html-report

Reports exported in XML can be visualized with a xUnit/jUnit Viewer, directly in your favorite CI/CD stack for example in order to see results run after run.

Advanced usage

Debug your testsuites

A venom.log file is generated for each venom run command.

There are two ways to debug a testsuite:

  • use -v flag on venom binary.
    • $ venom run -v test.yml will output details for each step
    • $ venom run -vv test.yml will generate dump.json files for each teststep.
    • $ venom run -vvv test.yml will generate pprof files for CPU profiling.
  • use info keyword in your teststep: test.yml file:
name: Exec testsuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo 'foo with a bar here'
    info:
      - this a first info
      - and a second...
- name: cat json
  steps:
  - script: cat exec/testa.json
    info: "the value of result.systemoutjson is {{.result.systemoutjson}}"
    assertions:
    - result.systemoutjson.foo ShouldContainSubstrin bar

$ venom run test.yml

# output:

 • Exec testsuite (exec.yml)
  • testA SUCCESS
    [info] this a first info (exec.yml:8)
    [info] and a second... (exec.yml:9)
  • testB SUCCESS
  • sleep 1 SUCCESS
  • cat json SUCCESS
    [info] the value of result.systemoutjson is map[foo:bar] (exec.yml:34)

Skip testcase and teststeps

It is possible to skip testcase according to some assertions. For instance, the following example will skip the last testcase.

name: "Skip testsuite"
vars:
  foo: bar

testcases:
- name: init
  steps:
  - type: exec
    script: echo {{.foo}}
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldContainSubstring bar

- name: do-not-skip-this
  skip: 
  - foo ShouldNotBeEmpty
  steps:
  - type: exec
    script: exit 0

- name: skip-this
  skip: 
    - foo ShouldBeEmpty
  steps:
  - type: exec
    script: command_not_found
    assertions:
    - result.code ShouldEqual 0

A skip statement may also be placed at steps level to partially execute a testcase. If all steps from a testcase are skipped, the testcase itself will also be treated as "skipped" rather than "passed"/"failed".

name: "Skip testsuite"
vars:
  foo: bar

testcases:
- name: skip-one-of-these
  steps:
  - name: do-not-skip-this
    type: exec
    script: exit 0
    assertions:
    - result.code ShouldEqual 0
    skip:
    - foo ShouldNotBeEmpty
  - name: skip-this
    type: exec
    script: exit 1
    assertions:
    - result.code ShouldEqual 0
    skip:
    - foo ShouldBeEmpty

Iterating over data

It is possible to iterate over data using range attribute.

The following data types are supported, each exposing contexted variables .index, .key and .value:

  • An array where each value will be iterated over ([]interface{})
    • .index/.key: current iteration index
    • .value: current iteration item value
  • A map where each key will be iterated over (map[string]interface{})
    • .index: current iteration index
    • .key: current iteration item key
    • .value: current iteration item value
  • An integer to perform target step n times (int)
    • .index/.key/.value: current iteration index
  • A templated string which results in one of the above typing (string)
    • It can be either inherited from vars file, or interpolated from a previous step result

For instance, the following example will iterate over an array of two items containing maps:

- name: range with hardcoded array
  steps:
  - type: exec
    range:
      - actual: hello
        expected: hello
      - actual: world
        expected: world
    script: echo "{{.value.actual}}"
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual "{{.value.expected}}"

More examples are available in tests/ranged.yml.

FAQ

Common errors with quotes

If you have this kind of error:

err:unable to parse file "foo.yaml": error converting YAML to JSON: yaml: line 8: did not find expected key

this is probably because you try to use a json value instead of a string. You should have more details in venom.log file.

Wrong:

...
vars:
  body: >-
      {
        "the-attribute": "the-value"
      }
...
steps:
- type: http
  body: "{{.body}}"
...

OK:

...
vars:
  body: >-
      {
        "the-attribute": "the-value"
      }
...
steps:
- type: http
  body: '{{.body}}'
...

Note the simple quote on the value of body.

Use venom in CI/CD pipelines

Venom can be used in dev environment or on your CI server. To display properly the venom output, you probably will have to export the environment variable IS_TTY=true before running venom.

Hacking

How to write your own executor?

How to compile?

$ make build

How to test?

Unit tests:
make test
Integration tests:

Prepare the stack:

make build OS=linux ARCH=amd64
cp dist/venom.linux-amd64 tests/venom
cd tests
make start-test-stack  # (wait a few seconds)
make build-test-binary-docker

Run integration tests:

make run-test

Cleanup:

make clean
make stop-test-stack

Contributing

Contribute with Gitpod

Please read the contributing guide to learn about how you can contribute to Venom ;-). There is no small contribution, don't hesitate!

Our awesome contributors:

License

Copyright 2022 OVH SAS

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	//Version is set with -ldflags "-X github.com/ovh/venom/venom.Version=$(VERSION)"
	Version = "snapshot"
	IsTest  = ""
)
View Source 
var Cyan = color.New(color.FgCyan).SprintFunc()
View Source 
var Gray = color.New(color.Attribute(90)).SprintFunc()
View Source 
var Green = color.New(color.FgGreen).SprintFunc()
View Source 
var Red = color.New(color.FgRed).SprintFunc()
View Source 
var Yellow = color.New(color.FgYellow).SprintFunc()

Functions

func BoolVarFromCtx added in v1.0.0 

func BoolVarFromCtx(ctx context.Context, varname string) bool

func Debug added in v1.0.0 

func Debug(ctx context.Context, format string, args ...interface{})

func Dump added in v1.0.0 

func Dump(va interface{}) (map[string]interface{}, error)

Dump dumps v as a map[string]interface{}.

func DumpString added in v1.0.0 

func DumpString(va interface{}) (map[string]string, error)

DumpString dumps v as a map[string]string{}, key in lowercase

func DumpStringPreserveCase added in v1.0.0 

func DumpStringPreserveCase(va interface{}) (map[string]string, error)

DumpStringPreserveCase dumps v as a map[string]string{}

func DumpWithPrefix added in v1.1.0 

func DumpWithPrefix(va interface{}, prefix string) (map[string]interface{}, error)

Dump dumps v as a map[string]interface{}.

func Error added in v1.0.0 

func Error(ctx context.Context, format string, args ...interface{})

func Fatal added in v1.0.0 

func Fatal(ctx context.Context, format string, args ...interface{})

func GetExecutorResult added in v1.0.0 

func GetExecutorResult(r interface{}) map[string]interface{}

func Info added in v1.0.0 

func Info(ctx context.Context, format string, args ...interface{})

func InitTestLogger added in v1.0.0 

func InitTestLogger(t *testing.T)

func IntVarFromCtx added in v1.0.0 

func IntVarFromCtx(ctx context.Context, varname string) int

func JSONUnmarshal added in v1.0.0 

func JSONUnmarshal(btes []byte, i interface{}) error

func OSExit added in v1.1.0 

func OSExit(exitCode int)

func RemoveNotPrintableChar added in v0.16.0 

func RemoveNotPrintableChar(in string) string

RemoveNotPrintableChar removes not printable character from a string

func StringMapInterfaceVarFromCtx added in v1.0.0 

func StringMapInterfaceVarFromCtx(ctx context.Context, varname string) map[string]interface{}

func StringMapStringVarFromCtx added in v1.0.0 

func StringMapStringVarFromCtx(ctx context.Context, varname string) map[string]string

func StringSliceVarFromCtx added in v1.0.0 

func StringSliceVarFromCtx(ctx context.Context, varname string) []string

func StringVarFromCtx added in v1.0.0 

func StringVarFromCtx(ctx context.Context, varname string) string

func VarFromCtx added in v1.0.0 

func VarFromCtx(ctx context.Context, varname string) interface{}

func Warn added in v1.0.0 

func Warn(ctx context.Context, format string, args ...interface{})

func Warning added in v1.0.0 

func Warning(ctx context.Context, format string, args ...interface{})

func WithFormatterLowerFirstKey added in v1.1.0 

func WithFormatterLowerFirstKey() dump.KeyFormatterFunc

Types

type Assertion added in v1.1.0 

type Assertion interface{}

Assertion (usually a string, but could be a dictionary when using logical operators)

type AssertionApplied added in v1.1.0 

type AssertionApplied struct {
	Assertion Assertion `json:"assertion" yml:"-"`
	IsOK      bool      `json:"isOK" yml:"-"`
}

type AssertionsApplied added in v1.1.0 

type AssertionsApplied struct {
	OK bool `json:"ok" yml:"-"`

	Assertions []AssertionApplied `json:"assertions" yml:"-"`
	// contains filtered or unexported fields
}

type AssignStep added in v0.27.0 

type AssignStep struct {
	Assignments map[string]Assignment `json:"vars" yaml:"vars" mapstructure:"vars"`
}

type Assignment added in v0.27.0 

type Assignment struct {
	From    string      `json:"from" yaml:"from"`
	Regex   string      `json:"regex" yaml:"regex"`
	Default interface{} `json:"default" yaml:"default"`
}

type ContextKey 

type ContextKey string

ContextKey can be added in context to store contextual infos. Also used by logger.

type Executor 

type Executor interface {
	// Run run a Test Step
	Run(context.Context, TestStep) (interface{}, error)
}

Executor execute a testStep.

type ExecutorRunner added in v1.0.0 

type ExecutorRunner interface {
	Executor

	ExecutorWithSetup
	Name() string
	Retry() int
	RetryIf() []string
	Delay() int
	Timeout() int
	Info() []string
	Type() string
	GetExecutor() Executor
	// contains filtered or unexported methods
}

type ExecutorWithSetup added in v1.0.0 

type ExecutorWithSetup interface {
	Setup(ctx context.Context, vars H) (context.Context, error)
	TearDown(ctx context.Context) error
}

type Failure 

type Failure struct {
	TestcaseClassname  string `xml:"-" json:"-" yaml:"-"`
	TestcaseName       string `xml:"-" json:"-" yaml:"-"`
	TestcaseLineNumber int    `xml:"-" json:"-" yaml:"-"`
	StepNumber         int    `xml:"-" json:"-" yaml:"-"`
	Assertion          string `xml:"-" json:"-" yaml:"-"`
	AssertionRequired  bool   `xml:"-" json:"-" yaml:"-"`
	Error              error  `xml:"-" json:"-" yaml:"-"`

	Value string `json:"value" yaml:"value,omitempty"`
}

Failure contains data related to a failed test.

func (Failure) String added in v1.0.0 

func (f Failure) String() string

type FailureXML added in v1.1.0 

type FailureXML struct {
	Value   string `xml:",cdata" json:"value" yaml:"value,omitempty"`
	Type    string `xml:"type,attr,omitempty" json:"type" yaml:"type,omitempty"`
	Message string `xml:"message,attr,omitempty" json:"message" yaml:"message,omitempty"`
}

type H added in v0.27.0 

type H map[string]interface{}

func AllVarsFromCtx added in v1.0.0 

func AllVarsFromCtx(ctx context.Context) H

func (*H) Add added in v0.27.0 

func (h *H) Add(k string, v interface{})

func (*H) AddAll added in v0.27.0 

func (h *H) AddAll(h2 H)

func (*H) AddAllWithPrefix added in v0.27.0 

func (h *H) AddAllWithPrefix(p string, h2 H)

func (*H) AddWithPrefix added in v0.27.0 

func (h *H) AddWithPrefix(p, k string, v interface{})

func (H) Clone added in v0.27.0 

func (h H) Clone() H

type InnerResult 

type InnerResult struct {
	Value string `xml:",cdata" json:"value" yaml:"value"`
}

InnerResult is used by TestCase

type Range added in v1.1.0 

type Range struct {
	Enabled    bool
	Items      []RangeData
	RawContent interface{} `json:"range"`
}

Range contains data related to iterable user values

type RangeData added in v1.1.0 

type RangeData struct {
	Key   string
	Value interface{}
}

RangeData contains a single iterable user value

type Skipped added in v0.16.0 

type Skipped struct {
	Value string `xml:",cdata" json:"value" yaml:"value,omitempty"`
}

Skipped contains data related to a skipped test.

type Status added in v1.2.0 

type Status string
const (
	StatusRun  Status = "RUN"
	StatusFail Status = "FAIL"
	StatusSkip Status = "SKIP"
	StatusPass Status = "PASS"
)

type StepAssertions 

type StepAssertions struct {
	Assertions []Assertion `json:"assertions,omitempty" yaml:"assertions,omitempty"`
}

StepAssertions contains step assertions

type TestCase 

type TestCase struct {
	TestCaseInput

	Skipped []Skipped `json:"skipped" yaml:"-"`
	Status  Status    `json:"status" yaml:"-"`

	Duration float64   `json:"duration" yaml:"-"`
	Start    time.Time `json:"start" yaml:"-"`
	End      time.Time `json:"end" yaml:"-"`

	TestStepResults []TestStepResult `json:"results" yaml:"-"`
	TestSuiteVars   H                `json:"-" yaml:"-"`

	IsExecutor  bool `json:"-" yaml:"-"`
	IsEvaluated bool `json:"-" yaml:"-"`
	// contains filtered or unexported fields
}

type TestCaseInput added in v1.1.0 

type TestCaseInput struct {
	Name         string            `json:"name" yaml:"name"`
	Vars         H                 `json:"vars" yaml:"vars"`
	Skip         []string          `json:"skip" yaml:"skip"`
	RawTestSteps []json.RawMessage `json:"steps" yaml:"steps"`
}

type TestCaseXML added in v1.1.0 

type TestCaseXML struct {
	XMLName   xml.Name     `xml:"testcase" json:"-" yaml:"-"`
	Classname string       `xml:"classname,attr,omitempty" json:"classname" yaml:"-"`
	Errors    []FailureXML `xml:"error,omitempty" json:"errors" yaml:"errors,omitempty"`
	Failures  []FailureXML `xml:"failure,omitempty" json:"failures" yaml:"failures,omitempty"`
	Name      string       `xml:"name,attr" json:"name" yaml:"name"`
	Skipped   []Skipped    `xml:"skipped,omitempty" json:"skipped" yaml:"skipped,omitempty"`
	Systemout InnerResult  `xml:"system-out,omitempty" json:"systemout" yaml:"systemout,omitempty"`
	Systemerr InnerResult  `xml:"system-err,omitempty" json:"systemerr" yaml:"systemerr,omitempty"`
	Time      float64      `xml:"time,attr,omitempty" json:"time" yaml:"time,omitempty"`
}

TestCase is a single test case with its result.

type TestStep 

type TestStep map[string]interface{}

TestStep represents a testStep

func (TestStep) IntValue added in v1.0.0 

func (t TestStep) IntValue(name string) (int, error)

func (TestStep) StringSliceValue added in v1.0.0 

func (t TestStep) StringSliceValue(name string) ([]string, error)

func (TestStep) StringValue added in v1.0.0 

func (t TestStep) StringValue(name string) (string, error)

type TestStepResult added in v1.1.0 

type TestStepResult struct {
	Name              string            `json:"name"`
	Errors            []Failure         `json:"errors"`
	Skipped           []Skipped         `json:"skipped" yaml:"skipped"`
	Status            Status            `json:"status" yaml:"status"`
	Raw               interface{}       `json:"raw" yaml:"raw"`
	Interpolated      interface{}       `json:"interpolated" yaml:"interpolated"`
	Number            int               `json:"number" yaml:"number"`
	RangedIndex       int               `json:"rangedIndex" yaml:"rangedIndex"`
	RangedEnable      bool              `json:"rangedEnable" yaml:"rangedEnable"`
	InputVars         map[string]string `json:"inputVars" yaml:"-"`
	ComputedVars      H                 `json:"computedVars" yaml:"-"`
	ComputedInfo      []string          `json:"computedInfos" yaml:"-"`
	AssertionsApplied AssertionsApplied `json:"assertionsApplied" yaml:"-"`
	Retries           int               `json:"retries" yaml:"retries"`

	Systemout string    `json:"systemout"`
	Systemerr string    `json:"systemerr"`
	Duration  float64   `json:"duration"`
	Start     time.Time `json:"start"`
	End       time.Time `json:"end"`
}

type TestSuite 

type TestSuite struct {
	Name      string     `json:"name" yaml:"name"`
	TestCases []TestCase `json:"testcases" yaml:"testcases"`
	Vars      H          `json:"vars" yaml:"vars"`

	// computed
	ShortName    string `json:"shortname" yaml:"-"`
	Filename     string `json:"filename" yaml:"-"`
	Filepath     string `json:"filepath" yaml:"-"`
	ComputedVars H      `json:"computed_vars" yaml:"-"`
	WorkDir      string `json:"workdir" yaml:"_"`
	Status       Status `json:"status" yaml:"status"`

	Duration float64   `json:"duration" yaml:"-"`
	Start    time.Time `json:"start" yaml:"-"`
	End      time.Time `json:"end" yaml:"-"`

	NbTestcasesFail int `json:"nbTestcasesFail"  yaml:"-"`
	NbTestcasesPass int `json:"nbTestcasesPass"  yaml:"-"`
	NbTestcasesSkip int `json:"nbTestcasesSkip"  yaml:"-"`
}

type TestSuiteInput added in v1.1.0 

type TestSuiteInput struct {
	Name      string          `json:"name" yaml:"name"`
	TestCases []TestCaseInput `json:"testcases" yaml:"testcases"`
	Vars      H               `json:"vars" yaml:"vars"`
}

type TestSuiteXML added in v1.1.0 

type TestSuiteXML struct {
	XMLName   xml.Name      `xml:"testsuite" json:"-" yaml:"-"`
	Disabled  int           `xml:"disabled,attr,omitempty" json:"disabled" yaml:""`
	Errors    int           `xml:"errors,attr,omitempty" json:"errors" yaml:"-"`
	Failures  int           `xml:"failures,attr,omitempty" json:"failures" yaml:"-"`
	Hostname  string        `xml:"hostname,attr,omitempty" json:"hostname" yaml:"-"`
	ID        string        `xml:"id,attr,omitempty" json:"id" yaml:"-"`
	Name      string        `xml:"name,attr" json:"name" yaml:"name"`
	Package   string        `xml:"package,attr,omitempty" json:"package" yaml:"-"`
	Skipped   int           `xml:"skipped,attr,omitempty" json:"skipped" yaml:"skipped,omitempty"`
	Total     int           `xml:"tests,attr" json:"total" yaml:"total,omitempty"`
	TestCases []TestCaseXML `xml:"testcase" json:"testcases" yaml:"testcases"`
	Version   string        `xml:"version,omitempty" json:"version" yaml:"version,omitempty"`
	Time      string        `xml:"time,attr,omitempty" json:"time" yaml:"-"`
	Timestamp string        `xml:"timestamp,attr,omitempty" json:"timestamp" yaml:"-"`
}

TestSuite is a single JUnit test suite which may contain many testcases.

type Tests 

type Tests struct {
	TestSuites       []TestSuite `json:"test_suites" yml:"tests_suites"`
	Status           Status      `json:"status" yml:"status"`
	NbTestsuitesFail int         `json:"nbTestsuitesFail"  yaml:"-"`
	NbTestsuitesPass int         `json:"nbTestsuitesPass"  yaml:"-"`
	NbTestsuitesSkip int         `json:"nbTestsuitesSkip"  yaml:"-"`
	Duration         float64     `json:"duration" yaml:"-"`
	Start            time.Time   `json:"start" yaml:"-"`
	End              time.Time   `json:"end" yaml:"-"`
}

type TestsHTML added in v1.1.0 

type TestsHTML struct {
	Tests     Tests  `json:"tests"`
	JSONValue string `json:"jsonValue"`
}

type TestsXML added in v1.1.0 

type TestsXML struct {
	XMLName    xml.Name       `xml:"testsuites" json:"-" yaml:"-"`
	TestSuites []TestSuiteXML `xml:"testsuite" json:"test_suites"`
}

type UserExecutor added in v1.0.0 

type UserExecutor struct {
	Executor     string            `json:"executor" yaml:"executor"`
	Input        H                 `json:"input" yaml:"input"`
	RawTestSteps []json.RawMessage `json:"steps" yaml:"steps"`
	Output       json.RawMessage   `json:"output" yaml:"output"`
	Filename     string            `json:"-" yaml:"-"`
}

func (UserExecutor) Run added in v1.0.0 

func (ux UserExecutor) Run(ctx context.Context, step TestStep) (interface{}, error)

Run is not implemented on user executor

func (UserExecutor) ZeroValueResult added in v1.0.0 

func (ux UserExecutor) ZeroValueResult() interface{}

type Venom added in v0.17.0 

type Venom struct {
	LogOutput io.Writer

	PrintFunc func(format string, a ...interface{}) (n int, err error)

	Tests Tests

	LibDir        string
	OutputFormat  string
	OutputDir     string
	StopOnFailure bool
	HtmlReport    bool
	Verbose       int
	// contains filtered or unexported fields
}

func New added in v0.17.0 

func New() *Venom

New instantiates a new venom on venom run cmd

func (*Venom) AddVariables added in v0.17.0 

func (v *Venom) AddVariables(variables map[string]interface{})

func (*Venom) GetExecutorRunner added in v1.0.0 

func (v *Venom) GetExecutorRunner(ctx context.Context, ts TestStep, h H) (context.Context, ExecutorRunner, error)

GetExecutorRunner initializes a test by name no type -> exec is default

func (*Venom) InitLogger added in v1.0.0 

func (v *Venom) InitLogger() error

InitLogger initializes venom logger

func (*Venom) OutputResult added in v0.17.0 

func (v *Venom) OutputResult() error

OutputResult output result to sdtout, files...

func (*Venom) Parse added in v0.17.0 

func (v *Venom) Parse(ctx context.Context, path []string) error

Parse parses tests suite to check context and variables

func (*Venom) Print added in v1.0.0 

func (v *Venom) Print(format string, a ...interface{})

func (*Venom) Println added in v1.0.0 

func (v *Venom) Println(format string, a ...interface{})

func (*Venom) PrintlnIndentedTrace added in v1.1.0 

func (v *Venom) PrintlnIndentedTrace(s string, indent string)

func (*Venom) PrintlnTrace added in v1.0.0 

func (v *Venom) PrintlnTrace(s string)

func (*Venom) Process added in v0.17.0 

func (v *Venom) Process(ctx context.Context, path []string) error

Process runs tests suite and return a Tests result

func (*Venom) RegisterExecutorBuiltin added in v1.0.0 

func (v *Venom) RegisterExecutorBuiltin(name string, e Executor)

RegisterExecutorBuiltin register builtin executors

func (*Venom) RegisterExecutorPlugin added in v1.0.0 

func (v *Venom) RegisterExecutorPlugin(name string, e Executor)

RegisterExecutorPlugin register plugin executors

func (*Venom) RegisterExecutorUser added in v1.0.0 

func (v *Venom) RegisterExecutorUser(name string, e Executor)

RegisterExecutorUser register User sxecutors

func (*Venom) RunTestStep added in v0.17.0 

func (v *Venom) RunTestStep(ctx context.Context, e ExecutorRunner, tc *TestCase, tsResult *TestStepResult, stepNumber int, rangedIndex int, step TestStep)

RunTestStep executes a venom testcase is a venom context

func (*Venom) RunUserExecutor added in v1.0.0 

func (v *Venom) RunUserExecutor(ctx context.Context, runner ExecutorRunner, tcIn *TestCase, tsIn *TestStepResult, step TestStep) (interface{}, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
venom
venom/root
venom/run
venom/update
venom/version
amqp
dbfixtures
exec
grpc
http
imap
kafka
mongo
mqtt
ovhapi
plugins/hello
plugins/odbc
rabbitmq
readfile
redis
smtp
sql
ssh
web

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.