arp

package module
v0.99.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 16, 2021 License: MIT Imports: 25 Imported by: 0

README

Arp

Short for Arpeggio, is a tool for automating and validating RESTful, RPC, and Websocket network calls. This tool is intended to be used for writing code free, quick, and flexible, integration or end-to-end tests.

Excluding the validation aspect, it can be used to orchestrate a sequence of network calls where the outputs of a call can be mapped to inputs of the subsequent calls.

Hilights

  • Tests are defined as YAML configuration files, no coding required
  • Validations are defined based on the expectation that an API call returns JSON formatted data. Each field can be validated using exact or partial matching.
  • Non-JSON formatted responses (e.g. binaries from download URLS, websocket requests) are automatically transformed into a JSON representable object that exposes simple properties for validation (size in bytes, sha256sum). If these properties are not enough, the response can be dumped to a local file and passed to an external program for validation as part of the test definition!
  • Individual fields from a test case's JSON response can be persisted and used as inputs or within the field validators for subsequent test cases via variables
  • Features an interactive mode for stepping through your test cases, retrying individual tests, viewing stored variables, and even hot-reloading of your test definition
The Inspiration

If you've ever had to write a backend API, you have probably written curl commands or configured PostMan to execute calls against your API for testing purposes. In this testing process, you have probably encountered one or more of the following scenarios:

  1. You don't care about the exact details of the API response, but are only looking to validate the shape of it
  2. You want to perform an exact match on the API response with a known good response
  3. Your API call depends on the response of another API call that you have to manually copy and paste into the input of your current test
  4. Repeat 3 as many times as it takes to get to the API state you are trying to validate.

This program helps solve those problems by allowing you to:

  • Define validations based on the shape of the response
  • Perform exact or fuzzy matching on values within the response
  • Filter for items in the response and pick out specific data to use as an input for future test cases

It basically tries to automate the process for why you are writing a curl command or setting PostMan up for in the first place. If you were going through the effort to use curl or PostMan for validation, it is not much more effort to extend that into a repeatable test case.

Definitions
  • Test: An individual test case that provides an input and validates a response
  • Test Suite: A file containing multiple tests
  • Validation Definition: A structure containing rules on how to validate a specific response property
  • Validation Matcher: A comparison value/placeholder to evaluate against a primitive data type within the response (e.g. bool, string, int)

Installation

export GO111MODULE=on 

git clone https://github.com/monstercat/arp.git && \
  cd arp && \
  go install ./cmd/arp

Usage

Usage of ./arp:
  -always-headers
        Always print the request and response headers in long test report output whether any matchers are defined for them or not.
  -colors
        Print test report with colors. (default true)
  -error-report
        Generate a test report that only contain failing test results.
  -file string
        Path to an individual test file to execute.
  -fixtures string
        Path to yaml file with data to include into the test scope via test variables. This file is also merged with each test file such that any YAML anchors defined within it are available for reference in the test files.
  -short
        Print a short report for executed tests containing only the validation results. (default true)
  -short-fail
        Keep the report short when errors are encountered rather than expanding with details.
  -step
        Run tests in interactive mode. Requires a test file to be provided with '-file'
  -tag value
        Only execute tests with tags matching this value. Tag input supports comma separated values which will execute tests that contain any on of those values. Subsequent tag parameters will AND with previous tag inputs to determine what tests will be run. Specifying no tag parameters will execute all tests.
  -test-root string
        Folder path containing all the test files to execute.
  -threads int
        Max number of test files to execute concurrently. (default 16)
  -tiny
        Print an even tinier report output than what the short flag provides. Only prints test status, name, and description. Failed tests will still be expanded.
  -var value
        Prepopulate the tests data store with a single KEY=VALUE pair. Multiple -var parameters can be provided for additional key/value pairs.

TLDR;

All tests in a directory can be executed with the -test-root flag like so:

./arp -test-root="<path to directory containing foo_test.yaml>"

Individual files can be executed using the -file flag:

./arp -file=<path>/foo_test.yaml

Sample Tests

Natural JSON Response

Here is a simple test case for a REST API returning a JSON response that can be executed:

REST Output

# sample.yaml
tests:
  - name: "List Users"
    description: "Listing all users"
    route: "https://reqres.in/api/users"
    method: GET
    response:
      code: 200
      payload:
        page:
          type: integer
          matches: $any
        per_page:
          type: integer
          matches: $any
        total:
          type: integer
          matches: $any
        total_pages:
          type: integer
          matches: $any
        data:
          type: array
          length: $notEmpty
          items:
            - type: object
              properties:
                id:
                  type: integer
                  matches: $any
                email:
                  type: string
                  matches: $any
                first_name:
                  type: string
                  matches: $any
                last_name:
                  type: string
                  matches: $any
                avatar:
                  type: string
                  matches: https://.*
Binary Response

When you get a non-JSON response from a network call, it gets transformed into a JSON representable object for validation:

Binary Output

tests:
  - name: Test Download
    description: Make sure we download the expected file
    route: https://www.dundeecity.gov.uk/sites/default/files/publications/civic_renewal_forms.zip
    method: GET
    response:
      code: 200
      # Set this to true so the binary payload can be formatted correctly for easy validation.
      binary: true
      # response will be saved to this file
      filePath: /tmp/myfile.zip
      payload:
        # We can validate the size in bytes if it is known ahead of time
        size:
          type: integer
          matches: $any
        # and/or we can validate the sha256 sum of the data
        sha256sum:
          type: string
          matches: $any
        # This field is always made available (whether a file path is 
        # provided or not) allowing an 'external' validator to process the file.
        saved:
          type: string
          matches: /tmp/myfile.zip
Test Structure Breakdown

This is a general overview of the test file structure. Individual examples with further details are also provided in their relevant doc sections.

# test_suite.yaml

# tests is an array of test case objects
tests:
    # name of the test
  - name: <string> 

    # what the test does
    description: <string> 
    
    # If set to true, this test will be skipped
    skip: <bool>

    # Used for both http calls and websocket connections
    route: <string> (<protocol>://<host>[:port]/<path>[?<params>&...])

    # For REST API calls only
    method: 'GET' | 'POST'

    # Headers to send with the request. These are sent on every REST API request and with the first Websocket client
    # connection.
    headers: # <object>
      # Each key refers to the specific header (e.g. Content-Type)
      <string>: <string>

    # A list of string identifiers to associate with the test. This can be used to filter what tests to execute at runtime with the `-tag` parameter
    tags:
      - <string>

    # Root object containing input to send with the request. 
    # This will change depending on the existence of any input modifiers (form_input, websockets, etc.)
    input:
      <string>: <object>|<array>|<bool>|<number>|<string>|<web socket messages>|<form inputs>
      
    # Protocol Modifier/Input modifier
    # If set to true, the contents of `input` will be sent as an HTML form with support for multipart upload.
    # See section 'API Inputs > Multipart/form-data' below for further details
    formInput: <boolean> 
    
    # Protocol/Input Modifier
    # If set to true, the test will spin up a websocket client and connect to the destination provided in `route`.
    #
    # A websocket test can send one or more payloads to fulfill an arbitary request transaction.
    #
    # The test will look for a specifically formatted `input` object based on the configuration below. See 
    # section 'API Inputs > Websocket' below for further details.
    websocket: <bool>
  
    # Protocol Modifier
    # If the following configs are provided, the test will spin up an RPC client and attempt to make a call
    # to the given address.
    rpc:
      protocol: HTTP | TCP
      address: <string>
      procedure: <string>

    # Root object containing instructions on how to validate the call response
    response:
      # Expected status code for an HTTP response. Not available for Websocket or RPC calls
      code: <integer>|<Integer Matcher>

      # Whether or not a binary response is expected. This will format any binary response into a basic representation
      # in JSON that validation matchers can be applied to. This object representation includes things like size in bytes and 
      # sha256 sum of the data
      # Only available for HTTP and RPC response validation
      binary: <bool>

      # File path to save any binary response data to. This can be used in conjunction with form uploads to test 
      # downloading and uploading of files
      filePath: <string>

      # Expected response headers to create matchers for. See the `Validations> Response Headers` section for more details.
      headers:
        <header name>: <Array Matcher>
        
      # Expected response matchers. Arp will always generate a response represented in JSON format that matchers can be
      # created for. This JSON representation may change depending on the nature of the response. See the `Validations` 
      # section for information on writing validators.
      payload:
        <string>: <Any Matcher>

API Inputs

There are currently 3 supported ways to provide inputs to your API request:

  • query parameters (REST)
  • JSON input (REST + RPC)
  • Multipart/form-data (REST: fields + multipart uploads + multi-file uploading)
  • Websockets (text and binary)
Query Parameters

Query parameters are simply added to the route property of your test case.

# sample.yaml
tests:
  - name: "List Users"
    description: "Listing all users on page 2"
    method: "GET"
    # add your parameters like you usually would
    route: "https://reqres.in/api/users?page=2"
---
JSON Input (REST/RPC)

JSON input can be provided using the input property of your test case for POST or any RPC request.

# create.yaml
tests:
  - name: "Create User"
    description: "Create a test user"
    method: "POST"
    route: "https://reqres.in/api/users"

    # pass in any json formatted input here
    input:
      name: "test user"
      job: "tester"
  
    response:
      code: 201
      payload:
        createdAt:
          type: string
          matches: $any
        id:
          type: string
          matches: $any

The input sent over the wire looks like:

{"name":  "test user", "job":  "tester"}
Multipart/form-data

You can specify that your input should be submitted as an HTML form by setting formInput: true in your test case. This mechanism can be used to upload one or more files. Form field names are defined by their key in the input property and are populated with the values they are mapped to. Entries that map to an array are treated as file form fields where each array element should be a file path that is to be uploaded with the form.

The request's Content-Type header will automatically be set for multipart forms.

For example, the following form

<form action="localhost/send" method="post" enctype="multipart/form-data">
    <input type="file" name="file" required="">
    <input type="text" name="description" value="" required="">
</form>

Can be submitted like so with the test

tests:
  - name: Test form submissions
    description: Upload a file through a form
    method: POST
    route: <some file uploader site>
    
    # set our input as form data
    formInput: true
    input:
      # populates a multipart form field called "files" in which one or more files will be uploaded
      file: 
        - /tmp/random.txt

      # populuates a multipart form field called "description" with the value "text file"
      description: "text file"
  
    response:
      code: 200
Websocket

You can set your test to make a websocket call by setting websocket: true in your test case. The first test case to have this flag enabled will be the one that creates the websocket client to the route specified in the test case. This client is then re-used for all test cases in the test suite that have websocket: true until the client is closed (input.closed: true).

The input object should be formatted as follows:

tests:
  ...
  
  websocket: true
  input:
    # array of websocket messages
    requests: 
        # <Websocket Message>
        # What message type to send the websocket request. Default: text
      - type: binary | text

        # How the payload is encoded when sending 'binary' websocket messages. Default: base64gzip
        #
        # * Base64 encoded embedded GZIP: your contents are gzipped and encoded as base64 explicitly for embedding in
        #        this test file. The test client will decode and extract the data to send the raw bytes over the wire
        # * Hex: you have an explicit set of bytes that you want to send. The test client will decode hex input and
        #        send the raw bytes over the wire.
        # * File: you have a local file that you want to send. The test client will read the entire file to memory and
        #        send the raw bytes over the wire.
        # * External: you want to call an external program/binary that will generate or encode your requests input at runtime.
        #           The standard output of the specified program will be streamed through the established websocket connection. If
        #           a non-zero exit code is returned, the test will fail and the executables STDERR contents will be provided
        #            
        encoding: base64gzip | hex | file | external

        # Payload to send in the request.
        payload: <object> | <string> | <external binary path>

        # Pass the following arguments to the binary specified in the `payload` when `encoding` is set to `external.
        args:
          - <string>
          - <string>
        
        # If set to true, the test client will not wait for a response and continue to send the next websocket message.
        # Default: false
        # Please note that setting `writeOnly: true` and making a call that DOES illicit a response will break 
        # the order in which test validations are run and likely fail your test.
        writeOnly: <bool>

        # If set to true, no message will be sent and the client will only wait for a response. 
        # Default: false
        # Having this and writeOnly enabled will create a no-op for a given message. 
        readOnly: <bool>

        # Use the writeOnly and readOnly flags to setup the ordering of who is expected to write or read
        # first when the websocket connection is created.
        
        # How the response data should be handled.
        # json: response data will attempted to be unmarshalled into a JSON object that can be validated.
        #       It will fallback to the binary json representation if the unmarshalling fails.
        # text: response data will be treated as plain text and wrapped in a json object for validation.
        #       This json object format follows { "payload": "<text response>" }.
        # binary: response data will be handled as a stream of data and the standard binary JSON response 
        #         will be provided for validation. This is useful for large responses that may not fit
        #         in memory.
        response: json | text | binary

        # Save the output of a binary response to a specific file path. This can then be passed into an
        # external validator to validate the binary contents.
        filePath: <string>
      - ...
      
    # If false, the next websocket enabled test will re-use the client from the last non-closed websocket test.
    # Set this to true if you want to force a new websocket session for the following test case
    close: <bool>

Multiple websocket writes can be performed within a single test case:

tests:
  - name: Testing websockets!
    description: Gonna make a websocket call
    route: ws://localhost:8080/echo
    websocket: true
    input:
      requests: 
          # first request sends a JSON object
        - type: text # set message type to text
          payload: 
            data: gonna send a json object
          response: json
          
          # second sends just plain text
        - type: text
          payload: just text
          response: text
          
          # Third sends plain text but expects binary data in the response
        - type: text
          payload: gimme a binary
          response: binary

          # fourth sends a binary message and expects a text response
        - type: binary
          # Set our payload encoding as hex for sake of embedding it in the test file.
          # The test client will decode the hex input and send the raw byte representation over the wire
          encoding: hex
          payload: 68656c6c6f2c2068657820776f726c64
          response: text

          # Using the read and write only flags to make this message a no-op
        - readOnly: true
          writeOnly: true
          payload: This is a no-op and nothing will be done

          # Send the standard output of a program as our websocket payload
        - type: text
          encoding: external
          payload: /bin/date
          args:
            - '-u'
            - '-R'

Validations

Each JSON data type has its own set of validation rules that can be applied. Some types have a short form available that support a more limited feature set of the regular validation definition. All short forms (other than strings) do not support variables from data store since the matcher type is derived from the value specified prior to the test execution.

Integers
payload:
  MyInteger:
    type: integer
    exists: <bool> # defaults to true
    matches: <matcher>

Supported matchers:

  • a specific integer value: e.g. -1, 0, 1, 2, 3, ...
  • The $any key word to match regardless of the numerical value
Short form

Only supports integer constant values.

payload:
  MyInteger: 42

# Is the same as
payload:
  MyInteger:
    type: integer
    exists: true
    matches: 42
Numbers
payload:
  MyNumber:
    type: number
    exists: <bool> # defaults to true
    matches: <matcher>

Supported matchers:

  • a specific numerical value: e.g. 1, 2, 3, 3.14, etc.
  • The $any key word to match regardless of the numerical value
Short form

Only supports numerical constant values.

payload:
  MyNumber: 42.1

# Is the same as
payload:
  MyNumber:
    type: number
    exists: true
    matches: 42.1
Booleans
payload:
  MyBool:
    type: bool
    exists: <bool> # defaults to true
    matches: <matcher>

Supported matchers:

  • true or false
  • The $any keyword to match regardless of the numerical value
Short form

Only supports boolean constant values.

payload:
  MyBool: true

# Is the same as
payload:
  MyBool:
    type: bool
    exists: true
    matches: true
Strings
payload:
  MyString:
    type: string
    exists: <bool> # defaults to true
    matches: <matcher>

Supported matchers:

  • any regular expression: e.g. ".*", "SomePartial.+", "[0-9]+"
  • The $any keyword to match any string (".*" expression)
  • The $notEmpty keyword to match non-empty strings (".+" expression)
Short form

Supports all string matchers.

payload:
  MyString: "meaning of life"

# Is the same as
payload:
  MyString:
    type: string
    exists: true
    matches: "meaning of life"

# Also supported
payload:
  MyString: <$any>|<$notEmpty>|<regexp>
Arrays
payload:
  MyArray:
    type: array
    length: <matcher>
    sorted: <bool> # defaults to true
    exists: <bool> # defaults to true
    items:
      - <sub validations>

Supported matchers (for length):

  • any integer value: e.g. 1, 2, 200, ...
  • The $notEmpty keyword
  • Length expressions like: $< 5, $<= 1, $> 1, $>= 500
Short form

Length matcher is fixed to $notEmpty.

payload:
  MyArray:
    - "test String"
    - type: object
      properties:
        Email: ".*@.*"

# Is the same as
payload:
  MyArray:
    type: array
    length: $notEmpty
    exists: true
    sorted: true
    items:
      - type: string
        matches: "test sttring"
      - type: object
        properties:
          Email:
            type: string
            matches: ".*@.*"

Array validation performs a length check independent of the item validations provided to it. This means you can validate that your array has returned a length of X but then only define a validator for the first element within it with the assumption that the rest of the items are the same.

For small arrays, you can validate that all the items are the same by simply repeating the validation for X number of expected items. E.g:

payload:
  MyArray:
    type: array
    length: 2
    items:
      # validate that each item returned is a string
      - type: string
        matches: $any
      - type: string
        matches: $any
Sorted Arrays

You can set the sorted property to false in the event that you are validating a large array response and are looking to seek out a specific item from it. This will have the validation perform a depth first search for the first node the validation matches on.

payload:
  MyArray:
    length: $notEmpty
    sorted: false
    items:
      # validate that array contains an object that has a Name property with `MySpecificPerson`
      - type: object
        properties:
          Name:
            type: string
            matches: MySpecificPerson

This mechanism can be used to 'pick' values from the response to put into the data store for use later. However, you need to ensure the most specific validation is executed first to locate the correct node. Otherwise you may get an unexpected node that the more general validation matched on first.

# Get the ID for MySpecificPerson and store it in the data store for a future test
payload:
  MyArray:
    length: $notEmpty
    sorted: false
    items:
      - type: object
        properties:
          Name:
            # Run this validation at a high priority than the ID since we are locating the ID based on the name
            priority: 0
            type: string
            matches: MySpecificPerson
          Id:
            # ID matcher is more generic (since we don't know the ID before hand). It needs to be executed AFTER
            # the name validation which will be used to locate the specific node in our unsorted response.
            priority: 1
            type: string
            matches: $any
            # subsequent tests can now reference the ID associated to 'MySpecificPerson' through the @{test_id} variable
            storeAs: test_id
Objects
payload:
  MyObject:
    type: object
    properties:
      FieldOne: <sub validation>

All objects are validated based on the keys present in the validations definition. If MyObject exists and the FieldOne property does not exist, a validation error will be raised.

Short form

No Short form is available for Object validations at this point in time. There is no way to determine whether an object will be a test definition or is part of the expected payload due to potential collisions with the type and property field on valid response objects.

Field Existence

If you want to validate that a field does not exist, you can add the exists property to your validation:

payload:
  MyObject:
    type: object
    properties:
      FieldOne:
        type: string
        matches: $any
      FieldTwo:
        type: string
        # if the FieldTwo key exists, this will raise a validation error
        exists: false
Response Code

Validating API response codes in the payload uses any valid integer validator.

# match 200 exactly
payload:
  code: 200

# match any non-500
payload:
  code:
    type: integer
    matches: '[1234][0-9]{2}'

# match any code
payload:
  code:
    type: integer
    matches: $any
Response Headers

You can define validations for response headers by defining your validators on the headers object of the response section in the test. All headers follow the format of Map[header key] -> []string

tests:
  - name: Get Charles ID
    description: Get the ID for user charles
    route:  https://reqres.in/api/users
    method: GET
    response:
      code: 200
      # ----------------------------------------------------------------
      # Make sure we're getting the right content type in our response
      headers:
        Content-Type:
          type: array
          length: $notEmpty
          items:
            - type: string
              matches: "application/json"
      # ----------------------------------------------------------------
      payload:
        data:
          type: array
          length: $notEmpty
          sorted: false
          items:
            - type: object
              properties:
                email:
                  priority: 0
                  type: string
                  matches: &charles_email 'charles.morris@reqres.in'
                id:
                  priority: 1
                  type: integer
                  matches: $any
                  storeAs: charles_id

You can clean up your header validators using the supported short forms for strings and arrays like so:

---
response:
  headers:
    Content-Type:
      - "application/json'
---
Binary Response Validation

You can write (limited) tests to validate binary specific response data. This is done by specifying binary:true in the response section of the test. The sha256 sum of the response data and its size in bytes are made available to matchers. Furthermore, the response can can be saved to a specific path on disk using the 'filePath' parameter which can then subsequently be used for future upload calls or external validation.

tests:
  - name: Test Download
    description: Make sure we download the expected file
    route: https://www.dundeecity.gov.uk/sites/default/files/publications/civic_renewal_forms.zip
    method: GET
    response:
      code: 200
      # set this to true so the binary payload can be formatted correctly for easy validation
      binary: true
      # response will be saved to this file
      filePath: /tmp/myfile.zip
      payload:
        # We can validate the size in bytes if it is known ahead of time
        size:
          type: integer
          matches: $any
        # and/or we can validate the sha256 sum of the data
        sha256sum:
          type: string
          matches: $any
        # This field is always made available in the event that the response is stored into a temp file
        # for tests expecting a JSON response but end up with a binary response.
        saved:
          type: string
          matches: /tmp/myfile.zip

If a call is made where non-binary data is expected but the response does contain binary data, the response will automatically fallback to the binary response format with some messages indicating the fallback was made. Your test will only fail if you had any validators defined on specific fields of the payload, otherwise you can continue to validate only the status code if you don't really care about the response.

  response: {
   "NOTICE": [
    "Unexpected non-JSON response was returned from this call triggering a fallback to its binary representation.",
    "Response data has been written to the path in the 'saved' field of this object."
   ],
   "saved": "/var/folders/47/56666ndd29n748s01t3f4cdr0000gn/T/binary-response-483156933",
   "sha256sum": "c8bceaae2017481d3e1fd5b47fc67d93f1e049c057461effabb9152b571d65b2",
   "size": 6615
  }
Websocket Response Validation

You can write tests to validate your websocket responses similar to how regular JSON and binary responses are validated. Since multiple writes/reads can happen in a given websocket test case, the response payload will contain array that correlates back to each websocket message object provided as an input.

# sample.yaml
# Running against the gorilla sample websocket server
# https://github.com/gorilla/websocket/blob/master/examples/echo/server.go
tests:
  - name: Testing websockets!
    description: Gonna make a websocket call
    route: ws://localhost:8080/echo
    websocket: true
    input:
      requests: 
          # first request sends a JSON object
        - type: text # set message type to text
          payload: 
            data: gonna send a json object
          response: json
          
          # second sends just plain text
        - type: text
          payload: just text
          response: text
          
          # Third sends plain text but expects binary data in the response
        - type: text
          payload: gimme a binary
          response: binary

          # fourth sends a binary message and expects a text response
        - type: binary
          # Set our payload encoding as hex for sake of embedding it in the test file.
          # The test client will decode the hex input and send the raw byte representation over the wire
          encoding: hex
          payload: 68656c6c6f2c2068657820776f726c64
          response: text
    response:
      payload:
        # Root object for all websocket responses
        responses:
          # validate the response of the first websocket text request
          - type: object
            properties:
              data: "gonna send a json object"
              
          # validate the response of the second websocket text request
          # even though we sent plain text and the server responded with plain text, it was wrapped 
          # in an object for validation purposes:
          # we sent: "just text" -> server echos "just text" -> test client wraps response {"payload": "just text"} -> validation
          - type: object
            properties:
              payload: "just text"
              
          # validate the binary response of the third websocket text request
          - type: object
            properties:
              size:
                type: integer
                matches: $any
              sha256sum: $any
          
          # the response from our binary sent data is just plain text
          - type: object
            properties:
              payload: 'hello, hex world'
Websocket Sessions

By default, a websocket connection will remain open in between test cases to preserve the same session for follow-up transactions. However, you can tell the test close the client to initiate a new session in a follow-up test by setting close: true in the test input.

The test client will automatically close after all tests in a test suite have been executed - no need to explicitly close the client on your last test.

Here's an example of session closing/sharing:

tests:
  - name: Testing websockets!
    description: Gonna make a websocket call
    route: ws://localhost:8080/echo
    websocket: true
    input:
      requests: 
        - payload: 
            object: gonna send a json object
          response: json
        - payload: just text
          response: text
        - payload: gimme a binary
          response: binary
          
      # Close the websocket client after this tests transactions have completed
      close: true          
    response:
      payload:
        responses:
          - type: object
            properties:
              object: "gonna send a json object"
          - type: object
            properties:
              payload: "just text"
          - type: object
            properties:
              size:
                type: integer
                matches: $any
              sha256sum: $any


  - name: Re-open the client
    description: New call with new client
    route: ws://localhost:8080/echo
    websocket: true
    input:
      requests: 
        - payload: 
            object: new client, new text
          response: json    
      # Close is not specified (defaults to false). The client created by THIS test will be used in the next
      # websocket test case.
    response:
      payload:
        responses:
          - type: object
            properties:
              object: $any
      
      
  - name: Re-use open client
    description: New call with old client
    route: ws://localhost:8080/echo
    websocket: true
    input:
      requests: 
        - payload: 
            object: old client, new text
          response: json     
      # Closing the websocket client connection created from the previous test. The next websocket test will create
      # a new client connection.
      close: true
    response:
      payload:
        responses:
          - type: object
            properties:
              object: $any
External Validator

You can specify an external validator if none of the above built in ones are sufficient for your test case. This is done by passing the response value to an external executable as a program parameter and validating the status code of the execution.

The main limitation is that all response values that are to be passed into external executables must be representable as strings. For binary responses, this can be achieved by using the built in mechanisms to save the binary data to a file and pass that file path into to the external validator.

Binary Data

Here is a modified form of the file download example that passes in the downloaded file to an external executable:

tests:
  - name: Test Download
    description: Make sure we download the expected file
    route: https://www.dundeecity.gov.uk/sites/default/files/publications/civic_renewal_forms.zip
    method: GET
    response:
      code: 200
      # set this to true so the binary payload can be formatted correctly for easy validation
      binary: true
      # response will be saved to this file
      filePath: /tmp/myfile.zip
      payload:
        # We can validate the size in bytes if it is known ahead of time
        size:
          type: integer
          matches: $any
        # and/or we can validate the sha256 sum of the data
        sha256sum:
          type: string
          matches: $any

        # This field stores the file path of our downloaded file, we can add the
        # external validator here!
        saved:
          type: external
          # what exit code we expect the program to return. The exit code will not be validated if this field is missing.
          returns: 0
          # we want to execute a python script that will do something with the
          # zip file and return a 0 exit code on success.
          bin: /usr/local/bin/python3
          args:
            - '@{TEST_DIR}/test.py'
            - /tmp/myfile.zip

Similarily with websockets:

tests:
  - name: Testing websockets!
    description: Gonna make a websocket call
    route: ws://localhost:8080/echo
    websocket: true
    input:
      requests: 
        - type: binary
          encoding: hex
          payload: 68656c6c6f2c2068657820776f726c64
          response: binary
          filePath: &ws_resp1 '/tmp/websocket_response_1'
    response:
      payload:
        responses:
          - type: object
            properties:
              saved:
                type: external
                bin: /usr/local/bin/python3
                args:
                  - '@{TEST_DIR}/test.py'
                  - *ws_resp1
Non-Binary Resopnse Data

For non-binary data that isn't being written to a file, there is no pre-determined string (like a filepath) that can be used to pass the value in as an argument to the external program. This can be solved with the storeAs field of the validator to set a data store variable with the value that can then be referenced in the arguments array.

Mirroring the above websocket test, lets modifiy it so it doesn't need to write any file. Suppose we call the server endpoint that echos everything we send back as base64, and now we have a script to decode the response and perform an equality check:

#!/usr/bin/env bash
# /tmp/checkBase64.sh
# Decodes the first base64 input and compares it with the second plain text input.

input="$1"
expected="$2"

result=$(echo "${input}" | base64 -d)

echo "Decoded: ${result}"
echo "Expected: ${expected}"

if [ "${result}" = "${expected}" ]; then
  # all good, return 0!
  exit 0
fi

# not equal, fail the validation!
exit 1

Our test case would look like this:

tests:
  - name: Base64 Echo Testing
    description: Make a websocket call that echos the input as base64

    # we are now calling an API that will echo everything back in base64
    route: ws://localhost:8080/base64
    websocket: true
    input:
      requests: 
        - type: text
          payload: Hello, world
          response: text
    response:
      payload:
        responses:
          - type: object
            properties:
              payload:
                type: external
                # store the response in the data store
                storeAs: encoded_response
                bin: '@{TEST_DIR}/checkBase64.sh'
                args:                  
                    # Now we can reference the value as a variable for our scripts input
                  - '@{encoded_response}'
                  - 'Hello, world'

Test Tags

Each test can have an array of arbitrary tags defined that can then be used filter test execution at runtime. This is useful for creating sets of tests that may be executed in one context but not another. These tags are defined in the tags field of the test definition like so:

# tests.yaml
tests:
  - name: Get Stuff
    description: This test performs a read only operation
    method: GET
    tags: 
      - read
      # create a local tag to identify a test that will work in a local environment
      - local
    ...

  - name: Write stuff
    description: This test performs a write operation
    method: POST
    tags:
      - write
    ...

Now suppose we want to execute our read-only tests:

# execute tests with tag 'read'
arp -file=./tests.yaml -tag=read

If we want to execute both read and write tests, we simply provide the tags as a comma separated list for a single argument. This will perform an OR check on the tags.

# execute tests with tag 'read' OR 'write' defined
arp -file=./tests.yaml -tag=read,write

We can add an additional -tag parameter if we want to only execute tests that are known to work in a local environment. This will perform an AND check with all the provided -tag inputs:

# execute tests with tags 'read' AND 'local' defined
arp -file=./tests.yaml -tag=read -tag=local

Combining these, we can now execute all read and write tests that work in our local environment:

# execute tests that have tags 'read' OR 'write', AND 'local' defined
arp -file=./tests.yaml -tag=read,write -tag=local

Data Storage

Each Test Suite has its own isolated data store that the tests can read and write variables to. Variables are read using @{myVarName} notation, and are saved using the storeAs property on your field matcher.

Variables can only be referenced in the following test definition fields:

  • input
tests:
  - name: Something
    input:
       api_token: @{MY_API_KEY}
---
  • headers
tests:
  - name: Something
    headers:
      SomeHeader: @{headerStuff}
---
  • route
tests:
  - name: Something
    route: '@{host}/user/@{user_id}'
---
  • validation matches fields as strings
---
payload:
  Name:
    type: string
    matches: '@{name}'
---

For example, if we wanted to store the ID of the user Charles from the sample test to use in a subsequent GET call. Our test would look like:

#sample.yaml
  - name: Get Charles ID
    description: Get the ID for user charles
    route:  https://reqres.in/api/users
    method: GET
    response:
      code: 200
      payload:
        data:
          type: array
          length: $notEmpty
          sorted: false
          items:
            - type: object
              properties:
                email:
                  # set email to a higher priority than ID so this matcher executes first to pick out our array element
                  priority: 0
                  type: string
                  # create an anchor here for the email so we don't have to type it out again
                  matches: &charles_email 'charles.morris@reqres.in'
                id:
                  # id matcher is set to a lower priority than email. This matcher is as generic as can be and can pick out the wrong node
                  # if evaluated first.
                  priority: 1
                  type: integer
                  matches: $any
                  # store the value in a variable called 'charles_id'
                  storeAs: charles_id
  - name: Get Charles Data
    description: Get the data for user Charles
    # Fetch the user using the ID we got from the previous test
    route: https://reqres.in/api/users/@{charles_id}
    method: GET
    response:
      code: 200
      payload:
        data:
          type: object
          properties:
            email:
              type: string
              # proof that we got the right user
              matches: *charles_email

DFS Store

Fixtures

The data store can be pre-populated with read only data prior to executing your tests with a file containing data definitions. This file should consist of a maps and arrays that terminate to a string value. E.g.

#fixtures.yaml
Hosts:
  Beta: 
    Local: http://localhost
    RealBeta: http://beta.NotLocalHost.com
  Prod: http://NotLocalHost.com
Search:
  - Bar
  - Bazz

# foo_test.yaml
tests:
  - name: List Foos by Bar
    # Access our host using the fixture mapping with dot ('.') notation
    # and index into our array using '[<index>]'
    route: '@{Hosts.Beta.Local}/foo?search=@{Search[0]}'
Environment Variables

The data store will also be pre-populated with your system's environment variables and can be access the same way as any other variable

# Env var
export MY_API_TOKEN="adfadfadfadfa"

# foo_test.yaml
tests:
  - name: List Foos
    input:
      # populate our apiToken input field with our environment variable value
      apiToken: @{MY_API_TOKEN}
    ...
Var Parameters

Alternatively, you can provide variables with one or more -var input parameters following the KEY=VALUE syntax:

./arp -test-root=. -var='MY_API_TOKEN=adfadfadfadfa' -var='SOMETHING_ELSE=not the token'
Variable Composition

Variables can also be composed of other variables, or in other words, you can have a variable in the name of a variable:

'@{Hosts.@{HOST_STAGE}}/foo'

Variables are resolved at run time starting from the most nested variable. If we had the following fixture file

# fixtures.yaml
Hosts:
    Beta: http://localhost
    Prod: http://NotLocalHost.com

And we set the HOST_STAGE to beta as an environment variable:

#Env Var
export HOST_STAGE="Beta"

@{Hosts.@{HOST_STAGE}}/foo would be resolved as follows:

  1. @{Hosts.@{HOST_STAGE}}/foo [@{HOST_STAGE} -> "Beta"] -> @{Hosts.Beta}/foo
  2. @{Hosts.Beta}/foo [@{Hosts.Beta} -> http://localhost] -> http://localhost/foo

Dynamic Inputs

The input properties of Test Cases also have the ability to use the output of an executed command as its value. This works similarly to the behavior of variables where it'll perform a straight value replacement (and recursive execution), but uses the syntax of $(<path> arg1 arg2 ...). The output is treated as a regular string and contains both the STDOUT and STDERR data. If the program encounters an error (program returns a non-zero exit code), the test case will not be executed.

For example, you can use /bin/date to provide the current date with your API call:

  input:
    date: '$(/bin/date -u -R)'

This syntax also supports the usage of variables. All variables are resolved prior to the execution of the command.

  input:
    something: '$(@{TEST_DIR}/myscript.sh)'

Or you can run some arbitrary shell command:

  input:
    result: '$(/bin/bash -c "echo \"hello, world\" | base64")'

Arp will recursively execute programs provided in the input starting with the inner most $(<command>) allowing for execution chaining like the following:

  input:
    result: '$(/bin/echo $(/bin/echo "first") $(/bin/echo "second"))'

The above example will execute in the following order:

  1. $(/bin/echo $(/bin/echo "first") $(/bin/echo "second")) -> [$(/bin/echo "first") -> first] -> $(/bin/echo first $(/bin/echo "second"))
  2. $(/bin/echo first $(/bin/echo "second")) -> [$(/bin/echo "second") -> second] -> $(/bin/echo first second)
  3. $(/bin/echo first second) -> "first second"

This unfortunately means that you won't be able to use subshells if you're looking to straight up embed a complicated bash command. In such a scenario it's recommended to put your command in an external script and invoke that script as a dynamic input.

This type of dynamic input is not recommended for providing large amounts of data as it will load the entire result in memory. For multi-part form and websockets requests, it's recommended to use their native binary or file operations that will allow for data streaming.

Interactive Mode

You can execute a test file in interactive mode by specifying your test file with the -file parameter along with -step.

Interactive Demo

Run Behavior

All Test Suites run in parallel with each other. The tests within each suite will run sequentially to force a linear dependency graph on storing and fetching variables in the data store. For most optimal performance, you can organize your tests in one of the following ways:

  1. ${test-root}/${api}.yaml - Good for short API calls that all flow into each other.
  2. ${test-root}/${api}/{action}.yaml - Good for separating tests with dependent calls from other tests with no dependencies within the same API scope

Pro-Tips:

Input Warnings

Data provided in a test case's input is resolved at test execution time and NOT while the config is parsed. Make sure to watch out for typos when dealing with websockets that use a mostly fixed form input.

Using Anchors

Our input is YAML and YAML supports anchors out of the box to reduce verbosity! Since the tests within yaml file are scoped under the 'tests' key, you can create arbitrary keys for anchoring elsewhere in your test file.

This can be useful for creating your own 'short forms' for types without sacrificing the configurability on them.

# foo_test.yaml

# common headers to pass with every request
Headers: &headers
  SomeHeader: MyHeader

# some common regular expression
# ISO date field
DateField: &date 
  type: string
  matches: '^(\d{4})-0?(\d+)-0?(\d+)[T ]0?(\d+):0?(\d+):0?(\d+)$' 

AnyString: &any-string
  type: string
  matches: $any

AnyInteger: &not500
  type: integer
  matches: '[234][0-9]{2}'


Input: &default-input
  apiToken: @{API_KEY}

# tests is the only reserved key in our file
tests:  
  - name: List Foos
    description: Should list all Foos with a Name ending with 'Bar'
    method: GET
    route: 'http://localhost/foo?search=Bar'
    headers:
      <<: *headers
      NewHeader: something not common
    input:
      <<: *default-input
    response:
      code: *not500
      # expecting a payload of {Foo: [{Date: "FooBar"}, "any string"]}
      payload:
        Foo:
          type: array
          length: 1
          items:
            - type: object
              properties:
                Date: *date
            - *any-string

The fixture and test files are concatenated together on load making anchors within the fixtures file available for reference in your test cases as well. This can be used to provide common anchors that can be used across all test files without having to redefine them in each test file.

Documentation

Index

Constants

View Source 
const (
	Any      = "$any"
	NotEmpty = "$notEmpty"
	LT       = "$<"
	LTE      = "$<="
	GT       = "$>"
	GTE      = "$>="
	EQ       = "$="

	// special keywords used in validation object definitions
	TEST_KEY_TYPE       = "type"
	TEST_KEY_PROPERTIES = "properties"
	TEST_KEY_LENGTH     = "length"
	TEST_KEY_ITEMS      = "items"
	TEST_KEY_SORTED     = "sorted"
	TEST_KEY_STORE      = "storeAs"
	TEST_KEY_PRIORITY   = "priority"
	TEST_KEY_MATCHES    = "matches"
	TEST_KEY_EXISTS     = "exists"

	TEST_EXEC_KEY_RETURN_CODE = "returns"
	TEST_EXEC_KEY_BIN_PATH    = "bin"
	TEST_EXEC_KEY_ARGS        = "args"

	ValueErrFmt            = "Expected value '%v' did not match the actual value '%v'"
	PatternErrFmt          = "Failed to match actual value '%v' with expected pattern: '%v'"
	NotEmptyErrFmt         = "Expected non-empty value, but got value '%v' instead."
	ArrayLengthErrFmt      = "Expected array with length %v %v but found length %v instead."
	ReceivedNullErrFmt     = "Received null value when non-null value was expected"
	ExpectedNullErrFmt     = "Expected null value when non-null value was returned"
	MalformedDefinitionFmt = "\nMalformed '%v' field detected on %v"
	MismatchedMatcher      = "Test expected a value type matching '%v' but response field is of type '%v'."
	BadVarMatcherFmt       = "Failed to resolve variable within matcher: %v"
	BadArrayElementFmt     = "\nExpected elements on '%v' to be objects"
	BadObjectFmt           = "\nExpected property '%v' to map to an object"

	// available field matchers
	TYPE_INT   = "integer"
	TYPE_NUM   = "number"
	TYPE_STR   = "string"
	TYPE_ARRAY = "array"
	TYPE_OBJ   = "object"
	TYPE_BOOL  = "bool"
	TYPE_EXEC  = "external"
)
View Source 
const (
	MissingDSKeyFmt   = "Attempted to retrieve data from data store that does not exist: key: %v"
	BadIndexDSFmt     = "Attempted to index into a data store value with a non-positive or non-integer value: %v"
	IndexExceedsDSFmt = "Index for data store value exceeds its max length: %v"
	StatusCodePath    = "response.StatusCode"
	HeadersPath       = "response.Header"
)
View Source 
const (
	// Test Config keys
	CFG_SKIP          = "skip"
	CFG_TAGS          = "tags"
	CFG_RESPONSE_CODE = "code"

	// Mime types
	MIME_JSON = "application/json"
	MIME_TEXT = "text/plain"

	//Headers
	HEADER_CONTENT_TYPE = "Content-Type"

	// MISC
	RESPONSE_PATH_FMT = "binary-response-*"

	//DataStore Vars
	DS_WS_CLIENT = "ws"
)
View Source 
const (
	WS_ENC_BASE64   = "base64gzip"
	WS_ENC_HEX      = "hex"
	WS_ENC_FILE     = "file"
	WS_ENC_EXTERNAL = "external"

	WS_MSG_TEXT = "text"
	WS_MSG_JSON = "json"
	WS_MSG_BIN  = "binary"
)

Variables

This section is empty.

Functions

func Base64GzipToByteReader 

func Base64GzipToByteReader(input string) (io.ReadCloser, error)

func ExecuteCommand 

func ExecuteCommand(input string) (interface{}, error)

func IndentStr 

func IndentStr(level int) string

func ObjectPrintf 

func ObjectPrintf(message string, obj interface{}) string

func PrintIndentedLn 

func PrintIndentedLn(indentLevel int, format string, args ...interface{})

func PrintReport 

func PrintReport(opts ReportOptions, passed bool, testingDuration time.Duration, results []MultiSuiteResult)

func PrintSingleTestReport 

func PrintSingleTestReport(opts ReportOptions, test *TestResult)

func PrintYamlObj 

func PrintYamlObj(object interface{}) (string, error)

func RecursiveExecuteCommand 

func RecursiveExecuteCommand(input interface{}) (interface{}, error)

Iterate through an object and execute any command strings that are located. Returns the input object with the command strings expanded to their results

func ShouldShowReport 

func ShouldShowReport(opts ReportOptions, test *TestResult) bool

func ToJsonObj 

func ToJsonObj(obj interface{}) (map[string]interface{}, error)

func YamlToJson 

func YamlToJson(i interface{}) interface{}

Types

type ArrayMatcher 

type ArrayMatcher struct {
	Length    *int64
	LengthStr *string
	Items     []interface{}
	ErrorStr  string
	Exists    bool
	DSName    string
	Sorted    bool
	Priority  int
}

func (*ArrayMatcher) Error 

func (m *ArrayMatcher) Error() string

func (*ArrayMatcher) GetPriority 

func (m *ArrayMatcher) GetPriority() int

func (*ArrayMatcher) Match 

func (m *ArrayMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*ArrayMatcher) Parse 

func (m *ArrayMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*ArrayMatcher) SetError 

func (m *ArrayMatcher) SetError(error string)

type BinResponseJson 

type BinResponseJson struct {
	Saved     string   `json:"saved"`
	Notice    []string `json:"NOTICE,omitempty"`
	Size      uint64   `json:"size"`
	SHA256Sum string   `json:"sha256sum"`
}

func (*BinResponseJson) GenericJSON 

func (bj *BinResponseJson) GenericJSON() map[string]interface{}

type BoolMatcher 

type BoolMatcher struct {
	Value    *bool
	Pattern  *string
	ErrorStr string
	Exists   bool
	DSName   string
	Priority int
}

func (*BoolMatcher) Error 

func (m *BoolMatcher) Error() string

func (*BoolMatcher) GetPriority 

func (m *BoolMatcher) GetPriority() int

func (*BoolMatcher) Match 

func (m *BoolMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*BoolMatcher) Parse 

func (m *BoolMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*BoolMatcher) SetError 

func (m *BoolMatcher) SetError(error string)

type ByteCountWriter 

type ByteCountWriter struct {
	ByteCount uint64
}

func (*ByteCountWriter) Write 

func (w *ByteCountWriter) Write(b []byte) (int, error)

type Colorizer 

type Colorizer struct {
	Enabled bool
}

func (*Colorizer) BrightBlue 

func (c *Colorizer) BrightBlue(input string) string

func (*Colorizer) BrightCyan 

func (c *Colorizer) BrightCyan(input string) string

func (*Colorizer) BrightGrey 

func (c *Colorizer) BrightGrey(input string) string

func (*Colorizer) BrightRed 

func (c *Colorizer) BrightRed(input string) string

func (*Colorizer) BrightWhite 

func (c *Colorizer) BrightWhite(input string) string

func (*Colorizer) BrightYellow 

func (c *Colorizer) BrightYellow(input string) string

func (*Colorizer) Cyan 

func (c *Colorizer) Cyan(input string) string

func (*Colorizer) Green 

func (c *Colorizer) Green(input string) string

func (*Colorizer) Red 

func (c *Colorizer) Red(input string) string

func (*Colorizer) Underline 

func (c *Colorizer) Underline(input string) string

func (*Colorizer) Yellow 

func (c *Colorizer) Yellow(input string) string

type CommandStack 

type CommandStack struct {
	Frames []CommandStackFrame
	Extra  string
}

func (*CommandStack) Pop 

func (s *CommandStack) Pop() *CommandStackFrame

func (*CommandStack) Push 

func (s *CommandStack) Push(f CommandStackFrame)

type CommandStackFrame 

type CommandStackFrame struct {
	StartPos    int
	EndPos      int
	CommandLine string
	Nested      int
}

func (*CommandStackFrame) IsValid 

func (f *CommandStackFrame) IsValid() bool

type DataStore 

type DataStore map[string]interface{}

func (*DataStore) ExpandVariable 

func (t *DataStore) ExpandVariable(input string) (interface{}, error)

type DepthMatchResponse 

type DepthMatchResponse struct {
	FoundNode DepthMatchResponseNode
	NodeChain []*DepthMatchResponseNode
}

type DepthMatchResponseNode 

type DepthMatchResponseNode struct {
	Status         bool
	Node           interface{}
	NodePath       string
	MatchedNodeKey bool
}

type ExecutableMatcher 

type ExecutableMatcher struct {
	ReturnCode *int
	BinPath    string
	PrgmArgs   []string
	ErrorStr   string
	Exists     bool
	DSName     string
	Priority   int
}

func (*ExecutableMatcher) Error 

func (m *ExecutableMatcher) Error() string

func (*ExecutableMatcher) GetPriority 

func (m *ExecutableMatcher) GetPriority() int

func (*ExecutableMatcher) Match 

func (m *ExecutableMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*ExecutableMatcher) Parse 

func (m *ExecutableMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*ExecutableMatcher) SetError 

func (m *ExecutableMatcher) SetError(error string)

type FieldMatcher 

type FieldMatcher interface {
	GetPriority() int
	Parse(parentNode interface{}, node map[interface{}]interface{}) error
	Match(field interface{}, datastore *DataStore) (bool, DataStore, error)
	Error() string
	SetError(error string)
}

type FieldMatcherConfig 

type FieldMatcherConfig struct {
	Matcher       FieldMatcher
	ObjectKeyPath FieldMatcherPath
}

type FieldMatcherPath 

type FieldMatcherPath struct {
	Keys           []FieldPathKey
	IsArrayElement bool
	Sorted         bool
	IsExecutable   bool
}

func (*FieldMatcherPath) GetParentPath 

func (f *FieldMatcherPath) GetParentPath() string

func (*FieldMatcherPath) GetPath 

func (f *FieldMatcherPath) GetPath() string

type FieldMatcherResult 

type FieldMatcherResult struct {
	Status          bool
	ObjectKeyPath   string
	Error           string
	ShowExtendedMsg bool
	IgnoreResult    bool
}

type FieldPathKey 

type FieldPathKey struct {
	Key          string
	IsArrayIndex bool
	IsObjectRoot bool
}

type FloatMatcher 

type FloatMatcher struct {
	Value    *float64
	Pattern  *string
	Exists   bool
	ErrorStr string
	DSName   string
	Priority int
}

func (*FloatMatcher) Error 

func (m *FloatMatcher) Error() string

func (*FloatMatcher) GetPriority 

func (m *FloatMatcher) GetPriority() int

func (*FloatMatcher) Match 

func (m *FloatMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*FloatMatcher) Parse 

func (m *FloatMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*FloatMatcher) SetError 

func (m *FloatMatcher) SetError(error string)

type InputReader 

type InputReader struct {
	FormWriter *multipart.Writer
	BodyReader io.Reader
	ErrorChan  chan error
}

type IntegerMatcher 

type IntegerMatcher struct {
	Value    *int64
	Pattern  *string
	Exists   bool
	ErrorStr string
	DSName   string
	Priority int
}

func (*IntegerMatcher) Error 

func (m *IntegerMatcher) Error() string

func (*IntegerMatcher) GetPriority 

func (m *IntegerMatcher) GetPriority() int

func (*IntegerMatcher) Match 

func (m *IntegerMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*IntegerMatcher) Parse 

func (m *IntegerMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*IntegerMatcher) SetError 

func (m *IntegerMatcher) SetError(error string)

type MultiSuiteResult 

type MultiSuiteResult struct {
	Passed      bool
	Error       error
	TestResults SuiteResult
	TestFile    string
}

type MultiTestSuite 

type MultiTestSuite struct {
	Suites  map[string]*TestSuite
	Verbose bool
}

func NewMultiSuiteTest 

func NewMultiSuiteTest(testDir string, fixtures string) (*MultiTestSuite, error)

func (*MultiTestSuite) ExecuteTests 

func (t *MultiTestSuite) ExecuteTests(threads int, testTags []string) (bool, []MultiSuiteResult, time.Duration, error)

func (*MultiTestSuite) LoadTests 

func (t *MultiTestSuite) LoadTests(testDir string, fixtures string) error

type NodeCacheObj 

type NodeCacheObj struct {
	Node      interface{}
	PathIndex int
}

type ObjectMatcher 

type ObjectMatcher struct {
	Properties map[interface{}]interface{}
	ErrorStr   string
	Exists     bool
	DSName     string
	Sorted     bool
	Priority   int
}

func (*ObjectMatcher) Error 

func (m *ObjectMatcher) Error() string

func (*ObjectMatcher) GetPriority 

func (m *ObjectMatcher) GetPriority() int

func (*ObjectMatcher) Match 

func (m *ObjectMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*ObjectMatcher) Parse 

func (m *ObjectMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*ObjectMatcher) SetError 

func (m *ObjectMatcher) SetError(error string)

type ReportOptions 

type ReportOptions struct {
	ShortErrors        bool
	Short              bool
	Tiny               bool
	AlwaysPrintHeaders bool
	ErrorsOnly         bool
	TestsPath          string
	Colors             Colorizer
	// Any failures while report is printed are suppresed and and indication
	// is provided that the result data may be incomplete
	InProgress bool
}

type ResponseMatcher 

type ResponseMatcher struct {
	DS     *DataStore
	Config []*FieldMatcherConfig
}

func (*ResponseMatcher) Match 

func (r *ResponseMatcher) Match(response interface{}) (bool, []*FieldMatcherResult, error)

Match Validates our test pattern against the actual JSON response

func (*ResponseMatcher) SortConfigs 

func (r *ResponseMatcher) SortConfigs()

type StringMatcher 

type StringMatcher struct {
	Value    *string
	ErrorStr string
	Exists   bool
	DSName   string
	Priority int
}

func (*StringMatcher) Error 

func (m *StringMatcher) Error() string

func (*StringMatcher) GetPriority 

func (m *StringMatcher) GetPriority() int

func (*StringMatcher) Match 

func (m *StringMatcher) Match(responseValue interface{}, datastore *DataStore) (bool, DataStore, error)

func (*StringMatcher) Parse 

func (m *StringMatcher) Parse(parentNode interface{}, node map[interface{}]interface{}) error

func (*StringMatcher) SetError 

func (m *StringMatcher) SetError(error string)

type SuiteResult 

type SuiteResult struct {
	Results  []*TestResult
	Passed   int
	Failed   int
	Total    int
	Duration time.Duration
}

type TestCase 

type TestCase struct {
	Config                TestCaseCfg
	IsRPC                 bool
	ResponseHeaderMatcher ResponseMatcher
	StatusCodeMatcher     ResponseMatcher
	ResponseMatcher       ResponseMatcher
	GlobalDataStore       *DataStore
	Tags                  map[string]bool
}

func (*TestCase) CloseWebsocket 

func (t *TestCase) CloseWebsocket()

func (*TestCase) Execute 

func (t *TestCase) Execute(testTags []string) (passed bool, result *TestResult, err error)

func (*TestCase) GetResolvedTestInput 

func (t *TestCase) GetResolvedTestInput() (interface{}, error)

Returns a new input object with all included variables resolved

func (*TestCase) GetRestInput 

func (t *TestCase) GetRestInput(input interface{}) (*InputReader, error)

func (*TestCase) GetTestHeaders 

func (t *TestCase) GetTestHeaders(inputReader *InputReader) (map[interface{}]interface{}, error)

func (*TestCase) GetTestRoute 

func (t *TestCase) GetTestRoute() (string, error)

func (*TestCase) GetTestRpcAddr 

func (t *TestCase) GetTestRpcAddr() (string, error)

func (*TestCase) GetWebsocketClient 

func (t *TestCase) GetWebsocketClient() (*websocket.Conn, string, error)

func (*TestCase) GetWebsocketInput 

func (t *TestCase) GetWebsocketInput(input interface{}) (*WSInput, error)

func (*TestCase) HasTag 

func (t *TestCase) HasTag(tagList string) bool

tag string can contain 1 or more tags separated by ",". This syntax will OR the tags.

func (*TestCase) LoadConfig 

func (t *TestCase) LoadConfig(test *TestCaseCfg) error

func (*TestCase) SkipTestOnTags 

func (t *TestCase) SkipTestOnTags(testTags []string) bool

func (*TestCase) StepExecWebsocket 

func (t *TestCase) StepExecWebsocket(step int, result *TestResult) (passed bool, remaining int, err error)

func (*TestCase) ValidateGeneric 

func (t *TestCase) ValidateGeneric(response map[string]interface{}) (bool, []*FieldMatcherResult, error)

func (*TestCase) ValidateREST 

func (t *TestCase) ValidateREST(statusCode int, response map[string]interface{}, headers map[string]interface{}) (bool, []*FieldMatcherResult, error)

type TestCaseCfg 

type TestCaseCfg struct {
	Name        string                      `yaml:"name"`
	Description string                      `yaml:"description"`
	ExitOnRun   bool                        `yaml:"exit"`
	Skip        bool                        `yaml:"skip"`
	Input       map[interface{}]interface{} `yaml:"input"`
	FormInput   bool                        `yaml:"formInput"`
	Tags        []string                    `yaml:"tags"`
	Headers     map[interface{}]interface{} `yaml:"headers"`
	Route       string                      `yaml:"route"`
	Method      string                      `yaml:"method"`
	RPC         TestCaseRpcCfg              `yaml:"rpc"`
	Websocket   bool                        `yaml:"websocket"`
	Response    TestCaseResponseCfg         `yaml:"response"`
}

type TestCaseResponseCfg 

type TestCaseResponseCfg struct {
	// status code could end up being either a number or an object defining a validation definition
	StatusCode interface{}                 `yaml:"code"`
	IsBinary   bool                        `yaml:"binary"`
	FilePath   string                      `yaml:"filePath"`
	Payload    map[interface{}]interface{} `yaml:"payload"`
	Headers    map[interface{}]interface{} `yaml:"headers"`
}

type TestCaseRpcCfg 

type TestCaseRpcCfg struct {
	Protocol  string `yaml:"protocol"`
	Address   string `yaml:"address"`
	Procedure string `yaml:"procedure"`
}

type TestResult 

type TestResult struct {
	TestCase        TestCase
	Fields          []*FieldMatcherResult
	Passed          bool
	Response        map[string]interface{}
	ResponseHeaders map[string]interface{}
	RequestHeaders  http.Header
	ResolvedRoute   string
	StatusCode      int
	StartTime       time.Time
	EndTime         time.Time
}

type TestSuite 

type TestSuite struct {
	Tests           []*TestCase
	GlobalDataStore DataStore
	Verbose         bool
}

func NewTestSuite 

func NewTestSuite(testFile string, fixtures string) (*TestSuite, error)

func (*TestSuite) Close 

func (t *TestSuite) Close()

func (*TestSuite) ExecuteTests 

func (t *TestSuite) ExecuteTests(testTags []string) (bool, SuiteResult, error)

func (*TestSuite) InitializeDataStore 

func (t *TestSuite) InitializeDataStore(fixtures string) error

func (*TestSuite) LoadFixtures 

func (t *TestSuite) LoadFixtures(fixtures string) (map[string]interface{}, error)

func (*TestSuite) LoadTests 

func (t *TestSuite) LoadTests(testFile string, fixtures string) (bool, error)

func (*TestSuite) ReloadFile 

func (t *TestSuite) ReloadFile(testFile string, fixtures string) (bool, error)

type TestSuiteCfg 

type TestSuiteCfg struct {
	Tests []TestCaseCfg `yaml:"tests"`
}

type VarStack 

type VarStack struct {
	Frames []VarStackFrame
	Extra  string
}

func (*VarStack) Pop 

func (s *VarStack) Pop() *VarStackFrame

func (*VarStack) Push 

func (s *VarStack) Push(f VarStackFrame)

type VarStackFrame 

type VarStackFrame struct {
	StartPos int
	EndPos   int
	VarName  string
	Nested   int
}

func (*VarStackFrame) IsValid 

func (f *VarStackFrame) IsValid() bool

type WSInput 

type WSInput struct {
	Requests []WSMessage `yaml:"requests" json:"requests"`
	Close    bool        `yaml:"close" json:"close"`
}

type WSMessage 

type WSMessage struct {
	Payload     interface{} `yaml:"payload" json:"payload"`
	Args        []string    `yaml:"args" json:"args"`
	WriteOnly   bool        `yaml:"WriteOnly" json:"WriteOnly"`
	ReadOnly    bool        `yaml:"readOnly" json:"readOnly"`
	Response    string      `yaml:"response" json:"response"`
	MessageType string      `yam:"type" json:"type"`
	Encoding    string      `yaml:"encoding" json:"encoding"`
	FilePath    string      `yaml:"filePath" json:"filePath"`
}

type WsResponseJson 

type WsResponseJson struct {
	Responses []map[string]interface{} `json:"responses"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
arp

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.