replicant

command module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2020 License: Apache-2.0 Imports: 3 Imported by: 0

README

Replicant

Go Report Card GoDoc Docker Cloud Automated build

Replicant is a synthetic testing service named after the bioengineered androids from Blade Runner. (all synthetics came from Blade Runner :)

It allows web application testing using chromedp, and api application testing using Go or Javascript. Provides a test manager, execution scheduler, api and facilities for emitting result data to external systems.

Status

Under heavy development and API changes are expected. Please file an issue if anything breaks.

Runing replicant

The replicant binary packs all functionality needed to run the server, executor and run local execution of tests for development or CI/CD purposes.

Locally for test development purposes
/path/to/replicant run --file api-test.yaml

If running locally from with the replicant binary a local chrome web browser with the development protocol can be specified:

/path/to/replicant run --chrome-remote-url http://127.0.0.1:9222  --file web-test.yaml

To have the local chrome browser started with the developer protocol enabled:

/path/to/chrome --remote-debugging-port=9222
Configuration options

Please see:

/path/to/replicant --help
Replicant server and executor locally with docker

The unbabel/replicant docker image packs everything needed to run and manage tests for both web apps and APIs. See the example docker-compose.yaml for more information.

docker stack deploy -c $PWD/docker-compose.yaml replicant

This will deploy the replicant server and 2 replicant executor nodes for web tests.

Web application testing

Web application testing support is based on the FQL (Ferret Query Language), documentation.

Test definition (can be also in json format)
POST http://127.0.0.1:8080/api/v1/run
content-type: application/yaml

name: duckduckgo-web-search
driver: web
schedule: '@every 60s'
timeout: 50s
retry_count: 2
inputs:
  url: "https://duckduckgo.com"
  user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36"
  timeout: 5000000
  text: "blade runner"
metadata:
  transaction: website-search
  application: duckduckgo
  environment: production
  component: website
script: |
  LET doc = DOCUMENT('{{ index . "url" }}', { driver: "cdp", userAgent: "{{ index . "user_agent" }}"})
  INPUT(doc, '#search_form_input_homepage', "{{ index . "text" }}")
  CLICK(doc, '#search_button_homepage')
  WAIT_NAVIGATION(doc)
  LET result = ELEMENT(doc, '#r1-0 > div > div.result__snippet.js-result-snippet').innerText
  RETURN {
    failed: result == "",
    message: "search result",
    data: result,
  }
Response
{
  "data": [
    {
      "uuid": "01DSSR5GH2BPX4G5FFCEVPEBKK",
      "name": "duckduckgo-web-search",
      "driver": "web",
      "failed": true,
      "message": "",
      "data": "",
      "time": "2019-11-16T09:19:39.554976Z",
      "metadata": {
        "application": "duckduckgo",
        "component": "website",
        "environment": "production",
        "transaction": "website-search"
      },
      "retry_count": 0,
      "with_callback": false,
      "duration_seconds": 6.967938203,
      "error": "operation timed out: WAIT_NAVIGATION(doc) at 4:0"
    }
  ]
}
API testing
Using the javascript driver

The following API is exposed by the javascript driver in order to perform HTTP calls and logging:

  • replicant.Log(string) log messages from the javascript test on the replicant server log.

  • replicant.NewResult() create a new response object to be returned as a result of the test, which should be modified accordingly to reflect the test result. The response must be returned as a serialized JSON object by calling its bounded method Response.JSON, E.g. return response.JSON().

Result type attributes:

{
		Data: "",
		Message: "",
		Failed: false,
}
  • replicant.http.NewRequest() creates a new HTTP request object for performing HTTP calls.

HttpRequest attributes:

{
		URL: "",
		Method: "",
		Body: "",
		Header: {},
		Params: {},
		FormData: {},
		SSLSkipVerify: false,
  • `replicant.http.Do(HttpRequest) performs a HTTP request and returns its response.

HttpResponse attributes:

{
	Status: ""
	StatusCode: 200
	Protocol: ""
	Body: ""
	Header: {}
	Error: ""
}
Test definition (can be also in JSON format)
POST http://127.0.0.1:8080/api/v1/run
content-type: application/yaml

name: duckduckgo-api-search
driver: javascript
schedule: '@every 60s'
timeout: 60s
retry_count: 2
inputs:
  url: "https://api.duckduckgo.com"
  text: "blade runner"
metadata:
  transaction: api-search
  application: duckduckgo
  environment: production
  component: api
script: |
  function Run(ctx) {
    req = replicant.http.NewRequest()
    req.URL = "{{ index . "url" }}"
    req.Params.q = "{{ index . "text" }}"
    req.Params.format = "json"
    req.Params.no_redirect = "1"
    resp = replicant.http.Do(req)
    data = JSON.parse(resp.Body)
    rr = replicant.NewResponse()
    switch(data.RelatedTopics && data.RelatedTopics.length > 0) {
      case true:
        rr.Data = data.RelatedTopics[0].Text
        rr.Message = resp.Status
        rr.Failed = false
        break
      case false:
        rr.Data = JSON.stringify(data)
        rr.Message = resp.Status
        rr.Failed = true
        break
    }
    return rr.JSON()
  }
Using the Go driver

Standard Go code can be used to create tests using following rules:

  • The package name must be transaction
  • The test function must implement the following signature: func Run(ctx context.Context) (message string, data string, err error).

Keep in mind that unlike the javascript driver which doesn't expose any I/O or lower level functionality for accessing the underlying OS, the Go driver currently exposes all of the Go standard library. Only use this driver if you are absolutely sure of what you are doing. This is planned to change in the future.

Test definition (can be also in JSON format)
POST http://127.0.0.1:8080/api/v1/run
content-type: application/yaml

name: duckduckgo-api-search
driver: go
schedule: '@every 60s'
timeout: 60s
retry_count: 2
inputs:
  url: "https://api.duckduckgo.com/"
  text: "blade runner"
metadata:
  transaction: api-search
  application: duckduckgo
  environment: production
  component: api
script: |
  package transaction
  import "bytes"
  import "context"
  import "fmt"
  import "net/http"
  import "io/ioutil"
  import "net/http"
  import "regexp"
  func Run(ctx context.Context) (m string, d string, err error) {
    req, err := http.NewRequest(http.MethodGet, "{{ index . "url" }}", nil)
      if err != nil {
        return "request build failed", "", err
    }
    req.Header.Add("Accept-Charset","utf-8")
    q := req.URL.Query()
    q.Add("q", "{{ index . "text" }}")
    q.Add("format", "json")
    q.Add("pretty", "1")
    q.Add("no_redirect", "1")
    req.URL.RawQuery = q.Encode()
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
      return "failed to send request", "", err
    }
    buf, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      return "failed to read response", "", err
    }
    rx, err := regexp.Compile(`"Text"\s*:\s*"(.*?)"`)
    if err != nil {
      return "failed to compile regexp", "", err
    }
    s := rx.FindSubmatch(buf)
    if len(s) < 2 {
      return "failed to find data", "", fmt.Errorf("no match")
    }
    return "search result", fmt.Sprintf("%s", s[1]), nil
  }
Response
{
  "data": [
    {
      "uuid": "01DSSR7ST5Q1Y2Y7HDSQDNS7Y7",
      "name": "duckduckgo-api-search",
      "driver": "go",
      "failed": false,
      "message": "search result",
      "data": "Blade Runner A 1982 American neo-noir science fiction film directed by Ridley Scott, written by Hampton...",
      "time": "2019-11-16T09:20:54.597852Z",
      "metadata": {
        "application": "duckduckgo",
        "component": "api",
        "environment": "production",
        "transaction": "api-search"
      },
      "retry_count": 0,
      "with_callback": false,
      "duration_seconds": 0.486582328,
      "error": ""
    }
  ]
}

API

Method Resource Action
POST /v1/transaction Add a managed transaction
GET /v1/transaction Get all managed transaction definitions
GET /v1/transaction/:name Get a managed transaction definition by name
DELETE /v1/transaction/:name Remove a managed transaction
POST /v1/run Run an ad-hoc transaction
POST /v1/run/:name Run a managed transaction by name
GET /v1/result Get all managed transaction last execution results
GET /v1/result/:name Get the latest result for a managed transaction by name
GET /metrics Get metrics (prometheus emitter must be enabled)
GET /debug/pprof Get available runtime profile data (debug enabled)
GET /debug/pprof/:profile Get profile data (for pprof, debug enabled)

TODO

  • Tests
  • Developer and user documentation
  • Add support for more conventional persistent stores
  • Vault integration for secrets (inputs)
  • Architecture and API documentation
  • Javascript driver transaction support

Acknowledgements

Contact

Bruno Moura brunotm@gmail.com

License

Replicant source code is available under the Apache Version 2.0 License

Documentation

The Go Gopher

There is no documentation for this package.

Source Files

View all Source files

Directories

Path Synopsis
Package api implements the replicant API handlers.
 Package api implements the replicant API handlers.
Package cmd implements replicant commands
 Package cmd implements replicant commands
go
Package gd implements a Go transaction driver.
 Package gd implements a Go transaction driver.
javascript
Package javascript implements a javascript transaction driver.
 Package javascript implements a javascript transaction driver.
web
Package web implements a web transaction driver based on chromedp/FQL.
 Package web implements a web transaction driver based on chromedp/FQL.
emitter
elasticsearch
Package elasticsearch implements a result emitter for elasticsearch.
 Package elasticsearch implements a result emitter for elasticsearch.
prometheus
Package prometheus implements a result exporter for prometheus.
 Package prometheus implements a result exporter for prometheus.
stdout
Package stdout implements a result exporter the standard output.
 Package stdout implements a result exporter the standard output.
internal
cmdutil
Package server implements utilities for working with spf13/cobra commands
 Package server implements utilities for working with spf13/cobra commands
executor
Package executor implements the replicant transaction execution service
 Package executor implements the replicant transaction execution service
scheduler
Package scheduler implements a task scheduler for transactions.
 Package scheduler implements a task scheduler for transactions.
tmpl
Package tmpl implements transaction template utils.
 Package tmpl implements transaction template utils.
webhook
Package webhook implements webhook based response listener.
 Package webhook implements webhook based response listener.
xz
Package xz implements internal utilities that mimics standard library counterparts.
 Package xz implements internal utilities that mimics standard library counterparts.
Package log implements a common structured logger.
 Package log implements a common structured logger.
Package manager implements a transaction manager for running and maintaining test transactions.
 Package manager implements a transaction manager for running and maintaining test transactions.
Package server implements the replicant server API.
 Package server implements the replicant server API.
Package store implements a simple interface and registry for transaction stores.
 Package store implements a simple interface and registry for transaction stores.
leveldb
Package leveldb implements a simple leveldb transaction store.
 Package leveldb implements a simple leveldb transaction store.
memory
Package memory implements a simple in-memory transaction store.
 Package memory implements a simple in-memory transaction store.
s3
Package s3 provides implementation of the storage layer under AWS S3 service.
 Package s3 provides implementation of the storage layer under AWS S3 service.
tests
Package tests implements a test suite for store drivers.
 Package tests implements a test suite for store drivers.
Package transaction implements the main types for defining, running and emitting replicant test data.
 Package transaction implements the main types for defining, running and emitting replicant test data.
callback
Package callback implements a facilities for listening to asynchronous test responses.
 Package callback implements a facilities for listening to asynchronous test responses.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.