replicant

README

Replicant is a synthetic transaction execution framework named after the bioengineered androids from Blade Runner. (all synthetics came from Blade Runner :)

It defines a common interface for transactions and results, provides a transaction manager, execution scheduler, api and facilities for emitting result data to external systems.

Status

Under heavy development and API changes are expected. Please file an issue if anything breaks.

Requirements

• Go 1.13
• External URL for API tests that require webhook based callbacks
• Chrome with remote debugging (CDP) either in headless mode or in foreground (useful for testing)

Examples

Running the server with the example config from the project root dir.

go run cmd/replicant/*.go -config $PWD/example-config.yaml

Web application testing (local development)

• The web application testing support is based on the FQL (Ferret Query Language), documentation.

• Start the Chrome browser with Chrome DevTools Protocol enabled: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 &

Test definition (can be also in json format)

POST http://127.0.0.1:8080/api/v1/run
content-type: application/yaml

name: duckduckgo-search
type: web
schedule: '@every 1m'
timeout: 200s
retry_count: 2
inputs:
  url: "https://duckduckgo.com"
  cdp_address: "http://127.0.0.1:9222"
  user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36"
  timeout: 5000000
  text: "blade runner"
metadata:
  application: duckduckgo-search
  environment: production
  component: web
script: |
  LET doc = DOCUMENT('{{ index . "url" }}', { driver: "cdp", userAgent: "{{ index . "user_agent" }}"})
  INPUT(doc, '#search_form_input_homepage', "{{ index . "text" }}")
  CLICK(doc, '#search_button_homepage')
  WAIT_NAVIGATION(doc)
  LET result = ELEMENT(doc, '#r1-0 > div > div.result__snippet.js-result-snippet').innerText
  RETURN {
    failed: result == "",
    message: "search result",
    data: result,
  }

Response

{
  "data": [
    {
      "name": "duckduckgo-search",
      "type": "web",
      "failed": false,
      "message": "search result",
      "data": "A blade runner must pursue and terminate four replicants who stole a ship in space, and have returned to Earth to find their creator.",
      "time": "2019-10-30T06:18:20.511246Z",
      "metadata": {
        "application": "duckduckgo-search",
        "component": "web",
        "environment": "production"
      },
      "retry_count": 0,
      "with_callback": false,
      "duration_seconds": 5.242629701
    }
  ]
}

API testing (local development)

• The api testing support is based on interpreted go code, documentation.

Test definition (can be also in json format)

POST http://127.0.0.1:8080/api/v1/run
content-type: application/yaml

name: duckduckgo-search
type: go
schedule: '@every 20s'
timeout: 200s
retry_count: 2
inputs:
  url: "https://api.duckduckgo.com/"
  text: "blade runner"
metadata:
  application: duckduckgo-search
  environment: production
  component: api
script: |
  package transaction
  import "bytes"
  import "context"
  import "fmt"
  import "net/http"
  import "io/ioutil"
  import "net/http"
  import "regexp"
  func Run(ctx context.Context) (m string, d string, err error) {
    req, err := http.NewRequest(http.MethodGet, "{{ index . "url" }}", nil)
      if err != nil {
        return "request build failed", "", err
    }
    req.Header.Add("Accept-Charset","utf-8")
    q := req.URL.Query()
    q.Add("q", "{{ index . "text" }}")
    q.Add("format", "json")
    q.Add("pretty", "1")
    q.Add("no_redirect", "1")
    req.URL.RawQuery = q.Encode()
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
      return "failed to send request", "", err
    }
    buf, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      return "failed to read response", "", err
    }
    rx, err := regexp.Compile(`"Text"\s*:\s*"(.*?)"`)
    if err != nil {
      return "failed to compile regexp", "", err
    }
    s := rx.FindSubmatch(buf)
    if len(s) < 2 {
      return "failed to find data", "", fmt.Errorf("failed to find data")
    }
    return "search result", fmt.Sprintf("%s", s[1]), nil
  }

Response

{
  "data": [
    {
      "name": "duckduckgo-search",
      "type": "go",
      "failed": false,
      "message": "search result",
      "data": "Blade Runner A 1982 American neo-noir science fiction film directed by Ridley Scott, written by Hampton...",
      "time": "2019-10-30T06:10:12.835481Z",
      "metadata": {
        "application": "duckduckgo-search",
        "component": "api",
        "environment": "production"
      },
      "retry_count": 0,
      "with_callback": false,
      "duration_seconds": 0.602482443
    }
  ]
}

API

Method	Resource	Action
POST	/v1/transaction	Add a managed transaction
GET	/v1/transaction	Get all managed transaction definitions
GET	/v1/transaction/:name	Get a managed transaction definition by name
DELETE	/v1/transaction/:name	Remove a managed transaction
POST	/v1/run	Run an ad-hoc transaction
POST	/v1/run/:name	Run a managed transaction by name
GET	/v1/result	Get all managed transaction last execution results
GET	/v1/result/:name	Get the latest result for a managed transaction by name
GET	/metrics	Get metrics (prometheus emitter must be enabled)
GET	/debug/pprof	Get available runtime profile data (debug enabled)
GET	/debug/pprof/:profile	Get profile data (for pprof, debug enabled)

TODO

• Tests
• Developer and user documentation
• Add support for more conventional persistent stores
• Vault integration for secrets (inputs)
• Architecture and API documentation
• Javascript driver transaction support

Related Projects

• Yaegi is Another Elegant Go Interpreter
• Ferret Declarative web scraping

Contact

Bruno Moura brunotm@gmail.com

License

Replicant source code is available under the Apache Version 2.0 License