handoff

package module
v0.0.0-...-6cca866 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 20, 2024 License: MIT Imports: 30 Imported by: 0

README

Handoff

Handoff is a library that allows you to bootstrap a server that runs scheduled and manually triggered e2e tests written in Go and is extensible through plugins.

Why Handoff?

More and more companies are building their software as distributed systems.

These are notoriously hard to test end-to-end as the number of services grow. At some point starting all of them and their dependencies locally on the developer's machine becomes unfeasible.

This means that if you want to make sure your system works as expected you need to run tests in an environment that is as close to production as possible, such as a development or staging cluster.

This is where Handoff comes in. You can run Handoff alongside your system as a standalone server that runs all your end-to-end tests either

  • on demand (via cli, api or ui) or
  • repeatedly through a configurable schedule

On top of that there is a baked in web ui where you can look up the test results and manually trigger new runs.

Other valuable (future/planned) features:

  • Matching and linking test runs to logs / traces / metrics generated by the systems under test (SUT) for easier debugging.
  • Running smoke tests on new deployments for automatic rollbacks if necessary (argocd, flux).
  • Fighting flaky tests via
  • auto detection
  • triggering multiple scheduled runs of the same test suite to provoke a test failure which can be debugged thereafter
  • Maintaining historic test run data for deeper insights on the stability of the platform and making it available via a /metrics endpoint.
  • Triggering of alerts on test failures (e.g. via pagerduty).
  • Automated notifications on test runs (e.g. via slack messages).
  • Github Integration: adding test run results to relevant PRs after they were merged and deployed.

Example

Bootstrapping a server is simple, all you need to do is run this code:

package main

func main() {
 h := handoff.New()
 h.Run()
}

To pass in test suites and scheduled runs you can do that by passing in handoff.WithTestSuite and handoff.WithScheduledRun options to handoff.New().

Another way is to register them via handoff.Register before calling handoff.New(). This is especially convenient when you want to have your tests in the same repository as the system under test (SUT), which means they would be in a different repository (unless you have a monorepo). In this case the test package could register the tests in an init function like so:

func init() {
    handoff.Register(ts, scheduledRuns)
}

and then all the handoff server needs to do is import the test package with a blank identifier:

import _ "github.com/my-org/my-service/tests"

For examples see [./cmd/example-server-bootstrap/main.go] and [./internal/packagetestexample].

Build

templ generate
go build ./cmd/example-server-bootstrap/
./example-server-bootstrap

Run

./example-server-bootstrap

Live reload

Instead of generating templ files and building the binary you can also run the example server with live reload:

air

Web

If your server is running you can open your browser to e.g. http://localhost:1337/suites.

This will show you all available test suites.

To create a new test run you can use the post request in ./requests.http

httpyac requests.http

Local dev cluster

Prerequisites:

  • Running Docker
  • Tilt + Kind installed and on the path

Run

kind create cluster --config=kind-config.yaml
tilt up

If you press <space> the tilt UI will open up.

Once handoff is green in the dashboard you should be able to open up the ui here: http://localhost:1337/.

Test best practices

  • Pass in the test context for longer running operations and check if it was cancelled.
  • Only log messages via t.Log/t.Logf as other log messages will not show up in the test logs.
  • Make sure that code in setup is idempotent as it can run more than once.

Planned features

  • (Feature) Write a tool "transformcli" that uses go:generate and go/ast to transform handoff tests and suites to standard go tests (suite -> test with subtests + init and cleanup)
  • (Feature) Users
    • Teams (assigned to test suites)
    • Favorite test suites (UI)
    • authentication providers (LDAP, Oauth2, ...)
  • (Feature) Test Suite Metadata (desription in markdown, links to e.g. github, documentation, handbook, ...) shown in UI
  • (Feature) Automatic test run retries/backoff on failures
  • (Feature) Add a limit to scheduled runs (stop after X runs or X mins)
  • (Feature) CLI program to run tests and wait for results, export results to json (via the server's http api)
  • (Feature) Dashboard UI that shows handoff statistics, running tests, resource usage (cpu, memory, active go routines...) etc
  • (Feature) Opt-in test timeouts through t.Context and / or providing wrapped handoff functions ( e.g. http clients) to be used in tests that implement test timeouts
  • (Feature) Add test-suite labels
  • (Feature) If test was run within the context of a PR maybe we can figure out who the author was (via commit emails) and send an email if a test failed.
  • (Feature) Allow running of handoff as headless/cli mode (without http server) that returns a code != 0 if a test has failed (e.g. in github actions CI)
  • (Feature) Add an option to the helm chart to support remote debugging through dlv
  • (Feature) Image for helm chart tests for automated helm release rollbacks
  • (Feature) Test suite namespaces for grouping
  • (Feature) Asynchronous plugin hooks with callbacks for slow operations (e.g. http calls)
  • (Plugin) Pagerduty - triger alerts/incidents on failed e2e tests
  • (Plugin) Slack - send messages to slack channels when tests pass / fail
  • (Plugin) Github - pr status checks
  • (Plugin) Jira - add test run results to a PR.
  • (Plugin) Prometheus / Loki / Tempo / ELK stack - find and fetch logs/traces/metrics that are created by tests (e.g. for easier debugging) - e.g. via correlation ids
  • (Technical) Idempotency in post requests via a key to avoid duplicate test runs
  • (Technical) Index data (e.g. with github.com/blevesearch/bleve) to be able to query test results.
  • (Technical) Server configuration through either ENV vars or cli flags
  • (Technical) Continue test runs on service restart
  • (Technical) Graceful server shutdown
  • (Technical) Well tested
  • (Technical) Registering of TestSuites and ScheduledRuns via imported packages
  • (Technical) Persistence layer
  • (Feature) Persist compressed test logs to save space
  • (Feature) Soft test fails that don't fail the entire testsuite. This can be used to help with the chicken/egg problem when you add new tests that target a new service version that is not deployed yet.
  • (Feature) Basic webui bundled in the service that shows test run results
  • (Feature) Configurable test run retention policy (TTL)
  • (Feature) Start test runs via POST requests
  • (Feature) Write test suites with multiple tests written in Go
  • (Feature) Manual retrying of failed tests
  • (Feature) Skip individual tests by calling t.Skip() within a test
  • (Feature) Scheduled / recurring test runs (e.g. for soak tests)
  • (Feature) Skip test subsets via regex filters passed into a test run
  • (Feature) Flaky test detection + metric label
  • (Feature) Support existing assertion libraries like stretch/testify
  • (Feature) Prometheus /metrics endpoint that exposes test metrics
  • (Feature) Basic support for plugins to hook into the test lifecycle

Potential features

  • (Technical) Websocket that streams test results (like test logs) - this could be used by the cli tool to get live updates on running tests
  • (Technical) Authenticated HTTP requests through TLS client certificates
  • (Feature) Grafana service dashboard template
  • (Feature) Service dashboards that show information of services k8s resources running in a cluster and their test suite runs
  • (Feature) Output go test json report
  • (Feature) Support running tests in languages other than go
  • (Feature) k8s operator / CRDs to configure test runs & schedules (we probably don't need this)

Open questions

  • How to add test timeouts (it's impossible to externally stop goroutines running user provided functions)?

Non goals

  • Implement a new assertion library. We aim to be compatible with existing ones.

Metrics

Metrics are exposed via the /metrics endpoint.

Name Type Description Labels
handoff_testsuites_running gauge The number of test suites currently running namespace, suite_name
handoff_testsuites_started_total counter The number of test suite runs started namespace, suite_name, result
handoff_tests_run_total counter The number of tests run namespace, suite_name, result

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RegisterSuites 

func RegisterSuites(suites []TestSuite)

Register registers test suites and schedules for the handoff server. Has to be called before `handoff.New()` to take effect.

Types

type AsyncHookCallback 

type AsyncHookCallback func(context map[string]any)

AsyncHookCallback allows async hooks to add additional context to a testsuite or testrun.

type AsyncTestFinishedListener 

type AsyncTestFinishedListener interface {
	Hook
	TestFinishedAsync(suite model.TestSuite, run model.TestSuiteRun, testName string, context map[string]any, callback AsyncHookCallback)
}

type AsyncTestSuiteFinishedListener 

type AsyncTestSuiteFinishedListener interface {
	Hook
	TestSuiteFinishedAsync(suite model.TestSuite, run model.TestSuiteRun, callback func(context map[string]any))
}

type Hook 

type Hook interface {
	Name() string
	Init() error
}

type Option 

type Option func(s *Server)

func WithHook 

func WithHook(p Hook) Option

func WithTestSuite 

func WithTestSuite(suite TestSuite) Option

type Server 

type Server struct {
	// contains filtered or unexported fields
}

func New 

func New(opts ...Option) *Server

New configures a new Handoff instance.

func (*Server) Run 

func (s *Server) Run(args []string) error

Run runs the server with the passed in flags. Usually you want to pass in `os.Args` here.

func (*Server) ServerPort 

func (h *Server) ServerPort() int

ServerPort returns the port that the server is using. This is useful when the port is randomly allocated on startup.

func (*Server) Shutdown 

func (s *Server) Shutdown() error

Shutdown shuts down the server and blocks until it is finished.

func (*Server) WaitForStartup 

func (s *Server) WaitForStartup()

WaitForStartup blocks until the server has started up.

type T 

type T struct {
	// contains filtered or unexported fields
}

func (*T) Attempt 

func (t *T) Attempt() int

func (*T) Cleanup 

func (t *T) Cleanup(c func())

func (*T) Context 

func (t *T) Context() context.Context

func (*T) Error 

func (t *T) Error(args ...any)

func (*T) Errorf 

func (t *T) Errorf(format string, args ...any)

func (*T) Fail 

func (t *T) Fail()

func (*T) FailNow 

func (t *T) FailNow()

func (*T) Failed 

func (t *T) Failed() bool

func (*T) Fatal 

func (t *T) Fatal(args ...any)

func (*T) Fatalf 

func (t *T) Fatalf(format string, args ...any)

func (*T) Helper 

func (t *T) Helper()

func (*T) Log 

func (t *T) Log(args ...any)

func (*T) Logf 

func (t *T) Logf(format string, args ...any)

func (*T) Name 

func (t *T) Name() string

func (*T) Result 

func (t *T) Result() model.Result

func (*T) SetTimeout 

func (t *T) SetTimeout(timeout time.Duration)

func (*T) SetValue 

func (t *T) SetValue(key string, value any)

func (*T) Setenv 

func (t *T) Setenv(key, value string)

func (*T) Skip 

func (t *T) Skip(args ...any)

func (*T) SkipNow 

func (t *T) SkipNow()

func (*T) Skipf 

func (t *T) Skipf(format string, args ...any)

func (*T) Skipped 

func (t *T) Skipped() bool

func (*T) SoftFailure 

func (t *T) SoftFailure()

func (*T) StartSpan 

func (t *T) StartSpan(name string, kv ...any) *model.Span

func (*T) TempDir 

func (t *T) TempDir() string

func (*T) Value 

func (t *T) Value(key string) any

type TB 

type TB = model.TB

type TestContext 

type TestContext = model.TestContext

type TestFinishedListener 

type TestFinishedListener interface {
	Hook
	TestFinished(suite model.TestSuite, run model.TestSuiteRun, testName string, context model.TestContext)
}

type TestFunc 

type TestFunc = model.TestFunc

Reexport to allow library users to reference these types

type TestSuite 

type TestSuite struct {
	// Name of the testsuite
	Name string `json:"name"`
	// Namespace allows grouping of test suites, e.g. by team name.
	Namespace       string
	MaxTestAttempts int
	Description     string
	Setup           func() error
	Teardown        func() error
	Timeout         time.Duration
	Tests           []TestFunc
}

TestSuite represents the external view of the Testsuite to allow users of the library to omit passing in redundant information like the name of the test which can be retrieved via reflection.. It is only used by the caller of the library and then mapped internally to enrich the struct with e.g. test function names.

type TestSuiteFinishedListener 

type TestSuiteFinishedListener interface {
	Hook
	TestSuiteFinished(suite model.TestSuite, run model.TestSuiteRun)
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
example-server-bootstrap
transformcli
handoff transform is a planned tool that automatically transforms handoff tests via go:generate into tests that can be run by the go standard toolchain but includes support for plugins and test suite setup/teardown.
 handoff transform is a planned tool that automatically transforms handoff tests via go:generate into tests that can be run by the go standard toolchain but includes support for plugins and test suite setup/teardown.
internal
hook
html
html/component
metric
model
The `model`s package is very atypical for projects written in go, but unfortunately cannot be avoided as it helps to avoid cyclic dependencies.
 The `model`s package is very atypical for projects written in go, but unfortunately cannot be avoided as it helps to avoid cyclic dependencies.
packagetestexample
storage

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.