README
¶
Load testing framework for golang
Disclaimer: This is still in initial development. It should be pretty stable at this point, but API changes may still occur before a 1.0.0 release is made.
Synopsis
Provides a framework for writing and running load test simulations that allow testing a system (say a web service) against a configurable load that consists of multiple individual tests being run sequentially or in sequence.
Installation
go get github.com/cjbearman/loadtest
Writing tests
A test would typically perform a single action against the system under test, such as performing a REST call. The test is automatically measured for time taken and must indicate a pass/fail outcome.
A test is any valid go struct that conforms to the Test interface.
type Test interface {
// Run will be called to execute the test
// The test should populate the rest object and, if needed, utilize testContext
// for parameters necessary for the operation of the specific iteration of the test
// If it returns a test context, a copy will be sent to any delegate runners attached
// to the current runner to trigger next tests
Run(result *Result, testContext TestContext) *TestContext
}
The Run method is called, with a Result object. The simplest test, which does nothing but pass, would look like this:
type DoesNothingTest struct {}
func (*DoesNothingTest) Run(result *Result, testContext TestContext) *TestContext {
result.Pass = true
return nil
}
The test context may contain information necessary to run the tests which would typically have been provided by prior tests.
Running tests
To run tests, you use a runner. All runners are configured with the following parameters
- The test to run (anything implementing the Test interface)
- The rate at which the test will run. Rate is expressed as the BHCA parameter, which is the target rate of executions over one hour. For example 3,600 BHCA would be one test per second.
- The maximum number of simultaneous tests that may be in progress at any one time (unlimited if configured at 0). If this number is exceeded, the test is throttled and another instance will not be started until the next scheduled interval is reached (if sufficient space is then available).
There are two types of runner.
- Initial runners run a test a pre-determined number of times, using the previously described parameters. Each test iteration receives an empty test context.
- Delegate runners are triggered by a completed test from any prior runner. Any runner may be given a list of "next" runners that should be triggered by a test from that runner that returns a context. When a test triggers delegate runners, a copy of the context from that test (along with any values written to the context) is passed to the test in the delegate runner.
Sequencing runners
Whilst tests and runners can be created and run entirely in code, it is usually prefferable to describe tests and runners using the sequencer, which accepts a YAML definition. This allows a single executable to run a variety of easily configurable test scenarios using the same set of tests, or subset thereof.
Since golang lacks the ability to identify all Test implementations via reflection, in order to use the sequencer it is necessary to register all tests you create with the registry by giving them a name with which they can be referenced from the sequencer.
import "github.com/cjbearman/loadtest/registry"
...
registry.Register("does-nothing", &DoesNothingTest{})
For a test to be registered in the registry, it must also implement TestCreator which is called to create a new unique copy of the test:
type TestCreator interface {
// NewInstance creates a new instance of a test
NewInstance() Test
}
The TestCreator implementation does not have to be the Test implementation, but that is usually most convenient. For example, see nop.go for an exmaple test that also defines this interface.
Bootstrapping your executable
All you need to do is to have your main() method register your tests and then create and execute a sequence using a YAML file. Here is how you would do that.
package main
import (
"log"
"os"
"github.com/cjbearman/loadtest/registry"
"github.com/cjbearman/loadtest/sequence"
)
func main() {
if len(os.Args) != 2 {
log.Fatalf("Please provide the name of a single yaml file on the command line")
}
// Register all test cases first
registry.Register("test-1", &TestOneImpl{})
registry.Register("test-2", &TestTwoImpl{})
registry.Register("test-3", &TestThreeImpl{})
// Now we can load the sequence (tests must be registered before this)
seq, err := sequence.NewSequenceFromFile(os.Args[1])
if err != nil {
log.Fatalf("Failed to process YAML sequence: %v", err)
}
// And run the sequence
seq.Execute()
}
Test initialization
Each runner will receive a single instance of a test. The Run method will be called once for each iteration of the test within that runner.
You can provide parameters to the test from your YAML by implementing the InitializableTest interface on your test:
type InitializableTest interface {
// Init is called after test creation and before the runner is executed
// if the test supports InitializableTest
Init(params map[string]interface{})
}
You can then store those parameters in the test struct and use them in the Run method. How to do this is described below
Whilst you can store other data in the struct from the Run method, be careful. There may be multiple instances of the Run method for the same struct simultaneously executing, so take care to use sync.Mutex to protect things.
Writing the YAML sequence
The YAML sequence consists of several sections:
Monitor flag
The monitor flag can be set to true to have a live monitor display of all executors as the sequence runs. If false, the sequence runs silently.
Common parameters
The common section accepts key/value configuration pairs that will be passed to the Init method of all InitializableTest implementations.
These are defined as:
- Key (mandatory) - The key for the configuration element
- Value (optional) - The value for the configuration element
- Env (optional) - The name of an environment variable from which the configuration element may be taken
You must provide either Value or Env, or both.
If you provide Env only, then the value of the key will be whatever is defined by that environment variable. If the environment variable is not present, the test will panic and the sequence will not run.
If you provide Value only, the value is used exclusively.
If you provide Env and Value, the value of the env var is used, if defined in the environment, else value is used.
Definitions section
The definition section defines the runners you want to use.
You must define a name for each runner.
Each runner is defined along with it's BHCA and max in progress (optional) settings. The test (as provided by the registry) name for the runner must be provided.
Optionally you can provide a params section which is formatted the same as the common section to provide parameters specific to this runner. If a parameter here has the same key as a common parameter, it's value will override the common parameter.
You may optionaly provide a next section which gives the name of the "next" executors to which test contexts will be passed for tests that return a context.
You may also provide filenames for three types of reports:
- Txt: A textual report of the runner including details of all tests. Any values in the test context at the conclusion of the text will be included.
- Csv: A CSV report containing the iteration number, start (ns/epoch) and end (ns/epoch) execution time of each test
- Svg: An SVG (scalable vector graphics) histogram representing the latency of tests run by the executor.
For Txt and Csv you may provide "console" instead of a filename, and the report will be output to the console at the conclusion of the sequence.
Execution section
The execution section defines how to run the executors.
All executors must be "run", at which point they start processing work. Multiple executors can be run concurrently.
All executors must be "wait"ed. Waiting on an executor will prevent the execution sequence from moving forward until all tests scheduled by that executor have completed. When waiting a delegate executor, the delegate executor will no longer accept new work once the wait state is entered.
For example, the following execution sequence:
- run runner1
- run runner2
- wait runner1
- wait runner2
- run runner3
- wait runner3
With the above sequence, runner1 and runner2 run simultaneously to completion. Once both have completed, runner3 will run.
Note that when using delegate runners, work may be sent to it before it starts to run and whilst it's running, work is queued and executed in FIFO order.
YAML example
# We want to see live monitoring
monitor: true
# Definition of common parameters that will be passed to ALL tests
common:
# KEY1 will have the fixed "value1"
- key: KEY1
value: value1
# KEY2 will have "value2" unless ENV_FOR_KEY_2 is defined, in which case the value is taken from that ENV VAR
- key: KEY2
value: value2
env: ENV_FOR_KEY_2
# Runner definitions
definitions:
# runner1 will operate at 3600 ops per hour (1 per sec)
# As an initial runner, it will run it's test 10 times (iterations)
# If the test (nop) returns true, then it's test context is handed off to runner2
- name: runner1
type: initial
bhca: 3600
iterations: 10
test: nop
next:
- runner2
# These parameters will only be passed to the test for this runner
params:
- key: KEY3
value: value3
# runner2 is a delegate, it is accepting tests from runner1
# It will permit only a single instance of the test to be running at any time
- name: runner2
type: delegate
bhca: 3600
max: 1
test: seq-test
# runner3 runs 10 more iterations of a test
- name: runner3
type: initial
bhca: 3600
iterations: 10
test: nop
reports:
# CSV report to file
csv: runner3.csv
# TXT report to console
txt: console
# SVG Histogram to file
svg: runner3.svg
# Executions will have runner1 and runner2 run concurrently.
# Runner3 will run once both runner1 and runner2 have finished
executions:
- run runner1
- run runner2
- wait runner1
- wait runner2
- run runner3
- wait runner3
HTTP Testing
The test package includes a test named HttpTest. This can be used for simple load generation against HTTP endpoints.
Make sure and register it in your executable
registry.Register("http", &test.HttpTest{})
You must provide the URL to be tested as "URL" parameter in the definition for the runner. It will assume a GET operation, and assert a pass condition provided the URL responds with 200 OK within 5 seconds.
There are various other parmaeters you can tweak through parameters to change the operation.
This is an example sequence using this test, to concurrently run operations against two endpoints, one expecting 200 OK, the other 404 Not Found:
monitor: true
definitions:
- name: page-found
type: initial
bhca: 360000
max: 10
iterations: 100
test: http
next:
- page-nf
params:
- key: URL
value: https://www.chrisjb.com/index.html
- name: page-nf
type: delegate
bhca: 360000
max: 10
test: http
params:
- key: URL
value: https://www.chrisjb.com/nf.html
- key: EXPECT
value: 404
executions:
- run page-found
- run page-nf
- wait page-found
- wait page-nf
Full example
Using the above YAML, you can look in example for a complete example.
> cd example
> go build loadtest.go
> ./loadtest http.yaml
Name : TOTAL PASS FAIL INPROG HWM Throt. Avg. Lat Max. Lat BHCA [COMPLETE]
----------------------------------------------------------------------------------------------------------
page-found : 100 100 0 0 10 4 41.150 289.847 275356 [COMPLETE]
page-nf : 100 100 0 0 1 0 27.263 62.437 36248 [COMPLETE]
The monitoring information (as shown above) will be displayed throughout the test. If your terminal supports it, colors will be used and it will be updated in place. If your terminal does not support it, it is repeatedly displayed at a slower rate.
The fields in the report are as follows:
- Name : The name of the runner
- TOTAL : Total tests run (blue)
- PASS : Total tests passed (green)
- FAIL : Total tests failed (turns red if >1)
- INPROG : Number of tests currently simultaneously in progress
- HWM : High water mark (maximum number of tests that have been in progress simultaneously at any time). Green, turns red if tests get throttled.
- Throt.: The count of throttles (test starts delayed because max iterations was reached)
- Avg. Lat: Average latency of tests
- Max. Lat: The latency of the longest running test
- BHCA: The busy hour call attempt rate being achieved
- [COMPLETE]: Flags [COMPLETE] when the runner completes
Documentation
¶
Overview ¶
Package loadtest provides tools for the execution of tests under load. Typically those tests will be REST or other simulation calls that need to be run at a predictable and controllable rate to validate the operation of some system against the load generated
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AtomicBool ¶
type AtomicBool struct {
// contains filtered or unexported fields
}
AtomicBool is an implementation of a synchronized boolean default value is false
func (*AtomicBool) Get ¶
func (b *AtomicBool) Get() bool
Get is used to return the value under synchronization
func (*AtomicBool) Set ¶
func (b *AtomicBool) Set(value bool)
Set is used to set the value under synchronization
type AtomicWork ¶
type AtomicWork struct {
// contains filtered or unexported fields
}
AtomicWork is a synchronized FIFO queue for work
func (*AtomicWork) GetRemainingCount ¶ added in v0.0.4
func (w *AtomicWork) GetRemainingCount() int
func (*AtomicWork) GetWork ¶
func (w *AtomicWork) GetWork() *TestContext
Pop pulls the next work item from the front of the queue, returning nil if no more work is available
func (*AtomicWork) Push ¶
func (w *AtomicWork) Push(context TestContext)
Push adds more work to the end of the queue
func (*AtomicWork) PushMulti ¶
func (w *AtomicWork) PushMulti(contexts []TestContext)
PushMulti adds multiple work items to the end of the queue
func (*AtomicWork) WorkRemaining ¶
func (w *AtomicWork) WorkRemaining() bool
type DelegateRunner ¶
type DelegateRunner interface {
Runner
Execute(work TestContext) error
}
DelegateRunner defines a runner that can accept work from other runners as a result of test completions
type InitContext ¶ added in v0.0.6
type InitContext map[string]interface{}
Init is also a map of key/value pairs that can be used to feed parameters to a test during initialization
type InitializableTest ¶
type InitializableTest interface {
Test
// Init is called after test creation and before the runner is executed
// if the test supports InitializableTest
Init(params InitContext)
}
type Monitorable ¶
type Monitorable interface {
// GetStats will be called by the monitor to retrieve stats from the monitorable
GetStats() Stat
// GetName should report the name of the instance being monitored
GetName() string
}
Monitorable is implemented by all runners to allow them to be monitored by the monitor
type Result ¶
type Result struct {
// Pass should be true if the test passed, otherwise false
Pass bool
// Msg should be an error indicating failure reason in the event that Pass=false
Msg error
// Start should be the unix epoch (nanosecond) that the test starts
// It is provided automatically by the runner, but can be refined by the testcase if needed
Start int64
// End should be the unix epoch (nanosecond) that the test finished
// It is provided automatically by the runner if left at 0
End int64
// Duration is calculated automatically by the runner as the period between Start and End
// the testcase should not write to this field
Duration int64
// Iteration is provided automatically by the runner and is the iteration sequence number
// of the test
Iteration int
// If a test needs to retry itself, it should return its own context here
// Will not work in initial runners (which are fixed iteration)
Retry *TestContext
// Attributes is a map that the test case can write information key/value pairs to
// these values may be used by reporters when reporting the test result
Attributes TestAttributes
}
Result is passed to each test and can be manipulated by the test to indicate the outcome and other relevant information
type Runner ¶
type Runner interface {
Monitorable
// Run is used to start the runner. It may only be called once.
// The next parameter may be used to provide one or more next runners to which
// tests may forward work using by calling TriggerNext upon their completion
Run(next []DelegateRunner)
// Wait may be called on a running runner and will wait for completion of that runner
// before returning, at which time the runner is considered complete
Wait() []Result
// GetResults can be used to retieve the results from a runner
// Calling on an uncompleted runner will panic.
GetResults() []Result
// GetHighWaterMark can be used to retrieve the maximum number of tests in progress simultaneously
// at any time during the runner's execution
// Calling on an uncompleted runner will panic.
GetHighWaterMark() int
// Closed will return true if the runner is completed
Closed() bool
// Closing will return true if Wait has been called but not yet returned
Closing() bool
// Running will return true if the runner is running or closing but not yet closed
Running() bool
// GetThrottled will return the number of calls throttled (delayed) due to the runner
// reaching max in progress tests at one or more points during executions.
// Calling on an uncompleted runner will panic.
GetThrottled() int
// GetTargetBHCA returns the target BHCA configured for the runner
GetTargetBHCA() int
// GetMaxInProgress returns the max in progress limit (0 = unlimited) configured
// for the runner
GetMaxInProgress() int
// GetName returns the runner name
GetName() string
}
Runner defines the interface elements common to all runners
type Stat ¶
type Stat struct {
// Pass represents the count of pass testcases and must be set
Pass int
// Fail represents the count of failed testcases and must be set
Fail int
// Remaining should be set to the number of remaining tests, if known
Remaining *int
// InProgress represents the count of inprogress testcases and must be set
// unless PFOnly is set
InProgress int
// Throttled represents the count of throttled calls due to max being reached
Throttled int
// HighWaterMark represents the high water mark (max in progress) seen during
// run
HWM int
// CumulativeRuntimeNS represents the cumulative run time of all completed testcases
// in nanoseconds and must be set unless PFOnly is set
CumulativeRuntimeNS int64
// MaxLatencyMS must be set to the ns taken by the longest running completed instance
// in nanoseconds and must be set unless PFOnly is set
MaxLatencyNS int64
// EffectiveBHCA must be set to the calculated achieved BHCA, unless PFOnly is set
EffectiveBHCA int64
// Completed should be set to true when a runner completes, unless NonCompletable is set
Completed bool
// NonCompletable indicates that the monitor is not an entity that completes
NonCompletable bool
// Aborted must be set if a runner was never started due to master abort signal
Aborted bool
// Number of retries
Retries int
}
Stat is representation of current operational state used to communicate between runners and monitor
type Test ¶
type Test interface {
// Run will be called to execute the test
// The test should populate the rest object and, if needed, utilize testContext
// for parameters necessary for the operation of the specific iteration of the test
// If it returns a test context, a copy will be sent to any delegate runners attached
// to the current runner to trigger next tests
Run(result *Result, testContext TestContext) *TestContext
}
Test is the interface that tests must implement A Result is passed and must be completed by the test The testContext will contain any contextual values passed to the test A Test is a singleton that will be executed multiple times, therefore the implementing struct should not carry state unless that state is common to all instances of the test
type TestAttributes ¶ added in v0.0.6
type TestAttributes map[string]interface{}
TestAttributes is also a map of key/value pairs that can be set in a test result for reporting purposes
type TestContext ¶
type TestContext map[string]interface{}
TestContext is a map of key/value paris that can be used to feed information to individual test executions
type TestCreator ¶
type TestCreator interface {
// NewInstance creates a new instance of a test
NewInstance() Test
}
TestCreator is used by test registry to create a new instance of a test TestCreators should be registered with the registry package to make them available for use by a sequence Typically a test will also implement TestCreator allowing it to create instances of itself
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package monitoring provides real time monitoring display of runner execution
|
Package monitoring provides real time monitoring display of runner execution |
|
Package registry provides a package that can be used to register tests that can then be retrieved by name this allows for config file driven testing
|
Package registry provides a package that can be used to register tests that can then be retrieved by name this allows for config file driven testing |
|
Package reporting provides features to allow for the generation of test reports
|
Package reporting provides features to allow for the generation of test reports |
|
Package runner provides runner implementations for running testcases at load.
|
Package runner provides runner implementations for running testcases at load. |
|
Package sequence provides the sequencer that runs tests from a YAML description.
|
Package sequence provides the sequencer that runs tests from a YAML description. |
|
Package util provides general purpose utilities
|
Package util provides general purpose utilities |