bender

package module
v0.0.0-...-7147e54 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 18, 2020 License: Apache-2.0 Imports: 6 Imported by: 0

README

Bender

Bender makes it easy to build load testing applications for services using protocols like HTTP, Thrift, Protocol Buffers and many more. Bender provides a library of flexible, powerful primitives that can be combined (with plain Go code) to build load testers customized to any use case and that evolve with your service over time.

Bender provides two different approaches to load testing. The first, LoadTestThroughput, gives the tester control over the throughput (QPS), but not over the concurrency. This one is very well suited for services that are open to the Internet, like web services, and even backend Thrift or Protocol Buffer services, since it will just keep sending requests, even if the service is struggling. The second approach, LoadTestConcurrency, gives the tester control over the concurrency, but not over the throughput. This approach is better suited to testing services that require lots of concurrent connections, and need to be tested for resource limits.

That Bender is a library makes it flexible and easy to extend, but means it takes longer to create an initial load tester. As a result, we've focused on creating easy-to-follow tutorials that are written for people unfamiliar with Go, along with documentation for all the major functions in the library.

Getting Started

The easiest way to get started with Bender is to use one of the tutorials:

Documentation

The package documentation is available on godoc.org. The function and data structure documentation is also available there.

Performance

We have only informal, anecdotal evidence for the maximum performance of Bender. For example, in a very simple load test using a Thrift server that just echoes a message, Bender was able to send 7,000 QPS from a single EC2 m3.2xlarge host. At higher throughput the Bender "overage" counter increased, indicating that either the Go runtime, the OS or the host was struggling to keep up.

We have found a few things that make a big difference when running load tests. First, the Go runtime needs some tuning. In particular, the Go GC is very immature, so we prefer to disable it using the GOGC=off environment variable. In addition, we have seen some gains from setting GOMAXPROCS to twice the number of CPUs.

Secondly, the Linux TCP stack for a default server installation is usually not tuned to high throughput servers or load testers. After some experimentation, we have settled on adding these lines to /etc/sysctl.conf, after which you can run sysctl -p to load them (although it is recommended to restart your host at this point to make sure these take effect).

# /etc/sysctl.conf
# Increase system file descriptor limit
fs.file-max = 100000

# Increase ephermeral IP ports
net.ipv4.ip_local_port_range = 10000 65000

# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

This is a slightly modified version of advice taken from this source: http://www.nateware.com/linux-network-tuning-for-2013.html#.VBjahC5dVyE

In addition, it helps to increase the open file limit with something like:

ulimit -n 100000

What Is Missing

Bender does not provide any support for sending load from more than one machine. If you need to send more load than a single machine can handle, or you need the requests to come from multiple physical hosts (or different networks, or whatever), you currently have to write your own tools. In addition, the histogram implementation used by Bender is inefficient to send over the network, unlike q-digest or t-digest, which we hope to implement in the future.

Bender does not provide any visualization tools, and has a relatively simple set of measurements, including a customizable histogram of latencies, an error rate and some other summary statistics. Bender does provide a complete log of everything that happens during a load test, so you can use existing tools to graph any aspect of that data, but nothing in Bender makes that easy right now.

Bender currently provides helper functions for DHCPv6, DNS, HTTP, Thrift and TFTP. We appreciate Pull Requests for other protocols.

The load testers we have written internally with Bender have a lot of common command line arguments, but we haven't finalized a set to share as part of the library.

Comparison to Other Load Testers

JMeter

JMeter provides a GUI to configure and run load tests, and can also be configured via XML (really, really not recommended by hand!) and run from the command line. JMeter uses the same approach as LoadTestConcurrency in Bender, which is not a good approach to load testing services (see the Bender docs and the Iago philosophy for more details on why that is). It isn't easy to extend JMeter to handle new protocols, so it doesn't have support for Thrift or Protobuf. It is relatively easy to extend other parts of JMeter by writing Java code, however, and the GUI makes it easy to plug all the pieces together.

Iago

Iago is Twitter's load testing library and it is the inspiration for Bender's LoadTestThroughput function. Iago is a Scala library written on top of Netty and the Twitter Finagle libraries. As a result, Iago is powerful, but difficult to understand, extend and configure. It was frustration with making Iago work that led to the creation of Bender.

The Grinder

The Grinder has the same load testing approach as JMeter, but allows scripting via Jython, which makes it more flexible and extensible. The Grinder uses threads, which limits the concurrency at which it can work, and makes it hard to implement things like Bender's LoadTestThroughput function. The Grinder does have support for conveniently running distributed load tests.

Copyright 2014-2018 Pinterest, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Attribution

Bender includes open source from the following sources:

Documentation

Overview

Package bender makes it easy to build load testing applications for services using protocols like HTTP, Thrift, Protocol Buffers and many others.

Bender provides two different approaches to load testing. The first, LoadTestThroughput, gives the tester control over the throughput (QPS), but not over the concurrency (number of goroutines). The second, LoadTestConcurrency, gives the tester control over the concurrency, but not over the throughput.

LoadTestThroughput simulates the load caused by concurrent clients sending requests to a service. It can be used to simulate a target throughput (QPS) and to measure the request latency and error rate at that throughput. The load tester will keep spawning goroutines to send requests, even if the service is sending errors or hanging, making this a good way to test the actual behavior of the service under heavy load. This is the same approach used by Twitter's Iago library, and is nearly always the right place to start when load testing services exposed (directly or indirectly) to the Internet.

LoadTestConcurrency simulates a fixed number of clients, each of which sends a request, waits for a response and then repeats. The downside to this approach is that increased latency from the service results in decreased throughput from the load tester, as the simulated clients are all waiting for responses. That makes this a poor way to test services, as real-world traffic doesn't behave this way. The best use for this function is to test services that need to handle a lot of concurrent connections, and for which you need to simulate many connections to test resource limits, latency and other metrics. This approach is used by load testers like the Grinder and JMeter, and has been critiqued well by Gil Tene in his talk "How Not To Measure Latency".

The next two sections provide more detail on the implementations of LoadTestThroughput and LoadTestConcurrency. The following sections provide descriptions for the common arguments to the load testing functions, and how they work, including the interval generators, request generators, request executors and event recorders.

LoadTestThroughput

The LoadTestThroughput function takes four arguments. The first is a function that generates nanosecond intervals which are used as request arrival times. The second is a channel of requests. The third is a function that knows how to send a request and validate the response. The inner loop of LoadTestThroughput looks like this: 

for {
    interval := intervals(time.Now().UnixNanos())
    time.Sleep(time.Duration(interval)
    request := <-requests
    go func() {
      err := requestExec(time.Now().UnixNano(), request)
    }()
}

The fourth argument to LoadTestThroughput is a channel which is used to output events. There are events for the start and end of the load test, the sending of each request and the receiving of each response and the wait time between sending requests. The wait message includes an "overage" time which is useful for monitoring the health of the load test program and underlying OS and host. The overage time measures the difference between the expected wait time (the interval time) and the actual wait time. On a heavily loaded host, or when there are long GC pauses, that difference can be large. Bender attempts to compensate for the overage by reducing the subsequent wait times, but under heavy load, the overage will continue to increase until it cannot be compensated for. At that point the wait events will report a monotonically increasing overage which means the load test isn't keeping up with the desired throughput.

A load test ends when the request channel is closed and all remaining requests in the channel have been executed.

LoadTestConcurrency

The LoadTestConcurrency function takes four arguments. The first is a semaphore that controls the maximum number of concurrently executing requests, and makes it possible to dynamically control that number over the lifetime of the load test. The second, third and fourth arguments are identical to those for LoadTestThroughput. The inner loop of LoadTestConcurrency does something like this: 

for {
    workerSem.Wait(1)
    request := <-requests
    go func() {
        err := requestExec(time.Now().UnixNano(), request)
        workerSem.Signal(1)
    }
}

Reducing the semaphore count will reduce the number of running connections as existing connections complete, so there can be some lag between calling workerSem.Wait(n) and the number of running connections actually decreasing by n. The worker semaphore does not protect you from reducing the number of workers below zero, which will cause undefined behavior from the load tester.

As with LoadTestThroughput, the load test ends when the request channel is closed and all remaining requests have been executed.

Interval Generators

An IntervalGenerator is a function that takes the current Unix epoch time (in nanoseconds) and returns a non-negative time (also in nanoseconds) until the next request should be sent. Bender provides functions to create interval generators for uniform and exponential distributions, each of which takes the target throughput (requests per second) and returns an IntervalGenerator. Neither of the included generators makes use of the function argument, but it is there for cases in which the simulated intervals are time dependent (you want to simulate the daily traffice variation of a web site, for example).

Request Channels

The request channel decouples creation of requests from execution of requests and allows them to run concurrently. A typical approach to creating a request channel is code like this: 

 c := make(chan *Request)
 go func() {
     for {
		 // create service request r with request ID rid
		 c <- &Request{rid, r}
     }
     close(c)
 }()

Requests can be generated randomly, read from files (like access logs) or generated any other way you like. The important part is that the request generation be done in a separate goroutine that communicates with the load tester via a channel. In addition, the channel must be closed to indicate that the load test is done.

The requests channel should almost certainly be buffered, unless you can generate requests much faster than they are sent (and not just on average). The easiest way to miss your target throughput with LoadTestThroughput is to be blocked waiting for requests to be generated, particularly when testing a large throughput.

Request Executors

A request executor is a function that takes the current Unix Epoch time (in nanoseconds) and a *Request, sends the request to the service, waits for the response, optionally validates it and returns an error or nil. This function is timed by the load tester, so it should do as little else as possible, and everything it does will be added to the reported service latency. Here, for example, is a very simple request executor for HTTP requests: 

func HttpRequestExecutor(_ int64, request *Request) error {
    url := request.Request.(string)
    _, err := http.Get(url)
    return err
}

The http package in Bender provides a function that generates executors that make use of the http packages Transport and Client classes and provide an easy way to validate the body of the http request.

RequestExecutors are called concurrently from multiple goroutines, and must be concurrency-safe.

Event Messages

The LoadTestThroughput and LoadTestConcurrency functions both take a channel of events (represented as interface{}) as a parameter. This channel is used to output events as they happen during the load test, including the following events:

StartEvent: sent once at the start of the load test.

EndEvent: sent once at the end of the load test, no more events are sent after this.

WaitEvent: sent only for LoadTestThroughput, see below for details.

StartRequestEvent: sent before a request is sent to the service, includes the request and the event time. Note that the event time is not the same as the start time for the request for stupid performance reasons. If you need to know the actual start time, see the EndRequestEvent.

EndRequestEvent: sent after a request has finished, includes the response, the actual start and end times for the request and any error returned by the RequestExecutor.

The WaitEvent includes the time until the next request is sent (in nanoseconds) and an "overage" time. When the inner loop sleeps, it subtracts the total time slept from the time it intended to sleep, and adds that to the overage. The overage, therefore, is a good proxy for how overloaded the load testing host is. If it grows over time, that means the load test is falling behind, and can't start enough goroutines to run all the requests it needs to. In that case you will need a more powerful load testing host, or need to distribute the load test across more hosts.

The event channel doesn't need to be buffered, but it may help if you find that Bender isn't sending as much throughput as you expect. In general, this depends a lot on how quickly you are consuming events from the channel, and how quickly the load tester is running. It is a good practice to proactively buffer this channel.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func LoadTestConcurrency 

func LoadTestConcurrency(workers *WorkerSemaphore, requests chan interface{}, requestExec RequestExecutor, recorder chan interface{})

LoadTestConcurrency starts a load test in which the caller controls the number of goroutines that are sending requests. See the package documentation for details on the arguments to this function.

func LoadTestThroughput 

func LoadTestThroughput(intervals IntervalGenerator, requests chan interface{}, requestExec RequestExecutor, recorder chan interface{})

LoadTestThroughput starts a load test in which the caller controls the interval between requests being sent. See the package documentation for details on the arguments to this function.

func Record 

func Record(c chan interface{}, recorders ...Recorder)

Record records messages from a channel using the given recorders.

Types

type EndEvent 

type EndEvent struct {
	// The Unix epoch times in nanoseconds at which the load test started and ended.
	Start, End int64
}

EndEvent is sent once at the end of the load test, after which no more events are sent.

type EndRequestEvent 

type EndRequestEvent struct {
	// The Unix epoch times (in nanoseconds) at which the request was started and finished
	Start, End int64
	// The response data returned by the request executor
	Response interface{}
	// An error or nil if there was no error
	Err error
}

EndRequestEvent is sent after a request has completed.

type IntervalGenerator 

type IntervalGenerator func(int64) int64

An IntervalGenerator is a function that takes the current Unix epoch time (in nanoseconds) and returns a non-negative time (also in nanoseconds) until the next request should be sent. Bender provides functions to create interval generators for uniform and exponential distributions, each of which takes the target throughput (requests per second) and returns an IntervalGenerator. Neither of the included generators makes use of the function argument, but it is there for cases in which the simulated intervals are time dependent (you want to simulate the daily traffice variation of a web site, for example).

func ExponentialIntervalGenerator 

func ExponentialIntervalGenerator(rate float64) IntervalGenerator

ExponentialIntervalGenerator creates an IntervalGenerator that outputs exponentially distributed intervals. The resulting arrivals constitute a Poisson process. The rate parameter is the average queries per second for the generator, and corresponds to the reciprocal of the lambda parameter to an exponential distribution. In English, if you want to generate 30 QPS on average, pass 30 as the value of rate.

func UniformIntervalGenerator 

func UniformIntervalGenerator(rate float64) IntervalGenerator

UniformIntervalGenerator creates and IntervalGenerator that outputs 1/rate every time it is called. Boring, right?

type Recorder 

type Recorder func(interface{})

A Recorder records a message.

func NewHistogramRecorder 

func NewHistogramRecorder(h *hist.Histogram) Recorder

NewHistogramRecorder creates a new hist.Histogram-based recorder.

func NewLoggingRecorder 

func NewLoggingRecorder(l *log.Logger) Recorder

NewLoggingRecorder creates a new log.Logger-based recorder.

type RequestExecutor 

type RequestExecutor func(int64, interface{}) (interface{}, error)

RequestExecutor is a function that takes the current Unix Epoch time (in nanoseconds) and a *Request, sends the request to the service, waits for the response, optionally validates it and returns an error or nil. This function is timed by the load tester, so it should do as little else as possible, and everything it does will be added to the reported service latency.

type StartEvent 

type StartEvent struct {
	// The Unix epoch time in nanoseconds at which the load test started.
	Start int64
}

StartEvent is sent once at the start of the load test.

type StartRequestEvent 

type StartRequestEvent struct {
	// The Unix epoch time (in nanoseconds) at which this event was created, which will be earlier
	// than the sending of the associated request (for performance reasons)
	Time int64
	// The request that will be sent, nothing good can come from modifying it
	Request interface{}
}

StartRequestEvent is sent before a request is executed. The sending of this event happens before the timing of the request starts, to avoid potential issues, so it contains the timestamp of the event send, and not the timestamp of the request start.

type WaitEvent 

type WaitEvent struct {
	// The next wait time (in nanoseconds) and the accumulated overage time (the difference between
	// the actual wait time and the intended wait time).
	Wait, Overage int64
}

WaitEvent is sent once for each request before sleeping for the given interval.

type WorkerSemaphore 

type WorkerSemaphore struct {
	// contains filtered or unexported fields
}

WorkerSemaphore controls the number of "workers" that can be running as part of a load test using LoadTestConcurrency.

func NewWorkerSemaphore 

func NewWorkerSemaphore() *WorkerSemaphore

NewWorkerSemaphore creates an empty WorkerSemaphore (no workers).

func (WorkerSemaphore) Signal 

func (s WorkerSemaphore) Signal(n int)

Signal adds a worker to the pool of workers that are currently sending requests. If no requests are outstanding, this will block until a request is ready to send.

func (WorkerSemaphore) Wait 

func (s WorkerSemaphore) Wait(n int) bool

Wait removes a worker from the pool. If all workers are busy, then this will wait until the next worker is finished, and remove it.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.