prombench

README

prombench

Benchmark and integration testing tool for Prometheus

Prombench generates load with synthetic exporters and then queries Prometheus to compare what we think we sent with what we hope was stored.

The eventual goal is to have Prombench monitor Prometheus internal metrics and increase the load until Prometheus can't keep up, then throttle back until it's just below the tipping point. For now it just generates a static load.

Usage

If the prometheus binary is in your PATH, you can run it without any arguments. It won't run if the test dir ("prombench" by default) already exists, so either remove it yourself between invocations or use the -rmtestdir option.

Each exporter generates 100 metrics with 100 labels, or 10000 real metrics. I can run about 30 such exporters on my system (8GB RAM, AMD FX(tm)-6300) at 1s scrape interval for a total load of 300K samples/second. Going above 300K samples/second requires more RAM and will probably require tuning Prometheus, see the tips in the storage part of the docs. You can override the default Prometheus command ("prometheus") with a path and command-line arguments at the end of the prombench command line, after --, e.g.

prombench -exporters inc:20 -- ~/src/prometheus/prometheus -storage.local.memory-chunks 2097152 -storage.local.max-chunks-to-persist 1048576 

Do not provide the -storage.local.retention Prometheus argument, use rather the prombench -test-duration argument. This allows the verification query to be scaled to how much data should still be present by the time it's run.

Exporters

The -exporters flag is a comma-separated list specifying which load exporters to use and how many, e.g. inc:2,static:3 would launch 2 inc exporters and 3 static exporters.

The inc exporter increments the value of each metric on each scrape.

The static exporter exports unchanging metrics.

The randcyclic exporter exports semi-random values.

The oscillate exporter toggles between two sets of values on each cycle. Unlike the others it doesn't actually go through the standard Prometheus client library except during initialization, so it has much lower CPU needs.

Scheduled tasks

The -run-every flag is a comma-separated list of commands to invoke at fixed intervals. For example, if you've installed go-torch and have setup your path appropriately, you might run

prombench -run-every '60s:go-torch -u http://localhost:9090 -f torch`date "+%s"`.svg'

to create per-minute flamegraphs based on debug/pprof data.

Dashboards

I've put up a rudimentary dashboard at Grafana.net which is helpful for seeing what's going on under the hood.

Documentation

Index

• Variables
• func Run(cfg Config)
• type Config
  • func (c Config) PrometheusInstance() (string, error)
• type ExporterSpec
  • func (e *ExporterSpec) Get() interface{}
  • func (e *ExporterSpec) Set(v string) error
  • func (e *ExporterSpec) String() string
• type ExporterSpecList
  • func (esl *ExporterSpecList) Get() interface{}
  • func (esl *ExporterSpecList) Set(v string) error
  • func (esl *ExporterSpecList) String() string
• type LoadExporterKind
  • func (i LoadExporterKind) String() string
• type RunIntervalSpec
  • func (r *RunIntervalSpec) Get() interface{}
  • func (r *RunIntervalSpec) Set(v string) error
  • func (r *RunIntervalSpec) String() string
• type RunIntervalSpecList
  • func (rsl *RunIntervalSpecList) Get() interface{}
  • func (rsl *RunIntervalSpecList) Set(v string) error
  • func (rsl *RunIntervalSpecList) String() string

Variables

var (
	QueryTime *prometheus.HistogramVec = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Namespace: "prombench",
			Subsystem: "query",
			Name:      "latency_seconds",
			Help:      "time to execute query",
			ConstLabels: prometheus.Labels{

				"benchmark": "insert-then-sum",
			},
		},
		[]string{"run_name", "query"},
	)
)

Functions

func Run

func Run(cfg Config)

Types

type Config

type Config struct {
	TestDirectory           string
	RmTestDirectory         bool
	FirstPort               int
	PrometheusPath          string
	ScrapeInterval          time.Duration
	TestDuration            time.Duration
	TestRetention           time.Duration
	ExtraArgs               []string
	Exporters               ExporterSpecList
	RunIntervals            RunIntervalSpecList
	MaxDeltaRatio           float64
	MaxQueryRetries         int
	AdaptiveInterval        time.Duration
	PrombenchListenAddress  string
	PrometheusListenAddress string
}

func (Config) PrometheusInstance

func (c Config) PrometheusInstance() (string, error)

TODO check for errors when the Config is created

type ExporterSpec

type ExporterSpec struct {
	Exporter LoadExporterKind
	Count    int
}

func (*ExporterSpec) Get

func (e *ExporterSpec) Get() interface{}

func (*ExporterSpec) Set

func (e *ExporterSpec) Set(v string) error

func (*ExporterSpec) String

func (e *ExporterSpec) String() string

type ExporterSpecList

type ExporterSpecList []ExporterSpec

func (*ExporterSpecList) Get

func (esl *ExporterSpecList) Get() interface{}

func (*ExporterSpecList) Set

func (esl *ExporterSpecList) Set(v string) error

func (*ExporterSpecList) String

func (esl *ExporterSpecList) String() string

type LoadExporterKind

type LoadExporterKind int

const (
	ExporterInc LoadExporterKind = iota
	ExporterStatic
	ExporterRandCyclic
	ExporterOscillate
)

func (LoadExporterKind) String

func (i LoadExporterKind) String() string

type RunIntervalSpec

type RunIntervalSpec struct {
	Command  string
	Interval time.Duration
}

func (*RunIntervalSpec) Get

func (r *RunIntervalSpec) Get() interface{}

func (*RunIntervalSpec) Set

func (r *RunIntervalSpec) Set(v string) error

func (*RunIntervalSpec) String

func (r *RunIntervalSpec) String() string

type RunIntervalSpecList

type RunIntervalSpecList []RunIntervalSpec

func (*RunIntervalSpecList) Get

func (rsl *RunIntervalSpecList) Get() interface{}

func (*RunIntervalSpecList) Set

func (rsl *RunIntervalSpecList) Set(v string) error

func (*RunIntervalSpecList) String

func (rsl *RunIntervalSpecList) String() string