testsuite

README

Prometheus Alert Generator Compliance

Specification

You can find the specification for this compliance here.

Running the tests

Step 1

Clone the repo

$ git clone https://github.com/prometheus/compliance.git
$ cd compliance/alert_generator

Step 2

Check that the rules file is up-to-date with

$ make check-rules
go run ./cmd/rule_config_builder/main.go -rules-file-path="./rules.yaml"
level=info ts=2022-02-24T14:14:19.596Z caller=main.go:53 msg="Rules file successfully generated" path=/home/ganesh/go/src/github.com/prometheus/compliance/alert_generator/rules.yaml

$ echo $?
0

Run make rules otherwise to generate fresh rules file.

Feed the rules.yaml file present in the directory into the software that you are testing with any additional tweaks required.

It is suggested to wait until one evaluation of all rule groups is done before starting the test.

Step 3

Check test-example.yaml for the example config to setup the test suite. That file has all the documentation on how to configure the test suite.

Feel free to modify the example config itself. Config for some softwares are provided as an example, more will be added.

Step 4

If you are running your software in a local environment, you can set the alertmanager URL to http://<host>:<port> with port being the one set in the test suite config. For example http://localhost:8080.

If you are testing a cloud offering, or if the local software setup cannot access the test suite's network, there are two alternatives:

Step 4a

You can run ./cmd/alert_generator_compliance_tester in docker and run it inside your infrastructure as a batch job. To do so, build the image with make docker and instead of step 6, simply run alert_generator_compliance_tester:latest with -config-file=<config file you have done in step 3>.

See example here

Step 4b

1. Open https://webhook.site/. It will give a unique link to use as a webhook. Let's take https://webhook.site/12345 as an example.
2. Set https://webhook.site/12345 to be the alertmanager URL in your software - hence all alerts will be set to this webhook.
3. On the webhook.site page, enable XHR Redirect on top, and in its settings, set the Target to http://<host>:<port> from the above, for example http://localhost:8080.

Note:

• The webhook.site page must be open at all times. It redirects the request locally, hence able to redirect to local network.
• In the test logs you might see "error in unmarshaling" error immediately followed by "alerts received" log. You can ignore the error log. It is something to do with how webhook.site redirects.
• Disable any kind of browser blockers for webhook.site if you are not getting alerts redirected to the test suite.

Step 5

If your software send alerts in a format that is not parsable by any of the provided parsers in alert_message_parsers.go, you can extend that file to include your custom parser and mention that name in the config file.

Step 6

Now that everything is set up, you can run the test as follows

$ make run CONFIG=./test-example.yaml

A successful test run will exit with Congrats! All tests passed message. An unsuccessful run will end with a description of what went wrong.

The test takes roughly 40 mins to run. But if all rule groups face any errors, the test will exit immediately.

---

Running on Prometheus as an example

1. Get the rules file from step 2 above and put it in the Prometheus config.
2. Set the alertmanager URL to localhost:8080.
3. Run Prometheus with --web.enable-remote-write-receiver flag to accept remote write.
4. Run the test with make run CONFIG=./test-prometheus.yaml

Documentation

Index

• Variables
• func DoGetRequest(u string, auth config.AuthConfig) ([]byte, error)
• func NewMulti(errs ...error) multiError
• func ParseAndGroupAlerts(b []byte) (map[string][]v1.Alert, error)
• func ParseAndGroupMetrics(b []byte) (map[string][]promql.Sample, error)
• func ParseAndGroupRules(b []byte) (map[string]*v1.RuleGroup, error)
• type AlertMessageParser
• type Alerts
• type Data
• type GETAlertsResponse
• type GETMetricsResponse
• type GETRulesResponse
• type Metrics
• type RemoteWriter
  • func NewRemoteWriter(cfg agconfig.Config, logger log.Logger) (*RemoteWriter, error)
  • func (rw *RemoteWriter) AddTimeSeries(ts []prompb.TimeSeries)
  • func (rw *RemoteWriter) Error() error
  • func (rw *RemoteWriter) Start() time.Time
  • func (rw *RemoteWriter) Stop()
  • func (rw *RemoteWriter) Wait()
• type RuleGroup
• type TestSuite
  • func NewTestSuite(opts TestSuiteOptions) (*TestSuite, error)
  • func (ts *TestSuite) Error() error
  • func (ts *TestSuite) Start()
  • func (ts *TestSuite) Stop()
  • func (ts *TestSuite) TestUntil() time.Time
  • func (ts *TestSuite) Wait()
  • func (ts *TestSuite) WasTestSuccessful() (yes bool, describe string)
• type TestSuiteOptions
• type Vector

Variables

var AlertMessageParsers = map[string]AlertMessageParser{

	"default": func(b []byte) ([]notifier.Alert, error) {
		var alerts []notifier.Alert
		err := json.Unmarshal(b, &alerts)
		return alerts, err
	},

	"alertmanager": func(b []byte) ([]notifier.Alert, error) {
		msg := webhook.Message{}
		err := json.Unmarshal(b, &msg)
		if err != nil {
			return nil, err
		}
		alerts := make([]notifier.Alert, 0, len(msg.Alerts))
		for _, al := range msg.Alerts {
			alerts = append(alerts, notifier.Alert{
				Labels:       labels.FromMap(al.Labels),
				Annotations:  labels.FromMap(al.Annotations),
				StartsAt:     al.StartsAt,
				EndsAt:       al.EndsAt,
				GeneratorURL: al.GeneratorURL,
			})
		}
		return alerts, nil
	},
}

AlertMessageParsers is a map of the parser name to the parser function. The parser name is the one to be used in the config file. You can extend this map to include your custom parser and it will be matched with the config file.

Functions

func DoGetRequest

func DoGetRequest(u string, auth config.AuthConfig) ([]byte, error)

TODO: add retries and set some timeouts.

func NewMulti

func NewMulti(errs ...error) multiError

NewMulti returns multiError with provided errors added if not nil.

func ParseAndGroupAlerts

func ParseAndGroupAlerts(b []byte) (map[string][]v1.Alert, error)

ParseAndGroupAlerts parses the alerts and groups by the rule group name. The alerts are assumed to have a `rulegroup` label.

func ParseAndGroupMetrics

func ParseAndGroupMetrics(b []byte) (map[string][]promql.Sample, error)

ParseAndGroupMetrics parses samples and groups by the rule group name. The metrics are assumed to have a `rulegroup` label.

func ParseAndGroupRules

func ParseAndGroupRules(b []byte) (map[string]*v1.RuleGroup, error)

ParseAndGroupRules parses the rules and groups by the rule group name. The rules are assumed to have a `rulegroup` label.

Types

type AlertMessageParser

type AlertMessageParser func(b []byte) ([]notifier.Alert, error)

type Alerts

type Alerts struct {
	Alerts []v1.Alert `json:"alerts"`
}

type Data

type Data struct {
	RuleGroups []*RuleGroup `json:"groups"`
}

type GETAlertsResponse

type GETAlertsResponse struct {
	Status string `json:"status"`
	Data   Alerts `json:"data"`
}

type GETMetricsResponse

type GETMetricsResponse struct {
	Status string  `json:"status"`
	Data   Metrics `json:"data"`
}

type GETRulesResponse

type GETRulesResponse struct {
	Status string `json:"status"`
	Data   Data   `json:"data"`
}

type Metrics

type Metrics struct {
	ResultType string   `json:"resultType"`
	Result     []Vector `json:"result"`
}

type RemoteWriter

type RemoteWriter struct {
	// contains filtered or unexported fields
}

RemoteWriter remote writes the time series provided AddTimeSeries() in sorted fashion w.r.t. the timestamps.

func NewRemoteWriter

func NewRemoteWriter(cfg agconfig.Config, logger log.Logger) (*RemoteWriter, error)

func (*RemoteWriter) AddTimeSeries

func (rw *RemoteWriter) AddTimeSeries(ts []prompb.TimeSeries)

AddTimeSeries adds more timeseries to the queue. The timestamp of the samples should be 0 based. It should not be called after calling Start().

func (*RemoteWriter) Error

func (rw *RemoteWriter) Error() error

func (*RemoteWriter) Start

func (rw *RemoteWriter) Start() time.Time

Start starts remote-writing the given timeseries. It returns the time corresponding to the 0 timestamp.

func (*RemoteWriter) Stop

func (rw *RemoteWriter) Stop()

func (*RemoteWriter) Wait

func (rw *RemoteWriter) Wait()

type RuleGroup

type RuleGroup struct {
	Name           string            `json:"name"`
	File           string            `json:"file"`
	Rules          []v1.AlertingRule `json:"rules"`
	Interval       float64           `json:"interval"`
	EvaluationTime float64           `json:"evaluationTime"`
	LastEvaluation time.Time         `json:"lastEvaluation"`
}

type TestSuite

type TestSuite struct {
	// contains filtered or unexported fields
}

TestSuite runs the entire test suite from start to end.

func NewTestSuite

func NewTestSuite(opts TestSuiteOptions) (*TestSuite, error)

func (*TestSuite) Error

func (ts *TestSuite) Error() error

Error() returns any error occured during execution of test and does not tell if the tests passed or failed.

func (*TestSuite) Start

func (ts *TestSuite) Start()

func (*TestSuite) Stop

func (ts *TestSuite) Stop()

func (*TestSuite) TestUntil

func (ts *TestSuite) TestUntil() time.Time

func (*TestSuite) Wait

func (ts *TestSuite) Wait()

func (*TestSuite) WasTestSuccessful

func (ts *TestSuite) WasTestSuccessful() (yes bool, describe string)

WasTestSuccessful tells if all the tests passed. It returns an explanation if any test failed. Before calling this method:

• Error() should be checked for no errors.
• The test should have finished (i.e. Wait() is not blocking).

type TestSuiteOptions

type TestSuiteOptions struct {
	Logger log.Logger
	// All the test cases to test.
	Cases []cases.TestCase

	Config config.Config

	AlertMessageParser AlertMessageParser
}

type Vector

type Vector struct {
	Metric labels.Labels  `json:"metric"`
	Value  [2]interface{} `json:"value"`
}