promqltest

README

The PromQL test scripting language

This package contains two things:

• an implementation of a test scripting language for PromQL engines
• a predefined set of tests written in that scripting language

The predefined set of tests can be run against any PromQL engine implementation by calling promqltest.RunBuiltinTests(). Any other test script can be run with promqltest.RunTest().

The rest of this document explains the test scripting language.

Each test script is written in plain text.

Comments can be given by prefixing the comment with a #, for example:

# This is a comment.

Each test file contains a series of commands. There are three kinds of commands:

• load
• clear
• eval

Each command is executed in the order given in the file.

load command

load adds some data to the test environment.

The syntax is as follows:

load <interval>
    <series> <points>
    ...
    <series> <points>

• <interval> is the step between points (eg. 1m or 30s)
• <series> is a Prometheus series name in the usual metric{label="value"} syntax
• <points> is a specification of the points to add for that series, following the same expanding syntax as for promtool unittest documented here

For example:

load 1m
    my_metric{env="prod"} 5 2+3x2 _ stale {{schema:1 sum:3 count:22 buckets:[5 10 7]}}

...will create a single series with labels my_metric{env="prod"}, with the following points:

• t=0: value is 5
• t=1m: value is 2
• t=2m: value is 5
• t=3m: value is 7
• t=4m: no point
• t=5m: stale marker
• t=6m: native histogram with schema 1, sum -3, count 22 and bucket counts 5, 10 and 7

Each load command is additive - it does not replace any data loaded in a previous load command. Use clear to remove all loaded data.

Native histograms with custom buckets (NHCB)

When loading a batch of classic histogram float series, you can optionally append the suffix _with_nhcb to convert them to native histograms with custom buckets and load both the original float series and the new histogram series.

clear command

clear removes all data previously loaded with load commands.

eval command

eval runs a query against the test environment and asserts that the result is as expected.

Both instant and range queries are supported.

The syntax is as follows:

# Instant query
eval instant at <time> <query>
    <series> <points>
    ...
    <series> <points>
    
# Range query
eval range from <start> to <end> step <step> <query>
    <series> <points>
    ...
    <series> <points>

• <time> is the timestamp to evaluate the instant query at (eg. 1m)
• <start> and <end> specify the time range of the range query, and use the same syntax as <time>
• <step> is the step of the range query, and uses the same syntax as <time> (eg. 30s)
• <series> and <points> specify the expected values, and follow the same syntax as for load above

For example:

eval instant at 1m sum by (env) (my_metric)
    {env="prod"} 5
    {env="test"} 20
    
eval range from 0 to 3m step 1m sum by (env) (my_metric)
    {env="prod"} 2 5 10 20
    {env="test"} 10 20 30 45

Instant queries also support asserting that the series are returned in exactly the order specified: use eval_ordered instant ... instead of eval instant .... This is not supported for range queries.

It is also possible to test that queries fail: use eval_fail instant ... or eval_fail range .... eval_fail optionally takes an expected error message string or regexp to assert that the error message is as expected.

For example:

# Assert that the query fails for any reason without asserting on the error message.
eval_fail instant at 1m ceil({__name__=~'testmetric1|testmetric2'})

# Assert that the query fails with exactly the provided error message string.
eval_fail instant at 1m ceil({__name__=~'testmetric1|testmetric2'})
    expected_fail_message vector cannot contain metrics with the same labelset

# Assert that the query fails with an error message matching the regexp provided.
eval_fail instant at 1m ceil({__name__=~'testmetric1|testmetric2'})
    expected_fail_regexp (vector cannot contain metrics .*|something else went wrong)

Documentation

Index

• Constants
• func HistogramTestExpression(h *histogram.FloatHistogram) string
• func LoadedStorage(t testutil.T, input string) *teststorage.TestStorage
• func NewTestEngine(tb testing.TB, enablePerStepStats bool, lookbackDelta time.Duration, ...) *promql.Engine
• func NewTestEngineWithOpts(tb testing.TB, opts promql.EngineOpts) *promql.Engine
• func RunBuiltinTests(t TBRun, engine promql.QueryEngine)
• func RunTest(t testutil.T, input string, engine promql.QueryEngine)
• type LazyLoader
  • func NewLazyLoader(input string, opts LazyLoaderOpts) (*LazyLoader, error)
  • func (ll *LazyLoader) Close() error
  • func (ll *LazyLoader) Context() context.Context
  • func (ll *LazyLoader) QueryEngine() *promql.Engine
  • func (ll *LazyLoader) Queryable() storage.Queryable
  • func (ll *LazyLoader) Storage() storage.Storage
  • func (ll *LazyLoader) WithSamplesTill(ts time.Time, fn func(error))
• type LazyLoaderOpts
• type TBRun

Constants

const (
	DefaultMaxSamplesPerQuery = 10000
)

Functions

func HistogramTestExpression

func HistogramTestExpression(h *histogram.FloatHistogram) string

HistogramTestExpression returns TestExpression() for the given histogram or "" if the histogram is nil.

func LoadedStorage

func LoadedStorage(t testutil.T, input string) *teststorage.TestStorage

LoadedStorage returns storage with generated data using the provided load statements. Non-load statements will cause test errors.

func NewTestEngine

func NewTestEngine(tb testing.TB, enablePerStepStats bool, lookbackDelta time.Duration, maxSamples int) *promql.Engine

NewTestEngine creates a promql.Engine with enablePerStepStats, lookbackDelta and maxSamples, and returns it.

func NewTestEngineWithOpts

func NewTestEngineWithOpts(tb testing.TB, opts promql.EngineOpts) *promql.Engine

NewTestEngineWithOpts creates a promql.Engine with opts and returns it.

func RunBuiltinTests

func RunBuiltinTests(t TBRun, engine promql.QueryEngine)

RunBuiltinTests runs an acceptance test suite against the provided engine.

func RunTest

func RunTest(t testutil.T, input string, engine promql.QueryEngine)

RunTest parses and runs the test against the provided engine.

Types

type LazyLoader

type LazyLoader struct {
	SubqueryInterval time.Duration
	// contains filtered or unexported fields
}

LazyLoader lazily loads samples into storage. This is specifically implemented for unit testing of rules.

func NewLazyLoader

func NewLazyLoader(input string, opts LazyLoaderOpts) (*LazyLoader, error)

NewLazyLoader returns an initialized empty LazyLoader.

func (*LazyLoader) Close

func (ll *LazyLoader) Close() error

Close closes resources associated with the LazyLoader.

func (*LazyLoader) Context

func (ll *LazyLoader) Context() context.Context

Context returns the LazyLoader's context.

func (*LazyLoader) QueryEngine

func (ll *LazyLoader) QueryEngine() *promql.Engine

QueryEngine returns the LazyLoader's query engine.

func (*LazyLoader) Queryable

func (ll *LazyLoader) Queryable() storage.Queryable

Queryable allows querying the LazyLoader's data. Note: only the samples till the max timestamp used in `WithSamplesTill` can be queried.

func (*LazyLoader) Storage

func (ll *LazyLoader) Storage() storage.Storage

Storage returns the LazyLoader's storage.

func (*LazyLoader) WithSamplesTill

func (ll *LazyLoader) WithSamplesTill(ts time.Time, fn func(error))

WithSamplesTill loads the samples till given timestamp and executes the given function.

type LazyLoaderOpts

type LazyLoaderOpts struct {
	// Both of these must be set to true for regular PromQL (as of
	// Prometheus v2.33). They can still be disabled here for legacy and
	// other uses.
	EnableAtModifier, EnableNegativeOffset bool
}

LazyLoaderOpts are options for the lazy loader.

type TBRun

type TBRun interface {
	testing.TB
	Run(string, func(*testing.T)) bool
}