README
¶
vmalert
vmalert
executes a list of the given alerting
or recording
rules against configured -datasource.url
compatible with Prometheus HTTP API. For sending alerting notifications
vmalert relies on Alertmanager configured via -notifier.url
flag.
Recording rules results are persisted via remote write
protocol and require -remoteWrite.url
to be configured.
Vmalert is heavily inspired by Prometheus
implementation and aims to be compatible with its syntax.
A single-node
or cluster version
of VictoriaMetrics are capable of proxying requests to vmalert via -vmalert.proxyURL
command-line flag.
Use this feature for the following cases:
- for proxying requests from Grafana Alerting UI;
- for accessing vmalert's UI through VictoriaMetrics Web interface.
Features
- Integration with VictoriaMetrics TSDB;
- VictoriaMetrics MetricsQL support and expressions validation;
- Prometheus alerting rules definition format support;
- Integration with Alertmanager starting from Alertmanager v0.16.0-aplha;
- Keeps the alerts state on restarts;
- Graphite datasource can be used for alerting and recording rules. See these docs;
- Recording and Alerting rules backfilling (aka
replay
). See these docs; - Lightweight and without extra dependencies.
- Supports reusable templates for annotations.
Limitations
vmalert
execute queries against remote datasource which has reliability risks because of the network. It is recommended to configure alerts thresholds and rules expressions with the understanding that network requests may fail;- by default, rules execution is sequential within one group, but persistence of execution results to remote storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous recording rule is reused in the next one;
QuickStart
To build vmalert
from sources:
git clone https://github.com/VictoriaMetrics/VictoriaMetrics
cd VictoriaMetrics
make vmalert
The build binary will be placed in VictoriaMetrics/bin
folder.
To start using vmalert
you will need the following things:
- list of rules - PromQL/MetricsQL expressions to execute;
- datasource address - reachable endpoint with Prometheus HTTP API support for running queries against;
- notifier address [optional] - reachable Alert Manager instance for processing, aggregating alerts, and sending notifications. Please note, notifier address also supports Consul and DNS Service Discovery via config file.
- remote write address [optional] - remote write compatible storage to persist rules and alerts state info. To persist results to multiple destinations use vmagent configured with multiple remote writes as a proxy;
- remote read address [optional] - MetricsQL compatible datasource to restore alerts state from.
Then configure vmalert
accordingly:
./bin/vmalert -rule=alert.rules \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource
-notifier.url=http://localhost:9093 \ # AlertManager URL (required if alerting rules are used)
-notifier.url=http://127.0.0.1:9093 \ # AlertManager replica URL
-remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules and alerts state info (required if recording rules are used)
-remoteRead.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource to restore alerts state from
-external.label=cluster=east-1 \ # External label to be applied for each rule
-external.label=replica=a # Multiple external labels may be set
Note there's a separate -remoteWrite.url
command-line flag to allow writing results of
alerting/recording rules into a different storage than the initial data that's
queried. This allows using vmalert
to aggregate data from a short-term,
high-frequency, high-cardinality storage into a long-term storage with
decreased cardinality and a bigger interval between samples.
See also stream aggregation.
See the full list of configuration flags in configuration section.
If you run multiple vmalert
services for the same datastore or AlertManager - do not forget
to specify different -external.label
command-line flags in order to define which vmalert
generated rules or alerts.
Configuration for recording
and alerting rules is very
similar to Prometheus rules and configured using YAML. Configuration examples may be found
in testdata folder.
Every rule
belongs to a group
and every configuration file may contain arbitrary number of groups:
groups:
[ - <rule_group> ]
Groups
Each group has the following attributes:
# The name of the group. Must be unique within a file.
name: <string>
# How often rules in the group are evaluated.
[ interval: <duration> | default = -evaluationInterval flag ]
# Limit the number of alerts an alerting rule and series a recording
# rule can produce. 0 is no limit.
[ limit: <int> | default = 0 ]
# How many rules execute at once within a group. Increasing concurrency may speed
# up round execution speed.
[ concurrency: <integer> | default = 1 ]
# Optional type for expressions inside the rules. Supported values: "graphite" and "prometheus".
# By default "prometheus" type is used.
[ type: <string> ]
# Optional list of HTTP URL parameters
# applied for all rules requests within a group
# For example:
# params:
# nocache: ["1"] # disable caching for vmselect
# denyPartialResponse: ["true"] # fail if one or more vmstorage nodes returned an error
# extra_label: ["env=dev"] # apply additional label filter "env=dev" for all requests
# see more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements
params:
[ <string>: [<string>, ...]]
# Optional list of HTTP headers in form `header-name: value`
# applied for all rules requests within a group
# For example:
# headers:
# - "CustomHeader: foo"
# - "CustomHeader2: bar"
# Headers set via this param have priority over headers set via `-datasource.headers` flag.
headers:
[ <string>, ...]
# Optional list of labels added to every rule within a group.
# It has priority over the external labels.
# Labels are commonly used for adding environment
# or tenant-specific tag.
labels:
[ <labelname>: <labelvalue> ... ]
rules:
[ - <rule> ... ]
Rules
Every rule contains expr
field for PromQL
or MetricsQL expression. vmalert
will execute the configured
expression and then act according to the Rule type.
There are two types of Rules:
- alerting -
Alerting rules allow defining alert conditions via
expr
field and to send notifications to Alertmanager if execution result is not empty. - recording -
Recording rules allow defining
expr
which result will be then backfilled to configured-remoteWrite.url
. Recording rules are used to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.
vmalert
forbids defining duplicates - rules with the same combination of name, expression, and labels
within one group.
Alerting rules
The syntax for alerting rule is the following:
# The name of the alert. Must be a valid metric name.
alert: <string>
# The expression to evaluate. The expression language depends on the type value.
# By default, PromQL/MetricsQL expression is used. If group.type="graphite", then the expression
# must contain valid Graphite expression.
expr: <string>
# Alerts are considered firing once they have been returned for this long.
# Alerts which have not yet been fired for long enough are considered pending.
# If param is omitted or set to 0 then alerts will be immediately considered
# as firing once they return.
[ for: <duration> | default = 0s ]
# Whether to print debug information into logs.
# Information includes alerts state changes and requests sent to the datasource.
# Please note, that if rule's query params contain sensitive
# information - it will be printed to logs.
# Is applicable to alerting rules only.
[ debug: <bool> | default = false ]
# Defines the number of rule's updates entries stored in memory
# and available for view on rule's Details page.
# Overrides `rule.updateEntriesLimit` value for this specific rule.
[ update_entries_limit: <integer> | default 0 ]
# Labels to add or overwrite for each alert.
labels:
[ <labelname>: <tmpl_string> ]
# Annotations to add to each alert.
annotations:
[ <labelname>: <tmpl_string> ]
Templating
It is allowed to use Go templating in annotations to format data, iterate over or execute expressions. The following variables are available in templating:
Variable | Description | Example |
---|---|---|
$value or .Value | The current alert's value. Avoid using value in labels, it may cause unexpected issues. | {% raw %}Number of connections is {{ $value }}{% endraw %} |
$activeAt or .ActiveAt | The moment of time when alert became active (pending or firing ). |
{% raw %}http://vm-grafana.com/panelId=xx?from={{($activeAt.Add (parseDurationTime "1h")).Unix}}&to={{($activeAt.Add (parseDurationTime "-1h")).Unix}}{% endraw %} |
$labels or .Labels | The list of labels of the current alert. Use as ".Labels.<label_name>". | {% raw %}Too high number of connections for {{ .Labels.instance }}{% endraw %} |
$alertID or .AlertID | The current alert's ID generated by vmalert. | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %} |
$groupID or .GroupID | The current alert's group ID generated by vmalert. | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}{% endraw %} |
$expr or .Expr | Alert's expression. Can be used for generating links to Grafana or other systems. | {% raw %}/api/v1/query?query={{ $expr|queryEscape }}{% endraw %} |
$for or .For | Alert's configured for param. | {% raw %}Number of connections is too high for more than {{ .For }}{% endraw %} |
$externalLabels or .ExternalLabels | List of labels configured via -external.label command-line flag. |
{% raw %}Issues with {{ $labels.instance }} (datacenter-{{ $externalLabels.dc }}){% endraw %} |
$externalURL or .ExternalURL | URL configured via -external.url command-line flag. Used for cases when vmalert is hidden behind proxy. |
{% raw %}Visit {{ $externalURL }} for more details{% endraw %} |
Additionally, vmalert
provides some extra templating functions listed here and reusable templates.
Template functions
vmalert
provides the following template functions, which can be used during templating:
args arg0 ... argN
- converts the input args into a map witharg0
, ...,argN
keys.externalURL
- returns the value of-external.url
command-line flag.first
- returns the first result from the input query results returned byquery
function.htmlEscape
- escapes special chars in input string, so it can be safely embedded as a plaintext into HTML.humanize
- converts the input number into human-readable format by adding metric prefixes. For example,100000
is converted into100K
.humanize1024
- converts the input number into human-readable format with 1024 base. For example,1024
is converted into 1ki`.humanizeDuration
- converts the input number in seconds into human-readable duration.humanizePercentage
- converts the input number to percentage. For example,0.123
is converted into12.3%
.humanizeTimestamp
- converts the input unix timestamp into human-readable time.jsonEscape
- JSON-encodes the input string.label name
- returns the value of the label with the givenname
from the input query result.match regex
- matches the input string against the providedregex
.parseDuration
- parses the input string into duration in seconds. For example,1h
is parsed into3600
.parseDurationTime
- parses the input string into time.Duration.pathEscape
- escapes the input string, so it can be safely put inside path part of URL.pathPrefix
- returns the path part of the-external.url
command-line flag.query
- executes the MetricsQL query against-datasource.url
and returns the query result. For example, {% raw %}{{ query "sort_desc(process_resident_memory_bytes)" | first | value }}
{% endraw %} executes thesort_desc(process_resident_memory_bytes)
query at-datasource.url
and returns the first result.queryEscape
- escapes the input string, so it can be safely put inside query arg part of URL.quotesEscape
- escapes the input string, so it can be safely embedded into JSON string.reReplaceAll regex repl
- replaces all the occurences of theregex
in input string with therepl
.safeHtml
- marks the input string as safe to use in HTML context without the need to html-escape it.sortByLabel name
- sorts the input query results by the label with the givenname
.stripDomain
- leaves the first part of the domain. For example,foo.bar.baz
is converted tofoo
. The port part is left in the output string. E.g.foo.bar:1234
is converted intofoo:1234
.stripPort
- stripsport
part fromhost:port
input string.strvalue
- returns the metric name from the input query result.title
- converts the first letters of every input word to uppercase.toLower
- converts all the chars in the input string to lowercase.toTime
- converts the input unix timestamp to time.Time.toUpper
- converts all the chars in the input string to uppercase.value
- returns the numeric value from the input query result.
Reusable templates
Like in Alertmanager you can define reusable templates
to share same templates across annotations. Just define the templates in a file and
set the path via -rule.templates
flag.
For example, template grafana.filter
can be defined as following:
{% raw %}
{{ define "grafana.filter" -}}
{{- $labels := .arg0 -}}
{{- range $name, $label := . -}}
{{- if (ne $name "arg0") -}}
{{- ( or (index $labels $label) "All" ) | printf "&var-%s=%s" $label -}}
{{- end -}}
{{- end -}}
{{- end -}}
{% endraw %}
And then used in annotations:
{% raw %}
groups:
- name: AlertGroupName
rules:
- alert: AlertName
expr: any_metric > 100
for: 30s
labels:
alertname: 'Any metric is too high'
severity: 'warning'
annotations:
dashboard: '{{ $externalURL }}/d/dashboard?orgId=1{{ template "grafana.filter" (args .CommonLabels "account_id" "any_label") }}'
{% endraw %}
The -rule.templates
flag supports wildcards so multiple files with templates can be loaded.
The content of -rule.templates
can be also hot reloaded.
Recording rules
The syntax for recording rules is following:
# The name of the time series to output to. Must be a valid metric name.
record: <string>
# The expression to evaluate. The expression language depends on the type value.
# By default, MetricsQL expression is used. If group.type="graphite", then the expression
# must contain valid Graphite expression.
expr: <string>
# Labels to add or overwrite before storing the result.
labels:
[ <labelname>: <labelvalue> ]
# Defines the number of rule's updates entries stored in memory
# and available for view on rule's Details page.
# Overrides `rule.updateEntriesLimit` value for this specific rule.
[ update_entries_limit: <integer> | default 0 ]
For recording rules to work -remoteWrite.url
must be specified.
Alerts state on restarts
vmalert
has no local storage, so alerts state is stored in the process memory. Hence, after restart of vmalert
the process alerts state will be lost. To avoid this situation, vmalert
should be configured via the following flags:
-remoteWrite.url
- URL to VictoriaMetrics (Single) or vminsert (Cluster).vmalert
will persist alerts state into the configured address in the form of time series namedALERTS
andALERTS_FOR_STATE
via remote-write protocol. These are regular time series and maybe queried from VM just as any other time series. The state is stored to the configured address on every rule evaluation.-remoteRead.url
- URL to VictoriaMetrics (Single) or vmselect (Cluster).vmalert
will try to restore alerts state from configured address by querying time series with nameALERTS_FOR_STATE
.
Both flags are required for proper state restoration. Restore process may fail if time series are missing
in configured -remoteRead.url
, weren't updated in the last 1h
(controlled by -remoteRead.lookback
)
or received state doesn't match current vmalert
rules configuration.
Multitenancy
There are the following approaches exist for alerting and recording rules across multiple tenants:
-
To run a separate
vmalert
instance per each tenant. The corresponding tenant must be specified in-datasource.url
command-line flag according to these docs. For example,/path/to/vmalert -datasource.url=http://vmselect:8481/select/123/prometheus
would run alerts againstAccountID=123
. For recording rules the-remoteWrite.url
command-line flag must contain the url for the specific tenant as well. For example,-remoteWrite.url=http://vminsert:8480/insert/123/prometheus
would write recording rules toAccountID=123
. -
To specify
tenant
parameter per each alerting and recording group if enterprise version of vmalert is used with-clusterMode
command-line flag. For example:
groups:
- name: rules_for_tenant_123
tenant: "123"
rules:
# Rules for accountID=123
- name: rules_for_tenant_456:789
tenant: "456:789"
rules:
# Rules for accountID=456, projectID=789
The results of alerting and recording rules contain vm_account_id
and vm_project_id
labels
if -clusterMode
is enabled. These labels can be used during templating,
and help to identify to which account or project the triggered alert or produced recording belongs.
If -clusterMode
is enabled, then -datasource.url
, -remoteRead.url
and -remoteWrite.url
must
contain only the hostname without tenant id. For example: -datasource.url=http://vmselect:8481
.
vmalert
automatically adds the specified tenant to urls per each recording rule in this case.
If -clusterMode
is enabled and the tenant
in a particular group is missing, then the tenant value
is obtained from -defaultTenant.prometheus
or -defaultTenant.graphite
depending on the type
of the group.
The enterprise version of vmalert is available in vmutils-*-enterprise.tar.gz
files
at release page and in *-enterprise
tags at Docker Hub.
Topology examples
The following sections are showing how vmalert
may be used and configured
for different scenarios.
Please note, not all flags in examples are required:
-remoteWrite.url
and-remoteRead.url
are optional and are needed only if you have recording rules or want to store alerts state onvmalert
restarts;-notifier.url
is optional and is needed only if you have alerting rules.
Single-node VictoriaMetrics
The simplest configuration where one single-node VM server is used for rules execution, storing recording rules results and alerts state.
vmalert
configuration flags:
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
![vmalert single](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_single.png)
Cluster VictoriaMetrics
In cluster mode
VictoriaMetrics has separate components for writing and reading path:
vminsert
and vmselect
components respectively. vmselect
is used for executing rules expressions
and vminsert
is used to persist recording rules results and alerts state.
Cluster mode could have multiple vminsert
and vmselect
components.
vmalert
configuration flags:
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions
-remoteWrite.url=http://vminsert:8480/insert/0/prometheus # vminsert addr to persist alerts state and recording rules results
-remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
![vmalert cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_cluster.png)
In case when you want to spread the load on these components - add balancers before them and configure
vmalert
with balancer's addresses. Please, see more about VM's cluster architecture
here.
HA vmalert
For HA user can run multiple identically configured vmalert
instances.
It means all of them will execute the same rules, write state and results to
the same destinations, and send alert notifications to multiple configured
Alertmanagers.
vmalert
configuration flags:
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
-notifier.url=http://alertmanager1:9093 \ # Multiple AlertManager addresses to send alerts when they trigger
-notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers
![vmalert ha](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_ha.png)
To avoid recording rules results and alerts state duplication in VictoriaMetrics server
don't forget to configure deduplication.
The recommended value for -dedup.minScrapeInterval
must be greater or equal to vmalert's evaluation_interval
.
If you observe inconsistent or "jumping" values in series produced by vmalert, try disabling -datasource.queryTimeAlignment
command line flag. Because of alignment, two or more vmalert HA pairs will produce results with the same timestamps.
But due of backfilling (data delivered to the datasource with some delay) values of such results may differ,
which would affect deduplication logic and result into "jumping" datapoints.
Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
all vmalert
s are having the same config.
Don't forget to configure cluster mode
for Alertmanagers for better reliability. List all Alertmanager URLs in vmalert's -notifier.url
to ensure high availability.
This example uses single-node VM server for the sake of simplicity. Check how to replace it with cluster VictoriaMetrics if needed.
Downsampling and aggregation via vmalert
vmalert
can't modify existing data. But it can run arbitrary PromQL/MetricsQL queries
via recording rules and backfill results to the configured -remoteWrite.url
.
This ability allows to aggregate data. For example, the following rule will calculate the average value for
metric http_requests
on the 5m
interval:
- record: http_requests:avg5m
expr: avg_over_time(http_requests[5m])
Every time this rule will be evaluated, vmalert
will backfill its results as a new time series http_requests:avg5m
to the configured -remoteWrite.url
.
vmalert
executes rules with specified interval (configured via flag -evaluationInterval
or as group's interval
param). The interval helps to control "resolution" of the produced series.
This ability allows to downsample data. For example, the following config will execute the rule only once every 5m
:
groups:
- name: my_group
interval: 5m
rules:
- record: http_requests:avg5m
expr: avg_over_time(http_requests[5m])
Ability of vmalert
to be configured with different -datasource.url
and -remoteWrite.url
command-line flags
allows reading data from one data source and backfilling results to another. This helps to build a system
for aggregating and downsampling the data.
The following example shows how to build a topology where vmalert
will process data from one cluster
and write results into another. Such clusters may be called as "hot" (low retention,
high-speed disks, used for operative monitoring) and "cold" (long term retention,
slower/cheaper disks, low resolution data). With help of vmalert
, user can setup
recording rules to process raw data from "hot" cluster (by applying additional transformations
or reducing resolution) and push results to "cold" cluster.
vmalert
configuration flags:
./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recording rules expressions
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheus # vminsert addr to persist recording rules results
![vmalert multi cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_multicluster.png)
Please note, replay feature may be used for transforming historical data.
Flags -remoteRead.url
and -notifier.url
are omitted since we assume only recording rules are used.
See also stream aggregation and downsampling.
Multiple remote writes
For persisting recording or alerting rule results vmalert
requires -remoteWrite.url
to be set.
But this flag supports only one destination. To persist rule results to multiple destinations
we recommend using vmagent as fan-out proxy:
![vmalert multiple remote write destinations](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_multiple_rw.png)
In this topology, vmalert
is configured to persist rule results to vmagent
. And vmagent
is configured to fan-out received data to two or more destinations.
Using vmagent
as a proxy provides additional benefits such as
data persisting when storage is unreachable,
or time series modification via relabeling.
Web
vmalert
runs a web-server (-httpListenAddr
) for serving metrics and alerts endpoints:
http://<vmalert-addr>
- UI;http://<vmalert-addr>/api/v1/rules
- list of all loaded groups and rules;http://<vmalert-addr>/api/v1/alerts
- list of all active alerts;http://<vmalert-addr>/vmalert/api/v1/alert?group_id=<group_id>&alert_id=<alert_id>
- get alert status in JSON format. Used as alert source in AlertManager.http://<vmalert-addr>/vmalert/alert?group_id=<group_id>&alert_id=<alert_id>
- get alert status in web UI.http://<vmalert-addr>/vmalert/rule?group_id=<group_id>&rule_id=<rule_id>
- get rule status in web UI.http://<vmalert-addr>/metrics
- application metrics.http://<vmalert-addr>/-/reload
- hot configuration reload.
vmalert
web UI can be accessed from single-node version of VictoriaMetrics
and from cluster version of VictoriaMetrics.
This may be used for better integraion with Grafana unified alerting system. See the following docs for details:
- How to query vmalert from single-node VictoriaMetrics
- How to query vmalert from VictoriaMetrics cluster
Graphite
vmalert sends requests to <-datasource.url>/render?format=json
during evaluation of alerting and recording rules
if the corresponding group or rule contains type: "graphite"
config option. It is expected that the <-datasource.url>/render
implements Graphite Render API for format=json
.
When using vmalert with both graphite
and prometheus
rules configured against cluster version of VM do not forget
to set -datasource.appendTypePrefix
flag to true
, so vmalert can adjust URL prefix automatically based on the query type.
Rules backfilling
vmalert supports alerting and recording rules backfilling (aka replay
). In replay mode vmalert
can read the same rules configuration as normal, evaluate them on the given time range and backfill
results via remote write to the configured storage. vmalert supports any PromQL/MetricsQL compatible
data source for backfilling.
How it works
In replay
mode vmalert works as a cli-tool and exits immediately after work is done.
To run vmalert in replay
mode:
./bin/vmalert -rule=path/to/your.rules \ # path to files with rules you usually use with vmalert
-datasource.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource
-remoteWrite.url=http://localhost:8428 \ # remote write compatible storage to persist results
-replay.timeFrom=2021-05-11T07:21:43Z \ # time from begin replay
-replay.timeTo=2021-05-29T18:40:43Z # time to finish replay
The output of the command will look like the following:
Replay mode:
from: 2021-05-11 07:21:43 +0000 UTC # set by -replay.timeFrom
to: 2021-05-29 18:40:43 +0000 UTC # set by -replay.timeTo
max data points per request: 1000 # set by -replay.maxDatapointsPerQuery
Group "ReplayGroup"
interval: 1m0s
requests to make: 27
max range per request: 16h40m0s
> Rule "type:vm_cache_entries:rate5m" (ID: 1792509946081842725)
27 / 27 [----------------------------------------------------------------------------------------------------] 100.00% 78 p/s
> Rule "go_cgo_calls_count:rate5m" (ID: 17958425467471411582)
27 / 27 [-----------------------------------------------------------------------------------------------------] 100.00% ? p/s
Group "vmsingleReplay"
interval: 30s
requests to make: 54
max range per request: 8h20m0s
> Rule "RequestErrorsToAPI" (ID: 17645863024999990222)
54 / 54 [-----------------------------------------------------------------------------------------------------] 100.00% ? p/s
> Rule "TooManyLogs" (ID: 9042195394653477652)
54 / 54 [-----------------------------------------------------------------------------------------------------] 100.00% ? p/s
2021-06-07T09:59:12.098Z info app/vmalert/replay.go:68 replay finished! Imported 511734 samples
In replay
mode all groups are executed sequentially one-by-one. Rules within the group are
executed sequentially as well (concurrency
setting is ignored). Vmalert sends rule's expression
to /query_range endpoint
of the configured -datasource.url
. Returned data is then processed according to the rule type and
backfilled to -remoteWrite.url
via remote Write protocol.
Vmalert respects evaluationInterval
value set by flag or per-group during the replay.
Vmalert automatically disables caching on VictoriaMetrics side by sending nocache=1
param. It allows
to prevent cache pollution and unwanted time range boundaries adjustment during backfilling.
Recording rules
The result of recording rules replay
should match with results of normal rules evaluation.
Alerting rules
The result of alerting rules replay
is time series reflecting alert's state.
To see if replayed
alert has fired in the past use the following PromQL/MetricsQL expression:
ALERTS{alertname="your_alertname", alertstate="firing"}
Execute the query against storage which was used for -remoteWrite.url
during the replay
.
Additional configuration
There are following non-required replay
flags:
-replay.maxDatapointsPerQuery
- the max number of data points expected to receive in one request. In two words, it affects the max time range for every/query_range
request. The higher the value, the fewer requests will be issued duringreplay
.-replay.ruleRetryAttempts
- when datasource fails to respond vmalert will make this number of retries per rule before giving up.-replay.rulesDelay
- delay between sequential rules execution. Important in cases if there are chaining (rules which depend on each other) rules. It is expected, that remote storage will be able to persist previously accepted data during the delay, so data will be available for the subsequent queries. Keep it equal or bigger than-remoteWrite.flushInterval
.-replay.disableProgressBar
- whether to disable progress bar which shows progress work. Progress bar may generate a lot of log records, which is not formatted as standard VictoriaMetrics logger. It could break logs parsing by external system and generate additional load on it.
See full description for these flags in ./vmalert -help
.
Limitations
- Graphite engine isn't supported yet;
query
template function is disabled for performance reasons (might be changed in future);limit
group's param has no effect during replay (might be changed in future);
Monitoring
vmalert
exports various metrics in Prometheus exposition format at http://vmalert-host:8880/metrics
page.
The default list of alerting rules for these metric can be found here.
We recommend setting up regular scraping of this page either through vmagent
or by Prometheus so that the exported
metrics may be analyzed later.
Use the official Grafana dashboard for vmalert
overview.
Graphs on this dashboard contain useful hints - hover the i
icon in the top left corner of each graph in order to read it.
If you have suggestions for improvements or have found a bug - please open an issue on github or add
a review to the dashboard.
Troubleshooting
vmalert executes configured rules within certain intervals. It is expected that at the moment when rule is executed,
the data is already present in configured -datasource.url
:
![vmalert expected evaluation](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_ts_normal.gif)
Usually, troubles start to appear when data in -datasource.url
is delayed or absent. In such cases, evaluations
may get empty response from datasource and produce empty recording rules or reset alerts state:
![vmalert evaluation when data is delayed](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_ts_data_delay.gif)
By default, recently written samples to VictoriaMetrics aren't visible for queries for up to 30s.
This behavior is controlled by -search.latencyOffset
command-line flag and the latency_offset
query ag at vmselect
.
Usually, this results into a 30s shift for recording rules results.
Note that too small value passed to -search.latencyOffset
or to latency_offest
query arg may lead to incomplete query results.
Try the following recommendations in such cases:
- Always configure group's
evaluationInterval
to be bigger or equal toscrape_interval
at which metrics are delivered to the datasource; - If you know in advance, that data in datasource is delayed - try changing vmalert's
-datasource.lookback
command-line flag to add a time shift for evaluations; - If time intervals between datapoints in datasource are irregular or
>=5min
- try changing vmalert's-datasource.queryStep
command-line flag to specify how far search query can lookback for the recent datapoint. The recommendation is to have the step at least two times bigger thanscrape_interval
, since there are no guarantees that scrape will not fail.
Sometimes, it is not clear why some specific alert fired or didn't fire. It is very important to remember, that
alerts with for: 0
fire immediately when their expression becomes true. And alerts with for > 0
will fire only
after multiple consecutive evaluations, and at each evaluation their expression must be true. If at least one evaluation
becomes false, then alert's state resets to the initial state.
If -remoteWrite.url
command-line flag is configured, vmalert will persist alert's state in form of time series
ALERTS
and ALERTS_FOR_STATE
to the specified destination. Such time series can be then queried via
vmui or Grafana to track how alerts state
changed in time.
vmalert stores last -rule.updateEntriesLimit
(or update_entries_limit
per-rule config)
state updates for each rule. To check updates, click on Details
link next to rule's name on /vmalert/groups
page
and check the Last updates
section:
![vmalert state](https://github.com/VictoriaMetrics/VictoriaMetrics/raw/v1.87.13-cluster/app/vmalert/vmalert_state.png)
Rows in the section represent ordered rule evaluations and their results. The column curl
contains an example of
HTTP request sent by vmalert to the -datasource.url
during evaluation. If specific state shows that there were
no samples returned and curl command returns data - then it is very likely there was no data in datasource on the
moment when rule was evaluated.
vmalert allows configuring more detailed logging for specific alerting rule. Just set debug: true
in rule's configuration
and vmalert will start printing additional log messages:
2022-09-15T13:35:41.155Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:41+02:00: query returned 0 samples (elapsed: 5.896041ms)
2022-09-15T13:35:56.149Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true&query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29+by%28instance%29+%3E+0&step=15s&time=1663248945"
2022-09-15T13:35:56.178Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: query returned 1 samples (elapsed: 28.368208ms)
2022-09-15T13:35:56.178Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true&query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29&step=15s&time=1663248945"
2022-09-15T13:35:56.179Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} created in state PENDING
...
2022-09-15T13:36:56.153Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
Profiling
vmalert
provides handlers for collecting the following Go profiles:
- Memory profile. It can be collected with the following command (replace
0.0.0.0
with hostname if needed):
curl http://0.0.0.0:8880/debug/pprof/heap > mem.pprof
- CPU profile. It can be collected with the following command (replace
0.0.0.0
with hostname if needed):
curl http://0.0.0.0:8880/debug/pprof/profile > cpu.pprof
The command for collecting CPU profile waits for 30 seconds before returning.
The collected profiles may be analyzed with go tool pprof. It is safe sharing the collected profiles from security point of view, since they do not contain sensitive information.
Configuration
Flags
Pass -help
to vmalert
in order to see the full list of supported
command-line flags with their descriptions.
The shortlist of configuration flags is the following: {% raw %}
-clusterMode
If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
-configCheckInterval duration
Interval for checking for changes in '-rule' or '-notifier.config' files. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes.
-datasource.appendTypePrefix
Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.
-datasource.basicAuth.password string
Optional basic auth password for -datasource.url
-datasource.basicAuth.passwordFile string
Optional path to basic auth password to use for -datasource.url
-datasource.basicAuth.username string
Optional basic auth username for -datasource.url
-datasource.bearerToken string
Optional bearer auth token to use for -datasource.url.
-datasource.bearerTokenFile string
Optional path to bearer token file to use for -datasource.url.
-datasource.disableKeepAlive
Whether to disable long-lived connections to the datasource. If true, disables HTTP keep-alives and will only use the connection to the server for a single HTTP request.
-datasource.headers string
Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. For example, -datasource.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -datasource.url. Multiple headers must be delimited by '^^': -datasource.headers='header1:value1^^header2:value2'
-datasource.lookback duration
Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
-datasource.maxIdleConnections int
Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
-datasource.oauth2.clientID string
Optional OAuth2 clientID to use for -datasource.url.
-datasource.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -datasource.url.
-datasource.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -datasource.url.
-datasource.oauth2.scopes string
Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'
-datasource.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -datasource.url.
-datasource.queryStep duration
How far a value can fallback to when evaluating queries. For example, if -datasource.queryStep=15s then param "step" with value "15s" will be added to every query. If set to 0, rule's evaluation interval will be used instead. (default 5m0s)
-datasource.queryTimeAlignment
Whether to align "time" parameter with evaluation interval.Alignment supposed to produce deterministic results despite of number of vmalert replicas or time they were started. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1257 (default true)
-datasource.roundDigits int
Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
-datasource.showURL
Whether to show -datasource.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
-datasource.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used
-datasource.tlsCertFile string
Optional path to client-side TLS certificate file to use when connecting to -datasource.url
-datasource.tlsInsecureSkipVerify
Whether to skip tls verification when connecting to -datasource.url
-datasource.tlsKeyFile string
Optional path to client-side TLS certificate key to use when connecting to -datasource.url
-datasource.tlsServerName string
Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used
-datasource.url string
Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL
-defaultTenant.graphite string
Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy .This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
-defaultTenant.prometheus string
Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
-disableAlertgroupLabel
Whether to disable adding group's Name as label to generated alerts and time series.
-dryRun
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP and UDP is used
-envflag.enable
Whether to enable reading flags from environment variables additionally to command line. Command line flag values have priority over values from environment vars. Flags are read only from command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
-evaluationInterval duration
How often to evaluate the rules (default 1m0s)
-external.alert.source string
External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr":{{$expr|jsonEscape|queryEscape}} },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]' . If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
-external.label array
Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
Supports an array of values separated by comma or specified via multiple flags.
-external.url string
External URL is used as alert's source for sent alerts to the notifier
-flagsAuthKey string
Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
-fs.disableMmap
Whether to use pread() instead of mmap() for reading data files. By default mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
-http.connTimeout duration
Incoming http connections are closed after the configured timeout. This may help to spread the incoming load among a cluster of services behind a load balancer. Please note that the real timeout may be bigger by up to 10% as a protection against the thundering herd problem (default 2m0s)
-http.disableResponseCompression
Disable compression of HTTP responses to save CPU resources. By default compression is enabled to save network bandwidth
-http.idleConnTimeout duration
Timeout for incoming idle http connections (default 1m0s)
-http.maxGracefulShutdownDuration duration
The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s)
-http.pathPrefix string
An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
-http.shutdownDelay duration
Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
-httpAuth.password string
Password for HTTP Basic Auth. The authentication is disabled if -httpAuth.username is empty
-httpAuth.username string
Username for HTTP Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
-httpListenAddr string
Address to listen for http connections. See also -httpListenAddr.useProxyProtocol (default ":8880")
-httpListenAddr.useProxyProtocol
Whether to use proxy protocol for connections accepted at -httpListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
-loggerDisableTimestamps
Whether to disable writing timestamps in logs
-loggerErrorsPerSecondLimit int
Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, the remaining errors are suppressed. Zero values disable the rate limit
-loggerFormat string
Format for logs. Possible values: default, json (default "default")
-loggerJSONFields string
Allows renaming fields in JSON formatted logs. Example: "ts:timestamp,msg:message" renames "ts" to "timestamp" and "msg" to "message". Supported fields: ts, level, caller, msg
-loggerLevel string
Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
-loggerOutput string
Output for the logs. Supported values: stderr, stdout (default "stderr")
-loggerTimezone string
Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
-loggerWarnsPerSecondLimit int
Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
-metricsAuthKey string
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
-notifier.basicAuth.password array
Optional basic auth password for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.basicAuth.passwordFile array
Optional path to basic auth password file for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.basicAuth.username array
Optional basic auth username for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.bearerToken array
Optional bearer token for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.bearerTokenFile array
Optional path to bearer token file for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.config string
Path to configuration file for notifiers
-notifier.oauth2.clientID array
Optional OAuth2 clientID to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.clientSecret array
Optional OAuth2 clientSecret to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.clientSecretFile array
Optional OAuth2 clientSecretFile to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.scopes array
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.tokenUrl array
Optional OAuth2 tokenURL to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.suppressDuplicateTargetErrors
Whether to suppress 'duplicate target' errors during discovery
-notifier.tlsCAFile array
Optional path to TLS CA file to use for verifying connections to -notifier.url. By default system CA is used
Supports an array of values separated by comma or specified via multiple flags.
-notifier.tlsCertFile array
Optional path to client-side TLS certificate file to use when connecting to -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.tlsInsecureSkipVerify array
Whether to skip tls verification when connecting to -notifier.url
Supports array of values separated by comma or specified via multiple flags.
-notifier.tlsKeyFile array
Optional path to client-side TLS certificate key to use when connecting to -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.tlsServerName array
Optional TLS server name to use for connections to -notifier.url. By default the server name from -notifier.url is used
Supports an array of values separated by comma or specified via multiple flags.
-notifier.url array
Prometheus Alertmanager URL, e.g. http://127.0.0.1:9093. List all Alertmanager URLs if it runs in the cluster mode to ensure high availability.
Supports an array of values separated by comma or specified via multiple flags.
-pprofAuthKey string
Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides httpAuth.* settings
-promscrape.consul.waitTime duration
Wait time used by Consul service discovery. Default value is used if not set
-promscrape.consulSDCheckInterval duration
Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#consul_sd_configs for details (default 30s)
-promscrape.discovery.concurrency int
The maximum number of concurrent requests to Prometheus autodiscovery API (Consul, Kubernetes, etc.) (default 100)
-promscrape.discovery.concurrentWaitTime duration
The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s)
-promscrape.dnsSDCheckInterval duration
Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#dns_sd_configs for details (default 30s)
-pushmetrics.extraLabel array
Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags.
-pushmetrics.interval duration
Interval for pushing metrics to -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
-remoteRead.basicAuth.password string
Optional basic auth password for -remoteRead.url
-remoteRead.basicAuth.passwordFile string
Optional path to basic auth password to use for -remoteRead.url
-remoteRead.basicAuth.username string
Optional basic auth username for -remoteRead.url
-remoteRead.bearerToken string
Optional bearer auth token to use for -remoteRead.url.
-remoteRead.bearerTokenFile string
Optional path to bearer token file to use for -remoteRead.url.
-remoteRead.disablePathAppend
Whether to disable automatic appending of '/api/v1/query' path to the configured -datasource.url and -remoteRead.url
-remoteRead.headers string
Optional HTTP headers to send with each request to the corresponding -remoteRead.url. For example, -remoteRead.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteRead.url. Multiple headers must be delimited by '^^': -remoteRead.headers='header1:value1^^header2:value2'
-remoteRead.ignoreRestoreErrors
Whether to ignore errors from remote storage when restoring alerts state on startup. DEPRECATED - this flag has no effect and will be removed in the next releases. (default true)
-remoteRead.lookback duration
Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)
-remoteRead.oauth2.clientID string
Optional OAuth2 clientID to use for -remoteRead.url.
-remoteRead.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -remoteRead.url.
-remoteRead.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -remoteRead.url.
-remoteRead.oauth2.scopes string
Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.
-remoteRead.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -remoteRead.url.
-remoteRead.showURL
Whether to show -remoteRead.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
-remoteRead.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -remoteRead.url. By default system CA is used
-remoteRead.tlsCertFile string
Optional path to client-side TLS certificate file to use when connecting to -remoteRead.url
-remoteRead.tlsInsecureSkipVerify
Whether to skip tls verification when connecting to -remoteRead.url
-remoteRead.tlsKeyFile string
Optional path to client-side TLS certificate key to use when connecting to -remoteRead.url
-remoteRead.tlsServerName string
Optional TLS server name to use for connections to -remoteRead.url. By default the server name from -remoteRead.url is used
-remoteRead.url vmalert
Optional URL to datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect.Remote read is used to restore alerts state.This configuration makes sense only if vmalert was configured with `remoteWrite.url` before and has been successfully persisted its state. E.g. http://127.0.0.1:8428. See also '-remoteRead.disablePathAppend', '-remoteRead.showURL'.
-remoteWrite.basicAuth.password string
Optional basic auth password for -remoteWrite.url
-remoteWrite.basicAuth.passwordFile string
Optional path to basic auth password to use for -remoteWrite.url
-remoteWrite.basicAuth.username string
Optional basic auth username for -remoteWrite.url
-remoteWrite.bearerToken string
Optional bearer auth token to use for -remoteWrite.url.
-remoteWrite.bearerTokenFile string
Optional path to bearer token file to use for -remoteWrite.url.
-remoteWrite.concurrency int
Defines number of writers for concurrent writing into remote querier (default 1)
-remoteWrite.disablePathAppend
Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.
-remoteWrite.flushInterval duration
Defines interval of flushes to remote write endpoint (default 5s)
-remoteWrite.headers string
Optional HTTP headers to send with each request to the corresponding -remoteWrite.url. For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteWrite.url. Multiple headers must be delimited by '^^': -remoteWrite.headers='header1:value1^^header2:value2'
-remoteWrite.maxBatchSize int
Defines defines max number of timeseries to be flushed at once (default 1000)
-remoteWrite.maxQueueSize int
Defines the max number of pending datapoints to remote write endpoint (default 100000)
-remoteWrite.oauth2.clientID string
Optional OAuth2 clientID to use for -remoteWrite.url.
-remoteWrite.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -remoteWrite.url.
-remoteWrite.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -remoteWrite.url.
-remoteWrite.oauth2.scopes string
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.
-remoteWrite.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -notifier.url.
-remoteWrite.sendTimeout duration
Timeout for sending data to the configured -remoteWrite.url. (default 30s)
-remoteWrite.showURL
Whether to show -remoteWrite.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
-remoteWrite.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. By default system CA is used
-remoteWrite.tlsCertFile string
Optional path to client-side TLS certificate file to use when connecting to -remoteWrite.url
-remoteWrite.tlsInsecureSkipVerify
Whether to skip tls verification when connecting to -remoteWrite.url
-remoteWrite.tlsKeyFile string
Optional path to client-side TLS certificate key to use when connecting to -remoteWrite.url
-remoteWrite.tlsServerName string
Optional TLS server name to use for connections to -remoteWrite.url. By default the server name from -remoteWrite.url is used
-remoteWrite.url string
Optional URL to VictoriaMetrics or vminsert where to persist alerts state and recording rules results in form of timeseries. For example, if -remoteWrite.url=http://127.0.0.1:8428 is specified, then the alerts state will be written to http://127.0.0.1:8428/api/v1/write . See also -remoteWrite.disablePathAppend, '-remoteWrite.showURL'.
-replay.disableProgressBar
Whether to disable rendering progress bars during the replay. Progress bar rendering might be verbose or break the logs parsing, so it is recommended to be disabled when not used in interactive mode.
-replay.maxDatapointsPerQuery int
Max number of data points expected in one request. It affects the max time range for every `/query_range` request during the replay. The higher the value, the less requests will be made during replay. (default 1000)
-replay.ruleRetryAttempts int
Defines how many retries to make before giving up on rule if request for it returns an error. (default 5)
-replay.rulesDelay duration
Delay between rules evaluation within the group. Could be important if there are chained rules inside of the groupand processing need to wait for previous rule results to be persisted by remote storage before evaluating the next rule.Keep it equal or bigger than -remoteWrite.flushInterval. (default 1s)
-replay.timeFrom string
The time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z'
-replay.timeTo string
The time filter in RFC3339 format to select timeseries with timestamp equal or lower than provided value. E.g. '2020-01-01T20:07:00Z'
-rule array
Path to the file with alert rules.
Supports patterns. Flag can be specified multiple times.
Examples:
-rule="/path/to/file". Path to a single file with alerting rules
-rule="dir/*.yaml" -rule="/*.yaml". Relative path to all .yaml files in "dir" folder,
absolute path to all .yaml files in root.
Rule files may contain %{ENV_VAR} placeholders, which are substituted by the corresponding env vars.
Supports an array of values separated by comma or specified via multiple flags.
-rule.configCheckInterval duration
Interval for checking for changes in '-rule' files. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes. DEPRECATED - see '-configCheckInterval' instead
-rule.maxResolveDuration duration
Limits the maximum duration for automatic alert expiration, which is by default equal to 3 evaluation intervals of the parent group.
-rule.resendDelay duration
Minimum amount of time to wait before resending an alert to notifier
-rule.templates array
Path or glob pattern to location with go template definitions
for rules annotations templating. Flag can be specified multiple times.
Examples:
-rule.templates="/path/to/file". Path to a single file with go templates
-rule.templates="dir/*.tpl" -rule.templates="/*.tpl". Relative path to all .tpl files in "dir" folder,
absolute path to all .tpl files in root.
Supports an array of values separated by comma or specified via multiple flags.
-rule.updateEntriesLimit int
Defines the max number of rule's state updates stored in-memory. Rule's updates are available on rule's Details page and are used for debugging purposes. The number of stored updates can be overriden per rule via update_entries_limit param. (default 20)
-rule.validateExpressions
Whether to validate rules expressions via MetricsQL engine (default true)
-rule.validateTemplates
Whether to validate annotation and label templates (default true)
-tls
Whether to enable TLS for incoming HTTP requests at -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set
-tlsCertFile string
Path to file with TLS certificate if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated
-tlsCipherSuites array
Optional list of TLS cipher suites for incoming requests over HTTPS if -tls is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants
Supports an array of values separated by comma or specified via multiple flags.
-tlsKeyFile string
Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
-tlsMinVersion string
Optional minimum TLS version to use for incoming requests over HTTPS if -tls is set. Supported values: TLS10, TLS11, TLS12, TLS13
-version
Show VictoriaMetrics version
{% endraw %}
Hot config reload
vmalert
supports "hot" config reload via the following methods:
- send SIGHUP signal to
vmalert
process; - send GET request to
/-/reload
endpoint; - configure
-configCheckInterval
flag for periodic reload on config change.
URL params
To set additional URL params for datasource.url
, remoteWrite.url
or remoteRead.url
just add them in address: -datasource.url=http://localhost:8428?nocache=1
.
To set additional URL params for specific group of rules modify
the params
group:
groups:
- name: TestGroup
params:
denyPartialResponse: ["true"]
extra_label: ["env=dev"]
Please note, params
are used only for executing rules expressions (requests to datasource.url
).
If there would be a conflict between URL params set in datasource.url
flag and params in group definition
the latter will have higher priority.
Notifier configuration file
Notifier also supports configuration via file specified with flag notifier.config
:
./bin/vmalert -rule=app/vmalert/config/testdata/rules.good.rules \
-datasource.url=http://localhost:8428 \
-notifier.config=app/vmalert/notifier/testdata/consul.good.yaml
The configuration file allows to configure static notifiers, discover notifiers via Consul and DNS: For example:
static_configs:
- targets:
- localhost:9093
- localhost:9095
consul_sd_configs:
- server: localhost:8500
services:
- alertmanager
dns_sd_configs:
- names:
- my.domain.com
type: 'A'
port: 9093
The list of configured or discovered Notifiers can be explored via UI. If Alertmanager runs in cluster mode then all its URLs needs to be available during discovery to ensure high availability.
The configuration file specification is the following:
# Per-target Notifier timeout when pushing alerts.
[ timeout: <duration> | default = 10s ]
# Prefix for the HTTP path alerts are pushed to.
[ path_prefix: <path> | default = / ]
# Configures the protocol scheme used for requests.
[ scheme: <scheme> | default = http ]
# Sets the `Authorization` header on every request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
[ username: <string> ]
[ password: <string> ]
[ password_file: <string> ]
# Optional `Authorization` header configuration.
authorization:
# Sets the authentication type.
[ type: <string> | default: Bearer ]
# Sets the credentials. It is mutually exclusive with
# `credentials_file`.
[ credentials: <secret> ]
# Sets the credentials to the credentials read from the configured file.
# It is mutually exclusive with `credentials`.
[ credentials_file: <filename> ]
# Configures the scrape request's TLS settings.
# see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config
tls_config:
[ <tls_config> ]
# Configures Bearer authentication token via string
bearer_token: <string>
# or by passing path to the file with token.
bearer_token_file: <string>
# Configures OAuth 2.0 authentication
# see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2
oauth2:
[ <oauth2_config> ]
# Optional list of HTTP headers in form `header-name: value`
# applied for all requests to notifiers
# For example:
# headers:
# - "CustomHeader: foo"
# - "CustomHeader2: bar"
headers:
[ <string>, ...]
# List of labeled statically configured Notifiers.
#
# Each list of targets may be additionally instructed with
# authorization params. Target's authorization params will
# inherit params from global authorization params if there
# are no conflicts.
static_configs:
[ - targets: ]
[ - '<host>' ]
[ oauth2 ]
[ basic_auth ]
[ authorization ]
[ tls_config ]
[ bearer_token ]
[ bearer_token_file ]
[ headers ]
# List of Consul service discovery configurations.
# See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
consul_sd_configs:
[ - <consul_sd_config> ... ]
# List of DNS service discovery configurations.
# See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config
dns_sd_configs:
[ - <dns_sd_config> ... ]
# List of relabel configurations for entities discovered via service discovery.
# Supports the same relabeling features as the rest of VictoriaMetrics components.
# See https://docs.victoriametrics.com/vmagent.html#relabeling
relabel_configs:
[ - <relabel_config> ... ]
# List of relabel configurations for alert labels sent via Notifier.
# Supports the same relabeling features as the rest of VictoriaMetrics components.
# See https://docs.victoriametrics.com/vmagent.html#relabeling
alert_relabel_configs:
[ - <relabel_config> ... ]
The configuration file can be hot-reloaded.
Contributing
vmalert
is mostly designed and built by VictoriaMetrics community.
Feel free to share your experience and ideas for improving this
software. Please keep simplicity as the main priority.
How to build from sources
It is recommended using binary releases
vmalert
is located invmutils-*
archives there.
Docker image
You can build vmalert
docker image from source and push it to your own docker repository.
Run the following commands from the root folder of the repository:
make package-vmalert
docker tag victoria-metrics/vmalert:version my-repo:my-version-name
docker push my-repo:my-version-name
To run the built image in victoria-metrics-k8s-stack
or VMAlert
CR object apply the following config change:
kind: VMAlert
spec:
image:
repository: my-repo
tag: my-version-name
Development build
- Install Go. The minimum supported version is Go 1.19.
- Run
make vmalert
from the root folder of the repository. It buildsvmalert
binary and puts it into thebin
folder.
Production build
- Install docker.
- Run
make vmalert-prod
from the root folder of the repository. It buildsvmalert-prod
binary and puts it into thebin
folder.
ARM build
ARM build may run on Raspberry Pi or on energy-efficient ARM servers.
Development ARM build
- Install Go. The minimum supported version is Go 1.19.
- Run
make vmalert-linux-arm
ormake vmalert-linux-arm64
from the root folder of the repository. It buildsvmalert-linux-arm
orvmalert-linux-arm64
binary respectively and puts it into thebin
folder.
Production ARM build
- Install docker.
- Run
make vmalert-linux-arm-prod
ormake vmalert-linux-arm64-prod
from the root folder of the repository. It buildsvmalert-linux-arm-prod
orvmalert-linux-arm64-prod
binary respectively and puts it into thebin
folder.
Documentation
¶
There is no documentation for this package.