README
¶
OpenSearch Output Plugin
This plugin writes to OpenSearch via HTTP
It supports OpenSearch releases from 1 and 2. Future comparability with 1.x is not guaranteed and instead will focus on 2.x support. Consider using the existing Elasticsearch plugin for 1.x.
Global configuration options
In addition to the plugin-specific configuration settings, plugins support additional global and plugin configuration settings. These settings are used to modify metrics, tags, and field or create aliases and configure ordering, etc. See the CONFIGURATION.md for more details.
Configuration
# Configuration for OpenSearch to send metrics to.
[[outputs.opensearch]]
## URLs
## The full HTTP endpoint URL for your OpenSearch instance. Multiple URLs can
## be specified as part of the same cluster, but only one URLs is used to
## write during each interval.
urls = ["http://node1.os.example.com:9200"]
## Index Name
## Target index name for metrics (OpenSearch will create if it not exists).
## This is a Golang template (see https://pkg.go.dev/text/template)
## You can also specify
## metric name (`{{.Name}}`), tag value (`{{.Tag "tag_name"}}`), field value (`{{.Field "field_name"}}`)
## If the tag does not exist, the default tag value will be empty string "".
## the timestamp (`{{.Time.Format "xxxxxxxxx"}}`).
## For example: "telegraf-{{.Time.Format \"2006-01-02\"}}-{{.Tag \"host\"}}" would set it to telegraf-2023-07-27-HostName
index_name = ""
## Timeout
## OpenSearch client timeout
# timeout = "5s"
## Sniffer
## Set to true to ask OpenSearch a list of all cluster nodes,
## thus it is not necessary to list all nodes in the urls config option
# enable_sniffer = false
## GZIP Compression
## Set to true to enable gzip compression
# enable_gzip = false
## Health Check Interval
## Set the interval to check if the OpenSearch nodes are available
## Setting to "0s" will disable the health check (not recommended in production)
# health_check_interval = "10s"
## Set the timeout for periodic health checks.
# health_check_timeout = "1s"
## HTTP basic authentication details.
# username = ""
# password = ""
## HTTP bearer token authentication details
# auth_bearer_token = ""
## Optional TLS Config
## Set to true/false to enforce TLS being enabled/disabled. If not set,
## enable TLS only if any of the other options are specified.
# tls_enable =
## Trusted root certificates for server
# tls_ca = "/path/to/cafile"
## Used for TLS client certificate authentication
# tls_cert = "/path/to/certfile"
## Used for TLS client certificate authentication
# tls_key = "/path/to/keyfile"
## Send the specified TLS server name via SNI
# tls_server_name = "kubernetes.example.com"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
## Template Config
## Manage templates
## Set to true if you want telegraf to manage its index template.
## If enabled it will create a recommended index template for telegraf indexes
# manage_template = true
## Template Name
## The template name used for telegraf indexes
# template_name = "telegraf"
## Overwrite Templates
## Set to true if you want telegraf to overwrite an existing template
# overwrite_template = false
## Document ID
## If set to true a unique ID hash will be sent as
## sha256(concat(timestamp,measurement,series-hash)) string. It will enable
## data resend and update metric points avoiding duplicated metrics with
## different id's
# force_document_id = false
## Value Handling
## Specifies the handling of NaN and Inf values.
## This option can have the following values:
## none -- do not modify field-values (default); will produce an error
## if NaNs or infs are encountered
## drop -- drop fields containing NaNs or infs
## replace -- replace with the value in "float_replacement_value" (default: 0.0)
## NaNs and inf will be replaced with the given number, -inf with the negative of that number
# float_handling = "none"
# float_replacement_value = 0.0
## Pipeline Config
## To use a ingest pipeline, set this to the name of the pipeline you want to use.
# use_pipeline = "my_pipeline"
## Pipeline Name
## Additionally, you can specify a tag name using the notation (`{{.Tag "tag_name"}}`)
## which will be used as the pipeline name (e.g. "{{.Tag "os_pipeline"}}").
## If the tag does not exist, the default pipeline will be used as the pipeline.
## If no default pipeline is set, no pipeline is used for the metric.
# default_pipeline = ""
Required parameters
urls: A list containing the full HTTP URL of one or more nodes from your OpenSearch instance.index_name: The target index for metrics. You can use the date format
For example: "telegraf-{{.Time.Format "2006-01-02"}}" would set it to
"telegraf-2023-07-27". You can also specify metric name ({{ .Name }}), tag
value ({{ .Tag \"tag_name\" }}), and field value
({{ .Field \"field_name\" }}).
If the tag does not exist, the default tag value will be empty string ""
Permissions
If you are using authentication within your OpenSearch cluster, you need to create an account and create a role with at least the manage role in the Cluster Privileges category. Otherwise, your account will not be able to connect to your OpenSearch cluster and send logs to your cluster. After that, you need to add "create_index" and "write" permission to your specific index pattern.
OpenSearch indexes and templates
Indexes per time-frame
This plugin can manage indexes per time-frame, as commonly done in other tools with OpenSearch. The timestamp of the metric collected will be used to decide the index destination. For more information about this usage on OpenSearch, check the docs.
Template management
Index templates are used in OpenSearch to define settings and mappings for the indexes and how the fields should be analyzed. For more information on how this works, see the docs.
This plugin can create a working template for use with telegraf metrics. It uses OpenSearch dynamic templates feature to set proper types for the tags and metrics fields. If the template specified already exists, it will not overwrite unless you configure this plugin to do so. Thus you can customize this template after its creation if necessary.
Example of an index template created by telegraf on OpenSearch 2.x:
{
"telegraf-2022.10.02" : {
"aliases" : { },
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"disk" : {
"properties" : {
"free" : {
"type" : "long"
},
"inodes_free" : {
"type" : "long"
},
"inodes_total" : {
"type" : "long"
},
"inodes_used" : {
"type" : "long"
},
"total" : {
"type" : "long"
},
"used" : {
"type" : "long"
},
"used_percent" : {
"type" : "float"
}
}
},
"measurement_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tag" : {
"properties" : {
"cpu" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"device" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"mode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1664693522789",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "TYugdmvsQfmxjzbGRJ8FIw",
"version" : {
"created" : "136247827"
},
"provided_name" : "telegraf-2022.10.02"
}
}
}
}
Example events
This plugin will format the events in the following way:
{
"@timestamp": "2017-01-01T00:00:00+00:00",
"measurement_name": "cpu",
"cpu": {
"usage_guest": 0,
"usage_guest_nice": 0,
"usage_idle": 71.85413456197966,
"usage_iowait": 0.256805341656516,
"usage_irq": 0,
"usage_nice": 0,
"usage_softirq": 0.2054442732579466,
"usage_steal": 0,
"usage_system": 15.04879301548127,
"usage_user": 12.634822807288275
},
"tag": {
"cpu": "cpu-total",
"host": "opensearhhost",
"dc": "datacenter1"
}
}
{
"@timestamp": "2017-01-01T00:00:00+00:00",
"measurement_name": "system",
"system": {
"load1": 0.78,
"load15": 0.8,
"load5": 0.8,
"n_cpus": 2,
"n_users": 2
},
"tag": {
"host": "opensearhhost",
"dc": "datacenter1"
}
}
Known issues
Integer values collected that are bigger than 2^63 and smaller than 1e21 (or in this exact same window of their negative counterparts) are encoded by golang JSON encoder in decimal format and that is not fully supported by OpenSearch dynamic field mapping. This causes the metrics with such values to be dropped in case a field mapping has not been created yet on the telegraf index. If that's the case you will see an exception on OpenSearch side like this:
{
"error": {
"root_cause": [
{"type": "mapper_parsing_exception", "reason": "failed to parse"}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "illegal_state_exception",
"reason": "No matching token for number_type [BIG_INTEGER]"
}
},
"status": 400
}
The correct field mapping will be created on the telegraf index as soon as a supported JSON value is received by OpenSearch, and subsequent insertions will work because the field mapping will already exist.
This issue is caused by the way OpenSearch tries to detect integer fields, and by how golang encodes numbers in JSON. There is no clear workaround for this at the moment.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Opensearch ¶
type Opensearch struct {
Username config.Secret `toml:"username"`
Password config.Secret `toml:"password"`
AuthBearerToken config.Secret `toml:"auth_bearer_token"`
EnableGzip bool `toml:"enable_gzip"`
EnableSniffer bool `toml:"enable_sniffer"`
FloatHandling string `toml:"float_handling"`
FloatReplacement float64 `toml:"float_replacement_value"`
ForceDocumentID bool `toml:"force_document_id"`
IndexName string `toml:"index_name"`
TemplateName string `toml:"template_name"`
ManageTemplate bool `toml:"manage_template"`
OverwriteTemplate bool `toml:"overwrite_template"`
DefaultPipeline string `toml:"default_pipeline"`
UsePipeline string `toml:"use_pipeline"`
Timeout config.Duration `toml:"timeout"`
HealthCheckInterval config.Duration `toml:"health_check_interval"`
HealthCheckTimeout config.Duration `toml:"health_check_timeout"`
URLs []string `toml:"urls"`
Log telegraf.Logger `toml:"-"`
tls.ClientConfig
// contains filtered or unexported fields
}
func (*Opensearch) Close ¶
func (o *Opensearch) Close() error
func (*Opensearch) Connect ¶
func (o *Opensearch) Connect() error
func (*Opensearch) GetIndexName ¶
func (o *Opensearch) GetIndexName(metric telegraf.Metric) (string, error)
func (*Opensearch) Init ¶
func (o *Opensearch) Init() error
func (*Opensearch) SampleConfig ¶
func (*Opensearch) SampleConfig() string
Source Files