elasticsearchexporter

README

Elasticsearch Exporter

Status
Stability	beta: traces, logs
Distributions	contrib, observiq

This exporter supports sending OpenTelemetry logs to Elasticsearch.

Configuration options

• endpoints: List of Elasticsearch URLs. If endpoints and cloudid is missing, the ELASTICSEARCH_URL environment variable will be used.
• cloudid (optional): ID of the Elastic Cloud Cluster to publish events to. The cloudid can be used instead of endpoints.
• num_workers (optional): Number of workers publishing bulk requests concurrently.
• index: The index or datastream name to publish events to. The default value is logs-generic-default. Note: To better differentiate between log indexes and traces indexes, index option are deprecated and replaced with below logs_index
• logs_index: The index or datastream name to publish events to. The default value is logs-generic-default
• logs_dynamic_index (optional): takes resource or log record attribute named elasticsearch.index.prefix and elasticsearch.index.suffix resulting dynamically prefixed / suffixed indexing based on logs_index. (priority: resource attribute > log record attribute)
  • enabled(default=false): Enable/Disable dynamic index for log records
• traces_index: The index or datastream name to publish traces to. The default value is traces-generic-default.
• traces_dynamic_index (optional): takes resource or span attribute named elasticsearch.index.prefix and elasticsearch.index.suffix resulting dynamically prefixed / suffixed indexing based on traces_index. (priority: resource attribute > span attribute)
  • enabled(default=false): Enable/Disable dynamic index for trace spans
• pipeline (optional): Optional Ingest Node pipeline ID used for processing documents published by the exporter.
• flush: Event bulk buffer flush settings
  • bytes (default=5242880): Write buffer flush limit.
  • interval (default=30s): Write buffer time limit.
• retry: Event retry settings
  • enabled (default=true): Enable/Disable event retry on error. Retry support is enabled by default.
  • max_requests (default=3): Number of HTTP request retries.
  • initial_interval (default=100ms): Initial waiting time if a HTTP request failed.
  • max_interval (default=1m): Max waiting time if a HTTP request failed.
• mapping: Events are encoded to JSON. The mapping allows users to configure additional mapping rules.
  • mode (default=ecs): The fields naming mode. valid modes are:
    • none: Use original fields and event structure from the OTLP event.
    • ecs: Try to map fields defined in the OpenTelemetry Semantic Conventions to Elastic Common Schema (ECS).
  • fields (optional): Configure additional fields mappings.
  • file (optional): Read additional field mappings from the provided YAML file.
  • dedup (default=true): Try to find and remove duplicate fields/attributes from events before publishing to Elasticsearch. Some structured logging libraries can produce duplicate fields (for example zap). Elasticsearch will reject documents that have duplicate fields.
  • dedot (default=true): When enabled attributes with . will be split into proper json objects.
• sending_queue
  • enabled (default = false)
  • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
  • queue_size (default = 1000): Maximum number of batches kept in memory before data; ignored if enabled is false;

HTTP settings

• read_buffer_size (default=0): Read buffer size.
• write_buffer_size (default=0): Write buffer size used when.
• timeout (default=90s): HTTP request time limit.
• headers (optional): Headers to be send with each HTTP request.

Security and Authentication settings

• user (optional): Username used for HTTP Basic Authentication.
• password (optional): Password used for HTTP Basic Authentication.
• api_key (optional): Authorization API Key.

TLS settings

• ca_file (optional): Root Certificate Authority (CA) certificate, for verifying the server's identity, if TLS is enabled.
• cert_file (optional): Client TLS certificate.
• key_file (optional): Client TLS key.
• insecure (optional): In gRPC when set to true, this is used to disable the client transport security. In HTTP, this disables verifying the server's certificate chain and host name.
• insecure_skip_verify (optional): Will enable TLS but not verify the certificate. is enabled.

Node Discovery

The Elasticsearch Exporter will check Elasticsearch regularly for available nodes and updates the list of hosts if discovery is enabled. Newly discovered nodes will automatically be used for load balancing.

• discover:
  • on_start (optional): If enabled the exporter queries Elasticsearch for all known nodes in the cluster on startup.
  • interval (optional): Interval to update the list of Elasticsearch nodes.

Example

exporters:
  elasticsearch/trace:
    endpoints: [https://elastic.example.com:9200]
    traces_index: trace_index
  elasticsearch/log:
    endpoints: [http://localhost:9200]
    logs_index: my_log_index
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 1000
······
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch/log]
    traces:
      receivers: [otlp]
      exporters: [elasticsearch/trace]
      processors: [batch]

Documentation

Overview

Package elasticsearchexporter contains an opentelemetry-collector exporter for Elasticsearch.

Package elasticsearchexporter contains an opentelemetry-collector exporter for Elasticsearch.

Package elasticsearchexporter contains an opentelemetry-collector exporter for Elasticsearch.

Package elasticsearchexporter contains an opentelemetry-collector exporter for Elasticsearch.

Index

• func NewFactory() exporter.Factory
• type AuthenticationSettings
• type Config
  • func (cfg *Config) Validate() error
• type DiscoverySettings
• type DynamicIndexSetting
• type FlushSettings
• type HTTPClientSettings
• type MappingMode
  • func (m MappingMode) String() string
• type MappingsSettings
• type RetrySettings

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for Elastic exporter.

Types

type AuthenticationSettings

type AuthenticationSettings struct {
	// User is used to configure HTTP Basic Authentication.
	User string `mapstructure:"user"`

	// Password is used to configure HTTP Basic Authentication.
	Password configopaque.String `mapstructure:"password"`

	// APIKey is used to configure ApiKey based Authentication.
	//
	// https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html
	APIKey configopaque.String `mapstructure:"api_key"`
}

AuthenticationSettings defines user authentication related settings.

type Config

type Config struct {
	exporterhelper.QueueSettings `mapstructure:"sending_queue"`
	// Endpoints holds the Elasticsearch URLs the exporter should send events to.
	//
	// This setting is required if CloudID is not set and if the
	// ELASTICSEARCH_URL environment variable is not set.
	Endpoints []string `mapstructure:"endpoints"`

	// CloudID holds the cloud ID to identify the Elastic Cloud cluster to send events to.
	// https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html
	//
	// This setting is required if no URL is configured.
	CloudID string `mapstructure:"cloudid"`

	// NumWorkers configures the number of workers publishing bulk requests.
	NumWorkers int `mapstructure:"num_workers"`

	// Index configures the index, index alias, or data stream name events should be indexed in.
	//
	// https://www.elastic.co/guide/en/elasticsearch/reference/current/indices.html
	// https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
	//
	// Deprecated: `index` is deprecated and replaced with `logs_index`.
	Index string `mapstructure:"index"`

	// This setting is required when logging pipelines used.
	LogsIndex string `mapstructure:"logs_index"`
	// fall back to pure LogsIndex, if 'elasticsearch.index.prefix' or 'elasticsearch.index.suffix' are not found in resource or attribute (prio: resource > attribute)
	LogsDynamicIndex DynamicIndexSetting `mapstructure:"logs_dynamic_index"`
	// This setting is required when traces pipelines used.
	TracesIndex string `mapstructure:"traces_index"`
	// fall back to pure TracesIndex, if 'elasticsearch.index.prefix' or 'elasticsearch.index.suffix' are not found in resource or attribute (prio: resource > attribute)
	TracesDynamicIndex DynamicIndexSetting `mapstructure:"traces_dynamic_index"`

	// Pipeline configures the ingest node pipeline name that should be used to process the
	// events.
	//
	// https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
	Pipeline string `mapstructure:"pipeline"`

	HTTPClientSettings `mapstructure:",squash"`
	Discovery          DiscoverySettings `mapstructure:"discover"`
	Retry              RetrySettings     `mapstructure:"retry"`
	Flush              FlushSettings     `mapstructure:"flush"`
	Mapping            MappingsSettings  `mapstructure:"mapping"`
}

Config defines configuration for Elastic exporter.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate validates the elasticsearch server configuration.

type DiscoverySettings

type DiscoverySettings struct {
	// OnStart, if set, instructs the exporter to look for available Elasticsearch
	// nodes the first time the exporter connects to the cluster.
	OnStart bool `mapstructure:"on_start"`

	// Interval instructs the exporter to renew the list of Elasticsearch URLs
	// with the given interval. URLs will not be updated if Interval is <=0.
	Interval time.Duration `mapstructure:"interval"`
}

DiscoverySettings defines Elasticsearch node discovery related settings. The exporter will check Elasticsearch regularly for available nodes and updates the list of hosts if discovery is enabled. Newly discovered nodes will automatically be used for load balancing.

DiscoverySettings should not be enabled when operating Elasticsearch behind a proxy or load balancer.

https://www.elastic.co/blog/elasticsearch-sniffing-best-practices-what-when-why-how

type DynamicIndexSetting

type DynamicIndexSetting struct {
	Enabled bool `mapstructure:"enabled"`
}

type FlushSettings

type FlushSettings struct {
	// Bytes sets the send buffer flushing limit.
	Bytes int `mapstructure:"bytes"`

	// Interval configures the max age of a document in the send buffer.
	Interval time.Duration `mapstructure:"interval"`
}

FlushSettings defines settings for configuring the write buffer flushing policy in the Elasticsearch exporter. The exporter sends a bulk request with all events already serialized into the send-buffer.

type HTTPClientSettings

type HTTPClientSettings struct {
	Authentication AuthenticationSettings `mapstructure:",squash"`

	// ReadBufferSize for HTTP client. See http.Transport.ReadBufferSize.
	ReadBufferSize int `mapstructure:"read_buffer_size"`

	// WriteBufferSize for HTTP client. See http.Transport.WriteBufferSize.
	WriteBufferSize int `mapstructure:"write_buffer_size"`

	// Timeout configures the HTTP request timeout.
	Timeout time.Duration `mapstructure:"timeout"`

	// Headers allows users to configure optional HTTP headers that
	// will be send with each HTTP request.
	Headers map[string]string `mapstructure:"headers,omitempty"`

	configtls.TLSClientSetting `mapstructure:"tls,omitempty"`
}

type MappingMode

type MappingMode int

const (
	MappingNone MappingMode = iota
	MappingECS
)

Enum values for MappingMode.

func (MappingMode) String

func (m MappingMode) String() string

type MappingsSettings

type MappingsSettings struct {
	// Mode configures the field mappings.
	Mode string `mapstructure:"mode"`

	// Additional field mappings.
	Fields map[string]string `mapstructure:"fields"`

	// File to read additional fields mappings from.
	File string `mapstructure:"file"`

	// Try to find and remove duplicate fields
	Dedup bool `mapstructure:"dedup"`

	Dedot bool `mapstructure:"dedot"`
}

type RetrySettings

type RetrySettings struct {
	// Enabled allows users to disable retry without having to comment out all settings.
	Enabled bool `mapstructure:"enabled"`

	// MaxRequests configures how often an HTTP request is retried before it is assumed to be failed.
	MaxRequests int `mapstructure:"max_requests"`

	// InitialInterval configures the initial waiting time if a request failed.
	InitialInterval time.Duration `mapstructure:"initial_interval"`

	// MaxInterval configures the max waiting time if consecutive requests failed.
	MaxInterval time.Duration `mapstructure:"max_interval"`
}

RetrySettings defines settings for the HTTP request retries in the Elasticsearch exporter. Failed sends are retried with exponential backoff.