humioexporter

README

Deprecated Humio Exporter

Status
Stability	deprecated
Supported pipeline types	traces
Distributions	contrib

Exports data to Humio using JSON over the HTTP Ingest API.

Humio now known as LogScale beginning with version 1.68 supports OTLP using HTTP and no longer requires a product specific exporter.

See #17013 for more information.

Getting Started

This exporter provides a set of global configuration options, as well as options specific to each telemetry data type. An example structure is illustrated below:

exporters:
  humio:
    endpoint: "my-global-endpoint"
    traces:
      ingest_token: "my-traces-token"

Required global options must always be specified, while options specific to each type of telemetry data are only required if that telemetry type has been enabled in a pipeline. For instance, the pipeline below will not require configuration options for logs or metrics:

service:
  pipelines:
    traces:
      receivers: [nop]
      processors: [nop]
      exporters: [humio]

The following sections outline the required and optional settings for both global options as well as options specific to each type of telemetry data.

Global options

This exporter requires as a minimum the following global configuration options:

• endpoint (no default): The global base URL on which the Humio backend can be reached, in the form host:port. For testing this locally with the Humio Docker image, the endpoint could be http://localhost:8080/. For use with the Humio cloud, the URLs are as follows, both of which use port 80:
  • EU: https://cloud.humio.com/
  • US: https://cloud.us.humio.com/

In addition, the following optional settings specific to this exporter can be overridden:

• disable_compression (default: false): Whether to stop compressing payloads with gzip before sending them to Humio. This should only be disabled if compression can be shown to have a negative impact on performance in your specific deployment.
• tag (default: none): The strategy to use for tagging telemetry data sent to Humio. By default, tagging is disabled, since it is a complex topic. See Tagging for more information, including possible values.

This exporter also supports inherited configuration options as described in Inherited Options. As defined in the TLS Configuration Settings , TLS is enabled by default. This can be disabled by overriding the following configuration options:

• insecure (default: false): Whether to enable client transport security for the exporter's HTTP connection. Not recommended for production deployments.
• insecure_skip_verify (default: false): Whether to skip verifying the server's certificate chain or not. Not recommended for production deployments.

Traces

For exporting traces, the following configuration options are required:

• ingest_token (no default): The token that has been issued in relation to the Humio repository to export traces into. This token grants write-only access to a single, specific Humio repository. See Ingest Tokens for more details.

In addition, the following optional settings can be overridden:

• unix_timestamps (default: false): Whether to use Unix or ISO 8601 formatted timestamps when exporting data to Humio. If this is set to true, timestamps will be represented in milliseconds (Unix time) in UTC, and the time zone of the event is stored separately in the payload sent to Humio.

Example Configuration

Below are two examples of configurations specific to this exporter, the first of which is the minimal required configuration for traces. For a more advanced example with all available configuration options, see This Example.

exporters:
  humio:
    endpoint: "https://cloud.humio.com/"
    traces:
      ingest_token: "00000000-0000-0000-0000-0000000000000"
  humio/advanced:
    endpoint: "http://localhost:8080/"
    timeout: 10s
    disable_compression: true
    tag: trace_id
    traces:
      ingest_token: "00000000-0000-0000-0000-0000000000000"
      unix_timestamps: true

Advaced Configuration

Inherited Options

This exporter, like many others, includes shared configuration helpers for the following advanced settings:

• HTTP Client Configuration
• TLS Configuration
• Queueing, Retry, and Timeout Configuration

Tagging

Tagging is a strategy in Humio to optimize search speeds by sharding ingested data into specific data sources. This allows for making queries that quickly rule out the majority of data to search through. See Humio Tagging for more details.

If tagging is disabled on this exporter, Humio will still use its own tags internally to organize the data. One such tag is the #type tag, which takes its value from the name of the parser assigned to the ingest token). This is the only officially recommended tag.

However, due to the nature of OpenTelemetry, the service name or trace ID can make decent tags as well, depending on the intended queries in the backend. For such tags to work well, however, you need to enable tag grouping. See Tag Grouping for a thorough discussion.

This exporter supports the following strategies for tags, which thus makes up the legal values for the tag option:

Strategy	Tag field in Humio
none	-
trace_id	#trace_id
service_name	#service_name

For instance, to enable the trace_id strategy, you must create a tag group as such:

curl $YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/taggrouping \
  -X POST \
  -H "Authorization: Bearer $YOUR_API_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '[ {"field": "trace_id","modulus": 16} ]'

Documentation

Index

• Constants
• func NewFactory() exporter.Factory
• type Config
  • func (c *Config) Validate() error
• type HumioLink
• type HumioSpan
• type HumioStructuredEvent
  • func (e *HumioStructuredEvent) MarshalJSON() ([]byte, error)
• type HumioStructuredEvents
• type HumioUnstructuredEvents
• type LogsConfig
• type TagOrganizer
• type Tagger
• type TracesConfig

Constants

const (
	// TagNone disables tagging of payloads to Humio
	TagNone tag = "none"

	// TagTraceID tags payloads to Humio using the trace ID, or an empty string if
	// the trace ID is missing
	TagTraceID tag = "trace_id"

	// TagServiceName tags payloads to Humio using the service name, or an empty
	// string if it is missing
	TagServiceName tag = "service_name"
)

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates an exporter factory for Humio

Types

type Config

type Config struct {
	// Inherited settings
	confighttp.HTTPClientSettings `mapstructure:",squash"`
	exporterhelper.QueueSettings  `mapstructure:"sending_queue"`
	exporterhelper.RetrySettings  `mapstructure:"retry_on_failure"`

	// Whether gzip compression should be disabled when sending data to Humio
	DisableCompression bool `mapstructure:"disable_compression"`

	// Name of tagging strategy used to target specific data sources for storage inside Humio
	Tag Tagger `mapstructure:"tag"`

	// Configuration options specific to logs
	Logs LogsConfig `mapstructure:"logs"`

	// Configuration options specific to traces
	Traces TracesConfig `mapstructure:"traces"`
	// contains filtered or unexported fields
}

Config represents the Humio configuration settings

func (*Config) Validate

func (c *Config) Validate() error

Validate ensures that a valid configuration has been provided, such that we can fail early

type HumioLink

type HumioLink struct {
	TraceID    string `json:"trace_id"`
	SpanID     string `json:"span_id"`
	TraceState string `json:"state,omitempty"`
}

HumioLink represents a relation between two spans

type HumioSpan

type HumioSpan struct {
	TraceID           string                 `json:"trace_id"`
	SpanID            string                 `json:"span_id"`
	ParentSpanID      string                 `json:"parent_id,omitempty"`
	Name              string                 `json:"name"`
	Kind              string                 `json:"kind"`
	Start             int64                  `json:"start"`
	End               int64                  `json:"end"`
	StatusCode        string                 `json:"status,omitempty"`
	StatusDescription string                 `json:"status_descr,omitempty"`
	ServiceName       string                 `json:"service"`
	Links             []*HumioLink           `json:"links,omitempty"`
	Attributes        map[string]interface{} `json:"attributes,omitempty"`
}

HumioSpan represents a span as it is stored inside Humio

type HumioStructuredEvent

type HumioStructuredEvent struct {
	// The time where the event occurred
	Timestamp time.Time

	// Whether to serialize the timestamp as Unix or ISO
	AsUnix bool

	// The event payload
	Attributes interface{}
}

HumioStructuredEvent represents a single structured event to send to Humio

func (*HumioStructuredEvent) MarshalJSON

func (e *HumioStructuredEvent) MarshalJSON() ([]byte, error)

MarshalJSON formats the timestamp in a HumioStructuredEvent as either an ISO string or a Unix timestamp in milliseconds with time zone

type HumioStructuredEvents

type HumioStructuredEvents struct {
	// Tags used to target specific data sources in Humio
	Tags map[string]string `json:"tags,omitempty"`

	// The series of structured events
	Events []*HumioStructuredEvent `json:"events"`
}

HumioStructuredEvents represents a payload of multiple structured events to send to Humio

type HumioUnstructuredEvents

type HumioUnstructuredEvents struct {
	// Key-value pairs to associate with the messages as metadata
	Fields map[string]string `json:"fields,omitempty"`

	// Tags used to target specific data sources in Humio
	Tags map[string]string `json:"tags,omitempty"`

	// The name of the parser to handle these messages inside Humio
	Type string `json:"type,omitempty"`

	// The series of unstructured messages
	Messages []string `json:"messages"`
}

HumioUnstructuredEvents represents a payload of multiple unstructured events (strings) to send to Humio

type LogsConfig

type LogsConfig struct {
	// Ingest token for identifying and authorizing with a Humio repository
	IngestToken string `mapstructure:"ingest_token"`

	// The name of a custom log parser to use, if no parser is associated with the ingest token
	LogParser string `mapstructure:"log_parser"`
}

LogsConfig represents the Humio configuration settings specific to logs

type TagOrganizer

type TagOrganizer struct {
	// contains filtered or unexported fields
}

TagOrganizer represents a type for organizing HumioStructuredEvents by tag while a tree of structured data from OpenTelemetry is being converted into Humio's format. Depending on the tagging strategy, the structure of resource- and instrumentation logs/spans is not necessarily optimal for sending to Humio

type Tagger

type Tagger interface {
	Tag() tag
}

Tagger represents a tagging strategy understood by the Humio exporter

type TracesConfig

type TracesConfig struct {
	// Ingest token for identifying and authorizing with a Humio repository
	IngestToken string `mapstructure:"ingest_token"`

	// Whether to use Unix timestamps, or to fall back to ISO 8601 formatted strings
	UnixTimestamps bool `mapstructure:"unix_timestamps"`
}

TracesConfig represents the Humio configuration settings specific to traces