sumologicexporter

README

Sumo Logic Exporter

Stability level: Beta

This exporter supports sending logs and metrics data to Sumo Logic.

We strongly recommend to use this exporter with sumologicextension.

Configuration is specified via the yaml in the following structure:

exporters:
  # ...
  sumologic:
    # unique URL generated for your HTTP Source, this is the address to send data to
    # deprecated, please use sumologicextension to manage your endpoints
    # if sumologicextension is not being used, the endpoint is required
    endpoint: <HTTP_Source_URL>
    # Compression encoding format, empty string means no compression, default = gzip
    compress_encoding: {gzip, deflate, ""}
    # max HTTP request body size in bytes before compression (if applied),
    # NOTE: this limit does not apply to data sent in otlp format,
    #   to limit size of otlp requests, please use the batch processor:
    #   https://github.com/open-telemetry/opentelemetry-collector/tree/v0.82.0/processor/batchprocessor
    # default = 1_048_576 (1MB)
    max_request_body_size: <max_request_body_size>

    # format to use when sending logs to Sumo Logic, default = otlp,
    # NOTE: only `otlp` is supported when used with sumologicextension
    log_format: {json, text, otlp}

    # format to use when sending metrics to Sumo Logic, default = otlp,
    # NOTE: only `otlp` is supported when used with sumologicextension
    metric_format: {otlp, prometheus}

    # Decompose OTLP Histograms into individual metrics, similar to how they're represented in Prometheus format.
    # The Sumo OTLP source currently doesn't support Histograms, and they are quietly dropped. This option produces
    # metrics similar to when metric_format is set to prometheus.
    decompose_otlp_histograms: {true, false}

    # format to use when sending traces to Sumo Logic,
    # currently only otlp is supported
    trace_format: {otlp}

    # timeout is the timeout for every attempt to send data to the backend,
    # maximum connection timeout is 55s, default = 5s
    timeout: <timeout>

    # defines if timestamp for logs should be set to 0,
    # it indicates that backend will extract timestamp from logs,
    # this option affects OTLP format only
    # default = true
    clear_logs_timestamp: {true, false}

    # name of resource attribute which should be dropped for records
    # this is for attribute used by routing processor
    # other attributes should be removed by processors in pipelines before
    # This is workaround for the following issue:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/7407
    # default = ``
    routing_atttribute_to_drop: <routing_atttribute_to_drop>

    json_logs:
      # defines which key will be used to attach the log body at.
      # This option affects JSON log format only.
      # By default this is "log".
      log_key: <log>
      # defines whether to include a timestamp field when sending
      # JSON logs, which would contain UNIX epoch timestamp in milliseconds.
      # This option affects JSON log format only.
      # default = true.
      add_timestamp: {true, false}
      # when add_timestamp is set to true then this key defines what is the name
      # of the timestamp key.
      # default = "timestamp".
      timestamp_key: <timestamp_key>
      # When flatten_body is set to true and log is a map,
      # log's body is going to be flattened and `log_key` won't be used
      # default = false
      flatten_body: {true, false}

    # instructs sumologicexporter to use an edpoint automatically generated by
    # sumologicextension;
    # to use direct endpoint, set it `auth` to `null` and set the endpoint configuration
    # option;
    # see sumologicextension documentation for details
    # default = sumologic
    auth:
      authenticator: <sumologicextension_name>

    # for below described queueing and retry related configuration please refer to:
    # https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md#configuration

    retry_on_failure:
      # default = true
      enabled: {true, false}
      # time to wait after the first failure before retrying;
      # ignored if enabled is false, default = 5s
      initial_interval: <initial_interval>
      # is the upper bound on backoff; ignored if enabled is false, default = 30s
      max_interval: <max_interval>
      # is the maximum amount of time spent trying to send a batch;
      # ignored if enabled is false, default = 120s
      max_elapsed_time: <max_elapsed_time>

    sending_queue:
      # default = false
      enabled: {true, false}
      # number of consumers that dequeue batches; ignored if enabled is false,
      # default = 10
      num_consumers: <num_consumers>
      # when set, enables persistence and uses the component specified as a storage extension for the persistent queue
      # make sure to configure and add a `file_storage` extension in `service.extensions`.
      # default = None
      storage: <storage_name>
      # maximum number of batches kept in memory before data;
      # ignored if enabled is false, default = 1000
      #
      # user should calculate this as num_seconds * requests_per_second where:
      # num_seconds is the number of seconds to buffer in case of a backend outage,
      # requests_per_second is the average number of requests per seconds.
      queue_size: <queue_size>

Metrics

The Sumo Logic Exporter exposes the following metrics:

• otelcol_exporter_requests_bytes (counter) - total size of HTTP requests (in bytes)
• otelcol_exporter_requests_duration (counter) - duration of HTTP requests (in milliseconds)
• otelcol_exporter_requests_records (counter) - total size of HTTP requests (in number of records)
• otelcol_exporter_requests_sent (counter) - number of HTTP requests

All of the above metrics have the following dimensions:

• endpoint - endpoint address
• exporter - exporter name
• pipeline - pipeline name (logs, metrics or traces)
• status_code - HTTP response status code (0 in case of error)

Example Configuration

Example with sumologicextension

extensions:
  sumologic:
    installation_token: <token>
    collector_name: my_collector

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      load:

exporters:
  sumologic:

processors:
  source:
    source_category: "custom category"
    source_name: "custom name"
    source_host: "%{k8s.pod.name}"

service:
  extensions: [sumologic]
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [source]
      exporters: [sumologic]

Example without sumologicextension

exporters:
  sumologic:
    endpoint: http://localhost:3000
    compress_encoding: "gzip"
    max_request_body_size: "1_048_576"  # 1MB
    log_format: "text"
    metric_format: "prometheus"
processors:
  source:
    source_category: "custom category"
    source_name: "custom name"
    source_host: "custom host"

Example with persistent queue

exporters:
  sumologic:
    endpoint: http://localhost:3000
    metric_format: prometheus
    sending_queue:
      enabled: true
      storage: file_storage

extensions:
  file_storage:
    directory: .

receivers:
  hostmetrics:
    collection_interval: 3s
    scrapers:
      load:

service:
  extensions:
  - file_storage
  pipelines:
    metrics:
      exporters:
      - sumologic
      receivers:
      - hostmetrics

Documentation

Index

• Constants
• func CreateDefaultHTTPClientSettings() confighttp.HTTPClientSettings
• func DecomposeHistograms(md pmetric.Metrics) pmetric.Metrics
• func NewFactory() exporter.Factory
• type CompressEncodingType
  • func (cet CompressEncodingType) Validate() error
• type Config
  • func (cfg *Config) Validate() error
• type JSONLogs
• type LogFormatType
• type MetricFormatType
• type PipelineType
• type TraceFormatType

Constants

const (
	// TextFormat represents log_format: text
	TextFormat LogFormatType = "text"
	// JSONFormat represents log_format: json
	JSONFormat LogFormatType = "json"
	// OTLPLogFormat represents log_format: otlp
	OTLPLogFormat LogFormatType = "otlp"
	// RemovedGraphiteFormat represents the no longer supported graphite metric format
	RemovedGraphiteFormat MetricFormatType = "graphite"
	// RemovedCarbon2Format represents the no longer supported carbon2 metric format
	RemovedCarbon2Format MetricFormatType = "carbon2"
	// PrometheusFormat represents metric_format: prometheus
	PrometheusFormat MetricFormatType = "prometheus"
	// OTLPMetricFormat represents metric_format: otlp
	OTLPMetricFormat MetricFormatType = "otlp"
	// OTLPTraceFormat represents trace_format: otlp
	OTLPTraceFormat TraceFormatType = "otlp"
	// GZIPCompression represents compress_encoding: gzip
	GZIPCompression CompressEncodingType = "gzip"
	// DeflateCompression represents compress_encoding: deflate
	DeflateCompression CompressEncodingType = "deflate"
	// NoCompression represents disabled compression
	NoCompression CompressEncodingType = ""
	// MetricsPipeline represents metrics pipeline
	MetricsPipeline PipelineType = "metrics"
	// LogsPipeline represents metrics pipeline
	LogsPipeline PipelineType = "logs"
	// TracesPipeline represents traces pipeline
	TracesPipeline PipelineType = "traces"

	// DefaultCompress defines default Compress
	DefaultCompress bool = true
	// DefaultCompressEncoding defines default CompressEncoding
	DefaultCompressEncoding CompressEncodingType = "gzip"
	// DefaultMaxRequestBodySize defines default MaxRequestBodySize in bytes
	DefaultMaxRequestBodySize int = 1 * 1024 * 1024
	// DefaultLogFormat defines default LogFormat
	DefaultLogFormat LogFormatType = OTLPLogFormat
	// DefaultMetricFormat defines default MetricFormat
	DefaultMetricFormat MetricFormatType = OTLPMetricFormat
	// DefaultClient defines default Client
	DefaultClient string = "otelcol"
	// DefaultClearTimestamp defines default ClearLogsTimestamp value
	DefaultClearLogsTimestamp bool = true
	// DefaultLogKey defines default LogKey value
	DefaultLogKey string = "log"
	// DefaultAddTimestamp defines default AddTimestamp value
	DefaultAddTimestamp bool = true
	// DefaultTimestampKey defines default TimestampKey value
	DefaultTimestampKey string = "timestamp"
	// DefaultFlattenBody defines default FlattenBody value
	DefaultFlattenBody bool = false
	// DefaultDropRoutingAttribute defines default DropRoutingAttribute
	DefaultDropRoutingAttribute string = ""
)

Functions

func CreateDefaultHTTPClientSettings

func CreateDefaultHTTPClientSettings() confighttp.HTTPClientSettings

CreateDefaultHTTPClientSettings returns default http client settings

func DecomposeHistograms

func DecomposeHistograms(md pmetric.Metrics) pmetric.Metrics

DecomposeHistograms decomposes any histograms present in the metric data into individual Sums and Gauges This is a noop if no Histograms are present, but otherwise makes a copy of the whole structure This exists because Sumo doesn't support OTLP histograms yet, and has the same semantics as the conversion to Prometheus format in prometheus_formatter.go

func NewFactory

func NewFactory() exporter.Factory

NewFactory returns a new factory for the sumologic exporter.

Types

type CompressEncodingType

type CompressEncodingType string

CompressEncodingType represents type of the pipeline

func (CompressEncodingType) Validate

func (cet CompressEncodingType) Validate() error

type Config

type Config struct {
	confighttp.HTTPClientSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
	exporterhelper.QueueSettings  `mapstructure:"sending_queue"`
	exporterhelper.RetrySettings  `mapstructure:"retry_on_failure"`

	// Compression encoding format, either empty string, gzip or deflate (default gzip)
	// Empty string means no compression
	CompressEncoding CompressEncodingType `mapstructure:"compress_encoding"`
	// Max HTTP request body size in bytes before compression (if applied).
	// By default 1MB is recommended.
	MaxRequestBodySize int `mapstructure:"max_request_body_size"`

	// Logs related configuration
	// Format to post logs into Sumo. (default json)
	//   * text - Logs will appear in Sumo Logic in text format.
	//   * json - Logs will appear in Sumo Logic in json format.
	//   * otlp - Logs will be send in otlp format and will appear in Sumo Logic in text format.
	LogFormat LogFormatType `mapstructure:"log_format"`

	// Metrics related configuration
	// The format of metrics you will be sending, either otlp or prometheus (Default is otlp)
	MetricFormat MetricFormatType `mapstructure:"metric_format"`

	// Decompose OTLP Histograms into individual metrics, similar to how they're represented in Prometheus format
	DecomposeOtlpHistograms bool `mapstructure:"decompose_otlp_histograms"`

	// Traces related configuration
	// The format of traces you will be sending, currently only otlp format is supported
	TraceFormat TraceFormatType `mapstructure:"trace_format"`

	// Attribute used by routingprocessor which should be dropped during data ingestion
	// This is workaround for the following issue:
	// https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/7407
	DropRoutingAttribute string `mapstructure:"routing_atttribute_to_drop"`

	// Sumo specific options
	// Name of the client
	Client string `mapstructure:"client"`

	// ClearTimestamp defines if timestamp for logs should be set to 0.
	// It indicates that backend will extract timestamp from logs.
	// This option affects OTLP format only.
	// By default this is true.
	ClearLogsTimestamp bool `mapstructure:"clear_logs_timestamp"`

	JSONLogs `mapstructure:"json_logs"`
}

Config defines configuration for Sumo Logic exporter.

func (*Config) Validate

func (cfg *Config) Validate() error

type JSONLogs

type JSONLogs struct {
	// LogKey defines which key will be used to attach the log body at.
	// This option affects JSON log format only.
	// By default this is "log".
	LogKey string `mapstructure:"log_key"`
	// AddTimestamp defines whether to include a timestamp field when sending
	// JSON logs, which would contain UNIX epoch timestamp in milliseconds.
	// This option affects JSON log format only.
	// By default this is true.
	AddTimestamp bool `mapstructure:"add_timestamp"`
	// When add_timestamp is set to true then this key defines what is the name
	// of the timestamp key.
	// By default this is "timestamp".
	TimestampKey string `mapstructure:"timestamp_key"`
	// When flatten_body is set to true and log is a map,
	// log's body is going to be flattened and `log_key` won't be used
	// By default this is false.
	FlattenBody bool `mapstructure:"flatten_body"`
}

type LogFormatType

type LogFormatType string

LogFormatType represents log_format

type MetricFormatType

type MetricFormatType string

MetricFormatType represents metric_format

type PipelineType

type PipelineType string

PipelineType represents type of the pipeline

type TraceFormatType

type TraceFormatType string

TraceFormatType represents trace_format