datasetexporter

README

DataSet Exporter

Status
Stability	alpha: logs, traces
Distributions	contrib
Issues
Code Owners	@atoulme, @martin-majlis-s1, @zdaratom, @tomaz-s1

This exporter sends logs to DataSet.

See the Getting Started guide.

Configuration

Required Settings

• dataset_url (no default): The URL of the DataSet API that ingests the data. Most likely https://app.scalyr.com.
• api_key (no default): The "Log Write" API Key required to use API. Instructions how to get API key.

If you do not want to specify api_key in the file, you can use the builtin functionality and use api_key: ${env:DATASET_API_KEY}.

Optional Settings

• buffer:
  • max_lifetime (default = 5s): The maximum delay between sending batches from the same source.
  • group_by (default = []): The list of attributes based on which events should be grouped.
  • retry_initial_interval (default = 5s): Time to wait after the first failure before retrying.
  • retry_max_interval (default = 30s): Is the upper bound on backoff.
  • retry_max_elapsed_time (default = 300s): Is the maximum amount of time spent trying to send a buffer.
• logs:
  • export_resource_info_on_event (default = false): Include resource info to DataSet Event while exporting Logs. This is especially useful when reducing DataSet billable log volume.
  • export_scope_info_on_event (default = false): Include LogRecord scope information (if available) on the DataSet event.
  • decompose_complex_message_field (default = true): Set this to false to disable decomposing complex body / message field types (e.g. a map) into separate fields.
• retry_on_failure: See retry_on_failure
• sending_queue: See sending_queue
• timeout: See timeout

Example

exporters:
  dataset/logs:
    # DataSet API URL, https://app.eu.scalyr.com for DataSet EU instance
    dataset_url: https://app.scalyr.com
    # API Key
    api_key: your_api_key
    buffer:
      # Send buffer to the API at least every 5s
      max_lifetime: 5s
      # Group data based on these attributes
      group_by:
        - attributes.container_id

  dataset/traces:
    # DataSet API URL, https://app.eu.scalyr.com for DataSet EU instance
    dataset_url: https://app.scalyr.com
    # API Key
    api_key: your_api_key
    buffer:
      max_lifetime: 15s
      group_by:
        - resource_service.instance.id

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      # add dataset among your exporters
      exporters: [dataset/logs]
    traces:
      receivers: [otlp]
      processors: [batch]
      # add dataset among your exporters
      exporters: [dataset/traces]

Documentation

Overview

Package datasetexporter implements an exporter that sends data to DataSet.

Index

• Constants
• func NewFactory() exporter.Factory
• type BufferSettings
• type Config
  • func (c *Config) String() string
  • func (c *Config) Unmarshal(conf *confmap.Conf) error
  • func (c *Config) Validate() error
• type DatasetExporter
• type ExporterConfig
• type LogsSettings
• type ResourceType
• type TraceAndSpan
  • func (ts TraceAndSpan) String() string
• type TracesSettings

Constants

const (
	Service = ResourceType("service")
	Process = ResourceType("process")
)

const ServiceNameKey = "service.name"

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory created new factory with DataSet exporters.

Types

type BufferSettings

type BufferSettings struct {
	MaxLifetime          time.Duration `mapstructure:"max_lifetime"`
	GroupBy              []string      `mapstructure:"group_by"`
	RetryInitialInterval time.Duration `mapstructure:"retry_initial_interval"`
	RetryMaxInterval     time.Duration `mapstructure:"retry_max_interval"`
	RetryMaxElapsedTime  time.Duration `mapstructure:"retry_max_elapsed_time"`
}

type Config

type Config struct {
	DatasetURL                     string              `mapstructure:"dataset_url"`
	APIKey                         configopaque.String `mapstructure:"api_key"`
	BufferSettings                 `mapstructure:"buffer"`
	TracesSettings                 `mapstructure:"traces"`
	LogsSettings                   `mapstructure:"logs"`
	exporterhelper.RetrySettings   `mapstructure:"retry_on_failure"`
	exporterhelper.QueueSettings   `mapstructure:"sending_queue"`
	exporterhelper.TimeoutSettings `mapstructure:"timeout"`
}

func (*Config) String

func (c *Config) String() string

String returns a string representation of the Config object. It includes all the fields and their values in the format "field_name: field_value".

func (*Config) Unmarshal

func (c *Config) Unmarshal(conf *confmap.Conf) error

func (*Config) Validate

func (c *Config) Validate() error

Validate checks if all required fields in Config are set and have valid values. If any of the required fields are missing or have invalid values, it returns an error.

type DatasetExporter

type DatasetExporter struct {
	// contains filtered or unexported fields
}

type ExporterConfig

type ExporterConfig struct {
	// contains filtered or unexported fields
}

type LogsSettings

type LogsSettings struct {
	// ExportResourceInfo is optional flag to signal that the resource info is being exported to DataSet while exporting Logs.
	// This is especially useful when reducing DataSet billable log volume.
	// Default value: false.
	ExportResourceInfo bool `mapstructure:"export_resource_info_on_event"`

	// ExportScopeInfo is an optional flag that signals if scope info should be exported (when available) with each event. If scope
	// information is not utilized, it makes sense to disable exporting it since it will result in increased billable log volume.
	ExportScopeInfo bool `mapstructure:"export_scope_info_on_event"`

	// DecomposeComplexMessageField is an optional flag to signal that message / body of complex types (e.g. a map) should be
	// decomposed / deconstructed into multiple fields. This is usually done outside of the main DataSet integration on the
	// client side (e.g. as part of the attribute processor or similar) or on the server side (DataSet server side JSON parser
	// for message field) and that's why this functionality is disabled by default.
	DecomposeComplexMessageField bool `mapstructure:"decompose_complex_message_field"`
}

type ResourceType

type ResourceType string

type TraceAndSpan

type TraceAndSpan [24]byte

func (TraceAndSpan) String

func (ts TraceAndSpan) String() string

type TracesSettings

type TracesSettings struct {
}