azuredataexplorerexporter

README

Azure Data Explorer Exporter

Status
Stability	beta: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@ag-ramachandran
Emeritus	@asaharn

This exporter sends metrics, logs and trace data to Azure Data Explorer, Azure Synapse Data Explorer and Real time analytics in Fabric

Configuration

The following settings are required:

• cluster_uri (no default): The cluster name of the provisioned ADX cluster to ingest the data.

One authentication method is required:

• Service principal:
  • application_id (no default): The client id to connect to the cluster and ingest data.
  • application_key (no default): The cluster secret corresponding to the client id.
  • tenant_id (no default): The tenant id where the application_id is referenced from.
• Managed identity:
  • managed_identity_id (no default): The managed identity id to authenticate with. Set to "system" for system-assigned managed identity. Set the MI client Id (GUID) for user-assigned managed identity.
• Default authentication:
  • use_azure_auth (default: false): Set to true to use the Azure default authentication.

The following settings can be optionally configured and have default values:

Note that the database tables are expected to be created upfront before the exporter is in operation , the definition of these are in the section Database and Table definition scripts

• db_name (default = "oteldb"): The ADX database where the tables are present to ingest the data.

• metrics_table_name (default = OTELMetrics): The target table in the database db_name that stores exported metric data.

• logs_table_name (default = OTELLogs): The target table in the database db_name that stores exported logs data.

• traces_table_name (default = OTELTraces): The target table in the database db_name that stores exported traces data.

  Optionally the following table mappings can be specified if the data needs to be mapped to a different table on ADX. This uses json table mapping that can be used to map attributes to target tables

• metrics_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.metrics_table_name

• logs_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.logs_table_name

• traces_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.traces_table_name

• ingestion_type (possible values=queued / managed, default = queued): ADX ingest can happen in managed streaming or queued modes.

Note: Streaming ingestion has to be enabled on ADX [configure the ADX cluster] in case of streaming option. Refer the query below to check if streaming is enabled

.show database <DB-Name> policy streamingingestion

An example configuration is provided as follows:

exporters:
  azuredataexplorer:
    # Kusto cluster uri
    cluster_uri: "https://CLUSTER.kusto.windows.net"
    # Client Id
    application_id: "f80da32c-108c-415c-a19e-643f461a677a"
    # The client secret for the client
    application_key: "xx-xx-xx-xx"
    # The tenant
    tenant_id: "21ff9e36-fbaa-43c8-98ba-00431ea10bc3"
    # A managed identity id to authenticate with. 
    # Set to "system" for system-assigned managed identity.
    # Set the MI client Id (GUID) for user-assigned managed identity.
    managed_identity_id: "z80da32c-108c-415c-a19e-643f461a677a"
    # Database for the logs
    db_name: "oteldb"
    # Metric table name
    metrics_table_name: "OTELMetrics"
    # Log table name
    logs_table_name: "OTELLogs"
    # Traces table
    traces_table_name: "OTELTraces"
    # Metric table mapping name
    metrics_table_json_mapping: "otelmetrics_mapping"
    # Log table mapping name
    logs_table_json_mapping: "otellogs_mapping"
    # Traces mapping table
    traces_table_json_mapping: "oteltraces_mapping"
    # Type of ingestion managed or queued
    ingestion_type : "managed"
    #other available exporter helper options, see more here: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md
    # timeout: 10s
    # sending_queue:
    #   enabled: true
    #   num_consumers: 2
    #   queue_size: 10
    # retry_on_failure:
    #   enabled: true
    #   initial_interval: 10s
    #   max_interval: 60s
    #   max_elapsed_time: 10m

Attribute mapping

This exporter maps OpenTelemetry trace, metric and log attributes to specific structures in the ADX tables. These can then be extended by usage of update policies in ADX if needed

Traces

ADX Table column	Description / OpenTelemetry attribute
TraceId	A valid trace identifier is a 16-byte array with at least one non-zero byte.
SpanId	A valid span identifier is an 8-byte array with at least one non-zero byte.
ParentId	A parent spanId, for the current span
SpanName	The span name
SpanStatus	Status Code of the Span.
SpanStatusMessage	Status Message of the Span
SpanKind	SpanKind describes the relationship between the Span, its parents, and its children in a Trace
StartTime	A start timestamp
EndTime	An end timestamp
TraceAttributes	Custom metric attributes set from the application. Also contains the instrumentation scope name and version
ResourceAttributes	The resource attributes JSON map as specified in open telemetry resource semantics
Events	A list of timestamped Events
Links	A list of Links to other Spans

Metrics

ADX Table column	Description / OpenTelemetry attribute
Timestamp	The timestamp of the datapoint
MetricName	The name of the datapoint
MetricType	The type / datapoint type of the metric (e.g. Sum, Histogram, Gauge etc.)
MetricUnit	Unit of measure of the metric
MetricDescription	Description about the metric
MetricValue	The metric value measured for the datapoint
MetricAttributes	Custom metric attributes set from the application. Also contains the instrumentation scope name and version
Host	The host.name extracted from Host resource semantics. If empty , the hostname of the exporter is used
ResourceAttributes	The resource attributes JSON map as specified in open telemetry resource semantics

Logs

ADX Table column	Description / OpenTelemetry attribute
Timestamp	The timestamp of the datapoint
ObservedTimestamp	Time when the event was observed.
TraceId	Request trace id.
SpanId	Request span id.
SeverityText	The severity text (also known as log level)
SeverityNumber	Numerical value of the severity.
Body	The body of the log record.
LogsAttributes	Custom metric attributes set from the application. Also contains the instrumentation scope name and version
ResourceAttributes	The resource attributes JSON map as specified in open telemetry resource semantics

Database and Table definition scripts

The following tables need to be created in the database specified in the configuration.

.create-merge table <Logs-Table-Name> (Timestamp:datetime, ObservedTimestamp:datetime, TraceID:string, SpanID:string, SeverityText:string, SeverityNumber:int, Body:string, ResourceAttributes:dynamic, LogsAttributes:dynamic) 

.create-merge table <Metrics-Table-Name> (Timestamp:datetime, MetricName:string, MetricType:string, MetricUnit:string, MetricDescription:string, MetricValue:real, Host:string, ResourceAttributes:dynamic,MetricAttributes:dynamic) 

.create-merge table <Traces-Table-Name> (TraceID:string, SpanID:string, ParentID:string, SpanName:string, SpanStatus:string, SpanKind:string, StartTime:datetime, EndTime:datetime, ResourceAttributes:dynamic, TraceAttributes:dynamic, Events:dynamic, Links:dynamic) 

// This is an optional column to store the status code and message as a dynamic field. This augments the status field with Status code and status message

.alter-merge table <Traces-Table-Name> (SpanStatusMessage:string) 

//Enable streaming ingestion( for managed streaming) for the created tables using
.alter table <Table-Name> policy streamingingestion enable

Optional configurations/Enhancements suggestions

• Using update policies , the collected data can further be processed as per application need. The following is an example where histogram metrics are exported to a histo specific table (buckets and aggregates)

.create table HistoBucketData (Timestamp: datetime, MetricName: string , MetricType: string , Value: double, LE: double, Host: string , ResourceAttributes: dynamic, MetricAttributes: dynamic )

.create function 
with ( docstring = "Histo bucket processing function", folder = "UpdatePolicyFunctions") ExtractHistoColumns()
{
    OTELMetrics
    | where MetricType == 'Histogram' and MetricName has "_bucket"
    | extend f=parse_json(MetricAttributes)
    | extend le=todouble(f.le)
    | extend M_name=replace_string(MetricName, '_bucket','')
    | project Timestamp, MetricName=M_name, MetricType, MetricValue, LE=le, Host, ResourceAttributes, MetricAttributes
}

.alter table HistoBucketData policy update 
@'[{ "IsEnabled": true, "Source": "OTELMetrics","Query": "ExtractHistoColumns()", "IsTransactional": false, "PropagateIngestionProperties": false}]'

//Below code creates a table which only contains count and sum values of Histogram metric type and attaches an update policy to it

 .create table HistoData (Timestamp: datetime, MetricName: string , MetricType: string , Count: double, Sum: double, Host: string , ResourceAttributes: dynamic, MetricAttributes: dynamic)

 .create function 
with ( docstring = "Histo sum count processing function", folder = "UpdatePolicyFunctions") ExtractHistoCountColumns()
{
   OTELMetrics
    | where MetricType =='Histogram'
    | where MetricName has "_count"
    | extend Count=MetricValue
    | extend M_name=replace_string(MetricName, '_bucket','')
    | join kind=inner (OTELMetrics
    | where MetricType =='Histogram'
    | where MetricName has "_sum"
    | project Sum = MetricValue , Timestamp)
 on Timestamp | project Timestamp, MetricName=M_name, MetricType, Count, Sum, Host, ResourceAttributes, MetricAttributes
}

.alter table HistoData policy update 
@'[{ "IsEnabled": true, "Source": "RawMetricsData","Query": "ExtractHistoCountColumns()", "IsTransactional": false, "PropagateIngestionProperties": false}]'

Opentelemetry Exporter Helper Configurations

The ADX exporter now includes support for Opentelemetry exporter helper configurations. This feature allows you to leverage the exporter helper capabilities(retries, timeout etc.) provided natively by Otel. Read more here.

Please note that this configuration is not enabled by default. To utilize the Opentelemetry exporter helper, you will need to add it manually to the configuration.

Example Configuration

# Example Opentelemetry Exporter Configuration
  timeout: 10s
  sending_queue:
      enabled: true
      num_consumers: 2
      queue_size: 10
  retry_on_failure:
      enabled: true
      initial_interval: 10s
      max_interval: 60s
      max_elapsed_time: 10m

Documentation

Index

• func NewFactory() exporter.Factory
• type AdxLog
• type AdxMetric
• type AdxTrace
• type Config
  • func (adxCfg *Config) Validate() error
• type Event
• type Link
• type Status

Functions

func NewFactory

func NewFactory() exporter.Factory

Creates a factory for the ADX Exporter

Types

type AdxLog

type AdxLog struct {
	Timestamp          string         // The timestamp of the occurrence. Formatted into string as RFC3339Nano
	ObservedTimestamp  string         // The timestamp of logs observed in opentelemetry collector.  Formatted into string as RFC3339Nano
	TraceID            string         // TraceId associated to the log
	SpanID             string         // SpanId associated to the log
	SeverityText       string         // The severity level of the log
	SeverityNumber     int32          // The severity number associated to the log
	Body               string         // The body/Text of the log
	ResourceAttributes map[string]any // JSON Resource attributes that can then be parsed.
	LogsAttributes     map[string]any // JSON attributes that can then be parsed.
}

type AdxMetric

type AdxMetric struct {
	Timestamp string // The timestamp of the occurrence. A metric is measured at a point of time. Formatted into string as RFC3339Nano
	// Including name, the Metric object is defined by the following properties:
	MetricName        string         // Name of the metric field
	MetricType        string         // The data point type (e.g. Sum, Gauge, Histogram ExponentialHistogram, Summary)
	MetricUnit        string         // The metric stream’s unit
	MetricDescription string         // The metric stream’s description
	MetricValue       float64        // the value of the metric
	MetricAttributes  map[string]any // JSON attributes that can then be parsed. Extrinsic properties
	// Additional properties
	Host               string         // The hostname for analysis of the metric. Extracted from https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/host/
	ResourceAttributes map[string]any // The originating Resource attributes. Refer https://opentelemetry.io/docs/reference/specification/resource/sdk/
}

This is derived from the specification https://opentelemetry.io/docs/reference/specification/metrics/datamodel/

type AdxTrace

type AdxTrace struct {
	TraceID            string         // TraceID associated to the Trace
	SpanID             string         // SpanID associated to the Trace
	ParentID           string         // ParentID associated to the Trace
	SpanName           string         // The SpanName of the Trace
	SpanStatus         string         // The SpanStatus Code associated to the Trace
	SpanStatusMessage  string         // The SpanStatusMessage associated to the Trace
	SpanKind           string         // The SpanKind of the Trace
	StartTime          string         // The start time of the occurrence. Formatted into string as RFC3339Nano
	EndTime            string         // The end time of the occurrence. Formatted into string as RFC3339Nano
	ResourceAttributes map[string]any // JSON Resource attributes that can then be parsed.
	TraceAttributes    map[string]any // JSON attributes that can then be parsed.
	Events             []*Event       // Array containing the events in a span
	Links              []*Link        // Array containing the link in a span
}

type Config

type Config struct {
	TimeoutSettings           exporterhelper.TimeoutConfig `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
	QueueSettings             exporterhelper.QueueConfig   `mapstructure:"sending_queue"`
	configretry.BackOffConfig `mapstructure:"retry_on_failure"`
	ClusterURI                string              `mapstructure:"cluster_uri"`
	ApplicationID             string              `mapstructure:"application_id"`
	ApplicationKey            configopaque.String `mapstructure:"application_key"`
	TenantID                  string              `mapstructure:"tenant_id"`
	ManagedIdentityID         string              `mapstructure:"managed_identity_id"`
	UseAzureAuth              bool                `mapstructure:"use_azure_auth"`
	Database                  string              `mapstructure:"db_name"`
	MetricTable               string              `mapstructure:"metrics_table_name"`
	LogTable                  string              `mapstructure:"logs_table_name"`
	TraceTable                string              `mapstructure:"traces_table_name"`
	MetricTableMapping        string              `mapstructure:"metrics_table_json_mapping"`
	LogTableMapping           string              `mapstructure:"logs_table_json_mapping"`
	TraceTableMapping         string              `mapstructure:"traces_table_json_mapping"`
	IngestionType             string              `mapstructure:"ingestion_type"`
}

Config defines configuration for Azure Data Explorer Exporter

func (*Config) Validate

func (adxCfg *Config) Validate() error

Validate checks if the exporter configuration is valid

type Event

type Event struct {
	EventName       string
	Timestamp       string
	EventAttributes map[string]any
}

type Link

type Link struct {
	TraceID            string
	SpanID             string
	TraceState         string
	SpanLinkAttributes map[string]any
}

type Status

type Status struct {
	Code    string
	Message string
}