azureeventhubreceiver

README

Azure Event Hub Receiver

Status
Stability	alpha: metrics, logs, traces
Distributions	contrib
Issues
Code Owners	@atoulme, @cparkins | Seeking more code owners!
Emeritus	@djaglowski

Overview

Azure resources and services can be configured to send their logs to an Azure Event Hub. The Azure Event Hub receiver pulls logs from an Azure Event Hub, transforms them, and pushes them through the collector pipeline.

Configuration

connection (Required)

A string describing the connection to an Azure event hub.

group (Optional)

The Consumer Group to read from. If empty will default to the default Consumer Group $Default

partition (Optional)

The partition to watch. If empty, it will watch explicitly all partitions.

Default: ""

offset (Optional)

The offset at which to start watching the event hub. If empty, it starts with the latest offset.

Default: ""

format (Optional)

Determines how to transform the Event Hub messages into OpenTelemetry logs. See the "Format" section below for details.

Default: "azure"

apply_semantic_conventions (optional)

Determines whether Azure Resource Logs are translated into OpenTelemetry Logs using semantic convention attribute names or not. When not applying semantic conventions, the log entry attribute names are copied without any changes.

Default: false (semantic conventions are not applied)

time_formats (optional)

All supported time format for logs, metrics and traces. Default is nil (unset), which means using the current iso8601 parser. The format is based on https://pkg.go.dev/time#Layout. If no time-zone info, will use UTC time. If all failed, it will use iso8601 format to parse.

Default: nil

Example Configuration

receivers:
  azureeventhub:
    connection: Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=superSecret1234=;EntityPath=hubName
    partition: foo
    group: bar
    offset: "1234-5566"
    format: "azure"
    # optional
    time_formats:
      # All supported time format. Default is empty string array, which means using the current iso8601 parser. The format is based on https://pkg.go.dev/time#Layout. If no time-zone info, will use UTC time.
      logs: ["01/02/2006 15:04:05","2006-01-02 15:04:05","2006-01-02T15:04:05Z07:00"]
      metrics: ["01/02/2006 15:04:05"]

This component can persist its state using the storage extension.

Format

raw

The "raw" format maps the AMQP properties and data into the attributes and body of an OpenTelemetry LogRecord, respectively. The body is represented as a raw byte array.

This format is not supported for Metrics.

azure

The "azure" format extracts the Azure log records from the AMQP message data, parses them, and maps the fields to OpenTelemetry attributes. The table below summarizes the mapping between the Azure common log format and the OpenTelemetry attributes.

Azure	OpenTelemetry
callerIpAddress (optional)	net.sock.peer.addr (attribute)
correlationId (optional)	azure.correlation.id (attribute)
category (optional)	azure.category (attribute)
durationMs (optional)	azure.duration (attribute)
Level (optional)	severity_number, severity_text (field)
location (optional)	cloud.region (attribute)
—	cloud.provider (attribute)
operationName (required)	azure.operation.name (attribute)
operationVersion (optional)	azure.operation.version (attribute)
properties (optional)	azure.properties (attribute, nested)
resourceId (required)	azure.resource.id (resource attribute)
resultDescription (optional)	azure.result.description (attribute)
resultSignature (optional)	azure.result.signature (attribute)
resultType (optional)	azure.result.type (attribute)
tenantId (required, tenant logs)	azure.tenant.id (attribute)
time or timeStamp (required)	time_unix_nano (time takes precedence)
identity (optional)	azure.identity (attribute, nested)

Notes:

• JSON does not distinguish between fixed and floating point numbers. All JSON numbers are encoded as doubles.

For Metrics the Azure Metric Records are an array of "records" with the following fields.

Azure	Open Telemetry
time	time_unix_nano (field)
resourceId	azure.resource.id (resource attribute)
metricName
timeGrain	start_time_unix_nano (field)
total	mapped to datapoint metricName + "_TOTAL"
count	mapped to datapoint metricName + "_COUNT"
minimum	mapped to datapoint metricName + "_MINIMUM"
maximum	mapped to datapoint metricName + "_MAXIMUM"
average	mapped to datapoint metricName + "_AVERAGE"

From this data a Metric of type Gauge is created with a Data Points that represents the values for the Metric including: Total, Minimum, Maximum, Average and Count.

Traces based on Azure Application Insights array of records from AppRequests & AppDependencies with the following fields.

Azure	Open Telemetry
Time	start_time(time_unix_nano(time))
	end_time(start_time + time_unix_nano(durationMs))
Name	span.name
OperationId	trace.id
ParentId	span.parentId
Id	span.id
AppRoleName	service.name

Documentation

Overview

Package azureeventhubreceiver listens to logs emitted by Azure Event hubs.

Index

• func NewFactory() receiver.Factory
• type AzureResourceLogsEventUnmarshaler
  • func (r AzureResourceLogsEventUnmarshaler) UnmarshalLogs(event *eventhub.Event) (plog.Logs, error)
• type Config
  • func (config *Config) Validate() error
• type TimeFormat

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory creates a factory for the Azure Event Hub receiver.

Types

type AzureResourceLogsEventUnmarshaler

type AzureResourceLogsEventUnmarshaler struct {
	// contains filtered or unexported fields
}

func (AzureResourceLogsEventUnmarshaler) UnmarshalLogs

func (r AzureResourceLogsEventUnmarshaler) UnmarshalLogs(event *eventhub.Event) (plog.Logs, error)

UnmarshalLogs takes a byte array containing a JSON-encoded payload with Azure log records and transforms it into an OpenTelemetry plog.Logs object. The data in the Azure log record appears as fields and attributes in the OpenTelemetry representation; the bodies of the OpenTelemetry log records are empty.

type Config

type Config struct {
	Connection               string        `mapstructure:"connection"`
	Partition                string        `mapstructure:"partition"`
	Offset                   string        `mapstructure:"offset"`
	StorageID                *component.ID `mapstructure:"storage"`
	Format                   string        `mapstructure:"format"`
	ConsumerGroup            string        `mapstructure:"group"`
	ApplySemanticConventions bool          `mapstructure:"apply_semantic_conventions"`
	TimeFormats              TimeFormat    `mapstructure:"time_formats"`
}

func (*Config) Validate

func (config *Config) Validate() error

Validate config

type TimeFormat

type TimeFormat struct {
	Logs    []string `mapstructure:"logs"`
	Metrics []string `mapstructure:"metrics"`
	Traces  []string `mapstructure:"traces"`
}