awskinesisexporter

README

Kinesis Exporter

Status
Stability	beta: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@Aneurysm9, @MovieStoreGuy

The kinesis exporter currently exports dynamic encodings to the configured kinesis stream. The exporter relies heavily on the kinesis.PutRecords api to reduce network I/O and and reduces records into smallest atomic representation to avoid hitting the hard limits placed on Records (No greater than 1Mb). This producer will block until the operation is done to allow for retryable and queued data to help during high loads.

The following settings are required:

• aws
  • stream_name (no default): The name of the Kinesis stream to export to.

The following settings can be optionally configured:

• aws
  • kinesis_endpoint (no default)
  • region (default = us-west-2): the region that the kinesis stream is deployed in
  • role (no default): The role to be used in order to send data to the kinesis stream
• encoding
  • name (default = otlp): defines the export type to be used to send to kinesis (available is otlp_proto, otlp_json, zipkin_proto, zipkin_json, jaeger_proto)
    • Note : otlp_json is considered experimental and should not be used for production environments.
  • compression (default = none): allows to set the compression type (defaults BestSpeed for all) before forwarding to kinesis (available is flate, gzip, zlib or none)
• max_records_per_batch (default = 500, PutRecords limit): The number of records that can be batched together then sent to kinesis.
• max_record_size (default = 1Mb, PutRecord(s) limit on record size): The max allowed size that can be exported to kinesis
• timeout (default = 5s): Is the timeout for every attempt to send data to the backend.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false
• sending_queue
  • enabled (default = true)
  • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
  • queue_size (default = 1000): Maximum number of batches kept in memory before dropping data; ignored if enabled is false; User should calculate this as num_seconds * requests_per_second where:
    • num_seconds is the number of seconds to buffer in case of a backend outage
    • requests_per_second is the average number of requests per seconds.

Example Configuration:

exporters:
  awskinesis:
    aws:
      stream_name: raw-trace-stream
      region: us-east-1
      role: arn:test-role

Documentation

Index

• func NewFactory() exporter.Factory
• type AWSConfig
• type Config
  • func (cfg *Config) Validate() error
• type Encoding
• type Exporter

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for Kinesis exporter.

Types

type AWSConfig

type AWSConfig struct {
	StreamName      string `mapstructure:"stream_name"`
	KinesisEndpoint string `mapstructure:"kinesis_endpoint"`
	Region          string `mapstructure:"region"`
	Role            string `mapstructure:"role"`
}

AWSConfig contains AWS specific configuration such as awskinesis stream, region, etc.

type Config

type Config struct {
	exporterhelper.TimeoutSettings `mapstructure:",squash"`
	exporterhelper.QueueSettings   `mapstructure:"sending_queue"`
	configretry.BackOffConfig      `mapstructure:"retry_on_failure"`

	Encoding           `mapstructure:"encoding"`
	AWS                AWSConfig `mapstructure:"aws"`
	MaxRecordsPerBatch int       `mapstructure:"max_records_per_batch"`
	MaxRecordSize      int       `mapstructure:"max_record_size"`
}

Config contains the main configuration options for the awskinesis exporter

func (*Config) Validate

func (cfg *Config) Validate() error

Validate checks if the exporter configuration is valid

type Encoding

type Encoding struct {
	Name        string `mapstructure:"name"`
	Compression string `mapstructure:"compression"`
}

type Exporter

type Exporter struct {
	// contains filtered or unexported fields
}

Exporter implements an OpenTelemetry trace exporter that exports all spans to AWS Kinesis