awss3exporter

README

AWS S3 Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib, observiq, splunk, sumo
Issues
Code Owners	@atoulme, @pdelewski

Schema supported

This exporter targets to support proto/json format.

Exporter Configuration

The following exporter configuration parameters are supported.

Name	Description	Default
region	AWS region.	"us-east-1"
s3_bucket	S3 bucket
s3_prefix	prefix for the S3 key (root directory inside bucket).
s3_partition	time granularity of S3 key: hour or minute	"minute"
role_arn	the Role ARN to be assumed
file_prefix	file prefix defined by user
marshaler	marshaler used to produce output data	otlp_json
endpoint	overrides the endpoint used by the exporter instead of constructing it from region and s3_bucket
s3_force_path_style	set this to true to force the request to use path-style addressing	false
disable_ssl	set this to true to disable SSL when sending requests	false

Marshaler

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

• otlp_json (default): the OpenTelemetry Protocol format, represented as json.
• otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.
• sumo_ic: the Sumo Logic Installed Collector Archive format. This format is supported only for logs.

Example Configuration

Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.

exporters:
  awss3:
    s3uploader:
        region: 'eu-central-1'
        s3_bucket: 'databucket'
        s3_prefix: 'metric'
        s3_partition: 'minute'

Logs and traces will be stored inside 'databucket' in the following path format.

metric/year=XXXX/month=XX/day=XX/hour=XX/minute=XX

AWS Credential Configuration

This exporter follows default credential resolution for the aws-sdk-go.

Follow the guidelines for the credential configuration.

Documentation

Overview

Package awss3exporter stores OpenTelemetry data as an AWS S3 exporter.

Index

• Constants
• Variables
• func NewFactory() exporter.Factory
• type Config
  • func (c *Config) Validate() error
• type MarshalerType
• type S3UploaderConfig

Constants

const (
	SourceCategoryKey = "_sourceCategory"
	SourceHostKey     = "_sourceHost"
	SourceNameKey     = "_sourceName"
)

Variables

var (
	ErrUnknownMarshaler = errors.New("unknown marshaler")
)

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for S3 exporter.

Types

type Config

type Config struct {
	S3Uploader    S3UploaderConfig `mapstructure:"s3uploader"`
	MarshalerName MarshalerType    `mapstructure:"marshaler"`

	FileFormat string `mapstructure:"file_format"`
}

Config contains the main configuration options for the s3 exporter

func (*Config) Validate

func (c *Config) Validate() error

type MarshalerType

type MarshalerType string

const (
	OtlpProtobuf MarshalerType = "otlp_proto"
	OtlpJSON     MarshalerType = "otlp_json"
	SumoIC       MarshalerType = "sumo_ic"
)

type S3UploaderConfig

type S3UploaderConfig struct {
	Region           string `mapstructure:"region"`
	S3Bucket         string `mapstructure:"s3_bucket"`
	S3Prefix         string `mapstructure:"s3_prefix"`
	S3Partition      string `mapstructure:"s3_partition"`
	FilePrefix       string `mapstructure:"file_prefix"`
	Endpoint         string `mapstructure:"endpoint"`
	RoleArn          string `mapstructure:"role_arn"`
	S3ForcePathStyle bool   `mapstructure:"s3_force_path_style"`
	DisableSSL       bool   `mapstructure:"disable_ssl"`
}

S3UploaderConfig contains aws s3 uploader related config to controls things like bucket, prefix, batching, connections, retries, etc.