awss3exporter

README

AWS S3 Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@atoulme, @pdelewski

Schema supported

This exporter targets to support proto/json format.

Exporter Configuration

The following exporter configuration parameters are supported.

Name	Description	Default
region	AWS region.	"us-east-1"
s3_bucket	S3 bucket
s3_prefix	prefix for the S3 key (root directory inside bucket).
s3_partition	time granularity of S3 key: hour or minute	"minute"
role_arn	the Role ARN to be assumed
file_prefix	file prefix defined by user
marshaler	marshaler used to produce output data	otlp_json
encoding	Encoding extension to use to marshal data. Overrides the marshaler configuration option if set.
encoding_file_extension	file format extension suffix when using the encoding configuration option. May be left empty for no suffix to be appended.
endpoint	(REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from region and s3_bucket
s3_force_path_style	set this to true to force the request to use path-style addressing	false
disable_ssl	set this to true to disable SSL when sending requests	false
compression	should the file be compressed	none

Marshaler

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

• otlp_json (default): the OpenTelemetry Protocol format, represented as json.
• otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.
• sumo_ic: the Sumo Logic Installed Collector Archive format. This format is supported only for logs.
• body: export the log body as string. This format is supported only for logs.

Encoding

Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.

See https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding.

Compression

• none (default): No compression will be applied
• gzip: Files will be compressed with gzip. This does not support sumo_icmarshaler.

Example Configuration

Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.

exporters:
  awss3:
    s3uploader:
        region: 'eu-central-1'
        s3_bucket: 'databucket'
        s3_prefix: 'metric'
        s3_partition: 'minute'

Logs and traces will be stored inside 'databucket' in the following path format.

metric/year=XXXX/month=XX/day=XX/hour=XX/minute=XX

AWS Credential Configuration

This exporter follows default credential resolution for the aws-sdk-go.

Follow the guidelines for the credential configuration.

OpenTelemetry Collector Helm Chart for Kubernetes

For example, when using OpenTelemetry Collector Helm Chart you could use extraEnvs in the values.yaml.

extraEnvs:
- name: AWS_ACCESS_KEY_ID
  value: "< YOUR AWS ACCESS KEY >"
- name: AWS_SECRET_ACCESS_KEY
  value: "< YOUR AWS SECRET ACCESS KEY >"

Documentation

Overview

Package awss3exporter stores OpenTelemetry data as an AWS S3 exporter.

Index

• Constants
• Variables
• func NewFactory() exporter.Factory
• type Config
  • func (c *Config) Validate() error
• type MarshalerType
• type S3UploaderConfig

Constants

const (
	SourceCategoryKey = "_sourceCategory"
	SourceHostKey     = "_sourceHost"
	SourceNameKey     = "_sourceName"
)

Variables

var ErrUnknownMarshaler = errors.New("unknown marshaler")

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for S3 exporter.

Types

type Config

type Config struct {
	S3Uploader    S3UploaderConfig `mapstructure:"s3uploader"`
	MarshalerName MarshalerType    `mapstructure:"marshaler"`

	// Encoding to apply. If present, overrides the marshaler configuration option.
	Encoding              *component.ID `mapstructure:"encoding"`
	EncodingFileExtension string        `mapstructure:"encoding_file_extension"`
}

Config contains the main configuration options for the s3 exporter

func (*Config) Validate

func (c *Config) Validate() error

type MarshalerType

type MarshalerType string

const (
	OtlpProtobuf MarshalerType = "otlp_proto"
	OtlpJSON     MarshalerType = "otlp_json"
	SumoIC       MarshalerType = "sumo_ic"
	Body         MarshalerType = "body"
)

type S3UploaderConfig

type S3UploaderConfig struct {
	Region string `mapstructure:"region"`
	// S3Bucket is the bucket name to be uploaded to.
	S3Bucket string `mapstructure:"s3_bucket"`
	// S3Prefix is the key (directory) prefix to written to inside the bucket
	S3Prefix string `mapstructure:"s3_prefix"`
	// S3Partition is used to provide the rollup on how data is written.
	// Valid values are: [hour,minute]
	S3Partition string `mapstructure:"s3_partition"`
	// FilePrefix is the filename prefix used for the file to avoid any potential collisions.
	FilePrefix string `mapstructure:"file_prefix"`
	// Endpoint is the URL used for communicated with S3.
	Endpoint string `mapstructure:"endpoint"`
	// RoleArn is the role policy to use when interacting with S3
	RoleArn string `mapstructure:"role_arn"`
	// S3ForcePathStyle sets the value for force path style.
	S3ForcePathStyle bool `mapstructure:"s3_force_path_style"`
	// DisableSLL forces communication to happen via HTTP instead of HTTPS.
	DisableSSL bool `mapstructure:"disable_ssl"`
	// Compression sets the algorithm used to process the payload
	// before uploading to S3.
	// Valid values are: `gzip` or no value set.
	Compression configcompression.Type `mapstructure:"compression"`
}

S3UploaderConfig contains aws s3 uploader related config to controls things like bucket, prefix, batching, connections, retries, etc.