awss3receiver

README

AWS S3 Receiver

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@atoulme, @adcharre

Overview

Receiver for retrieving trace previously stored in S3 by the AWS S3 Exporter.

Configuration

The following exporter configuration parameters are supported.

Name	Description	Default	Required
starttime	The time at which to start retrieving data.		Required
endtime	The time at which to stop retrieving data.		Required
s3downloader:
region	AWS region.	"us-east-1"	Optional
s3_bucket	S3 bucket		Required
s3_prefix	prefix for the S3 key (root directory inside bucket).		Required
s3_partition	time granularity of S3 key: hour or minute	"minute"	Optional
file_prefix	file prefix defined by user		Optional
endpoint	overrides the endpoint used by the exporter instead of constructing it from region and s3_bucket		Optional
endpoint_partition_id	partition id to use if endpoint is specified.	"aws"	Optional
s3_force_path_style	set this to true to force the request to use path-style addressing	false	Optional
encodings:	An array of entries with the following properties:		Optional
extension	Extension to use for decoding a key with a matching suffix.		Required
suffix	Key suffix to match against.		Required
notifications:
opampextension	Name of the OpAMP Extension to use to send ingest progress notifications.

Time format for starttime and endtime

The starttime and endtime fields are used to specify the time range for which to retrieve data. The time format is either RFC3339,YYYY-MM-DD HH:MM or simply YYYY-MM-DD, in which case the time is assumed to be 00:00.

Encodings

By default, the receiver understands the following encodings:

• otlp_json (OpenTelemetry Protocol format represented as json) with a suffix of .json
• otlp_proto (OpenTelemetry Protocol format represented as Protocol Buffers) with a suffix of .binpb

The encodings options allows you to specify Encoding Extensions to use to decode keys with matching suffixes.

Example Configuration

extension:
  # example of text encoding extension
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"
    
receivers:
  awss3:
    starttime: "2024-01-01 01:00"
    endtime: "2024-01-02"
    s3downloader:
        region: "us-west-1"
        s3_bucket: "mybucket"
        s3_prefix: "trace"
        s3_partition: "minute"
    encodings:
      - extension: text_encoding
        suffix: ".txt"

Notifications

The receiver can send notifications of ingest progress to an OpAmp server using the custom message capability of "org.opentelemetry.collector.receiver.awss3" and message type "TimeBasedIngestStatus". The format of the notifications is a ProtoBuf formatted OLTP logs message with a single Log Record. The body of the record is set to status and the timestamp of the record is used to hold the ingest time. The record also has the following attributes:

Attribute	Description
telemetry_type	The type of telemetry being ingested. One of "traces", "metrics", or "logs".
ingest_status	The status of the data ingestion. One of "ingesting", "failed", or "completed".
start_time	The time to start retrieving data as an Int64, nanoseconds since Unix epoch.
end_time	The time to stop retrieving data as an Int64, nanoseconds since Unix epoch.
failure_message	Error message if ingest_status is "failed".

The "ingesting" status is sent at the beginning of the ingest process before data has been retrieved for the specified time. If during the processing of the data an error occurs a status message with ingest_status set to "failed" status with the time of the data being ingested when the failure occurred. If the ingest process completes successfully a status message with ingest_status set to "completed" is sent.

Documentation

Overview

Package awss3receiver implements a receiver that can be used by the Opentelemetry collector to retrieve traces previously stored in S3 by the AWS S3 Exporter.

Index

• Constants
• func NewFactory() receiver.Factory
• type Config
  • func (c Config) Validate() error
• type Encoding
• type GetObjectAPI
• type ListObjectsAPI
• type ListObjectsV2Pager
• type Notifications
• type S3DownloaderConfig

Constants

const (
	S3PartitionMinute = "minute"
	S3PartitionHour   = "hour"
)

const (
	IngestStatusCompleted = "completed"
	IngestStatusFailed    = "failed"
	IngestStatusIngesting = "ingesting"
	CustomCapability      = "org.opentelemetry.collector.receiver.awss3"
)

Functions

func NewFactory

func NewFactory() receiver.Factory

Types

type Config

type Config struct {
	S3Downloader  S3DownloaderConfig `mapstructure:"s3downloader"`
	StartTime     string             `mapstructure:"starttime"`
	EndTime       string             `mapstructure:"endtime"`
	Encodings     []Encoding         `mapstructure:"encodings"`
	Notifications Notifications      `mapstructure:"notifications"`
}

Config defines the configuration for the file receiver.

func (Config) Validate

func (c Config) Validate() error

type Encoding

type Encoding struct {
	Extension component.ID `mapstructure:"extension"`
	Suffix    string       `mapstructure:"suffix"`
}

type GetObjectAPI

type GetObjectAPI interface {
	GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}

type ListObjectsAPI

type ListObjectsAPI interface {
	NewListObjectsV2Paginator(params *s3.ListObjectsV2Input) ListObjectsV2Pager
}

type ListObjectsV2Pager

type ListObjectsV2Pager interface {
	HasMorePages() bool
	NextPage(context.Context, ...func(*s3.Options)) (*s3.ListObjectsV2Output, error)
}

type Notifications

type Notifications struct {
	OpAMP *component.ID `mapstructure:"opampextension"`
}

type S3DownloaderConfig

type S3DownloaderConfig struct {
	Region              string `mapstructure:"region"`
	S3Bucket            string `mapstructure:"s3_bucket"`
	S3Prefix            string `mapstructure:"s3_prefix"`
	S3Partition         string `mapstructure:"s3_partition"`
	FilePrefix          string `mapstructure:"file_prefix"`
	Endpoint            string `mapstructure:"endpoint"`
	EndpointPartitionID string `mapstructure:"endpoint_partition_id"`
	S3ForcePathStyle    bool   `mapstructure:"s3_force_path_style"`
}

S3DownloaderConfig contains aws s3 downloader related config to controls things like bucket, prefix, batching, connections, retries, etc.