README
¶
Exporter Helper
This is a helper exporter that other exporters can depend on. Today, it primarily offers queued retries and resource attributes to metric labels conversion.
⚠ This exporter should not be added to a service pipeline.
Configuration
The following configuration options can be modified:
retry_on_failureenabled(default = true)initial_interval(default = 5s): Time to wait after the first failure before retrying; ignored ifenabledisfalsemax_interval(default = 30s): Is the upper bound on backoff; ignored ifenabledisfalsemax_elapsed_time(default = 300s): Is the maximum amount of time spent trying to send a batch; ignored ifenabledisfalse
sending_queueenabled(default = true)num_consumers(default = 10): Number of consumers that dequeue batches; ignored ifenabledisfalsequeue_size(default = 5000): Maximum number of batches kept in memory before dropping; ignored ifenabledisfalseUser should calculate this asnum_seconds * requests_per_second / requests_per_batchwhere:num_secondsis the number of seconds to buffer in case of a backend outagerequests_per_secondis the average number of requests per secondsrequests_per_batchis the average number of requests per batch (if the batch processor is used, the metricbatch_send_sizecan be used for estimation)
timeout(default = 5s): Time to wait per individual attempt to send data to a backend
Persistent Queue
Status: alpha
⚠ The capability is under development. To use it, a storage extension needs to be set up.
To use the persistent queue, the following setting needs to be set:
sending_queuestorage(default = none): When set, enables persistence and uses the component specified as a storage extension for the persistent queue
The maximum number of batches stored to disk can be controlled using sending_queue.queue_size parameter (which,
similarly as for in-memory buffering, defaults to 5000 batches).
When persistent queue is enabled, the batches are being buffered using the provided storage extension - filestorage is a popular and safe choice. If the collector instance is killed while having some items in the persistent queue, on restart the items will be be picked and the exporting is continued.
┌─Consumer #1─┐
│ ┌───┐ │
──────Deleted────── ┌───►│ │ 1 │ ├───► Success
Waiting in channel x x x │ │ └───┘ │
for consumer ───┐ x x x │ │ │
│ x x x │ └─────────────┘
▼ x x x │
┌─────────────────────────────────────────x─────x───┐ │ ┌─Consumer #2─┐
│ x x x │ │ │ ┌───┐ │
│ ┌───┐ ┌───┐ ┌───┐ ┌─x─┐ ┌───┐ ┌─x─┐ ┌─x─┐ │ │ │ │ 2 │ ├───► Permanent -> X
│ n+1 │ n │ ... │ 6 │ │ 5 │ │ 4 │ │ 3 │ │ 2 │ │ 1 │ ├────┼───►│ └───┘ │ failure
│ └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ │ │ │ │
│ │ │ └─────────────┘
└───────────────────────────────────────────────────┘ │
▲ ▲ ▲ ▲ │ ┌─Consumer #3─┐
│ │ │ │ │ │ ┌───┐ │
│ │ │ │ │ │ │ 3 │ ├───► (in progress)
write read └─────┬─────┘ ├───►│ └───┘ │
index index │ │ │ │
▲ │ │ └─────────────┘
│ │ │
│ currently │ ┌─Consumer #4─┐
│ dispatched │ │ ┌───┐ │ Temporary
│ └───►│ │ 4 │ ├───► failure
│ │ └───┘ │ │
│ │ │ │
│ └─────────────┘ │
│ ▲ │
│ └── Retry ───────┤
│ │
│ │
└────────────────────────────────────── Requeuing ◄────── Retry limit exceeded ───┘
Example:
receivers:
otlp:
protocols:
grpc:
exporters:
otlp:
endpoint: <ENDPOINT>
sending_queue:
storage: file_storage/otc
extensions:
file_storage/otc:
directory: /var/lib/storage/otc
timeout: 10s
service:
extensions: [file_storage]
pipelines:
metrics:
receivers: [otlp]
exporters: [otlp]
logs:
receivers: [otlp]
exporters: [otlp]
traces:
receivers: [otlp]
exporters: [otlp]
Documentation
¶
Overview ¶
Package exporterhelper provides helper functions for exporters.
Index ¶
- func NewLogsExporter(_ context.Context, set component.ExporterCreateSettings, cfg config.Exporter, ...) (component.LogsExporter, error)
- func NewMetricsExporter(_ context.Context, set component.ExporterCreateSettings, cfg config.Exporter, ...) (component.MetricsExporter, error)
- func NewThrottleRetry(err error, delay time.Duration) error
- func NewTracesExporter(_ context.Context, set component.ExporterCreateSettings, cfg config.Exporter, ...) (component.TracesExporter, error)
- type Option
- func WithCapabilities(capabilities consumer.Capabilities) Option
- func WithQueue(queueSettings QueueSettings) Option
- func WithRetry(retrySettings RetrySettings) Option
- func WithShutdown(shutdown component.ShutdownFunc) Option
- func WithStart(start component.StartFunc) Option
- func WithTimeout(timeoutSettings TimeoutSettings) Option
- type QueueSettings
- type RetrySettings
- type TimeoutSettings
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewLogsExporter ¶
func NewLogsExporter( _ context.Context, set component.ExporterCreateSettings, cfg config.Exporter, pusher consumer.ConsumeLogsFunc, options ...Option, ) (component.LogsExporter, error)
NewLogsExporter creates a component.LogsExporter that records observability metrics and wraps every request with a Span.
func NewMetricsExporter ¶
func NewMetricsExporter( _ context.Context, set component.ExporterCreateSettings, cfg config.Exporter, pusher consumer.ConsumeMetricsFunc, options ...Option, ) (component.MetricsExporter, error)
NewMetricsExporter creates a component.MetricsExporter that records observability metrics and wraps every request with a Span.
func NewThrottleRetry ¶ added in v0.7.0
NewThrottleRetry creates a new throttle retry error.
func NewTracesExporter ¶ added in v0.25.0
func NewTracesExporter( _ context.Context, set component.ExporterCreateSettings, cfg config.Exporter, pusher consumer.ConsumeTracesFunc, options ...Option, ) (component.TracesExporter, error)
NewTracesExporter creates a component.TracesExporter that records observability metrics and wraps every request with a Span.
Types ¶
type Option ¶ added in v0.16.0
type Option func(*baseSettings)
Option apply changes to baseSettings.
func WithCapabilities ¶ added in v0.27.0
func WithCapabilities(capabilities consumer.Capabilities) Option
WithCapabilities overrides the default Capabilities() function for a Consumer. The default is non-mutable data. TODO: Verify if we can change the default to be mutable as we do for processors.
func WithQueue ¶ added in v0.7.0
func WithQueue(queueSettings QueueSettings) Option
WithQueue overrides the default QueueSettings for an exporter. The default QueueSettings is to disable queueing.
func WithRetry ¶ added in v0.7.0
func WithRetry(retrySettings RetrySettings) Option
WithRetry overrides the default RetrySettings for an exporter. The default RetrySettings is to disable retries.
func WithShutdown ¶
func WithShutdown(shutdown component.ShutdownFunc) Option
WithShutdown overrides the default Shutdown function for an exporter. The default shutdown function does nothing and always returns nil.
func WithStart ¶ added in v0.6.0
WithStart overrides the default Start function for an exporter. The default start function does nothing and always returns nil.
func WithTimeout ¶ added in v0.7.0
func WithTimeout(timeoutSettings TimeoutSettings) Option
WithTimeout overrides the default TimeoutSettings for an exporter. The default TimeoutSettings is 5 seconds.
type QueueSettings ¶ added in v0.7.0
type QueueSettings struct {
// Enabled indicates whether to not enqueue batches before sending to the consumerSender.
Enabled bool `mapstructure:"enabled"`
// NumConsumers is the number of consumers from the queue.
NumConsumers int `mapstructure:"num_consumers"`
// QueueSize is the maximum number of batches allowed in queue at a given time.
QueueSize int `mapstructure:"queue_size"`
// StorageID if not empty, enables the persistent storage and uses the component specified
// as a storage extension for the persistent queue
StorageID *config.ComponentID `mapstructure:"storage"`
}
QueueSettings defines configuration for queueing batches before sending to the consumerSender.
func NewDefaultQueueSettings ¶ added in v0.46.0
func NewDefaultQueueSettings() QueueSettings
NewDefaultQueueSettings returns the default settings for QueueSettings.
func (*QueueSettings) Validate ¶ added in v0.45.0
func (qCfg *QueueSettings) Validate() error
Validate checks if the QueueSettings configuration is valid
type RetrySettings ¶ added in v0.7.0
type RetrySettings struct {
// Enabled indicates whether to not retry sending batches in case of export failure.
Enabled bool `mapstructure:"enabled"`
// InitialInterval the time to wait after the first failure before retrying.
InitialInterval time.Duration `mapstructure:"initial_interval"`
// MaxInterval is the upper bound on backoff interval. Once this value is reached the delay between
// consecutive retries will always be `MaxInterval`.
MaxInterval time.Duration `mapstructure:"max_interval"`
// MaxElapsedTime is the maximum amount of time (including retries) spent trying to send a request/batch.
// Once this value is reached, the data is discarded.
MaxElapsedTime time.Duration `mapstructure:"max_elapsed_time"`
}
RetrySettings defines configuration for retrying batches in case of export failure. The current supported strategy is exponential backoff.
func NewDefaultRetrySettings ¶ added in v0.46.0
func NewDefaultRetrySettings() RetrySettings
NewDefaultRetrySettings returns the default settings for RetrySettings.
type TimeoutSettings ¶ added in v0.7.0
type TimeoutSettings struct {
// Timeout is the timeout for every attempt to send data to the backend.
Timeout time.Duration `mapstructure:"timeout"`
}
TimeoutSettings for timeout. The timeout applies to individual attempts to send data to the backend.
func NewDefaultTimeoutSettings ¶ added in v0.46.0
func NewDefaultTimeoutSettings() TimeoutSettings
NewDefaultTimeoutSettings returns the default settings for TimeoutSettings.
Source Files
Directories