exporterhelper

package
v0.85.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 11, 2023 License: Apache-2.0 Imports: 22 Imported by: 251

README

Exporter Helper

This is a helper exporter that other exporters can depend on. Today, it primarily offers queued retry capabilities.

⚠ This exporter should not be added to a service pipeline.

Configuration

The following configuration options can be modified:

  • retry_on_failure
    • enabled (default = true)
    • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
    • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
    • max_elapsed_time (default = 300s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false
  • sending_queue
    • enabled (default = true)
    • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
    • queue_size (default = 1000): Maximum number of batches kept in memory before dropping; ignored if enabled is false User should calculate this as num_seconds * requests_per_second / requests_per_batch where:
      • num_seconds is the number of seconds to buffer in case of a backend outage
      • requests_per_second is the average number of requests per seconds
      • requests_per_batch is the average number of requests per batch (if the batch processor is used, the metric batch_send_size can be used for estimation)
  • timeout (default = 5s): Time to wait per individual attempt to send data to a backend

The initial_interval, max_interval, max_elapsed_time, and timeout options accept duration strings, valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Persistent Queue

Status: alpha

⚠ The capability is under development. To use it, a storage extension needs to be set up.

To use the persistent queue, the following setting needs to be set:

  • sending_queue
    • storage (default = none): When set, enables persistence and uses the component specified as a storage extension for the persistent queue

The maximum number of batches stored to disk can be controlled using sending_queue.queue_size parameter (which, similarly as for in-memory buffering, defaults to 1000 batches).

When persistent queue is enabled, the batches are being buffered using the provided storage extension - filestorage is a popular and safe choice. If the collector instance is killed while having some items in the persistent queue, on restart the items will be be picked and the exporting is continued.

                                                              ┌─Consumer #1─┐
                                                              │    ┌───┐    │
                              ──────Deleted──────        ┌───►│    │ 1 │    ├───► Success
        Waiting in channel    x           x     x        │    │    └───┘    │
        for consumer ───┐     x           x     x        │    │             │
                        │     x           x     x        │    └─────────────┘
                        ▼     x           x     x        │
┌─────────────────────────────────────────x─────x───┐    │    ┌─Consumer #2─┐
│                             x           x     x   │    │    │    ┌───┐    │
│     ┌───┐     ┌───┐ ┌───┐ ┌─x─┐ ┌───┐ ┌─x─┐ ┌─x─┐ │    │    │    │ 2 │    ├───► Permanent -> X
│ n+1 │ n │ ... │ 6 │ │ 5 │ │ 4 │ │ 3 │ │ 2 │ │ 1 │ ├────┼───►│    └───┘    │      failure
│     └───┘     └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ │    │    │             │
│                                                   │    │    └─────────────┘
└───────────────────────────────────────────────────┘    │
   ▲              ▲     ▲           ▲                    │    ┌─Consumer #3─┐
   │              │     │           │                    │    │    ┌───┐    │
   │              │     │           │                    │    │    │ 3 │    ├───► (in progress)
 write          read    └─────┬─────┘                    ├───►│    └───┘    │
 index          index         │                          │    │             │
   ▲                          │                          │    └─────────────┘
   │                          │                          │
   │                      currently                      │    ┌─Consumer #4─┐
   │                      dispatched                     │    │    ┌───┐    │     Temporary
   │                                                     └───►│    │ 4 │    ├───►  failure
   │                                                          │    └───┘    │         │
   │                                                          │             │         │
   │                                                          └─────────────┘         │
   │                                                                 ▲                │
   │                                                                 └── Retry ───────┤
   │                                                                                  │
   │                                                                                  │
   └────────────────────────────────────── Requeuing  ◄────── Retry limit exceeded ───┘

Example:

receivers:
  otlp:
    protocols:
      grpc:
exporters:
  otlp:
    endpoint: <ENDPOINT>
    sending_queue:
      storage: file_storage/otc
extensions:
  file_storage/otc:
    directory: /var/lib/storage/otc
    timeout: 10s
service:
  extensions: [file_storage]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      exporters: [otlp]

Documentation

Overview

Package exporterhelper provides helper functions for exporters.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewLogsExporter 

func NewLogsExporter(
	_ context.Context,
	set exporter.CreateSettings,
	cfg component.Config,
	pusher consumer.ConsumeLogsFunc,
	options ...Option,
) (exporter.Logs, error)

NewLogsExporter creates an exporter.Logs that records observability metrics and wraps every request with a Span.

func NewLogsRequestExporter added in v0.84.0 

func NewLogsRequestExporter(
	_ context.Context,
	set exporter.CreateSettings,
	converter LogsConverter,
	options ...Option,
) (exporter.Logs, error)

NewLogsRequestExporter creates new logs exporter based on custom LogsConverter and RequestSender. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func NewMetricsExporter 

func NewMetricsExporter(
	_ context.Context,
	set exporter.CreateSettings,
	cfg component.Config,
	pusher consumer.ConsumeMetricsFunc,
	options ...Option,
) (exporter.Metrics, error)

NewMetricsExporter creates an exporter.Metrics that records observability metrics and wraps every request with a Span.

func NewMetricsRequestExporter added in v0.84.0 

func NewMetricsRequestExporter(
	_ context.Context,
	set exporter.CreateSettings,
	converter MetricsConverter,
	options ...Option,
) (exporter.Metrics, error)

NewMetricsRequestExporter creates a new metrics exporter based on a custom MetricsConverter and RequestSender. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func NewThrottleRetry 

func NewThrottleRetry(err error, delay time.Duration) error

NewThrottleRetry creates a new throttle retry error.

func NewTracesExporter 

func NewTracesExporter(
	_ context.Context,
	set exporter.CreateSettings,
	cfg component.Config,
	pusher consumer.ConsumeTracesFunc,
	options ...Option,
) (exporter.Traces, error)

NewTracesExporter creates an exporter.Traces that records observability metrics and wraps every request with a Span.

func NewTracesRequestExporter added in v0.84.0 

func NewTracesRequestExporter(
	_ context.Context,
	set exporter.CreateSettings,
	converter TracesConverter,
	options ...Option,
) (exporter.Traces, error)

NewTracesRequestExporter creates a new traces exporter based on a custom TracesConverter and RequestSender. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

Types

type LogsConverter added in v0.84.0 

type LogsConverter interface {
	// RequestFromLogs converts plog.Logs data into a request.
	RequestFromLogs(context.Context, plog.Logs) (Request, error)
}

LogsConverter provides an interface for converting plog.Logs into a request. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type MetricsConverter added in v0.84.0 

type MetricsConverter interface {
	// RequestFromMetrics converts pdata.Metrics into a request.
	RequestFromMetrics(context.Context, pmetric.Metrics) (Request, error)
}

MetricsConverter provides an interface for converting pmetric.Metrics into a request. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type Option 

type Option func(*baseExporter)

Option apply changes to baseExporter.

func WithCapabilities 

func WithCapabilities(capabilities consumer.Capabilities) Option

WithCapabilities overrides the default Capabilities() function for a Consumer. The default is non-mutable data. TODO: Verify if we can change the default to be mutable as we do for processors.

func WithQueue 

func WithQueue(config QueueSettings) Option

WithQueue overrides the default QueueSettings for an exporter. The default QueueSettings is to disable queueing. This option cannot be used with the new exporter helpers New[Traces|Metrics|Logs]RequestExporter.

func WithRetry 

func WithRetry(retrySettings RetrySettings) Option

WithRetry overrides the default RetrySettings for an exporter. The default RetrySettings is to disable retries.

func WithShutdown 

func WithShutdown(shutdown component.ShutdownFunc) Option

WithShutdown overrides the default Shutdown function for an exporter. The default shutdown function does nothing and always returns nil.

func WithStart 

func WithStart(start component.StartFunc) Option

WithStart overrides the default Start function for an exporter. The default start function does nothing and always returns nil.

func WithTimeout 

func WithTimeout(timeoutSettings TimeoutSettings) Option

WithTimeout overrides the default TimeoutSettings for an exporter. The default TimeoutSettings is 5 seconds.

type QueueSettings 

type QueueSettings struct {
	// Enabled indicates whether to not enqueue batches before sending to the consumerSender.
	Enabled bool `mapstructure:"enabled"`
	// NumConsumers is the number of consumers from the queue.
	NumConsumers int `mapstructure:"num_consumers"`
	// QueueSize is the maximum number of batches allowed in queue at a given time.
	QueueSize int `mapstructure:"queue_size"`
	// StorageID if not empty, enables the persistent storage and uses the component specified
	// as a storage extension for the persistent queue
	StorageID *component.ID `mapstructure:"storage"`
}

QueueSettings defines configuration for queueing batches before sending to the consumerSender.

func NewDefaultQueueSettings 

func NewDefaultQueueSettings() QueueSettings

NewDefaultQueueSettings returns the default settings for QueueSettings.

func (*QueueSettings) Validate 

func (qCfg *QueueSettings) Validate() error

Validate checks if the QueueSettings configuration is valid

type Request added in v0.84.0 

type Request interface {
	// Export exports the request to an external endpoint.
	Export(ctx context.Context) error
}

Request represents a single request that can be sent to an external endpoint. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type RequestItemsCounter added in v0.84.0 

type RequestItemsCounter interface {
	// ItemsCount returns a number of basic items in the request where item is the smallest piece of data that can be
	// sent. For example, for OTLP exporter, this value represents the number of spans,
	// metric data points or log records.
	ItemsCount() int
}

RequestItemsCounter is an optional interface that can be implemented by Request to provide a number of items in the request. This is a recommended interface to implement for exporters. It is required for batching and queueing based on number of items. Also, it's used for reporting number of items in collector's logs, metrics and traces. If not implemented, collector's logs, metrics and traces will report 0 items. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type RetrySettings 

type RetrySettings struct {
	// Enabled indicates whether to not retry sending batches in case of export failure.
	Enabled bool `mapstructure:"enabled"`
	// InitialInterval the time to wait after the first failure before retrying.
	InitialInterval time.Duration `mapstructure:"initial_interval"`
	// RandomizationFactor is a random factor used to calculate next backoffs
	// Randomized interval = RetryInterval * (1 ± RandomizationFactor)
	RandomizationFactor float64 `mapstructure:"randomization_factor"`
	// Multiplier is the value multiplied by the backoff interval bounds
	Multiplier float64 `mapstructure:"multiplier"`
	// MaxInterval is the upper bound on backoff interval. Once this value is reached the delay between
	// consecutive retries will always be `MaxInterval`.
	MaxInterval time.Duration `mapstructure:"max_interval"`
	// MaxElapsedTime is the maximum amount of time (including retries) spent trying to send a request/batch.
	// Once this value is reached, the data is discarded.
	MaxElapsedTime time.Duration `mapstructure:"max_elapsed_time"`
}

RetrySettings defines configuration for retrying batches in case of export failure. The current supported strategy is exponential backoff.

func NewDefaultRetrySettings 

func NewDefaultRetrySettings() RetrySettings

NewDefaultRetrySettings returns the default settings for RetrySettings.

type TimeoutSettings 

type TimeoutSettings struct {
	// Timeout is the timeout for every attempt to send data to the backend.
	Timeout time.Duration `mapstructure:"timeout"`
}

TimeoutSettings for timeout. The timeout applies to individual attempts to send data to the backend.

func NewDefaultTimeoutSettings 

func NewDefaultTimeoutSettings() TimeoutSettings

NewDefaultTimeoutSettings returns the default settings for TimeoutSettings.

type TracesConverter added in v0.84.0 

type TracesConverter interface {
	// RequestFromTraces converts ptrace.Traces into a Request.
	RequestFromTraces(context.Context, ptrace.Traces) (Request, error)
}

TracesConverter provides an interface for converting ptrace.Traces into a request. This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.