cortex

README

OpenTelemetry Go SDK Prometheus Remote Write Exporter for Cortex

⚠️ This project is deprecated. Use the go.opentelemetry.io/otel/exporters/otlp/otlpmetric exporter as a replacement to send data to a collector which can then export with its PRW exporter.

This module contains an exporter that sends cumulative metrics data from the OpenTelemetry Go SDK to Cortex using the Prometheus Remote Write API. While it is aimed at Cortex, it should work with other backends that ingest data with the same API.

This exporter is push-based and integrates with the OpenTelemetry Go SDK's push Controller. The Controller periodically collects data and passes it to this exporter. The exporter then converts this data into TimeSeries, a format that Cortex accepts, and sends it to Cortex through HTTP POST requests. The request body is formatted according to the protocol defined by the Prometheus Remote Write API. See Prometheus's remote storage integration documentation for more details on the Remote Write API.

See the example submodule for a working example of this exporter.

Table of Contents

• OpenTelemetry Go SDK Prometheus Remote Write Exporter for Cortex
• Table of Contents
  • Installation
  • Setting up the Exporter
  • Configuring the Exporter
  • Securing the Exporter
    • Authentication
    • TLS
  • Instrument to Aggregation Mapping
  • Error Handling
  • Retry Logic
  • Design Document
  • Future Enhancements

Installation

go get -u go.opentelemetry.io/contrib/exporters/metric/cortex

Setting up the Exporter

Users only need to call the InstallNewPipeline function to setup the exporter. It requires a Config struct and returns a push Controller and error. If the error is nil, the setup is successful and the user can begin creating instruments. No other action is needed.

// Create a Config struct named `config`.

controller, err := cortex.InstallNewPipeline(config)
if err != nil {
    return err
}
defer controller.Stop(context.Background())

// Make instruments and record data using `global.MeterProvider`.

Configuring the Exporter

The Exporter requires certain information, such as the endpoint URL and push interval duration, to function properly. This information is stored in a Config struct, which is passed into the Exporter during the setup pipeline.

There are two options for creating a Config struct:

1. Use the utils submodule to read settings from a YAML file into a new Config struct

   • Call utils.NewConfig(...) to create the struct. More details can be found in the utils module's README
   • Config structs have a Validate() method that sets defaults and checks for errors, but it isn't necessary to call it as utils.NewConfig() does that already.

2. Create the Config struct manually

   • Users should call the Config struct's Validate() method to set default values and check for errors.

// (Option 1) Create Config struct using utils module.
config, err := utils.NewConfig("config.yml")
if err != nil {
    return err
}

// (Option 2) Create Config struct manually.
configTwo := cortex.Config {
  Endpoint:      "http://localhost:9009/api/prom/push",
	RemoteTimeout: 30 * time.Second,
	PushInterval: 5 * time.Second,
	Headers: map[string]string{
		"test": "header",
	},
}
// Validate() should be called when creating the Config struct manually.
err = config.Validate()
if err != nil {
	return err
}

The Config struct supports many different configuration options. Here is the Config struct definition as well as the supported YAML properties. The mapstructure tags are used by the utils submodule.

type Config struct {
	Endpoint            string            `mapstructure:"url"`
	RemoteTimeout       time.Duration     `mapstructure:"remote_timeout"`
	Name                string            `mapstructure:"name"`
	BasicAuth           map[string]string `mapstructure:"basic_auth"`
	BearerToken         string            `mapstructure:"bearer_token"`
	BearerTokenFile     string            `mapstructure:"bearer_token_file"`
	TLSConfig           map[string]string `mapstructure:"tls_config"`
	ProxyURL            string            `mapstructure:"proxy_url"`
	PushInterval        time.Duration     `mapstructure:"push_interval"`
	Quantiles           []float64         `mapstructure:"quantiles"`
	HistogramBoundaries []float64         `mapstructure:"histogram_boundaries"`
	Headers             map[string]string `mapstructure:"headers"`
	Client              *http.Client
}

Supported YAML Properties

This is sourced from the Prometheus Remote Write Configuration documentation.

# The URL of the endpoint to send samples to.
url: <string>

# Timeout for requests to the remote write endpoint.
[ remote_timeout: <duration> | default = 30s ]

# Name of the remote write config, which if specified must be unique among remote write configs. The name will be used in metrics and logging in place of a generated value to help users distinguish between remote write configs.
[ name: <string>]

# Sets the `Authorization` header on every remote write request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
  [ username: <string>]
  [ password: <string>]
  [ password_file: <string> ]

# Sets the `Authorization` header on every remote write request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]

# Sets the `Authorization` header on every remote write request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]

# Configures the remote write request's TLS settings.
tls_config:
  # CA certificate to validate API server certificate with.
  [ ca_file: <filename>]

  # Certificate and key files for client cert authentication to the   server.
  [ cert_file: <filename> ]
  [ key_file: <filename> ]

  # ServerName extension to indicate the name of the server.
  # https://tools.ietf.org/html/rfc4366#section-3.1
  [ server_name: <string> ]

  # Disable validation of the server certificate.
  [ insecure_skip_verify: <boolean> ]

# Optional proxy URL.
[ proxy_url: <string>]

# Quantiles for Distribution aggregations
[ quantiles: ]
  - <string>
  - <string>
  - ...

# Histogram Buckets
[ histogram_buckets: ]
  - <string>
  - <string>
  - ...

Securing the Exporter

Authentication

The exporter provides two forms of authentication which are shown below. Users can add their own custom authentication by providing their own HTTP Client through the Config struct and customizing it as needed.

1. Basic Authentication

    // Basic Authentication properties in the Config struct.
    cortex.Config{
      // ...
      BasicAuth: map[string]string{
        "username":      "user",
        "password":      "password",
        "password_file": "passwordFile",
    }
   

    # Basic Authentication properties in the YAML file.
    basic_auth:
      username: user
      password: password
      password_file: passwordfile
   

   Basic authentication sets a HTTP Authorization header containing a base64 encoded username/password pair. See RFC 7617 for more information. Note that the password and password file are mutually exclusive. The Config struct's Validate() method will return an error if both are set.

2. Bearer Token Authentication

    // Bearer Token Authentication properties in the Config struct.
   cortex.Config{
      BearerToken:     "token",
      BearerTokenFile: "tokenfile",
    }
   

    # Bearer Token Authentication properties in the YAML file.
    bearer_token: token
    bearer_token_file: tokenfile
   

   Bearer token authentication sets a HTTP Authorization header containing a bearer token. See RFC 6750 for more information. Note that the bearer token and bearer token file are mutually exclusive. The Config struct's Validate() method will return an error if both are set.

TLS

Users can add TLS to the exporter's HTTP Client through the Config struct by providing certificate and key files. The certificate type does not matter. See TestBuildClient() and TestMutualTLS() in auth_test.go for an example of how TLS can be used with the exporter. TestMutualTLS() checks certificates between the exporter and a server both ways.

// TLS properties in the Config struct.
cortex.Config{
  TLSConfig: map[string]string{
    "ca_file":              "cafile",
    "cert_file":            "certfile",
    "key_file":             "keyfile",
    "server_name":          "server",
    "insecure_skip_verify": "0",
  },
}

# TLS properties in the YAML file.
tls_config:
  ca_file: cafile
  cert_file: certfile
  key_file: keyfile
  server_name: server
  insecure_skip_verify: true

Instrument to Aggregation Mapping

The exporter uses the simple selector's NewWithHistogramDistribution(). This means that instruments are mapped to aggregations as shown in the table below.

Instrument	Aggregation
Counter	Sum
UpDownCounter	Sum
ValueRecorder	Histogram
SumObserver	Sum
UpDownSumObserver	Sum
ValueObserver	Histogram

Although only the Sum and Histogram aggregations are currently being used, the exporter supports 4 different aggregations:

1. Sum
2. LastValue
3. Distribution
4. Histogram

Error Handling

In general, errors are returned to the calling function / method. Eventually, errors make their way up to the push Controller where it calls the exporter's Export() method. The push Controller passes the errors to the OpenTelemetry Go SDK's global error handler.

The exception is when the exporter fails to send an HTTP request to Cortex. Regardless of status code, the error is ignored. See the retry logic section below for more details.

Retry Logic

The exporter does not implement any retry logic since the exporter sends cumulative metrics data, which means that data will be preserved even if some exports fail.

For example, consider a situation where a user increments a Counter instrument 5 times and an export happens between each increment. If the exports happen like so:

  SUCCESS FAIL FAIL SUCCESS SUCCESS
  1       2    3    4       5

Then the received data will be:

1 4 5

The end result is the same since the aggregations are cumulative.

Design Document

Design Document

The document is not in this module as it contains large images which will increase the size of the overall repo significantly.

Future Enhancements

• Add configuration option for different selectors

  Users may not want to use the default Histogram selector and should be able to choose which selector they want to use.

Documentation

Overview

Package cortex provides a Cortex exporter. It provides functionality to sends cumulative metric data Cortex using the Prometheus Remote Write API.

Deprecated: This package is no longer supported. Use the go.opentelemetry.io/otel/exporters/otlp/otlpmetric exporter as a replacement to send data to a collector which can then export with its PRW exporter.

Index

• Variables
• func InstallNewPipeline(config Config, options ...controller.Option) (*controller.Controller, error)
• func NewExportPipeline(config Config, options ...controller.Option) (*controller.Controller, error)
• func SemVersion() string
• func Version() string
• type Config
  • func (c *Config) Validate() error
• type Exporter
  • func NewRawExporter(config Config) (*Exporter, error)
  • func (e *Exporter) ConvertToTimeSeries(res *resource.Resource, checkpointSet export.InstrumentationLibraryReader) ([]prompb.TimeSeries, error)
  • func (e *Exporter) Export(_ context.Context, res *resource.Resource, ...) error
  • func (e *Exporter) TemporalityFor(*sdkapi.Descriptor, aggregation.Kind) aggregation.Temporality

Variables

var (
	// ErrTwoPasswords occurs when the YAML file contains both `password` and
	// `password_file`.
	ErrTwoPasswords = fmt.Errorf("cannot have two passwords in the YAML file")

	// ErrTwoBearerTokens occurs when the YAML file contains both `bearer_token` and
	// `bearer_token_file`.
	ErrTwoBearerTokens = fmt.Errorf("cannot have two bearer tokens in the YAML file")

	// ErrConflictingAuthorization occurs when the YAML file contains both BasicAuth and
	// bearer token authorization
	ErrConflictingAuthorization = fmt.Errorf("cannot have both basic auth and bearer token authorization")

	// ErrNoBasicAuthUsername occurs when no username was provided for basic
	// authentication.
	ErrNoBasicAuthUsername = fmt.Errorf("no username provided for basic authentication")

	// ErrNoBasicAuthPassword occurs when no password or password file was provided for
	// basic authentication.
	ErrNoBasicAuthPassword = fmt.Errorf("no password or password file provided for basic authentication")

	// ErrInvalidQuantiles occurs when the supplied quantiles are not between 0 and 1.
	ErrInvalidQuantiles = fmt.Errorf("cannot have quantiles that are less than 0 or greater than 1")
)

var ErrFailedToReadFile = fmt.Errorf("failed to read password / bearer token file")

ErrFailedToReadFile occurs when a password / bearer token file exists, but could not be read.

Functions

func InstallNewPipeline

func InstallNewPipeline(config Config, options ...controller.Option) (*controller.Controller, error)

InstallNewPipeline registers a push Controller's MeterProvider globally.

func NewExportPipeline

func NewExportPipeline(config Config, options ...controller.Option) (*controller.Controller, error)

NewExportPipeline sets up a complete export pipeline with a push Controller and Exporter.

func SemVersion

func SemVersion() string

SemVersion is the semantic version to be supplied to tracer/meter creation.

func Version

func Version() string

Version is the current release version of the Cortex exporter.

Types

type Config

type Config struct {
	Endpoint            string            `mapstructure:"url"`
	RemoteTimeout       time.Duration     `mapstructure:"remote_timeout"`
	Name                string            `mapstructure:"name"`
	BasicAuth           map[string]string `mapstructure:"basic_auth"`
	BearerToken         string            `mapstructure:"bearer_token"`
	BearerTokenFile     string            `mapstructure:"bearer_token_file"`
	TLSConfig           map[string]string `mapstructure:"tls_config"`
	ProxyURL            *url.URL          `mapstructure:"proxy_url"`
	PushInterval        time.Duration     `mapstructure:"push_interval"`
	Quantiles           []float64         `mapstructure:"quantiles"`
	HistogramBoundaries []float64         `mapstructure:"histogram_boundaries"`
	Headers             map[string]string `mapstructure:"headers"`
	Client              *http.Client
}

Config contains properties the Exporter uses to export metrics data to Cortex.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks a Config struct for missing required properties and property conflicts. Additionally, it adds default values to missing properties when there is a default.

type Exporter

type Exporter struct {
	// contains filtered or unexported fields
}

Exporter forwards metrics to a Cortex instance

func NewRawExporter

func NewRawExporter(config Config) (*Exporter, error)

NewRawExporter validates the Config struct and creates an Exporter with it.

func (*Exporter) ConvertToTimeSeries

func (e *Exporter) ConvertToTimeSeries(res *resource.Resource, checkpointSet export.InstrumentationLibraryReader) ([]prompb.TimeSeries, error)

ConvertToTimeSeries converts a InstrumentationLibraryReader to a slice of TimeSeries pointers Based on the aggregation type, ConvertToTimeSeries will call helper functions like convertFromSum to generate the correct number of TimeSeries.

func (*Exporter) Export

func (e *Exporter) Export(_ context.Context, res *resource.Resource, checkpointSet export.InstrumentationLibraryReader) error

Export forwards metrics to Cortex from the SDK

func (*Exporter) TemporalityFor

func (e *Exporter) TemporalityFor(*sdkapi.Descriptor, aggregation.Kind) aggregation.Temporality

TemporalityFor returns CumulativeExporter so the Processor correctly aggregates data