discovery

package
v0.113.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 21, 2024 License: Apache-2.0 Imports: 44 Imported by: 0

README

Discovery confmap.Provider

This feature currently has an alpha stability level.
Backwards incompatible changes to components and custom discovery configuration may occur.

The Discovery confmap.Provider provides the ability to define Collector service config through individual component yaml mappings in a config.d directory:

graph LR
  config.d[/config.d/] --> 1>service.yaml]
  subgraph 1a[service.yaml]
    1 --> 1a1[[pipelines:<br>metrics:<br>receivers:<br>- otlp<br>exporters:<br>- debug]]
  end
  config.d --> 2[/exporters/]
  subgraph 2a[exporters]
    2 --> 2a1>otlp.yaml]
    2a1 --> 2b1[[otlp:<br>endpoint: 1.2.3.4:2345]]
    2 --> 2a2>logging.yaml]
    2a2 --> 2b2[[debug:<br>verbosity: detailed]]
  end
  config.d --> 3[/extensions/]
  subgraph 3a[extensions]
    3 --> 3a1>zpages.yaml]
    3a1 --> 3b1[[zpages:<br>endpoint: 0.0.0.0:12345]]
    3 --> 3a2>health-check.yaml]
    3a2 --> 3b2[[health_check:<br>path: /health]]
  end
  config.d --> 4[/processors/]
  subgraph 4a[processors]
    4 --> 4a1>batch.yaml]
    4a1 --> 4b1[[batch:<br>]]
    4 --> 4a2>resource-detection.yaml]
    4a2 --> 4b2[[resourcedetection:<br>detectors:<br>- system]]
  end
  config.d --> 5[/receivers/]
  subgraph 5a[receivers]
    5 --> 5a1>otlp.yaml]
    5a1 --> 5b1[[otlp:<br>protocols:<br>grpc:]]
  end

This component is currently supported in the Collector settings via the following commandline options:

option environment variable default description
--configd none disabled Whether to enable config.d functionality for final Collector config content.
--config-dir SPLUNK_CONFIG_DIR /etc/otel/collector/config.d The root config.d directory to walk for component directories and yaml mapping files.
--dry-run none disabled Whether to report the final assembled config contents to stdout before immediately exiting. This can be used with or without config.d

To source only config.d content and not an additional or default configuration file, the --config option or SPLUNK_CONFIG environment variable must be set to /dev/null or an arbitrary empty file:

$ # run the Collector without a config file using components from a local ./config.d config directory,
$ # printing the config to stdout before exiting instead of starting the Collector service:
$ bin/otelcol --config /dev/null --configd --config-dir ./config.d --dry-run
2023/02/24 19:54:23 settings.go:331: Set config to [/dev/null]
2023/02/24 19:54:23 settings.go:384: Set ballast to 168 MiB
2023/02/24 19:54:23 settings.go:400: Set memory limit to 460 MiB
exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: 1.2.3.4:2345
extensions:
  health_check:
    path: /health
  zpages:
    endpoint: 0.0.0.0:1234
processors:
  batch: {}
  resourcedetection:
    detectors:
    - system
receivers:
  otlp:
    protocols:
      grpc: null
service:
  pipelines:
    metrics:
      exporters:
      - debug
      receivers:
      - otlp

Discovery Mode

This component also provides a --discovery [--dry-run] [--discovery-properties=<properties.yaml>] option compatible with config.d that attempts to instantiate any .discovery.yaml receivers using corresponding .discovery.yaml observers in a "preflight" Collector service. Discovery mode will:

  1. Load and attempt to start any observers in config.d/extensions/<name>.discovery.yaml.
  2. Load and attempt to start any receiver blocks in config.d/receivers/<name>.discovery.yaml in a Discovery Receiver instance to receive discovery events from all successfully started observers.
  3. Wait 10s or the configured SPLUNK_DISCOVERY_DURATION environment variable time.Duration.
  4. Embed any receiver instances' configs resulting in a discovery.status of successful inside a receiver_creator/discovery receiver's configuration to be passed to the final Collector service config in a new or existing service::pipelines::metrics::receivers sequence (or outputted w/ --dry-run). Any required observers will be added to service::extensions.
  5. Log any receiver resulting in a discovery.status of partial with the configured guidance for setting any relevant discovery properties.
  6. Stop all temporary components before continuing on to the actual Collector service (or exiting early with --dry-run).

Unlike config.d component files, which are direct configuration entries for the desired component, Discovery component configs have an enabled boolean and config parent mapping field to determine use and configure the functionality of the components:

# <some-observer-type-with-optional-name.discovery.yaml>
<observer_type>(/<observer_name>):
  enabled: <true | false> # true by default
  config:
    <embedded observer config>

# <some-receiver-type-with-optional-name.discovery.yaml>
<receiver_type>(/<receiver_name>):
  enabled: <true | false> # true by default
  rule:
    <observer_type>(/<observer_name>): <receiver creator rule for this observer>
  config:
    default:
      <default embedded receiver config>
    <observer_type>(/<observer_name>):
      <observer-specific config items, merged with `default`>
  status:
    metrics:
      <discovery receiver metric status entries>
    statements:
      <discovery receiver statement status entries>

By default, the discovery mode is provided with pre-made discovery config components in bundle.d.

The following components have bundled discovery configurations in the last Splunk OpenTelemetry Collector release:

I. Receivers

II. Extensions

Discovery properties

Configuring discovery components is performed by merging discovery properties with the config.d receivers and extensions *.discovery.yaml files. Discovery properties are of the form:

splunk.discovery.receivers.<receiver-type(/name)>.config.<field>(<::subfield>)*: <value>
splunk.discovery.extensions.<observer-type(/name)>.config.<field>(<::subfield>)*: <value>
splunk.discovery.receivers.<receiver-type(/name)>.enabled: <true or false>
splunk.discovery.extensions.<observer-type(/name)>.enabled: <true or false>

# Examples
splunk.discovery.receivers.prometheus_simple.config.labels::my_label: my_label_value
splunk.discovery.receivers.prometheus_simple.enabled: true

splunk.discovery.extensions.docker_observer.config.endpoint: tcp://localhost:8080
splunk.discovery.extensions.k8s_observer.enabled: false

These properties can be in config.d/properties.discovery.yaml or specified at run time with --set command line options.

You can also specify a --discovery-properties=<filepath.yaml> argument to disregard config.d/properties.discovery.yaml properties and load properties not to be shared with another Collector service, while still benefiting from existing discovery component definitions.

The config.d/properties.discovery.yaml file supports specifying the property values directly as well within a mapped form. The property values can also be environment variables:

# --set form will take priority to mapped values
splunk.discovery.receivers.prometheus_simple.config.labels::my_label: my_label_value
splunk.discovery.receivers.prometheus_simple.enabled: true

splunk.discovery.receivers.smartagent/postgresql.config.params::username: '${env:PG_USERNAME}'
splunk.discovery.receivers.smartagent/postgresql.config.params::password: '${env:PG_PASSWORD}'

# mapped property form
splunk.discovery:
  extensions:
    docker_observer:
      enabled: false
      config:
        endpoint: tcp://localhost:54321
  receivers:
    prometheus_simple:
      enabled: false # will be overwritten by above --set form (discovery is attempted for the receiver)

Each discovery property also has an equivalent environment variable form using _x<hex pair>_ encoded delimiters for non-word characters [^a-zA-Z0-9_]:

SPLUNK_DISCOVERY_RECEIVERS_receiver_x2d_type_x2f_receiver_x2d_name_CONFIG_field_x3a__x3a_subfield=value
SPLUNK_DISCOVERY_EXTENSIONS_observer_x2d_type_x2f_observer_x2d_name_CONFIG_field_x3a__x3a_subfield=value
SPLUNK_DISCOVERY_RECEIVERS_receiver_x2d_type_x2f_receiver_x2d_name_ENABLED=<true or false>
SPLUNK_DISCOVERY_EXTENSIONS_observer_x2d_type_x2f_observer_x2d_name_ENABLED=<true or false>

# Examples
SPLUNK_DISCOVERY_RECEIVERS_prometheus_simple_CONFIG_labels_x3a__x3a_my_label="my_username"
SPLUNK_DISCOVERY_RECEIVERS_prometheus_simple_ENABLED=true

SPLUNK_DISCOVERY_EXTENSIONS_docker_observer_CONFIG_endpoint="tcp://localhost:8080"
SPLUNK_DISCOVERY_EXTENSIONS_k8s_observer_ENABLED=false

The priority order for discovery config values from lowest to highest is:

  1. Pre-made bundle.d component config content (lowest).
  2. config.d/<receivers or extensions>/*.discovery.yaml component config file content.
  3. config.d/properties.discovery.yaml properties file mapped form content.
  4. config.d/properties.discovery.yaml properties file --set form content.
  5. SPLUNK_DISCOVERY_<xyz> property environment variables available to the collector process.
  6. --set splunk.discovery.<xyz> property commandline options (highest).

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {

	// Service is for pipelines and final settings.
	// It must be in the root config directory and named "service.yaml"
	Service ServiceEntry
	// Exporters is a map of exporters to use in final config.
	// They must be in `config.d/exporters` directory.
	Exporters map[component.ID]ExporterEntry
	// Extensions is a map of extensions to use in final config.
	// They must be in `config.d/extensions` directory.
	Extensions map[component.ID]ExtensionEntry
	// DiscoveryObservers is a map of observer extensions to use in discovery.
	// They must be in `config.d/extensions` directory and end with ".discovery.yaml".
	DiscoveryObservers map[component.ID]ObserverEntry
	// Processors is a map of extensions to use in final config.
	// They must be in `config.d/processors` directory.
	Processors map[component.ID]ProcessorEntry
	// Receivers is a map of receiver entries to use in final config
	// They must be in `config.d/receivers` directory.
	Receivers map[component.ID]ReceiverEntry
	// ReceiversToDiscover is a map of receiver entries to use in discovery mode's
	// underlying discovery receiver. They must be in `config.d/receivers` directory and
	// end with ".discovery.yaml".
	ReceiversToDiscover map[component.ID]ReceiverToDiscoverEntry
	// DiscoveryProperties is a mapping of discovery properties to their values for
	// configuring discovery mode components.
	// It must be in the root config directory and named "properties.discovery.yaml".
	DiscoveryProperties PropertiesEntry
	// contains filtered or unexported fields
}

Config is a model for stitching together the final Collector configuration with additional discovery component fields for use w/ discovery mode. It allows individual yaml files to be added to a config.d directory and be sourced in the final config such that small changes don't apply to a central configuration file, and possibly eliminates the need for one overall (still in design and dependent on aliasing and array insertion operators).

func NewConfig 

func NewConfig(logger *zap.Logger) *Config

func (*Config) Load 

func (c *Config) Load(configDPath string) error

Load will walk the file tree from the configDPath root, loading the component files as they are discovered, determined by their parent directory and filename.

func (*Config) LoadFS added in v0.72.0 

func (c *Config) LoadFS(dirfs fs.FS) error

LoadFS will walk the provided filesystem, loading the component files as they are discovered, determined by their parent directory and filename.

func (*Config) LoadProperties added in v0.81.0 

func (c *Config) LoadProperties(path string) error

type Entry 

type Entry map[string]any

func (Entry) Self added in v0.63.0 

func (e Entry) Self() Entry

func (Entry) ToStringMap 

func (e Entry) ToStringMap() map[string]any

type ExporterEntry 

type ExporterEntry struct {
	Entry `yaml:",inline"`
}

func (ExporterEntry) ErrorF 

func (ExporterEntry) ErrorF(path string, err error) error

type ExtensionEntry 

type ExtensionEntry struct {
	Entry `yaml:",inline"`
}

func (ExtensionEntry) ErrorF 

func (ExtensionEntry) ErrorF(path string, err error) error

type ObserverEntry 

type ObserverEntry struct {
	Enabled *bool
	Config  Entry
	Entry   `yaml:",inline"`
}

func (ObserverEntry) ErrorF 

func (ObserverEntry) ErrorF(path string, err error) error

type ProcessorEntry 

type ProcessorEntry struct {
	Entry `yaml:",inline"`
}

func (ProcessorEntry) ErrorF 

func (ProcessorEntry) ErrorF(path string, err error) error

type PropertiesEntry added in v0.74.0 

type PropertiesEntry struct {
	Entry `yaml:",inline"`
}

func (PropertiesEntry) ErrorF added in v0.74.0 

func (PropertiesEntry) ErrorF(path string, err error) error

type Provider added in v0.63.0 

type Provider struct {
	// contains filtered or unexported fields
}

func New added in v0.63.0 

func New() (*Provider, error)

func (*Provider) ConfigDProviderFactory added in v0.101.0 

func (m *Provider) ConfigDProviderFactory() confmap.ProviderFactory

func (*Provider) ConfigDScheme added in v0.63.0 

func (m *Provider) ConfigDScheme() string

func (*Provider) DiscoveryModeProviderFactory added in v0.101.0 

func (m *Provider) DiscoveryModeProviderFactory() confmap.ProviderFactory

func (*Provider) DiscoveryModeScheme added in v0.63.0 

func (m *Provider) DiscoveryModeScheme() string

func (*Provider) PropertiesFileProviderFactory added in v0.101.0 

func (m *Provider) PropertiesFileProviderFactory() confmap.ProviderFactory

func (*Provider) PropertiesFileScheme added in v0.81.0 

func (m *Provider) PropertiesFileScheme() string

func (*Provider) PropertyProviderFactory added in v0.101.0 

func (m *Provider) PropertyProviderFactory() confmap.ProviderFactory

func (*Provider) PropertyScheme added in v0.70.0 

func (m *Provider) PropertyScheme() string

type ReceiverEntry 

type ReceiverEntry struct {
	Entry `yaml:",inline"`
}

func (ReceiverEntry) ErrorF 

func (ReceiverEntry) ErrorF(path string, err error) error

type ReceiverToDiscoverEntry 

type ReceiverToDiscoverEntry struct {
	// Receiver creator rules by observer extension ID
	Rule map[component.ID]string
	// Platform/observer specific config by observer extension ID.
	// These are merged w/ "default" component.ID in a "config" map
	Config map[component.ID]map[string]any
	// Whether to attempt to discover this receiver
	Enabled *bool
	// The remaining items used to merge applicable rule and config
	Entry `yaml:",inline"`
}

func (ReceiverToDiscoverEntry) ErrorF 

func (ReceiverToDiscoverEntry) ErrorF(path string, err error) error

func (ReceiverToDiscoverEntry) ToStringMap 

func (r ReceiverToDiscoverEntry) ToStringMap() map[string]any

type ServiceEntry 

type ServiceEntry struct {
	Entry `yaml:",inline"`
}

func (ServiceEntry) ErrorF 

func (ServiceEntry) ErrorF(path string, err error) error

Source Files

View all Source files

Directories

Path Synopsis
cmd/discoverybundler

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.