README
¶
Config Source Manager
The configsource.Manager handles the use of multiple config source objects
simplifying configuration parsing,
monitoring for updates, and the overall life-cycle of the used config sources.
Usage Overview
The public API of the configsource.Manager is described in godoc comments at the
manager.go file.
Below is an overview of its usage, which consists of two separate phases:
- Configuration Processing
- Watching for Updates
Configuration Processing
The configsource.Manager receives as input a set of config source factories and a local config.Map that
will be used to generate the resulting, or effective, configuration also in the form of a config.Map,
that can be used by code that is oblivious to the usage of config sources.
+-----------------------------------------------------+
| config.Map |
|-----------------------------------------------------|
| |
| logical YAML config: |
| +-------------------------------------------------+ |
| |config_sources: | |
| | include: | | +---------------------------------+
| | # `include` is an example of a config | | +---------------------------------+|
| | # source that can read from files. | | +---------------------------------+||
| | env: | | +---------------------------------+|||
| | # `env` is another possible config | | +---------------------------------+||||
| | # source that insert YAML from env vars. | | | |||||
| | | | | |||||
| |# Below the standard YAML configuration but | | | |||||
| |# some data still to be retrieved from the | | | |||||
| |# config sources. | | | |||||
| | | | | configsource Factory |||||
| |receivers: ${include:/cfgs/rcvrs/def.yaml} | | | |||||
| | | | | |||||
| |exporters ${env:EXPORTERS_DEFINITION} | | | |||||
| | | | | |||||
| |service: | | | |||||
| | pipelines: | | | |||||
| | trace: | | | ||||+
| | receivers: ${include:/cfgs/rcvrs/use.yaml} | | | |||+
| | exporters: ${env:EXPORTERS_IN_USE} | | | ||+
| +-------------------------------------------------+ | | |+
+-----------------------------------------------------+ +---------------------------------+
| |
| |
+----------------------------------------------------------------+
|
|
v
+-------------------------------+
| |
| configsource.Manager |
| |
+-------------------------------+
|
|
"Resolve" |
|
|
|
|
|
v
+-----------------------------------------------------+
| config.Map |
|-----------------------------------------------------|
| |
| logica YAML config: |
| +-------------------------------------------------+ |
| |receivers: | |
| | zipkin: | |
| | jaeger: | |
| | otlp: | |
| |exporters: | |
| | zipkin: | |
| | jaeger: | |
| | otlp: | |
| |service: | |
| | pipelines: | |
| | trace: | |
| | receivers: [zipkin, jaeger, otlp] | |
| | exporters: [otlp] | |
| +-------------------------------------------------+ |
+-----------------------------------------------------+
The Resolve method proceeds in the following steps:
- Create the
configsource.ConfigSourceobjects defined theconfig_sourcessection of the initial configuration; - For each config node (key) of the initial configuration:
- Skip config node if it is under the
config_sourcessection (this excludesconfig_sourcesfrom the resulting config); - Parse the node value transforming any config source invocation, or environment variable, into the retrieved data;
- Add the key and the value retrieved above into the resulting configuration;
- Skip config node if it is under the
- Return the resulting, aka effective, configuration.
Processing Config Source Invocations
For each config source invocation, e.g. ${include:/cfgs/rcvrs/use.yaml}, the code proceeds as in the following steps:
- Get the corresponding
configsource.ConfigSourceobject by its name, given in the initial configuration under theconfig_sourcessection; - Calls the
Retrievemethod, of the selectedconfigsource.ConfigSource, according to the selector and parameters given in the invocation;
+---------+ +--------------+ +-----------+
| Manager | | ConfigSource | | Retrieved |
+---------+ +--------------+ +-----------+
| | |
| | |
| | Retrieve |
|-------------------------------------------->+--+
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
|<--------------------------------------------+--+
| "Retrieved object" | |
| | |
Watching for Updates
After the configuration was processed the configsource.Manager can be used as a single point to watch for updates in
the configuration data retrieved via the config sources used to process the “initial” configuration and to generate
the“effective” one.
The configsource.Manager does that by wrapping calls to the WatchForUpdate() method of each configsource.Retreived
instance that also implements the configsource.Watchable interface.
Documentation
¶
Overview ¶
Package configsource is an internal package that implements methods for injecting, watching, and updating data from ConfigSource into configuration.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Manager ¶ added in v0.25.0
type Manager struct {
// contains filtered or unexported fields
}
Manager is used to inject data from config sources into a configuration and also to monitor for updates on the items injected into the configuration. All methods of a Manager must be called only once and have an expected sequence:
1. NewManager to create a new instance; 2. Resolve to inject the data from config sources into a configuration; 3. WatchForUpdate in a goroutine to wait for configuration updates; 4. WaitForWatcher to wait until the watchers are in place; 5. Close to close the instance;
The current syntax to reference a config source in a YAML is provisional. Currently single-line:
param_to_be_retrieved: $<cfgSrcName>:<selector>[?<params_url_query_format>]
bracketed single-line:
param_to_be_retrieved: ${<cfgSrcName>:<selector>[?<params_url_query_format>]}
and multi-line are supported:
param_to_be_retrieved: | $<cfgSrcName>: <selector> [<params_multi_line_YAML>]
The <cfgSrcName> is a name string used to identify the config source instance to be used to retrieve the value.
The <selector> is the mandatory parameter required when retrieving data from a config source.
Not all config sources need the optional parameters, they are used to provide extra control when retrieving and preparing the data to be injected into the configuration.
For single-line format <params_url_query_format> uses the same syntax as URL query parameters. Hypothetical example in a YAML file:
component:
config_field: $file:/etc/secret.bin?binary=true
For multi-line format <params_multi_line_YAML> uses syntax as a YAML inside YAML. Possible usage example in a YAML file:
component:
config_field: | $yamltemplate: /etc/log_template.yaml logs_path: /var/logs/ timeout: 10s
Not all config sources need these optional parameters, they are used to provide extra control when retrieving and data to be injected into the configuration.
Assuming a config source named "env" that retrieve environment variables and one named "file" that retrieves contents from individual files, here are some examples:
component: # Retrieves the value of the environment variable LOGS_DIR. logs_dir: $env:LOGS_DIR # Retrieves the value from the file /etc/secret.bin and injects its contents as a []byte. bytes_from_file: $file:/etc/secret.bin?binary=true # Retrieves the value from the file /etc/text.txt and injects its contents as a string. # Hypothetically the "file" config source by default tries to inject the file contents # as a string if params doesn't specify that "binary" is true. text_from_file: $file:/etc/text.txt
Bracketed single-line should be used when concatenating a suffix to the value retrieved by the config source. Example:
component:
# Retrieves the value of the environment variable LOGS_DIR and appends /component.log to it.
log_file_fullname: ${env:LOGS_DIR}/component.log
Environment variables are expanded before passed to the config source when used in the selector or the optional parameters. Example:
component: # Retrieves the value from the file text.txt located on the path specified by the environment # variable DATA_PATH. The name of the environment variable is the string after the delimiter # until the first character different than '_' and non-alpha-numeric. text_from_file: $file:$DATA_PATH/text.txt
Since environment variables and config sources both use the '$', with or without brackets, as a prefix for their expansion it is necessary to have a way to distinguish between them. For the non-bracketed syntax the code will peek at the first character other than alpha-numeric and '_' after the '$'. If that character is a ':' it will treat it as a config source and as environment variable otherwise. For example:
component: field_0: $PATH:/etc/logs # Injects the data from a config sourced named "PATH" using the selector "/etc/logs". field_1: $PATH/etc/logs # Expands the environment variable "PATH" and adds the suffix "/etc/logs" to it.
So if you need to include an environment followed by ':' the bracketed syntax must be used instead:
component:
field_0: ${PATH}:/etc/logs # Expands the environment variable "PATH" and adds the suffix ":/etc/logs" to it.
For the bracketed syntax the presence of ':' inside the brackets indicates that code will treat the bracketed contents as a config source. For example:
component:
field_0: ${file:/var/secret.txt} # Injects the data from a config sourced named "file" using the selector "/var/secret.txt".
field_1: ${file}:/var/secret.txt # Expands the environment variable "file" and adds the suffix ":/var/secret.txt" to it.
If the character following the '$' is in the set {'*', '#', '$', '@', '!', '?', '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} the code will consider it to be the name of an environment variable to expand, or config source if followed by ':'. Do not use any of these characters as the first char on the name of a config source or an environment variable (even if allowed by the system) to avoid unexpected results.
For an overview about the internals of the Manager refer to the package README.md.
func NewManager ¶ added in v0.25.0
NewManager creates a new instance of a Manager to be used to inject data from ConfigSource objects into a configuration and watch for updates on the injected data.
func (*Manager) Close ¶ added in v0.25.0
Close terminates the WatchForUpdate function and closes all Session objects used in the configuration. It should be called
func (*Manager) Resolve ¶ added in v0.25.0
Resolve inspects the given confmap.Conf and resolves all config sources referenced in the configuration, returning a confmap.Conf in which all env vars and config sources on the given input config map are resolved to actual literal values of the env vars or config sources. This method must be called only once per lifetime of a Manager object.
func (*Manager) WaitForWatcher ¶ added in v0.25.0
func (m *Manager) WaitForWatcher()
WaitForWatcher blocks until the watchers used by WatchForUpdate are all ready. This is used to ensure that the watchers are in place before proceeding.
func (*Manager) WatchForUpdate ¶ added in v0.25.0
WatchForUpdate must watch for updates on any of the values retrieved from config sources and injected into the configuration. Typically this method is launched in a goroutine, the method WaitForWatcher blocks until the WatchForUpdate goroutine is running and ready.
Source Files