confmap

README

High Level Design

This document is work in progress.

Conf

The Conf represents the raw configuration for a service (e.g. OpenTelemetry Collector).

Provider

The Provider provides configuration, and allows to watch/monitor for changes. Any Provider has a <scheme> associated with it, and will provide configs for configURI that follow the ":<opaque_data>" format. This format is compatible with the URI definition (see RFC 3986). The <scheme> MUST be always included in the configURI. The scheme for any Provider MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified in file URI syntax.

Converter

The Converter allows implementing conversion logic for the provided configuration. One of the most common use-case is to migrate/transform the configuration after a backwards incompatible change.

Resolver

The Resolver handles the use of multiple Providers and Converters simplifying configuration parsing, monitoring for updates, and the overall life-cycle of the used config providers. The Resolver provides two main functionalities: Configuration Resolving and Watching for Updates.

Configuration Resolving

The Resolver receives as input a set of Providers, a list of Converters, and a list of configuration identifier configURI that will be used to generate the resulting, or effective, configuration in the form of a Conf, that can be used by code that is oblivious to the usage of Providers and Converters.

Providers are used to provide an entire configuration when the configURI is given directly to the Resolver, or an individual value (partial configuration) when the configURI is embedded into the Conf as a values using the syntax ${configURI}.

              Resolver                   Provider
   Resolve       │                          │
────────────────►│                          │
                 │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │          Conf            │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Merge                 │
              │  │◄──┘                      │
              └─ │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │    Partial Conf Value    │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  embedded    │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Replace               │
              │  │◄──┘                      │
              └─ │                          │
                 │            Converter     │
              ┌─ │     Convert    │         │
              │  ├───────────────►│         │
    foreach   │  │                │         │
   Converter  │  │◄───────────────┤         │
              └─ │                          │
                 │                          │
◄────────────────┤                          │

The Resolve method proceeds in the following steps:

1. Start with an empty "result" of Conf type.
2. For each config URI retrieves individual configurations, and merges it into the "result".
3. For each embedded config URI retrieves individual value, and replaces it into the "result".
4. For each "Converter", call "Convert" for the "result".
5. Return the "result", aka effective, configuration.

Watching for Updates

After the configuration was processed, the Resolver can be used as a single point to watch for updates in the configuration retrieved via the Provider used to retrieve the “initial” configuration and to generate the “effective” one.

         Resolver              Provider
            │                     │
   Watch    │                     │
───────────►│                     │
            │                     │
            .                     .
            .                     .
            .                     .
            │      onChange       │
            │◄────────────────────┤
◄───────────┤                     │

The Resolver does that by passing an onChange func to each Provider.Retrieve call and capturing all watch events.

Documentation

Index

• Constants
• type ChangeEvent
• type CloseFunc
• type Conf
  • func New() *Conf
  • func NewFromStringMap(data map[string]interface{}) *Conf
  • func (l *Conf) AllKeys() []string
  • func (l *Conf) Get(key string) interface{}
  • func (l *Conf) IsSet(key string) bool
  • func (l *Conf) Merge(in *Conf) error
  • func (l *Conf) Sub(key string) (*Conf, error)
  • func (l *Conf) ToStringMap() map[string]interface{}
  • func (l *Conf) Unmarshal(result interface{}) error
  • func (l *Conf) UnmarshalExact(result interface{}) error
• type Converter
• type Provider
• type Resolver
  • func NewResolver(set ResolverSettings) (*Resolver, error)
  • func (mr *Resolver) Resolve(ctx context.Context) (*Conf, error)
  • func (mr *Resolver) Shutdown(ctx context.Context) error
  • func (mr *Resolver) Watch() <-chan error
• type ResolverSettings
• type Retrieved
  • func NewRetrieved(rawConf interface{}, opts ...RetrievedOption) (*Retrieved, error)
  • func (r *Retrieved) AsConf() (*Conf, error)
  • func (r *Retrieved) AsRaw() (interface{}, error)
  • func (r *Retrieved) Close(ctx context.Context) error
• type RetrievedOption
  • func WithRetrievedClose(closeFunc CloseFunc) RetrievedOption
• type Unmarshaler
• type WatcherFunc

Constants

const (
	// KeyDelimiter is used as the default key delimiter in the default koanf instance.
	KeyDelimiter = "::"
)

Types

type ChangeEvent

type ChangeEvent struct {
	// Error is nil if the config is changed and needs to be re-fetched.
	// Any non-nil error indicates that there was a problem with watching the config changes.
	Error error
}

ChangeEvent describes the particular change event that happened with the config. TODO: see if this can be eliminated.

type CloseFunc

type CloseFunc func(context.Context) error

CloseFunc a function equivalent to Retrieved.Close.

type Conf

type Conf struct {
	// contains filtered or unexported fields
}

Conf represents the raw configuration map for the OpenTelemetry Collector. The confmap.Conf can be unmarshalled into the Collector's config using the "service" package.

func New

func New() *Conf

New creates a new empty confmap.Conf instance.

func NewFromStringMap

func NewFromStringMap(data map[string]interface{}) *Conf

NewFromStringMap creates a confmap.Conf from a map[string]interface{}.

func (*Conf) AllKeys

func (l *Conf) AllKeys() []string

AllKeys returns all keys holding a value, regardless of where they are set. Nested keys are returned with a KeyDelimiter separator.

func (*Conf) Get

func (l *Conf) Get(key string) interface{}

Get can retrieve any value given the key to use.

func (*Conf) IsSet

func (l *Conf) IsSet(key string) bool

IsSet checks to see if the key has been set in any of the data locations. IsSet is case-insensitive for a key.

func (*Conf) Merge

func (l *Conf) Merge(in *Conf) error

Merge merges the input given configuration into the existing config. Note that the given map may be modified.

func (*Conf) Sub

func (l *Conf) Sub(key string) (*Conf, error)

Sub returns new Conf instance representing a sub-config of this instance. It returns an error is the sub-config is not a map[string]interface{} (use Get()), and an empty Map if none exists.

func (*Conf) ToStringMap

func (l *Conf) ToStringMap() map[string]interface{}

ToStringMap creates a map[string]interface{} from a Parser.

func (*Conf) Unmarshal

func (l *Conf) Unmarshal(result interface{}) error

Unmarshal unmarshalls the config into a struct. Tags on the fields of the structure must be properly set.

func (*Conf) UnmarshalExact

func (l *Conf) UnmarshalExact(result interface{}) error

UnmarshalExact unmarshalls the config into a struct, erroring if a field is nonexistent.

type Converter

type Converter interface {
	// Convert applies the conversion logic to the given "conf".
	Convert(ctx context.Context, conf *Conf) error
}

Converter is a converter interface for the confmap.Conf that allows distributions (in the future components as well) to build backwards compatible config converters.

type Provider

type Provider interface {
	// Retrieve goes to the configuration source and retrieves the selected data which
	// contains the value to be injected in the configuration and the corresponding watcher that
	// will be used to monitor for updates of the retrieved value.
	//
	// `uri` must follow the "<scheme>:<opaque_data>" format. This format is compatible
	// with the URI definition (see https://datatracker.ietf.org/doc/html/rfc3986). The "<scheme>"
	// must be always included in the `uri`. The "<scheme>" supported by any provider:
	//   - MUST consist of a sequence of characters beginning with a letter and followed by any
	//     combination of letters, digits, plus ("+"), period ("."), or hyphen ("-").
	//     See https://datatracker.ietf.org/doc/html/rfc3986#section-3.1.
	//   - MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified
	//     in https://tools.ietf.org/id/draft-kerwin-file-scheme-07.html#syntax.
	//   - For testing, all implementation MUST check that confmaptest.ValidateProviderScheme returns no error.
	//
	// `watcher` callback is called when the config changes. watcher may be called from
	// a different go routine. After watcher is called Retrieved.Get should be called
	// to get the new config. See description of Retrieved for more details.
	// watcher may be nil, which indicates that the caller is not interested in
	// knowing about the changes.
	//
	// If ctx is cancelled should return immediately with an error.
	// Should never be called concurrently with itself or with Shutdown.
	Retrieve(ctx context.Context, uri string, watcher WatcherFunc) (*Retrieved, error)

	// Scheme returns the location scheme used by Retrieve.
	Scheme() string

	// Shutdown signals that the configuration for which this Provider was used to
	// retrieve values is no longer in use and the Provider should close and release
	// any resources that it may have created.
	//
	// This method must be called when the Collector service ends, either in case of
	// success or error. Retrieve cannot be called after Shutdown.
	//
	// Should never be called concurrently with itself or with Retrieve.
	// If ctx is cancelled should return immediately with an error.
	Shutdown(ctx context.Context) error
}

Provider is an interface that helps to retrieve a config map and watch for any changes to the config map. Implementations may load the config from a file, a database or any other source.

The typical usage is the following:

r, err := provider.Retrieve("file:/path/to/config")
// Use r.Map; wait for watcher to be called.
r.Close()
r, err = provider.Retrieve("file:/path/to/config")
// Use r.Map; wait for watcher to be called.
r.Close()
// repeat retrieve/wait/close cycle until it is time to shut down the Collector process.
// ...
provider.Shutdown()

type Resolver

type Resolver struct {
	sync.Mutex
	// contains filtered or unexported fields
}

Resolver resolves a configuration as a Conf.

func NewResolver

func NewResolver(set ResolverSettings) (*Resolver, error)

NewResolver returns a new Resolver that resolves configuration from multiple URIs.

To resolve a configuration the following steps will happen:

1. Retrieves individual configurations from all given "URIs", and merge them in the retrieve order.
2. Once the Conf is merged, apply the converters in the given order.

After the configuration was resolved the `Resolver` can be used as a single point to watch for updates in the configuration data retrieved via the config providers used to process the "initial" configuration and to generate the "effective" one. The typical usage is the following:

Resolver.Resolve(ctx)
Resolver.Watch() // wait for an event.
Resolver.Resolve(ctx)
Resolver.Watch() // wait for an event.
// repeat Resolve/Watch cycle until it is time to shut down the Collector process.
Resolver.Shutdown(ctx)

`uri` must follow the "<scheme>:<opaque_data>" format. This format is compatible with the URI definition (see https://datatracker.ietf.org/doc/html/rfc3986). An empty "<scheme>" defaults to "file" schema.

func (*Resolver) Resolve

func (mr *Resolver) Resolve(ctx context.Context) (*Conf, error)

Resolve returns the configuration as a Conf, or error otherwise.

Should never be called concurrently with itself, Watch or Shutdown.

func (*Resolver) Shutdown

func (mr *Resolver) Shutdown(ctx context.Context) error

Shutdown signals that the provider is no longer in use and the that should close and release any resources that it may have created. It terminates the Watch channel.

Should never be called concurrently with itself or Get.

func (*Resolver) Watch

func (mr *Resolver) Watch() <-chan error

Watch blocks until any configuration change was detected or an unrecoverable error happened during monitoring the configuration changes.

Error is nil if the configuration is changed and needs to be re-fetched. Any non-nil error indicates that there was a problem with watching the configuration changes.

Should never be called concurrently with itself or Get.

type ResolverSettings

type ResolverSettings struct {
	// URIs locations from where the Conf is retrieved, and merged in the given order.
	// It is required to have at least one location.
	URIs []string

	// Providers is a map of pairs <scheme, Provider>.
	// It is required to have at least one Provider.
	Providers map[string]Provider

	// MapConverters is a slice of Converter.
	Converters []Converter
}

ResolverSettings are the settings to configure the behavior of the Resolver.

type Retrieved

type Retrieved struct {
	// contains filtered or unexported fields
}

Retrieved holds the result of a call to the Retrieve method of a Provider object.

func NewRetrieved

func NewRetrieved(rawConf interface{}, opts ...RetrievedOption) (*Retrieved, error)

NewRetrieved returns a new Retrieved instance that contains the data from the raw deserialized config. The rawConf can be one of the following types:

• Primitives: int, int32, int64, float32, float64, bool, string;
• []interface{};
• map[string]interface{};

func (*Retrieved) AsConf

func (r *Retrieved) AsConf() (*Conf, error)

AsConf returns the retrieved configuration parsed as a Conf.

func (*Retrieved) AsRaw

func (r *Retrieved) AsRaw() (interface{}, error)

AsRaw returns the retrieved configuration parsed as an interface{} which can be one of the following types:

• Primitives: int, int32, int64, float32, float64, bool, string;
• []interface{} - every member follows the same rules as the given interface{};
• map[string]interface{} - every value follows the same rules as the given interface{};

func (*Retrieved) Close

func (r *Retrieved) Close(ctx context.Context) error

Close and release any watchers that Provider.Retrieve may have created.

Should block until all resources are closed, and guarantee that `onChange` is not going to be called after it returns except when `ctx` is cancelled.

Should never be called concurrently with itself.

type RetrievedOption

type RetrievedOption func(*retrievedSettings)

RetrievedOption options to customize Retrieved values.

func WithRetrievedClose

func WithRetrievedClose(closeFunc CloseFunc) RetrievedOption

WithRetrievedClose overrides the default Retrieved.Close function. The default Retrieved.Close function does nothing and always returns nil.

type Unmarshaler

type Unmarshaler interface {
	// Unmarshal a Conf into the struct in a custom way.
	// The Conf for this specific component may be nil or empty if no config available.
	Unmarshal(component *Conf) error
}

Unmarshaler interface may be implemented by types to customize their behavior when being unmarshaled from a Conf.

type WatcherFunc

type WatcherFunc func(*ChangeEvent)