confmap

package module
v0.0.0-...-76a0d45 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 27, 2024 License: Apache-2.0 Imports: 16 Imported by: 16

README

High Level Design

This document is work in progress.

Conf

The Conf represents the raw configuration for a service (e.g. OpenTelemetry Collector).

Provider

The Provider provides configuration, and allows to watch/monitor for changes. Any Provider has a <scheme> associated with it, and will provide configs for configURI that follow the ":<opaque_data>" format. This format is compatible with the URI definition (see RFC 3986). The <scheme> MUST be always included in the configURI. The scheme for any Provider MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified in file URI syntax.

Converter

The Converter allows implementing conversion logic for the provided configuration. One of the most common use-case is to migrate/transform the configuration after a backwards incompatible change.

Resolver

The Resolver handles the use of multiple Providers and Converters simplifying configuration parsing, monitoring for updates, and the overall life-cycle of the used config providers. The Resolver provides two main functionalities: Configuration Resolving and Watching for Updates.

Configuration Resolving

The Resolver receives as input a set of Providers, a list of Converters, and a list of configuration identifier configURI that will be used to generate the resulting, or effective, configuration in the form of a Conf, that can be used by code that is oblivious to the usage of Providers and Converters.

Providers are used to provide an entire configuration when the configURI is given directly to the Resolver, or an individual value (partial configuration) when the configURI is embedded into the Conf as a values using the syntax ${configURI}.

Limitation:

  • When embedding a ${configURI} the uri cannot contain dollar sign ("$") character unless it embeds another uri.
  • The number of URIs is limited to 100.
              Resolver                   Provider
   Resolve       │                          │
────────────────►│                          │
                 │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │          Conf            │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Merge                 │
              │  │◄──┘                      │
              └─ │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │    Partial Conf Value    │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  embedded    │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Replace               │
              │  │◄──┘                      │
              └─ │                          │
                 │            Converter     │
              ┌─ │     Convert    │         │
              │  ├───────────────►│         │
    foreach   │  │                │         │
   Converter  │  │◄───────────────┤         │
              └─ │                          │
                 │                          │
◄────────────────┤                          │

The Resolve method proceeds in the following steps:

  1. Start with an empty "result" of Conf type.
  2. For each config URI retrieves individual configurations, and merges it into the "result".
  3. For each embedded config URI retrieves individual value, and replaces it into the "result".
  4. For each "Converter", call "Convert" for the "result".
  5. Return the "result", aka effective, configuration.
Watching for Updates

After the configuration was processed, the Resolver can be used as a single point to watch for updates in the configuration retrieved via the Provider used to retrieve the “initial” configuration and to generate the “effective” one.

         Resolver              Provider
            │                     │
   Watch    │                     │
───────────►│                     │
            │                     │
            .                     .
            .                     .
            .                     .
            │      onChange       │
            │◄────────────────────┤
◄───────────┤                     │

The Resolver does that by passing an onChange func to each Provider.Retrieve call and capturing all watch events.

Documentation

Overview

Example (EmbeddedManualUnmarshaling)

We can unmarshal an embedded struct with a custom `Unmarshal` method. 

package main

import (
	"fmt"
	"slices"

	"github.com/oodle-ai/opentelemetry-collector/confmap"
)

type NetworkScrape struct {
	Enabled  bool     `mapstructure:"enabled"`
	Networks []string `mapstructure:"networks"`
	Wifi     bool     `mapstructure:"wifi"`
}

func (n *NetworkScrape) Unmarshal(c *confmap.Conf) error {
	if err := c.Unmarshal(n, confmap.WithIgnoreUnused()); err != nil {
		return err
	}
	if slices.Contains(n.Networks, "wlan0") {
		n.Wifi = true
	}
	return nil
}

type RouterScrape struct {
	NetworkScrape `mapstructure:",squash"`
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"networks": []string{"eth0", "eth1", "wlan0"},
		"enabled":  true,
	})
	scrapeInfo := &RouterScrape{}
	if err := conf.Unmarshal(scrapeInfo); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nNetworks: %q\nWifi: %v\nEnabled: %v\n", scrapeInfo.Networks, scrapeInfo.Wifi, scrapeInfo.Enabled)
}

Output:

Configuration contains the following:
Networks: ["eth0" "eth1" "wlan0"]
Wifi: true
Enabled: true
Example (EmbeddedUnmarshaling)

We can unmarshal embedded structs with mapstructure field annotations. 

package main

import (
	"fmt"
	"time"

	"github.com/oodle-ai/opentelemetry-collector/confmap"
)

type DiskScrape struct {
	Disk   string        `mapstructure:"disk"`
	Scrape time.Duration `mapstructure:"scrape"`
}

type CPUScrape struct {
	Enabled bool `mapstructure:"enabled"`
}

type ComputerScrape struct {
	DiskScrape `mapstructure:",squash"`
	CPUScrape  `mapstructure:",squash"`
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"disk":    "c",
		"scrape":  "5s",
		"enabled": true,
	})
	scrapeInfo := &ComputerScrape{}
	if err := conf.Unmarshal(scrapeInfo); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nDisk: %q\nScrape: %s\nEnabled: %v\n", scrapeInfo.Disk, scrapeInfo.Scrape, scrapeInfo.Enabled)
}

Output:

Configuration contains the following:
Disk: "c"
Scrape: 5s
Enabled: true
Example (ManualUnmarshaling) 
package main

import (
	"fmt"
	"time"

	"github.com/oodle-ai/opentelemetry-collector/confmap"
)

type ManualScrapeInfo struct {
	Disk   string
	Scrape time.Duration
}

func (m *ManualScrapeInfo) Unmarshal(c *confmap.Conf) error {
	m.Disk = c.Get("disk").(string)
	if c.Get("vinyl") == "33" {
		m.Scrape = 10 * time.Second
	} else {
		m.Scrape = 2 * time.Second
	}
	return nil
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"disk":  "Beatles",
		"vinyl": "33",
	})
	scrapeInfo := &ManualScrapeInfo{}
	if err := conf.Unmarshal(scrapeInfo, confmap.WithIgnoreUnused()); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nDisk: %q\nScrape: %s\n", scrapeInfo.Disk, scrapeInfo.Scrape)
}

Output:

Configuration contains the following:
Disk: "Beatles"
Scrape: 10s
Example (SimpleUnmarshaling)

We can annotate a struct with mapstructure field annotations. 

package main

import (
	"fmt"
	"time"

	"github.com/oodle-ai/opentelemetry-collector/confmap"
)

type DiskScrape struct {
	Disk   string        `mapstructure:"disk"`
	Scrape time.Duration `mapstructure:"scrape"`
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"disk":   "c",
		"scrape": "5s",
	})
	scrapeInfo := &DiskScrape{}
	if err := conf.Unmarshal(scrapeInfo); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nDisk: %q\nScrape: %s\n", scrapeInfo.Disk, scrapeInfo.Scrape)
}

Output:

Configuration contains the following:
Disk: "c"
Scrape: 5s

Index

Examples

Constants

View Source 
const (
	// KeyDelimiter is used as the default key delimiter in the default koanf instance.
	KeyDelimiter = "::"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type ChangeEvent 

type ChangeEvent struct {
	// Error is nil if the config is changed and needs to be re-fetched.
	// Any non-nil error indicates that there was a problem with watching the config changes.
	Error error
}

ChangeEvent describes the particular change event that happened with the config. TODO: see if this can be eliminated.

type CloseFunc 

type CloseFunc func(context.Context) error

CloseFunc a function equivalent to Retrieved.Close.

type Conf 

type Conf struct {
	// contains filtered or unexported fields
}

Conf represents the raw configuration map for the OpenTelemetry Collector. The confmap.Conf can be unmarshalled into the Collector's config using the "service" package.

func New 

func New() *Conf

New creates a new empty confmap.Conf instance.

func NewFromStringMap 

func NewFromStringMap(data map[string]any) *Conf

NewFromStringMap creates a confmap.Conf from a map[string]any.

func (*Conf) AllKeys 

func (l *Conf) AllKeys() []string

AllKeys returns all keys holding a value, regardless of where they are set. Nested keys are returned with a KeyDelimiter separator.

func (*Conf) Get 

func (l *Conf) Get(key string) any

Get can retrieve any value given the key to use.

func (*Conf) IsSet 

func (l *Conf) IsSet(key string) bool

IsSet checks to see if the key has been set in any of the data locations.

func (*Conf) Marshal 

func (l *Conf) Marshal(rawVal any, _ ...MarshalOption) error

Marshal encodes the config and merges it into the Conf.

func (*Conf) Merge 

func (l *Conf) Merge(in *Conf) error

Merge merges the input given configuration into the existing config. Note that the given map may be modified.

func (*Conf) Sub 

func (l *Conf) Sub(key string) (*Conf, error)

Sub returns new Conf instance representing a sub-config of this instance. It returns an error is the sub-config is not a map[string]any (use Get()), and an empty Map if none exists.

func (*Conf) ToStringMap 

func (l *Conf) ToStringMap() map[string]any

ToStringMap creates a map[string]any from a Parser.

func (*Conf) Unmarshal 

func (l *Conf) Unmarshal(result any, opts ...UnmarshalOption) error

Unmarshal unmarshalls the config into a struct using the given options. Tags on the fields of the structure must be properly set.

type Converter 

type Converter interface {
	// Convert applies the conversion logic to the given "conf".
	Convert(ctx context.Context, conf *Conf) error
}

Converter is a converter interface for the confmap.Conf that allows distributions (in the future components as well) to build backwards compatible config converters.

type ConverterFactory 

type ConverterFactory = moduleFactory[Converter, ConverterSettings]

ConverterFactory defines a factory that can be used to instantiate new instances of a Converter.

func NewConverterFactory 

func NewConverterFactory(f CreateConverterFunc) ConverterFactory

NewConverterFactory can be used to create a ConverterFactory.

type ConverterSettings 

type ConverterSettings struct {
	// Logger is a zap.Logger that will be passed to Converters.
	// Converters should be able to rely on the Logger being non-nil;
	// when instantiating a Converter with a ConverterFactory,
	// nil Logger references should be replaced with a no-op Logger.
	Logger *zap.Logger
}

ConverterSettings are the settings to initialize a Converter.

type CreateConverterFunc 

type CreateConverterFunc = createConfmapFunc[Converter, ConverterSettings]

CreateConverterFunc is a function that creates a Converter instance.

type CreateProviderFunc 

type CreateProviderFunc = createConfmapFunc[Provider, ProviderSettings]

CreateProviderFunc is a function that creates a Provider instance.

type MarshalOption 

type MarshalOption interface {
	// contains filtered or unexported methods
}

type Marshaler 

type Marshaler interface {
	// Marshal the config into a Conf in a custom way.
	// The Conf will be empty and can be merged into.
	Marshal(component *Conf) error
}

Marshaler defines an optional interface for custom configuration marshaling. A configuration struct can implement this interface to override the default marshaling.

type Provider 

type Provider interface {
	// Retrieve goes to the configuration source and retrieves the selected data which
	// contains the value to be injected in the configuration and the corresponding watcher that
	// will be used to monitor for updates of the retrieved value.
	//
	// `uri` must follow the "<scheme>:<opaque_data>" format. This format is compatible
	// with the URI definition (see https://datatracker.ietf.org/doc/html/rfc3986). The "<scheme>"
	// must be always included in the `uri`. The "<scheme>" supported by any provider:
	//   - MUST consist of a sequence of characters beginning with a letter and followed by any
	//     combination of letters, digits, plus ("+"), period ("."), or hyphen ("-").
	//     See https://datatracker.ietf.org/doc/html/rfc3986#section-3.1.
	//   - MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified
	//     in https://tools.ietf.org/id/draft-kerwin-file-scheme-07.html#syntax.
	//   - For testing, all implementation MUST check that confmaptest.ValidateProviderScheme returns no error.
	//
	// `watcher` callback is called when the config changes. watcher may be called from
	// a different go routine. After watcher is called Retrieved.Get should be called
	// to get the new config. See description of Retrieved for more details.
	// watcher may be nil, which indicates that the caller is not interested in
	// knowing about the changes.
	//
	// If ctx is cancelled should return immediately with an error.
	// Should never be called concurrently with itself or with Shutdown.
	Retrieve(ctx context.Context, uri string, watcher WatcherFunc) (*Retrieved, error)

	// Scheme returns the location scheme used by Retrieve.
	Scheme() string

	// Shutdown signals that the configuration for which this Provider was used to
	// retrieve values is no longer in use and the Provider should close and release
	// any resources that it may have created.
	//
	// This method must be called when the Collector service ends, either in case of
	// success or error. Retrieve cannot be called after Shutdown.
	//
	// Should never be called concurrently with itself or with Retrieve.
	// If ctx is cancelled should return immediately with an error.
	Shutdown(ctx context.Context) error
}

Provider is an interface that helps to retrieve a config map and watch for any changes to the config map. Implementations may load the config from a file, a database or any other source.

The typical usage is the following: 

r, err := provider.Retrieve("file:/path/to/config")
// Use r.Map; wait for watcher to be called.
r.Close()
r, err = provider.Retrieve("file:/path/to/config")
// Use r.Map; wait for watcher to be called.
r.Close()
// repeat retrieve/wait/close cycle until it is time to shut down the Collector process.
// ...
provider.Shutdown()

type ProviderFactory 

type ProviderFactory = moduleFactory[Provider, ProviderSettings]

ProviderFactory defines a factory that can be used to instantiate new instances of a Provider.

func NewProviderFactory 

func NewProviderFactory(f CreateProviderFunc) ProviderFactory

NewProviderFactory can be used to create a ProviderFactory.

type ProviderSettings 

type ProviderSettings struct {
	// Logger is a zap.Logger that will be passed to Providers.
	// Providers should be able to rely on the Logger being non-nil;
	// when instantiating a Provider with a ProviderFactory,
	// nil Logger references should be replaced with a no-op Logger.
	Logger *zap.Logger
}

ProviderSettings are the settings to initialize a Provider.

type Resolver 

type Resolver struct {
	// contains filtered or unexported fields
}

Resolver resolves a configuration as a Conf.

func NewResolver 

func NewResolver(set ResolverSettings) (*Resolver, error)

NewResolver returns a new Resolver that resolves configuration from multiple URIs.

To resolve a configuration the following steps will happen:

  1. Retrieves individual configurations from all given "URIs", and merge them in the retrieve order.
  2. Once the Conf is merged, apply the converters in the given order.

After the configuration was resolved the `Resolver` can be used as a single point to watch for updates in the configuration data retrieved via the config providers used to process the "initial" configuration and to generate the "effective" one. The typical usage is the following: 

Resolver.Resolve(ctx)
Resolver.Watch() // wait for an event.
Resolver.Resolve(ctx)
Resolver.Watch() // wait for an event.
// repeat Resolve/Watch cycle until it is time to shut down the Collector process.
Resolver.Shutdown(ctx)

`uri` must follow the "<scheme>:<opaque_data>" format. This format is compatible with the URI definition (see https://datatracker.ietf.org/doc/html/rfc3986). An empty "<scheme>" defaults to "file" schema.

func (*Resolver) Resolve 

func (mr *Resolver) Resolve(ctx context.Context) (*Conf, error)

Resolve returns the configuration as a Conf, or error otherwise. Should never be called concurrently with itself, Watch or Shutdown.

func (*Resolver) Shutdown 

func (mr *Resolver) Shutdown(ctx context.Context) error

Shutdown signals that the provider is no longer in use and the that should close and release any resources that it may have created. It terminates the Watch channel.

Should never be called concurrently with itself or Get.

func (*Resolver) Watch 

func (mr *Resolver) Watch() <-chan error

Watch blocks until any configuration change was detected or an unrecoverable error happened during monitoring the configuration changes.

Error is nil if the configuration is changed and needs to be re-fetched. Any non-nil error indicates that there was a problem with watching the configuration changes.

Should never be called concurrently with itself or Get.

type ResolverSettings 

type ResolverSettings struct {
	// URIs locations from where the Conf is retrieved, and merged in the given order.
	// It is required to have at least one location.
	URIs []string

	// ProviderFactories is a list of Provider creation functions.
	// It is required to have at least one ProviderFactory
	// if a Provider is not given.
	ProviderFactories []ProviderFactory

	// ProviderSettings contains settings that will be passed to Provider
	// factories when instantiating Providers.
	ProviderSettings ProviderSettings

	// ConverterFactories is a slice of Converter creation functions.
	ConverterFactories []ConverterFactory

	// ConverterSettings contains settings that will be passed to Converter
	// factories when instantiating Converters.
	ConverterSettings ConverterSettings
}

ResolverSettings are the settings to configure the behavior of the Resolver.

type Retrieved 

type Retrieved struct {
	// contains filtered or unexported fields
}

Retrieved holds the result of a call to the Retrieve method of a Provider object.

func NewRetrieved 

func NewRetrieved(rawConf any, opts ...RetrievedOption) (*Retrieved, error)

NewRetrieved returns a new Retrieved instance that contains the data from the raw deserialized config. The rawConf can be one of the following types:

  • Primitives: int, int32, int64, float32, float64, bool, string;
  • []any;
  • map[string]any;

func (*Retrieved) AsConf 

func (r *Retrieved) AsConf() (*Conf, error)

AsConf returns the retrieved configuration parsed as a Conf.

func (*Retrieved) AsRaw 

func (r *Retrieved) AsRaw() (any, error)

AsRaw returns the retrieved configuration parsed as an any which can be one of the following types:

  • Primitives: int, int32, int64, float32, float64, bool, string;
  • []any - every member follows the same rules as the given any;
  • map[string]any - every value follows the same rules as the given any;

func (*Retrieved) Close 

func (r *Retrieved) Close(ctx context.Context) error

Close and release any watchers that Provider.Retrieve may have created.

Should block until all resources are closed, and guarantee that `onChange` is not going to be called after it returns except when `ctx` is cancelled.

Should never be called concurrently with itself.

type RetrievedOption 

type RetrievedOption func(*retrievedSettings)

RetrievedOption options to customize Retrieved values.

func WithRetrievedClose 

func WithRetrievedClose(closeFunc CloseFunc) RetrievedOption

WithRetrievedClose overrides the default Retrieved.Close function. The default Retrieved.Close function does nothing and always returns nil.

type UnmarshalOption 

type UnmarshalOption interface {
	// contains filtered or unexported methods
}

func WithIgnoreUnused 

func WithIgnoreUnused() UnmarshalOption

WithIgnoreUnused sets an option to ignore errors if existing keys in the original Conf were unused in the decoding process (extra keys).

type Unmarshaler 

type Unmarshaler interface {
	// Unmarshal a Conf into the struct in a custom way.
	// The Conf for this specific component may be nil or empty if no config available.
	// This method should only be called by decoding hooks when calling Conf.Unmarshal.
	Unmarshal(component *Conf) error
}

Unmarshaler interface may be implemented by types to customize their behavior when being unmarshaled from a Conf.

type WatcherFunc 

type WatcherFunc func(*ChangeEvent)

Source Files

View all Source files

Directories

Path Synopsis
Package confmaptest helps loading confmap.Conf to test packages implementing using the configuration.
 Package confmaptest helps loading confmap.Conf to test packages implementing using the configuration.
converter
expandconverter Module
internal
envvar
mapstructure
provider
internal
internal/configurablehttpprovider
envprovider Module
fileprovider Module
httpprovider Module
httpsprovider Module
yamlprovider Module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.