dials

README

Dials

Dials is an extensible configuration package for Go.

Installation

go get github.com/vimeo/dials@latest

Prerequisites

Dials requires Go 1.18 or later.

What is Dials?

Dials is a configuration package for Go applications. It supports several different configuration sources including:

• Cue, JSON, YAML, and TOML config files
• environment variables
• command line flags (for both Go's flag package and pflag package)
• watched config files and re-reading when there are changes to the watched files
• default values

Why choose Dials?

Dials is a configuration solution that supports several configuration sources so you only have to focus on the business logic. Define the configuration struct and select the configuration sources and Dials will do the rest. Dials is designed to be extensible so if the built-in sources don't meet your needs, you can write your own and still get all the other benefits. Moreover, setting defaults doesn't require additional function calls. Just populate the config struct with the default values and pass the struct to Dials. Dials also allows the flexibility to choose the precedence order to determine which sources can overwrite the configuration values. Additionally, Dials has special handling of structs that implement encoding.TextUnmarshaler so structs (like IP and time) can be properly parsed.

Using Dials

Reading from config files, environment variables, and command line flags

Dials requires very minimal configuration to populate values from several sources through the ez package. Define the config struct and provide a method ConfigPath() (string, bool) to indicate the path to the config file. The package defines several functions that help populate the config struct by reading from config files, environment variables, and command line flags.

package main

import (
	"context"
	"fmt"

	"github.com/vimeo/dials/ez"
)

// Config is the configuration struct needed for the application
type Config struct {
	// When a struct tag more specifically corresponding to a source is
	// present, it takes precedence over the `dials` tag. Note that just
	// because there is a `yaml` struct tag present doesn't mean that other
	// sources can't fill this field.
	Val1 string `dials:"Val1" yaml:"b"`
	// the dials tag can be used as an alias so when the name in the config file
	// changes, the code doesn't have to change.
	Val2 int `dials:"val_2"`
	// Dials will register a flag with the name matching the dials tag.
	// Without any struct tags, dials will decode the go camel case field name
	// and encode it using lower-kebab-case and use the encoded name as the flag
	// name (ex: val-3). To specify a different flag name, use the `dialsflag`
	// tag. Now, Dials will register a flag with "some-val" name instead.
	// The `dialsdesc` tag is used to provide help message for the flag.
	Val3 bool `dialsflag:"some-val" dialsdesc:"enable auth"`
	// Path holds the value of the path to the config file. Dials follows the
	// *nix convention for environment variables and will look for the dials tag
	// or field name in all caps when struct tags aren't specified. Without any
	// struct tags, it would lookup the PATH environment variable. To specify a
	// different env variable, use the `dialsenv` tag. Now Dials will lookup
	// "configpath" env value to populate the Path field.
	Path string `dialsenv:"configpath"`
}

// ConfigPath returns the path to the config file that Dials should read. This 
// is particularly helpful when it's desirable to specify the file's path via
// environment variables or command line flags. Dials will first populate the 
// configuration struct from environment variables and command line flags
// and then read the config file that the ConfigPath() method returns
func (c *Config) ConfigPath() (string, bool) {
	// can alternatively return empty string and false if the state of the
	// struct doesn't specify a config file to read. We would recommend to use
	// dials.Config directly (shown in the next example) instead of the ez
	// package if you just want to use environment variables and flags without a
	// file source
	return c.Path, true
}

func main() {
	defCfg := Config{}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// The following function will populate the config struct by reading the
	// config files, environment variables, and command line flags (order matches
	// the function name) with increasing precedence. In other words, the flag
	// source (last) would overwrite the YAML source (first) were they both to
	// attempt to set the same struct field. There are several options that can be
	// passed in as well to indicate whether the file will be watched and updates
	// to the file should update the config struct and if the flags name component
	// separation should use different encoding than lower-kebab-case
	d, dialsErr := ez.YAMLConfigEnvFlag(ctx, &defCfg, ez.Params[Config]{WatchConfigFile: true})
	if dialsErr != nil {
		// error handling
	}

	// View returns a pointer to the fully stacked configuration file
	// The stacked configuration is populated from the config file, environment
	// variables and commandline flags.
	cfg := d.View()
	fmt.Printf("Config: %+v\n", cfg)
}

For reading from JSON or TOML config files along with environment variables and command line flags, use the ez.JSONConfigEnvFlag or ez.TOMLConfigEnvFlag functions.

If the above code is run with the following YAML file:

b: valueb
val_2: 2
val-3: false

and the following command (make sure to change the configpath value to point to your path)

export configpath=path/to/config/file
export VAL_2=5
go run main.go --some-val

the output will be Config: &{Val1:valueb Val2:5 Val3:true}

Note that even though val_2 has a value of 2 in the yaml file, the config value output for Val2 is 5 because environment variables take precedence.

Configure your configuration settings

If the predefined functions in the ez package don't meet your needs, you can specify the sources you would like in the order of precedence you prefer. Not much setup is needed to configure this. Choose the predefined sources and add the appropriate dials tags to the config struct.

package main

import (
	"context"
	"fmt"

	"github.com/vimeo/dials"
	"github.com/vimeo/dials/sources/env"
	"github.com/vimeo/dials/sources/file"
	"github.com/vimeo/dials/sources/flag"
	"github.com/vimeo/dials/decoders/yaml"
)

type Config struct {
	Val1 string `dials:"Val1" yaml:"b"`
	// The `dialsenv` tag is used to override the name used by the environment
	// source. If you want to use a single, consistent name across several
	// sources, set the `dials` tag instead
	Val2 int `dialsenv:"VAL_2"`
	// the `dialsflag` tag is used for command line flag values and the dialsdesc
	// tag provides the flag help text
	Val3 bool `dialsflag:"val-3" dialsdesc:"maximum number of idle connections to DB"`
}

func main() {
	defaultConfig := &Config{
		// Val1 has a default value of "hello" and will be overwritten by the
		// sources if there is a corresponding field for Val1
		Val1: "hello",
	}

	// Define a file source if you want to read from a config file. To read
	// from other source files such as Cue, JSON, and TOML, use "&cue.Decoder{}",
    // "&json.Decoder{}" or "&toml.Decoder{}"
	fileSrc, fileErr := file.NewSource("path/to/config", &yaml.Decoder{})
	if fileErr != nil {
		// error handling
	}

	// Define a `dials.Source` for command line flags. Consider using the dials
	// pflag library if the application uses the github.com/spf13/pflag package
	flagSet, flagErr := flag.NewCmdLineSet(flag.DefaultFlagNameConfig(), defaultConfig)
	if flagErr != nil {
		// error handling
	}

	// use the environment source to get values from environment variables
	envSrc := &env.Source{}

	// the Config struct will be populated in the order in which the sources are
	// passed in the Config function with increasing precedence. So the fileSrc
	// value will overwrite the flagSet value if they both were to set the
	// same field
	d, err := dials.Config(context.Background(), defaultConfig, envSrc, flagSet, fileSrc)
	if err != nil {
		// error handling
	}

	// View returns a pointer to the fully stacked configuration.
	// The stacked configuration is populated from the config file, environment
	// variables and commandline flags. Can alternatively use
	cfg := d.View()
	fmt.Printf("Config: %+v\n", cfg)
}

If the above code is run with the following YAML file (make sure to change the path in the code):

b: valueb
val-3: false

and the following commands

export VAL_2=5
go run main.go --val-3

the output will be Config: &{Val1:valueb Val2:5 Val3:true}.

Note that even when val-3 is defined in the yaml file and the file source takes precedence, only the value from command line flag populates the config due to the special dialsflag tag. The val-3 name is only used by the flag source. The file source will still use the field name. You can update the yaml file to val3: false to have the file source overwrite the field. Alternatively, we recommend using the dials tag to have consistent naming across all sources.

Watching file source

If you wish to watch the config file and make updates to your configuration, use the watching source. This functionality is available in the ez package by using the WithWatchingConfigFile(true) option (the default is false). The WatchingSource can be used when you want to further customize the configuration as well. Please note that the Watcher interface is likely to change in the near future.

	// NewWatchSource also has watch options that can be passed to use a ticker
	// for polling, set a logger, and more
	watchingFileSource, fsErr := file.NewWatchingSource(
		"path/to/config", &yaml.Decoder{})

	if fsErr != nil {
		// error handling
		return
	}

	// Use a non-background context with the WatchingSource to prevent goroutine leaks
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()


	// additional sources can be passed along with the watching file source and the
	// precedence order will still be dictated by the order in which the sources are
	// defined in the Config function.
	d, err := dials.Config(ctx, defCfg, watchingFileSource)
	if err != nil {
		// error handling
	}

	conf, serial := d.ViewVersion()

	// You can get notified whenever the config changes by registering a callback.
	// If a new version has become available since the serial argument was
	// returned by ViewVersion(), it will be called immediately to bring
	// the callback up to date.
	// You can call the returned unregister function to unregister at a later point.
	unreg := d.RegisterCallback(ctx, serial, func(ctx context.Context, oldCfg, newCfg *Config) {
		// log, increment metrics, re-index a map, etc.
	})
	// If the passed context expires before registration succeeds, a nil
	// unregister callback will be returned.
	if unreg != nil {
		defer unreg()
	}

Aliased Configuration Values

Dials supports aliases for fields where you want to change the name and support an orderly transition from an old name to a new name. Just as there is a dials struct tag there is also a dialsalias struct tag that you can use as an alternate name. Any other casing or transformation rules on the original tags also applies to the aliases. Only one of the original or alias versions of the field may be set at any time. Setting both results in an error. Aliases are supported in the other source-specific tags like dialsenv, dialsflag, and dialspflag as well by appending "alias" to the flag name. The ez package also wraps the appropriate file source's decoder so config files can also make use of aliases.

Aliases are implemented using the Mangler interface. The env, flag, and pflag sources support aliasing without any additional options.

Flags

When setting commandline flags using either the pflag or flag sources, additional flag-types become available for simple slices and maps.

Slices

Slices of integer-types get parsed as comma-separated values using Go's parsing rules (with whitespace stripped off each component) e.g. --a=1,2,3 parses as []int{1,2,3}

Slices of strings get parsed as comma-separated values if the individual values are alphanumeric, and must be quoted in conformance with Go's strconv.Unquote for more complicated values e.g. --a=abc parses as []string{"abc"}, --a=a,b,c parses as []string{"a", "b", "c"}, while --a="bbbb,ffff" has additional quoting (ignoring any shell), so it becomes []string{"bbbb,ffff"}

Slice-typed flags may be specified multiple times, and the values will be concatenated. As a result, a commandline with "--a=b", "--a=c" may be parsed as []string{b,c}.

Maps

Maps are parsed like Slices, with the addition of : separators between keys and values. (strconv.Unquote-compatible quoting is mandatory for more complicated strings as well)

e.g. --a=b:c parses as map[string]string{"b": "c"}

Source

The Source interface is implemented by different configuration sources that populate the configuration struct. Dials currently supports environment variables, command line flags, and config file sources. When the dials.Config method is going through the different Sources to extract the values, it calls the Value method on each of these sources. This allows for the logic of the Source to be encapsulated while giving the application access to the values populated by each Source. Please note that the Value method on the Source interface and the Watcher interface are likely to change in the near future.

Decoder

Decoders are modular, allowing users to mix and match Decoders and Sources. Dials currently supports Decoders that decode different data formats (JSON, YAML, and TOML) and insert the values into the appropriate fields in the config struct. Decoders can be expanded from that use case and users can write their own Decoders to perform the tasks they like (more info in the section below).

Decoder is called when the supported Source calls the Decode method to unmarshal the data into the config struct and returns the populated struct. There are two sources that the Decoders can be used with: files (including watched files) and static.StringSource. Please note that the Decoder interface is likely to change in the near future.

Write your own Source and Decoder

If you wish to define your own source, implement the Source interface and pass the source to the dials.Config function. If you want the Source to interact with a Decoder, call Decode in the Value method of the Source.

Since Decoders are modular, keep the logic of Decoder encapsulated and separate from the Source. Source and Decoder implementations should be orthogonal and Decoders should not be Source specific. For example, you can have an HTTP or File Source that can interact with the JSON decoder to unmarshal the data to a struct.

Putting it all together

1. The dials.Config function makes a deep copy of the configuration struct and makes each field a pointer (even the fields in nested structs) with special handling for structs that implement encoding.TextUnmarshaler.
2. Call the Value method on each Source and stores the returned value.
3. The final step is to to compose the final config struct by overlaying the values from all the different Sources and accounting for the precedence order. Since the fields are pointers, we can directly assign pointers while overlaying. Overlay even has safety checks for deduplicating maps sharing a backing pointer and for structs with self-referential pointers.

So when you write your own Source, you just have to pass the Source in to the dials.Config function and Dials will take care of deep copying and pointerifying the struct and composing the final struct with overlay.

Contributors

Dials is a production of Vimeo's Core Services team

• @sergiosalvatore
• @dfinkel
• @sachinagada

Documentation

Index

• type CfgSerial
• type Decoder
• type Dials
  • func Config[T any](ctx context.Context, t *T, sources ...Source) (*Dials[T], error)
  • func (d *Dials[T]) EnableVerification(ctx context.Context) (*T, CfgSerial[T], error)
  • func (d *Dials[T]) Events() <-chan *T
  • func (d *Dials[T]) Fill(blankConfig *T)
  • func (d *Dials[T]) RegisterCallback(ctx context.Context, serial CfgSerial[T], cb NewConfigHandler[T]) UnregisterCBFunc
  • func (d *Dials[T]) View() *T
  • func (d *Dials[T]) ViewVersion() (*T, CfgSerial[T])
• type NewConfigHandler
• type Params
  • func (p Params[T]) Config(ctx context.Context, t *T, sources ...Source) (*Dials[T], error)
• type Source
• type Type
  • func NewType(t reflect.Type) *Type
  • func (t *Type) Type() reflect.Type
• type UnregisterCBFunc
• type VerifiedConfig
• type WatchArgs
• type WatchedErrorHandler
• type Watcher

Examples

• Dials.RegisterCallback

Types

type CfgSerial

type CfgSerial[T any] struct {
	// contains filtered or unexported fields
}

CfgSerial is an opaque object unique to a config-version

type Decoder

type Decoder interface {
	Decode(io.Reader, *Type) (reflect.Value, error)
}

Decoder interface is implemented by different data formats to read the config files, decode the data, and insert the values in the config struct. Dials currently includes implementations for YAML, JSON, and TOML data formats.

type Dials

type Dials[T any] struct {
	// contains filtered or unexported fields
}

Dials is the main access point for your configuration.

func Config

func Config[T any](ctx context.Context, t *T, sources ...Source) (*Dials[T], error)

Config populates the passed in config struct by reading the values from the different Sources. The order of the sources denotes the precedence of the formats so the last source passed to the function has the ability to override fields that were set by previous sources This top-level function is present for convenience and backwards compatibility when there is no need to specify an error-handler.

func (*Dials[T]) EnableVerification

func (d *Dials[T]) EnableVerification(ctx context.Context) (*T, CfgSerial[T], error)

EnableVerification enables verification on dials if DelayInitialVerification was set on the Params struct. Returns the config that was verified and a CfgSerial or the error from calling Verify() (if the config type implements VerifiedConfig.

If DelayInitialVerification is not set, returns successfully without verifying the config.

If verification succeeds on the currently installed configuration, all subsequent configuration versions will be verified. (based on re-stacking versions from watching sources)

When there are watching sources (including Blank) the global callbacks may be suppressed with the Params.CallGlobalCallbacksAfterVerificationEnabled option. This suppression expires after verification is re-enabled by this method.

Note: if the context expires while this call is awaiting a response from the background "monitor" goroutine, verification may still happen, but whether it transitions out of the delayed verification state is indeterminate.

func (*Dials[T]) Events

func (d *Dials[T]) Events() <-chan *T

Events returns a channel that will get a message every time the configuration is updated.

func (*Dials[T]) Fill

func (d *Dials[T]) Fill(blankConfig *T)

Fill populates the passed struct with the current value of the configuration. It is a thin wrapper around assignment deprecated: assign return value from View() instead

func (*Dials[T]) RegisterCallback

func (d *Dials[T]) RegisterCallback(ctx context.Context, serial CfgSerial[T], cb NewConfigHandler[T]) UnregisterCBFunc

RegisterCallback registers the callback cb to receive notifications whenever a new configuration is installed. If the "current" version is later than the one represented by the value of CfgSerial, a notification will be delivered immediately. This call is only blocking if the callback handling has filled up an internal channel. (likely because an already-registered callback is slow or blocking) serial must be obtained from [Dials.ViewVersion()]. Catch-up callbacks are suppressed if passed passed an invalid CfgSerial (including the zero-value)

May return a nil UnregisterCBFunc if the context expires

The returned UnregisterCBFunc will block until the relevant callback has been removed from the set of callbacks.

Example

// setup a cancelable context so the monitor goroutine gets shutdown.
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

type testConfig struct {
	Foo string
}

type ptrifiedConfig struct {
	Foo *string
}

base := testConfig{
	Foo: "foo",
}
// Push a new value, that should overlay on top of the base
foozleStr := "foozle"
foozleConfig := ptrifiedConfig{
	Foo: &foozleStr,
}

w := fakeWatchingSource{fakeSource: fakeSource{outVal: foozleConfig}}
d, dialsErr := Config(ctx, &base, &w)
if dialsErr != nil {
	panic("unexpected error: " + dialsErr.Error())
}

cfg, serialToken := d.ViewVersion()
fmt.Printf("Foo: %s\n", cfg.Foo)

unregCB := d.RegisterCallback(ctx, serialToken, func(ctx context.Context, oldCfg, newCfg *testConfig) {
	panic("not getting called this time")
})

unregCB(ctx)

Output:

Foo: foozle

func (*Dials[T]) View

func (d *Dials[T]) View() *T

View returns the configuration struct populated.

func (*Dials[T]) ViewVersion

func (d *Dials[T]) ViewVersion() (*T, CfgSerial[T])

View returns the configuration struct populated, and an opaque token.

type NewConfigHandler

type NewConfigHandler[T any] func(ctx context.Context, oldConfig, newConfig *T)

NewConfigHandler is a callback that's called after a new config is installed. Callbacks are run on a dedicated Goroutine, so one can do expensive/blocking work in this callback, however, execution should not last longer than the interval between new configs.

type Params

type Params[T any] struct {
	// OnWatchedError is called when either of several conditions are met:
	//  - There is an error re-stacking the configuration
	//  - One of the Sources implementing the Watcher interface reports an error
	//  - a Verify() method fails after re-stacking when a new version is
	//    provided by a watching source
	OnWatchedError WatchedErrorHandler[T]

	// SkipInitialVerification skips the initial call to `Verify()` on any
	// configurations that implement the [VerifiedConfig] interface.
	//
	// In cases where later updates from Watching sources are depended upon to
	// provide a configuration that will be allowed by Verify(), one should set
	// this to true.  See `sourcewrap.Blank` for more details.
	//
	// Unlike DelayInitialVerification, this field only skips the initial Verify()
	// call, so all watching sources (including Blank) trigger configuration
	// verification.
	SkipInitialVerification bool

	// OnNewConfig is called when a new (valid) configuration is installed.
	//
	// OnNewConfig runs on the same "callback" goroutine as the
	// OnWatchedError callback, with callbacks being executed in-order.
	// In the event that a call to OnNewConfig blocks too long, some calls
	// may be dropped.
	OnNewConfig NewConfigHandler[T]

	// DelayInitialVerification skips calls to Verify() until the EnableVerification()
	// method is called.
	//
	// Some systems require coalescing the data from multiple Sources, which require
	// initialization parameters from other sources (e.g. filenames).
	//
	// Notably, many use-cases involving sourcewrap.Blank may require multiple steps
	// to initialize, during which time the configuration will be incomplete and may
	// not validate.
	DelayInitialVerification bool

	// CallGlobalCallbacksAfterVerificationEnabled suppresses calling any registered
	// global while in the delayed-verification mode.
	//
	// In particular, global callbacks (those registered in this struct) are
	// suppressed under two conditions:
	//  - DelayInitialVerification was set to true when Config was called
	//  - EnableVerification has not been called (without it returning an error)
	CallGlobalCallbacksAfterVerificationEnabled bool
}

Params provides options for setting Dials's behavior in some cases.

func (Params[T]) Config

func (p Params[T]) Config(ctx context.Context, t *T, sources ...Source) (*Dials[T], error)

Config populates the passed in config struct by reading the values from the different Sources. The order of the sources denotes the precedence of the formats so the last source passed to the function has the ability to override fields that were set by previous sources

If present, a Verify() method will be called after each stacking attempt. Blocking/expensive work should not be done in this method. (see the comment on Verify()) in VerifiedConfig for details)

If complicated/blocking initialization/verification is necessary, one can either:

• If not using any watching sources, do any verification with the returned config from Config.
• If using at least one watching source, configure a goroutine to watch the channel returned by the `Dials.Events()` method that does its own installation after verifying the config.

More complicated verification/initialization should be done by consuming from the channel returned by `Events()`.

type Source

type Source interface {
	// Value provides the current value for the configuration.
	// Value methods should not create any long-lived resources or spin off
	// long-lived goroutines.
	// Config() will cancel the context passed to this method upon Config's
	// return.
	// Implementations that need to handle state changes with long-lived
	// background goroutines should implement the Watcher interface, which
	// explicitly provides a way to supply state updates.
	Value(context.Context, *Type) (reflect.Value, error)
}

Source interface is implemented by each configuration source that is used to populate the config struct such as environment variables, command line flags, config files, and more

type Type

type Type struct {
	// contains filtered or unexported fields
}

Type is a wrapper for a reflect.Type.

func NewType

func NewType(t reflect.Type) *Type

NewType constructs a new dials Type for a reflect.Type.

func (*Type) Type

func (t *Type) Type() reflect.Type

Type describes a config struct type, usually it is already pointerified

type UnregisterCBFunc

type UnregisterCBFunc func(ctx context.Context) bool

UnregisterCBFunc unregisters a callback from the dials object it was registered with.

type VerifiedConfig

type VerifiedConfig interface {
	// Verify() should return a non-nil error if the configuration is
	// invalid.
	// As this method is called any time the configuration sources are
	// restacked, it should not do any complex or blocking work.
	Verify() error
}

VerifiedConfig implements the Verify method, allowing Dials to execute the Verify method before returning/installing a new version of the configuration.

type WatchArgs

type WatchArgs interface {
	// ReportNewValue reports a new value. The base implementation returns an
	// error if the internal reporting channel is full and the context
	// expires/is-canceled, however, wrapping implementations are free to
	// return any other error as appropriate.
	ReportNewValue(ctx context.Context, val reflect.Value) error
	// Done indicates that this watcher has stopped and will not send any
	// more updates.
	Done(ctx context.Context)
	// ReportError reports a problem in the watcher. Returns an error if
	// the internal reporting channel is full and the context
	// expires/is-canceled.
	ReportError(ctx context.Context, err error) error

	// BlockingReportNewValue reports a new value. Returns an error if the internal
	// reporting channel is full and the context expires/is-canceled.
	// Blocks until the new value has been or returns an error.
	//
	// Most Source implementations should use ReportNewValue(). This was added to
	// support [github.com/vimeo/dials/sourcewrap.Blank]. This should only be used
	// in similar cases.
	BlockingReportNewValue(ctx context.Context, val reflect.Value) error
}

WatchArgs provides methods for a Watcher implementation to update the state of a Dials instance.

type WatchedErrorHandler

type WatchedErrorHandler[T any] func(ctx context.Context, err error, oldConfig, newConfig *T)

WatchedErrorHandler is a callback that's called when something fails when dials is operating in a watching mode. If non-nil, both oldConfig and newConfig are guaranteed to be populated with the same pointer-type that was passed to `Config()`. newConfig will be nil for errors that prevent stacking.

type Watcher

type Watcher interface {
	// Watch will be called in the primary goroutine calling Config(). If
	// Watcher implementations need a persistent goroutine, they should
	// spawn it themselves.
	Watch(context.Context, *Type, WatchArgs) error
}

Watcher should be implemented by Sources that allow their configuration to be watched for changes.