configloader

README

configloader-go

configloader is a Go library for loading config from multiple files (like config.toml and config_override.toml), with defaults and environment variable overrides. It provides the following features:

• Info on the provenance of each config field value -- which file the value came from, or if it was an env var override, or a default, or absent.
• Ability to flag fields as optional. And error will result if required fields are absent.
• Detection of vestigial fields in the config files -- fields which are unknown to the code.
• Ability to supply default field values.
• Environment variable field overriding.
• Additional expected-type checking, with detailed error messages. (Although this is of limited value over the checking in encoding/json and BurntSushi/toml).
• No more dependencies than you need.

See the GoDoc documentation.

Why not Viper?

1. It pulls in 20+ dependencies, including config languages you aren't using. (We vendor all our dependencies, so this is especially glaring.)
2. It doesn't support multiple files.

Installation

go get -u github.com/Psiphon-Inc/configloader-go

Example

For a full sample, see the "recommended" sample app.

import (
  "github.com/Psiphon-Inc/configloader-go"
  "github.com/Psiphon-Inc/configloader-toml"
)

type Config struct {
  Server struct{
    ListenPort string
  }

  Log struct {
    Level string
    Format string `conf:"optional"`
  }

  Stats struct {
    SampleCount int `toml:"sample_count"`
  }
}

envVarOverrides := []configloader.EnvOverride{
  {
    EnvVar: "SERVER_PORT",
    Key: configloader.Key{"Server", "ListenPort"},
  },
}

defaults := []configloader.Default{
  {
    Key: configloader.Key{"Log", "Level"},
    Val: "info",
  },
  {
    Key: configloader.Key{"Stats", "SampleCount"},
    Val: 1000,
  },
}

var config Config

metadata, err := configloader.Load(
  toml.Codec, // Specifies config file format
  configReaders, configReaderNames,
  defaults,
  envVarOverrides,
  &config)

// Record the config info. May help diagnose problems later.
log.Print(metadata.ConfigMap) // or log.Print(config)
log.Print(metadata.Provenances)

Future work

• Type checking inside slices (and better slice handling generally).

• HCL (or HCL2) support. Note that we'll either need better top-level slice support, or specify a limitation of no-top-level slices (which are easy to get with HCL).

• Re-evaluate whether the type checking is worthwhile at all or if it should just be left to the unmarshaler. (https://github.com/Psiphon-Inc/configloader-go/issues/1)

• Consider providing defaults via struct tags.

License

BSD 3-Clause License

Documentation

Overview

Package configloader makes loading config information easier, more flexible, and more powerful. It enables loading from multiple files, defaults, and environment overrides. TOML and JSON are supported out-of-the-box, but other formats can be easily used.

It is recommended that the examples be perused to assist usage: https://github.com/Psiphon-Inc/configloader-go/tree/master/examples

Result Structs and Maps

The value to be populated can be a struct or a map[string]interface{}. A struct is preferable, as it provides information about what fields should be expected, which are optional, and so on.

Struct Field Tags

Field name aliases can be specified using type-specific tags, like `toml:"alias"` or `json:"alias"`. Fields can be ignored using type-specific tags as well, like `toml:"-"` or `json:"-"`.

configloader also provides its own tag type, of the form `conf:"optional,specific_type"`, described below. The name "conf" is configurable with the TagName variable.

Optional and Required Fields

All fields in a result struct are by default required. A field can be marked as optional with the struct tag `conf:"optional"`. A field is also considered optional if it is present in the defaults argument.

Defaults

Default values for otherwise absent fields can be passed to Load(). Default values are only applied if the field receives no value from either the config files (readers) or an environment variable.

Fields with defaults provided in this manner are implicitly considered optional fields.

If a default value depends on the the values of other fields, then it should be flagged as optional via the struct tag, loaded from config, then checked with metadata.IsDefined() to see if it was set (in a file or environment override), and populated appropriately if it wasn't.

It is possible but not recommended to provide defaults by pre-populating the struct or map result. It's also possible but not recommended to check metadata.IsDefined() in accessors and return a default if not defined. Both of these approaches will result in the provenance being "[absent]" rather than "[default]".

Specify Field Type

The type to used for comparison can be specified with a struct tag, like `conf:",float32"` (before the comma is "optional", or not). It will be compared against the Type and Kind of the field. (There may not be any good use for this. If we come across one, add it here. Otherwise re-think the existence of this feature. See issue: https://github.com/Psiphon-Inc/configloader-go/issues/1)

Support for TextUnmarshaler

configloader detects fields that implement encoding/TextUnmarshaler and expects to find string values for those fields. This means that support for TextUnmarshaler is expected from the underlying unmarshaler.

Index

• Variables
• func FindFiles(fileLocations ...FileLocation) (readers []io.Reader, closers []io.Closer, readerNames []string, err error)
• type Codec
• type Default
• type EnvOverride
• type FileLocation
• type Key
  • func (k Key) MarshalText() (text []byte, err error)
  • func (k Key) String() string
• type Metadata
  • func Load(codec Codec, readers []io.Reader, readerNames []string, defaults []Default, ...) (md Metadata, err error)
  • func (md *Metadata) IsDefined(key ...string) (bool, error)
• type Provenance
  • func (prov Provenance) String() string
• type Provenances
  • func (provs Provenances) String() string

Variables

var TagName = "conf"

TagName is used in struct tags like `conf:"optional"`. Can be modified if the caller desires.

Functions

func FindFiles

func FindFiles(fileLocations ...FileLocation) (readers []io.Reader, closers []io.Closer, readerNames []string, err error)

FindFiles assists with figuring out which config files should be used.

fileLocations is the location info for the files that will contribute to this config. All files will be used, and each will be merged on top of the previous ones. The first file must exist (in at least one of the search paths), but subsequent files are optional. The intention is that the first file is the primary config, and the other files optionally override that.

The returned readers and readerNames are intended to be passed directly to configloader.Load(). The closers should be closed after Load() is called, perhaps like this:

defer func() {
  for i := range closers {
    closers[i].Close()
  }
}()

The reason the closers are separate from the readers (instead of []io.ReadClosers) is both to ease passing into Load() and to help ensure the closing happens (via and "unused variable" compile error).

Types

type Codec

type Codec interface {
	reflection.Codec

	Marshal(v interface{}) ([]byte, error)
	Unmarshal(data []byte, v interface{}) error

	// Codec-specific checks (OVER AND ABOVE decoder.fieldTypesConsistent).
	// For example, encoding.json makes all numbers float64.
	// noDeeper should be true if the structure should not be checked any deeper.
	// err must be non-nil if the types are _not_ consistent.
	// check and gold will never be nil.
	FieldTypesConsistent(check, gold *reflection.StructField) (noDeeper bool, err error)
}

Codec is the interface that specific config file language support must implement. See the json and toml packages for examples.

type Default

type Default struct {
	// The key of the field that will start with the default value.
	Key Key

	// The value the field should be given if it doesn't receive any other value.
	Val interface{}
}

Default is used to provide a default value for a field if it is otherwise absent.

type EnvOverride

type EnvOverride struct {
	// The environment variable. Case-sensitive.
	EnvVar string

	// The key of the field that should be overridden.
	Key Key

	// A function to convert from the string obtained from the environment to the type
	// required by the field. For example:
	//   func(v string) interface{} {
	// 	   return strconv.Atoi(v)
	//   }
	Conv func(envString string) (interface{}, error)
}

EnvOverride indicates that a field should be overridden by an environment variable value, if it exists.

type FileLocation

type FileLocation struct {
	// The filename will be searched for relative to each of the search paths. If one of
	// the search paths is "", then Filename will also be searched for as an absolute path.
	Filename string

	// The file will be search for through the SearchPaths. These are in order -- the
	// search will stop on the first match.
	SearchPaths []string
}

FileLocation is the name of a (potential) config file, and the places where it should be looked for.

type Key

type Key []string

Key is a field path into a struct or map. For most cases it can contain the field names used in the result struct, or the aliases used in the config file. A struct path key might look like Key{"Stats", "SampleCount"}. A config alias key might look like Key{"stats", "sample_count"}.

func (Key) MarshalText

func (k Key) MarshalText() (text []byte, err error)

MarshalText implements encoding.TextMarshaler. To be used with JSON logging (especially of Provenances).

func (Key) String

func (k Key) String() string

Convert k to a string appropriate for keying a map (so, unique and consistent).

type Metadata

type Metadata struct {

	// A map version of the resulting config.
	// It is good practice to log either this map or the config struct for later debugging help,
	// BUT ONLY IF THEY DON'T CONTAIN SECRETS.
	// (If the result is already a map, this is identical.)
	ConfigMap map[string]interface{}

	// The sources of each config field.
	// It is good practice to log either this map or the config struct for later debugging help.
	Provenances Provenances
	// contains filtered or unexported fields
}

Metadata contains information about the loaded config.

func Load

func Load(codec Codec, readers []io.Reader, readerNames []string, defaults []Default, envOverrides []EnvOverride, result interface{},
) (
	md Metadata, err error,
)

Load gathers config data from readers, defaults, and environment overrides, and populates result with the values. It provides log-able provenance information for each field in the metadata.

codec implements config-file-type-specific helpers. It's possible to use a custom implementation, but you probably want to use one of the configloader-go sub-packages (like json or toml).

readers will be used to populate the config. Later readers in the slice will take precedence and values from them will clobber the earlier.

readerNames contains useful names for the readers. This is intended to be the filenames obtained from FindFiles(). This is partly a human-readable convenience for provenances and partly essential to know exactly which files were used (as FindFiles look across multiple search paths).

defaults will be used to populate the result before any other sources.

envOverrides is a mapping from environment variable name to config key path. These are applied after all other sources.

result may be struct or map[string]interface{}.

Some of the reasons an error may be returned:

• A required field is absent
• A field was found in the config sources that is not present in the result struct
• The type of a value in the config sources didn't match the expected type in the result struct
• One of the readers couldn't be read
• Some other codec unmarshaling problem

func (*Metadata) IsDefined

func (md *Metadata) IsDefined(key ...string) (bool, error)

IsDefined checks if the given key was defined in the loaded struct (including from defaults or environment variable overrides). Error is returned if the key is not valid for the result struct. (So error is never returned if the result is a map.)

type Provenance

type Provenance struct {

	// The key of the field this is the provenance for.
	Key Key

	// The source of the value of the field. It can be one of the following:
	//   "path/to/file.toml": If the value came from a file and readerNames was provided to Load()
	//   "[0]": If the value came from a file and readerNames was not provided to Load()
	//   "[default]": If the field received the default value passed to Load()
	//   "[absent]": If the field was not set at all
	//   "$ENV_VAR_NAME": If the field value came from an environment variable override
	Src string
	// contains filtered or unexported fields
}

Provenance indicates the source that the value of a field ultimately came from.

func (Provenance) String

func (prov Provenance) String() string

String converts the provenance to a string. Useful for debugging, logging, or examples.

type Provenances

type Provenances []Provenance

Provenances provides the sources (provenances) for all of the fields in the resulting struct or map. It is good practice to log this value for later debugging help.

func (Provenances) String

func (provs Provenances) String() string

String converts the provenances to a string. Useful for debugging, logging, or examples.