README
¶
configloader-go
configloader is a Go library for loading config from multiple files (like config.toml and config_override.toml), with defaults and environment variable overrides. It provides the following features:
- Info on the provenance of each config field value -- which file the value came from, or if it was an env var override, or a default, or absent.
- Ability to flag fields as optional. And error will result if required fields are absent.
- Detection of vestigial fields in the config files -- fields which are unknown to the code.
- Ability to supply default field values.
- Environment variable field overriding.
- Additional expected-type checking, with detailed error messages. (Although this is of limited value over the checking in encoding/json and BurntSushi/toml).
- No more dependencies than you need.
See the GoDoc documentation.
Why not Viper?
- It pulls in 20+ dependencies, including config languages you aren't using. (We vendor all our dependencies, so this is especially glaring.)
- It doesn't support multiple files.
Installation
go get -u github.com/Psiphon-Inc/configloader-go
Example
For a full sample, see the "recommended" sample app.
import (
"github.com/Psiphon-Inc/configloader-go"
"github.com/Psiphon-Inc/configloader-toml"
)
type Config struct {
Server struct{
ListenPort string
}
Log struct {
Level string
Format string `conf:"optional"`
}
Stats struct {
SampleCount int `toml:"sample_count"`
}
}
envVarOverrides := []configloader.EnvOverride{
{
EnvVar: "SERVER_PORT",
Key: configloader.Key{"Server", "ListenPort"},
},
}
defaults := []configloader.Default{
{
Key: configloader.Key{"Log", "Level"},
Val: "info",
},
{
Key: configloader.Key{"Stats", "SampleCount"},
Val: 1000,
},
}
var config Config
metadata, err := configloader.Load(
toml.Codec, // Specifies config file format
configReaders, configReaderNames,
defaults,
envVarOverrides,
&config)
// Record the config info. May help diagnose problems later.
log.Print(metadata.ConfigMap) // or log.Print(config)
log.Print(metadata.Provenances)
Future work
-
Type checking inside slices (and better slice handling generally).
-
HCL (or HCL2) support. Note that we'll either need better top-level slice support, or specify a limitation of no-top-level slices (which are easy to get with HCL).
-
Re-evaluate whether the type checking is worthwhile at all or if it should just be left to the unmarshaler. (https://github.com/Psiphon-Inc/configloader-go/issues/1)
-
Consider providing defaults via struct tags.
License
BSD 3-Clause License
Documentation
¶
Overview ¶
Package configloader makes loading config information easier, more flexible, and more powerful. It enables loading from multiple files, defaults, and environment overrides. TOML and JSON are supported out-of-the-box, but other formats can be easily used.
It is recommended that the examples be perused to assist usage: https://github.com/Psiphon-Inc/configloader-go/tree/master/examples
Result Structs and Maps ¶
The value to be populated can be a struct or a map[string]interface{}. A struct is preferable, as it provides information about what fields should be expected, which are optional, and so on.
Struct Field Tags ¶
Field name aliases can be specified using type-specific tags, like `toml:"alias"` or `json:"alias"`. Fields can be ignored using type-specific tags as well, like `toml:"-"` or `json:"-"`.
configloader also provides its own tag type, of the form `conf:"optional,specific_type"`, described below. The name "conf" is configurable with the TagName variable.
Optional and Required Fields ¶
All fields in a result struct are by default required. A field can be marked as optional with the struct tag `conf:"optional"`. A field is also considered optional if it is present in the defaults argument.
Defaults ¶
Default values for otherwise absent fields can be passed to Load(). Default values are only applied if the field receives no value from either the config files (readers) or an environment variable.
Fields with defaults provided in this manner are implicitly considered optional fields.
If a default value depends on the the values of other fields, then it should be flagged as optional via the struct tag, loaded from config, then checked with metadata.IsDefined() to see if it was set (in a file or environment override), and populated appropriately if it wasn't.
It is possible but not recommended to provide defaults by pre-populating the struct or map result. It's also possible but not recommended to check metadata.IsDefined() in accessors and return a default if not defined. Both of these approaches will result in the provenance being "[absent]" rather than "[default]".
Specify Field Type ¶
The type to used for comparison can be specified with a struct tag, like `conf:",float32"` (before the comma is "optional", or not). It will be compared against the Type and Kind of the field. (There may not be any good use for this. If we come across one, add it here. Otherwise re-think the existence of this feature. See issue: https://github.com/Psiphon-Inc/configloader-go/issues/1)
Support for TextUnmarshaler ¶
configloader detects fields that implement encoding/TextUnmarshaler and expects to find string values for those fields. This means that support for TextUnmarshaler is expected from the underlying unmarshaler.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var TagName = "conf"
TagName is used in struct tags like `conf:"optional"`. Can be modified if the caller desires.
Functions ¶
func FindFiles ¶
func FindFiles(fileLocations ...FileLocation) (readers []io.Reader, closers []io.Closer, readerNames []string, err error)
FindFiles assists with figuring out which config files should be used.
fileLocations is the location info for the files that will contribute to this config. All files will be used, and each will be merged on top of the previous ones. The first file must exist (in at least one of the search paths), but subsequent files are optional. The intention is that the first file is the primary config, and the other files optionally override that.
The returned readers and readerNames are intended to be passed directly to configloader.Load(). The closers should be closed after Load() is called, perhaps like this:
defer func() {
for i := range closers {
closers[i].Close()
}
}()
The reason the closers are separate from the readers (instead of []io.ReadClosers) is both to ease passing into Load() and to help ensure the closing happens (via and "unused variable" compile error).
Types ¶
type Codec ¶
type Codec interface {
reflection.Codec
Marshal(v interface{}) ([]byte, error)
Unmarshal(data []byte, v interface{}) error
// Codec-specific checks (OVER AND ABOVE decoder.fieldTypesConsistent).
// For example, encoding.json makes all numbers float64.
// noDeeper should be true if the structure should not be checked any deeper.
// err must be non-nil if the types are _not_ consistent.
// check and gold will never be nil.
FieldTypesConsistent(check, gold *reflection.StructField) (noDeeper bool, err error)
}
Codec is the interface that specific config file language support must implement. See the json and toml packages for examples.
type Default ¶
type Default struct {
// The key of the field that will start with the default value.
Key Key
// The value the field should be given if it doesn't receive any other value.
Val interface{}
}
Default is used to provide a default value for a field if it is otherwise absent.
type EnvOverride ¶
type EnvOverride struct {
// The environment variable. Case-sensitive.
EnvVar string
// The key of the field that should be overridden.
Key Key
// A function to convert from the string obtained from the environment to the type
// required by the field. For example:
// func(v string) interface{} {
// return strconv.Atoi(v)
// }
Conv func(envString string) (interface{}, error)
}
EnvOverride indicates that a field should be overridden by an environment variable value, if it exists.
type FileLocation ¶
type FileLocation struct {
// The filename will be searched for relative to each of the search paths. If one of
// the search paths is "", then Filename will also be searched for as an absolute path.
Filename string
// The file will be search for through the SearchPaths. These are in order -- the
// search will stop on the first match.
SearchPaths []string
}
FileLocation is the name of a (potential) config file, and the places where it should be looked for.
type Key ¶
type Key []string
Key is a field path into a struct or map. For most cases it can contain the field names used in the result struct, or the aliases used in the config file. A struct path key might look like Key{"Stats", "SampleCount"}. A config alias key might look like Key{"stats", "sample_count"}.
func (Key) MarshalText ¶
MarshalText implements encoding.TextMarshaler. To be used with JSON logging (especially of Provenances).
type Metadata ¶
type Metadata struct {
// A map version of the resulting config.
// It is good practice to log either this map or the config struct for later debugging help,
// BUT ONLY IF THEY DON'T CONTAIN SECRETS.
// (If the result is already a map, this is identical.)
ConfigMap map[string]interface{}
// The sources of each config field.
// It is good practice to log either this map or the config struct for later debugging help.
Provenances Provenances
// contains filtered or unexported fields
}
Metadata contains information about the loaded config.
func Load ¶
func Load(codec Codec, readers []io.Reader, readerNames []string, defaults []Default, envOverrides []EnvOverride, result interface{}, ) ( md Metadata, err error, )
Load gathers config data from readers, defaults, and environment overrides, and populates result with the values. It provides log-able provenance information for each field in the metadata.
codec implements config-file-type-specific helpers. It's possible to use a custom implementation, but you probably want to use one of the configloader-go sub-packages (like json or toml).
readers will be used to populate the config. Later readers in the slice will take precedence and values from them will clobber the earlier.
readerNames contains useful names for the readers. This is intended to be the filenames obtained from FindFiles(). This is partly a human-readable convenience for provenances and partly essential to know exactly which files were used (as FindFiles look across multiple search paths).
defaults will be used to populate the result before any other sources.
envOverrides is a mapping from environment variable name to config key path. These are applied after all other sources.
result may be struct or map[string]interface{}.
Some of the reasons an error may be returned:
- A required field is absent
- A field was found in the config sources that is not present in the result struct
- The type of a value in the config sources didn't match the expected type in the result struct
- One of the readers couldn't be read
- Some other codec unmarshaling problem
type Provenance ¶
type Provenance struct {
// The key of the field this is the provenance for.
Key Key
// The source of the value of the field. It can be one of the following:
// "path/to/file.toml": If the value came from a file and readerNames was provided to Load()
// "[0]": If the value came from a file and readerNames was not provided to Load()
// "[default]": If the field received the default value passed to Load()
// "[absent]": If the field was not set at all
// "$ENV_VAR_NAME": If the field value came from an environment variable override
Src string
// contains filtered or unexported fields
}
Provenance indicates the source that the value of a field ultimately came from.
func (Provenance) String ¶
func (prov Provenance) String() string
String converts the provenance to a string. Useful for debugging, logging, or examples.
type Provenances ¶
type Provenances []Provenance
Provenances provides the sources (provenances) for all of the fields in the resulting struct or map. It is good practice to log this value for later debugging help.
func (Provenances) String ¶
func (provs Provenances) String() string
String converts the provenances to a string. Useful for debugging, logging, or examples.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
Package json provides JSON Codec methods for use with configloader.
|
Package json provides JSON Codec methods for use with configloader. |
|
Package reflection provides GetStructFields, allowing for the collection of structural information about structs or maps.
|
Package reflection provides GetStructFields, allowing for the collection of structural information about structs or maps. |
|
Package toml provides TOML Codec methods for use with configloader.
|
Package toml provides TOML Codec methods for use with configloader. |