goschtalt

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 18, 2022 License: Apache-2.0 Imports: 16 Imported by: 0

README

goschtalt

A simple configuration library that supports multiple files and formats.

Build Status codecov.io Go Report Card GitHub Release GoDoc

Goals & Themes

  • Favor small, simple designs.
  • Keep dependencies to a minimum.
  • Favor user customization options over building everything in.
  • Leverage go's new fs.FS interface for collecting files.

API Stability

This package has not yet released to 1.x yet, so APIs are subject to change for a bit longer.

Extensions

These are just the extensions the goschtalt team maintains. Others may be available and it's fairly easy to write your own. Extensions have their own go.mod files that independently track dependencies to keep dependencies only based on what you need, not what could be used.

Configuration Decoders

The decoders convert a file format into a useful object tree. The meta.Object has many convenience functions that make adding decoders pretty simple. Generally, the hardest part is determining where you are processing in the original file.

Status GoDoc Extension Description
Go Report Card GoDoc decoders/env An environment variable based configuration decoder.
Go Report Card GoDoc decoders/json A JSON configuration decoder.
Go Report Card GoDoc decoders/properties A properties configuration decoder.
Go Report Card GoDoc decoders/yaml A YAML/YML configuration decoder
Configuration Encoders

The encoders are used to output configuration into a file format. Ideally you want a format that accepts comments so it's easier see where the configurations originated from.

Status GoDoc Extension Description
Go Report Card GoDoc encoders/yaml A YAML/YML configuration encoder.

Dependencies

There are only two production dependencies in the core goschtalt code beyond the go standard library. The rest are testing dependencies.

Production dependencies:

  • github.com/mitchellh/hashstructure
  • github.com/mitchellh/mapstructure

Examples

Coming soon.

Documentation

Overview

Package goschtalt is a lightweight and flexible configuration registry that makes it easy to configure an application.

Goschtalt is a fresh take on application configuration now that Go has improved filesystem abstraction, modules, and the Option pattern. At its core, Goschtalt is a low dependency library that provides configuration values via a small and customizable API. The configuration values can be merged using either the default semantics or specified on a parameter by parameter basis.

What problems is Goschtalt trying to solve?

  • Merging of multiple configuration files and configuration sources allow for better flexibility for deployment.
  • Configuration is incrementally compiled allowing later configuration to use earlier configuration.
  • Clear explanation how the configuration was derived.
  • User customizable via Options.

Features

  • Users choose which configuration file decoders they want.
  • Configuration fields may be labeled as 'secret' to enable secret redaction during output of portions of the configuration tree.
  • Configuration fields may instruct the merge process of how the new field should merge with the existing field. ('replace', 'keep', 'fail', 'append', 'prepend', 'clear')
  • Configuration file groups include a reference to the specific io.fs, so configuration may come from anything that implements that interface.
  • Package defaults are set via goschtalt.DefaultOptions, but can be replaced when invoking a new goschtalt.Config object.
  • Default values are supported at runtime.
  • Variable expansion in the configuration tree is supported for both environment variables as well as custom values.
  • No singleton objects.
  • Low dependency count.

Where do I find configuration file encoders/decoders?

The project contains several packages that are versioned together, but are otherwise independent (different go modules). They can be found here:

https://github.com/schmidtw/goschtalt/tree/main/extensions/ 

decoders/env        - decoder for environment variables
decoders/json       - decoder for json files
decoders/properties - decoder for properties files
decoders/yaml       - decoder for yaml files
encoders/yaml       - encoder for yaml files

How do I decorate my configuration files to take full advantage of goschtalt?

For most of the decoders you can specify instructions for goschtalt's handling of the data fields by annotating the key portion. Here's a simple example in yaml: 

foo:
  bar((prepend)):
    - 1
    - 2

If this configuration data is merged with an existing configuration set: 

foo:
  bar:
    - 3
    - 4

The resulting configuration will be: 

foo:
  bar:
    - 1
    - 2
    - 3
    - 4

The commands available are consistent, but vary based on the type of the value being defined.

All types (maps, arrays, values) support:

  • replace - replaces any existing values encountered by this merge
  • keep - keeps the existing values encountered by this merge
  • fail - causes the merge to return an error and stop processing
  • clear - causes all of the existing configuration tree to be deleted
  • secret - this special command marks the field as secret

Maps support the following instructions:

  • splice - merge the leaf nodes if possible instead of replacing the map entirely

Arrays support the following instructions:

  • append - append this array to the existing array
  • prepend - prepend this array to the existing array

Default merging behaviors:

  • maps - splice when possible, replace if splicing isn't possible
  • arrays - append
  • values - replace

An example showing using a secret: 

foo:
  bar ((append secret)):
    - 3
    - 4

The order of the instructions doesn't matter, nor does extra spaces around the instructions. You may comma separate them, or you may just use a space. But you can only have one or two instructions (one MUST be secret if there are two.

A bit more on secrets.

Secrets are primarily there so that if you want to output your configuration and everything is marked as secret correctly, you can get a redacted configuration file with minimal work. It's also handy if you output your configuration values into a log so you don't accidentally leak your secrets.

How do I write my own configuration decoder?

Examples of decoders exist in the extensions/decoders directory. Of interest are the `env` decoder that provides an Option, and the `yaml` decoder that is simply a decoder.

What's with the name?

In psychology, gestalt is a way of thinking about data via patterns and configuration. It's a also somewhat common word. gostalt is pretty good, except there were still several things that used it, including a go framework. This goschtalt project is the only response google returned as of Aug 12, 2022.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ErrDecoding      = errors.New("decoding error")
	ErrEncoding      = errors.New("encoding error")
	ErrNotCompiled   = errors.New("the Compile() function must be called first")
	ErrCodecNotFound = errors.New("encoder/decoder not found")
	ErrInvalidInput  = errors.New("input is invalid")
	ErrFileMissing   = errors.New("required file is missing")
)
View Source 
var DefaultOptions = []Option{}

DefaultOptions allows a simple place where decoders can automatically register themselves, as well as a simple way to find what is configured by default. Most extensions will register themselves using init(). It is safe to change this value at pretty much any time & compile afterwards; just know this value is not mutex protected so if you are changing it after init() the synchronization is up to the caller.

Functions

func Unmarshal 

func Unmarshal[T any](c *Config, key string, opts ...UnmarshalOption) (T, error)

Unmarshal provides a generics based strict typed approach to fetching parts of the configuration tree.

func UnmarshalFn 

func UnmarshalFn[T any](key string, opts ...UnmarshalOption) func(*Config) (T, error)

UnmarshalFn returns a function that takes a goschtalt Config structure and returns a function that allows for unmarshalling of a portion of the tree specified by the key into a zero value type.

This function is specifically helpful with DI frameworks like Uber's fx framework.

In this short example, the type myStruct is created and populated with the configuring values found under the "conf" key in the goschtalt configuration. 

app := fx.New(
	fx.Provide(
		goschtalt.UnmarshalFn[myStruct]("conf"),
	),
)

Types

type Config 

type Config struct {
	// contains filtered or unexported fields
}

Config is a configurable, prioritized, merging configuration registry.

func New 

func New(opts ...Option) (*Config, error)

New creates a new goschtalt configuration instance with any number of options.

func (*Config) Compile 

func (c *Config) Compile() error

Compile reads in all the files configured using the options provided, and merges the configuration trees into a single map for later use.

func (*Config) CompiledAt 

func (c *Config) CompiledAt() time.Time

CompiledAt returns when the configuration was compiled.

func (*Config) Explain 

func (c *Config) Explain() string

Explain returns a human focused explanation of how the configuration was arrived at. Each time the options change or the configuration is compiled the explanation will be updated.

func (*Config) Extensions 

func (c *Config) Extensions() []string

Extensions returns the extensions this config object supports.

func (*Config) Hash 

func (c *Config) Hash() uint64

Hash returns the hash of the configuration; even if the configuration is empty.

func (*Config) Marshal 

func (c *Config) Marshal(opts ...MarshalOption) ([]byte, error)

Marshal renders the into the format specified ('json', 'yaml' or other extensions the Codecs provide and if adding comments should be attempted. If a format does not support comments, an error is returned. The result of the call is a slice of bytes with the information rendered into it.

func (*Config) OrderList 

func (c *Config) OrderList(list []string) []string

OrderList is a helper function that sorts a caller provided list of filenames exectly the same way the Config object would sort them when reading and merging the records when the configuration is being compiled. It also filters the list based on the decoders present.

Example 
// SPDX-FileCopyrightText: 2022 Weston Schmidt <weston_schmidt@alumni.purdue.edu>
// SPDX-License-Identifier: Apache-2.0

package main

import (
	"fmt"

	"github.com/schmidtw/goschtalt"
	"github.com/schmidtw/goschtalt/pkg/decoder"
	"github.com/schmidtw/goschtalt/pkg/meta"
)

// This is a fake decoder that allows the OrderList() to correctly work without
// needing to bring any external decoders into goschtalt.
type fake struct{}

func (f fake) Decode(_ decoder.Context, _ []byte, _ *meta.Object) error { return nil }
func (f fake) Extensions() []string                                     { return []string{"yml", "yaml", "json"} }

func main() {
	g, err := goschtalt.New(goschtalt.WithDecoder(fake{}))
	if err != nil {
		panic(err)
	}

	err = g.Compile()
	if err != nil {
		panic(err)
	}

	files := []string{
		"file_2.json",
		"file_1.yml",
		"file_1.txt",
		"S99_file.yml",
		"S58_file.json",
	}
	list := g.OrderList(files)

	fmt.Println("files:")
	for _, file := range list {
		fmt.Printf("\t%s\n", file)
	}

}

Output:

files:
	S58_file.json
	S99_file.yml
	file_1.yml
	file_2.json

func (*Config) ShowOrder 

func (c *Config) ShowOrder() ([]string, error)

ShowOrder is a helper function that provides the order the configuration records were combined based on the present configuration. This can only be called after the Compile() has been called.

func (*Config) Unmarshal 

func (c *Config) Unmarshal(key string, result any, opts ...UnmarshalOption) error

Unmarshal performs the act of looking up the specified section of the tree and decoding the tree into the result. Additional options can be specified to adjust the behavior.

func (*Config) With 

func (c *Config) With(opts ...Option) error

With takes a list of options and applies them. Use of With() is optional as New() can take all the same options as well. If AutoCompile() is not specified Compile() will need to be called to see changes in the configuration based on the new options.

type DecoderConfigOption 

type DecoderConfigOption interface {
	fmt.Stringer
	UnmarshalOption
	ValueOption
	// contains filtered or unexported methods
}

DecoderConfigOption is used to configure the mapstructure used for decoding structures into the values used by the internal goschtalt tree.

All of these options are directly concerned with the mitchellh/mapstructure package. For additional details please see: https://github.com/mitchellh/mapstructure

func DecodeHook 

func DecodeHook(hook mapstructure.DecodeHookFunc) DecoderConfigOption

DecodeHook, will be called before any decoding and any type conversion (if WeaklyTypedInput is on). This lets you modify the values before they're set down onto the resulting struct. The DecodeHook is called for every map and value in the input. This means that if a struct has embedded fields with squash tags the decode hook is called only once with all of the input data, not once for each embedded struct.

If an error is returned, the entire decode will fail with that error.

Defaults to nothing set.

func ErrorUnset 

func ErrorUnset(unset ...bool) DecoderConfigOption

If ErrorUnset is true, then it is an error for there to exist fields in the result that were not set in the decoding process (extra fields). This only applies to decoding to a struct. This will affect all nested structs as well.

Defaults to false.

func ErrorUnused 

func ErrorUnused(unused ...bool) DecoderConfigOption

If ErrorUnused is true, then it is an error for there to exist keys in the original map that were unused in the decoding process (extra keys).

Defaults to false.

func Exactly 

func Exactly(this mapstructure.DecoderConfig) DecoderConfigOption

Exactly allows setting nearly all the mapstructure.DecoderConfig values to whatever value is desired. A few fields aren't available (Metadata, Squash, Result) but the rest are honored.

This option will mainly be useful in a scope where the code has no idea what options have been set & needs something very specific.

func IgnoreUntaggedFields 

func IgnoreUntaggedFields(ignore ...bool) DecoderConfigOption

IgnoreUntaggedFields ignores all struct fields without explicit TagName, comparable to `mapstructure:"-"` as default behavior.

func MatchName 

func MatchName(fn func(key, field string) bool) DecoderConfigOption

MatchName is the function used to match the map key to the struct field name or tag. Defaults to `strings.EqualFold`. This can be used to implement case-sensitive tag values, support snake casing, etc.

Defaults to nil.

func TagName 

func TagName(name string) DecoderConfigOption

The tag name that mapstructure reads for field names.

This defaults to "mapstructure".

func WeaklyTypedInput 

func WeaklyTypedInput(weak ...bool) DecoderConfigOption

If WeaklyTypedInput is true, the decoder will make the following "weak" conversions:

  • bools to string (true = "1", false = "0")
  • numbers to string (base 10)
  • bools to int/uint (true = 1, false = 0)
  • strings to int/uint (base implied by prefix)
  • int to bool (true if value != 0)
  • string to bool (accepts: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False. Anything else is an error)
  • empty array = empty map and vice versa
  • negative numbers to overflowed uint values (base 10)
  • slice of maps to a merged map
  • single values are converted to slices if required. Each element is weakly decoded. For example: "4" can become []int{4} if the target type is an int slice.

Defaults to false.

func ZeroFields 

func ZeroFields(zero ...bool) DecoderConfigOption

ZeroFields, if set to true, will zero fields before writing them. For example, a map will be emptied before decoded values are put in it. If this is false, a map will be merged.

Defaults to false.

type ExpandOption 

type ExpandOption interface {
	// contains filtered or unexported methods
}

ExpandOption provides the means to configure options around variable expansion.

func WithDelimiters 

func WithDelimiters(start, end string) ExpandOption

WithDelimiters provides a way to define different delimiters for the start and end of a variable for matching purposes.

func WithMaximum 

func WithMaximum(maximum int) ExpandOption

WithMaximum provides a way to overwrite the maximum number of times variables are expanded. Any value less than 1 will default to 10000 as a precaution against getting trapped in an infinite loop.

func WithOrigin 

func WithOrigin(origin string) ExpandOption

WithOrigin provides the origin name to add showing where a value in the configuration tree originates from.

type MarshalOption 

type MarshalOption interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

MarshalOption provides specific configuration for the process of producing a document based on the present information in the goschtalt object.

func FormatAs 

func FormatAs(extension string) MarshalOption

FormatAs specifies the final document format extension to use when performing the operation.

func IncludeOrigins 

func IncludeOrigins(origins ...bool) MarshalOption

IncludeOrigins enables or disables providing the origin for each configuration value present. The default behavior is not to include origins.

func RedactSecrets 

func RedactSecrets(redact ...bool) MarshalOption

RedactSecrets enables the replacement of secret portions of the tree with REDACTED. Passing a redact value of false disables this behavior. The default behavior is to redact secrets.

type Option 

type Option interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

Option configures specific behavior of Config as well as the locations used for the configurations compiled. There are 3 basic groups of options:

  • Configuration of locations where to collect configuration
  • Addition of encoders/decoders
  • Define default behaviors

func AddBuffer 

func AddBuffer(recordName string, in []byte) Option

AddBuffer adds a buffer of bytes for inclusion when compiling the configuration. The format of the bytes is determined by the extension of the recordName field. The recordName field is also used for sorting this configuration value relative to other configuration values.

func AddBufferFn 

func AddBufferFn(recordName string, fn func(recordName string, un UnmarshalFunc) ([]byte, error)) Option

AddBufferFn adds a function that is called during compile time of the configuration. The recordName of this record is passed into the fn function that is called as well as an UnmarshalFunc that represents the existing state of the merged configuration prior to adding the buffer that results in the call to fn.

The format of th ebytes is determined by the extension of the recordName field. The recordName field is also used for sorting this configuration value relative to other configuration values.

func AddDir 

func AddDir(fs fs.FS, path string) Option

AddDir adds a directory (excluding all subdirectories) for inclusion when compiling the configuration. Any files that cannot be processed will be ignored. It is not an error if any files are missing, or if all the files cannot be processed.

Use AddFile() if you need to require a file to be present.

All the files that can be processed with a decoder will be compiled into the configuration.

func AddDirs added in v0.0.2 

func AddDirs(fs fs.FS, paths ...string) Option

AddDirs adds a list of directories (excluding all subdirectories) for inclusion when compiling the configuration. Any files that cannot be processed will be ignored. It is not an error if any files are missing, or if all the files cannot be processed.

Use AddFile() if you need to require a file to be present.

All the files that can be processed with a decoder will be compiled into the configuration.

func AddFile 

func AddFile(fs fs.FS, filename string) Option

AddFile adds exactly one file to the list of files to be compiled into a configuration. The filename must be relative to the fs. If the file specified cannot be processed it is considered an error.

func AddFiles 

func AddFiles(fs fs.FS, filenames ...string) Option

AddFiles adds any number of files to the list of files to be compiled into a configuration. The filenames must be relative to the fs. Any files that cannot be processed will be ignored. It is not an error if any files are missing, or if all the files cannot be processed.

Use AddFile() if you need to require a file to be present.

All the files that can be processed with a decoder will be compiled into the configuration.

func AddTree 

func AddTree(fs fs.FS, path string) Option

AddTree adds a directory tree (including all subdirectories) for inclusion when compiling the configuration. Any files that cannot be processed will be ignored. It is not an error if any files are missing, or if all the files cannot be processed.

Use AddFile() if you need to require a file to be present.

All the files that can be processed with a decoder will be compiled into the configuration.

func AddTrees added in v0.0.2 

func AddTrees(fs fs.FS, paths ...string) Option

AddTrees adds a list of directory trees (including all subdirectories) for inclusion when compiling the configuration. Any files that cannot be processed will be ignored. It is not an error if any files are missing, or if all the files cannot be processed.

Use AddFile() if you need to require a file to be present.

All the files that can be processed with a decoder will be compiled into the configuration.

func AddValue 

func AddValue(recordName, key string, val any, opts ...ValueOption) Option

AddValues provides a simple way to set additional configuration values at runtime.

func AddValueFn 

func AddValueFn(recordName, key string, fn func(recordName string, unmarshal UnmarshalFunc) (any, error), opts ...ValueOption) Option

AddValues provides a simple way to set additional configuration values at runtime via a function call. Note that the provided fn will be called each time the configuration is compiled, allowing the value returned to change if desired.

func AlterKeyCase 

func AlterKeyCase(alter func(string) string) Option

AlterKeyCase defines how the keys should be altered prior to use. This option enables enforcing key case to be all upper case, all lower case, no change or whatever is needed.

Passing nil alter value is interpreted as "do not alter" the key case and is the same as passing: 

func(s string) string { return s }

Examples: 

AlterKeyCase(strings.ToLower)
AlterKeyCase(strings.ToUpper)

func AutoCompile 

func AutoCompile(enable ...bool) Option

AutoCompile instructs New() and With() to also compile the configuration after all the options are applied if enable is true or omitted. Passing an enable value of false disables the extra behavior.

func DefaultMarshalOptions 

func DefaultMarshalOptions(opts ...MarshalOption) Option

DefaultMarshalOptions allows customization of the desired options for all invocations of the Marshal() function. This should make consistent use use of the Marshal() call easier.

func DefaultUnmarshalOptions 

func DefaultUnmarshalOptions(opts ...UnmarshalOption) Option

DefaultUnmarshalOptions allows customization of the desired options for all invocations of the Unmarshal() function. This should make consistent use use of the Unmarshal() call easier.

func DefaultValueOptions 

func DefaultValueOptions(opts ...ValueOption) Option

DefaultValueOptions allows customization of the desired options for all invocations of the TODO:() function. This should make consistent use use of the TODO:() call easier.

func DisableDefaultPackageOptions 

func DisableDefaultPackageOptions() Option

DisableDefaultPackageOptions provides a way to explicitly not use any preconfigured default values by this package and instead use just the options specified.

func Expand 

func Expand(mapper func(string) string, opts ...ExpandOption) Option

Expand provides a way to expand variables in values throughout the configuration tree. Expand() can be called multiple times to expand variables based on additional configurations and mappers.

The initial discovery of a variable to expand in the configuration tree value is determined by the Start and End delimiters options provided. The default delimiters are "${" and "}" respectively. Further expansions of values replaces ${var} or $var in the string based on the mapping function provided.

Expand directives are evaluated in the order specified.

func ExpandEnv 

func ExpandEnv(opts ...ExpandOption) Option

ExpandEnv is a simple way to add automatic environment variable expansion after the configuration has been compiled.

func SetKeyDelimiter 

func SetKeyDelimiter(delimiter string) Option

SetKeyDelimiter provides the delimiter used for determining key parts. A string with length of at least 1 must be provided. The default value is '.'.

func SortRecordsCustomFn 

func SortRecordsCustomFn(less func(a, b string) bool) Option

SortRecordsCustomFn provides a way to specify how you want the files sorted prior to their merge. This function provides a way to provide a completely custom sorting algorithm.

The default is SortRecordsNaturally.

See also: SortRecordsLexically, SortRecordsNaturally

func SortRecordsLexically 

func SortRecordsLexically() Option

SortRecordsLexically provides a built in sorter based on lexical order.

The default is SortRecordsNaturally.

See also: SortRecordsCustomFn, SortRecordsNaturally

func SortRecordsNaturally 

func SortRecordsNaturally() Option

SortRecordsNaturally provides a built in sorter based on natural order. More information about natural sort order: https://en.wikipedia.org/wiki/Natural_sort_order

Notes:

  • Don't use floating point numbers. They are treated like 2 integers separated by the '.' rune.
  • Any leading 0 values are dropped from the number.

Example sort order: 

01_foo.yml
2_foo.yml
98_foo.yml
99 dogs.yml
99_Abc.yml
99_cli.yml
99_mine.yml
100_alpha.yml

The default is SortRecordsNaturally.

See also: SortRecordsCustomFn, SortRecordsLexically

func WithDecoder 

func WithDecoder(d decoder.Decoder) Option

WithDecoder registers a Decoder for the specific file extensions provided. Attempting to register a duplicate extension is not supported.

See also: WithEncoder

func WithEncoder 

func WithEncoder(enc encoder.Encoder) Option

WithEncoder registers a Encoder for the specific file extensions provided. Attempting to register a duplicate extension is not supported.

See also: WithDecoder

func WithError 

func WithError(err error) Option

WithError provides a way for plugins to return an error during option processing. This option will always produce the specified error; including if the err value is nil.

type UnmarshalFunc 

type UnmarshalFunc func(key string, result any, opts ...UnmarshalOption) error

UnmarshalFunc provides a special use Unmarshal() function during AddBufferFn() and AddValueFn() option provided callbacks. This pattern allows the specified function access to the configuration values up to this point. Expansion of any Expand() or ExpandEnv() options is also applied to the configuration tree provided.

type UnmarshalOption 

type UnmarshalOption interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

UnmarshalOption provides specific configuration for the process of producing a document based on the present information in the goschtalt object.

func Optional 

func Optional(optional ...bool) UnmarshalOption

Optional provides a way to allow the requested configuration to not be present and return an empty structure without an error instead of failing. If the optional parameter is not passed, the value is assumed to be true.

The default behavior is to require the request to be present.

See also: Required

func Required 

func Required(required ...bool) UnmarshalOption

Required provides a way to allow the requested configuration to be required and return an error if it is missing. If the optional parameter is not passed, the value is assumed to be true.

The default behavior is to require the request to be present.

See also: Optional

type ValueOption 

type ValueOption interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

ValueOption provides the means to configure options around variable mapping as well as if the specific value being added should be a default or a normal configuration value.

See also DecoderConfigOption which can be used as ValueOption options.

func AsDefault 

func AsDefault(asDefault ...bool) ValueOption

AsDefault specifies that this value is a default value & is applied prior to any other configuration values. Default values are applied in the order the options are specified.

Source Files

View all Source files

Directories

Path Synopsis
examples
simple Module
extensions
cli/simple Module
decoders/cli Module
decoders/env Module
decoders/json Module
decoders/properties Module
decoders/yaml Module
encoders/yaml Module
internal
natsort
pkg
decoder
encoder
meta

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.