configstore

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 12, 2019 License: BSD-3-Clause Imports: 16 Imported by: 90

README

configstore

The configstore library aims to facilitate configuration discovery and management. It mixes configuration items coming from various (abstracted) data sources, called providers.

GoDoc Go Report Card

Items

An item is composed of 3 fields:

  • Key: The name of the item. Does not have to be unique. The provider is responsible for giving a sensible initial value.
  • Value: The content of the item. This can be either manipulated as a plain scalar string, or as a marshaled (JSON or YAML) blob for complex objects.
  • Priority: An abstract integer value to use when priorizing between items sharing the same key. The provider is responsible for giving a sensible initial value.

Configuration format

The item keys are NOT case-sensitive. Also, - and _ characters are equivalent.

The exact input format of the configuration depends on the provider. Providers can either be loaded manually in your code, or controlled by the env variable CONFIGURATION_FROM.

Example main.go
func main() {
    configstore.InitFromEnvironment()

    val, err := configstore.GetItemValue("foo")
    if err != nil {
        panic(err)
    }
    fmt.Println(val)
}

Outputs:

bar
Reading from a file

Env:

CONFIGURATION_FROM=file:foo.cfg

Contents of foo.cfg file:

- key: foo
  priority: 12
  value: bar

Key/value pairs are read from a single file in yaml.

Reading from env

Env:

CONFIGURATION_FROM=env:CONFIG
CONFIG_FOO=bar

Key/value pairs are read from the environment, with an optional prefix. Remember that key names are case-insensitive, and that _ and - are equivalent in key names.

Reading from a file hierarchy

Env:

CONFIGURATION_FROM=filetree:configdir

Contents of configdir directory:

foo

Contents of configdir/foo file:

bar

Key/value pairs are read by traversing a root directory. Each file in the dir represents an item: the filename is the key, the contents are the value. To have several items sharing the same key, you can use a single level of sub-directory as such: configdir/foo/bar1, configdir/foo/bar2, ... The filenames bar1/bar2 are not used in the resulting items.

Reading from a custom source

These built-in providers implement common sources of configuration, but configstore can be expanded with other data sources. See Example: multiple providers.

Example 101

file.txt:

- key: foo
  value: bar
- key: baz
  value: bazz

func main() {
    configstore.File("/path/to/file.txt")
    v, err := configstore.GetItemValue("foo")
    fmt.Println(v, err)
}

This very basic example describes how to get a string out of a configuration file (which can be JSON or YAML). To do more advanced configuration manipulation, see the next examples.

Example: multiple providers

Configuration Providers represent an abstract data source. Their only role is to return a list of items.

Some built-in implementations are available (in-memory, file, env, ...), but the library exposes a way to register a provider factory, to extend it and bridge with any other existing system.

Example mixing several providers

// custom provider with hardcoded values
func MyProviderFunc() (configstore.ItemList, error) {
    ret := configstore.ItemList{
        Items: []configstore.Item{
            // an item has 3 components: key, value, priority
            // they are defined by the provider, but can be modified later by the library user
            configstore.NewItem("key1", `value1-higher-prio`, 6),
            configstore.NewItem("key1", `value1-lower-prio`, 5),
            configstore.NewItem("key2", `value2`, 5),
        },
    }
    return ret, nil
}

func main() {

    configstore.RegisterProvider("myprovider", MyProviderFunc)
    configstore.File("/path/to/file.txt")
    configstore.Env("CONFIG_")

    // blends items from all sources
    items, err := configstore.GetItemList()
    if err != nil {
        panic(err)
    }

    for _, i := range items.Items {
        val, err := i.Value()
        if err != nil {
            panic(err)
        }
        fmt.Println(i.Key(), val, i.Priority())
    }
}

Example: advanced filtering

When calling configstore.GetItemList(), the caller gets an ItemList.

This object contains all the configuration items. To manipulate it, you can use a ItemFilter object, which provides convenient helper functions to select and reorder the items.

All objects are safe to use even when the item list is empty.

Example of use:

func main() {
    items, err := configstore.GetItemList()
    if err != nil {
        panic(err)
    }

    // we start by building a filter to manipulate our configuration items
    // we will apply it on our items list later
    filter := configstore.Filter()

    // get the databases
    filter = filter.Slice("database")

    // now we have a list of database objects, let's assume the payload resembles this:
    // {"name": "foo", "ip": "192.168.0.1", "port": 5432, "type": "RO"}
    // {"name": "foo", "ip": "192.168.0.1", "port": 5433, "type": "RW"}
    // {"name": "bar", "ip": "192.168.0.1", "port": 5434, "type": "RO"}
    //
    // the "database" initial key provides too little information to extract the data relating to a specific DB
    // we need to drill down into the value

    // we need to unmarshal the JSON representation of the whole sublist
    // we pass a factory function that instantiates objects of the correct concrete type
    filter = filter.Unmarshal(func() interface{} { return &Database{} })

    // now we want to actually index and lookup by database name, instead of the generic "database"
    // we apply a rekey function that does payload inspection
    filter = filter.Rekey(rekeyByName)

    // we have duplicate elements: database "foo" is present twice
    // we want to favor the RW instance if possible
    // we apply a reordering function that does payload inspection
    filter = filter.Reorder(prioritizeRW)

    // we only need 1 of each, we squash to only keep the single highest priority of each key
    filter = filter.Squash()

    // now we have only 2 items left:
    // {"name": "foo", "ip": "192.168.0.1", "port": 5433, "type": "RW"}
    // {"name": "bar", "ip": "192.168.0.1", "port": 5434, "type": "RO"}

    // we can finally apply it on our list
    items = filter.Apply(items)

    // the same thing, more concise:
    filter = configstore.Filter().Slice("database").Unmarshal(func() interface{} { return &Database{} }).Rekey(rekeyByName).Reorder(prioritizeRW).Squash()
    items, err = filter.GetItemList() // shortcut: applies the filter to the full list from configstore.GetItemList()
    if err != nil {
        panic(err)
    }
    // declaring your filter separately like this lets you define it globally and execute it later
    // that way, you can use its description (String()) to generate usage information.
}

type Database struct {
    Name string `json:"name"`
    IP   string `json:"ip"`
    Port int    `json:"port"`
    Type string `json:"type"`
}

func rekeyByName(s *configstore.Item) string {
    i, err := s.Unmarshaled()
    // we see here the error that was produced when we called *ItemList.Unmarshal(...)*
    // we ignore it for now, it will be handled when the *main()* retrieves the object.
    if err == nil {
        return i.(*Database).Name
    }
    return s.Key()
}

func prioritizeRW(s *configstore.Item) int64 {
    i, err := s.Unmarshaled()
    if err == nil {
        if i.(*Database).Type == "RW" {
            return s.Priority() + 1
        }
    }
    return s.Priority()
}

Documentation

Index

Constants

View Source 
const (
	// ConfigEnvVar defines the environment variable used to set up the configuration providers via InitFromEnvironment
	ConfigEnvVar = "CONFIGURATION_FROM"
)

Variables

This section is empty.

Functions

func AllowProviderOverride 

func AllowProviderOverride()

AllowProviderOverride allows multiple calls to RegisterProvider() with the same provider name. This is useful for controlled test cases, but is not recommended in the context of a real application.

func Env added in v0.2.0 

func Env(prefix string)

func ErrorProvider 

func ErrorProvider(name string, err error)

ErrorProvider registers a configstore provider which always returns an error.

func File 

func File(filename string)

File registers a configstore provider which reads from the file given in parameter (static content).

func FileCustom 

func FileCustom(filename string, fn func([]byte) ([]Item, error))

FileCustom registers a configstore provider which reads from the file given in parameter, and loads the content using the given unmarshal function

func FileCustomRefresh 

func FileCustomRefresh(filename string, fn func([]byte) ([]Item, error))

FileCustomRefresh registers a configstore provider which reads from the file given in parameter, and loads the content using the given unmarshal function; and watches file stat for auto refresh

func FileList 

func FileList(dirname string)

FileList registers a configstore provider which reads from the files contained in the directory given in parameter. The content of the files should be JSON/YAML similar to the File provider.

func FileRefresh 

func FileRefresh(filename string)

FileRefresh registers a configstore provider which readfs from the file given in parameter (provider watches file stat for auto refresh, watchers get notified).

func FileTree 

func FileTree(dirname string)

FileTree registers a configstore provider which reads from the files contained in the directory given in parameter. A limited hierarchy is supported: files can either be top level (in which case the file name will be used as the item key), or nested in a single sub-directory (in which case the sub-directory name will be used as item key for all the files contained in it). The content of the files should be the plain data, with no envelope. Capitalization can be used to indicate item priority for sub-directories containing multiple items which should be differentiated. Capitalized = higher priority.

func GetItemValue 

func GetItemValue(key string) (string, error)

GetItemValue fetches the full item list, merging the results from all providers, then returns a single item's value by key.

func GetItemValueBool 

func GetItemValueBool(key string) (bool, error)

GetItemValueBool fetches the full item list, merging the results from all providers, then returns a single item's value by key.

func GetItemValueDuration 

func GetItemValueDuration(key string) (time.Duration, error)

GetItemValueDuration fetches the full item list, merging the results from all providers, then returns a single item's value by key.

func GetItemValueFloat 

func GetItemValueFloat(key string) (float64, error)

GetItemValueFloat fetches the full item list, merging the results from all providers, then returns a single item's value by key.

func GetItemValueInt 

func GetItemValueInt(key string) (int64, error)

GetItemValueInt fetches the full item list, merging the results from all providers, then returns a single item's value by key.

func GetItemValueUint 

func GetItemValueUint(key string) (uint64, error)

GetItemValueUint fetches the full item list, merging the results from all providers, then returns a single item's value by key.

func InitFromEnvironment 

func InitFromEnvironment()

InitFromEnvironment initializes configuration providers via their name and an optional argument. Suitable provider factories should have been registered via RegisterProviderFactory for this to work. Built-in providers (File, FileList, FileTree, ...) are registered by default.

Valid example: CONFIGURATION_FROM=file:/etc/myfile.conf,file:/etc/myfile2.conf,filelist:/home/foobar/configs

func NotifyIsMuted added in v0.1.5 

func NotifyIsMuted() bool

NotifyIsMuted reports whether notifications are currently muted.

func NotifyMute added in v0.1.5 

func NotifyMute()

NotifyMute prevents configstore from notifying watchers on configuration changes, until MotifyUnmute() is called.

func NotifyUnmute added in v0.1.5 

func NotifyUnmute()

NotifyUnmute allows configstore to resume notifications to watchers on configuration changes. This will trigger a notification to catch up any change done during the time spent mute.

func NotifyWatchers 

func NotifyWatchers()

NotifyWatchers is used by providers to notify of configuration changes. It unblocks all the watchers which are ranging over a watch channel.

func RegisterProvider 

func RegisterProvider(name string, f Provider)

RegisterProvider registers a provider

func RegisterProviderFactory 

func RegisterProviderFactory(name string, f func(string))

RegisterProviderFactory registers a factory function so that InitFromEnvironment can properly instantiate configuration providers via name + argument.

func Watch 

func Watch() chan struct{}

Watch returns a channel which you can range over. You will get unblocked every time a provider notifies of a configuration change.

Types

type ErrAmbiguousItem 

type ErrAmbiguousItem string

func (ErrAmbiguousItem) Error 

func (e ErrAmbiguousItem) Error() string

type ErrItemNotFound 

type ErrItemNotFound string

func (ErrItemNotFound) Error 

func (e ErrItemNotFound) Error() string

type ErrProvider 

type ErrProvider string

func (ErrProvider) Error 

func (e ErrProvider) Error() string

type ErrUninitializedItemList 

type ErrUninitializedItemList string

func (ErrUninitializedItemList) Error 

func (e ErrUninitializedItemList) Error() string

type InMemoryProvider 

type InMemoryProvider struct {
	// contains filtered or unexported fields
}

InMemoryProvider implements an in-memory configstore provider.

func InMemory 

func InMemory(name string) *InMemoryProvider

InMemory registers an InMemoryProvider with a given arbitrary name and returns it. You can append any number of items to it, see Add().

func (*InMemoryProvider) Add 

func (inmem *InMemoryProvider) Add(s ...Item) *InMemoryProvider

Add appends an item to the in-memory list.

func (*InMemoryProvider) Items 

func (inmem *InMemoryProvider) Items() (ItemList, error)

Items returns the in-memory item list. This is the function that gets called by configstore.

type Item 

type Item struct {
	// contains filtered or unexported fields
}

Item is a key/value pair with a priority attached. The initial priority is set by the provider, but can be modified (see Reorder).

func GetItem 

func GetItem(key string) (Item, error)

GetItem retrieves the full item list, merging the results from all providers, then returns a single item by key. If 0 or >=2 items are present with that key, it will return an error.

func NewItem 

func NewItem(key, value string, priority int64) Item

NewItem creates a item object from key / value / priority values. It is meant to be used by provider implementations.

func (*Item) Key 

func (s *Item) Key() string

Key returns the item key.

func (Item) Priority 

func (s Item) Priority() int64

Priority returns the item priority.

func (*Item) UnmarshalJSON 

func (s *Item) UnmarshalJSON(b []byte) error

UnmarshalJSON respects json.Unmarshaler

func (Item) Unmarshaled 

func (s Item) Unmarshaled() (interface{}, error)

Unmarshaled returns the unmarshaled object produced by ItemFilter.Unmarshal, along with any error that was encountered in list processing (unmarshal, transform).

func (Item) Value 

func (s Item) Value() (string, error)

Value returns the item value, along with any error that was encountered in list processing (unmarshal, transform).

func (Item) ValueBool 

func (s Item) ValueBool() (bool, error)

ValueBool returns the item value, along with any error that was encountered in list processing (unmarshal, transform).

func (Item) ValueBytes 

func (s Item) ValueBytes() ([]byte, error)

ValueBytes returns the item value, along with any error that was encountered in list processing (unmarshal, transform). Data to be returned should be base64 encoded

func (Item) ValueDuration 

func (s Item) ValueDuration() (time.Duration, error)

ValueDuration returns the item value, along with any error that was encountered in list processing (unmarshal, transform).

func (Item) ValueFloat 

func (s Item) ValueFloat() (float64, error)

ValueFloat returns the item value, along with any error that was encountered in list processing (unmarshal, transform).

func (Item) ValueInt 

func (s Item) ValueInt() (int64, error)

ValueInt returns the item value, along with any error that was encountered in list processing (unmarshal, transform).

func (Item) ValueUint 

func (s Item) ValueUint() (uint64, error)

ValueUint returns the item value, along with any error that was encountered in list processing (unmarshal, transform).

type ItemFilter 

type ItemFilter struct {
	// contains filtered or unexported fields
}

ItemFilter holds a list of manipulation steps to operate on an item list. It can be declared globally then used/applied on specific item lists later. By declaring it before actual use, you can make it available to other packages which can then use it to describe your filters / configuration (e.g. main for usage). See String().

func Filter 

func Filter() *ItemFilter

Filter creates a new empty filter object.

func (*ItemFilter) Apply 

func (s *ItemFilter) Apply(items *ItemList) *ItemList

Apply applies the filter on an existing item list.

func (*ItemFilter) GetFirstItem 

func (s *ItemFilter) GetFirstItem() (Item, error)

GetFirstItem fetches the full item list, applies the filter, then returns the first item of the list.

func (*ItemFilter) GetItem 

func (s *ItemFilter) GetItem(key string) (Item, error)

GetItem fetches the full item list, applies the filter, then returns a single item by key.

func (*ItemFilter) GetItemList 

func (s *ItemFilter) GetItemList() (*ItemList, error)

GetItemList fetches the full item list, applies the filter, and returns the result.

func (*ItemFilter) GetItemValue 

func (s *ItemFilter) GetItemValue(key string) (string, error)

GetItemValue fetches the full item list, applies the filter, then returns a single item's value by key.

func (*ItemFilter) GetItemValueBool 

func (s *ItemFilter) GetItemValueBool(key string) (bool, error)

GetItemValueBool fetches the full item list, applies the filter, then returns a single item's value by key.

func (*ItemFilter) GetItemValueDuration 

func (s *ItemFilter) GetItemValueDuration(key string) (time.Duration, error)

GetItemValueDuration fetches the full item list, applies the filter, then returns a single item's value by key.

func (*ItemFilter) GetItemValueFloat 

func (s *ItemFilter) GetItemValueFloat(key string) (float64, error)

GetItemValueFloat fetches the full item list, applies the filter, then returns a single item's value by key.

func (*ItemFilter) GetItemValueInt 

func (s *ItemFilter) GetItemValueInt(key string) (int64, error)

GetItemValueInt fetches the full item list, applies the filter, then returns a single item's value by key.

func (*ItemFilter) GetItemValueUint 

func (s *ItemFilter) GetItemValueUint(key string) (uint64, error)

GetItemValueUint fetches the full item list, applies the filter, then returns a single item's value by key.

func (*ItemFilter) MustGetFirstItem 

func (s *ItemFilter) MustGetFirstItem() Item

MustGetFirstItem is similar to GetFirstItem, but always returns an Item object. The eventual error is stored inside, and returned when accessing the item's value. Useful for chaining calls.

func (*ItemFilter) MustGetItem 

func (s *ItemFilter) MustGetItem(key string) Item

MustGetItem is similar to GetItem, but always returns an Item object. The eventual error is stored inside, and returned when accessing the item's value. Useful for chaining calls.

func (*ItemFilter) Rekey 

func (s *ItemFilter) Rekey(rekeyF func(*Item) string) *ItemFilter

Rekey modifies item keys. The function parameter is called for each item in the item list, and the returned string is used as the new key.

func (*ItemFilter) Reorder 

func (s *ItemFilter) Reorder(reorderF func(*Item) int64) *ItemFilter

Reorder modifies item priority. The function parameter is called for each item in the item list, and the returned integer is used as the new priority.

func (*ItemFilter) Slice 

func (s *ItemFilter) Slice(key string, keyF ...func(string) string) *ItemFilter

Slice filters the list items, keeping only those matching key. You can optionally pass a list of modifier functions, to be invoked when applying the filter.

func (*ItemFilter) Squash 

func (s *ItemFilter) Squash() *ItemFilter

Squash filters the items in the item list, keeping only the items with the highest priority for each key.

func (*ItemFilter) String 

func (s *ItemFilter) String() string

String returns a description of the filter.

func (*ItemFilter) Transform 

func (s *ItemFilter) Transform(transformF func(*Item) (string, error)) *ItemFilter

Transform modifies item values. The function parameter is called for each item in the item list, and the returned string + error are the values which will be returned by item.Value().

func (*ItemFilter) Unmarshal 

func (s *ItemFilter) Unmarshal(f func() interface{}) *ItemFilter

Unmarshal tries to unmarshal (from JSON or YAML) all the items in the item list into objects returned by the factory f(). The results and errors will be stored to be handled later. See item.Unmarshaled().

type ItemList 

type ItemList struct {
	Items []Item
	// contains filtered or unexported fields
}

ItemList is a list of items which can be manipulated by an ItemFilter

func GetItemList 

func GetItemList() (*ItemList, error)

GetItemList retrieves the full item list, merging the results from all providers. It does NOT cache, it's the responsability of the providers to keep an in-ram representation if desired.

func (*ItemList) GetItem 

func (s *ItemList) GetItem(key string) (Item, error)

GetItem returns a single item, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) GetItemValue 

func (s *ItemList) GetItemValue(key string) (string, error)

GetItemValue returns a single item value, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) GetItemValueBool 

func (s *ItemList) GetItemValueBool(key string) (bool, error)

GetItemValueBool returns a single item value, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) GetItemValueDuration 

func (s *ItemList) GetItemValueDuration(key string) (time.Duration, error)

GetItemValueDuration returns a single item value, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) GetItemValueFloat 

func (s *ItemList) GetItemValueFloat(key string) (float64, error)

GetItemValueFloat returns a single item value, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) GetItemValueInt 

func (s *ItemList) GetItemValueInt(key string) (int64, error)

GetItemValueInt returns a single item value, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) GetItemValueUint 

func (s *ItemList) GetItemValueUint(key string) (uint64, error)

GetItemValueUint returns a single item value, by key. If 0 or >=2 items are present with that key, it will return an error.

func (*ItemList) Keys 

func (s *ItemList) Keys() []string

Keys returns a list of the different keys present in the item list.

func (*ItemList) Len 

func (s *ItemList) Len() int

Implements sort.Interface. NOT CONCURRENT SAFE.

func (*ItemList) Less 

func (s *ItemList) Less(i, j int) bool

Implements sort.Interface NOT CONCURRENT SAFE.

func (*ItemList) Swap 

func (s *ItemList) Swap(i, j int)

Implements sort.Interface NOT CONCURRENT SAFE.

type Provider 

type Provider func() (ItemList, error)

A Provider retrieves config items and makes them available to the configstore, Their implementations can vary wildly (HTTP API, file, env, hardcoded test, ...) and their results will get merged by the configstore library. It's the responsability of the application using configstore to register suitable providers.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.