config

README

config

config is a package to hold all configuration values for each Flow component. This package centralizes configuration management providing access to the entire FlowConfig and utilities to add a new config value, corresponding CLI flag, and validation.

Package structure

The root config package contains the FlowConfig struct and the default config file default-config.yml. The default-config.yml file is the default configuration that is loaded when the config package is initialize. The default-config.yml is a snapshot of all the configuration values defined for Flow. Each subpackage contains configuration structs and utilities for components and their related subcomponents. These packages also contain the CLI flags for each configuration value. The network package is a good example of this pattern. The network component is a large component made of many other large components and subcomponents. Each configuration struct is defined for all of these network related components in the network subpackage and CLI flags.

Overriding default values

The entire default config can be overridden using the --config-file CLI flag. When set the config package will attempt to parse the specified config file and override all the values defined. A single default value can be overridden by setting the CLI flag for that specific config. For example --network-connection-pruning=false will override the default network connection pruning config to false. Override entire config file.

go build -tags relic -o flow-access-node ./cmd/access
./flow-access-node --config-file=config/config.yml

Override a single configuration value.

go build -tags relic -o flow-access-node ./cmd/access
./flow-access-node --network-connection-pruning=false

Adding a new config value

Adding a new config to the FlowConfig can be done in a few easy steps.

1. Create a new subpackage in the config package for the new configuration structs to live. Although it is encouraged to put all configuration sub-packages in the config package so that configuration can be updated in one place these sub-packages can live anywhere. This package will define the configuration structs and CLI flags for overriding.

   mkdir example_config 
   

2. Add a new CLI flag for the config value.

   const workersCLIFlag = "app-workers"
   flags.String(workersCLIFlag, 1, "number of app workers")
   

   The network package can be used as a good example of how to structure CLI flag initialization. All flags are initialized in a single function InitializeNetworkFlags, this function is then used during flag initialization of the config package.
3. Add the config as a new field to an existing configuration struct or create a new one. Each configuration struct must be a field on the FlowConfig struct so that it is unmarshalled during configuration initialization. Each field on a configuration struct must contain the following field tags.
   1. validate - validate tag is used to perform validation on field structs using the validator package. In the example below you will notice the validate:"gt=0" tag, this will ensure that the value of AppWorkers is greater than 0. The top level FlowConfig struct has a Validate method that performs struct validation. This validation is done with the validator package, each validate tag on ever struct field and sub struct field will be validated and validation errors are returned.
   2. mapstructure - mapstructure tag is used for unmarshalling and must match the CLI flag name defined in step or else the field will not be set when the config is unmarshalled.

        type MyComponentConfig struct {
            AppWorkers int `validate:"gt=0" mapstructure:"app-workers"`
        }
   

   It's important to make sure that the CLI flag name matches the mapstructure field tag to avoid parsing errors.
4. Add the new config and a default value to the default-config.yml file. Ensure that the new property added matches the configuration struct structure for the subpackage the config belongs to.

     config-file: "./default-config.yml"
     network-config:
     ...
     my-component:
       app-workers: 1
   

5. Finally, if a new struct was created add it as a new field to the FlowConfig. In the previous steps we added a new config struct and added a new property to the default-config.yml for this struct my-component. This property name must match the mapstructure field tag for the struct when added to the FlowConfig.

   // FlowConfig Flow configuration.
   type FlowConfig struct {
       ConfigFile    string          `validate:"filepath" mapstructure:"config-file"`
       NetworkConfig *network.Config `mapstructure:"network-config"`
       MyComponentConfig *mypackage.MyComponentConfig `mapstructure:"my-component"`
   }
   

Nested structs

In an effort to keep the configuration yaml structure readable some configuration will be in nested properties. When this is the case the mapstructure squash tag can be used so that the corresponding nested struct will be flattened before the configuration is unmarshalled. This is used in the network package which is a collection of configuration structs nested on the network.Config struct.

type Config struct {
    // UnicastRateLimitersConfig configuration for all unicast rate limiters.
    UnicastRateLimitersConfig `mapstructure:",squash"`
    ...
}

UnicastRateLimitersConfig is a nested struct that defines configuration for unicast rate limiter component. In our configuration yaml structure you will see that all network configs are defined under the network-config property.

Setting Aliases

Most configs will not be defined on the top layer FlowConfig but instead be defined on nested structs and in nested properties of the configuration yaml. When the default config is initially loaded the underlying config viper store will store each configuration with a key that is prefixed with each parent property. For example, because network-connection-pruning is found on the network-config property of the configuration yaml, the key used by the config store to store this config value will be prefixed with network e.g. network.network-connection-pruning

Later in the config process we bind the underlying config store with our pflag set, this allows us to override default values using CLI flags. At this time the underlying config store would have 2 separate keys network-connection-pruning and network.network-connection-pruning for the same configuration value. This is because we don't use the network prefix for the CLI flags used to override network configs. As a result, an alias must be set from network.network-connection-pruning -> network-connection-pruning so that they both point to the value loaded from the CLI flag. See SetAliases in the network package for a reference.

Documentation

Index

• func BindPFlags(c *FlowConfig, flags *pflag.FlagSet) (bool, error)
• func InitializePFlagSet(flags *pflag.FlagSet, config *FlowConfig)
• func LogConfig(logger *zerolog.Event, flags *pflag.FlagSet) map[string]struct{}
• func Unmarshall(flowConfig *FlowConfig) error
• type FlowConfig
  • func DefaultConfig() (*FlowConfig, error)
  • func (fc *FlowConfig) Validate() error

Functions

func BindPFlags

func BindPFlags(c *FlowConfig, flags *pflag.FlagSet) (bool, error)

BindPFlags binds the configuration to the cli pflag set. This should be called after all pflags have been parsed. If the --config-file flag has been set the config will be loaded from the specified config file. Args:

c: The Flow configuration that will be used to unmarshall the configuration values into after binding pflags.
This needs to be done because pflags may override a configuration value.

Returns:

error: if there is any error encountered binding pflags or unmarshalling the config struct, all errors are considered irrecoverable.
bool: true if --config-file flag was set and config file was loaded, false otherwise.

Note: As configuration management is improved, this func should accept the entire Flow config as the arg to unmarshall new config values into.

func InitializePFlagSet

func InitializePFlagSet(flags *pflag.FlagSet, config *FlowConfig)

InitializePFlagSet initializes all CLI flags for the Flow node base configuration on the provided pflag set. Args:

*pflag.FlagSet: the pflag set of the Flow node.
*FlowConfig: the config used to set default values on the flags

Note: in subsequent PR's all flag initialization for Flow node should be moved to this func.

func LogConfig

func LogConfig(logger *zerolog.Event, flags *pflag.FlagSet) map[string]struct{}

LogConfig logs configuration keys and values if they were overridden with a config file. It also returns a map of keys for which the values were set by a config file.

Parameters:

• logger: *zerolog.Event to which the configuration keys and values will be logged.
• flags: *pflag.FlagSet containing the set flags.

Returns:

• map[string]struct{}: map of keys for which the values were set by a config file.

func Unmarshall

func Unmarshall(flowConfig *FlowConfig) error

Unmarshall unmarshalls the Flow configuration into the provided FlowConfig struct. Args:

flowConfig: the flow config struct used for unmarshalling.

Returns:

error: if there is any error encountered unmarshalling the configuration, all errors are considered irrecoverable.

Types

type FlowConfig

type FlowConfig struct {
	// ConfigFile used to set a path to a config.yml file used to override the default-config.yml file.
	ConfigFile    string          `validate:"filepath" mapstructure:"config-file"`
	NetworkConfig *netconf.Config `mapstructure:"network-config"`
}

FlowConfig Flow configuration.

func DefaultConfig

func DefaultConfig() (*FlowConfig, error)

DefaultConfig initializes the flow configuration. All default values for the Flow configuration are stored in the default-config.yml file. These values can be overridden by node operators by setting the corresponding cli flag. DefaultConfig should be called before any pflags are parsed, this will allow the configuration to initialize with defaults from default-config.yml. Returns:

*FlowConfig: an instance of the network configuration fully initialized to the default values set in the config file
error: if there is any error encountered while initializing the configuration, all errors are considered irrecoverable.

func (*FlowConfig) Validate

func (fc *FlowConfig) Validate() error

Validate checks validity of the Flow config. Errors indicate that either the configuration is broken, incompatible with the node's internal state, or that the node's internal state is corrupted. In all cases, continuation is impossible.