config

README

config

config is a package to hold all configuration values for each Flow component. This package centralizes configuration management providing access to the entire FlowConfig and utilities to add a new config value, corresponding CLI flag, and validation.

Package structure

The root config package contains the FlowConfig struct and the default config file default-config.yml. The default-config.yml file is the default configuration that is loaded when the config package is initialized. The default-config.yml is a snapshot of all the configuration values defined for Flow. Each subpackage contains configuration structs and utilities for components and their related subcomponents. These packages also contain the CLI flags for each configuration value. The netconf package is a good example of this pattern. The network component is a large component made of many other large components and subcomponents. Each configuration struct is defined for all of these network related components in the netconf subpackage and CLI flags.

Overriding default values

The entire default config can be overridden using the --config-file CLI flag. When set the config package will attempt to parse the specified config file and override all the values defined. A single default value can be overridden by setting the CLI flag for that specific config. For example --networking-connection-pruning=false will override the default network connection pruning config to false. Override entire config file.

go build -tags relic -o flow-access-node ./cmd/access
./flow-access-node --config-file=config/config.yml

Override a single configuration value.

go build -tags relic -o flow-access-node ./cmd/access
./flow-access-node --networking-connection-pruning=false

Adding a new config value

Adding a new config to the FlowConfig can be done in a few easy steps.

The network package can be used as a good example of how to add CLI flags and will be used in the steps below.

1. Create a new configuration package for the new configuration structs and CLI flags. Although it is encouraged to define all package configuration structs and CLI flags as a subpackage of the config package, the configuration package can live anywhere. This configuration package will define the configuration structs and CLI flags for overriding.

   mkdir example_config 
   

   For the network package we have a configuration subpackage created in network/netconf.

2. Add a new CLI flag for the config value.

   const workersCLIFlag = "app-workers"
   flags.String(workersCLIFlag, 1, "number of app workers")
   

   All network package CLI flags are defined in network/netconf/flags.go in:

   • const block
   • AllFlagNames function
   • InitializeNetworkFlags function

   InitializeNetworkFlags is used during initialization of all flags in the InitializePFlagSet function in the config/base_flags.go.

3. Add the configuration as a new field to an existing configuration struct or create a new configuration struct. Each configuration struct must be a field on the FlowConfig struct so that it is unmarshalled during configuration initialization. Each field on a configuration struct must contain the following field tags.

   1. validate - validate tag is used to perform validation on field structs using the validator package. In the example below you will notice the validate:"gt=0" tag, this will ensure that the value of AppWorkers is greater than 0. The top level FlowConfig struct has a Validate method that performs struct validation. This validation is done with the validator package, each validate tag on every struct field and sub struct field will be validated and validation errors are returned.
   2. mapstructure - mapstructure tag is used for unmarshalling and must match the CLI flag name defined in step or else the field will not be set when the config is unmarshalled.

        type MyComponentConfig struct {
            AppWorkers int `validate:"gt=0" mapstructure:"app-workers"`
        }
   

   It's important to make sure that each mapstructure field tag matches the CLI flag name in config/default-config.yml to avoid parsing errors.

   All network package configuration structs are defined in network/netconf/config.go

4. Add the new configuration properties and default values to config/default-config.yml. Ensure that each new property added matches the mapstructure value of the configuration struct field.

     config-file: "./default-config.yml"
     network-config:
     ...
     my-component:
       app-workers: 1
   

   All network package configuration values are defined under network-config in default-config.yml

5. If a new struct was created in step 3, add it as a new field to FlowConfig struct in config/config.go. In the previous steps we added a new config struct and added a new property to config/default-config.yml for the my-component struct. This property name must match the mapstructure field tag for the struct when added to FlowConfig.

   // FlowConfig Flow configuration.
   type FlowConfig struct {
       ConfigFile    string          `validate:"filepath" mapstructure:"config-file"`
       NetworkConfig *network.Config `mapstructure:"network-config"`
       MyComponentConfig *mypackage.MyComponentConfig `mapstructure:"my-component"`
   }
   

   The network package configuration struct, NetworkConfig, is already embedded as a field in FlowConfig struct. This means that new fields can be added to the network package configuration struct without having to update the FlowConfig struct.

Nested structs

In an effort to keep the configuration yaml structure readable some configuration properties will be defined in nested structs. When this is the case the mapstructure squash tag must be used in the parent struct so that the corresponding nested struct will be flattened before the configuration is unmarshalled. This is used in the network package which is a collection of configuration structs nested in the network.Config struct:

type Config struct {
    UnicastConfig                 `mapstructure:",squash"`
    p2pconf.ResourceManagerConfig `mapstructure:",squash"`
    ConnectionManagerConfig       `mapstructure:",squash"`
    
    p2pconf.GossipSubConfig `mapstructure:",squash"`
    AlspConfig              `mapstructure:",squash"`
    ...
}

Each nested struct must have the mapstructure squash tag so that the nested struct will be flattened before unmarshalling. The nesting can be as deep as needed. For example, the UnicastConfig struct in the Config struct has a nested UnicastRateLimitersConfig struct that also uses the squash tag:

type UnicastConfig struct {
    // UnicastRateLimitersConfig configuration for all unicast rate limiters.
    UnicastRateLimitersConfig `mapstructure:",squash"`
    ...
}

Setting Aliases

Most configs will not be defined on the top layer FlowConfig but instead be defined on nested structs and in nested properties of the configuration yaml. When the default config is initially loaded the underlying config viper store will store each configuration with a key that is prefixed with each parent property. For example, because networking-connection-pruning is found on the network-config property of the configuration yaml, the key used by the config store to store this config value will be prefixed with network-config e.g. network-config.networking-connection-pruning

Later in the config process we bind the underlying config store with our pflag set, this allows us to override default values using CLI flags. At this time the underlying config store would have 2 separate keys networking-connection-pruning and network-config.networking-connection-pruning for the same configuration value. This is because we don't use the network prefix for the CLI flags used to override network configs. As a result, an alias must be set from network-config.networking-connection-pruning -> networking-connection-pruning so that they both point to the value loaded from the CLI flag. See SetAliases in network/netconf/flags.go in the network package for a reference.

Documentation

Index

• func BindPFlags(c *FlowConfig, flags *pflag.FlagSet) (bool, error)
• func InitializePFlagSet(flags *pflag.FlagSet, config *FlowConfig)
• func LogConfig(logger *zerolog.Event, flags *pflag.FlagSet) map[string]struct{}
• func Unmarshall(flowConfig *FlowConfig) error
• type FlowConfig
  • func DefaultConfig() (*FlowConfig, error)
  • func (fc *FlowConfig) Validate() error

Functions

func BindPFlags

func BindPFlags(c *FlowConfig, flags *pflag.FlagSet) (bool, error)

BindPFlags binds the configuration to the cli pflag set. This should be called after all pflags have been parsed. If the --config-file flag has been set the config will be loaded from the specified config file. Args:

c: The Flow configuration that will be used to unmarshall the configuration values into after binding pflags.
This needs to be done because pflags may override a configuration value.

Returns:

error: if there is any error encountered binding pflags or unmarshalling the config struct, all errors are considered irrecoverable.
bool: true if --config-file flag was set and config file was loaded, false otherwise.

Note: As configuration management is improved, this func should accept the entire Flow config as the arg to unmarshall new config values into.

func InitializePFlagSet

func InitializePFlagSet(flags *pflag.FlagSet, config *FlowConfig)

InitializePFlagSet initializes all CLI flags for the Flow node base configuration on the provided pflag set. Args:

*pflag.FlagSet: the pflag set of the Flow node.
*FlowConfig: the config used to set default values on the flags

Note: in subsequent PR's all flag initialization for Flow node should be moved to this func.

func LogConfig

func LogConfig(logger *zerolog.Event, flags *pflag.FlagSet) map[string]struct{}

LogConfig logs configuration keys and values if they were overridden with a config file. It also returns a map of keys for which the values were set by a config file.

Parameters:

• logger: *zerolog.Event to which the configuration keys and values will be logged.
• flags: *pflag.FlagSet containing the set flags.

Returns:

• map[string]struct{}: map of keys for which the values were set by a config file.

func Unmarshall

func Unmarshall(flowConfig *FlowConfig) error

Unmarshall unmarshalls the Flow configuration into the provided FlowConfig struct. Args:

flowConfig: the flow config struct used for unmarshalling.

Returns:

error: if there is any error encountered unmarshalling the configuration, all errors are considered irrecoverable.

Types

type FlowConfig

type FlowConfig struct {
	// ConfigFile used to set a path to a config.yml file used to override the default-config.yml file.
	ConfigFile    string          `validate:"filepath" mapstructure:"config-file"`
	NetworkConfig *netconf.Config `mapstructure:"network-config"`
}

FlowConfig Flow configuration.

func DefaultConfig

func DefaultConfig() (*FlowConfig, error)

DefaultConfig initializes the flow configuration. All default values for the Flow configuration are stored in the default-config.yml file. These values can be overridden by node operators by setting the corresponding cli flag. DefaultConfig should be called before any pflags are parsed, this will allow the configuration to initialize with defaults from default-config.yml. Returns:

*FlowConfig: an instance of the network configuration fully initialized to the default values set in the config file
error: if there is any error encountered while initializing the configuration, all errors are considered irrecoverable.

func (*FlowConfig) Validate

func (fc *FlowConfig) Validate() error

Validate checks validity of the Flow config. Errors indicate that either the configuration is broken, incompatible with the node's internal state, or that the node's internal state is corrupted. In all cases, continuation is impossible.