econfig

package
v1.4.22 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 28, 2023 License: BSD-3-Clause Imports: 13 Imported by: 18

README

Docs: GoDoc

econfig provides methods to set values on a Config struct through a (TOML) config file or command-line args (flags in Go terminology), with support for setting Network params and values on any other struct as well (e.g., an Env to be constructed later in a ConfigEnv method).

  • Standard usage:

    • cfg := &ss.Config
    • cfg.Defaults() -- sets hard-coded defaults -- user should define and call this method first. It is better to use the def: field tag however because it then shows in -h or --help usage and in the GoGi GUI.
    • econfig.Config(cfg, "config.toml") -- sets config values according to the standard order, with given file name specifying the default config file name.

  • Has support for nested Include paths, which are processed in the natural deepest-first order. The processed Config struct field will contain a list of all such files processed. Config must implement the IncludesPtr() *[]string method which satisfies the Includer interface, and returns a pointer to an Includes []string field containing a list of config files to include. The default IncludePaths includes current dir (.) and configs directory, which is recommended location to store different configs.

  • Order of setting in econfig.Config:

    • Apply any def: field tag default values.
    • Look for --config, --cfg arg, specifying config file(s) on the command line (comma separated if multiple, with no spaces).
    • Fall back on default config file name passed to Config function, if arg not found.
    • Read any Include[s] files in config file in deepest-first (natural) order, then the specified config file last -- includee overwrites included settings.
    • Process command-line args based on Config field names, with . separator for sub-fields.

  • All field name references in toml files and command-line args are case-insensitive. For args (flags) kebab-case (with either - or _ delimiter) can be used. For bool args, use "No" prefix in any form (e.g., "NoRunLog" or "no-run-log"). Instead of polluting the flags space with all the different options, custom args processing code is used.

  • Args in sub-structs are automatically available with just the field name and also nested within the name of the parent struct field -- for example, -Run.NEpochs and just -NEpochs (or -nepochs lowercase). Use nest:"+" to force a field to only be available in its nested form, in case of conflict of names without nesting (which are logged).

  • Is a replacement for ecmd and includes the helper methods for saving log files etc.

  • A map[string]any type can be used for deferred raw params to be applied later (Network, Env etc). Example: Network = {'.PFCLayer:Layer.Inhib.Layer.Gi' = '2.4', '#VSPatchPrjn:Prjn.Learn.LRate' = '0.01'} where the key expression contains the params selector : path to variable.

  • Supports full set of Open (file), OpenFS (takes fs.FS arg, e.g., for embedded), Read (bytes) methods for loading config files. The overall Config() version uses OpenWithIncludes which processes includes -- others are just for single files. Also supports Write and Save methods for saving from current state.

  • If needed, different config file encoding formats can be supported, with TOML being the default (currently only TOML).

Special fields, supported types, and field tags

  • To enable include file processing, add a Includes []string field and a func (cfg *Config) IncludesPtr() *[]string { return &cfg.Includes } method. The include file(s) are read first before the current one. A stack of such includes is created and processed in the natural order encountered, so each includer is applied after the includees, recursively. Note: use --config to specify the first config file read -- the Includes field is excluded from arg processing because it would be processed after the point where include files are processed.

  • Field map[string]any -- allows raw parsing of values that can be applied later. Use this for Network, Env etc fields.

  • Field tag def:"value", used in the GoGi GUI, sets the initial default value and is shown for the -h or --help usage info.

  • kit registered "enum" const types, with names automatically parsed from string values (including bit flags). Must use the goki stringer version to generate FromString() method, and register the type like this: var KitTestEnum = kit.Enums.AddEnum(TestEnumN, kit.NotBitFlag, nil) -- see enum.go file for example.

Standard Config Example

Here's the Config struct from axon/examples/ra25, which can provide a useful starting point. It uses Params, Run and Log sub-structs to better organize things. For sims with extensive Env config, that should be added as a separate sub-struct as well. The view:"add-fields" struct tag shows all of the fields in one big dialog in the GUI -- if you want separate ones, omit that.

// ParamConfig has config parameters related to sim params
type ParamConfig struct {
	Network map[string]any `desc:"network parameters"`
	Set     string         `desc:"ParamSet name to use -- must be valid name as listed in compiled-in params or loaded params"`
	File    string         `desc:"Name of the JSON file to input saved parameters from."`
	Tag     string         `desc:"extra tag to add to file names and logs saved from this run"`
	Note    string         `desc:"user note -- describe the run params etc -- like a git commit message for the run"`
	SaveAll bool           `desc:"Save a snapshot of all current param and config settings in a directory named params_<datestamp> then quit -- useful for comparing to later changes and seeing multiple views of current params"`
}

// RunConfig has config parameters related to running the sim
type RunConfig struct {
	GPU          bool   `def:"true" desc:"use the GPU for computation -- generally faster even for small models if NData ~16"`
	Threads      int    `def:"0" desc:"number of parallel threads for CPU computation -- 0 = use default"`
	Run          int    `def:"0" desc:"starting run number -- determines the random seed -- runs counts from there -- can do all runs in parallel by launching separate jobs with each run, runs = 1"`
	Runs         int    `def:"5" min:"1" desc:"total number of runs to do when running Train"`
	Epochs       int    `def:"100" desc:"total number of epochs per run"`
	NZero        int    `def:"2" desc:"stop run after this number of perfect, zero-error epochs"`
	NTrials      int    `def:"32" desc:"total number of trials per epoch.  Should be an even multiple of NData."`
	NData        int    `def:"16" min:"1" desc:"number of data-parallel items to process in parallel per trial -- works (and is significantly faster) for both CPU and GPU.  Results in an effective mini-batch of learning."`
	TestInterval int    `def:"5" desc:"how often to run through all the test patterns, in terms of training epochs -- can use 0 or -1 for no testing"`
	PCAInterval  int    `def:"5" desc:"how frequently (in epochs) to compute PCA on hidden representations to measure variance?"`
	StartWts     string `desc:"if non-empty, is the name of weights file to load at start of first run -- for testing"`
}

// LogConfig has config parameters related to logging data
type LogConfig struct {
	SaveWts   bool `desc:"if true, save final weights after each run"`
	Epoch     bool `def:"true" desc:"if true, save train epoch log to file, as .epc.tsv typically"`
	Run       bool `def:"true" desc:"if true, save run log to file, as .run.tsv typically"`
	Trial     bool `def:"false" desc:"if true, save train trial log to file, as .trl.tsv typically. May be large."`
	TestEpoch bool `def:"false" desc:"if true, save testing epoch log to file, as .tst_epc.tsv typically.  In general it is better to copy testing items over to the training epoch log and record there."`
	TestTrial bool `def:"false" desc:"if true, save testing trial log to file, as .tst_trl.tsv typically. May be large."`
	NetData   bool `desc:"if true, save network activation etc data from testing trials, for later viewing in netview"`
}

// Config is a standard Sim config -- use as a starting point.
type Config struct {
	Includes []string    `desc:"specify include files here, and after configuration, it contains list of include files added"`
	GUI      bool        `def:"true" desc:"open the GUI -- does not automatically run -- if false, then runs automatically and quits"`
	Debug    bool        `desc:"log debugging information"`
	Params   ParamConfig `view:"add-fields" desc:"parameter related configuration options"`
	Run      RunConfig   `view:"add-fields" desc:"sim running related configuration options"`
	Log      LogConfig   `view:"add-fields" desc:"data logging related configuration options"`
}

func (cfg *Config) IncludesPtr() *[]string { return &cfg.Includes }

Key design considerations

  • Can set config values from command-line args and/or config file (TOML being the preferred format) (or env vars)

    • current axon models only support args. obelisk models only support TOML. conflicts happen.

  • Sims use a Config struct with fields that represents the definitive value of all arg / config settings (vs a map[string]any)

    • struct provides compile time error checking (and IDE completion) -- very important and precludes map.
    • Add Config to Sim so it is visible in the GUI for easy visual debugging etc (current args map is organized by types -- makes it hard to see everything).

  • Enable setting Network or Env params directly:

    • Use Network., Env., TrainEnv., TestEnv. etc prefixes followed by standard params selectors (e.g., Layer.Act.Gain) or paths to fields in relevant env. These can be added to Config as map[string]any and then applied during ConfigNet, ConfigEnv etc.

  • TOML Go implementations are case insensitive (TOML spec says case sensitive..) -- makes sense to use standard Go CamelCase conventions as in every other Go struct.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// DefaultEncoding is the default encoding format for config files.
	// currently toml is the only supported format, but others could be added
	// if needed.
	DefaultEncoding = "toml"

	// IncludePaths is a list of file paths to try for finding config files
	// specified in Include field or via the command line --config --cfg or -c args.
	// Set this prior to calling Config -- default is current directory '.' and 'configs'
	IncludePaths = []string{".", "configs"}

	//	NonFlagArgs are the command-line args that remain after all the flags have
	// been processed.  This is set after the call to Config.
	NonFlagArgs = []string{}

	// ConfigFile is the name of the config file actually loaded, specified by the
	// -config or -cfg command-line arg or the default file given in Config
	ConfigFile string

	// Help is variable target for -help or -h args
	Help bool
)
View Source 
var KiT_TestEnum = kit.Enums.AddEnum(TestEnumN, kit.NotBitFlag, nil)

This registers the enum with Kit which then allows full set / get from strings.

Functions

func CommandArgs 

func CommandArgs(allArgs map[string]reflect.Value)

CommandArgs adds non-field args that control the config process: -config -cfg -help -h

func Config 

func Config(cfg any, defaultFile string) ([]string, error)

Config is the overall config setting function, processing config files and command-line arguments, in the following order:

  • Apply any `def:` field tag default values.
  • Look for `--config`, `--cfg`, or `-c` arg, specifying a config file on the command line.
  • Fall back on default config file name passed to `Config` function, if arg not found.
  • Read any `Include[s]` files in config file in deepest-first (natural) order, then the specified config file last.
  • Process command-line args based on Config field names, with `.` separator for sub-fields.
  • Boolean flags are set on with plain -flag; use No prefix to turn off (or explicitly set values to true or false).

Also processes -help or -h and prints usage and quits immediately.

func FieldArgNames 

func FieldArgNames(obj any, allArgs map[string]reflect.Value)

FieldArgNames adds to given args map all the different ways the field names can be specified as arg flags, mapping to the reflect.Value

func IncludeStack 

func IncludeStack(cfg Includer) ([]string, error)

IncludeStack returns the stack of include files in the natural order in which they are encountered (nil if none). Files should then be read in reverse order of the slice. Returns an error if any of the include files cannot be found on IncludePath. Does not alter cfg.

func LogFileName 

func LogFileName(logName, netName, runName string) string

LogFileName returns a standard log file name as netName_runName_logName.tsv

func OpenFS 

func OpenFS(cfg any, fsys fs.FS, file string) error

OpenFS reads config from given TOML file, using the fs.FS filesystem -- e.g., for embed files.

func OpenWithIncludes 

func OpenWithIncludes(cfg any, file string) error

OpenWithIncludes reads config from given config file, looking on IncludePaths for the file, and opens any Includes specified in the given config file in the natural include order so includee overwrites included settings. Is equivalent to Open if there are no Includes. Returns an error if any of the include files cannot be found on IncludePath.

func ParseArg 

func ParseArg(s string, args []string, allArgs map[string]reflect.Value, errNotFound bool) (a []string, err error)

func ParseArgs 

func ParseArgs(cfg any, args []string, allArgs map[string]reflect.Value, errNotFound bool) ([]string, error)

ParseArgs parses given args using map of all available args setting the value accordingly, and returning any leftover args. setting errNotFound = true causes args that are not in allArgs to trigger an error. Otherwise, it just skips those.

func Save 

func Save(cfg any, file string) error

Save writes TOML to given file.

func SetArgValue 

func SetArgValue(name string, fval reflect.Value, value string) error

SetArgValue sets given arg name to given value, into settable reflect.Value

func SetFromArgs 

func SetFromArgs(cfg any, args []string) (nonFlags []string, err error)

SetFromArgs sets Config values from command-line args, based on the field names in the Config struct. Returns any args that did not start with a `-` flag indicator. For more robust error processing, it is assumed that all flagged args (-) must refer to fields in the config, so any that fail to match trigger an error. Errors can also result from parsing. Errors are automatically logged because these are user-facing.

func SetFromDefaults 

func SetFromDefaults(cfg any) error

SetFromDefaults sets Config values from field tag `def:` values. Parsing errors are automatically logged.

func SetLogFile 

func SetLogFile(logs *elog.Logs, configOn bool, mode etime.Modes, time etime.Times, logName, netName, runName string)

SetLogFile sets the log file for given mode and time, using given logName (extension), netName and runName, if the Config flag is set.

func Usage 

func Usage(cfg any) string

Usage returns the usage string for args based on given Config object

Types

type Includer 

type Includer interface {
	// IncludesPtr returns a pointer to the Includes []string field containing file(s) to include
	// before processing the current config file.
	IncludesPtr() *[]string
}

Includer facilitates processing include files in Config objects.

type TestEnum 

type TestEnum int32

TestEnum is an enum type for testing 

const (
	TestValue1 TestEnum = iota

	TestValue2

	TestEnumN
)

func (*TestEnum) FromString 

func (i *TestEnum) FromString(s string) error

func (TestEnum) MarshalJSON 

func (ev TestEnum) MarshalJSON() ([]byte, error)

func (TestEnum) MarshalText 

func (ev TestEnum) MarshalText() ([]byte, error)

TOML uses Text interface

func (TestEnum) String 

func (i TestEnum) String() string

func (*TestEnum) UnmarshalJSON 

func (ev *TestEnum) UnmarshalJSON(b []byte) error

func (*TestEnum) UnmarshalText 

func (ev *TestEnum) UnmarshalText(b []byte) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.