config

Documentation

Overview

Config controls the overall configuration of the application.

It is generated by first attempting to read a configuration file and then overwriting those values with anything found in environment variables. Environment variables always come last and have the highest priority. As per (https://12factor.net/config).

All environment variables are prefixed with "GOFER". Ex: GOFER_DEBUG=true

Most envvar configuration abides by common envvar string=string formatting: Ex. `export GOFER_DEBUG=true`. Complex envvars might take a json string as value.

Example: export GOFER_EXTENSIONS="{"name":"test"},{"name":"test2"}"

You can print out a current description of current environment variable configuration by using the cli command:

`gofer service printenv`

Index

• func GetAPIEnvVars() []string
• func GetCLIEnvVars() []string
• type API
  • func DefaultAPIConfig() *API
  • func InitAPIConfig(userDefinedPath string, loadDefaults, validate, devMode bool) (*API, error)
• type CLI
  • func DefaultCLIConfig() *CLI
  • func InitCLIConfig(flagPath string, loadDefaults bool) (*CLI, error)
• type Development
  • func DefaultDevelopmentConfig() *Development
  • func FullDevelopmentConfig() *Development
• type Docker
  • func DefaultDockerConfig() *Docker
• type Extensions
  • func DefaultExtensionsConfig() *Extensions
• type ExternalEventsAPI
  • func DefaultExternalEventsAPIConfig() *ExternalEventsAPI
• type Frontend
• type ObjectStore
  • func DefaultObjectStoreConfig() *ObjectStore
• type Scheduler
  • func DefaultSchedulerConfig() *Scheduler
• type SecretStore
  • func DefaultSecretStoreConfig() *SecretStore
• type Server
  • func DefaultServerConfig() *Server
• type Sqlite
• type SqliteSecret

Functions

func GetAPIEnvVars

func GetAPIEnvVars() []string

func GetCLIEnvVars

func GetCLIEnvVars() []string

Types

type API

type API struct {
	// Controls the ability to extension runs. This setting can be toggled while the server is running.
	IgnorePipelineRunEvents bool `koanf:"ignore_pipeline_run_events"`

	// The limit automatically imposed if the pipeline does not define a limit. 0 is unlimited.
	RunParallelismLimit int `koanf:"run_parallelism_limit"`

	// How many total versions of an individual pipelines to keep.
	// The oldest version of a pipeline over this limit gets deleted.
	// 0 means don't delete versions.
	PipelineVersionLimit int `koanf:"pipeline_version_limit"`

	// Controls how long Gofer will hold onto events before discarding them. This is important factor in disk space
	// and memory footprint.
	//
	// Example: Rough math on a 5,000 pipeline Gofer instance with a full 6 months of retention
	//  puts the memory and storage footprint at about 9GB.
	EventLogRetention time.Duration `koanf:"event_log_retention"`

	// How often the background process for pruning events should run.
	EventPruneInterval time.Duration `koanf:"event_prune_interval"`

	// Log level affects the entire application's logs including launched extensions.
	LogLevel string `koanf:"log_level"`

	// The total amount of runs before logs of the oldest log starts being deleted.
	TaskRunLogExpiry int `koanf:"task_run_log_expiry"`

	// Directory to store task run log files.
	TaskRunLogsDir string `koanf:"task_run_logs_dir"`

	// TaskRunStopTimeout controls the time the scheduler will wait for a normal user container(non-extension containers)
	// to stop. When the timeout is reached the container will be forcefully terminated.
	// You can use a negative duration("-1s") to convey that no timeout should be specified and the scheduler
	// should wait however long it takes the container to respond to the terminate signal.
	// This is usually passed to the scheduler when a request to cancel a task run is being made.
	TaskRunStopTimeout time.Duration `koanf:"task_run_stop_timeout"`

	Development       *Development       `koanf:"development"`
	Extensions        *Extensions        `koanf:"extensions"`
	ExternalEventsAPI *ExternalEventsAPI `koanf:"external_events_api"`
	ObjectStore       *ObjectStore       `koanf:"object_store"`
	Scheduler         *Scheduler         `koanf:"scheduler"`
	SecretStore       *SecretStore       `koanf:"secret_store"`
	Server            *Server            `koanf:"server"`
}

API defines config settings for the gofer server

func DefaultAPIConfig

func DefaultAPIConfig() *API

func InitAPIConfig

func InitAPIConfig(userDefinedPath string, loadDefaults, validate, devMode bool) (*API, error)

Get the final configuration for the server. This involves correctly finding and ordering different possible paths for the configuration file:

1. The function is intended to be called with paths gleaned from the -config flag in the cli.
2. If the user does not use the -config path of the path does not exist, then we default to a few hard coded config path locations.
3. Then try to see if the user has set an envvar for the config file, which overrides all previous config file paths.
4. Finally, whatever configuration file path is found first is the processed.

Whether or not we use the configuration file we then search the environment for all environment variables:

• Environment variables are loaded after the config file and therefore overwrite any conflicting keys.
• All configuration that goes into a configuration file can also be used as an environment variable.

type CLI

type CLI struct {
	Namespace string `koanf:"namespace"`
	Detail    bool   `koanf:"detail"`
	Format    string `koanf:"format"`
	Host      string `koanf:"host"`
	NoColor   bool   `koanf:"no_color"`
	Token     string `koanf:"token"`
}

func DefaultCLIConfig

func DefaultCLIConfig() *CLI

DefaultCLIConfig returns a pre-populated configuration struct that is used as the base for super imposing user configuration settings.

func InitCLIConfig

func InitCLIConfig(flagPath string, loadDefaults bool) (*CLI, error)

Get configuration for command line. This involves correctly finding and ordering different possible paths for the configuration file:

1. The function is intended to be called with paths gleaned from the -config flag in the cli.
2. If the user does not use the -config path of the path does not exist, then we default to a few hard coded config path locations.
3. Then try to see if the user has set an envvar for the config file, which overrides all previous config file paths.
4. Finally, whatever configuration file path is found first is the processed.

Whether or not we use the configuration file we then search the environment for all environment variables:

• Environment variables are loaded after the config file and therefore overwrite any conflicting keys.
• All configuration that goes into a configuration file can also be used as an environment variable.

type Development

type Development struct {
	PrettyLogging   bool `koanf:"pretty_logging"`
	BypassAuth      bool `koanf:"bypass_auth"`
	UseLocalhostTLS bool `koanf:"use_localhost_tls"`

	// Use a pre-filled insecure encryption key.
	DefaultEncryption bool `koanf:"default_encryption"`

	// Pass the "skip_tls_verify" environment variable to extensions
	// so that they can talk to Gofer without verifying the cert.
	ExtensionSkipTLSVerify bool `koanf:"extension_skip_tls_verify"`
}

func DefaultDevelopmentConfig

func DefaultDevelopmentConfig() *Development

func FullDevelopmentConfig

func FullDevelopmentConfig() *Development

type Docker

type Docker struct {
	// Prune runs a reoccuring `docker system prune` job to avoid filling the local disk with docker images.
	Prune bool `koanf:"prune"`

	// The period of time in between runs of `docker system prune`
	PruneInterval time.Duration `koanf:"prune_interval"`
}

func DefaultDockerConfig

func DefaultDockerConfig() *Docker

type Extensions

type Extensions struct {
	// InstallBaseExtensions attempts to automatically install the cron and interval extensions on first startup.
	InstallBaseExtensions bool `koanf:"install_base_extensions"`

	// StopTimeout controls the time the scheduler will wait for a extension container to stop. After this period
	// Gofer will attempt to force stop the extension container.
	StopTimeout time.Duration `koanf:"stop_timeout"`

	// TLSCertPath is the file path of the extension TLS certificate.
	TLSCertPath string `koanf:"tls_cert_path"`

	// TLSKeyPath is the file path of the extension TLS key.
	TLSKeyPath string `koanf:"tls_key_path"`
}

Extensions represents the configuration for Gofer Extensions. Extensions are used to generate events in which pipelines should run.

func DefaultExtensionsConfig

func DefaultExtensionsConfig() *Extensions

type ExternalEventsAPI

type ExternalEventsAPI struct {
	Enable bool `koanf:"enable"`

	// URL for the server to bind to. Ex: localhost:8080
	Host string `koanf:"host"`
}

ExternalEventsAPI controls how the settings around the HTTP service that handles external extension events.

func DefaultExternalEventsAPIConfig

func DefaultExternalEventsAPIConfig() *ExternalEventsAPI

type Frontend

type Frontend struct {
	Enable bool `koanf:"enable"`
}

Frontend represents configuration for frontend basecoat

type ObjectStore

type ObjectStore struct {
	// The ObjectStore engine used by the backend.
	// Possible values are: bolt
	Engine string `koanf:"engine"`

	Sqlite *Sqlite `koanf:"sqlite"`

	// Pipeline Objects last forever but are limited in number. This is the total amount of items that can be stored
	// per pipeline before gofer starts deleting objects.
	PipelineObjectLimit int `koanf:"pipeline_object_limit"`

	// Objects stored at the run level are unlimited in number, but only last for a certain number of runs.
	// The number below controls how many runs until the run objects for the oldest run will be deleted.
	// Ex. an object stored on run number #5 with an expiry of 2 will be deleted on run #7 regardless of run
	// health.
	RunObjectExpiry int `koanf:"run_object_expiry"`
}

ObjectStore defines config settings for gofer ObjectStore. The ObjectStore stores temporary objects for pipelines and runs.

func DefaultObjectStoreConfig

func DefaultObjectStoreConfig() *ObjectStore

type Scheduler

type Scheduler struct {
	// The database engine used by the scheduler
	// possible values are: docker
	Engine string  `koanf:"engine"`
	Docker *Docker `koanf:"docker"`
}

Scheduler defines config settings for gofer scheduler. The scheduler is the backend for how containers are run.

func DefaultSchedulerConfig

func DefaultSchedulerConfig() *Scheduler

type SecretStore

type SecretStore struct {
	// The ObjectStore engine used by the backend.
	// Possible values are: sqlite
	Engine string `koanf:"engine"`

	Sqlite *SqliteSecret `koanf:"sqlite"`
}

SecretStore defines the configuration for Gofer's secret backend.

func DefaultSecretStoreConfig

func DefaultSecretStoreConfig() *SecretStore

type Server

type Server struct {
	// URL where the Gofer server is located. Shared with entities that need to talk to the Gofer API.
	Address string `koanf:"address"`

	// URL for the server to bind to. Ex: localhost:8080
	Host string `koanf:"host"`

	// How long the GRPC service should wait on in-progress connections before hard closing everything out.
	ShutdownTimeout time.Duration `koanf:"shutdown_timeout"`

	// Path to Gofer's sqlite database.
	StoragePath string `koanf:"storage_path"`

	// The total amount of results the database will attempt to pass back when a limit is not explicitly given.
	StorageResultsLimit int `koanf:"storage_results_limit"`

	TLSCertPath string `koanf:"tls_cert_path"`
	TLSKeyPath  string `koanf:"tls_key_path"`
}

Server represents lower level HTTP/GRPC server settings.

func DefaultServerConfig

func DefaultServerConfig() *Server

DefaultServerConfig returns a pre-populated configuration struct that is used as the base for super imposing user configuration settings.

type Sqlite

type Sqlite struct {
	Path string `koanf:"path"` // file path for database file
}

Sqlite

type SqliteSecret

type SqliteSecret struct {
	Path string `koanf:"path"` // file path for database file
	// EncryptionKey is a 32-bit random string of characters used to encrypt data at rest.
	EncryptionKey string `split_words:"true" koanf:"encryption_key"`
}

SqliteSecret