cogs

README

installation:

With go:

• clone this repo, cd into it
• go build -o $GOPATH/bin ./cmd/cogs

Without go, PLatform can be Linux/Windows/Darwin:

PL="Darwin" VR="0.6.0" \
  curl -SLk \ 
  "github.com/Bestowinc/cogs/releases/download/v${VR}/cogs_${VR}_${PL}_x86_64.tar.gz" | \
  tar xvz -C /usr/local/bin cogs

COGS COnfiguration manaGement S

Usage:
  cogs gen <ctx> <cog-file> [options]

Options:
  -h --help        Show this screen.
  --version        Show version.
  --no-enc, -n     Skips fetching encrypted vars.
  --envsubst, -e   Perform environmental substitution on the given cog file.
  --keys=<key,>    Include specific keys, comma separated.
  --not=<key,>     Exclude specific keys, comma separated.
  --out=<type>     Configuration output type [default: json].
                   <type>: json, toml, yaml, dotenv, raw.
  
  --export, -x     If --out=dotenv: Prepends "export " to each line.
  --preserve, -p   If --out=dotenv: Preserves variable casing.
  --sep=<sep>      If --out=raw:    Delimits values with a <sep>arator.

annotated spec:

name = "basic_example"

# key value pairs for a context are defined under <ctx>.vars
[docker.vars]
var = "var_value"
other_var = "other_var_value"

[sops]
# a default path to be inherited can be defined under <ctx>.path
path = ["./test_files/manifest.yaml", "subpath"]
[sops.vars]
# a <var>.path key can map to four valid types:
# 1. path value is "string_value" - indicating a single file to look through
# 2. path value is [] - thus <ctx>.path will be inherited
# 3. path value is a ["two_index, "array"] - either index being [] or "string_value":
# -  [[], "subpath"] - path will be inherited from <ctx>.path if present
# -  ["path", []] - subpath will be inherited from <ctx>.path if present
# 4. ["path", "subpath"] - nothing will be inherited
var1.path = ["./test_files/manifest.yaml", "subpath"]
var2.path = []
var3.path = [[], "other_subpath"]
# dangling variable should return {"empty_var": ""} since only name override was defined
empty_var.name = "some_name"
# key value pairs for an encrypted context are defined under <ctx>.enc.vars
[sops.enc.vars]
yaml_enc.path = "./test_files/test.enc.yaml"
dotenv_enc = {path = "./test_files/test.enc.env", name = "DOTENV_ENC"}
json_enc.path = "./test_files/test.enc.json"

[kustomize]
path = ["./test_files/kustomization.yaml", "configMapGenerator.[0].literals"]
# a default deserialization path to be inherited can be defined under <ctx>.path
# once <var>.path has been traversed, attempt to deserialize the returned object
# as if it was in dotenv format
type = "dotenv"
[kustomize.vars]
# var1.name = "VAR_1" means that the key alias "VAR_1" will
# be searched for to retrieve the var1 value
var1 = {path = [], name = "VAR_1"}
var2 = {path = [], name = "VAR_2"}
var3 = {path = [[], "jsonMap"], type = "json"}

goals:

1. Allow a flat style of managing configurations across disparate contexts and different formats (plaintext vs. encrypted)

   • aggregates plaintext config values and SOPS secrets in one manifest
     • ex: local development vs. docker vs. production environments

2. Introduce an automated and cohesive way to validate and correlate configurations

   • TODO: allow a gradual introduction of new variable names by automating:
     • introduction of new name for same value (DB_SECRETS -> DATABASE_SECRETS)
     • and deprecation of old name (managing deletion of old DB_SECRETS references)

scope of support:

• microservice configuration
• parse YAML manifests
• valid viper package input (so able to output JSON, YAML, and TOML)
• SOPS secret management
• docker-compose YAML env config scheme

subcommands

• cogs gen

  • outputs a flat and serialized K:V array

• cogs migrate TODO

  • cogs migrate <OLD_KEY_NAME> <NEW_KEY_NAME> [<envs>...]
  • cogs migrate --commit <OLD_KEY_NAME> <NEW_KEY_NAME> (<envs>...)

Aims to allow a gradual and automated migration of key names without risking sensitive environments:

# config.yaml pre migration
DB_SECRETS: "secret_pw"

Should happen in two main steps:

1. cogs migrate DB_SECRETS DATABASE_SECRETS

• should default to creating the new key name in all environments
• creates new variable in remote file or cog manifest

# config.yaml during migration
DB_SECRETS: "secret_pw"
DATABASE_SECRETS: "secret_pw"

1. cogs migrate --commit DB_SECRETS DATABASE_SECRETS <env>...

• removes old key name for all <envs> specified

# config.yaml post migration
DATABASE_SECRETS: "secret_pw"

• should apply to plaintext K/Vs and SOPS encrypted values

running example data locally:

• gpg --import ./test_files/sops_functional_tests_key.asc should be run to import the test private key used for encrypted dummy data
• Building binary locally : go build -o $GOPATH/bin ./cmd/cogs
• Kustomize style var retrieval: cogs gen kustomize ./basic.cog.toml
• Encrypted var retrieval: cogs gen sops ./basic.cog.toml
• some-service.cog.toml shows how a toml definition correlates to the JSON counterpart

further references

TOML spec

envsubst cheatsheet:

Expression	Meaning
${var}	Value of var (same as $var)
${var-${DEFAULT}}	If var not set, evaluate expression as ${DEFAULT}
${var:-${DEFAULT}}	If var not set or is empty, evaluate expression as ${DEFAULT}
${var=${DEFAULT}}	If var not set, evaluate expression as ${DEFAULT}
${var:=${DEFAULT}}	If var not set or is empty, evaluate expression as ${DEFAULT}
$$var	Escape expressions. Result will be $var.
${var^^}	Uppercase value of $var
${var,,}	Lowercase value of $var
${#var}	Value of $var string length
${var^}
${var,}
${var:position}
${var:position:length}
${var#substring}
${var##substring}
${var%substring}
${var%%substring}
${var/substring/replacement}
${var//substring/replacement}
${var/#substring/replacement}
${var/%substring/replacement}

Notes:

• envsubst warning, make sure that any --envsubst tags retain a file's membership as valid TOML:

# yes
[env.vars]
thing = "${THING_VAR}"

# NO
[sloppy.vars]${NO}
thing = "${THING_VAR}"

Documentation

Index

• Variables
• func Generate(envName, cogPath string, outputType Format) (map[string]interface{}, error)
• func IsEnvFile(path string) bool
• func IsJSONFile(path string) bool
• func IsTOMLFile(path string) bool
• func IsYAMLFile(path string) bool
• func OutputCfg(cfg *Cfg, format Format) (interface{}, error)
• type Cfg
  • func (c Cfg) String() string
• type Format
  • func FormatForCfg(cfg *Cfg) (format Format)
  • func FormatForPath(path string) Format
  • func (t Format) Validate() error
• type Gear
  • func (g *Gear) ResolveMap(env RawEnv) (map[string]interface{}, error)
  • func (g *Gear) SetName(name string)
• type RawEnv
• type Resolver
• type Visitor
  • func NewDotenvVisitor(buf []byte) (Visitor, error)
  • func NewJSONVisitor(buf []byte) (Visitor, error)
  • func NewTOMLVisitor(buf []byte) (Visitor, error)
  • func NewYAMLVisitor(buf []byte) (Visitor, error)

Variables

var EnvSubst bool = false

EnvSubst decides whether to use environmental substitution or not

var NoEnc bool = false

NoEnc decides whether to output encrypted variables or now

Functions

func Generate

func Generate(envName, cogPath string, outputType Format) (map[string]interface{}, error)

Generate is a top level command that takes an env argument and cogfilepath to return a string map

func IsEnvFile

func IsEnvFile(path string) bool

IsEnvFile returns true if a given file path corresponds to a .env file

func IsJSONFile

func IsJSONFile(path string) bool

IsJSONFile returns true if a given file path corresponds to a JSON file

func IsTOMLFile

func IsTOMLFile(path string) bool

IsTOMLFile returns true if a given file path corresponds to a TOML file

func IsYAMLFile

func IsYAMLFile(path string) bool

IsYAMLFile returns true if a given file path corresponds to a YAML file

func OutputCfg

func OutputCfg(cfg *Cfg, format Format) (interface{}, error)

OutputCfg returns the corresponding value for a given Cfg struct

Types

type Cfg

type Cfg struct {
	Name         string      // defaults to key name in cog file unless var.name="other_name" is used
	Value        string      // Cfg.ComplexValue should be nil if Cfg.Value is a non-empty string("")
	ComplexValue interface{} // Cfg.Value should be empty string("") if Cfg.ComplexValue is non-nil
	Path         string      // filepath string where Cfg can be resolved
	SubPath      string      // object traversal string used to resolve Cfg if not at top level of document (yq syntax)
	// contains filtered or unexported fields
}

Cfg holds all the data needed to generate one string key value pair

func (Cfg) String

func (c Cfg) String() string

String holds the string representation of a Cfg struct

type Format

type Format string

Format represents the final marshalled k/v output type from a resolved Gear

const (
	JSON   Format = "json"
	YAML   Format = "yaml"
	TOML   Format = "toml"
	Dotenv Format = "dotenv"
	Raw    Format = "raw"
)

Formats for respective object notation

func FormatForCfg

func FormatForCfg(cfg *Cfg) (format Format)

FormatForCfg returns the correct format given the readType

func FormatForPath

func FormatForPath(path string) Format

FormatForPath returns the correct format given the path to a file

func (Format) Validate

func (t Format) Validate() error

Validate ensures that a string maps to a valid Format

type Gear

type Gear struct {
	Name string
	// contains filtered or unexported fields
}

Gear represents one of the contexts in a cog manifest. The term "gear" is used to refer to the operating state of a machine (similar to how a microservice can operate locally or in a remote environment) rather than a gear object. The term "switching gears" is an apt representation of how one Cog manifest file can have many contexts/environments

func (*Gear) ResolveMap

func (g *Gear) ResolveMap(env RawEnv) (map[string]interface{}, error)

ResolveMap outputs the flat associative string, resolving potential filepath pointers held by Cfg objects by calling the .ResolveValue() method

func (*Gear) SetName

func (g *Gear) SetName(name string)

SetName sets the gear name to the provided string

type RawEnv

type RawEnv map[string]interface{}

RawEnv is meant to represent the topmost untraversed level of a cog environment

type Resolver

type Resolver interface {
	ResolveMap(RawEnv) (map[string]interface{}, error)
	SetName(string)
}

Resolver is meant to define an object that returns the final string map to be used in a configuration resolving any paths and sub paths defined in the underling config map

type Visitor

type Visitor interface {
	SetValue(*Cfg) error
}

Visitor allows a query path to return the underlying value for a given visitor

func NewDotenvVisitor

func NewDotenvVisitor(buf []byte) (Visitor, error)

NewDotenvVisitor returns a visitor object that satisfies the Visitor interface attempting to turn a supposed dotenv byte slice into a *yaml.Node object

func NewJSONVisitor

func NewJSONVisitor(buf []byte) (Visitor, error)

NewJSONVisitor returns a visitor object that satisfies the Visitor interface attempting to turn a supposed JSON byte slice into a *yaml.Node object

func NewTOMLVisitor

func NewTOMLVisitor(buf []byte) (Visitor, error)

NewTOMLVisitor returns a visitor object that satisfies the Visitor interface attempting to turn a supposed TOML byte slice into a *yaml.Node object

func NewYAMLVisitor

func NewYAMLVisitor(buf []byte) (Visitor, error)

NewYAMLVisitor returns a visitor object that satisfies the Visitor interface