generate

README

File generation

The package in this folder is used to generate boilerplate configuration files for all environments.

Files in this category are e.g.:

• files in environment-specific overlay folders
• environment-specific value files (e.g. in a service folder services/service-A/values/cluster-specific/...)

See General file generation for details.

In addition, this package provides a mechanism to render go-templates directly with the ParseTemplate function. This function accepts a go-template file, a list of value files and a desired target (or output) file. See Custom template rendering for details.

Custom template rendering

Here is an example for a customly rendered go-template:

# setup files
tmp_dir="tmp"
mkdir "${tmp_dir}"

cat > "${tmp_dir}/template" << EOF
my first value: {{.key1}}
something else - {{.key2.key3}}
EOF

cat > "${tmp_dir}/values_1.yaml" << EOF
key1: value1
EOF

cat > "${tmp_dir}/values_2.yaml" << EOF
key2:
  key3: value2
EOF

# render template
coco generate custom \
  --value "${tmp_dir}/values_1.yaml" \
  --value "${tmp_dir}/values_2.yaml" \
  --target "${tmp_dir}/output" \
  "${tmp_dir}/template"

# output
cat "${tmp_dir}/output"
# expected output:

# > my first value: value1
# > something else - value2

# cleanup

rm -rf "${tmp_dir}"

General file generation

Why is this needed?

The file generation shall solve the following issues

• missing environment-specific configurations when adding or changing a service
• easing the creation of new environments by auto-generating what is possible
• preventing configuration drift for environment-specific configurations in service values

How can the needs be adressed?

With the generate package we can run file-generation which allows for the following features:

• diff detection: find missing configurations for all service-environment combinations
• drift prevention: all environment-specific configurations are recreated by the tool, which prevents accidental configuration drift between environments

How it works

In general, the file generation command depends on a set of global inputs (the files in the top-level values folder). For each values file to be generated there needs to be a folder with the desired name. Within the folder should be a coco.yaml configuration file, with the following structure:

type: environment
values:
  - names of value.yaml files
  - with path relative to coco.yaml

The value files need to be .yaml files, however in the list the file ending must not be provided. These input files govern which files will be generated and what values are generated automatically. Note that all keys will be merged and if multiple value files contain the same key, the values lower in the list overwrite previous values.

Naming rules

The structure of generated files is defined by a local template file (identified by its ending .tmpl) that appears on the same level as the files that will be generated.

If the template file has no name and just the ending (i.e. .tmpl), the generated file will take the full cluster-name, and otherwise the template name is prefixed as e.g.:

.tmpl      ->  full_cluster_name.yaml
name.tmpl  ->  name--full_cluster_name.yaml

Similarely whole subfolders can be template folders and similar naming rules as above apply:

.tmpl/      ->  full_cluster_name/
name.tmpl/  ->  name--full_cluster_name/

In a .tmpl folder all files will be treated as template files and they will be rendered and copied to the generated subfolders. In contrast to template files outside of .tmpl folders these files will not undergo the renaming procedure. This means that their names persist in every generated environment folder.

Exceptions

Version differences

Per default, file generation will only take control over generated files that hold a compatible version as the file generation tool itself, e.g. a file with the first line # Code generated by CLI 'coco generate ...' (version: v1.2.3); DO NOT EDIT. will only be changed by coco in version v2.0.0. > ACTUAL_VERSION > v1.2.3.

Manual overwrites

Normally generated files cannot be manipulated by hand since their content will be overwritten once the file-generation CLI is run again.

If there is a need to create an exception for this overwrite mechanism, the following key indications or 1-line indications can be used to prevent the delition of the marked lines.

persistent: line # HumanInput
alsoPersistent: !HumanInput line
line: that will be overwritten

Example (helm value files)

Setup

This example deals with environment-specific values for rendering a helm-chart like they can be found e.g. here. The relevant inputs for file-generation CLI are in the following files

+-- services
|   +-- serviceA
|   |   +-- values
|   |   |   +-- cluster-specific
|   |   |   |   +-- .tmpl
|   |   |   |   +-- cluster_1.yaml
...
+-- values
|   +-- cluster_1
|   |   +-- coco.yaml
|   |   +-- value1.yaml
|   +-- cluster_2
|   |   +-- coco.yaml
|   |   +-- value2.yaml
|   |   +-- value22.yaml

Here the .tmpl file contains a golang template. This template will be rendered by CoCo with the values for each environment specified in values-cluster/${full_cluster_name}.yaml. The resulting file will be stored under services/serviceA/cluster-values/${full_cluster_name}.yaml.

Note that there exist already files under services/serviceA/cluster-specific/cluster_1.yaml. The content of these files will be overwritten unless exceptions have been specified - see Exceptions. The existing file cluster_1.yaml has the form:

# Code generated by CLI 'coco generate ...' (version: v99.99.99); DO NOT EDIT.

MyFavorite: Value # HumanInput

generalValue: oldValue
{{- if eq  .clusterName "cluster_1" -}}
cluster_1_value: true
{{-end }}

And finally, the .tmpl has the form

generalValue: {{ .generalValue }}

Results

After the file-generation CLI runs, the following file structure will exists

+-- services
|   +-- external-dns
|   |   +-- values
|   |   |   +-- cluster-specific
|   |   |   |   +-- .tmpl
|   |   |   |   +-- cluster_1.yaml
|   |   |   |   +-- cluster_2.yaml
...
+-- values
|   +-- cluster_1
|   |   +-- coco.yaml
|   |   +-- value1.yaml
|   +-- cluster_2
|   |   +-- coco.yaml
|   |   +-- value2.yaml
|   |   +-- value22.yaml

and the newly generated files contain:

cluster_1.yaml:

# Code generated by CLI 'coco generate ...' (version: v99.99.99); DO NOT EDIT.

MyFavorite: Value # HumanInput

generalValue: gValue
cluster_1_value: true

cluster_2.yaml:

# Code generated by CLI 'coco generate ...' (version: v99.99.99); DO NOT EDIT.

generalValue: gValue

Example (kustomize overlay folder)

Setup

This example deals with environment-specific overlay folders for rendering a kustomize application. The relevant inputs for file-generation CLI are in the following files

+-- services
|   +-- serviceB
|   |   +-- overlays
|   |   |   +-- .tmpl
|   |   |   |   +-- configmap.yaml
|   |   |   |   +-- ingress.yaml
|   |   |   |   +-- kustomization.yaml
|   |   |   |   +-- vars-cm.yaml
...
+-- values
|   +-- cluster_1
|   |   +-- coco.yaml
|   |   +-- value1.yaml
|   +-- cluster_2
|   |   +-- coco.yaml
|   |   +-- value2.yaml
|   |   +-- value22.yaml

After file-generation, this gives the following folders

+-- services
|   +-- serviceB
|   |   +-- overlays
|   |   |   +-- .tmpl
                ...
|   |   |   +-- cluster_1
|   |   |   |   +-- configmap.yaml
|   |   |   |   +-- ingress.yaml
|   |   |   |   +-- kustomization.yaml
|   |   |   |   +-- vars-cm.yaml
|   |   |   +-- cluster_2/
                ...
...
+-- values
|   +-- cluster_1
|   |   +-- coco.yaml
|   |   +-- value1.yaml
|   +-- cluster_2
|   |   +-- coco.yaml
|   |   +-- value2.yaml
|   |   +-- value22.yaml

where we droped the remaining files for brevity.

Example subfolder, relative path, merging and overwriting of values

Setup

+-- services
|   +-- serviceA
|   |   +-- values
|   |   |   +-- .tmpl
...
+-- values
|   +-- cluster_1
|   |   +-- coco.yaml
|   |   +-- value1.yaml
|   |   +-- cluster_2
|   |   |   +-- coco.yaml
|   |   |   +-- value2.yaml

The value file in the cluster_1 folder contains the following keys and values:

generalValue: generalValue
parentValue: parentValue

With the coco.yaml file:

type: environment
name: parentFolderCluster
values:
  - value1

Whereas the values file in the subfolder look like this:

generalValue: specificValue
subValue: subValue

With the coco.yaml file:

type: environment
name: subfolderCluster
values:
  - ../value1
  - value2

And finally, the .tmpl has the form

generalValue: {{ .generalValue }}
parentValue: {{ .parentValue }}
subValue: {{ .subValue }}

Results

+-- services
|   +-- serviceA
|   |   +-- values
|   |   |   +-- .tmpl
|   |   |   +-- parentFolderCluster.yaml
|   |   |   +-- subfolderCluster.yaml
...
+-- values
|   +-- cluster_1
|   |   +-- coco.yaml
|   |   +-- value1.yaml
|   |   +-- cluster_2
|   |   |   +-- coco.yaml
|   |   |   +-- value2.yaml

Where 'parentFolderCluster.yaml' contains these key-value pairs:

generalValue: generalValue
parentValue: parentValue

Whereas 'subfolderCluster.yaml' looks like this:

generalValue: specificValue
parentValue: parentValue
subValue: subValue

Note that the value2 key 'generalValue' overwrites the corresponding key from 'value1' since it is listed below in '/values/cluster_1/cluster_2/coco.yaml' and still inherits the 'parentValue.

Documentation

Index

• func Generate(basepath, templateIdentifier, persistenceFlag, configFileName string, ...) error
• func ParseTemplate(filename string, valueFiles []string, target string) error

Functions

func Generate

func Generate(
	basepath, templateIdentifier, persistenceFlag, configFileName string,
	v *version.Version,
	clusterValues, envFilters, folderFilters, excludeFolders []string,
	logLvl log.Level, takeControl bool,
) error

Generate is the main entry function for the generate package which governs file generation from templates as described in the ./readme.md. Inputs:

• basepath: root from where template files will be identified in all sub folders
• templateIdentifier: substring in a filename that identifies this file as a template
• persistenceFlag: identifier in generated yaml files that this line shall not be overwritten
• clusterValues: folder in which value files for file generation are located
• envFilters: filters down the list of environments for which file generation is performed
• folderFilters: filters down the list of template locations to specific sub-folders
• version: coco version (for comparisons with the version in the existing generated files)
• takeControl: overwrite to do file generation also on files that have a different version
• logLvl: specifies the log level that will be used

func ParseTemplate

func ParseTemplate(filename string, valueFiles []string, target string) error