cuegen

README

Cuegen

Cuegen is a tool to build kubernetes resources with CUE. It solves the problems of composing and versioning of "charts", as well as easily importing external data (scripts, certificates, keys, etc.) into CUE charts. For this purpose, it extends the rich possibilities that CUE already provides with the ability to "compose" charts from various sources and load resources, controlled by attributes.

If CUE and creating k8s resources with CUE is new to you, the CUE homepage, the k8s tutorial, as well as examples in this repository (in that order) are good starting points.

Table of Contents

• Cuegen
  • Table of Contents
  • Features
  • Install
  • Usage
  • Configuration
  • Environment Variables
  • Components
  • Attributes
    • Attribute "@readfile"
    • Attribute "@readmap"
    • Attribute "@read"
      • with file Attribute value
      • with directory Attribute value
  • Order Workaround
  • Changelog

Features

• Compose manifests from various, versioned sources (local directories, git-repositories)
• Load file contents into CUE values, e.g. a script into a ConfigMap key
• Load whole directories as key/values data at once, e.g. into a ConfigMap
• Load structured data (JSON/YAML/env) into a CUE struct
• Automatically decrypt SOPS-encrypted data
• Load remote charts

Install

Download the latest release or build with go1.20rc3 or later:

go install github.com/noris-network/cuegen@latest

Usage

cuegen path/to/cuegen.yaml
# or
cuegen path/to/directory-containing-cuegen-dot-yaml
# or
cuegen https://git.example.com/deployments/myapp.git

Have a look at the examples for some ready-to-run examples.

Cuegen can be used as config management plugin in ArgoCD. A container with the plugin.yaml configuration is available at docker hub. The cuegen-cmp plugin container expects the cuegen config file to be named cuegen.yaml.

Configuration

A configuration file (preferred name: cuegen.cue, schema) is required to run cuegen. For backwards compatibility the yaml format will still be supported in the future.

cuegen: {
  objectsPath:    "objects"          // this will be dumped to YAML
  secretDataPath: "secret.*.data"    // values matching this path will be loaded as []byte and
                                     //     automatically be base64 encoded when dumped to YAML
  checkPath:                         // this path is checked to contain only concrete values
  checkPaths: [                      // when more than one path needs to be checked, the
    "values",                        //     'objectsPath' with always be checked.
    "global",
  ]
  components: [                      // merge cue files from these soures into the main entrypoint
    "https://$GITLAB_TOKEN@gitlab.noris.net/mcs/components/cuegen/mongodb.git?ref=v6.0.4-mcs.1",
    "https://$GITLAB_TOKEN@gitlab.noris.net/mcs/components/cuegen/wekan.git?ref=v6.71-mcs.0",
  ]
  debug: false                       // print some info useful for debugging
}

Environment Variables

Some environment variables can help working with cuegen:

CUEGEN_DEBUG              turn on debug output with "true"
CUEGEN_HTTP_PASSWORD      password for git authentication
CUEGEN_HTTP_USERNAME      username for git authentication
DUMP_OVERLAYS_TO          directory to dump overlays to (debug only)
SOPS_AGE_KEY              age key for decryption
SOPS_AGE_KEY_FILE         age key file for decryption
YQ_PRETTYPRINT            !="": run yaml output thru `yq -P`
                          starting with `/`: use as path to yq

Components

Components can be

• directories outside the main chart
• zip files
• git repositories

Local paths are not restricted. As this could be a security problem, this will change in a future release.

Environment variables in components are expanded, that can be used to e.g. add authentication to git urls or prefix local paths. For cloning private repositories via Http url, CUEGEN_HTTP_USERNAME and CUEGEN_HTTP_PASSWORD can also be set in the environment.

Components Example:

cuegen: components: [
  "../database-chart",
  "../common-static-resources.zip",
  "https://github.com/nxcc/cuegen-test-chart-one?ref=v0.1.0",
  "https://github.com/noris-network/cuegen?ref=v0.2.1#examples/configmap",
]

Because of the way CUE and cuegen work, all *.cue files need to be in the root of the components directory. No special files need to be present for a chart, and although cuegen does not require or evaluate them, it is recommended to add metadata (see examples). For git repositories, a branch or tag name can be specified with the ref parameter. If a zip file has only one directory in it's root, this level is skipped and cuegen uses all files in that directory.

Attributes

The directory containing cuegen.yaml is the base directory, all paths are relative to this.

Attribute "@readfile"

Attribute @readfile loads contents of given file(s) into a CUE value. An optional suffix controlls whitespace (=nl: ensure \n at end, =trim: trim whitespace from begin and end).

Load version.txt into Version, trim any whitespace from begin and end:

Version: string @readfile(version.txt=trim)

Load my.crt and chain.crt concatenated into Certificate. A \n is ensured to be placed after my.crt:

Certificate: string @readfile(my.crt=nl, chain.crt)

Attribute "@readmap"

Attribute @readmap loads contents of given file(s) into a CUE value as key/value data. When the cue path matches secretDataPath, values are inserted as []byte. This will automatically base64 encode them when dumped to YAML. This is useful for Secrets. This can also be forced by an =bytes suffix.

Load values from data.json as []bytes:

secret: foo: {
  data: {} @readmap(data.json=bytes)
}

Attribute "@read"

with file Attribute value

Attribute @read(filename) tries to load structured data (JSON/YAML/env) into a CUE struct.

Load key/value data from env.env and data.yaml into configMap.myconfig.data:

configMap: "myconfig": {
    data: {} @read(env.env, data.yaml)
}

with directory Attribute value

Attribute @read(directory) tries to load regular files from the given directory as key/values into a CUE struct.

Load all files from directory scripts as key/values into configMap.scripts.data:

configMap: "scripts": {
    data: {} @read(scripts)
}

Order Workaround

Until issue 2555 is resolved in CUE, there is a temporary workaround.

Changelog

• v0.1.0 - Initial release
• v0.1.1 - Improved attribute lookup
• v0.2.0 - Added checks when reading cuegen.yaml
• v0.2.1 - Improved error messages
• v0.3.0 - Added ability to read subpaths from git repos
• v0.3.1 - No code changes, trigger cmp build
• v0.3.2 - No code changes, bump go version to 1.20
• v0.4.0 - switch default config to cuegen.cue, use cue v0.5.0-beta5
• v0.4.1 - Make components & checkPaths optional
• v0.4.2 - downgrade cue to v0.5.0-beta.2 (performance regression)
• v0.4.3 - fix running as kustomize plugin
• v0.4.4 - improve handling of git urls
• v0.5.0 - add cuegen default config
• v0.6.0 - add dumpOverlays option
• v0.7.0 - upgrade cue to v0.5.0 (many fixes, rare performance regression still present)
• v0.7.1 - fix secret handling of @readfile
• v0.7.2 - internal cleanup
• v0.8.0 - allow remote cuegen directories, rm kustomize plugin support
• v0.9.0 - upgrade cue to v0.6.0
• v0.10.0 - add YQ_PRETTYPRINT to filter output thru yq -P
• v0.11.0 - add DUMP_OVERLAYS_TO to dump overlays to directory (debug)
• v0.11.1 - upgrade sops
• v0.12.0 - add workaround for file order bug
• v0.13.0 - No code changes, bump cue version to 0.7.0
• v0.13.1 - bump deps
• v0.14.x - "silent release" of v1alpha2 schema...
• v0.14.4 - No code changes, bump cue version to 0.8.0