go-rancher-gen

README

go-rancher-gen

rancher-gen is a file generator that renders templates using Rancher Metadata.

Core features:

• Powerful template syntax that embraces Rancher services, containers and hosts as first-class objects
• Ability to run arbitrary commands when a file has been updated (e.g. to reload an application's configuration)
• Ability to run check commands on staged files before updating the destination files
• Ability to specify multiple template sets using a TOML config file

Usage

Command Line

rancher-gen [options] source [dest]

options

Flag	Description
config	Path to an optional config file. Options specified on the CLI always take precedence.
metadata-version	Metadata version string used when querying the Rancher Metadata API. Default: latest.
include-inactive	Not yet implemented
interval	Interval (in seconds) for polling the Metadata API for changes. Default: 5.
onetime	Process all templates once and exit. Default: false.
log-level	Verbosity of log output. Default: info.
check-cmd	Command to check the content before updating the destination. Use the {{staging}} placeholder to reference the staging file.
notify-cmd	Command to run after the destination file has been updated.
notify-lbl	Run command once for each container with this label.
notify-output	Print the result of the notify command to STDOUT.
version	Show application version and exit.

source

Path to the template.

dest

Path to the destination file. If omitted, then the generated content is printed to STDOUT.

Examples

rancher-gen --onetime --notify-cmd="/usr/sbin/service nginx reload" \
/etc/rancher-gen/nginx.tmpl /etc/nginx/nginx.conf

rancher-gen --interval 2 --check-cmd="/usr/sbin/nginx -t -c {{staging}}" \
--notify-cmd="/usr/sbin/service nginx reload" /etc/rancher-gen/nginx.tmpl /etc/nginx/nginx.conf

Configuration file

You can optionally pass a configuration file to rancher-gen. The configuration file is a TOML file. It allows you to specify multiple template sets grouped by template sections. You can specify the same options as on the command line. Options specified on the command line or via environment variables take precedence over the corresponding values in the configuration file. An example file is available here.

How to dynamically configure your applications with Rancher Metadata

You can bundle rancher-gen with the application image or run it as a service sidekick, that exposes the generated configuration file in a shared volume.

Bundled with application image

Download the binary from the release page. Add the binary to your Docker image and provide a mechanism that runs rancher-gen on container start and then executes the main application. This functionality could be provided by a Bash script executed as image ENTRYPOINT. If you want to reload the application whenever the Metadata referenced in the template changes, you can use a container process supervisor (e.g. S6-overlay) to keep rancher-gen running in the background and notify the application when it needs to reload the configuration (by sending it a SIGHUP for example).

Sidekick Container

Create a new Docker image using janeczku/rancher-gen:latest as base. Add the template(s) and configuration file(s) to the image. Expose the configuration folder as VOLUME. Run rancher-gen on container start, specifying relevant options as command line parameters.

Example acme/nginx-config sidekick image

FROM janeczku/rancher-gen:latest
COPY config.toml /etc/rancher-gen/
COPY nginx.tmpl /etc/rancher-gen/
VOLUME /etc/nginx
CMD ["--config", "/etc/rancher-gen/config.toml"]

Example Rancher Compose file

nginx:
  image: nginx:latest
  volumes_from:
  - config-sidekick
  labels:
    io.rancher.sidekicks: template-sidekick
config-sidekick:
  image: acme/nginx-config

Notify Labels

Specifying a notify-lbl config parameter will run the check-cmd and notify-cmd hook once for each container matching the notify-lbl value. This parameter can be a single label name (ie. notify-lbl = "mylabel") or a specific label:value pair (ie. notify-lbl = "mylabel:myvalue"). Label names and values are colon-separated.

Container fields like {{.Name}} and {{.Address}} (see the Container model for reference) values can be used as template variables in the notify-cmd string. This allows the notify-cmd to be tailored to specific containers. Labels can be accessed by period separation (ie. {{.Labels.my.label}}. See the following example which appends text to a file:

Example config.toml file

[[template]]
source = "/etc/rancher-gen/nginx.tmpl"
dest = "/etc/rancher-gen/nginx.conf"
check-cmd = "echo $(date)\t check >> /etc/rancher-gen/check.log"
notify-cmd = "echo $(date)\t notify N:{{Name}} A:{{Address}} >> /etc/rancher-gen/notify.log"
notify-lbl = "testlabel:1"
notify-output = true

This example will run notify-cmd once for each unique container with a label name of testlabel and a label value of 1, and the container Name and Address will substitue for values during runtime.

Example Output

Mon Feb 13 22:58:39 UTC 2017 notify N:whoami-whoami-1 A:10.42.85.195
Mon Feb 13 22:58:39 UTC 2017 notify N:whoami2-whoami-1 A:10.42.130.246
Mon Feb 13 22:58:39 UTC 2017 notify N:whoami3-whoami-1 A:10.42.234.149

Template Language

Templates are Go text templates. In addition to the built-in functions, rancher-gen exposes functions and methods to easily discover Rancher services, containers and hosts.

Service Discovery Objects

type Service struct {
	Name        string
	Stack       string
	Kind        string
	Vip         string
	Fqdn        string
	Ports       []Port
	Labels      LabelMap
	Metadata    MetadataMap
	Containers  []Container
}

type Port struct {
	PublicPort   string
	InternalPort string
	Protocol     string
}

type Container struct {
	Name        string
	Address     string
	Stack       string
	Service     string
	Health      string
	State       string
	Labels      LabelMap
	Host        Host
}

type Host struct {
	UUID        string
	Name        string
	Address     string
	Hostname    string
	Labels      LabelMap
}

The LabelMap and MetadataMap types implement methods for easily checking the existence of specific keys and accessing their values:

Labels.Exists(key string) bool
Returns true if the given label key exists in the map.

Labels.GetValue(key, default string) string
Returns the value of the given label key. The function accepts an optional default value that is returned when the key doesn't exist or is set to an empty string.

Metadata.Exists(key string) bool
Returns true if the given metadata key exists in the map.

Metadata.GetValue(key, default interface{}) interface{}
Returns the value of the given label key. The function accepts an optional default value that is returned when the key doesn't exist.

Examples:

Check if the label exists:

{{range services}}
{{if .Labels.Exists "foo"}}
{{do something}}
{{end}}
{{end}}

Get the value of a Metadata key:

{{range services}}
Metadata foo: {{.Metadata.GetValue "foo"}}
{{end}}

Using a default value:

{{range services}}
Label foo: {{.Labels.GetValue "foo" "default value"}}
{{end}}

Service Discovery Functions

host

Lookup a specific host

Optional argument
UUID string
Return Type
Host

If the argument is omitted the local host is returned:

{{host}}

hosts

Lookup hosts

Optional parameters
labelSelector string
Returned Type
[]Host

The function returns a slice of Host which can be used for ranging in a template:

{{range hosts}}
host {{.Name}} {{.Address}}
{{end}}

which would produce something like:

host aws-sm-01 148.210.10.10
host aws-sm-02 148.210.10.11

One or multiple label selectors can be passed as arguments to limit the result to hosts with matching labels. The syntax of the label selector is @label-key=label-value.

The following function returns only hosts that have a label "foo" with the value "bar":

{{hosts "@foo=bar"}}

The label selector syntax supports a regex pattern on it's right side. E.g. to lookup hosts that have a specific label regardless of the value:

{{hosts "@foo=.*"}}

If the argument is omitted all hosts are returned:

{{hosts}}

service

Lookup a specific service

Optional parameter
serviceIdentifier string
Returned Type
Service

The function returns a Service struct. You can use the Containers field for ranging over all containers belonging to the service:

{{with service "web.production"}}
{{range .Containers}}
http://{{.Address}}:9090
{{end}}
{{end}}

which would produce something like:

http://10.12.20.111:9090
http://10.12.20.122:9090

The syntax of the serviceIdentifier parameter is service-name[.stack-name]:

{{service "web.production"}}

If the stack name is omitted the service is looked up in the local stack:

{{service "web"}}

If no argument is given the local service is returned:

{{service}}

services

Lookup services matching the given stack and label selectors

Optional parameters
stackSelector string
labelSelector string
Return Type
[]Service

Just like with the hosts function multiple label selectors can be passed to select services with matching labels:

{{services "@foo=bar"}}

The stack selector parameter uses the syntax .stack-name:

{{services ".production"}}

Stack and label selectors can be combined like this:

{{services ".production" "@foo=bar"}}

If arguments are omitted then all services are returned:

{{services}}

Helper Functions and Pipes

whereLabelExists

Filter a slice of hosts, services or containers returning the items that have the given label key.

Parameters
labelKey string
input []Host, []Service or []Container
Return Type
same as input

whereLabelEquals

Filter a slice of hosts, services or containers returning the items that have the given label key and value.

Arguments
labelKey string
labelValue string
input []Host, []Service or []Container
Return Type
same as input

{{$ervice := service "web.production"}}
{{range $container := whereLabelEquals "foo" "bar" $service.Containers}}
{{do something with $container}}
{{end}}

whereLabelMatches

Filter a slice of hosts, services or containers returning the items that have the given label and a value matching the regex pattern.

Arguments
labelKey string
regexPattern string
input []Host, []Service or []Container
Return Type
same as input

groupByLabel

This function takes a slice of hosts, services or containers and groups the items by their value of the given label. It returns a map with label values as key and a slice of corresponding elements items as value.

Arguments
label-key string
input []Host,[]Service,[]Container
Return Type
map[string][]Host/[]Service/[]Container

{{range $labelValue, $hosts := hosts | groupByLabel "foo"}}
{{$labelValue}}
{{range $hosts}}
IP: {{.Address}}
{{end}}
{{end}}

base

Alias for the path.Base function

filename: {{$service.Metadata.GetValue "targetPath" | base}}

See Go's path.Base() for more information.

dir

Alias for the path.Dir function

See Go's path.Dir() for more information.

exists

Test for existence of path or file

{{ exists (/path/to/file) }}

env

Returns the value of the given environment variable or an empty string if the variable isn't set

{{env "FOO_VAR"}}

timestamp

Alias for time.Now

# Generated by rancher-gen {{timestamp}}

The timestamp can be formatted as required by invoking the Format method:

# Generated by rancher-gen {{timestamp.Format "Jan 2, 2006 15:04"}}

See Go's time.Format() for more information about formatting the date according to the layout of the reference time.

split

Alias for strings.Split

{{$items := split $someString ":"}}

See Go's strings.Split() for more information.

join

Alias for strings.Join
Takes the given slice of strings as a pipe and joins them on the provided string:

{{$items | join ","}}

See Go's strings.Join() for more information.

toLower

Alias for strings.ToLower
Takes the argument as a string and converts it to lowercase.

{{$svc.Metadata.GetValue "foo" | toLower}}

See Go's strings.ToLower() for more information.

toUpper

Alias for strings.ToUpper
Takes the argument as a string and converts it to uppercase.

{{$svc.Metadata.GetValue "foo" | toUpper}}

See Go's strings.ToUpper() for more information.

contains

Alias for strings.Contains

See Go's strings.Contains() for more information.

replace

Alias for strings.Replace

{{$foo := $svc.Labels.GetValue "foo"}}
foo: {{replace $foo "-" "_" -1}}

See Go's strings.Replace() for more information.

Examples

TODO

Development

Prerequisites:

1. Go
2. Running Rancher instance
3. [rancher-compose(https://docs.rancher.com/rancher/v1.4/en/cattle/rancher-compose/) (and associated environment variables)

Setup:

1. Ensure Go is installed, cd into your go-work (or equivalent) directory within $GOPATH
2. git clone this repo, cd into repo
3. Ensure test directory exists: mkdir -p test

Development workflow can happen as follows:

1. Build project: make build
2. Create Docker image: make dev-image
3. Run in Rancher: rancher-compose up (add -d if needed)
4. Inspect, and rancher-compose rm when finished. Repeat.

See make for more commands.