consul-template

README

Consul Template

This project provides a convenient way to populate values from Consul into the filesystem using the consul-template daemon.

The daemon consul-template queries a Consul instance and updates any number of specified templates on the filesystem. As an added bonus, consul-template can optionally run arbitrary commands when the update process completes. See the Examples section for some scenarios were this functionality might prove useful.

The documentation in this README corresponds to the master branch of Consul Template. It may contain unreleased features or different APIs than the most recently released version. Please see the Git tag that corresponds to your version of Consul Template for the proper documentation.

Installation

You can download a released consul-template artifact from the Consul Template release page on GitHub. If you wish to compile from source, you will need to have buildtools and Go installed:

$ git clone https://github.com/hashicorp/consul-template.git
$ cd consul-template
$ make

This process will create bin/consul-template which make be invoked as a binary.

Usage

Options

Option	Description
auth	The basic authentication username (and optional password), separated by a colon. There is no default value.
consul*	The location of the Consul instance to query (may be an IP address or FQDN) with port.
max-stale	The maximum staleness of a query. If specified, Consul will distribute work among all servers instead of just the leader. The default value is 0 (none).
ssl	Use HTTPS while talking to Consul. Requires the Consul server to be configured to serve secure connections. The default value is false.
ssl-verify	Verify certificates when connecting via SSL. This requires the use of -ssl. The default value is true.
syslog	Send log output to syslog (in addition to stdout and stderr). The default value is false.
syslog-facility	The facility to use when sending to syslog. This requires the use of -syslog. The default value is LOCAL0.
token	The Consul API token. There is no default value.
template*	The input template, output path, and optional command separated by a colon (:). This option is additive and may be specified multiple times for multiple templates.
wait	The minimum(:maximum) to wait before rendering a new template to disk and triggering a command, separated by a colon (:). If the optional maximum value is omitted, it is assumed to be 4x the required minimum value. There is no default value.
retry	The amount of time to wait if Consul returns an error when communicating with the API. The default value is 5 seconds.
config	The path to a configuration file or directory of configuration files on disk, relative to the current working directory. Values specified on the CLI take precedence over values specified in the configuration file. There is no default value.
log-level	The log level for output. This applies to the stdout/stderr logging as well as syslog logging (if enabled). Valid values are "debug", "info", "warn", and "err". The default value is "warn".
dry	Dump generated templates to the console. If specified, generated templates are not committed to disk and commands are not invoked. (CLI-only)
once	Run Consul Template once and exit (as opposed to the default behavior of daemon). (CLI-only)
version	Output version information and quit. (CLI-only)

* = Required parameter

Command Line

The CLI interface supports all of the options detailed above.

Query the nyc1 demo Consul instance, rendering the template on disk at /tmp/template.ctmpl to /tmp/result, running Consul Template as a service until stopped:

$ consul-template \
  -consul demo.consul.io \
  -template "/tmp/template.ctmpl:/tmp/result"

Query a local Consul instance, rendering the template and restarting nginx if the template has changed, once, polling 30s if Consul is unavailable:

$ consul-template \
  -consul 127.0.0.1:8500 \
  -template "/tmp/template.ctmpl:/var/www/nginx.conf:service nginx restart" \
  -retry 30s \
  -once

Query a Consul instance, rendering multiple templates and commands as a service until stopped:

$ consul-template \
  -consul my.consul.internal:6124 \
  -template "/tmp/nginx.ctmpl:/var/nginx/nginx.conf:service nginx restart" \
  -template "/tmp/redis.ctmpl:/var/redis/redis.conf:service redis restart" \
  -template "/tmp/haproxy.ctmpl:/var/haproxy/haproxy.conf"

Query a Consul instance that requires authentication, dumping the templates to stdout instead of writing to disk. In this example, the second and third parameters to the -template option are required but ignored. The file will not be written to disk and the optional command will not be executed:

$ consul-template \
  -consul my.consul.internal:6124 \
  -template "/tmp/template.ctmpl:/tmp/result:service nginx restart"
  -dry

Configuration File(s)

The Consul Template configuration files are written in HashiCorp Configuration Language (HCL). By proxy, this means the Consul Template configuration file is JSON-compatible. For more information, please see the HCL specification.

The Configuration file syntax interface supports all of the options detailed above, unless otherwise noted in the table.

consul = "127.0.0.1:8500"
token = "abcd1234"
retry = "10s"
max_stale = "10m"

auth {
  enabled = true
  username = "test"
  password = "test"
}

ssl {
  enabled = true
  verify = false
}

syslog {
  enabled = true
  facility = "LOCAL5"
}

template {
  source = "/path/on/disk/to/template"
  destination = "/path/on/disk/where/template/will/render"
  command = "optional command to run when the template is updated"
}

template {
  // Multiple template definitions are supported
}

Query the nyc1 demo Consul instance, rendering the template on disk at /tmp/template.ctmpl to /tmp/result, running Consul Template as a service until stopped:

consul = "nyc1.demo.consul.io"

template {
  source = "/tmp/template.ctmpl"
  destination  = "/tmp/result"
}

If a directory is given instead of a file, all files in the directory (recursively) will be merged in lexical order. So if multiple files declare a "consul" key for instance, the last one will be used.

Commands specified on the command line take precedence over those defined in a config file!

Templating Language

Consul Template consumes template files in the Go Template format. If you are not familiar with the syntax, we recommend reading the documentation, but it is similar in appearance to Mustache, Handlebars, or Liquid.

In addition to the Go-provided template functions, Consul Template exposes the following functions:

API Functions

datacenters

Query Consul for all datacenters in the catalog. Datacenters are queried using the following syntax:

{{datacenters}}

file

Read and render the contents of a local file on disk. If the file cannot be read, an error will occur. Files are read using the following syntax:

{{file "/path/to/local/file"}}

This example will render the entire contents of the file at /path/to/local/file into the template.

key

Query Consul for the value at the given key. If the key cannot be converted to a string-like value, an error will occur. Keys are queried using the following syntax:

{{key "service/redis/maxconns@east-aws"}}

The example above is querying Consul for the service/redis/maxconns in the east-aws datacenter. If you omit the datacenter attribute, the local Consul datacenter will be queried:

{{key "service/redis/maxconns"}}

The beauty of Consul is that the key-value structure is entirely up to you!

ls

Query Consul for all top-level key-value pairs at the given prefix. If any of the values cannot be converted to a string-like value, an error will occur:

{{range ls "service/redis@east-aws"}}
{{.Key}} {{.Value}}{{end}}

If the Consul instance had the correct structure at service/redis in the east-aws datacenter, the resulting template could look like:

minconns 2
maxconns 12

If you omit the datacenter attribute on ls, the local Consul datacenter will be queried.

nodes

Query Consul for all nodes in the catalog. Nodes are queried using the following syntax:

{{nodes}}

This example will query Consul's default datacenter. You can specify an optional parameter to the nodes call to specify the datacenter:

{{nodes "@east-aws"}}

This will query Consul for all nodes in the east-aws datacenter.

service

Query Consul for the service group(s) matching the given pattern. Services are queried using the following syntax:

{{service "release.webapp@east-aws:8000"}}

The example above is querying Consul for healthy "webapp" services, with the "release" tag, in the "east-aws" datacenter, using port "8000". The tag, datacenter and port attributes are optional. To query all nodes of the "webapp" service (regardless of tag and port) for the current datacenter:

{{service "webapp"}}

The function returns a []*Service struct which can be used for ranging in a template:

{{range service "webapp@datacenter"}}
server {{.Name}} {{.Address}}:{{.Port}}{{end}}

which would produce something like:

server nyc_web_01 123.456.789.10:8080
server nyc_web_02 456.789.101.213:8080

By default only healthy services are returned. If you want to get all services, you can pass the "any" option:

{{service "webapp" "any"}}

This will return all services registered to the agent, regardless of their status.

If you want to filter services by a specific health or health(s), you can specify a comma-separated list of health check statuses:

{{service "webapp" "passing, warning"}}

This will returns services which are deemed "passing" or "warning" according to their node and service-level checks defined in Consul. Please note that the comma implies an "or", not an "and".

Specifying more than one status filter while "any" is used will return an error, since "any" is the superset of all status filters.

There is an architectural difference between the following:

{{service "webapp"}}
{{service "webapp" "passing"}}

The former will return all services which Consul considers "healthy" and passing. The latter will return all services registered with the Consul agent and perform client-side filtering. As a general rule, you should not use the "passing" argument alone if you want only healthy services - simply omit the second argument instead. However, the extra argument is useful if you want "passing or warning" services like:

{{service "webapp" "passing, warning"}}

The service's status is also exposed if you need to do additional filtering:

{{range service "webapp" "any"}}
{{if eq .Status "critical"}}
// Critical state!{{end}}
{{if eq .Status "passing"}}
// Ok{{end}}

To put a service into maintenance mode in Consul around executing the command, simply wrap your command in a consul maint call:

#!/bin/sh
set -e
consul maint -enable -service webapp -reason "Consul Template updated"
service nginx reload
consul maint -disable -service webapp

Alternatively, if you do not have the Consul agent installed, you can make the API requests directly (advanced):

#!/bin/sh
set -e
curl -X PUT "http://$CONSUL_HTTP_ADDR/v1/agent/service/maintenance/webapp?enable=true&reason=Consul+Template+Updated"
service nginx reload
curl -X PUT "http://$CONSUL_HTTP_ADDR/v1/agent/service/maintenance/webapp?enable=false"

services

Query Consul for all services in the catalog. Services are queried using the following syntax:

{{services}}

This example will query Consul's default datacenter. You can specify an optional parameter to the services call to specify the datacenter:

{{services "@east-aws"}}

Please be advised: the services function is different than service, which accepts more parameters and queries the health endpoints for a list of services. This endpoint queries the Consul catalog which returns a map of services to its list of tags. For example:

{{range services}}
{{.Name}}
{{range .Tags}}
  {{.}}{{end}}
{{end}}

tree

Query Consul for all key-value pairs at the given prefix. If any of the values cannot be converted to a string-like value, an error will occur:

{{range tree "service/redis@east-aws"}}
{{.Key}} {{.Value}}{{end}}

If the Consul instance had the correct structure at service/redis in the east-aws datacenter, the resulting template could look like:

minconns 2
maxconns 12
nested/config/value "value"

Unlike ls, tree returns all keys under the prefix, just like the Unix tree command.

If you omit the datacenter attribute on tree, the local Consul datacenter will be queried.

---

Helper Functions

byKey

Takes the list of key pairs returned from a tree function and creates a map that groups pairs by their top-level directory. For example, if the Consul KV store contained the following structure:

groups/elasticsearch/es1
groups/elasticsearch/es2
groups/elasticsearch/es3
services/elasticsearch/check_elasticsearch
services/elasticsearch/check_indexes

With the following template:

{{range $key, $pairs := tree "groups" | byKey}}{{$key}}:
{{range $pair := $pairs}}  {{.Key}}={{.Value}}
{{end}}{{end}}

The result would be:

elasticsearch:
  es1=1
  es2=1
  es3=1

Note that the top-most key is stripped from the Key value. Keys that have no prefix after stripping are removed from the list.

The resulting pairs are keyed as a map, so it is possible to look up a single value by key:

{{$weights := tree "weights"}}
{{range service "release.webapp"}}
  {{$weight := or (index $weights .Node) 100}}
  server {{.Node}} {{.Address}}:{{.Port}} weight {{$weight}}{{end}}

byTag

Takes the list of services returned by the service function and creates a map that groups services by tag.

{{range $tag, $services := service "webapp" | byTag}}{{$tag}}
{{range $services}} server {{.Name}} {{.Address}}:{{.Port}}
{{end}}{{end}}

env

Reads the given environment variable accessible to the current process.

{{env "CLUSTER_ID"}}

This function can be chained to manipulate the output:

{{env "CLUSTER_ID" | toLower}}

loop

Accepts varying parameters and differs its behavior based on those parameters.

If loop is given one integer, it will return a goroutine that begins at zero and loops up to but not including the given integer:

{{range loop 5}}
# Comment{{end}}

If given two integers, this function will return a goroutine that begins at the first integer and loops up to but not including the second integer:

{{range $i := loop 5 8}}
stanza-{{$i}}{{end}}

which would render:

stanza-5
stanza-6
stanza-7

Note: It is not possible to get the index and the element since the function returns a goroutine, not a slice. In other words, the following is not valid:

# Will NOT work!
{{range $i, $e := loop 5 8}}
# ...{{end}}

parseJSON

Takes the given input (usually the value from a key) and parses the result as JSON:

{{with $d := key "user/info" | parseJSON}}{{$d.name}}{{end}}

Note: Consul Template evaluates the template multiple times, and on the first evaluation the value of the key will be empty (because no data has been loaded yet). This means that templates must guard against empty responses. For example:

{{with $d := key "user/info" | parseJSON}}
{{if $d}}
...
{{end}}
{{end}}

It just works for simple keys. But fails if you want to iterate over keys or use index function. Wrapping code that access object with {{ if $d }}...{{end}} is good enough.

Alternatively you can read data from a local JSON file:

{{with $d := file "/path/to/local/data.json" | parseJSON}}{{$d.some_key}}{{end}}

regexReplaceAll

Takes the argument as a regular expression and replaces all occurences of the regex with the given string. As in go, you can use variables like $1 to refer to subexpressions in the replacement string.

{{"foo.bar" | regexReplaceAll "foo([.a-z]+)" "$1"}}

replaceAll

Takes the argument as a string and replaces all occurences of the given string with the given string.

{{"foo.bar" | replaceAll "." "_"}}

This function can be chained with other functions as well:

{{service "webapp"}}{{.Name | replaceAll ":" "_"}}{{end}}

timestamp

Returns the current timestamp as a string (UTC). If no arguments are given, the result is the current RFC3339 timestamp:

{{timestamp}} // e.g. 1970-01-01T00:00:00Z

If the optional parameter is given, it is used to format the timestamp using the magic reference date Mon Jan 2 15:04:05 -0700 MST 2006:

{{timestamp "2006-01-02"}} // e.g. 1970-01-01

See Go's time.Format() for more information.

toLower

Takes the argument as a string and converts it to lowercase.

{{key "user/name" | toLower}}

See Go's strings.ToLower() for more information.

toTitle

Takes the argument as a string and converts it to titlecase.

{{key "user/name" | toTitle}}

See Go's strings.Title() for more information.

toUpper

Takes the argument as a string and converts it to uppercase.

{{key "user/name" | toUpper}}

See Go's strings.ToUpper() for more information.

Caveats

Termination on Error

By default Consul Template is highly fault-tolerant. If Consul is unreachable or a template changes, Consul Template will happily continue running. The only exception to this rule is if the optional command exits non-zero. In this case, Consul Template will also exit non-zero. The reason for this decision is so the user can easily configure something like Upstart or God to manage Consul Template as a service.

If you want Consul Template to continue watching for changes, even if the optional command argument fails, you can append || true to your command. For example:

$ consul-template \
  -template "in.ctmpl:out.file:service nginx restart || true"

In this example, even if the Nginx restart command returns non-zero, the overall function will still return an OK exit code; Consul Template will continue to run as a service. Additionally, if you have complex logic for restarting your service, you can intelligently choose when you want Consul Template to exit and when you want it to continue to watch for changes. For these types of complex scripts, we recommend using a custom sh or bash script instead of putting the logic directly in the consul-template command or configuration file.

File Permissions

Consul Template uses Go's file modification libraries under the hood. If a file at the destination path already exists, Consul Template will do its best to preserve the existing file permissions. For non-existent files, Go will default to the system default. If you require specific file permissions on the output file, you can use the optional command parameter and chmod, for example:

consul-template \
  -template "/tmp/nginx.ctmpl:/var/nginx/nginx.conf:chmod 644 /var/nginx/nginx.conf && sudo restart nginx"

template {
  source = "/tmp/nginx.ctmpl"
  destination = "/var/nginx/nginx.conf"
  command = "chmod 644 /var/nginx/nginx.conf && sudo restart nginx"
}

Command Environment

The current processes environment is used when executing commands with the following additional environment variables:

• CONSUL_HTTP_ADDR
• CONSUL_HTTP_TOKEN
• CONSUL_HTTP_AUTH
• CONSUL_HTTP_SSL
• CONSUL_HTTP_SSL_VERIFY

These environment variables are exported with their current values when the command executes. Other Consul tooling reads these environment variables, providing smooth integration with other Consul tools (like consul maint or consul lock). Additionally, exposing these environment variables gives power users the ability to further customize their command script.

Examples

HAProxy

HAProxy is a very common load balancer. You can read more about the HAProxy configuration file syntax in the HAProxy documentation, but here is an example template for rendering an HAProxy configuration file with Consul Template:

global
    daemon
    maxconn {{key "service/haproxy/maxconn"}}

defaults
    mode {{key "service/haproxy/mode"}}{{range ls "service/haproxy/timeouts"}}
    timeout {{.Key}} {{.Value}}{{end}}

listen http-in
    bind *:8000{{range service "release.webapp"}}
    server {{.Node}} {{.Address}}:{{.Port}}{{end}}

Save this file to disk as haproxy.ctmpl and run the consul-template daemon:

$ consul-template \
  -consul demo.consul.io \
  -template haproxy.ctmpl:/etc/haproxy/haproxy.conf
  -dry

Depending on the state of the demo Consul instance, you could see the following output:

global
    daemon
    maxconn 4

defaults
    mode default
    timeout 5

listen http-in
    bind *:8000
    server nyc3-worker-2 104.131.109.224:80
    server nyc3-worker-3 104.131.59.59:80
    server nyc3-worker-1 104.131.86.92:80

For more information on how to save this result to disk or for the full list of functionality available inside a Consul template file, please consult the API documentation.

Varnish

Varnish is an common caching engine that can also act as a proxy. You can read more about the Varnish configuration file syntax in the Varnish documentation, but here is an example template for rendering a Varnish configuration file with Consul Template:

import directors;
{{range service "consul"}}
backend {{.Name}}_{{.ID}} {
    .host = "{{.Address}}";
    .port = "{{.Port}}";"
}{{end}}

sub vcl_init {
  new bar = directors.round_robin();
{{range service "consul"}}
  bar.add_backend({{.Name}}_{{.ID}});{{end}}
}

sub vcl_recv {
  set req.backend_hint = bar.backend();
}

Save this file to disk as varnish.ctmpl and run the consul-template daemon:

$ consul-template \
  -consul demo.consul.io \
  -template varnish.ctmpl:/etc/varnish/varnish.conf \
  -dry

You should see the following output:

import directors;

backend consul_consul {
    .host = "104.131.109.106";
    .port = "8300";"
}

sub vcl_init {
  new bar = directors.round_robin();

  bar.add_backend(consul_consul);
}

sub vcl_recv {
  set req.backend_hint = bar.backend();
}

Apache httpd

Apache httpd is a popular web server. You can read more about the Apache httpd configuration file syntax in the Apache httpd documentation, but here is an example template for rendering part of an Apache httpd configuration file that is responsible for configuring a reverse proxy with dynamic end points based on service tags with Consul Template:

{{range $tag, $service := service "web" | byTag}}
# "{{$tag}}" api providers.
<Proxy balancer://{{$tag}}>
{{range $service}}  BalancerMember http://{{.Address}}:{{.Port}}
{{end}} ProxySet lbmethod=bybusyness
</Proxy>
Redirect permanent /api/{{$tag}} /api/{{$tag}}/
ProxyPass /api/{{$tag}}/ balancer://{{$tag}}/
ProxyPassReverse /api/{{$tag}}/ balancer://{{$tag}}/
{{end}}

Just like the previous examples, save this file to disk and run the consul-template daemon:

$ consul-template \
  -consul demo.consul.io \
  -template httpd.ctmpl:/etc/httpd/sites-available/balancer.conf

You should see output similar to the following:

# "frontend" api providers.
<Proxy balancer://frontend>
  BalancerMember http://104.131.109.106:8080
  BalancerMember http://104.131.109.113:8081
  ProxySet lbmethod=bybusyness
</Proxy>
Redirect permanent /api/frontend /api/frontend/
ProxyPass /api/frontend/ balancer://frontend/
ProxyPassReverse /api/frontend/ balancer://frontend/

# "api" api providers.
<Proxy balancer://api>
  BalancerMember http://104.131.108.11:8500
  ProxySet lbmethod=bybusyness
</Proxy>
Redirect permanent /api/api /api/api/
ProxyPass /api/api/ balancer://api/
ProxyPassReverse /api/api/ balancer://api/

Querying all services

As of Consul Template 0.6.0, it is possible to have a complex dependency graph with dependent services. As such, it is possible to query and watch all services in Consul:

{{range services}}# {{.Name}}{{range service .Name}}
{{.Address}}{{end}}

{{end}}

Just like the previous examples, save this file to disk and run the consul-template daemon:

$ consul-template \
  -consul demo.consul.io \
  -template everything.ctmpl:/tmp/inventory

You should see output similar to the following:

# consul
104.131.121.232

# redis
104.131.86.92
104.131.109.224
104.131.59.59

# web
104.131.86.92
104.131.109.224
104.131.59.59

Debugging

Consul Template can print verbose debugging output. To set the log level for Consul Template, use the -log-level flag:

$ consul-template -log-level info ...

<timestamp> [INFO] (cli) received redis from Watcher
<timestamp> [INFO] (cli) invoking Runner
# ...

You can also specify the level as debug:

$ consul-template -log-level debug ...

<timestamp> [DEBUG] (cli) creating Runner
<timestamp> [DEBUG] (cli) creating Consul API client
<timestamp> [DEBUG] (cli) creating Watcher
<timestamp> [DEBUG] (cli) looping for data
<timestamp> [DEBUG] (watcher) starting watch
<timestamp> [DEBUG] (watcher) all pollers have started, waiting for finish
<timestamp> [DEBUG] (redis) starting poll
<timestamp> [DEBUG] (service redis) querying Consul with &{...}
<timestamp> [DEBUG] (service redis) Consul returned 2 services
<timestamp> [DEBUG] (redis) writing data to channel
<timestamp> [DEBUG] (redis) starting poll
<timestamp> [INFO] (cli) received redis from Watcher
<timestamp> [INFO] (cli) invoking Runner
<timestamp> [DEBUG] (service redis) querying Consul with &{...}
# ...

FAQ

Q: How is this different than confd?
A: The answer is simple: Service Discovery as a first class citizen. You are also encouraged to read this Pull Request on the project for more background information. We think confd is a great project, but Consul Template fills a missing gap.

Q: How is this different than Puppet/Chef/Ansible/Salt?
A: Configuration management tools are designed to be used in unison with Consul Template. Instead of rendering a stale configuration file, use your configuration management software to render a dynamic template that will be populated by Consul.

Contributing

To hack on Consul Template, you will need a modern Go environment. To compile the consul-template binary and run the test suite, simply execute:

$ make

This will compile the consul-template binary into bin/consul-template and run the test suite.

If you just want to run the tests:

$ make

Or to run a specific test in the suite:

go test ./... -run SomeTestFunction_name

Submit Pull Requests and Issues to the Consul Template project on GitHub.