dockergen

README

docker-gen

Host Install

Linux/OSX binaries for release 0.7.5

Separate Container Install

It can also be run as two separate containers using the jeremiec82/my-nginx-gen image, together with virtually any other image.

This is how you could run the official nginx image and have docker-gen generate a reverse proxy config in the same way that nginx-proxy works. You may want to do this to prevent having the docker socket bound to a publicly exposed container service.

More information on my-nginx-gen repository.

Original readme from jwilder

docker-gen is a file generator that renders templates using docker container meta-data.

It can be used to generate various kinds of files for:

• Centralized logging - fluentd, logstash or other centralized logging tools that tail the containers JSON log file or files within the container.
• Log Rotation - logrotate files to rotate container JSON log files
• Reverse Proxy Configs - nginx, haproxy, etc. reverse proxy configs to route requests from the host to containers
• Service Discovery - Scripts (python, bash, etc..) to register containers within etcd, hipache, etc..

===

Installation

There are three common ways to run docker-gen:

• on the host
• bundled in a container with another application
• separate standalone containers

Host Install

Download the version you need, untar, and install to your PATH.

$ wget https://github.com/jwilder/docker-gen/releases/download/0.7.4/docker-gen-linux-amd64-0.7.4.tar.gz
$ tar xvzf docker-gen-linux-amd64-0.7.4.tar.gz
$ ./docker-gen

Bundled Container Install

Docker-gen can be bundled inside of a container along-side applications.

jwilder/nginx-proxy trusted build is an example of running docker-gen within a container along-side nginx. jwilder/docker-register is an example of running docker-gen within a container to do service registration with etcd.

===

Usage

$ docker-gen
Usage: docker-gen [options] template [dest]

Generate files from docker container meta-data

Options:
  -config value
      config files with template directives. Config files will be merged if this option is specified multiple times. (default [])
  -endpoint string
      docker api endpoint (tcp|unix://..). Default unix:///var/run/docker.sock
  -interval int
      notify command interval (secs)
  -keep-blank-lines
      keep blank lines in the output file
  -notify restart xyz
      run command after template is regenerated (e.g restart xyz)
  -notify-output
      log the output(stdout/stderr) of notify command
  -notify-sighup container-ID
      send HUP signal to container.  Equivalent to 'docker kill -s HUP container-ID'
  -only-exposed
      only include containers with exposed ports
  -only-published
      only include containers with published ports (implies -only-exposed)
  -include-stopped
      include stopped containers
  -tlscacert string
      path to TLS CA certificate file (default "/Users/jason/.docker/machine/machines/default/ca.pem")
  -tlscert string
      path to TLS client certificate file (default "/Users/jason/.docker/machine/machines/default/cert.pem")
  -tlskey string
      path to TLS client key file (default "/Users/jason/.docker/machine/machines/default/key.pem")
  -tlsverify
      verify docker daemon's TLS certicate (default true)
  -version
      show version
  -watch
      watch for container changes
  -wait
      minimum (and/or maximum) duration to wait after each container change before triggering

Arguments:
  template - path to a template to generate
  dest - path to a write the template. If not specfied, STDOUT is used

Environment Variables:
  DOCKER_HOST - default value for -endpoint
  DOCKER_CERT_PATH - directory path containing key.pem, cert.pm and ca.pem
  DOCKER_TLS_VERIFY - enable client TLS verification]

If no <dest> file is specified, the output is sent to stdout. Mainly useful for debugging.

Configuration file

Using the -config flag from above you can tell docker-gen to use the specified config file instead of command-line options. Multiple templates can be defined and they will be executed in the order that they appear in the config file.

An example configuration file, docker-gen.cfg can be found in the examples folder.

Configuration File Syntax

[[config]]
Starts a configuration section

dest = "path/to/a/file"
path to a write the template. If not specfied, STDOUT is used

notifycmd = "/etc/init.d/foo reload"
run command after template is regenerated (e.g restart xyz)

onlyexposed = true
only include containers with exposed ports

template = "/path/to/a/template/file.tmpl"
path to a template to generate

watch = true
watch for container changes

wait = "500ms:2s"
debounce changes with a min:max duration. Only applicable if watch = true


[config.NotifyContainers]
Starts a notify container section

containername = 1
container name followed by the signal to send

container_id = 1
or the container id can be used followed by the signal to send

Putting it all together here is an example configuration file.

[[config]]
template = "/etc/nginx/nginx.conf.tmpl"
dest = "/etc/nginx/sites-available/default"
onlyexposed = true
notifycmd = "/etc/init.d/nginx reload"

[[config]]
template = "/etc/logrotate.conf.tmpl"
dest = "/etc/logrotate.d/docker"
watch = true

[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"

[config.NotifyContainers]
nginx = 1  # 1 is a signal number to be sent; here SIGHUP
e75a60548dc9 = 1  # a key can be either container name (nginx) or ID

===

Templating

The templates used by docker-gen are written using the Go text/template language. In addition to the built-in functions supplied by Go, docker-gen provides a number of additional functions to make it simpler (or possible) to generate your desired output.

Emit Structure

Within the templates, the object emitted by docker-gen will be a structure consisting of following Go structs:

type RuntimeContainer struct {
    ID           string
    Addresses    []Address
    Networks     []Network
    Gateway      string
    Name         string
    Hostname     string
    Image        DockerImage
    Env          map[string]string
    Volumes      map[string]Volume
    Node         SwarmNode
    Labels       map[string]string
    IP           string
    IP6LinkLocal string
    IP6Global    string
    Mounts       []Mount
    State        State
}

type Address struct {
    IP           string
    IP6LinkLocal string
    IP6Global    string
    Port         string
    HostPort     string
    Proto        string
    HostIP       string
}

type Network struct {
    IP                  string
    Name                string
    Gateway             string
    EndpointID          string
    IPv6Gateway         string
    GlobalIPv6Address   string
    MacAddress          string
    GlobalIPv6PrefixLen int
    IPPrefixLen         int
}

type DockerImage struct {
    Registry   string
    Repository string
    Tag        string
}

type Mount struct {
  Name        string
  Source      string
  Destination string
  Driver      string
  Mode        string
  RW          bool
}

type Volume struct {
    Path      string
    HostPath  string
    ReadWrite bool
}

type SwarmNode struct {
    ID      string
    Name    string
    Address Address
}

type State struct {
  Running bool
}

// Accessible from the root in templates as .Docker
type Docker struct {
    Name                 string
    NumContainers        int
    NumImages            int
    Version              string
    ApiVersion           string
    GoVersion            string
    OperatingSystem      string
    Architecture         string
    CurrentContainerID   string
}

// Host environment variables accessible from root in templates as .Env

For example, this is a JSON version of an emitted RuntimeContainer struct:

{
   "ID":"71e9768075836eb38557adcfc71a207386a0c597dbeda240cf905df79b18cebf",
   "Addresses":[
      {
         "IP":"172.17.0.4",
         "Port":"22",
         "Proto":"tcp",
         "HostIP":"192.168.10.24",
         "HostPort":"2222"
      }
   ],
   "Gateway":"172.17.42.1",
   "Node": {
       "ID":"I2VY:P7PF:TZD5:PGWB:QTI7:QDSP:C5UD:DYKR:XKKK:TRG2:M2BL:DFUN",
       "Name":"docker-test",
       "Address": {
           "IP":"192.168.10.24"
       }
   },
   "Labels": {
       "operatingsystem":"Ubuntu 14.04.2 LTS",
       "storagedriver":"devicemapper",
       "anything_foo":"something_bar"
   },
   "IP":"172.17.0.4",
   "Name":"docker_register",
   "Hostname":"71e976807583",
   "Image":{
      "Registry":"jwilder",
      "Repository":"docker-register"
   },
   "Env":{
      "ETCD_HOST":"172.17.42.1:4001",
      "PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "DOCKER_HOST":"unix:///var/run/docker.sock",
      "HOST_IP":"172.17.42.1"
   },
   "Volumes":{
      "/mnt":{
         "Path":"/mnt",
         "HostPath":"/Users/joebob/tmp",
         "ReadWrite":true
      }
   }
}

Functions

• closest $array $value: Returns the longest matching substring in $array that matches $value
• coalesce ...: Returns the first non-nil argument.
• contains $map $key: Returns true if $map contains $key. Takes maps from string to string.
• dict $key $value ...: Creates a map from a list of pairs. Each $key value must be a string, but the $value can be any type (or nil). Useful for passing more than one value as a pipeline context to subtemplates.
• dir $path: Returns an array of filenames in the specified $path.
• exists $path: Returns true if $path refers to an existing file or directory. Takes a string.
• first $array: Returns the first value of an array or nil if the arry is nil or empty.
• groupBy $containers $fieldPath: Groups an array of RuntimeContainer instances based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value, which must be a string. Returns a map from the value of the field path expression to an array of containers having that value. Containers that do not have a value for the field path in question are omitted.
• groupByKeys $containers $fieldPath: Returns the same as groupBy but only returns the keys of the map.
• groupByMulti $containers $fieldPath $sep: Like groupBy, but the string value specified by $fieldPath is first split by $sep into a list of strings. A container whose $fieldPath value contains a list of strings will show up in the map output under each of those strings.
• groupByLabel $containers $label: Returns the same as groupBy but grouping by the given label's value.
• hasPrefix $prefix $string: Returns whether $prefix is a prefix of $string.
• hasSuffix $suffix $string: Returns whether $suffix is a suffix of $string.
• intersect $slice1 $slice2: Returns the strings that exist in both string slices.
• json $value: Returns the JSON representation of $value as a string.
• keys $map: Returns the keys from $map. If $map is nil, a nil is returned. If $map is not a map, an error will be thrown.
• last $array: Returns the last value of an array.
• parseBool $string: parseBool returns the boolean value represented by the string. It accepts 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False. Any other value returns an error. Alias for strconv.ParseBool
• replace $string $old $new $count: Replaces up to $count occurences of $old with $new in $string. Alias for strings.Replace
• sha1 $string: Returns the hexadecimal representation of the SHA1 hash of $string.
• split $string $sep: Splits $string into a slice of substrings delimited by $sep. Alias for strings.Split
• splitN $string $sep $count: Splits $string into a slice of substrings delimited by $sep, with number of substrings returned determined by $count. Alias for strings.SplitN
• trimPrefix $prefix $string: If $prefix is a prefix of $string, return $string with $prefix trimmed from the beginning. Otherwise, return $string unchanged.
• trimSuffix $suffix $string: If $suffix is a suffix of $string, return $string with $suffix trimmed from the end. Otherwise, return $string unchanged.
• trim $string: Removes whitespace from both sides of $string.
• when $condition $trueValue $falseValue: Returns the $trueValue when the $condition is true and the $falseValue otherwise
• where $items $fieldPath $value: Filters an array or slice based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value. Returns an array of items having that value.
• whereNot $items $fieldPath $value: Filters an array or slice based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value. Returns an array of items not having that value.
• whereExist $items $fieldPath: Like where, but returns only items where $fieldPath exists (is not nil).
• whereNotExist $items $fieldPath: Like where, but returns only items where $fieldPath does not exist (is nil).
• whereAny $items $fieldPath $sep $values: Like where, but the string value specified by $fieldPath is first split by $sep into a list of strings. The comparison value is a string slice with possible matches. Returns items which OR intersect these values.
• whereAll $items $fieldPath $sep $values: Like whereAny, except all $values must exist in the $fieldPath.
• whereLabelExists $containers $label: Filters a slice of containers based on the existence of the label $label.
• whereLabelDoesNotExist $containers $label: Filters a slice of containers based on the non-existence of the label $label.
• whereLabelValueMatches $containers $label $pattern: Filters a slice of containers based on the existence of the label $label with values matching the regular expression $pattern.

===

Examples

• Automated Nginx Reverse Proxy for Docker
• Docker Log Management With Fluentd
• Docker Service Discovery Using Etcd and Haproxy

NGINX Reverse Proxy Config

jwilder/nginx-proxy trusted build.

Start nginx-proxy:

$ docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock -t jwilder/nginx-proxy

Then start containers with a VIRTUAL_HOST env variable:

$ docker run -e VIRTUAL_HOST=foo.bar.com -t ...

If you wanted to run docker-gen directly on the host, you could do it with:

$ docker-gen -only-published -watch -notify "/etc/init.d/nginx reload" templates/nginx.tmpl /etc/nginx/sites-enabled/default

Fluentd Log Management

This template generate a fluentd.conf file used by fluentd. It would then ship log files off the host.

$ docker-gen -watch -notify "restart fluentd" templates/fluentd.tmpl /etc/fluent/fluent.conf

Service Discovery in Etcd

This template is an example of generating a script that is then executed. This template generates a python script that is then executed which register containers in Etcd using its HTTP API.

$ docker-gen -notify "/bin/bash /tmp/etcd.sh" -interval 10 templates/etcd.tmpl /tmp/etcd.sh

Development

This project uses glock for managing 3rd party dependencies. You'll need to install glock into your workspace before hacking on docker-gen.

$ git clone <your fork>
$ cd <your fork>
$ make get-deps
$ make

TODO

• Add event status for handling start and stop events differently

License

MIT

Documentation

Index

• func GenerateFile(config Config, containers Context) bool
• func GetCurrentContainerID() string
• func GetEndpoint(endpoint string) (string, error)
• func NewDockerClient(endpoint string, tlsVerify bool, tlsCert, tlsCaCert, tlsKey string) (*docker.Client, error)
• func NewGenerator(gc GeneratorConfig) (*generator, error)
• func SetDockerEnv(d *docker.Env)
• func SetServerInfo(d *docker.DockerInfo)
• type Address
• type Config
• type ConfigFile
  • func (c *ConfigFile) FilterWatches() ConfigFile
• type Context
  • func (c *Context) Docker() Docker
  • func (c *Context) Env() map[string]string
• type Docker
• type DockerContainer
• type DockerImage
  • func (i *DockerImage) String() string
• type GeneratorConfig
• type Mount
• type Network
• type RuntimeContainer
  • func (r *RuntimeContainer) Equals(o RuntimeContainer) bool
  • func (r *RuntimeContainer) PublishedAddresses() []Address
• type State
• type SwarmNode
• type Volume
• type Wait
  • func ParseWait(s string) (*Wait, error)
  • func (w *Wait) UnmarshalText(text []byte) error

Functions

func GenerateFile

func GenerateFile(config Config, containers Context) bool

func GetCurrentContainerID

func GetCurrentContainerID() string

func GetEndpoint

func GetEndpoint(endpoint string) (string, error)

func NewDockerClient

func NewDockerClient(endpoint string, tlsVerify bool, tlsCert, tlsCaCert, tlsKey string) (*docker.Client, error)

func NewGenerator

func NewGenerator(gc GeneratorConfig) (*generator, error)

func SetDockerEnv

func SetDockerEnv(d *docker.Env)

func SetServerInfo

func SetServerInfo(d *docker.DockerInfo)

Types

type Address

type Address struct {
	IP           string
	IP6LinkLocal string
	IP6Global    string
	Port         string
	HostPort     string
	Proto        string
	HostIP       string
}

type Config

type Config struct {
	Template         string
	Dest             string
	Watch            bool
	Wait             *Wait
	NotifyCmd        string
	NotifyOutput     bool
	NotifyContainers map[string]docker.Signal
	OnlyExposed      bool
	OnlyPublished    bool
	IncludeStopped   bool
	Interval         int
	KeepBlankLines   bool
}

type ConfigFile

type ConfigFile struct {
	Config []Config
}

func (*ConfigFile) FilterWatches

func (c *ConfigFile) FilterWatches() ConfigFile

type Context

type Context []*RuntimeContainer

func (*Context) Docker

func (c *Context) Docker() Docker

func (*Context) Env

func (c *Context) Env() map[string]string

type Docker

type Docker struct {
	Name               string
	NumContainers      int
	NumImages          int
	Version            string
	ApiVersion         string
	GoVersion          string
	OperatingSystem    string
	Architecture       string
	CurrentContainerID string
}

type DockerContainer

type DockerContainer struct {
}

type DockerImage

type DockerImage struct {
	Registry   string
	Repository string
	Tag        string
}

func (*DockerImage) String

func (i *DockerImage) String() string

type GeneratorConfig

type GeneratorConfig struct {
	Endpoint string

	TLSCert   string
	TLSKey    string
	TLSCACert string
	TLSVerify bool
	All       bool

	ConfigFile ConfigFile
}

type Mount

type Mount struct {
	Name        string
	Source      string
	Destination string
	Driver      string
	Mode        string
	RW          bool
}

type Network

type Network struct {
	IP                  string
	Name                string
	Gateway             string
	EndpointID          string
	IPv6Gateway         string
	GlobalIPv6Address   string
	MacAddress          string
	GlobalIPv6PrefixLen int
	IPPrefixLen         int
}

type RuntimeContainer

type RuntimeContainer struct {
	ID           string
	Addresses    []Address
	Networks     []Network
	Gateway      string
	Name         string
	Hostname     string
	Image        DockerImage
	Env          map[string]string
	Volumes      map[string]Volume
	Node         SwarmNode
	Labels       map[string]string
	IP           string
	IP6LinkLocal string
	IP6Global    string
	Mounts       []Mount
	State        State
}

func (*RuntimeContainer) Equals

func (r *RuntimeContainer) Equals(o RuntimeContainer) bool

func (*RuntimeContainer) PublishedAddresses

func (r *RuntimeContainer) PublishedAddresses() []Address

type State

type State struct {
	Running bool
}

type SwarmNode

type SwarmNode struct {
	ID      string
	Name    string
	Address Address
}

type Volume

type Volume struct {
	Path      string
	HostPath  string
	ReadWrite bool
}

type Wait

type Wait struct {
	Min time.Duration
	Max time.Duration
}

func ParseWait

func ParseWait(s string) (*Wait, error)

func (*Wait) UnmarshalText

func (w *Wait) UnmarshalText(text []byte) error