logstash

README

I've restarted from the existing looplab/logspout-logstash and incorporated some existing forks' changes as well as my own. I've made this repo canonical to my own needs within Docker/Rancher for an ELK stack.

logspout-logstash

A minimalistic adapter for Logspout to write to Logstash. It also export some information about the Rancher environment it is running in if Rancher labels are available on the container as well as export all other Docker labels.

All labels not prefixed by io.rancher will be included with the . separator replaced by a _ to prevent error in Elastic.

Started from the instructions here on how to build your own Logspout container with custom modules.

Docker Logspout-Logstash

Docker image for dm/logspout-logstash, available in docker hub dmacedo/logspout-logstash:latest

Docker Tags

dmacedo/logspout-logstash tagged images:

• latest: Latest stable release
• v1.0.0: First release v1.0.0

Usage

Build docker image:

cd docker && make build

To run the image, set LOGSTASH_HOST env (defaults to logstash.local):

cd docker && LOGSTASH_HOST=logstash.local make run

Additional documentation

Use by setting a docker environment variable ROUTE_URIS=logstash://host:port to the Logstash server.

The default protocol is UDP, but it is possible to change to TCP by adding +tcp after the logstash protocol when starting your container.

In your logstash config, set the input codec to json e.g:

input {
  udp {
    port  => 5000
    codec => json
  }
  tcp {
    port  => 5000
    codec => json
  }
}

Available configuration options

For example, to get into the Logstash event's @tags field, use the LOGSTASH_TAGS container environment variable. Multiple tags can be passed by using comma-separated values.

  # Add any number of arbitrary tags to your event
  -e LOGSTASH_TAGS="docker,production"

The output into logstash should be like:

    "tags": [
      "docker",
      "production"
    ],

You can also add arbitrary logstash fields to the event using the LOGSTASH_FIELDS container environment variable:

  # Add any number of arbitrary fields to your event
  -e LOGSTASH_FIELDS="myfield=something,anotherfield=something_else"

The output into logstash should be like:

    "myfield": "something",
    "another_field": "something_else",

Both configuration options can be set for every individual container, or for the logspout-logstash container itself where they then become a default for all containers if not overridden there.

By setting the environment variable DOCKER_LABELS to a non-empty value, logspout-logstash will add all docker container labels as fields:

"docker": {
    "hostname": "866e2ca94f5f",
    "id": "866e2ca94f5fe11d57add5a78232c53dfb6187f04f6e150ec15f0ae1e1737731",
    "image": "centos:7",
    "labels": {
        "a_label": "yes",
        "build-date": "20161214",
        "license": "GPLv2",
        "name": "CentOS Base Image",
        "pleasework": "okay",
        "some_label_with_dots": "more.dots",
        "vendor": "CentOS"
    },
    "name": "/ecstatic_murdock"
}

To be compatible with Elasticsearch, dots in labels will be replaced with underscores.

Retrying

Two environment variables control the behavior of Logspout when the Logstash target isn't available:

RETRY_STARTUP causes Logspout to retry forever if Logstash isn't available at startup, and RETRY_SEND will retry sending log lines when Logstash becomes unavailable while Logspout is running.

Note that RETRY_SEND will work only if UDP is used for the log transport and the destination doesn't change; in any other case RETRY_SEND should be disabled, restart and reconnect instead and let RETRY_STARTUP deal with the situation.

With both retry options, log lines will be lost when Logstash isn't available. Set the environment variables to any nonempty value to enable retrying. The default is disabled.

This table shows all available configurations:

Environment Variable	Input Type	Default Value
LOGSTASH_TAGS	array	None
LOGSTASH_FIELDS	map	None
DOCKER_LABELS	any	""
RETRY_STARTUP	any	""
RETRY_SEND	any	""

Documentation

Index

• func GetContainerTags(c *docker.Container, a *LogstashAdapter) []string
• func GetLogstashFields(c *docker.Container, a *LogstashAdapter) map[string]string
• func NewLogstashAdapter(route *router.Route) (router.LogAdapter, error)
• type DockerInfo
• type LogstashAdapter
  • func (a *LogstashAdapter) Stream(logstream chan *router.Message)
• type RancherContainer
• type RancherInfo
  • func GetRancherInfo(c *docker.Container) *RancherInfo
• type RancherStack

Functions

func GetContainerTags

func GetContainerTags(c *docker.Container, a *LogstashAdapter) []string

Get container tags configured with the environment variable LOGSTASH_TAGS

func GetLogstashFields

func GetLogstashFields(c *docker.Container, a *LogstashAdapter) map[string]string

func NewLogstashAdapter

func NewLogstashAdapter(route *router.Route) (router.LogAdapter, error)

NewLogstashAdapter creates a LogstashAdapter with UDP as the default transport.

Types

type DockerInfo

type DockerInfo struct {
	Name     string            `json:"name"`
	ID       string            `json:"id"`
	Image    string            `json:"image"`
	Hostname string            `json:"hostname"`
	Labels   map[string]string `json:"labels"`
}

type LogstashAdapter

type LogstashAdapter struct {
	// contains filtered or unexported fields
}

LogstashAdapter is an adapter that streams UDP JSON to Logstash.

func (*LogstashAdapter) Stream

func (a *LogstashAdapter) Stream(logstream chan *router.Message)

Stream implements the router.LogAdapter interface.

type RancherContainer

type RancherContainer struct {
	Name      string `json:"name"`           // io.rancher.container.name
	UUID      string `json:"uuid"`           // io.rancher.container.uuid
	IP        string `json:"ip,omitempty"`   // io.rancher.container.ip
	StartOnce string `json:"once,omitempty"` // io.rancher.container.start_once
}

type RancherInfo

type RancherInfo struct {
	Environment string           `json:"environment,omitempty"`
	Container   RancherContainer `json:"container"`
	Stack       RancherStack     `json:"stack"`
}

func GetRancherInfo

func GetRancherInfo(c *docker.Container) *RancherInfo

type RancherStack

type RancherStack struct {
	Service    string `json:"service"`              // io.rancher.stack_service.name
	Name       string `json:"name"`                 // io.rancher.stack.name
	Full       string `json:"full"`                 // io.rancher.stack_service.name
	Global     string `json:"global,omitempty"`     // io.rancher.scheduler.global
	Deployment string `json:"deployment,omitempty"` // io.rancher.service.deployment.unit
}