logstash

README

logspout-rancher-ledger

A big thanks to @blippar for work on https://github.com/blippar/logspout-logstash-rancher. This plugin is based on their work, but with MAJOR changes to populating the Rancher info for the event data shipped to logstash

A minimalistic adapter for github.com/gliderlabs/logspout for containers running in Rancher 2.0 k8s cluster to write to Logstash.

In the local directory there is an example ELK stack and logspout compose files that can be used to deploy to rancher. You will need to tweak the logspout-sample.yaml file

Changes

I have created a new field index that is populated by the label io.rancher.stack.name. If it is empty or does not exist the index field is populated with container name label

if rancherInfo.Stack.StackName != "" {
        data["index"] = rancherInfo.Stack.StackName
    } else {
        data["index"] = rancherInfo.Container.Name
}

All the metadata that gets passed in the event data is now pulled from the rancher API.

// Uses the passed docker id to find the rancher Id
func GetRancherId(cID string) *client.Container {

    // This adds a filter to search for the specific container we just received an event from
    filters :=map[string]interface{}{"externalId": cID}

    listOpts := &client.ListOpts{Filters: filters}

    container, err := rancher.Container.List(listOpts)

    if err != nil {
        log.Print(err)
    }

    // There should only ever be 1 container in the list thanks to our filter
    return &container.Data[0]
}

The metadata also get cached in an in-memory map[string]*RancherInfo NOTE As of yet I do not know of a good way to remove the data from "cache" when a container gets deleted.

func Cache(con *RancherInfo) {
	cCache[con.Container.DockerID] = con
}

func ExistsInCache(containerID string) bool {
	for k, _ := range cCache {
		if k == containerID {
			return true
		}
	}

	return false
}

func GetFromCache(cID string) *RancherInfo {
	return cCache[cID]
}

• Removed Logstash Tags -- Due to pulling the k8s namespace which is in essence what was generally getting passed as tags.
• API admin account keys MUST to be passed as env variables to query the API for container. ()This could be avoided by passing the service account labels in rancher, but logspout will send logs outside the scope of the environment that the container is deployed in and cause errors)

// Setting package global Rancher API setting
var cattleUrl = os.Getenv("CATTLE_URL")
var cattleAccessKey = os.Getenv("CATTLE_ACCESS_KEY")
var cattleSecretKey = os.Getenv("CATTLE_SECRET_KEY")

Building

Change to custom directory and docker built -t logspout-rancher-ledger .

Logstash config

The below example is also in the local/logstash_conf dir for use with the local docker-compose.yml file which you will need to edit the volume mount path for.

input {
  tcp {
    port => 5000
  }
}

filter {
  json {
    source => "message"
  }
}

output{
  elasticsearch {
    hosts => ["elasticsearch"]
    index => "%{index}-%{+YYYY.MM.dd}"
  }

}

Available configuration options

You can also add arbitrary logstash fields to the event using the LOGSTASH_FIELDS container environment variable:

  # Add any number of arbitrary fields to your event
  -e LOGSTASH_FIELDS="myfield=something,anotherfield=something_else"

The output into logstash should be like:

    "myfield": "something",
    "another_field": "something_else",

Both configuration options can be set for every individual container, or for the logspout-logstash container itself where they then become a default for all containers if not overridden there.

This table shows all available configurations:

Environment Variable	Input Type	Default Value
LOGSTASH_TAGS	array	None
LOGSTASH_FIELDS	map	None

Documentation

Index

• func Cache(con *RancherInfo)
• func DeleteFromCache(cId string) bool
• func ExistsInCache(containerID string) bool
• func GetLogstashFields(c *docker.Container, a *LogstashAdapter) map[string]string
• func GetRancherId(cID string) *client.Container
• func NewLogstashAdapter(route *router.Route) (router.LogAdapter, error)
• type DockerInfo
• type LogstashAdapter
  • func (a *LogstashAdapter) Stream(logstream chan *router.Message)
• type RancherContainer
• type RancherInfo
  • func GetFromCache(cID string) *RancherInfo
  • func GetRancherInfo(c *docker.Container) *RancherInfo
• type RancherStack

Functions

func Cache

func Cache(con *RancherInfo)

Add the RancherInfo to the cache

func DeleteFromCache

func DeleteFromCache(cId string) bool

func ExistsInCache

func ExistsInCache(containerID string) bool

Check if the container data already exists in the cached map

func GetLogstashFields

func GetLogstashFields(c *docker.Container, a *LogstashAdapter) map[string]string

Parse the logstash fields env variables

func GetRancherId

func GetRancherId(cID string) *client.Container

Uses the passed docker id to find the rancher Id

func NewLogstashAdapter

func NewLogstashAdapter(route *router.Route) (router.LogAdapter, error)

NewLogstashAdapter creates a LogstashAdapter with UDP as the default transport.

Types

type DockerInfo

type DockerInfo struct {
	Name     string `json:"name"`
	ID       string `json:"id"`
	Image    string `json:"image"`
	Hostname string `json:"hostname"`
}

Container Docker info for event data

type LogstashAdapter

type LogstashAdapter struct {
	// contains filtered or unexported fields
}

LogstashAdapter is an adapter that streams UDP JSON to Logstash.

func (*LogstashAdapter) Stream

func (a *LogstashAdapter) Stream(logstream chan *router.Message)

Stream implements the router.LogAdapter interface.

type RancherContainer

type RancherContainer struct {
	Name     string `json:"name"`           // io.rancher.container.name
	IP       string `json:"ip,omitempty"`   // io.rancher.container.ip
	ID       string `json:"once,omitempty"` // io.rancher.container.start_once
	HostID   string `json:"hostId,omitempty"`
	DockerID string `json:"dockerId,omitempty"`
}

Rancher container data for event

type RancherInfo

type RancherInfo struct {
	Environment string            `json:"environment,omitempty"`
	Container   *RancherContainer `json:"container,omitempty"`
	Stack       *RancherStack     `json:"stack,omitempty"`
}

Rancher data for evetn data

func GetFromCache

func GetFromCache(cID string) *RancherInfo

Get the container data from the map

func GetRancherInfo

func GetRancherInfo(c *docker.Container) *RancherInfo

Get the rancher meteadata from the api/cahce

type RancherStack

type RancherStack struct {
	Service    string `json:"service,omitempty"`   // Service Name from API
	StackName  string `json:"stackName,omitempty"` // io.rancher.stack.name
	StackState string `json:"stackState,omitempty"`
}

Rancher stack inf for event