kinesis

README

logspout-kinesis

A Logspout adapter for writing Docker container logs to Amazon Kinesis.

usage

After you created your custom Logspout build (see below), you can just run it as:

$ 	docker run --rm \
		-e LOGSPOUT=ignore \
		--name="logspout" \
		--volume=/var/run/docker.sock:/var/run/docker.sock \
		mycompany/logspout kinesis://

You will need the Amazon credentials in the form of environment variables, or whatever suits you. You will also need to set the environment variables KINESIS_STREAM_TEMPLATE and KINESIS_PARTITION_KEY_TEMPLATE, see below.

streams and partition key configuration

You can decide to route your logs to one or multiple Kinesis Streams (and their names) via the the KINESIS_STREAM_TEMPLATE environment variable. Golang's text/template package is used for templating, where the model available for templating is the Message struct.

For instance, you can decide to assign a Kinesis stream for each of you app (assuming you have a app label on your running containers:

$ export KINESIS_STREAM_TEMPLATE={{ index .Container.Config.Labels "app" }}

Or you can use environment variables, with our provided lookUp template function:

$ export KINESIS_STREAM_TEMPLATE={{ lookUp .Container.Config.Env "APP_ID" }}

This will search through .Container.Config.Env for APP_ID=* and return the value after the =.

IMPORTANT: if executing the stream name template results in an empty string, logspout-kinesis won't stream the log message. If debug logging is activated (see below), it will tell you so.

You can similarly decide the format of your partition key for each of your stream (here the app name and process type):

$ export KINESIS_PARTITION_KEY_TEMPLATE={{ index .Container.Config.Labels "app" }}.{{ index .Container.Config.Labels "process.type" }}

IMPORTANT: if the partition key end up being an empty string, logspout-kinesis will default to set it as a uuid. If debug logging is activated (see below), it will tell you so.

stream creation

By default, logspout-kinesis will create a stream if it is missing from Kinesis.

stream tagging

By default, logspout-kinesis will tag a stream it just created.

You can set the KINESIS_STREAM_TAG_KEY environment variable to set the tag key, and the KINESIS_STREAM_TAG_VALUE variable to set the template you want to use for the tag value.

A complete example would be:

KINESIS_STREAM_TAG_KEY="app"
KINESIS_STREAM_TAG_VALUE={{ lookUp .Container.Config.Env "EMPIRE_APPNAME" }}

logging

To activate logging, set the KINESIS_DEBUG environment variable to true.

build

logspout-kinesis is a custom logspout module. To use it, create an empty Dockerfile based on gliderlabs/logspout, and import this logspout-kinesis package into a new modules.go file. The gliderlabs/logspout base image will ONBUILD COPY and replace the original modules.go.

The following example creates a minimal logspout image capable of writing Dockers logs to Kinesis:

In modules.go:

package main

import (
	_ "github.com/gliderlabs/logspout/httpstream"
	_ "github.com/gliderlabs/logspout/routesapi"
	_ "github.com/remind101/logspout-kinesis"
)

In Dockerfile:

FROM gliderlabs/logspout:v3

Final step, build the image:

$ docker build -t mycompany/logspout .

More information on custom modules is available at the Logspout repo: Custom Logspout Modules

Documentation

Index

• Constants
• Variables
• func NewAdapter(route *router.Route) (router.LogAdapter, error)
• type Adapter
  • func (a *Adapter) Stream(logstream chan *router.Message)
• type Client
• type DroppedInputError
  • func (e *DroppedInputError) Error() string
• type Flusher
• type MissingEnvVarError
  • func (e *MissingEnvVarError) Error() string
• type Stream
  • func NewStream(name string, tags *map[string]*string, pKeyTmpl *template.Template) *Stream
  • func (s *Stream) Start()
  • func (s *Stream) Write(m *router.Message) error
• type StreamNotReadyError
  • func (e *StreamNotReadyError) Error() string

Constants

const (
	// PutRecordsLimit is the maximum number of records allowed for a PutRecords request.
	PutRecordsLimit int = 500

	// PutRecordsSizeLimit is the maximum allowed size per PutRecords request.
	PutRecordsSizeLimit int = 5 * 1024 * 1024 // 5MB

	// RecordSizeLimit is the maximum allowed size per record.
	RecordSizeLimit int = 1 * 1024 * 1024 // 1MB
)

Variables

var (
	// ErrorHandler handles the reporting of an error.
	ErrorHandler = logErr

	// ErrMissingTagKey is returned when the tag key environment variable doesn't match.
	ErrMissingTagKey = errors.New("the tag key is empty, check your template KINESIS_STREAM_TAG_KEY")

	// ErrMissingTagValue is returned when the tag value environment variable doesn't match.
	ErrMissingTagValue = errors.New("the tag value is empty, check your template KINESIS_STREAM_TAG_VALUE")
)

var ErrEmptyTmpl = errors.New("the template is empty")

ErrEmptyTmpl is returned when the template is empty.

var ErrRecordTooBig = errors.New("data byte size is over the limit")

ErrRecordTooBig is raised when a record is too big to be sent.

Functions

func NewAdapter

func NewAdapter(route *router.Route) (router.LogAdapter, error)

NewAdapter creates a kinesis adapter. Called during init.

Types

type Adapter

type Adapter struct {
	Streams    map[string]*Stream
	StreamTmpl *template.Template
	TagTmpl    *template.Template
	PKeyTmpl   *template.Template
}

Adapter represents the logspout adapter for Kinesis.

func (*Adapter) Stream

func (a *Adapter) Stream(logstream chan *router.Message)

Stream handles the routing of a message to Kinesis.

type Client

type Client interface {
	Create(*kinesis.CreateStreamInput) (bool, error)
	Status(*kinesis.DescribeStreamInput) string
	Tag(*kinesis.AddTagsToStreamInput) error
	PutRecords(inp *kinesis.PutRecordsInput) (*kinesis.PutRecordsOutput, error)
}

Client is a wrapper for the AWS Kinesis client.

type DroppedInputError

type DroppedInputError struct {
	Stream string
	Count  int
}

DroppedInputError is returned when an input is dropped.

func (*DroppedInputError) Error

func (e *DroppedInputError) Error() string

type Flusher

type Flusher interface {
	// contains filtered or unexported methods
}

Flusher flushes the inputs to Amazon Kinesis.

type MissingEnvVarError

type MissingEnvVarError struct {
	EnvVar string
}

MissingEnvVarError is return when an environment variable is missing.

func (*MissingEnvVarError) Error

func (e *MissingEnvVarError) Error() string

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

Stream represents a stream that will send messages to its writer.

func NewStream

func NewStream(name string, tags *map[string]*string, pKeyTmpl *template.Template) *Stream

NewStream instantiates a new stream.

func (*Stream) Start

func (s *Stream) Start()

Start runs the goroutines making calls to create and tag the stream on AWS.

func (*Stream) Write

func (s *Stream) Write(m *router.Message) error

Write sends the message to the writer if the stream is ready i.e created and tagged.

type StreamNotReadyError

type StreamNotReadyError struct {
	Stream string
}

StreamNotReadyError is returned while the stream is being created.

func (*StreamNotReadyError) Error

func (e *StreamNotReadyError) Error() string