logrustash

README

Logstash hook for logrus

Use this hook to send the logs to Logstash over both UDP and TCP.

Important notes

This a fork from github.com/bshuster-repo/logrus-logstash-hook repo.

ripcurld0 going to rewrite original hook but there is no estimates when it will be ready for using in production. And more important is that he declines all pull requests with new features. So the main goal of this fork is to add some new features and use logstash hook until ripcurld0 finish his work.

Added features:

• Async mode. You can send log messages without blocking logic.
• Reconnect.

Usage

package main

import (
    "github.com/sirupsen/logrus"
    "github.com/ylamothe/logrustash"
)

func main() {
        log := logrus.New()
        hook, err := logrustash.NewHook("tcp", "172.17.0.2:9999", "myappName")

        if err != nil {
                log.Fatal(err)
        }
        log.Hooks.Add(hook)
        ctx := log.WithFields(logrus.Fields{
          "method": "main",
        })
        ...
        ctx.Info("Hello World!")
}

This is how it will look like:

{
    "@timestamp" => "2016-02-29T16:57:23.000Z",
      "@version" => "1",
         "level" => "info",
       "message" => "Hello World!",
        "method" => "main",
          "host" => "172.17.0.1",
          "port" => 45199,
          "type" => "myappName"
}

Async mode

Create hook with NewAsync... factory methods if you want to send logs in async mode.

Example:

log := logrus.New()
hook, err := logrustash.NewAsyncHook("tcp", "172.17.0.2:9999", "myappName")
if err != nil {
        log.Fatal(err)
}
log.Hooks.Add(hook)

In the very rare cases buffer can be clogged. By default all new messages will be dropped until buffer frees.

If you don't want to lose messages you can change this behaviour:

log := logrus.New()
hook, err := logrustash.NewAsyncHook("tcp", "172.17.0.2:9999", "myappName")
if err != nil {
        log.Fatal(err)
}

hook.WaitUntilBufferFrees = true
log.Hooks.Add(hook)

Reconnect

Doesn't work if you create hook with your own connection. Don't use this factory methods if you want to have auto reconnect:

• NewHookWithFieldsAndConn
• NewAsyncHookWithFieldsAndConn
• NewHookWithFieldsAndConnAndPrefix
• NewAsyncHookWithFieldsAndConnAndPrefix

When occurs not temporary net error hook will automatically try to create new connection to logstash.

With each new consecutive attempt to reconnect, delay before next reconnect will grow up by formula:

ReconnectBaseDelay * ReconnectDelayMultiplier^reconnectRetries

Be careful using reconnects without async mode because delay can increase significantly and this will blocks your logic.

Example for async mode:

hook, err := logrustash.NewAsyncHook("tcp", "172.17.0.2:9999", "myappName")
if err != nil {
        log.Fatal(err)
}

hook.ReconnectBaseDelay = time.Second // Wait for one second before first reconnect.
hook.ReconnectDelayMultiplier = 2
hook.MaxReconnectRetries = 10

log.Hooks.Add(hook)

With this configuration hook will wait 1024 (2^10) seconds before last reconnect. When message buffer will full all new messages will be dropped (depends on WaitUntilBufferFrees parameter).

Example for sync mode:

hook, err := logrustash.NewHook("tcp", "172.17.0.2:9999", "myappName")
if err != nil {
        log.Fatal(err)
}

hook.ReconnectBaseDelay = time.Second // Wait for one second before first reconnect.
hook.ReconnectDelayMultiplier = 1
hook.MaxReconnectRetries = 3

log.Hooks.Add(hook)

WIth this configuration we will have constant reconnect delay in 1 second.

Hook Fields

Fields can be added to the hook, which will always be in the log context. This can be done when creating the hook:

hook, err := logrustash.NewHookWithFields("tcp", "172.17.0.2:9999", "myappName", logrus.Fields{
        "hostname":    os.Hostname(),
        "serviceName": "myServiceName",
})

Or afterwards:

hook.WithFields(logrus.Fields{
        "hostname":    os.Hostname(),
        "serviceName": "myServiceName",
})

This allows you to set up the hook so logging is available immediately, and add important fields as they become available.

Single fields can be added/updated using 'WithField':

hook.WithField("status", "running")

Field prefix

The hook allows you to send logging to logstash and also retain the default std output in text format. However to keep this console output readable some fields might need to be omitted from the default non-hooked log output. Each hook can be configured with a prefix used to identify fields which are only to be logged to the logstash connection. For example if you don't want to see the hostname and serviceName on each log line in the console output you can add a prefix:

hook, err := logrustash.NewHookWithFields("tcp", "172.17.0.2:9999", "myappName", logrus.Fields{
        "_hostname":    os.Hostname(),
        "_serviceName": "myServiceName",
})
...
hook.WithPrefix("_")

There are also constructors available which allow you to specify the prefix from the start. The std-out will not have the '_hostname' and '_servicename' fields, and the logstash output will, but the prefix will be dropped from the name.

TODO

• Add more tests.

Authors

Name	Github	Twitter
Boaz Shuster	ripcurld0	@ripcurld0
Alexander Borisov	cheshir	cheshirysh

License

MIT.

Documentation

Index

• type Hook
  • func NewAsyncFilterHook() *Hook
  • func NewAsyncFilterHookWithPrefix(prefix string) *Hook
  • func NewAsyncHook(protocol, address, appName string) (*Hook, error)
  • func NewAsyncHookWithConn(conn net.Conn, appName string) (*Hook, error)
  • func NewAsyncHookWithFields(protocol, address, appName string, alwaysSentFields logrus.Fields) (*Hook, error)
  • func NewAsyncHookWithFieldsAndConn(conn net.Conn, appName string, alwaysSentFields logrus.Fields) (*Hook, error)
  • func NewAsyncHookWithFieldsAndConnAndPrefix(conn net.Conn, appName string, alwaysSentFields logrus.Fields, prefix string) (*Hook, error)
  • func NewAsyncHookWithFieldsAndPrefix(protocol, address, appName string, alwaysSentFields logrus.Fields, ...) (*Hook, error)
  • func NewFilterHook() *Hook
  • func NewFilterHookWithPrefix(prefix string) *Hook
  • func NewHook(protocol, address, appName string) (*Hook, error)
  • func NewHookWithConn(conn net.Conn, appName string) (*Hook, error)
  • func NewHookWithFields(protocol, address, appName string, alwaysSentFields logrus.Fields) (*Hook, error)
  • func NewHookWithFieldsAndConn(conn net.Conn, appName string, alwaysSentFields logrus.Fields) (*Hook, error)
  • func NewHookWithFieldsAndConnAndPrefix(conn net.Conn, appName string, alwaysSentFields logrus.Fields, prefix string) (*Hook, error)
  • func NewHookWithFieldsAndPrefix(protocol, address, appName string, alwaysSentFields logrus.Fields, ...) (*Hook, error)
  • func (h *Hook) Fire(entry *logrus.Entry) error
  • func (h *Hook) Levels() []logrus.Level
  • func (h *Hook) WithField(key string, value interface{})
  • func (h *Hook) WithFields(fields logrus.Fields)
  • func (h *Hook) WithPrefix(prefix string)
• type LogstashFormatter
  • func (f *LogstashFormatter) Format(entry *logrus.Entry) ([]byte, error)
  • func (f *LogstashFormatter) FormatWithPrefix(entry *logrus.Entry, prefix string) ([]byte, error)

Types

type Hook

type Hook struct {
	sync.RWMutex

	TimeFormat string

	WaitUntilBufferFrees     bool
	Timeout                  time.Duration // Timeout for sending message.
	MaxSendRetries           int           // Declares how many times we will try to resend message.
	ReconnectBaseDelay       time.Duration // First reconnect delay.
	ReconnectDelayMultiplier float64       // Base multiplier for delay before reconnect.
	MaxReconnectRetries      int           // Declares how many times we will try to reconnect.
	// contains filtered or unexported fields
}

Hook represents a connection to a Logstash instance

func NewAsyncFilterHook

func NewAsyncFilterHook() *Hook

NewAsyncFilterHook makes a new hook which does not forward to logstash, but simply enforces the prefix rules. Logs will be sent asynchronously.

func NewAsyncFilterHookWithPrefix

func NewAsyncFilterHookWithPrefix(prefix string) *Hook

NewAsyncFilterHookWithPrefix make a new hook which does not forward to logstash, but simply enforces the specified prefix. Logs will be sent asynchronously.

func NewAsyncHook

func NewAsyncHook(protocol, address, appName string) (*Hook, error)

NewAsyncHook creates a new hook to a Logstash instance, which listens on `protocol`://`address`. Logs will be sent asynchronously.

func NewAsyncHookWithConn

func NewAsyncHookWithConn(conn net.Conn, appName string) (*Hook, error)

NewAsyncHookWithConn creates a new hook to a Logstash instance, using the supplied connection. Logs will be sent asynchronously.

func NewAsyncHookWithFields

func NewAsyncHookWithFields(protocol, address, appName string, alwaysSentFields logrus.Fields) (*Hook, error)

NewAsyncHookWithFields creates a new hook to a Logstash instance, which listens on `protocol`://`address`. alwaysSentFields will be sent with every log entry. Logs will be sent asynchronously.

func NewAsyncHookWithFieldsAndConn

func NewAsyncHookWithFieldsAndConn(conn net.Conn, appName string, alwaysSentFields logrus.Fields) (*Hook, error)

NewAsyncHookWithFieldsAndConn creates a new hook to a Logstash instance using the supplied connection. Logs will be sent asynchronously.

func NewAsyncHookWithFieldsAndConnAndPrefix

func NewAsyncHookWithFieldsAndConnAndPrefix(conn net.Conn, appName string, alwaysSentFields logrus.Fields, prefix string) (*Hook, error)

NewAsyncHookWithFieldsAndConnAndPrefix creates a new hook to a Logstash instance using the suppolied connection and prefix. Logs will be sent asynchronously.

func NewAsyncHookWithFieldsAndPrefix

func NewAsyncHookWithFieldsAndPrefix(protocol, address, appName string, alwaysSentFields logrus.Fields, prefix string) (*Hook, error)

NewAsyncHookWithFieldsAndPrefix creates a new hook to a Logstash instance, which listens on `protocol`://`address`. alwaysSentFields will be sent with every log entry. prefix is used to select fields to filter. Logs will be sent asynchronously.

func NewFilterHook

func NewFilterHook() *Hook

NewFilterHook makes a new hook which does not forward to logstash, but simply enforces the prefix rules.

func NewFilterHookWithPrefix

func NewFilterHookWithPrefix(prefix string) *Hook

NewFilterHookWithPrefix make a new hook which does not forward to logstash, but simply enforces the specified prefix.

func NewHook

func NewHook(protocol, address, appName string) (*Hook, error)

NewHook creates a new hook to a Logstash instance, which listens on `protocol`://`address`.

func NewHookWithConn

func NewHookWithConn(conn net.Conn, appName string) (*Hook, error)

NewHookWithConn creates a new hook to a Logstash instance, using the supplied connection.

func NewHookWithFields

func NewHookWithFields(protocol, address, appName string, alwaysSentFields logrus.Fields) (*Hook, error)

NewHookWithFields creates a new hook to a Logstash instance, which listens on `protocol`://`address`. alwaysSentFields will be sent with every log entry.

func NewHookWithFieldsAndConn

func NewHookWithFieldsAndConn(conn net.Conn, appName string, alwaysSentFields logrus.Fields) (*Hook, error)

NewHookWithFieldsAndConn creates a new hook to a Logstash instance using the supplied connection.

func NewHookWithFieldsAndConnAndPrefix

func NewHookWithFieldsAndConnAndPrefix(conn net.Conn, appName string, alwaysSentFields logrus.Fields, prefix string) (*Hook, error)

NewHookWithFieldsAndConnAndPrefix creates a new hook to a Logstash instance using the suppolied connection and prefix.

func NewHookWithFieldsAndPrefix

func NewHookWithFieldsAndPrefix(protocol, address, appName string, alwaysSentFields logrus.Fields, prefix string) (*Hook, error)

NewHookWithFieldsAndPrefix creates a new hook to a Logstash instance, which listens on `protocol`://`address`. alwaysSentFields will be sent with every log entry. prefix is used to select fields to filter.

func (*Hook) Fire

func (h *Hook) Fire(entry *logrus.Entry) error

Fire send message to logstash. In async mode log message will be dropped if message buffer is full. If you want wait until message buffer frees – set WaitUntilBufferFrees to true.

func (*Hook) Levels

func (h *Hook) Levels() []logrus.Level

Levels specifies "active" log levels. Log messages with this levels will be sent to logstash.

func (*Hook) WithField

func (h *Hook) WithField(key string, value interface{})

WithField add field with value that will be sent with each message

func (*Hook) WithFields

func (h *Hook) WithFields(fields logrus.Fields)

WithFields add fields with values that will be sent with each message

func (*Hook) WithPrefix

func (h *Hook) WithPrefix(prefix string)

WithPrefix sets a prefix filter to use in all subsequent logging

type LogstashFormatter

type LogstashFormatter struct {
	Type string // if not empty use for logstash type field.

	// TimestampFormat sets the format used for timestamps.
	TimestampFormat string
}

LogstashFormatter generates json in logstash format. Logstash site: http://logstash.net/

func (*LogstashFormatter) Format

func (f *LogstashFormatter) Format(entry *logrus.Entry) ([]byte, error)

Format formats log message.

func (*LogstashFormatter) FormatWithPrefix

func (f *LogstashFormatter) FormatWithPrefix(entry *logrus.Entry, prefix string) ([]byte, error)

FormatWithPrefix removes prefix from keys and formats log message.