logkafka

README

logkafka

Status

import "github.com/nxsre/logrus-kafka-hook"

Package logkafka provides a Logrus Hook implementation for publishing log messages on a Kafka Topic.

logkafka uses a builder pattern to configure the Hook. This provides simplicity and flexibility when configuring a Hook. There are convenience functions for creating a sarama.AsyncProducer and logrus.Formatter.

Basic Usage

// use SimpleProducer to create an AsyncProducer
producer := logkafka.SimpleProducer([]string{"127.0.0.1:9092"}, sarama.CompressionSnappy, sarama.WaitForLocal, nil)

// use DefaultFormatter to create a JSONFormatter with pre-defined fields or override with any fields
// create the Hook and use the builder functions to apply configurations
hook := logkafka.New().WithFormatter(logkafka.DefaultFormatter(logrus.Fields{"type": "myappName"})).WithProducer(producer)

// create a new logger and add the hook
log := logrus.New()
log.Hooks.Add(hook)

log.Debug("Hello World")

Example of formatted message published to Kafka

{
	"@timestamp": "2018-04-20T04:03:00Z",
	"@version": "1",
	"level": "info",
	"message": "Hello World",
	"type": "myappName"
}

Documentation

Overview

Package logkafka provides a Logrus Hook implementation for publishing log messages on a Kafka Topic.

logkafka uses a builder pattern to configure the Hook. This provides simplicity and flexibility when configuring a Hook. There are convenience functions for creating a sarama.AsyncProducer and logrus.Formatter.

Basic Usage

// use SimpleProducer to create an AsyncProducer
producer := logkafka.SimpleProducer([]string{"127.0.0.1:9092"}, sarama.CompressionSnappy, sarama.WaitForLocal, nil)

// use DefaultFormatter to create a JSONFormatter with pre-defined fields or override with any fields
// create the Hook and use the builder functions to apply configurations
hook := logkafka.NewHook().WithFormatter(logkafka.DefaultFormatter(logrus.Fields{"type": "myappName"})).WithProducer(producer)

// create a new logger and add the hook
log := logrus.NewHook()
log.Hooks.Add(hook)

log.Debug("Hello World")

Example of formatted message published to Kafka

{
  "@timestamp": "2018-04-20T04:03:00Z",
  "@version": "1",
  "level": "info",
  "message": "Hello World",
  "type": "myappName"
}

Index

• Constants
• Variables
• func DefaultFormatter(fields logrus.Fields) logrus.Formatter
• func NewLogger(hook *Hook) *logrus.Logger
• func SimpleProducer(brokers []string, compression sarama.CompressionCodec, ack sarama.RequiredAcks, ...) (sarama.AsyncProducer, error)
• type Hook
  • func NewHook() *Hook
  • func (h *Hook) Fire(entry *logrus.Entry) error
  • func (h *Hook) Levels() []logrus.Level
  • func (h *Hook) WithFormatter(formatter logrus.Formatter) *Hook
  • func (h *Hook) WithLevels(levels []logrus.Level) *Hook
  • func (h *Hook) WithProducer(producer sarama.AsyncProducer) *Hook
  • func (h *Hook) WithTopic(topic string) *Hook
• type StructuredFormatter
  • func (f StructuredFormatter) Format(e *logrus.Entry) ([]byte, error)

Constants

const RFC3339NanoFixed = "2006-01-02T15:04:05.000000000Z07:00"

RFC3339NanoFixed is time.RFC3339Nano with nanoseconds padded using zeros to ensure the formatted time is always the same number of characters.

Variables

var ErrNoProducer = errors.New("no producer defined")

ErrNoProducer is used when hook Fire method is called and no producer was configured.

Functions

func DefaultFormatter

func DefaultFormatter(fields logrus.Fields) logrus.Formatter

DefaultFormatter returns a default structured formatter: A JSON format with "@version" set to "1" (unless set differently in `fields`, "type" to "log" (unless set differently in `fields`), "@timestamp" to the log time and "message" to the log message.

**Note:** to set a different configuration use the `StructuredFormatter` structure.

func NewLogger

func NewLogger(hook *Hook) *logrus.Logger

func SimpleProducer

func SimpleProducer(brokers []string, compression sarama.CompressionCodec, ack sarama.RequiredAcks, tlscfg *tls.Config) (sarama.AsyncProducer, error)

SimpleProducer accepts a minimal set of configurations and creates an AsyncProducer.

Types

type Hook

type Hook struct {
	// contains filtered or unexported fields
}

Hook represents a logrus hook for Kafka.

func NewHook

func NewHook() *Hook

NewHook creates a new logrus hook for Kafka.

Defaults:

Formatter: *logrus.TextFormatter*
Levels: *logrus.AllLevels*
Topic: *"logs"*

func (*Hook) Fire

func (h *Hook) Fire(entry *logrus.Entry) error

Fire writes the entry as a message on Kafka.

func (*Hook) Levels

func (h *Hook) Levels() []logrus.Level

Levels returns all log levels that are enabled for writing messages to Kafka.

func (*Hook) WithFormatter

func (h *Hook) WithFormatter(formatter logrus.Formatter) *Hook

WithFormatter adds a formatter to the created Hook.

func (*Hook) WithLevels

func (h *Hook) WithLevels(levels []logrus.Level) *Hook

WithLevels adds levels to the created Hook.

func (*Hook) WithProducer

func (h *Hook) WithProducer(producer sarama.AsyncProducer) *Hook

WithProducer adds a producer to the created Hook.

func (*Hook) WithTopic

func (h *Hook) WithTopic(topic string) *Hook

WithTopic adds a topic to the created Hook.

type StructuredFormatter

type StructuredFormatter struct {
	logrus.Formatter
	logrus.Fields
}

StructuredFormatter represents a structured format. It has logrus.Formatter which formats the entry and logrus.Fields which are added to the JSON message if not given in the entry data.

**Note:** use the `DefaultFormatter` function to create a default StructuredFormatter.

func (StructuredFormatter) Format

func (f StructuredFormatter) Format(e *logrus.Entry) ([]byte, error)

Format formats an entry to a structured format according to the given Formatter and Fields.

**Note:** the given entry is copied and not changed during the formatting process.