logtailer

README

logtailer

Summary

A simple log tailer written in go. Originally written by Parse to consume production log data of various formats and feed it into Facebook's analytics systems for day-to-day operations. logtailer uses a modular approach to consuming logs and directing output. To support new log types or change existing behavior, simply implement the Profile interface to suit your needs. The reference implementations in this release consume logs directly and output parsed lines as stdout.

Reference implementations include:

• a dummy profile used for demonstration. Consumes the input log file and prints to stdout
• a mongodb log parser based on a Programmable Expression Grammar (PEG). At Parse we found the PEG parser to perform better, and more accurately, than any regex-based pattern we could come up with, due to the complex nature of MongoDB log lines. The PEG parser focuses on actual operations (queries, inserts, commands, etc) and ignores other noise. At Parse, we processed 4B operations/day with this tailer. The mongodb tailer converts lines into a consistent JSON format that can be processed by other analytics systems.
• an sshd log parser that converts ssh login events to JSON

Building

Has been tested on go version 1.5.3, but will probably work with earlier versions.

$ go install github.com/ParsePlatform/logtailer/cmd/logtailer

External Dependencies

Logtailer was written to run cron once per minute. Since logs rotate less frequently than that, it relies on the logtail2 command readily available in the Ubuntu repositories. When repeatedly run with the same log file for input, logtail2 ensures that only new lines are consumed. To make this work, ensure that the logtailer run directory exists:

mkdir -p /var/run/logtailer
chown <your tailer user> /var/run/logtailer

Alternatively, logtailer accepts stdin as input. Simply specify - to the log_file flag when invoking logtailer.

Testing

The simplest test of the binary is to invoke the dummy profile with some simple input. It should be echoed back, along with some statistics that go to stderr.

echo -n '1\n2\n3\n' | ./logtailer dummy -log_file -
1
2
3
{"Records":3,"ParseErrors":0,"SendErrors":0}

Tuning

By default, the logtailer creates a worker for each CPU on the system. You can override this by setting the num_workers flag.

Documentation

Overview

Package logtailer provides an easy way to write log file munging programs

Index

• type Logtailer
  • func NewLogtailer(profile profiles.Profile, logFile string, stateDir string, logger *log.Logger) *Logtailer
  • func (lt *Logtailer) PrepEnvironment() error
  • func (lt *Logtailer) Run(numWorkers int) (*Stats, error)
  • func (lt *Logtailer) Stop()
• type Splitter
• type Stats
  • func (s *Stats) IsHealthy() bool
  • func (s *Stats) String() string

Examples

• NewLogtailer

Types

type Logtailer

type Logtailer struct {
	Logger   *log.Logger
	Profile  profiles.Profile
	LogFile  string
	StateDir string
	DryRun   bool
	// contains filtered or unexported fields
}

Logtailer holds the state of a logtailer program which represents the consumption of an input source with a particular profile

func NewLogtailer

func NewLogtailer(profile profiles.Profile, logFile string, stateDir string, logger *log.Logger) *Logtailer

NewLogtailer prepares a new Logtailer from a profile, input logfile, state directory, and a logger.

Example

tmpFile, _ := ioutil.TempFile("", "")
defer os.Remove(tmpFile.Name())

logger := log.New(os.Stderr, "logtailer", log.LstdFlags)
tailer := NewLogtailer(&dummy.DummyProfile{}, tmpFile.Name(), "/tmp/", logger)
stats, _ := tailer.Run(1)
fmt.Println(stats)

Output:

{"Records":0,"ParseErrors":0,"SendErrors":0}

func (*Logtailer) PrepEnvironment

func (lt *Logtailer) PrepEnvironment() error

PrepEnvironment ensures the specified log file and state directories exist.

func (*Logtailer) Run

func (lt *Logtailer) Run(numWorkers int) (*Stats, error)

Run starts the consumption of the input source and starts `numWorkers` separate goroutines to process lines.

If the log lines are ordered `numWorkers` should be 1.

func (*Logtailer) Stop

func (lt *Logtailer) Stop()

Stop stops consuming new input

type Splitter

type Splitter interface {
	Split(data []byte, atEOF bool) (advance int, token []byte, err error)
}

Splitter supplies a custom function for a bufio.Scanner

type Stats

type Stats struct {
	Records     int
	ParseErrors int
	SendErrors  int
	sync.Mutex
}

Stats holds basic metrics about the logtailer run.

func (*Stats) IsHealthy

func (s *Stats) IsHealthy() bool

IsHealthy returns true if the stats appear healthy.

func (*Stats) String

func (s *Stats) String() string