README
¶
mtail - extract whitebox monitoring data from application logs for collection into a timeseries database
mtail is a tool for extracting metrics from application logs to be exported
into a timeseries database or timeseries calculator for alerting and
dashboarding.
It fills a monitoring niche by being the glue between applications that do not export their own internal state (other than via logs) and existing monitoring systems, such that system operators do not need to patch those applications to instrument them or writing custom extraction code for every such application.
The extraction is controlled by mtail programs which define patterns and actions:
# simple line counter
counter line_count
/$/ {
line_count++
}
Metrics are exported for scraping by a collector as JSON or Prometheus format over HTTP, or can be periodically sent to a collectd, StatsD, or Graphite collector socket.
Read the programming guide if you want to learn how to write mtail programs.
Mailing list: https://groups.google.com/forum/#!forum/mtail-users
Installation
There are various ways of installing mtail.
Precompiled binaries
Precompiled binaries for released versions are available in the Releases page on Github. Using the latest production release binary is the recommended way of installing mtail.
Windows, OSX and Linux binaries are available.
Building from source
To build mtail from the source code yourself you need to have a working Go
environment with version 1.9 or greater installed. mtail is go getable and
go installable from this repository but is best if you use the Makefile to
build it.
go get -u github.com/google/mtail
cd $GOPATH/src/github.com/google/mtail
make install
If you develop the compiler you will need some additional tools
like goyacc to be able to rebuild the parser.
See the Build instructions for more details.
A Dockerfile is included in this repository for local development as an
alternative to installing Go in your environment, and takes care of all the
build dependency installation, if you don't care for that.
Deployment
mtail works best when it paired with a timeseries-based calculator and
alerting tool, like Prometheus.
So what you do is you take the metrics from the log files and you bring them down to the monitoring system?
It deals with the instrumentation so the engineers don't have to! It has the extraction skills! It is good at dealing with log files!!
Read More
Full documentation at http://google.github.io/mtail/
Read more about writing mtail programs:
Read more about hacking on mtail
Read more about deploying mtail and your programs in a monitoring environment
- Deploying
- Interoperability with other systems
- Troubleshooting
- FAQ
After that, if you have any questions, please email (and optionally join) the mailing list: https://groups.google.com/forum/#!forum/mtail-users or file a new issue.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
mdot
Command mdot turns an mtail program AST into a graphviz graph on standard output.
|
Command mdot turns an mtail program AST into a graphviz graph on standard output. |
|
mgen
Command mgen generates mtail programs for fuzz testing by following a simple grammar.
|
Command mgen generates mtail programs for fuzz testing by following a simple grammar. |
|
internal
|
|
|
exporter
Package exporter provides the interface for getting metrics out of mtail, into your monitoring system of choice.
|
Package exporter provides the interface for getting metrics out of mtail, into your monitoring system of choice. |
|
metrics
Package metrics provides storage for metrics being recorded by mtail programs.
|
Package metrics provides storage for metrics being recorded by mtail programs. |
|
tailer
Package tailer provides a class that is responsible for tailing log files and extracting new log lines to be passed into the virtual machines.
|
Package tailer provides a class that is responsible for tailing log files and extracting new log lines to be passed into the virtual machines. |
|
testutil
Reimport the go-cmp package as the name 'cmp' conflicts with the cmp instruction in the vm.
|
Reimport the go-cmp package as the name 'cmp' conflicts with the cmp instruction in the vm. |
|
vm
Package vm provides a compiler and virtual machine environment for executing mtail programs.
|
Package vm provides a compiler and virtual machine environment for executing mtail programs. |
|
vm/code
Package code contains the bytecode instructions for the mtail virtual machine.
|
Package code contains the bytecode instructions for the mtail virtual machine. |
|
vm/parser
Package parser implements the parse phase of the mtail program compilation.
|
Package parser implements the parse phase of the mtail program compilation. |
|
vm/position
Package position implements a data structure for storing source code positions.
|
Package position implements a data structure for storing source code positions. |
|
watcher
Package watcher provides a way of watching for filesystem events and notifying observers when they occur.
|
Package watcher provides a way of watching for filesystem events and notifying observers when they occur. |