logquery

README

LogQuery

LogQuery is a tool for reading logfmt formatted data, then running SQL queries on it using SQLite.

Demo:

$ cat sample.txt
event=error code=123 msg="something bad happened" timestamp="1970-01-01 15:23"
event=error code=123 msg="something bad happened" timestamp="1970-01-01 17:35"
event=error code=456 msg="something went wrong" timestamp="1970-01-01 18:56"
$ logquery "SELECT * FROM log WHERE code=123" < sample.txt
timestamp="1970-01-01 15:23" code=123 event="error" msg="something bad happened"
timestamp="1970-01-01 17:35" code=123 event="error" msg="something bad happened"
$ logquery -o csv "SELECT * FROM log WHERE code=123" < sample.txt
event,msg,timestamp,code
error,something bad happened,1970-01-01 15:23,123
error,something bad happened,1970-01-01 17:35,123

Installation

make install

Or

go get -u git.sr.ht/~charles/logquery

Usage

Usage: logquery <query> [<path>]

Arguments:
  <query>     SQLite query to run on the input log.
  [<path>]    Input path to read from instead of standard in.

Flags:
  -h, --help                      Show context-sensitive help.
  -f, --output-format="logfmt"    Specify the output format for results. Must be one of: logfmt, csv.
  -o, --output="-"                Specify the output file to use instead of standard out.
      --cpuprofile="-"            Run the program with CPU profiling and write the results to this file.
      --memprofile="-"            Run the program with memory profiling and write the results to this file.
      --version

LogFmt Parser

LogQuery also implements its own LogFmt parser via the parser module.

Performance

The initial version of LogQuery was written in Python, and achieved a performance of roughly 11k records/second. It has since been rewritten Go, and now runs at roughly 70k records/second. Both benchmarks were run on a Thinkpad T430 with an i7-3632QM, 16GB DDR3 memory, and an SSD with ZFS (with encryption and compression enabled).

Limitations

• LogQuery currently works by reading the entire input stream into an in-memory database. This means it can take a while for large input files.

Future Work

• Support more output formats
• Support CSV input

Documentation

Overview

Package logquery implements core functionality for the logquery tool. Note that the CLI entry point is in cmd/logquery/main.go.

Index

• func RunQuery(stream io.Reader, output io.Writer, query string, format OutputFormat) error
• type OutputFormat

Functions

func RunQuery

func RunQuery(stream io.Reader, output io.Writer, query string, format OutputFormat) error

RunQuery executes the specified SQLite query on the logfmt data provided on the given stream input, writing any results to the given output stream using the specified format.

Types

type OutputFormat

type OutputFormat string

OutputFormat is used to represent possible formats that can be used when generating output.

const CSV OutputFormat = "csv"

CSV indicates that output should be formatted in CSV.

const LogFmt OutputFormat = "logfmt"

LogFmt indicates output should be formatted in logfmt.