Lixie
Lixie is infrastructure awareness tool. The name does not really matter and
it also does not mean anything (I am not going to use 'dog ate my homework'
excuse but instead use the modern excuse - Mistral came up with it).
The goal of Lixie is to be single binary, zero external dependency
tool. Currently there are two planned modes:
Log filtering - high level idea
Most of the modern logs are, to put it bluntly, spam. Typical SIEM
approaches rely on matching specific log rules of interest, and ignoring
the rest. However, this leaves the 'unknown unknown' problem on the table,
as if you don't have rule for it, you are not aware of it happening either.
Lixie's approach is inverse - everything is worth looking at, at least
once. Lixie provides a way of cultivating ruleset about which lines are
interesting, which are not, and showing the logs based on the produced
filtering criteria. If integrated with Vector, it can also enrich the logs
with the filtering results, and subsequently Vector pipeline can e.g. drop
spam, or redirect it to something with shorter retention period.
Functionality
-
Lixie can pull logs from Loki, categorize them, and show them.
-
Lixie has rule editor (and human readable dump format) for the log
classification rules
Demo
Here is an example. Note that only
toplevel links work - anything beyond that gets redirected to your own
local Lixie instance which probably does not exist.
Planned TODO (There is no guarantee these actually get implemented)
Features
Big features
-
Optimize rule evaluation (it could be also used when generating rules for
Vector); identify the stuff with most cardinality, match by that first
(and order the rules based on that too; it doesn't hurt to make it user
visible)
-
idea: # of field matchers / # of regexp field matchers = score (and
then do iteratively by fields)
-
As an option, could also parallelize it?
-
Rethink how rules are stored; just big json file can get bit unwieldy?
-
summary log rules page
- show by e.g. source, and some nice statistics about hits?
Robustness
-
Implement more unit tests
-
Properly vendor static resources (bootstrap CSS, htmx JS)
-
identify better logging pattern than printf :p
Prettiness