cwlq

README

cwlq

cwlq is a tool/package for querying logs (of Amazon CloudWatch Logs) stored in various datasources.

Usage

$ cwlq s3://myrds-audit-logs/2022/12/11/ --parser rdsaudit --filter "message.host == '10.0.1.123'" --filter "message.object contains 'INSERT'"

Support datasource

Amazon S3

s3://bucket/path/to

Local file or directory

local://path/to local:///root/path/to

Fake datasource

fake://rdsaudit?duration=3sec

Amazon CloudWatch Logs directly

WIP

Note Perhaps it would be better to use CloudWatch Logs Insights.

Support Parser

rdsaudit

Parser for gziped logs via MariaDB Audit Plugin for Amazon RDS.

Field	Example	Description
timestamp	1670717181000	The Unix time stamp for the logged event with microsecond precision.
message.timestamp	20221211 00:06:21	The Unix time stamp for the logged event with microsecond precision????
message.serverhost	ip-10-0-0-123	The name of the instance that the event is logged for.
message.username	redash	The connected user name of the user.
message.host	10.0.1.123	The host that the user connected from.
message.connectionid	502547196	The connection ID number for the logged operation.
message.queryid	84996781288	The query ID number, which can be used for finding the relational table events and related queries. For TABLE events, multiple lines are added.
message.operation	QUERY	The recorded action type. Possible values are: CONNECT, QUERY, READ, WRITE, CREATE, ALTER, RENAME, and DROP.
message.database	dbname	The active database, as set by the USE command.
message.object	SELECT * FROM accounts;	For QUERY events, this value indicates the query that the database performed. For TABLE events, it indicates the table name.
message.retcode	0	The return code of the logged operation.
message.connection_type	1	The security state of the connection to the server.
raw	``	Raw data of log event.

Install

homebrew tap:

$ brew install pepabo/tap/cwlq

manually:

Download binany from releases page

go install:

$ go install github.com/pepabo/cwlq@latest

Documentation

Index

• type Cwlf
  • func New(dsn, parserType string, filters []string) (*Cwlf, error)
  • func (c *Cwlf) Filtered() int64
  • func (c *Cwlf) Outer(o outer.Outer)
  • func (c *Cwlf) Run(ctx context.Context) (err error)
  • func (c *Cwlf) Total() int64

Types

type Cwlf

type Cwlf struct {
	D datasource.Datasource
	P parser.Parser
	F *filter.Filter
	O outer.Outer
}

func New

func New(dsn, parserType string, filters []string) (*Cwlf, error)

func (*Cwlf) Filtered

func (c *Cwlf) Filtered() int64

func (*Cwlf) Outer

func (c *Cwlf) Outer(o outer.Outer)

func (*Cwlf) Run

func (c *Cwlf) Run(ctx context.Context) (err error)

func (*Cwlf) Total

func (c *Cwlf) Total() int64