springparse

README

SpringParse

Motivation

As of now, we can use fluentd daemonset to have a pod on every eks node. Mount the /var/log/containers and tail any incoming logs and send it to elasticsearch. However, the log format isn't ideal. For example, the below log will simply stick the sprintboot logs to the key value log. The log level, executor, thread, and loggername are not searchable in elasticsearch that way. fluentd have log transformation via regexp but that plugin is not suffice for the use case as regexing logs for transformation is too difficult or simply not possible.

{
  "log": "2020-03-30 19:57:33.702  INFO 1 --- [       Thread-3] c.v.b.t.fooWebhookEventConsumer      : FOOLOG",
  "stream": "stdout",
  "time": "2020-03-30T19:57:33.703403463Z"
}

Usage

For details on setting environment variables and configurations please refer to documentation

Example

Golang version

springparse is currently compatible with golang version from 1.12+.

Todo

• implement more prometheus metrics

Documentation

Index

• type Client
  • func New() *Client
  • func (r *Client) Server() error
  • func (r *Client) ShowConfiguration()
  • func (r *Client) SpringParse()
  • func (r *Client) ValidateAwsEnv() error
  • func (r *Client) ValidateOtherEnv() error
• type Runner

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client the struct that runs the program

func New

func New() *Client

New creates a client struct with map initialized

func (*Client) Server

func (r *Client) Server() error

Server host a metric server so SRE can grab metrics

func (*Client) ShowConfiguration

func (r *Client) ShowConfiguration()

ShowConfiguration shows the environment variables being set

func (*Client) SpringParse

func (r *Client) SpringParse()

SpringParse This is the main program

func (*Client) ValidateAwsEnv

func (r *Client) ValidateAwsEnv() error

ValidateAwsEnv validates all aws related configration and set defaults from environment variables

func (*Client) ValidateOtherEnv

func (r *Client) ValidateOtherEnv() error

ValidateOtherEnv validates all system info from environment variables.

type Runner

type Runner struct {
	Buffer   *elasticOut
	BufferId string
}

Runner Runner spawn by go routine