loges

package module
v0.0.0-...-8d6c616 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2018 License: MIT Imports: 25 Imported by: 3

README

Logging Utility & Daemon

Read log data from Inputs [Tail Files, Stdin, Monit], perform transforms [Combine lines(multi-line-errors)] and output to [ElasticSearch, Stdout] Recognize lines with metrics and send to Monitoring [Graphite, InfluxDB, ..]

Drawing

Why?

We had 2 needs: 1) to send data to Elasticsearch for viewing in http://kibana.org/ and 2) if possible, unify the Logging/Metrics systems data-collection-forwarding.

There are probably better tools out there for this but putting together the specific combination of: (LogStash format in Elasticsearch, Tail files, Read Kafka, Metrics read from log files) didn't happen, see alternates below.

Features

  • Inputs
    • Stdin
    • Tail Logs (multiple files)
    • Monit (metrics via http)
    • Http (custom)
  • Transforms:
    • Logstash http://logstash.net/
    • Colorizer for console
    • Concat into single line when needed (e.g. error stack trace)
    • Separate Metrics Log Lines from regular log lines
    • Custom plugins
  • Log Line Outputs
    • Stdout (optional colorized)
    • Elasticsearch
  • Metric Outputs
    • Graphite

Alternatives

Usage

loges --source=monit,tail --filter=stdfiles --out=elasticsearch --metrics=graphite \
   /path/to/my/file \
   /path/to/another/file

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FirstNonWhitespace 

func FirstNonWhitespace(bs []byte) (r rune, ok bool)

Returns the first non-whitespace rune in the byte slice. Returns false if the input is not valid UTF-8 or the input contains no runes except whitespace. Returns

func FlattenMonit 

func FlattenMonit(r *MonitRequest) (url.Values, int64)

takes a monit object struture, and converts to a map[string]interface{} which it serializes to json, to send over a wire

func HttpRun 

func HttpRun(httpPort string, msgsOut chan *LineEvent)

We are starting an http -> collector 

accepts monit data

func IsJson 

func IsJson(by []byte) bool

func IsJsonArray 

func IsJsonArray(by []byte) bool

Determines if the bytes is a json array, only looks at prefix 

not parsing the entire thing

func IsJsonObject 

func IsJsonObject(by []byte) bool

func MakeCustomHandler 

func MakeCustomHandler(msgsOut chan *LineEvent) http.HandlerFunc

func MakeFileFlattener 

func MakeFileFlattener(filename string, msgChan chan *LineEvent) func(string)

This formatter reads go files and performs:

  1. Squashes multiple lines into one (as needed), Tries to squash panics(go) into one line
  2. Reads out the LineType/Level [DEBUG,INFO,METRIC] into a field

This expects log files in this format 

2013-05-25 13:25:32.475 authctx.go:169: [DEBUG] sink       Building sink for kafka from factory method

func MakeMonitHandler 

func MakeMonitHandler(msgsOut chan *LineEvent) http.HandlerFunc

func MonitParse 

func MonitParse(data []byte) (url.Values, int64)

accepts a smokr message of monit format, and parses and converts to a name/value pair format

func RunTransforms 

func RunTransforms(parallel int, msgChan chan *LineEvent)

just run the transforms

func StdinPruducer 

func StdinPruducer(msgChan chan *LineEvent)

sends messages from stdin for consumption

func TailFile 

func TailFile(filename string, config tail.Config, done chan bool, msgChan chan *LineEvent)

func ToElasticSearch 

func ToElasticSearch(msgChan chan *LineEvent, esType, esHost, ttl string,
	exitIfNoMsgs time.Duration, sendMetrics bool)

read the message channel and send to elastic search Uses the background Bulk Indexor, which has the **Possibility** of losing data if app panic/quits (but is much faster than non-bulk) 

@exitIfNoMsgs :  Should we panic if we don't see messages after this duration?

func ToStdout 

func ToStdout(msgChan chan *LineEvent, colorize bool)

read the message channel and send to stdout

func TransformRegister 

func TransformRegister(txform LineTransform)

func UpdateLogstashIndex 

func UpdateLogstashIndex()

update the index occasionally

Types

type Event 

type Event struct {
	Source      string                 `json:"@source"`
	Type        string                 `json:"@type"`
	Timestamp   time.Time              `json:"@timestamp"`
	Message     string                 `json:"@message"`
	Tags        []string               `json:"@tags,omitempty"`
	IndexFields map[string]interface{} `json:"@idx,omitempty"`
	Fields      map[string]interface{} `json:"@fields,omitempty"`
	Raw         *json.RawMessage       `json:"jsonfields,omitempty"`
	// contains filtered or unexported fields
}

A Logstash formatted event

func NewEvent 

func NewEvent(eventType, source, message string) *Event

func NewTsEvent 

func NewTsEvent(eventType, source, message string, t time.Time) *Event

New event using existing time stamp

func (*Event) Id 

func (e *Event) Id() string

func (*Event) Index 

func (e *Event) Index() string

func (*Event) SetId 

func (e *Event) SetId(id string)

Set the id instead of using mongo objectid as Id

func (*Event) String 

func (e *Event) String() string

type GraphiteRunner 

type GraphiteRunner struct {
	Mu sync.Mutex
	// contains filtered or unexported fields
}

func NewGraphiteRunner 

func NewGraphiteRunner(addr string) *GraphiteRunner

func (*GraphiteRunner) Close 

func (g *GraphiteRunner) Close()

type LineEvent 

type LineEvent struct {
	Data      []byte
	Prefix    string
	Ts        time.Time // Date string if found
	LogLevel  string    // [METRIC, INFO,  DEBUG]
	Source    string    // Source = filename if file, else monit, etc
	Offset    uint64
	Item      interface{}
	WriteErrs uint16
}

Representing data about a log line from file we are tailing

func (*LineEvent) IsJson 

func (l *LineEvent) IsJson() bool

func (*LineEvent) IsMetric 

func (l *LineEvent) IsMetric() bool

type LineTransform 

type LineTransform func(*LineEvent) *Event

func FileFormatter 

func FileFormatter(logstashType string, tags []string) LineTransform

file format [date source jsonmessage] parser

func FluentdFormatter 

func FluentdFormatter(logstashType string, tags []string) LineTransform

Fluentd format [date source jsonmessage] parser

func GraphiteTransform 

func GraphiteTransform(logstashType, addr, prefix string, metricsToEs bool) LineTransform

type MonitBlock 

type MonitBlock struct {
	Percent float32 `xml:"percent"`
	Usage   string  `xml:"usage"`
	Total   string  `xml:"total"`
}

<block> 

<percent>57.5</percent>
<usage>2051.3 MB</usage>
<total>37544.3 MB</total>

</block>

func (*MonitBlock) PercentTotalStr 

func (m *MonitBlock) PercentTotalStr() string

type MonitCpu 

type MonitCpu struct {
	Percent      float32 `xml:"percent"`
	PercentTotal float32 `xml:"percenttotal"`
}

func (*MonitCpu) PercentTotalStr 

func (m *MonitCpu) PercentTotalStr() string

type MonitINode 

type MonitINode struct {
	Percent float32 `xml:"percent"`
	Usage   float32 `xml:"usage"`
	Total   float32 `xml:"total"`
}

<inode> 

<percent>57.5</percent>
<usage>36786.6</usage>
<total>70127.3</total>

</inode>

func (*MonitINode) PercentTotalStr 

func (m *MonitINode) PercentTotalStr() string

type MonitMemory 

type MonitMemory struct {
	Percent       float32 `xml:"percent"`
	PercentTotal  float32 `xml:"percenttotal"`
	Kilobyte      int     `xml:"kilobyte"`
	KilobyteTotal int     `xml:"kilobytetotal"`
}

func (*MonitMemory) PercentTotalStr 

func (m *MonitMemory) PercentTotalStr() string

type MonitPlatform 

type MonitPlatform struct {
	Name    string `xml:"name"`
	Release string `xml:"release"`
	Version string `xml:"version"`
	Machine string `xml:"machine"`
	Cpu     int    `xml:"cpu"`
	Memory  int    `xml:"memory"`
	Swap    int    `xml:"swap"`
}

type MonitRequest 

type MonitRequest struct {
	Id            int            `xml:"id,attr"`
	Incarnation   string         `xml:"incarnation,attr"`
	Version       string         `xml:"version,attr"`
	Server        MonitServer    `xml:"server"`
	Platform      MonitPlatform  `xml:"platform"`
	Services      []MonitService `xml:"services>service"`
	ServiceGroups interface{}    `xml:"servicegroups"`
}

type MonitServer 

type MonitServer struct {
	Uptime      int    `xml:"uptime"`
	Poll        int    `xml:"poll"`
	StartDelay  int    `xml:"startdelay"`
	Host        string `xml:"localhostname"`
	ControlFile string `xml:"controlfile"`
}

type MonitService 

type MonitService struct {
	Type   int         `xml:"type"` // 0 = filesystem, 3 = service, 5= System(os)
	Name   string      `xml:"name,attr"`
	Ts     int         `xml:"collected_sec"`
	Tsu    int         `xml:"collected_usec"`
	Status int         `xml:"status"`
	Memory MonitMemory `xml:"memory"`
	Cpu    MonitCpu    `xml:"cpu"`
	Block  MonitBlock  `xml:"block"`
	INode  MonitINode  `xml:"inode"`
}

type NvMetrics 

type NvMetrics struct {
	url.Values
}

func NewNvMetrics 

func NewNvMetrics(qs string) (NvMetrics, error)

func (*NvMetrics) MetricTypeVal 

func (uv *NvMetrics) MetricTypeVal(name string) (string, string)

func (*NvMetrics) Value 

func (uv *NvMetrics) Value(name string) (interface{}, error)

Source Files

View all Source files

Directories

Path Synopsis
examples
customformat

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.