log

README

Log Service

A simple implementation of a log repository that receives all log messages via gRPC and store them in a bleve repository.

REST API

TODO

Documentation

Overview

Package log provides a persistence layer for json-formatted logs generated by the application.

It is intended to provide an out-of-the-box solution for storing and querying the logs, but should be replaced by more scalable solutions like ELK.

Index

• func BleveDeleteLogs(idx bleve.Index, str string) (int64, error)
• func BleveDuplicateIndex(from bleve.Index, to bleve.Index) error
• func BleveListLogs(idx bleve.Index, str string, page int32, size int32) (chan log.ListLogResponse, error)
• func BlevePutLog(idx bleve.Index, line map[string]string) error
• func FromLogMsgToMap(doc *log.LogMessage, m map[string]interface{})
• func FromLogMsgToStringMap(doc *log.LogMessage, m map[string]string)
• func UnmarshallLogMsgFromDoc(doc *document.Document, msg *log.LogMessage)
• type IndexableLog
  • func MarshallLogMsg(line map[string]string) (*IndexableLog, error)
• type MessageRepository
• type SyslogServer
  • func NewSyslogServer(bleveIndexPath string, mappingName string, deleteOnClose ...bool) (*SyslogServer, error)
  • func (s *SyslogServer) AggregatedLogs(msgId string, timeRangeType string, refTime int32) (chan log.TimeRangeResponse, error)
  • func (s *SyslogServer) Close()
  • func (s *SyslogServer) DeleteLogs(query string) (int64, error)
  • func (s *SyslogServer) ListLogs(str string, page, size int32) (chan log.ListLogResponse, error)
  • func (s *SyslogServer) PutLog(line map[string]string) error
  • func (s *SyslogServer) Resync() error

Functions

func BleveDeleteLogs

func BleveDeleteLogs(idx bleve.Index, str string) (int64, error)

BleveDeleteLogs queries the bleve index, based on the passed query string and deletes the results

func BleveDuplicateIndex

func BleveDuplicateIndex(from bleve.Index, to bleve.Index) error

func BleveListLogs

func BleveListLogs(idx bleve.Index, str string, page int32, size int32) (chan log.ListLogResponse, error)

BleveListLogs queries the bleve index, based on the passed query string. It returns the results as a stream of log.ListLogResponse with the values of the indexed fields for each corresponding hit. Results are ordered by descending timestamp rather than by score.

func BlevePutLog

func BlevePutLog(idx bleve.Index, line map[string]string) error

BlevePutLog stores a new log msg in a bleve index. It expects a map[string]string retrieved from a deserialized proto log message.

func FromLogMsgToMap

func FromLogMsgToMap(doc *log.LogMessage, m map[string]interface{})

FromLogMsgToMap uses reflection to transform a log.LogMessage in a map[string]interface{}. It also converts MsgId to corresponding label and timestamp as second in time.

func FromLogMsgToStringMap

func FromLogMsgToStringMap(doc *log.LogMessage, m map[string]string)

FromLogMsgToStringMap uses reflection to transform a log.LogMessage in a map[string]string.

func UnmarshallLogMsgFromDoc

func UnmarshallLogMsgFromDoc(doc *document.Document, msg *log.LogMessage)

UnmarshallLogMsgFromDoc populates the LogMessage from the passed bleve document.

Types

type IndexableLog

type IndexableLog struct {
	Nano int
	log.LogMessage
}

IndexableLog extends default log.LogMessage struct to add index specific methods

func MarshallLogMsg

func MarshallLogMsg(line map[string]string) (*IndexableLog, error)

MarshallLogMsg creates an IndexableLog object and populates the inner LogMessage with known fields of the passed JSON line.

type MessageRepository

type MessageRepository interface {
	PutLog(map[string]string) error
	ListLogs(string, int32, int32) (chan log.ListLogResponse, error)
	DeleteLogs(string) (int64, error)
	AggregatedLogs(string, string, int32) (chan log.TimeRangeResponse, error)
	Resync() error
}

MessageRepository exposes interface methods to manage the log messages provided by Pydio.

type SyslogServer

type SyslogServer struct {
	Index bleve.Index
	// contains filtered or unexported fields
}

SyslogServer is the syslog specific implementation of the Log server

func NewSyslogServer

func NewSyslogServer(bleveIndexPath string, mappingName string, deleteOnClose ...bool) (*SyslogServer, error)

NewSyslogServer creates and configures a default Bleve instance to store technical logs

func (*SyslogServer) AggregatedLogs

func (s *SyslogServer) AggregatedLogs(msgId string, timeRangeType string, refTime int32) (chan log.TimeRangeResponse, error)

AggregatedLogs performs a faceted query in the syslog repository. UNIMPLEMENTED.

func (*SyslogServer) Close

func (s *SyslogServer) Close()

func (*SyslogServer) DeleteLogs

func (s *SyslogServer) DeleteLogs(query string) (int64, error)

func (*SyslogServer) ListLogs

func (s *SyslogServer) ListLogs(str string, page, size int32) (chan log.ListLogResponse, error)

ListLogs performs a query in the bleve index, based on the passed query string. It returns results as a stream of log.ListLogResponse for each corresponding hit. Results are ordered by descending timestamp rather than by score.

func (*SyslogServer) PutLog

func (s *SyslogServer) PutLog(line map[string]string) error

PutLog adds a new LogMessage in the syslog index.

func (*SyslogServer) Resync

func (s *SyslogServer) Resync() error