log

README

Log Service

A simple implementation of a log repository that receives all log messages via gRPC and store them in a bleve repository.

REST API

TODO

Documentation

Overview

Package log provides a persistence layer for json-formatted logs generated by the application.

It is intended to provide an out-of-the-box solution for storing and querying the logs, but should be replaced by more scalable solutions like ELK.

Index

• Constants
• func BleveDeleteLogs(idx bleve.Index, str string) (int64, error)
• func BleveDuplicateIndex(from bleve.Index, inserts chan interface{}, logger func(string)) error
• func BleveListLogs(idx bleve.Index, str string, page int32, size int32) (chan log.ListLogResponse, error)
• func BlevePutLog(idx bleve.Index, line map[string]string) error
• func UnmarshallLogMsgFromDoc(doc *document.Document, msg *log.LogMessage)
• func UnmarshallLogMsgFromFields(m map[string]interface{}, msg *log.LogMessage)
• type IndexableLog
  • func MarshallLogMsg(line map[string]string) (*IndexableLog, error)
• type MessageRepository
• type SyslogServer
  • func NewSyslogServer(indexPath string, mappingName string, rotationSize int64) (*SyslogServer, error)
  • func (s *SyslogServer) AggregatedLogs(msgId string, timeRangeType string, refTime int32) (chan log.TimeRangeResponse, error)
  • func (s *SyslogServer) Close()
  • func (s *SyslogServer) DeleteLogs(query string) (int64, error)
  • func (s *SyslogServer) ListLogs(str string, page, size int32) (chan log.ListLogResponse, error)
  • func (s *SyslogServer) Open(indexPath string, mappingName string) error
  • func (s *SyslogServer) PutLog(line map[string]string) error
  • func (s *SyslogServer) Resync(logger *zap.Logger) error
  • func (s *SyslogServer) Truncate(max int64, logger *zap.Logger) error

Constants

const (
	MinRotationSize     = 68 * 1024
	DefaultRotationSize = int64(200 * 1024 * 1024)
)

Functions

func BleveDeleteLogs

func BleveDeleteLogs(idx bleve.Index, str string) (int64, error)

BleveDeleteLogs queries the bleve index, based on the passed query string and deletes the results

func BleveDuplicateIndex

func BleveDuplicateIndex(from bleve.Index, inserts chan interface{}, logger func(string)) error

func BleveListLogs

func BleveListLogs(idx bleve.Index, str string, page int32, size int32) (chan log.ListLogResponse, error)

BleveListLogs queries the bleve index, based on the passed query string. It returns the results as a stream of log.ListLogResponse with the values of the indexed fields for each corresponding hit. Results are ordered by descending timestamp rather than by score.

func BlevePutLog

func BlevePutLog(idx bleve.Index, line map[string]string) error

BlevePutLog stores a new log msg in a bleve index. It expects a map[string]string retrieved from a deserialized proto log message.

func UnmarshallLogMsgFromDoc

func UnmarshallLogMsgFromDoc(doc *document.Document, msg *log.LogMessage)

UnmarshallLogMsgFromDoc populates the LogMessage from the passed bleve document.

func UnmarshallLogMsgFromFields

func UnmarshallLogMsgFromFields(m map[string]interface{}, msg *log.LogMessage)

Types

type IndexableLog

type IndexableLog struct {
	Nano int
	log.LogMessage
}

IndexableLog extends default log.LogMessage struct to add index specific methods

func MarshallLogMsg

func MarshallLogMsg(line map[string]string) (*IndexableLog, error)

MarshallLogMsg creates an IndexableLog object and populates the inner LogMessage with known fields of the passed JSON line.

type MessageRepository

type MessageRepository interface {
	PutLog(map[string]string) error
	ListLogs(string, int32, int32) (chan log.ListLogResponse, error)
	DeleteLogs(string) (int64, error)
	AggregatedLogs(string, string, int32) (chan log.TimeRangeResponse, error)
	Resync(logger *zap.Logger) error
	Truncate(max int64, logger *zap.Logger) error
}

MessageRepository exposes interface methods to manage the log messages provided by Pydio.

type SyslogServer

type SyslogServer struct {
	SearchIndex bleve.IndexAlias
	// contains filtered or unexported fields
}

SyslogServer is the syslog specific implementation of the Log server

func NewSyslogServer

func NewSyslogServer(indexPath string, mappingName string, rotationSize int64) (*SyslogServer, error)

NewSyslogServer creates and configures a default Bleve instance to store technical logs Setting rotationSize to -1 fully disables rotation

func (*SyslogServer) AggregatedLogs

func (s *SyslogServer) AggregatedLogs(msgId string, timeRangeType string, refTime int32) (chan log.TimeRangeResponse, error)

AggregatedLogs performs a faceted query in the syslog repository. UNIMPLEMENTED.

func (*SyslogServer) Close

func (s *SyslogServer) Close()

func (*SyslogServer) DeleteLogs

func (s *SyslogServer) DeleteLogs(query string) (int64, error)

DeleteLogs truncate logs based on a search query

func (*SyslogServer) ListLogs

func (s *SyslogServer) ListLogs(str string, page, size int32) (chan log.ListLogResponse, error)

ListLogs performs a query in the bleve index, based on the passed query string. It returns results as a stream of log.ListLogResponse for each corresponding hit. Results are ordered by descending timestamp rather than by score.

func (*SyslogServer) Open

func (s *SyslogServer) Open(indexPath string, mappingName string) error

func (*SyslogServer) PutLog

func (s *SyslogServer) PutLog(line map[string]string) error

PutLog adds a new LogMessage in the syslog index.

func (*SyslogServer) Resync

func (s *SyslogServer) Resync(logger *zap.Logger) error

Resync creates a copy of current index. It has been originally used for switching analyze format from bleve to scorch.

func (*SyslogServer) Truncate

func (s *SyslogServer) Truncate(max int64, logger *zap.Logger) error

Truncate gathers size of existing indexes, starting from last. When max is reached it starts deleting all previous indexes.