harvester

Documentation

Overview

Package harvester harvests different inputs for new information. Currently two harvester types exist:

• log

• stdin

  The log harvester reads a file line by line. In case the end of a file is found with an incomplete line, the line pointer stays at the beginning of the incomplete line. As soon as the line is completed, it is read and returned.

  The stdin harvesters reads data from stdin.

Index

• Variables
• func MatchAny(matchers []match.Matcher, text string) bool
• type Harvester
  • func NewHarvester(cfg *common.Config, state file.State, states *file.States, outlet Outlet) (*Harvester, error)
  • func (h *Harvester) Harvest(r reader.Reader)
  • func (h *Harvester) SendStateUpdate()
  • func (h *Harvester) Setup() (reader.Reader, error)
  • func (h *Harvester) Stop()
• type LogFile
  • func NewLogFile(fs source.FileSource, config harvesterConfig) (*LogFile, error)
  • func (r *LogFile) Close()
  • func (r *LogFile) Read(buf []byte) (int, error)
• type Outlet

Variables

var (
	ErrFileTruncate = errors.New("detected file being truncated")
	ErrRenamed      = errors.New("file was renamed")
	ErrRemoved      = errors.New("file was removed")
	ErrInactive     = errors.New("file inactive")
	ErrClosed       = errors.New("reader closed")
)

Functions

func MatchAny

func MatchAny(matchers []match.Matcher, text string) bool

MatchAny checks if the text matches any of the regular expressions

Types

type Harvester

type Harvester struct {
	ID uuid.UUID
	// contains filtered or unexported fields
}

func NewHarvester

func NewHarvester(
	cfg *common.Config,
	state file.State,
	states *file.States,
	outlet Outlet,
) (*Harvester, error)

func (*Harvester) Harvest

func (h *Harvester) Harvest(r reader.Reader)

Harvest reads files line by line and sends events to the defined output

func (*Harvester) SendStateUpdate

func (h *Harvester) SendStateUpdate()

sendStateUpdate send an empty event with the current state to update the registry close_timeout does not apply here to make sure a harvester is closed properly. In case the output is blocked the harvester will stay open to make sure no new harvester is started. As soon as the output becomes available again, the finished state is written and processing can continue.

func (*Harvester) Setup

func (h *Harvester) Setup() (reader.Reader, error)

Setup opens the file handler and creates the reader for the harvester

func (*Harvester) Stop

func (h *Harvester) Stop()

Stop stops harvester and waits for completion

type LogFile

type LogFile struct {
	// contains filtered or unexported fields
}

func NewLogFile

func NewLogFile(
	fs source.FileSource,
	config harvesterConfig,
) (*LogFile, error)

func (*LogFile) Close

func (r *LogFile) Close()

func (*LogFile) Read

func (r *LogFile) Read(buf []byte) (int, error)

Read reads from the reader and updates the offset The total number of bytes read is returned.

type Outlet

type Outlet interface {
	SetSignal(signal <-chan struct{})
	OnEventSignal(data *util.Data) bool
	OnEvent(data *util.Data) bool
}