logs

README

logs-agent

logs-agent collects logs and submits them to datadog's infrastructure.

Structure

logs reads the config files, and instantiates what's needed. Each log line comes from a source (e.g. file, network, docker), and then enters one of the available pipeline - tailer|listener|container -> decoder -> processor -> sender -> auditor

Tailer tails a file and submits data to the processors

Listener listens on local network (TCP, UDP, Unix) and submits data to the processors

Container scans docker logs from stdout/stderr and submits data to the processors

Decoder converts bytes arrays into messages

Processor updates the messages, filtering, redacting or adding metadata, and submits to the forwarder

Sender submits the messages to the intake, and notifies the auditor

Auditor notes that messages were properly submitted, stores offsets for agent restarts

Tests

# Run the unit tests
inv test --targets=./pkg/logs --timeout=10

Documentation

Index

• func BlockUntilAutoConfigRanOnce(getAC func() *autodiscovery.AutoConfig, timeout time.Duration)
• func Flush(ctx context.Context)
• func GetMessageReceiver() *diagnostic.BufferedMessageReceiver
• func GetStatus() status.Status
• func IsAgentRunning() bool
• func Start(getAC func() *autodiscovery.AutoConfig) error
• func StartServerless(getAC func() *autodiscovery.AutoConfig, logsChan chan aws.LogMessage, ...) error
• func Stop()
• type Agent
  • func NewAgent(sources *config.LogSources, services *service.Services, ...) *Agent
  • func NewServerless(sources *config.LogSources, services *service.Services, ...) *Agent
  • func (a *Agent) Flush(ctx context.Context)
  • func (a *Agent) Start()
  • func (a *Agent) Stop()

Functions

func BlockUntilAutoConfigRanOnce

func BlockUntilAutoConfigRanOnce(getAC func() *autodiscovery.AutoConfig, timeout time.Duration)

BlockUntilAutoConfigRanOnce blocks until the AutoConfig has been run once. It also returns after the given timeout.

func Flush

func Flush(ctx context.Context)

Flush flushes synchronously the running instance of the Logs Agent. Use a WithTimeout context in order to have a flush that can be cancelled.

func GetMessageReceiver

func GetMessageReceiver() *diagnostic.BufferedMessageReceiver

GetMessageReceiver returns the diagnostic message receiver

func GetStatus

func GetStatus() status.Status

GetStatus returns logs-agent status

func IsAgentRunning

func IsAgentRunning() bool

IsAgentRunning returns true if the logs-agent is running.

func Start

func Start(getAC func() *autodiscovery.AutoConfig) error

Start starts logs-agent getAC is a func returning the prepared AutoConfig. It is nil until the AutoConfig is ready, please consider using BlockUntilAutoConfigRanOnce instead of directly using it. The parameter serverless indicates whether or not this Logs Agent is running in a serverless environment.

func StartServerless

func StartServerless(getAC func() *autodiscovery.AutoConfig, logsChan chan aws.LogMessage, extraTags []string) error

StartServerless starts a Serverless instance of the Logs Agent.

func Stop

func Stop()

Stop stops properly the logs-agent to prevent data loss, it only returns when the whole pipeline is flushed.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent represents the data pipeline that collects, decodes, processes and sends logs to the backend + ------------------------------------------------------ + | | | Collector -> Decoder -> Processor -> Sender -> Auditor | | | + ------------------------------------------------------ +

func NewAgent

func NewAgent(sources *config.LogSources, services *service.Services, processingRules []*logstypes.ProcessingRule, endpoints *logstypes.Endpoints) *Agent

NewAgent returns a new Logs Agent

func NewServerless

func NewServerless(sources *config.LogSources, services *service.Services, processingRules []*logstypes.ProcessingRule, endpoints *logstypes.Endpoints) *Agent

NewServerless returns a Logs Agent instance to run in a serverless environment. The Serverless Logs Agent has only one input being the channel to receive the logs to process. It is using a NullAuditor because we've nothing to do after having sent the logs to the intake.

func (*Agent) Flush

func (a *Agent) Flush(ctx context.Context)

Flush flushes synchronously the pipelines managed by the Logs Agent.

func (*Agent) Start

func (a *Agent) Start()

Start starts all the elements of the data pipeline in the right order to prevent data loss

func (*Agent) Stop

func (a *Agent) Stop()

Stop stops all the elements of the data pipeline in the right order to prevent data loss