logs

README

logs-agent

logs-agent collects logs and submits them to datadog's infrastructure.

Structure

logs reads the config files, and instantiates what's needed. Each log line comes from a source (e.g. file, network, docker), and then enters one of the available pipeline - tailer|listener|container -> decoder -> processor -> sender -> auditor

Tailer tails a file and submits data to the processors

Listener listens on local network (TCP, UDP, Unix) and submits data to the processors

Container scans docker logs from stdout/stderr and submits data to the processors

Decoder converts bytes arrays into messages

Processor updates the messages, filtering, redacting or adding metadata, and submits to the forwarder

Sender submits the messages to the intake, and notifies the auditor

Auditor notes that messages were properly submitted, stores offsets for agent restarts

Tests

# Run the unit tests
inv test --targets=./pkg/logs --timeout=10

Documentation

Index

• func GetScheduler() *scheduler.Scheduler
• func GetStatus() status.Status
• func IsAgentRunning() bool
• func Start() error
• func Stop()
• type Agent
  • func NewAgent(sources *config.LogSources, services *service.Services, ...) *Agent
  • func (a *Agent) Start()
  • func (a *Agent) Stop()
• type Transport

Functions

func GetScheduler

func GetScheduler() *scheduler.Scheduler

GetScheduler returns the logs-config scheduler if set.

func GetStatus

func GetStatus() status.Status

GetStatus returns logs-agent status

func IsAgentRunning

func IsAgentRunning() bool

IsAgentRunning returns true if the logs-agent is running.

func Start

func Start() error

Start starts logs-agent

func Stop

func Stop()

Stop stops properly the logs-agent to prevent data loss, it only returns when the whole pipeline is flushed.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent represents the data pipeline that collects, decodes, processes and sends logs to the backend + ------------------------------------------------------ + | | | Collector -> Decoder -> Processor -> Sender -> Auditor | | | + ------------------------------------------------------ +

func NewAgent

func NewAgent(sources *config.LogSources, services *service.Services, processingRules []*config.ProcessingRule, endpoints *config.Endpoints) *Agent

NewAgent returns a new Logs Agent

func (*Agent) Start

func (a *Agent) Start()

Start starts all the elements of the data pipeline in the right order to prevent data loss

func (*Agent) Stop

func (a *Agent) Stop()

Stop stops all the elements of the data pipeline in the right order to prevent data loss

type Transport

type Transport string

Transport is the transport used by logs-agent, i.e TCP or HTTP

const (
	// TransportHTTP indicates logs-agent is using HTTP transport
	TransportHTTP Transport = "HTTP"
	// TransportTCP indicates logs-agent is using TCP transport
	TransportTCP Transport = "TCP"
)

var (

	// CurrentTransport is the current transport used by logs-agent, i.e TCP or HTTP
	CurrentTransport Transport
)