ilogtail

README

Alibaba iLogtail - The Lightweight Collector of SLS in Alibaba Cloud | 中文版本

iLogtail was born for observable scenarios and has many production-level features such as lightweight, high performance, and automated configuration, which are widely used internally by Alibaba Group and tens of thousands of external Alibaba Cloud customers. You can deploy it in physical machines, Kubernetes and other environments to collect telemetry data, such as logs, traces and metrics.

Abstract

The core advantages of iLogtail :

• Support a variety of Logs, Traces, Metrics data collection, and friendly to container and Kubernetes environment support.
• The resource cost of data collection is quite low, 5-20 times better than similar telemetry data collection Agent performance.
• High stability, used in the production of Alibaba and tens of thousands of Alibaba Cloud customers, and collecting dozens of petabytes of observable data every day with nearly tens of millions deployments.
• Support plugin expansion, such as collection, processing, aggregation, and sending modules.
• Support configuration remote management and provide a variety of ways, such as SLS console, SDK, K8s Operator, etc.
• Supports multiple advanced features such as self-monitoring, flow control, resource control, alarms, and statistics collection.

iLogtail supports the collection of a variety of telemetry data and transmission to a variety of different backends, such as SLS observable platform. The data supported for collection are mainly as follows:

• Logs
  • Collect static log files
  • Dynamic collect the files when running with containerized environment
  • Dynamic collect Stdout when running with containerized environment
• Traces
  • OpenTelemetry protocol
  • Skywalking V2 protocol
  • Skywalking V3 protocol
  • ...
• Metrics
  • Node metrics
  • Process metrics
  • Gpu metrics
  • Nginx metrics
  • Support fetch prometheus metrics
  • Support transfer telegraf metrics
  • ...

Quick Start

This repository is the golang part of iLogtail，it contains most of the features of iLogtail. And, it can work by itself or work with Ilogtail-C(open source soon) using CGO.

1. Start with local

make build && sh bin/ilogtail

NOTE: for some higher linux version, you have to install systemd-devel in advance

#centos
yum install systemd-devel

#ubuntu
apt-get update && apt-get install -y libsystemd-dev

1. Start with Alibaba Cloud
   Please read this doc.

Documentation

For documentation on the latest version see the documentation index

• Input Plugins
• Processor Plugins
• Aggregator Plugins
• Flusher Plugins

Contribution

There are many ways to contribute:

• Fix and report bugs
• Improve Documentation
• Review code and feature proposals
• Contribute plugins

Contact Us

You can report bugs, make suggestions or participate in discussions through Github Issues, or contact us with the following ways:

• DingTalk：iLogtail社区
• WeChat：日志服务
• Bilibili：阿里云SLS
• Zhihu：阿里云日志服务

Our Users

Tens of thousands of companies use iLogtail in Alibaba Cloud, IDC, or other clouds. More details please see here.

Licence

Apache 2.0 License

Documentation

Index

• Constants
• Variables
• func AddAggregatorCreator(name string, creator AggregatorCreator)
• func AddFlusherCreator(name string, creator FlusherCreator)
• func AddMetricCreator(name string, creator MetricCreator)
• func AddProcessorCreator(name string, creator ProcessorCreator)
• func AddServiceCreator(name string, creator ServiceCreator)
• type Aggregator
• type AggregatorCreator
• type Collector
• type CommonContext
• type Context
• type CounterMetric
• type Flusher
• type FlusherCreator
• type LatencyMetric
• type LogGroupQueue
• type MetricCreator
• type MetricInput
• type Processor
• type ProcessorCreator
• type ServiceCreator
• type ServiceInput
• type StringMetric

Constants

const (
	MetricInputType  = iota
	ServiceInputType = iota
	FilterType       = iota
	ProcessorType    = iota
	AggregatorType   = iota
)

logtail plugin type define

Variables

var Aggregators = map[string]AggregatorCreator{}

var Flushers = map[string]FlusherCreator{}

var MetricInputs = map[string]MetricCreator{}

var Processors = map[string]ProcessorCreator{}

var ServiceInputs = map[string]ServiceCreator{}

Functions

func AddAggregatorCreator

func AddAggregatorCreator(name string, creator AggregatorCreator)

func AddFlusherCreator

func AddFlusherCreator(name string, creator FlusherCreator)

func AddMetricCreator

func AddMetricCreator(name string, creator MetricCreator)

func AddProcessorCreator

func AddProcessorCreator(name string, creator ProcessorCreator)

func AddServiceCreator

func AddServiceCreator(name string, creator ServiceCreator)

Types

type Aggregator

type Aggregator interface {
	// Init called for init some system resources, like socket, mutex...
	// return flush interval(ms) and error flag, if interval is 0, use default interval
	Init(Context, LogGroupQueue) (int, error)

	// Description returns a one-sentence description on the Input.
	Description() string

	// Add the metric to the aggregator.
	Add(log *protocol.Log) error

	// Flush pushes the current aggregates to the accumulator.
	Flush() []*protocol.LogGroup

	// Reset resets the aggregators caches and aggregates.
	Reset()
}

Aggregator is an interface for implementing an Aggregator plugin. the RunningAggregator wraps this interface and guarantees that Add, Push, and Reset can not be called concurrently, so locking is not required when implementing an Aggregator plugin.

type AggregatorCreator

type AggregatorCreator func() Aggregator

type Collector

type Collector interface {
	AddData(tags map[string]string,
		fields map[string]string,
		t ...time.Time)

	AddDataArray(tags map[string]string,
		columns []string,
		values []string,
		t ...time.Time)

	AddRawLog(log *protocol.Log)
}

Collector ...

type CommonContext

type CommonContext struct {
	Project    string
	Logstore   string
	ConfigName string
}

type Context

type Context interface {
	InitContext(project, logstore, configName string)

	GetConfigName() string
	GetProject() string
	GetLogstore() string
	GetRuntimeContext() context.Context
	RegisterCounterMetric(metric CounterMetric)
	RegisterStringMetric(metric StringMetric)
	RegisterLatencyMetric(metric LatencyMetric)

	MetricSerializeToPB(log *protocol.Log)

	SaveCheckPoint(key string, value []byte) error
	GetCheckPoint(key string) (value []byte, exist bool)
	SaveCheckPointObject(key string, obj interface{}) error
	GetCheckPointObject(key string, obj interface{}) (exist bool)
}

Context for plugin

type CounterMetric

type CounterMetric interface {
	Name() string

	Add(v int64)

	// Clear same with set
	Clear(v int64)

	Get() int64

	Serialize(log *protocol.Log)
}

type Flusher

type Flusher interface {
	// Init called for init some system resources, like socket, mutex...
	Init(Context) error

	// Description returns a one-sentence description on the Input.
	Description() string

	// IsReady checks if flusher is ready to accept more data.
	// @projectName, @logstoreName, @logstoreKey: meta of the corresponding data.
	// Note: If SetUrgent is called, please make some adjustment so that IsReady
	//   can return true to accept more data in time and config instance can be
	//   stopped gracefully.
	IsReady(projectName string, logstoreName string, logstoreKey int64) bool

	// Flush flushes data to destination, such as SLS, console, file, etc.
	// It is expected to return no error at most time because IsReady will be called
	// before it to make sure there is space for next data.
	Flush(projectName string, logstoreName string, configName string, logGroupList []*protocol.LogGroup) error

	// SetUrgent indicates the flusher that it will be destroyed soon.
	// @flag indicates if main program (Logtail mostly) will exit after calling this.
	//
	// Note: there might be more data to flush after SetUrgent is called, and if flag
	//   is true, these data will be passed to flusher through IsReady/Flush before
	//   program exits.
	//
	// Recommendation: set some state flags in this method to guide the behavior
	//   of other methods.
	SetUrgent(flag bool)

	// Stop stops flusher and release resources.
	// It is time for flusher to do cleaning jobs, includes:
	// 1. Flush cached but not flushed data. For flushers that contain some kinds of
	//   aggregation or buffering, it is important to flush cached out now, otherwise
	//   data will lost.
	// 2. Release opened resources: goroutines, file handles, connections, etc.
	// 3. Maybe more, it depends.
	// In a word, flusher should only have things that can be recycled by GC after this.
	Stop() error
}

Flusher ... Sample flusher implementation: see plugin_manager/flusher_sls.gox.

type FlusherCreator

type FlusherCreator func() Flusher

type LatencyMetric

type LatencyMetric interface {
	Name() string

	Begin()

	Clear()

	End()
	// nano second
	Get() int64

	Serialize(log *protocol.Log)
}

type LogGroupQueue

type LogGroupQueue interface {
	// no blocking
	Add(loggroup *protocol.LogGroup) error
	AddWithWait(loggroup *protocol.LogGroup, duration time.Duration) error
}

LogGroupQueue for aggregator, Non blocked if aggregator's buffer is full, aggregator can add LogGroup to this queue return error if LogGroupQueue is full

type MetricCreator

type MetricCreator func() MetricInput

type MetricInput

type MetricInput interface {
	// Init called for init some system resources, like socket, mutex...
	// return call interval(ms) and error flag, if interval is 0, use default interval
	Init(Context) (int, error)

	// Description returns a one-sentence description on the Input
	Description() string

	// Collect takes in an accumulator and adds the metrics that the Input
	// gathers. This is called every "interval"
	Collect(Collector) error
}

MetricInput ...

type Processor

type Processor interface {
	// Init called for init some system resources, like socket, mutex...
	Init(Context) error

	// Description returns a one-sentence description on the Input
	Description() string

	// Apply the filter to the given metric
	ProcessLogs(logArray []*protocol.Log) []*protocol.Log
}

Processor also can be a filter

type ProcessorCreator

type ProcessorCreator func() Processor

type ServiceCreator

type ServiceCreator func() ServiceInput

type ServiceInput

type ServiceInput interface {
	// Init called for init some system resources, like socket, mutex...
	// return interval(ms) and error flag, if interval is 0, use default interval
	Init(Context) (int, error)

	// Description returns a one-sentence description on the Input
	Description() string

	// Start starts the ServiceInput's service, whatever that may be
	Start(Collector) error

	// Stop stops the services and closes any necessary channels and connections
	Stop() error
}

ServiceInput ...

type StringMetric

type StringMetric interface {
	Name() string

	Set(v string)

	Get() string

	Serialize(log *protocol.Log)
}