k8s

README

Kubernetes plugin

It reads Kubernetes logs and also adds pod meta-information. Also, it joins split logs into a single event.

Source log file should be named in the following format:
[pod-name]_[namespace]_[container-name]-[container-id].log

E.g. my_pod-1566485760-trtrq_my-namespace_my-container-4e0301b633eaa2bfdcafdeba59ba0c72a3815911a6a820bf273534b0f32d98e0.log

An information which plugin adds:

• k8s_node – node name where pod is running;
• k8s_node_label_* – node labels;
• k8s_pod – pod name;
• k8s_namespace – pod namespace name;
• k8s_container – pod container name;
• k8s_label_* – pod labels.

Example:

pipelines:
  example_k8s_pipeline:
    input:
      type: k8s
      offsets_file: /data/offsets.yaml
      file_config:                        // customize file plugin
        persistence_mode: sync
        read_buffer_size: 2048

Config params

split_event_size int default=1000000

Docker splits long logs by 16kb chunks. The plugin joins them back, but if an event is longer than this value in bytes, it will be split after all.

Due to the optimization process it's not a strict rule. Events may be split even if they won't exceed the limit.

allowed_pod_labels []string

If set, it defines which pod labels to add to the event, others will be ignored.

allowed_node_labels []string

If set, it defines which node labels to add to the event, others will be ignored.

only_node bool default=false

Skips retrieving Kubernetes meta information using Kubernetes API and adds only k8s_node field.

watching_dir string default=/var/log/containers

Kubernetes dir with container logs. It's like watching_dir parameter from file plugin config.

offsets_file string required

The filename to store offsets of processed files. It's like offsets_file parameter from file plugin config.

file_config file.Config

Under the hood this plugin uses file plugin to collect logs from files. So you can change any file plugin config parameter using file_config section. Check out an example.

Generated using insane-doc

Documentation

Index

• Variables
• func Factory() (pipeline.AnyPlugin, pipeline.AnyConfig)
• func MultilineActionFactory() (pipeline.AnyPlugin, pipeline.AnyConfig)
• type Config
• type MultilineAction
  • func (p *MultilineAction) Do(event *pipeline.Event) pipeline.ActionResult
  • func (p *MultilineAction) Start(config pipeline.AnyConfig, params *pipeline.ActionPluginParams)
  • func (p *MultilineAction) Stop()
• type Plugin
  • func (p *Plugin) Commit(event *pipeline.Event)
  • func (p *Plugin) Start(config pipeline.AnyConfig, params *pipeline.InputPluginParams)
  • func (p *Plugin) Stop()

Variables

var (
	MaintenanceInterval = 15 * time.Second

	MetaWaitTimeout = 60 * time.Second

	// some debugging shit
	DisableMetaUpdates = false
)

Functions

func Factory

func Factory() (pipeline.AnyPlugin, pipeline.AnyConfig)

func MultilineActionFactory

func MultilineActionFactory() (pipeline.AnyPlugin, pipeline.AnyConfig)

Types

type Config

type Config struct {

	//> @3@4@5@6
	//>
	//> Docker splits long logs by 16kb chunks. The plugin joins them back, but if an event is longer than this value in bytes, it will be split after all.
	//> > Due to the optimization process it's not a strict rule. Events may be split even if they won't exceed the limit.
	SplitEventSize int `json:"split_event_size" default:"1000000"` //*

	//> @3@4@5@6
	//>
	//> If set, it defines which pod labels to add to the event, others will be ignored.
	AllowedPodLabels  []string `json:"allowed_pod_labels" slice:"true"` //*
	AllowedPodLabels_ map[string]bool

	//> @3@4@5@6
	//>
	//> If set, it defines which node labels to add to the event, others will be ignored.
	AllowedNodeLabels  []string `json:"allowed_node_labels" slice:"true"` //*
	AllowedNodeLabels_ map[string]bool

	//> @3@4@5@6
	//>
	//> Skips retrieving Kubernetes meta information using Kubernetes API and adds only `k8s_node` field.
	OnlyNode bool `json:"only_node" default:"false"` //*

	//> @3@4@5@6
	//>
	//> Kubernetes dir with container logs. It's like `watching_dir` parameter from [file plugin](/plugin/input/file/README.md) config.
	WatchingDir  string `json:"watching_dir" default:"/var/log/containers"` //*
	WatchingDir_ string

	//> @3@4@5@6
	//>
	//> The filename to store offsets of processed files. It's like `offsets_file` parameter from [file plugin](/plugin/input/file/README.md) config.
	OffsetsFile string `json:"offsets_file" required:"true"` //*

	//> @3@4@5@6
	//>
	//> Under the hood this plugin uses [file plugin](/plugin/input/file/README.md) to collect logs from files. So you can change any [file plugin](/plugin/input/file/README.md) config parameter using `file_config` section. Check out an example.
	FileConfig file.Config `json:"file_config" child:"true"` //*
}

type MultilineAction

type MultilineAction struct {
	// contains filtered or unexported fields
}

func (*MultilineAction) Do

func (p *MultilineAction) Do(event *pipeline.Event) pipeline.ActionResult

func (*MultilineAction) Start

func (p *MultilineAction) Start(config pipeline.AnyConfig, params *pipeline.ActionPluginParams)

func (*MultilineAction) Stop

func (p *MultilineAction) Stop()

type Plugin

type Plugin struct {
	// contains filtered or unexported fields
}

func (*Plugin) Commit

func (p *Plugin) Commit(event *pipeline.Event)

func (*Plugin) Start

func (p *Plugin) Start(config pipeline.AnyConfig, params *pipeline.InputPluginParams)

func (*Plugin) Stop

func (p *Plugin) Stop()