docker

README

Docker Input Plugin

The docker plugin uses the Docker Engine API to gather metrics on running docker containers.

The docker plugin uses the Official Docker Client to gather stats from the Engine API.

Configuration

# Read metrics about docker containers
[[inputs.docker]]
  ## Docker Endpoint
  ##   To use TCP, set endpoint = "tcp://[ip]:[port]"
  ##   To use environment variables (ie, docker-machine), set endpoint = "ENV"
  endpoint = "unix:///var/run/docker.sock"

  ## Set to true to collect Swarm metrics(desired_replicas, running_replicas)
  ## Note: configure this in one of the manager nodes in a Swarm cluster.
  ## configuring in multiple Swarm managers results in duplication of metrics.
  gather_services = false

  ## Only collect metrics for these containers. Values will be appended to
  ## container_name_include.
  ## Deprecated (1.4.0), use container_name_include
  container_names = []

  ## Set the source tag for the metrics to the container ID hostname, eg first 12 chars
  source_tag = false

  ## Containers to include and exclude. Collect all if empty. Globs accepted.
  container_name_include = []
  container_name_exclude = []

  ## Container states to include and exclude. Globs accepted.
  ## When empty only containers in the "running" state will be captured.
  ## example: container_state_include = ["created", "restarting", "running", "removing", "paused", "exited", "dead"]
  ## example: container_state_exclude = ["created", "restarting", "running", "removing", "paused", "exited", "dead"]
  # container_state_include = []
  # container_state_exclude = []

  ## Timeout for docker list, info, and stats commands
  timeout = "5s"

  ## Whether to report for each container per-device blkio (8:0, 8:1...),
  ## network (eth0, eth1, ...) and cpu (cpu0, cpu1, ...) stats or not.
  ## Usage of this setting is discouraged since it will be deprecated in favor of 'perdevice_include'.
  ## Default value is 'true' for backwards compatibility, please set it to 'false' so that 'perdevice_include' setting
  ## is honored.
  perdevice = true

  ## Specifies for which classes a per-device metric should be issued
  ## Possible values are 'cpu' (cpu0, cpu1, ...), 'blkio' (8:0, 8:1, ...) and 'network' (eth0, eth1, ...)
  ## Please note that this setting has no effect if 'perdevice' is set to 'true'
  # perdevice_include = ["cpu"]

  ## Whether to report for each container total blkio and network stats or not.
  ## Usage of this setting is discouraged since it will be deprecated in favor of 'total_include'.
  ## Default value is 'false' for backwards compatibility, please set it to 'true' so that 'total_include' setting
  ## is honored.
  total = false

  ## Specifies for which classes a total metric should be issued. Total is an aggregated of the 'perdevice' values.
  ## Possible values are 'cpu', 'blkio' and 'network'
  ## Total 'cpu' is reported directly by Docker daemon, and 'network' and 'blkio' totals are aggregated by this plugin.
  ## Please note that this setting has no effect if 'total' is set to 'false'
  # total_include = ["cpu", "blkio", "network"]

  ## docker labels to include and exclude as tags.  Globs accepted.
  ## Note that an empty array for both will include all labels as tags
  docker_label_include = []
  docker_label_exclude = []

  ## Which environment variables should we use as a tag
  tag_env = ["JAVA_HOME", "HEAP_SIZE"]

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Environment Configuration

When using the "ENV" endpoint, the connection is configured using the cli Docker environment variables.

Security

Giving telegraf access to the Docker daemon expands the attack surface that could result in an attacker gaining root access to a machine. This is especially relevant if the telegraf configuration can be changed by untrusted users.

Docker Daemon Permissions

Typically, telegraf must be given permission to access the docker daemon unix socket when using the default endpoint. This can be done by adding the telegraf unix user (created when installing a Telegraf package) to the docker unix group with the following command:

sudo usermod -aG docker telegraf

If telegraf is run within a container, the unix socket will need to be exposed within the telegraf container. This can be done in the docker CLI by add the option -v /var/run/docker.sock:/var/run/docker.sock or adding the following lines to the telegraf container definition in a docker compose file:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

source tag

Selecting the containers measurements can be tricky if you have many containers with the same name. To alleviate this issue you can set the below value to true

source_tag = true

This will cause all measurements to have the source tag be set to the first 12 characters of the container id. The first 12 characters is the common hostname for containers that have no explicit hostname set, as defined by docker.

Kubernetes Labels

Kubernetes may add many labels to your containers, if they are not needed you may prefer to exclude them:

  docker_label_exclude = ["annotation.kubernetes*"]

Docker-compose Labels

Docker-compose will add labels to your containers. You can limit restrict labels to selected ones, e.g.

  docker_label_include = [
    "com.docker.compose.config-hash",
    "com.docker.compose.container-number",
    "com.docker.compose.oneoff",
    "com.docker.compose.project",
    "com.docker.compose.service",
  ]

Metrics

• docker
  • tags:
    • unit
    • engine_host
    • server_version
  • fields:
    • n_used_file_descriptors
    • n_cpus
    • n_containers
    • n_containers_running
    • n_containers_stopped
    • n_containers_paused
    • n_images
    • n_goroutines
    • n_listener_events
    • memory_total
    • pool_blocksize (requires devicemapper storage driver) (deprecated see: docker_devicemapper)

The docker_data and docker_metadata measurements are available only for some storage drivers such as devicemapper.

• docker_data (deprecated see: docker_devicemapper)

  • tags:
    • unit
    • engine_host
    • server_version
  • fields:
    • available
    • total
    • used

• docker_metadata (deprecated see: docker_devicemapper)

  • tags:
    • unit
    • engine_host
    • server_version
  • fields:
    • available
    • total
    • used

The above measurements for the devicemapper storage driver can now be found in the new docker_devicemapper measurement

• docker_devicemapper

  • tags:
    • engine_host
    • server_version
    • pool_name
  • fields:
    • pool_blocksize_bytes
    • data_space_used_bytes
    • data_space_total_bytes
    • data_space_available_bytes
    • metadata_space_used_bytes
    • metadata_space_total_bytes
    • metadata_space_available_bytes
    • thin_pool_minimum_free_space_bytes

• docker_container_mem

  • tags:
    • engine_host
    • server_version
    • container_image
    • container_name
    • container_status
    • container_version
  • fields:
    • total_pgmajfault
    • cache
    • mapped_file
    • total_inactive_file
    • pgpgout
    • rss
    • total_mapped_file
    • writeback
    • unevictable
    • pgpgin
    • total_unevictable
    • pgmajfault
    • total_rss
    • total_rss_huge
    • total_writeback
    • total_inactive_anon
    • rss_huge
    • hierarchical_memory_limit
    • total_pgfault
    • total_active_file
    • active_anon
    • total_active_anon
    • total_pgpgout
    • total_cache
    • inactive_anon
    • active_file
    • pgfault
    • inactive_file
    • total_pgpgin
    • max_usage
    • usage
    • failcnt
    • limit
    • container_id

• docker_container_cpu

  • tags:
    • engine_host
    • server_version
    • container_image
    • container_name
    • container_status
    • container_version
    • cpu
  • fields:
    • throttling_periods
    • throttling_throttled_periods
    • throttling_throttled_time
    • usage_in_kernelmode
    • usage_in_usermode
    • usage_system
    • usage_total
    • usage_percent
    • container_id

• docker_container_net

  • tags:
    • engine_host
    • server_version
    • container_image
    • container_name
    • container_status
    • container_version
    • network
  • fields:
    • rx_dropped
    • rx_bytes
    • rx_errors
    • tx_packets
    • tx_dropped
    • rx_packets
    • tx_errors
    • tx_bytes
    • container_id

• docker_container_blkio

  • tags:
    • engine_host
    • server_version
    • container_image
    • container_name
    • container_status
    • container_version
    • device
  • fields:
    • io_service_bytes_recursive_async
    • io_service_bytes_recursive_read
    • io_service_bytes_recursive_sync
    • io_service_bytes_recursive_total
    • io_service_bytes_recursive_write
    • io_serviced_recursive_async
    • io_serviced_recursive_read
    • io_serviced_recursive_sync
    • io_serviced_recursive_total
    • io_serviced_recursive_write
    • container_id

The docker_container_health measurements report on a containers HEALTHCHECK status if configured.

• docker_container_health (container must use the HEALTHCHECK)

  • tags:
    • engine_host
    • server_version
    • container_image
    • container_name
    • container_status
    • container_version
  • fields:
    • health_status (string)
    • failing_streak (integer)

• docker_container_status

  • tags:
    • engine_host
    • server_version
    • container_image
    • container_name
    • container_status
    • container_version
  • fields:
    • container_id
    • oomkilled (boolean)
    • pid (integer)
    • exitcode (integer)
    • started_at (integer)
    • finished_at (integer)
    • uptime_ns (integer)

• docker_swarm

  • tags:
    • service_id
    • service_name
    • service_mode
  • fields:
    • tasks_desired
    • tasks_running

Example Output

docker,engine_host=debian-stretch-docker,server_version=17.09.0-ce n_containers=6i,n_containers_paused=0i,n_containers_running=1i,n_containers_stopped=5i,n_cpus=2i,n_goroutines=41i,n_images=2i,n_listener_events=0i,n_used_file_descriptors=27i 1524002041000000000
docker,engine_host=debian-stretch-docker,server_version=17.09.0-ce,unit=bytes memory_total=2101661696i 1524002041000000000
docker_container_mem,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,engine_host=debian-stretch-docker,server_version=17.09.0-ce active_anon=8327168i,active_file=2314240i,cache=27402240i,container_id="adc4ba9593871bf2ab95f3ffde70d1b638b897bb225d21c2c9c84226a10a8cf4",hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=25088000i,limit=2101661696i,mapped_file=20582400i,max_usage=36646912i,pgfault=4193i,pgmajfault=214i,pgpgin=9243i,pgpgout=520i,rss=8327168i,rss_huge=0i,total_active_anon=8327168i,total_active_file=2314240i,total_cache=27402240i,total_inactive_anon=0i,total_inactive_file=25088000i,total_mapped_file=20582400i,total_pgfault=4193i,total_pgmajfault=214i,total_pgpgin=9243i,total_pgpgout=520i,total_rss=8327168i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=36528128i,usage_percent=0.4342225020025297,writeback=0i 1524002042000000000
docker_container_cpu,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,cpu=cpu-total,engine_host=debian-stretch-docker,server_version=17.09.0-ce container_id="adc4ba9593871bf2ab95f3ffde70d1b638b897bb225d21c2c9c84226a10a8cf4",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=40000000i,usage_in_usermode=100000000i,usage_percent=0,usage_system=6394210000000i,usage_total=117319068i 1524002042000000000
docker_container_cpu,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,cpu=cpu0,engine_host=debian-stretch-docker,server_version=17.09.0-ce container_id="adc4ba9593871bf2ab95f3ffde70d1b638b897bb225d21c2c9c84226a10a8cf4",usage_total=20825265i 1524002042000000000
docker_container_cpu,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,cpu=cpu1,engine_host=debian-stretch-docker,server_version=17.09.0-ce container_id="adc4ba9593871bf2ab95f3ffde70d1b638b897bb225d21c2c9c84226a10a8cf4",usage_total=96493803i 1524002042000000000
docker_container_net,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,engine_host=debian-stretch-docker,network=eth0,server_version=17.09.0-ce container_id="adc4ba9593871bf2ab95f3ffde70d1b638b897bb225d21c2c9c84226a10a8cf4",rx_bytes=1576i,rx_dropped=0i,rx_errors=0i,rx_packets=20i,tx_bytes=0i,tx_dropped=0i,tx_errors=0i,tx_packets=0i 1524002042000000000
docker_container_blkio,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,device=254:0,engine_host=debian-stretch-docker,server_version=17.09.0-ce container_id="adc4ba9593871bf2ab95f3ffde70d1b638b897bb225d21c2c9c84226a10a8cf4",io_service_bytes_recursive_async=27398144i,io_service_bytes_recursive_read=27398144i,io_service_bytes_recursive_sync=0i,io_service_bytes_recursive_total=27398144i,io_service_bytes_recursive_write=0i,io_serviced_recursive_async=529i,io_serviced_recursive_read=529i,io_serviced_recursive_sync=0i,io_serviced_recursive_total=529i,io_serviced_recursive_write=0i 1524002042000000000
docker_container_health,container_image=telegraf,container_name=zen_ritchie,container_status=running,container_version=unknown,engine_host=debian-stretch-docker,server_version=17.09.0-ce failing_streak=0i,health_status="healthy" 1524007529000000000
docker_swarm,service_id=xaup2o9krw36j2dy1mjx1arjw,service_mode=replicated,service_name=test tasks_desired=3,tasks_running=3 1508968160000000000

Documentation

Overview

Package docker contains few helper functions copied from https://github.com/docker/cli/blob/master/cli/command/container/stats_helpers.go

Index

• Constants
• Variables
• func CalculateCPUPercentUnix(previousCPU, previousSystem uint64, v *types.StatsJSON) float64
• func CalculateMemPercentUnixNoCache(limit float64, usedNoCache float64) float64
• func CalculateMemUsageUnixNoCache(mem types.MemoryStats) float64
• type Client
  • func NewClient(host string, tlsConfig *tls.Config) (Client, error)
  • func NewEnvClient() (Client, error)
• type Docker
  • func (d *Docker) Gather(acc telegraf.Accumulator) error
  • func (d *Docker) Init() error
  • func (*Docker) SampleConfig() string
• type SocketClient
  • func (c *SocketClient) Close() error
  • func (c *SocketClient) ContainerInspect(ctx context.Context, containerID string) (types.ContainerJSON, error)
  • func (c *SocketClient) ContainerList(ctx context.Context, options types.ContainerListOptions) ([]types.Container, error)
  • func (c *SocketClient) ContainerStats(ctx context.Context, containerID string, stream bool) (types.ContainerStats, error)
  • func (c *SocketClient) Info(ctx context.Context) (types.Info, error)
  • func (c *SocketClient) NodeList(ctx context.Context, options types.NodeListOptions) ([]swarm.Node, error)
  • func (c *SocketClient) ServiceList(ctx context.Context, options types.ServiceListOptions) ([]swarm.Service, error)
  • func (c *SocketClient) TaskList(ctx context.Context, options types.TaskListOptions) ([]swarm.Task, error)

Constants

const (
	KB = 1000
	MB = 1000 * KB
	GB = 1000 * MB
	TB = 1000 * GB
	PB = 1000 * TB
)

KB, MB, GB, TB, PB...human friendly

Variables

var NodeList = []swarm.Node{
	{
		ID: "0cl4jturcyd1ks3fwpd010kor",
		Status: swarm.NodeStatus{
			State: "ready",
		},
	},
	{
		ID: "0cl4jturcyd1ks3fwpd010kor",
		Status: swarm.NodeStatus{
			State: "ready",
		},
	},
}

var ServiceList = []swarm.Service{
	{
		ID: "qolkls9g5iasdiuihcyz9rnx2",
		Spec: swarm.ServiceSpec{
			Annotations: swarm.Annotations{
				Name: "test1",
			},
			Mode: swarm.ServiceMode{
				Replicated: &swarm.ReplicatedService{
					Replicas: &two,
				},
			},
		},
	},
	{
		ID: "qolkls9g5iasdiuihcyz9rn3",
		Spec: swarm.ServiceSpec{
			Annotations: swarm.Annotations{
				Name: "test2",
			},
			Mode: swarm.ServiceMode{
				Global: &swarm.GlobalService{},
			},
		},
	},
}

var TaskList = []swarm.Task{
	{
		ID:        "kwh0lv7hwwbh",
		ServiceID: "qolkls9g5iasdiuihcyz9rnx2",
		NodeID:    "0cl4jturcyd1ks3fwpd010kor",
		Status: swarm.TaskStatus{
			State: "running",
		},
		DesiredState: "running",
	},
	{
		ID:        "u78m5ojbivc3",
		ServiceID: "qolkls9g5iasdiuihcyz9rnx2",
		NodeID:    "0cl4jturcyd1ks3fwpd010kor",
		Status: swarm.TaskStatus{
			State: "running",
		},
		DesiredState: "running",
	},
	{
		ID:        "1n1uilkhr98l",
		ServiceID: "qolkls9g5iasdiuihcyz9rn3",
		NodeID:    "0cl4jturcyd1ks3fwpd010kor",
		Status: swarm.TaskStatus{
			State: "running",
		},
		DesiredState: "running",
	},
}

Functions

func CalculateCPUPercentUnix

func CalculateCPUPercentUnix(previousCPU, previousSystem uint64, v *types.StatsJSON) float64

func CalculateMemPercentUnixNoCache

func CalculateMemPercentUnixNoCache(limit float64, usedNoCache float64) float64

func CalculateMemUsageUnixNoCache

func CalculateMemUsageUnixNoCache(mem types.MemoryStats) float64

CalculateMemUsageUnixNoCache calculate memory usage of the container. Cache is intentionally excluded to avoid misinterpretation of the output.

On Docker 19.03 and older, the result is `mem.Usage - mem.Stats["cache"]`. On new docker with cgroup v1 host, the result is `mem.Usage - mem.Stats["total_inactive_file"]`. On new docker with cgroup v2 host, the result is `mem.Usage - mem.Stats["inactive_file"]`.

This definition is designed to be consistent with past values and the latest docker CLI * https://github.com/docker/cli/blob/6e2838e18645e06f3e4b6c5143898ccc44063e3b/cli/command/container/stats_helpers.go#L239

Types

type Client

type Client interface {
	Info(ctx context.Context) (types.Info, error)
	ContainerList(ctx context.Context, options types.ContainerListOptions) ([]types.Container, error)
	ContainerStats(ctx context.Context, containerID string, stream bool) (types.ContainerStats, error)
	ContainerInspect(ctx context.Context, containerID string) (types.ContainerJSON, error)
	ServiceList(ctx context.Context, options types.ServiceListOptions) ([]swarm.Service, error)
	TaskList(ctx context.Context, options types.TaskListOptions) ([]swarm.Task, error)
	NodeList(ctx context.Context, options types.NodeListOptions) ([]swarm.Node, error)
	Close() error
}

func NewClient

func NewClient(host string, tlsConfig *tls.Config) (Client, error)

func NewEnvClient

func NewEnvClient() (Client, error)

type Docker

type Docker struct {
	Endpoint       string
	ContainerNames []string `toml:"container_names" deprecated:"1.4.0;use 'container_name_include' instead"`

	GatherServices bool `toml:"gather_services"`

	Timeout          config.Duration
	PerDevice        bool     `toml:"perdevice" deprecated:"1.18.0;use 'perdevice_include' instead"`
	PerDeviceInclude []string `toml:"perdevice_include"`
	Total            bool     `toml:"total" deprecated:"1.18.0;use 'total_include' instead"`
	TotalInclude     []string `toml:"total_include"`
	TagEnvironment   []string `toml:"tag_env"`
	LabelInclude     []string `toml:"docker_label_include"`
	LabelExclude     []string `toml:"docker_label_exclude"`

	ContainerInclude []string `toml:"container_name_include"`
	ContainerExclude []string `toml:"container_name_exclude"`

	ContainerStateInclude []string `toml:"container_state_include"`
	ContainerStateExclude []string `toml:"container_state_exclude"`

	IncludeSourceTag bool `toml:"source_tag"`

	Log telegraf.Logger

	tlsint.ClientConfig
	// contains filtered or unexported fields
}

Docker object

func (*Docker) Gather

func (d *Docker) Gather(acc telegraf.Accumulator) error

Gather metrics from the docker server.

func (*Docker) Init

func (d *Docker) Init() error

func (*Docker) SampleConfig

func (*Docker) SampleConfig() string

type SocketClient

type SocketClient struct {
	// contains filtered or unexported fields
}

func (*SocketClient) Close

func (c *SocketClient) Close() error

func (*SocketClient) ContainerInspect

func (c *SocketClient) ContainerInspect(ctx context.Context, containerID string) (types.ContainerJSON, error)

func (*SocketClient) ContainerList

func (c *SocketClient) ContainerList(ctx context.Context, options types.ContainerListOptions) ([]types.Container, error)

func (*SocketClient) ContainerStats

func (c *SocketClient) ContainerStats(ctx context.Context, containerID string, stream bool) (types.ContainerStats, error)

func (*SocketClient) Info

func (c *SocketClient) Info(ctx context.Context) (types.Info, error)

func (*SocketClient) NodeList

func (c *SocketClient) NodeList(ctx context.Context, options types.NodeListOptions) ([]swarm.Node, error)

func (*SocketClient) ServiceList

func (c *SocketClient) ServiceList(ctx context.Context, options types.ServiceListOptions) ([]swarm.Service, error)

func (*SocketClient) TaskList

func (c *SocketClient) TaskList(ctx context.Context, options types.TaskListOptions) ([]swarm.Task, error)