docker-alertd

README

Docker-Alertd

What Does It Do?

docker-alertd monitors docker containers on a host machine and sends alerts via email when usage limits have been breached.

Currently, alerts can be sent based on:

1. Container existence (regardless of running state)
2. Running state (running or existed)
3. Memory usage (in MB)
4. CPU Usage (as a percentage)
5. Minimum Process running in container

Step 1: Install

Method: Build from source

Assuming that you already have go installed on your machine, you can just go get it.

go get github.com/deltaskelta/docker-alertd

If you would like to download a pre-compiled binary, head to the release downloads

latest release - precompiled binary download

Step 2: Make a Configuration File

Docker-Alertd needs a configuration file in order to run, it will search the directory which contains the binary, and the home directory for a config file automatically. A special config file location can be specified with the --config flag.

A base config file can be generated with the command:

docker-alertd initconfig -d /path/to/config/location/

Example .docker-alertd.yaml file

---
# The duration and interations settings, if omitted, have a default value of 100ms between
# docker API calls and an indefinite number of iterations which will run the monitor forever
#duration: 100				# duration in ms between docker API calls
#iterations: 0				# number of iterations to run

# 'containers' is an array of dictionaries that each contain the name of a container to
# monitor, and the metrics which it should be monitored by. If there are no metrics
# present, then it will just be monitored to make sure that is is currently up.

# This will monitor only that the container exists, running or not...
# containers:
#   - name: mycontainer

containers:
  - name: container1
    expectedRunning: true

  - name: container2
    expectedRunning: true
    maxCpu: 20
    maxMem: 20
    minProcs: 4

# If email settings are present and active, then email alerts will be sent when an alert
# is triggered.
emailSettings:
  active: true
  smtp: smtp.someserver.com
  password: s00p3rS33cret
  port: 587
  from: auto@freshpowpow.com
  subject: "DOCKER_ALERTD"
  to:
    - jeff@gnarfresh.com

Configuration Variables

Containers

duration: the duration to wait between docker engine API calls.

iterations: the number of iterations that docker-alertd should run (0 = run forever)

name: the container name or ID

maxCpu: the maximum cpu usage threshold (as a percentage), if the container uses more CPU, an alert will be triggered.

maxMem: the maximum memory usage threshold (in MB). If the container uses more system memory than this, an alert will be triggered.

minProcs: the minimum number of running processes (PID's) in the container. If a the number of running processes dips below this level (when a process fails), an alert will be triggered.

Email Settings

active: whether email settings are active or not

smtp: the smtp server to connect to

password: the password to use for smtp authentication

port: the port to connect to the smtp server

from: the email address to send from

subject: the subject line of emails sent

to: an array of email addresses to send the alerts to

Slack Settings

webhookURL: the webhookURL provided by slack after you authorize an app on a slack channel. See slack apps

Step 3: Run the program

/path/to/binary/docker-alertd --config ~/path/to/configuration/file/.docker-alertd.yaml

This will start the program and log the output to stdout. It can be stopped with CTRL-C.

Example Output:

2017/09/18 12:11:44 starting docker-alertd
------------------------------
2017/09/18 12:11:44 ALERT:
2017/09/18 12:11:44 container1: Existence check failure: Error: No such container: container1
2017/09/18 12:11:44 container2: Existence check failure: Error: No such container: container2
...

Step 4. Set up as a background process (optional)

If you wish to have docker-alertd run as a background process, it needs to be setup as a background process as per your operating system.

As A Systemd Service (for Linux systems with systemd)

If you have a systemd based system then you can refer to docker-alertd.service.example the example systemd service file and this tutorial

With Launchd (MacOS)

Refer to the launchd plist example file file and the launchd reference

With Sys V Init (various Linux systems without systemd)

Refer to this Sys V Init tutorial

Testing Alert Authentication

Docker-Alertd comes with a testalert command which will search for a nonexistant container name and send an alert to the appropriate places, use this for testing that smtp or other authentication settings are correct