ayd

README

Ayd - The easiest alive monitoring tool. You can start monitoring if your system live or not, by only one easy command like below.

$ ayd ping:192.168.1.1 https://example.com

Features

• Check service status using:

  • HTTP/HTTPS
  • FTP/FTPS
  • ICMP echo (ping)
  • TCP connect
  • DNS resolve
  • file/directory existance
  • execute external command (or script file)
  • plugin

• The status page for browser, console, or programs.

• Send alert if target failure or recovered.

Good at

• Make a status page for temporary usage.

  You can start it by only one command! And, stop it via just Ctrl-C!

• Make a status page for a small system.

  Single binary server, single log file, there is no database!

Not good at

• Complex customization, extension.

  There is a few extension way, but extensibility is not the goal of this project.

• Investigate more detail.

  This program is just for check your service is dead or alive.

Quick start

1. Download the latest version from release page.

2. Extract downloaded package and put to somewhere that registered to PATH.

3. Run the server with target URLs (and schedule if need) as arguments.

$ ayd https://your-service.example.com ping:another-host.example.com

1. Check your service status.

You can see the status page on http://localhost:9000/, and you can use HTTP APIs.

Reference

Ayd check the target specified as URLs if alive or not, and report to the alert target that specified as URLs if status changed. A command to start Ayd looks like below.

$ ayd  -a exec:/path/to/alert.sh  10m  ping:192.168.1.1 http://example.com
      └────────────┬────────────┘└─┬─┘└───────────────┬───────────────────┘
                   │               │              Target URLs
                   │               │   Check if 192.168.1.1 is responding to ping,
                   │               │   and if http://example.com is serving.
                   │               │
                   │            Schedule
                   │        Check targets every 10 minutes.
                   │
               Alert URL
       Execute /path/to/alert.sh if the target status changed.

The common schemes for target URL or alrt URL, are supported by Ayd itself. You can also add other schemes using plugin.

Ayd checks the target every 5 minutes in default, but you can change it by placing schedule specification before target URL.

While Ayd running, it provides simple dashboard and some APIs you can check the status. Also, the log file is formatted as JSON, so you can read it or aggregate it easily using common tools like jq.

• URL scheme
• Scheduling
• Status page and endpoints
• Log file
• Text encoding
• Tips
  • Daemonize
  • Change listen port
  • Use HTTPS status page on status page
  • Use Basic Authentication on status page
  • One-shot mode

URL scheme

Ayd supports below URL schemes in default.

scheme	as Target	as Alert
http: / https:	✔	✔
ftp: / ftps:	✔	➖
ping:	✔	➖
tcp:	✔	➖
dns:	✔	➖
file:	✔	✔
exec:	✔	✔
source:	✔	✔

You can use extra scheme with plugin if you want to.

http: / https:

Fetch HTTP/HTTPS page and check if the status code is 2xx or not.

You can use GET, HEAD, POST, OPTIONS, and CONNECT method by specifying like http-post://... or https-head://.... The default method is GET.

Ayd will Follow redirect maximum 10 times.

HTTP will timeout in 10 minutes and report as failure.

examples:

• http://example.com/
• https://example.com/
• http-head://example.com/path/to/somewhere
• https-options://example.com/abc?def=ghi

as Alert

If you use HTTP/HTTP as an alert URL, Ayd adds some queries to send information about the incident.

query name	example	description
ayd_time	2001-02-03T16:05:06+09:00	The timestamp when status changed
ayd_status	FAILURE, DEGRADE, UNKNOWN, HEALTHY	The current status of the target
ayd_latency	123.456	The latency of the latest checking
ayd_target	https://target.example.com	The target URL
ayd_message	hello world	The latest message of the target
ayd_extra	{"hello":"world"}	The Extra values in JSON format

ftp: / ftps:

Send LIST or MLSD command of FTP and check the result.

It uses anonymous as username and password if absent those in the URL.

FTP will timeout in 10 minutes and report as failure.

examples:

• ftp://example.com/
• ftp://example.com/path/to/directory
• ftps://foo:bar@example.com/path/to/file.txt

as Alert

FTP/FTPS does not support to used as an alert URL.

ping:

Send ICMP echo request (a.k.a. ping command) and check if the target is connected or not.

Ayd sends 3 packets in 1 second and expects all packets to return. These parameter can changed by AYD_PING_PACKETS and AYD_PING_PERIOD environment variable.

In Linux or MacOS, Ayd use non-privileged ICMP in default. So, you can use ping even if rootless. But this way is not work on some platforms for instance docker container. Please set yes to AYD_PRIVILEGED environment variable to use privileged ICMP.

You can specify IPv4 or IPv6 with ping4: or ping6: scheme.

Ping will timeout in 30 seconds after sent all packets and report as failure.

examples:

• ping:example.com
• ping:192.168.1.1
• ping:192.168.1.10#my-server

as Alert

ping does not support to used as an alert URL.

tcp:

Connect to TCP and check if the service is listening or not.

tcp:// will select IPv4 or IPv6 automatically. But you can also use tcp4:// or tcp6:// to choose IP protocol version.

TCP will timeout in 10 seconds and report as failure.

examples:

• tcp://example.com:3309
• tcp4://127.0.0.1:3309
• tcp6://[::1]:3309
• tcp://192.168.1.10:80#my-server

as Alert

TCP does not support to used as an alert URL.

dns:

Resolve hostname via DNS and check if the host exists or not.

You can specify record type as a type query like dns:example.com?type=A, or as a scheme like dns-cname:example.com. Supported type is A, AAAA, CNAME, MX, NS, and TXT.

There are dns4: and dns6: scheme for shorthand of A and AAAA type.

You also can specify the DNS server as a host name of URL like dns://8.8.8.8/example.com.

DNS will timeout in 10 seconds and report as failure.

examples:

• dns:example.com
• dns4:example.com
• dns:example.com?type=AAAA
• dns-cname:example.com
• dns://8.8.8.8/example.com

as Alert

DNS does not support to used as an alert URL.

file:

Check the file or the directory existence. It only checks existence, so it does not report error even if it has no enough permission to read the target.

examples:

• file:./path/to/something
• file:/path/to/somewhere
• file:/dev/sdc1#does-storage-connected?

as Alert

If you use this as an alert target, it writes logs that the same as normal log to the target path, but only logs when status changed.

exec:

Execute external command and check if the exit code is 0 or not. The exit code 0 means HEALTHY, otherwise mean FAILURE. If couldn't execute command, Ayd reports as UNKNOWN.

The command's stdout and stderr will be captured as a message of the status check record. It is recommended to keep output as short as possible for log readability reason.

You can specify the first argument as the fragment of URL like below.

exec:/path/to/command#this-is-argument

Above target URL works like below command in the shell. (In accurate, Ayd does not use shell to execute command so you can not use some features for example shell variable.)

$ /path/to/command this-is-argument

And, you can specify environment arguments as the query of URL like below.

exec:/path/to/command?something=foobar&hello=world

Above target URL works like below command in the shell.

$ export something=foobar
$ export hello=world
$ /path/to/command

Exec will timeout in 1 hour and report as failure.

examples:

• exec:./check.exe
• exec:/usr/local/bin/check.sh

Extra report format

exec: commands can report latency of service, or status of service in the output of the command. Please write output like below if you want to use this function.

::latency::123.456
::status::failure
hello world

This output is reporting latency is 123.456ms, status is FAILURE, and message is hello world.

• ::latency::: Reports the latency of service in milliseconds.
• ::status::: Reports the status of service in healthy, degrade, failure, aborted, or unknown.

Ayd uses the last value if found multiple reports in single output.

as Alert

If you use exec: as an alert URL, Ayd sets some environment variables about the incident. The name of variable and meaning is the same as the queries of HTTP scheme as alert.

source:

This is a special scheme for loading targets from a file, a remote host, or a command. Load each line in the file as a target URL and check all targets.

Source file looks like below.

# servers
ping:somehost.example.com
ping:anotherhost.example.com
ping:yet.anotherhost.example.com

# services
https://service1.example.com
https://service2.example.com

# you can also read another file
source:./another-list.txt

The line that starts with # will ignored as a comment.

Source file should encoded by UTF-8 with/without BOM or UTF-16 with BOM, but in Windows, you can use legacy encoding. Please see also text encoding chapter.

examples:

• source:./targets.txt
• source:/path/to/targets.txt

source+http: / source+https:

source+http: and source+https: is variants of source:. These fetch HTTP/HTTPS and load the response as a source file.

WARNING: Please don't use it if you can't completely trust the HTTP server because this scheme can be a backdoor of your server. For example, Ayd will execute everything even if HTTP server responses exec:rm#/your/important/directory

examples:

• source+https://example.com/targets.txt

source+ftp: / source+ftps:

source+ftp: and source+ftps: is variants of source: that very similar to source+http:. These download source file via FTP/FTPS and load it.

WARNING: Please don't use it if you can't completely trust the source file in the FTP server because this scheme can be a backdoor of your server. For example, Ayd will execute everything even if the FTP server responses exec:rm#/your/important/directory

examples:

• source+ftps://example.com/targets.txt

source+exec:

source+exec: is another variant of source:. It is execute script as the same way as exec: and load the output as a source file.

examples:

• source+exec:./make-targets-list.exe
• source+exec:/usr/local/bin/targets.sh

as Alert

Even if use it as an alert URL, the behavior is almost the same, but send alert to the all URLs loaded.

Plugin

A plugin is a executable file named like ayd-xxx-probe or ayd-xxx-alert, installed to the PATH directory.

Ayd looks for ayd-xxx-probe for a target or ayd-xxx-alert for an alert, if URL scheme is xxx:, xxx-yyy:, or xxx+yyy:. You can change scheme via changing xxx, but you can't use ayd, alert, and the scheme that is supported by Ayd itself. And you can use yyy part to change plugin behavior, the same as http: or dns:.

The longest plugin name has priority if you installed multiple plugins. For example, ayd-xxx-yyy-probe has high priority than ayd-xxx-probe.

The plugin prints result to stdout, in the same format as log file.

Ayd expects UTF-8 text as outputs of plugins. But in Windows, you can use system's default character encoding. Please see also text encoding chapter.

Execution of a plugin will timeout in maximum 1 hour and report as failure.

The differences from plugin to exec: are below.

	exec:	plugin
URL Scheme	exec: only	anything
executable file directory	anywhere	only in the PATH directory
set argument and environment variable in URL	can	can not
receive raw target URL	can not	can
record about multiple targets like as source	can not	can

There is a library for create plugin.

Probe plugin

Probe plugin which to check the target, receives the target URL as the only one argument of the command.

For example, target URL foobar:your-target has the same mean as below command.

$ ayd-foobar-probe "foobar:your-target"

Alert plugin

Alert plugin which to send alerts, receives below arguments.

1. Alert URL.
2. Timestamp in RFC3339 format.
3. Current status of the target.
4. Current latency in milliseconds.
5. Target URL that causes the incident.
6. Message from the latest probe.
7. Extra values in JSON format.

For example, the alert URL foobar:your-alert for plugin ayd-foobar-alert will be called like a below command.

$ ayd-foobar-alert                      \
    "foobar:your-alert"                 \
    "2001-02-30T16:05:06+09:00"         \
    "FAILURE"                           \
    "1.234"                             \
    "ping:your-target"                  \
    "this is message of the record"     \
    '{"extra values":"in json format"}'

The output of the probe plugin will parsed the same way to log file, but all target URL will add alert: prefix and won't not show in status page.

plugin list

Probe plugin

• SMB (samba)
• NTP

Alert plugin

• e-mail (SMTP)

  

• Slack

  

Scheduling

Ayd will check targets every 5 minutes in default. You can place the schedule specifications before the target URLs like below if you want to change scheduling.

$ ayd 10m https://your-service.example.com \
      1h  https://another-service.example.com https://yet-another-service.example.com

The above command means to check your-service.example.com every 10 minutes, and to check another-service.example.com and yet-another-service.example.com every 1 hour.

You can also use the Cron style spec as a schedule spec like below.

$ ayd '*/5  6-21 * *'     https://your-service.example.com \
      '*/10 *    * * 1-5' https://another-service.example.com

The above command means checking your-service.example.com every 5 minutes from 6 a.m. to 9 p.m, and checking another-service.example.com every 10 minutes from monday to friday.

 ┌─────── minute (0 - 59)
 │ ┌────── hour (0 - 23)
 │ │ ┌───── day of the month (1 - 31)
 │ │ │ ┌──── month (1 - 12)
 │ │ │ │ ┌─── [optional] day of the week (0 - 6 (sunday - saturday))
 │ │ │ │ │
'* * * * *'

Status page and endpoints

Ayd has these pages/endpoints.

path	description
/status.html	Human friendly status page in HTML.
/status.txt	Human friendly status page in plain text.
/status.json	Machine readable status page in JSON format.
/incidents.html	Human friendly incident history page in HTML.
/incidents.rss	Incident history feed in RSS 2.0 format.
/incidents.csv	Incident history in CSV format.
/incidents.json	Incident history in JSON format.
/log.html	Raw log data in HTML page.
/log.csv	Raw log file in CSV format.
/log.json	Raw log file in JSON format.
/targets.txt	The list of target URLs, separated by \n.
/targets.json	The list of target URLs in JSON format.
/metrics	Minimal status page for use by Prometheus.
/healthz	Health status page for checking status of Ayd itself.

Change encoding of /stauts.txt

/status.txt accepts charset query to changing the encoding of result. The query value is unicode or ascii. In default, the encoding is unicode.

examples:

• http://localhost:9000/status.txt, http://localhost:9000/status.txt?charset=unicode: Reply is in UTF-8 text.
• http://localhost:9000/status.txt?charset=ascii: Reply is in ASCII text.

Be careful, the target URL or the message won't convert even if set charset=ascii. The response could include non-ascii text.

Filter log entries in /log.html, /log.csv, and /log.json

The log endpoints accept since, until, target, and query query to filtering log entries.

since and until is the queries to filtering by date-time, in RFC3339 format like 2001-02-03T16:05:06+09:00. In default, Ayd replies logs that from 7 days ago to current time.

And, target is the query to filtering by target URL.

query is space delimited query for filtering records. It works as perfect matching for status, partial match for target URL and message text. And, you can use filters for latency like <10ms or >=1s.

examples:

• http://localhost:9000/log.csv?since=2000-01-01T00:00:00Z&until=2001-01-01T00:00:00Z: The logs from 2000-01-01 to 2000-12-31.
• http://localhost:9000/log.csv?since=2021-01-01T00:00:00Z&target=ping:localhost: The logs about ping:localhost since 2021-01-01.
• http://localhost:9000/log.json?query=-healthy%20ping:: The logs within recent 7 days that only about unhealthy(-healthy) ping(ping:) targets.

Log file

The log file of Ayd is stored in JSON Lines format, encoded UTF-8. Each record has at least 4 fields.

• time when status check started, in RFC3339 format like 2001-02-30T16:05:06+00:00.

• status of the record that HEALTHY, DEGRADE, FAILURE, UNKNOWN, or ABORTED.

  • HEALTHY means service seems working well.

  • DEGRADE means service seems working but partially degraded. You should do something to the target system because the target is not completely healthy.

  • FAILURE means service seems failure or stopped. You should do something to the target system because the target may be broken if received this status.

  • UNKNOWN means Ayd is failed to status checking. For example, not found test script, failed to resolve service name, etc. You should check the target system, other systems like DNS, or Ayd settings because maybe something worse happened if received this status.

  • ABORTED means Ayd terminated during status checking. For example, Ayd reports this when terminated Ayd with Ctrl-C. You do not have to action about this status because it happens by your operation. (might be you have to check Ayd settings if you do not know why caused this)

• latency of the service in milliseconds.

  Some probes like ping: reports average latency, and other probes reports total value..

• target URL.

  This URL is the same to passed one as argument, but normalized. For example, ping:somehost?hello=world to be ping:somehost because ping: does not use query values.

• (optional) message, the detail of status, the reason for failure, or the output of the executed script.

Log records can have other extra fields.

For example, log lines look like below.

{"time":"2001-02-30T16:00:00+09:00", "status":"FAILURE", "latency":0.544, "target":"http://localhost", "message":"Get \"http://localhost\": dial tcp [::1]:80: connect: connection refused"}
{"time":"2001-02-30T16:05:00+09:00", "status":"UNKNOWN", "latency":0.000, "target":"tcp:somehost:1234", "message":"lookup somehost on 192.168.1.1:53: no such host"}
{"time":"2001-02-30T16:10:00+09:00", "status":"HEALTHY", "latency":0.375, "target":"ping:anotherhost", "message":"All packets came back", "packets_recv":3, "packets_sent:3, "rtt_avg":0.38, "rtt_max":0.47, "rtt_min":0.31}

Ayd will save the log file named ayd.log into the current directory in default. You can change this with -f option like below.

$ ayd -f /path/to/ayd.log ping:example.com

There is no feature to log rotate. Please consider using the log rotation tool if you have a plan to use it for a long time.

If you use -f - option, Ayd does not write log file. This is not recommended for production use because Ayd can't restore last status when restore if don't save log file. But, this is may useful for use Ayd as a parts of script file.

If you want use log file in other format like CSV, you can download via HTTP endpoint, or you can use ayd conv subcommand like below.

$ cat ayd.log | ayd conv > ayd_log.csv

$ ayd conv ./ayd.log -o ayd_log.csv

$ ayd conv -l ./ayd.log -o ayd_log.ltsv

Text encoding

Ayd expects UTF-8 with/without BOM or UTF-8 with BOM as input character encoding.

But in Windows, you can use the system's default character encoding too, for example CP1252 or CP932. Ayd tries to decode as UTF-8 first, and then tries to use the system's default encoding. If the text has the BOM, Ayd always follow it.

The characters couldn't decode will replaced by U+FFFD that means unrecognized character before save to the log file. That means;

• The log file is always valid UTF-8 even if your external command or plugin writes invalid characters.
• You can lose information if external commands or plugins write invalid characters as current encoding.

Tips

Daemonize

Docker

You can use docker image for execute Ayd. This image includes Ayd command and some plugins.

$ docker run --restart=always -v /var/log/ayd:/var/log/ayd macrat/ayd http://your-target.example.com

Of course, you can also use docker-compose or Kubernetes, etc. Please see ayd-docker repository for more information about this contianer image.

Systemd

If you using systemd, it is easy to daemonize Ayd.

Please put ayd command to /usr/local/bin/ayd (you can use another place if you want), and write a setting like below to /etc/systemd/system/ayd.service.

[Unit]
Description=Ayd status monitoring service
After=network.target remote-fs.target

[Service]
ExecStart=/usr/local/bin/ayd -f /var/log/ayd.log \
    http://your-target.example.com
#   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ please change target

[Install]
WantedBy=multi-user.target

And then, you can enable this service.

$ sudo systemctl daemon-reload
$ sudo systemctl start ayd
$ sudo systemctl enable ayd

Change listen port

You can change the HTTP server listen port with -p option. In default, Ayd uses port 9000.

Use HTTPS on status page

You can set certificate file and key file via -c option and -k option.

$ ayd -c ./your-certificate.crt -k ./your-certificate.key ping:localhost

The HTTP/2 will be enabled if set these options.

Use Basic Authentication on status page

Ayd has very simple authentication mechanism using Basic Authentication. You can use it like below.

$ ayd -u user:p@ssword ping:localhost

For above example, you can access status page using user as username and p@ssword as password.

This is not very secure because you have to write a plain password in the command argument. (Attacker can peek arguments of other process easily if you have access to the server terminal) But, this is very easy to setup, and at least, it works well against end user who doesn't have access to the server. If you need more secure option, please consider use reverse proxy like Nginx.

One-shot mode

If you want to use Ayd in a script, you can use -1 option. Ayd will check status just once and exit when passed -1 option.

Exit status code will be 0 if all targets are healthy. If some targets are unhealthy, the status code will be 1. And, if your arguments are wrong (or can't resolve host names, or exec scripts not found), the status code will be 2.