woodwatch

README

woodwatch

woodwatch is a small Go program that can be used to POST webhooks when peers stop sending ICMP echo requests (pings) for long enough to be considered down.

Lots of systems have the ability to send pings and so they all work with woodwatch out of the box! No client software needs to be installed.

Despite using ICMP woodwatch can be run without root privileges with Linux capabilities. A small bless.sh script is included to give cap_net_raw capabilities to the woodwatch binary.

The Internet is flaky. woodwatch tries to eliminate basic jitter and sporadic packet loss by supporting configurable up/down thresholds. The thresholds can be configured globally and also per-monitored-peer.

Notify all the things! woodwatch's webbhooks work out of the box with Slack for channel notifications when a peer goes up/down. Like the thresholds webhooks can be configured globally and also per-monitored-peer.

Installation

1. Pick a woodwatch release and download the .tar.gz for your architecture (most probably Linux_x86_64.tar.gz).

   wget https://github.com/cpu/woodwatch/releases/download/v0.0.1/woowdwatch_v0.0.1_Linux_x86_64.tar.gz
   

2. Extract the release archive and cd into it.

   mkdir /tmp/woodwatch && tar xf woodwatch_*.tar.gz -C /tmp/woodwatch --strip-components=1 && cd /tmp/woodwatch
   

3. Put the woodwatch binary in /usr/local/bin.

   sudo cp woodwatch /usr/local/bin
   

4. Make the bless.sh script executable and use it on the installed woodwatch binary to give it cap_net_raw.

   chmod +x ./bless.sh && sudo ./bless.sh /usr/local/bin/woodwatch
   

5. Create the woodwatch config directory.

   sudo mkdir -p /etc/woodwatch
   

6. Copy the example config in place.

   sudo cp example.config.json /etc/woodwatch/config.json
   

7. Customize the example config.

   sudo $EDITOR /etc/woodwatch/config.json
   

8. Add a woodwatch user with no shell, a disabled password, and no home directory.

   sudo adduser --disabled-password --no-create-home --shell=/bin/false --gecos "" woodwatch
   

9. Give the woodwatch user ownership of the woodwatch config directory.

   sudo chown -R woodwatch:woodwatch /etc/woodwatch
   

10. Install the example systemd service.

    sudo cp example.woodwatch.service /etc/systemd/system/woodwatch.service
    

11. Reload the systemd manager configuration.

    sudo systemctl daemon-reload
    

12. Enable the woodwatch service to start at boot.

    sudo systemctl enable woodwatch
    

13. Start the woodwatch service.

    sudo systemctl start woodwatch
    

14. Check the woodwatch service logs to ensure there are no errors.

    journalctl -u woodwatch --no-pager -e
    

Configuration

Global Configuration

• UpThreshold - an unsigned integer expressing how many checks without a peer timeout must occur before the peer is considered up.
• DownThreshold - an unsigned integer expressing how many checks with a peer timeout must occur before the peer is considered down.
• MonitorCycle - a required duration string expressing how often peers are checked for timeouts. This should be shorter than the PeerTimeout.
• PeerTimeout - a required duration string expressing how long must elapse between seeing ICMP echo requests from a peer before it is considered timed out. This should be longer than the MonitorCycle.
• Webhook - an optional string specifying a URL to be POSTed for notable events (or all state change events if -verbose is used).
• Peers - one or more objects describing a peer configuration.

Peer Configuration

• Name - a required string representing the name of the peer. Use Slack emoji like :satellite: to make your webhook events more memorable.
• Network - a required CIDR notation network that the peer will be sending ICMP echo requests from. E.g. 192.168.1.0/24 to expect pings from 192.168.1.1 through 192.168.1.254. You may find a CIDR calculator helpful.
• UpThreshold - an optional unsigned integer to override the global UpThreshold for this peer.
• DownThreshold - an optional unsigned integer to override the global DownThreshold for this peer.
• Webhook - an optional string specifying a URL to override the global Webhook for this peer.

Example Configuration

{
  "UpThreshold": 3,
  "DownThreshold": 3,
  "MonitorCycle": "2s",
  "PeerTimeout": "4s",
  "Webhook": "http://localhost:9090/woodwatch-hook"
  "Peers": [
    {
      "Name": "LAN",
      "Network": "192.168.1.0/24",
      "UpThreshold": 2,
      "DownThreshold": 5,
      "Webhook": "http://localhost:9090/custom-lan-hook"
    }
  ]
}

The above configuration will have woodwatch monitor a LAN for connectivity by expecting periodic ICMP echo requests from any host in the 192.168.1.0/24 network, at least every 4s.

After 2 checks (4s) having seen the expected pings the LAN network will be considered Up and a webhook POST will be sent to http://localhost:9090/custom-lan-hook.

If the pings stop being sent, after 5 checks (10s) the LAN network will be considered Down and a webhook POST will be sent to http://localhost:9090/custom-lan-hook.

Example Webhook POSTs

For the example configuration shared above the configured webhook for the LAN peer will receive an Up event as a HTTP POST request like:

POST /custom-lan-hook HTTP/1.1
Host: localhost:9090
User-Agent: cpu.woodwatch 0.0.1 (linux; amd64)
Content-Length: 299
Content-Type: application/json
Accept-Encoding: gzip

{
  "title": "Peer LAN is Up",
  "text": "LAN (last seen 2019-02-24 11:22:44 AM -0500) was previously Maybe Up (2 of 2) and is now Up",
  "timestamp": "2019-02-24T11:22:45.045655028-05:00",
  "lastSeen": "2019-02-24T11:22:44.660459371-05:00",
  "newState": "Up",
  "prevState": "Maybe Up (2 of 2)"
}

Similarly for a Down event the configured webhook for the LAN peer will receive an HTTP POST request like:

POST /custom-lan-hook HTTP/1.1
Host: localhost:9090
User-Agent: cpu.woodwatch 0.0.1 (linux; amd64)
Content-Length: 309
Content-Type: application/json
Accept-Encoding: gzip

{
  "title": "Peer LAN is Down",
  "text": "LAN (last seen 2019-02-24 11:22:54 AM -0500) was previously Maybe Down (5 of 5) and is now Down",
  "timestamp": "2019-02-24T11:23:09.045820003-05:00",
  "lastSeen": "2019-02-24T11:22:54.695991071-05:00",
  "newState": "Down",
  "prevState": "Maybe Down (5 of 5)"
}

Development

woodwatch is built to support Go 1.11.x and uses Go 1.11 modules and vendored dependencies. Presently the only dependency outside of the Go stdlib is x/net/. Releases are built and published with GoReleaser.

Presently woodwatch supports Linux and the x86_64, arm64, armv7 and arm6 architectures. Woodwatch does not support other OSes at this time because:

1. The x/net/icmp.ListenPacket function can only bind a raw ICMP endpoint on Darwin and Linux.
2. Running woodwatch without root requires Linux capabilities, specifically CAP_NET_RAW. It's a very bad idea to run woodwatch as root. Don't do it!

Linting

To run linters locally install golangci-lint:

   go get -u github.com/golangci/golangci-lint/cmd/golangci-lint

and then start the linters by running golangci-lint in the root of the project directory:

   golangci-lint run

Unit tests

In the root of the project directory run:

   go test -race ./...

Echo Webhook POSTs

You might find it useful to test woodwatch webhooks by running a small webserver that echoes all received POST requests. One option for this is the Node.js http-echo-server. You can run the server as follows:

   PORT=9090 http-echo-server

Configure a webhook URL using the http-echo-server by setting the URL in your config file:

   ...
   "UpHook": "http://localhost:9090/woodwatch-up-hook",
   ...

Making a snapshot release

First make sure you have installed GoReleaser. Then run:

   goreleaser --snapshot --skip-publish --rm-dist

This will result in a dist/ directory structure similar to the following:

   dist
   ├── checksums.txt
   ├── config.yaml
   ├── linux_amd64
   │   └── woodwatch
   ├── linux_arm_6
   │   └── woodwatch
   ├── linux_arm64
   │   └── woodwatch
   ├── linux_arm_7
   │   └── woodwatch
   ├── woodwatch_v0.0.0-next_Linux_arm64.tar.gz
   ├── woodwatch_v0.0.0-next_Linux_armv6.tar.gz
   ├── woodwatch_v0.0.0-next_Linux_armv7.tar.gz
   └── woodwatch_v0.0.0-next_Linux_x86_64.tar.gz

Documentation

Overview

Package woodwatch provides a server for monitoring peers by ICMP echo request keepalives.

Index

• Variables
• type Config
  • func LoadConfig(data []byte) (Config, error)
  • func LoadConfigFile(filename string) (Config, error)
  • func (c Config) Valid() error
• type PeerConfig
  • func (pc PeerConfig) Valid() error
• type Server
  • func NewServer(log *log.Logger, verbose bool, addr string, c Config) (*Server, error)
  • func (s *Server) Close() error
  • func (s *Server) Listen() error

Variables

var (
	// ErrNoPeerName is returned from PeerConfig.Valid() when the PeerConfig doesn't
	// have a Name.
	ErrNoPeerName = errors.New("All PeerConfigs must have a Name")
	// ErrNoPeerNetwork is returned from PeerConfig.Valid() when the PeerConfig
	// doesn't have a Network.
	ErrNoPeerNetwork = errors.New("All PeerConfigs must have a Network")
)

var (
	// ErrServerAlreadyListening is returned from Server.Listen when the Server is
	// already listening.
	ErrServerAlreadyListening = errors.New("Listen() can only be called once")
	// ErrServerNotListening is returned from Server.Close when the Server is not
	// listening.
	ErrServerNotListening = errors.New("Close() must be called after Listen()")
	// ErrEmptyListenAddress is returned from Server.Listen when the Server's
	// listen address is empty.
	ErrEmptyListenAddress = errors.New("Listen address must not be empty")
	// ErrTooFewPeers is returned from NewServer when there aren't enough
	// peers provided.
	ErrTooFewPeers = errors.New("One or more Peers must be configured")
)

Types

type Config

type Config struct {
	// UpThreshold is how many cycles a peer needs to be sending ICMP echo
	// requests without timeout before it is considered up. Individual PeerConfigs
	// may set their own UpThreshold.
	UpThreshold uint
	// DownThreshold is how many cycles a peer needs to miss sending ICMP echo
	// requests before it is considered down. Individual PeerConfigs may set their
	// own DownThreshold.
	DownThreshold uint
	// MonitorCycle is a mandatory string describing the duration between checking
	// if a Peer has sent ICMP echo requests within the PeerTimeout. E.g. "4s",
	// "1m".
	MonitorCycle string
	// PeerTimeout is a mandatory string describing the duration within a Peer
	// must have sent ICMP echo requests to be considered seen recently during
	// a monitor cycle. E.g. "8s", "2m".
	PeerTimeout string
	// Webhook is an optional webhook URL to be POSTed for events. Individual
	// PeerConfigs may set their own Webhook.
	Webhook string
	// Peers is one or more PeerConfigs describing a peer to be monitored.
	Peers []PeerConfig
}

Config describes the global woodwatch configuration and the peers to be monitored.

func LoadConfig

func LoadConfig(data []byte) (Config, error)

LoadConfig loads a woodwatch.Config from the given data bytes.

func LoadConfigFile

func LoadConfigFile(filename string) (Config, error)

LoadConfigFile reads the data bytes from the file located at the provided filename and uses LoadConfig to return a woodwatch.Config from the file bytes.

func (Config) Valid

func (c Config) Valid() error

Valid checks that a woodwatch Config is valid. If no peers are specified ErrTooFewPeers is returned. Each of the Peers specified will have their PeerConfig.Valid() function called and any errors will be returned. The MonitorCycle and PeerTimeout will both be parsed as time.Duration instances and any errors will be returned.

type PeerConfig

type PeerConfig struct {
	// Name is the name of the peer. Supports :slack: emoji!
	Name string
	// Network is the string representation of a CIDR network. To be considered up
	// the peer must periodically send ICMP echo requests from a host within this
	// CIDR network. E.g. "192.168.1.0/24".
	Network string
	// UpThreshold is how many cycles the peer needs to be sending ICMP echo
	// requests without timeout before it is considered up. If zero the global
	// UpThreshold is used.
	UpThreshold uint
	// DownThreshold is how many cycles the peer needs to miss sending ICMP echo
	// requests before it is considered down. If zero the global DownThreshold is
	// used.
	DownThreshold uint
	// Webhook is an optional webhook to be POSTed for events. If not provided the
	// global Webhook is used.
	Webhook string
}

PeerConfig is a struct holding configuration related to monitoring a Peer.

func (PeerConfig) Valid

func (pc PeerConfig) Valid() error

Valid checks that a PeerConfig has a Name and Network or returns ErrNoPeerName/ErrNoPeerNetwork if the PeerConfig is not valid.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is a struct for monitoring peers for keepalives received on a icmp.PacketConn.

func NewServer

func NewServer(
	log *log.Logger,
	verbose bool,
	addr string,
	c Config) (*Server, error)

NewServer constructs a woodwatch.Server for the given arguments and config or returns an error. The Server will not be running and listening for ICMP messages until it is explicitly started by calling Server.Listen()

func (*Server) Close

func (s *Server) Close() error

Close closes the Server's PacketConn and stops listening for ICMP messages on the Server's listen address. If Close is called before Listen it will return ErrServerNotListening.

func (*Server) Listen

func (s *Server) Listen() error

Listen opens a PacketConn for the Server's listen address that will listen for ICMP packets. If Listen is called on a Server with an empty listen address it will return ErrEmptyListeningAddress. If Listen is called more than once it will return ErrServerAlreadyListening for all calls after the first.