poller

README

Poller

A composable monitoring framework written in Go.

It's very alpha and prone to change, so please bear with it. The README is currently outdated as things are in the process of being reorganized.

What is it?

Poller's job is to monitor http applications by submitting GET requests to URL you define in a config file. These URLs are called "Checks".

Once a check is done, the result is sent to one or many backends of your choice.

Current supported backends are stdout, syslog, librato and statsd.

How to configure it

Configuration is done via:

• Environment variables for backend.
• JSON for list of checks
• Flags passed when executing the program.

A typical json file for checks looks like this

[
    {
        "key": "com_google",                // Key should be unique among all checks specified
        "url": "http://google.com",         // URL of the check
        "interval": "10s",                  // Check will be perfom every 10s. Format available here: http://godoc.org/time#ParseDuration
        "alert": true,                      // (optional) Enable "alerts" for this checks.
        "alertDelay": "60s"                 // (required if alert is set) Wait 60s (or 6 other checks after the first downtime) before sending an alert
    },
    {
        "key": "fr_yahoo",
        "url": "http://yahoo.fr",
        "interval": "10s"
    },
    {
        "key": "connect_sensiolabs_com_api",
        "url": "https://connect.sensiolabs.com/api/",
        "interval": "60s",
        "headers": {
            "Accept": "application/vnd.com.sensiolabs.connect+xml"  // (optional) Added HTTP header
        }
    }
]

The JSON config file is optional as checks can be added thanks to the HTTP endpoint /checks.

Running ./poller --help will prints a list of available options.

How to monitor it?

A /health http endpoint is available. If poller is answering a 200, then all is good!

How to receive alerts when a check is down?

Alerting and alerting delay is customisable for each check (Please read the example configuration). "Alerters" are enabled from the command line with the --alerts flag.

# Enable the smtp alerter
./poller --alerts="smtp"

# Enable both the pagerduty and smtp alerter
./poller --alerts="smtp,pagerduty"

SMTP Alerter

The SMTP Alerter is enabled when you run poller like this:

./poller --alerts="smtp"

SMTP alerter is configured using these environment variables:

• SMTP_HOST: (required) ie: localhost
• SMTP_PORT: (required) ie: 25
• SMTP_AUTH: (optional) "MD5" or "PLAIN"
• SMTP_USERNAME: (optional)
• SMTP_PASSWORD: (optional)
• SMTP_PLAIN_IDENTITY: (optional)
• SMTP_RECIPIENT: (required) ie: monitoring@example.org
• SMTP_FROM: (required) ie: poller@example.org

PagerDuty Alerter

The PagerDuty alerter is enabled when you run poller like this:

./poller --alerts="pagerduty"

PagerDuty's alerter is configured using these environment variables:

• PAGERDUTY_SERVICE_KEY: (required) PagerDuty's Service Key. Please read the Getting started section.

How to add checks while poller is running

Poller supports live configuration changes thanks to the /checks http endpoint. Send a PUT request with a valid config JSON in the body of the request and poller will append the checks to its list.

Backends configuration

Here is a list of supported backend and how to configure them with environment variables. Backends are enabled from the command-line thanks to the --backend flag.

# Enable the stdout backend
./poller --backends="stdout"

# Enable both the librato and stdout backend
./poller --backends="librato,stdout"

Librato

• LIBRATO_USER (required): Username of your librato account
• LIBRATO_TOKEN (required): API token of your librato account
• LIBRATO_SOURCE (optional): Source name for your metrics. Defaults to poller
• LIBRATO_PREFIX (optional): Prefix of your metrics. Defaults to poller.checks.

Given your check's key is foobar and LIBRATO_PREFIX is acme.:

On success:

• 1 will be sent to metrics acme.foobar.up
• The duration in milliseconds will be sent to foobar.duration

On error or timeout:

• 0 will be sent to metrics acme.foobar.up
• The duration in milliseconds will be sent to foobar.duration

Stdout

No configuration is necessary. Output will look like this:

2012/01/24 11:35:16 com_google UP 345.271ms
2012/01/24 11:35:17 fr_yahoo DOWN 1.518175s
2012/01/24 11:35:17 fr_yahoo ALERT Down since 2006-01-02 15:04:05.999999999 -0700 MST

Statsd

Statsd backend uses of these environment variables:

• STATSD_HOST (required): Host of your statsd instance
• STATSD_PORT (optional): Port of your statsd instance. Defaults to 8125.
• STATSD_PROTOCOL (optional): Either tcp or udp. Defaults to udp.
• STATSD_PREFIX (optional): Prefix of your metrics. Defaults to poller.checks.

The metrics are sent the same way as the Librato backend.

Syslog

You can configure the syslog backend with these:

• SYSLOG_NETWORK (optional): "tcp", "tcp4" (IPv4-only), "tcp6" (IPv6-only), "udp", "udp4" (IPv4-only), "udp6" (IPv6-only), "ip", "ip4" (IPv4-only), "ip6" (IPv6-only), "unix" and "unixpacket". Defaults to nothing.
• SYSLOG_ADDRESS (optional): Address of your syslog daemon. Defaults to nothing.
• SYSLOG_PREFIX (optional): This will be added to your log. Defaults to "poller".

Yes, you can send checks results to loggly by using the syslog backend.

Output formatting is the same as the stdout backend.

Technical documentation

Poller's documentation is available on godoc: http://godoc.org/github.com/marcw/poller

Why Go?

A Go application has the advantage of being concurrent, fast, cross-compilable and easily deployable. I think that's a lot of good reasons to create a monitoring system out of this language.

Install go

note: OSX users, do NOT install Go from homebrew. At time of writing, the package is broken and won't let you cross-compile.

Please follow instructions from: http://golang.org/doc/install/source

Please set a $GOROOT env var in your ~/.bashrc. file. If you unpacked the go source in /home/you/go:

export GOROOT=/home/you/go

Please also set a $GOPATH env var in your ~/.bashrc file. $GOPATH is where go will look for packages and where you clone private projects. An example of $GOPATH would be /home/you/work/go.

export GOPATH=/home/you/work/go

Enable cross compilation with Go

Follow this really good blogpost: http://dave.cheney.net/2012/09/08/an-introduction-to-cross-compilation-with-go

License

The Poller code is free to use and distribute, under the MIT license.

Documentation

Index

• func NewConfigHttpHandler(config *Config) http.Handler
• type Alerter
  • func NewPagerDutyAlerter() (Alerter, error)
  • func NewSmtpAlerter() (Alerter, error)
• type Backend
  • func NewLibratoBackend(user, token, source, prefix string) (Backend, error)
  • func NewStatsdBackend(host, port, protocol, prefix string) (Backend, error)
  • func NewStdoutBackend() Backend
  • func NewSyslogBackend(network, raddr, prefix string) (Backend, error)
• type Check
  • func NewCheck(key, interval string, alert bool, alertDelay string, notifyFix bool, ...) (*Check, error)
  • func NewCheckFromJSON(data []byte) (*Check, error)
  • func (c *Check) AlertDescription() string
  • func (c *Check) JSON() ([]byte, error)
  • func (c *Check) ShouldAlert() bool
  • func (c *Check) ShouldNotifyFix() bool
  • func (c *Check) Type() CheckType
• type CheckType
• type Config
  • func NewConfig(store Store, scheduler Scheduler) *Config
  • func (c *Config) Add(check *Check) error
  • func (c *Config) Schedule()
• type Event
  • func NewEvent(check *Check) *Event
  • func (e *Event) Down()
  • func (e *Event) IsUp() bool
  • func (e *Event) Up()
• type Poller
  • func NewDirectPoller() Poller
• type Probe
  • func NewHttpProbe(ua string, timeout time.Duration) Probe
  • func NewUdpProbe(timeout time.Duration) Probe
• type Scheduler
  • func NewSimpleScheduler() Scheduler
• type Store
  • func NewInMemoryStore() Store

Functions

func NewConfigHttpHandler

func NewConfigHttpHandler(config *Config) http.Handler

Create a handler function that is usable by http.Handle. This handler will be able response to GET, POST and PUT requests. * GET the list of checks as a JSON array * POST will create a new check and add it to the CheckList. * After any of POST or PUT operation, the configuration is persisted to it's store.

Types

type Alerter

type Alerter interface {
	Alert(event *Event)
}

An Alerter raises an alert based on the event it received. An alert is a communication to a system or a user with the information about current's and past check's states. For concrete implementation, see the "github.com/marcw/poller/alert" package.

func NewPagerDutyAlerter

func NewPagerDutyAlerter() (Alerter, error)

func NewSmtpAlerter

func NewSmtpAlerter() (Alerter, error)

type Backend

type Backend interface {
	Log(e *Event)
	Close()
}

A backend log checks event. For concrete implementation, see the "github.com/marcw/poller/backend" package.

func NewLibratoBackend

func NewLibratoBackend(user, token, source, prefix string) (Backend, error)

func NewStatsdBackend

func NewStatsdBackend(host, port, protocol, prefix string) (Backend, error)

Instanciate a new Backend that will send data to a statsd instance

func NewStdoutBackend

func NewStdoutBackend() Backend

func NewSyslogBackend

func NewSyslogBackend(network, raddr, prefix string) (Backend, error)

Create a new SyslogBackend instance.

type Check

type Check struct {
	Key string // Key (should be unique among same Scheduler

	Interval time.Duration // Interval between each check

	UpSince    time.Time     // Time since the service is up
	DownSince  time.Time     // Time since the service is down
	WasDownFor time.Duration // Time since the service was down
	WasUpFor   time.Duration // Time since the service was up

	Alert     bool // Raise alert if service is down
	Alerted   bool // Is backend already alerted?
	NotifyFix bool // Notify if service is back up

	AlertDelay time.Duration // Delay before raising an alert (zero value = NOW)
	Config     *bag.Bag
	// contains filtered or unexported fields
}

func NewCheck

func NewCheck(key, interval string, alert bool, alertDelay string, notifyFix bool, config map[string]interface{}) (*Check, error)

func NewCheckFromJSON

func NewCheckFromJSON(data []byte) (*Check, error)

NewCheckFromJSON() instantiates a new Check from a JSON representation.

func (*Check) AlertDescription

func (c *Check) AlertDescription() string

func (*Check) JSON

func (c *Check) JSON() ([]byte, error)

Returns a JSON representation of the Check.

func (*Check) ShouldAlert

func (c *Check) ShouldAlert() bool

Check if it's time to send the alert. Returns true if it is.

func (*Check) ShouldNotifyFix

func (c *Check) ShouldNotifyFix() bool

func (*Check) Type

func (c *Check) Type() CheckType

type CheckType

type CheckType string

const (
	CheckTypeUDP  CheckType = "udp"
	CheckTypeHTTP CheckType = "http"
)

type Config

type Config struct {
	// contains filtered or unexported fields
}

The Config struct holds and links together a CheckList, a Scheduler and a configuration Store.

func NewConfig

func NewConfig(store Store, scheduler Scheduler) *Config

Instantiates a new Config with an empty CheckList

func (*Config) Add

func (c *Config) Add(check *Check) error

func (*Config) Schedule

func (c *Config) Schedule()

type Event

type Event struct {
	Check      *Check        // check
	Duration   time.Duration // total duration of check
	StatusCode int           // http status code, if any
	Time       time.Time     // time of check

	Alert     bool // true if backend should raise an alert
	NotifyFix bool // true if backend should notify of service being up again
	// contains filtered or unexported fields
}

Represents the state of a Check after being polled

func NewEvent

func NewEvent(check *Check) *Event

func (*Event) Down

func (e *Event) Down()

func (*Event) IsUp

func (e *Event) IsUp() bool

func (*Event) Up

func (e *Event) Up()

type Poller

type Poller interface {
	Run(Scheduler, Backend, Probe, Alerter)
}

A Poller is the glue between a Scheduler, a Backend, a Probe and a Alerter.

func NewDirectPoller

func NewDirectPoller() Poller

NewDirectPoller() returns a "no-frills" Poller instance. It waits for the next scheduled check, poll it, log it and if alerting is needed, pass it through the alerter.

type Probe

type Probe interface {
	Test(c *Check) *Event
}

A Probe is a specialized way to poll a check. ie: a HttpProbe will specialize in polling HTTP resources.

func NewHttpProbe

func NewHttpProbe(ua string, timeout time.Duration) Probe

func NewUdpProbe

func NewUdpProbe(timeout time.Duration) Probe

type Scheduler

type Scheduler interface {
	Schedule(check *Check) error
	Stop(key string)
	StopAll()
	Start()
	Next() <-chan *Check
}

func NewSimpleScheduler

func NewSimpleScheduler() Scheduler

Instantiates a SimpleScheduler which scheduling strategy's fairly basic. For each scheduled check, a new time.Timer is created in its own goroutine.

type Store

type Store interface {
	Add(*Check) error
	Get(key string) (*Check, error)
	Remove(key string) error
	Len() (int, error)
	ScheduleAll(Scheduler) error
}

A Store defines a place where configuration can be loaded/persisted.

func NewInMemoryStore

func NewInMemoryStore() Store

Instantiates a new CheckList