function

README

Auto Remedy Dispatcher

The Auto Remedy Dispatcher is a receiver for Prometheus alertmanager alerts, and looks up possible remedies for alerts, and then applies the winning remedy.

Deploying

The dispatcher is deployed an OpenFAAS function, using the command

faas build -f dispatcher.yml
faas deploy -f dispatcher.yml --env=combine_output=false

The combine_output=false makes the stderr log to the docker log, so you can view it with

kubectl logs -n openfaas-fn <dispatcher pod>

Configuration

Each remedy has an array of remedy configurations, which are used to decide which nodes match the remedy based on a regexp filter.

E.g. the NodeDown remedy which performs a hardware reset on the servers should have a limit on how often it can be applied:

[
  {
    "name": "db node down",
    "filter": "-db.*",
    "ratelimit": "2"
  },
  {
    "name": "node down",
    "filter": ".*",
    "ratelimit": "10"
  },
]

Development

If you want to develop in a fully functional environment locally, use minikube. There you can deploy openfaas, and use the prometheus instance which is bundled with it.

After openfaas is deployed, run

eval $(minikube docker-env)
export OPENFAAS_URL=http://$(minikube ip):31112

then deploy the dispatcher with

faas deploy -f dispatch.yml

The first time it is deployed you need to run

kubectl edit -n openfaas-fn deployment.apps/dispatcher

change imagePullPolicy from Always to Never, as it otherwise tries to download the docker image from the docker hub.

Configure amtool

Make alertmanager accessible

kubectl edit -n openfaas service/alertmanager

set type: NodePort

go get github.com/prometheus/alertmanager/cmd/amtool

create ~/.config/amtool/config.yml

alertmanager.url: http://$(minikube ip):$(kubectl get service -n openfaas alertmanager --no-headers -o custom-columns=port:spec.ports[*].nodePort)

amtool alert add \
  FooAlert \
  node=x.y.z \
  --annotation=runbook='http://x.y.z' \
  --summary='foo has happened'

Documentation

Index

• func Handle(req handler.Request) (handler.Response, error)
• type ConfigFetcher
  • func (cf *ConfigFetcher) Fetch(alert, node string) (*RemedyConfig, error)
• type Configurator
• type Dispatcher
  • func NewDispatcher(conf *DispatcherConfig) *Dispatcher
  • func (d *Dispatcher) Dispatch(alert template.Alert) error
• type DispatcherConfig
  • func NewDispatcherConfig(r Rediser) *DispatcherConfig
• type Ratelimit
  • func (r *Ratelimit) Limited(key string, max int64) (bool, error)
• type Ratelimiter
• type Redis
  • func (r *Redis) Expire(key string, duration time.Duration) (bool, error)
  • func (r *Redis) GetInt64(key string) (int64, error)
  • func (r *Redis) Incr(key string) (int64, error)
  • func (r *Redis) LRange(key string, start, stop int64) ([]string, error)
• type Rediser
• type RemedyConfig

Functions

func Handle

func Handle(req handler.Request) (handler.Response, error)

Handle a function invocation

Types

type ConfigFetcher

type ConfigFetcher struct {
	// contains filtered or unexported fields
}

func (*ConfigFetcher) Fetch

func (cf *ConfigFetcher) Fetch(alert, node string) (*RemedyConfig, error)

Fetch returns the first remedy configuration for alert that matches node, or nil if no config matched

type Configurator

type Configurator interface {
	Fetch(string, string) (*RemedyConfig, error)
}

type Dispatcher

type Dispatcher struct {
	*DispatcherConfig
}

func NewDispatcher

func NewDispatcher(conf *DispatcherConfig) *Dispatcher

func (*Dispatcher) Dispatch

func (d *Dispatcher) Dispatch(alert template.Alert) error

type DispatcherConfig

type DispatcherConfig struct {
	// contains filtered or unexported fields
}

func NewDispatcherConfig

func NewDispatcherConfig(r Rediser) *DispatcherConfig

type Ratelimit

type Ratelimit struct {
	// contains filtered or unexported fields
}

Ratelimit is a distributed rate limiter, which uses this pattern: https://redislabs.com/redis-best-practices/basic-rate-limiting/

func (*Ratelimit) Limited

func (r *Ratelimit) Limited(key string, max int64) (bool, error)

Limited checks if key has reached the max rate limit per hour

type Ratelimiter

type Ratelimiter interface {
	Limited(key string, max int64) (bool, error)
}

type Redis

type Redis struct {
	// contains filtered or unexported fields
}

func (*Redis) Expire

func (r *Redis) Expire(key string, duration time.Duration) (bool, error)

func (*Redis) GetInt64

func (r *Redis) GetInt64(key string) (int64, error)

GetInt64 wraps the redis call, returns -1 when key is not found

func (*Redis) Incr

func (r *Redis) Incr(key string) (int64, error)

func (*Redis) LRange

func (r *Redis) LRange(key string, start, stop int64) ([]string, error)

type Rediser

type Rediser interface {
	GetInt64(string) (int64, error)
	Incr(string) (int64, error)
	Expire(string, time.Duration) (bool, error)
	LRange(string, int64, int64) ([]string, error)
}

type RemedyConfig

type RemedyConfig struct {
	Name       string `json:"name"`
	NodeFilter string `json:"filter"`    // regexp
	Remedy     string `json:"remedy"`    // name of the faas function to invoke
	RateLimit  int64  `json:"ratelimit"` // max invocations per minute

}