prepalert

README

prepalert

Toil reduction tool to prepare before responding to Mackerel alerts

preplert consists of two parts: a webhook server that receives Mackerel webhooks and sends the payload to Amazon SQS, and a worker that queries various data based on the webhooks and pastes information for alert response as a GraphAnnotation.

Install

Homebrew (macOS and Linux)

$ brew install mashiike/tap/prepalert

Binary packages

Releases

QuickStart

Set your Mackerel API key to the environment variable MACKEREL_APIKEY.
and Execute the following command:

$ prepalert init 

Or the following command:

$ prepalert --coinfig <output config path> init

Usage

NAME:
   prepalert - A webhook server for prepare alert memo

USAGE:
   prepalert -config <config file> [command options]

VERSION:
   current

COMMANDS:
   exec      Generate a virtual webhook from past alert to execute the rule
   init      create inital config
   run       run server (default command)
   validate  validate the configuration
   help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --address value                    run address (default: ":8080") [$PREPALERT_ADDRESS]
   --batch-size value                 run local sqs batch size (default: 1) [$PREPALERT_BATCH_SIZE]
   --config value, -c value           config path (default: ".") [$CONFIG, $PREPALERT_CONFIG]
   --help, -h                         show help (default: false)
   --log-level value                  output log-level (default: "info") [$PREPALERT_LOG_LEVEL]
   --mackerel-apikey value, -k value  for access mackerel API (default: *********) [$MACKEREL_APIKEY, $PREPALERT_MACKEREL_APIKEY]
   --mode value                       run mode (default: "http") [$PREPALERT_MODE]
   --prefix value                     run server prefix (default: "/") [$PREPALERT_PREFIX]
   --version, -v                      print the version (default: false)

If the command is omitted, the run command is executed.

Configurations

Configuration file is HCL (HashiCorp Configuration Language) format. prepalert init can generate a initial configuration file.

The most small configuration file is as follows:

prepalert {
    required_version = ">=v0.2.0"
    sqs_queue_name   = "prepalert"
    service          = "prod"
}

rule "any_alert" {
    alert {
        any = true
    }

    infomation = <<EOF
How do you respond to alerts?
Describe information about your alert response here.
(This area can use Go's template notation.)
EOF
}

Usage with AWS Lambda (serverless)

prepalert works with AWS Lambda and Amazon SQS.

Lambda Function requires a webhook and a worker

sequenceDiagram
  autonumber
  Mackerel->>+http lambda function : POST /
  http lambda function ->>+Amazon SQS: SendMessage
  Amazon SQS-->- http lambda function: 200 Ok
  http lambda function-->- Mackerel: 200 Ok
  Amazon SQS ->>+ worker lambda function: trigger by AWS Lambda
  worker lambda function ->>+ Data Source: query
  Data Source -->- worker lambda function: query results
  worker lambda function  ->>+ Mackerel: Create Graph Annotation
  Mackerel-->- worker lambda function : 200 Ok
  worker lambda function ->>-  Amazon SQS: Success Delete message

Let's solidify the Lambda package with the following zip arcive (runtime provided.al2)

lambda.zip
├── bootstrap    # build binary
└── config.hcl   # configuration file

A related document is https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

for example.

deploy two lambda functions, prepalert-http and prepalert-worker in lambda directory
The example of lambda directory uses lambroll for deployment.

For more information on the infrastructure around lambda functions, please refer to example.tf.

LICENSE

MIT License

Copyright (c) 2022 IKEDA Masashi

Documentation

Index

• type Alert
• type App
  • func New(apikey string, cfg *hclconfig.Config) (*App, error)
  • func (app *App) CheckBasicAuth(r *http.Request) bool
  • func (app *App) EnableBackend() bool
  • func (app *App) EnableBasicAuth() bool
  • func (app *App) Exec(ctx context.Context, alertID string) error
  • func (app *App) HandleSQS(ctx context.Context, event *events.SQSEvent) (*grat.BatchItemFailureResponse, error)
  • func (app *App) NewWebhookBody(ctx context.Context, alertID string) (*WebhookBody, error)
  • func (app *App) ProcessRule(ctx context.Context, rule *Rule, body *WebhookBody) error
  • func (app *App) ProcessRules(ctx context.Context, body *WebhookBody) error
  • func (app *App) Run(ctx context.Context, opts RunOptions) error
  • func (app *App) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (app *App) WithQueryRunningContext(ctx context.Context, reqID uint64, message *events.SQSMessage) context.Context
• type Host
• type QueryData
• type RenderInfomationData
• type Role
• type Rule
  • func NewRule(client *mackerel.Client, cfg *hclconfig.RuleBlock) (*Rule, error)
  • func (rule *Rule) BuildInfomation(ctx context.Context, body *WebhookBody) (string, error)
  • func (rule *Rule) Match(body *WebhookBody) bool
  • func (rule *Rule) Name() string
  • func (rule *Rule) RenderInfomation(ctx context.Context, data *RenderInfomationData) (string, error)
• type RunOptions
• type Service
• type WebhookBody

Types

type Alert

type Alert struct {
	OpenedAt          int64   `json:"openedAt"`
	ClosedAt          int64   `json:"closedAt"`
	CreatedAt         int64   `json:"createdAt"`
	CriticalThreshold float64 `json:"criticalThreshold"`
	Duration          int64   `json:"duration"`
	IsOpen            bool    `json:"isOpen"`
	MetricLabel       string  `json:"metricLabel"`
	MetricValue       float64 `json:"metricValue"`
	MonitorName       string  `json:"monitorName"`
	MonitorOperator   string  `json:"monitorOperator"`
	Status            string  `json:"status"`
	Trigger           string  `json:"trigger"`
	ID                string  `json:"id"`
	URL               string  `json:"url"`
	WarningThreshold  float64 `json:"warningThreshold"`
}

type App

type App struct {
	// contains filtered or unexported fields
}

func New

func New(apikey string, cfg *hclconfig.Config) (*App, error)

func (*App) CheckBasicAuth

func (app *App) CheckBasicAuth(r *http.Request) bool

func (*App) EnableBackend

func (app *App) EnableBackend() bool

func (*App) EnableBasicAuth

func (app *App) EnableBasicAuth() bool

func (*App) Exec

func (app *App) Exec(ctx context.Context, alertID string) error

func (*App) HandleSQS

func (app *App) HandleSQS(ctx context.Context, event *events.SQSEvent) (*grat.BatchItemFailureResponse, error)

func (*App) NewWebhookBody

func (app *App) NewWebhookBody(ctx context.Context, alertID string) (*WebhookBody, error)

func (*App) ProcessRule

func (app *App) ProcessRule(ctx context.Context, rule *Rule, body *WebhookBody) error

func (*App) ProcessRules

func (app *App) ProcessRules(ctx context.Context, body *WebhookBody) error

func (*App) Run

func (app *App) Run(ctx context.Context, opts RunOptions) error

func (*App) ServeHTTP

func (app *App) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*App) WithQueryRunningContext

func (app *App) WithQueryRunningContext(ctx context.Context, reqID uint64, message *events.SQSMessage) context.Context

type Host

type Host struct {
	ID        string  `json:"id"`
	Name      string  `json:"name"`
	URL       string  `json:"url"`
	Type      string  `json:"type"`
	Status    string  `json:"status"`
	Memo      string  `json:"memo"`
	IsRetired bool    `json:"isRetired"`
	Roles     []*Role `json:"roles"`
}

type QueryData

type QueryData struct {
	*WebhookBody
	Params interface{}
}

type RenderInfomationData

type RenderInfomationData struct {
	*WebhookBody
	QueryResults map[string]*queryrunner.QueryResult
	Params       interface{}
}

type Role

type Role struct {
	Fullname    string `json:"fullname"`
	ServiceName string `json:"serviceName"`
	ServiceURL  string `json:"serviceUrl"`
	RoleName    string `json:"roleName"`
	RoleURL     string `json:"roleUrl"`
}

type Rule

type Rule struct {
	// contains filtered or unexported fields
}

func NewRule

func NewRule(client *mackerel.Client, cfg *hclconfig.RuleBlock) (*Rule, error)

func (*Rule) BuildInfomation

func (rule *Rule) BuildInfomation(ctx context.Context, body *WebhookBody) (string, error)

func (*Rule) Match

func (rule *Rule) Match(body *WebhookBody) bool

func (*Rule) Name

func (rule *Rule) Name() string

func (*Rule) RenderInfomation

func (rule *Rule) RenderInfomation(ctx context.Context, data *RenderInfomationData) (string, error)

type RunOptions

type RunOptions struct {
	Mode      string
	Address   string
	Prefix    string
	BatchSize int
}

type Service

type Service struct {
	ID    string  `json:"id"`
	Memo  string  `json:"memo"`
	Name  string  `json:"name"`
	OrgID string  `json:"orgId"`
	Roles []*Role `json:"roles"`
}

type WebhookBody

type WebhookBody struct {
	OrgName  string   `json:"orgName"`
	Event    string   `json:"event"`
	ImageURL string   `json:"imageUrl"`
	Memo     string   `json:"memo"`
	Host     *Host    `json:"host,omitempty"`
	Service  *Service `json:"service,omitempty"`
	Alert    *Alert   `json:"alert"`
}