chgm

README

ClusterHasGoneMissing Investigation

Alert firing investigation

1. PagerDuty webhook receives CHGM alert from Dead Man's Snitch.
2. CAD Tekton pipeline is triggered via PagerDuty sending a webhook to Tekton EventListener.
3. Logs into AWS account of cluster and checks for stopped/terminated instances.
   • If unable to access AWS account, posts "cluster credentials are missing" limited support reason.
4. If stopped/terminated instances are found, pulls AWS CloudTrail events for those instances.
   • If no stopped/terminated instances are found, escalates to SRE for further investigation.
5. If the user of the event is:
   • Authorized (SRE or OSD managed), runs the network verifier and escalates the alert to SRE for futher investigation.
     • Note: Authorized users have prefix RH-SRE, osdManagedAdmin, or have the ManagedOpenShift-Installer-Role.
   • Not authorized (not SRE or OSD managed), posts the appropriate limited support reason and silences the alert.
6. Adds notes with investigation details to the PagerDuty alert.

CHGM investigation overview

Documentation

Overview

Package chgm contains functionality for the chgm investigation

Index

• func Investigate(r *investigation.Resources) error
• type CloudTrailEventRaw

Functions

func Investigate

func Investigate(r *investigation.Resources) error

Investigate runs the investigation for a triggered chgm pagerduty event

Types

type CloudTrailEventRaw

type CloudTrailEventRaw struct {
	EventVersion string `json:"eventVersion"`
	UserIdentity struct {
		Type           string `json:"type"`
		SessionContext struct {
			SessionIssuer struct {
				Type     string `json:"type"`
				UserName string `json:"userName"`
			} `json:"sessionIssuer"`
		} `json:"sessionContext"`
	} `json:"userIdentity"`
}

CloudTrailEventRaw will help marshal the cloudtrail.Event.CloudTrailEvent string TODO: tidy uo the struct when we know exactly what we need