sredocs
sredocs is to turn postmortems, charters and other SRE docs into machine readable data for analysis while retaining human readability properties.
This repository contains code for downloading postmortems and/or charters following certain template(s) from Google Docs, parsing the data to CSV and support for uploading the result to Google BigQuery.
Here is a demo dashboard for incident analysis using this, Google Data Studio and BigQuery. It contains an interactive report with information related to five dummy postmortems.
sredocs currently parses postmortems using a single specific template (sample) and charter that will be published separately later.
Download
Here is how to use the download mode:
- Go to https://console.developers.google.com. I'd recommend creating a new project called "sredocs".
- Switch to that project. Search for "Google Drive API" then enable it.
- Go to APIs & Services -> Credentials.
- Click on "Create credentials" then select Service account key.
- Give your Service account key a name, e.g "sredocs".
- Take note of its address (looks like an email address). Do not assign it a role while creating it.
- Save the private key (it will be automatically downloaded).
- Switch to Google Drive.
- Give your service account address, e.g sredocs@...iam.gservice... view access to the folder with SRE docs.
- Pass the private key to sredocs via -download_credentials_path.
- Run
mkdir <download_dir>
- Run
sredocs -mode=download -cloud_credentials=<creds.json> -download_folder=<docs_folder> -download_destination=<download_dir>
Parse
Here is how to use the parse mode:
- Run
mkdir <parsed_dir>
- Run
sredocs -mode=parse -parse_kind=auto -parse_path=<download_dir> -parse_output_path=<parsed_dir>
Upload
Here is how to use the upload mode:
- Follow the instructions for Download first.
- Go to https://console.developers.google.com.
- Navigate to IAM & Admin.
- Click on the Add button.
- Search for your service account, e.g sredocs@... Give it the BigQuery Admin role.
- Run
sredocs -mode=upload -cloud_credentials=<creds.json> -upload_path=<parsed_dir> -upload_project=<org:project> -upload_dataset=<sredocs> -upload_table=<sredocs_20190603> -upload_truncate=true
BYOP
You can also bring your own parser.
type Parser interface {
CompileRegex(fields []string) ([]*regexp.Regexp, error)
Parse(fields []string, b []byte) (*bytes.Buffer, error)
CSVHeader(regexps []*regexp.Regexp) []string
NamedGroup(field string) string
Save(buf *bytes.Buffer, filename string) error
}
You will have to modify charter/* and postmortem/* to switch away from the DefaultParser.
You can also extend sredocs by adding a new kind of document by using charter/* as a template.