nomad-custodian

README

Nomad Custodian

Inspired by Cloud Custodian, this simple CLI will help Nomad administrators manage job resources with cost optimization and maintenance in mind.

Features

• Scale in all job task group counts to count=1 during off business hours
• Scale out all jobs to original counts
• Delete all jobs
• Backup all jobs as JSON files

How to use

list

Running nomad-custodian list will list the meta tags and current task group counts for each job.

$ nomad-custodian list
Number of jobs running: 4

+  Job: couchbase    Status: running
   Field             Value
   Count             2
+  Job: demo-webapp  Status: running
   Field             Value
   Count             3
+  Job: example      Status: running
   Field             Value
   Count             2
+  Job: nginx        Status: pending
   Field             Value
   Count             2
   custodian-ignore  1

Listing Batch Type Jobs

nomad-custodian list --job-type batch

+  Job: ynab-bitcoin-sync  Status: running
   Field                   Value
   Count                   1
   Periodic                */30 * * * *     Every 30 minutes

scale-in

Excluding --force or -f with the scale-in and scale-out commands will provide a preview of what will change. For example, running nomad-custodian scale-in will provide the below output.

$ nomad-custodian scale-in
Job: couchbase, running
  What's Changing                  From  To
  Meta[custodian-action]                 scaled-in
  Meta[custodian-couchbase-count]        2
  Meta[custodian-revert-version]         1
  Count                            2     1

Job: demo-webapp, running
  What's Changing                 From  To
  Meta[custodian-action]                scaled-in
  Meta[custodian-demo-count]            3
  Meta[custodian-revert-version]        2
  Count                           3     1

Job: example, running
  What's Changing                 From  To
  Meta[custodian-action]                scaled-in
  Meta[custodian-cache-count]           2
  Meta[custodian-revert-version]        2
  Count                           2     1

Jobs Skipped  Scale Status  Ignore
nginx                       true

Including the --force flag will produce similar output as the plan but the changes will take place.

$ nomad-custodian scale-in --force
Job: couchbase, running
  What's Changing                  From  To
  Meta[custodian-action]                 scaled-in
  Meta[custodian-couchbase-count]        2
  Meta[custodian-revert-version]         1
  Count                            2     1

Job: demo-webapp, running
  What's Changing                 From  To
  Meta[custodian-action]                scaled-in
  Meta[custodian-demo-count]            3
  Meta[custodian-revert-version]        2
  Count                           3     1

Job: example, running
  What's Changing                 From  To
  Meta[custodian-action]                scaled-in
  Meta[custodian-cache-count]           2
  Meta[custodian-revert-version]        2
  Count                           2     1

Jobs Skipped  Scale Status  Ignore
nginx                       true

scale-out

The scale-out command is similar to the scale-in command in terms of output.

$ nomad-custodian scale-out -f
Job: couchbase, running
  What's Changing                  From       To
  Meta[custodian-action]           scaled-in
  Meta[custodian-couchbase-count]  2
  Meta[custodian-revert-version]   1
  Count                            1          2

Job: demo-webapp, running
  What's Changing                 From       To
  Meta[custodian-action]          scaled-in
  Meta[custodian-demo-count]      3
  Meta[custodian-revert-version]  2
  Count                           1          3

Job: example, running
  What's Changing                 From       To
  Meta[custodian-action]          scaled-in
  Meta[custodian-cache-count]     2
  Meta[custodian-revert-version]  2
  Count                           1          2

Jobs Skipped  Scale Status  Ignore
nginx                       true

backup-jobs

The backup-jobs command provides an easy way to locally backup all the jobs registered in Nomad as JSON files. A new time stamped directory is created each time the command is executed.

$ nomad-custodian backup-jobs
mkdir jobs-backup: file exists
Job couchbase written to couchbase.json
Job demo-webapp written to demo-webapp.json
Job example written to example.json
Job nginx written to nginx.json

$ ls jobs-backup/1578492852
couchbase.json   demo-webapp.json example.json     nginx.json

delete-all-jobs

The delete-all-jobs helps make bulk deregistering of jobs (and purging if --purge or -p is included) from Nomad.

nomad-custodian delete-all-jobs -f -p
Are you sure you want to continue? (y/N): y
Job couchbase deregister response: ece44f6c-e518-bbbe-7f06-41ee4f3b61c8Action: Deregister, Job: couchbase
Job demo-webapp deregister response: 97f82a9d-ddd1-dc31-1be6-e5e81440b00fAction: Deregister, Job: demo-webapp
Job example deregister response: b8c9885e-d87c-9d2c-fbdc-2b1f42a57422Action: Deregister, Job: example

Jobs Skipped  Scale Status  Ignore
nginx                       true

Safety Controls

Prevent any custodian actions:

Nothing available at the moment

Prevent changes on specific jobs:

job "nginx" {
  datacenters = ["dc1"]

  meta {
    nomad-custodian-ignore = true
  }
  ...

Development

To build the binary:

make build

Log Level

Log level can be set by using the below environment variable.

export CRON_DESCRIPTOR_LOG_LEVEL=debug

Improvement / Feature Ideas

• Service and UI components
  • Service could be deployed to the same cluster or a management cluster
  • UI would provide same functionality as the CLI
• Filtering capabilities
  • Namespaces
  • Job names
  • Time of day
• Globally prevent custodian changes
  • Enforce with Consul KV check
  • Enforce some other way with Nomad

Documentation

Overview

Copyright © 2020 John Suarez jsuar@users.noreply.github.com

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.