libra

README

Libra

Libra autoscales Nomad task groups so you don't have to. View the API documentation here.

Design

Libra takes heavy inspiration from the design of Nomad itself and uses a client/server model. Much like Nomad, the Libra CLI makes HTTP API calls to the Libra server.

The skeleton of this project is from jippi/nomad-auto-scale, and could not have been completed without the architecture exemplified there.

How do I add a command?

1. Add an API endpoint in /api/.
2. Register the endpoint with the server in /command/server.go
3. Document the endpoint in /api/README.md
4. Add a new command in /command/ that calls the API endpoint.
5. Register the command with the CLI in /commands.go

Todo:

• Randomly stagger cron jobs to avoid conflict
• Improve configuration management (perhaps add a submission API)
• Handle Nomad errors more robustly

Configuration

You can (and probably should) configure five environment variables as well, LIBRA_ADDR, LIBRA_CONFIG_DIR, GRAPHITE_PASSWORD, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY.

Libra gets most of its configuration from HCL config files located in a config directory (default /etc/libra). Here's an example config.hcl file:

// Nomad Client configuration
nomad {
  address = "http://localhost:4646"
}

backend "test-backend" {
  kind     = "cloudwatch"
  region   = "us-east-1"
}

backend "other-backend" {
  kind     = "graphite"
  host     = "https://my-grafana.hosted-metrics.grafana.net"
  username = "api_key"  
}

// Scale for the job "nginx-prod"
// job and group must correspond to a valid Nomad job and group that is running in the Nomad cluster
job "nginx-prod" {
  // For group "nginx"
  group "nginx" {
    // (required) The minimum nuber of tasks to run for this job
    min_count = 1

    // (required) The maximum number of tasks to run for this job
    max_count = 3

    // Scale by a rule
    rule "cloudwatch asg cpu usage upper bound" {
      // (required) What backend to use, this will define which configuration
      // is valid and which checks you can execute
      backend = "test-backend"

      // (required) The CloudWatch dimensions as a map of name and value
      dimensions = {
          "AutoScalingGroupName" = "infra-httpapi-asg",
      }     

      // (required) The CloudWatch metric name and namespace
      metric_namespace = "AWS/EC2"
      metric_name = "CPUUtilization"

      // (required) The comparison to check, one of:
      //
      //   - above (>)
      //   - below (<)
      //   - equal (==)
      //   - not_equal (!=)
      //   - above_or_equal (>=)
      //   - below_or_equal (<=)
      comparison = "above_or_equal"

      // (required) The value to compare to, this should be a float
      comparison_value = 90.0

      // (optional) How often this rule should be checked, by default it will be checked every minute
      cron = "* * * * *"

      action       = "increase_count"
      action_value = 1
    }

    rule "cloudwatch asg cpu usage lower bound" {
      backend          = "test-backend"
      dimensions = {
        "AutoScalingGroupName" = "infra-httpapi-asg",
      }
      metric_namespace = "AWS/EC2"
      metric_name      = "CPUUtilization"
      comparison       = "below"
      comparison_value = 20.0
      cron             = "* * * * *"
      action           = "decrease_count"
      action_value     = 1
    }
    
    rule "graphite nomad statsd cpu lower bound" {
      backend          = "other-backend"
      metric_name      = "stats.*.nomad.*.domain.*.allocs.statsd.statsd.*.*.cpu.total_percent"
      comparison       = "below"
      comparison_value = 20.0
      cron             = "* * * * *"
      action           = "decrease_count"
      action_value     = 1
    }
  }
}