shutter

README

shutter

A single daemon based graceful shutdown manager for EC2 instances under an Autoscaling Group and a Lifecycle Hook.

How to use

It accepts basic settings via a yaml file. You can specify the file path by -file. And you can also run shutter as a daemon by using -daemon.

$ ./shutter -h
Usage of ./shutter:
  -daemon
        do as daemon
  -file string
        a config file path
  -instanceid string
        EC2 instance id (optional, used if daemon = false)

The yaml configuration layout is here: Especially about command part, you can use these replacement will be replaced by actual instance values on runtime.

• ${INSTANCE_ID}, an EC2 instance id
• ${PUBLIC_IP_ADDRESS}, a public IP address for an EC2 instance
• ${PRIVATE_IP_ADDRESS}, a private IP address for an EC2 instance

aws_region: us-east-1
watcher:
  autoscaling_group_name: test
  interval_sec: 60
finisher:
  lifecycle_hook_name: test
  lifecycle_action_result: ABANDON
  terminate:
    # it should have idempotency
    command: "ping -c 5 ${PUBLIC_IP_ADDRESS}"
  wait:
    command: "ping -c 5 ${PUBLIC_IP_ADDRESS}"
    interval_sec: 60
    max_tries: 30

Required minimum IAM policy

autoscaling:CompleteLifecycleAction
autoscaling:DescribeAutoScalingInstances
autoscaling:DescribeAutoScalingGroups
ec2:DescribeInstances

shutter internals

State management

shutter manages terminating instance by a structure like FSM. It has below states:

• init

  • noop

• terminate

  • Run a command to start termination. e.g. trigger graceful shutdown scripts

• wait

  • Wait until a command is succeeded. e.g. trigger script checks there's no active client

• complete

  • Complete an instance on an autoscaling group

• finished

  • noop, all tasks are succeeded

• aborted

  • noop, any task failed

Documentation

Index

• Variables
• func DoCommand(commands string) (*os.ProcessState, error)
• func DoForever(client AwsClient, config *Config, logger *zap.Logger) error
• func DoOnce(client AwsClient, config *Config, logger *zap.Logger) error
• func DoOnceWithInstanceId(client AwsClient, config *Config, logger *zap.Logger, instanceId string) error
• func NewFinisher(client AwsClient, c *Config, logger *zap.Logger, instanceId string) *finisher
• func NewWatcher(client AwsClient, config *Config, logger *zap.Logger) *watcher
• func Render(commands string, instance *ec2.Instance) string
• type AwsClient
  • func NewAwsClient(config *Config) (AwsClient, error)
• type Config
  • func NewConfig(name string) (*Config, error)
• type Finisher
• type Terminate
• type Wait
• type Watcher

Variables

var (
	ErrInvalidAPIResponse = errors.New("AWS API response is invalid")
	ErrAlreadyTerminated  = errors.New("The instance has already terminated")
)

Functions

func DoCommand

func DoCommand(commands string) (*os.ProcessState, error)

func DoForever

func DoForever(client AwsClient, config *Config, logger *zap.Logger) error

func DoOnce

func DoOnce(client AwsClient, config *Config, logger *zap.Logger) error

func DoOnceWithInstanceId

func DoOnceWithInstanceId(client AwsClient, config *Config, logger *zap.Logger, instanceId string) error

func NewFinisher

func NewFinisher(client AwsClient, c *Config, logger *zap.Logger, instanceId string) *finisher

func NewWatcher

func NewWatcher(client AwsClient, config *Config, logger *zap.Logger) *watcher

func Render

func Render(commands string, instance *ec2.Instance) string

Types

type AwsClient

type AwsClient interface {
	DescribeAutoscalingGroup(name string) (*autoscaling.Group, error)
	DescribeInstance(instanceId string) (*ec2.Instance, error)
	DescribeInstanceDetails(instanceId string) (*autoscaling.InstanceDetails, error)
	CompleteLifecycleAction(instanceId string, lifecycleActionResult, lifecycleHook string) error
}

A minimum AWS client

func NewAwsClient

func NewAwsClient(config *Config) (AwsClient, error)

type Config

type Config struct {
	AwsRegion string   `yaml:"aws_region"`
	Watcher   Watcher  `yaml:"watcher"`
	Finisher  Finisher `yaml:"finisher"`
}

func NewConfig

func NewConfig(name string) (*Config, error)

type Finisher

type Finisher struct {
	LifecycleHookName     string    `yaml:"lifecycle_hook_name"`
	LifecycleActionResult string    `yaml:"lifecycle_action_result"`
	Terminate             Terminate `yaml:"terminate"`
	Wait                  Wait      `yaml:"wait"`
}

type Terminate

type Terminate struct {
	Command string `yaml:"command"`
}

type Wait

type Wait struct {
	Command     string        `yaml:"command"`
	IntervalSec time.Duration `yaml:"interval_sec"`
	MaxTries    int64         `yaml:"max_tries"`
}

type Watcher

type Watcher struct {
	AutoscalingGroupName string        `yaml:"autoscaling_group_name"`
	IntervalSec          time.Duration `yaml:"interval_sec"`
}