ecs-drain-lambda
Based on the original idea from AWS Blog post and GitHub. With the following differences:
Why?
When updating AMI for the ECS instances then ASG replaces them without "Draining" , which may cause a short downtime of deployed containers. This function automates the ECS Cluster Instances Drain process.
How does it work?
ecs-drain-lambda function:
-
Receives CloudWatch event:
- ANY AutoScaling Lifecycle Terminate event ( EC2 Auto Scaling Lifecycle Hooks for
autoscaling:EC2_INSTANCE_TERMINATING
event should be configured on your ASG ) from CloudWatch Events
or
- ANY Spot Instance Interruption Notice. Imporatant, AWS doesn't guarantees that instance will be drained in time, instance could be terminated before the notice arrival.
-
Gets the ID of the instance that has to be terminated
-
Looks for the ECS Cluster name in the UserData in the following format: ECS_CLUSTER=xxxxxxxxx
-
If some ECS Tasks are running on the instance, starts the Drain
process
-
Waits for all the ECS Tasks to shutdown
-
Completes Lifecycle Hook, which lets ASG proceed with instance termination
Requirements
-
Serverless Framework
-
Golang
-
GNU Make
-
Configured EC2 Auto Scaling Lifecycle Hooks for autoscaling:EC2_INSTANCE_TERMINATING
event on your ASG
Example CloudFormation resource:
ASGTerminateHook:
Type: "AWS::AutoScaling::LifecycleHook"
Properties:
AutoScalingGroupName: !Ref ECSAutoScalingGroup
DefaultResult: "ABANDON"
HeartbeatTimeout: "900"
LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING"
How to use
Note: by default us-east-1
region is selected, if you need to deploy it to the
different region you can use sls deploy -v --region ${AWS_REGION}
Limitations
-
Function waits for 15 minutes for Drain to complete and fails with the timeout after
-
If function fails, then the default lifecycle hook action will be triggered (ABANDON
or CONTINUE
depending on your Hook configuration), either result will end up with eventual instance termination.
Documentation
If the instance is terminating, both ABANDON and CONTINUE allow the instance to terminate. However, ABANDON stops any remaining actions, such as other lifecycle hooks, while CONTINUE allows any other lifecycle hooks to complete.