sip

README

go-scale-in-protection

Monitor workers' statuses to enable or disable instance scale-in protection accordingly.

Inspiration

The need for this monitor arises from https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html#scale-sqs-queue-scale-in-protection:

while (true)
{
  SetInstanceProtection(False);
  Work = GetNextWorkUnit();
  SetInstanceProtection(True);
  ProcessWorkUnit(Work);
  SetInstanceProtection(False);
}

Essentially, if you have any number of workers who can be either ACTIVE or IDLE, you generally want to enable scale-in protection when any of your worker is actively doing some work, while once all the workers have become idle, you would want to disable scale-in protection to let the Auto Scaling group reclaim your instance naturally.

Note: there is a possibility that your instance is terminated in-between the GetWorkUnit() and the ProcessWorkUnit(Work) calls. Generally if your visibility timeout is low enough, this is not an issue as a different worker would be able to pick up the message again.

Usage

There is only one very simple ScaleInProtector struct. You can declare a dependency on the package and use it; and I would recommend copying, pasting, and customising it to your needs as well.

go get github.com/nguyengg/go-scale-in-protection

package main

import (
	"context"
	"errors"
	sip "github.com/nguyengg/go-scale-in-protection"
	"log"
	"os"
	"os/signal"
	"strconv"
	"sync"
	"syscall"
	"time"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Kill, os.Interrupt, syscall.SIGTERM)
	defer stop()

	s := &sip.ScaleInProtector{
		// both InstanceId and AutoScalingGroupName are optional.
		// the monitor knows how to use the default aws.Config to figure out its own instance Id (via IMDSv2) and the
		// Auto Scaling group name containing that instance id.
		InstanceId:           "i-1234",
		AutoScalingGroupName: "my-asg",
		// when the last worker becomes idle, wait for another 15" before marking the instance as safe to terminate.
		IdleAtLeast: 15 * time.Minute,
	}

	// you can start the workers first or start the monitor's main loop first, either works.
	// the monitor doesn't need to know beforehand how many workers there are but you should signal active at least once
	// because the monitor starts out assuming all workers are idle.
	workerCount := 5
	var wg sync.WaitGroup
	for i := range workerCount {
		wg.Add(1)
		go func(id string) {
			defer wg.Done()

			for {
				// get some work.

				// mark active.
				s.SignalActive(id)

				// do the work.

				// mark idle.
				s.SignalIdle(id)
			}
		}(strconv.Itoa(i))
	}

	// always starts the monitor's main loop in a goroutine because it doesn't return until context is cancelled or it
	// encounters an error.
	wg.Add(1)
	go func() {
		if err := s.StartMonitoring(ctx); err != nil {
			if errors.Is(err, context.Canceled) {
				return
			}

			log.Fatal(err)
		}
	}()

	wg.Wait()
}

Documentation

Index

• type AutoScalingAPIClient
• type ScaleInProtector
  • func (s *ScaleInProtector) IsProtectedFromScaleIn() bool
  • func (s *ScaleInProtector) SignalActive(id string)
  • func (s *ScaleInProtector) SignalIdle(id string)
  • func (s *ScaleInProtector) StartMonitoring(ctx context.Context) (err error)

Types

type AutoScalingAPIClient

type AutoScalingAPIClient interface {
	DescribeAutoScalingInstances(context.Context, *autoscaling.DescribeAutoScalingInstancesInput, ...func(*autoscaling.Options)) (*autoscaling.DescribeAutoScalingInstancesOutput, error)
	SetInstanceProtection(context.Context, *autoscaling.SetInstanceProtectionInput, ...func(*autoscaling.Options)) (*autoscaling.SetInstanceProtectionOutput, error)
}

AutoScalingAPIClient extracts the subset of autoscaling.Client APIs that ScaleInProtector uses.

type ScaleInProtector

type ScaleInProtector struct {
	// InstanceId is the instance Id to enable or disable scale-in protection.
	//
	// If not specified, an imds.Client created from the default aws.Config (config.LoadDefaultConfig) will be used to
	// detect the instance Id of the host. If one cannot be detected, StartMonitoring will return a non-nil error.
	InstanceId string
	// AutoScalingGroupName is the name of the Auto Scaling group that contains the instance specified by InstanceId.
	//
	// If not specified, the given AutoScaling will be used to find the Auto Scaling group that contains the instance
	// specified by InstanceId. If one cannot be found, StartMonitoring will return a non-nil error. If both InstanceId
	// and AutoScalingGroupName are given but the Auto Scaling group does not contain the instance, an error is also
	// returned (you cannot have the monitor effects an instance different from the one it's running on for safety). If
	// you want an option to disable this check, send a PR.
	AutoScalingGroupName string
	// AutoScaling is the client that will be used to make Auto Scaling service calls.
	//
	// If not given, an autoscaling.Client created from the default aws.Config (config.LoadDefaultConfig) will be used.
	AutoScaling AutoScalingAPIClient
	// IdleAtLeast specifies the amount of time all workers must have been idle before scale-in protection may be
	// disabled.
	//
	// The delay starts from when the last worker becomes idle. If you want to measure from the moment the first worker
	// becomes idle, send a PR (trailing vs. leading delay).
	IdleAtLeast time.Duration
	// Logger is used to log whenever the scale-in protection changes.
	//
	// Defaults to log.Default.
	Logger *log.Logger
	// contains filtered or unexported fields
}

ScaleInProtector monitors active statuses of several workers to enable or disable scale-in protection accordingly.

The zero value ScaleInProtector is ready for use. StartMonitoring should be called in a separate goroutine to start the monitoring loop. Each worker then calls either SignalActive or SignalIdle at the appropriate time, passing the worker identifier as the sole argument.

func (*ScaleInProtector) IsProtectedFromScaleIn

func (s *ScaleInProtector) IsProtectedFromScaleIn() bool

IsProtectedFromScaleIn returns the internal flag reflecting whether scale-in protection is enabled or not.

It is entirely possible for the monitor to think it has scale-in protection enabled while an external action may have disabled it and vice versa.

func (*ScaleInProtector) SignalActive

func (s *ScaleInProtector) SignalActive(id string)

SignalActive should be called by a worker passing its identifier when it has an active job.

It is safe to call this method even if StartMonitoring hasn't been called. The method may block if there are concurrent calls to either SignalActive and SignalIdle. Once StartMonitoring has been called, the method may also block until the scale-in protection change has been effected.

func (*ScaleInProtector) SignalIdle

func (s *ScaleInProtector) SignalIdle(id string)

SignalIdle should be called by a worker passing its identifier when it has become idle.

It is safe to call this method even if StartMonitoring hasn't been called. The method may block if there are concurrent calls to either SignalActive and SignalIdle. Once StartMonitoring has been called, the method may also block until the scale-in protection change has been effected.

func (*ScaleInProtector) StartMonitoring

func (s *ScaleInProtector) StartMonitoring(ctx context.Context) (err error)

StartMonitoring starts the monitoring loop.

The method should be called in a separate goroutine because it will not return until the given context is cancelled; context.Context.Err is always returned in this case. Errors from making Auto Scaling service calls also cause the loop to terminate.

Panics if StartMonitoring has been called more than once.