shutdown

README

shutdown

Providing shutdown callbacks for graceful app shutdown

Motivation

We've opensourced this library, because we wanted a clean pattern built into our applications for handling certian node state via callbacks before it's shut down.

In our case we've configured AWS's autoscaling to publish a termination message on SQS on a autoscale scale-in event. Our app listens for such a message and upon recieving it, executes shutdown callbacks. In the mean time the application is letting autoscaler know it's node is still alive by emiting heartbeats and finally once the callbacks are executed, it let's autoscaler know it can proceed with shutting down the node.

We decided to use the same callback pattern in case of handling POSIX signals.

Installation

go get github.com/marmotedu/iam/pkg/shutdown

Documentation

github.com/marmotedu/iam/pkg/shutdown documentation is available on godoc.

Both ShutdownManagers are also documented:

• PosixSignalManager
• AwsManager

Example - AWS Autoscale, Scale-in Event

Graceful shutdown will listen for SQS messages on example-sqs-queue. If a termination message has current EC2 instance id, it will run all callbacks in separate go routines.

While callbacks are running, it will call aws api RecordLifecycleActionHeartbeatInput autoscaler every 15 minutes. When callbacks are finished, the application will call aws api CompleteLifecycleAction. The callback will delay only, if shutdown was initiated by awsmanager. If the message does not have current instance id, it will forward the message to correct instance via http on port 7999.

package main

import (
	"fmt"
	"time"

	"github.com/marmotedu/iam/pkg/shutdown"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/awsmanager"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
	// initialize shutdown with ping time
	gs := shutdown.New()

	// add posix shutdown manager
	gs.AddShutdownManager(posixsignal.NewPosixSignalManager())

	// set error handler
	gs.SetErrorHandler(shutdown.ErrorFunc(func(err error) {
		fmt.Println("Error:", err)
	}))

	// add aws shutdown manager
	gs.AddShutdownManager(awsmanager.NewAwsManager(&awsmanager.AwsManagerConfig{
		SqsQueueName:      "example-sqs-queue",
		LifecycleHookName: "example-lifecycle-hook",
		Port:              7999,
	}))

	// add your tasks that implement ShutdownCallback
	gs.AddShutdownCallback(shutdown.ShutdownFunc(func(shutdownManager string) error {
		fmt.Println("Shutdown callback start")
		if shutdownManager == awsmanager.Name {
			time.Sleep(time.Hour)
		}
		fmt.Println("Shutdown callback finished")
		return nil
	}))

	// start shutdown managers
	if err := gs.Start(); err != nil {
		fmt.Println("Start:", err)
		return
	}

	// do other stuff
	time.Sleep(time.Hour * 2)
}

Example - POSIX signals

Graceful shutdown will listen for posix SIGINT and SIGTERM signals. When they are received it will run all callbacks in separate go routines. When callbacks return, the application will exit with os.Exit(0)

package main

import (
	"fmt"
	"time"

	"github.com/marmotedu/iam/pkg/shutdown"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
	// initialize shutdown
	gs := shutdown.New()

	// add posix shutdown manager
	gs.AddShutdownManager(posixsignal.NewPosixSignalManager())

	// add your tasks that implement ShutdownCallback
	gs.AddShutdownCallback(shutdown.ShutdownFunc(func(string) error {
		fmt.Println("Shutdown callback start")
		time.Sleep(time.Second)
		fmt.Println("Shutdown callback finished")
		return nil
	}))

	// start shutdown managers
	if err := gs.Start(); err != nil {
		fmt.Println("Start:", err)
		return
	}

	// do other stuff
	time.Sleep(time.Hour)
}

Example - posix signals with error handler

The same as above, except now we set an ErrorHandler that prints the error returned from ShutdownCallback.

package main

import (
	"fmt"
	"time"
	"errors"

	"github.com/marmotedu/iam/pkg/shutdown"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
	// initialize shutdown
	gs := shutdown.New()

	// add posix shutdown manager
	gs.AddShutdownManager(posixsignal.NewPosixSignalManager())

	// set error handler
	gs.SetErrorHandler(shutdown.ErrorFunc(func(err error) {
		fmt.Println("Error:", err)
	}))

	// add your tasks that implement ShutdownCallback
	gs.AddShutdownCallback(shutdown.ShutdownFunc(func(string) error {
		fmt.Println("Shutdown callback start")
		time.Sleep(time.Second)
		fmt.Println("Shutdown callback finished")
		return errors.New("my-error")
	}))

	// start shutdown managers
	if err := gs.Start(); err != nil {
		fmt.Println("Start:", err)
		return
	}

	// do other stuff
	time.Sleep(time.Hour)
}

Licence

See LICENCE file in the root of the repository.

Documentation

Overview

Package shutdown Providing shutdown callbacks for graceful app shutdown

Installation

To install run:

go get github.com/marmotedu/iam/pkg/shutdown

Example - posix signals

Graceful shutdown will listen for posix SIGINT and SIGTERM signals. When they are received it will run all callbacks in separate go routines. When callbacks return, the application will exit with os.Exit(0)

package main

import (
	"fmt"
	"time"

	"github.com/marmotedu/iam/pkg/shutdown"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
	// initialize shutdown
	gs := shutdown.New()

	// add posix shutdown manager
	gs.AddShutdownManager(posixsignal.NewPosixSignalManager())

	// add your tasks that implement ShutdownCallback
	gs.AddShutdownCallback(shutdown.ShutdownFunc(func(string) error {
		fmt.Println("Shutdown callback start")
		time.Sleep(time.Second)
		fmt.Println("Shutdown callback finished")
		return nil
	}))

	// start shutdown managers
	if err := gs.Start(); err != nil {
		fmt.Println("Start:", err)
		return
	}

	// do other stuff
	time.Sleep(time.Hour)
}

Example - posix signals with error handler

The same as above, except now we set an ErrorHandler that prints the error returned from ShutdownCallback.

package main

import (
	"fmt"
	"time"
	"errors"

	"github.com/marmotedu/iam/pkg/shutdown"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
	// initialize shutdown
	gs := shutdown.New()

	// add posix shutdown manager
	gs.AddShutdownManager(posixsignal.NewPosixSignalManager())

	// set error handler
	gs.SetErrorHandler(shutdown.ErrorFunc(func(err error) {
		fmt.Println("Error:", err)
	}))

	// add your tasks that implement ShutdownCallback
	gs.AddShutdownCallback(shutdown.ShutdownFunc(func(string) error {
		fmt.Println("Shutdown callback start")
		time.Sleep(time.Second)
		fmt.Println("Shutdown callback finished")
		return errors.New("my-error")
	}))

	// start shutdown managers
	if err := gs.Start(); err != nil {
		fmt.Println("Start:", err)
		return
	}

	// do other stuff
	time.Sleep(time.Hour)
}

Example - aws

Graceful shutdown will listen for SQS messages on "example-sqs-queue". If a termination message has current EC2 instance id, it will run all callbacks in separate go routines. While callbacks are running it will call aws api RecordLifecycleActionHeartbeatInput autoscaler every 15 minutes. When callbacks return, the application will call aws api CompleteLifecycleAction. The callback will delay only if shutdown was initiated by awsmanager. If the message does not have current instance id, it will forward the message to correct instance via http on port 7999.

package main

import (
	"fmt"
	"time"

	"github.com/marmotedu/iam/pkg/shutdown"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/awsmanager"
	"github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
	// initialize shutdown with ping time
	gs := shutdown.New()

	// add posix shutdown manager
	gs.AddShutdownManager(posixsignal.NewPosixSignalManager())

	// set error handler
	gs.SetErrorHandler(shutdown.ErrorFunc(func(err error) {
		fmt.Println("Error:", err)
	}))

	// add aws shutdown manager
	gs.AddShutdownManager(awsmanager.NewAwsManager(&awsmanager.AwsManagerConfig{
		SqsQueueName:      "example-sqs-queue",
		LifecycleHookName: "example-lifecycle-hook",
		Port:              7999,
	}))

	// add your tasks that implement ShutdownCallback
	gs.AddShutdownCallback(shutdown.ShutdownFunc(func(shutdownManager string) error {
		fmt.Println("Shutdown callback start")
		if shutdownManager == awsmanager.Name {
			time.Sleep(time.Hour)
		}
		fmt.Println("Shutdown callback finished")
		return nil
	}))

	// start shutdown managers
	if err := gs.Start(); err != nil {
		fmt.Println("Start:", err)
		return
	}

	// do other stuff
	time.Sleep(time.Hour * 2)
}

Index

• type ErrorFunc
  • func (f ErrorFunc) OnError(err error)
• type ErrorHandler
• type GSInterface
• type GracefulShutdown
  • func New() *GracefulShutdown
  • func (gs *GracefulShutdown) AddShutdownCallback(shutdownCallback ShutdownCallback)
  • func (gs *GracefulShutdown) AddShutdownManager(manager ShutdownManager)
  • func (gs *GracefulShutdown) ReportError(err error)
  • func (gs *GracefulShutdown) SetErrorHandler(errorHandler ErrorHandler)
  • func (gs *GracefulShutdown) Start() error
  • func (gs *GracefulShutdown) StartShutdown(sm ShutdownManager)
• type ShutdownCallback
• type ShutdownFunc
  • func (f ShutdownFunc) OnShutdown(shutdownManager string) error
• type ShutdownManager

Types

type ErrorFunc

type ErrorFunc func(err error)

ErrorFunc is a helper type, so you can easily provide anonymous functions as ErrorHandlers.

func (ErrorFunc) OnError

func (f ErrorFunc) OnError(err error)

OnError defines the action needed to run when error occurred.

type ErrorHandler

type ErrorHandler interface {
	OnError(err error)
}

ErrorHandler is an interface you can pass to SetErrorHandler to handle asynchronous errors.

type GSInterface

type GSInterface interface {
	StartShutdown(sm ShutdownManager)
	ReportError(err error)
	AddShutdownCallback(shutdownCallback ShutdownCallback)
}

GSInterface is an interface implemented by GracefulShutdown, that gets passed to ShutdownManager to call StartShutdown when shutdown is requested.

type GracefulShutdown

type GracefulShutdown struct {
	// contains filtered or unexported fields
}

GracefulShutdown is main struct that handles ShutdownCallbacks and ShutdownManagers. Initialize it with New.

func New

func New() *GracefulShutdown

New initializes GracefulShutdown.

func (*GracefulShutdown) AddShutdownCallback

func (gs *GracefulShutdown) AddShutdownCallback(shutdownCallback ShutdownCallback)

AddShutdownCallback adds a ShutdownCallback that will be called when shutdown is requested.

You can provide anything that implements ShutdownCallback interface, or you can supply a function like this:

AddShutdownCallback(shutdown.ShutdownFunc(func() error {
	// callback code
	return nil
}))

func (*GracefulShutdown) AddShutdownManager

func (gs *GracefulShutdown) AddShutdownManager(manager ShutdownManager)

AddShutdownManager adds a ShutdownManager that will listen to shutdown requests.

func (*GracefulShutdown) ReportError

func (gs *GracefulShutdown) ReportError(err error)

ReportError is a function that can be used to report errors to ErrorHandler. It is used in ShutdownManagers.

func (*GracefulShutdown) SetErrorHandler

func (gs *GracefulShutdown) SetErrorHandler(errorHandler ErrorHandler)

SetErrorHandler sets an ErrorHandler that will be called when an error is encountered in ShutdownCallback or in ShutdownManager.

You can provide anything that implements ErrorHandler interface, or you can supply a function like this:

SetErrorHandler(shutdown.ErrorFunc(func (err error) {
	// handle error
}))

func (*GracefulShutdown) Start

func (gs *GracefulShutdown) Start() error

Start calls Start on all added ShutdownManagers. The ShutdownManagers start to listen to shutdown requests. Returns an error if any ShutdownManagers return an error.

func (*GracefulShutdown) StartShutdown

func (gs *GracefulShutdown) StartShutdown(sm ShutdownManager)

StartShutdown is called from a ShutdownManager and will initiate shutdown. first call ShutdownStart on Shutdownmanager, call all ShutdownCallbacks, wait for callbacks to finish and call ShutdownFinish on ShutdownManager.

type ShutdownCallback

type ShutdownCallback interface {
	OnShutdown(string) error
}

ShutdownCallback is an interface you have to implement for callbacks. OnShutdown will be called when shutdown is requested. The parameter is the name of the ShutdownManager that requested shutdown.

type ShutdownFunc

type ShutdownFunc func(string) error

ShutdownFunc is a helper type, so you can easily provide anonymous functions as ShutdownCallbacks.

func (ShutdownFunc) OnShutdown

func (f ShutdownFunc) OnShutdown(shutdownManager string) error

OnShutdown defines the action needed to run when shutdown triggered.

type ShutdownManager

type ShutdownManager interface {
	GetName() string
	Start(gs GSInterface) error
	ShutdownStart() error
	ShutdownFinish() error
}

ShutdownManager is an interface implemnted by ShutdownManagers. GetName returns the name of ShutdownManager. ShutdownManagers start listening for shutdown requests in Start. When they call StartShutdown on GSInterface, first ShutdownStart() is called, then all ShutdownCallbacks are executed and once all ShutdownCallbacks return, ShutdownFinish is called.