shutdown

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2017 License: MIT Imports: 10 Imported by: 10

README

shutdown

Shutdown management library for Go

This package helps you manage shutdown code centrally, and provides functionality to execute code when a controlled shutdown occurs.

This will enable you to save data, notify other services that your application is shutting down.

GoDoc Build Status

concept

Managing shutdowns can be very tricky, often leading to races, crashes and strange behavior. This package will help you manage the shutdown process and will attempt to fix some of the common problems when dealing with shutting down.

The shutdown package allow you to block shutdown while certain parts of your code is running. This is helpful to ensure that operations are not interrupted.

The second part of the shutdown process is notifying goroutines in a select loop and calling functions in your code that handles various shutdown procedures, like closing databases, notifying other servers, deleting temporary files, etc.

The second part of the process has three stages, which will enable you to do your shutdown in stages. This will enable you to rely on some parts, like logging, to work in the first two stages. There is no rules for what you should put in which stage, but things executing in stage one can safely rely on stage two not being executed yet.

All operations have timeouts. This is to fix another big issue with shutdowns; applications that hang on shutdown. The timeout is for each stage of the shutdown process, and can be adjusted to your application needs. If a timeout is exceeded the next shutdown stage will be initiated regardless.

Finally, you can always cancel a notifier, which will remove it from the shutdown queue.

changes

nil notifiers

It was tricky to detect cases where shutdown had started when you requested notifiers.

To help for that common case, the library now returns a nil Notifier if shutdown has already reached the stage you are requesting a notifier for.

This is backwards compatible, but makes it much easier to test for such a case:

    f := shutdown.First()
    if f == nil {
        // Already shutting down.
        return
    }
"context" support

Support for the context package has been added. This allows you to easily wrap shutdown cancellation to your contexts using shutdown.CancelCtx(parent Context).

This functions equivalent to calling context.WithCancel and you must release resources the same way, by calling the returned CancelFunc.

For legacy codebases we will seamlessly integrate with golang.org/x/net/context. Be sure to update to the latest version using go get -u golang.org/x/net/context, since Go 1.7 compatibility is a recent update.

v2

This is version 2 of the shutdown package. It contains some breaking changes to simplify the use of shutdown functions.

usage

First get the libary with go get -u github.com/klauspost/shutdown2, and add it as an import to your code with import github.com/klauspost/shutdown2.

The next thing you probably want to do is to register Ctrl+c and system terminate. This will make all shutdown handlers run when any of these are sent to your program:

	shutdown.OnSignal(0, os.Interrupt, syscall.SIGTERM)

If you don't like the default timeout duration of 5 seconds, you can change it by calling the SetTimeout function:

  shutdown.SetTimeout(time.Second * 1)

Now the maximum delay for shutdown is 4 seconds. The timeout is applied to each of the stages and that is also the maximum time to wait for the shutdown to begin. If you need to adjust a single stage, use SetTimeoutN function.

Next you can register functions to run when shutdown runs:

  logFile := os.Create("log.txt")

  // Execute the function in the first stage of the shutdown process
  _ = shutdown.FirstFn(func(){
    logFile.Close()
  })

  // Execute this function in the second part of the shutdown process
  _ = shutdown.SecondFn(func(){
    _ = os.Delete("log.txt")
  })

As noted there are three stages. All functions in one stage are executed in parallel. The package will wait for all functions in one stage to have finished before moving on to the next one.
So your code cannot rely on any particular order of execution inside a single stage, but you are guaranteed that the First stage is finished before any functions from stage two are executed.

You can send a parameter to your function, which is delivered as an interface{}. This way you can re-use the same function for similar tasks. See simple-func.go in the examples folder.

This example above uses functions that are called, but you can also request channels that are notified on shutdown. This allows you do have shutdown handling in blocked select statements like this:

  go func() {
    // Get a stage 1 notification
    finish := shutdown.First()
    select {
      case n:= <-finish:
        log.Println("Closing")
        close(n)
        return
  }

If you suspect that shutdown may already be running, you should check the returned notifier. If shutdown has already been initiated, and has reached or surpassed the stage you are requesting a notifier for, nil will be returned.

    // Get a stage 1 notification
    finish := shutdown.First()
    // If shutdown is at Stage 1 or later, nil will be returned 
    if finish == nil {
        log.Println("Already shutting down")
        return
    }
    select {
      case n:= <-finish:
        log.Println("Closing")
        close(n)
        return
  }

If you for some reason don't need a notifier anymore you can cancel it. When a notifier has been cancelled it will no longer receive notifications, and the shutdown code will no longer wait for it on exit.

  go func() {
    // Get a stage 1 notification
    finish := shutdown.First()
    select {
      case n:= <-finish:
        close(n)
        return
      case <-otherchan:
        finish.Cancel()
        return
  }

Functions are cancelled the same way by cancelling the returned notifier. Be aware that if shutdown has been initiated you can no longer cancel notifiers, so you may need to aquire a shutdown lock (see below).

If you want to Cancel a notifier, but shutdown may have started, you can use the CancelWait function. It will cancel a Notifier, or wait for it to become active if shutdown has been started.

If you get back a nil notifier because shutdown has already reached that stage, calling CancelWait will return at once.

  go func() {
    // Get a stage 1 notification
    finish := shutdown.First()    
    if finish == nil {
        return 
    }
    select {
      case n:= <-finish:
        close(n)
        return
      case <-otherchan:
        // Cancel the finish notifier, or wait until Stage 1 is complete.
        finish.CancelWait() 
        return
  }

The final thing you can do is to lock shutdown in parts of your code you do not want to be interrupted by a shutdown, or if the code relies on resources that are destroyed as part of the shutdown process.

A simple example can be seen in this http handler:

	http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		// Acquire a lock.
		lock := shutdown.Lock()
		// While this is held server will not shut down (except after timeout)
		if lock == nil {
			// Shutdown has started, return that the service is unavailable
			w.WriteHeader(http.StatusServiceUnavailable)
			return
		}
		// Defer unlocking the lock.
		defer lock()
		io.WriteString(w, "Server running")
	})

If shutdown is started, either by a signal or by another goroutine, it will wait until the lock is released. It is important always to release the lock, if shutdown.Lock() returns a non-nil value. Otherwise the server will have to wait until the timeout has passed before it starts shutting down, which may not be what you want.

As a convenience we also supply wrappers for http.Handler and http.HandlerFunc, which will do the same for you.

Each lock keeps track of its own creation time and will warn you if any lock exceeds the deadline time set for the pre-shutdown stage. This will help you identify issues that may be with your code, where it takes longer to complete than the allowed time, or you have forgotten to unlock any aquired lock.

Finally you can call shutdown.Exit(exitcode) to call all exit handlers and exit your application. This will wait for all locks to be released and notify all shutdown handlers and exit with the given exit code. If you want to do the exit yourself you can call the shutdown.Shutdown(), which does the same, but doesn't exit. Beware that you don't hold a lock when you call Exit/Shutdown.

Do note that calling os.Exit() or unhandled panics does not execute your shutdown handlers.

Also there are some things to be mindful of:

  • Notifiers can be created inside shutdown code, but only for stages following the current. So stage 1 notifiers can create stage 2 notifiers, but if they create a stage 1 notifier this will never be called.
  • Timeout can be changed once shutdown has been initiated, but it will only affect the following stages.
  • Notifiers returned from a function (eg. FirstFn) can be used for selects. They will be notified, but the shutdown manager will not wait for them to finish, so using them for this is not recommended.
  • If a panic occurs inside a shutdown function call in your code, the panic will be recovered and ignored and the shutdown will proceed. A message along with the backtrace is printed to Logger. If you want to handle panics, you must do it in your code.
  • When shutdown is initiated, it cannot be stopped.

When you design with this do take care that this library is for controlled shutdown of your application. If you application crashes no shutdown handlers are run, so panics will still be fatal. You can of course still call the Shutdown() function if you recover a panic, but the library does nothing like this automatically.

logging

By default logging is done to the standard log package. You can replace the Logger with your own before you start using the package. You can also send a "Printf" style function to the SetLogPrinter. This will allow you to easy hook up things like (*testing.T).Logf or specific loggers to intercept output.

You can set a custom WarningPrefix and ErrorPrefix in the package variables.

When you keep LogLockTimeout enabled, you will also get detailed information about your lock timeouts, including a file:line indication where the notifier/lock was created. It is recommended to keep this enabled for easier debugging.

If a line number isn't enough information you can pass something that can identify your shutdown.FirstFn(func() {select{}}, "Some Context"), will print "Some Context" when the function fails to return. The context is simply fmt.Printf("%v", ctx) when the function is created, so you can pass arbitrary objects.

You can use SetLogPrinter(func(string, ...interface{}){}) to disable logging.

why 3 stages?

By limiting the design to "only" three stages enable you to clearly make design choices, and force you to run as many things as possible in parallel. With this you can write simple design docs. Lets look at a webserver example:

  • Preshutdown: Finish accepted requests, refuse new ones.
  • Stage 1: Notify clients, flush data to database, notify upstream servers we are offline.
  • Stage 2: Flush database bulk writers, messages, close databases. (no database writes)
  • Stage 3: Flush/close log/metrics writer. (no log writes)

My intention is that this makes the shutdown process easier to manage, and encourage more concurrency, because you don't create a long daisy-chain of events, and doesn't force you to look through all your code to insert a single event correctly.

Don't think of the 3-stages as something that must do all stages of your shutdown. A single function call can of course (and is intended to) contain several "substages". Shutting down the database can easily be several stages, but you only register a single stage in the shutdown manager. The important part is that nothing else in the same stage can use the database.

examples

There are examples in the examples folder.

license

This code is published under an MIT license. See LICENSE file for more information.

Documentation

Overview

Package shutdown provides management of your shutdown process.

The package will enable you to get notifications for your application and handle the shutdown process.

See more information about the how to use it in the README.md file

Package home: https://github.com/klauspost/shutdown2

Example (Functions)

Get a notifier and perform our own function when we shutdown 

_ = FirstFn(func() {
	// This function is called on shutdown
	fmt.Println("First shutdown stage called")
})

// Will print the parameter when Shutdown() is called

Output:

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// Logger used for output.
	// This can be exchanged with your own.
	Logger = LogPrinter(log.New(os.Stderr, "[shutdown]: ", log.LstdFlags))

	// LoggerMu is a mutex for the Logger
	LoggerMu sync.Mutex

	// StagePS indicates the pre shutdown stage when waiting for locks to be released.
	StagePS = Stage{0}

	// Stage1 Indicates first stage of timeouts.
	Stage1 = Stage{1}

	// Stage2 Indicates second stage of timeouts.
	Stage2 = Stage{2}

	// Stage3 indicates third stage of timeouts.
	Stage3 = Stage{3}

	// WarningPrefix is printed before warnings.
	WarningPrefix = "WARN: "

	// ErrorPrefix is printed before errors.
	ErrorPrefix = "ERROR: "

	// LogLockTimeouts enables log timeout warnings
	// and notifier status updates.
	// Should not be changed once shutdown has started.
	LogLockTimeouts = true

	// StatusTimer is the time between logging which notifiers are waiting to finish.
	// Should not be changed once shutdown has started.
	StatusTimer = time.Minute
)

Functions

func CancelCtx 

func CancelCtx(parent context.Context) (ctx context.Context, cancel context.CancelFunc)

CancelCtx will cancel the supplied context when shutdown starts. The returned context must be cancelled when done similar to https://golang.org/pkg/context/#WithCancel

func CancelCtxN 

func CancelCtxN(parent context.Context, s Stage) (ctx context.Context, cancel context.CancelFunc)

CancelCtxN will cancel the supplied context at a supplied shutdown stage. The returned context must be cancelled when done similar to https://golang.org/pkg/context/#WithCancel

func Exit 

func Exit(code int)

Exit performs shutdown operations and exits with the given exit code.

func Lock 

func Lock(ctx ...interface{}) func()

Lock will signal that you have a function running, that you do not want to be interrupted by a shutdown.

The lock is created with a timeout equal to the length of the preshutdown stage at the time of creation. When that amount of time has expired the lock will be removed, and a warning will be printed.

If the function returns nil shutdown has already been initiated, and you did not get a lock. You should therefore not call the returned function.

If the function did not return nil, you should call the function to unlock the lock.

You should not hold a lock when you start a shutdown.

For easier debugging you can send a context that will be printed if the lock times out. All supplied context is printed with '%v' formatting.

Example

Note that the same effect of this example can also be achieved using the WrapHandlerFunc helper. 

http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
	// Get a lock while we have the lock, the server will not shut down.
	lock := Lock()
	if lock != nil {
		defer lock()
	} else {
		// We are currently shutting down, return http.StatusServiceUnavailable
		w.WriteHeader(http.StatusServiceUnavailable)
		return
	}
	// ...
})
http.ListenAndServe(":8080", nil)

Output:

func OnSignal 

func OnSignal(exitCode int, sig ...os.Signal)

OnSignal will start the shutdown when any of the given signals arrive

A good shutdown default is 

shutdown.OnSignal(0, os.Interrupt, syscall.SIGTERM)

which will do shutdown on Ctrl+C and when the program is terminated.

func OnTimeout added in v1.1.0 

func OnTimeout(fn func(Stage, string))

OnTimeout allows you to get a notification if a shutdown stage times out. The stage and the context of the hanging shutdown/lock function is returned.

func SetLogPrinter 

func SetLogPrinter(fn func(format string, v ...interface{}))

SetLogPrinter will use the specified function to write logging information.

func SetTimeout 

func SetTimeout(d time.Duration)

SetTimeout sets maximum delay to wait for each stage to finish. When the timeout has expired for a stage the next stage will be initiated.

func SetTimeoutN 

func SetTimeoutN(s Stage, d time.Duration)

SetTimeoutN set maximum delay to wait for a specific stage to finish. When the timeout expired for a stage the next stage will be initiated. The stage can be obtained by using the exported variables called 'Stage1, etc.

Example

Change timeout for a single stage 

// Set timout for all stages
SetTimeout(time.Second)

// But give second stage more time
SetTimeoutN(Stage2, time.Second*10)

Output:

func Shutdown 

func Shutdown()

Shutdown will signal all notifiers in three stages. It will first check that all locks have been released - see Lock()

func Started 

func Started() bool

Started returns true if shutdown has been started. Note that shutdown can have been started before you check the value.

func Wait 

func Wait()

Wait will wait until shutdown has finished. This can be used to keep a main function from exiting until shutdown has been called, either by a goroutine or a signal.

Example

This is an example, that could be your main function.

We wait for jobs to finish in another goroutine, from where we initialize the shutdown.

This is of course not a real-world problem, but there are many cases where you would want to initialize shutdown from other places than your main function, and where you would still like it to be able to do some final cleanup. 

x := make([]struct{}, 10)
var wg sync.WaitGroup

wg.Add(len(x))
for i := range x {
	go func(i int) {
		time.Sleep(time.Millisecond * time.Duration(i))
		wg.Done()
	}(i)
}

// ignore this reset, for test purposes only
reset()

// Wait for the jobs above to finish
go func() {
	wg.Wait()
	fmt.Println("jobs done")
	Shutdown()
}()

// Since this is main, we wait for a shutdown to occur before
// exiting.
Wait()
fmt.Println("exiting main")

// Note than the output will always be in this order.

Output:

jobs done
exiting main

func WrapHandler 

func WrapHandler(h http.Handler) http.Handler

WrapHandler will return an http Handler That will lock shutdown until all have completed and will return http.StatusServiceUnavailable if shutdown has been initiated.

Example

This example creates a fileserver and wraps the handler, so all request will finish before shutdown is started.

If requests take too long to finish the shutdown will proceed and clients will be disconnected when the server shuts down. To modify the timeout use SetTimeoutN(Preshutdown, duration) 

// Set a custom timeout, if the 5 second default doesn't fit your needs.
SetTimeoutN(StagePS, time.Second*30)
// Catch OS signals
OnSignal(0, os.Interrupt, syscall.SIGTERM)

// Create a fileserver handler
fh := http.FileServer(http.Dir("/examples"))

// Wrap the handler function
http.Handle("/", WrapHandler(fh))

// Start the server
http.ListenAndServe(":8080", nil)

Output:

func WrapHandlerFunc 

func WrapHandlerFunc(h http.HandlerFunc) http.HandlerFunc

WrapHandlerFunc will return an http.HandlerFunc that will lock shutdown until all have completed. The handler will return http.StatusServiceUnavailable if shutdown has been initiated.

Example

This example creates a custom function handler and wraps the handler, so all request will finish before shutdown is started.

If requests take too long to finish (see the shutdown will proceed and clients will be disconnected when the server shuts down. To modify the timeout use SetTimeoutN(Preshutdown, duration) 

// Set a custom timeout, if the 5 second default doesn't fit your needs.
SetTimeoutN(StagePS, time.Second*30)
// Catch OS signals
OnSignal(0, os.Interrupt, syscall.SIGTERM)

// Example handler function
fn := func(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
}

// Wrap the handler function
http.HandleFunc("/", WrapHandlerFunc(fn))

// Start the server
http.ListenAndServe(":8080", nil)

Output:

Types

type LogPrinter 

type LogPrinter interface {
	Printf(format string, v ...interface{})
}

LogPrinter is an interface for writing logging information. The writer must handle concurrent writes.

type Notifier 

type Notifier chan chan struct{}

Notifier is a channel, that will be sent a channel once the application shuts down. When you have performed your shutdown actions close the channel you are given.

Example

Get a notifier and perform our own code when we shutdown 

shutdown := First()
select {
case n := <-shutdown:
	// Do shutdown code ...

	// Signal we are done
	close(n)
}

Output:

func First 

func First(ctx ...interface{}) Notifier

First returns a notifier that will be called in the first stage of shutdowns. If shutdown has started and this stage has already been reached, nil will be returned. The context is printed if LogLockTimeouts is enabled.

func FirstFn 

func FirstFn(fn func(), ctx ...interface{}) Notifier

FirstFn executes a function in the first stage of the shutdown If shutdown has started and this stage has already been reached, nil will be returned. The context is printed if LogLockTimeouts is enabled.

func PreShutdown 

func PreShutdown(ctx ...interface{}) Notifier

PreShutdown will return a Notifier that will be fired as soon as the shutdown. is signalled, before locks are released. This allows to for instance send signals to upstream servers not to send more requests. The context is printed if LogLockTimeouts is enabled.

func PreShutdownFn 

func PreShutdownFn(fn func(), ctx ...interface{}) Notifier

PreShutdownFn registers a function that will be called as soon as the shutdown. is signalled, before locks are released. This allows to for instance send signals to upstream servers not to send more requests. The context is printed if LogLockTimeouts is enabled.

func Second 

func Second(ctx ...interface{}) Notifier

Second returns a notifier that will be called in the second stage of shutdowns. If shutdown has started and this stage has already been reached, nil will be returned. The context is printed if LogLockTimeouts is enabled.

func SecondFn 

func SecondFn(fn func(), ctx ...interface{}) Notifier

SecondFn executes a function in the second stage of the shutdown. If shutdown has started and this stage has already been reached, nil will be returned. The context is printed if LogLockTimeouts is enabled.

func Third 

func Third(ctx ...interface{}) Notifier

Third returns a notifier that will be called in the third stage of shutdowns. If shutdown has started and this stage has already been reached, nil will be returned. The context is printed if LogLockTimeouts is enabled.

func ThirdFn 

func ThirdFn(fn func(), ctx ...interface{}) Notifier

ThirdFn executes a function in the third stage of the shutdown. If shutdown has started and this stage has already been reached, nil will be returned. The context is printed if LogLockTimeouts is enabled.

func (Notifier) Cancel 

func (s Notifier) Cancel()

Cancel a Notifier. This will remove a notifier from the shutdown queue, and it will not be signalled when shutdown starts. If the shutdown has already started this will not have any effect, but a goroutine will wait for the notifier to be triggered.

func (Notifier) CancelWait 

func (s Notifier) CancelWait()

CancelWait will cancel a Notifier, or wait for it to become active if shutdown has been started. This will remove a notifier from the shutdown queue, and it will not be signalled when shutdown starts. If the notifier is nil (requested after its stage has started), it will return at once. If the shutdown has already started, this will wait for the notifier to be called and close it.

type Stage 

type Stage struct {
	// contains filtered or unexported fields
}

Stage contains stage information. Valid values for this is exported as variables StageN.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.