README
¶
Graceful process restarts in Go
It is sometimes useful to update the running code and / or configuration of a network service, without disrupting existing connections. Usually, this is achieved by starting a new process, somehow transferring clients to it and then exiting the old process.
There are many ways to implement graceful upgrades. They vary wildly in the trade-offs they make, and how much control they afford the user. This library has the following goals:
- No old code keeps running after a successful upgrade
- The new process has a grace period for performing initialisation
- Crashing during initialisation is OK
- Only a single upgrade is ever run in parallel
tableflip does not work on Windows.
It's easy to get started:
upg, err := tableflip.New(tableflip.Options{})
if err != nil {
panic(err)
}
defer upg.Stop()
go func() {
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGHUP)
for range sig {
err := upg.Upgrade()
if err != nil {
log.Println("Upgrade failed:", err)
continue
}
log.Println("Upgrade succeeded")
}
}()
ln, err := upg.Fds.Listen("tcp", "localhost:8080")
if err != nil {
log.Fatalln("Can't listen:", err)
}
var server http.Server
go server.Serve(ln)
if err := upg.Ready(); err != nil {
panic(err)
}
<-upg.Exit()
time.AfterFunc(30*time.Second, func() {
os.Exit(1)
})
_ = server.Shutdown(context.Background())
Documentation
¶
Overview ¶
Package tableflip implements zero downtime upgrades.
An upgrade spawns a new copy of argv[0] and passes file descriptors of used listening sockets to the new process. The old process exits once the new process signals readiness. Thus new code can use sockets allocated in the old process. This is similar to the approach used by nginx, but as a library.
At any point in time there are one or two processes, with at most one of them in non-ready state. A successful upgrade fully replaces all old configuration and code.
To use this library with systemd you need to use the PIDFile option in the service file.
[Unit] Description=Service using tableflip [Service] ExecStart=/path/to/binary -some-flag /path/to/pid-file ExecReload=/bin/kill -HUP $MAINPID PIDFile=/path/to/pid-file
Then pass /path/to/pid-file to New. You can use systemd-run to test your implementation:
systemd-run --user -p PIDFile=/path/to/pid-file /path/to/binary
systemd-run will print a unit name, which you can use with systemctl to inspect the service.
NOTES:
Requires at least Go 1.9, since there is a race condition on the pipes used for communication between parent and child.
If you're seeing "can't start process: no such file or directory", you're probably using "go run main.go", for graceful reloads to work, you'll need use "go build main.go".
Example (HttpShutdown) ¶
This shows how to use the upgrader with the graceful shutdown facilities of net/http.
package main
import (
"context"
"flag"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/cloudflare/tableflip"
)
func main() {
var (
listenAddr = flag.String("listen", "localhost:8080", "`Address` to listen on")
pidFile = flag.String("pid-file", "", "`Path` to pid file")
)
flag.Parse()
log.SetPrefix(fmt.Sprintf("%d ", os.Getpid()))
upg, err := tableflip.New(tableflip.Options{
PIDFile: *pidFile,
})
if err != nil {
panic(err)
}
defer upg.Stop()
// Do an upgrade on SIGHUP
go func() {
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGHUP)
for range sig {
err := upg.Upgrade()
if err != nil {
log.Println("Upgrade failed:", err)
}
}
}()
ln, err := upg.Fds.Listen("tcp", *listenAddr)
if err != nil {
log.Fatalln("Can't listen:", err)
}
server := http.Server{
// Set timeouts, etc.
}
go func() {
err := server.Serve(ln)
if err != http.ErrServerClosed {
log.Println("HTTP server:", err)
}
}()
log.Printf("ready")
if err := upg.Ready(); err != nil {
panic(err)
}
<-upg.Exit()
// Make sure to set a deadline on exiting the process
// after upg.Exit() is closed. No new upgrades can be
// performed if the parent doesn't exit.
time.AfterFunc(30*time.Second, func() {
log.Println("Graceful shutdown timed out")
os.Exit(1)
})
// Wait for connections to drain.
server.Shutdown(context.Background())
}
Output:
Example (TcpServer) ¶
This shows how to use the Upgrader with a listener based service.
package main
import (
"flag"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"github.com/cloudflare/tableflip"
)
func main() {
var (
listenAddr = flag.String("listen", "localhost:8080", "`Address` to listen on")
pidFile = flag.String("pid-file", "", "`Path` to pid file")
)
flag.Parse()
log.SetPrefix(fmt.Sprintf("%d ", os.Getpid()))
upg, err := tableflip.New(tableflip.Options{
PIDFile: *pidFile,
})
if err != nil {
panic(err)
}
defer upg.Stop()
// Do an upgrade on SIGHUP
go func() {
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGHUP)
for range sig {
err := upg.Upgrade()
if err != nil {
log.Println("upgrade failed:", err)
}
}
}()
ln, err := upg.Fds.Listen("tcp", *listenAddr)
if err != nil {
log.Fatalln("Can't listen:", err)
}
go func() {
defer ln.Close()
log.Printf("listening on %s", ln.Addr())
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
c.SetDeadline(time.Now().Add(time.Second))
c.Write([]byte("It is a mistake to think you can solve any major problems just with potatoes.\n"))
c.Close()
}()
}
}()
log.Printf("ready")
if err := upg.Ready(); err != nil {
panic(err)
}
<-upg.Exit()
}
Output:
Index ¶
- Constants
- type Conn
- type Fds
- func (f *Fds) AddConn(network, addr string, conn Conn) error
- func (f *Fds) AddFile(name string, file *os.File) error
- func (f *Fds) AddListener(network, addr string, ln Listener) error
- func (f *Fds) Conn(network, addr string) (net.Conn, error)
- func (f *Fds) File(name string) (*os.File, error)
- func (f *Fds) Listen(network, addr string) (net.Listener, error)
- func (f *Fds) Listener(network, addr string) (net.Listener, error)
- type Listener
- type Options
- type Upgrader
Examples ¶
Constants ¶
const DefaultUpgradeTimeout time.Duration = time.Minute
DefaultUpgradeTimeout is the duration before the Upgrader kills the new process if no readiness notification was received.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Fds ¶
type Fds struct {
// contains filtered or unexported fields
}
Fds holds all file descriptors inherited from the parent process.
func (*Fds) AddConn ¶
AddConn adds a connection.
It is safe to close conn after calling this method.
func (*Fds) AddFile ¶
AddFile adds a file.
Until Go 1.12, file will be in blocking mode after this call.
func (*Fds) AddListener ¶
AddListener adds a listener.
It is safe to close ln after calling the method. Any existing listener with the same address is overwitten.
func (*Fds) Conn ¶
Conn returns an inherited connection or nil.
It is safe to close the returned Conn.
type Options ¶
type Options struct {
// Time after which an upgrade is considered failed. Defaults to
// DefaultUpgradeTimeout.
UpgradeTimeout time.Duration
// The PID of a ready process is written to this file.
PIDFile string
}
Options control the behaviour of the Upgrader.
type Upgrader ¶
type Upgrader struct {
*Fds
// contains filtered or unexported fields
}
Upgrader handles zero downtime upgrades and passing files between processes.
func New ¶
New creates a new Upgrader. Files are passed from the parent and may be empty.
Only the first call to this function will succeed.
func (*Upgrader) Exit ¶
func (u *Upgrader) Exit() <-chan struct{}
Exit returns a channel which is closed when the process should exit.
func (*Upgrader) HasParent ¶
HasParent checks if the current process is an upgrade or the first invocation.
func (*Upgrader) Ready ¶
Ready signals that the current process is ready to accept connections. It must be called to finish the upgrade.
All fds which were inherited but not used are closed after the call to Ready.
func (*Upgrader) Stop ¶
func (u *Upgrader) Stop()
Stop prevents any more upgrades from happening, and closes the exit channel.
If this function is called before a call to Upgrade() has succeeded, it is assumed that the process is being shut down completely. All Unix sockets known to Upgrader.Fds are then unlinked from the filesystem.
Source Files