tableroll

README

tableroll — Coordinate upgrades between processes

tableroll is a graceful upgrade process for network services. It allows zero-downtime upgrades (such that the listening socket never drops a connection) between multiple Go processes.

It is inspired heavily by cloudflare's tableflip library. The primary difference between 'tableflip' and 'tableroll' is that 'tableroll' does not require updates to re-use the existing executable binary nor does it enforce any process heirarchy between the old and new processes.

It is expected that the old and new procsses in a tableroll upgrade will both be managed by an external service manager, such as a systemd template unit.

Instead of coordinating upgrades between a parent and child process, tableroll coordinates upgrades between a number of processes that agree on a well-known filesystem path ahead of time, and which all have access to that path.

Usage

tableroll's usage is similar to tableflip's usage.

In general, your process should do the following:

1. Construct an upgrader using tableroll.New
2. Create or add all managed listeners / connections / files via upgrader.Fds.
3. Mark itself as ready to accept connections using upgrader.Ready
4. Wait for a request to exit using the upgrader.UpgradeComplete channel
5. Close all managed listeners and drain all connections (e.g. using server.Shutdown on http.Server)

One example usage might be the following:

Usage Example

The following example shows a simple usage of tableroll.

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"os"
	"time"

	"github.com/inconshreveable/log15"
	"github.com/ngrok/tableroll"
)

func main() {
	ctx := context.Background()
	logger := log15.New()
	if err := os.MkdirAll("/tmp/testroll", 0700); err != nil {
		log.Fatalf("can't create coordination dir: %v", err)
	}
	upg, err := tableroll.New(ctx, "/tmp/testroll", tableroll.WithLogger(logger))
	if err != nil {
		panic(err)
	}
	ln, err := upg.Fds.Listen("tcp", "127.0.0.1:8080")
	if err != nil {
		log.Fatalf("can't listen: %v", err)
	}

	server := &http.Server{
		Handler: http.HandlerFunc(func(r http.ResponseWriter, req *http.Request) {
			logger.Info("got http connection")
			time.Sleep(10 * time.Second)
			r.Write([]byte(fmt.Sprintf("hello from %v!\n", os.Getpid())))
		}),
	}
	go server.Serve(ln)

	if err := upg.Ready(); err != nil {
		panic(err)
	}
	<-upg.UpgradeComplete()

	time.AfterFunc(30*time.Second, func() {
		os.Exit(1)
	})

	_ = server.Shutdown(context.Background())

	logger.Info("server shutdown")
}

When this program is run, it will listen on port 8080 for http connections. If you start another copy of it, the newer copy will take over. If you have pending http requests in-flight, they'll be handled by the old process before it shuts down.

Documentation

Overview

Package tableroll implements zero downtime upgrades between two independently managed processes.

An upgrade is coordinated over a well-known coordination directory. Any number of processes may be run at once that coordinate upgrades on that directory, and between those many processes, one is chosen to own all shareable / upgradeable file descriptors. Each upgrade uniquely involves two processes, and unix exclusive locks on the filesystem ensure that.

Each process under tableroll should be able to signal readiness, which will indicate to tableroll that it is safe for previous processes to cease listening for new connections and begin draining existing ones.

Unlike other upgrade mechanisms in this space, it is expected that a new binary is started independently, such as in a new container, not as a child of the existing one. How a new upgrade is started is entirely out of scope of this library. Both copies of the process must have access to the same coordination directory, but apart from that, there are no stringent requirements.

Index

• Constants
• Variables
• type Conn
• type Fds
  • func (f *Fds) Conn(id string) (net.Conn, error)
  • func (f *Fds) DialWith(id, network, address string, ...) (net.Conn, error)
  • func (f *Fds) File(id string) (*os.File, error)
  • func (f *Fds) Listen(ctx context.Context, id string, cfg *net.ListenConfig, network, addr string) (net.Listener, error)
  • func (f *Fds) ListenWith(id, network, addr string, ...) (net.Listener, error)
  • func (f *Fds) Listener(id string) (net.Listener, error)
  • func (f *Fds) OpenFileWith(id string, name string, openFunc func(name string) (*os.File, error)) (*os.File, error)
  • func (f *Fds) Remove(id string) error
  • func (f *Fds) String() string
• type Listener
• type Option
  • func WithLogger(l log15.Logger) Option
  • func WithUpgradeTimeout(t time.Duration) Option
• type Upgrader
  • func New(ctx context.Context, coordinationDir string, opts ...Option) (*Upgrader, error)
  • func (u *Upgrader) Ready() error
  • func (u *Upgrader) Stop()
  • func (u *Upgrader) UpgradeComplete() <-chan struct{}

Constants

const DefaultUpgradeTimeout time.Duration = time.Minute

DefaultUpgradeTimeout is the duration in which the upgrader expects the sibling to send a 'Ready' notification after passing over all its file descriptors; If the sibling does not send that it is ready in that duration, this Upgrader will close the sibling's connection and wait for additional connections.

Variables

var (
	// ErrUpgradeInProgress indicates that an upgrade is in progress. This state
	// is not necessarily terminal.
	// This error will be returned if an attempt is made to mutate the file
	// descriptor store while the upgrader is currently attempting to transfer
	// all file descriptors elsewhere.
	ErrUpgradeInProgress = errors.New("an upgrade is currently in progress")
	// ErrUpgradeCompleted indicates that an upgrade has already happened. This
	// state is terminal.
	// This error will be returned if an attempt is made to mutate the file
	// descriptor store after an upgrade has already completed.
	ErrUpgradeCompleted = errors.New("an upgrade has completed")
	// ErrUpgraderStopped indicates the upgrader's Stop method has been called.
	// This state is terminal.
	// This error will be returned if an atttempt is made to mutate the file
	// descriptor store after stopping the upgrader.
	ErrUpgraderStopped = errors.New("the upgrader has been marked as stopped")
)

Types

type Conn

type Conn interface {
	net.Conn
	syscall.Conn
}

Conn can be shared between processes.

type Fds

type Fds struct {
	// contains filtered or unexported fields
}

Fds holds all shareable file descriptors, whether created in this process or inherited from a previous one. It provides methods for adding and removing file descriptors from the store.

func (*Fds) Conn

func (f *Fds) Conn(id string) (net.Conn, error)

Conn returns an inherited connection or nil.

It is the caller's responsibility to close the returned Conn at the appropriate time, typically when the Upgrader indicates draining and exiting is expected.

func (*Fds) DialWith

func (f *Fds) DialWith(id, network, address string, dialFn func(network, address string) (net.Conn, error)) (net.Conn, error)

DialWith takess an id and a function that returns a connection (akin to net.Dial). If an inherited connection with that id exists, it will be returned. Otherwise, the provided function will be called and the resulting connection stored with that id and returned.

func (*Fds) File

func (f *Fds) File(id string) (*os.File, error)

File returns an inherited file or nil.

The descriptor may be in blocking mode.

func (*Fds) Listen

func (f *Fds) Listen(ctx context.Context, id string, cfg *net.ListenConfig, network, addr string) (net.Listener, error)

Listen returns a listener inherited from the parent process, or creates a new one. It is expected that the caller will close the returned listener once the Upgrader indicates draining is desired. The arguments are passed to net.Listen, and their meaning is described there.

func (*Fds) ListenWith

func (f *Fds) ListenWith(id, network, addr string, listenerFunc func(network, addr string) (net.Listener, error)) (net.Listener, error)

ListenWith returns a listener with the given id inherited from the previous owner, or if it doesn't exist creates a new one using the provided function. The listener function should return quickly since it will block any upgrade requests from being serviced. Note that any unix sockets will have "SetUnlinkOnClose(false)" called on them. Callers may choose to switch them back to 'true' if appropriate. The listener function is compatible with net.Listen.

func (*Fds) Listener

func (f *Fds) Listener(id string) (net.Listener, error)

Listener returns an inherited listener with the given ID, or nil.

It is the caller's responsibility to close the returned listener once connections should be drained.

func (*Fds) OpenFileWith

func (f *Fds) OpenFileWith(id string, name string, openFunc func(name string) (*os.File, error)) (*os.File, error)

OpenFileWith retrieves the given file from the store, and if it's not present opens and adds it. The required openFunc is compatible with `os.Open`.

func (*Fds) Remove

func (f *Fds) Remove(id string) error

Remove removes the given file descriptor from the fds store.

func (*Fds) String

func (f *Fds) String() string

type Listener

type Listener interface {
	net.Listener
	syscall.Conn
}

Listener can be shared between processes.

type Option

type Option func(u *Upgrader)

Option is an option function for Upgrader. See Rob Pike's post on the topic for more information on this pattern: https://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html

func WithLogger

func WithLogger(l log15.Logger) Option

WithLogger configures the logger to use for tableroll operations. By default, nothing will be logged.

func WithUpgradeTimeout

func WithUpgradeTimeout(t time.Duration) Option

WithUpgradeTimeout allows configuring the update timeout. If a time of 0 is specified, the default will be used.

type Upgrader

type Upgrader struct {
	Fds *Fds
	// contains filtered or unexported fields
}

Upgrader handles zero downtime upgrades and passing files between processes.

func New

func New(ctx context.Context, coordinationDir string, opts ...Option) (*Upgrader, error)

New constructs a tableroll upgrader. The first argument is a directory. All processes in an upgrade chain must use the same coordination directory. The provided directory must exist and be writeable by the process using tableroll. Canonically, this directory is `/run/${program}/tableroll/`. Any number of options to configure tableroll may also be provided. If the passed in context is cancelled, any attempt to connect to an existing owner will be cancelled. To stop servicing upgrade requests and complete stop the upgrader, the `Stop` method should be called.

func (*Upgrader) Ready

func (u *Upgrader) Ready() error

Ready signals that the current process is ready to accept connections. It must be called to finish the upgrade.

All fds which were inherited but not used are closed after the call to Ready.

func (*Upgrader) Stop

func (u *Upgrader) Stop()

Stop prevents any more upgrades from happening, and closes the upgrade complete channel.

func (*Upgrader) UpgradeComplete

func (u *Upgrader) UpgradeComplete() <-chan struct{}

UpgradeComplete returns a channel which is closed when the managed file descriptors have been passed to the next process, and the next process has indicated it is ready.