overseer

README

overseer

overseer is a package for creating monitorable, gracefully restarting, self-upgrading binaries in Go (golang). The main goal of this project is to facilitate the creation of self-upgrading binaries which play nice with standard process managers, secondly it should expose a small and simple API with reasonable defaults.

Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process.

Features

• Simple
• Works with process managers (systemd, upstart, supervisor, etc)
• Graceful, zero-down time restarts
• Easy self-upgrading binaries

Install

go get github.com/jpillora/overseer

Quick example

This program works with process managers, supports graceful, zero-down time restarts and self-upgrades its own binary.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/jpillora/overseer"
	"github.com/jpillora/overseer/fetcher"
)

//create another main() to run the overseer process
//and then convert your old main() into a 'prog(state)'
func main() {
	overseer.Run(overseer.Config{
		Program: prog,
		Address: ":3000",
		Fetcher: &fetcher.HTTP{
			URL:      "http://localhost:4000/binaries/myapp",
			Interval: 1 * time.Second,
		},
	})
}

//prog(state) runs in a child process
func prog(state overseer.State) {
	log.Printf("app (%s) listening...", state.ID)
	http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "app (%s) says hello\n", state.ID)
	}))
	http.Serve(state.Listener, nil)
}

How it works:

• overseer uses the main process to check for and install upgrades and a child process to run Program.
• The main process retrieves the files of the listeners described by Address/es.
• The child process is provided with these files which is converted into a Listener/s for the Program to consume.
• All child process pipes are connected back to the main process.
• All signals received on the main process are forwarded through to the child process.
• Fetcher runs in a goroutine and checks for updates at preconfigured interval. When Fetcher returns a valid binary stream (io.Reader), the master process saves it to a temporary location, verifies it, replaces the current binary and initiates a graceful restart.
• The fetcher.HTTP accepts a URL, it polls this URL with HEAD requests and until it detects a change. On change, we GET the URL and stream it back out to overseer. See also fetcher.S3.
• Once a binary is received, it is run with a simple echo token to confirm it is a overseer binary.
• Except for scheduled restarts, the active child process exiting will cause the main process to exit with the same code. So, overseer is not a process manager.

See Configuration options here and the runtime State available to your program here.

More examples

See the example/ directory and run example.sh, you should see the following output:

$ cd example/
$ sh example.sh
BUILT APP (1)
RUNNING APP
app#1 (c7940a5bfc3f0e8633d3bf775f54bb59f50b338e) listening...
app#1 (c7940a5bfc3f0e8633d3bf775f54bb59f50b338e) says hello
app#1 (c7940a5bfc3f0e8633d3bf775f54bb59f50b338e) says hello
BUILT APP (2)
app#2 (3dacb8bc673c1b4d38f8fb4fad5b017671aa8a67) listening...
app#2 (3dacb8bc673c1b4d38f8fb4fad5b017671aa8a67) says hello
app#2 (3dacb8bc673c1b4d38f8fb4fad5b017671aa8a67) says hello
app#1 (c7940a5bfc3f0e8633d3bf775f54bb59f50b338e) says hello
app#1 (c7940a5bfc3f0e8633d3bf775f54bb59f50b338e) exiting...
BUILT APP (3)
app#3 (b7614e7ff42eed8bb334ed35237743b0e4041678) listening...
app#3 (b7614e7ff42eed8bb334ed35237743b0e4041678) says hello
app#3 (b7614e7ff42eed8bb334ed35237743b0e4041678) says hello
app#2 (3dacb8bc673c1b4d38f8fb4fad5b017671aa8a67) says hello
app#2 (3dacb8bc673c1b4d38f8fb4fad5b017671aa8a67) exiting...
app#3 (b7614e7ff42eed8bb334ed35237743b0e4041678) says hello

Note: app#1 stays running until the last request is closed.

Only use graceful restarts

func main() {
	overseer.Run(overseer.Config{
		Program: prog,
		Address: ":3000",
	})
}

Send main a SIGUSR2 (Config.RestartSignal) to manually trigger a restart

Only use auto-upgrades, no restarts

func main() {
	overseer.Run(overseer.Config{
		Program: prog,
		NoRestart: true,
		Fetcher: &fetcher.HTTP{
			URL:      "http://localhost:4000/binaries/myapp",
			Interval: 1 * time.Second,
		},
	})
}

Your binary will be upgraded though it will require manual restart from the user, suitable for creating self-upgrading command-line applications.

Multi-platform binaries using a dynamic fetch URL

func main() {
	overseer.Run(overseer.Config{
		Program: prog,
		Fetcher: &fetcher.HTTP{
			URL: "http://localhost:4000/binaries/app-"+runtime.GOOS+"-"+runtime.GOARCH,
			//e.g.http://localhost:4000/binaries/app-linux-amd64
		},
	})
}

Known issues

• The master process's overseer.Config cannot be changed via an upgrade, the master process must be restarted.
  • Therefore, Addresses can only be changed by restarting the main process.
• Currently shells out to mv for moving files because mv handles cross-partition moves unlike os.Rename.
• Package init() functions will run twice on start, once in the main process and once in the child process.

More documentation

• Core overseer package
• Common fetcher.Interface
  • File fetcher
  • HTTP fetcher
  • S3 fetcher
  • Github fetcher

Third-party Fetchers

• overseer-bindiff A binary diff fetcher and builder

Contributing

See CONTRIBUTING.md

Documentation

Overview

Package overseer implements daemonizable self-upgrading binaries in Go (golang).

Index

• Variables
• func IsSupported() bool
• func Restart()
• func Run(c Config)
• func RunErr(c Config) error
• func SanityCheck()
• func WMIQueryWithContext(ctx context.Context, query string, dst interface{}, ...) error
• type Config
• type State
• type Win32_Process
  • func GetWin32Proc(pid int32) ([]Win32_Process, error)
  • func GetWin32ProcWithContext(ctx context.Context, pid int32) ([]Win32_Process, error)

Variables

var (
	SIGUSR1 = syscall.SIGTERM
	SIGUSR2 = syscall.SIGTERM
	SIGTERM = syscall.SIGTERM
)

var (
	//DisabledState is a placeholder state for when
	//overseer is disabled and the program function
	//is run manually.
	DisabledState = State{Enabled: false}
)

var (
	Timeout = 3 * time.Second
)

Functions

func IsSupported

func IsSupported() bool

IsSupported returns whether overseer is supported on the current OS.

func Restart

func Restart()

Restart programmatically triggers a graceful restart. If NoRestart is enabled, then this will essentially be a graceful shutdown.

func Run

func Run(c Config)

Run executes overseer, if an error is encountered, overseer fallsback to running the program directly (unless Required is set).

func RunErr

func RunErr(c Config) error

RunErr allows manual handling of any overseer errors.

func SanityCheck

func SanityCheck()

SanityCheck manually runs the check to ensure this binary is compatible with overseer. This tries to ensure that a restart is never performed against a bad binary, as it would require manual intervention to rectify. This is automatically done on overseer.Run() though it can be manually run prior whenever necessary.

func WMIQueryWithContext

func WMIQueryWithContext(ctx context.Context, query string, dst interface{}, connectServerArgs ...interface{}) error

Types

type Config

type Config struct {
	//Required will prevent overseer from fallback to running
	//running the program in the main process on failure.
	Required bool
	//Program's main function
	Program func(state State)
	//Program's zero-downtime socket listening address (set this or Addresses)
	Address string
	//Program's zero-downtime socket listening addresses (set this or Address)
	Addresses []string
	//RestartSignal will manually trigger a graceful restart. Defaults to SIGUSR2.
	RestartSignal os.Signal
	//TerminateTimeout controls how long overseer should
	//wait for the program to terminate itself. After this
	//timeout, overseer will issue a SIGKILL.
	TerminateTimeout time.Duration
	//MinFetchInterval defines the smallest duration between Fetch()s.
	//This helps to prevent unwieldy fetch.Interfaces from hogging
	//too many resources. Defaults to 1 second.
	MinFetchInterval time.Duration
	//PreUpgrade runs after a binary has been retrieved, user defined checks
	//can be run here and returning an error will cancel the upgrade.
	PreUpgrade func(tempBinaryPath string) error
	//Debug enables all [overseer] logs.
	Debug bool
	//NoWarn disables warning [overseer] logs.
	NoWarn bool
	//NoRestart disables all restarts, this option essentially converts
	//the RestartSignal into a "ShutdownSignal".
	NoRestart bool
	//NoRestartAfterFetch disables automatic restarts after each upgrade.
	//Though manual restarts using the RestartSignal can still be performed.
	NoRestartAfterFetch bool
	//Fetcher will be used to fetch binaries.
	Fetcher fetcher.Interface
}

Config defines overseer's run-time configuration

type State

type State struct {
	//whether overseer is running enabled. When enabled,
	//this program will be running in a child process and
	//overseer will perform rolling upgrades.
	Enabled bool
	//ID is a SHA-1 hash of the current running binary
	ID string
	//StartedAt records the start time of the program
	StartedAt time.Time
	//Listener is the first net.Listener in Listeners
	Listener net.Listener
	//Listeners are the set of acquired sockets by the master
	//process. These are all passed into this program in the
	//same order they are specified in Config.Addresses.
	Listeners []net.Listener
	//Program's first listening address
	Address string
	//Program's listening addresses
	Addresses []string
	//GracefulShutdown will be filled when its time to perform
	//a graceful shutdown.
	GracefulShutdown chan bool
	//Path of the binary currently being executed
	BinPath string
}

State contains the current run-time state of overseer

type Win32_Process

type Win32_Process struct {
	Name                  string
	ExecutablePath        *string
	CommandLine           *string
	Priority              uint32
	CreationDate          *time.Time
	ProcessID             uint32
	ThreadCount           uint32
	Status                *string
	ReadOperationCount    uint64
	ReadTransferCount     uint64
	WriteOperationCount   uint64
	WriteTransferCount    uint64
	CSCreationClassName   string
	CSName                string
	Caption               *string
	CreationClassName     string
	Description           *string
	ExecutionState        *uint16
	HandleCount           uint32
	KernelModeTime        uint64
	MaximumWorkingSetSize *uint32
	MinimumWorkingSetSize *uint32
	OSCreationClassName   string
	OSName                string
	OtherOperationCount   uint64
	OtherTransferCount    uint64
	PageFaults            uint32
	PageFileUsage         uint32
	ParentProcessID       uint32
	PeakPageFileUsage     uint32
	PeakVirtualSize       uint64
	PeakWorkingSetSize    uint32
	PrivatePageCount      uint64
	TerminationDate       *time.Time
	UserModeTime          uint64
	WorkingSetSize        uint64
}

func GetWin32Proc

func GetWin32Proc(pid int32) ([]Win32_Process, error)

func GetWin32ProcWithContext

func GetWin32ProcWithContext(ctx context.Context, pid int32) ([]Win32_Process, error)