healthcheck

Documentation

Overview

Package healthcheck helps you implement liveness and readiness checks for your application. It supports synchronous and asynchronous (background) checks. It can optionally report each check's status as a set of Prometheus gauge metrics for cluster-wide monitoring and alerting.

It also includes a small library of generic checks for DNS, TCP, and HTTP reachability as well as Goroutine usage.

Example

_, upstreamURL := upstream() // Mock some upstream Server

// Create a Handler that we can use to register liveness and readiness checks.
healthchecks := NewHandler()

// Add a readiness check to make sure an upstream dependency resolves in DNS.
// If this fails we don't want to receive requests, but we shouldn't be
// restarted or rescheduled.
upstreamHost := upstreamURL.Hostname()
healthchecks.AddReadinessCheck(
	"upstream-dependency-dns",
	DNSResolveCheck(upstreamHost, 50*time.Millisecond))

// Add a liveness check to detect Goroutine leaks. If this fails we want
// to be restarted/rescheduled.
healthchecks.AddLivenessCheck(
	"goroutine-threshold",
	GoroutineCountCheck(100),
)

// Serve http://0.0.0.0:8080/live and http://0.0.0.0:8080/ready endpoints.
// go http.ListenAndServe("0.0.0.0:8080", healthchecks)

// Make a request to the readiness endpoint and print the response.
fmt.Print(dumpRequest(healthchecks, "GET", "/ready"))

Output:

HTTP/1.1 200 OK
Connection: close
Content-Type: application/json; charset=utf-8

{
    "goroutine-threshold": "OK",
    "upstream-dependency-dns": "OK"
}

Example (Advanced)

upstream, _ := upstream() // Mock some upstream Server

// Create a Handler that we can use to register liveness and readiness checks.
healthchecks := NewHandler()

// Add a readiness check against the health of an upstream HTTP dependency
healthchecks.AddReadinessCheck(
	"upstream-dependency-http",
	HTTPGetCheck(upstream.URL, 500*time.Millisecond))

// Implement a custom check with a 50 millisecond timeout.
healthchecks.AddLivenessCheck(
	"custom-check-with-timeout",
	Timeout(func() error {
		// Simulate some work that could take a long time
		time.Sleep(time.Millisecond * 100)
		return nil
	}, 50*time.Millisecond),
)

// Expose the readiness endpoints on a custom path /healthz mixed into
// our main application mux.
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
	_, _ = w.Write([]byte("Hello, world!"))
})
mux.HandleFunc("/healthz", healthchecks.ReadyEndpoint)

// Sleep for just a moment to make sure our Async handler had a chance to run
time.Sleep(500 * time.Millisecond)

// Make a sample request to the /healthz endpoint and print the response.
fmt.Println(dumpRequest(mux, "GET", "/healthz"))

Output:

HTTP/1.1 503 Service Unavailable
Connection: close
Content-Type: application/json; charset=utf-8

{
    "custom-check-with-timeout": "timed out after 50ms",
    "upstream-dependency-http": "OK"
}

Example (Database)

// Connect to a database/sql database
database := connectToDatabase()

// Create a Handler that we can use to register liveness and readiness checks.
healthchecks := NewHandler()

// Add a readiness check to we don't receive requests unless we can reach
// the database with a ping in <1 second.
healthchecks.AddReadinessCheck("database", DatabasePingCheck(database, 1*time.Second))

// Serve http://0.0.0.0:8080/live and http://0.0.0.0:8080/ready endpoints.
// go http.ListenAndServe("0.0.0.0:8080", healthchecks)

// Make a request to the readiness endpoint and print the response.
fmt.Print(dumpRequest(healthchecks, "GET", "/ready"))

Output:

HTTP/1.1 200 OK
Connection: close
Content-Type: application/json; charset=utf-8

{
    "database": "OK"
}

Example (Metrics)

// Create a new Prometheus registry (you'd likely already have one of these).
registry := prometheus.NewRegistry()

// Create a metrics-exposing Handler for the Prometheus registry
// It wraps the default handler to add metrics.
healthchecks := NewMetricsHandler(NewHandler(), registry)

// Add a simple readiness check that always fails.
healthchecks.AddReadinessCheck(
	"failing-check",
	func() error {
		return fmt.Errorf("example failure")
	},
)

// Add a liveness check that always succeeds
healthchecks.AddLivenessCheck(
	"successful-check",
	func() error {
		return nil
	},
)

// Create an "admin" listener on 0.0.0.0:9402
internal := http.NewServeMux()
// go http.ListenAndServe("0.0.0.0:9402", internal)

// Expose prometheus metrics on /metrics
internal.Handle("/metrics", promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))

// Expose a liveness check on /live and readiness check on /ready
internal.HandleFunc("/live", healthchecks.LiveEndpoint)
internal.HandleFunc("/ready", healthchecks.ReadyEndpoint)

// Make a request to the metrics endpoint and print the response.
fmt.Println(dumpRequest(internal, "GET", "/metrics"))

Output:

HTTP/1.1 200 OK
Connection: close
Content-Type: text/plain; version=0.0.4; charset=utf-8

# HELP healthcheck Indicates if check is healthy (1 is healthy, 0 is unhealthy)
# TYPE healthcheck gauge
healthcheck{check="live",name="successful-check"} 1
healthcheck{check="ready",name="failing-check"} 0

Index

• type Check
  • func DNSResolveCheck(host string, timeout time.Duration) Check
  • func DatabasePingCheck(database *sql.DB, timeout time.Duration) Check
  • func GoroutineCountCheck(threshold int) Check
  • func HTTPCheck(url string, method string, status int, timeout time.Duration) Check
  • func HTTPCheckClient(client *http.Client, url string, method string, status int, ...) Check
  • func HTTPGetCheck(url string, timeout time.Duration) Check
  • func TCPDialCheck(addr string, timeout time.Duration) Check
  • func Timeout(check Check, timeout time.Duration) Check
• type Handler
  • func NewHandler() Handler
  • func NewMetricsHandler(handler Handler, registry prometheus.Registerer) Handler

Examples

• Package
• Package (Advanced)
• Package (Database)
• Package (Metrics)

Types

type Check

type Check func() error

Check is a health/readiness check.

func DNSResolveCheck

func DNSResolveCheck(host string, timeout time.Duration) Check

DNSResolveCheck returns a Check that makes sure the provided host can resolve to at least one IP address within the specified timeout.

func DatabasePingCheck

func DatabasePingCheck(database *sql.DB, timeout time.Duration) Check

DatabasePingCheck returns a Check that validates connectivity to a database/sql.DB using Ping().

func GoroutineCountCheck

func GoroutineCountCheck(threshold int) Check

GoroutineCountCheck returns a Check that fails if too many goroutines are running (which could indicate a resource leak).

func HTTPCheck

func HTTPCheck(url string, method string, status int, timeout time.Duration) Check

HTTPCheck returns a Check that performs a HTTP request against the specified URL. The Check fails if the response times out or returns an unexpected status code.

func HTTPCheckClient

func HTTPCheckClient(client *http.Client, url string, method string, status int, timeout time.Duration) Check

HTTPCheckClient returns a Check that performs a HTTP request against the specified URL. The Check fails if the response times out or returns an unexpected status code. On top of that it uses a custom client specified by the caller.

func HTTPGetCheck

func HTTPGetCheck(url string, timeout time.Duration) Check

HTTPGetCheck returns a Check that performs an HTTP GET request against the specified URL. The check fails if the response times out or returns a non-200 status code.

func TCPDialCheck

func TCPDialCheck(addr string, timeout time.Duration) Check

TCPDialCheck returns a Check that checks TCP connectivity to the provided endpoint.

func Timeout

func Timeout(check Check, timeout time.Duration) Check

Timeout adds a timeout to a Check. If the underlying check takes longer than the timeout, it returns an error.

type Handler

type Handler interface {
	// The Handler is an http.Handler, so it can be exposed directly and handle
	// /live and /ready endpoints.
	http.Handler

	// AddLivenessCheck adds a check that indicates that this instance of the
	// application should be destroyed or restarted. A failed liveness check
	// indicates that this instance is unhealthy, not some upstream dependency.
	// Every liveness check is also included as a readiness check.
	AddLivenessCheck(name string, check Check)

	// AddReadinessCheck adds a check that indicates that this instance of the
	// application is currently unable to serve requests because of an upstream
	// or some transient failure. If a readiness check fails, this instance
	// should no longer receiver requests, but should not be restarted or
	// destroyed.
	AddReadinessCheck(name string, check Check)

	// LiveEndpoint is the HTTP handler for just the /live endpoint, which is
	// useful if you need to attach it into your own HTTP handler tree.
	LiveEndpoint(http.ResponseWriter, *http.Request)

	// ReadyEndpoint is the HTTP handler for just the /ready endpoint, which is
	// useful if you need to attach it into your own HTTP handler tree.
	ReadyEndpoint(http.ResponseWriter, *http.Request)
}

Handler is an http.Handler with additional methods that register health and readiness checks. It handles handle "/live" and "/ready" HTTP endpoints.

func NewHandler

func NewHandler() Handler

NewHandler creates a new basic Handler

func NewMetricsHandler

func NewMetricsHandler(handler Handler, registry prometheus.Registerer) Handler

NewMetricsHandler returns a healthy Handler that writes the current check status into the provided Prometheus registry.