httpbackoff

README

httpbackoff

Automatic http retries for intermittent failures, with exponential backoff, based on https://github.com/cenkalti/backoff.

The reason for a separate library, is that this library handles http status codes to know whether to retry or not. HTTP codes in range 500-599 are retried. Connection failures are also retried. Status codes 400-499 are considered permanent errors and are not retried.

The Retry function performs the http request and retries if temporary errors occur. It takes a single parameter as its input - a function to perform the http request. This function must return (resp *http.Response, tempError error, permError error) where tempError must be non-nil if a temporary error occurs (e.g. dropped connection), and permError must be non-nil if an error occurs that does not warrant retrying the request (e.g. badly formed url).

For example, the following code that is not using retries:

res, err := http.Get("http://www.google.com/robots.txt")

can be rewritten as:

res, attempts, err := httpbackoff.Retry(func() (*http.Response, error, error) {
	resp, err := http.Get("http://www.google.com/robots.txt")
	// assume all errors are temporary
    return resp, err, nil
})

Please note the additional return value attempts is an int specifying how many http calls were made (i.e. = 1 if no retries, otherwise > 1).

The go http package has 9 functions that return (*http.Reponse, error) that can potentially be retried:

• http://golang.org/pkg/net/http/#Client.Do
• http://golang.org/pkg/net/http/#Client.Get
• http://golang.org/pkg/net/http/#Client.Head
• http://golang.org/pkg/net/http/#Client.Post
• http://golang.org/pkg/net/http/#Client.PostForm
• http://golang.org/pkg/net/http/#Get
• http://golang.org/pkg/net/http/#Head
• http://golang.org/pkg/net/http/#Post
• http://golang.org/pkg/net/http/#PostForm

To simplify using these functions, 9 utility functions have been written that wrap these. Therefore you can simplify this example above further with:

res, _, err := httpbackoff.Get("http://www.google.com/robots.txt")

Configuring back off settings

To use cusom back off settings (for example, in testing, you might want to fail quickly), instead of calling the package functions, you can call methods of HTTPRetryClient with the same name:

package main

import (
	"log"
	"net/http/httputil"
	"time"

	"github.com/cenkalti/backoff"
	"github.com/taskcluster/httpbackoff"
)

func main() {
	// Note, you only need to create a client if you want to customise
	// the back off settings. In this example, we want to, but if you
	// wish to use the reasonable default settings, no need to do this.
	retryClient := httpbackoff.Client{
		BackOffSettings: &backoff.ExponentialBackOff{
			InitialInterval:     1 * time.Millisecond,
			RandomizationFactor: 0.2,
			Multiplier:          1.2,
			MaxInterval:         5 * time.Millisecond,
			MaxElapsedTime:      20 * time.Millisecond,
			Clock:               backoff.SystemClock,
		},
	}

	res, _, err := retryClient.Get("http://www.google.com/robots.txt")
	if err != nil {
		log.Fatalf("%s", err)
	}
	data, err := httputil.DumpResponse(res, true)
	if err != nil {
		log.Fatalf("%s", err)
	}
	log.Print(string(data))
}

Testing

The package has tests, which run in travis. See http://travis-ci.org/taskcluster/httpbackoff.

Concurrency

As far as I am aware, there is nothing in this library that prevents it from being used concurrently. Please let me know if you find any problems!

Contributing

Contributions are welcome. Please fork, and issue a Pull Request back with an explanation of your changes. Also please include tests for any functional changes.

Documentation

Overview

This package provides exponential backoff support for making HTTP requests.

It uses the github.com/cenkalti/backoff algorithm.

Network failures and HTTP 5xx status codes qualify for retries.

HTTP calls that return HTTP 4xx status codes do not get retried.

If the last HTTP request made does not result in a 2xx HTTP status code, an error is returned, together with the data.

There are several utility methods that wrap the standard net/http package calls.

Any function that takes no arguments and returns (*http.Response, error) can be retried using this library's Retry function.

The methods in this library should be able to run concurrently in multiple go routines.

Example Usage

Consider this trivial HTTP GET request:

res, err := http.Get("http://www.google.com/robots.txt")

This can be rewritten as follows, enabling automatic retries:

res, attempts, err := httpbackoff.Get("http://www.google.com/robots.txt")

The variable attempts stores the number of http calls that were made (one plus the number of retries).

Index

• func ClientDo(c *http.Client, req *http.Request) (resp *http.Response, attempts int, err error)
• func ClientGet(c *http.Client, url string) (resp *http.Response, attempts int, err error)
• func ClientHead(c *http.Client, url string) (resp *http.Response, attempts int, err error)
• func ClientPost(c *http.Client, url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)
• func ClientPostForm(c *http.Client, url string, data url.Values) (resp *http.Response, attempts int, err error)
• func Get(url string) (resp *http.Response, attempts int, err error)
• func Head(url string) (resp *http.Response, attempts int, err error)
• func Post(url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)
• func PostForm(url string, data url.Values) (resp *http.Response, attempts int, err error)
• func Retry(httpCall func() (resp *http.Response, tempError error, permError error)) (*http.Response, int, error)
• type BadHttpResponseCode
  • func (err BadHttpResponseCode) Error() string
• type Client
  • func (httpRetryClient *Client) ClientDo(c *http.Client, req *http.Request) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) ClientGet(c *http.Client, url string) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) ClientHead(c *http.Client, url string) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) ClientPost(c *http.Client, url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) ClientPostForm(c *http.Client, url string, data url.Values) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) Get(url string) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) Head(url string) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) Post(url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) PostForm(url string, data url.Values) (resp *http.Response, attempts int, err error)
  • func (httpRetryClient *Client) Retry(httpCall func() (resp *http.Response, tempError error, permError error)) (*http.Response, int, error)

Functions

func ClientDo

func ClientDo(c *http.Client, req *http.Request) (resp *http.Response, attempts int, err error)

ClientDo works the same as HTTPRetryClient.ClientDo, but uses the default exponential back off settings

func ClientGet

func ClientGet(c *http.Client, url string) (resp *http.Response, attempts int, err error)

ClientGet works the same as HTTPRetryClient.ClientGet, but uses the default exponential back off settings

func ClientHead

func ClientHead(c *http.Client, url string) (resp *http.Response, attempts int, err error)

ClientHead works the same as HTTPRetryClient.ClientHead, but uses the default exponential back off settings

func ClientPost

func ClientPost(c *http.Client, url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)

ClientPost works the same as HTTPRetryClient.ClientPost, but uses the default exponential back off settings

func ClientPostForm

func ClientPostForm(c *http.Client, url string, data url.Values) (resp *http.Response, attempts int, err error)

ClientPostForm works the same as HTTPRetryClient.ClientPostForm, but uses the default exponential back off settings

func Get

func Get(url string) (resp *http.Response, attempts int, err error)

Get works the same as HTTPRetryClient.Get, but uses the default exponential back off settings

func Head

func Head(url string) (resp *http.Response, attempts int, err error)

Head works the same as HTTPRetryClient.Head, but uses the default exponential back off settings

func Post

func Post(url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)

Post works the same as HTTPRetryClient.Post, but uses the default exponential back off settings

func PostForm

func PostForm(url string, data url.Values) (resp *http.Response, attempts int, err error)

PostForm works the same as HTTPRetryClient.PostForm, but uses the default exponential back off settings

func Retry

func Retry(httpCall func() (resp *http.Response, tempError error, permError error)) (*http.Response, int, error)

Retry works the same as HTTPRetryClient.Retry, but uses the default exponential back off settings

Types

type BadHttpResponseCode

type BadHttpResponseCode struct {
	HttpResponseCode int
	Message          string
}

Any non 2xx HTTP status code is considered a bad response code, and will result in a BadHttpResponseCode.

func (BadHttpResponseCode) Error

func (err BadHttpResponseCode) Error() string

Returns an error message for this bad HTTP response code

type Client

type Client struct {
	BackOffSettings *backoff.ExponentialBackOff
}

func (*Client) ClientDo

func (httpRetryClient *Client) ClientDo(c *http.Client, req *http.Request) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Client.Do where attempts is the number of http calls made (one plus number of retries).

func (*Client) ClientGet

func (httpRetryClient *Client) ClientGet(c *http.Client, url string) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Client.Get where attempts is the number of http calls made (one plus number of retries).

func (*Client) ClientHead

func (httpRetryClient *Client) ClientHead(c *http.Client, url string) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Client.Head where attempts is the number of http calls made (one plus number of retries).

func (*Client) ClientPost

func (httpRetryClient *Client) ClientPost(c *http.Client, url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Client.Post where attempts is the number of http calls made (one plus number of retries).

func (*Client) ClientPostForm

func (httpRetryClient *Client) ClientPostForm(c *http.Client, url string, data url.Values) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Client.PostForm where attempts is the number of http calls made (one plus number of retries).

func (*Client) Get

func (httpRetryClient *Client) Get(url string) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Get where attempts is the number of http calls made (one plus number of retries).

func (*Client) Head

func (httpRetryClient *Client) Head(url string) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Head where attempts is the number of http calls made (one plus number of retries).

func (*Client) Post

func (httpRetryClient *Client) Post(url string, bodyType string, body []byte) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#Post where attempts is the number of http calls made (one plus number of retries).

func (*Client) PostForm

func (httpRetryClient *Client) PostForm(url string, data url.Values) (resp *http.Response, attempts int, err error)

Retry wrapper for http://golang.org/pkg/net/http/#PostForm where attempts is the number of http calls made (one plus number of retries).

func (*Client) Retry

func (httpRetryClient *Client) Retry(httpCall func() (resp *http.Response, tempError error, permError error)) (*http.Response, int, error)

Retry is the core library method for retrying http calls.

httpCall should be a function that performs the http operation, and returns (resp *http.Response, tempError error, permError error). Errors that should cause retries should be returned as tempError. Permanent errors that should not result in retries should be returned as permError. Retries are performed using the exponential backoff algorithm from the github.com/cenkalti/backoff package. Retry automatically treats HTTP 5xx status codes as a temporary error, and any other non-2xx HTTP status codes as a permanent error. Thus httpCall function does not need to handle the HTTP status code of resp, since Retry will take care of it.

Concurrent use of this library method is supported.