pester

README

pester

pester wraps Go's standard lib http client to provide several options to increase resiliency in your request. If you experience poor network conditions or requests could experience varied delays, you can now pester the endpoint for data.

• Retry on errors
• Backoff

Simple Example

Use pester where you would use the http client calls. By default, pester will retry the endpoint 3 times with the DefaultBackoff strategy of waiting 1 second between retries.

/* swap in replacement, just switch
   http.{Get|Post|PostForm|Head|Do} to
   pester.{Get|Post|PostForm|Head|Do}
*/
resp, err := pester.Get("http://sethammons.com")

Backoff Strategy

Provide your own backoff strategy, or use one of the provided built in strategies:

• DefaultBackoff: 1 second
• LinearBackoff: n seconds where n is the retry number
• LinearJitterBackoff: n seconds where n is the retry number, +/- 0-33%
• ExponentialBackoff: n seconds where n is 2^(retry number)
• ExponentialJitterBackoff: n seconds where n is 2^(retry number), +/- 0-33%

client := pester.New()
client.Backoff = func(retry int) time.Duration {
    // set up something dynamic or use a look up table
    return time.Duration(retry) * time.Minute
}

Complete example

For a complete and working example, see the sample directory. pester allows you to use a constructor to control:

• backoff strategy
• retries
• keeping a log for debugging

package main

import (
    "log"
    "net/http"
    "strings"

    "github.com/sethgrid/pester"
)

func main() {
    log.Println("Starting...")

    { // drop in replacement for http.Get and other client methods
        resp, err := pester.Get("http://example.com")
        if err != nil {
            log.Println("error GETing example.com", err)
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }

    { // control the resiliency
        client := pester.New()
        client.MaxRetries = 5
        client.Backoff = pester.ExponentialBackoff
        client.KeepLog = true

        resp, err := client.Get("http://example.com")
        if err != nil {
            log.Println("error GETing example.com", client.LogString())
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }

    { // use the pester version of http.Client.Do
        req, err := http.NewRequest("POST", "http://example.com", strings.NewReader("data"))
        if err != nil {
            log.Fatal("Unable to create a new http request", err)
        }
        resp, err := pester.Do(req)
        if err != nil {
            log.Println("error POSTing example.com", err)
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }
}

Example Log

pester also allows you to control the resiliency and can optionally log the errors.

c := pester.New()
c.KeepLog = true

nonExistantURL := "http://localhost:9000/foo"
_, _ = c.Get(nonExistantURL)

fmt.Println(c.LogString())
/*
Output:

1432402837 Get [GET] http://localhost:9000/foo attempt-1 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
1432402838 Get [GET] http://localhost:9000/foo attempt-2 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
1432402839 Get [GET] http://localhost:9000/foo attempt-3 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
*/

Tests

You can run tests in the root directory with $ go test. There is a benchmark-like test available with $ cd benchmarks; go test. You can see pester in action with $ cd sample; go run main.go.

For watching open file descriptors, you can run watch "lsof -i -P | grep main" if you started the app with go run main.go. I did this for watching for FD leaks. My method was to alter sample/main.go to only run one case (pester.Get with set backoff stategy, concurrency and retries increased) and adding a sleep after the result came back. This let me verify if FDs were getting left open when they should have closed. If you know a better way, let me know! I was able to see that FDs are now closing when they should :)

Are we there yet? Are we there yet? Are we there yet? Are we there yet? ...

Documentation

Overview

Package pester provides additional resiliency over the standard http client methods by allowing you to control concurrency, retries, and a backoff strategy.

Index

• Variables
• func DefaultBackoff(_ int) time.Duration
• func Do(req *http.Request) (resp *http.Response, err error)
• func ExponentialBackoff(i int) time.Duration
• func ExponentialJitterBackoff(i int) time.Duration
• func Get(url string) (resp *http.Response, err error)
• func Head(url string) (resp *http.Response, err error)
• func LinearBackoff(i int) time.Duration
• func LinearJitterBackoff(i int) time.Duration
• func Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)
• func PostForm(url string, data url.Values) (resp *http.Response, err error)
• type BackoffStrategy
• type Client
  • func New() *Client
  • func NewExtendedClient(hc *http.Client) *Client
  • func (c *Client) Do(req *http.Request) (resp *http.Response, err error)
  • func (c *Client) EmbedHTTPClient(hc *http.Client)
  • func (c *Client) FormatError(e ErrEntry) string
  • func (c *Client) Get(url string) (resp *http.Response, err error)
  • func (c *Client) Head(url string) (resp *http.Response, err error)
  • func (c *Client) LogErrCount() int
  • func (c *Client) LogString() string
  • func (c *Client) Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)
  • func (c *Client) PostForm(url string, data url.Values) (resp *http.Response, err error)
  • func (c *Client) SetRetryOnHTTP429(flag bool)
• type ErrEntry
• type LogHook

Variables

var DefaultClient = &Client{MaxRetries: 3, Backoff: DefaultBackoff, ErrLog: []ErrEntry{}}

DefaultClient provides sensible defaults

var ErrReadingBody = errors.New("error reading body")

ErrReadingBody happens when we cannot read the body bytes

var ErrReadingRequestBody = errors.New("error reading request body")

ErrReadingRequestBody happens when we cannot read the request body bytes

var ErrUnexpectedMethod = errors.New("unexpected client method, must be one of Do, Get, Head, Post, or PostFrom")

ErrUnexpectedMethod occurs when an http.Client method is unable to be mapped from a calling method in the pester client

Functions

func DefaultBackoff

func DefaultBackoff(_ int) time.Duration

DefaultBackoff always returns 1 second

func Do

func Do(req *http.Request) (resp *http.Response, err error)

Do provides the same functionality as http.Client.Do and creates its own constructor

func ExponentialBackoff

func ExponentialBackoff(i int) time.Duration

ExponentialBackoff returns ever increasing backoffs by a power of 2

func ExponentialJitterBackoff

func ExponentialJitterBackoff(i int) time.Duration

ExponentialJitterBackoff returns ever increasing backoffs by a power of 2 with +/- 0-33% to prevent sychronized reuqests.

func Get

func Get(url string) (resp *http.Response, err error)

Get provides the same functionality as http.Client.Get and creates its own constructor

func Head

func Head(url string) (resp *http.Response, err error)

Head provides the same functionality as http.Client.Head and creates its own constructor

func LinearBackoff

func LinearBackoff(i int) time.Duration

LinearBackoff returns increasing durations, each a second longer than the last

func LinearJitterBackoff

func LinearJitterBackoff(i int) time.Duration

LinearJitterBackoff returns increasing durations, each a second longer than the last with +/- 0-33% to prevent sychronized reuqests.

func Post

func Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)

Post provides the same functionality as http.Client.Post and creates its own constructor

func PostForm

func PostForm(url string, data url.Values) (resp *http.Response, err error)

PostForm provides the same functionality as http.Client.PostForm and creates its own constructor

Types

type BackoffStrategy

type BackoffStrategy func(retry int) time.Duration

BackoffStrategy is used to determine how long a retry request should wait until attempted

type Client

type Client struct {
	Transport     http.RoundTripper
	CheckRedirect func(req *http.Request, via []*http.Request) error
	Jar           http.CookieJar
	Timeout       time.Duration

	// pester specific
	MaxRetries         int
	Backoff            BackoffStrategy
	KeepLog            bool
	LogHook            LogHook
	RetryAttemptHeader string

	ErrLog         []ErrEntry
	RetryOnHTTP429 bool
	// contains filtered or unexported fields
}

Client wraps the http client and exposes all the functionality of the http.Client. Additionally, Client provides pester specific values for handling resiliency.

func New

func New() *Client

New constructs a new DefaultClient with sensible default values

func NewExtendedClient

func NewExtendedClient(hc *http.Client) *Client

NewExtendedClient allows you to pass in an http.Client that is previously set up and extends it to have Pester's features of concurrency and retries.

func (*Client) Do

func (c *Client) Do(req *http.Request) (resp *http.Response, err error)

Do provides the same functionality as http.Client.Do

func (*Client) EmbedHTTPClient

func (c *Client) EmbedHTTPClient(hc *http.Client)

EmbedHTTPClient allows you to extend an existing Pester client with an underlying http.Client, such as https://godoc.org/golang.org/x/oauth2/google#DefaultClient

func (*Client) FormatError

func (c *Client) FormatError(e ErrEntry) string

FormatError formats the Error to human readable string

func (*Client) Get

func (c *Client) Get(url string) (resp *http.Response, err error)

Get provides the same functionality as http.Client.Get

func (*Client) Head

func (c *Client) Head(url string) (resp *http.Response, err error)

Head provides the same functionality as http.Client.Head

func (*Client) LogErrCount

func (c *Client) LogErrCount() int

LogErrCount is a helper method used primarily for test validation

func (*Client) LogString

func (c *Client) LogString() string

LogString provides a string representation of the errors the client has seen

func (*Client) Post

func (c *Client) Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)

Post provides the same functionality as http.Client.Post

func (*Client) PostForm

func (c *Client) PostForm(url string, data url.Values) (resp *http.Response, err error)

PostForm provides the same functionality as http.Client.PostForm

func (*Client) SetRetryOnHTTP429

func (c *Client) SetRetryOnHTTP429(flag bool)

SetRetryOnHTTP429 sets RetryOnHTTP429 for clients

type ErrEntry

type ErrEntry struct {
	Time    time.Time
	Method  string
	URL     string
	Verb    string
	Attempt int
	Err     error
}

ErrEntry is used to provide the LogString() data and is populated each time an error happens if KeepLog is set. ErrEntry.Retry is deprecated in favor of ErrEntry.Attempt

type LogHook

type LogHook func(e ErrEntry)

LogHook is used to log attempts as they happen. This function is never called, however, if KeepLog is set to true.