pester

README

pester

pester wraps Go's standard lib http client to provide several options to increase resiliency in your request. If you experience poor network conditions or requests could experience varied delays, you can now pester the endpoint for data.

• Send out multiple requests and get the first back (only used for GET calls)
• Retry on errors
• Backoff

Simple Example

Use pester where you would use the http client calls. By default, pester will use a concurrency of 1, and retry the endpoint 3 times with the DefaultBackoff strategy of waiting 1 second between retries.

/* swap in replacement, just switch
   http.{Get|Post|PostForm|Head|Do} to
   pester.{Get|Post|PostForm|Head|Do}
*/
resp, err := pester.Get("http://sethammons.com")

Backoff Strategy

Provide your own backoff strategy, or use one of the provided built in strategies:

• DefaultBackoff: 1 second
• LinearBackoff: n seconds where n is the retry number
• LinearJitterBackoff: n seconds where n is the retry number, +/- 0-33%
• ExponentialBackoff: n seconds where n is 2^(retry number)
• ExponentialJitterBackoff: n seconds where n is 2^(retry number), +/- 0-33%

client := pester.New()
client.Backoff = func(retry int) time.Duration {
    // set up something dynamic or use a look up table
    return time.Duration(retry) * time.Minute
}

Complete example

For a complete and working example, see the sample directory. pester allows you to use a constructor to control:

• backoff strategy
• retries
• concurrency
• keeping a log for debugging

package main

import (
    "log"
    "net/http"
    "strings"

    "github.com/sethgrid/pester"
)

func main() {
    log.Println("Starting...")

    { // drop in replacement for http.Get and other client methods
        resp, err := pester.Get("http://example.com")
        if err != nil {
            log.Println("error GETing example.com", err)
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }

    { // control the resiliency
        client := pester.New()
        client.Concurrency = 3
        client.MaxRetries = 5
        client.Backoff = pester.ExponentialBackoff
        client.KeepLog = true

        resp, err := client.Get("http://example.com")
        if err != nil {
            log.Println("error GETing example.com", client.LogString())
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }

    { // use the pester version of http.Client.Do
        req, err := http.NewRequest("POST", "http://example.com", strings.NewReader("data"))
        if err != nil {
            log.Fatal("Unable to create a new http request", err)
        }
        resp, err := pester.Do(req)
        if err != nil {
            log.Println("error POSTing example.com", err)
        }
        defer resp.Body.Close()
        log.Printf("example.com %s", resp.Status)
    }
}

Example Log

pester also allows you to control the resiliency and can optionally log the errors.

c := pester.New()
c.KeepLog = true

nonExistantURL := "http://localhost:9000/foo"
_, _ = c.Get(nonExistantURL)

fmt.Println(c.LogString())
/*
Output:

1432402837 Get [GET] http://localhost:9000/foo request-0 retry-0 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
1432402838 Get [GET] http://localhost:9000/foo request-0 retry-1 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
1432402839 Get [GET] http://localhost:9000/foo request-0 retry-2 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
*/

Tests

You can run tests in the root directory with $ go test. There is a benchmark-like test available with $ cd benchmarks; go test. You can see pester in action with $ cd sample; go run main.go.

For watching open file descriptors, you can run watch "lsof -i -P | grep main" if you started the app with go run main.go. I did this for watching for FD leaks. My method was to alter sample/main.go to only run one case (pester.Get with set backoff stategy, concurrency and retries increased) and adding a sleep after the result came back. This let me verify if FDs were getting left open when they should have closed. If you know a better way, let me know! I was able to see that FDs are now closing when they should :)

Are we there yet? Are we there yet? Are we there yet? Are we there yet? ...

Documentation

Overview

Package pester provides additional resiliency over the standard http client methods by allowing you to control request rates, retries, and a backoff strategy.

Index

• Variables
• func DefaultBackoff(_ int) time.Duration
• func Do(req *http.Request) (resp *http.Response, err error)
• func ExponentialBackoff(i int) time.Duration
• func ExponentialJitterBackoff(i int) time.Duration
• func Get(url string) (resp *http.Response, err error)
• func Head(url string) (resp *http.Response, err error)
• func LinearBackoff(i int) time.Duration
• func LinearJitterBackoff(i int) time.Duration
• func Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)
• func PostForm(url string, data url.Values) (resp *http.Response, err error)
• type BackoffStrategy
• type Client
  • func New() *Client
  • func (c *Client) Do(req *http.Request) (resp *http.Response, err error)
  • func (c *Client) FormatError(e ErrEntry) string
  • func (c *Client) Get(url string) (resp *http.Response, err error)
  • func (c *Client) Head(url string) (resp *http.Response, err error)
  • func (c *Client) Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)
  • func (c *Client) PostForm(url string, data url.Values) (resp *http.Response, err error)
  • func (c *Client) SetCheckRedirect(checkRedirect func(req *http.Request, via []*http.Request) error)
  • func (c *Client) SetJar(jar http.CookieJar)
  • func (c *Client) SetRetryOnHTTP429(flag bool)
  • func (c *Client) SetTimeout(timeout time.Duration)
  • func (c *Client) SetTransport(transport http.RoundTripper)
• type ContextLogHook
• type ErrEntry
• type LogHook
• type ResponseError

Variables

var (
	// ErrUnexpectedMethod occurs when an http.Client method is unable to be mapped from a calling method in the pester client
	ErrUnexpectedMethod = errors.New("unexpected client method, must be one of Do, Get, Head, Post, or PostFrom")
	// ErrReadingBody happens when we cannot read the body bytes
	// Deprecated: use ErrReadingRequestBody
	ErrReadingBody = errors.New("error reading body")
	// ErrReadingRequestBody happens when we cannot read the request body bytes
	ErrReadingRequestBody = errors.New("error reading request body")
)

var DefaultClient = &Client{MaxRetries: 3, Backoff: DefaultBackoff, ErrLog: []ErrEntry{}}

Functions

func DefaultBackoff

func DefaultBackoff(_ int) time.Duration

DefaultBackoff always returns 1 second

func Do

func Do(req *http.Request) (resp *http.Response, err error)

Do provides the same functionality as http.Client.Do and creates its own constructor

func ExponentialBackoff

func ExponentialBackoff(i int) time.Duration

ExponentialBackoff returns ever increasing backoffs by a power of 2

func ExponentialJitterBackoff

func ExponentialJitterBackoff(i int) time.Duration

ExponentialJitterBackoff returns ever increasing backoffs by a power of 2 with +/- 0-33% to prevent sychronized reuqests.

func Get

func Get(url string) (resp *http.Response, err error)

Get provides the same functionality as http.Client.Get and creates its own constructor

func Head

func Head(url string) (resp *http.Response, err error)

Head provides the same functionality as http.Client.Head and creates its own constructor

func LinearBackoff

func LinearBackoff(i int) time.Duration

LinearBackoff returns increasing durations, each a second longer than the last

func LinearJitterBackoff

func LinearJitterBackoff(i int) time.Duration

LinearJitterBackoff returns increasing durations, each a second longer than the last with +/- 0-33% to prevent sychronized reuqests.

func Post

func Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)

Post provides the same functionality as http.Client.Post and creates its own constructor

func PostForm

func PostForm(url string, data url.Values) (resp *http.Response, err error)

PostForm provides the same functionality as http.Client.PostForm and creates its own constructor

Types

type BackoffStrategy

type BackoffStrategy func(retry int) time.Duration

BackoffStrategy is used to determine how long a retry request should wait until attempted

type Client

type Client struct {
	Ratelimiter *rate.Limiter

	// pester specific
	MaxRetries     int
	Backoff        BackoffStrategy
	LogHook        LogHook
	ContextLogHook ContextLogHook

	ErrLog         []ErrEntry
	RetryOnHTTP429 bool
	// contains filtered or unexported fields
}

Client wraps the http client and exposes all the functionality of the http.Client. Additionally, Client provides pester specific values for handling resiliency.

func New

func New() *Client

New constructs a new DefaultClient with sensible default values

func (*Client) Do

func (c *Client) Do(req *http.Request) (resp *http.Response, err error)

Do provides the same functionality as http.Client.Do

func (*Client) FormatError

func (c *Client) FormatError(e ErrEntry) string

Format the Error to human readable string

func (*Client) Get

func (c *Client) Get(url string) (resp *http.Response, err error)

Get provides the same functionality as http.Client.Get

func (*Client) Head

func (c *Client) Head(url string) (resp *http.Response, err error)

Head provides the same functionality as http.Client.Head

func (*Client) Post

func (c *Client) Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error)

Post provides the same functionality as http.Client.Post

func (*Client) PostForm

func (c *Client) PostForm(url string, data url.Values) (resp *http.Response, err error)

PostForm provides the same functionality as http.Client.PostForm

func (*Client) SetCheckRedirect

func (c *Client) SetCheckRedirect(checkRedirect func(req *http.Request, via []*http.Request) error)

func (*Client) SetJar

func (c *Client) SetJar(jar http.CookieJar)

func (*Client) SetRetryOnHTTP429

func (c *Client) SetRetryOnHTTP429(flag bool)

set RetryOnHTTP429 for clients,

func (*Client) SetTimeout

func (c *Client) SetTimeout(timeout time.Duration)

func (*Client) SetTransport

func (c *Client) SetTransport(transport http.RoundTripper)

type ContextLogHook

type ContextLogHook func(ctx context.Context, e ErrEntry)

ContextLogHook does the same as LogHook but with passed Context

type ErrEntry

type ErrEntry struct {
	Time    time.Time
	Method  string
	URL     string
	Verb    string
	Request int
	Retry   int
	Attempt int
	Err     error
}

ErrEntry is used to provide the LogString() data and is populated each time an error happens if KeepLog is set. ErrEntry.Retry is deprecated in favor of ErrEntry.Attempt

type LogHook

type LogHook func(e ErrEntry)

LogHook is used to log attempts as they happen. This function is never called, however, if KeepLog is set to true.

type ResponseError

type ResponseError struct {
	Field   string `json:"field"`
	Message string `json:"message"`
	Code    int    `json:"code"`
	Value   string `json:"value"`
}