utils

README

web-utils

[DEPRECATED] New features will be implemented in https://github.com/ShevaXu/golang.

web-utils provides useful web-dev features following some best practices for production, including:

HTTP Utils

A SafeClient with

• timeout setting for underlying http.Client;
• request retries (can be timeout only);
• exponential backoff.

Just utils.StdClient() to get a preset-client or cl := utils.SafeClient{...} for a custom one.

Semaphore

For Bounding resource use.

s := NewSemaphore(10)

go func() {
    ctx := context.Background() // for cancellation
    if s.Obtain(ctx) {
        defer s.Release()
        // do whatever 
    }
}()

For weighted semaphore, see []this implementation](https://github.com/golang/sync/blob/master/semaphore/semaphore.go).

Asserting

Tiny functions for testing.

a := assert.NewAssert()

a.True(...)
a.Equal(...)
a.NotEqual(...)
a.Nil(...)
a.NotNil(...)
a.NoError(...)

Install

go get github.com/ShevaXu/web-utils

TODO

• context support for client;
• rate-limit (leaky bucket);
• circuit-breaker.

Documentation

Overview

Package utils provides useful features to Go's http.Client following some best practices for production, such as timeout, retries and backoff.

Index

• func IsTimeoutErr(e error) bool
• func NewFormPost(url string, v url.Values, f RequestHook) (*http.Request, error)
• func NewJSONPost(url string, v interface{}, f RequestHook) (*http.Request, error)
• func ShouldRetry(statusCode int) bool
• type Backoff
  • func (b *Backoff) Next(previous int) int
• type HTTPClient
• type RequestHook
• type SafeClient
  • func StdClient() *SafeClient
  • func (c *SafeClient) DoRequest(method, url string, content []byte, maxTries int, f RequestHook) (tries, status int, body []byte, err error)
  • func (c *SafeClient) PostFormWithRetry(url string, v url.Values, maxTries int, f RequestHook) (tries, status int, body []byte, err error)
  • func (c *SafeClient) PostJSONWithRetry(url string, v interface{}, maxTries int, f RequestHook) (tries, status int, body []byte, err error)
  • func (c *SafeClient) RequestWithClose(req *http.Request) (status int, body []byte, err error)
  • func (c *SafeClient) RequestWithRetry(req *http.Request, maxTries int) (tries, status int, body []byte, err error)
• type Semaphore
  • func NewSemaphore(n int) Semaphore

Functions

func IsTimeoutErr

func IsTimeoutErr(e error) bool

IsTimeoutErr checks if an error is timeout by cast it to net.Error.

func NewFormPost

func NewFormPost(url string, v url.Values, f RequestHook) (*http.Request, error)

NewFormPost returns a Request with default "Content-type: text/plain".

func NewJSONPost

func NewJSONPost(url string, v interface{}, f RequestHook) (*http.Request, error)

NewJSONPost returns a Request with json encoded and header set; additional headers or cookies can be set through the RequestHook.

func ShouldRetry

func ShouldRetry(statusCode int) bool

ShouldRetry determines if the client should repeat the request without modifications at any later time; returns true for http 408 and 5xx status.

Types

type Backoff

type Backoff struct {
	BaseSleep, MaxSleep int
}

Backoff implements the exponential backoff algorithm with jitter for client sending remote calls. It use an alternative method described in https://www.awsarchitectureblog.com/2015/03/backoff.html:

func (*Backoff) Next

func (b *Backoff) Next(previous int) int

Next returns the next sleep time computed by the previous one; the Decorrelated Jitter is: sleep = min(cap, random_between(base, sleep * 3)).

type HTTPClient

type HTTPClient interface {
	RequestWithRetry(req *http.Request, maxTries int) (tries, status int, body []byte, err error)
	DoRequest(method, url string, content []byte, maxTries int, f RequestHook) (tries, status int, body []byte, err error)
}

HTTPClient provides additional features upon http.Client, e.g., io Reader handle and request retry; it also normalize the HTTP response.

type RequestHook

type RequestHook func(req *http.Request)

RequestHook can modify the Request anyway it wants.

type SafeClient

type SafeClient struct {
	TimeoutOnly bool
	http.Client // embedded
	Backoff
}

SafeClient implements HTTPClient; it wraps a http.Client underneath (safe for concurrent use by multiple goroutines).

func StdClient

func StdClient() *SafeClient

StdClient gives a ready-to-use SafeClient instance.

func (*SafeClient) DoRequest

func (c *SafeClient) DoRequest(method, url string, content []byte, maxTries int, f RequestHook) (tries, status int, body []byte, err error)

DoRequest is the generalized version of RequestWithRetry that initialize a Request each time to ensure Body get consumed. Additional headers or cookies can be set through the RequestHook.

func (*SafeClient) PostFormWithRetry

func (c *SafeClient) PostFormWithRetry(url string, v url.Values, maxTries int, f RequestHook) (tries, status int, body []byte, err error)

PostFormWithRetry is a convenient method for form POST requests.

func (*SafeClient) PostJSONWithRetry

func (c *SafeClient) PostJSONWithRetry(url string, v interface{}, maxTries int, f RequestHook) (tries, status int, body []byte, err error)

PostJSONWithRetry is a convenient method for JSON POST requests.

func (*SafeClient) RequestWithClose

func (c *SafeClient) RequestWithClose(req *http.Request) (status int, body []byte, err error)

RequestWithClose sends the request and returns statusCode and raw body. It reads and closes Response.Body, return any error occurs.

func (*SafeClient) RequestWithRetry

func (c *SafeClient) RequestWithRetry(req *http.Request, maxTries int) (tries, status int, body []byte, err error)

RequestWithRetry wraps RequestWithClose and exponential-backoff retries in following conditions: 1. timeout error occurs (mostly client-side); 2. server-side should-retry statusCode returned. It returns the last response if tries run out. NOTICE: retry works for request with no body only before go1.9.

type Semaphore

type Semaphore interface {
	// Obtain puts one into the semaphore, returns true if succeeds.
	// It blocks utils succeeds or the context cancelled.
	// Obtaining from a closed semaphore should return false.
	Obtain(context.Context) bool

	// Release takes one from the semaphore, returns true if succeeds.
	// It should never blocks.
	Release() bool

	// Capacity returns semaphore's max concurrent resources.
	Capacity() int

	// Count returns semaphore's current used resources.
	Count() int

	// Close stops obtaining resources from semaphore,
	// it makes Obtain() return false ever since.
	Close()

	// Closed tells if semaphore is closed.
	Closed() bool
}

Semaphore is bounded resources abstraction. Ref: https://github.com/golang/go/wiki/BoundingResourceUse

func NewSemaphore

func NewSemaphore(n int) Semaphore

NewSemaphore returns an internal semaphore. This is the exported interface for using semaphore.