httpclient

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2024 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package httpclient contains helpers for doing HTTP requests (for example, round trippers for retrying and rate limiting).

Index

Examples

Constants

View Source 
const (
	DefaultRateLimitingBurst       = 1
	DefaultRateLimitingWaitTimeout = 15 * time.Second
)

Default parameter values for RateLimitingRoundTripper.

View Source 
const (
	DefaultMaxRetryAttempts                  = 10
	DefaultExponentialBackoffInitialInterval = time.Second
	DefaultExponentialBackoffMultiplier      = 2
)

Default parameter values for RetryableRoundTripper.

View Source 
const RetryAttemptNumberHeader = "X-Retry-Attempt"

RetryAttemptNumberHeader is an HTTP header name that will contain the serial number of the retry attempt.

View Source 
const UnlimitedRetryAttempts = -1

UnlimitedRetryAttempts should be used as RetryableRoundTripperOpts.MaxRetryAttempts value when we want to stop retries only by RetryableRoundTripperOpts.BackoffPolicy.

Variables

View Source 
var DefaultBackoffPolicy = retry.PolicyFunc(func() backoff.BackOff {
	bf := backoff.NewExponentialBackOff()
	bf.InitialInterval = DefaultExponentialBackoffInitialInterval
	bf.Multiplier = DefaultExponentialBackoffMultiplier
	bf.Reset()
	return bf
})

DefaultBackoffPolicy is a default backoff policy.

Functions

func CheckErrorIsTemporary 

func CheckErrorIsTemporary(err error) bool

CheckErrorIsTemporary checks either error is temporary or not.

func CloneHTTPHeader 

func CloneHTTPHeader(in http.Header) http.Header

CloneHTTPHeader creates a deep copy of an http.Header.

func CloneHTTPRequest 

func CloneHTTPRequest(req *http.Request) *http.Request

CloneHTTPRequest creates a shallow copy of the request along with a deep copy of the Headers.

func DefaultCheckRetry 

func DefaultCheckRetry(
	ctx context.Context, resp *http.Response, roundTripErr error, doneRetryAttempts int,
) (needRetry bool, err error)

DefaultCheckRetry represents default function to determine either retry is needed or not.

Types

type AuthBearerRoundTripper 

type AuthBearerRoundTripper struct {
	Delegate     http.RoundTripper
	AuthProvider AuthProvider
	// contains filtered or unexported fields
}

AuthBearerRoundTripper implements http.RoundTripper interface and sets Authorization HTTP header in all outgoing requests.

func NewAuthBearerRoundTripper 

func NewAuthBearerRoundTripper(delegate http.RoundTripper, authProvider AuthProvider) *AuthBearerRoundTripper

NewAuthBearerRoundTripper creates a new AuthBearerRoundTripper.

func NewAuthBearerRoundTripperWithOpts 

func NewAuthBearerRoundTripperWithOpts(delegate http.RoundTripper, authProvider AuthProvider,
	opts AuthBearerRoundTripperOpts) *AuthBearerRoundTripper

NewAuthBearerRoundTripperWithOpts creates a new AuthBearerRoundTripper with options.

func (*AuthBearerRoundTripper) RoundTrip 

func (rt *AuthBearerRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip executes a single HTTP transaction, returning a Response for the provided Request.

type AuthBearerRoundTripperError 

type AuthBearerRoundTripperError struct {
	Inner error
}

AuthBearerRoundTripperError is returned in RoundTrip method of AuthBearerRoundTripper when the original request cannot be potentially retried.

func (*AuthBearerRoundTripperError) Error 

func (e *AuthBearerRoundTripperError) Error() string

func (*AuthBearerRoundTripperError) Unwrap 

func (e *AuthBearerRoundTripperError) Unwrap() error

Unwrap returns the next error in the error chain.

type AuthBearerRoundTripperOpts 

type AuthBearerRoundTripperOpts struct {
	TokenScope []string
}

AuthBearerRoundTripperOpts is options for AuthBearerRoundTripper.

type AuthProvider 

type AuthProvider interface {
	GetToken(ctx context.Context, scope ...string) (string, error)
}

AuthProvider provide auth information that used for barear authrization

type CheckRetryFunc 

type CheckRetryFunc func(ctx context.Context, resp *http.Response, roundTripErr error, doneRetryAttempts int) (bool, error)

CheckRetryFunc is a function that is called right after RoundTrip() method and determines if the next retry attempt is needed.

type RateLimitingRoundTripper 

type RateLimitingRoundTripper struct {
	Delegate http.RoundTripper

	RateLimit   int
	Burst       int
	WaitTimeout time.Duration
	Adaptation  RateLimitingRoundTripperAdaptation
	// contains filtered or unexported fields
}

RateLimitingRoundTripper wraps implementing http.RoundTripper interface object and provides adaptive (can use limit from response's HTTP header) rate limiting mechanism for outgoing requests.

func NewRateLimitingRoundTripper 

func NewRateLimitingRoundTripper(delegate http.RoundTripper, rateLimit int) (*RateLimitingRoundTripper, error)

NewRateLimitingRoundTripper creates a new RateLimitingRoundTripper with specified rate limit.

Example

ExampleNewRateLimitingRoundTripper demonstrates the use of RateLimitingRoundTripper with default parameters.

Add "// Output:" in the end of the function and run: 

$ go test ./httpclient -v -run ExampleNewRateLimitingRoundTripper

Output will be like: 

[Req#1] 204 (0ms)
[Req#2] 204 (502ms)
[Req#3] 204 (497ms)
[Req#4] 204 (500ms)
[Req#5] 204 (503ms)

// Note: error handling is intentionally omitted so as not to overcomplicate the example.
// It is strictly necessary to handle all errors in real code.

server := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
	rw.WriteHeader(http.StatusNoContent)
}))

// Let's make transport that may do maximum 2 requests per second.
tr, _ := NewRateLimitingRoundTripper(http.DefaultTransport, 2)
httpClient := &http.Client{Transport: tr}

start := time.Now()
prev := time.Now()
for i := 0; i < 5; i++ {
	resp, _ := httpClient.Get(server.URL)
	_ = resp.Body.Close()
	now := time.Now()
	_, _ = fmt.Fprintf(os.Stderr, "[Req#%d] %d (%dms)\n", i+1, resp.StatusCode, now.Sub(prev).Milliseconds())
	prev = now
}
delta := time.Since(start) - time.Second*2
if delta > time.Millisecond*20 {
	fmt.Println("Total time is much greater than 2s")
} else {
	fmt.Println("Total time is about 2s")
}

Output:

Total time is about 2s

func NewRateLimitingRoundTripperWithOpts 

func NewRateLimitingRoundTripperWithOpts(
	delegate http.RoundTripper, rateLimit int, opts RateLimitingRoundTripperOpts,
) (*RateLimitingRoundTripper, error)

NewRateLimitingRoundTripperWithOpts creates a new RateLimitingRoundTripper with specified rate limit and options. For options that are not presented, the default values will be used.

Example

ExampleNewRateLimitingRoundTripperWithOpts demonstrates the use of RateLimitingRoundTripper. 

// Note: error handling is intentionally omitted so as not to overcomplicate the example.
// It is strictly necessary to handle all errors in real code.

server := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
	rw.WriteHeader(http.StatusNoContent)
}))

// Let's make transport that may do maximum 2 requests per second.
tr, _ := NewRateLimitingRoundTripperWithOpts(http.DefaultTransport, 2, RateLimitingRoundTripperOpts{
	WaitTimeout: time.Millisecond * 100, // Wait maximum 100ms.
})
httpClient := &http.Client{Transport: tr}

for i := 0; i < 2; i++ {
	resp, err := httpClient.Get(server.URL)
	if err != nil {
		var waitErr *RateLimitingWaitError
		if errors.As(err, &waitErr) {
			fmt.Printf("trying to do too many requests")
		}
		continue
	}
	_ = resp.Body.Close()
}

Output:

trying to do too many requests

func (*RateLimitingRoundTripper) RoundTrip 

func (rt *RateLimitingRoundTripper) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip executes a single HTTP transaction, returning a Response for the provided Request.

type RateLimitingRoundTripperAdaptation 

type RateLimitingRoundTripperAdaptation struct {
	ResponseHeaderName string
	SlackPercent       int
}

RateLimitingRoundTripperAdaptation represents a params to adapt rate limiting in accordance with value in response.

type RateLimitingRoundTripperOpts 

type RateLimitingRoundTripperOpts struct {
	Burst       int
	WaitTimeout time.Duration
	Adaptation  RateLimitingRoundTripperAdaptation
}

RateLimitingRoundTripperOpts represents an options for RateLimitingRoundTripper.

type RateLimitingWaitError 

type RateLimitingWaitError struct {
	Inner error
}

RateLimitingWaitError is returned in RoundTrip method of RateLimitingRoundTripper when rate limit is exceeded.

func (*RateLimitingWaitError) Error 

func (e *RateLimitingWaitError) Error() string

func (*RateLimitingWaitError) Unwrap 

func (e *RateLimitingWaitError) Unwrap() error

Unwrap returns the next error in the error chain.

type RetryableRoundTripper 

type RetryableRoundTripper struct {
	// Delegate is an object that implements http.RoundTripper interface
	// and is used for sending HTTP requests under the hood.
	Delegate http.RoundTripper

	// Logger is used for logging.
	// When it's necessary to use context-specific logger, LoggerProvider should be used instead.
	Logger log.FieldLogger

	// LoggerProvider is a function that provides a context-specific logger.
	// One of the typical use cases is to use a retryable client in the context of a request handler,
	// where the logger should produce logs with request-specific information (e.g., request ID).
	LoggerProvider func(ctx context.Context) log.FieldLogger

	// MaxRetryAttempts determines how many maximum retry attempts can be done.
	// The total number of sending HTTP request may be MaxRetryAttempts + 1 (the first request is not a retry attempt).
	// If its value is UnlimitedRetryAttempts, it's supposed that retry mechanism will be stopped by BackoffPolicy.
	// By default, DefaultMaxRetryAttempts const is used.
	MaxRetryAttempts int

	// CheckRetry is called right after RoundTrip() method and determines if the next retry attempt is needed.
	// By default, DefaultCheckRetry function is used.
	CheckRetry CheckRetryFunc

	// IgnoreRetryAfter determines if Retry-After HTTP header of the response is parsed and
	// used as a wait time before doing the next retry attempt.
	// If it's true or response doesn't contain Retry-After HTTP header, BackoffPolicy will be used for computing delay.
	IgnoreRetryAfter bool

	// BackoffPolicy is used for computing wait time before doing the next retry attempt
	// when the given response doesn't contain Retry-After HTTP header or IgnoreRetryAfter is true.
	// By default, DefaultBackoffPolicy is used.
	BackoffPolicy retry.Policy
}

RetryableRoundTripper wraps an object that implements http.RoundTripper interface and provides a retrying mechanism for HTTP requests.

func NewRetryableRoundTripper 

func NewRetryableRoundTripper(delegate http.RoundTripper) (*RetryableRoundTripper, error)

NewRetryableRoundTripper returns a new instance of RetryableRoundTripper.

Example

ExampleNewRetryableRoundTripper demonstrates the use of RetryableRoundTripper with default parameters.

To execute this example: 1) Add "// Output: Got 200 after 10 retry attempts" in the end of function. 2) Run: 

$ go test ./httpclient -v -run ExampleNewRetryableRoundTripper

Stderr will contain something like this: 

[Req#1] wait time: 0.001s
[Req#2] wait time: 1.109s
[Req#3] wait time: 2.882s
[Req#4] wait time: 4.661s
[Req#5] wait time: 7.507s
[Req#6] wait time: 14.797s
[Req#7] wait time: 37.984s
[Req#8] wait time: 33.942s
[Req#9] wait time: 39.394s
[Req#10] wait time: 35.820s
[Req#11] wait time: 48.060s

// Note: error handling is intentionally omitted so as not to overcomplicate the example.
// It is strictly necessary to handle all errors in real code.

reqTimes := make(chan time.Time, DefaultMaxRetryAttempts+1)

server := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
	reqTimes <- time.Now()
	if n, err := strconv.Atoi(r.Header.Get(RetryAttemptNumberHeader)); err == nil && n == DefaultMaxRetryAttempts {
		rw.WriteHeader(http.StatusOK)
		_, _ = rw.Write([]byte("ok, you win..."))
		return
	}
	rw.WriteHeader(http.StatusServiceUnavailable)
}))

prevTime := time.Now()

// Create RetryableRoundTripper with default params:
//	+ maximum retry attempts = 10 (DefaultMaxRetryAttempts)
//	+ respect Retry-After response HTTP header
//	+ exponential backoff policy (DefaultBackoffPolicy, multiplier = 2, initial interval = 1s)
tr, _ := NewRetryableRoundTripper(http.DefaultTransport)
httpClient := &http.Client{Transport: tr}

resp, err := httpClient.Get(server.URL)
if err != nil {
	log.Fatal(err)
}
_ = resp.Body.Close()
close(reqTimes)

reqsCount := 0
for reqTime := range reqTimes {
	reqsCount++
	_, _ = fmt.Fprintf(os.Stderr, "[Req#%d] wait time: %.3fs\n", reqsCount, reqTime.Sub(prevTime).Seconds())
	prevTime = reqTime
}
doneRetryAttempts := reqsCount - 1
fmt.Printf("Got %d after %d retry attempts", resp.StatusCode, doneRetryAttempts)

Output:

func NewRetryableRoundTripperWithOpts 

func NewRetryableRoundTripperWithOpts(
	delegate http.RoundTripper, opts RetryableRoundTripperOpts,
) (*RetryableRoundTripper, error)

NewRetryableRoundTripperWithOpts creates a new instance of RateLimitingRoundTripper with specified options.

func (*RetryableRoundTripper) RoundTrip 

func (rt *RetryableRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip performs request with retry logic. nolint: gocyclo

type RetryableRoundTripperError 

type RetryableRoundTripperError struct {
	Inner error
}

RetryableRoundTripperError is returned in RoundTrip method of RetryableRoundTripper when the original request cannot be potentially retried.

func (*RetryableRoundTripperError) Error 

func (e *RetryableRoundTripperError) Error() string

func (*RetryableRoundTripperError) Unwrap 

func (e *RetryableRoundTripperError) Unwrap() error

Unwrap returns the next error in the error chain.

type RetryableRoundTripperOpts 

type RetryableRoundTripperOpts struct {
	// Logger is used for logging.
	// When it's necessary to use context-specific logger, LoggerProvider should be used instead.
	Logger log.FieldLogger

	// LoggerProvider is a function that provides a context-specific logger.
	// One of the typical use cases is to use a retryable client in the context of a request handler,
	// where the logger should produce logs with request-specific information (e.g., request ID).
	LoggerProvider func(ctx context.Context) log.FieldLogger

	// MaxRetryAttempts determines how many maximum retry attempts can be done.
	// The total number of sending HTTP request may be MaxRetryAttempts + 1 (the first request is not a retry attempt).
	// If its value is UnlimitedRetryAttempts, it's supposed that retry mechanism will be stopped by BackoffPolicy.
	// By default, DefaultMaxRetryAttempts const is used.
	MaxRetryAttempts int

	// CheckRetry is called right after RoundTrip() method and determines if the next retry attempt is needed.
	// By default, DefaultCheckRetry function is used.
	CheckRetryFunc CheckRetryFunc

	// IgnoreRetryAfter determines if Retry-After HTTP header of the response is parsed and
	// used as a wait time before doing the next retry attempt.
	// If it's true or response doesn't contain Retry-After HTTP header, BackoffPolicy will be used for computing delay.
	IgnoreRetryAfter bool

	// BackoffPolicy is used for computing wait time before doing the next retry attempt
	// when the given response doesn't contain Retry-After HTTP header or IgnoreRetryAfter is true.
	// By default, DefaultBackoffPolicy is used.
	BackoffPolicy retry.Policy
}

RetryableRoundTripperOpts represents an options for RetryableRoundTripper.

type UserAgentRoundTripper 

type UserAgentRoundTripper struct {
	Delegate       http.RoundTripper
	UserAgent      string
	UpdateStrategy UserAgentUpdateStrategy
}

UserAgentRoundTripper implements http.RoundTripper interface and sets User-Agent HTTP header in all outgoing requests.

func NewUserAgentRoundTripper 

func NewUserAgentRoundTripper(delegate http.RoundTripper, userAgent string) *UserAgentRoundTripper

NewUserAgentRoundTripper creates a new UserAgentRoundTripper.

func NewUserAgentRoundTripperWithOpts 

func NewUserAgentRoundTripperWithOpts(
	delegate http.RoundTripper, userAgent string, opts UserAgentRoundTripperOpts,
) *UserAgentRoundTripper

NewUserAgentRoundTripperWithOpts creates a new UserAgentRoundTripper with specified options.

func (*UserAgentRoundTripper) RoundTrip 

func (rt *UserAgentRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip executes a single HTTP transaction, returning a Response for the provided Request.

type UserAgentRoundTripperOpts 

type UserAgentRoundTripperOpts struct {
	UpdateStrategy UserAgentUpdateStrategy
}

UserAgentRoundTripperOpts represents an options for UserAgentRoundTripper.

type UserAgentUpdateStrategy 

type UserAgentUpdateStrategy int

UserAgentUpdateStrategy represents a strategy for updating User-Agent HTTP header. 

const (
	UserAgentUpdateStrategySetIfEmpty UserAgentUpdateStrategy = iota
	UserAgentUpdateStrategyAppend
	UserAgentUpdateStrategyPrepend
)

User-Agent update strategies.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.