README
¶
We copied this code from here. Basically we needed to change the RetryNotify function such that we can optionally stop retrying when we encounter certain types of errors.
Following is the original readme:
Exponential Backoff 
This is a Go port of the exponential backoff algorithm from Google's HTTP Client Library for Java.
Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate. The retries exponentially increase and stop increasing when a certain threshold is met.
Usage
See https://godoc.org/github.com/cenkalti/backoff#pkg-examples
Contributing
- I would like to keep this library as small as possible.
- Please don't send a PR without opening an issue and discussing it first.
- If proposed change is not a common use case, I will probably not accept it.
Documentation
¶
Overview ¶
Package backoff implements backoff algorithms for retrying operations.
Use Retry function for retrying operations that may fail. If Retry does not meet your needs, copy/paste the function into your project and modify as you wish.
There is also Ticker type similar to time.Ticker. You can use it if you need to work with channels.
See Examples section below for usage examples.
Index ¶
- Constants
- Variables
- func GetRandomValueFromInterval(randomizationFactor, random float64, currentInterval time.Duration) time.Duration
- func Retry(o Operation, b BackOff) error
- func RetryNotify(operation Operation, b BackOff, notify Notify) error
- func RetryUntilCancel(ctx context.Context, operation Operation, b BackOff, notify Notify) error
- type BackOff
- type Clock
- type ConstantBackOff
- type ExponentialBackOff
- type Notify
- type Operation
- type StopBackOff
- type Ticker
- type ZeroBackOff
Examples ¶
Constants ¶
const ( DefaultInitialInterval = 500 * time.Millisecond DefaultRandomizationFactor = 0.5 DefaultMultiplier = 1.5 DefaultMaxInterval = 60 * time.Second DefaultMaxElapsedTime = 15 * time.Minute )
Default values for ExponentialBackOff.
const Stop time.Duration = -1
Stop indicates that no more retries should be made for use in NextBackOff().
Variables ¶
var ErrContinue = errors.New("looping through backoff")
ErrContinue is a sentinel error designed to be used with Retry/RetryNotify/RetryUntilCancel and NotifyContinue.
Retry/RetryNotify/RetryUntilCancel: if operation() returns ErrContinue, RetryUntilCancel will reset its backoff and call notify() (if set) with the error.
NotifyContinue: by extension, NotifyContinue always returns nil (causing operation() to be retried) when operation() returns ErrContinue (the typical usage of ErrContinue).
The two combined allow semantics similar to 'continue' in a regular loop: operation() is re-run from the beginning, as a loop's body would be.
var SystemClock = systemClock{}
SystemClock implements Clock interface that uses time.Now().
Functions ¶
func GetRandomValueFromInterval ¶
func GetRandomValueFromInterval(randomizationFactor, random float64, currentInterval time.Duration) time.Duration
Returns a random value from the following interval:
[randomizationFactor * currentInterval, randomizationFactor * currentInterval].
func Retry ¶
Retry the operation o until it does not return error or BackOff stops. o is guaranteed to be run at least once. It is the caller's responsibility to reset b after Retry returns.
Retry sleeps the goroutine for the duration returned by BackOff after a failed operation returns.
Example ¶
package main
import (
"github.com/pachyderm/pachyderm/v2/src/internal/backoff"
)
func main() {
// An operation that may fail.
operation := func() error {
return nil // or an error
}
err := backoff.Retry(operation, backoff.NewExponentialBackOff())
if err != nil {
// Handle error.
return
}
// Operation is successful.
}
Output:
func RetryNotify ¶
RetryNotify calls notify function with the error and wait duration for each failed attempt before sleep.
Types ¶
type BackOff ¶
type BackOff interface {
// NextBackOff returns the duration to wait before retrying the operation,
// or backoff.Stop to indicate that no more retries should be made.
//
// Example usage:
//
// duration := backoff.NextBackOff();
// if (duration == backoff.Stop) {
// // Do not retry operation.
// } else {
// // Sleep for duration and retry operation.
// }
//
NextBackOff() time.Duration
// Reset to initial state.
Reset()
}
BackOff is a backoff policy for retrying an operation.
type ConstantBackOff ¶
type ConstantBackOff struct {
Interval time.Duration
// After MaxElapsedTime the ConstantBackOff stops.
// It never stops if MaxElapsedTime == 0.
MaxElapsedTime time.Duration
// contains filtered or unexported fields
}
ConstantBackOff is a backoff policy that always returns the same backoff delay. This is in contrast to an exponential backoff policy, which returns a delay that grows longer as you call NextBackOff() over and over again.
Note: Implementation is not thread-safe
func NewConstantBackOff ¶
func NewConstantBackOff(d time.Duration) *ConstantBackOff
NewConstantBackOff ...
func RetryEvery ¶
func RetryEvery(d time.Duration) *ConstantBackOff
RetryEvery is an alias for NewConstantBackoff, with a nicer name for inline calls
func (*ConstantBackOff) For ¶
func (b *ConstantBackOff) For(maxElapsed time.Duration) *ConstantBackOff
For sets b.MaxElapsedTime to 'maxElapsed' and returns b
func (*ConstantBackOff) GetElapsedTime ¶
func (b *ConstantBackOff) GetElapsedTime() time.Duration
GetElapsedTime returns the elapsed time since an ExponentialBackOff instance is created and is reset when Reset() is called.
The elapsed time is computed using time.Now().UnixNano().
func (*ConstantBackOff) NextBackOff ¶
func (b *ConstantBackOff) NextBackOff() time.Duration
NextBackOff ...
type ExponentialBackOff ¶
type ExponentialBackOff struct {
InitialInterval time.Duration
RandomizationFactor float64
Multiplier float64
MaxInterval time.Duration
// After MaxElapsedTime the ExponentialBackOff stops.
// It never stops if MaxElapsedTime == 0.
MaxElapsedTime time.Duration
Clock Clock
CurrentInterval time.Duration
StartTime time.Time
}
ExponentialBackOff is a backoff implementation that increases the backoff period for each retry attempt using a randomization function that grows exponentially.
NextBackOff() is calculated using the following formula:
randomized interval =
RetryInterval * (random value in range [1 - RandomizationFactor, 1 + RandomizationFactor])
In other words NextBackOff() will range between the randomization factor percentage below and above the retry interval.
For example, given the following parameters:
RetryInterval = 2 RandomizationFactor = 0.5 Multiplier = 2
the actual backoff period used in the next retry attempt will range between 1 and 3 seconds, multiplied by the exponential, that is, between 2 and 6 seconds.
Note: MaxInterval caps the RetryInterval and not the randomized interval.
If the time elapsed since an ExponentialBackOff instance is created goes past the MaxElapsedTime, then the method NextBackOff() starts returning backoff.Stop.
The elapsed time can be reset by calling Reset().
Example: Given the following default arguments, for 10 tries the sequence will be, and assuming we go over the MaxElapsedTime on the 10th try:
Request # RetryInterval (seconds) Randomized Interval (seconds) 1 0.5 [0.25, 0.75] 2 0.75 [0.375, 1.125] 3 1.125 [0.562, 1.687] 4 1.687 [0.8435, 2.53] 5 2.53 [1.265, 3.795] 6 3.795 [1.897, 5.692] 7 5.692 [2.846, 8.538] 8 8.538 [4.269, 12.807] 9 12.807 [6.403, 19.210] 10 19.210 backoff.Stop
Note: Implementation is not thread-safe.
func New60sBackOff ¶
func New60sBackOff() *ExponentialBackOff
New60sBackOff returns a backoff identical to New10sBackOff except with a longer MaxElapsedTime This may be more useful for watcher and controllers (e.g. the PPS master or the worker) than New10sBackOff, which is a length of time that makes more sense for the critical paths of slow RPCs (e.g. PutFile)
func NewExponentialBackOff ¶
func NewExponentialBackOff() *ExponentialBackOff
NewExponentialBackOff creates an instance of ExponentialBackOff using default values.
func NewInfiniteBackOff ¶
func NewInfiniteBackOff() *ExponentialBackOff
NewInfiniteBackOff creates an instance of ExponentialBackOff that never ends.
func NewTestingBackOff ¶
func NewTestingBackOff() *ExponentialBackOff
NewTestingBackOff returns a backoff tuned towards waiting for a Pachyderm state change in a test
func (*ExponentialBackOff) GetElapsedTime ¶
func (b *ExponentialBackOff) GetElapsedTime() time.Duration
GetElapsedTime returns the elapsed time since an ExponentialBackOff instance is created and is reset when Reset() is called.
The elapsed time is computed using time.Now().UnixNano().
func (*ExponentialBackOff) NextBackOff ¶
func (b *ExponentialBackOff) NextBackOff() time.Duration
NextBackOff calculates the next backoff interval using the formula:
Randomized interval = RetryInterval +/- (RandomizationFactor * RetryInterval)
func (*ExponentialBackOff) Reset ¶
func (b *ExponentialBackOff) Reset()
Reset the interval back to the initial retry interval and restarts the timer.
type Notify ¶
Notify is a notify-on-error function. It receives an operation error and backoff delay if the operation failed (with an error).
If the notify function returns an error itself, we stop retrying and return the error.
NOTE that if the backoff policy stated to stop retrying, the notify function isn't called.
func NotifyContinue ¶
func NotifyContinue(inner interface{}) Notify
NotifyContinue is a convenience function for use with RetryUntilCancel. If 'inner' is set to a Notify function, it's called if 'err' is anything other than ErrContinue. If 'inner' is a string or any other non-nil value of another type, any error other than ErrContinue is logged, and the loop is re-run (in this case, there is no way to escape the backoff--RetryUntilCancel must be used to avoid an infinite loop). If 'inner' is nil, any error other than ErrContinue is returned.
This is useful for e.g. monitoring functions that want to repeatedly execute the same control loop until their context is cancelled.
func NotifyCtx ¶
NotifyCtx is a convenience function for use with RetryNotify that exits if 'ctx' is closed, and otherwise logs the error and retries.
Note that an alternative, if the only goal is to retry until 'ctx' is closed, is to use RetryUntilCancel, which will not even call the given 'notify' function if its context is cancelled (RetryUntilCancel with notify=nil is similar to RetryNotify with NotifyCtx, except that with the former, the backoff will be ended early if the ctx is cancelled, whereas the latter will sleep for the full backoff duration and then call the operation again).
type Operation ¶
type Operation func() error
An Operation is executing by Retry() or RetryNotify(). The operation will be retried using a backoff policy if it returns an error.
func MustLoop ¶
MustLoop is a convenience function for use with RetryUntilCancel. It wraps 'operation' in another Operation that returns ErrContinue if the inner operation returns nil. If used with RetryUntilCancel and NotifyContinue, this guarantees that 'operation'is retried until 'ctx' is cancelled, even if it returns nil.
type StopBackOff ¶
type StopBackOff struct{}
StopBackOff is a fixed backoff policy that always returns backoff.Stop for NextBackOff(), meaning that the operation should never be retried.
type Ticker ¶
Ticker holds a channel that delivers `ticks' of a clock at times reported by a BackOff.
Ticks will continue to arrive when the previous operation is still running, so operations that take a while to fail could run in quick succession.
Example ¶
package main
import (
"fmt"
"github.com/pachyderm/pachyderm/v2/src/internal/backoff"
)
func main() {
// An operation that may fail.
operation := func() error {
return nil // or an error
}
ticker := backoff.NewTicker(backoff.NewExponentialBackOff())
var err error
// Ticks will continue to arrive when the previous operation is still running,
// so operations that take a while to fail could run in quick succession.
for range ticker.C {
if err = operation(); err != nil {
fmt.Println(err, "will retry...")
continue
}
ticker.Stop()
break
}
if err != nil {
// Operation has failed.
return
}
}
Output:
type ZeroBackOff ¶
type ZeroBackOff struct{}
ZeroBackOff is a fixed backoff policy whose backoff time is always zero, meaning that the operation is retried immediately without waiting, indefinitely.
Source Files