Documentation
¶
Overview ¶
Package pacer makes pacing and retrying API calls easy
Index ¶
- func IsRetryAfter(err error) (retryAfter time.Duration, isRetryAfter bool)
- func RetryAfterError(err error, retryAfter time.Duration) error
- type AmazonCloudDrive
- type AmazonCloudDriveOption
- type AttackConstant
- type Burst
- type Calculator
- type DecayConstant
- type Default
- type DefaultOption
- type GoogleDrive
- type GoogleDriveOption
- type InvokerFunc
- type MaxSleep
- type MinSleep
- type Option
- type Paced
- type Pacer
- type S3
- type S3Option
- type State
- type TokenDispenser
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsRetryAfter ¶
IsRetryAfter returns true if the the error or any of it's Cause's is an error returned by RetryAfterError. It also returns the associated Duration if possible.
Types ¶
type AmazonCloudDrive ¶
type AmazonCloudDrive struct {
// contains filtered or unexported fields
}
AmazonCloudDrive is a specialized pacer for Amazon Drive
It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to 0..2**retries seconds.
See https://developer.amazon.com/public/apis/experience/cloud-drive/content/restful-api-best-practices
func NewAmazonCloudDrive ¶
func NewAmazonCloudDrive(opts ...AmazonCloudDriveOption) *AmazonCloudDrive
NewAmazonCloudDrive returns a new AmazonCloudDrive Calculator with default values
func (*AmazonCloudDrive) Calculate ¶
func (c *AmazonCloudDrive) Calculate(state State) time.Duration
Calculate takes the current Pacer state and return the wait time until the next try.
func (*AmazonCloudDrive) Update ¶
func (c *AmazonCloudDrive) Update(opts ...AmazonCloudDriveOption)
Update applies the Calculator options.
type AmazonCloudDriveOption ¶
type AmazonCloudDriveOption interface {
ApplyAmazonCloudDrive(*AmazonCloudDrive)
}
AmazonCloudDriveOption is the interface implemented by all options for the AmazonCloudDrive Calculator
type AttackConstant ¶
type AttackConstant uint
AttackConstant configures the attack constant of a Calculator
func (AttackConstant) ApplyDefault ¶
func (o AttackConstant) ApplyDefault(c *Default)
ApplyDefault updates the value on the Calculator
func (AttackConstant) ApplyS3 ¶
func (o AttackConstant) ApplyS3(c *S3)
ApplyS3 updates the value on the Calculator
type Burst ¶
type Burst int
Burst configures the number of API calls to allow without sleeping
func (Burst) ApplyGoogleDrive ¶
func (o Burst) ApplyGoogleDrive(c *GoogleDrive)
ApplyGoogleDrive updates the value on the Calculator
type Calculator ¶
type Calculator interface {
// Calculate takes the current Pacer state and returns the sleep time after which
// the next Pacer call will be done.
Calculate(state State) time.Duration
}
Calculator is a generic calculation function for a Pacer.
type DecayConstant ¶
type DecayConstant uint
DecayConstant configures the decay constant time of a Calculator
func (DecayConstant) ApplyDefault ¶
func (o DecayConstant) ApplyDefault(c *Default)
ApplyDefault updates the value on the Calculator
func (DecayConstant) ApplyS3 ¶
func (o DecayConstant) ApplyS3(c *S3)
ApplyS3 updates the value on the Calculator
type Default ¶
type Default struct {
// contains filtered or unexported fields
}
Default is a truncated exponential attack and decay.
On retries the sleep time is doubled, on non errors then sleeptime decays according to the decay constant as set with SetDecayConstant.
The sleep never goes below that set with SetMinSleep or above that set with SetMaxSleep.
func NewDefault ¶
func NewDefault(opts ...DefaultOption) *Default
NewDefault creates a Calculator used by Pacer as the default.
func (*Default) Calculate ¶
Calculate takes the current Pacer state and return the wait time until the next try.
func (*Default) Update ¶
func (c *Default) Update(opts ...DefaultOption)
Update applies the Calculator options.
type DefaultOption ¶
type DefaultOption interface {
ApplyDefault(*Default)
}
DefaultOption is the interface implemented by all options for the Default Calculator
type GoogleDrive ¶
type GoogleDrive struct {
// contains filtered or unexported fields
}
GoogleDrive is a specialized pacer for Google Drive
It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to (2 ^ n) + random_number_milliseconds seconds.
See https://developers.google.com/drive/v2/web/handle-errors#exponential-backoff
func NewGoogleDrive ¶
func NewGoogleDrive(opts ...GoogleDriveOption) *GoogleDrive
NewGoogleDrive returns a new GoogleDrive Calculator with default values
func (*GoogleDrive) Calculate ¶
func (c *GoogleDrive) Calculate(state State) time.Duration
Calculate takes the current Pacer state and return the wait time until the next try.
func (*GoogleDrive) Update ¶
func (c *GoogleDrive) Update(opts ...GoogleDriveOption)
Update applies the Calculator options.
type GoogleDriveOption ¶
type GoogleDriveOption interface {
ApplyGoogleDrive(*GoogleDrive)
}
GoogleDriveOption is the interface implemented by all options for the GoogleDrive Calculator
type InvokerFunc ¶
InvokerFunc is the signature of the wrapper function used to invoke the target function in Pacer.
type MaxSleep ¶
MaxSleep configures the maximum sleep time of a Calculator
func (MaxSleep) ApplyDefault ¶
ApplyDefault updates the value on the Calculator
type MinSleep ¶
MinSleep configures the minimum sleep time of a Calculator
func (MinSleep) ApplyAmazonCloudDrive ¶
func (o MinSleep) ApplyAmazonCloudDrive(c *AmazonCloudDrive)
ApplyAmazonCloudDrive updates the value on the Calculator
func (MinSleep) ApplyDefault ¶
ApplyDefault updates the value on the Calculator
func (MinSleep) ApplyGoogleDrive ¶
func (o MinSleep) ApplyGoogleDrive(c *GoogleDrive)
ApplyGoogleDrive updates the value on the Calculator
type Option ¶
type Option func(*pacerOptions)
Option can be used in New to configure the Pacer.
func CalculatorOption ¶
func CalculatorOption(c Calculator) Option
CalculatorOption sets a Calculator for the new Pacer.
func InvokerOption ¶
func InvokerOption(invoker InvokerFunc) Option
InvokerOption sets a InvokerFunc for the new Pacer.
func MaxConnectionsOption ¶
MaxConnectionsOption sets the maximum connections number for the new Pacer.
func RetriesOption ¶
RetriesOption sets the retries number for the new Pacer.
type Paced ¶
Paced is a function which is called by the Call and CallNoRetry methods. It should return a boolean, true if it would like to be retried, and an error. This error may be returned or returned wrapped in a RetryError.
type Pacer ¶
type Pacer struct {
// contains filtered or unexported fields
}
Pacer is the primary type of the pacer package. It allows to retry calls with a configurable delay in between.
func (*Pacer) Call ¶
Call paces the remote operations to not exceed the limits and retry on rate limit exceeded
This calls fn, expecting it to return a retry flag and an error. This error may be returned wrapped in a RetryError if the number of retries is exceeded.
func (*Pacer) CallNoRetry ¶
CallNoRetry paces the remote operations to not exceed the limits and return a retry error on rate limit exceeded
This calls fn and wraps the output in a RetryError if it would like it to be retried
func (*Pacer) ModifyCalculator ¶
func (p *Pacer) ModifyCalculator(f func(Calculator))
ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.
func (*Pacer) SetCalculator ¶
func (p *Pacer) SetCalculator(c Calculator)
SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.
It will choose the default algorithm if nil is passed in.
func (*Pacer) SetMaxConnections ¶
SetMaxConnections sets the maximum number of concurrent connections. Setting the value to 0 will allow unlimited number of connections. Should not be changed once you have started calling the pacer. By default this will be set to fs.Config.Checkers.
func (*Pacer) SetRetries ¶
SetRetries sets the max number of retries for Call
type S3 ¶
type S3 struct {
// contains filtered or unexported fields
}
S3 implements a pacer compatible with our expectations of S3, where it tries to not delay at all between successful calls, but backs off in the default fashion in response to any errors. The assumption is that errors should be exceedingly rare (S3 seems to have largely solved the sort of stability questions rclone is likely to run into), and in the happy case it can handle calls with no delays between them.
Basically defaultPacer, but with some handling of sleepTime going to/from 0ms
type S3Option ¶
type S3Option interface {
ApplyS3(*S3)
}
S3Option is the interface implemented by all options for the S3 Calculator
type State ¶
type State struct {
SleepTime time.Duration // current time to sleep before adding the pacer token back
ConsecutiveRetries int // number of consecutive retries, will be 0 when the last invoker call returned false
LastError error // the error returned by the last invoker call or nil
}
State represents the public Pacer state that will be passed to the configured Calculator
type TokenDispenser ¶
type TokenDispenser struct {
// contains filtered or unexported fields
}
TokenDispenser is for controlling concurrency
func NewTokenDispenser ¶
func NewTokenDispenser(n int) *TokenDispenser
NewTokenDispenser makes a pool of n tokens
func (*TokenDispenser) Get ¶
func (td *TokenDispenser) Get()
Get gets a token from the pool - don't forget to return it with Put
Source Files