pacer

package
v1.58.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 29, 2022 License: MIT Imports: 5 Imported by: 129

Documentation

Overview

Package pacer makes pacing and retrying API calls easy

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsRetryAfter

func IsRetryAfter(err error) (retryAfter time.Duration, isRetryAfter bool)

IsRetryAfter returns true if the error or any of it's Cause's is an error returned by RetryAfterError. It also returns the associated Duration if possible.

func RetryAfterError

func RetryAfterError(err error, retryAfter time.Duration) error

RetryAfterError returns a wrapped error that can be used by Calculator implementations

Types

type AmazonCloudDrive

type AmazonCloudDrive struct {
	// contains filtered or unexported fields
}

AmazonCloudDrive is a specialized pacer for Amazon Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to 0..2**retries seconds.

See https://developer.amazon.com/public/apis/experience/cloud-drive/content/restful-api-best-practices

func NewAmazonCloudDrive

func NewAmazonCloudDrive(opts ...AmazonCloudDriveOption) *AmazonCloudDrive

NewAmazonCloudDrive returns a new AmazonCloudDrive Calculator with default values

func (*AmazonCloudDrive) Calculate

func (c *AmazonCloudDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*AmazonCloudDrive) Update

func (c *AmazonCloudDrive) Update(opts ...AmazonCloudDriveOption)

Update applies the Calculator options.

type AmazonCloudDriveOption

type AmazonCloudDriveOption interface {
	ApplyAmazonCloudDrive(*AmazonCloudDrive)
}

AmazonCloudDriveOption is the interface implemented by all options for the AmazonCloudDrive Calculator

type AttackConstant

type AttackConstant uint

AttackConstant configures the attack constant of a Calculator

func (AttackConstant) ApplyDefault

func (o AttackConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (AttackConstant) ApplyS3

func (o AttackConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type AzureIMDS added in v1.54.0

type AzureIMDS struct {
}

AzureIMDS is a pacer for the Azure instance metadata service.

func NewAzureIMDS added in v1.54.0

func NewAzureIMDS() *AzureIMDS

NewAzureIMDS returns a new Azure IMDS calculator.

func (*AzureIMDS) Calculate added in v1.54.0

func (c *AzureIMDS) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

type Burst

type Burst int

Burst configures the number of API calls to allow without sleeping

func (Burst) ApplyGoogleDrive

func (o Burst) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

type Calculator

type Calculator interface {
	// Calculate takes the current Pacer state and returns the sleep time after which
	// the next Pacer call will be done.
	Calculate(state State) time.Duration
}

Calculator is a generic calculation function for a Pacer.

type DecayConstant

type DecayConstant uint

DecayConstant configures the decay constant time of a Calculator

func (DecayConstant) ApplyDefault

func (o DecayConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (DecayConstant) ApplyS3

func (o DecayConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Default

type Default struct {
	// contains filtered or unexported fields
}

Default is a truncated exponential attack and decay.

On retries the sleep time is doubled, on non errors then sleeptime decays according to the decay constant as set with SetDecayConstant.

The sleep never goes below that set with SetMinSleep or above that set with SetMaxSleep.

func NewDefault

func NewDefault(opts ...DefaultOption) *Default

NewDefault creates a Calculator used by Pacer as the default.

func (*Default) Calculate

func (c *Default) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*Default) Update

func (c *Default) Update(opts ...DefaultOption)

Update applies the Calculator options.

type DefaultOption

type DefaultOption interface {
	ApplyDefault(*Default)
}

DefaultOption is the interface implemented by all options for the Default Calculator

type GoogleDrive

type GoogleDrive struct {
	// contains filtered or unexported fields
}

GoogleDrive is a specialized pacer for Google Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to (2 ^ n) + random_number_milliseconds seconds.

See https://developers.google.com/drive/v2/web/handle-errors#exponential-backoff

func NewGoogleDrive

func NewGoogleDrive(opts ...GoogleDriveOption) *GoogleDrive

NewGoogleDrive returns a new GoogleDrive Calculator with default values

func (*GoogleDrive) Calculate

func (c *GoogleDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*GoogleDrive) Update

func (c *GoogleDrive) Update(opts ...GoogleDriveOption)

Update applies the Calculator options.

type GoogleDriveOption

type GoogleDriveOption interface {
	ApplyGoogleDrive(*GoogleDrive)
}

GoogleDriveOption is the interface implemented by all options for the GoogleDrive Calculator

type InvokerFunc

type InvokerFunc func(try, tries int, f Paced) (bool, error)

InvokerFunc is the signature of the wrapper function used to invoke the target function in Pacer.

type MaxSleep

type MaxSleep time.Duration

MaxSleep configures the maximum sleep time of a Calculator

func (MaxSleep) ApplyDefault

func (o MaxSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MaxSleep) ApplyS3

func (o MaxSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type MinSleep

type MinSleep time.Duration

MinSleep configures the minimum sleep time of a Calculator

func (MinSleep) ApplyAmazonCloudDrive

func (o MinSleep) ApplyAmazonCloudDrive(c *AmazonCloudDrive)

ApplyAmazonCloudDrive updates the value on the Calculator

func (MinSleep) ApplyDefault

func (o MinSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MinSleep) ApplyGoogleDrive

func (o MinSleep) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

func (MinSleep) ApplyS3

func (o MinSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Option

type Option func(*pacerOptions)

Option can be used in New to configure the Pacer.

func CalculatorOption

func CalculatorOption(c Calculator) Option

CalculatorOption sets a Calculator for the new Pacer.

func InvokerOption

func InvokerOption(invoker InvokerFunc) Option

InvokerOption sets an InvokerFunc for the new Pacer.

func MaxConnectionsOption

func MaxConnectionsOption(maxConnections int) Option

MaxConnectionsOption sets the maximum connections number for the new Pacer.

func RetriesOption

func RetriesOption(retries int) Option

RetriesOption sets the retries number for the new Pacer.

type Paced

type Paced func() (bool, error)

Paced is a function which is called by the Call and CallNoRetry methods. It should return a boolean, true if it would like to be retried, and an error. This error may be returned or returned wrapped in a RetryError.

type Pacer

type Pacer struct {
	// contains filtered or unexported fields
}

Pacer is the primary type of the pacer package. It allows to retry calls with a configurable delay in between.

func New

func New(options ...Option) *Pacer

New returns a Pacer with sensible defaults.

func (*Pacer) Call

func (p *Pacer) Call(fn Paced) (err error)

Call paces the remote operations to not exceed the limits and retry on rate limit exceeded

This calls fn, expecting it to return a retry flag and an error. This error may be returned wrapped in a RetryError if the number of retries is exceeded.

func (*Pacer) CallNoRetry

func (p *Pacer) CallNoRetry(fn Paced) error

CallNoRetry paces the remote operations to not exceed the limits and return a retry error on rate limit exceeded

This calls fn and wraps the output in a RetryError if it would like it to be retried

func (*Pacer) ModifyCalculator

func (p *Pacer) ModifyCalculator(f func(Calculator))

ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.

func (*Pacer) SetCalculator

func (p *Pacer) SetCalculator(c Calculator)

SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.

It will choose the default algorithm if nil is passed in.

func (*Pacer) SetMaxConnections

func (p *Pacer) SetMaxConnections(n int)

SetMaxConnections sets the maximum number of concurrent connections. Setting the value to 0 will allow unlimited number of connections. Should not be changed once you have started calling the pacer. By default this will be set to fs.Config.Checkers.

func (*Pacer) SetRetries

func (p *Pacer) SetRetries(retries int)

SetRetries sets the max number of retries for Call

type S3

type S3 struct {
	// contains filtered or unexported fields
}

S3 implements a pacer compatible with our expectations of S3, where it tries to not delay at all between successful calls, but backs off in the default fashion in response to any errors. The assumption is that errors should be exceedingly rare (S3 seems to have largely solved the sort of stability questions rclone is likely to run into), and in the happy case it can handle calls with no delays between them.

Basically defaultPacer, but with some handling of sleepTime going to/from 0ms

func NewS3

func NewS3(opts ...S3Option) *S3

NewS3 returns a new S3 Calculator with default values

func (*S3) Calculate

func (c *S3) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*S3) Update

func (c *S3) Update(opts ...S3Option)

Update applies the Calculator options.

type S3Option

type S3Option interface {
	ApplyS3(*S3)
}

S3Option is the interface implemented by all options for the S3 Calculator

type State

type State struct {
	SleepTime          time.Duration // current time to sleep before adding the pacer token back
	ConsecutiveRetries int           // number of consecutive retries, will be 0 when the last invoker call returned false
	LastError          error         // the error returned by the last invoker call or nil
}

State represents the public Pacer state that will be passed to the configured Calculator

type TokenDispenser

type TokenDispenser struct {
	// contains filtered or unexported fields
}

TokenDispenser is for controlling concurrency

func NewTokenDispenser

func NewTokenDispenser(n int) *TokenDispenser

NewTokenDispenser makes a pool of n tokens

func (*TokenDispenser) Get

func (td *TokenDispenser) Get()

Get gets a token from the pool - don't forget to return it with Put

func (*TokenDispenser) Put

func (td *TokenDispenser) Put()

Put returns a token

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL