pacer

package
v1.58.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 29, 2022 License: MIT Imports: 5 Imported by: 129

Documentation

Overview

Package pacer makes pacing and retrying API calls easy

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsRetryAfter 

func IsRetryAfter(err error) (retryAfter time.Duration, isRetryAfter bool)

IsRetryAfter returns true if the error or any of it's Cause's is an error returned by RetryAfterError. It also returns the associated Duration if possible.

func RetryAfterError 

func RetryAfterError(err error, retryAfter time.Duration) error

RetryAfterError returns a wrapped error that can be used by Calculator implementations

Types

type AmazonCloudDrive 

type AmazonCloudDrive struct {
	// contains filtered or unexported fields
}

AmazonCloudDrive is a specialized pacer for Amazon Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to 0..2**retries seconds.

See https://developer.amazon.com/public/apis/experience/cloud-drive/content/restful-api-best-practices

func NewAmazonCloudDrive 

func NewAmazonCloudDrive(opts ...AmazonCloudDriveOption) *AmazonCloudDrive

NewAmazonCloudDrive returns a new AmazonCloudDrive Calculator with default values

func (*AmazonCloudDrive) Calculate 

func (c *AmazonCloudDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*AmazonCloudDrive) Update 

func (c *AmazonCloudDrive) Update(opts ...AmazonCloudDriveOption)

Update applies the Calculator options.

type AmazonCloudDriveOption 

type AmazonCloudDriveOption interface {
	ApplyAmazonCloudDrive(*AmazonCloudDrive)
}

AmazonCloudDriveOption is the interface implemented by all options for the AmazonCloudDrive Calculator

type AttackConstant 

type AttackConstant uint

AttackConstant configures the attack constant of a Calculator

func (AttackConstant) ApplyDefault 

func (o AttackConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (AttackConstant) ApplyS3 

func (o AttackConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type AzureIMDS added in v1.54.0 

type AzureIMDS struct {
}

AzureIMDS is a pacer for the Azure instance metadata service.

func NewAzureIMDS added in v1.54.0 

func NewAzureIMDS() *AzureIMDS

NewAzureIMDS returns a new Azure IMDS calculator.

func (*AzureIMDS) Calculate added in v1.54.0 

func (c *AzureIMDS) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

type Burst 

type Burst int

Burst configures the number of API calls to allow without sleeping

func (Burst) ApplyGoogleDrive 

func (o Burst) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

type Calculator 

type Calculator interface {
	// Calculate takes the current Pacer state and returns the sleep time after which
	// the next Pacer call will be done.
	Calculate(state State) time.Duration
}

Calculator is a generic calculation function for a Pacer.

type DecayConstant 

type DecayConstant uint

DecayConstant configures the decay constant time of a Calculator

func (DecayConstant) ApplyDefault 

func (o DecayConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (DecayConstant) ApplyS3 

func (o DecayConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Default 

type Default struct {
	// contains filtered or unexported fields
}

Default is a truncated exponential attack and decay.

On retries the sleep time is doubled, on non errors then sleeptime decays according to the decay constant as set with SetDecayConstant.

The sleep never goes below that set with SetMinSleep or above that set with SetMaxSleep.

func NewDefault 

func NewDefault(opts ...DefaultOption) *Default

NewDefault creates a Calculator used by Pacer as the default.

func (*Default) Calculate 

func (c *Default) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*Default) Update 

func (c *Default) Update(opts ...DefaultOption)

Update applies the Calculator options.

type DefaultOption 

type DefaultOption interface {
	ApplyDefault(*Default)
}

DefaultOption is the interface implemented by all options for the Default Calculator

type GoogleDrive 

type GoogleDrive struct {
	// contains filtered or unexported fields
}

GoogleDrive is a specialized pacer for Google Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to (2 ^ n) + random_number_milliseconds seconds.

See https://developers.google.com/drive/v2/web/handle-errors#exponential-backoff

func NewGoogleDrive 

func NewGoogleDrive(opts ...GoogleDriveOption) *GoogleDrive

NewGoogleDrive returns a new GoogleDrive Calculator with default values

func (*GoogleDrive) Calculate 

func (c *GoogleDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*GoogleDrive) Update 

func (c *GoogleDrive) Update(opts ...GoogleDriveOption)

Update applies the Calculator options.

type GoogleDriveOption 

type GoogleDriveOption interface {
	ApplyGoogleDrive(*GoogleDrive)
}

GoogleDriveOption is the interface implemented by all options for the GoogleDrive Calculator

type InvokerFunc 

type InvokerFunc func(try, tries int, f Paced) (bool, error)

InvokerFunc is the signature of the wrapper function used to invoke the target function in Pacer.

type MaxSleep 

type MaxSleep time.Duration

MaxSleep configures the maximum sleep time of a Calculator

func (MaxSleep) ApplyDefault 

func (o MaxSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MaxSleep) ApplyS3 

func (o MaxSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type MinSleep 

type MinSleep time.Duration

MinSleep configures the minimum sleep time of a Calculator

func (MinSleep) ApplyAmazonCloudDrive 

func (o MinSleep) ApplyAmazonCloudDrive(c *AmazonCloudDrive)

ApplyAmazonCloudDrive updates the value on the Calculator

func (MinSleep) ApplyDefault 

func (o MinSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MinSleep) ApplyGoogleDrive 

func (o MinSleep) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

func (MinSleep) ApplyS3 

func (o MinSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Option 

type Option func(*pacerOptions)

Option can be used in New to configure the Pacer.

func CalculatorOption 

func CalculatorOption(c Calculator) Option

CalculatorOption sets a Calculator for the new Pacer.

func InvokerOption 

func InvokerOption(invoker InvokerFunc) Option

InvokerOption sets an InvokerFunc for the new Pacer.

func MaxConnectionsOption 

func MaxConnectionsOption(maxConnections int) Option

MaxConnectionsOption sets the maximum connections number for the new Pacer.

func RetriesOption 

func RetriesOption(retries int) Option

RetriesOption sets the retries number for the new Pacer.

type Paced 

type Paced func() (bool, error)

Paced is a function which is called by the Call and CallNoRetry methods. It should return a boolean, true if it would like to be retried, and an error. This error may be returned or returned wrapped in a RetryError.

type Pacer 

type Pacer struct {
	// contains filtered or unexported fields
}

Pacer is the primary type of the pacer package. It allows to retry calls with a configurable delay in between.

func New 

func New(options ...Option) *Pacer

New returns a Pacer with sensible defaults.

func (*Pacer) Call 

func (p *Pacer) Call(fn Paced) (err error)

Call paces the remote operations to not exceed the limits and retry on rate limit exceeded

This calls fn, expecting it to return a retry flag and an error. This error may be returned wrapped in a RetryError if the number of retries is exceeded.

func (*Pacer) CallNoRetry 

func (p *Pacer) CallNoRetry(fn Paced) error

CallNoRetry paces the remote operations to not exceed the limits and return a retry error on rate limit exceeded

This calls fn and wraps the output in a RetryError if it would like it to be retried

func (*Pacer) ModifyCalculator 

func (p *Pacer) ModifyCalculator(f func(Calculator))

ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.

func (*Pacer) SetCalculator 

func (p *Pacer) SetCalculator(c Calculator)

SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.

It will choose the default algorithm if nil is passed in.

func (*Pacer) SetMaxConnections 

func (p *Pacer) SetMaxConnections(n int)

SetMaxConnections sets the maximum number of concurrent connections. Setting the value to 0 will allow unlimited number of connections. Should not be changed once you have started calling the pacer. By default this will be set to fs.Config.Checkers.

func (*Pacer) SetRetries 

func (p *Pacer) SetRetries(retries int)

SetRetries sets the max number of retries for Call

type S3 

type S3 struct {
	// contains filtered or unexported fields
}

S3 implements a pacer compatible with our expectations of S3, where it tries to not delay at all between successful calls, but backs off in the default fashion in response to any errors. The assumption is that errors should be exceedingly rare (S3 seems to have largely solved the sort of stability questions rclone is likely to run into), and in the happy case it can handle calls with no delays between them.

Basically defaultPacer, but with some handling of sleepTime going to/from 0ms

func NewS3 

func NewS3(opts ...S3Option) *S3

NewS3 returns a new S3 Calculator with default values

func (*S3) Calculate 

func (c *S3) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*S3) Update 

func (c *S3) Update(opts ...S3Option)

Update applies the Calculator options.

type S3Option 

type S3Option interface {
	ApplyS3(*S3)
}

S3Option is the interface implemented by all options for the S3 Calculator

type State 

type State struct {
	SleepTime          time.Duration // current time to sleep before adding the pacer token back
	ConsecutiveRetries int           // number of consecutive retries, will be 0 when the last invoker call returned false
	LastError          error         // the error returned by the last invoker call or nil
}

State represents the public Pacer state that will be passed to the configured Calculator

type TokenDispenser 

type TokenDispenser struct {
	// contains filtered or unexported fields
}

TokenDispenser is for controlling concurrency

func NewTokenDispenser 

func NewTokenDispenser(n int) *TokenDispenser

NewTokenDispenser makes a pool of n tokens

func (*TokenDispenser) Get 

func (td *TokenDispenser) Get()

Get gets a token from the pool - don't forget to return it with Put

func (*TokenDispenser) Put 

func (td *TokenDispenser) Put()

Put returns a token

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.