pool

Documentation

Overview

Package pool provides specific implementations of the amboy.Runner interface that serve as the worker pools for tasks in work queues.

Intentionally, most of the important logic about job execution and dispatching happens in the Queue implementation, and the Runner implementations are simplistic.

Local Workers Pool

The LocalWorkers is a simple worker pool implementation that spawns a collection of (n) workers and dispatches jobs to worker threads, that consume work items from the Queue's Next() method.

Rate Limiting Pools

Amboy includes two rate limiting pools, to control the flow of tasks processed by the queue. The "simple" implementation sleeps for a configurable interval in-between each task, while the averaged tool, uses an exponential weighted average and a targeted number of tasks to complete over an interval to achieve a reasonable flow of tasks through the runner.

Index

• func NewAbortablePool(opts *WorkerOptions) amboy.AbortableRunner
• func NewLocalWorkers(opts *WorkerOptions) amboy.Runner
• func NewMovingAverageRateLimitedWorkers(targetNum int, period time.Duration, opts *WorkerOptions) (amboy.AbortableRunner, error)
• func NewNoop() amboy.Runner
• func NewSimpleRateLimitedWorkers(sleepInterval time.Duration, opts *WorkerOptions) (amboy.Runner, error)
• func NewSingle(logger grip.Logger) amboy.Runner
• type WorkerOptions

Functions

func NewAbortablePool

func NewAbortablePool(opts *WorkerOptions) amboy.AbortableRunner

NewAbortablePool produces a simple implementation of a worker pool that provides access to cancel running jobs. The cancellation functions work by creating context cancelation function and then canceling the contexts passed to the jobs specifically.

func NewLocalWorkers

func NewLocalWorkers(opts *WorkerOptions) amboy.Runner

NewLocalWorkers is a constructor for pool of worker processes that execute jobs from a queue locally, and takes arguments for the number of worker processes and a amboy.Queue object.

func NewMovingAverageRateLimitedWorkers

func NewMovingAverageRateLimitedWorkers(targetNum int, period time.Duration, opts *WorkerOptions) (amboy.AbortableRunner, error)

NewMovingAverageRateLimitedWorkers returns a worker pool implementation that attempts to run a target number of tasks over a specified period to provide a more stable dispatching rate. It uses an exponentially weighted average of task time when determining the rate, which favors recent tasks over previous tasks.

Returns an error if the size or target numbers are less than one and if the period is less than a millisecond.

func NewNoop

func NewNoop() amboy.Runner

NewNoop creates a runner implementation that has no workers, but satisfies the workers and semantics of the Runner interface to support queues deployments that have insert only queues.

func NewSimpleRateLimitedWorkers

func NewSimpleRateLimitedWorkers(sleepInterval time.Duration, opts *WorkerOptions) (amboy.Runner, error)

NewSimpleRateLimitedWorkers returns a worker pool that sleeps for the specified interval after completing each task. After that interval, the runner will run the next available task as soon as its ready.

The constructor returns an error if the size (number of workers) is less than 1 or the interval is less than a millisecond.

func NewSingle

func NewSingle(logger grip.Logger) amboy.Runner

NewSingle returns an amboy.Runner implementation with single-worker in the pool.

Types

type WorkerOptions

type WorkerOptions struct {
	Logger     grip.Logger
	Queue      amboy.Queue
	NumWorkers int
}

WorkerOptions describes the arguments passed to the constructors of worker pools. Queue must not be nil.