sqs-worker-pool

README

Program sqs-worker-pool starts multiple worker processes for each matching
SQS queue depending on queue depth.

For details see https://godoc.org/github.com/Doist/sqs-worker-pool

Documentation

Overview

Program sqs-worker-pool starts multiple worker processes for each matching SQS queue depending on queue depth.

Its main purpose is maintaining pool of workers for multiple queues with uneven load, when constantly running multiple workers may be a waste of resources.

Upon start it fetches list of all SQS queues matching given conditions (see -include and -exclude flags), then starts worker pool for each matched queue. Each worker pool polls queue depth once a minute and calculates target number of workers. Target number of workers in a pool is calculated as queue depth / worker-load, but capped to max-workers. For non-empty queues, target number is always in [1, max-workers] range. Pool does not terminate workers by itself, workers should terminate if they see that queue became idle.

Program starts a single process as a worker for each queue, this process can be used as a dispatcher to run different processes depending on queue name. Worker process is called with queue name as its first positional argument and SQS queue url as a second positional argument. These values are also passed via environment as NAME and URL variables respectively.

Upon receiving INT or TERM signals, program tries gracefully shutting down all running workers by sending them TERM first, and KILL 3 seconds later.

sqs-worker-pool is built using AWS SDK, so it looks up required credentials in a usual way: via local confgiuration, environment variables, IAM role.