README
¶
job-manager
job-manager is a server that coordinates the execution of distributed jobs over HTTP.
The gist is that consumers poll the server for jobs, and send job status back upon completion. Checkins can be used to annotate the job with data pre-completion. A command-line tool, jobctl, is provided for queue administration. That with labels and JSON Schema validation support processors that undergo a lot of iteration well.
I made this initially to experiment with some higher-level features in a job queue, but it should be capable of comparable performance to other postgres-backed job queues. In most cases, the job server itself is not a performance bottleneck compared to the backend, however multiple replicas can be run for high availability.
install
Using go:
$ go get github.com/jeffrom/job-manager/...
Via docker: docker pull jeffrom/job-manager
Or download a github release.
migrations
To run postgresql migrations:
$ jobctl migrate
Migrations are implemented using golang-migrate. Note that this command requests a job-manager server to execute the migration using its configured credentials. This requires additional permissions in Postgresql.
features
- cli controller
- straightforward Rest API (protobuf support)
- job server is stateless / scales horizontally
- implement your own backend, comes with postgresql
- in-memory backend for development & testing purposes (not really working right now)
- claim windows: only dequeue to consumers with matching claims for a configurable duration
- check ins
- store result data
- queue labels
- versioned queue configuration
- easily update queue configurations via cicd with
jobctl apply - json schema validation for job arguments, data, results
- the usual job queue features: retries, exponential backoff, durability
- graceful shutdown
- not fast, predictable 😎
clients
There is a go client:
$ go get github.com/jeffrom/job-manager/mjob
develop
To start a development server on your laptop:
$ make dev
Run tests and static analysis:
$ make test
$ make lint
Point jobctl at the local server dev proxy:
$ export HOST=:4000
$ jobctl stats
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
jobctl/commands
Package commands contains jobctl's cobra commands.
|
Package commands contains jobctl's cobra commands. |
|
Package integration contains integration tests for job-manager.
|
Package integration contains integration tests for job-manager. |
|
mjob
module
|
|
|
pkg
|
|
|
backend
Package backend manages storing data.
|
Package backend manages storing data. |
|
backend/mem
Package mem is a minimal reference implementation of backend.Interface suitable for use in tests.
|
Package mem is a minimal reference implementation of backend.Interface suitable for use in tests. |
|
backend/pg
Package pg implements backend.Interface using Postgres.
|
Package pg implements backend.Interface using Postgres. |
|
backend/pg/migrations
Package migrations contains migration data.
|
Package migrations contains migration data. |
|
config
Package config handles job-manager configuration helpers.
|
Package config handles job-manager configuration helpers. |
|
internal
Package internal contains code that we don't want to export outside of job-manager.
|
Package internal contains code that we don't want to export outside of job-manager. |
|
logger
Package logger contains a logger for use by other job-manager packages.
|
Package logger contains a logger for use by other job-manager packages. |
|
testenv
Package testenv contains helpers for setting up job-manager test environments.
|
Package testenv contains helpers for setting up job-manager test environments. |
|
web
Package web contains code to run webservers.
|
Package web contains code to run webservers. |
|
web/handler
Package handler contains http handlers.
|
Package handler contains http handlers. |
|
web/middleware
Package middleware contains various http middlewares.
|
Package middleware contains various http middlewares. |
|
Package release contains metadata for the current job-manager release.
|
Package release contains metadata for the current job-manager release. |