README
¶
job-manager
job-manager is a server that coordinates the execution of distributed jobs over HTTP.
The gist is that consumers poll the server for jobs, and send job status back upon completion. Checkins can be used to annotate the job with data pre-completion. A command-line tool, jobctl, is provided for queue administration. Labels, JSON Schema validation, and a gitops-style queue administration are supported to better support complex workflows across multiple development teams.
I made this initially to experiment with some higher-level features in a job queue, but it should be capable of comparable performance to postgres-backed job queues. In most cases, the job server itself is rarely a bottleneck, but multiple replicas can be run for high availability. Typically, the backend, not the api server, will be the bottleneck in most workloads.
install
Using go:
$ go get github.com/jeffrom/job-manager/...
Via docker: docker pull jeffmartin1117/job-manager
Or download a github release.
migrations
To run postgresql migrations:
$ jobctl migrate
Migrations are implemented using golang-migrate.
features
- cli controller
- straightforward Rest API (protobuf support)
- job server is stateless / scales horizontally
- implement your own backend, comes with postgresql
- in-memory backend for development & testing purposes (not really working right now)
- claim windows: only dequeue to consumers with matching claims for a configurable duration
- check ins
- store result data
- queue labels
- versioned queue configuration
- easily update queue configurations via cicd with
jobctl apply - json schema validation for job arguments, data, results
- the usual job queue features: retries, exponential backoff, durability
- graceful shutdown
- not fast, predictable 😎
clients
For now, just go:
$ go get github.com/jeffrom/job-manager/mjob
develop
To start a development server on your laptop:
$ make dev
Run tests and static analysis:
$ make test
$ make lint
Point jobctl at the local server dev proxy:
$ export HOST=:4000
$ jobctl stats
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
jobctl/commands
Package commands contains jobctl's cobra commands.
|
Package commands contains jobctl's cobra commands. |
|
Package integration contains integration tests for job-manager.
|
Package integration contains integration tests for job-manager. |
|
mjob
module
|
|
|
pkg
|
|
|
backend
Package backend manages storing data.
|
Package backend manages storing data. |
|
backend/mem
Package mem is a minimal reference implementation of backend.Interface suitable for use in tests.
|
Package mem is a minimal reference implementation of backend.Interface suitable for use in tests. |
|
backend/pg
Package pg implements backend.Interface using Postgres.
|
Package pg implements backend.Interface using Postgres. |
|
backend/pg/migrations
Package migrations contains migration data.
|
Package migrations contains migration data. |
|
config
Package config handles job-manager configuration helpers.
|
Package config handles job-manager configuration helpers. |
|
internal
Package internal contains code that we don't want to export outside of job-manager.
|
Package internal contains code that we don't want to export outside of job-manager. |
|
logger
Package logger contains a logger for use by other job-manager packages.
|
Package logger contains a logger for use by other job-manager packages. |
|
testenv
Package testenv contains helpers for setting up job-manager test environments.
|
Package testenv contains helpers for setting up job-manager test environments. |
|
web
Package web contains code to run webservers.
|
Package web contains code to run webservers. |
|
web/handler
Package handler contains http handlers.
|
Package handler contains http handlers. |
|
web/middleware
Package middleware contains various http middlewares.
|
Package middleware contains various http middlewares. |
|
Package release contains metadata for the current job-manager release.
|
Package release contains metadata for the current job-manager release. |