gofer

README

Gofer: Run short-lived jobs easily.

Gofer is an opinionated, cloud-native, container-focused, continuous thing do-er, that focuses on simplicity and usability for both developers and ops.

Features:

• Deploy it as a single static binary
• Write your pipelines in Go or Rust
• Pluggable: Write your own triggers, shared tasks, and more in any language (through GRPC).
• DAG(Directed Acyclic Graph) support.
• Reliability tooling: A/B test, version, and canary new pipelines.
• Bring your own everything! Secret store, object store, container scheduler. Gofer has the interfaces to support all of them.

Philosophy:

It uses a philosophy similar to concourse, leveraging the industry popular Docker container as the packaging method for code running on infrastructure. The benefits of this is simplicity. No foreign agents, no cluster setup, no yaml mess. Everything is based on running a container, mirroring what most companies already understand and use on a day to day basis.

You can read more about Gofer and it's philosophy here.

Demo:

Documentation & Getting Started

If you want to fully dive into Gofer, check out the documentation site!

Install

Extended installation information is available through the documentation site.

Download a specific release:

You can view and download releases by version here.

Download the latest release:

• Linux: wget https://github.com/clintjedwards/gofer/releases/latest/download/gofer

Build from source:

You'll need to install protoc and its associated golang/grpc modules first

1. git clone https://github.com/clintjedwards/gofer && cd gofer
2. make build OUTPUT=/tmp/gofer

The Gofer binary comes with a CLI to manage the server as well as act as a client.

Why not use <insert favorite tool> instead ?

Tool	Category	Why not?
Jenkins	General thing-doer	Supports generally anything you might want to do ever, but because of this it can be operationally hard to manage, usually has massive security issues and isn't by default opinionated enough to provide users a good interface into how they should be managing their workloads.
Buildkite/CircleCI/Github actions/etc	Gitops cloud builders	Gitops focused cloud build tooling is great for most situations and probably what most companies should start out using. The issue is that running your workloads can be hard to test since these tools use custom agents to manage those jobs. This causes local testing to be difficult as the custom agents generally work very differently locally. Many times users will fight with yaml and make commits just to test that their job does what they need due to their being no way to determine that beforehand.
ArgoCD	Kubernetes focused CI/CD tooling	In the right direction with its focus on running containers on already established container orchstrators, but Argo is tied to gitops making it hard to test locally, and also closely tied to Kubernetes.
ConcourseCI	Container focused thing do-er	Concourse is great and where much of this inspiration for this project comes from. It sports a sleek CLI, great UI, and cloud-native primatives that makes sense. The drawback of concourse is that it uses a custom way of managing docker containers that can be hard to reason about. This makes testing locally difficult and running in production means that your short-lived containers exist on a platform that the rest of your company is not used to running containers on.
Airflow	ETL systems	I haven't worked with large scale data systems enough to know deeply about how ETL systems came to be, but (maybe naively) they seem to fit into the same paradigm of "run x thing every time y happens". Airflow was particularly rough to operate in the early days of its release with security and UX around DAG runs/management being nearly non-existent. As an added bonus the scheduler regularly crashed from poorly written user workloads making it a reliability nightmare. Additionally, Airflow's models of combining the execution logic of your DAGs with your code led to issues of testing and iterating locally. Instead of having tooling specifically for data workloads, instead it might be easier for both data teams and ops teams to work in the model of distributed cron as Gofer does. Write your stream processing using dedicated tooling/libraries like Benthos (or in whatever language you're most familiar with), wrap it in a Docker container, and use Gofer to manage which containers should run when, where, and how often. This gives you easy testing, separation of responsibilities, and no python decorator spam around your logic.
Cadence	ETL systems	I like Uber's cadence, it does a great job at providing a platform that does distributed cron and has some really nifty features by choosing to interact with your workflows at the code level. The ability to bake in sleeps and polls just like you would regular code is awesome. But just like Airflow, I don't want to marry my scheduling platform with my business logic. I write the code as I would for a normal application context and I just need something to run that code. When we unmarry the business logic and the scheduling platform we are able to treat it just like we treat all our other code, which means code workflows(testing, for example) we were all already used to and the ability to foster code reuse for these same processes.

Dev Setup

Gofer is setup such that the base run mode is the development mode. So simply running the binary without any additional flags allows easy authless development.

You'll need to install the following first:

To run Gofer dev mode:

• Docker

To build protocol buffers:

• protoc
• protoc gen plugins go/grpc

Run from the Makefile

Gofer uses flags, env vars, and files to manage configuration (in order of most important). The Makefile already includes all the commands and flags you need to run in dev mode by simply running make run.

In case you want to run without the make file simply run:

go build -o /tmp/gofer
DEBUG=true; SEMVER=0.0.0; /tmp/gofer service start

Authors

• Clint Edwards - Github

This software is provided as-is. It's a hobby project, done in my free time, and I don't get paid for doing it.