turbine

README

Turbine

Fast, distributed message queue for Golang and MongoDB

There are many, many message queue tools. Perhaps you should use one of those instead. But Turbine fills a unique niche in that it:

1. is a distributed queue that can be shared between several producer and consumer servers simultaneously
2. supports swappable storage providers, with the first provider being MongoDB.
3. supports fast, in-memory queues using Golang channels
4. can retry failed jobs (with exponential backoff)
5. can schedule jobs to in the future

Pushing Tasks to the Queue

// Create a Task with any parameters
task := queue.NewTask(
    "TaskName" // Task Name
    map[string]any{ // Parameters
        "foo": "bar",
        "baz": 42,
    }
)

// Publish the task to the Queue (this will happen quickly)
if err := queue.Publish(task); err != nil {
    // only errors related to queuing the task
}

Run the Queue

// Create and start a queue
q := queue.New(
    queue.WithConsumers(), // one or more "consumer" functions (below)
    queue.WithStorage(), // optional storage adapter persists tasks 
    queue.WithTimeout()
)

Consuming Tasks from the Queue

Mongo Storage Provider