benthos

README

Benthos is a service that reads messages from any sources and writes them to any sinks. It bridges services and tools together in ways that can simplify your platform or reduce development time.

A range of optional buffer strategies are available, allowing you to select a balance between latency, protection against back pressure and file based persistence, or nothing at all (direct bridge).

Supported Protocols

Currently supported input/output targets:

• ZMQ4 (PUSH, PULL, SUB, PUB)
• Nanomsg/Scalability Protocols (PUSH, PULL, SUB, PUB)
• RabbitMQ (AMQP 0.91)
• NSQ
• NATS
• Kafka
• HTTP 1.1 POST/GET
• STDIN/STDOUT
• File

Setting up multiple outputs or inputs is done by choosing a routing strategy (fan in, fan out, round robin, etc.)

For a full and up to date list of all inputs, buffer options, processors, and outputs you can find them in the docs, or print them from the binary:

# Print inputs, buffers and output options
benthos --list-inputs --list-buffers --list-outputs --list-processors | less

Mixing multiple part message protocols with single part can be done in different ways, for more guidance check out this doc.

Install

Build with Go:

go get github.com/Jeffail/benthos/cmd/...

Or, download from here.

Run

benthos -c ./config.yaml

Config

Benthos has inputs, optional processors, an optional buffer, and outputs, which are all set in a single config file.

Check out the samples in ./config, or create a fully populated default configuration file:

benthos --print-yaml > config.yaml
benthos --print-json > config.json

The configuration file should contain a section for an input, output, and a buffer. For example, if we wanted to output to a ZMQ4 push socket our output section in a YAML config might look like this:

output:
  type: zmq4
  zmq4:
    addresses:
      - tcp://*:1234
    socket_type: PUSH

There are also configuration sections for logging and metrics, if you print an example config you will see the available options.

For a list of metrics within Benthos check out this spec.

ZMQ4 Support

Benthos supports ZMQ4 for both data input and output. To add this you need to install libzmq4 and use the compile time flag when building benthos:

go install -tags "ZMQ4" ./cmd/...

Vendoring

Benthos uses dep for managing dependencies. To get started make sure you have dep installed:

go get -u github.com/golang/dep/cmd/dep

And then run dep ensure. You can decrease the size of vendor by only storing needed files with dep prune.

Docker

There's a Dockerfile for creating a benthos docker image. This is built from scratch and so you'll need to build without CGO (CGO_ENABLED=0) for your benthos build to run within it. Create it like this:

CGO_ENABLED=0 make docker
docker run --rm benthos

Then use the image:

docker run --rm -v ~/benthos.yaml:/config.yaml -v /tmp/data:/data -p 8080:8080 \
	benthos -c /config.yaml