benthos

README

Benthos is a stream multiplexer service driven by a config file in either YAML or JSON format and deploys as a single binary with zero runtime dependencies.

A variety of configurable message processors can be chained together for solving common streaming problems such as content based multiplexing, filtering, modifying, batching, splitting, (de)compressing, (un)archiving, etc.

A range of optional buffer strategies are available, allowing you to select a balance between latency, protection against back pressure and file based persistence, or nothing at all (direct bridge).

For more details check out the general documentation. For some applied examples such as streaming and deduplicating the Twitter firehose to Kafka check out the cookbook section.

Supported Protocols

Currently supported input/output targets:

• Amazon (S3, SQS)
• Elasticsearch
• File
• HTTP(S)
• Kafka
• MQTT
• Nanomsg
• NATS
• NATS Streaming
• NSQ
• RabbitMQ (AMQP 0.91)
• Redis
• Stdin/Stdout
• ZMQ4

Setting up multiple outputs or inputs is done by choosing a routing strategy (fan-in, fan-out, round-robin, etc.)

It is possible to enable a REST API to dynamically change inputs and outputs at runtime, which you can read about here.

For a full and up to date list of all inputs, buffer options, processors, and outputs you can find them in the docs, or print them from the binary:

# Print inputs, buffers and output options
benthos --list-inputs --list-buffers --list-outputs --list-processors | less

Mixing multiple part message protocols with single part can be done in different ways, for more guidance check out this doc.

Install

Build with Go:

go get github.com/Jeffail/benthos/cmd/benthos

Or, pull the docker image:

docker pull jeffail/benthos

Or, download from here.

Run

benthos -c ./config.yaml

Or, with docker:

# Send HTTP /POST data to Kafka:
docker run --rm \
	-e "BENTHOS_INPUT=http_server" \
	-e "BENTHOS_OUTPUT=kafka" \
	-e "KAFKA_OUTPUT_BROKER_ADDRESSES=kafka-server:9092" \
	-e "KAFKA_OUTPUT_TOPIC=benthos_topic" \
	-p 4195:4195 \
	jeffail/benthos

# Using your own config file:
docker run --rm -v /path/to/your/config.yaml:/benthos.yaml jeffail/benthos

Streams

Benthos can be run in --streams mode which, instead of running a single stream of inputs to outputs, opens up a REST HTTP API for creating and managing multiple streams. Each stream has its own input, buffer, pipeline and output sections which contains an isolated stream of data with its own lifetime.

Config

Benthos has inputs, optional processors, an optional buffer, and outputs, which are all set in a single config file.

Check out the samples in ./config, or create a fully populated default configuration file:

benthos --print-yaml > config.yaml
benthos --print-json | jq '.' > config.json

If we wanted to pipe Stdin to a ZMQ push socket our YAML config might look like this:

input:
  type: stdin
output:
  type: zmq4
  zmq4:
    addresses:
      - tcp://*:1234
    socket_type: PUSH

There are also configuration sections for logging and metrics, if you print an example config you will see the available options.

For a list of metrics within Benthos check out this spec.

Environment Variables

You can use environment variables to replace fields in your config files.

ZMQ4 Support

Benthos supports ZMQ4 for both data input and output. To add this you need to install libzmq4 and use the compile time flag when building Benthos:

go install -tags "ZMQ4" ./cmd/...

Vendoring

Benthos uses dep for managing dependencies. To get started make sure you have dep installed:

go get -u github.com/golang/dep/cmd/dep

And then run dep ensure. You can decrease the size of vendor by only storing needed files with dep prune.

Docker

There's a multi-stage Dockerfile for creating a Benthos docker image which results in a minimal image from scratch. You can build it with:

make docker

Then use the image:

docker run --rm \
	-v /path/to/your/benthos.yaml:/config.yaml \
	-v /tmp/data:/data \
	-p 4195:4195 \
	benthos -c /config.yaml

There are a few examples here that show you some ways of setting up Benthos containers using docker-compose.