manual-partitioner

README

kafka-plugin Manual-Partitioner

A simple example demonstrating the usage of Kafka plugin API with a focus on manual partitioning.

Requirements

To start the example you have to have Kafka broker running first. Furthermore, the example assumes that each auto-created topic has at least 3 partitions (num.partitions >= 3 in server.properties).

If you don't have Kafka installed locally, you can use the following docker image in combination with a prepared server.properties config file:

sudo docker create -p 2181:2181 -p 9092:9092 --name kafka --rm \
   --env ADVERTISED_HOST=172.17.0.1 --env ADVERTISED_PORT=9092 spotify/kafka
KAFKA_VERSION=$(docker inspect -f '{{ .Config.Env }}' kafka |  tr ' ' '\n' | grep KAFKA_VERSION | sed 's/^.*=//')
SCALA_VERSION=$(docker inspect -f '{{ .Config.Env }}' kafka |  tr ' ' '\n' | grep SCALA_VERSION | sed 's/^.*=//')
sudo docker cp server.properties kafka:/opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION}/config/server.properties
sudo docker start kafka

It will bring up Kafka broker listening on port 9092 for client communication.

Usage

To run the example, type:

go run main.go deps.go [-kafka-config <config-filepath>]

If kafka-config is unspecified, the application will try to search for kafka.conf in the current working directory. If the configuration file cannot be loaded, the initialization of the kafka plugin will be skipped and the example scenario will thus not execute in its entirety.

Documentation

Overview

Example "manual-partitioner" demonstrates the use of Kafka plugin API with a focus on manual partitioning.