partitioner

README

Kafka Partitioners for Sarama

Default Java/Scala - Murmur2Partitioner

This is a go implementation to match the current recommended default partitioner:

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java

It implements a murmur2 hash that matches the partitioning provided by the official Java/Scala client.

The default hash implementation in sarama is FNV-1a which is not compatible with the standard kafka client.

Deprecated Java/Scala - JavaHashPartitioner

TBC

Testing

# Get the Kafka lib
curl -O http://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.1.0/kafka-clients-0.10.1.0.jar

curl -O http://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jarhttps://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
curl -O http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar
curl -O http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.21/slf4j-log4j12-1.7.21.jar
curl -O http://repo1.maven.org/maven2/org/mockito/mockito-core/2.2.9/mockito-core-2.2.9.jar
curl -O http://repo1.maven.org/maven2/net/bytebuddy/byte-buddy/1.5.0/byte-buddy-1.5.0.jar
curl -O http://repo1.maven.org/maven2/org/objenesis/objenesis/2.4/objenesis-2.4.jar
curl -O http://repo1.maven.org/maven2/net/bytebuddy/byte-buddy-agent/1.5.0/byte-buddy-agent-1.5.0.jar

mkdir mockito-extensions
echo "mock-maker-inline" > mockito-extensions/org.mockito.plugins.MockMaker

Documentation

Index

• func Murmur2Partition(bytes []byte, numPartitions int32) int32
• func MurmurHash2(data []byte) (h int32)
• func NewMurmur2Partitioner(topic string) sarama.Partitioner
• type Murmur2Partitioner
  • func (p *Murmur2Partitioner) Partition(message *sarama.ProducerMessage, numPartitions int32) (int32, error)
  • func (p *Murmur2Partitioner) RequiresConsistency() bool

Functions

func Murmur2Partition

func Murmur2Partition(bytes []byte, numPartitions int32) int32

func MurmurHash2

func MurmurHash2(data []byte) (h int32)

The original MurmurHash2 32-bit algorithm by Austin Appleby. Taken from https://github.com/aviddiviner/go-murmur by David Irvine Adapted to match the behavior of the Java Kafka Client

func NewMurmur2Partitioner

func NewMurmur2Partitioner(topic string) sarama.Partitioner

NewMurmur2Partitioner creates a partitioner

Types

type Murmur2Partitioner

type Murmur2Partitioner struct {
	// contains filtered or unexported fields
}

Murmur2Partitioner is a matching implemention for the current (10.1) kafka partitioner. It will generate the same partition ID's a the default Kafka client and hence can be as a replacement with correct inter op

func (*Murmur2Partitioner) Partition

func (p *Murmur2Partitioner) Partition(message *sarama.ProducerMessage, numPartitions int32) (int32, error)

Called by Samza to get the partition for a message. Will return a random partition if the message key is nil

func (*Murmur2Partitioner) RequiresConsistency

func (p *Murmur2Partitioner) RequiresConsistency() bool

RequiresConsistency is always true for this implemention