hmq-kafka

README

Partition IoT Platform With MQTT And Kafka

About

Golang MQTT Broker, Version 3.1.1, and Compatible for eclipse paho client and mosquitto-client

1, Each MQTT broker publishes and subscribes to a partition of kafak multiple topics, one broker corresponds to a partition

2, When the device is connected, the broker saves the correspondence between the device and the partition, which is convenient for the backend service query.

3, When broker receive message from device, according to the producer's topic configuration, sends message to the specified kafka topic

4, After the backend service receives the kafka message, if the service needs to response device, it can directly reply according to the partition to which the received message belongs, or query the response from the Redis to the specified partition, mqtt topic and qos should be saved in the message header with key "topic" and "qos"

Usage of hmq-kafka:

Usage: hmq-kafka [options]

Broker Options:
    -w,  --worker <number>            Worker num to process message, perfer (client num)/10. (default 1024)
    -p,  --port <port>                Use port for clients (default: 1883)
         --host <host>                Network host to listen on. (default "0.0.0.0")
    -ws, --wsport <port>              Use port for websocket monitoring
    -wsp,--wspath <path>              Use path for websocket monitoring
    -c,  --config <file>              Configuration file

Logging Options:
    -d, --debug <bool>                Enable debugging output (default false)
    -D                                Debug enabled

Common Options:
    -h, --help                        Show this message

hmq.json

{
	"workerNum": 4096,
	"port": "1883",
	"host": "0.0.0.0",
	"debug": true,
	"tlsPort": "8883",
	"tlsHost": "0.0.0.0",
	"wsPort": "1888",
	"wsPath": "/ws",
	"wsTLS": false,
	"tlsInfo": {
		"verify": true,
		"caFile": "ssl/ca/ca.pem",
		"certFile": "ssl/server/cert.pem",
		"keyFile": "ssl/server/key.pem"
	},
	"plugins": [
		"authhttp"
	],
	"redisAddr": "127.0.0.1:6379",
	"kafkaConfig": {
		"addr": [
			"127.0.0.1:9090"
		],
		"consumer": {
			"partition": -1,
			"topic": [
				"topic"
			]
		},
		"producer": {
			"onConnect": "onConnect",
			"onPublish": "onPublish",
			"onSubscribe": "onSubscribe",
			"onDisconnect": "onDisconnect",
			"onUnsubscribe": "onUnsubscribe",
			"regexpMap": [
				{
					"^/(.+)/(.+)/upload/(.*)$": "upload"
				}
			]
		}
	}
}

Features and Future

• Supports QOS 0 and 1

• Containerization

• Supports retained messages

• Supports will messages

• Websocket Support

• TLS/SSL Support

• Kafka Deliver

• AuthHTTP Support

  • Auth Connect
  • Auth ACL
  • Cache Support

• HTTP API

  • Disconnect Connect

• Connect Persist In Redis

Performance

• High throughput

• High concurrency

• Low memory and CPU

License

• Apache License Version 2.0

Reference

• Surgermq.(https://github.com/surgemq/surgemq)

Benchmark Tool

• https://github.com/inovex/mqtt-stresser
• https://github.com/krylovsk/mqtt-benchmark

Documentation

Overview

Copyright (c) 2018, joy.zhou <chowyu08@gmail.com>

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.