kafka-grpc-stream

README

kafka-grpc-stream

Scaleable streaming grpc service that allows external web clients to subscribe, filter, and stream data from kafka topics. More details coming soon!

Features

• 1.4.0 - Added TTL to key registry keys
• 1.3.0 - Max age and min offset filters
• 1.2.1 - More stable load balancer
• 1.2.0 - Monitoring and CLI tool
• 1.1.0 - Buffered broadcasting
• 1.0.0 - Key glob filtering
• 0.0.5 - Manual subscribe to a topic/partition server through grpc client
• 0.0.4 - Key registration (maps keys to partitions)
• 0.0.3 - Server-side host-session routing

Local Development

Pre-requisites

• go1.22.5
• docker >= 20.10.17
• make >= 4.3

Running

Starts a local etcd, redpanda server and the kafka-grpc-stream server.

docker compose up

CLI

Connect to the running server using the built-in CLI tool. Default returns all results.

docker compose exec -it server cli [--keys KEY1 --keys KEY2 ...]

Redpanda

Access the local redpanda console at http://localhost:8000

Etcd

Access the etcd via command line:

docker compose exec etcdctl help

Testing

You can run the entire test suite using make:

make test

And the e2e test suite:

make e2e-test

...or you can have more fine-grained control of tests by running ginkgo

docker compose run --rm dev ginkgo help

Generators

Mocks

Mocks are generated with mockery. Update .mockery.yaml with the interface/s to want to mock:

make mock

Protobuf

Protobufs are manages in the protobuf directory. To generate protobufs:

make proto

Examples

• Tuning Demo: A grafana dashboard for tuning a single kafka-grpc-stream server instance.

Build Reference

• Redpanda
• Redpanda Console
• Redpanda Connect
• Etcd
• CAdvisor
• Prometheus
• Grafana
• Alpine
• Golang
• golangci-lint
• protoc
• protoc-gen-go
• protoc-gen-go-grpc
• mockery
• go-test-coverage