go-kafka-http

README

Kafka HTTP

使用CircleCI进程CI Pipeline校验，Go Report Card进行Golang项目代码分析

1. CircleCI: https://circleci.com/gh/chenguolin/go-api-service/tree/master
2. Go Report Card: https://goreportcard.com/report/github.com/chenguolin/go-api-service

用Gin实现的HTTP API服务，提供HTTP接口用于访问Kafka集群，提供以下类型接口 (默认使用38765端口)

要求: kafka server版本 >= v0.10.0.x，某些命令需要更高版本（例如create topic需要v0.10.1.x）

1. /v1/list/topics 查询topic列表
2. /v1/list/consumer_groups 查询consumer group列表
3. /v1/describe/topic 查看某个topic信息
4. /v1/describe/consumer_group 查看某个consumer group信息
5. /v1/create/topic 创建一个新的topic
6. /v1/create/partition topic创建新的partition
7. /v1/delete/topic 删除某个topic
8. /v1/delete/message 删除消息
9. /v1/consumer/messages 消费topic
10. /v1/producer/messages 生产数据

Usage

使用请参考 Kafka HTTP API服务

Performance

1. wrk -t12 -c400 -d30s "http://localhost:38765/v1/list/topics?brokers=127.0.0.1:9092"

Running 30s test @ http://localhost:38765/v1/list/topics?brokers=127.0.0.1:9092
  12 threads and 400 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   308.50ms  168.99ms   1.02s    61.88%
    Req/Sec   111.89     80.33   370.00     71.25%
  38818 requests in 30.10s, 33.21MB read
  Socket errors: connect 0, read 9, write 1, timeout 0
Requests/sec:   1289.60
Transfer/sec:      1.10MB

1. wrk -t12 -c400 -d30s "http://localhost:38765/v1/consumer/messages?brokers=127.0.0.1:9092&topic=kafka_topic_test&consumer_group=cgl&count=3"

Running 30s test @ http://localhost:38765/v1/consumer/messages?brokers=127.0.0.1:9092&topic=kafka_topic_test&consumer_group=cgl&count=3
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us     nan%
    Req/Sec    60.09     76.83   313.00     83.10%
  3571 requests in 30.10s, 1.18MB read
  Socket errors: connect 0, read 3, write 2, timeout 3571
Requests/sec:    118.66
Transfer/sec:     40.00KB

1. wrk -t4 -c2000 -d30s -T5s --script=post.lua --latency "http://localhost:38765/v1/producer/messages"

Running 30s test @ http://localhost:38765/v1/producer/messages
   4 threads and 2000 connections
   Thread Stats   Avg      Stdev     Max   +/- Stdev
     Latency     1.71s   776.75ms   4.15s    66.35%
     Req/Sec   377.20    484.01     3.38k    87.77%
   Latency Distribution
      50%    1.38s
      75%    2.31s
      90%    2.93s
      99%    3.26s
   33209 requests in 30.10s, 8.65MB read
   Socket errors: connect 0, read 314, write 3, timeout 1
 Requests/sec:   1103.14
 Transfer/sec:    294.10KB

TODO

1. 目前没有做权限方面的支持，严格来说按照公有云的使用姿势，HTTP API接口需要支持 AK/SK 签名认证，具体可以参考 HTTP API 接口设计指南

CI Workflow Check

1. gocyclo: 校验代码复杂度
2. gofmt: 校验代码是否都已经格式化
3. golint: 校验代码风格规范是否按照指定标准
4. gosimple: 校验代码是否可以简化
5. govet: 代码静态校验
6. misspell: 校验是否有英文单词拼写错误
7. unused: 校验是否有未使用变量、常量、函数、结构体等
8. gotest: 单元测试校验

Gitlab CI

如果使用Gitlab，也可以通过.gitlab-ci.yml运行Gitlab CI Pipeline，详细介绍请参考下面2篇文章

1. Gitlab 安装使用
2. Gitlab CI和CD配置