kafka-idle-topics

README

= kafka-idle-topics

image:https://travis-ci.com/abraham-leal/kafka-idle-topics.svg?branch=main["Build Status", link="https://travis-ci.com/abraham-leal/kafka-idle-topics"]

Tooling to find topics that are not utilized in your Kafka cluster.

Ensure efficient usage of your topics in https://www.confluent.io/confluent-cloud/[Confluent Cloud] by discovering topics that aren't being used.

`kafka-idle-topics` evaluates an idle topic through these criteria:

- It has no Consumer Groups (including inactive consumer groups that have offsets committed).
- It has no active production during the time of assessment (30 seconds while running by default).
- It has no data within its partitions.

To download, go to `Run` for different ways of usage. It is also possible to download a release.
The output will be written to a file in the same directory. The name of this file is configurable.

== Usage

[source,bash]
----
Usage of ./kafka-idle-topics:
  -allowList value
    	A comma delimited list of topics to evaluate. It also accepts a path to a file containing a list of topics.
  -bootstrap-servers string
    	Address to the target Kafka Cluster. Accepts multiple endpoints separated by a comma. Can be set using env variable KAFKA_BOOTSTRAP
  -disallowList value
    	A comma delimited list of topics to exclude from evaluation. It also accepts a path to a file containing a list of topics.
  -filename string
    	Custom filename for the output if needed. (default "idleTopics.txt")
  -hideInternalTopics
    	Hide internal topics from assessment.
  -hideTopicsPrefixes value
    	Disqualify provided prefixes from assessment. A comma delimited list. It also accepts a path to a file containing a list.
  -idleMinutes int
    	Amount of minutes a topic should be idle to report it. Can be set using env variable KAFKA_IDLE_MINUTES
  -kafkaSecurity string
    	Type of connection to attempt. Options: plain_tls, plain (no tls), tls (one-way), none. (default "none")
  -password string
    	Password in the PLAIN module. Can be set using env variable KAFKA_PASSWORD
  -productionAssessmentTimeMs int
    	Timeframe to assess active production. (default 30000)
  -skip string
    	Filtering to skip. Options are: production, consumption, storage. This can be a comma-delimited list.
  -username string
    	Username in the PLAIN module. Can be set using env variable KAFKA_USERNAME
  -version
    	Print the current version and exit
----

== Security

`kafka-idle-topics` supports the following security set-ups:

- `plain_tls`: SASL_SSL w/ PLAIN  
- `plain`: SASL_PLAINTEXT w/ PLAIN (No TLS)
- `tls`: SSL (One-way TLS)
- `none`: PLAINTEXT (Default)  

To add certificates the tool can trust for TLS connections, ensure they are located in the subdirectories go seeks.
Commonly it is `/etc/ssl/certs/`.

== Run

You can run `kafka-idle-topics` in multiple configurations:

=== Docker

Example with Confluent Cloud:

[source,bash]
----
docker run \
  -e KAFKA_BOOTSTRAP=<<BOOTSTRAP-SERVERS>>\
  -e KAFKA_USERNAME=<<API-KEY>>\
  -e KAFKA_PASSWORD=<<API-SECRET>>\
  abrahamleal/kafka-idle-topics:latest 
----

=== Locally

Downloading with Go, example with Confluent Cloud:

[source,bash]
----
go get github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics
go install github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics
$GOPATH/bin/kafka-idle-topics -bootstrap-servers <<bootstrap>> -username <<API-KEY>> -password <<API-SECRET>> -kafkaSecurity plain_tls
----

=== Build From Source

Git clone, example with Confluent Cloud:

[source,bash]
----
git clone https://github.com/abraham-leal/kafka-idle-topics
cd kafka-idle-topics/
go build ./cmd/kafka-idle-topics/*
./kafka-idle-topics -bootstrap-servers <<bootstrap>> -username <<API-KEY>> -password <<API-SECRET>> -kafkaSecurity plain_tls
----

=== Filtering the evaluation

It is now possible to filter the topics which are evaluated.
Setting `-allowList` or/and `-disallowList` flags will accept either a comma delimited string, or a file containing
comma delimited entries for topic names.
If specifying a file, make sure it has an extension (such as `.txt`).
A topic specified in `-disallowList` and `-allowList` will be disallowed by default.

Additionally, you may set `-hideTopicsPrefixes` to hide custom prefixes from the evaluation.

== Feature Requests / Issue Reporting

This repo tracks feature requests and issues through Github Issues.
If you'd like to see something fixed that was not caught by testing, or you'd like to see a new feature, please feel free
to file a Github issue in this repo, I'll review and answer at best effort.

Additionally, if you'd like to contribute a fix/feature, please feel free to open a PR for review.