README
¶
Jokk
Jokk/Jåkk is a basic Kafka inspection tool.
The word jokk or jåkk is pronounced as you would the word for an enthusiast or participant in a specified activity (i.e. jock).
Jokk means a river or stream in the Samic langague and the name is chosen for this tool since the aim is that it will provide an insight into the stream of data in Kafka.
Rationale
I have been using Kafka for several years and although there are tools to inspect what is happening inside of Kafka I wanted to build a free-to-use tool that provides the feature set that I am after. That, plus I get to practice my Golang programming skills. Two birds with one stone.
Getting started
Clone this repo, install Go, and build the repo:
cd jokk
go build
For more build options see build.sh:
./build.sh -h
Or if you prefer you can download a pre-built package for your choice of OS.
Configuration
First you must let jokk know what Kafka environments you want to inspect. To do so, simply copy the file jokk.toml.example to jokk.toml and add the required information. The local Kafka environment settings are pre-filled and if you have a local instance running you can access it by setting the environment flag to -e local.
Running Jokk
Jokk supports two modes; command line mode and interactive mode. The sections below describe how to use each of these modes.
Command Line Mode
Below is a non=complete list of commands. For a complete list of the available commands type: ./jokk -h.
Currently issuing the above command will generate the following output:
Usage:
jokk [OPTIONS] <command>
Application Options:
-c, --credentials-file= File that contains the credentials (default: ./jokk.toml)
-n, --environment= Dictates what configuration settings to use (from the jokk.toml file)
-f, --filter= Apply filter to narrow search result
-s, --start-time= Start time format 'YYYY-MM-DD HH:MM:SS' (not applicable to all commands)
-e, --end-time= End time format 'YYYY-MM-DD HH:MM:SS' (not applicable to all commands)
-r, --record-format= Formatting to apply when storing messages (JSON/raw) (default: JSON)
-v, --verbose Display verbose information when available
Help Options:
-h, --help Show this help message
Available commands:
addTopic Add a topic to the Kafka cluster
clearTopic Clear messages from a topic in the Kafka cluster (use -f/filter to determine topic)
deleteTopic Delete a topic from the Kafka cluster (use -f/filter to determine topic)
importMessages Import/publish messages to a topic from a file (use -f/filter to determine topic)
interactive Interactive mode
listTopics List topics and related information
storeMessages Store messages from a topic to a file (use -f/filter to determine topic)
topicInfo Detailed topic info (use -f/filter to determine topic(s))
viewMessages View messages in a topic (use -f/filter to determine topic)
Create a jokk.toml file
There is an example file jokk.toml.example that contains a template for how to create so-called environments that can be pointed to when running Jokk. In this example file there are two environments, local and remote, but you can, of course, create as many as you like and name them as you wish. For example, for my own personal use I have "local", "dev", "integration", "qa", and "prod".
Copy the file over to jokk.toml and amend it in the way you deem necessary.
The examples below use -n local but you can substitute this with whatever environments you have provided in the jokk.toml file.
List topics
./jokk -n local listTopics
To get a more detailed view run with the verbose (-v) flag:
./jokk -n local -v listTopics
Topic info
Run the following command to get detailed info about the topic sometopic. The -f flag stands for filter and it can be used to narrow down what topic you would like to get information about. If no -f is available (or more than one topic match the filter) a numbered list will be shown for a quick select.
./jokk -n local -f topicx topicInfo
Example output:
./jokk -n local -f topicx topicInfo
2022-07-11T17:24:51-06:00 INF Welcome to Jokk
2022-07-11T17:24:51-06:00 INF running settings for environment: local
2022-07-11T17:24:51-06:00 INF calling host: localhost:9092
2022-07-11T17:24:51-06:00 INF found more than one topic [2] matching the filter: topicx
2022-07-11T17:24:51-06:00 INF 1: topicx.y
2022-07-11T17:24:51-06:00 INF 2: topicx.z
pick a number (0 to exit): 1
2022-07-11T17:24:52-06:00 INF
+--------------------------------------------+------+------------+-------------+------+-----------------------+--------+-----------+--------+----------+-----+----------+---------+---------+
| TOPIC | MSGS | PARTITIONS | REPL FACTOR | P ID | P OFFSETS [OLD - NEW] | P MSGS | P % DISTR | LEADER | REPLICAS | ISR | MSGS 24h | MSGS 1h | MSGS 1m |
+--------------------------------------------+------+------------+-------------+------+-----------------------+--------+-----------+--------+----------+-----+----------+---------+---------+
| topicx.y | 0 | 3 | 1 | | | | | | | | | | |
| | | | | 0 | [219259 - 219259] | 0 | 0.00 | 1 | [1] | [1] | 0 | 0 | 0 |
| | | | | 1 | [0 - 0] | 0 | 0.00 | 1 | [1] | [1] | 0 | 0 | 0 |
| | | | | 2 | [0 - 0] | 0 | 0.00 | 1 | [1] | [1] | 0 | 0 | 0 |
+--------------------------------------------+------+------------+-------------+------+-----------------------+--------+-----------+--------+----------+-----+----------+---------+---------+P
Add topic
A simple dialogue will guide you through the creation of a topic:
./jokk -n local addTopic
2022-07-11T17:29:01-06:00 INF Welcome to Jokk
2022-07-11T17:29:01-06:00 INF running settings for environment: local
2022-07-11T17:29:01-06:00 INF calling host: localhost:9092
2022-07-11T17:29:01-06:00 INF topic creation process (enter 0 to exit)
enter topic name: topicx.y
number of partitions: 3
replication factor: 1
2022-07-11T17:29:10-06:00 INF Topic topicx.y created
Delete topic
Similiarly to when adding a topic, the delete topic also runs via a dialogue (note that -f can be used to narrow the options down):
./jokk -n local -f topicx deleteTopic
2022-07-11T17:31:02-06:00 INF Welcome to Jokk
2022-07-11T17:31:02-06:00 INF running settings for environment: local
2022-07-11T17:31:02-06:00 INF calling host: localhost:9092
2022-07-11T17:31:02-06:00 INF found more than one topic [2] matching the filter: topicx
2022-07-11T17:31:02-06:00 INF 1: topicx.y
2022-07-11T17:31:02-06:00 INF 2: topicx.z
pick a number (0 to exit): 1
2022-07-11T17:31:05-06:00 INF Topic topicx.y deleted
Clear topic
Clears (deletes) existing messages from a topic. This feature uses the existing offsets in partitions to determine what messages should be cleared. This means that if you have incoming messages while running this command there is no way to tell exactly what messages will be deleted and which ones will not (given that there may be newer offsets generated by new messages.)
./jokk -n local -f topicx clearTopic
View messages
Provides a way to peek inside the messages available in a topic.
./jokk -n local viewMessages -f topicx -s "2022-07-20 18:00:00" -e "2022-07-20 19:30:00"
2022-07-20T21:04:05-04:00 INF Welcome to Jokk
2022-07-20T21:04:05-04:00 INF running settings for environment: local
2022-07-20T21:04:05-04:00 INF calling host: localhost:9092
2022-07-20T21:04:07-04:00 INF Viewing messages from - to: 2022-07-20 18:00:00 - 2022-07-20 19:30:00
2022-07-20T21:04:07-04:00 INF [Time : Offset: Value] 2022-07-20 18:01:28.971 -0400 EDT : <msg payload>
View another = enter (N to exit):
2022-07-20T21:04:15-04:00 INF [Time : Offset: Value] 2022-07-20 18:01:28.98 -0400 EDT : <msg payload>
View another = enter (N to exit): N
Store messages
Stores messages in a topic in a JSON format to disc.
./jokk -n local storeMessages -f topicx -s "2022-07-20 18:00:00" -e "2022-07-20 19:30:00"
2022-07-20T21:07:35-04:00 INF Welcome to Jokk
2022-07-20T21:07:35-04:00 INF running settings for environment: local
2022-07-20T21:07:35-04:00 INF calling host: localhost:9092
Enter a file name to use (.json will automatically be added): test
2022-07-20T21:07:41-04:00 INF Finished writing messages to file: test.json
The JSON file content will look like this:
[
{
"Headers": [],
"Timestamp": "2022-07-20T18:01:28.971-04:00",
"BlockTimestamp": "0001-01-01T00:00:00Z",
"Key": null,
"Value": <msg payload>,
"Topic": "topicx",
"Partition": 0,
"Offset": 933
},{
"Headers": [],
"Timestamp": "2022-07-20T18:01:28.98-04:00",
"BlockTimestamp": "0001-01-01T00:00:00Z",
"Key": null,
"Value": <msg payload>,
"Topic": "topicx",
"Partition": 0,
"Offset": 934
},
...
]
Import/Publish messages
Imports messages from file to a topic. The layout of the imported file must follow the same as in the store messages output.
./jokk -n local importMessages
2022-08-13T18:02:56-06:00 INF Welcome to Jokk
2022-08-13T18:02:56-06:00 INF running settings for environment: local
2022-08-13T18:02:56-06:00 INF calling host: localhost:9092
Enter a file name to use: msg.json
2022-08-13T18:02:58-06:00 INF 1: topicx.y
2022-08-13T18:02:58-06:00 INF 2: topicx.z
pick a number (0 to exit): 1
2022-08-13T18:03:00-06:00 INF Imported 36 messages to topic topicx.y
Interactive Mode
Instead of running every single task from the command line you can start Jokk in so-called interactive mode. This will open a text based UI from which you can run all commands by pressing commands on your keyboard. It's essentially a very basic UI for interacting with the underlying functionality in Jokk.
./jokk -n local interactive
This will display something like this:
See the area at the bottom named "Available Commands" for what commands are accessible in the current context.
Documentation
¶
There is no documentation for this package.
Source Files
Directories