README
¶
New Relic Infrastructure Integration for Kafka
The New Relic Infrastructure Integration for Kafka captures critical performance metrics and inventory reported by Kafka clusters. Data on Brokers, Topics, Java Producers, and Java Consumers is collected.
Inventory data is obtained mainly from Zookeeper nodes and metrics are collected through JMX.
See our documentation web site for more details.
Requirements
JMX is required to be enabled on the following entities in order to be able to collect metrics:
- Brokers
- Java Producers
- Java Consumers
Information on configuring JMX can be found here.
Installation
- download an archive file for the
KafkaIntegration - extract
kafka-definition.ymland/bindirectory into/var/db/newrelic-infra/newrelic-integrations - add execute permissions for the binary file
nri-kafka(if required) - extract
kafka-config.yml.sampleinto/etc/newrelic-infra/integrations.d
Usage
This is the description about how to run the Kafka Integration with New Relic Infrastructure agent, so it is required to have the agent installed (see agent installation).
In order to use the Kafka Integration it is required to configure kafka-config.yml.sample file. Firstly, rename the file to kafka-config.yml. Then, depending on your needs, specify all instances that you want to monitor. Once this is done, restart the Infrastructure agent.
You can view your data in Insights by creating your own custom NRQL queries. To do so use the KafkaBrokerSample, KafkaTopicSample, KafkaProducerSample, or KafkaConsumerSample event type.
Compatibility
- Supported OS: No limitations
- Kafka versions: 0.8+
Integration Development usage
Assuming that you have source code you can build and run the Kafka Integration locally.
- Go to directory of the Kafka Integration and build it
$ make
- The command above will execute tests for the Kafka Integration and build an executable file called
nri-kafkainbindirectory.
$ ./bin/nri-kafka
- If you want to know more about usage of
./nri-kafkacheck
$ ./bin/nri-kafka -help
For managing external dependencies govendor tool is used. It is required to lock all external dependencies to specific version (if possible) into vendor directory.
Directories
¶
| Path | Synopsis |
|---|---|
|
args
Package args contains a structs for a passed in argument list.
|
Package args contains a structs for a passed in argument list. |
|
brokercollect
Package brokercollect handles collection of Broker inventory and metric data
|
Package brokercollect handles collection of Broker inventory and metric data |
|
conoffsetcollect
Package conoffsetcollect handles collection of consumer offsets for consumer groups
|
Package conoffsetcollect handles collection of consumer offsets for consumer groups |
|
jmxwrapper
Package jmxwrapper contains varaibles for using github.com/newrelic/infra-integrations-sdk/jmx package while allowing everything to be mocked for testing.
|
Package jmxwrapper contains varaibles for using github.com/newrelic/infra-integrations-sdk/jmx package while allowing everything to be mocked for testing. |
|
metrics
Package metrics contains definitions for all JMX collected Metrics, and core collection methods for Brokers, Consumers, and Producers.
|
Package metrics contains definitions for all JMX collected Metrics, and core collection methods for Brokers, Consumers, and Producers. |
|
prodconcollect
Package prodconcollect handles collection of Consumer and Producer metric data
|
Package prodconcollect handles collection of Consumer and Producer metric data |
|
testutils
Package testutils contains setup functions for tests that initialize common variables in the utils package.
|
Package testutils contains setup functions for tests that initialize common variables in the utils package. |
|
topiccollect
Package topiccollect handles collection of Topic inventory and metric data
|
Package topiccollect handles collection of Topic inventory and metric data |
|
zookeeper
Package zookeeper has a common interface and mock objects to implement and test Zookeeper connections.
|
Package zookeeper has a common interface and mock objects to implement and test Zookeeper connections. |