README
¶
DISCONTINUATION OF PROJECT.
This project will no longer be maintained by Intel.
This project has been identified as having known security escapes.
Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.
Intel no longer accepts patches to this project.
DISCONTINUATION OF PROJECT
This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.
Snap publisher plugin - Kafka
Allows publishing of data to Apache Kafka
It's used in the snap framework.
Getting Started
System Requirements
- Uses sarama golang client for Kafka by Shopify
Installation
Download Kafka plugin binary:
You can get the pre-built binaries for your OS and architecture at plugin's GitHub Releases page.
To build the plugin binary:
Fork https://github.com/intelsdi-x/snap-plugin-publisher-kafka
Clone repo into $GOPATH/src/github.com/intelsdi-x/:
$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-kafka.git
Build the plugin by running make within the cloned repo:
$ make
This builds the plugin in ./build
Configuration and Usage
- Set up the snap framework
Task Manifest Config
In task manifest, in config section of Kafka publisher the following settings can be declared:
| Key | Type | Default value | Description |
|---|---|---|---|
| topic | string | "snap" | The topic to send messages |
| brokers | string | "localhost:9092" | Semicolon delimited list of "server:port" brokers Kafka's standard broker communication port is 9092 |
Notice To publish to remote Kafka endpoint, please ensure that all listed brokers are available from the host where snap-plugin-publisher-kafka is running. You can simply check it by running telnet providing address and port where Kafka advertised broker is running, for example telnet 123.45.67.89 9092.
Documentation
Kafka Quickstart
This is a minimal-configuration needed to run the Kafka broker service on Docker
Run ZooKeeper server on docker
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one.
$ docker run -d --name zookeeper jplock/zookeeper:3.4.6
Check if ZooKeeper docker is running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b0ddbdd75cd jplock/zookeeper:3.4.6 "/opt/zookeeper/bin/ 38 seconds ago Up 38 seconds 2181/tcp, 2888/tcp, 3888/tcp zookeeper
Run Kafka server on docker
docker run -d --name kafka --link zookeeper:zookeeper ches/kafka
Verify the running docker containers:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dfb3cdfb3f87 ches/kafka:latest "/start.sh" 7 seconds ago Up 6 seconds 7203/tcp, 9092/tcp kafka
9b0ddbdd75cd jplock/zookeeper:3.4.6 "/opt/zookeeper/bin/ 3 minutes ago Up 3 minutes 2181/tcp, 2888/tcp, 3888/tcp zookeeper
Get Kafka advertised hostname:
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' kafka
172.17.0.14
Read more about Kafka on http://kafka.apache.org
Published data
The plugin publishes all collected metrics serialized as JSON to Kafka. An example of published data is below:
[
{
"timestamp": "2016-07-25T11:27:59.795548989+02:00",
"namespace": "/intel/mock/bar",
"data": 82,
"unit": "",
"tags": {
"plugin_running_on": "my-machine"
},
"version": 0,
"last_advertised_time": "2016-07-25T11:27:21.852064032+02:00"
},
{
"timestamp": "2016-07-25T11:27:59.795549268+02:00",
"namespace": "/intel/mock/foo",
"data": 72,
"unit": "",
"tags": {
"plugin_running_on": "my-machine"
},
"version": 0,
"last_advertised_time": "2016-07-25T11:27:21.852063228+02:00"
}
]
Examples
Example of running psutil collector plugin and publishing data to Kafka.
Set up the Snap framework
Ensure Snap daemon is running:
- initd:
service snap-telemetry start - systemd:
systemctl start snap-telemetry - command line:
sudo snapteld -l 1 -t 0 &
Download and load Snap plugins (paths to binary files for Linux/amd64):
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-kafka/latest/linux/x86_64/snap-plugin-publisher-kafka
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-psutil/latest/linux/x86_64/snap-plugin-collector-psutil
$ snaptel plugin load snap-plugin-publisher-kafka
$ snaptel plugin load snap-plugin-collector-psutil
Create a task manifest (see exemplary tasks),
for example psutil-kafka.json with following content:
{
"version": 1,
"schedule": {
"type": "simple",
"interval": "1s"
},
"workflow": {
"collect": {
"metrics": {
"/intel/psutil/load/load1": {},
"/intel/psutil/load/load15": {}
},
"publish": [
{
"plugin_name": "kafka",
"config": {
"topic": "test",
"brokers": "172.17.0.14:9092"
}
}
]
}
}
}
Create a task:
$ snaptel task create -t psutil-kafka.json
Watch created task:
$ snaptel task watch <task_id>
To stop previously created task:
$ snaptel task stop <task_id>
Roadmap
There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.
If you have a feature request, please add it as an issue and/or submit a pull request.
Community Support
This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework
Contributing
We love contributions!
There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.
License
Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.
Acknowledgements
- Author: Nicholas Weaver
And thank you! Your contribution, through code and participation, is incredibly important to us.
Documentation
¶
There is no documentation for this package.
Source Files
Directories