fd21f93358
added a plugin option zookeeper_chroot to set up the kafka endpoint in zookeeper, which may not be / (default). This chroot is then configured in the consumergroup config.Zookeeper.Chroot This is workaround the fact that this plugins does not handle the urls like "zookeeper_server:port/chroot" As the peers are stored in an array, it makes no sens to have them beeing URL. Peers should all be members of the same cluster, so they all have the same chroot. |
||
|---|---|---|
| .. | ||
| README.md | ||
| kafka_consumer.go | ||
| kafka_consumer_integration_test.go | ||
| kafka_consumer_test.go | ||
README.md
Kafka Consumer Input Plugin
The Kafka consumer plugin polls a specified Kafka topic and adds messages to InfluxDB. The plugin assumes messages follow the line protocol. Consumer Group is used to talk to the Kafka cluster so multiple instances of telegraf can read from the same topic in parallel.
Configuration
# Read metrics from Kafka topic(s)
[[inputs.kafka_consumer]]
## topic(s) to consume
topics = ["telegraf"]
## an array of Zookeeper connection strings
zookeeper_peers = ["localhost:2181"]
## the name of the consumer group
consumer_group = "telegraf_metrics_consumers"
## Maximum number of metrics to buffer between collection intervals
metric_buffer = 100000
## Offset (must be either "oldest" or "newest")
offset = "oldest"
## Data format to consume. This can be "json", "influx" or "graphite"
## Each data format has it's own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
Testing
Running integration tests requires running Zookeeper & Kafka. The following commands assume you're on OS X & using boot2docker or docker-machine through Docker Toolbox.
To start Kafka & Zookeeper:
docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`boot2docker ip || docker-machine ip <your_machine_name>` --env ADVERTISED_PORT=9092 spotify/kafka