dom

telegraf

History

Cameron Sparr e436b2d720 Cleanup & standardize config file changes: - -sample-config will now comment out all but a few default plugins. - config file parse errors will output path to bad conf file. - cleanup 80-char line-length and some other style issues. - default package conf file will now have all plugins, but commented out. closes #199 closes #944		2016-04-01 10:59:53 -06:00
..
README.md	Seems to be a toml parse bug around triple pounds	2016-02-18 14:36:03 -07:00
kafka_consumer.go	Cleanup & standardize config file	2016-04-01 10:59:53 -06:00
kafka_consumer_integration_test.go	Flush based on buffer size rather than time	2016-02-16 22:25:22 -07:00
kafka_consumer_test.go	Flush based on buffer size rather than time	2016-02-16 22:25:22 -07:00

README.md

Kafka Consumer Input Plugin

The Kafka consumer plugin polls a specified Kafka topic and adds messages to InfluxDB. The plugin assumes messages follow the line protocol. Consumer Group is used to talk to the Kafka cluster so multiple instances of telegraf can read from the same topic in parallel.

Configuration

# Read metrics from Kafka topic(s)
[[inputs.kafka_consumer]]
  ## topic(s) to consume
  topics = ["telegraf"]
  ## an array of Zookeeper connection strings
  zookeeper_peers = ["localhost:2181"]
  ## the name of the consumer group
  consumer_group = "telegraf_metrics_consumers"
  ## Maximum number of metrics to buffer between collection intervals
  metric_buffer = 100000
  ## Offset (must be either "oldest" or "newest")
  offset = "oldest"

  ## Data format to consume. This can be "json", "influx" or "graphite"
  ## Each data format has it's own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

Testing

Running integration tests requires running Zookeeper & Kafka. The following commands assume you're on OS X & using boot2docker or docker-machine through Docker Toolbox.

To start Kafka & Zookeeper:

docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`boot2docker ip || docker-machine ip <your_machine_name>` --env ADVERTISED_PORT=9092 spotify/kafka