0.3.0 Removing internal parallelism: twemproxy and rabbitmq

closes #449

.gitignore

@@ -1,3 +1,4 @@
-pkg/
 tivan
 .vagrant
+telegraf
+.idea

CHANGELOG.md

@@ -1,27 +1,335 @@
+## v0.3.0 [unreleased]
+
+### Release Notes
+- **breaking change** the `io` plugin has been renamed `diskio`
+- **breaking change** Plugin measurements aggregated into a single measurement.
+- **breaking change** `jolokia` plugin: must use global tag/drop/pass parameters
+for configuration.
+- `twemproxy` plugin: `prefix` option removed.
+- `procstat` cpu measurements are now prepended with `cpu_time_` instead of
+only `cpu_`
+- The prometheus plugin schema has not been changed (measurements have not been
+aggregated).
+
+### Features
+- Plugin measurements aggregated into a single measurement.
+- Added ability to specify per-plugin tags
+- Added ability to specify per-plugin measurement suffix and prefix.
+(`name_prefix` and `name_suffix`)
+- Added ability to override base plugin name. (`name_override`)
+
+### Bugfixes
+
+## v0.2.5 [unreleased]
+
+### Features
+- [#427](https://github.com/influxdb/telegraf/pull/427): zfs plugin: pool stats added. Thanks @allenpetersen!
+- [#428](https://github.com/influxdb/telegraf/pull/428): Amazon Kinesis output. Thanks @jimmystewpot!
+- [#449](https://github.com/influxdb/telegraf/pull/449): influxdb plugin, thanks @mark-rushakoff
+
+### Bugfixes
+- [#430](https://github.com/influxdb/telegraf/issues/430): Network statistics removed in elasticsearch 2.1. Thanks @jipperinbham!
+- [#452](https://github.com/influxdb/telegraf/issues/452): Elasticsearch open file handles error. Thanks @jipperinbham!
+
+## v0.2.4 [2015-12-08]
+
+### Features
+- [#412](https://github.com/influxdb/telegraf/pull/412): Additional memcached stats. Thanks @mgresser!
+- [#410](https://github.com/influxdb/telegraf/pull/410): Additional redis metrics. Thanks @vlaadbrain!
+- [#414](https://github.com/influxdb/telegraf/issues/414): Jolokia plugin auth parameters
+- [#415](https://github.com/influxdb/telegraf/issues/415): memcached plugin: support unix sockets
+- [#418](https://github.com/influxdb/telegraf/pull/418): memcached plugin additional unit tests.
+- [#408](https://github.com/influxdb/telegraf/pull/408): MailChimp plugin.
+- [#382](https://github.com/influxdb/telegraf/pull/382): Add system wide network protocol stats to `net` plugin.
+- [#401](https://github.com/influxdb/telegraf/pull/401): Support pass/drop/tagpass/tagdrop for outputs. Thanks @oldmantaiter!
+
+### Bugfixes
+- [#405](https://github.com/influxdb/telegraf/issues/405): Prometheus output cardinality issue
+- [#388](https://github.com/influxdb/telegraf/issues/388): Fix collection hangup when cpu times decrement.
+
+## v0.2.3 [2015-11-30]
+
+### Release Notes
+- **breaking change** The `kafka` plugin has been renamed to `kafka_consumer`.
+and most of the config option names have changed.
+This only affects the kafka consumer _plugin_ (not the
+output). There were a number of problems with the kafka plugin that led to it
+only collecting data once at startup, so the kafka plugin was basically non-
+functional.
+- Plugins can now be specified as a list, and multiple plugin instances of the
+same type can be specified, like this:
+
+```
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  drop = ["cpu_time"]
+```
+
+- Riemann output added
+- Aerospike plugin: tag changed from `host` -> `aerospike_host`
+
+### Features
+- [#379](https://github.com/influxdb/telegraf/pull/379): Riemann output, thanks @allenj!
+- [#375](https://github.com/influxdb/telegraf/pull/375): kafka_consumer service plugin.
+- [#392](https://github.com/influxdb/telegraf/pull/392): Procstat plugin can now accept pgrep -f pattern, thanks @ecarreras!
+- [#383](https://github.com/influxdb/telegraf/pull/383): Specify plugins as a list.
+- [#354](https://github.com/influxdb/telegraf/pull/354): Add ability to specify multiple metrics in one statsd line. Thanks @MerlinDMC!
+
+### Bugfixes
+- [#371](https://github.com/influxdb/telegraf/issues/371): Kafka consumer plugin not functioning.
+- [#389](https://github.com/influxdb/telegraf/issues/389): NaN value panic
+
+## v0.2.2 [2015-11-18]
+
+### Release Notes
+- 0.2.1 has a bug where all lists within plugins get duplicated, this includes
+lists of servers/URLs. 0.2.2 is being released solely to fix that bug
+
+### Bugfixes
+- [#377](https://github.com/influxdb/telegraf/pull/377): Fix for duplicate slices in plugins.
+
+## v0.2.1 [2015-11-16]
+
+### Release Notes
+- Telegraf will no longer use docker-compose for "long" unit test, it has been
+changed to just run docker commands in the Makefile. See `make docker-run` and
+`make docker-kill`. `make test` will still run all unit tests with docker.
+- Long unit tests are now run in CircleCI, with docker & race detector
+- Redis plugin tag has changed from `host` to `server`
+- HAProxy plugin tag has changed from `host` to `server`
+- UDP output now supported
+- Telegraf will now compile on FreeBSD
+- Users can now specify outputs as lists, specifying multiple outputs of the
+same type.
+
+### Features
+- [#325](https://github.com/influxdb/telegraf/pull/325): NSQ output. Thanks @jrxFive!
+- [#318](https://github.com/influxdb/telegraf/pull/318): Prometheus output. Thanks @oldmantaiter!
+- [#338](https://github.com/influxdb/telegraf/pull/338): Restart Telegraf on package upgrade. Thanks @linsomniac!
+- [#337](https://github.com/influxdb/telegraf/pull/337): Jolokia plugin, thanks @saiello!
+- [#350](https://github.com/influxdb/telegraf/pull/350): Amon output.
+- [#365](https://github.com/influxdb/telegraf/pull/365): Twemproxy plugin by @codeb2cc
+- [#317](https://github.com/influxdb/telegraf/issues/317): ZFS plugin, thanks @cornerot!
+- [#364](https://github.com/influxdb/telegraf/pull/364): Support InfluxDB UDP output.
+- [#370](https://github.com/influxdb/telegraf/pull/370): Support specifying multiple outputs, as lists.
+- [#372](https://github.com/influxdb/telegraf/pull/372): Remove gosigar and update go-dockerclient for FreeBSD support. Thanks @MerlinDMC!
+
+### Bugfixes
+- [#331](https://github.com/influxdb/telegraf/pull/331): Dont overwrite host tag in redis plugin.
+- [#336](https://github.com/influxdb/telegraf/pull/336): Mongodb plugin should take 2 measurements.
+- [#351](https://github.com/influxdb/telegraf/issues/317): Fix continual "CREATE DATABASE" in writes
+- [#360](https://github.com/influxdb/telegraf/pull/360): Apply prefix before ShouldPass check. Thanks @sotfo!
+
+## v0.2.0 [2015-10-27]
+
+### Release Notes
+- The -test flag will now only output 2 collections for plugins that need it
+- There is a new agent configuration option: `flush_interval`. This option tells
+Telegraf how often to flush data to InfluxDB and other output sinks. For example,
+users can set `interval = "2s"` and `flush_interval = "60s"` for Telegraf to
+collect data every 2 seconds, and flush every 60 seconds.
+- `precision` and `utc` are no longer valid agent config values. `precision` has
+moved to the `influxdb` output config, where it will continue to default to "s"
+- debug and test output will now print the raw line-protocol string
+- Telegraf will now, by default, round the collection interval to the nearest
+even interval. This means that `interval="10s"` will collect every :00, :10, etc.
+To ease scale concerns, flushing will be "jittered" by a random amount so that
+all Telegraf instances do not flush at the same time. Both of these options can
+be controlled via the `round_interval` and `flush_jitter` config options.
+- Telegraf will now retry metric flushes twice
+
+### Features
+- [#205](https://github.com/influxdb/telegraf/issues/205): Include per-db redis keyspace info
+- [#226](https://github.com/influxdb/telegraf/pull/226): Add timestamps to points in Kafka/AMQP outputs. Thanks @ekini
+- [#90](https://github.com/influxdb/telegraf/issues/90): Add Docker labels to tags in docker plugin
+- [#223](https://github.com/influxdb/telegraf/pull/223): Add port tag to nginx plugin. Thanks @neezgee!
+- [#227](https://github.com/influxdb/telegraf/pull/227): Add command intervals to exec plugin. Thanks @jpalay!
+- [#241](https://github.com/influxdb/telegraf/pull/241): MQTT Output. Thanks @shirou!
+- Memory plugin: cached and buffered measurements re-added
+- Logging: additional logging for each collection interval, track the number
+of metrics collected and from how many plugins.
+- [#240](https://github.com/influxdb/telegraf/pull/240): procstat plugin, thanks @ranjib!
+- [#244](https://github.com/influxdb/telegraf/pull/244): netstat plugin, thanks @shirou!
+- [#262](https://github.com/influxdb/telegraf/pull/262): zookeeper plugin, thanks @jrxFive!
+- [#237](https://github.com/influxdb/telegraf/pull/237): statsd service plugin, thanks @sparrc
+- [#273](https://github.com/influxdb/telegraf/pull/273): puppet agent plugin, thats @jrxFive!
+- [#280](https://github.com/influxdb/telegraf/issues/280): Use InfluxDB client v2.
+- [#281](https://github.com/influxdb/telegraf/issues/281): Eliminate need to deep copy Batch Points.
+- [#286](https://github.com/influxdb/telegraf/issues/286): bcache plugin, thanks @cornerot!
+- [#287](https://github.com/influxdb/telegraf/issues/287): Batch AMQP output, thanks @ekini!
+- [#301](https://github.com/influxdb/telegraf/issues/301): Collect on even intervals
+- [#298](https://github.com/influxdb/telegraf/pull/298): Support retrying output writes
+- [#300](https://github.com/influxdb/telegraf/issues/300): aerospike plugin. Thanks @oldmantaiter!
+- [#322](https://github.com/influxdb/telegraf/issues/322): Librato output. Thanks @jipperinbham!
+
+### Bugfixes
+- [#228](https://github.com/influxdb/telegraf/pull/228): New version of package will replace old one. Thanks @ekini!
+- [#232](https://github.com/influxdb/telegraf/pull/232): Fix bashism run during deb package installation. Thanks @yankcrime!
+- [#261](https://github.com/influxdb/telegraf/issues/260): RabbitMQ panics if wrong credentials given. Thanks @ekini!
+- [#245](https://github.com/influxdb/telegraf/issues/245): Document Exec plugin example. Thanks @ekini!
+- [#264](https://github.com/influxdb/telegraf/issues/264): logrotate config file fixes. Thanks @linsomniac!
+- [#290](https://github.com/influxdb/telegraf/issues/290): Fix some plugins sending their values as strings.
+- [#289](https://github.com/influxdb/telegraf/issues/289): Fix accumulator panic on nil tags.
+- [#302](https://github.com/influxdb/telegraf/issues/302): Fix `[tags]` getting applied, thanks @gotyaoi!
+
+## v0.1.9 [2015-09-22]
+
+### Release Notes
+- InfluxDB output config change: `url` is now `urls`, and is a list. Config files
+will still be backwards compatible if only `url` is specified.
+- The -test flag will now output two metric collections
+- Support for filtering telegraf outputs on the CLI -- Telegraf will now
+allow filtering of output sinks on the command-line using the `-outputfilter`
+flag, much like how the `-filter` flag works for plugins.
+- Support for filtering on config-file creation -- Telegraf now supports
+filtering to -sample-config command. You can now run
+`telegraf -sample-config -filter cpu -outputfilter influxdb` to get a config
+file with only the cpu plugin defined, and the influxdb output defined.
+- **Breaking Change**: The CPU collection plugin has been refactored to fix some
+bugs and outdated dependency issues. At the same time, I also decided to fix
+a naming consistency issue, so cpu_percentageIdle will become cpu_usage_idle.
+Also, all CPU time measurements now have it indicated in their name, so cpu_idle
+will become cpu_time_idle. Additionally, cpu_time measurements are going to be
+dropped in the default config.
+- **Breaking Change**: The memory plugin has been refactored and some measurements
+have been renamed for consistency. Some measurements have also been removed from being outputted. They are still being collected by gopsutil, and could easily be
+re-added in a "verbose" mode if there is demand for it.
+
+### Features
+- [#143](https://github.com/influxdb/telegraf/issues/143): InfluxDB clustering support
+- [#181](https://github.com/influxdb/telegraf/issues/181): Makefile GOBIN support. Thanks @Vye!
+- [#203](https://github.com/influxdb/telegraf/pull/200): AMQP output. Thanks @ekini!
+- [#182](https://github.com/influxdb/telegraf/pull/182): OpenTSDB output. Thanks @rplessl!
+- [#187](https://github.com/influxdb/telegraf/pull/187): Retry output sink connections on startup.
+- [#220](https://github.com/influxdb/telegraf/pull/220): Add port tag to apache plugin. Thanks @neezgee!
+- [#217](https://github.com/influxdb/telegraf/pull/217): Add filtering for output sinks
+and filtering when specifying a config file.
+
+### Bugfixes
+- [#170](https://github.com/influxdb/telegraf/issues/170): Systemd support
+- [#175](https://github.com/influxdb/telegraf/issues/175): Set write precision before gathering metrics
+- [#178](https://github.com/influxdb/telegraf/issues/178): redis plugin, multiple server thread hang bug
+- Fix net plugin on darwin
+- [#84](https://github.com/influxdb/telegraf/issues/84): Fix docker plugin on CentOS. Thanks @neezgee!
+- [#189](https://github.com/influxdb/telegraf/pull/189): Fix mem_used_perc. Thanks @mced!
+- [#192](https://github.com/influxdb/telegraf/issues/192): Increase compatibility of postgresql plugin. Now supports versions 8.1+
+- [#203](https://github.com/influxdb/telegraf/issues/203): EL5 rpm support. Thanks @ekini!
+- [#206](https://github.com/influxdb/telegraf/issues/206): CPU steal/guest values wrong on linux.
+- [#212](https://github.com/influxdb/telegraf/issues/212): Add hashbang to postinstall script. Thanks @ekini!
+- [#212](https://github.com/influxdb/telegraf/issues/212): Fix makefile warning. Thanks @ekini!
+
+## v0.1.8 [2015-09-04]
+
+### Release Notes
+- Telegraf will now write data in UTC at second precision by default
+- Now using Go 1.5 to build telegraf
+
+### Features
+- [#150](https://github.com/influxdb/telegraf/pull/150): Add Host Uptime metric to system plugin
+- [#158](https://github.com/influxdb/telegraf/pull/158): Apache Plugin. Thanks @KPACHbIuLLIAnO4
+- [#159](https://github.com/influxdb/telegraf/pull/159): Use second precision for InfluxDB writes
+- [#165](https://github.com/influxdb/telegraf/pull/165): Add additional metrics to mysql plugin. Thanks @nickscript0
+- [#162](https://github.com/influxdb/telegraf/pull/162): Write UTC by default, provide option
+- [#166](https://github.com/influxdb/telegraf/pull/166): Upload binaries to S3
+- [#169](https://github.com/influxdb/telegraf/pull/169): Ping plugin
+
+### Bugfixes
+
+## v0.1.7 [2015-08-28]
+
+### Features
+- [#38](https://github.com/influxdb/telegraf/pull/38): Kafka output producer.
+- [#133](https://github.com/influxdb/telegraf/pull/133): Add plugin.Gather error logging. Thanks @nickscript0!
+- [#136](https://github.com/influxdb/telegraf/issues/136): Add a -usage flag for printing usage of a single plugin.
+- [#137](https://github.com/influxdb/telegraf/issues/137): Memcached: fix when a value contains a space
+- [#138](https://github.com/influxdb/telegraf/issues/138): MySQL server address tag.
+- [#142](https://github.com/influxdb/telegraf/pull/142): Add Description and SampleConfig funcs to output interface
+- Indent the toml config file for readability
+
+### Bugfixes
+- [#128](https://github.com/influxdb/telegraf/issues/128): system_load measurement missing.
+- [#129](https://github.com/influxdb/telegraf/issues/129): Latest pkg url fix.
+- [#131](https://github.com/influxdb/telegraf/issues/131): Fix memory reporting on linux & darwin. Thanks @subhachandrachandra!
+- [#140](https://github.com/influxdb/telegraf/issues/140): Memory plugin prec->perc typo fix. Thanks @brunoqc!
+
+## v0.1.6 [2015-08-20]
+
+### Features
+- [#112](https://github.com/influxdb/telegraf/pull/112): Datadog output. Thanks @jipperinbham!
+- [#116](https://github.com/influxdb/telegraf/pull/116): Use godep to vendor all dependencies
+- [#120](https://github.com/influxdb/telegraf/pull/120): Httpjson plugin. Thanks @jpalay & @alvaromorales!
+
+### Bugfixes
+- [#113](https://github.com/influxdb/telegraf/issues/113): Update README with Telegraf/InfluxDB compatibility
+- [#118](https://github.com/influxdb/telegraf/pull/118): Fix for disk usage stats in Windows. Thanks @srfraser!
+- [#122](https://github.com/influxdb/telegraf/issues/122): Fix for DiskUsage segv fault. Thanks @srfraser!
+- [#126](https://github.com/influxdb/telegraf/issues/126): Nginx plugin not catching net.SplitHostPort error
+
+## v0.1.5 [2015-08-13]
+
+### Features
+- [#54](https://github.com/influxdb/telegraf/pull/54): MongoDB plugin. Thanks @jipperinbham!
+- [#55](https://github.com/influxdb/telegraf/pull/55): Elasticsearch plugin. Thanks @brocaar!
+- [#71](https://github.com/influxdb/telegraf/pull/71): HAProxy plugin. Thanks @kureikain!
+- [#72](https://github.com/influxdb/telegraf/pull/72): Adding TokuDB metrics to MySQL. Thanks vadimtk!
+- [#73](https://github.com/influxdb/telegraf/pull/73): RabbitMQ plugin. Thanks @ianunruh!
+- [#77](https://github.com/influxdb/telegraf/issues/77): Automatically create database.
+- [#79](https://github.com/influxdb/telegraf/pull/56): Nginx plugin. Thanks @codeb2cc!
+- [#86](https://github.com/influxdb/telegraf/pull/86): Lustre2 plugin. Thanks srfraser!
+- [#91](https://github.com/influxdb/telegraf/pull/91): Unit testing
+- [#92](https://github.com/influxdb/telegraf/pull/92): Exec plugin. Thanks @alvaromorales!
+- [#98](https://github.com/influxdb/telegraf/pull/98): LeoFS plugin. Thanks @mocchira!
+- [#103](https://github.com/influxdb/telegraf/pull/103): Filter by metric tags. Thanks @srfraser!
+- [#106](https://github.com/influxdb/telegraf/pull/106): Options to filter plugins on startup. Thanks @zepouet!
+- [#107](https://github.com/influxdb/telegraf/pull/107): Multiple outputs beyong influxdb. Thanks @jipperinbham!
+- [#108](https://github.com/influxdb/telegraf/issues/108): Support setting per-CPU and total-CPU gathering.
+- [#111](https://github.com/influxdb/telegraf/pull/111): Report CPU Usage in cpu plugin. Thanks @jpalay!
+
+### Bugfixes
+- [#85](https://github.com/influxdb/telegraf/pull/85): Fix GetLocalHost testutil function for mac users
+- [#89](https://github.com/influxdb/telegraf/pull/89): go fmt fixes
+- [#94](https://github.com/influxdb/telegraf/pull/94): Fix for issue #93, explicitly call sarama.v1 -> sarama
+- [#101](https://github.com/influxdb/telegraf/issues/101): switch back from master branch if building locally
+- [#99](https://github.com/influxdb/telegraf/issues/99): update integer output to new InfluxDB line protocol format
+
+## v0.1.4 [2015-07-09]
+
+### Features
+- [#56](https://github.com/influxdb/telegraf/pull/56): Update README for Kafka plugin. Thanks @EmilS!
+
+### Bugfixes
+- [#50](https://github.com/influxdb/telegraf/pull/50): Fix init.sh script to use telegraf directory. Thanks @jseriff!
+- [#52](https://github.com/influxdb/telegraf/pull/52): Update CHANGELOG to reference updated directory. Thanks @benfb!
+
 ## v0.1.3 [2015-07-05]
 
 ### Features
-- [#35](https://github.com/influxdb/influxdb/pull/35): Add Kafka plugin. Thanks @EmilS!
-- [#47](https://github.com/influxdb/influxdb/pull/47): Add RethinkDB plugin. Thanks @jipperinbham!
+- [#35](https://github.com/influxdb/telegraf/pull/35): Add Kafka plugin. Thanks @EmilS!
+- [#47](https://github.com/influxdb/telegraf/pull/47): Add RethinkDB plugin. Thanks @jipperinbham!
 
 ### Bugfixes
-- [#45](https://github.com/influxdb/influxdb/pull/45): Skip disk tags that don't have a value. Thanks @jhofeditz!
-- [#43](https://github.com/influxdb/influxdb/pull/43): Fix bug in MySQL plugin. Thanks @marcosnils!
+- [#45](https://github.com/influxdb/telegraf/pull/45): Skip disk tags that don't have a value. Thanks @jhofeditz!
+- [#43](https://github.com/influxdb/telegraf/pull/43): Fix bug in MySQL plugin. Thanks @marcosnils!
 
 ## v0.1.2 [2015-07-01]
 
 ### Features
-- [#12](https://github.com/influxdb/influxdb/pull/12): Add Linux/ARM to the list of built binaries. Thanks @voxxit!
-- [#14](https://github.com/influxdb/influxdb/pull/14): Clarify the S3 buckets that Telegraf is pushed to.
-- [#16](https://github.com/influxdb/influxdb/pull/16): Convert Redis to use URI, support Redis AUTH. Thanks @jipperinbham!
-- [#21](https://github.com/influxdb/influxdb/pull/21): Add memcached plugin. Thanks @Yukki!
+- [#12](https://github.com/influxdb/telegraf/pull/12): Add Linux/ARM to the list of built binaries. Thanks @voxxit!
+- [#14](https://github.com/influxdb/telegraf/pull/14): Clarify the S3 buckets that Telegraf is pushed to.
+- [#16](https://github.com/influxdb/telegraf/pull/16): Convert Redis to use URI, support Redis AUTH. Thanks @jipperinbham!
+- [#21](https://github.com/influxdb/telegraf/pull/21): Add memcached plugin. Thanks @Yukki!
 
 ### Bugfixes
-- [#13](https://github.com/influxdb/influxdb/pull/13): Fix the packaging script.
-- [#19](https://github.com/influxdb/influxdb/pull/19): Add host name to metric tags. Thanks @sherifzain!
-- [#20](https://github.com/influxdb/influxdb/pull/20): Fix race condition with accumulator mutex. Thanks @nkatsaros!
-- [#23](https://github.com/influxdb/influxdb/pull/23): Change name of folder for packages. Thanks @colinrymer!
-- [#32](https://github.com/influxdb/influxdb/pull/32): Fix spelling of memoory -> memory. Thanks @tylernisonoff!
+- [#13](https://github.com/influxdb/telegraf/pull/13): Fix the packaging script.
+- [#19](https://github.com/influxdb/telegraf/pull/19): Add host name to metric tags. Thanks @sherifzain!
+- [#20](https://github.com/influxdb/telegraf/pull/20): Fix race condition with accumulator mutex. Thanks @nkatsaros!
+- [#23](https://github.com/influxdb/telegraf/pull/23): Change name of folder for packages. Thanks @colinrymer!
+- [#32](https://github.com/influxdb/telegraf/pull/32): Fix spelling of memoory -> memory. Thanks @tylernisonoff!
 
 ## v0.1.1 [2015-06-19]
 

CONFIGURATION.md

@@ -0,0 +1,177 @@
+# Telegraf Configuration
+
+## Plugin Configuration
+
+There are some configuration options that are configurable per plugin:
+
+* **name_override**: Override the base name of the measurement.
+(Default is the name of the plugin).
+* **name_prefix**: Specifies a prefix to attach to the measurement name.
+* **name_suffix**: Specifies a suffix to attach to the measurement name.
+* **tags**: A map of tags to apply to a specific plugin's measurements.
+
+### Plugin Filters
+
+There are also filters that can be configured per plugin:
+
+* **pass**: An array of strings that is used to filter metrics generated by the
+current plugin. Each string in the array is tested as a glob match against field names
+and if it matches, the field is emitted.
+* **drop**: The inverse of pass, if a field name matches, it is not emitted.
+* **tagpass**: tag names and arrays of strings that are used to filter
+measurements by the current plugin. Each string in the array is tested as a glob
+match against the tag name, and if it matches the measurement is emitted.
+* **tagdrop**: The inverse of tagpass. If a tag matches, the measurement is not emitted.
+This is tested on measurements that have passed the tagpass test.
+* **interval**: How often to gather this metric. Normal plugins use a single
+global interval, but if one particular plugin should be run less or more often,
+you can configure that here.
+
+### Plugin Configuration Examples
+
+This is a full working config that will output CPU data to an InfluxDB instance
+at 192.168.59.103:8086, tagging measurements with dc="denver-1". It will output
+measurements at a 10s interval and will collect per-cpu data, dropping any
+fields which begin with `time_`.
+
+```toml
+[tags]
+  dc = "denver-1"
+
+[agent]
+  interval = "10s"
+
+# OUTPUTS
+[outputs]
+[[outputs.influxdb]]
+  url = "http://192.168.59.103:8086" # required.
+  database = "telegraf" # required.
+  precision = "s"
+
+# PLUGINS
+[plugins]
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  # filter all fields beginning with 'time_'
+  drop = ["time_*"]
+```
+
+### Plugin Config: tagpass and tagdrop
+
+```toml
+[plugins]
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  drop = ["cpu_time"]
+  # Don't collect CPU data for cpu6 & cpu7
+  [plugins.cpu.tagdrop]
+    cpu = [ "cpu6", "cpu7" ]
+
+[[plugins.disk]]
+  [plugins.disk.tagpass]
+    # tagpass conditions are OR, not AND.
+    # If the (filesystem is ext4 or xfs) OR (the path is /opt or /home)
+    # then the metric passes
+    fstype = [ "ext4", "xfs" ]
+    # Globs can also be used on the tag values
+    path = [ "/opt", "/home*" ]
+```
+
+### Plugin Config: pass and drop
+
+```toml
+# Drop all metrics for guest & steal CPU usage
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+  drop = ["usage_guest", "usage_steal"]
+
+# Only store inode related metrics for disks
+[[plugins.disk]]
+  pass = ["inodes*"]
+```
+
+### Plugin config: prefix, suffix, and override
+
+This plugin will emit measurements with the name `cpu_total`
+
+```toml
+[[plugins.cpu]]
+  name_suffix = "_total"
+  percpu = false
+  totalcpu = true
+```
+
+This will emit measurements with the name `foobar`
+
+```toml
+[[plugins.cpu]]
+  name_override = "foobar"
+  percpu = false
+  totalcpu = true
+```
+
+### Plugin config: tags
+
+This plugin will emit measurements with two additional tags: `tag1=foo` and
+`tag2=bar`
+
+```toml
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+  [plugins.cpu.tags]
+    tag1 = "foo"
+    tag2 = "bar"
+```
+
+### Multiple plugins of the same type
+
+Additional plugins (or outputs) of the same type can be specified,
+just define more instances in the config file:
+
+```toml
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  drop = ["cpu_time*"]
+```
+
+## Output Configuration
+
+Telegraf also supports specifying multiple output sinks to send data to,
+configuring each output sink is different, but examples can be
+found by running `telegraf -sample-config`.
+
+Outputs also support the same configurable options as plugins
+(pass, drop, tagpass, tagdrop), added in 0.2.4
+
+```toml
+[[outputs.influxdb]]
+  urls = [ "http://localhost:8086" ]
+  database = "telegraf"
+  precision = "s"
+  # Drop all measurements that start with "aerospike"
+  drop = ["aerospike*"]
+
+[[outputs.influxdb]]
+  urls = [ "http://localhost:8086" ]
+  database = "telegraf-aerospike-data"
+  precision = "s"
+  # Only accept aerospike data:
+  pass = ["aerospike*"]
+
+[[outputs.influxdb]]
+  urls = [ "http://localhost:8086" ]
+  database = "telegraf-cpu0-data"
+  precision = "s"
+  # Only store measurements where the tag "cpu" matches the value "cpu0"
+  [outputs.influxdb.tagpass]
+    cpu = ["cpu0"]
+```

CONTRIBUTING.md

@@ -0,0 +1,293 @@
+## Sign the CLA
+
+Before we can merge a pull request, you will need to sign the CLA,
+which can be found [on our website](http://influxdb.com/community/cla.html)
+
+## Plugins
+
+This section is for developers who want to create new collection plugins.
+Telegraf is entirely plugin driven. This interface allows for operators to
+pick and chose what is gathered as well as makes it easy for developers
+to create new ways of generating metrics.
+
+Plugin authorship is kept as simple as possible to promote people to develop
+and submit new plugins.
+
+### Plugin Guidelines
+
+* A plugin must conform to the `plugins.Plugin` interface.
+* Each generated metric automatically has the name of the plugin that generated
+it prepended. This is to keep plugins honest.
+* Plugins should call `plugins.Add` in their `init` function to register themselves.
+See below for a quick example.
+* To be available within Telegraf itself, plugins must add themselves to the
+`github.com/influxdb/telegraf/plugins/all/all.go` file.
+* The `SampleConfig` function should return valid toml that describes how the
+plugin can be configured. This is include in `telegraf -sample-config`.
+* The `Description` function should say in one line what this plugin does.
+
+### Plugin interface
+
+```go
+type Plugin interface {
+    SampleConfig() string
+    Description() string
+    Gather(Accumulator) error
+}
+
+type Accumulator interface {
+    Add(measurement string,
+        value interface{},
+        tags map[string]string,
+        timestamp ...time.Time)
+    AddFields(measurement string,
+        fields map[string]interface{},
+        tags map[string]string,
+        timestamp ...time.Time)
+}
+```
+
+### Accumulator
+
+The way that a plugin emits metrics is by interacting with the Accumulator.
+
+The `Add` function takes 3 arguments:
+* **measurement**: A string description of the metric. For instance `bytes_read` or `faults`.
+* **value**: A value for the metric. This accepts 5 different types of value:
+  * **int**: The most common type. All int types are accepted but favor using `int64`
+  Useful for counters, etc.
+  * **float**: Favor `float64`, useful for gauges, percentages, etc.
+  * **bool**: `true` or `false`, useful to indicate the presence of a state. `light_on`, etc.
+  * **string**: Typically used to indicate a message, or some kind of freeform information.
+  * **time.Time**: Useful for indicating when a state last occurred, for instance `light_on_since`.
+* **tags**: This is a map of strings to strings to describe the where or who
+about the metric. For instance, the `net` plugin adds a tag named `"interface"`
+set to the name of the network interface, like `"eth0"`.
+
+The `AddFieldsWithTime` allows multiple values for a point to be passed. The values
+used are the same type profile as **value** above. The **timestamp** argument
+allows a point to be registered as having occurred at an arbitrary time.
+
+Let's say you've written a plugin that emits metrics about processes on the current host.
+
+```go
+
+type Process struct {
+    CPUTime float64
+    MemoryBytes int64
+    PID int
+}
+
+func Gather(acc plugins.Accumulator) error {
+    for _, process := range system.Processes() {
+        tags := map[string]string {
+            "pid": fmt.Sprintf("%d", process.Pid),
+        }
+
+        acc.Add("cpu", process.CPUTime, tags, time.Now())
+        acc.Add("memory", process.MemoryBytes, tags, time.Now())
+    }
+}
+```
+
+### Plugin Example
+
+```go
+package simple
+
+// simple.go
+
+import "github.com/influxdb/telegraf/plugins"
+
+type Simple struct {
+    Ok bool
+}
+
+func (s *Simple) Description() string {
+    return "a demo plugin"
+}
+
+func (s *Simple) SampleConfig() string {
+    return "ok = true # indicate if everything is fine"
+}
+
+func (s *Simple) Gather(acc plugins.Accumulator) error {
+    if s.Ok {
+        acc.Add("state", "pretty good", nil)
+    } else {
+        acc.Add("state", "not great", nil)
+    }
+
+    return nil
+}
+
+func init() {
+    plugins.Add("simple", func() plugins.Plugin { return &Simple{} })
+}
+```
+
+## Service Plugins
+
+This section is for developers who want to create new "service" collection
+plugins. A service plugin differs from a regular plugin in that it operates
+a background service while Telegraf is running. One example would be the `statsd`
+plugin, which operates a statsd server.
+
+Service Plugins are substantially more complicated than a regular plugin, as they
+will require threads and locks to verify data integrity. Service Plugins should
+be avoided unless there is no way to create their behavior with a regular plugin.
+
+Their interface is quite similar to a regular plugin, with the addition of `Start()`
+and `Stop()` methods.
+
+### Service Plugin Guidelines
+
+* Same as the `Plugin` guidelines, except that they must conform to the
+`plugins.ServicePlugin` interface.
+
+### Service Plugin interface
+
+```go
+type ServicePlugin interface {
+    SampleConfig() string
+    Description() string
+    Gather(Accumulator) error
+    Start() error
+    Stop()
+}
+```
+
+## Outputs
+
+This section is for developers who want to create a new output sink. Outputs
+are created in a similar manner as collection plugins, and their interface has
+similar constructs.
+
+### Output Guidelines
+
+* An output must conform to the `outputs.Output` interface.
+* Outputs should call `outputs.Add` in their `init` function to register themselves.
+See below for a quick example.
+* To be available within Telegraf itself, plugins must add themselves to the
+`github.com/influxdb/telegraf/outputs/all/all.go` file.
+* The `SampleConfig` function should return valid toml that describes how the
+output can be configured. This is include in `telegraf -sample-config`.
+* The `Description` function should say in one line what this output does.
+
+### Output interface
+
+```go
+type Output interface {
+    Connect() error
+    Close() error
+    Description() string
+    SampleConfig() string
+    Write(points []*client.Point) error
+}
+```
+
+### Output Example
+
+```go
+package simpleoutput
+
+// simpleoutput.go
+
+import "github.com/influxdb/telegraf/outputs"
+
+type Simple struct {
+    Ok bool
+}
+
+func (s *Simple) Description() string {
+    return "a demo output"
+}
+
+func (s *Simple) SampleConfig() string {
+    return "url = localhost"
+}
+
+func (s *Simple) Connect() error {
+    // Make a connection to the URL here
+    return nil
+}
+
+func (s *Simple) Close() error {
+    // Close connection to the URL here
+    return nil
+}
+
+func (s *Simple) Write(points []*client.Point) error {
+    for _, pt := range points {
+        // write `pt` to the output sink here
+    }
+    return nil
+}
+
+func init() {
+    outputs.Add("simpleoutput", func() outputs.Output { return &Simple{} })
+}
+
+```
+
+## Service Outputs
+
+This section is for developers who want to create new "service" output. A
+service output differs from a regular output in that it operates a background service
+while Telegraf is running. One example would be the `prometheus_client` output,
+which operates an HTTP server.
+
+Their interface is quite similar to a regular output, with the addition of `Start()`
+and `Stop()` methods.
+
+### Service Output Guidelines
+
+* Same as the `Output` guidelines, except that they must conform to the
+`plugins.ServiceOutput` interface.
+
+### Service Output interface
+
+```go
+type ServiceOutput interface {
+    Connect() error
+    Close() error
+    Description() string
+    SampleConfig() string
+    Write(points []*client.Point) error
+    Start() error
+    Stop()
+}
+```
+
+## Unit Tests
+
+### Execute short tests
+
+execute `make test-short`
+
+### Execute long tests
+
+As Telegraf collects metrics from several third-party services it becomes a
+difficult task to mock each service as some of them have complicated protocols
+which would take some time to replicate.
+
+To overcome this situation we've decided to use docker containers to provide a
+fast and reproducible environment to test those services which require it.
+For other situations
+(i.e: https://github.com/influxdb/telegraf/blob/master/plugins/redis/redis_test.go )
+a simple mock will suffice.
+
+To execute Telegraf tests follow these simple steps:
+
+- Install docker following [these](https://docs.docker.com/installation/)
+instructions
+- execute `make test`
+
+**OSX users**: you will need to install `boot2docker` or `docker-machine`.
+The Makefile will assume that you have a `docker-machine` box called `default` to
+get the IP address.
+
+### Unit test troubleshooting
+
+Try cleaning up your test environment by executing `make docker-kill` and
+re-running

Godeps

@@ -0,0 +1,52 @@
+git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git dbd8d5c40a582eb9adacde36b47932b3a3ad0034
+github.com/Shopify/sarama 159e9990b0796511607dd0d7aaa3eb37d1829d16
+github.com/Sirupsen/logrus 446d1c146faa8ed3f4218f056fcd165f6bcfda81
+github.com/amir/raidman 6a8e089bbe32e6b907feae5ba688841974b3c339
+github.com/armon/go-metrics 06b60999766278efd6d2b5d8418a58c3d5b99e87
+github.com/aws/aws-sdk-go 999b1591218c36d5050d1ba7266eba956e65965f
+github.com/beorn7/perks b965b613227fddccbfffe13eae360ed3fa822f8d
+github.com/boltdb/bolt b34b35ea8d06bb9ae69d9a349119252e4c1d8ee0
+github.com/cenkalti/backoff 4dc77674aceaabba2c7e3da25d4c823edfb73f99
+github.com/dancannon/gorethink a124c9663325ed9f7fb669d17c69961b59151e6e
+github.com/davecgh/go-spew 5215b55f46b2b919f50a1df0eaa5886afe4e3b3d
+github.com/eapache/go-resiliency f341fb4dca45128e4aa86389fa6a675d55fe25e1
+github.com/eapache/queue ded5959c0d4e360646dc9e9908cff48666781367
+github.com/fsouza/go-dockerclient 7177a9e3543b0891a5d91dbf7051e0f71455c8ef
+github.com/go-ini/ini 9314fb0ef64171d6a3d0a4fa570dfa33441cba05
+github.com/go-sql-driver/mysql d512f204a577a4ab037a1816604c48c9c13210be
+github.com/gogo/protobuf e492fd34b12d0230755c45aa5fb1e1eea6a84aa9
+github.com/golang/protobuf 68415e7123da32b07eab49c96d2c4d6158360e9b
+github.com/golang/snappy 723cc1e459b8eea2dea4583200fd60757d40097a
+github.com/gonuts/go-shellquote e842a11b24c6abfb3dd27af69a17f482e4b483c2
+github.com/hailocab/go-hostpool 0637eae892be221164aff5fcbccc57171aea6406
+github.com/hashicorp/go-msgpack fa3f63826f7c23912c15263591e65d54d080b458
+github.com/hashicorp/raft d136cd15dfb7876fd7c89cad1995bc4f19ceb294
+github.com/hashicorp/raft-boltdb d1e82c1ec3f15ee991f7cc7ffd5b67ff6f5bbaee
+github.com/influxdb/influxdb 69a7664f2d4b75aec300b7cbfc7e57c971721f04
+github.com/jmespath/go-jmespath c01cf91b011868172fdcd9f41838e80c9d716264
+github.com/klauspost/crc32 0aff1ea9c20474c3901672b5b6ead0ac611156de
+github.com/lib/pq 11fc39a580a008f1f39bb3d11d984fb34ed778d9
+github.com/matttproud/golang_protobuf_extensions d0c3fe89de86839aecf2e0579c40ba3bb336a453
+github.com/mreiferson/go-snappystream 028eae7ab5c4c9e2d1cb4c4ca1e53259bbe7e504
+github.com/naoina/go-stringutil 6b638e95a32d0c1131db0e7fe83775cbea4a0d0b
+github.com/naoina/toml 751171607256bb66e64c9f0220c00662420c38e9
+github.com/nsqio/go-nsq 2118015c120962edc5d03325c680daf3163a8b5f
+github.com/pborman/uuid cccd189d45f7ac3368a0d127efb7f4d08ae0b655
+github.com/pmezard/go-difflib e8554b8641db39598be7f6342874b958f12ae1d4
+github.com/prometheus/client_golang 67994f177195311c3ea3d4407ed0175e34a4256f
+github.com/prometheus/client_model fa8ad6fec33561be4280a8f0514318c79d7f6cb6
+github.com/prometheus/common 56b90312e937d43b930f06a59bf0d6a4ae1944bc
+github.com/prometheus/procfs 406e5b7bfd8201a36e2bb5f7bdae0b03380c2ce8
+github.com/samuel/go-zookeeper 218e9c81c0dd8b3b18172b2bbfad92cc7d6db55f
+github.com/shirou/gopsutil fc932d9090f13a84fb4b3cb8baa124610cab184c
+github.com/streadway/amqp b4f3ceab0337f013208d31348b578d83c0064744
+github.com/stretchr/objx 1a9d0bb9f541897e62256577b352fdbc1fb4fd94
+github.com/stretchr/testify e3a8ff8ce36581f87a15341206f205b1da467059
+github.com/wvanbergen/kafka 1a8639a45164fcc245d5c7b4bd3ccfbd1a0ffbf3
+github.com/wvanbergen/kazoo-go 0f768712ae6f76454f987c3356177e138df258f8
+golang.org/x/crypto 7b85b097bf7527677d54d3220065e966a0e3b613
+golang.org/x/net 1796f9b8b7178e3c7587dff118d3bb9d37f9b0b3
+gopkg.in/dancannon/gorethink.v1 a124c9663325ed9f7fb669d17c69961b59151e6e
+gopkg.in/fatih/pool.v2 cba550ebf9bce999a02e963296d4bc7a486cb715
+gopkg.in/mgo.v2 e30de8ac9ae3b30df7065f766c71f88bba7d4e49
+gopkg.in/yaml.v2 f7716cbe52baa25d2e9b0d0da546fcf909fc16b4

LICENSE_OF_DEPENDENCIES.md

@@ -0,0 +1,33 @@
+# List
+- github.com/Shopify/sarama [MIT LICENSE](https://github.com/Shopify/sarama/blob/master/MIT-LICENSE)
+- github.com/Sirupsen/logrus [MIT LICENSE](https://github.com/Sirupsen/logrus/blob/master/LICENSE)
+- github.com/armon/go-metrics [MIT LICENSE](https://github.com/armon/go-metrics/blob/master/LICENSE)
+- github.com/boltdb/bolt [MIT LICENSE](https://github.com/boltdb/bolt/blob/master/LICENSE)
+- github.com/cenkalti/backoff [MIT LICENSE](https://github.com/cenkalti/backoff/blob/master/LICENSE)
+- github.com/dancannon/gorethink [APACHE LICENSE](https://github.com/dancannon/gorethink/blob/master/LICENSE)
+- github.com/eapache/go-resiliency [MIT LICENSE](https://github.com/eapache/go-resiliency/blob/master/LICENSE)
+- github.com/eapache/queue [MIT LICENSE](https://github.com/eapache/queue/blob/master/LICENSE)
+- github.com/fsouza/go-dockerclient [BSD LICENSE](https://github.com/fsouza/go-dockerclient/blob/master/LICENSE)
+- github.com/go-sql-driver/mysql [MPL LICENSE](https://github.com/go-sql-driver/mysql/blob/master/LICENSE)
+- github.com/gogo/protobuf [BSD LICENSE](https://github.com/gogo/protobuf/blob/master/LICENSE)
+- github.com/golang/protobuf [BSD LICENSE](https://github.com/golang/protobuf/blob/master/LICENSE)
+- github.com/golang/snappy [BSD LICENSE](https://github.com/golang/snappy/blob/master/LICENSE)
+- github.com/gonuts/go-shellquote (No License, but the project it was forked from https://github.com/kballard/go-shellquote is [MIT](https://github.com/kballard/go-shellquote/blob/master/LICENSE)).
+- github.com/hashicorp/go-msgpack [BSD LICENSE](https://github.com/hashicorp/go-msgpack/blob/master/LICENSE)
+- github.com/hashicorp/raft [MPL LICENSE](https://github.com/hashicorp/raft/blob/master/LICENSE)
+- github.com/hashicorp/raft-boltdb [MPL LICENSE](https://github.com/hashicorp/raft-boltdb/blob/master/LICENSE)
+- github.com/lib/pq [MIT LICENSE](https://github.com/lib/pq/blob/master/LICENSE.md)
+- github.com/matttproud/golang_protobuf_extensions [APACHE LICENSE](https://github.com/matttproud/golang_protobuf_extensions/blob/master/LICENSE)
+- github.com/naoina/go-stringutil [MIT LICENSE](https://github.com/naoina/go-stringutil/blob/master/LICENSE)
+- github.com/naoina/toml [MIT LICENSE](https://github.com/naoina/toml/blob/master/LICENSE)
+- github.com/prometheus/client_golang [APACHE LICENSE](https://github.com/prometheus/client_golang/blob/master/LICENSE)
+- github.com/samuel/go-zookeeper [BSD LICENSE](https://github.com/samuel/go-zookeeper/blob/master/LICENSE)
+- github.com/stretchr/objx [MIT LICENSE](github.com/stretchr/objx)
+- github.com/stretchr/testify [MIT LICENSE](https://github.com/stretchr/testify/blob/master/LICENCE.txt)
+- github.com/wvanbergen/kafka [MIT LICENSE](https://github.com/wvanbergen/kafka/blob/master/LICENSE)
+- github.com/wvanbergen/kazoo-go [MIT LICENSE](https://github.com/wvanbergen/kazoo-go/blob/master/MIT-LICENSE)
+- gopkg.in/dancannon/gorethink.v1 [APACHE LICENSE](https://github.com/dancannon/gorethink/blob/v1.1.2/LICENSE)
+- gopkg.in/mgo.v2 [BSD LICENSE](https://github.com/go-mgo/mgo/blob/v2/LICENSE)
+- golang.org/x/crypto/* [BSD LICENSE](https://github.com/golang/crypto/blob/master/LICENSE)
+- internal Glob function [MIT LICENSE](https://github.com/ryanuber/go-glob/blob/master/LICENSE)
+

Makefile

@@ -0,0 +1,98 @@
+UNAME := $(shell sh -c 'uname')
+VERSION := $(shell sh -c 'git describe --always --tags')
+ifdef GOBIN
+PATH := $(GOBIN):$(PATH)
+else
+PATH := $(subst :,/bin:,$(GOPATH))/bin:$(PATH)
+endif
+
+# Standard Telegraf build
+default: prepare build
+
+# Only run the build (no dependency grabbing)
+build:
+	go build -o telegraf -ldflags \
+		"-X main.Version=$(VERSION)" \
+		./cmd/telegraf/telegraf.go
+
+# Build with race detector
+dev: prepare
+	go build -race -o telegraf -ldflags \
+		"-X main.Version=$(VERSION)" \
+		./cmd/telegraf/telegraf.go
+
+# Build linux 64-bit, 32-bit and arm architectures
+build-linux-bins: prepare
+	GOARCH=amd64 GOOS=linux go build -o telegraf_linux_amd64 \
+								-ldflags "-X main.Version=$(VERSION)" \
+								./cmd/telegraf/telegraf.go
+	GOARCH=386 GOOS=linux go build -o telegraf_linux_386 \
+								-ldflags "-X main.Version=$(VERSION)" \
+								./cmd/telegraf/telegraf.go
+	GOARCH=arm GOOS=linux go build -o telegraf_linux_arm \
+								-ldflags "-X main.Version=$(VERSION)" \
+								./cmd/telegraf/telegraf.go
+
+# Get dependencies and use gdm to checkout changesets
+prepare:
+	go get ./...
+	go get github.com/sparrc/gdm
+	gdm restore
+
+# Run all docker containers necessary for unit tests
+docker-run:
+ifeq ($(UNAME), Darwin)
+	docker run --name kafka \
+		-e ADVERTISED_HOST=$(shell sh -c 'boot2docker ip || docker-machine ip default') \
+		-e ADVERTISED_PORT=9092 \
+		-p "2181:2181" -p "9092:9092" \
+		-d spotify/kafka
+endif
+ifeq ($(UNAME), Linux)
+	docker run --name kafka \
+		-e ADVERTISED_HOST=localhost \
+		-e ADVERTISED_PORT=9092 \
+		-p "2181:2181" -p "9092:9092" \
+		-d spotify/kafka
+endif
+	docker run --name mysql -p "3306:3306" -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -d mysql
+	docker run --name memcached -p "11211:11211" -d memcached
+	docker run --name postgres -p "5432:5432" -d postgres
+	docker run --name rabbitmq -p "15672:15672" -p "5672:5672" -d rabbitmq:3-management
+	docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
+	docker run --name redis -p "6379:6379" -d redis
+	docker run --name aerospike -p "3000:3000" -d aerospike
+	docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
+	docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
+	docker run --name riemann -p "5555:5555" -d blalor/riemann
+
+# Run docker containers necessary for CircleCI unit tests
+docker-run-circle:
+	docker run --name kafka \
+		-e ADVERTISED_HOST=localhost \
+		-e ADVERTISED_PORT=9092 \
+		-p "2181:2181" -p "9092:9092" \
+		-d spotify/kafka
+	docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
+	docker run --name aerospike -p "3000:3000" -d aerospike
+	docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
+	docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
+	docker run --name riemann -p "5555:5555" -d blalor/riemann
+
+# Kill all docker containers, ignore errors
+docker-kill:
+	-docker kill nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann
+	-docker rm nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann
+
+# Run full unit tests using docker containers (includes setup and teardown)
+test: docker-kill docker-run
+	# Sleeping for kafka leadership election, TSDB setup, etc.
+	sleep 60
+	# SUCCESS, running tests
+	go test -race ./...
+
+# Run "short" unit tests
+test-short:
+	go test -short ./...
+
+.PHONY: test

README.md

@@ -1,21 +1,48 @@
-# Telegraf - A native agent for InfluxDB
+# Telegraf - A native agent for InfluxDB [![Circle CI](https://circleci.com/gh/influxdb/telegraf.svg?style=svg)](https://circleci.com/gh/influxdb/telegraf)
 
-Telegraf is an agent written in Go for collecting metrics from the system it's running on or from other services and writing them into InfluxDB.
+Telegraf is an agent written in Go for collecting metrics from the system it's
+running on, or from other services, and writing them into InfluxDB.
 
-Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics from well known services (like Hadoop, or Postgres, or Redis) and third party APIs (like Mailchimp, AWS CloudWatch, or Google Analytics).
+Design goals are to have a minimal memory footprint with a plugin system so
+that developers in the community can easily add support for collecting metrics
+from well known services (like Hadoop, Postgres, or Redis) and third party
+APIs (like Mailchimp, AWS CloudWatch, or Google Analytics).
 
-We'll eagerly accept pull requests for new plugins and will manage the set of plugins that Telegraf supports. See the bottom of this doc for instructions on writing new plugins.
+We'll eagerly accept pull requests for new plugins and will manage the set of
+plugins that Telegraf supports. See the
+[contributing guide](CONTRIBUTING.md) for instructions on
+writing new plugins.
 
-## Quickstart
+## Installation:
 
-* Build from source or download telegraf:
+### Linux deb and rpm packages:
 
-### Linux packages for Debian/Ubuntu and RHEL/CentOS:
+Latest:
+* http://get.influxdb.org/telegraf/telegraf_0.2.4_amd64.deb
+* http://get.influxdb.org/telegraf/telegraf-0.2.4-1.x86_64.rpm
 
-```
-http://get.influxdb.org/telegraf/telegraf_0.1.2_amd64.deb
-http://get.influxdb.org/telegraf/telegraf-0.1.2-1.x86_64.rpm
-```
+##### Package instructions:
+
+* Telegraf binary is installed in `/opt/telegraf/telegraf`
+* Telegraf daemon configuration file is in `/etc/opt/telegraf/telegraf.conf`
+* On sysv systems, the telegraf daemon can be controlled via
+`service telegraf [action]`
+* On systemd systems (such as Ubuntu 15+), the telegraf daemon can be
+controlled via `systemctl [action] telegraf`
+
+### Linux binaries:
+
+Latest:
+* http://get.influxdb.org/telegraf/telegraf_linux_amd64_0.2.4.tar.gz
+* http://get.influxdb.org/telegraf/telegraf_linux_386_0.2.4.tar.gz
+* http://get.influxdb.org/telegraf/telegraf_linux_arm_0.2.4.tar.gz
+
+##### Binary instructions:
+
+These are standalone binaries that can be unpacked and executed on any linux
+system. They can be unpacked and renamed in a location such as
+`/usr/local/bin` for convenience. A config file will need to be generated,
+see "How to use it" below.
 
 ### OSX via Homebrew:
 
@@ -24,154 +51,121 @@ brew update
 brew install telegraf
 ```
 
+### From Source:
+
+Telegraf manages dependencies via [gdm](https://github.com/sparrc/gdm),
+which gets installed via the Makefile
+if you don't have it already. You also must build with golang version 1.4+.
+
+1. [Install Go](https://golang.org/doc/install)
+2. [Setup your GOPATH](https://golang.org/doc/code.html#GOPATH)
+3. Run `go get github.com/influxdb/telegraf`
+4. Run `cd $GOPATH/src/github.com/influxdb/telegraf`
+5. Run `make`
+
 ### How to use it:
 
-* Run `telegraf -sample-config > telegraf.toml` to create an initial configuration
-* Edit the configuration to match your needs
-* Run `telegraf -config telegraf.toml -test` to output one full measurement sample to STDOUT
-* Run `telegraf -config telegraf.toml` to gather and send metrics to InfluxDB
-
+* Run `telegraf -sample-config > telegraf.conf` to create an initial configuration.
+* Or run `telegraf -sample-config -filter cpu:mem -outputfilter influxdb > telegraf.conf`.
+to create a config file with only CPU and memory plugins defined, and InfluxDB
+output defined.
+* Edit the configuration to match your needs.
+* Run `telegraf -config telegraf.conf -test` to output one full measurement
+sample to STDOUT. NOTE: you may want to run as the telegraf user if you are using
+the linux packages `sudo -u telegraf telegraf -config telegraf.conf -test`
+* Run `telegraf -config telegraf.conf` to gather and send metrics to configured outputs.
+* Run `telegraf -config telegraf.conf -filter system:swap`.
+to run telegraf with only the system & swap plugins defined in the config.
 
 ## Telegraf Options
 
-Telegraf has a few options you can configure under the `agent` section of the config. If you don't see an `agent` section run `telegraf -sample-config > telegraf.toml` to create a valid initial configuration:
+Telegraf has a few options you can configure under the `agent` section of the
+config.
 
-* **hostname**: The hostname is passed as a tag. By default this will be the value retured by `hostname` on the machine running Telegraf. You can override that value here.
-* **interval**: How ofter to gather metrics. Uses a simple number + unit parser, ie "10s" for 10 seconds or "5m" for 5 minutes.
-* **debug**: Set to true to gather and send metrics to STDOUT as well as InfluxDB.
+* **hostname**: The hostname is passed as a tag. By default this will be
+the value returned by `hostname` on the machine running Telegraf.
+You can override that value here.
+* **interval**: How often to gather metrics. Uses a simple number +
+unit parser, e.g. "10s" for 10 seconds or "5m" for 5 minutes.
+* **debug**: Set to true to gather and send metrics to STDOUT as well as
+InfluxDB.
+
+## Configuration
+
+See the [configuration guide](CONFIGURATION.md) for a rundown of the more advanced
+configuration options.
 
 ## Supported Plugins
 
+**You can view usage instructions for each plugin by running**
+`telegraf -usage <pluginname>`.
+
 Telegraf currently has support for collecting metrics from:
 
-* System (memory, CPU, network, etc.)
-* Docker
-* MySQL
-* Prometheus (client libraries and exporters)
-* PostgreSQL
-* Redis
+* aerospike
+* apache
+* bcache
+* disque
+* elasticsearch
+* exec (generic JSON-emitting executable plugin)
+* haproxy
+* httpjson (generic JSON-emitting http service plugin)
+* influxdb
+* jolokia
+* leofs
+* lustre2
+* mailchimp
+* memcached
+* mongodb
+* mysql
+* nginx
+* phpfpm
+* ping
+* postgresql
+* procstat
+* prometheus
+* puppetagent
+* rabbitmq
+* redis
+* rethinkdb
+* twemproxy
+* zfs
+* zookeeper
+* system
+    * cpu
+    * mem
+    * net
+    * netstat
+    * disk
+    * diskio
+    * swap
 
-We'll be adding support for many more over the coming months. Read on if you want to add support for another service or third-party API.
+## Supported Service Plugins
 
-## Plugin Options
+Telegraf can collect metrics via the following services:
 
-There are 3 configuration options that are configurable per plugin:
+* statsd
+* kafka_consumer
 
-* **pass**: An array of strings that is used to filter metrics generated by the current plugin. Each string in the array is tested as a prefix against metrics and if it matches, the metric is emitted.
-* **drop**: The inverse of pass, if a metric matches, it is not emitted.
-* **interval**: How often to gather this metric. Normal plugins use a single global interval, but if one particular plugin should be run less or more often, you can configure that here.
+We'll be adding support for many more over the coming months. Read on if you
+want to add support for another service or third-party API.
 
-## Plugins
+## Supported Outputs
 
-This section is for developers that want to create new collection plugins. Telegraf is entirely plugin driven. This interface allows for operators to
-pick and chose what is gathered as well as makes it easy for developers
-to create new ways of generating metrics.
+* influxdb
+* nsq
+* kafka
+* datadog
+* opentsdb
+* amqp (rabbitmq)
+* mqtt
+* librato
+* prometheus
+* amon
+* riemann
 
-Plugin authorship is kept as simple as possible to promote people to develop
-and submit new plugins.
-
-## Guidelines
-
-* A plugin must conform to the `plugins.Plugin` interface.
-* Telegraf promises to run each plugin's Gather function serially. This means
-developers don't have to worry about thread safety within these functions.
-* Each generated metric automatically has the name of the plugin that generated
-it prepended. This is to keep plugins honest.
-* Plugins should call `plugins.Add` in their `init` function to register themselves.
-See below for a quick example.
-* To be available within Telegraf itself, plugins must add themselves to the `github.com/influxdb/telegraf/plugins/all/all.go` file.
-* The `SampleConfig` function should return valid toml that describes how the plugin can be configured. This is include in `telegraf -sample-config`.
-* The `Description` function should say in one line what this plugin does.
-
-### Plugin interface
-
-```go
-type Plugin interface {
-  SampleConfig() string
-  Description() string
-  Gather(Accumulator) error
-}
-
-type Accumulator interface {
-  Add(measurement string, value interface{}, tags map[string]string)
-  AddValuesWithTime(measurement string, values map[string]interface{}, tags map[string]string, timestamp time.Time)
-}
-```
-
-### Accumulator
-
-The way that a plugin emits metrics is by interacting with the Accumulator.
-
-The `Add` function takes 3 arguments:
-* **measurement**: A string description of the metric. For instance `bytes_read` or `faults`.
-* **value**: A value for the metric. This accepts 5 different types of value:
-  * **int**: The most common type. All int types are accepted but favor using `int64`
-  Useful for counters, etc.
-  * **float**: Favor `float64`, useful for gauges, percentages, etc.
-  * **bool**: `true` or `false`, useful to indicate the presence of a state. `light_on`, etc.
-  * **string**: Typically used to indicate a message, or some kind of freeform information.
-  * **time.Time**: Useful for indicating when a state last occurred, for instance `light_on_since`.
-* **tags**: This is a map of strings to strings to describe the where or who about the metric. For instance, the `net` plugin adds a tag named `"interface"` set to the name of the network interface, like `"eth0"`.
-
-The `AddValuesWithTime` allows multiple values for a point to be passed. The values
-used are the same type profile as **value** above. The **timestamp** argument
-allows a point to be registered as having occurred at an arbitrary time.
-
-Let's say you've written a plugin that emits metrics about processes on the current host.
-
-```go
-
-type Process struct {
-  CPUTime float64
-  MemoryBytes int64
-  PID int
-}
-
-func Gather(acc plugins.Accumulator) error {
-  for _, process := range system.Processes() {
-    tags := map[string]string {
-      "pid": fmt.Sprintf("%d", process.Pid),
-    }
-
-    acc.Add("cpu", process.CPUTime, tags)
-    acc.Add("memory", process.MemoryBytes, tags)
-  }
-}
-```
-
-### Example
-
-```go
-package simple
-
-// simple.go
-
-import "github.com/influxdb/telegraf/plugins"
-
-type Simple struct {
-  Ok bool
-}
-
-func (s *Simple) Description() string {
-  return "a demo plugin"
-}
-
-func (s *Simple) SampleConfig() string {
-  return "ok = true # indicate if everything is fine"
-}
-
-func (s *Simple) Gather(acc plugins.Accumulator) error {
-  if s.Ok {
-    acc.Add("state", "pretty good", nil)
-  } else {
-    acc.Add("state", "not great", nil)
-  }
-
-  return nil
-}
-
-func init() {
-  plugins.Add("simple", func() plugins.Plugin { return &Simple{} })
-}
-```
+## Contributing
 
+Please see the
+[contributing guide](CONTRIBUTING.md)
+for details on contributing a plugin or output to Telegraf.

Vagrantfile

@@ -1,122 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
-VAGRANTFILE_API_VERSION = "2"
-
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  # All Vagrant configuration is done here. The most common configuration
-  # options are documented and commented below. For a complete reference,
-  # please see the online documentation at vagrantup.com.
-
-  # Every Vagrant virtual environment requires a box to build off of.
-  config.vm.box = "ubuntu/trusty64"
-
-  # Disable automatic box update checking. If you disable this, then
-  # boxes will only be checked for updates when the user runs
-  # `vagrant box outdated`. This is not recommended.
-  # config.vm.box_check_update = false
-
-  # Create a forwarded port mapping which allows access to a specific port
-  # within the machine from a port on the host machine. In the example below,
-  # accessing "localhost:8080" will access port 80 on the guest machine.
-  # config.vm.network "forwarded_port", guest: 80, host: 8080
-
-  # Create a private network, which allows host-only access to the machine
-  # using a specific IP.
-  # config.vm.network "private_network", ip: "192.168.33.10"
-
-  # Create a public network, which generally matched to bridged network.
-  # Bridged networks make the machine appear as another physical device on
-  # your network.
-  # config.vm.network "public_network"
-
-  # If true, then any SSH connections made will enable agent forwarding.
-  # Default value: false
-  # config.ssh.forward_agent = true
-
-  # Share an additional folder to the guest VM. The first argument is
-  # the path on the host to the actual folder. The second argument is
-  # the path on the guest to mount the folder. And the optional third
-  # argument is a set of non-required options.
-  config.vm.synced_folder "~/go", "/home/vagrant/go"
-
-  # Provider-specific configuration so you can fine-tune various
-  # backing providers for Vagrant. These expose provider-specific options.
-  # Example for VirtualBox:
-  #
-  config.vm.provider "virtualbox" do |vb|
-  #   # Don't boot with headless mode
-  #   vb.gui = true
-  #
-  #   # Use VBoxManage to customize the VM. For example to change memory:
-    vb.customize ["modifyvm", :id, "--memory", "1024"]
-  end
-  #
-  # View the documentation for the provider you're using for more
-  # information on available options.
-
-  # Enable provisioning with CFEngine. CFEngine Community packages are
-  # automatically installed. For example, configure the host as a
-  # policy server and optionally a policy file to run:
-  #
-  # config.vm.provision "cfengine" do |cf|
-  #   cf.am_policy_hub = true
-  #   # cf.run_file = "motd.cf"
-  # end
-  #
-  # You can also configure and bootstrap a client to an existing
-  # policy server:
-  #
-  # config.vm.provision "cfengine" do |cf|
-  #   cf.policy_server_address = "10.0.2.15"
-  # end
-
-  # Enable provisioning with Puppet stand alone.  Puppet manifests
-  # are contained in a directory path relative to this Vagrantfile.
-  # You will need to create the manifests directory and a manifest in
-  # the file default.pp in the manifests_path directory.
-  #
-  # config.vm.provision "puppet" do |puppet|
-  #   puppet.manifests_path = "manifests"
-  #   puppet.manifest_file  = "site.pp"
-  # end
-
-  # Enable provisioning with chef solo, specifying a cookbooks path, roles
-  # path, and data_bags path (all relative to this Vagrantfile), and adding
-  # some recipes and/or roles.
-  #
-  # config.vm.provision "chef_solo" do |chef|
-  #   chef.cookbooks_path = "../my-recipes/cookbooks"
-  #   chef.roles_path = "../my-recipes/roles"
-  #   chef.data_bags_path = "../my-recipes/data_bags"
-  #   chef.add_recipe "mysql"
-  #   chef.add_role "web"
-  #
-  #   # You may also specify custom JSON attributes:
-  #   chef.json = { mysql_password: "foo" }
-  # end
-
-  # Enable provisioning with chef server, specifying the chef server URL,
-  # and the path to the validation key (relative to this Vagrantfile).
-  #
-  # The Opscode Platform uses HTTPS. Substitute your organization for
-  # ORGNAME in the URL and validation key.
-  #
-  # If you have your own Chef Server, use the appropriate URL, which may be
-  # HTTP instead of HTTPS depending on your configuration. Also change the
-  # validation key to validation.pem.
-  #
-  # config.vm.provision "chef_client" do |chef|
-  #   chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
-  #   chef.validation_key_path = "ORGNAME-validator.pem"
-  # end
-  #
-  # If you're using the Opscode platform, your validator client is
-  # ORGNAME-validator, replacing ORGNAME with your organization name.
-  #
-  # If you have your own Chef Server, the default validation client name is
-  # chef-validator, unless you changed the configuration.
-  #
-  #   chef.validation_client_name = "ORGNAME-validator"
-end

accumulator.go

@@ -2,99 +2,183 @@ package telegraf
 
 import (
 	"fmt"
-	"sort"
-	"strings"
+	"log"
+	"math"
 	"sync"
 	"time"
 
-	"github.com/influxdb/influxdb/client"
+	"github.com/influxdb/telegraf/internal/config"
+
+	"github.com/influxdb/influxdb/client/v2"
 )
 
-type BatchPoints struct {
-	mu sync.Mutex
+type Accumulator interface {
+	Add(measurement string, value interface{},
+		tags map[string]string, t ...time.Time)
+	AddFields(measurement string, fields map[string]interface{},
+		tags map[string]string, t ...time.Time)
 
-	client.BatchPoints
+	SetDefaultTags(tags map[string]string)
+	AddDefaultTag(key, value string)
 
-	Debug bool
+	Prefix() string
+	SetPrefix(prefix string)
 
-	Prefix string
-
-	Config *ConfiguredPlugin
+	Debug() bool
+	SetDebug(enabled bool)
 }
 
-func (bp *BatchPoints) Add(measurement string, val interface{}, tags map[string]string) {
-	bp.mu.Lock()
-	defer bp.mu.Unlock()
-
-	measurement = bp.Prefix + measurement
-
-	if bp.Config != nil {
-		if !bp.Config.ShouldPass(measurement) {
-			return
-		}
-	}
-
-	if bp.Debug {
-		var tg []string
-
-		for k, v := range tags {
-			tg = append(tg, fmt.Sprintf("%s=\"%s\"", k, v))
-		}
-
-		sort.Strings(tg)
-
-		fmt.Printf("> [%s] %s value=%v\n", strings.Join(tg, " "), measurement, val)
-	}
-
-	bp.Points = append(bp.Points, client.Point{
-		Measurement: measurement,
-		Tags:        tags,
-		Fields: map[string]interface{}{
-			"value": val,
-		},
-	})
+func NewAccumulator(
+	pluginConfig *config.PluginConfig,
+	points chan *client.Point,
+) Accumulator {
+	acc := accumulator{}
+	acc.points = points
+	acc.pluginConfig = pluginConfig
+	return &acc
 }
 
-func (bp *BatchPoints) AddValuesWithTime(
+type accumulator struct {
+	sync.Mutex
+
+	points chan *client.Point
+
+	defaultTags map[string]string
+
+	debug bool
+
+	pluginConfig *config.PluginConfig
+
+	prefix string
+}
+
+func (ac *accumulator) Add(
 	measurement string,
-	values map[string]interface{},
+	value interface{},
 	tags map[string]string,
-	timestamp time.Time,
+	t ...time.Time,
 ) {
-	bp.mu.Lock()
-	defer bp.mu.Unlock()
-
-	measurement = bp.Prefix + measurement
-
-	if bp.Config != nil {
-		if !bp.Config.ShouldPass(measurement) {
-			return
-		}
-	}
-
-	if bp.Debug {
-		var tg []string
-
-		for k, v := range tags {
-			tg = append(tg, fmt.Sprintf("%s=\"%s\"", k, v))
-		}
-
-		var vals []string
-
-		for k, v := range values {
-			vals = append(vals, fmt.Sprintf("%s=%v", k, v))
-		}
-
-		sort.Strings(tg)
-		sort.Strings(vals)
-
-		fmt.Printf("> [%s] %s %s\n", strings.Join(tg, " "), measurement, strings.Join(vals, " "))
-	}
-
-	bp.Points = append(bp.Points, client.Point{
-		Measurement: measurement,
-		Tags:        tags,
-		Fields:      values,
-		Time:        timestamp,
-	})
+	fields := make(map[string]interface{})
+	fields["value"] = value
+	ac.AddFields(measurement, fields, tags, t...)
+}
+
+func (ac *accumulator) AddFields(
+	measurement string,
+	fields map[string]interface{},
+	tags map[string]string,
+	t ...time.Time,
+) {
+	if !ac.pluginConfig.Filter.ShouldTagsPass(tags) {
+		return
+	}
+
+	// Override measurement name if set
+	if len(ac.pluginConfig.NameOverride) != 0 {
+		measurement = ac.pluginConfig.NameOverride
+	}
+	// Apply measurement prefix and suffix if set
+	if len(ac.pluginConfig.MeasurementPrefix) != 0 {
+		measurement = ac.pluginConfig.MeasurementPrefix + measurement
+	}
+	if len(ac.pluginConfig.MeasurementSuffix) != 0 {
+		measurement = measurement + ac.pluginConfig.MeasurementSuffix
+	}
+
+	if tags == nil {
+		tags = make(map[string]string)
+	}
+	// Apply plugin-wide tags if set
+	for k, v := range ac.pluginConfig.Tags {
+		if _, ok := tags[k]; !ok {
+			tags[k] = v
+		}
+	}
+	// Apply daemon-wide tags if set
+	for k, v := range ac.defaultTags {
+		if _, ok := tags[k]; !ok {
+			tags[k] = v
+		}
+	}
+
+	result := make(map[string]interface{})
+	for k, v := range fields {
+		// Filter out any filtered fields
+		if ac.pluginConfig != nil {
+			if !ac.pluginConfig.Filter.ShouldPass(k) {
+				continue
+			}
+		}
+		result[k] = v
+
+		// Validate uint64 and float64 fields
+		switch val := v.(type) {
+		case uint64:
+			// InfluxDB does not support writing uint64
+			if val < uint64(9223372036854775808) {
+				result[k] = int64(val)
+			} else {
+				result[k] = int64(9223372036854775807)
+			}
+		case float64:
+			// NaNs are invalid values in influxdb, skip measurement
+			if math.IsNaN(val) || math.IsInf(val, 0) {
+				if ac.debug {
+					log.Printf("Measurement [%s] field [%s] has a NaN or Inf "+
+						"field, skipping",
+						measurement, k)
+				}
+				continue
+			}
+		}
+	}
+	fields = nil
+	if len(result) == 0 {
+		return
+	}
+
+	var timestamp time.Time
+	if len(t) > 0 {
+		timestamp = t[0]
+	} else {
+		timestamp = time.Now()
+	}
+
+	if ac.prefix != "" {
+		measurement = ac.prefix + measurement
+	}
+
+	pt, err := client.NewPoint(measurement, tags, result, timestamp)
+	if err != nil {
+		log.Printf("Error adding point [%s]: %s\n", measurement, err.Error())
+		return
+	}
+	if ac.debug {
+		fmt.Println("> " + pt.String())
+	}
+	ac.points <- pt
+}
+
+func (ac *accumulator) SetDefaultTags(tags map[string]string) {
+	ac.defaultTags = tags
+}
+
+func (ac *accumulator) AddDefaultTag(key, value string) {
+	ac.defaultTags[key] = value
+}
+
+func (ac *accumulator) Prefix() string {
+	return ac.prefix
+}
+
+func (ac *accumulator) SetPrefix(prefix string) {
+	ac.prefix = prefix
+}
+
+func (ac *accumulator) Debug() bool {
+	return ac.debug
+}
+
+func (ac *accumulator) SetDebug(debug bool) {
+	ac.debug = debug
 }

agent.go

@@ -1,195 +1,160 @@
 package telegraf
 
 import (
+	"crypto/rand"
 	"fmt"
 	"log"
-	"net/url"
+	"math/big"
 	"os"
-	"sort"
 	"sync"
 	"time"
 
-	"github.com/influxdb/influxdb/client"
+	"github.com/influxdb/telegraf/internal/config"
+	"github.com/influxdb/telegraf/outputs"
 	"github.com/influxdb/telegraf/plugins"
+
+	"github.com/influxdb/influxdb/client/v2"
 )
 
-type runningPlugin struct {
-	name   string
-	plugin plugins.Plugin
-	config *ConfiguredPlugin
-}
-
+// Agent runs telegraf and collects data based on the given config
 type Agent struct {
-	Interval Duration
-	Debug    bool
-	Hostname string
-
-	Config *Config
-
-	plugins []*runningPlugin
-
-	conn *client.Client
+	Config *config.Config
 }
 
-func NewAgent(config *Config) (*Agent, error) {
-	agent := &Agent{Config: config, Interval: Duration{10 * time.Second}}
-
-	err := config.ApplyAgent(agent)
-	if err != nil {
-		return nil, err
+// NewAgent returns an Agent struct based off the given Config
+func NewAgent(config *config.Config) (*Agent, error) {
+	a := &Agent{
+		Config: config,
 	}
 
-	if agent.Hostname == "" {
+	if a.Config.Agent.Hostname == "" {
 		hostname, err := os.Hostname()
 		if err != nil {
 			return nil, err
 		}
 
-		agent.Hostname = hostname
+		a.Config.Agent.Hostname = hostname
 	}
 
-	if config.Tags == nil {
-		config.Tags = map[string]string{}
-	}
+	config.Tags["host"] = a.Config.Agent.Hostname
 
-	config.Tags["host"] = agent.Hostname
-
-	return agent, nil
+	return a, nil
 }
 
-func (agent *Agent) Connect() error {
-	config := agent.Config
+// Connect connects to all configured outputs
+func (a *Agent) Connect() error {
+	for _, o := range a.Config.Outputs {
+		switch ot := o.Output.(type) {
+		case outputs.ServiceOutput:
+			if err := ot.Start(); err != nil {
+				log.Printf("Service for output %s failed to start, exiting\n%s\n",
+					o.Name, err.Error())
+				return err
+			}
+		}
 
-	u, err := url.Parse(config.URL)
-	if err != nil {
-		return err
+		if a.Config.Agent.Debug {
+			log.Printf("Attempting connection to output: %s\n", o.Name)
+		}
+		err := o.Output.Connect()
+		if err != nil {
+			log.Printf("Failed to connect to output %s, retrying in 15s\n", o.Name)
+			time.Sleep(15 * time.Second)
+			err = o.Output.Connect()
+			if err != nil {
+				return err
+			}
+		}
+		if a.Config.Agent.Debug {
+			log.Printf("Successfully connected to output: %s\n", o.Name)
+		}
 	}
-
-	c, err := client.NewClient(client.Config{
-		URL:       *u,
-		Username:  config.Username,
-		Password:  config.Password,
-		UserAgent: config.UserAgent,
-	})
-
-	if err != nil {
-		return err
-	}
-
-	agent.conn = c
-
 	return nil
 }
 
-func (a *Agent) LoadPlugins() ([]string, error) {
-	var names []string
-
-	for _, name := range a.Config.PluginsDeclared() {
-		creator, ok := plugins.Plugins[name]
-		if !ok {
-			return nil, fmt.Errorf("Undefined but requested plugin: %s", name)
+// Close closes the connection to all configured outputs
+func (a *Agent) Close() error {
+	var err error
+	for _, o := range a.Config.Outputs {
+		err = o.Output.Close()
+		switch ot := o.Output.(type) {
+		case outputs.ServiceOutput:
+			ot.Stop()
 		}
-
-		plugin := creator()
-
-		config, err := a.Config.ApplyPlugin(name, plugin)
-		if err != nil {
-			return nil, err
-		}
-
-		a.plugins = append(a.plugins, &runningPlugin{name, plugin, config})
-		names = append(names, name)
 	}
-
-	sort.Strings(names)
-
-	return names, nil
+	return err
 }
 
-func (a *Agent) crankParallel() error {
-	points := make(chan *BatchPoints, len(a.plugins))
-
+// gatherParallel runs the plugins that are using the same reporting interval
+// as the telegraf agent.
+func (a *Agent) gatherParallel(pointChan chan *client.Point) error {
 	var wg sync.WaitGroup
 
-	for _, plugin := range a.plugins {
-		if plugin.config.Interval != 0 {
+	start := time.Now()
+	counter := 0
+	for _, plugin := range a.Config.Plugins {
+		if plugin.Config.Interval != 0 {
 			continue
 		}
 
 		wg.Add(1)
-		go func(plugin *runningPlugin) {
+		counter++
+		go func(plugin *config.RunningPlugin) {
 			defer wg.Done()
 
-			var acc BatchPoints
-			acc.Debug = a.Debug
-			acc.Prefix = plugin.name + "_"
-			acc.Config = plugin.config
+			acc := NewAccumulator(plugin.Config, pointChan)
+			acc.SetDebug(a.Config.Agent.Debug)
+			// acc.SetPrefix(plugin.Name + "_")
+			acc.SetDefaultTags(a.Config.Tags)
 
-			plugin.plugin.Gather(&acc)
+			if err := plugin.Plugin.Gather(acc); err != nil {
+				log.Printf("Error in plugin [%s]: %s", plugin.Name, err)
+			}
 
-			points <- &acc
 		}(plugin)
 	}
 
+	if counter == 0 {
+		return nil
+	}
+
 	wg.Wait()
 
-	close(points)
-
-	var acc BatchPoints
-	acc.Tags = a.Config.Tags
-	acc.Time = time.Now()
-	acc.Database = a.Config.Database
-
-	for sub := range points {
-		acc.Points = append(acc.Points, sub.Points...)
-	}
-
-	_, err := a.conn.Write(acc.BatchPoints)
-	return err
+	elapsed := time.Since(start)
+	log.Printf("Gathered metrics, (%s interval), from %d plugins in %s\n",
+		a.Config.Agent.Interval, counter, elapsed)
+	return nil
 }
 
-func (a *Agent) crank() error {
-	var acc BatchPoints
-
-	acc.Debug = a.Debug
-
-	for _, plugin := range a.plugins {
-		acc.Prefix = plugin.name + "_"
-		acc.Config = plugin.config
-		err := plugin.plugin.Gather(&acc)
-		if err != nil {
-			return err
-		}
-	}
-
-	acc.Tags = a.Config.Tags
-	acc.Time = time.Now()
-	acc.Database = a.Config.Database
-
-	_, err := a.conn.Write(acc.BatchPoints)
-	return err
-}
-
-func (a *Agent) crankSeparate(shutdown chan struct{}, plugin *runningPlugin) error {
-	ticker := time.NewTicker(plugin.config.Interval)
+// gatherSeparate runs the plugins that have been configured with their own
+// reporting interval.
+func (a *Agent) gatherSeparate(
+	shutdown chan struct{},
+	plugin *config.RunningPlugin,
+	pointChan chan *client.Point,
+) error {
+	ticker := time.NewTicker(plugin.Config.Interval)
 
 	for {
-		var acc BatchPoints
+		var outerr error
+		start := time.Now()
 
-		acc.Debug = a.Debug
+		acc := NewAccumulator(plugin.Config, pointChan)
+		acc.SetDebug(a.Config.Agent.Debug)
+		// acc.SetPrefix(plugin.Name + "_")
+		acc.SetDefaultTags(a.Config.Tags)
 
-		acc.Prefix = plugin.name + "_"
-		acc.Config = plugin.config
-		err := plugin.plugin.Gather(&acc)
-		if err != nil {
-			return err
+		if err := plugin.Plugin.Gather(acc); err != nil {
+			log.Printf("Error in plugin [%s]: %s", plugin.Name, err)
 		}
 
-		acc.Tags = a.Config.Tags
-		acc.Time = time.Now()
-		acc.Database = a.Config.Database
+		elapsed := time.Since(start)
+		log.Printf("Gathered metrics, (separate %s interval), from %s in %s\n",
+			plugin.Config.Interval, plugin.Name, elapsed)
 
-		a.conn.Write(acc.BatchPoints)
+		if outerr != nil {
+			return outerr
+		}
 
 		select {
 		case <-shutdown:
@@ -200,86 +165,226 @@ func (a *Agent) crankSeparate(shutdown chan struct{}, plugin *runningPlugin) err
 	}
 }
 
-func (a *Agent) TestAllPlugins() error {
-	var names []string
-
-	for name, _ := range plugins.Plugins {
-		names = append(names, name)
-	}
-
-	sort.Strings(names)
-
-	var acc BatchPoints
-	acc.Debug = true
-
-	fmt.Printf("* Testing all plugins with default configuration\n")
-
-	for _, name := range names {
-		plugin := plugins.Plugins[name]()
-
-		fmt.Printf("* Plugin: %s\n", name)
-
-		acc.Prefix = name + "_"
-		err := plugin.Gather(&acc)
-		if err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
+// Test verifies that we can 'Gather' from all plugins with their configured
+// Config struct
 func (a *Agent) Test() error {
-	var acc BatchPoints
+	shutdown := make(chan struct{})
+	defer close(shutdown)
+	pointChan := make(chan *client.Point)
 
-	acc.Debug = true
+	// dummy receiver for the point channel
+	go func() {
+		for {
+			select {
+			case <-pointChan:
+				// do nothing
+			case <-shutdown:
+				return
+			}
+		}
+	}()
 
-	for _, plugin := range a.plugins {
-		acc.Prefix = plugin.name + "_"
-		acc.Config = plugin.config
+	for _, plugin := range a.Config.Plugins {
+		acc := NewAccumulator(plugin.Config, pointChan)
+		acc.SetDebug(true)
+		// acc.SetPrefix(plugin.Name + "_")
 
-		fmt.Printf("* Plugin: %s\n", plugin.name)
-		if plugin.config.Interval != 0 {
-			fmt.Printf("* Internal: %s\n", plugin.config.Interval)
+		fmt.Printf("* Plugin: %s, Collection 1\n", plugin.Name)
+		if plugin.Config.Interval != 0 {
+			fmt.Printf("* Internal: %s\n", plugin.Config.Interval)
 		}
 
-		err := plugin.plugin.Gather(&acc)
-		if err != nil {
+		if err := plugin.Plugin.Gather(acc); err != nil {
 			return err
 		}
-	}
 
+		// Special instructions for some plugins. cpu, for example, needs to be
+		// run twice in order to return cpu usage percentages.
+		switch plugin.Name {
+		case "cpu", "mongodb":
+			time.Sleep(500 * time.Millisecond)
+			fmt.Printf("* Plugin: %s, Collection 2\n", plugin.Name)
+			if err := plugin.Plugin.Gather(acc); err != nil {
+				return err
+			}
+		}
+
+	}
 	return nil
 }
 
-func (a *Agent) Run(shutdown chan struct{}) error {
-	if a.conn == nil {
-		err := a.Connect()
-		if err != nil {
-			return err
+// writeOutput writes a list of points to a single output, with retries.
+// Optionally takes a `done` channel to indicate that it is done writing.
+func (a *Agent) writeOutput(
+	points []*client.Point,
+	ro *config.RunningOutput,
+	shutdown chan struct{},
+	wg *sync.WaitGroup,
+) {
+	defer wg.Done()
+	if len(points) == 0 {
+		return
+	}
+	retry := 0
+	retries := a.Config.Agent.FlushRetries
+	start := time.Now()
+
+	for {
+		filtered := ro.FilterPoints(points)
+		err := ro.Output.Write(filtered)
+		if err == nil {
+			// Write successful
+			elapsed := time.Since(start)
+			log.Printf("Flushed %d metrics to output %s in %s\n",
+				len(filtered), ro.Name, elapsed)
+			return
+		}
+
+		select {
+		case <-shutdown:
+			return
+		default:
+			if retry >= retries {
+				// No more retries
+				msg := "FATAL: Write to output [%s] failed %d times, dropping" +
+					" %d metrics\n"
+				log.Printf(msg, ro.Name, retries+1, len(points))
+				return
+			} else if err != nil {
+				// Sleep for a retry
+				log.Printf("Error in output [%s]: %s, retrying in %s",
+					ro.Name, err.Error(), a.Config.Agent.FlushInterval.Duration)
+				time.Sleep(a.Config.Agent.FlushInterval.Duration)
+			}
+		}
+
+		retry++
+	}
+}
+
+// flush writes a list of points to all configured outputs
+func (a *Agent) flush(
+	points []*client.Point,
+	shutdown chan struct{},
+	wait bool,
+) {
+	var wg sync.WaitGroup
+	for _, o := range a.Config.Outputs {
+		wg.Add(1)
+		go a.writeOutput(points, o, shutdown, &wg)
+	}
+	if wait {
+		wg.Wait()
+	}
+}
+
+// flusher monitors the points input channel and flushes on the minimum interval
+func (a *Agent) flusher(shutdown chan struct{}, pointChan chan *client.Point) error {
+	// Inelegant, but this sleep is to allow the Gather threads to run, so that
+	// the flusher will flush after metrics are collected.
+	time.Sleep(time.Millisecond * 100)
+
+	ticker := time.NewTicker(a.Config.Agent.FlushInterval.Duration)
+	points := make([]*client.Point, 0)
+
+	for {
+		select {
+		case <-shutdown:
+			log.Println("Hang on, flushing any cached points before shutdown")
+			a.flush(points, shutdown, true)
+			return nil
+		case <-ticker.C:
+			a.flush(points, shutdown, false)
+			points = make([]*client.Point, 0)
+		case pt := <-pointChan:
+			points = append(points, pt)
 		}
 	}
+}
 
+// jitterInterval applies the the interval jitter to the flush interval using
+// crypto/rand number generator
+func jitterInterval(ininterval, injitter time.Duration) time.Duration {
+	var jitter int64
+	outinterval := ininterval
+	if injitter.Nanoseconds() != 0 {
+		maxjitter := big.NewInt(injitter.Nanoseconds())
+		if j, err := rand.Int(rand.Reader, maxjitter); err == nil {
+			jitter = j.Int64()
+		}
+		outinterval = time.Duration(jitter + ininterval.Nanoseconds())
+	}
+
+	if outinterval.Nanoseconds() < time.Duration(500*time.Millisecond).Nanoseconds() {
+		log.Printf("Flush interval %s too low, setting to 500ms\n", outinterval)
+		outinterval = time.Duration(500 * time.Millisecond)
+	}
+
+	return outinterval
+}
+
+// Run runs the agent daemon, gathering every Interval
+func (a *Agent) Run(shutdown chan struct{}) error {
 	var wg sync.WaitGroup
 
-	for _, plugin := range a.plugins {
-		if plugin.config.Interval != 0 {
+	a.Config.Agent.FlushInterval.Duration = jitterInterval(a.Config.Agent.FlushInterval.Duration,
+		a.Config.Agent.FlushJitter.Duration)
+
+	log.Printf("Agent Config: Interval:%s, Debug:%#v, Hostname:%#v, "+
+		"Flush Interval:%s\n",
+		a.Config.Agent.Interval, a.Config.Agent.Debug,
+		a.Config.Agent.Hostname, a.Config.Agent.FlushInterval)
+
+	// channel shared between all plugin threads for accumulating points
+	pointChan := make(chan *client.Point, 1000)
+
+	// Round collection to nearest interval by sleeping
+	if a.Config.Agent.RoundInterval {
+		i := int64(a.Config.Agent.Interval.Duration)
+		time.Sleep(time.Duration(i - (time.Now().UnixNano() % i)))
+	}
+	ticker := time.NewTicker(a.Config.Agent.Interval.Duration)
+
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		if err := a.flusher(shutdown, pointChan); err != nil {
+			log.Printf("Flusher routine failed, exiting: %s\n", err.Error())
+			close(shutdown)
+		}
+	}()
+
+	for _, plugin := range a.Config.Plugins {
+
+		// Start service of any ServicePlugins
+		switch p := plugin.Plugin.(type) {
+		case plugins.ServicePlugin:
+			if err := p.Start(); err != nil {
+				log.Printf("Service for plugin %s failed to start, exiting\n%s\n",
+					plugin.Name, err.Error())
+				return err
+			}
+			defer p.Stop()
+		}
+
+		// Special handling for plugins that have their own collection interval
+		// configured. Default intervals are handled below with gatherParallel
+		if plugin.Config.Interval != 0 {
 			wg.Add(1)
-			go func(plugin *runningPlugin) {
+			go func(plugin *config.RunningPlugin) {
 				defer wg.Done()
-				a.crankSeparate(shutdown, plugin)
+				if err := a.gatherSeparate(shutdown, plugin, pointChan); err != nil {
+					log.Printf(err.Error())
+				}
 			}(plugin)
 		}
 	}
 
 	defer wg.Wait()
 
-	ticker := time.NewTicker(a.Interval.Duration)
-
 	for {
-		err := a.crankParallel()
-		if err != nil {
-			log.Printf("Error in plugins: %s", err)
+		if err := a.gatherParallel(pointChan); err != nil {
+			log.Printf(err.Error())
 		}
 
 		select {

agent_test.go

@@ -1,61 +1,156 @@
 package telegraf
 
-/*
-func TestAgent_DrivesMetrics(t *testing.T) {
-	var (
-		plugin  plugins.MockPlugin
-	)
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+	"time"
 
-	defer plugin.AssertExpectations(t)
-	defer metrics.AssertExpectations(t)
+	"github.com/influxdb/telegraf/internal/config"
 
-	a := &Agent{
-		plugins: []plugins.Plugin{&plugin},
-		Config:  &Config{},
-	}
+	// needing to load the plugins
+	_ "github.com/influxdb/telegraf/plugins/all"
+	// needing to load the outputs
+	_ "github.com/influxdb/telegraf/outputs/all"
+)
 
-	plugin.On("Add", "foo", 1.2, nil).Return(nil)
-	plugin.On("Add", "bar", 888, nil).Return(nil)
+func TestAgent_LoadPlugin(t *testing.T) {
+	c := config.NewConfig()
+	c.PluginFilters = []string{"mysql"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ := NewAgent(c)
+	assert.Equal(t, 1, len(a.Config.Plugins))
 
-	err := a.crank()
-	require.NoError(t, err)
+	c = config.NewConfig()
+	c.PluginFilters = []string{"foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 0, len(a.Config.Plugins))
+
+	c = config.NewConfig()
+	c.PluginFilters = []string{"mysql", "foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 1, len(a.Config.Plugins))
+
+	c = config.NewConfig()
+	c.PluginFilters = []string{"mysql", "redis"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Plugins))
+
+	c = config.NewConfig()
+	c.PluginFilters = []string{"mysql", "foo", "redis", "bar"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Plugins))
 }
 
-func TestAgent_AppliesTags(t *testing.T) {
-	var (
-		plugin  plugins.MockPlugin
-		metrics MockMetrics
-	)
+func TestAgent_LoadOutput(t *testing.T) {
+	c := config.NewConfig()
+	c.OutputFilters = []string{"influxdb"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ := NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Outputs))
 
-	defer plugin.AssertExpectations(t)
-	defer metrics.AssertExpectations(t)
+	c = config.NewConfig()
+	c.OutputFilters = []string{}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 3, len(a.Config.Outputs))
 
-	a := &Agent{
-		plugins: []plugins.Plugin{&plugin},
-		metrics: &metrics,
-		Config: &Config{
-			Tags: map[string]string{
-				"dc": "us-west-1",
-			},
-		},
-	}
+	c = config.NewConfig()
+	c.OutputFilters = []string{"foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 0, len(a.Config.Outputs))
 
-	m1 := cypress.Metric()
-	m1.Add("name", "foo")
-	m1.Add("value", 1.2)
+	c = config.NewConfig()
+	c.OutputFilters = []string{"influxdb", "foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Outputs))
 
-	msgs := []*cypress.Message{m1}
+	c = config.NewConfig()
+	c.OutputFilters = []string{"influxdb", "kafka"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 3, len(a.Config.Outputs))
 
-	m2 := cypress.Metric()
-	m2.Timestamp = m1.Timestamp
-	m2.Add("name", "foo")
-	m2.Add("value", 1.2)
-	m2.AddTag("dc", "us-west-1")
-
-	plugin.On("Read").Return(msgs, nil)
-	metrics.On("Receive", m2).Return(nil)
-
-	err := a.crank()
-	require.NoError(t, err)
+	c = config.NewConfig()
+	c.OutputFilters = []string{"influxdb", "foo", "kafka", "bar"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 3, len(a.Config.Outputs))
+}
+
+func TestAgent_ZeroJitter(t *testing.T) {
+	flushinterval := jitterInterval(time.Duration(10*time.Second),
+		time.Duration(0*time.Second))
+
+	actual := flushinterval.Nanoseconds()
+	exp := time.Duration(10 * time.Second).Nanoseconds()
+
+	if actual != exp {
+		t.Errorf("Actual %v, expected %v", actual, exp)
+	}
+}
+
+func TestAgent_ZeroInterval(t *testing.T) {
+	min := time.Duration(500 * time.Millisecond).Nanoseconds()
+	max := time.Duration(5 * time.Second).Nanoseconds()
+
+	for i := 0; i < 1000; i++ {
+		flushinterval := jitterInterval(time.Duration(0*time.Second),
+			time.Duration(5*time.Second))
+		actual := flushinterval.Nanoseconds()
+
+		if actual > max {
+			t.Errorf("Didn't expect interval %d to be > %d", actual, max)
+			break
+		}
+		if actual < min {
+			t.Errorf("Didn't expect interval %d to be < %d", actual, min)
+			break
+		}
+	}
+}
+
+func TestAgent_ZeroBoth(t *testing.T) {
+	flushinterval := jitterInterval(time.Duration(0*time.Second),
+		time.Duration(0*time.Second))
+
+	actual := flushinterval
+	exp := time.Duration(500 * time.Millisecond)
+
+	if actual != exp {
+		t.Errorf("Actual %v, expected %v", actual, exp)
+	}
+}
+
+func TestAgent_JitterMax(t *testing.T) {
+	max := time.Duration(32 * time.Second).Nanoseconds()
+
+	for i := 0; i < 1000; i++ {
+		flushinterval := jitterInterval(time.Duration(30*time.Second),
+			time.Duration(2*time.Second))
+		actual := flushinterval.Nanoseconds()
+		if actual > max {
+			t.Errorf("Didn't expect interval %d to be > %d", actual, max)
+			break
+		}
+	}
+}
+
+func TestAgent_JitterMin(t *testing.T) {
+	min := time.Duration(30 * time.Second).Nanoseconds()
+
+	for i := 0; i < 1000; i++ {
+		flushinterval := jitterInterval(time.Duration(30*time.Second),
+			time.Duration(2*time.Second))
+		actual := flushinterval.Nanoseconds()
+		if actual < min {
+			t.Errorf("Didn't expect interval %d to be < %d", actual, min)
+			break
+		}
+	}
 }
-*/

circle.yml

@@ -0,0 +1,20 @@
+machine:
+  services:
+    - docker
+  post:
+    - sudo service zookeeper stop
+    - go version
+    - go version | grep 1.5.1 || sudo rm -rf /usr/local/go
+    - wget https://storage.googleapis.com/golang/go1.5.1.linux-amd64.tar.gz
+    - sudo tar -C /usr/local -xzf go1.5.1.linux-amd64.tar.gz
+    - go version
+
+dependencies:
+  cache_directories:
+    - "~/telegraf-build/src"
+  override:
+    - docker info
+
+test:
+  override:
+    - bash scripts/circle-test.sh

cmd/telegraf/telegraf.go

@@ -9,71 +9,113 @@ import (
 	"strings"
 
 	"github.com/influxdb/telegraf"
+	"github.com/influxdb/telegraf/internal/config"
+	_ "github.com/influxdb/telegraf/outputs/all"
 	_ "github.com/influxdb/telegraf/plugins/all"
 )
 
-var fDebug = flag.Bool("debug", false, "show metrics as they're generated to stdout")
+var fDebug = flag.Bool("debug", false,
+	"show metrics as they're generated to stdout")
 var fTest = flag.Bool("test", false, "gather metrics, print them out, and exit")
 var fConfig = flag.String("config", "", "configuration file to load")
+var fConfigDirectory = flag.String("configdirectory", "",
+	"directory containing additional *.conf files")
 var fVersion = flag.Bool("version", false, "display the version")
-var fSampleConfig = flag.Bool("sample-config", false, "print out full sample configuration")
+var fSampleConfig = flag.Bool("sample-config", false,
+	"print out full sample configuration")
 var fPidfile = flag.String("pidfile", "", "file to write our pid to")
+var fPLuginFilters = flag.String("filter", "",
+	"filter the plugins to enable, separator is :")
+var fOutputFilters = flag.String("outputfilter", "",
+	"filter the outputs to enable, separator is :")
+var fUsage = flag.String("usage", "",
+	"print usage for a plugin, ie, 'telegraf -usage mysql'")
 
-var Version = "unreleased"
-var Commit = ""
+// Telegraf version
+//	-ldflags "-X main.Version=`git describe --always --tags`"
+var Version string
 
 func main() {
 	flag.Parse()
 
+	var pluginFilters []string
+	if *fPLuginFilters != "" {
+		pluginsFilter := strings.TrimSpace(*fPLuginFilters)
+		pluginFilters = strings.Split(":"+pluginsFilter+":", ":")
+	}
+
+	var outputFilters []string
+	if *fOutputFilters != "" {
+		outputFilter := strings.TrimSpace(*fOutputFilters)
+		outputFilters = strings.Split(":"+outputFilter+":", ":")
+	}
+
 	if *fVersion {
-		fmt.Printf("InfluxDB Telegraf agent - Version %s\n", Version)
+		v := fmt.Sprintf("Telegraf - Version %s", Version)
+		fmt.Println(v)
 		return
 	}
 
 	if *fSampleConfig {
-		telegraf.PrintSampleConfig()
+		config.PrintSampleConfig(pluginFilters, outputFilters)
+		return
+	}
+
+	if *fUsage != "" {
+		if err := config.PrintPluginConfig(*fUsage); err != nil {
+			if err2 := config.PrintOutputConfig(*fUsage); err2 != nil {
+				log.Fatalf("%s and %s", err, err2)
+			}
+		}
 		return
 	}
 
 	var (
-		config *telegraf.Config
-		err    error
+		c   *config.Config
+		err error
 	)
 
 	if *fConfig != "" {
-		config, err = telegraf.LoadConfig(*fConfig)
+		c = config.NewConfig()
+		c.OutputFilters = outputFilters
+		c.PluginFilters = pluginFilters
+		err = c.LoadConfig(*fConfig)
 		if err != nil {
 			log.Fatal(err)
 		}
 	} else {
-		config = telegraf.DefaultConfig()
+		fmt.Println("Usage: Telegraf")
+		flag.PrintDefaults()
+		return
 	}
 
-	ag, err := telegraf.NewAgent(config)
+	if *fConfigDirectory != "" {
+		err = c.LoadDirectory(*fConfigDirectory)
+		if err != nil {
+			log.Fatal(err)
+		}
+	}
+	if len(c.Outputs) == 0 {
+		log.Fatalf("Error: no outputs found, did you provide a valid config file?")
+	}
+	if len(c.Plugins) == 0 {
+		log.Fatalf("Error: no plugins found, did you provide a valid config file?")
+	}
+
+	ag, err := telegraf.NewAgent(c)
 	if err != nil {
 		log.Fatal(err)
 	}
 
 	if *fDebug {
-		ag.Debug = true
-	}
-
-	plugins, err := ag.LoadPlugins()
-	if err != nil {
-		log.Fatal(err)
+		ag.Config.Agent.Debug = true
 	}
 
 	if *fTest {
-		if *fConfig != "" {
-			err = ag.Test()
-		} else {
-			err = ag.TestAllPlugins()
-		}
-
+		err = ag.Test()
 		if err != nil {
 			log.Fatal(err)
 		}
-
 		return
 	}
 
@@ -83,28 +125,17 @@ func main() {
 	}
 
 	shutdown := make(chan struct{})
-
 	signals := make(chan os.Signal)
-
 	signal.Notify(signals, os.Interrupt)
-
 	go func() {
 		<-signals
 		close(shutdown)
 	}()
 
-	log.Print("InfluxDB Agent running")
-	log.Printf("Loaded plugins: %s", strings.Join(plugins, " "))
-	if ag.Debug {
-		log.Printf("Debug: enabled")
-		log.Printf("Agent Config: Interval:%s, Debug:%#v, Hostname:%#v\n",
-			ag.Interval, ag.Debug, ag.Hostname)
-	}
-
-	if config.URL != "" {
-		log.Printf("Sending metrics to: %s", config.URL)
-		log.Printf("Tags enabled: %v", config.ListTags())
-	}
+	log.Printf("Starting Telegraf (version %s)\n", Version)
+	log.Printf("Loaded outputs: %s", strings.Join(c.OutputNames(), " "))
+	log.Printf("Loaded plugins: %s", strings.Join(c.PluginNames(), " "))
+	log.Printf("Tags enabled: %s", c.ListTags())
 
 	if *fPidfile != "" {
 		f, err := os.Create(*fPidfile)

config.go

@@ -1,302 +0,0 @@
-package telegraf
-
-import (
-	"errors"
-	"fmt"
-	"io/ioutil"
-	"sort"
-	"strings"
-	"time"
-
-	"github.com/influxdb/telegraf/plugins"
-	"github.com/naoina/toml"
-	"github.com/naoina/toml/ast"
-)
-
-type Duration struct {
-	time.Duration
-}
-
-func (d *Duration) UnmarshalTOML(b []byte) error {
-	dur, err := time.ParseDuration(string(b[1 : len(b)-1]))
-	if err != nil {
-		return err
-	}
-
-	d.Duration = dur
-
-	return nil
-}
-
-type Config struct {
-	URL       string
-	Username  string
-	Password  string
-	Database  string
-	UserAgent string
-	Tags      map[string]string
-
-	agent   *ast.Table
-	plugins map[string]*ast.Table
-}
-
-func (c *Config) Plugins() map[string]*ast.Table {
-	return c.plugins
-}
-
-type ConfiguredPlugin struct {
-	Name string
-
-	Drop []string
-	Pass []string
-
-	Interval time.Duration
-}
-
-func (cp *ConfiguredPlugin) ShouldPass(measurement string) bool {
-	if cp.Pass != nil {
-		for _, pat := range cp.Pass {
-			if strings.HasPrefix(measurement, pat) {
-				return true
-			}
-		}
-
-		return false
-	}
-
-	if cp.Drop != nil {
-		for _, pat := range cp.Drop {
-			if strings.HasPrefix(measurement, pat) {
-				return false
-			}
-		}
-
-		return true
-	}
-
-	return true
-}
-
-func (c *Config) ApplyAgent(v interface{}) error {
-	if c.agent != nil {
-		return toml.UnmarshalTable(c.agent, v)
-	}
-
-	return nil
-}
-
-func (c *Config) ApplyPlugin(name string, v interface{}) (*ConfiguredPlugin, error) {
-	cp := &ConfiguredPlugin{Name: name}
-
-	if tbl, ok := c.plugins[name]; ok {
-
-		if node, ok := tbl.Fields["pass"]; ok {
-			if kv, ok := node.(*ast.KeyValue); ok {
-				if ary, ok := kv.Value.(*ast.Array); ok {
-					for _, elem := range ary.Value {
-						if str, ok := elem.(*ast.String); ok {
-							cp.Pass = append(cp.Pass, str.Value)
-						}
-					}
-				}
-			}
-		}
-
-		if node, ok := tbl.Fields["drop"]; ok {
-			if kv, ok := node.(*ast.KeyValue); ok {
-				if ary, ok := kv.Value.(*ast.Array); ok {
-					for _, elem := range ary.Value {
-						if str, ok := elem.(*ast.String); ok {
-							cp.Drop = append(cp.Drop, str.Value)
-						}
-					}
-				}
-			}
-		}
-
-		if node, ok := tbl.Fields["interval"]; ok {
-			if kv, ok := node.(*ast.KeyValue); ok {
-				if str, ok := kv.Value.(*ast.String); ok {
-					dur, err := time.ParseDuration(str.Value)
-					if err != nil {
-						return nil, err
-					}
-
-					cp.Interval = dur
-				}
-			}
-		}
-
-		delete(tbl.Fields, "drop")
-		delete(tbl.Fields, "pass")
-		delete(tbl.Fields, "interval")
-		return cp, toml.UnmarshalTable(tbl, v)
-	}
-
-	return cp, nil
-}
-
-func (c *Config) PluginsDeclared() []string {
-	var plugins []string
-
-	for name, _ := range c.plugins {
-		plugins = append(plugins, name)
-	}
-
-	sort.Strings(plugins)
-
-	return plugins
-}
-
-func DefaultConfig() *Config {
-	return &Config{}
-}
-
-var ErrInvalidConfig = errors.New("invalid configuration")
-
-func LoadConfig(path string) (*Config, error) {
-	data, err := ioutil.ReadFile(path)
-	if err != nil {
-		return nil, err
-	}
-
-	tbl, err := toml.Parse(data)
-	if err != nil {
-		return nil, err
-	}
-
-	c := &Config{
-		plugins: make(map[string]*ast.Table),
-	}
-
-	for name, val := range tbl.Fields {
-		subtbl, ok := val.(*ast.Table)
-		if !ok {
-			return nil, ErrInvalidConfig
-		}
-
-		switch name {
-		case "influxdb":
-			err := toml.UnmarshalTable(subtbl, c)
-			if err != nil {
-				return nil, err
-			}
-		case "agent":
-			c.agent = subtbl
-		default:
-			c.plugins[name] = subtbl
-		}
-	}
-
-	return c, nil
-}
-
-func (c *Config) ListTags() string {
-	var tags []string
-
-	for k, v := range c.Tags {
-		tags = append(tags, fmt.Sprintf("%s=%s", k, v))
-	}
-
-	sort.Strings(tags)
-
-	return strings.Join(tags, " ")
-}
-
-type hasConfig interface {
-	BasicConfig() string
-}
-
-type hasDescr interface {
-	Description() string
-}
-
-var header = `# Telegraf configuration
-
-# If this file is missing an [agent] section, you must first generate a
-# valid config with 'telegraf -sample-config > telegraf.toml'
-
-# Telegraf is entirely plugin driven. All metrics are gathered from the
-# declared plugins.
-
-# Even if a plugin has no configuration, it must be declared in here
-# to be active. Declaring a plugin means just specifying the name
-# as a section with no variables. To deactivate a plugin, comment
-# out the name and any variables.
-
-# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
-# file would generate.
-
-# One rule that plugins conform to is wherever a connection string
-# can be passed, the values '' and 'localhost' are treated specially.
-# They indicate to the plugin to use their own builtin configuration to
-# connect to the local system.
-
-# NOTE: The configuration has a few required parameters. They are marked
-# with 'required'. Be sure to edit those to make this configuration work.
-
-# Configuration for influxdb server to send metrics to
-[influxdb]
-# The full HTTP endpoint URL for your InfluxDB instance
-url = "http://localhost:8086" # required.
-
-# The target database for metrics. This database must already exist
-database = "telegraf" # required.
-
-# username = "telegraf"
-# password = "metricsmetricsmetricsmetrics"
-
-# Set the user agent for the POSTs (can be useful for log differentiation)
-# user_agent = "telegraf"
-# tags = { "dc": "us-east-1" }
-
-# Tags can also be specified via a normal map, but only one form at a time:
-
-# [influxdb.tags]
-# dc = "us-east-1"
-
-# Configuration for telegraf itself
-# [agent]
-# interval = "10s"
-# debug = false
-# hostname = "prod3241"
-
-# PLUGINS
-
-`
-
-func PrintSampleConfig() {
-	fmt.Printf(header)
-
-	var names []string
-
-	for name, _ := range plugins.Plugins {
-		names = append(names, name)
-	}
-
-	sort.Strings(names)
-
-	for _, name := range names {
-		creator := plugins.Plugins[name]
-
-		plugin := creator()
-
-		fmt.Printf("# %s\n[%s]\n", plugin.Description(), name)
-
-		var config string
-
-		config = strings.TrimSpace(plugin.SampleConfig())
-
-		if config == "" {
-			fmt.Printf("  # no configuration\n\n")
-		} else {
-			fmt.Printf("\n")
-			lines := strings.Split(config, "\n")
-			for _, line := range lines {
-				fmt.Printf("%s\n", line)
-			}
-
-			fmt.Printf("\n")
-		}
-	}
-}

etc/config.sample.toml

@@ -1,134 +0,0 @@
-# Telegraf configuration
-
-# If this file is missing an [agent] section, you must first generate a
-# valid config with 'telegraf -sample-config > telegraf.toml'
-
-# Telegraf is entirely plugin driven. All metrics are gathered from the
-# declared plugins.
-
-# Even if a plugin has no configuration, it must be declared in here
-# to be active. Declaring a plugin means just specifying the name
-# as a section with no variables. To deactivate a plugin, comment
-# out the name and any variables.
-
-# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
-# file would generate.
-
-# One rule that plugins conform to is wherever a connection string
-# can be passed, the values '' and 'localhost' are treated specially.
-# They indicate to the plugin to use their own builtin configuration to
-# connect to the local system.
-
-# NOTE: The configuration has a few required parameters. They are marked
-# with 'required'. Be sure to edit those to make this configuration work.
-
-# Configuration for influxdb server to send metrics to
-[influxdb]
-# The full HTTP endpoint URL for your InfluxDB instance
-url = "http://localhost:8086" # required.
-
-# The target database for metrics. This database must already exist
-database = "telegraf" # required.
-
-# username = "telegraf"
-# password = "metricsmetricsmetricsmetrics"
-
-# Set the user agent for the POSTs (can be useful for log differentiation)
-# user_agent = "telegraf"
-# tags = { "dc": "us-east-1" }
-
-# Tags can also be specified via a normal map, but only one form at a time:
-
-# [influxdb.tags]
-# dc = "us-east-1"
-
-# Configuration for telegraf itself
-# [agent]
-# interval = "10s"
-# debug = false
-# hostname = "prod3241"
-
-# PLUGINS
-
-# Read metrics about cpu usage
-[cpu]
-  # no configuration
-
-# Read metrics about disk usage by mount point
-[disk]
-  # no configuration
-
-# Read metrics about docker containers
-[docker]
-  # no configuration
-
-# Read metrics about disk IO by device
-[io]
-  # no configuration
-
-# Read metrics about memory usage
-[mem]
-  # no configuration
-
-# Read metrics from one or many mysql servers
-[mysql]
-
-# specify servers via a url matching:
-#  [username[:password]@][protocol[(address)]]/[?tls=[true|false|skip-verify]]
-#  e.g. root:root@http://10.0.0.18/?tls=false
-#
-# If no servers are specified, then localhost is used as the host.
-servers = ["localhost"]
-
-# Read metrics about network interface usage
-[net]
-
-# By default, telegraf gathers stats from any up interface (excluding loopback)
-# Setting interfaces will tell it to gather these explicit interfaces,
-# regardless of status.
-#
-# interfaces = ["eth0", ... ]
-
-# Read metrics from one or many postgresql servers
-[postgresql]
-
-# specify servers via an array of tables
-[[postgresql.servers]]
-
-# specify address via a url matching:
-#   postgres://[pqgotest[:password]]@localhost?sslmode=[disable|verify-ca|verify-full]
-# or a simple string:
-#   host=localhost user=pqotest password=... sslmode=...
-# 
-# All connection parameters are optional. By default, the host is localhost
-# and the user is the currently running user. For localhost, we default
-# to sslmode=disable as well.
-# 
-
-address = "sslmode=disable"
-
-# A list of databases to pull metrics about. If not specified, metrics for all
-# databases are gathered.
-
-# databases = ["app_production", "blah_testing"]
-
-# [[postgresql.servers]]
-# address = "influx@remoteserver"
-
-# Read metrics from one or many redis servers
-[redis]
-
-# An array of address to gather stats about. Specify an ip on hostname
-# with optional port. ie localhost, 10.10.3.33:18832, etc.
-#
-# If no servers are specified, then localhost is used as the host.
-servers = ["localhost"]
-
-# Read metrics about swap memory usage
-[swap]
-  # no configuration
-
-# Read metrics about system load
-[system]
-  # no configuration
-

etc/logrotate.d/telegraf

@@ -0,0 +1,11 @@
+/var/log/telegraf/telegraf.log 
+{
+    rotate 6
+    daily
+    missingok
+    dateext
+    copytruncate
+    notifempty
+    compress
+}
+

etc/telegraf.conf

@@ -0,0 +1,123 @@
+# Telegraf configuration
+
+# Telegraf is entirely plugin driven. All metrics are gathered from the
+# declared plugins.
+
+# Even if a plugin has no configuration, it must be declared in here
+# to be active. Declaring a plugin means just specifying the name
+# as a section with no variables. To deactivate a plugin, comment
+# out the name and any variables.
+
+# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
+# file would generate.
+
+# One rule that plugins conform to is wherever a connection string
+# can be passed, the values '' and 'localhost' are treated specially.
+# They indicate to the plugin to use their own builtin configuration to
+# connect to the local system.
+
+# NOTE: The configuration has a few required parameters. They are marked
+# with 'required'. Be sure to edit those to make this configuration work.
+
+# Tags can also be specified via a normal map, but only one form at a time:
+[tags]
+  # dc = "us-east-1"
+
+# Configuration for telegraf agent
+[agent]
+  # Default data collection interval for all plugins
+  interval = "10s"
+  # Rounds collection interval to 'interval'
+  # ie, if interval="10s" then always collect on :00, :10, :20, etc.
+  round_interval = true
+
+  # Default data flushing interval for all outputs. You should not set this below
+  # interval. Maximum flush_interval will be flush_interval + flush_jitter
+  flush_interval = "10s"
+  # Jitter the flush interval by a random amount. This is primarily to avoid
+  # large write spikes for users running a large number of telegraf instances.
+  # ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
+  flush_jitter = "0s"
+
+  # Run telegraf in debug mode
+  debug = false
+  # Override default hostname, if empty use os.Hostname()
+  hostname = ""
+
+
+###############################################################################
+#                                  OUTPUTS                                    #
+###############################################################################
+
+[outputs]
+
+# Configuration for influxdb server to send metrics to
+[[outputs.influxdb]]
+  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
+  # Multiple urls can be specified but it is assumed that they are part of the same
+  # cluster, this means that only ONE of the urls will be written to each interval.
+  # urls = ["udp://localhost:8089"] # UDP endpoint example
+  urls = ["http://localhost:8086"] # required
+  # The target database for metrics (telegraf will create it if not exists)
+  database = "telegraf" # required
+  # Precision of writes, valid values are n, u, ms, s, m, and h
+  # note: using second precision greatly helps InfluxDB compression
+  precision = "s"
+
+  # Connection timeout (for the connection with InfluxDB), formatted as a string.
+  # If not provided, will default to 0 (no timeout)
+  # timeout = "5s"
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
+  # user_agent = "telegraf"
+  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
+  # udp_payload = 512
+
+
+###############################################################################
+#                                  PLUGINS                                    #
+###############################################################################
+
+[plugins]
+
+# Read metrics about cpu usage
+[[plugins.cpu]]
+  # Whether to report per-cpu stats or not
+  percpu = true
+  # Whether to report total system cpu stats or not
+  totalcpu = true
+  # Comment this line if you want the raw CPU time metrics
+  drop = ["cpu_time"]
+
+# Read metrics about disk usage by mount point
+[[plugins.disk]]
+  # By default, telegraf gather stats for all mountpoints.
+  # Setting mountpoints will restrict the stats to the specified mountpoints.
+  # Mountpoints=["/"]
+
+# Read metrics about disk IO by device
+[[plugins.diskio]]
+  # By default, telegraf will gather stats for all devices including
+  # disk partitions.
+  # Setting devices will restrict the stats to the specified devcies.
+  # Devices=["sda","sdb"]
+  # Uncomment the following line if you do not need disk serial numbers.
+  # SkipSerialNumber = true
+
+# Read metrics about memory usage
+[[plugins.mem]]
+  # no configuration
+
+# Read metrics about swap memory usage
+[[plugins.swap]]
+  # no configuration
+
+# Read metrics about system load & uptime
+[[plugins.system]]
+  # no configuration
+
+
+###############################################################################
+#                              SERVICE PLUGINS                                #
+###############################################################################

internal/config/config.go

@@ -0,0 +1,709 @@
+package config
+
+import (
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"path/filepath"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/influxdb/telegraf/plugins"
+
+	"github.com/naoina/toml"
+	"github.com/naoina/toml/ast"
+
+	"github.com/influxdb/influxdb/client/v2"
+)
+
+// Config specifies the URL/user/password for the database that telegraf
+// will be logging to, as well as all the plugins that the user has
+// specified
+type Config struct {
+	Tags          map[string]string
+	PluginFilters []string
+	OutputFilters []string
+
+	Agent   *AgentConfig
+	Plugins []*RunningPlugin
+	Outputs []*RunningOutput
+}
+
+func NewConfig() *Config {
+	c := &Config{
+		// Agent defaults:
+		Agent: &AgentConfig{
+			Interval:      internal.Duration{Duration: 10 * time.Second},
+			RoundInterval: true,
+			FlushInterval: internal.Duration{Duration: 10 * time.Second},
+			FlushRetries:  2,
+			FlushJitter:   internal.Duration{Duration: 5 * time.Second},
+		},
+
+		Tags:          make(map[string]string),
+		Plugins:       make([]*RunningPlugin, 0),
+		Outputs:       make([]*RunningOutput, 0),
+		PluginFilters: make([]string, 0),
+		OutputFilters: make([]string, 0),
+	}
+	return c
+}
+
+type AgentConfig struct {
+	// Interval at which to gather information
+	Interval internal.Duration
+
+	// RoundInterval rounds collection interval to 'interval'.
+	//     ie, if Interval=10s then always collect on :00, :10, :20, etc.
+	RoundInterval bool
+
+	// Interval at which to flush data
+	FlushInterval internal.Duration
+
+	// FlushRetries is the number of times to retry each data flush
+	FlushRetries int
+
+	// FlushJitter tells
+	FlushJitter internal.Duration
+
+	// TODO(cam): Remove UTC and Precision parameters, they are no longer
+	// valid for the agent config. Leaving them here for now for backwards-
+	// compatability
+	UTC       bool `toml:"utc"`
+	Precision string
+
+	// Option for running in debug mode
+	Debug    bool
+	Hostname string
+}
+
+// TagFilter is the name of a tag, and the values on which to filter
+type TagFilter struct {
+	Name   string
+	Filter []string
+}
+
+type RunningOutput struct {
+	Name   string
+	Output outputs.Output
+	Config *OutputConfig
+}
+
+type RunningPlugin struct {
+	Name   string
+	Plugin plugins.Plugin
+	Config *PluginConfig
+}
+
+// Filter containing drop/pass and tagdrop/tagpass rules
+type Filter struct {
+	Drop []string
+	Pass []string
+
+	TagDrop []TagFilter
+	TagPass []TagFilter
+
+	IsActive bool
+}
+
+// PluginConfig containing a name, interval, and filter
+type PluginConfig struct {
+	Name              string
+	NameOverride      string
+	MeasurementPrefix string
+	MeasurementSuffix string
+	Tags              map[string]string
+	Filter            Filter
+	Interval          time.Duration
+}
+
+// OutputConfig containing name and filter
+type OutputConfig struct {
+	Name   string
+	Filter Filter
+}
+
+// Filter returns filtered slice of client.Points based on whether filters
+// are active for this RunningOutput.
+func (ro *RunningOutput) FilterPoints(points []*client.Point) []*client.Point {
+	if !ro.Config.Filter.IsActive {
+		return points
+	}
+
+	var filteredPoints []*client.Point
+	for i := range points {
+		if !ro.Config.Filter.ShouldPass(points[i].Name()) || !ro.Config.Filter.ShouldTagsPass(points[i].Tags()) {
+			continue
+		}
+		filteredPoints = append(filteredPoints, points[i])
+	}
+	return filteredPoints
+}
+
+// ShouldPass returns true if the metric should pass, false if should drop
+// based on the drop/pass filter parameters
+func (f Filter) ShouldPass(fieldkey string) bool {
+	if f.Pass != nil {
+		for _, pat := range f.Pass {
+			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			// Cam, 2015-12-07
+			if strings.HasPrefix(fieldkey, pat) || internal.Glob(pat, fieldkey) {
+				return true
+			}
+		}
+		return false
+	}
+
+	if f.Drop != nil {
+		for _, pat := range f.Drop {
+			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			// Cam, 2015-12-07
+			if strings.HasPrefix(fieldkey, pat) || internal.Glob(pat, fieldkey) {
+				return false
+			}
+		}
+
+		return true
+	}
+	return true
+}
+
+// ShouldTagsPass returns true if the metric should pass, false if should drop
+// based on the tagdrop/tagpass filter parameters
+func (f Filter) ShouldTagsPass(tags map[string]string) bool {
+	if f.TagPass != nil {
+		for _, pat := range f.TagPass {
+			if tagval, ok := tags[pat.Name]; ok {
+				for _, filter := range pat.Filter {
+					if internal.Glob(filter, tagval) {
+						return true
+					}
+				}
+			}
+		}
+		return false
+	}
+
+	if f.TagDrop != nil {
+		for _, pat := range f.TagDrop {
+			if tagval, ok := tags[pat.Name]; ok {
+				for _, filter := range pat.Filter {
+					if internal.Glob(filter, tagval) {
+						return false
+					}
+				}
+			}
+		}
+		return true
+	}
+
+	return true
+}
+
+// Plugins returns a list of strings of the configured plugins.
+func (c *Config) PluginNames() []string {
+	var name []string
+	for _, plugin := range c.Plugins {
+		name = append(name, plugin.Name)
+	}
+	return name
+}
+
+// Outputs returns a list of strings of the configured plugins.
+func (c *Config) OutputNames() []string {
+	var name []string
+	for _, output := range c.Outputs {
+		name = append(name, output.Name)
+	}
+	return name
+}
+
+// ListTags returns a string of tags specified in the config,
+// line-protocol style
+func (c *Config) ListTags() string {
+	var tags []string
+
+	for k, v := range c.Tags {
+		tags = append(tags, fmt.Sprintf("%s=%s", k, v))
+	}
+
+	sort.Strings(tags)
+
+	return strings.Join(tags, " ")
+}
+
+var header = `# Telegraf configuration
+
+# Telegraf is entirely plugin driven. All metrics are gathered from the
+# declared plugins.
+
+# Even if a plugin has no configuration, it must be declared in here
+# to be active. Declaring a plugin means just specifying the name
+# as a section with no variables. To deactivate a plugin, comment
+# out the name and any variables.
+
+# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
+# file would generate.
+
+# One rule that plugins conform to is wherever a connection string
+# can be passed, the values '' and 'localhost' are treated specially.
+# They indicate to the plugin to use their own builtin configuration to
+# connect to the local system.
+
+# NOTE: The configuration has a few required parameters. They are marked
+# with 'required'. Be sure to edit those to make this configuration work.
+
+# Tags can also be specified via a normal map, but only one form at a time:
+[tags]
+  # dc = "us-east-1"
+
+# Configuration for telegraf agent
+[agent]
+  # Default data collection interval for all plugins
+  interval = "10s"
+  # Rounds collection interval to 'interval'
+  # ie, if interval="10s" then always collect on :00, :10, :20, etc.
+  round_interval = true
+
+  # Default data flushing interval for all outputs. You should not set this below
+  # interval. Maximum flush_interval will be flush_interval + flush_jitter
+  flush_interval = "10s"
+  # Jitter the flush interval by a random amount. This is primarily to avoid
+  # large write spikes for users running a large number of telegraf instances.
+  # ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
+  flush_jitter = "0s"
+
+  # Run telegraf in debug mode
+  debug = false
+  # Override default hostname, if empty use os.Hostname()
+  hostname = ""
+
+
+###############################################################################
+#                                  OUTPUTS                                    #
+###############################################################################
+
+[outputs]
+`
+
+var pluginHeader = `
+
+###############################################################################
+#                                  PLUGINS                                    #
+###############################################################################
+
+[plugins]
+`
+
+var servicePluginHeader = `
+
+###############################################################################
+#                              SERVICE PLUGINS                                #
+###############################################################################
+`
+
+// PrintSampleConfig prints the sample config
+func PrintSampleConfig(pluginFilters []string, outputFilters []string) {
+	fmt.Printf(header)
+
+	// Filter outputs
+	var onames []string
+	for oname := range outputs.Outputs {
+		if len(outputFilters) == 0 || sliceContains(oname, outputFilters) {
+			onames = append(onames, oname)
+		}
+	}
+	sort.Strings(onames)
+
+	// Print Outputs
+	for _, oname := range onames {
+		creator := outputs.Outputs[oname]
+		output := creator()
+		printConfig(oname, output, "outputs")
+	}
+
+	// Filter plugins
+	var pnames []string
+	for pname := range plugins.Plugins {
+		if len(pluginFilters) == 0 || sliceContains(pname, pluginFilters) {
+			pnames = append(pnames, pname)
+		}
+	}
+	sort.Strings(pnames)
+
+	// Print Plugins
+	fmt.Printf(pluginHeader)
+	servPlugins := make(map[string]plugins.ServicePlugin)
+	for _, pname := range pnames {
+		creator := plugins.Plugins[pname]
+		plugin := creator()
+
+		switch p := plugin.(type) {
+		case plugins.ServicePlugin:
+			servPlugins[pname] = p
+			continue
+		}
+
+		printConfig(pname, plugin, "plugins")
+	}
+
+	// Print Service Plugins
+	fmt.Printf(servicePluginHeader)
+	for name, plugin := range servPlugins {
+		printConfig(name, plugin, "plugins")
+	}
+}
+
+type printer interface {
+	Description() string
+	SampleConfig() string
+}
+
+func printConfig(name string, p printer, op string) {
+	fmt.Printf("\n# %s\n[[%s.%s]]", p.Description(), op, name)
+	config := p.SampleConfig()
+	if config == "" {
+		fmt.Printf("\n  # no configuration\n")
+	} else {
+		fmt.Printf(config)
+	}
+}
+
+func sliceContains(name string, list []string) bool {
+	for _, b := range list {
+		if b == name {
+			return true
+		}
+	}
+	return false
+}
+
+// PrintPluginConfig prints the config usage of a single plugin.
+func PrintPluginConfig(name string) error {
+	if creator, ok := plugins.Plugins[name]; ok {
+		printConfig(name, creator(), "plugins")
+	} else {
+		return errors.New(fmt.Sprintf("Plugin %s not found", name))
+	}
+	return nil
+}
+
+// PrintOutputConfig prints the config usage of a single output.
+func PrintOutputConfig(name string) error {
+	if creator, ok := outputs.Outputs[name]; ok {
+		printConfig(name, creator(), "outputs")
+	} else {
+		return errors.New(fmt.Sprintf("Output %s not found", name))
+	}
+	return nil
+}
+
+func (c *Config) LoadDirectory(path string) error {
+	directoryEntries, err := ioutil.ReadDir(path)
+	if err != nil {
+		return err
+	}
+	for _, entry := range directoryEntries {
+		if entry.IsDir() {
+			continue
+		}
+		name := entry.Name()
+		if len(name) < 6 || name[len(name)-5:] != ".conf" {
+			continue
+		}
+		err := c.LoadConfig(filepath.Join(path, name))
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// LoadConfig loads the given config file and applies it to c
+func (c *Config) LoadConfig(path string) error {
+	data, err := ioutil.ReadFile(path)
+	if err != nil {
+		return err
+	}
+
+	tbl, err := toml.Parse(data)
+	if err != nil {
+		return err
+	}
+
+	for name, val := range tbl.Fields {
+		subTable, ok := val.(*ast.Table)
+		if !ok {
+			return errors.New("invalid configuration")
+		}
+
+		switch name {
+		case "agent":
+			if err = toml.UnmarshalTable(subTable, c.Agent); err != nil {
+				log.Printf("Could not parse [agent] config\n")
+				return err
+			}
+		case "tags":
+			if err = toml.UnmarshalTable(subTable, c.Tags); err != nil {
+				log.Printf("Could not parse [tags] config\n")
+				return err
+			}
+		case "outputs":
+			for outputName, outputVal := range subTable.Fields {
+				switch outputSubTable := outputVal.(type) {
+				case *ast.Table:
+					if err = c.addOutput(outputName, outputSubTable); err != nil {
+						return err
+					}
+				case []*ast.Table:
+					for _, t := range outputSubTable {
+						if err = c.addOutput(outputName, t); err != nil {
+							return err
+						}
+					}
+				default:
+					return fmt.Errorf("Unsupported config format: %s",
+						outputName)
+				}
+			}
+		case "plugins":
+			for pluginName, pluginVal := range subTable.Fields {
+				switch pluginSubTable := pluginVal.(type) {
+				case *ast.Table:
+					if err = c.addPlugin(pluginName, pluginSubTable); err != nil {
+						return err
+					}
+				case []*ast.Table:
+					for _, t := range pluginSubTable {
+						if err = c.addPlugin(pluginName, t); err != nil {
+							return err
+						}
+					}
+				default:
+					return fmt.Errorf("Unsupported config format: %s",
+						pluginName)
+				}
+			}
+		// Assume it's a plugin for legacy config file support if no other
+		// identifiers are present
+		default:
+			if err = c.addPlugin(name, subTable); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func (c *Config) addOutput(name string, table *ast.Table) error {
+	if len(c.OutputFilters) > 0 && !sliceContains(name, c.OutputFilters) {
+		return nil
+	}
+	creator, ok := outputs.Outputs[name]
+	if !ok {
+		return fmt.Errorf("Undefined but requested output: %s", name)
+	}
+	output := creator()
+
+	outputConfig, err := buildOutput(name, table)
+	if err != nil {
+		return err
+	}
+
+	if err := toml.UnmarshalTable(table, output); err != nil {
+		return err
+	}
+
+	ro := &RunningOutput{
+		Name:   name,
+		Output: output,
+		Config: outputConfig,
+	}
+	c.Outputs = append(c.Outputs, ro)
+	return nil
+}
+
+func (c *Config) addPlugin(name string, table *ast.Table) error {
+	if len(c.PluginFilters) > 0 && !sliceContains(name, c.PluginFilters) {
+		return nil
+	}
+	// Legacy support renaming io plugin to diskio
+	if name == "io" {
+		name = "diskio"
+	}
+
+	creator, ok := plugins.Plugins[name]
+	if !ok {
+		return fmt.Errorf("Undefined but requested plugin: %s", name)
+	}
+	plugin := creator()
+
+	pluginConfig, err := buildPlugin(name, table)
+	if err != nil {
+		return err
+	}
+
+	if err := toml.UnmarshalTable(table, plugin); err != nil {
+		return err
+	}
+
+	rp := &RunningPlugin{
+		Name:   name,
+		Plugin: plugin,
+		Config: pluginConfig,
+	}
+	c.Plugins = append(c.Plugins, rp)
+	return nil
+}
+
+// buildFilter builds a Filter (tagpass/tagdrop/pass/drop) to
+// be inserted into the OutputConfig/PluginConfig to be used for prefix
+// filtering on tags and measurements
+func buildFilter(tbl *ast.Table) Filter {
+	f := Filter{}
+
+	if node, ok := tbl.Fields["pass"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if ary, ok := kv.Value.(*ast.Array); ok {
+				for _, elem := range ary.Value {
+					if str, ok := elem.(*ast.String); ok {
+						f.Pass = append(f.Pass, str.Value)
+						f.IsActive = true
+					}
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["drop"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if ary, ok := kv.Value.(*ast.Array); ok {
+				for _, elem := range ary.Value {
+					if str, ok := elem.(*ast.String); ok {
+						f.Drop = append(f.Drop, str.Value)
+						f.IsActive = true
+					}
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["tagpass"]; ok {
+		if subtbl, ok := node.(*ast.Table); ok {
+			for name, val := range subtbl.Fields {
+				if kv, ok := val.(*ast.KeyValue); ok {
+					tagfilter := &TagFilter{Name: name}
+					if ary, ok := kv.Value.(*ast.Array); ok {
+						for _, elem := range ary.Value {
+							if str, ok := elem.(*ast.String); ok {
+								tagfilter.Filter = append(tagfilter.Filter, str.Value)
+							}
+						}
+					}
+					f.TagPass = append(f.TagPass, *tagfilter)
+					f.IsActive = true
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["tagdrop"]; ok {
+		if subtbl, ok := node.(*ast.Table); ok {
+			for name, val := range subtbl.Fields {
+				if kv, ok := val.(*ast.KeyValue); ok {
+					tagfilter := &TagFilter{Name: name}
+					if ary, ok := kv.Value.(*ast.Array); ok {
+						for _, elem := range ary.Value {
+							if str, ok := elem.(*ast.String); ok {
+								tagfilter.Filter = append(tagfilter.Filter, str.Value)
+							}
+						}
+					}
+					f.TagDrop = append(f.TagDrop, *tagfilter)
+					f.IsActive = true
+				}
+			}
+		}
+	}
+
+	delete(tbl.Fields, "drop")
+	delete(tbl.Fields, "pass")
+	delete(tbl.Fields, "tagdrop")
+	delete(tbl.Fields, "tagpass")
+	return f
+}
+
+// buildPlugin parses plugin specific items from the ast.Table,
+// builds the filter and returns a
+// PluginConfig to be inserted into RunningPlugin
+func buildPlugin(name string, tbl *ast.Table) (*PluginConfig, error) {
+	cp := &PluginConfig{Name: name}
+	if node, ok := tbl.Fields["interval"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				dur, err := time.ParseDuration(str.Value)
+				if err != nil {
+					return nil, err
+				}
+
+				cp.Interval = dur
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["name_prefix"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				cp.MeasurementPrefix = str.Value
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["name_suffix"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				cp.MeasurementSuffix = str.Value
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["name_override"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				cp.NameOverride = str.Value
+			}
+		}
+	}
+
+	cp.Tags = make(map[string]string)
+	if node, ok := tbl.Fields["tags"]; ok {
+		if subtbl, ok := node.(*ast.Table); ok {
+			if err := toml.UnmarshalTable(subtbl, cp.Tags); err != nil {
+				log.Printf("Could not parse tags for plugin %s\n", name)
+			}
+		}
+	}
+
+	delete(tbl.Fields, "name_prefix")
+	delete(tbl.Fields, "name_suffix")
+	delete(tbl.Fields, "name_override")
+	delete(tbl.Fields, "interval")
+	delete(tbl.Fields, "tags")
+	cp.Filter = buildFilter(tbl)
+	return cp, nil
+}
+
+// buildOutput parses output specific items from the ast.Table, builds the filter and returns an
+// OutputConfig to be inserted into RunningPlugin
+// Note: error exists in the return for future calls that might require error
+func buildOutput(name string, tbl *ast.Table) (*OutputConfig, error) {
+	oc := &OutputConfig{
+		Name:   name,
+		Filter: buildFilter(tbl),
+	}
+	return oc, nil
+}

internal/config/config_test.go

@@ -0,0 +1,296 @@
+package config
+
+import (
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/plugins"
+	"github.com/influxdb/telegraf/plugins/exec"
+	"github.com/influxdb/telegraf/plugins/memcached"
+	"github.com/influxdb/telegraf/plugins/procstat"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestConfig_LoadSinglePlugin(t *testing.T) {
+	c := NewConfig()
+	c.LoadConfig("./testdata/single_plugin.toml")
+
+	memcached := plugins.Plugins["memcached"]().(*memcached.Memcached)
+	memcached.Servers = []string{"localhost"}
+
+	mConfig := &PluginConfig{
+		Name: "memcached",
+		Filter: Filter{
+			Drop: []string{"other", "stuff"},
+			Pass: []string{"some", "strings"},
+			TagDrop: []TagFilter{
+				TagFilter{
+					Name:   "badtag",
+					Filter: []string{"othertag"},
+				},
+			},
+			TagPass: []TagFilter{
+				TagFilter{
+					Name:   "goodtag",
+					Filter: []string{"mytag"},
+				},
+			},
+			IsActive: true,
+		},
+		Interval: 5 * time.Second,
+	}
+
+	assert.Equal(t, memcached, c.Plugins[0].Plugin,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Plugins[0].Config,
+		"Testdata did not produce correct memcached metadata.")
+}
+
+func TestConfig_LoadDirectory(t *testing.T) {
+	c := NewConfig()
+	err := c.LoadConfig("./testdata/single_plugin.toml")
+	if err != nil {
+		t.Error(err)
+	}
+	err = c.LoadDirectory("./testdata/subconfig")
+	if err != nil {
+		t.Error(err)
+	}
+
+	memcached := plugins.Plugins["memcached"]().(*memcached.Memcached)
+	memcached.Servers = []string{"localhost"}
+
+	mConfig := &PluginConfig{
+		Name: "memcached",
+		Filter: Filter{
+			Drop: []string{"other", "stuff"},
+			Pass: []string{"some", "strings"},
+			TagDrop: []TagFilter{
+				TagFilter{
+					Name:   "badtag",
+					Filter: []string{"othertag"},
+				},
+			},
+			TagPass: []TagFilter{
+				TagFilter{
+					Name:   "goodtag",
+					Filter: []string{"mytag"},
+				},
+			},
+			IsActive: true,
+		},
+		Interval: 5 * time.Second,
+	}
+	assert.Equal(t, memcached, c.Plugins[0].Plugin,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Plugins[0].Config,
+		"Testdata did not produce correct memcached metadata.")
+
+	ex := plugins.Plugins["exec"]().(*exec.Exec)
+	ex.Commands = []*exec.Command{
+		&exec.Command{
+			Command: "/usr/bin/myothercollector --foo=bar",
+			Name:    "myothercollector",
+		},
+	}
+	eConfig := &PluginConfig{Name: "exec"}
+	assert.Equal(t, ex, c.Plugins[1].Plugin,
+		"Merged Testdata did not produce a correct exec struct.")
+	assert.Equal(t, eConfig, c.Plugins[1].Config,
+		"Merged Testdata did not produce correct exec metadata.")
+
+	memcached.Servers = []string{"192.168.1.1"}
+	assert.Equal(t, memcached, c.Plugins[2].Plugin,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Plugins[2].Config,
+		"Testdata did not produce correct memcached metadata.")
+
+	pstat := plugins.Plugins["procstat"]().(*procstat.Procstat)
+	pstat.Specifications = []*procstat.Specification{
+		&procstat.Specification{
+			PidFile: "/var/run/grafana-server.pid",
+		},
+		&procstat.Specification{
+			PidFile: "/var/run/influxdb/influxd.pid",
+		},
+	}
+
+	pConfig := &PluginConfig{Name: "procstat"}
+
+	assert.Equal(t, pstat, c.Plugins[3].Plugin,
+		"Merged Testdata did not produce a correct procstat struct.")
+	assert.Equal(t, pConfig, c.Plugins[3].Config,
+		"Merged Testdata did not produce correct procstat metadata.")
+}
+
+func TestFilter_Empty(t *testing.T) {
+	f := Filter{}
+
+	measurements := []string{
+		"foo",
+		"bar",
+		"barfoo",
+		"foo_bar",
+		"foo.bar",
+		"foo-bar",
+		"supercalifradjulisticexpialidocious",
+	}
+
+	for _, measurement := range measurements {
+		if !f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to pass", measurement)
+		}
+	}
+}
+
+func TestFilter_Pass(t *testing.T) {
+	f := Filter{
+		Pass: []string{"foo*", "cpu_usage_idle"},
+	}
+
+	passes := []string{
+		"foo",
+		"foo_bar",
+		"foo.bar",
+		"foo-bar",
+		"cpu_usage_idle",
+	}
+
+	drops := []string{
+		"bar",
+		"barfoo",
+		"bar_foo",
+		"cpu_usage_busy",
+	}
+
+	for _, measurement := range passes {
+		if !f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to pass", measurement)
+		}
+	}
+
+	for _, measurement := range drops {
+		if f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to drop", measurement)
+		}
+	}
+}
+
+func TestFilter_Drop(t *testing.T) {
+	f := Filter{
+		Drop: []string{"foo*", "cpu_usage_idle"},
+	}
+
+	drops := []string{
+		"foo",
+		"foo_bar",
+		"foo.bar",
+		"foo-bar",
+		"cpu_usage_idle",
+	}
+
+	passes := []string{
+		"bar",
+		"barfoo",
+		"bar_foo",
+		"cpu_usage_busy",
+	}
+
+	for _, measurement := range passes {
+		if !f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to pass", measurement)
+		}
+	}
+
+	for _, measurement := range drops {
+		if f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to drop", measurement)
+		}
+	}
+}
+
+func TestFilter_TagPass(t *testing.T) {
+	filters := []TagFilter{
+		TagFilter{
+			Name:   "cpu",
+			Filter: []string{"cpu-*"},
+		},
+		TagFilter{
+			Name:   "mem",
+			Filter: []string{"mem_free"},
+		}}
+	f := Filter{
+		TagPass: filters,
+	}
+
+	passes := []map[string]string{
+		{"cpu": "cpu-total"},
+		{"cpu": "cpu-0"},
+		{"cpu": "cpu-1"},
+		{"cpu": "cpu-2"},
+		{"mem": "mem_free"},
+	}
+
+	drops := []map[string]string{
+		{"cpu": "cputotal"},
+		{"cpu": "cpu0"},
+		{"cpu": "cpu1"},
+		{"cpu": "cpu2"},
+		{"mem": "mem_used"},
+	}
+
+	for _, tags := range passes {
+		if !f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to pass", tags)
+		}
+	}
+
+	for _, tags := range drops {
+		if f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to drop", tags)
+		}
+	}
+}
+
+func TestFilter_TagDrop(t *testing.T) {
+	filters := []TagFilter{
+		TagFilter{
+			Name:   "cpu",
+			Filter: []string{"cpu-*"},
+		},
+		TagFilter{
+			Name:   "mem",
+			Filter: []string{"mem_free"},
+		}}
+	f := Filter{
+		TagDrop: filters,
+	}
+
+	drops := []map[string]string{
+		{"cpu": "cpu-total"},
+		{"cpu": "cpu-0"},
+		{"cpu": "cpu-1"},
+		{"cpu": "cpu-2"},
+		{"mem": "mem_free"},
+	}
+
+	passes := []map[string]string{
+		{"cpu": "cputotal"},
+		{"cpu": "cpu0"},
+		{"cpu": "cpu1"},
+		{"cpu": "cpu2"},
+		{"mem": "mem_used"},
+	}
+
+	for _, tags := range passes {
+		if !f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to pass", tags)
+		}
+	}
+
+	for _, tags := range drops {
+		if f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to drop", tags)
+		}
+	}
+}

internal/config/testdata/single_plugin.toml

@@ -0,0 +1,9 @@
+[[plugins.memcached]]
+  servers = ["localhost"]
+  pass = ["some", "strings"]
+  drop = ["other", "stuff"]
+  interval = "5s"
+  [plugins.memcached.tagpass]
+    goodtag = ["mytag"]
+  [plugins.memcached.tagdrop]
+    badtag = ["othertag"]

internal/config/testdata/subconfig/exec.conf

@@ -0,0 +1,8 @@
+[[plugins.exec]]
+  # specify commands via an array of tables
+  [[plugins.exec.commands]]
+  # the command to run
+  command = "/usr/bin/myothercollector --foo=bar"
+
+  # name of the command (used as a prefix for measurements)
+  name = "myothercollector"

internal/config/testdata/subconfig/memcached.conf

@@ -0,0 +1,9 @@
+[[plugins.memcached]]
+  servers = ["192.168.1.1"]
+  pass = ["some", "strings"]
+  drop = ["other", "stuff"]
+  interval = "5s"
+  [plugins.memcached.tagpass]
+    goodtag = ["mytag"]
+  [plugins.memcached.tagdrop]
+    badtag = ["othertag"]

internal/config/testdata/subconfig/procstat.conf

@@ -0,0 +1,5 @@
+[[plugins.procstat]]
+  [[plugins.procstat.specifications]]
+  pid_file = "/var/run/grafana-server.pid"
+  [[plugins.procstat.specifications]]
+  pid_file = "/var/run/influxdb/influxd.pid"

internal/config/testdata/telegraf-agent.toml

@@ -0,0 +1,334 @@
+# Telegraf configuration
+
+# Telegraf is entirely plugin driven. All metrics are gathered from the
+# declared plugins.
+
+# Even if a plugin has no configuration, it must be declared in here
+# to be active. Declaring a plugin means just specifying the name
+# as a section with no variables. To deactivate a plugin, comment
+# out the name and any variables.
+
+# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
+# file would generate.
+
+# One rule that plugins conform to is wherever a connection string
+# can be passed, the values '' and 'localhost' are treated specially.
+# They indicate to the plugin to use their own builtin configuration to
+# connect to the local system.
+
+# NOTE: The configuration has a few required parameters. They are marked
+# with 'required'. Be sure to edit those to make this configuration work.
+
+# Tags can also be specified via a normal map, but only one form at a time:
+[tags]
+  # dc = "us-east-1"
+
+# Configuration for telegraf agent
+[agent]
+  # Default data collection interval for all plugins
+  interval = "10s"
+
+  # If utc = false, uses local time (utc is highly recommended)
+  utc = true
+
+  # Precision of writes, valid values are n, u, ms, s, m, and h
+  # note: using second precision greatly helps InfluxDB compression
+  precision = "s"
+
+  # run telegraf in debug mode
+  debug = false
+
+  # Override default hostname, if empty use os.Hostname()
+  hostname = ""
+
+
+###############################################################################
+#                                  OUTPUTS                                    #
+###############################################################################
+
+[outputs]
+
+# Configuration for influxdb server to send metrics to
+[[outputs.influxdb]]
+  # The full HTTP endpoint URL for your InfluxDB instance
+  # Multiple urls can be specified for InfluxDB cluster support. Server to
+  # write to will be randomly chosen each interval.
+  urls = ["http://localhost:8086"] # required.
+
+  # The target database for metrics. This database must already exist
+  database = "telegraf" # required.
+
+  # Connection timeout (for the connection with InfluxDB), formatted as a string.
+  # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+  # If not provided, will default to 0 (no timeout)
+  # timeout = "5s"
+
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+
+  # Set the user agent for the POSTs (can be useful for log differentiation)
+  # user_agent = "telegraf"
+
+[[outputs.influxdb]]
+  urls = ["udp://localhost:8089"]
+  database = "udp-telegraf"
+
+# Configuration for the Kafka server to send metrics to
+[[outputs.kafka]]
+  # URLs of kafka brokers
+  brokers = ["localhost:9092"]
+  # Kafka topic for producer messages
+  topic = "telegraf"
+  # Telegraf tag to use as a routing key
+  #  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+
+
+###############################################################################
+#                                  PLUGINS                                    #
+###############################################################################
+
+[plugins]
+
+# Read Apache status information (mod_status)
+[[plugins.apache]]
+# An array of Apache status URI to gather stats.
+urls = ["http://localhost/server-status?auto"]
+
+# Read metrics about cpu usage
+[[plugins.cpu]]
+  # Whether to report per-cpu stats or not
+  percpu = true
+  # Whether to report total system cpu stats or not
+  totalcpu = true
+  # Comment this line if you want the raw CPU time metrics
+  drop = ["cpu_time"]
+
+# Read metrics about disk usage by mount point
+[[plugins.diskio]]
+  # no configuration
+
+# Read metrics from one or many disque servers
+[[plugins.disque]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port and password. ie disque://localhost, disque://10.10.3.33:18832,
+  # 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read stats from one or more Elasticsearch servers or clusters
+[[plugins.elasticsearch]]
+  # specify a list of one or more Elasticsearch servers
+  servers = ["http://localhost:9200"]
+
+  # set local to false when you want to read the indices stats from all nodes
+  # within the cluster
+  local = true
+
+# Read flattened metrics from one or more commands that output JSON to stdout
+[[plugins.exec]]
+  # specify commands via an array of tables
+  [[exec.commands]]
+  # the command to run
+  command = "/usr/bin/mycollector --foo=bar"
+
+  # name of the command (used as a prefix for measurements)
+  name = "mycollector"
+
+# Read metrics of haproxy, via socket or csv stats page
+[[plugins.haproxy]]
+  # An array of address to gather stats about. Specify an ip on hostname
+  # with optional port. ie localhost, 10.10.3.33:1936, etc.
+  #
+  # If no servers are specified, then default to 127.0.0.1:1936
+  servers = ["http://myhaproxy.com:1936", "http://anotherhaproxy.com:1936"]
+  # Or you can also use local socket(not work yet)
+  # servers = ["socket:/run/haproxy/admin.sock"]
+
+# Read flattened metrics from one or more JSON HTTP endpoints
+[[plugins.httpjson]]
+  # Specify services via an array of tables
+  [[httpjson.services]]
+
+    # a name for the service being polled
+    name = "webserver_stats"
+
+    # URL of each server in the service's cluster
+    servers = [
+      "http://localhost:9999/stats/",
+      "http://localhost:9998/stats/",
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "GET"
+
+    # HTTP parameters (all values must be strings)
+    [httpjson.services.parameters]
+      event_type = "cpu_spike"
+      threshold = "0.75"
+
+# Read metrics about disk IO by device
+[[plugins.io]]
+  # no configuration
+
+# read metrics from a Kafka topic
+[[plugins.kafka_consumer]]
+  # topic(s) to consume
+  topics = ["telegraf"]
+  # an array of Zookeeper connection strings
+  zookeeper_peers = ["localhost:2181"]
+  # the name of the consumer group
+  consumer_group = "telegraf_metrics_consumers"
+  # Maximum number of points to buffer between collection intervals
+  point_buffer = 100000
+  # Offset (must be either "oldest" or "newest")
+  offset = "oldest"
+
+# Read metrics from a LeoFS Server via SNMP
+[[plugins.leofs]]
+  # An array of URI to gather stats about LeoFS.
+  # Specify an ip or hostname with port. ie 127.0.0.1:4020
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 4020 as the port.
+  servers = ["127.0.0.1:4021"]
+
+# Read metrics from local Lustre service on OST, MDS
+[[plugins.lustre2]]
+  # An array of /proc globs to search for Lustre stats
+  # If not specified, the default will work on Lustre 2.5.x
+  #
+  # ost_procfiles = ["/proc/fs/lustre/obdfilter/*/stats", "/proc/fs/lustre/osd-ldiskfs/*/stats"]
+  # mds_procfiles = ["/proc/fs/lustre/mdt/*/md_stats"]
+
+# Read metrics about memory usage
+[[plugins.mem]]
+  # no configuration
+
+# Read metrics from one or many memcached servers
+[[plugins.memcached]]
+  # An array of address to gather stats about. Specify an ip on hostname
+  # with optional port. ie localhost, 10.0.0.1:11211, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read metrics from one or many MongoDB servers
+[[plugins.mongodb]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port add password. ie mongodb://user:auth_key@10.10.3.30:27017,
+  # mongodb://10.10.3.33:18832, 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 27107 as the port.
+  servers = ["127.0.0.1:27017"]
+
+# Read metrics from one or many mysql servers
+[[plugins.mysql]]
+  # specify servers via a url matching:
+  #  [username[:password]@][protocol[(address)]]/[?tls=[true|false|skip-verify]]
+  #  e.g.
+  #    servers = ["root:root@http://10.0.0.18/?tls=false"]
+  #    servers = ["root:passwd@tcp(127.0.0.1:3306)/"]
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read metrics about network interface usage
+[[plugins.net]]
+  # By default, telegraf gathers stats from any up interface (excluding loopback)
+  # Setting interfaces will tell it to gather these explicit interfaces,
+  # regardless of status.
+  #
+  # interfaces = ["eth0", ... ]
+
+# Read Nginx's basic status information (ngx_http_stub_status_module)
+[[plugins.nginx]]
+  # An array of Nginx stub_status URI to gather stats.
+  urls = ["http://localhost/status"]
+
+# Ping given url(s) and return statistics
+[[plugins.ping]]
+  # urls to ping
+  urls = ["www.google.com"] # required
+  # number of pings to send (ping -c <COUNT>)
+  count = 1 # required
+  # interval, in s, at which to ping. 0 == default (ping -i <PING_INTERVAL>)
+  ping_interval = 0.0
+  # ping timeout, in s. 0 == no timeout (ping -t <TIMEOUT>)
+  timeout = 0.0
+  # interface to send ping from (ping -I <INTERFACE>)
+  interface = ""
+
+# Read metrics from one or many postgresql servers
+[[plugins.postgresql]]
+  # specify servers via an array of tables
+  [[postgresql.servers]]
+
+  # specify address via a url matching:
+  #   postgres://[pqgotest[:password]]@localhost[/dbname]?sslmode=[disable|verify-ca|verify-full]
+  # or a simple string:
+  #   host=localhost user=pqotest password=... sslmode=... dbname=app_production
+  #
+  # All connection parameters are optional. By default, the host is localhost
+  # and the user is the currently running user. For localhost, we default
+  # to sslmode=disable as well.
+  #
+  # Without the dbname parameter, the driver will default to a database
+  # with the same name as the user. This dbname is just for instantiating a
+  # connection with the server and doesn't restrict the databases we are trying
+  # to grab metrics for.
+  #
+
+  address = "sslmode=disable"
+
+  # A list of databases to pull metrics about. If not specified, metrics for all
+  # databases are gathered.
+
+  # databases = ["app_production", "blah_testing"]
+
+  # [[postgresql.servers]]
+  # address = "influx@remoteserver"
+
+# Read metrics from one or many prometheus clients
+[[plugins.prometheus]]
+  # An array of urls to scrape metrics from.
+  urls = ["http://localhost:9100/metrics"]
+
+# Read metrics from one or many RabbitMQ servers via the management API
+[[plugins.rabbitmq]]
+  # Specify servers via an array of tables
+  [[rabbitmq.servers]]
+  # name = "rmq-server-1" # optional tag
+  # url = "http://localhost:15672"
+  # username = "guest"
+  # password = "guest"
+
+  # A list of nodes to pull metrics about. If not specified, metrics for
+  # all nodes are gathered.
+  # nodes = ["rabbit@node1", "rabbit@node2"]
+
+# Read metrics from one or many redis servers
+[[plugins.redis]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port add password. ie redis://localhost, redis://10.10.3.33:18832,
+  # 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read metrics from one or many RethinkDB servers
+[[plugins.rethinkdb]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port add password. ie rethinkdb://user:auth_key@10.10.3.30:28105,
+  # rethinkdb://10.10.3.33:18832, 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 28015 as the port.
+  servers = ["127.0.0.1:28015"]
+
+# Read metrics about swap memory usage
+[[plugins.swap]]
+  # no configuration
+
+# Read metrics about system load & uptime
+[[plugins.system]]
+  # no configuration

internal/internal.go

@@ -0,0 +1,153 @@
+package internal
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+	"time"
+)
+
+// Duration just wraps time.Duration
+type Duration struct {
+	Duration time.Duration
+}
+
+// UnmarshalTOML parses the duration from the TOML config file
+func (d *Duration) UnmarshalTOML(b []byte) error {
+	dur, err := time.ParseDuration(string(b[1 : len(b)-1]))
+	if err != nil {
+		return err
+	}
+
+	d.Duration = dur
+
+	return nil
+}
+
+var NotImplementedError = errors.New("not implemented yet")
+
+type JSONFlattener struct {
+	Fields map[string]interface{}
+}
+
+// FlattenJSON flattens nested maps/interfaces into a fields map
+func (f *JSONFlattener) FlattenJSON(
+	fieldname string,
+	v interface{},
+) error {
+	if f.Fields == nil {
+		f.Fields = make(map[string]interface{})
+	}
+	fieldname = strings.Trim(fieldname, "_")
+	switch t := v.(type) {
+	case map[string]interface{}:
+		for k, v := range t {
+			err := f.FlattenJSON(fieldname+"_"+k+"_", v)
+			if err != nil {
+				return err
+			}
+		}
+	case float64:
+		f.Fields[fieldname] = t
+	case bool, string, []interface{}:
+		// ignored types
+		return nil
+	default:
+		return fmt.Errorf("JSON Flattener: got unexpected type %T with value %v (%s)",
+			t, t, fieldname)
+	}
+	return nil
+}
+
+// ReadLines reads contents from a file and splits them by new lines.
+// A convenience wrapper to ReadLinesOffsetN(filename, 0, -1).
+func ReadLines(filename string) ([]string, error) {
+	return ReadLinesOffsetN(filename, 0, -1)
+}
+
+// ReadLines reads contents from file and splits them by new line.
+// The offset tells at which line number to start.
+// The count determines the number of lines to read (starting from offset):
+//   n >= 0: at most n lines
+//   n < 0: whole file
+func ReadLinesOffsetN(filename string, offset uint, n int) ([]string, error) {
+	f, err := os.Open(filename)
+	if err != nil {
+		return []string{""}, err
+	}
+	defer f.Close()
+
+	var ret []string
+
+	r := bufio.NewReader(f)
+	for i := 0; i < n+int(offset) || n < 0; i++ {
+		line, err := r.ReadString('\n')
+		if err != nil {
+			break
+		}
+		if i < int(offset) {
+			continue
+		}
+		ret = append(ret, strings.Trim(line, "\n"))
+	}
+
+	return ret, nil
+}
+
+// Glob will test a string pattern, potentially containing globs, against a
+// subject string. The result is a simple true/false, determining whether or
+// not the glob pattern matched the subject text.
+//
+// Adapted from https://github.com/ryanuber/go-glob/blob/master/glob.go
+// thanks Ryan Uber!
+func Glob(pattern, measurement string) bool {
+	// Empty pattern can only match empty subject
+	if pattern == "" {
+		return measurement == pattern
+	}
+
+	// If the pattern _is_ a glob, it matches everything
+	if pattern == "*" {
+		return true
+	}
+
+	parts := strings.Split(pattern, "*")
+
+	if len(parts) == 1 {
+		// No globs in pattern, so test for match
+		return pattern == measurement
+	}
+
+	leadingGlob := strings.HasPrefix(pattern, "*")
+	trailingGlob := strings.HasSuffix(pattern, "*")
+	end := len(parts) - 1
+
+	for i, part := range parts {
+		switch i {
+		case 0:
+			if leadingGlob {
+				continue
+			}
+			if !strings.HasPrefix(measurement, part) {
+				return false
+			}
+		case end:
+			if len(measurement) > 0 {
+				return trailingGlob || strings.HasSuffix(measurement, part)
+			}
+		default:
+			if !strings.Contains(measurement, part) {
+				return false
+			}
+		}
+
+		// Trim evaluated text from measurement as we loop over the pattern.
+		idx := strings.Index(measurement, part) + len(part)
+		measurement = measurement[idx:]
+	}
+
+	// All parts of the pattern matched
+	return true
+}

internal/internal_test.go

@@ -0,0 +1,44 @@
+package internal
+
+import "testing"
+
+func testGlobMatch(t *testing.T, pattern, subj string) {
+	if !Glob(pattern, subj) {
+		t.Errorf("%s should match %s", pattern, subj)
+	}
+}
+
+func testGlobNoMatch(t *testing.T, pattern, subj string) {
+	if Glob(pattern, subj) {
+		t.Errorf("%s should not match %s", pattern, subj)
+	}
+}
+
+func TestEmptyPattern(t *testing.T) {
+	testGlobMatch(t, "", "")
+	testGlobNoMatch(t, "", "test")
+}
+
+func TestPatternWithoutGlobs(t *testing.T) {
+	testGlobMatch(t, "test", "test")
+}
+
+func TestGlob(t *testing.T) {
+	for _, pattern := range []string{
+		"*test",           // Leading glob
+		"this*",           // Trailing glob
+		"*is*a*",          // Lots of globs
+		"**test**",        // Double glob characters
+		"**is**a***test*", // Varying number of globs
+	} {
+		testGlobMatch(t, pattern, "this_is_a_test")
+	}
+
+	for _, pattern := range []string{
+		"test*", // Implicit substring match should fail
+		"*is",   // Partial match should fail
+		"*no*",  // Globs without a match between them should fail
+	} {
+		testGlobNoMatch(t, pattern, "this_is_a_test")
+	}
+}

outputs/all/all.go

@@ -0,0 +1,16 @@
+package all
+
+import (
+	_ "github.com/influxdb/telegraf/outputs/amon"
+	_ "github.com/influxdb/telegraf/outputs/amqp"
+	_ "github.com/influxdb/telegraf/outputs/datadog"
+	_ "github.com/influxdb/telegraf/outputs/influxdb"
+	_ "github.com/influxdb/telegraf/outputs/kafka"
+	_ "github.com/influxdb/telegraf/outputs/kinesis"
+	_ "github.com/influxdb/telegraf/outputs/librato"
+	_ "github.com/influxdb/telegraf/outputs/mqtt"
+	_ "github.com/influxdb/telegraf/outputs/nsq"
+	_ "github.com/influxdb/telegraf/outputs/opentsdb"
+	_ "github.com/influxdb/telegraf/outputs/prometheus_client"
+	_ "github.com/influxdb/telegraf/outputs/riemann"
+)

outputs/amon/README.md

@@ -0,0 +1,9 @@
+# Amon Output Plugin
+
+This plugin writes to [Amon](https://www.amon.cx)
+and requires an `serverkey` and `amoninstance` URL which can be obtained [here](https://www.amon.cx/docs/monitoring/)
+for the account.
+
+If the point value being sent cannot be converted to a float64, the metric is skipped.
+
+Metrics are grouped by converting any `_` characters to `.` in the Point Name.

outputs/amon/amon.go

@@ -0,0 +1,157 @@
+package amon
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+	"strings"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Amon struct {
+	ServerKey    string
+	AmonInstance string
+	Timeout      internal.Duration
+
+	client *http.Client
+}
+
+var sampleConfig = `
+  # Amon Server Key
+  server_key = "my-server-key" # required.
+
+  # Amon Instance URL
+  amon_instance = "https://youramoninstance" # required
+
+  # Connection timeout.
+  # timeout = "5s"
+`
+
+type TimeSeries struct {
+	Series []*Metric `json:"series"`
+}
+
+type Metric struct {
+	Metric string   `json:"metric"`
+	Points [1]Point `json:"points"`
+}
+
+type Point [2]float64
+
+func (a *Amon) Connect() error {
+	if a.ServerKey == "" || a.AmonInstance == "" {
+		return fmt.Errorf("serverkey and amon_instance are required fields for amon output")
+	}
+	a.client = &http.Client{
+		Timeout: a.Timeout.Duration,
+	}
+	return nil
+}
+
+func (a *Amon) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	ts := TimeSeries{}
+	tempSeries := []*Metric{}
+	metricCounter := 0
+
+	for _, pt := range points {
+		mname := strings.Replace(pt.Name(), "_", ".", -1)
+		if amonPts, err := buildPoints(pt); err == nil {
+			for fieldName, amonPt := range amonPts {
+				metric := &Metric{
+					Metric: mname + "_" + strings.Replace(fieldName, "_", ".", -1),
+				}
+				metric.Points[0] = amonPt
+				tempSeries = append(tempSeries, metric)
+				metricCounter++
+			}
+		} else {
+			log.Printf("unable to build Metric for %s, skipping\n", pt.Name())
+		}
+	}
+
+	ts.Series = make([]*Metric, metricCounter)
+	copy(ts.Series, tempSeries[0:])
+	tsBytes, err := json.Marshal(ts)
+	if err != nil {
+		return fmt.Errorf("unable to marshal TimeSeries, %s\n", err.Error())
+	}
+	req, err := http.NewRequest("POST", a.authenticatedUrl(), bytes.NewBuffer(tsBytes))
+	if err != nil {
+		return fmt.Errorf("unable to create http.Request, %s\n", err.Error())
+	}
+	req.Header.Add("Content-Type", "application/json")
+
+	resp, err := a.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("error POSTing metrics, %s\n", err.Error())
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode < 200 || resp.StatusCode > 209 {
+		return fmt.Errorf("received bad status code, %d\n", resp.StatusCode)
+	}
+
+	return nil
+}
+
+func (a *Amon) SampleConfig() string {
+	return sampleConfig
+}
+
+func (a *Amon) Description() string {
+	return "Configuration for Amon Server to send metrics to."
+}
+
+func (a *Amon) authenticatedUrl() string {
+
+	return fmt.Sprintf("%s/api/system/%s", a.AmonInstance, a.ServerKey)
+}
+
+func buildPoints(pt *client.Point) (map[string]Point, error) {
+	pts := make(map[string]Point)
+	for k, v := range pt.Fields() {
+		var p Point
+		if err := p.setValue(v); err != nil {
+			return pts, fmt.Errorf("unable to extract value from Fields, %s", err.Error())
+		}
+		p[0] = float64(pt.Time().Unix())
+		pts[k] = p
+	}
+	return pts, nil
+}
+
+func (p *Point) setValue(v interface{}) error {
+	switch d := v.(type) {
+	case int:
+		p[1] = float64(int(d))
+	case int32:
+		p[1] = float64(int32(d))
+	case int64:
+		p[1] = float64(int64(d))
+	case float32:
+		p[1] = float64(d)
+	case float64:
+		p[1] = float64(d)
+	default:
+		return fmt.Errorf("undeterminable type")
+	}
+	return nil
+}
+
+func (a *Amon) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("amon", func() outputs.Output {
+		return &Amon{}
+	})
+}

outputs/amon/amon_test.go

@@ -0,0 +1,89 @@
+package amon
+
+import (
+	"fmt"
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/influxdb/influxdb/client/v2"
+)
+
+func TestBuildPoint(t *testing.T) {
+	var tagtests = []struct {
+		ptIn  *client.Point
+		outPt Point
+		err   error
+	}{
+		{
+			testutil.TestPoint(float64(0.0)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				0.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float64(1.0)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				1.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int(10)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				10.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int32(112345)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int64(112345)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float32(11234.5)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint("11234.5"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			fmt.Errorf("unable to extract value from Fields, undeterminable type"),
+		},
+	}
+	for _, tt := range tagtests {
+		pt, err := buildPoint(tt.ptIn)
+		if err != nil && tt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", tt.ptIn.Name(), err)
+		}
+		if tt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", tt.ptIn.Name(), tt.err.Error())
+		}
+		if !reflect.DeepEqual(pt, tt.outPt) && tt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", tt.ptIn.Name(), tt.outPt, pt)
+		}
+	}
+}

outputs/amqp/README.md

@@ -0,0 +1,9 @@
+# AMQP Output Plugin
+
+This plugin writes to a AMQP exchange using tag, defined in configuration file
+as RoutingTag, as a routing key.
+
+If RoutingTag is empty, then empty routing key will be used.
+Metrics are grouped in batches by RoutingTag.
+
+This plugin doesn't bind exchange to a queue, so it should be done by consumer.

outputs/amqp/amqp.go

@@ -0,0 +1,161 @@
+package amqp
+
+import (
+	"bytes"
+	"fmt"
+	"log"
+	"sync"
+	"time"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/streadway/amqp"
+)
+
+type AMQP struct {
+	// AMQP brokers to send metrics to
+	URL string
+	// AMQP exchange
+	Exchange string
+	// Routing Key Tag
+	RoutingTag string `toml:"routing_tag"`
+	// InfluxDB database
+	Database string
+	// InfluxDB retention policy
+	RetentionPolicy string
+	// InfluxDB precision
+	Precision string
+
+	channel *amqp.Channel
+	sync.Mutex
+	headers amqp.Table
+}
+
+const (
+	DefaultRetentionPolicy = "default"
+	DefaultDatabase        = "telegraf"
+	DefaultPrecision       = "s"
+)
+
+var sampleConfig = `
+  # AMQP url
+  url = "amqp://localhost:5672/influxdb"
+  # AMQP exchange
+  exchange = "telegraf"
+  # Telegraf tag to use as a routing key
+  #  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+
+  # InfluxDB retention policy
+  #retention_policy = "default"
+  # InfluxDB database
+  #database = "telegraf"
+  # InfluxDB precision
+  #precision = "s"
+`
+
+func (q *AMQP) Connect() error {
+	q.Lock()
+	defer q.Unlock()
+
+	q.headers = amqp.Table{
+		"precision":        q.Precision,
+		"database":         q.Database,
+		"retention_policy": q.RetentionPolicy,
+	}
+
+	connection, err := amqp.Dial(q.URL)
+	if err != nil {
+		return err
+	}
+	channel, err := connection.Channel()
+	if err != nil {
+		return fmt.Errorf("Failed to open a channel: %s", err)
+	}
+
+	err = channel.ExchangeDeclare(
+		q.Exchange, // name
+		"topic",    // type
+		true,       // durable
+		false,      // delete when unused
+		false,      // internal
+		false,      // no-wait
+		nil,        // arguments
+	)
+	if err != nil {
+		return fmt.Errorf("Failed to declare an exchange: %s", err)
+	}
+	q.channel = channel
+	go func() {
+		log.Printf("Closing: %s", <-connection.NotifyClose(make(chan *amqp.Error)))
+		log.Printf("Trying to reconnect")
+		for err := q.Connect(); err != nil; err = q.Connect() {
+			log.Println(err)
+			time.Sleep(10 * time.Second)
+		}
+
+	}()
+	return nil
+}
+
+func (q *AMQP) Close() error {
+	return q.channel.Close()
+}
+
+func (q *AMQP) SampleConfig() string {
+	return sampleConfig
+}
+
+func (q *AMQP) Description() string {
+	return "Configuration for the AMQP server to send metrics to"
+}
+
+func (q *AMQP) Write(points []*client.Point) error {
+	q.Lock()
+	defer q.Unlock()
+	if len(points) == 0 {
+		return nil
+	}
+	var outbuf = make(map[string][][]byte)
+
+	for _, p := range points {
+		// Combine tags from Point and BatchPoints and grab the resulting
+		// line-protocol output string to write to AMQP
+		var value, key string
+		value = p.String()
+
+		if q.RoutingTag != "" {
+			if h, ok := p.Tags()[q.RoutingTag]; ok {
+				key = h
+			}
+		}
+		outbuf[key] = append(outbuf[key], []byte(value))
+
+	}
+	for key, buf := range outbuf {
+		err := q.channel.Publish(
+			q.Exchange, // exchange
+			key,        // routing key
+			false,      // mandatory
+			false,      // immediate
+			amqp.Publishing{
+				Headers:     q.headers,
+				ContentType: "text/plain",
+				Body:        bytes.Join(buf, []byte("\n")),
+			})
+		if err != nil {
+			return fmt.Errorf("FAILED to send amqp message: %s", err)
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("amqp", func() outputs.Output {
+		return &AMQP{
+			Database:        DefaultDatabase,
+			Precision:       DefaultPrecision,
+			RetentionPolicy: DefaultRetentionPolicy,
+		}
+	})
+}

outputs/amqp/amqp_test.go

@@ -0,0 +1,28 @@
+package amqp
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	var url = "amqp://" + testutil.GetLocalHost() + ":5672/"
+	q := &AMQP{
+		URL:      url,
+		Exchange: "telegraf_test",
+	}
+
+	// Verify that we can connect to the AMQP broker
+	err := q.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the amqp broker
+	err = q.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/datadog/README.md

@@ -0,0 +1,9 @@
+# Datadog Output Plugin
+
+This plugin writes to the [Datadog Metrics API](http://docs.datadoghq.com/api/#metrics)
+and requires an `apikey` which can be obtained [here](https://app.datadoghq.com/account/settings#api)
+for the account.
+
+If the point value being sent cannot be converted to a float64, the metric is skipped.
+
+Metrics are grouped by converting any `_` characters to `.` in the Point Name.

outputs/datadog/datadog.go

@@ -0,0 +1,179 @@
+package datadog
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+	"net/url"
+	"sort"
+	"strings"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Datadog struct {
+	Apikey  string
+	Timeout internal.Duration
+
+	apiUrl string
+	client *http.Client
+}
+
+var sampleConfig = `
+  # Datadog API key
+  apikey = "my-secret-key" # required.
+
+  # Connection timeout.
+  # timeout = "5s"
+`
+
+type TimeSeries struct {
+	Series []*Metric `json:"series"`
+}
+
+type Metric struct {
+	Metric string   `json:"metric"`
+	Points [1]Point `json:"points"`
+	Host   string   `json:"host"`
+	Tags   []string `json:"tags,omitempty"`
+}
+
+type Point [2]float64
+
+const datadog_api = "https://app.datadoghq.com/api/v1/series"
+
+func NewDatadog(apiUrl string) *Datadog {
+	return &Datadog{
+		apiUrl: apiUrl,
+	}
+}
+
+func (d *Datadog) Connect() error {
+	if d.Apikey == "" {
+		return fmt.Errorf("apikey is a required field for datadog output")
+	}
+	d.client = &http.Client{
+		Timeout: d.Timeout.Duration,
+	}
+	return nil
+}
+
+func (d *Datadog) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	ts := TimeSeries{}
+	tempSeries := []*Metric{}
+	metricCounter := 0
+
+	for _, pt := range points {
+		mname := strings.Replace(pt.Name(), "_", ".", -1)
+		if amonPts, err := buildPoints(pt); err == nil {
+			for fieldName, amonPt := range amonPts {
+				metric := &Metric{
+					Metric: mname + strings.Replace(fieldName, "_", ".", -1),
+				}
+				metric.Points[0] = amonPt
+				tempSeries = append(tempSeries, metric)
+				metricCounter++
+			}
+		} else {
+			log.Printf("unable to build Metric for %s, skipping\n", pt.Name())
+		}
+	}
+
+	ts.Series = make([]*Metric, metricCounter)
+	copy(ts.Series, tempSeries[0:])
+	tsBytes, err := json.Marshal(ts)
+	if err != nil {
+		return fmt.Errorf("unable to marshal TimeSeries, %s\n", err.Error())
+	}
+	req, err := http.NewRequest("POST", d.authenticatedUrl(), bytes.NewBuffer(tsBytes))
+	if err != nil {
+		return fmt.Errorf("unable to create http.Request, %s\n", err.Error())
+	}
+	req.Header.Add("Content-Type", "application/json")
+
+	resp, err := d.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("error POSTing metrics, %s\n", err.Error())
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode < 200 || resp.StatusCode > 209 {
+		return fmt.Errorf("received bad status code, %d\n", resp.StatusCode)
+	}
+
+	return nil
+}
+
+func (d *Datadog) SampleConfig() string {
+	return sampleConfig
+}
+
+func (d *Datadog) Description() string {
+	return "Configuration for DataDog API to send metrics to."
+}
+
+func (d *Datadog) authenticatedUrl() string {
+	q := url.Values{
+		"api_key": []string{d.Apikey},
+	}
+	return fmt.Sprintf("%s?%s", d.apiUrl, q.Encode())
+}
+
+func buildPoints(pt *client.Point) (map[string]Point, error) {
+	pts := make(map[string]Point)
+	for k, v := range pt.Fields() {
+		var p Point
+		if err := p.setValue(v); err != nil {
+			return pts, fmt.Errorf("unable to extract value from Fields, %s", err.Error())
+		}
+		p[0] = float64(pt.Time().Unix())
+		pts[k] = p
+	}
+	return pts, nil
+}
+
+func buildTags(ptTags map[string]string) []string {
+	tags := make([]string, len(ptTags))
+	index := 0
+	for k, v := range ptTags {
+		tags[index] = fmt.Sprintf("%s:%s", k, v)
+		index += 1
+	}
+	sort.Strings(tags)
+	return tags
+}
+
+func (p *Point) setValue(v interface{}) error {
+	switch d := v.(type) {
+	case int:
+		p[1] = float64(int(d))
+	case int32:
+		p[1] = float64(int32(d))
+	case int64:
+		p[1] = float64(int64(d))
+	case float32:
+		p[1] = float64(d)
+	case float64:
+		p[1] = float64(d)
+	default:
+		return fmt.Errorf("undeterminable type")
+	}
+	return nil
+}
+
+func (d *Datadog) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("datadog", func() outputs.Output {
+		return NewDatadog(datadog_api)
+	})
+}

outputs/datadog/datadog_test.go

@@ -0,0 +1,176 @@
+package datadog
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var (
+	fakeUrl    = "http://test.datadog.com"
+	fakeApiKey = "123456"
+)
+
+func fakeDatadog() *Datadog {
+	d := NewDatadog(fakeUrl)
+	d.Apikey = fakeApiKey
+	return d
+}
+
+func TestUriOverride(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		json.NewEncoder(w).Encode(`{"status":"ok"}`)
+	}))
+	defer ts.Close()
+
+	d := NewDatadog(ts.URL)
+	d.Apikey = "123456"
+	err := d.Connect()
+	require.NoError(t, err)
+	err = d.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}
+
+func TestBadStatusCode(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusInternalServerError)
+		json.NewEncoder(w).Encode(`{ 'errors': [
+    	'Something bad happened to the server.',
+    	'Your query made the server very sad.'
+  		]
+		}`)
+	}))
+	defer ts.Close()
+
+	d := NewDatadog(ts.URL)
+	d.Apikey = "123456"
+	err := d.Connect()
+	require.NoError(t, err)
+	err = d.Write(testutil.MockBatchPoints().Points())
+	if err == nil {
+		t.Errorf("error expected but none returned")
+	} else {
+		require.EqualError(t, fmt.Errorf("received bad status code, 500\n"), err.Error())
+	}
+}
+
+func TestAuthenticatedUrl(t *testing.T) {
+	d := fakeDatadog()
+
+	authUrl := d.authenticatedUrl()
+	assert.EqualValues(t, fmt.Sprintf("%s?api_key=%s", fakeUrl, fakeApiKey), authUrl)
+}
+
+func TestBuildTags(t *testing.T) {
+	var tagtests = []struct {
+		ptIn    map[string]string
+		outTags []string
+	}{
+		{
+			map[string]string{"one": "two", "three": "four"},
+			[]string{"one:two", "three:four"},
+		},
+		{
+			map[string]string{"aaa": "bbb"},
+			[]string{"aaa:bbb"},
+		},
+		{
+			map[string]string{},
+			[]string{},
+		},
+	}
+	for _, tt := range tagtests {
+		tags := buildTags(tt.ptIn)
+		if !reflect.DeepEqual(tags, tt.outTags) {
+			t.Errorf("\nexpected %+v\ngot %+v\n", tt.outTags, tags)
+		}
+	}
+}
+
+func TestBuildPoint(t *testing.T) {
+	var tagtests = []struct {
+		ptIn  *client.Point
+		outPt Point
+		err   error
+	}{
+		{
+			testutil.TestPoint(0.0, "test1"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				0.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(1.0, "test2"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				1.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(10, "test3"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				10.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int32(112345), "test4"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int64(112345), "test5"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float32(11234.5), "test6"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint("11234.5", "test7"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			fmt.Errorf("unable to extract value from Fields, undeterminable type"),
+		},
+	}
+	for _, tt := range tagtests {
+		pt, err := buildPoint(tt.ptIn)
+		if err != nil && tt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", tt.ptIn.Name(), err)
+		}
+		if tt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", tt.ptIn.Name(), tt.err.Error())
+		}
+		if !reflect.DeepEqual(pt, tt.outPt) && tt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", tt.ptIn.Name(), tt.outPt, pt)
+		}
+	}
+}

outputs/influxdb/README.md

@@ -0,0 +1,12 @@
+# InfluxDB Output Plugin
+
+This plugin writes to [InfluxDB](https://www.influxdb.com) via HTTP or UDP.
+
+Required parameters:
+
+* `urls`: List of strings, this is for InfluxDB clustering
+support. On each flush interval, Telegraf will randomly choose one of the urls
+to write to. Each URL should start with either `http://` or `udp://`
+* `database`: The name of the database to write to.
+
+

outputs/influxdb/influxdb.go

@@ -0,0 +1,162 @@
+package influxdb
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"math/rand"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type InfluxDB struct {
+	// URL is only for backwards compatability
+	URL        string
+	URLs       []string `toml:"urls"`
+	Username   string
+	Password   string
+	Database   string
+	UserAgent  string
+	Precision  string
+	Timeout    internal.Duration
+	UDPPayload int `toml:"udp_payload"`
+
+	conns []client.Client
+}
+
+var sampleConfig = `
+  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
+  # Multiple urls can be specified but it is assumed that they are part of the same
+  # cluster, this means that only ONE of the urls will be written to each interval.
+  # urls = ["udp://localhost:8089"] # UDP endpoint example
+  urls = ["http://localhost:8086"] # required
+  # The target database for metrics (telegraf will create it if not exists)
+  database = "telegraf" # required
+  # Precision of writes, valid values are n, u, ms, s, m, and h
+  # note: using second precision greatly helps InfluxDB compression
+  precision = "s"
+
+  # Connection timeout (for the connection with InfluxDB), formatted as a string.
+  # If not provided, will default to 0 (no timeout)
+  # timeout = "5s"
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
+  # user_agent = "telegraf"
+  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
+  # udp_payload = 512
+`
+
+func (i *InfluxDB) Connect() error {
+	var urls []string
+	for _, u := range i.URLs {
+		urls = append(urls, u)
+	}
+
+	// Backward-compatability with single Influx URL config files
+	// This could eventually be removed in favor of specifying the urls as a list
+	if i.URL != "" {
+		urls = append(urls, i.URL)
+	}
+
+	var conns []client.Client
+	for _, u := range urls {
+		switch {
+		case strings.HasPrefix(u, "udp"):
+			parsed_url, err := url.Parse(u)
+			if err != nil {
+				return err
+			}
+
+			if i.UDPPayload == 0 {
+				i.UDPPayload = client.UDPPayloadSize
+			}
+			c, err := client.NewUDPClient(client.UDPConfig{
+				Addr:        parsed_url.Host,
+				PayloadSize: i.UDPPayload,
+			})
+			if err != nil {
+				return err
+			}
+			conns = append(conns, c)
+		default:
+			// If URL doesn't start with "udp", assume HTTP client
+			c, err := client.NewHTTPClient(client.HTTPConfig{
+				Addr:      u,
+				Username:  i.Username,
+				Password:  i.Password,
+				UserAgent: i.UserAgent,
+				Timeout:   i.Timeout.Duration,
+			})
+			if err != nil {
+				return err
+			}
+
+			// Create Database if it doesn't exist
+			_, e := c.Query(client.Query{
+				Command: fmt.Sprintf("CREATE DATABASE IF NOT EXISTS %s", i.Database),
+			})
+
+			if e != nil {
+				log.Println("Database creation failed: " + e.Error())
+			}
+
+			conns = append(conns, c)
+		}
+	}
+
+	i.conns = conns
+	rand.Seed(time.Now().UnixNano())
+	return nil
+}
+
+func (i *InfluxDB) Close() error {
+	// InfluxDB client does not provide a Close() function
+	return nil
+}
+
+func (i *InfluxDB) SampleConfig() string {
+	return sampleConfig
+}
+
+func (i *InfluxDB) Description() string {
+	return "Configuration for influxdb server to send metrics to"
+}
+
+// Choose a random server in the cluster to write to until a successful write
+// occurs, logging each unsuccessful. If all servers fail, return error.
+func (i *InfluxDB) Write(points []*client.Point) error {
+	bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
+		Database:  i.Database,
+		Precision: i.Precision,
+	})
+
+	for _, point := range points {
+		bp.AddPoint(point)
+	}
+
+	// This will get set to nil if a successful write occurs
+	err := errors.New("Could not write to any InfluxDB server in cluster")
+
+	p := rand.Perm(len(i.conns))
+	for _, n := range p {
+		if e := i.conns[n].Write(bp); e != nil {
+			log.Println("ERROR: " + e.Error())
+		} else {
+			err = nil
+			break
+		}
+	}
+	return err
+}
+
+func init() {
+	outputs.Add("influxdb", func() outputs.Output {
+		return &InfluxDB{}
+	})
+}

outputs/influxdb/influxdb_test.go

@@ -0,0 +1,41 @@
+package influxdb
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestUDPInflux(t *testing.T) {
+	i := InfluxDB{
+		URLs: []string{"udp://localhost:8089"},
+	}
+
+	err := i.Connect()
+	require.NoError(t, err)
+	err = i.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}
+
+func TestHTTPInflux(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprintln(w, `{"results":[{}]}`)
+	}))
+	defer ts.Close()
+
+	i := InfluxDB{
+		URLs: []string{ts.URL},
+	}
+
+	err := i.Connect()
+	require.NoError(t, err)
+	err = i.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/kafka/kafka.go

@@ -0,0 +1,85 @@
+package kafka
+
+import (
+	"errors"
+	"fmt"
+
+	"github.com/Shopify/sarama"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Kafka struct {
+	// Kafka brokers to send metrics to
+	Brokers []string
+	// Kafka topic
+	Topic string
+	// Routing Key Tag
+	RoutingTag string `toml:"routing_tag"`
+
+	producer sarama.SyncProducer
+}
+
+var sampleConfig = `
+  # URLs of kafka brokers
+  brokers = ["localhost:9092"]
+  # Kafka topic for producer messages
+  topic = "telegraf"
+  # Telegraf tag to use as a routing key
+  #  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+`
+
+func (k *Kafka) Connect() error {
+	producer, err := sarama.NewSyncProducer(k.Brokers, nil)
+	if err != nil {
+		return err
+	}
+	k.producer = producer
+	return nil
+}
+
+func (k *Kafka) Close() error {
+	return k.producer.Close()
+}
+
+func (k *Kafka) SampleConfig() string {
+	return sampleConfig
+}
+
+func (k *Kafka) Description() string {
+	return "Configuration for the Kafka server to send metrics to"
+}
+
+func (k *Kafka) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	for _, p := range points {
+		// Combine tags from Point and BatchPoints and grab the resulting
+		// line-protocol output string to write to Kafka
+		value := p.String()
+
+		m := &sarama.ProducerMessage{
+			Topic: k.Topic,
+			Value: sarama.StringEncoder(value),
+		}
+		if h, ok := p.Tags()[k.RoutingTag]; ok {
+			m.Key = sarama.StringEncoder(h)
+		}
+
+		_, _, err := k.producer.SendMessage(m)
+		if err != nil {
+			return errors.New(fmt.Sprintf("FAILED to send kafka message: %s\n",
+				err))
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("kafka", func() outputs.Output {
+		return &Kafka{}
+	})
+}

outputs/kafka/kafka_test.go

@@ -0,0 +1,28 @@
+package kafka
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	brokers := []string{testutil.GetLocalHost() + ":9092"}
+	k := &Kafka{
+		Brokers: brokers,
+		Topic:   "Test",
+	}
+
+	// Verify that we can connect to the Kafka broker
+	err := k.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the kafka broker
+	err = k.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/kinesis/README.md

@@ -0,0 +1,61 @@
+## Amazon Kinesis Output for Telegraf
+
+This is an experimental plugin that is still in the early stages of development. It will batch up all of the Points
+in one Put request to Kinesis. This should save the number of API requests by a considerable level.
+
+## About Kinesis
+
+This is not the place to document all of the various Kinesis terms however it
+maybe useful for users to review Amazons official documentation which is available
+[here](http://docs.aws.amazon.com/kinesis/latest/dev/key-concepts.html).
+
+## Amazon Authentication
+
+This plugin uses a credential chain for Authentication with the Kinesis API endpoint. In the following order the plugin
+will attempt to authenticate.
+1. [IAMS Role](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
+2. [Environment Variables](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk)
+3. [Shared Credentials](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk)
+
+
+## Config
+
+For this output plugin to function correctly the following variables must be configured.
+
+* region
+* streamname
+* partitionkey
+
+### region
+
+The region is the Amazon region that you wish to connect to. Examples include but are not limited to
+* us-west-1
+* us-west-2
+* us-east-1
+* ap-southeast-1
+* ap-southeast-2
+
+### streamname
+
+The streamname is used by the plugin to ensure that data is sent to the correct Kinesis stream. It is important to
+note that the stream *MUST* be pre-configured for this plugin to function correctly. If the stream does not exist the
+plugin will result in telegraf exiting with an exit code of 1.
+
+### partitionkey
+
+This is used to group data within a stream. Currently this plugin only supports a single partitionkey.
+Manually configuring different hosts, or groups of hosts with manually selected partitionkeys might be a workable
+solution to scale out.
+
+### format
+
+The format configuration value has been designated to allow people to change the format of the Point as written to
+Kinesis. Right now there are two supported formats string and custom.
+
+#### string
+
+String is defined using the default Point.String() value and translated to []byte for the Kinesis stream.
+
+#### custom
+
+Custom is a string defined by a number of values in the FormatMetric() function.

outputs/kinesis/kinesis.go

@@ -0,0 +1,179 @@
+package kinesis
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"os"
+	"sync/atomic"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
+	"github.com/aws/aws-sdk-go/aws/ec2metadata"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/kinesis"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type KinesisOutput struct {
+	Region       string `toml:"region"`
+	StreamName   string `toml:"streamname"`
+	PartitionKey string `toml:"partitionkey"`
+	Format       string `toml:"format"`
+	Debug        bool   `toml:"debug"`
+	svc          *kinesis.Kinesis
+}
+
+var sampleConfig = `
+  # Amazon REGION of kinesis endpoint.
+  region = "ap-southeast-2"
+  # Kinesis StreamName must exist prior to starting telegraf.
+  streamname = "StreamName"
+  # PartitionKey as used for sharding data.
+  partitionkey = "PartitionKey"
+  # format of the Data payload in the kinesis PutRecord, supported
+  # String and Custom.
+  format = "string"
+  # debug will show upstream aws messages.
+  debug = false
+`
+
+func (k *KinesisOutput) SampleConfig() string {
+	return sampleConfig
+}
+
+func (k *KinesisOutput) Description() string {
+	return "Configuration for the AWS Kinesis output."
+}
+
+func checkstream(l []*string, s string) bool {
+	// Check if the StreamName exists in the slice returned from the ListStreams API request.
+	for _, stream := range l {
+		if *stream == s {
+			return true
+		}
+	}
+	return false
+}
+
+func (k *KinesisOutput) Connect() error {
+	// We attempt first to create a session to Kinesis using an IAMS role, if that fails it will fall through to using
+	// environment variables, and then Shared Credentials.
+	if k.Debug {
+		log.Printf("kinesis: Establishing a connection to Kinesis in %+v", k.Region)
+	}
+	Config := &aws.Config{
+		Region: aws.String(k.Region),
+		Credentials: credentials.NewChainCredentials(
+			[]credentials.Provider{
+				&ec2rolecreds.EC2RoleProvider{Client: ec2metadata.New(session.New())},
+				&credentials.EnvProvider{},
+				&credentials.SharedCredentialsProvider{},
+			}),
+	}
+	svc := kinesis.New(session.New(Config))
+
+	KinesisParams := &kinesis.ListStreamsInput{
+		Limit: aws.Int64(100),
+	}
+
+	resp, err := svc.ListStreams(KinesisParams)
+
+	if err != nil {
+		log.Printf("kinesis: Error in ListSteams API call : %+v \n", err)
+	}
+
+	if checkstream(resp.StreamNames, k.StreamName) {
+		if k.Debug {
+			log.Printf("kinesis: Stream Exists")
+		}
+		k.svc = svc
+		return nil
+	} else {
+		log.Printf("kinesis : You have configured a StreamName %+v which does not exist. exiting.", k.StreamName)
+		os.Exit(1)
+	}
+	return err
+}
+
+func (k *KinesisOutput) Close() error {
+	return errors.New("Error")
+}
+
+func FormatMetric(k *KinesisOutput, point *client.Point) (string, error) {
+	if k.Format == "string" {
+		return point.String(), nil
+	} else {
+		m := fmt.Sprintf("%+v,%+v,%+v",
+			point.Name(),
+			point.Tags(),
+			point.String())
+		return m, nil
+	}
+}
+
+func writekinesis(k *KinesisOutput, r []*kinesis.PutRecordsRequestEntry) time.Duration {
+	start := time.Now()
+	payload := &kinesis.PutRecordsInput{
+		Records:    r,
+		StreamName: aws.String(k.StreamName),
+	}
+
+	if k.Debug {
+		resp, err := k.svc.PutRecords(payload)
+		if err != nil {
+			log.Printf("kinesis: Unable to write to Kinesis : %+v \n", err.Error())
+		}
+		log.Printf("%+v \n", resp)
+
+	} else {
+		_, err := k.svc.PutRecords(payload)
+		if err != nil {
+			log.Printf("kinesis: Unable to write to Kinesis : %+v \n", err.Error())
+		}
+	}
+	return time.Since(start)
+}
+
+func (k *KinesisOutput) Write(points []*client.Point) error {
+	var sz uint32 = 0
+
+	if len(points) == 0 {
+		return nil
+	}
+
+	r := []*kinesis.PutRecordsRequestEntry{}
+
+	for _, p := range points {
+		atomic.AddUint32(&sz, 1)
+
+		metric, _ := FormatMetric(k, p)
+		d := kinesis.PutRecordsRequestEntry{
+			Data:         []byte(metric),
+			PartitionKey: aws.String(k.PartitionKey),
+		}
+		r = append(r, &d)
+
+		if sz == 500 {
+			// Max Messages Per PutRecordRequest is 500
+			elapsed := writekinesis(k, r)
+			log.Printf("Wrote a %+v point batch to Kinesis in %+v.\n", sz, elapsed)
+			atomic.StoreUint32(&sz, 0)
+			r = nil
+		}
+	}
+
+	writekinesis(k, r)
+
+	return nil
+}
+
+func init() {
+	outputs.Add("kinesis", func() outputs.Output {
+		return &KinesisOutput{}
+	})
+}

outputs/kinesis/kinesis_test.go

@@ -0,0 +1,39 @@
+package kinesis
+
+import (
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+	"testing"
+)
+
+func TestFormatMetric(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	k := &KinesisOutput{
+		Format: "string",
+	}
+
+	p := testutil.MockBatchPoints().Points()[0]
+
+	valid_string := "test1,tag1=value1 value=1 1257894000000000000"
+	func_string, err := FormatMetric(k, p)
+
+	if func_string != valid_string {
+		t.Error("Expected ", valid_string)
+	}
+	require.NoError(t, err)
+
+	k = &KinesisOutput{
+		Format: "custom",
+	}
+
+	valid_custom := "test1,map[tag1:value1],test1,tag1=value1 value=1 1257894000000000000"
+	func_custom, err := FormatMetric(k, p)
+
+	if func_custom != valid_custom {
+		t.Error("Expected ", valid_custom)
+	}
+	require.NoError(t, err)
+}

outputs/librato/README.md

@@ -0,0 +1,12 @@
+# Librato Output Plugin
+
+This plugin writes to the [Librato Metrics API](http://dev.librato.com/v1/metrics#metrics)
+and requires an `api_user` and `api_token` which can be obtained [here](https://metrics.librato.com/account/api_tokens)
+for the account.
+
+The `source_tag` option in the Configuration file is used to send contextual information from
+Point Tags to the API.
+
+If the point value being sent cannot be converted to a float64, the metric is skipped.
+
+Currently, the plugin does not send any associated Point Tags.

outputs/librato/librato.go

@@ -0,0 +1,175 @@
+package librato
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Librato struct {
+	ApiUser   string
+	ApiToken  string
+	SourceTag string
+	Timeout   internal.Duration
+
+	apiUrl string
+	client *http.Client
+}
+
+var sampleConfig = `
+  # Librator API Docs
+  # http://dev.librato.com/v1/metrics-authentication
+
+  # Librato API user
+  api_user = "telegraf@influxdb.com" # required.
+
+  # Librato API token
+  api_token = "my-secret-token" # required.
+
+  # Tag Field to populate source attribute (optional)
+  # This is typically the _hostname_ from which the metric was obtained.
+  source_tag = "hostname"
+
+  # Connection timeout.
+  # timeout = "5s"
+`
+
+type Metrics struct {
+	Gauges []*Gauge `json:"gauges"`
+}
+
+type Gauge struct {
+	Name        string  `json:"name"`
+	Value       float64 `json:"value"`
+	Source      string  `json:"source"`
+	MeasureTime int64   `json:"measure_time"`
+}
+
+const librato_api = "https://metrics-api.librato.com/v1/metrics"
+
+func NewLibrato(apiUrl string) *Librato {
+	return &Librato{
+		apiUrl: apiUrl,
+	}
+}
+
+func (l *Librato) Connect() error {
+	if l.ApiUser == "" || l.ApiToken == "" {
+		return fmt.Errorf("api_user and api_token are required fields for librato output")
+	}
+	l.client = &http.Client{
+		Timeout: l.Timeout.Duration,
+	}
+	return nil
+}
+
+func (l *Librato) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	metrics := Metrics{}
+	tempGauges := []*Gauge{}
+	metricCounter := 0
+
+	for _, pt := range points {
+		if gauges, err := l.buildGauges(pt); err == nil {
+			for _, gauge := range gauges {
+				tempGauges = append(tempGauges, gauge)
+				metricCounter++
+			}
+		} else {
+			log.Printf("unable to build Gauge for %s, skipping\n", pt.Name())
+		}
+	}
+
+	metrics.Gauges = make([]*Gauge, metricCounter)
+	copy(metrics.Gauges, tempGauges[0:])
+	metricsBytes, err := json.Marshal(metrics)
+	if err != nil {
+		return fmt.Errorf("unable to marshal Metrics, %s\n", err.Error())
+	}
+	req, err := http.NewRequest("POST", l.apiUrl, bytes.NewBuffer(metricsBytes))
+	if err != nil {
+		return fmt.Errorf("unable to create http.Request, %s\n", err.Error())
+	}
+	req.Header.Add("Content-Type", "application/json")
+	req.SetBasicAuth(l.ApiUser, l.ApiToken)
+
+	resp, err := l.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("error POSTing metrics, %s\n", err.Error())
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != 200 {
+		return fmt.Errorf("received bad status code, %d\n", resp.StatusCode)
+	}
+
+	return nil
+}
+
+func (l *Librato) SampleConfig() string {
+	return sampleConfig
+}
+
+func (l *Librato) Description() string {
+	return "Configuration for Librato API to send metrics to."
+}
+
+func (l *Librato) buildGauges(pt *client.Point) ([]*Gauge, error) {
+	gauges := []*Gauge{}
+	for fieldName, value := range pt.Fields() {
+		gauge := &Gauge{
+			Name:        pt.Name() + "_" + fieldName,
+			MeasureTime: pt.Time().Unix(),
+		}
+		if err := gauge.setValue(value); err != nil {
+			return gauges, fmt.Errorf("unable to extract value from Fields, %s\n",
+				err.Error())
+		}
+		if l.SourceTag != "" {
+			if source, ok := pt.Tags()[l.SourceTag]; ok {
+				gauge.Source = source
+			} else {
+				return gauges,
+					fmt.Errorf("undeterminable Source type from Field, %s\n",
+						l.SourceTag)
+			}
+		}
+	}
+	return gauges, nil
+}
+
+func (g *Gauge) setValue(v interface{}) error {
+	switch d := v.(type) {
+	case int:
+		g.Value = float64(int(d))
+	case int32:
+		g.Value = float64(int32(d))
+	case int64:
+		g.Value = float64(int64(d))
+	case float32:
+		g.Value = float64(d)
+	case float64:
+		g.Value = float64(d)
+	default:
+		return fmt.Errorf("undeterminable type %+v", d)
+	}
+	return nil
+}
+
+func (l *Librato) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("librato", func() outputs.Output {
+		return NewLibrato(librato_api)
+	})
+}

outputs/librato/librato_test.go

@@ -0,0 +1,212 @@
+package librato
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/stretchr/testify/require"
+)
+
+var (
+	fakeUrl   = "http://test.librato.com"
+	fakeUser  = "telegraf@influxdb.com"
+	fakeToken = "123456"
+)
+
+func fakeLibrato() *Librato {
+	l := NewLibrato(fakeUrl)
+	l.ApiUser = fakeUser
+	l.ApiToken = fakeToken
+	return l
+}
+
+func TestUriOverride(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer ts.Close()
+
+	l := NewLibrato(ts.URL)
+	l.ApiUser = "telegraf@influxdb.com"
+	l.ApiToken = "123456"
+	err := l.Connect()
+	require.NoError(t, err)
+	err = l.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}
+
+func TestBadStatusCode(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusServiceUnavailable)
+		json.NewEncoder(w).Encode(`{
+      "errors": {
+        "system": [
+          "The API is currently down for maintenance. It'll be back shortly."
+        ]
+      }
+    }`)
+	}))
+	defer ts.Close()
+
+	l := NewLibrato(ts.URL)
+	l.ApiUser = "telegraf@influxdb.com"
+	l.ApiToken = "123456"
+	err := l.Connect()
+	require.NoError(t, err)
+	err = l.Write(testutil.MockBatchPoints().Points())
+	if err == nil {
+		t.Errorf("error expected but none returned")
+	} else {
+		require.EqualError(t, fmt.Errorf("received bad status code, 503\n"), err.Error())
+	}
+}
+
+func TestBuildGauge(t *testing.T) {
+	var gaugeTests = []struct {
+		ptIn     *client.Point
+		outGauge *Gauge
+		err      error
+	}{
+		{
+			testutil.TestPoint(0.0, "test1"),
+			&Gauge{
+				Name:        "test1",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       0.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(1.0, "test2"),
+			&Gauge{
+				Name:        "test2",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       1.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(10, "test3"),
+			&Gauge{
+				Name:        "test3",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       10.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int32(112345), "test4"),
+			&Gauge{
+				Name:        "test4",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int64(112345), "test5"),
+			&Gauge{
+				Name:        "test5",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float32(11234.5), "test6"),
+			&Gauge{
+				Name:        "test6",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       11234.5,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint("11234.5", "test7"),
+			&Gauge{
+				Name:        "test7",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       11234.5,
+			},
+			fmt.Errorf("unable to extract value from Fields, undeterminable type"),
+		},
+	}
+
+	l := NewLibrato(fakeUrl)
+	for _, gt := range gaugeTests {
+		gauge, err := l.buildGauge(gt.ptIn)
+		if err != nil && gt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", gt.ptIn.Name(), err)
+		}
+		if gt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", gt.ptIn.Name(), gt.err.Error())
+		}
+		if !reflect.DeepEqual(gauge, gt.outGauge) && gt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", gt.ptIn.Name(), gt.outGauge, gauge)
+		}
+	}
+}
+
+func TestBuildGaugeWithSource(t *testing.T) {
+	pt1, _ := client.NewPoint(
+		"test1",
+		map[string]string{"hostname": "192.168.0.1"},
+		map[string]interface{}{"value": 0.0},
+		time.Date(2010, time.November, 10, 23, 0, 0, 0, time.UTC),
+	)
+	pt2, _ := client.NewPoint(
+		"test2",
+		map[string]string{"hostnam": "192.168.0.1"},
+		map[string]interface{}{"value": 1.0},
+		time.Date(2010, time.December, 10, 23, 0, 0, 0, time.UTC),
+	)
+	var gaugeTests = []struct {
+		ptIn     *client.Point
+		outGauge *Gauge
+		err      error
+	}{
+
+		{
+			pt1,
+			&Gauge{
+				Name:        "test1",
+				MeasureTime: time.Date(2010, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       0.0,
+				Source:      "192.168.0.1",
+			},
+			nil,
+		},
+		{
+			pt2,
+			&Gauge{
+				Name:        "test2",
+				MeasureTime: time.Date(2010, time.December, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       1.0,
+			},
+			fmt.Errorf("undeterminable Source type from Field, hostname"),
+		},
+	}
+
+	l := NewLibrato(fakeUrl)
+	l.SourceTag = "hostname"
+	for _, gt := range gaugeTests {
+		gauge, err := l.buildGauge(gt.ptIn)
+		if err != nil && gt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", gt.ptIn.Name(), err)
+		}
+		if gt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", gt.ptIn.Name(), gt.err.Error())
+		}
+		if !reflect.DeepEqual(gauge, gt.outGauge) && gt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", gt.ptIn.Name(), gt.outGauge, gauge)
+		}
+	}
+}

outputs/mqtt/mqtt.go

@@ -0,0 +1,190 @@
+package mqtt
+
+import (
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"io/ioutil"
+	"strings"
+	"sync"
+
+	paho "git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+const MaxClientIdLen = 8
+const MaxRetryCount = 3
+const ClientIdPrefix = "telegraf"
+
+type MQTT struct {
+	Servers     []string `toml:"servers"`
+	Username    string
+	Password    string
+	Database    string
+	Timeout     internal.Duration
+	TopicPrefix string
+
+	Client *paho.Client
+	Opts   *paho.ClientOptions
+	sync.Mutex
+}
+
+var sampleConfig = `
+  servers = ["localhost:1883"] # required.
+
+  # MQTT outputs send metrics to this topic format
+  #    "<topic_prefix>/host/<hostname>/<pluginname>/"
+  #   ex: prefix/host/web01.example.com/mem/available
+  # topic_prefix = "prefix"
+
+  # username and password to connect MQTT server.
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+`
+
+func (m *MQTT) Connect() error {
+	var err error
+	m.Lock()
+	defer m.Unlock()
+
+	m.Opts, err = m.CreateOpts()
+	if err != nil {
+		return err
+	}
+
+	m.Client = paho.NewClient(m.Opts)
+	if token := m.Client.Connect(); token.Wait() && token.Error() != nil {
+		return token.Error()
+	}
+
+	return nil
+}
+
+func (m *MQTT) Close() error {
+	if m.Client.IsConnected() {
+		m.Client.Disconnect(20)
+	}
+	return nil
+}
+
+func (m *MQTT) SampleConfig() string {
+	return sampleConfig
+}
+
+func (m *MQTT) Description() string {
+	return "Configuration for MQTT server to send metrics to"
+}
+
+func (m *MQTT) Write(points []*client.Point) error {
+	m.Lock()
+	defer m.Unlock()
+	if len(points) == 0 {
+		return nil
+	}
+	hostname, ok := points[0].Tags()["host"]
+	if !ok {
+		hostname = ""
+	}
+
+	for _, p := range points {
+		var t []string
+		if m.TopicPrefix != "" {
+			t = append(t, m.TopicPrefix)
+		}
+		tm := strings.Split(p.Name(), "_")
+		if len(tm) < 2 {
+			tm = []string{p.Name(), "stat"}
+		}
+
+		t = append(t, "host", hostname, tm[0], tm[1])
+		topic := strings.Join(t, "/")
+
+		value := p.String()
+		err := m.publish(topic, value)
+		if err != nil {
+			return fmt.Errorf("Could not write to MQTT server, %s", err)
+		}
+	}
+
+	return nil
+}
+
+func (m *MQTT) publish(topic, body string) error {
+	token := m.Client.Publish(topic, 0, false, body)
+	token.Wait()
+	if token.Error() != nil {
+		return token.Error()
+	}
+	return nil
+}
+
+func (m *MQTT) CreateOpts() (*paho.ClientOptions, error) {
+	opts := paho.NewClientOptions()
+
+	clientId := getRandomClientId()
+	opts.SetClientID(clientId)
+
+	TLSConfig := &tls.Config{InsecureSkipVerify: false}
+	ca := "" // TODO
+	scheme := "tcp"
+	if ca != "" {
+		scheme = "ssl"
+		certPool, err := getCertPool(ca)
+		if err != nil {
+			return nil, err
+		}
+		TLSConfig.RootCAs = certPool
+	}
+	TLSConfig.InsecureSkipVerify = true // TODO
+	opts.SetTLSConfig(TLSConfig)
+
+	user := m.Username
+	if user == "" {
+		opts.SetUsername(user)
+	}
+	password := m.Password
+	if password != "" {
+		opts.SetPassword(password)
+	}
+
+	if len(m.Servers) == 0 {
+		return opts, fmt.Errorf("could not get host infomations")
+	}
+	for _, host := range m.Servers {
+		server := fmt.Sprintf("%s://%s", scheme, host)
+
+		opts.AddBroker(server)
+	}
+	opts.SetAutoReconnect(true)
+	return opts, nil
+}
+
+func getRandomClientId() string {
+	const alphanum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+	var bytes = make([]byte, MaxClientIdLen)
+	rand.Read(bytes)
+	for i, b := range bytes {
+		bytes[i] = alphanum[b%byte(len(alphanum))]
+	}
+	return ClientIdPrefix + "-" + string(bytes)
+}
+
+func getCertPool(pemPath string) (*x509.CertPool, error) {
+	certs := x509.NewCertPool()
+
+	pemData, err := ioutil.ReadFile(pemPath)
+	if err != nil {
+		return nil, err
+	}
+	certs.AppendCertsFromPEM(pemData)
+	return certs, nil
+}
+
+func init() {
+	outputs.Add("mqtt", func() outputs.Output {
+		return &MQTT{}
+	})
+}

outputs/mqtt/mqtt_test.go

@@ -0,0 +1,27 @@
+package mqtt
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	var url = testutil.GetLocalHost() + ":1883"
+	m := &MQTT{
+		Servers: []string{url},
+	}
+
+	// Verify that we can connect to the MQTT broker
+	err := m.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the mqtt broker
+	err = m.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/nsq/README.md

@@ -0,0 +1,4 @@
+# NSQ Output Plugin
+
+This plugin writes to a specified NSQD instance, usually local to the producer. It requires
+a `server` name and a `topic` name.

outputs/nsq/nsq.go

@@ -0,0 +1,71 @@
+package nsq
+
+import (
+	"fmt"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/nsqio/go-nsq"
+)
+
+type NSQ struct {
+	Server   string
+	Topic    string
+	producer *nsq.Producer
+}
+
+var sampleConfig = `
+  # Location of nsqd instance listening on TCP
+  server = "localhost:4150"
+  # NSQ topic for producer messages
+  topic = "telegraf"
+`
+
+func (n *NSQ) Connect() error {
+	config := nsq.NewConfig()
+	producer, err := nsq.NewProducer(n.Server, config)
+
+	if err != nil {
+		return err
+	}
+
+	n.producer = producer
+	return nil
+}
+
+func (n *NSQ) Close() error {
+	n.producer.Stop()
+	return nil
+}
+
+func (n *NSQ) SampleConfig() string {
+	return sampleConfig
+}
+
+func (n *NSQ) Description() string {
+	return "Send telegraf measurements to NSQD"
+}
+
+func (n *NSQ) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	for _, p := range points {
+		// Combine tags from Point and BatchPoints and grab the resulting
+		// line-protocol output string to write to NSQ
+		value := p.String()
+
+		err := n.producer.Publish(n.Topic, []byte(value))
+
+		if err != nil {
+			return fmt.Errorf("FAILED to send NSQD message: %s", err)
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("nsq", func() outputs.Output {
+		return &NSQ{}
+	})
+}

outputs/nsq/nsq_test.go

@@ -0,0 +1,28 @@
+package nsq
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	server := []string{testutil.GetLocalHost() + ":4150"}
+	n := &NSQ{
+		Server: server[0],
+		Topic:  "telegraf",
+	}
+
+	// Verify that we can connect to the NSQ daemon
+	err := n.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the NSQ daemon
+	err = n.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/opentsdb/README.md

@@ -0,0 +1,78 @@
+# OpenTSDB Output Plugin
+
+This plugin writes to a OpenTSDB instance using the "telnet" mode
+
+## Transfer "Protocol" in the telnet mode
+
+The expected input from OpenTSDB is specified in the following way:
+
+```
+put <metric> <timestamp> <value> <tagk1=tagv1[ tagk2=tagv2 ...tagkN=tagvN]>
+```
+
+The telegraf output plugin adds an optional prefix to the metric keys so 
+that a subamount can be selected.
+
+```
+put <[prefix.]metric> <timestamp> <value> <tagk1=tagv1[ tagk2=tagv2 ...tagkN=tagvN]>
+```
+
+### Example 
+
+```
+put nine.telegraf.system_load1 1441910356 0.430000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_load5 1441910356 0.580000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_load15 1441910356 0.730000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_uptime 1441910356 3655970.000000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_uptime_format 1441910356  dc=homeoffice host=irimame scope=green
+put nine.telegraf.mem_total 1441910356 4145426432 dc=homeoffice host=irimame scope=green
+...
+put nine.telegraf.io_write_bytes 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.io_read_time 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.io_write_time 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.io_io_time 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.ping_packets_transmitted 1441910366  dc=homeoffice host=irimame scope=green url=www.google.com
+put nine.telegraf.ping_packets_received 1441910366  dc=homeoffice host=irimame scope=green url=www.google.com
+put nine.telegraf.ping_percent_packet_loss 1441910366 0.000000 dc=homeoffice host=irimame scope=green url=www.google.com
+put nine.telegraf.ping_average_response_ms 1441910366 24.006000 dc=homeoffice host=irimame scope=green url=www.google.com
+...
+```
+
+## 
+
+The OpenTSDB interface can be simulated with this reader:
+
+```
+// opentsdb_telnet_mode_mock.go 
+package main
+
+import (
+	"io"
+	"log"
+	"net"
+	"os"
+)
+
+func main() {
+	l, err := net.Listen("tcp", "localhost:4242")
+	if err != nil {
+		log.Fatal(err)
+	}
+	defer l.Close()
+	for {
+		conn, err := l.Accept()
+		if err != nil {
+			log.Fatal(err)
+		}
+		go func(c net.Conn) {
+			defer c.Close()
+			io.Copy(os.Stdout, c)
+		}(conn)
+	}
+}
+
+```
+
+## Allowed values for metrics
+
+OpenTSDB allows `integers` and `floats` as input values

outputs/opentsdb/opentsdb.go

@@ -0,0 +1,168 @@
+package opentsdb
+
+import (
+	"fmt"
+	"net"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type OpenTSDB struct {
+	Prefix string
+
+	Host string
+	Port int
+
+	Debug bool
+}
+
+var sampleConfig = `
+  # prefix for metrics keys
+  prefix = "my.specific.prefix."
+
+  ## Telnet Mode ##
+  # DNS name of the OpenTSDB server in telnet mode
+  host = "opentsdb.example.com"
+
+  # Port of the OpenTSDB server in telnet mode
+  port = 4242
+
+  # Debug true - Prints OpenTSDB communication
+  debug = false
+`
+
+type MetricLine struct {
+	Metric    string
+	Timestamp int64
+	Value     string
+	Tags      string
+}
+
+func (o *OpenTSDB) Connect() error {
+	// Test Connection to OpenTSDB Server
+	uri := fmt.Sprintf("%s:%d", o.Host, o.Port)
+	tcpAddr, err := net.ResolveTCPAddr("tcp", uri)
+	if err != nil {
+		return fmt.Errorf("OpenTSDB: TCP address cannot be resolved")
+	}
+	connection, err := net.DialTCP("tcp", nil, tcpAddr)
+	if err != nil {
+		return fmt.Errorf("OpenTSDB: Telnet connect fail")
+	}
+	defer connection.Close()
+	return nil
+}
+
+func (o *OpenTSDB) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	now := time.Now()
+
+	// Send Data with telnet / socket communication
+	uri := fmt.Sprintf("%s:%d", o.Host, o.Port)
+	tcpAddr, _ := net.ResolveTCPAddr("tcp", uri)
+	connection, err := net.DialTCP("tcp", nil, tcpAddr)
+	if err != nil {
+		return fmt.Errorf("OpenTSDB: Telnet connect fail")
+	}
+	defer connection.Close()
+
+	for _, pt := range points {
+		for _, metric := range buildMetrics(pt, now, o.Prefix) {
+			messageLine := fmt.Sprintf("put %s %v %s %s\n",
+				metric.Metric, metric.Timestamp, metric.Value, metric.Tags)
+			if o.Debug {
+				fmt.Print(messageLine)
+			}
+			_, err := connection.Write([]byte(messageLine))
+			if err != nil {
+				return fmt.Errorf("OpenTSDB: Telnet writing error %s", err.Error())
+			}
+		}
+	}
+
+	return nil
+}
+
+func buildTags(ptTags map[string]string) []string {
+	tags := make([]string, len(ptTags))
+	index := 0
+	for k, v := range ptTags {
+		tags[index] = fmt.Sprintf("%s=%s", k, v)
+		index += 1
+	}
+	sort.Strings(tags)
+	return tags
+}
+
+func buildMetrics(pt *client.Point, now time.Time, prefix string) []*MetricLine {
+	ret := []*MetricLine{}
+	for fieldName, value := range pt.Fields() {
+		metric := &MetricLine{
+			Metric:    fmt.Sprintf("%s%s_%s", prefix, pt.Name(), fieldName),
+			Timestamp: now.Unix(),
+		}
+
+		metricValue, buildError := buildValue(value)
+		if buildError != nil {
+			fmt.Printf("OpenTSDB: %s\n", buildError.Error())
+			continue
+		}
+		metric.Value = metricValue
+		tagsSlice := buildTags(pt.Tags())
+		metric.Tags = fmt.Sprint(strings.Join(tagsSlice, " "))
+		ret = append(ret, metric)
+	}
+	return ret
+}
+
+func buildValue(v interface{}) (string, error) {
+	var retv string
+	switch p := v.(type) {
+	case int64:
+		retv = IntToString(int64(p))
+	case uint64:
+		retv = UIntToString(uint64(p))
+	case float64:
+		retv = FloatToString(float64(p))
+	default:
+		return retv, fmt.Errorf("unexpected type %T with value %v for OpenTSDB", v, v)
+	}
+	return retv, nil
+}
+
+func IntToString(input_num int64) string {
+	return strconv.FormatInt(input_num, 10)
+}
+
+func UIntToString(input_num uint64) string {
+	return strconv.FormatUint(input_num, 10)
+}
+
+func FloatToString(input_num float64) string {
+	return strconv.FormatFloat(input_num, 'f', 6, 64)
+}
+
+func (o *OpenTSDB) SampleConfig() string {
+	return sampleConfig
+}
+
+func (o *OpenTSDB) Description() string {
+	return "Configuration for OpenTSDB server to send metrics to"
+}
+
+func (o *OpenTSDB) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("opentsdb", func() outputs.Output {
+		return &OpenTSDB{}
+	})
+}

outputs/opentsdb/opentsdb_test.go

@@ -0,0 +1,71 @@
+package opentsdb
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestBuildTagsTelnet(t *testing.T) {
+	var tagtests = []struct {
+		ptIn    map[string]string
+		outTags []string
+	}{
+		{
+			map[string]string{"one": "two", "three": "four"},
+			[]string{"one=two", "three=four"},
+		},
+		{
+			map[string]string{"aaa": "bbb"},
+			[]string{"aaa=bbb"},
+		},
+		{
+			map[string]string{"one": "two", "aaa": "bbb"},
+			[]string{"aaa=bbb", "one=two"},
+		},
+		{
+			map[string]string{},
+			[]string{},
+		},
+	}
+	for _, tt := range tagtests {
+		tags := buildTags(tt.ptIn)
+		if !reflect.DeepEqual(tags, tt.outTags) {
+			t.Errorf("\nexpected %+v\ngot %+v\n", tt.outTags, tags)
+		}
+	}
+}
+
+func TestWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	o := &OpenTSDB{
+		Host:   testutil.GetLocalHost(),
+		Port:   4242,
+		Prefix: "prefix.test.",
+	}
+
+	// Verify that we can connect to the OpenTSDB instance
+	err := o.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to OpenTSDB
+	err = o.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+
+	// Verify postive and negative test cases of writing data
+	bp := testutil.MockBatchPoints()
+	bp.AddPoint(testutil.TestPoint(float64(1.0), "justametric.float"))
+	bp.AddPoint(testutil.TestPoint(int64(123456789), "justametric.int"))
+	bp.AddPoint(testutil.TestPoint(uint64(123456789012345), "justametric.uint"))
+	bp.AddPoint(testutil.TestPoint("Lorem Ipsum", "justametric.string"))
+	bp.AddPoint(testutil.TestPoint(float64(42.0), "justametric.anotherfloat"))
+
+	err = o.Write(bp.Points())
+	require.NoError(t, err)
+
+}

outputs/prometheus_client/README.md

@@ -0,0 +1,6 @@
+# Prometheus Client Service Output Plugin
+
+This plugin starts a Prometheus Client, listening on a port defined in the
+configuration file.
+
+It exposes all metrics on `/metrics` to be polled by a Prometheus server.

outputs/prometheus_client/prometheus_client.go

@@ -0,0 +1,125 @@
+package prometheus_client
+
+import (
+	"fmt"
+	"log"
+	"net/http"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+type PrometheusClient struct {
+	Listen  string
+	metrics map[string]*prometheus.UntypedVec
+}
+
+var sampleConfig = `
+  # Address to listen on
+  # listen = ":9126"
+`
+
+func (p *PrometheusClient) Start() error {
+	if p.Listen == "" {
+		p.Listen = "localhost:9126"
+	}
+
+	http.Handle("/metrics", prometheus.Handler())
+	server := &http.Server{
+		Addr: p.Listen,
+	}
+
+	p.metrics = make(map[string]*prometheus.UntypedVec)
+	go server.ListenAndServe()
+	return nil
+}
+
+func (p *PrometheusClient) Stop() {
+	// TODO: Use a listener for http.Server that counts active connections
+	//       that can be stopped and closed gracefully
+}
+
+func (p *PrometheusClient) Connect() error {
+	// This service output does not need to make any further connections
+	return nil
+}
+
+func (p *PrometheusClient) Close() error {
+	// This service output does not need to close any of its connections
+	return nil
+}
+
+func (p *PrometheusClient) SampleConfig() string {
+	return sampleConfig
+}
+
+func (p *PrometheusClient) Description() string {
+	return "Configuration for the Prometheus client to spawn"
+}
+
+func (p *PrometheusClient) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	for _, point := range points {
+		var labels []string
+		key := point.Name()
+
+		for k, _ := range point.Tags() {
+			if len(k) > 0 {
+				labels = append(labels, k)
+			}
+		}
+
+		if _, ok := p.metrics[key]; !ok {
+			p.metrics[key] = prometheus.NewUntypedVec(
+				prometheus.UntypedOpts{
+					Name: key,
+					Help: fmt.Sprintf("Telegraf collected point '%s'", key),
+				},
+				labels,
+			)
+			prometheus.MustRegister(p.metrics[key])
+		}
+
+		l := prometheus.Labels{}
+		for tk, tv := range point.Tags() {
+			l[tk] = tv
+		}
+
+		for _, val := range point.Fields() {
+			switch val := val.(type) {
+			default:
+				log.Printf("Prometheus output, unsupported type. key: %s, type: %T\n",
+					key, val)
+			case int64:
+				m, err := p.metrics[key].GetMetricWith(l)
+				if err != nil {
+					log.Printf("ERROR Getting metric in Prometheus output, "+
+						"key: %s, labels: %v,\nerr: %s\n",
+						key, l, err.Error())
+					continue
+				}
+				m.Set(float64(val))
+			case float64:
+				m, err := p.metrics[key].GetMetricWith(l)
+				if err != nil {
+					log.Printf("ERROR Getting metric in Prometheus output, "+
+						"key: %s, labels: %v,\nerr: %s\n",
+						key, l, err.Error())
+					continue
+				}
+				m.Set(val)
+			}
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("prometheus_client", func() outputs.Output {
+		return &PrometheusClient{}
+	})
+}

outputs/prometheus_client/prometheus_client_test.go

@@ -0,0 +1,98 @@
+package prometheus_client
+
+import (
+	"testing"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/plugins/prometheus"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var pTesting *PrometheusClient
+
+func TestPrometheusWritePointEmptyTag(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	p := &prometheus.Prometheus{
+		Urls: []string{"http://localhost:9126/metrics"},
+	}
+	tags := make(map[string]string)
+	pt1, _ := client.NewPoint(
+		"test_point_1",
+		tags,
+		map[string]interface{}{"value": 0.0})
+	pt2, _ := client.NewPoint(
+		"test_point_2",
+		tags,
+		map[string]interface{}{"value": 1.0})
+	var points = []*client.Point{
+		pt1,
+		pt2,
+	}
+	require.NoError(t, pTesting.Write(points))
+
+	expected := []struct {
+		name  string
+		value float64
+		tags  map[string]string
+	}{
+		{"test_point_1", 0.0, tags},
+		{"test_point_2", 1.0, tags},
+	}
+
+	var acc testutil.Accumulator
+
+	require.NoError(t, p.Gather(&acc))
+	for _, e := range expected {
+		assert.NoError(t, acc.ValidateValue(e.name, e.value))
+	}
+}
+
+func TestPrometheusWritePointTag(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	p := &prometheus.Prometheus{
+		Urls: []string{"http://localhost:9126/metrics"},
+	}
+	tags := make(map[string]string)
+	tags["testtag"] = "testvalue"
+	pt1, _ := client.NewPoint(
+		"test_point_3",
+		tags,
+		map[string]interface{}{"value": 0.0})
+	pt2, _ := client.NewPoint(
+		"test_point_4",
+		tags,
+		map[string]interface{}{"value": 1.0})
+	var points = []*client.Point{
+		pt1,
+		pt2,
+	}
+	require.NoError(t, pTesting.Write(points))
+
+	expected := []struct {
+		name  string
+		value float64
+	}{
+		{"test_point_3", 0.0},
+		{"test_point_4", 1.0},
+	}
+
+	var acc testutil.Accumulator
+
+	require.NoError(t, p.Gather(&acc))
+	for _, e := range expected {
+		assert.True(t, acc.CheckTaggedValue(e.name, e.value, tags))
+	}
+}
+
+func init() {
+	pTesting = &PrometheusClient{Listen: "localhost:9126"}
+	pTesting.Start()
+}

outputs/registry.go

@@ -0,0 +1,43 @@
+package outputs
+
+import (
+	"github.com/influxdb/influxdb/client/v2"
+)
+
+type Output interface {
+	// Connect to the Output
+	Connect() error
+	// Close any connections to the Output
+	Close() error
+	// Description returns a one-sentence description on the Output
+	Description() string
+	// SampleConfig returns the default configuration of the Output
+	SampleConfig() string
+	// Write takes in group of points to be written to the Output
+	Write(points []*client.Point) error
+}
+
+type ServiceOutput interface {
+	// Connect to the Output
+	Connect() error
+	// Close any connections to the Output
+	Close() error
+	// Description returns a one-sentence description on the Output
+	Description() string
+	// SampleConfig returns the default configuration of the Output
+	SampleConfig() string
+	// Write takes in group of points to be written to the Output
+	Write(points []*client.Point) error
+	// Start the "service" that will provide an Output
+	Start() error
+	// Stop the "service" that will provide an Output
+	Stop()
+}
+
+type Creator func() Output
+
+var Outputs = map[string]Creator{}
+
+func Add(name string, creator Creator) {
+	Outputs[name] = creator
+}

outputs/riemann/riemann.go

@@ -0,0 +1,101 @@
+package riemann
+
+import (
+	"errors"
+	"fmt"
+	"os"
+
+	"github.com/amir/raidman"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Riemann struct {
+	URL       string
+	Transport string
+
+	client *raidman.Client
+}
+
+var sampleConfig = `
+  # URL of server
+  url = "localhost:5555"
+  # transport protocol to use either tcp or udp
+  transport = "tcp"
+`
+
+func (r *Riemann) Connect() error {
+	c, err := raidman.Dial(r.Transport, r.URL)
+
+	if err != nil {
+		return err
+	}
+
+	r.client = c
+	return nil
+}
+
+func (r *Riemann) Close() error {
+	r.client.Close()
+	return nil
+}
+
+func (r *Riemann) SampleConfig() string {
+	return sampleConfig
+}
+
+func (r *Riemann) Description() string {
+	return "Configuration for the Riemann server to send metrics to"
+}
+
+func (r *Riemann) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	var events []*raidman.Event
+	for _, p := range points {
+		evs := buildEvents(p)
+		for _, ev := range evs {
+			events = append(events, ev)
+		}
+	}
+
+	var senderr = r.client.SendMulti(events)
+	if senderr != nil {
+		return errors.New(fmt.Sprintf("FAILED to send riemann message: %s\n",
+			senderr))
+	}
+
+	return nil
+}
+
+func buildEvents(p *client.Point) []*raidman.Event {
+	events := []*raidman.Event{}
+	for fieldName, value := range p.Fields() {
+		host, ok := p.Tags()["host"]
+		if !ok {
+			hostname, err := os.Hostname()
+			if err != nil {
+				host = "unknown"
+			} else {
+				host = hostname
+			}
+		}
+
+		event := &raidman.Event{
+			Host:    host,
+			Service: p.Name() + "_" + fieldName,
+			Metric:  value,
+		}
+		events = append(events, event)
+	}
+
+	return events
+}
+
+func init() {
+	outputs.Add("riemann", func() outputs.Output {
+		return &Riemann{}
+	})
+}

outputs/riemann/riemann_test.go

@@ -0,0 +1,27 @@
+package riemann
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	url := testutil.GetLocalHost() + ":5555"
+
+	r := &Riemann{
+		URL:       url,
+		Transport: "tcp",
+	}
+
+	err := r.Connect()
+	require.NoError(t, err)
+
+	err = r.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

plugins/aerospike/README.md

@@ -0,0 +1,265 @@
+## Telegraf Plugin: Aerospike
+
+#### Plugin arguments:
+- **servers** string array: List of aerospike servers to query (def: 127.0.0.1:3000)
+
+#### Description
+
+The aerospike plugin queries aerospike server(s) and get node statistics. It also collects stats for
+all the configured namespaces.
+
+For what the measurements mean, please consult the [Aerospike Metrics Reference Docs](http://www.aerospike.com/docs/reference/metrics).
+
+The metric names, to make it less complicated in querying, have replaced all `-` with `_` as Aerospike metrics come in both forms (no idea why).
+
+# Measurements:
+#### Aerospike Statistics [values]:
+
+Meta:
+- units: Integer
+
+Measurement names:
+- batch_index_queue
+- batch_index_unused_buffers
+- batch_queue
+- batch_tree_count
+- client_connections
+- data_used_bytes_memory
+- index_used_bytes_memory
+- info_queue
+- migrate_progress_recv
+- migrate_progress_send
+- migrate_rx_objs
+- migrate_tx_objs
+- objects
+- ongoing_write_reqs
+- partition_absent
+- partition_actual
+- partition_desync
+- partition_object_count
+- partition_ref_count
+- partition_replica
+- proxy_in_progress
+- query_agg_avg_rec_count
+- query_avg_rec_count
+- query_lookup_avg_rec_count
+- queue
+- record_locks
+- record_refs
+- sindex_used_bytes_memory
+- sindex_gc_garbage_cleaned
+- system_free_mem_pct
+- total_bytes_disk
+- total_bytes_memory
+- tree_count
+- scans_active
+- uptime
+- used_bytes_disk
+- used_bytes_memory
+- cluster_size
+- waiting_transactions
+
+#### Aerospike Statistics [cumulative]:
+
+Meta:
+- units: Integer
+
+Measurement names:
+- batch_errors
+- batch_index_complete
+- batch_index_errors
+- batch_index_initiate
+- batch_index_timeout
+- batch_initiate
+- batch_timeout
+- err_duplicate_proxy_request
+- err_out_of_space
+- err_replica_non_null_node
+- err_replica_null_node
+- err_rw_cant_put_unique
+- err_rw_pending_limit
+- err_rw_request_not_found
+- err_storage_queue_full
+- err_sync_copy_null_master
+- err_sync_copy_null_node
+- err_tsvc_requests
+- err_write_fail_bin_exists
+- err_write_fail_generation
+- err_write_fail_generation_xdr
+- err_write_fail_incompatible_type
+- err_write_fail_key_exists
+- err_write_fail_key_mismatch
+- err_write_fail_not_found
+- err_write_fail_noxdr
+- err_write_fail_parameter
+- err_write_fail_prole_delete
+- err_write_fail_prole_generation
+- err_write_fail_prole_unknown
+- err_write_fail_unknown
+- fabric_msgs_rcvd
+- fabric_msgs_sent
+- heartbeat_received_foreign
+- heartbeat_received_self
+- migrate_msgs_recv
+- migrate_msgs_sent
+- migrate_num_incoming_accepted
+- migrate_num_incoming_refused
+- proxy_action
+- proxy_initiate
+- proxy_retry
+- proxy_retry_new_dest
+- proxy_retry_q_full
+- proxy_retry_same_dest
+- proxy_unproxy
+- query_abort
+- query_agg
+- query_agg_abort
+- query_agg_err
+- query_agg_success
+- query_bad_records
+- query_fail
+- query_long_queue_full
+- query_long_running
+- query_lookup_abort
+- query_lookup_err
+- query_lookups
+- query_lookup_success
+- query_reqs
+- query_short_queue_full
+- query_short_running
+- query_success
+- query_tracked
+- read_dup_prole
+- reaped_fds
+- rw_err_ack_badnode
+- rw_err_ack_internal
+- rw_err_ack_nomatch
+- rw_err_dup_cluster_key
+- rw_err_dup_internal
+- rw_err_dup_send
+- rw_err_write_cluster_key
+- rw_err_write_internal
+- rw_err_write_send
+- sindex_ucgarbage_found
+- sindex_gc_locktimedout
+- sindex_gc_inactivity_dur
+- sindex_gc_activity_dur
+- sindex_gc_list_creation_time
+- sindex_gc_list_deletion_time
+- sindex_gc_objects_validated
+- sindex_gc_garbage_found
+- stat_cluster_key_err_ack_dup_trans_reenqueue
+- stat_cluster_key_err_ack_rw_trans_reenqueue
+- stat_cluster_key_prole_retry
+- stat_cluster_key_regular_processed
+- stat_cluster_key_trans_to_proxy_retry
+- stat_deleted_set_object
+- stat_delete_success
+- stat_duplicate_operation
+- stat_evicted_objects
+- stat_evicted_objects_time
+- stat_evicted_set_objects
+- stat_expired_objects
+- stat_nsup_deletes_not_shipped
+- stat_proxy_errs
+- stat_proxy_reqs
+- stat_proxy_reqs_xdr
+- stat_proxy_success
+- stat_read_errs_notfound
+- stat_read_errs_other
+- stat_read_reqs
+- stat_read_reqs_xdr
+- stat_read_success
+- stat_rw_timeout
+- stat_slow_trans_queue_batch_pop
+- stat_slow_trans_queue_pop
+- stat_slow_trans_queue_push
+- stat_write_errs
+- stat_write_errs_notfound
+- stat_write_errs_other
+- stat_write_reqs
+- stat_write_reqs_xdr
+- stat_write_success
+- stat_xdr_pipe_miss
+- stat_xdr_pipe_writes
+- stat_zero_bin_records
+- storage_defrag_corrupt_record
+- storage_defrag_wait
+- transactions
+- basic_scans_succeeded
+- basic_scans_failed
+- aggr_scans_succeeded
+- aggr_scans_failed
+- udf_bg_scans_succeeded
+- udf_bg_scans_failed
+- udf_delete_err_others
+- udf_delete_reqs
+- udf_delete_success
+- udf_lua_errs
+- udf_query_rec_reqs
+- udf_read_errs_other
+- udf_read_reqs
+- udf_read_success
+- udf_replica_writes
+- udf_scan_rec_reqs
+- udf_write_err_others
+- udf_write_reqs
+- udf_write_success
+- write_master
+- write_prole
+
+#### Aerospike Statistics [percentage]:
+
+Meta:
+- units: percent (out of 100)
+
+Measurement names:
+- free_pct_disk
+- free_pct_memory
+
+# Measurements:
+#### Aerospike Namespace Statistics [values]:
+
+Meta:
+- units: Integer
+- tags: `namespace=<namespace>`
+
+Measurement names:
+- available_bin_names
+- available_pct
+- current_time
+- data_used_bytes_memory
+- index_used_bytes_memory
+- master_objects
+- max_evicted_ttl
+- max_void_time
+- non_expirable_objects
+- objects
+- prole_objects
+- sindex_used_bytes_memory
+- total_bytes_disk
+- total_bytes_memory
+- used_bytes_disk
+- used_bytes_memory
+
+#### Aerospike Namespace Statistics [cumulative]:
+
+Meta:
+- units: Integer
+- tags: `namespace=<namespace>`
+
+Measurement names:
+- evicted_objects
+- expired_objects
+- set_deleted_objects
+- set_evicted_objects
+
+#### Aerospike Namespace Statistics [percentage]:
+
+Meta:
+- units: percent (out of 100)
+- tags: `namespace=<namespace>`
+
+Measurement names:
+- free_pct_disk
+- free_pct_memory

plugins/aerospike/aerospike.go

@@ -0,0 +1,342 @@
+package aerospike
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"github.com/influxdb/telegraf/plugins"
+	"net"
+	"strconv"
+	"strings"
+	"sync"
+)
+
+const (
+	MSG_HEADER_SIZE = 8
+	MSG_TYPE        = 1 // Info is 1
+	MSG_VERSION     = 2
+)
+
+var (
+	STATISTICS_COMMAND = []byte("statistics\n")
+	NAMESPACES_COMMAND = []byte("namespaces\n")
+)
+
+type aerospikeMessageHeader struct {
+	Version uint8
+	Type    uint8
+	DataLen [6]byte
+}
+
+type aerospikeMessage struct {
+	aerospikeMessageHeader
+	Data []byte
+}
+
+// Taken from aerospike-client-go/types/message.go
+func (msg *aerospikeMessage) Serialize() []byte {
+	msg.DataLen = msgLenToBytes(int64(len(msg.Data)))
+	buf := bytes.NewBuffer([]byte{})
+	binary.Write(buf, binary.BigEndian, msg.aerospikeMessageHeader)
+	binary.Write(buf, binary.BigEndian, msg.Data[:])
+	return buf.Bytes()
+}
+
+type aerospikeInfoCommand struct {
+	msg *aerospikeMessage
+}
+
+// Taken from aerospike-client-go/info.go
+func (nfo *aerospikeInfoCommand) parseMultiResponse() (map[string]string, error) {
+	responses := make(map[string]string)
+	offset := int64(0)
+	begin := int64(0)
+
+	dataLen := int64(len(nfo.msg.Data))
+
+	// Create reusable StringBuilder for performance.
+	for offset < dataLen {
+		b := nfo.msg.Data[offset]
+
+		if b == '\t' {
+			name := nfo.msg.Data[begin:offset]
+			offset++
+			begin = offset
+
+			// Parse field value.
+			for offset < dataLen {
+				if nfo.msg.Data[offset] == '\n' {
+					break
+				}
+				offset++
+			}
+
+			if offset > begin {
+				value := nfo.msg.Data[begin:offset]
+				responses[string(name)] = string(value)
+			} else {
+				responses[string(name)] = ""
+			}
+			offset++
+			begin = offset
+		} else if b == '\n' {
+			if offset > begin {
+				name := nfo.msg.Data[begin:offset]
+				responses[string(name)] = ""
+			}
+			offset++
+			begin = offset
+		} else {
+			offset++
+		}
+	}
+
+	if offset > begin {
+		name := nfo.msg.Data[begin:offset]
+		responses[string(name)] = ""
+	}
+	return responses, nil
+}
+
+type Aerospike struct {
+	Servers []string
+}
+
+var sampleConfig = `
+  # Aerospike servers to connect to (with port)
+  # Default: servers = ["localhost:3000"]
+  #
+  # This plugin will query all namespaces the aerospike
+  # server has configured and get stats for them.
+  servers = ["localhost:3000"]
+ `
+
+func (a *Aerospike) SampleConfig() string {
+	return sampleConfig
+}
+
+func (a *Aerospike) Description() string {
+	return "Read stats from an aerospike server"
+}
+
+func (a *Aerospike) Gather(acc plugins.Accumulator) error {
+	if len(a.Servers) == 0 {
+		return a.gatherServer("127.0.0.1:3000", acc)
+	}
+
+	var wg sync.WaitGroup
+
+	var outerr error
+
+	for _, server := range a.Servers {
+		wg.Add(1)
+		go func(server string) {
+			defer wg.Done()
+			outerr = a.gatherServer(server, acc)
+		}(server)
+	}
+
+	wg.Wait()
+	return outerr
+}
+
+func (a *Aerospike) gatherServer(host string, acc plugins.Accumulator) error {
+	aerospikeInfo, err := getMap(STATISTICS_COMMAND, host)
+	if err != nil {
+		return fmt.Errorf("Aerospike info failed: %s", err)
+	}
+	readAerospikeStats(aerospikeInfo, acc, host, "")
+	namespaces, err := getList(NAMESPACES_COMMAND, host)
+	if err != nil {
+		return fmt.Errorf("Aerospike namespace list failed: %s", err)
+	}
+	for ix := range namespaces {
+		nsInfo, err := getMap([]byte("namespace/"+namespaces[ix]+"\n"), host)
+		if err != nil {
+			return fmt.Errorf("Aerospike namespace '%s' query failed: %s", namespaces[ix], err)
+		}
+		readAerospikeStats(nsInfo, acc, host, namespaces[ix])
+	}
+	return nil
+}
+
+func getMap(key []byte, host string) (map[string]string, error) {
+	data, err := get(key, host)
+	if err != nil {
+		return nil, fmt.Errorf("Failed to get data: %s", err)
+	}
+	parsed, err := unmarshalMapInfo(data, string(key))
+	if err != nil {
+		return nil, fmt.Errorf("Failed to unmarshal data: %s", err)
+	}
+
+	return parsed, nil
+}
+
+func getList(key []byte, host string) ([]string, error) {
+	data, err := get(key, host)
+	if err != nil {
+		return nil, fmt.Errorf("Failed to get data: %s", err)
+	}
+	parsed, err := unmarshalListInfo(data, string(key))
+	if err != nil {
+		return nil, fmt.Errorf("Failed to unmarshal data: %s", err)
+	}
+
+	return parsed, nil
+}
+
+func get(key []byte, host string) (map[string]string, error) {
+	var err error
+	var data map[string]string
+
+	asInfo := &aerospikeInfoCommand{
+		msg: &aerospikeMessage{
+			aerospikeMessageHeader: aerospikeMessageHeader{
+				Version: uint8(MSG_VERSION),
+				Type:    uint8(MSG_TYPE),
+				DataLen: msgLenToBytes(int64(len(key))),
+			},
+			Data: key,
+		},
+	}
+
+	cmd := asInfo.msg.Serialize()
+	addr, err := net.ResolveTCPAddr("tcp", host)
+	if err != nil {
+		return data, fmt.Errorf("Lookup failed for '%s': %s", host, err)
+	}
+
+	conn, err := net.DialTCP("tcp", nil, addr)
+	if err != nil {
+		return data, fmt.Errorf("Connection failed for '%s': %s", host, err)
+	}
+	defer conn.Close()
+
+	_, err = conn.Write(cmd)
+	if err != nil {
+		return data, fmt.Errorf("Failed to send to '%s': %s", host, err)
+	}
+
+	msgHeader := bytes.NewBuffer(make([]byte, MSG_HEADER_SIZE))
+	_, err = readLenFromConn(conn, msgHeader.Bytes(), MSG_HEADER_SIZE)
+	if err != nil {
+		return data, fmt.Errorf("Failed to read header: %s", err)
+	}
+	err = binary.Read(msgHeader, binary.BigEndian, &asInfo.msg.aerospikeMessageHeader)
+	if err != nil {
+		return data, fmt.Errorf("Failed to unmarshal header: %s", err)
+	}
+
+	msgLen := msgLenFromBytes(asInfo.msg.aerospikeMessageHeader.DataLen)
+
+	if int64(len(asInfo.msg.Data)) != msgLen {
+		asInfo.msg.Data = make([]byte, msgLen)
+	}
+
+	_, err = readLenFromConn(conn, asInfo.msg.Data, len(asInfo.msg.Data))
+	if err != nil {
+		return data, fmt.Errorf("Failed to read from connection to '%s': %s", host, err)
+	}
+
+	data, err = asInfo.parseMultiResponse()
+	if err != nil {
+		return data, fmt.Errorf("Failed to parse response from '%s': %s", host, err)
+	}
+
+	return data, err
+}
+
+func readAerospikeStats(
+	stats map[string]string,
+	acc plugins.Accumulator,
+	host string,
+	namespace string,
+) {
+	fields := make(map[string]interface{})
+	tags := map[string]string{
+		"aerospike_host": host,
+		"namespace":      "_service",
+	}
+
+	if namespace != "" {
+		tags["namespace"] = namespace
+	}
+	for key, value := range stats {
+		// We are going to ignore all string based keys
+		val, err := strconv.ParseInt(value, 10, 64)
+		if err == nil {
+			if strings.Contains(key, "-") {
+				key = strings.Replace(key, "-", "_", -1)
+			}
+			fields[key] = val
+		}
+	}
+	acc.AddFields("aerospike", fields, tags)
+}
+
+func unmarshalMapInfo(infoMap map[string]string, key string) (map[string]string, error) {
+	key = strings.TrimSuffix(key, "\n")
+	res := map[string]string{}
+
+	v, exists := infoMap[key]
+	if !exists {
+		return res, fmt.Errorf("Key '%s' missing from info", key)
+	}
+
+	values := strings.Split(v, ";")
+	for i := range values {
+		kv := strings.Split(values[i], "=")
+		if len(kv) > 1 {
+			res[kv[0]] = kv[1]
+		}
+	}
+
+	return res, nil
+}
+
+func unmarshalListInfo(infoMap map[string]string, key string) ([]string, error) {
+	key = strings.TrimSuffix(key, "\n")
+
+	v, exists := infoMap[key]
+	if !exists {
+		return []string{}, fmt.Errorf("Key '%s' missing from info", key)
+	}
+
+	values := strings.Split(v, ";")
+	return values, nil
+}
+
+func readLenFromConn(c net.Conn, buffer []byte, length int) (total int, err error) {
+	var r int
+	for total < length {
+		r, err = c.Read(buffer[total:length])
+		total += r
+		if err != nil {
+			break
+		}
+	}
+	return
+}
+
+// Taken from aerospike-client-go/types/message.go
+func msgLenToBytes(DataLen int64) [6]byte {
+	b := make([]byte, 8)
+	binary.BigEndian.PutUint64(b, uint64(DataLen))
+	res := [6]byte{}
+	copy(res[:], b[2:])
+	return res
+}
+
+// Taken from aerospike-client-go/types/message.go
+func msgLenFromBytes(buf [6]byte) int64 {
+	nbytes := append([]byte{0, 0}, buf[:]...)
+	DataLen := binary.BigEndian.Uint64(nbytes)
+	return int64(DataLen)
+}
+
+func init() {
+	plugins.Add("aerospike", func() plugins.Plugin {
+		return &Aerospike{}
+	})
+}

plugins/aerospike/aerospike_test.go

@@ -0,0 +1,112 @@
+package aerospike
+
+import (
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"reflect"
+	"testing"
+)
+
+func TestAerospikeStatistics(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	a := &Aerospike{
+		Servers: []string{testutil.GetLocalHost() + ":3000"},
+	}
+
+	var acc testutil.Accumulator
+
+	err := a.Gather(&acc)
+	require.NoError(t, err)
+
+	// Only use a few of the metrics
+	asMetrics := []string{
+		"transactions",
+		"stat_write_errs",
+		"stat_read_reqs",
+		"stat_write_reqs",
+	}
+
+	for _, metric := range asMetrics {
+		assert.True(t, acc.HasIntValue(metric), metric)
+	}
+
+}
+
+func TestAerospikeMsgLenFromToBytes(t *testing.T) {
+	var i int64 = 8
+	assert.True(t, i == msgLenFromBytes(msgLenToBytes(i)))
+}
+
+func TestReadAerospikeStatsNoNamespace(t *testing.T) {
+	// Also test for re-writing
+	var acc testutil.Accumulator
+	stats := map[string]string{
+		"stat-write-errs": "12345",
+		"stat_read_reqs":  "12345",
+	}
+	readAerospikeStats(stats, &acc, "host1", "")
+	for k := range stats {
+		if k == "stat-write-errs" {
+			k = "stat_write_errs"
+		}
+		assert.True(t, acc.HasMeasurement(k))
+		assert.True(t, acc.CheckValue(k, int64(12345)))
+	}
+}
+
+func TestReadAerospikeStatsNamespace(t *testing.T) {
+	var acc testutil.Accumulator
+	stats := map[string]string{
+		"stat_write_errs": "12345",
+		"stat_read_reqs":  "12345",
+	}
+	readAerospikeStats(stats, &acc, "host1", "test")
+
+	tags := map[string]string{
+		"aerospike_host": "host1",
+		"namespace":      "test",
+	}
+	for k := range stats {
+		assert.True(t, acc.ValidateTaggedValue(k, int64(12345), tags) == nil)
+	}
+}
+
+func TestAerospikeUnmarshalList(t *testing.T) {
+	i := map[string]string{
+		"test": "one;two;three",
+	}
+
+	expected := []string{"one", "two", "three"}
+
+	list, err := unmarshalListInfo(i, "test2")
+	assert.True(t, err != nil)
+
+	list, err = unmarshalListInfo(i, "test")
+	assert.True(t, err == nil)
+	equal := true
+	for ix := range expected {
+		if list[ix] != expected[ix] {
+			equal = false
+			break
+		}
+	}
+	assert.True(t, equal)
+}
+
+func TestAerospikeUnmarshalMap(t *testing.T) {
+	i := map[string]string{
+		"test": "key1=value1;key2=value2",
+	}
+
+	expected := map[string]string{
+		"key1": "value1",
+		"key2": "value2",
+	}
+	m, err := unmarshalMapInfo(i, "test")
+	assert.True(t, err == nil)
+	assert.True(t, reflect.DeepEqual(m, expected))
+}

plugins/all/all.go

@@ -1,11 +1,37 @@
 package all
 
 import (
+	_ "github.com/influxdb/telegraf/plugins/aerospike"
+	_ "github.com/influxdb/telegraf/plugins/apache"
+	_ "github.com/influxdb/telegraf/plugins/bcache"
+	_ "github.com/influxdb/telegraf/plugins/disque"
+	_ "github.com/influxdb/telegraf/plugins/elasticsearch"
+	_ "github.com/influxdb/telegraf/plugins/exec"
+	_ "github.com/influxdb/telegraf/plugins/haproxy"
+	_ "github.com/influxdb/telegraf/plugins/httpjson"
+	_ "github.com/influxdb/telegraf/plugins/influxdb"
+	_ "github.com/influxdb/telegraf/plugins/jolokia"
 	_ "github.com/influxdb/telegraf/plugins/kafka_consumer"
+	_ "github.com/influxdb/telegraf/plugins/leofs"
+	_ "github.com/influxdb/telegraf/plugins/lustre2"
+	_ "github.com/influxdb/telegraf/plugins/mailchimp"
 	_ "github.com/influxdb/telegraf/plugins/memcached"
+	_ "github.com/influxdb/telegraf/plugins/mongodb"
 	_ "github.com/influxdb/telegraf/plugins/mysql"
+	_ "github.com/influxdb/telegraf/plugins/nginx"
+	_ "github.com/influxdb/telegraf/plugins/phpfpm"
+	_ "github.com/influxdb/telegraf/plugins/ping"
 	_ "github.com/influxdb/telegraf/plugins/postgresql"
+	_ "github.com/influxdb/telegraf/plugins/procstat"
 	_ "github.com/influxdb/telegraf/plugins/prometheus"
+	_ "github.com/influxdb/telegraf/plugins/puppetagent"
+	_ "github.com/influxdb/telegraf/plugins/rabbitmq"
 	_ "github.com/influxdb/telegraf/plugins/redis"
+	_ "github.com/influxdb/telegraf/plugins/rethinkdb"
+	_ "github.com/influxdb/telegraf/plugins/statsd"
 	_ "github.com/influxdb/telegraf/plugins/system"
+	_ "github.com/influxdb/telegraf/plugins/trig"
+	_ "github.com/influxdb/telegraf/plugins/twemproxy"
+	_ "github.com/influxdb/telegraf/plugins/zfs"
+	_ "github.com/influxdb/telegraf/plugins/zookeeper"
 )

plugins/apache/README.md

@@ -0,0 +1,45 @@
+# Telegraf plugin: Apache
+
+#### Plugin arguments:
+- **urls** []string: List of apache-status URLs to collect from.
+
+#### Description
+
+The Apache plugin collects from the /server-status?auto URL. See
+[apache.org/server-status?auto](http://www.apache.org/server-status?auto) for an
+example. And
+[here](http://httpd.apache.org/docs/2.2/mod/mod_status.html) for the apache
+mod_status documentation.
+
+# Measurements:
+
+Meta:
+- tags: `port=<port>`, `server=url`
+
+- apache_TotalAccesses
+- apache_TotalkBytes
+- apache_CPULoad
+- apache_Uptime
+- apache_ReqPerSec
+- apache_BytesPerSec
+- apache_BytesPerReq
+- apache_BusyWorkers
+- apache_IdleWorkers
+- apache_ConnsTotal
+- apache_ConnsAsyncWriting
+- apache_ConnsAsyncKeepAlive
+- apache_ConnsAsyncClosing
+
+### Scoreboard measurements
+
+- apache_scboard_waiting
+- apache_scboard_starting
+- apache_scboard_reading
+- apache_scboard_sending
+- apache_scboard_keepalive
+- apache_scboard_dnslookup
+- apache_scboard_closing
+- apache_scboard_logging
+- apache_scboard_finishing
+- apache_scboard_idle_cleanup
+- apache_scboard_open

plugins/apache/apache.go

@@ -0,0 +1,170 @@
+package apache
+
+import (
+	"bufio"
+	"fmt"
+	"net"
+	"net/http"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Apache struct {
+	Urls []string
+}
+
+var sampleConfig = `
+  # An array of Apache status URI to gather stats.
+  urls = ["http://localhost/server-status?auto"]
+`
+
+func (n *Apache) SampleConfig() string {
+	return sampleConfig
+}
+
+func (n *Apache) Description() string {
+	return "Read Apache status information (mod_status)"
+}
+
+func (n *Apache) Gather(acc plugins.Accumulator) error {
+	var wg sync.WaitGroup
+	var outerr error
+
+	for _, u := range n.Urls {
+		addr, err := url.Parse(u)
+		if err != nil {
+			return fmt.Errorf("Unable to parse address '%s': %s", u, err)
+		}
+
+		wg.Add(1)
+		go func(addr *url.URL) {
+			defer wg.Done()
+			outerr = n.gatherUrl(addr, acc)
+		}(addr)
+	}
+
+	wg.Wait()
+
+	return outerr
+}
+
+var tr = &http.Transport{
+	ResponseHeaderTimeout: time.Duration(3 * time.Second),
+}
+
+var client = &http.Client{Transport: tr}
+
+func (n *Apache) gatherUrl(addr *url.URL, acc plugins.Accumulator) error {
+	resp, err := client.Get(addr.String())
+	if err != nil {
+		return fmt.Errorf("error making HTTP request to %s: %s", addr.String(), err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("%s returned HTTP status %s", addr.String(), resp.Status)
+	}
+
+	tags := getTags(addr)
+
+	sc := bufio.NewScanner(resp.Body)
+	fields := make(map[string]interface{})
+	for sc.Scan() {
+		line := sc.Text()
+		if strings.Contains(line, ":") {
+			parts := strings.SplitN(line, ":", 2)
+			key, part := strings.Replace(parts[0], " ", "", -1), strings.TrimSpace(parts[1])
+
+			switch key {
+			case "Scoreboard":
+				for field, value := range n.gatherScores(part) {
+					fields[field] = value
+				}
+			default:
+				value, err := strconv.ParseFloat(part, 64)
+				if err != nil {
+					continue
+				}
+				fields[key] = value
+			}
+		}
+	}
+	acc.AddFields("apache", fields, tags)
+
+	return nil
+}
+
+func (n *Apache) gatherScores(data string) map[string]interface{} {
+	var waiting, open int = 0, 0
+	var S, R, W, K, D, C, L, G, I int = 0, 0, 0, 0, 0, 0, 0, 0, 0
+
+	for _, s := range strings.Split(data, "") {
+
+		switch s {
+		case "_":
+			waiting++
+		case "S":
+			S++
+		case "R":
+			R++
+		case "W":
+			W++
+		case "K":
+			K++
+		case "D":
+			D++
+		case "C":
+			C++
+		case "L":
+			L++
+		case "G":
+			G++
+		case "I":
+			I++
+		case ".":
+			open++
+		}
+	}
+
+	fields := map[string]interface{}{
+		"scboard_waiting":      float64(waiting),
+		"scboard_starting":     float64(S),
+		"scboard_reading":      float64(R),
+		"scboard_sending":      float64(W),
+		"scboard_keepalive":    float64(K),
+		"scboard_dnslookup":    float64(D),
+		"scboard_closing":      float64(C),
+		"scboard_logging":      float64(L),
+		"scboard_finishing":    float64(G),
+		"scboard_idle_cleanup": float64(I),
+		"scboard_open":         float64(open),
+	}
+	return fields
+}
+
+// Get tag(s) for the apache plugin
+func getTags(addr *url.URL) map[string]string {
+	h := addr.Host
+	host, port, err := net.SplitHostPort(h)
+	if err != nil {
+		host = addr.Host
+		if addr.Scheme == "http" {
+			port = "80"
+		} else if addr.Scheme == "https" {
+			port = "443"
+		} else {
+			port = ""
+		}
+	}
+	return map[string]string{"server": host, "port": port}
+}
+
+func init() {
+	plugins.Add("apache", func() plugins.Plugin {
+		return &Apache{}
+	})
+}

plugins/apache/apache_test.go

@@ -0,0 +1,80 @@
+package apache
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var apacheStatus = `
+Total Accesses: 129811861
+Total kBytes: 5213701865
+CPULoad: 6.51929
+Uptime: 941553
+ReqPerSec: 137.87
+BytesPerSec: 5670240
+BytesPerReq: 41127.4
+BusyWorkers: 270
+IdleWorkers: 630
+ConnsTotal: 1451
+ConnsAsyncWriting: 32
+ConnsAsyncKeepAlive: 945
+ConnsAsyncClosing: 205
+Scoreboard: WW_____W_RW_R_W__RRR____WR_W___WW________W_WW_W_____R__R_WR__WRWR_RRRW___R_RWW__WWWRW__R_RW___RR_RW_R__W__WR_WWW______WWR__R___R_WR_W___RW______RR________________W______R__RR______W________________R____R__________________________RW_W____R_____W_R_________________R____RR__W___R_R____RW______R____W______W_W_R_R______R__R_R__________R____W_______WW____W____RR__W_____W_R_______W__________W___W____________W_______WRR_R_W____W_____R____W_WW_R____RRW__W............................................................................................................................................................................................................................................................................................................WRRWR____WR__RR_R___RWR_________W_R____RWRRR____R_R__RW_R___WWW_RW__WR_RRR____W___R____WW_R__R___RR_W_W_RRRRWR__RRWR__RRW_W_RRRW_R_RR_W__RR_RWRR_R__R___RR_RR______R__RR____R_____W_R_R_R__R__R__________W____WW_R___R_R___R_________RR__RR____RWWWW___W_R________R_R____R_W___W___R___W_WRRWW_______R__W_RW_______R________RR__R________W_______________________W_W______________RW_________WR__R___R__R_______________WR_R_________W___RW_____R____________W____......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
+`
+
+func TestHTTPApache(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		fmt.Fprintln(w, apacheStatus)
+	}))
+	defer ts.Close()
+
+	a := Apache{
+		Urls: []string{ts.URL},
+	}
+
+	var acc testutil.Accumulator
+	err := a.Gather(&acc)
+	require.NoError(t, err)
+
+	testInt := []struct {
+		measurement string
+		value       float64
+	}{
+		{"TotalAccesses", 1.29811861e+08},
+		{"TotalkBytes", 5.213701865e+09},
+		{"CPULoad", 6.51929},
+		{"Uptime", 941553},
+		{"ReqPerSec", 137.87},
+		{"BytesPerSec", 5.67024e+06},
+		{"BytesPerReq", 41127.4},
+		{"BusyWorkers", 270},
+		{"IdleWorkers", 630},
+		{"ConnsTotal", 1451},
+		{"ConnsAsyncWriting", 32},
+		{"ConnsAsyncKeepAlive", 945},
+		{"ConnsAsyncClosing", 205},
+		{"scboard_waiting", 630},
+		{"scboard_starting", 0},
+		{"scboard_reading", 157},
+		{"scboard_sending", 113},
+		{"scboard_keepalive", 0},
+		{"scboard_dnslookup", 0},
+		{"scboard_closing", 0},
+		{"scboard_logging", 0},
+		{"scboard_finishing", 0},
+		{"scboard_idle_cleanup", 0},
+		{"scboard_open", 2850},
+	}
+
+	for _, test := range testInt {
+		assert.True(t, acc.CheckValue(test.measurement, test.value))
+	}
+}

plugins/bcache/README.md

@@ -0,0 +1,89 @@
+# Telegraf plugin: bcache
+
+Get bcache stat from stats_total directory and dirty_data file.
+
+# Measurements
+
+Meta:
+
+- tags: `backing_dev=dev bcache_dev=dev`
+
+Measurement names:
+
+- dirty_data
+- bypassed
+- cache_bypass_hits
+- cache_bypass_misses
+- cache_hit_ratio
+- cache_hits
+- cache_miss_collisions
+- cache_misses
+- cache_readaheads
+
+### Description
+
+```
+dirty_data
+  Amount of dirty data for this backing device in the cache. Continuously
+  updated unlike the cache set's version, but may be slightly off.
+  
+bypassed
+  Amount of IO (both reads and writes) that has bypassed the cache
+  
+
+cache_bypass_hits
+cache_bypass_misses
+  Hits and misses for IO that is intended to skip the cache are still counted,
+  but broken out here.
+  
+cache_hits
+cache_misses
+cache_hit_ratio
+  Hits and misses are counted per individual IO as bcache sees them; a
+  partial hit is counted as a miss.
+  
+cache_miss_collisions
+  Counts instances where data was going to be inserted into the cache from a
+  cache miss, but raced with a write and data was already present (usually 0
+  since the synchronization for cache misses was rewritten)
+  
+cache_readaheads
+  Count of times readahead occurred.
+```
+
+# Example output
+
+Using this configuration:
+
+```
+[bcache]
+  # Bcache sets path
+  # If not specified, then default is:
+  # bcachePath = "/sys/fs/bcache"
+  #
+  # By default, telegraf gather stats for all bcache devices
+  # Setting devices will restrict the stats to the specified
+  # bcache devices.
+  # bcacheDevs = ["bcache0", ...]
+```
+
+When run with:
+
+```
+./telegraf -config telegraf.conf -filter bcache -test
+```
+
+It produces:
+
+```
+* Plugin: bcache, Collection 1
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_dirty_data value=11639194
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_bypassed value=5167704440832
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_bypass_hits value=146270986
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_bypass_misses value=0
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_hit_ratio value=90
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_hits value=511941651
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_miss_collisions value=157678
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_misses value=50647396
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_readaheads value=0
+```

plugins/bcache/bcache.go

@@ -0,0 +1,141 @@
+package bcache
+
+import (
+	"errors"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Bcache struct {
+	BcachePath string
+	BcacheDevs []string
+}
+
+var sampleConfig = `
+  # Bcache sets path
+  # If not specified, then default is:
+  # bcachePath = "/sys/fs/bcache"
+  #
+  # By default, telegraf gather stats for all bcache devices
+  # Setting devices will restrict the stats to the specified
+  # bcache devices.
+  # bcacheDevs = ["bcache0", ...]
+`
+
+func (b *Bcache) SampleConfig() string {
+	return sampleConfig
+}
+
+func (b *Bcache) Description() string {
+	return "Read metrics of bcache from stats_total and dirty_data"
+}
+
+func getTags(bdev string) map[string]string {
+	backingDevFile, _ := os.Readlink(bdev)
+	backingDevPath := strings.Split(backingDevFile, "/")
+	backingDev := backingDevPath[len(backingDevPath)-2]
+
+	bcacheDevFile, _ := os.Readlink(bdev + "/dev")
+	bcacheDevPath := strings.Split(bcacheDevFile, "/")
+	bcacheDev := bcacheDevPath[len(bcacheDevPath)-1]
+
+	return map[string]string{"backing_dev": backingDev, "bcache_dev": bcacheDev}
+}
+
+func prettyToBytes(v string) uint64 {
+	var factors = map[string]uint64{
+		"k": 1 << 10,
+		"M": 1 << 20,
+		"G": 1 << 30,
+		"T": 1 << 40,
+		"P": 1 << 50,
+		"E": 1 << 60,
+	}
+	var factor uint64
+	factor = 1
+	prefix := v[len(v)-1 : len(v)]
+	if factors[prefix] != 0 {
+		v = v[:len(v)-1]
+		factor = factors[prefix]
+	}
+	result, _ := strconv.ParseFloat(v, 32)
+	result = result * float64(factor)
+
+	return uint64(result)
+}
+
+func (b *Bcache) gatherBcache(bdev string, acc plugins.Accumulator) error {
+	tags := getTags(bdev)
+	metrics, err := filepath.Glob(bdev + "/stats_total/*")
+	if len(metrics) < 0 {
+		return errors.New("Can't read any stats file")
+	}
+	file, err := ioutil.ReadFile(bdev + "/dirty_data")
+	if err != nil {
+		return err
+	}
+	rawValue := strings.TrimSpace(string(file))
+	value := prettyToBytes(rawValue)
+
+	fields := make(map[string]interface{})
+	fields["dirty_data"] = value
+
+	for _, path := range metrics {
+		key := filepath.Base(path)
+		file, err := ioutil.ReadFile(path)
+		rawValue := strings.TrimSpace(string(file))
+		if err != nil {
+			return err
+		}
+		if key == "bypassed" {
+			value := prettyToBytes(rawValue)
+			fields[key] = value
+		} else {
+			value, _ := strconv.ParseUint(rawValue, 10, 64)
+			fields[key] = value
+		}
+	}
+	acc.AddFields("bcache", fields, tags)
+	return nil
+}
+
+func (b *Bcache) Gather(acc plugins.Accumulator) error {
+	bcacheDevsChecked := make(map[string]bool)
+	var restrictDevs bool
+	if len(b.BcacheDevs) != 0 {
+		restrictDevs = true
+		for _, bcacheDev := range b.BcacheDevs {
+			bcacheDevsChecked[bcacheDev] = true
+		}
+	}
+
+	bcachePath := b.BcachePath
+	if len(bcachePath) == 0 {
+		bcachePath = "/sys/fs/bcache"
+	}
+	bdevs, _ := filepath.Glob(bcachePath + "/*/bdev*")
+	if len(bdevs) < 1 {
+		return errors.New("Can't find any bcache device")
+	}
+	for _, bdev := range bdevs {
+		if restrictDevs {
+			bcacheDev := getTags(bdev)["bcache_dev"]
+			if !bcacheDevsChecked[bcacheDev] {
+				continue
+			}
+		}
+		b.gatherBcache(bdev, acc)
+	}
+	return nil
+}
+
+func init() {
+	plugins.Add("bcache", func() plugins.Plugin {
+		return &Bcache{}
+	})
+}

plugins/bcache/bcache_test.go

@@ -0,0 +1,153 @@
+package bcache
+
+import (
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	dirty_data            = "1.5G"
+	bypassed              = "4.7T"
+	cache_bypass_hits     = "146155333"
+	cache_bypass_misses   = "0"
+	cache_hit_ratio       = "90"
+	cache_hits            = "511469583"
+	cache_miss_collisions = "157567"
+	cache_misses          = "50616331"
+	cache_readaheads      = "2"
+)
+
+var (
+	testBcachePath           = os.TempDir() + "/telegraf/sys/fs/bcache"
+	testBcacheUuidPath       = testBcachePath + "/663955a3-765a-4737-a9fd-8250a7a78411"
+	testBcacheDevPath        = os.TempDir() + "/telegraf/sys/devices/virtual/block/bcache0"
+	testBcacheBackingDevPath = os.TempDir() + "/telegraf/sys/devices/virtual/block/md10"
+)
+
+type metrics struct {
+	name  string
+	value uint64
+}
+
+func TestBcacheGeneratesMetrics(t *testing.T) {
+	err := os.MkdirAll(testBcacheUuidPath, 0755)
+	require.NoError(t, err)
+
+	err = os.MkdirAll(testBcacheDevPath, 0755)
+	require.NoError(t, err)
+
+	err = os.MkdirAll(testBcacheBackingDevPath+"/bcache", 0755)
+	require.NoError(t, err)
+
+	err = os.Symlink(testBcacheBackingDevPath+"/bcache", testBcacheUuidPath+"/bdev0")
+	require.NoError(t, err)
+
+	err = os.Symlink(testBcacheDevPath, testBcacheUuidPath+"/bdev0/dev")
+	require.NoError(t, err)
+
+	err = os.MkdirAll(testBcacheUuidPath+"/bdev0/stats_total", 0755)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/dirty_data", []byte(dirty_data), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/bypassed", []byte(bypassed), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_bypass_hits", []byte(cache_bypass_hits), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_bypass_misses", []byte(cache_bypass_misses), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_hit_ratio", []byte(cache_hit_ratio), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_hits", []byte(cache_hits), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_miss_collisions", []byte(cache_miss_collisions), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_misses", []byte(cache_misses), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_readaheads", []byte(cache_readaheads), 0644)
+	require.NoError(t, err)
+
+	intMetrics := []*metrics{
+		{
+			name:  "dirty_data",
+			value: 1610612736,
+		},
+		{
+			name:  "bypassed",
+			value: 5167704440832,
+		},
+		{
+			name:  "cache_bypass_hits",
+			value: 146155333,
+		},
+		{
+			name:  "cache_bypass_misses",
+			value: 0,
+		},
+		{
+			name:  "cache_hit_ratio",
+			value: 90,
+		},
+		{
+			name:  "cache_hits",
+			value: 511469583,
+		},
+		{
+			name:  "cache_miss_collisions",
+			value: 157567,
+		},
+		{
+			name:  "cache_misses",
+			value: 50616331,
+		},
+		{
+			name:  "cache_readaheads",
+			value: 2,
+		},
+	}
+
+	tags := map[string]string{
+		"backing_dev": "md10",
+		"bcache_dev":  "bcache0",
+	}
+
+	var acc testutil.Accumulator
+
+	//all devs
+	b := &Bcache{BcachePath: testBcachePath}
+
+	err = b.Gather(&acc)
+	require.NoError(t, err)
+
+	for _, metric := range intMetrics {
+		assert.True(t, acc.HasUIntValue(metric.name), metric.name)
+		assert.True(t, acc.CheckTaggedValue(metric.name, metric.value, tags))
+	}
+
+	//one exist dev
+	b = &Bcache{BcachePath: testBcachePath, BcacheDevs: []string{"bcache0"}}
+
+	err = b.Gather(&acc)
+	require.NoError(t, err)
+
+	for _, metric := range intMetrics {
+		assert.True(t, acc.HasUIntValue(metric.name), metric.name)
+		assert.True(t, acc.CheckTaggedValue(metric.name, metric.value, tags))
+	}
+
+	err = os.RemoveAll(os.TempDir() + "/telegraf")
+	require.NoError(t, err)
+}

plugins/disque/disque.go

@@ -0,0 +1,204 @@
+package disque
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Disque struct {
+	Servers []string
+
+	c   net.Conn
+	buf []byte
+}
+
+var sampleConfig = `
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port and password. ie disque://localhost, disque://10.10.3.33:18832,
+  # 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+`
+
+func (r *Disque) SampleConfig() string {
+	return sampleConfig
+}
+
+func (r *Disque) Description() string {
+	return "Read metrics from one or many disque servers"
+}
+
+var Tracking = map[string]string{
+	"uptime_in_seconds":          "uptime",
+	"connected_clients":          "clients",
+	"blocked_clients":            "blocked_clients",
+	"used_memory":                "used_memory",
+	"used_memory_rss":            "used_memory_rss",
+	"used_memory_peak":           "used_memory_peak",
+	"total_connections_received": "total_connections_received",
+	"total_commands_processed":   "total_commands_processed",
+	"instantaneous_ops_per_sec":  "instantaneous_ops_per_sec",
+	"latest_fork_usec":           "latest_fork_usec",
+	"mem_fragmentation_ratio":    "mem_fragmentation_ratio",
+	"used_cpu_sys":               "used_cpu_sys",
+	"used_cpu_user":              "used_cpu_user",
+	"used_cpu_sys_children":      "used_cpu_sys_children",
+	"used_cpu_user_children":     "used_cpu_user_children",
+	"registered_jobs":            "registered_jobs",
+	"registered_queues":          "registered_queues",
+}
+
+var ErrProtocolError = errors.New("disque protocol error")
+
+// Reads stats from all configured servers accumulates stats.
+// Returns one of the errors encountered while gather stats (if any).
+func (g *Disque) Gather(acc plugins.Accumulator) error {
+	if len(g.Servers) == 0 {
+		url := &url.URL{
+			Host: ":7711",
+		}
+		g.gatherServer(url, acc)
+		return nil
+	}
+
+	var wg sync.WaitGroup
+
+	var outerr error
+
+	for _, serv := range g.Servers {
+		u, err := url.Parse(serv)
+		if err != nil {
+			return fmt.Errorf("Unable to parse to address '%s': %s", serv, err)
+		} else if u.Scheme == "" {
+			// fallback to simple string based address (i.e. "10.0.0.1:10000")
+			u.Scheme = "tcp"
+			u.Host = serv
+			u.Path = ""
+		}
+		wg.Add(1)
+		go func(serv string) {
+			defer wg.Done()
+			outerr = g.gatherServer(u, acc)
+		}(serv)
+	}
+
+	wg.Wait()
+
+	return outerr
+}
+
+const defaultPort = "7711"
+
+func (g *Disque) gatherServer(addr *url.URL, acc plugins.Accumulator) error {
+	if g.c == nil {
+
+		_, _, err := net.SplitHostPort(addr.Host)
+		if err != nil {
+			addr.Host = addr.Host + ":" + defaultPort
+		}
+
+		c, err := net.Dial("tcp", addr.Host)
+		if err != nil {
+			return fmt.Errorf("Unable to connect to disque server '%s': %s", addr.Host, err)
+		}
+
+		if addr.User != nil {
+			pwd, set := addr.User.Password()
+			if set && pwd != "" {
+				c.Write([]byte(fmt.Sprintf("AUTH %s\r\n", pwd)))
+
+				r := bufio.NewReader(c)
+
+				line, err := r.ReadString('\n')
+				if err != nil {
+					return err
+				}
+				if line[0] != '+' {
+					return fmt.Errorf("%s", strings.TrimSpace(line)[1:])
+				}
+			}
+		}
+
+		g.c = c
+	}
+
+	g.c.Write([]byte("info\r\n"))
+
+	r := bufio.NewReader(g.c)
+
+	line, err := r.ReadString('\n')
+	if err != nil {
+		return err
+	}
+
+	if line[0] != '$' {
+		return fmt.Errorf("bad line start: %s", ErrProtocolError)
+	}
+
+	line = strings.TrimSpace(line)
+
+	szStr := line[1:]
+
+	sz, err := strconv.Atoi(szStr)
+	if err != nil {
+		return fmt.Errorf("bad size string <<%s>>: %s", szStr, ErrProtocolError)
+	}
+
+	var read int
+
+	fields := make(map[string]interface{})
+	tags := map[string]string{"host": addr.String()}
+	for read < sz {
+		line, err := r.ReadString('\n')
+		if err != nil {
+			return err
+		}
+
+		read += len(line)
+
+		if len(line) == 1 || line[0] == '#' {
+			continue
+		}
+
+		parts := strings.SplitN(line, ":", 2)
+
+		name := string(parts[0])
+
+		metric, ok := Tracking[name]
+		if !ok {
+			continue
+		}
+
+		val := strings.TrimSpace(parts[1])
+
+		ival, err := strconv.ParseUint(val, 10, 64)
+		if err == nil {
+			fields[metric] = ival
+			continue
+		}
+
+		fval, err := strconv.ParseFloat(val, 64)
+		if err != nil {
+			return err
+		}
+
+		fields[metric] = fval
+	}
+	acc.AddFields("disque", fields, tags)
+	return nil
+}
+
+func init() {
+	plugins.Add("disque", func() plugins.Plugin {
+		return &Disque{}
+	})
+}

plugins/disque/disque_test.go

@@ -0,0 +1,250 @@
+package disque
+
+import (
+	"bufio"
+	"fmt"
+	"net"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestDisqueGeneratesMetrics(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	l, err := net.Listen("tcp", "localhost:0")
+	require.NoError(t, err)
+
+	defer l.Close()
+
+	go func() {
+		c, err := l.Accept()
+		if err != nil {
+			return
+		}
+
+		buf := bufio.NewReader(c)
+
+		for {
+			line, err := buf.ReadString('\n')
+			if err != nil {
+				return
+			}
+
+			if line != "info\r\n" {
+				return
+			}
+
+			fmt.Fprintf(c, "$%d\n", len(testOutput))
+			c.Write([]byte(testOutput))
+		}
+	}()
+
+	addr := fmt.Sprintf("disque://%s", l.Addr().String())
+
+	r := &Disque{
+		Servers: []string{addr},
+	}
+
+	var acc testutil.Accumulator
+
+	err = r.Gather(&acc)
+	require.NoError(t, err)
+
+	checkInt := []struct {
+		name  string
+		value uint64
+	}{
+		{"uptime", 1452705},
+		{"clients", 31},
+		{"blocked_clients", 13},
+		{"used_memory", 1840104},
+		{"used_memory_rss", 3227648},
+		{"used_memory_peak", 89603656},
+		{"total_connections_received", 5062777},
+		{"total_commands_processed", 12308396},
+		{"instantaneous_ops_per_sec", 18},
+		{"latest_fork_usec", 1644},
+		{"registered_jobs", 360},
+		{"registered_queues", 12},
+	}
+
+	for _, c := range checkInt {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mem_fragmentation_ratio", 1.75},
+		{"used_cpu_sys", 19585.73},
+		{"used_cpu_user", 11255.96},
+		{"used_cpu_sys_children", 1.75},
+		{"used_cpu_user_children", 1.91},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+}
+
+func TestDisqueCanPullStatsFromMultipleServers(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	l, err := net.Listen("tcp", "localhost:0")
+	require.NoError(t, err)
+
+	defer l.Close()
+
+	go func() {
+		c, err := l.Accept()
+		if err != nil {
+			return
+		}
+
+		buf := bufio.NewReader(c)
+
+		for {
+			line, err := buf.ReadString('\n')
+			if err != nil {
+				return
+			}
+
+			if line != "info\r\n" {
+				return
+			}
+
+			fmt.Fprintf(c, "$%d\n", len(testOutput))
+			c.Write([]byte(testOutput))
+		}
+	}()
+
+	addr := fmt.Sprintf("disque://%s", l.Addr().String())
+
+	r := &Disque{
+		Servers: []string{addr},
+	}
+
+	var acc testutil.Accumulator
+
+	err = r.Gather(&acc)
+	require.NoError(t, err)
+
+	checkInt := []struct {
+		name  string
+		value uint64
+	}{
+		{"uptime", 1452705},
+		{"clients", 31},
+		{"blocked_clients", 13},
+		{"used_memory", 1840104},
+		{"used_memory_rss", 3227648},
+		{"used_memory_peak", 89603656},
+		{"total_connections_received", 5062777},
+		{"total_commands_processed", 12308396},
+		{"instantaneous_ops_per_sec", 18},
+		{"latest_fork_usec", 1644},
+		{"registered_jobs", 360},
+		{"registered_queues", 12},
+	}
+
+	for _, c := range checkInt {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mem_fragmentation_ratio", 1.75},
+		{"used_cpu_sys", 19585.73},
+		{"used_cpu_user", 11255.96},
+		{"used_cpu_sys_children", 1.75},
+		{"used_cpu_user_children", 1.91},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+}
+
+const testOutput = `# Server
+disque_version:0.0.1
+disque_git_sha1:b5247598
+disque_git_dirty:0
+disque_build_id:379fda78983a60c6
+os:Linux 3.13.0-44-generic x86_64
+arch_bits:64
+multiplexing_api:epoll
+gcc_version:4.8.2
+process_id:32420
+run_id:1cfdfa4c6bc3f285182db5427522a8a4c16e42e4
+tcp_port:7711
+uptime_in_seconds:1452705
+uptime_in_days:16
+hz:10
+config_file:/usr/local/etc/disque/disque.conf
+
+# Clients
+connected_clients:31
+client_longest_output_list:0
+client_biggest_input_buf:0
+blocked_clients:13
+
+# Memory
+used_memory:1840104
+used_memory_human:1.75M
+used_memory_rss:3227648
+used_memory_peak:89603656
+used_memory_peak_human:85.45M
+mem_fragmentation_ratio:1.75
+mem_allocator:jemalloc-3.6.0
+
+# Jobs
+registered_jobs:360
+
+# Queues
+registered_queues:12
+
+# Persistence
+loading:0
+aof_enabled:1
+aof_state:on
+aof_rewrite_in_progress:0
+aof_rewrite_scheduled:0
+aof_last_rewrite_time_sec:0
+aof_current_rewrite_time_sec:-1
+aof_last_bgrewrite_status:ok
+aof_last_write_status:ok
+aof_current_size:41952430
+aof_base_size:9808
+aof_pending_rewrite:0
+aof_buffer_length:0
+aof_rewrite_buffer_length:0
+aof_pending_bio_fsync:0
+aof_delayed_fsync:1
+
+# Stats
+total_connections_received:5062777
+total_commands_processed:12308396
+instantaneous_ops_per_sec:18
+total_net_input_bytes:1346996528
+total_net_output_bytes:1967551763
+instantaneous_input_kbps:1.38
+instantaneous_output_kbps:1.78
+rejected_connections:0
+latest_fork_usec:1644
+
+# CPU
+used_cpu_sys:19585.73
+used_cpu_user:11255.96
+used_cpu_sys_children:1.75
+used_cpu_user_children:1.91
+`

plugins/elasticsearch/README.md

@@ -0,0 +1,320 @@
+# Elasticsearch plugin
+
+#### Plugin arguments:
+- **servers** []string: list of one or more Elasticsearch servers
+- **local** boolean: If false, it will read the indices stats from all nodes
+- **cluster_health** boolean: If true, it will also obtain cluster level stats
+
+#### Description
+
+The [elasticsearch](https://www.elastic.co/) plugin queries endpoints to obtain
+[node](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html)
+and optionally [cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) stats.
+
+Example:
+
+```
+    [elasticsearch]
+
+    servers = ["http://localhost:9200"]
+
+    local = true
+
+    cluster_health = true
+```
+
+# Measurements
+#### cluster measurements (utilizes fields instead of single values):
+
+contains `status`, `timed_out`, `number_of_nodes`, `number_of_data_nodes`,
+`active_primary_shards`, `active_shards`, `relocating_shards`,
+`initializing_shards`, `unassigned_shards` fields
+- elasticsearch_cluster_health
+
+contains `status`, `number_of_shards`, `number_of_replicas`,
+`active_primary_shards`, `active_shards`, `relocating_shards`,
+`initializing_shards`, `unassigned_shards` fields
+- elasticsearch_indices
+
+#### node measurements:
+
+field data circuit breaker measurement names:
+- elasticsearch_breakers_fielddata_estimated_size_in_bytes value=0
+- elasticsearch_breakers_fielddata_overhead value=1.03
+- elasticsearch_breakers_fielddata_tripped value=0
+- elasticsearch_breakers_fielddata_limit_size_in_bytes value=623326003
+- elasticsearch_breakers_request_estimated_size_in_bytes value=0
+- elasticsearch_breakers_request_overhead value=1.0
+- elasticsearch_breakers_request_tripped value=0
+- elasticsearch_breakers_request_limit_size_in_bytes value=415550668
+- elasticsearch_breakers_parent_overhead value=1.0
+- elasticsearch_breakers_parent_tripped value=0
+- elasticsearch_breakers_parent_limit_size_in_bytes value=727213670
+- elasticsearch_breakers_parent_estimated_size_in_bytes value=0
+
+File system information, data path, free disk space, read/write measurement names:
+- elasticsearch_fs_timestamp value=1436460392946
+- elasticsearch_fs_total_free_in_bytes value=16909316096
+- elasticsearch_fs_total_available_in_bytes value=15894814720
+- elasticsearch_fs_total_total_in_bytes value=19507089408
+
+indices size, document count, indexing and deletion times, search times,
+field cache size, merges and flushes measurement names:
+- elasticsearch_indices_id_cache_memory_size_in_bytes value=0
+- elasticsearch_indices_completion_size_in_bytes value=0
+- elasticsearch_indices_suggest_total value=0
+- elasticsearch_indices_suggest_time_in_millis value=0
+- elasticsearch_indices_suggest_current value=0
+- elasticsearch_indices_query_cache_memory_size_in_bytes value=0
+- elasticsearch_indices_query_cache_evictions value=0
+- elasticsearch_indices_query_cache_hit_count value=0
+- elasticsearch_indices_query_cache_miss_count value=0
+- elasticsearch_indices_store_size_in_bytes value=37715234
+- elasticsearch_indices_store_throttle_time_in_millis value=215
+- elasticsearch_indices_merges_current_docs value=0
+- elasticsearch_indices_merges_current_size_in_bytes value=0
+- elasticsearch_indices_merges_total value=133
+- elasticsearch_indices_merges_total_time_in_millis value=21060
+- elasticsearch_indices_merges_total_docs value=203672
+- elasticsearch_indices_merges_total_size_in_bytes value=142900226
+- elasticsearch_indices_merges_current value=0
+- elasticsearch_indices_filter_cache_memory_size_in_bytes value=7384
+- elasticsearch_indices_filter_cache_evictions value=0
+- elasticsearch_indices_indexing_index_total value=84790
+- elasticsearch_indices_indexing_index_time_in_millis value=29680
+- elasticsearch_indices_indexing_index_current value=0
+- elasticsearch_indices_indexing_noop_update_total value=0
+- elasticsearch_indices_indexing_throttle_time_in_millis value=0
+- elasticsearch_indices_indexing_delete_tota value=13879
+- elasticsearch_indices_indexing_delete_time_in_millis value=1139
+- elasticsearch_indices_indexing_delete_current value=0
+- elasticsearch_indices_get_exists_time_in_millis value=0
+- elasticsearch_indices_get_missing_total value=1
+- elasticsearch_indices_get_missing_time_in_millis value=2
+- elasticsearch_indices_get_current value=0
+- elasticsearch_indices_get_total value=1
+- elasticsearch_indices_get_time_in_millis value=2
+- elasticsearch_indices_get_exists_total value=0
+- elasticsearch_indices_refresh_total value=1076
+- elasticsearch_indices_refresh_total_time_in_millis value=20078
+- elasticsearch_indices_percolate_current value=0
+- elasticsearch_indices_percolate_memory_size_in_bytes value=-1
+- elasticsearch_indices_percolate_queries value=0
+- elasticsearch_indices_percolate_total value=0
+- elasticsearch_indices_percolate_time_in_millis value=0
+- elasticsearch_indices_translog_operations value=17702
+- elasticsearch_indices_translog_size_in_bytes value=17
+- elasticsearch_indices_recovery_current_as_source value=0
+- elasticsearch_indices_recovery_current_as_target value=0
+- elasticsearch_indices_recovery_throttle_time_in_millis value=0
+- elasticsearch_indices_docs_count value=29652
+- elasticsearch_indices_docs_deleted value=5229
+- elasticsearch_indices_flush_total_time_in_millis value=2401
+- elasticsearch_indices_flush_total value=115
+- elasticsearch_indices_fielddata_memory_size_in_bytes value=12996
+- elasticsearch_indices_fielddata_evictions value=0
+- elasticsearch_indices_search_fetch_current value=0
+- elasticsearch_indices_search_open_contexts value=0
+- elasticsearch_indices_search_query_total value=1452
+- elasticsearch_indices_search_query_time_in_millis value=5695
+- elasticsearch_indices_search_query_current value=0
+- elasticsearch_indices_search_fetch_total value=414
+- elasticsearch_indices_search_fetch_time_in_millis value=146
+- elasticsearch_indices_warmer_current value=0
+- elasticsearch_indices_warmer_total value=2319
+- elasticsearch_indices_warmer_total_time_in_millis value=448
+- elasticsearch_indices_segments_count value=134
+- elasticsearch_indices_segments_memory_in_bytes value=1285212
+- elasticsearch_indices_segments_index_writer_memory_in_bytes value=0
+- elasticsearch_indices_segments_index_writer_max_memory_in_bytes value=172368955
+- elasticsearch_indices_segments_version_map_memory_in_bytes value=611844
+- elasticsearch_indices_segments_fixed_bit_set_memory_in_bytes value=0
+
+HTTP connection measurement names:
+- elasticsearch_http_current_open value=3
+- elasticsearch_http_total_opened value=3
+
+JVM stats, memory pool information, garbage collection, buffer pools measurement names:
+- elasticsearch_jvm_timestamp value=1436460392945
+- elasticsearch_jvm_uptime_in_millis value=202245
+- elasticsearch_jvm_mem_non_heap_used_in_bytes value=39634576
+- elasticsearch_jvm_mem_non_heap_committed_in_bytes value=40841216
+- elasticsearch_jvm_mem_pools_young_max_in_bytes value=279183360
+- elasticsearch_jvm_mem_pools_young_peak_used_in_bytes value=71630848
+- elasticsearch_jvm_mem_pools_young_peak_max_in_bytes value=279183360
+- elasticsearch_jvm_mem_pools_young_used_in_bytes value=32685760
+- elasticsearch_jvm_mem_pools_survivor_peak_used_in_bytes value=8912888
+- elasticsearch_jvm_mem_pools_survivor_peak_max_in_bytes value=34865152
+- elasticsearch_jvm_mem_pools_survivor_used_in_bytes value=8912880
+- elasticsearch_jvm_mem_pools_survivor_max_in_bytes value=34865152
+- elasticsearch_jvm_mem_pools_old_peak_max_in_bytes value=724828160
+- elasticsearch_jvm_mem_pools_old_used_in_bytes value=11110928
+- elasticsearch_jvm_mem_pools_old_max_in_bytes value=724828160
+- elasticsearch_jvm_mem_pools_old_peak_used_in_bytes value=14354608
+- elasticsearch_jvm_mem_heap_used_in_bytes value=52709568
+- elasticsearch_jvm_mem_heap_used_percent value=5
+- elasticsearch_jvm_mem_heap_committed_in_bytes value=259522560
+- elasticsearch_jvm_mem_heap_max_in_bytes value=1038876672
+- elasticsearch_jvm_threads_peak_count value=45
+- elasticsearch_jvm_threads_count value=44
+- elasticsearch_jvm_gc_collectors_young_collection_count value=2
+- elasticsearch_jvm_gc_collectors_young_collection_time_in_millis value=98
+- elasticsearch_jvm_gc_collectors_old_collection_count value=1
+- elasticsearch_jvm_gc_collectors_old_collection_time_in_millis value=24
+- elasticsearch_jvm_buffer_pools_direct_count value=40
+- elasticsearch_jvm_buffer_pools_direct_used_in_bytes value=6304239
+- elasticsearch_jvm_buffer_pools_direct_total_capacity_in_bytes value=6304239
+- elasticsearch_jvm_buffer_pools_mapped_count value=0
+- elasticsearch_jvm_buffer_pools_mapped_used_in_bytes value=0
+- elasticsearch_jvm_buffer_pools_mapped_total_capacity_in_bytes value=0
+
+TCP information measurement names:
+- elasticsearch_network_tcp_in_errs value=0
+- elasticsearch_network_tcp_passive_opens value=16
+- elasticsearch_network_tcp_curr_estab value=29
+- elasticsearch_network_tcp_in_segs value=113
+- elasticsearch_network_tcp_out_segs value=97
+- elasticsearch_network_tcp_retrans_segs value=0
+- elasticsearch_network_tcp_attempt_fails value=0
+- elasticsearch_network_tcp_active_opens value=13
+- elasticsearch_network_tcp_estab_resets value=0
+- elasticsearch_network_tcp_out_rsts value=0
+
+Operating system stats, load average, cpu, mem, swap measurement names:
+- elasticsearch_os_swap_used_in_bytes value=0
+- elasticsearch_os_swap_free_in_bytes value=487997440
+- elasticsearch_os_timestamp value=1436460392944
+- elasticsearch_os_uptime_in_millis value=25092
+- elasticsearch_os_cpu_sys value=0
+- elasticsearch_os_cpu_user value=0
+- elasticsearch_os_cpu_idle value=99
+- elasticsearch_os_cpu_usage value=0
+- elasticsearch_os_cpu_stolen value=0
+- elasticsearch_os_mem_free_percent value=74
+- elasticsearch_os_mem_used_percent value=25
+- elasticsearch_os_mem_actual_free_in_bytes value=1565470720
+- elasticsearch_os_mem_actual_used_in_bytes value=534159360
+- elasticsearch_os_mem_free_in_bytes value=477761536
+- elasticsearch_os_mem_used_in_bytes value=1621868544
+
+Process statistics, memory consumption, cpu usage, open file descriptors measurement names:
+- elasticsearch_process_mem_resident_in_bytes value=246382592
+- elasticsearch_process_mem_share_in_bytes value=18747392
+- elasticsearch_process_mem_total_virtual_in_bytes value=4747890688
+- elasticsearch_process_timestamp value=1436460392945
+- elasticsearch_process_open_file_descriptors value=160
+- elasticsearch_process_cpu_total_in_millis value=15480
+- elasticsearch_process_cpu_percent value=2
+- elasticsearch_process_cpu_sys_in_millis value=1870
+- elasticsearch_process_cpu_user_in_millis value=13610
+
+Statistics about each thread pool, including current size, queue and rejected tasks measurement names:
+- elasticsearch_thread_pool_merge_threads value=6
+- elasticsearch_thread_pool_merge_queue value=4
+- elasticsearch_thread_pool_merge_active value=5
+- elasticsearch_thread_pool_merge_rejected value=2
+- elasticsearch_thread_pool_merge_largest value=5
+- elasticsearch_thread_pool_merge_completed value=1
+- elasticsearch_thread_pool_bulk_threads value=4
+- elasticsearch_thread_pool_bulk_queue value=5
+- elasticsearch_thread_pool_bulk_active value=7
+- elasticsearch_thread_pool_bulk_rejected value=3
+- elasticsearch_thread_pool_bulk_largest value=1
+- elasticsearch_thread_pool_bulk_completed value=4
+- elasticsearch_thread_pool_warmer_threads value=2
+- elasticsearch_thread_pool_warmer_queue value=7
+- elasticsearch_thread_pool_warmer_active value=3
+- elasticsearch_thread_pool_warmer_rejected value=2
+- elasticsearch_thread_pool_warmer_largest value=3
+- elasticsearch_thread_pool_warmer_completed value=1
+- elasticsearch_thread_pool_get_largest value=2
+- elasticsearch_thread_pool_get_completed value=1
+- elasticsearch_thread_pool_get_threads value=1
+- elasticsearch_thread_pool_get_queue value=8
+- elasticsearch_thread_pool_get_active value=4
+- elasticsearch_thread_pool_get_rejected value=3
+- elasticsearch_thread_pool_index_threads value=6
+- elasticsearch_thread_pool_index_queue value=8
+- elasticsearch_thread_pool_index_active value=4
+- elasticsearch_thread_pool_index_rejected value=2
+- elasticsearch_thread_pool_index_largest value=3
+- elasticsearch_thread_pool_index_completed value=6
+- elasticsearch_thread_pool_suggest_threads value=2
+- elasticsearch_thread_pool_suggest_queue value=7
+- elasticsearch_thread_pool_suggest_active value=2
+- elasticsearch_thread_pool_suggest_rejected value=1
+- elasticsearch_thread_pool_suggest_largest value=8
+- elasticsearch_thread_pool_suggest_completed value=3
+- elasticsearch_thread_pool_fetch_shard_store_queue value=7
+- elasticsearch_thread_pool_fetch_shard_store_active value=4
+- elasticsearch_thread_pool_fetch_shard_store_rejected value=2
+- elasticsearch_thread_pool_fetch_shard_store_largest value=4
+- elasticsearch_thread_pool_fetch_shard_store_completed value=1
+- elasticsearch_thread_pool_fetch_shard_store_threads value=1
+- elasticsearch_thread_pool_management_threads value=2
+- elasticsearch_thread_pool_management_queue value=3
+- elasticsearch_thread_pool_management_active value=1
+- elasticsearch_thread_pool_management_rejected value=6
+- elasticsearch_thread_pool_management_largest value=2
+- elasticsearch_thread_pool_management_completed value=22
+- elasticsearch_thread_pool_percolate_queue value=23
+- elasticsearch_thread_pool_percolate_active value=13
+- elasticsearch_thread_pool_percolate_rejected value=235
+- elasticsearch_thread_pool_percolate_largest value=23
+- elasticsearch_thread_pool_percolate_completed value=33
+- elasticsearch_thread_pool_percolate_threads value=123
+- elasticsearch_thread_pool_listener_active value=4
+- elasticsearch_thread_pool_listener_rejected value=8
+- elasticsearch_thread_pool_listener_largest value=1
+- elasticsearch_thread_pool_listener_completed value=1
+- elasticsearch_thread_pool_listener_threads value=1
+- elasticsearch_thread_pool_listener_queue value=2
+- elasticsearch_thread_pool_search_rejected value=7
+- elasticsearch_thread_pool_search_largest value=2
+- elasticsearch_thread_pool_search_completed value=4
+- elasticsearch_thread_pool_search_threads value=5
+- elasticsearch_thread_pool_search_queue value=7
+- elasticsearch_thread_pool_search_active value=2
+- elasticsearch_thread_pool_fetch_shard_started_threads value=3
+- elasticsearch_thread_pool_fetch_shard_started_queue value=1
+- elasticsearch_thread_pool_fetch_shard_started_active value=5
+- elasticsearch_thread_pool_fetch_shard_started_rejected value=6
+- elasticsearch_thread_pool_fetch_shard_started_largest value=4
+- elasticsearch_thread_pool_fetch_shard_started_completed value=54
+- elasticsearch_thread_pool_refresh_rejected value=4
+- elasticsearch_thread_pool_refresh_largest value=8
+- elasticsearch_thread_pool_refresh_completed value=3
+- elasticsearch_thread_pool_refresh_threads value=23
+- elasticsearch_thread_pool_refresh_queue value=7
+- elasticsearch_thread_pool_refresh_active value=3
+- elasticsearch_thread_pool_optimize_threads value=3
+- elasticsearch_thread_pool_optimize_queue value=4
+- elasticsearch_thread_pool_optimize_active value=1
+- elasticsearch_thread_pool_optimize_rejected value=2
+- elasticsearch_thread_pool_optimize_largest value=7
+- elasticsearch_thread_pool_optimize_completed value=3
+- elasticsearch_thread_pool_snapshot_largest value=1
+- elasticsearch_thread_pool_snapshot_completed value=0
+- elasticsearch_thread_pool_snapshot_threads value=8
+- elasticsearch_thread_pool_snapshot_queue value=5
+- elasticsearch_thread_pool_snapshot_active value=6
+- elasticsearch_thread_pool_snapshot_rejected value=2
+- elasticsearch_thread_pool_generic_threads value=1
+- elasticsearch_thread_pool_generic_queue value=4
+- elasticsearch_thread_pool_generic_active value=6
+- elasticsearch_thread_pool_generic_rejected value=3
+- elasticsearch_thread_pool_generic_largest value=2
+- elasticsearch_thread_pool_generic_completed value=27
+- elasticsearch_thread_pool_flush_threads value=3
+- elasticsearch_thread_pool_flush_queue value=8
+- elasticsearch_thread_pool_flush_active value=0
+- elasticsearch_thread_pool_flush_rejected value=1
+- elasticsearch_thread_pool_flush_largest value=5
+- elasticsearch_thread_pool_flush_completed value=3
+
+Transport statistics about sent and received bytes in cluster communication measurement names:
+- elasticsearch_transport_server_open value=13
+- elasticsearch_transport_rx_count value=6
+- elasticsearch_transport_rx_size_in_bytes value=1380
+- elasticsearch_transport_tx_count value=6
+- elasticsearch_transport_tx_size_in_bytes value=1380

plugins/elasticsearch/elasticsearch.go

@@ -0,0 +1,226 @@
+package elasticsearch
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+const statsPath = "/_nodes/stats"
+const statsPathLocal = "/_nodes/_local/stats"
+const healthPath = "/_cluster/health"
+
+type node struct {
+	Host       string            `json:"host"`
+	Name       string            `json:"name"`
+	Attributes map[string]string `json:"attributes"`
+	Indices    interface{}       `json:"indices"`
+	OS         interface{}       `json:"os"`
+	Process    interface{}       `json:"process"`
+	JVM        interface{}       `json:"jvm"`
+	ThreadPool interface{}       `json:"thread_pool"`
+	FS         interface{}       `json:"fs"`
+	Transport  interface{}       `json:"transport"`
+	HTTP       interface{}       `json:"http"`
+	Breakers   interface{}       `json:"breakers"`
+}
+
+type clusterHealth struct {
+	ClusterName         string                 `json:"cluster_name"`
+	Status              string                 `json:"status"`
+	TimedOut            bool                   `json:"timed_out"`
+	NumberOfNodes       int                    `json:"number_of_nodes"`
+	NumberOfDataNodes   int                    `json:"number_of_data_nodes"`
+	ActivePrimaryShards int                    `json:"active_primary_shards"`
+	ActiveShards        int                    `json:"active_shards"`
+	RelocatingShards    int                    `json:"relocating_shards"`
+	InitializingShards  int                    `json:"initializing_shards"`
+	UnassignedShards    int                    `json:"unassigned_shards"`
+	Indices             map[string]indexHealth `json:"indices"`
+}
+
+type indexHealth struct {
+	Status              string `json:"status"`
+	NumberOfShards      int    `json:"number_of_shards"`
+	NumberOfReplicas    int    `json:"number_of_replicas"`
+	ActivePrimaryShards int    `json:"active_primary_shards"`
+	ActiveShards        int    `json:"active_shards"`
+	RelocatingShards    int    `json:"relocating_shards"`
+	InitializingShards  int    `json:"initializing_shards"`
+	UnassignedShards    int    `json:"unassigned_shards"`
+}
+
+const sampleConfig = `
+  # specify a list of one or more Elasticsearch servers
+  servers = ["http://localhost:9200"]
+
+  # set local to false when you want to read the indices stats from all nodes
+  # within the cluster
+  local = true
+
+  # set cluster_health to true when you want to also obtain cluster level stats
+  cluster_health = false
+`
+
+// Elasticsearch is a plugin to read stats from one or many Elasticsearch
+// servers.
+type Elasticsearch struct {
+	Local         bool
+	Servers       []string
+	ClusterHealth bool
+	client        *http.Client
+}
+
+// NewElasticsearch return a new instance of Elasticsearch
+func NewElasticsearch() *Elasticsearch {
+	return &Elasticsearch{client: http.DefaultClient}
+}
+
+// SampleConfig returns sample configuration for this plugin.
+func (e *Elasticsearch) SampleConfig() string {
+	return sampleConfig
+}
+
+// Description returns the plugin description.
+func (e *Elasticsearch) Description() string {
+	return "Read stats from one or more Elasticsearch servers or clusters"
+}
+
+// Gather reads the stats from Elasticsearch and writes it to the
+// Accumulator.
+func (e *Elasticsearch) Gather(acc plugins.Accumulator) error {
+	for _, serv := range e.Servers {
+		var url string
+		if e.Local {
+			url = serv + statsPathLocal
+		} else {
+			url = serv + statsPath
+		}
+		if err := e.gatherNodeStats(url, acc); err != nil {
+			return err
+		}
+		if e.ClusterHealth {
+			e.gatherClusterStats(fmt.Sprintf("%s/_cluster/health?level=indices", serv), acc)
+		}
+	}
+	return nil
+}
+
+func (e *Elasticsearch) gatherNodeStats(url string, acc plugins.Accumulator) error {
+	nodeStats := &struct {
+		ClusterName string           `json:"cluster_name"`
+		Nodes       map[string]*node `json:"nodes"`
+	}{}
+	if err := e.gatherData(url, nodeStats); err != nil {
+		return err
+	}
+	for id, n := range nodeStats.Nodes {
+		tags := map[string]string{
+			"node_id":      id,
+			"node_host":    n.Host,
+			"node_name":    n.Name,
+			"cluster_name": nodeStats.ClusterName,
+		}
+
+		for k, v := range n.Attributes {
+			tags["node_attribute_"+k] = v
+		}
+
+		stats := map[string]interface{}{
+			"indices":     n.Indices,
+			"os":          n.OS,
+			"process":     n.Process,
+			"jvm":         n.JVM,
+			"thread_pool": n.ThreadPool,
+			"fs":          n.FS,
+			"transport":   n.Transport,
+			"http":        n.HTTP,
+			"breakers":    n.Breakers,
+		}
+
+		now := time.Now()
+		for p, s := range stats {
+			f := internal.JSONFlattener{}
+			err := f.FlattenJSON("", s)
+			if err != nil {
+				return err
+			}
+			acc.AddFields("elasticsearch_"+p, f.Fields, tags, now)
+		}
+	}
+	return nil
+}
+
+func (e *Elasticsearch) gatherClusterStats(url string, acc plugins.Accumulator) error {
+	clusterStats := &clusterHealth{}
+	if err := e.gatherData(url, clusterStats); err != nil {
+		return err
+	}
+	measurementTime := time.Now()
+	clusterFields := map[string]interface{}{
+		"status":                clusterStats.Status,
+		"timed_out":             clusterStats.TimedOut,
+		"number_of_nodes":       clusterStats.NumberOfNodes,
+		"number_of_data_nodes":  clusterStats.NumberOfDataNodes,
+		"active_primary_shards": clusterStats.ActivePrimaryShards,
+		"active_shards":         clusterStats.ActiveShards,
+		"relocating_shards":     clusterStats.RelocatingShards,
+		"initializing_shards":   clusterStats.InitializingShards,
+		"unassigned_shards":     clusterStats.UnassignedShards,
+	}
+	acc.AddFields(
+		"elasticsearch_cluster_health",
+		clusterFields,
+		map[string]string{"name": clusterStats.ClusterName},
+		measurementTime,
+	)
+
+	for name, health := range clusterStats.Indices {
+		indexFields := map[string]interface{}{
+			"status":                health.Status,
+			"number_of_shards":      health.NumberOfShards,
+			"number_of_replicas":    health.NumberOfReplicas,
+			"active_primary_shards": health.ActivePrimaryShards,
+			"active_shards":         health.ActiveShards,
+			"relocating_shards":     health.RelocatingShards,
+			"initializing_shards":   health.InitializingShards,
+			"unassigned_shards":     health.UnassignedShards,
+		}
+		acc.AddFields(
+			"elasticsearch_indices",
+			indexFields,
+			map[string]string{"index": name},
+			measurementTime,
+		)
+	}
+	return nil
+}
+
+func (e *Elasticsearch) gatherData(url string, v interface{}) error {
+	r, err := e.client.Get(url)
+	if err != nil {
+		return err
+	}
+	defer r.Body.Close()
+	if r.StatusCode != http.StatusOK {
+		// NOTE: we are not going to read/discard r.Body under the assumption we'd prefer
+		// to let the underlying transport close the connection and re-establish a new one for
+		// future calls.
+		return fmt.Errorf("elasticsearch: API responded with status-code %d, expected %d",
+			r.StatusCode, http.StatusOK)
+	}
+	if err = json.NewDecoder(r.Body).Decode(v); err != nil {
+		return err
+	}
+	return nil
+}
+
+func init() {
+	plugins.Add("elasticsearch", func() plugins.Plugin {
+		return NewElasticsearch()
+	})
+}

plugins/elasticsearch/elasticsearch_test.go

@@ -0,0 +1,108 @@
+package elasticsearch
+
+import (
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type transportMock struct {
+	statusCode int
+	body       string
+}
+
+func newTransportMock(statusCode int, body string) http.RoundTripper {
+	return &transportMock{
+		statusCode: statusCode,
+		body:       body,
+	}
+}
+
+func (t *transportMock) RoundTrip(r *http.Request) (*http.Response, error) {
+	res := &http.Response{
+		Header:     make(http.Header),
+		Request:    r,
+		StatusCode: t.statusCode,
+	}
+	res.Header.Set("Content-Type", "application/json")
+	res.Body = ioutil.NopCloser(strings.NewReader(t.body))
+	return res, nil
+}
+
+func TestElasticsearch(t *testing.T) {
+	es := NewElasticsearch()
+	es.Servers = []string{"http://example.com:9200"}
+	es.client.Transport = newTransportMock(http.StatusOK, statsResponse)
+
+	var acc testutil.Accumulator
+	if err := es.Gather(&acc); err != nil {
+		t.Fatal(err)
+	}
+
+	tags := map[string]string{
+		"cluster_name":          "es-testcluster",
+		"node_attribute_master": "true",
+		"node_id":               "SDFsfSDFsdfFSDSDfSFDSDF",
+		"node_name":             "test.host.com",
+		"node_host":             "test",
+	}
+
+	testTables := []map[string]float64{
+		indicesExpected,
+		osExpected,
+		processExpected,
+		jvmExpected,
+		threadPoolExpected,
+		fsExpected,
+		transportExpected,
+		httpExpected,
+		breakersExpected,
+	}
+
+	for _, testTable := range testTables {
+		for k, v := range testTable {
+			assert.NoError(t, acc.ValidateTaggedValue(k, v, tags))
+		}
+	}
+}
+
+func TestGatherClusterStats(t *testing.T) {
+	es := NewElasticsearch()
+	es.Servers = []string{"http://example.com:9200"}
+	es.ClusterHealth = true
+	es.client.Transport = newTransportMock(http.StatusOK, clusterResponse)
+
+	var acc testutil.Accumulator
+	require.NoError(t, es.Gather(&acc))
+
+	var clusterHealthTests = []struct {
+		measurement string
+		fields      map[string]interface{}
+		tags        map[string]string
+	}{
+		{
+			"cluster_health",
+			clusterHealthExpected,
+			map[string]string{"name": "elasticsearch_telegraf"},
+		},
+		{
+			"indices",
+			v1IndexExpected,
+			map[string]string{"index": "v1"},
+		},
+		{
+			"indices",
+			v2IndexExpected,
+			map[string]string{"index": "v2"},
+		},
+	}
+
+	for _, exp := range clusterHealthTests {
+		assert.NoError(t, acc.ValidateTaggedFields(exp.measurement, exp.fields, exp.tags))
+	}
+}

plugins/elasticsearch/testdata_test.go

@@ -0,0 +1,759 @@
+package elasticsearch
+
+const clusterResponse = `
+{
+   "cluster_name": "elasticsearch_telegraf",
+   "status": "green",
+   "timed_out": false,
+   "number_of_nodes": 3,
+   "number_of_data_nodes": 3,
+   "active_primary_shards": 5,
+   "active_shards": 15,
+   "relocating_shards": 0,
+   "initializing_shards": 0,
+   "unassigned_shards": 0,
+   "indices": {
+      "v1": {
+         "status": "green",
+         "number_of_shards": 10,
+         "number_of_replicas": 1,
+         "active_primary_shards": 10,
+         "active_shards": 20,
+         "relocating_shards": 0,
+         "initializing_shards": 0,
+         "unassigned_shards": 0
+      },
+      "v2": {
+         "status": "red",
+         "number_of_shards": 10,
+         "number_of_replicas": 1,
+         "active_primary_shards": 0,
+         "active_shards": 0,
+         "relocating_shards": 0,
+         "initializing_shards": 0,
+         "unassigned_shards": 20
+      }
+   }
+}
+`
+
+var clusterHealthExpected = map[string]interface{}{
+	"status":                "green",
+	"timed_out":             false,
+	"number_of_nodes":       3,
+	"number_of_data_nodes":  3,
+	"active_primary_shards": 5,
+	"active_shards":         15,
+	"relocating_shards":     0,
+	"initializing_shards":   0,
+	"unassigned_shards":     0,
+}
+
+var v1IndexExpected = map[string]interface{}{
+	"status":                "green",
+	"number_of_shards":      10,
+	"number_of_replicas":    1,
+	"active_primary_shards": 10,
+	"active_shards":         20,
+	"relocating_shards":     0,
+	"initializing_shards":   0,
+	"unassigned_shards":     0,
+}
+
+var v2IndexExpected = map[string]interface{}{
+	"status":                "red",
+	"number_of_shards":      10,
+	"number_of_replicas":    1,
+	"active_primary_shards": 0,
+	"active_shards":         0,
+	"relocating_shards":     0,
+	"initializing_shards":   0,
+	"unassigned_shards":     20,
+}
+
+const statsResponse = `
+{
+  "cluster_name": "es-testcluster",
+  "nodes": {
+    "SDFsfSDFsdfFSDSDfSFDSDF": {
+      "timestamp": 1436365550135,
+      "name": "test.host.com",
+      "transport_address": "inet[/127.0.0.1:9300]",
+      "host": "test",
+      "ip": [
+        "inet[/127.0.0.1:9300]",
+        "NONE"
+      ],
+      "attributes": {
+        "master": "true"
+      },
+      "indices": {
+        "docs": {
+          "count": 29652,
+          "deleted": 5229
+        },
+        "store": {
+          "size_in_bytes": 37715234,
+          "throttle_time_in_millis": 215
+        },
+        "indexing": {
+          "index_total": 84790,
+          "index_time_in_millis": 29680,
+          "index_current": 0,
+          "delete_total": 13879,
+          "delete_time_in_millis": 1139,
+          "delete_current": 0,
+          "noop_update_total": 0,
+          "is_throttled": false,
+          "throttle_time_in_millis": 0
+        },
+        "get": {
+          "total": 1,
+          "time_in_millis": 2,
+          "exists_total": 0,
+          "exists_time_in_millis": 0,
+          "missing_total": 1,
+          "missing_time_in_millis": 2,
+          "current": 0
+        },
+        "search": {
+          "open_contexts": 0,
+          "query_total": 1452,
+          "query_time_in_millis": 5695,
+          "query_current": 0,
+          "fetch_total": 414,
+          "fetch_time_in_millis": 146,
+          "fetch_current": 0
+        },
+        "merges": {
+          "current": 0,
+          "current_docs": 0,
+          "current_size_in_bytes": 0,
+          "total": 133,
+          "total_time_in_millis": 21060,
+          "total_docs": 203672,
+          "total_size_in_bytes": 142900226
+        },
+        "refresh": {
+          "total": 1076,
+          "total_time_in_millis": 20078
+        },
+        "flush": {
+          "total": 115,
+          "total_time_in_millis": 2401
+        },
+        "warmer": {
+          "current": 0,
+          "total": 2319,
+          "total_time_in_millis": 448
+        },
+        "filter_cache": {
+          "memory_size_in_bytes": 7384,
+          "evictions": 0
+        },
+        "id_cache": {
+          "memory_size_in_bytes": 0
+        },
+        "fielddata": {
+          "memory_size_in_bytes": 12996,
+          "evictions": 0
+        },
+        "percolate": {
+          "total": 0,
+          "time_in_millis": 0,
+          "current": 0,
+          "memory_size_in_bytes": -1,
+          "memory_size": "-1b",
+          "queries": 0
+        },
+        "completion": {
+          "size_in_bytes": 0
+        },
+        "segments": {
+          "count": 134,
+          "memory_in_bytes": 1285212,
+          "index_writer_memory_in_bytes": 0,
+          "index_writer_max_memory_in_bytes": 172368955,
+          "version_map_memory_in_bytes": 611844,
+          "fixed_bit_set_memory_in_bytes": 0
+        },
+        "translog": {
+          "operations": 17702,
+          "size_in_bytes": 17
+        },
+        "suggest": {
+          "total": 0,
+          "time_in_millis": 0,
+          "current": 0
+        },
+        "query_cache": {
+          "memory_size_in_bytes": 0,
+          "evictions": 0,
+          "hit_count": 0,
+          "miss_count": 0
+        },
+        "recovery": {
+          "current_as_source": 0,
+          "current_as_target": 0,
+          "throttle_time_in_millis": 0
+        }
+      },
+      "os": {
+        "timestamp": 1436460392944,
+        "load_average": [
+          0.01,
+          0.04,
+          0.05
+        ],
+        "mem": {
+          "free_in_bytes": 477761536,
+          "used_in_bytes": 1621868544,
+          "free_percent": 74,
+          "used_percent": 25,
+          "actual_free_in_bytes": 1565470720,
+          "actual_used_in_bytes": 534159360
+        },
+        "swap": {
+          "used_in_bytes": 0,
+          "free_in_bytes": 487997440
+        }
+      },
+      "process": {
+        "timestamp": 1436460392945,
+        "open_file_descriptors": 160,
+        "cpu": {
+          "percent": 2,
+          "sys_in_millis": 1870,
+          "user_in_millis": 13610,
+          "total_in_millis": 15480
+        },
+        "mem": {
+          "total_virtual_in_bytes": 4747890688
+        }
+      },
+      "jvm": {
+        "timestamp": 1436460392945,
+        "uptime_in_millis": 202245,
+        "mem": {
+          "heap_used_in_bytes": 52709568,
+          "heap_used_percent": 5,
+          "heap_committed_in_bytes": 259522560,
+          "heap_max_in_bytes": 1038876672,
+          "non_heap_used_in_bytes": 39634576,
+          "non_heap_committed_in_bytes": 40841216,
+          "pools": {
+            "young": {
+              "used_in_bytes": 32685760,
+              "max_in_bytes": 279183360,
+              "peak_used_in_bytes": 71630848,
+              "peak_max_in_bytes": 279183360
+            },
+            "survivor": {
+              "used_in_bytes": 8912880,
+              "max_in_bytes": 34865152,
+              "peak_used_in_bytes": 8912888,
+              "peak_max_in_bytes": 34865152
+            },
+            "old": {
+              "used_in_bytes": 11110928,
+              "max_in_bytes": 724828160,
+              "peak_used_in_bytes": 14354608,
+              "peak_max_in_bytes": 724828160
+            }
+          }
+        },
+        "threads": {
+          "count": 44,
+          "peak_count": 45
+        },
+        "gc": {
+          "collectors": {
+            "young": {
+              "collection_count": 2,
+              "collection_time_in_millis": 98
+            },
+            "old": {
+              "collection_count": 1,
+              "collection_time_in_millis": 24
+            }
+          }
+        },
+        "buffer_pools": {
+          "direct": {
+            "count": 40,
+            "used_in_bytes": 6304239,
+            "total_capacity_in_bytes": 6304239
+          },
+          "mapped": {
+            "count": 0,
+            "used_in_bytes": 0,
+            "total_capacity_in_bytes": 0
+          }
+        }
+      },
+      "thread_pool": {
+        "percolate": {
+          "threads": 123,
+          "queue": 23,
+          "active": 13,
+          "rejected": 235,
+          "largest": 23,
+          "completed": 33
+        },
+        "fetch_shard_started": {
+          "threads": 3,
+          "queue": 1,
+          "active": 5,
+          "rejected": 6,
+          "largest": 4,
+          "completed": 54
+        },
+        "listener": {
+          "threads": 1,
+          "queue": 2,
+          "active": 4,
+          "rejected": 8,
+          "largest": 1,
+          "completed": 1
+        },
+        "index": {
+          "threads": 6,
+          "queue": 8,
+          "active": 4,
+          "rejected": 2,
+          "largest": 3,
+          "completed": 6
+        },
+        "refresh": {
+          "threads": 23,
+          "queue": 7,
+          "active": 3,
+          "rejected": 4,
+          "largest": 8,
+          "completed": 3
+        },
+        "suggest": {
+          "threads": 2,
+          "queue": 7,
+          "active": 2,
+          "rejected": 1,
+          "largest": 8,
+          "completed": 3
+        },
+        "generic": {
+          "threads": 1,
+          "queue": 4,
+          "active": 6,
+          "rejected": 3,
+          "largest": 2,
+          "completed": 27
+        },
+        "warmer": {
+          "threads": 2,
+          "queue": 7,
+          "active": 3,
+          "rejected": 2,
+          "largest": 3,
+          "completed": 1
+        },
+        "search": {
+          "threads": 5,
+          "queue": 7,
+          "active": 2,
+          "rejected": 7,
+          "largest": 2,
+          "completed": 4
+        },
+        "flush": {
+          "threads": 3,
+          "queue": 8,
+          "active": 0,
+          "rejected": 1,
+          "largest": 5,
+          "completed": 3
+        },
+        "optimize": {
+          "threads": 3,
+          "queue": 4,
+          "active": 1,
+          "rejected": 2,
+          "largest": 7,
+          "completed": 3
+        },
+        "fetch_shard_store": {
+          "threads": 1,
+          "queue": 7,
+          "active": 4,
+          "rejected": 2,
+          "largest": 4,
+          "completed": 1
+        },
+        "management": {
+          "threads": 2,
+          "queue": 3,
+          "active": 1,
+          "rejected": 6,
+          "largest": 2,
+          "completed": 22
+        },
+        "get": {
+          "threads": 1,
+          "queue": 8,
+          "active": 4,
+          "rejected": 3,
+          "largest": 2,
+          "completed": 1
+        },
+        "merge": {
+          "threads": 6,
+          "queue": 4,
+          "active": 5,
+          "rejected": 2,
+          "largest": 5,
+          "completed": 1
+        },
+        "bulk": {
+          "threads": 4,
+          "queue": 5,
+          "active": 7,
+          "rejected": 3,
+          "largest": 1,
+          "completed": 4
+        },
+        "snapshot": {
+          "threads": 8,
+          "queue": 5,
+          "active": 6,
+          "rejected": 2,
+          "largest": 1,
+          "completed": 0
+        }
+      },
+      "fs": {
+        "timestamp": 1436460392946,
+        "total": {
+          "total_in_bytes": 19507089408,
+          "free_in_bytes": 16909316096,
+          "available_in_bytes": 15894814720
+        },
+        "data": [
+          {
+            "path": "/usr/share/elasticsearch/data/elasticsearch/nodes/0",
+            "mount": "/usr/share/elasticsearch/data",
+            "type": "ext4",
+            "total_in_bytes": 19507089408,
+            "free_in_bytes": 16909316096,
+            "available_in_bytes": 15894814720
+          }
+        ]
+      },
+      "transport": {
+        "server_open": 13,
+        "rx_count": 6,
+        "rx_size_in_bytes": 1380,
+        "tx_count": 6,
+        "tx_size_in_bytes": 1380
+      },
+      "http": {
+        "current_open": 3,
+        "total_opened": 3
+      },
+      "breakers": {
+        "fielddata": {
+          "limit_size_in_bytes": 623326003,
+          "limit_size": "594.4mb",
+          "estimated_size_in_bytes": 0,
+          "estimated_size": "0b",
+          "overhead": 1.03,
+          "tripped": 0
+        },
+        "request": {
+          "limit_size_in_bytes": 415550668,
+          "limit_size": "396.2mb",
+          "estimated_size_in_bytes": 0,
+          "estimated_size": "0b",
+          "overhead": 1.0,
+          "tripped": 0
+        },
+        "parent": {
+          "limit_size_in_bytes": 727213670,
+          "limit_size": "693.5mb",
+          "estimated_size_in_bytes": 0,
+          "estimated_size": "0b",
+          "overhead": 1.0,
+          "tripped": 0
+        }
+      }
+    }
+  }
+}
+`
+
+var indicesExpected = map[string]float64{
+	"indices_id_cache_memory_size_in_bytes":             0,
+	"indices_completion_size_in_bytes":                  0,
+	"indices_suggest_total":                             0,
+	"indices_suggest_time_in_millis":                    0,
+	"indices_suggest_current":                           0,
+	"indices_query_cache_memory_size_in_bytes":          0,
+	"indices_query_cache_evictions":                     0,
+	"indices_query_cache_hit_count":                     0,
+	"indices_query_cache_miss_count":                    0,
+	"indices_store_size_in_bytes":                       37715234,
+	"indices_store_throttle_time_in_millis":             215,
+	"indices_merges_current_docs":                       0,
+	"indices_merges_current_size_in_bytes":              0,
+	"indices_merges_total":                              133,
+	"indices_merges_total_time_in_millis":               21060,
+	"indices_merges_total_docs":                         203672,
+	"indices_merges_total_size_in_bytes":                142900226,
+	"indices_merges_current":                            0,
+	"indices_filter_cache_memory_size_in_bytes":         7384,
+	"indices_filter_cache_evictions":                    0,
+	"indices_indexing_index_total":                      84790,
+	"indices_indexing_index_time_in_millis":             29680,
+	"indices_indexing_index_current":                    0,
+	"indices_indexing_noop_update_total":                0,
+	"indices_indexing_throttle_time_in_millis":          0,
+	"indices_indexing_delete_total":                     13879,
+	"indices_indexing_delete_time_in_millis":            1139,
+	"indices_indexing_delete_current":                   0,
+	"indices_get_exists_time_in_millis":                 0,
+	"indices_get_missing_total":                         1,
+	"indices_get_missing_time_in_millis":                2,
+	"indices_get_current":                               0,
+	"indices_get_total":                                 1,
+	"indices_get_time_in_millis":                        2,
+	"indices_get_exists_total":                          0,
+	"indices_refresh_total":                             1076,
+	"indices_refresh_total_time_in_millis":              20078,
+	"indices_percolate_current":                         0,
+	"indices_percolate_memory_size_in_bytes":            -1,
+	"indices_percolate_queries":                         0,
+	"indices_percolate_total":                           0,
+	"indices_percolate_time_in_millis":                  0,
+	"indices_translog_operations":                       17702,
+	"indices_translog_size_in_bytes":                    17,
+	"indices_recovery_current_as_source":                0,
+	"indices_recovery_current_as_target":                0,
+	"indices_recovery_throttle_time_in_millis":          0,
+	"indices_docs_count":                                29652,
+	"indices_docs_deleted":                              5229,
+	"indices_flush_total_time_in_millis":                2401,
+	"indices_flush_total":                               115,
+	"indices_fielddata_memory_size_in_bytes":            12996,
+	"indices_fielddata_evictions":                       0,
+	"indices_search_fetch_current":                      0,
+	"indices_search_open_contexts":                      0,
+	"indices_search_query_total":                        1452,
+	"indices_search_query_time_in_millis":               5695,
+	"indices_search_query_current":                      0,
+	"indices_search_fetch_total":                        414,
+	"indices_search_fetch_time_in_millis":               146,
+	"indices_warmer_current":                            0,
+	"indices_warmer_total":                              2319,
+	"indices_warmer_total_time_in_millis":               448,
+	"indices_segments_count":                            134,
+	"indices_segments_memory_in_bytes":                  1285212,
+	"indices_segments_index_writer_memory_in_bytes":     0,
+	"indices_segments_index_writer_max_memory_in_bytes": 172368955,
+	"indices_segments_version_map_memory_in_bytes":      611844,
+	"indices_segments_fixed_bit_set_memory_in_bytes":    0,
+}
+
+var osExpected = map[string]float64{
+	"os_swap_used_in_bytes":       0,
+	"os_swap_free_in_bytes":       487997440,
+	"os_timestamp":                1436460392944,
+	"os_mem_free_percent":         74,
+	"os_mem_used_percent":         25,
+	"os_mem_actual_free_in_bytes": 1565470720,
+	"os_mem_actual_used_in_bytes": 534159360,
+	"os_mem_free_in_bytes":        477761536,
+	"os_mem_used_in_bytes":        1621868544,
+}
+
+var processExpected = map[string]float64{
+	"process_mem_total_virtual_in_bytes": 4747890688,
+	"process_timestamp":                  1436460392945,
+	"process_open_file_descriptors":      160,
+	"process_cpu_total_in_millis":        15480,
+	"process_cpu_percent":                2,
+	"process_cpu_sys_in_millis":          1870,
+	"process_cpu_user_in_millis":         13610,
+}
+
+var jvmExpected = map[string]float64{
+	"jvm_timestamp":                                     1436460392945,
+	"jvm_uptime_in_millis":                              202245,
+	"jvm_mem_non_heap_used_in_bytes":                    39634576,
+	"jvm_mem_non_heap_committed_in_bytes":               40841216,
+	"jvm_mem_pools_young_max_in_bytes":                  279183360,
+	"jvm_mem_pools_young_peak_used_in_bytes":            71630848,
+	"jvm_mem_pools_young_peak_max_in_bytes":             279183360,
+	"jvm_mem_pools_young_used_in_bytes":                 32685760,
+	"jvm_mem_pools_survivor_peak_used_in_bytes":         8912888,
+	"jvm_mem_pools_survivor_peak_max_in_bytes":          34865152,
+	"jvm_mem_pools_survivor_used_in_bytes":              8912880,
+	"jvm_mem_pools_survivor_max_in_bytes":               34865152,
+	"jvm_mem_pools_old_peak_max_in_bytes":               724828160,
+	"jvm_mem_pools_old_used_in_bytes":                   11110928,
+	"jvm_mem_pools_old_max_in_bytes":                    724828160,
+	"jvm_mem_pools_old_peak_used_in_bytes":              14354608,
+	"jvm_mem_heap_used_in_bytes":                        52709568,
+	"jvm_mem_heap_used_percent":                         5,
+	"jvm_mem_heap_committed_in_bytes":                   259522560,
+	"jvm_mem_heap_max_in_bytes":                         1038876672,
+	"jvm_threads_peak_count":                            45,
+	"jvm_threads_count":                                 44,
+	"jvm_gc_collectors_young_collection_count":          2,
+	"jvm_gc_collectors_young_collection_time_in_millis": 98,
+	"jvm_gc_collectors_old_collection_count":            1,
+	"jvm_gc_collectors_old_collection_time_in_millis":   24,
+	"jvm_buffer_pools_direct_count":                     40,
+	"jvm_buffer_pools_direct_used_in_bytes":             6304239,
+	"jvm_buffer_pools_direct_total_capacity_in_bytes":   6304239,
+	"jvm_buffer_pools_mapped_count":                     0,
+	"jvm_buffer_pools_mapped_used_in_bytes":             0,
+	"jvm_buffer_pools_mapped_total_capacity_in_bytes":   0,
+}
+
+var threadPoolExpected = map[string]float64{
+	"thread_pool_merge_threads":                 6,
+	"thread_pool_merge_queue":                   4,
+	"thread_pool_merge_active":                  5,
+	"thread_pool_merge_rejected":                2,
+	"thread_pool_merge_largest":                 5,
+	"thread_pool_merge_completed":               1,
+	"thread_pool_bulk_threads":                  4,
+	"thread_pool_bulk_queue":                    5,
+	"thread_pool_bulk_active":                   7,
+	"thread_pool_bulk_rejected":                 3,
+	"thread_pool_bulk_largest":                  1,
+	"thread_pool_bulk_completed":                4,
+	"thread_pool_warmer_threads":                2,
+	"thread_pool_warmer_queue":                  7,
+	"thread_pool_warmer_active":                 3,
+	"thread_pool_warmer_rejected":               2,
+	"thread_pool_warmer_largest":                3,
+	"thread_pool_warmer_completed":              1,
+	"thread_pool_get_largest":                   2,
+	"thread_pool_get_completed":                 1,
+	"thread_pool_get_threads":                   1,
+	"thread_pool_get_queue":                     8,
+	"thread_pool_get_active":                    4,
+	"thread_pool_get_rejected":                  3,
+	"thread_pool_index_threads":                 6,
+	"thread_pool_index_queue":                   8,
+	"thread_pool_index_active":                  4,
+	"thread_pool_index_rejected":                2,
+	"thread_pool_index_largest":                 3,
+	"thread_pool_index_completed":               6,
+	"thread_pool_suggest_threads":               2,
+	"thread_pool_suggest_queue":                 7,
+	"thread_pool_suggest_active":                2,
+	"thread_pool_suggest_rejected":              1,
+	"thread_pool_suggest_largest":               8,
+	"thread_pool_suggest_completed":             3,
+	"thread_pool_fetch_shard_store_queue":       7,
+	"thread_pool_fetch_shard_store_active":      4,
+	"thread_pool_fetch_shard_store_rejected":    2,
+	"thread_pool_fetch_shard_store_largest":     4,
+	"thread_pool_fetch_shard_store_completed":   1,
+	"thread_pool_fetch_shard_store_threads":     1,
+	"thread_pool_management_threads":            2,
+	"thread_pool_management_queue":              3,
+	"thread_pool_management_active":             1,
+	"thread_pool_management_rejected":           6,
+	"thread_pool_management_largest":            2,
+	"thread_pool_management_completed":          22,
+	"thread_pool_percolate_queue":               23,
+	"thread_pool_percolate_active":              13,
+	"thread_pool_percolate_rejected":            235,
+	"thread_pool_percolate_largest":             23,
+	"thread_pool_percolate_completed":           33,
+	"thread_pool_percolate_threads":             123,
+	"thread_pool_listener_active":               4,
+	"thread_pool_listener_rejected":             8,
+	"thread_pool_listener_largest":              1,
+	"thread_pool_listener_completed":            1,
+	"thread_pool_listener_threads":              1,
+	"thread_pool_listener_queue":                2,
+	"thread_pool_search_rejected":               7,
+	"thread_pool_search_largest":                2,
+	"thread_pool_search_completed":              4,
+	"thread_pool_search_threads":                5,
+	"thread_pool_search_queue":                  7,
+	"thread_pool_search_active":                 2,
+	"thread_pool_fetch_shard_started_threads":   3,
+	"thread_pool_fetch_shard_started_queue":     1,
+	"thread_pool_fetch_shard_started_active":    5,
+	"thread_pool_fetch_shard_started_rejected":  6,
+	"thread_pool_fetch_shard_started_largest":   4,
+	"thread_pool_fetch_shard_started_completed": 54,
+	"thread_pool_refresh_rejected":              4,
+	"thread_pool_refresh_largest":               8,
+	"thread_pool_refresh_completed":             3,
+	"thread_pool_refresh_threads":               23,
+	"thread_pool_refresh_queue":                 7,
+	"thread_pool_refresh_active":                3,
+	"thread_pool_optimize_threads":              3,
+	"thread_pool_optimize_queue":                4,
+	"thread_pool_optimize_active":               1,
+	"thread_pool_optimize_rejected":             2,
+	"thread_pool_optimize_largest":              7,
+	"thread_pool_optimize_completed":            3,
+	"thread_pool_snapshot_largest":              1,
+	"thread_pool_snapshot_completed":            0,
+	"thread_pool_snapshot_threads":              8,
+	"thread_pool_snapshot_queue":                5,
+	"thread_pool_snapshot_active":               6,
+	"thread_pool_snapshot_rejected":             2,
+	"thread_pool_generic_threads":               1,
+	"thread_pool_generic_queue":                 4,
+	"thread_pool_generic_active":                6,
+	"thread_pool_generic_rejected":              3,
+	"thread_pool_generic_largest":               2,
+	"thread_pool_generic_completed":             27,
+	"thread_pool_flush_threads":                 3,
+	"thread_pool_flush_queue":                   8,
+	"thread_pool_flush_active":                  0,
+	"thread_pool_flush_rejected":                1,
+	"thread_pool_flush_largest":                 5,
+	"thread_pool_flush_completed":               3,
+}
+
+var fsExpected = map[string]float64{
+	"fs_timestamp":                1436460392946,
+	"fs_total_free_in_bytes":      16909316096,
+	"fs_total_available_in_bytes": 15894814720,
+	"fs_total_total_in_bytes":     19507089408,
+}
+
+var transportExpected = map[string]float64{
+	"transport_server_open":      13,
+	"transport_rx_count":         6,
+	"transport_rx_size_in_bytes": 1380,
+	"transport_tx_count":         6,
+	"transport_tx_size_in_bytes": 1380,
+}
+
+var httpExpected = map[string]float64{
+	"http_current_open": 3,
+	"http_total_opened": 3,
+}
+
+var breakersExpected = map[string]float64{
+	"breakers_fielddata_estimated_size_in_bytes": 0,
+	"breakers_fielddata_overhead":                1.03,
+	"breakers_fielddata_tripped":                 0,
+	"breakers_fielddata_limit_size_in_bytes":     623326003,
+	"breakers_request_estimated_size_in_bytes":   0,
+	"breakers_request_overhead":                  1.0,
+	"breakers_request_tripped":                   0,
+	"breakers_request_limit_size_in_bytes":       415550668,
+	"breakers_parent_overhead":                   1.0,
+	"breakers_parent_tripped":                    0,
+	"breakers_parent_limit_size_in_bytes":        727213670,
+	"breakers_parent_estimated_size_in_bytes":    0,
+}

plugins/exec/README.md

@@ -0,0 +1,42 @@
+# Exec Plugin
+
+The exec plugin can execute arbitrary commands which output JSON. Then it flattens JSON and finds
+all numeric values, treating them as floats.
+
+For example, if you have a json-returning command called mycollector, you could
+setup the exec plugin with:
+
+```
+[[exec.commands]]
+command = "/usr/bin/mycollector --output=json"
+name = "mycollector"
+interval = 10
+```
+
+The name is used as a prefix for the measurements.
+
+The interval is used to determine how often a particular command should be run. Each
+time the exec plugin runs, it will only run a particular command if it has been at least
+`interval` seconds since the exec plugin last ran the command.
+
+
+# Sample
+
+Let's say that we have a command named "mycollector", which gives the following output:
+```json
+{
+    "a": 0.5,
+    "b": {
+        "c": "some text",
+        "d": 0.1,
+        "e": 5
+    }
+}
+```
+
+The collected metrics will be:
+```
+exec_mycollector_a value=0.5
+exec_mycollector_b_d value=0.1
+exec_mycollector_b_e value=5
+```

plugins/exec/exec.go

@@ -0,0 +1,98 @@
+package exec
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"os/exec"
+
+	"github.com/gonuts/go-shellquote"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+const sampleConfig = `
+  # the command to run
+  command = "/usr/bin/mycollector --foo=bar"
+
+  # name of the command (used as a prefix for measurements)
+  name = "mycollector"
+`
+
+type Exec struct {
+	Command string
+	Name    string
+
+	runner Runner
+}
+
+type Runner interface {
+	Run(*Exec) ([]byte, error)
+}
+
+type CommandRunner struct{}
+
+func (c CommandRunner) Run(e *Exec) ([]byte, error) {
+	split_cmd, err := shellquote.Split(e.Command)
+	if err != nil || len(split_cmd) == 0 {
+		return nil, fmt.Errorf("exec: unable to parse command, %s", err)
+	}
+
+	cmd := exec.Command(split_cmd[0], split_cmd[1:]...)
+	var out bytes.Buffer
+	cmd.Stdout = &out
+
+	if err := cmd.Run(); err != nil {
+		return nil, fmt.Errorf("exec: %s for command '%s'", err, e.Command)
+	}
+
+	return out.Bytes(), nil
+}
+
+func NewExec() *Exec {
+	return &Exec{runner: CommandRunner{}}
+}
+
+func (e *Exec) SampleConfig() string {
+	return sampleConfig
+}
+
+func (e *Exec) Description() string {
+	return "Read flattened metrics from one or more commands that output JSON to stdout"
+}
+
+func (e *Exec) Gather(acc plugins.Accumulator) error {
+	out, err := e.runner.Run(e)
+	if err != nil {
+		return err
+	}
+
+	var jsonOut interface{}
+	err = json.Unmarshal(out, &jsonOut)
+	if err != nil {
+		return fmt.Errorf("exec: unable to parse output of '%s' as JSON, %s",
+			e.Command, err)
+	}
+
+	f := internal.JSONFlattener{}
+	err = f.FlattenJSON("", jsonOut)
+	if err != nil {
+		return err
+	}
+
+	var msrmnt_name string
+	if e.Name == "" {
+		msrmnt_name = "exec"
+	} else {
+		msrmnt_name = "exec_" + e.Name
+	}
+	acc.AddFields(msrmnt_name, f.Fields, nil)
+	return nil
+}
+
+func init() {
+	plugins.Add("exec", func() plugins.Plugin {
+		return NewExec()
+	})
+}

plugins/exec/exec_test.go

@@ -0,0 +1,262 @@
+package exec
+
+import (
+	"fmt"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"math"
+	"testing"
+	"time"
+)
+
+// Midnight 9/22/2015
+const baseTimeSeconds = 1442905200
+
+const validJson = `
+{
+    "status": "green",
+    "num_processes": 82,
+    "cpu": {
+        "status": "red",
+        "nil_status": null,
+        "used": 8234,
+        "free": 32
+    },
+    "percent": 0.81,
+    "users": [0, 1, 2, 3]
+}`
+
+const malformedJson = `
+{
+    "status": "green",
+`
+
+type runnerMock struct {
+	out []byte
+	err error
+}
+
+type clockMock struct {
+	now time.Time
+}
+
+func newRunnerMock(out []byte, err error) Runner {
+	return &runnerMock{
+		out: out,
+		err: err,
+	}
+}
+
+func (r runnerMock) Run(command *Command) ([]byte, error) {
+	if r.err != nil {
+		return nil, r.err
+	}
+	return r.out, nil
+}
+
+func newClockMock(now time.Time) Clock {
+	return &clockMock{now: now}
+}
+
+func (c clockMock) Now() time.Time {
+	return c.now
+}
+
+func TestExec(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+20, 0))
+	command := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mycollector_num_processes", 82},
+		{"mycollector_cpu_used", 8234},
+		{"mycollector_cpu_free", 32},
+		{"mycollector_percent", 0.81},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	assert.Equal(t, deltaPoints, 4, "non-numeric measurements should be ignored")
+}
+
+func TestExecMalformed(t *testing.T) {
+	runner := newRunnerMock([]byte(malformedJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+20, 0))
+	command := Command{
+		Command:   "badcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.Error(t, err)
+
+	assert.Equal(t, deltaPoints, 0, "No new points should have been added")
+}
+
+func TestCommandError(t *testing.T) {
+	runner := newRunnerMock(nil, fmt.Errorf("exit status code 1"))
+	clock := newClockMock(time.Unix(baseTimeSeconds+20, 0))
+	command := Command{
+		Command:   "badcommand",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.Error(t, err)
+
+	assert.Equal(t, deltaPoints, 0, "No new points should have been added")
+}
+
+func TestExecNotEnoughTime(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+5, 0))
+	command := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	assert.Equal(t, deltaPoints, 0, "No new points should have been added")
+}
+
+func TestExecUninitializedLastRunAt(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds, 0))
+	command := Command{
+		Command:  "testcommand arg1",
+		Name:     "mycollector",
+		Interval: math.MaxInt32,
+		// Uninitialized lastRunAt should default to time.Unix(0, 0), so this should
+		// run no matter what the interval is
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mycollector_num_processes", 82},
+		{"mycollector_cpu_used", 8234},
+		{"mycollector_cpu_free", 32},
+		{"mycollector_percent", 0.81},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	assert.Equal(t, deltaPoints, 4, "non-numeric measurements should be ignored")
+}
+func TestExecOneNotEnoughTimeAndOneEnoughTime(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+5, 0))
+	notEnoughTimeCommand := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+	enoughTimeCommand := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  3,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&notEnoughTimeCommand, &enoughTimeCommand},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mycollector_num_processes", 82},
+		{"mycollector_cpu_used", 8234},
+		{"mycollector_cpu_free", 32},
+		{"mycollector_percent", 0.81},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	assert.Equal(t, deltaPoints, 4, "Only one command should have been run")
+}

plugins/haproxy/haproxy.go

@@ -0,0 +1,364 @@
+package haproxy
+
+import (
+	"encoding/csv"
+	"fmt"
+	"github.com/influxdb/telegraf/plugins"
+	"io"
+	"net/http"
+	"net/url"
+	"strconv"
+	"sync"
+	"time"
+)
+
+//CSV format: https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.1
+const (
+	HF_PXNAME         = 0  // 0. pxname [LFBS]: proxy name
+	HF_SVNAME         = 1  // 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend, any name for server/listener)
+	HF_QCUR           = 2  //2. qcur [..BS]: current queued requests. For the backend this reports the number queued without a server assigned.
+	HF_QMAX           = 3  //3. qmax [..BS]: max value of qcur
+	HF_SCUR           = 4  // 4. scur [LFBS]: current sessions
+	HF_SMAX           = 5  //5. smax [LFBS]: max sessions
+	HF_SLIM           = 6  //6. slim [LFBS]: configured session limit
+	HF_STOT           = 7  //7. stot [LFBS]: cumulative number of connections
+	HF_BIN            = 8  //8. bin [LFBS]: bytes in
+	HF_BOUT           = 9  //9. bout [LFBS]: bytes out
+	HF_DREQ           = 10 //10. dreq [LFB.]: requests denied because of security concerns.
+	HF_DRESP          = 11 //11. dresp [LFBS]: responses denied because of security concerns.
+	HF_EREQ           = 12 //12. ereq [LF..]: request errors. Some of the possible causes are:
+	HF_ECON           = 13 //13. econ [..BS]: number of requests that encountered an error trying to
+	HF_ERESP          = 14 //14. eresp [..BS]: response errors. srv_abrt will be counted here also.  Some other errors are: - write error on the client socket (won't be counted for the server stat) - failure applying filters to the response.
+	HF_WRETR          = 15 //15. wretr [..BS]: number of times a connection to a server was retried.
+	HF_WREDIS         = 16 //16. wredis [..BS]: number of times a request was redispatched to another server. The server value counts the number of times that server was switched away from.
+	HF_STATUS         = 17 //17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)...)
+	HF_WEIGHT         = 18 //18. weight [..BS]: total weight (backend), server weight (server)
+	HF_ACT            = 19 //19. act [..BS]: number of active servers (backend), server is active (server)
+	HF_BCK            = 20 //20. bck [..BS]: number of backup servers (backend), server is backup (server)
+	HF_CHKFAIL        = 21 //21. chkfail [...S]: number of failed checks. (Only counts checks failed when the server is up.)
+	HF_CHKDOWN        = 22 //22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts transitions to the whole backend being down, rather than the sum of the counters for each server.
+	HF_LASTCHG        = 23 //23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
+	HF_DOWNTIME       = 24 //24. downtime [..BS]: total downtime (in seconds). The value for the backend is the downtime for the whole backend, not the sum of the server downtime.
+	HF_QLIMIT         = 25 //25. qlimit [...S]: configured maxqueue for the server, or nothing in the value is 0 (default, meaning no limit)
+	HF_PID            = 26 //26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
+	HF_IID            = 27 //27. iid [LFBS]: unique proxy id
+	HF_SID            = 28 //28. sid [L..S]: server id (unique inside a proxy)
+	HF_THROTTLE       = 29 //29. throttle [...S]: current throttle percentage for the server, when slowstart is active, or no value if not in slowstart.
+	HF_LBTOT          = 30 //30. lbtot [..BS]: total number of times a server was selected, either for new sessions, or when re-dispatching. The server counter is the number of times that server was selected.
+	HF_TRACKED        = 31 //31. tracked [...S]: id of proxy/server if tracking is enabled.
+	HF_TYPE           = 32 //32. type [LFBS]: (0                                                                                                                                                                                                  = frontend, 1 = backend, 2 = server, 3 = socket/listener)
+	HF_RATE           = 33 //33. rate [.FBS]: number of sessions per second over last elapsed second
+	HF_RATE_LIM       = 34 //34. rate_lim [.F..]: configured limit on new sessions per second
+	HF_RATE_MAX       = 35 //35. rate_max [.FBS]: max number of new sessions per second
+	HF_CHECK_STATUS   = 36 //36. check_status [...S]: status of last health check, one of:
+	HF_CHECK_CODE     = 37 //37. check_code [...S]: layer5-7 code, if available
+	HF_CHECK_DURATION = 38 //38. check_duration [...S]: time in ms took to finish last health check
+	HF_HRSP_1xx       = 39 //39. hrsp_1xx [.FBS]: http responses with 1xx code
+	HF_HRSP_2xx       = 40 //40. hrsp_2xx [.FBS]: http responses with 2xx code
+	HF_HRSP_3xx       = 41 //41. hrsp_3xx [.FBS]: http responses with 3xx code
+	HF_HRSP_4xx       = 42 //42. hrsp_4xx [.FBS]: http responses with 4xx code
+	HF_HRSP_5xx       = 43 //43. hrsp_5xx [.FBS]: http responses with 5xx code
+	HF_HRSP_OTHER     = 44 //44. hrsp_other [.FBS]: http responses with other codes (protocol error)
+	HF_HANAFAIL       = 45 //45. hanafail [...S]: failed health checks details
+	HF_REQ_RATE       = 46 //46. req_rate [.F..]: HTTP requests per second over last elapsed second
+	HF_REQ_RATE_MAX   = 47 //47. req_rate_max [.F..]: max number of HTTP requests per second observed
+	HF_REQ_TOT        = 48 //48. req_tot [.F..]: total number of HTTP requests received
+	HF_CLI_ABRT       = 49 //49. cli_abrt [..BS]: number of data transfers aborted by the client
+	HF_SRV_ABRT       = 50 //50. srv_abrt [..BS]: number of data transfers aborted by the server (inc. in eresp)
+	HF_COMP_IN        = 51 //51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
+	HF_COMP_OUT       = 52 //52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
+	HF_COMP_BYP       = 53 //53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor (CPU/BW limit)
+	HF_COMP_RSP       = 54 //54. comp_rsp [.FB.]: number of HTTP responses that were compressed
+	HF_LASTSESS       = 55 //55. lastsess [..BS]: number of seconds since last session assigned to server/backend
+	HF_LAST_CHK       = 56 //56. last_chk [...S]: last health check contents or textual error
+	HF_LAST_AGT       = 57 //57. last_agt [...S]: last agent check contents or textual error
+	HF_QTIME          = 58 //58. qtime [..BS]:
+	HF_CTIME          = 59 //59. ctime [..BS]:
+	HF_RTIME          = 60 //60. rtime [..BS]: (0 for TCP)
+	HF_TTIME          = 61 //61. ttime [..BS]: the average total session time in ms over the 1024 last requests
+)
+
+type haproxy struct {
+	Servers []string
+
+	client *http.Client
+}
+
+var sampleConfig = `
+  # An array of address to gather stats about. Specify an ip on hostname
+  # with optional port. ie localhost, 10.10.3.33:1936, etc.
+  #
+  # If no servers are specified, then default to 127.0.0.1:1936
+  servers = ["http://myhaproxy.com:1936", "http://anotherhaproxy.com:1936"]
+  # Or you can also use local socket(not work yet)
+  # servers = ["socket:/run/haproxy/admin.sock"]
+`
+
+func (r *haproxy) SampleConfig() string {
+	return sampleConfig
+}
+
+func (r *haproxy) Description() string {
+	return "Read metrics of haproxy, via socket or csv stats page"
+}
+
+// Reads stats from all configured servers accumulates stats.
+// Returns one of the errors encountered while gather stats (if any).
+func (g *haproxy) Gather(acc plugins.Accumulator) error {
+	if len(g.Servers) == 0 {
+		return g.gatherServer("http://127.0.0.1:1936", acc)
+	}
+
+	var wg sync.WaitGroup
+
+	var outerr error
+
+	for _, serv := range g.Servers {
+		wg.Add(1)
+		go func(serv string) {
+			defer wg.Done()
+			outerr = g.gatherServer(serv, acc)
+		}(serv)
+	}
+
+	wg.Wait()
+
+	return outerr
+}
+
+func (g *haproxy) gatherServer(addr string, acc plugins.Accumulator) error {
+	if g.client == nil {
+
+		client := &http.Client{}
+		g.client = client
+	}
+
+	u, err := url.Parse(addr)
+	if err != nil {
+		return fmt.Errorf("Unable parse server address '%s': %s", addr, err)
+	}
+
+	req, err := http.NewRequest("GET", fmt.Sprintf("%s://%s%s/;csv", u.Scheme, u.Host, u.Path), nil)
+	if u.User != nil {
+		p, _ := u.User.Password()
+		req.SetBasicAuth(u.User.Username(), p)
+	}
+
+	res, err := g.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("Unable to connect to haproxy server '%s': %s", addr, err)
+	}
+
+	if res.StatusCode != 200 {
+		return fmt.Errorf("Unable to get valid stat result from '%s': %s", addr, err)
+	}
+
+	return importCsvResult(res.Body, acc, u.Host)
+}
+
+func importCsvResult(r io.Reader, acc plugins.Accumulator, host string) error {
+	csv := csv.NewReader(r)
+	result, err := csv.ReadAll()
+	now := time.Now()
+
+	for _, row := range result {
+		fields := make(map[string]interface{})
+		tags := map[string]string{
+			"server": host,
+			"proxy":  row[HF_PXNAME],
+			"sv":     row[HF_SVNAME],
+		}
+		for field, v := range row {
+			switch field {
+			case HF_QCUR:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["qcur"] = ival
+				}
+			case HF_QMAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["qmax"] = ival
+				}
+			case HF_SCUR:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["scur"] = ival
+				}
+			case HF_SMAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["smax"] = ival
+				}
+			case HF_STOT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["stot"] = ival
+				}
+			case HF_BIN:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["bin"] = ival
+				}
+			case HF_BOUT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["bout"] = ival
+				}
+			case HF_DREQ:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["dreq"] = ival
+				}
+			case HF_DRESP:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["dresp"] = ival
+				}
+			case HF_EREQ:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["ereq"] = ival
+				}
+			case HF_ECON:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["econ"] = ival
+				}
+			case HF_ERESP:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["eresp"] = ival
+				}
+			case HF_WRETR:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["wretr"] = ival
+				}
+			case HF_WREDIS:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["wredis"] = ival
+				}
+			case HF_ACT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["active_servers"] = ival
+				}
+			case HF_BCK:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["backup_servers"] = ival
+				}
+			case HF_DOWNTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["downtime"] = ival
+				}
+			case HF_THROTTLE:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["throttle"] = ival
+				}
+			case HF_LBTOT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["lbtot"] = ival
+				}
+			case HF_RATE:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["rate"] = ival
+				}
+			case HF_RATE_MAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["rate_max"] = ival
+				}
+			case HF_CHECK_DURATION:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["check_duration"] = ival
+				}
+			case HF_HRSP_1xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.1xx"] = ival
+				}
+			case HF_HRSP_2xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.2xx"] = ival
+				}
+			case HF_HRSP_3xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.3xx"] = ival
+				}
+			case HF_HRSP_4xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.4xx"] = ival
+				}
+			case HF_HRSP_5xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.5xx"] = ival
+				}
+			case HF_REQ_RATE:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["req_rate"] = ival
+				}
+			case HF_REQ_RATE_MAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["req_rate_max"] = ival
+				}
+			case HF_REQ_TOT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["req_tot"] = ival
+				}
+			case HF_CLI_ABRT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["cli_abort"] = ival
+				}
+			case HF_SRV_ABRT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["srv_abort"] = ival
+				}
+			case HF_QTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["qtime"] = ival
+				}
+			case HF_CTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["ctime"] = ival
+				}
+			case HF_RTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["rtime"] = ival
+				}
+			case HF_TTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["ttime"] = ival
+				}
+			}
+		}
+		acc.AddFields("haproxy", fields, tags, now)
+	}
+	return err
+}
+
+func init() {
+	plugins.Add("haproxy", func() plugins.Plugin {
+		return &haproxy{}
+	})
+}

plugins/haproxy/haproxy_test.go

@@ -0,0 +1,163 @@
+package haproxy
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"net/http"
+	"net/http/httptest"
+)
+
+func TestHaproxyGeneratesMetricsWithAuthentication(t *testing.T) {
+	//We create a fake server to return test data
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		username, password, ok := r.BasicAuth()
+		if !ok {
+			w.WriteHeader(http.StatusNotFound)
+			fmt.Fprint(w, "Unauthorized")
+			return
+		}
+
+		if username == "user" && password == "password" {
+			fmt.Fprint(w, csvOutputSample)
+		} else {
+			w.WriteHeader(http.StatusNotFound)
+			fmt.Fprint(w, "Unauthorized")
+		}
+	}))
+	defer ts.Close()
+
+	//Now we tested again above server, with our authentication data
+	r := &haproxy{
+		Servers: []string{strings.Replace(ts.URL, "http://", "http://user:password@", 1)},
+	}
+
+	var acc testutil.Accumulator
+
+	err := r.Gather(&acc)
+	require.NoError(t, err)
+
+	tags := map[string]string{
+		"server": ts.Listener.Addr().String(),
+		"proxy":  "be_app",
+		"sv":     "host0",
+	}
+
+	assert.NoError(t, acc.ValidateTaggedValue("stot", uint64(171014), tags))
+
+	checkInt := []struct {
+		name  string
+		value uint64
+	}{
+
+		{"qmax", 81},
+		{"scur", 288},
+		{"smax", 713},
+		{"bin", 5557055817},
+		{"bout", 24096715169},
+		{"dreq", 1102},
+		{"dresp", 80},
+		{"ereq", 95740},
+		{"econ", 0},
+		{"eresp", 0},
+		{"wretr", 17},
+		{"wredis", 19},
+		{"active_servers", 1},
+		{"backup_servers", 0},
+		{"downtime", 0},
+		{"throttle", 13},
+		{"lbtot", 114},
+		{"rate", 18},
+		{"rate_max", 102},
+		{"check_duration", 1},
+		{"http_response.1xx", 0},
+		{"http_response.2xx", 1314093},
+		{"http_response.3xx", 537036},
+		{"http_response.4xx", 123452},
+		{"http_response.5xx", 11966},
+		{"req_rate", 35},
+		{"req_rate_max", 140},
+		{"req_tot", 1987928},
+		{"cli_abort", 0},
+		{"srv_abort", 0},
+		{"qtime", 0},
+		{"ctime", 2},
+		{"rtime", 23},
+		{"ttime", 545},
+	}
+
+	for _, c := range checkInt {
+		assert.Equal(t, true, acc.CheckValue(c.name, c.value))
+	}
+
+	//Here, we should get error because we don't pass authentication data
+	r = &haproxy{
+		Servers: []string{ts.URL},
+	}
+
+	err = r.Gather(&acc)
+	require.Error(t, err)
+}
+
+func TestHaproxyGeneratesMetricsWithoutAuthentication(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		fmt.Fprint(w, csvOutputSample)
+	}))
+	defer ts.Close()
+
+	r := &haproxy{
+		Servers: []string{ts.URL},
+	}
+
+	var acc testutil.Accumulator
+
+	err := r.Gather(&acc)
+	require.NoError(t, err)
+
+	tags := map[string]string{
+		"proxy":  "be_app",
+		"server": ts.Listener.Addr().String(),
+		"sv":     "host0",
+	}
+
+	assert.NoError(t, acc.ValidateTaggedValue("stot", uint64(171014), tags))
+	assert.NoError(t, acc.ValidateTaggedValue("scur", uint64(1), tags))
+	assert.NoError(t, acc.ValidateTaggedValue("rate", uint64(3), tags))
+	assert.Equal(t, true, acc.CheckValue("bin", uint64(5557055817)))
+}
+
+//When not passing server config, we default to localhost
+//We just want to make sure we did request stat from localhost
+func TestHaproxyDefaultGetFromLocalhost(t *testing.T) {
+	r := &haproxy{}
+
+	var acc testutil.Accumulator
+
+	err := r.Gather(&acc)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "127.0.0.1:1936/;csv")
+}
+
+const csvOutputSample = `
+# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
+fe_app,FRONTEND,,81,288,713,2000,1094063,5557055817,24096715169,1102,80,95740,,,17,19,OPEN,,,,,,,,,2,16,113,13,114,,0,18,0,102,,,,0,1314093,537036,123452,11966,1360,,35,140,1987928,,,0,0,0,0,,,,,,,,
+be_static,host0,0,0,0,3,,3209,1141294,17389596,,0,,0,0,0,0,no check,1,1,0,,,,,,2,17,1,,3209,,2,0,,7,,,,0,218,1497,1494,0,0,0,,,,0,0,,,,,2,,,0,2,23,545,
+be_static,BACKEND,0,0,0,3,200,3209,1141294,17389596,0,0,,0,0,0,0,UP,1,1,0,,0,70698,0,,2,17,0,,3209,,1,0,,7,,,,0,218,1497,1494,0,0,,,,,0,0,0,0,0,0,2,,,0,2,23,545,
+be_static,host0,0,0,0,1,,28,17313,466003,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,1,,28,,2,0,,1,L4OK,,1,0,17,6,5,0,0,0,,,,0,0,,,,,2103,,,0,1,1,36,
+be_static,host4,0,0,0,1,,28,15358,1281073,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,2,,28,,2,0,,1,L4OK,,1,0,20,5,3,0,0,0,,,,0,0,,,,,2076,,,0,1,1,54,
+be_static,host5,0,0,0,1,,28,17547,1970404,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,3,,28,,2,0,,1,L4OK,,0,0,20,5,3,0,0,0,,,,0,0,,,,,1495,,,0,1,1,53,
+be_static,host6,0,0,0,1,,28,14105,1328679,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,4,,28,,2,0,,1,L4OK,,0,0,18,8,2,0,0,0,,,,0,0,,,,,1418,,,0,0,1,49,
+be_static,host7,0,0,0,1,,28,15258,1965185,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,5,,28,,2,0,,1,L4OK,,0,0,17,8,3,0,0,0,,,,0,0,,,,,935,,,0,0,1,28,
+be_static,host8,0,0,0,1,,28,12934,1034779,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,6,,28,,2,0,,1,L4OK,,0,0,17,9,2,0,0,0,,,,0,0,,,,,582,,,0,1,1,66,
+be_static,host9,0,0,0,1,,28,13434,134063,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,7,,28,,2,0,,1,L4OK,,0,0,17,8,3,0,0,0,,,,0,0,,,,,539,,,0,0,1,80,
+be_static,host1,0,0,0,1,,28,7873,1209688,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,8,,28,,2,0,,1,L4OK,,0,0,22,6,0,0,0,0,,,,0,0,,,,,487,,,0,0,1,36,
+be_static,host2,0,0,0,1,,28,13830,1085929,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,9,,28,,2,0,,1,L4OK,,0,0,19,6,3,0,0,0,,,,0,0,,,,,338,,,0,1,1,38,
+be_static,host3,0,0,0,1,,28,17959,1259760,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,10,,28,,2,0,,1,L4OK,,1,0,20,6,2,0,0,0,,,,0,0,,,,,92,,,0,1,1,17,
+be_static,BACKEND,0,0,0,2,200,307,160276,13322728,0,0,,0,0,0,0,UP,11,11,0,,0,70698,0,,2,18,0,,307,,1,0,,4,,,,0,205,73,29,0,0,,,,,0,0,0,0,0,0,92,,,0,1,3,381,
+be_app,host0,0,0,1,32,,171014,510913516,2193856571,,0,,0,1,1,0,UP,100,1,0,1,0,70698,0,,2,19,1,,171013,,2,3,,12,L7OK,301,10,0,119534,48051,2345,1056,0,0,,,,73,1,,,,,0,Moved Permanently,,0,2,312,2341,
+be_app,host4,0,0,2,29,,171013,499318742,2195595896,12,34,,0,2,0,0,UP,100,1,0,2,0,70698,0,,2,19,2,,171013,,2,3,,12,L7OK,301,12,0,119572,47882,2441,1088,0,0,,,,84,2,,,,,0,Moved Permanently,,0,2,316,2355,
+`

plugins/httpjson/README.md

@@ -0,0 +1,148 @@
+# HTTP JSON Plugin
+
+The httpjson plugin can collect data from remote URLs which respond with JSON. Then it flattens JSON and finds all numeric values, treating them as floats.
+
+For example, if you have a service called _mycollector_, which has HTTP endpoint for gathering stats at http://my.service.com/_stats, you would configure the HTTP JSON
+plugin like this:
+
+```
+[[httpjson.services]]
+  name = "mycollector"
+
+  servers = [
+    "http://my.service.com/_stats"
+  ]
+
+  # HTTP method to use (case-sensitive)
+  method = "GET"
+```
+
+`name` is used as a prefix for the measurements.
+
+`method` specifies HTTP method to use for requests.
+
+You can also specify which keys from server response should be considered tags:
+
+```
+[[httpjson.services]]
+  ...
+
+  tag_keys = [
+    "role",
+    "version"
+  ]
+```
+
+You can also specify additional request parameters for the service:
+
+```
+[[httpjson.services]]
+  ...
+
+ [httpjson.services.parameters]
+    event_type = "cpu_spike"
+    threshold = "0.75"
+
+```
+
+
+# Example:
+
+Let's say that we have a service named "mycollector" configured like this:
+
+```
+[httpjson]
+  [[httpjson.services]]
+    name = "mycollector"
+
+    servers = [
+      "http://my.service.com/_stats"
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "GET"
+
+    tag_keys = ["service"]
+```
+
+which responds with the following JSON:
+
+```json
+{
+    "service": "service01",
+    "a": 0.5,
+    "b": {
+        "c": "some text",
+        "d": 0.1,
+        "e": 5
+    }
+}
+```
+
+The collected metrics will be:
+```
+httpjson_mycollector_a,service='service01',server='http://my.service.com/_stats' value=0.5
+httpjson_mycollector_b_d,service='service01',server='http://my.service.com/_stats' value=0.1
+httpjson_mycollector_b_e,service='service01',server='http://my.service.com/_stats' value=5
+```
+
+# Example 2, Multiple Services:
+
+There is also the option to collect JSON from multiple services, here is an
+example doing that.
+
+```
+[httpjson]
+  [[httpjson.services]]
+    name = "mycollector1"
+
+    servers = [
+      "http://my.service1.com/_stats"
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "GET"
+
+  [[httpjson.services]]
+    name = "mycollector2"
+
+    servers = [
+      "http://service.net/json/stats"
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "POST"
+```
+
+The services respond with the following JSON:
+
+mycollector1:
+```json
+{
+    "a": 0.5,
+    "b": {
+        "c": "some text",
+        "d": 0.1,
+        "e": 5
+    }
+}
+```
+
+mycollector2:
+```json
+{
+    "load": 100,
+    "users": 1335
+}
+```
+
+The collected metrics will be:
+
+```
+httpjson_mycollector1_a,server='http://my.service.com/_stats' value=0.5
+httpjson_mycollector1_b_d,server='http://my.service.com/_stats' value=0.1
+httpjson_mycollector1_b_e,server='http://my.service.com/_stats' value=5
+
+httpjson_mycollector2_load,server='http://service.net/json/stats' value=100
+httpjson_mycollector2_users,server='http://service.net/json/stats' value=1335
+```

plugins/httpjson/httpjson.go

@@ -0,0 +1,217 @@
+package httpjson
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/url"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type HttpJson struct {
+	Name       string
+	Servers    []string
+	Method     string
+	TagKeys    []string
+	Parameters map[string]string
+	client     HTTPClient
+}
+
+type HTTPClient interface {
+	// Returns the result of an http request
+	//
+	// Parameters:
+	// req: HTTP request object
+	//
+	// Returns:
+	// http.Response:  HTTP respons object
+	// error        :  Any error that may have occurred
+	MakeRequest(req *http.Request) (*http.Response, error)
+}
+
+type RealHTTPClient struct {
+	client *http.Client
+}
+
+func (c RealHTTPClient) MakeRequest(req *http.Request) (*http.Response, error) {
+	return c.client.Do(req)
+}
+
+var sampleConfig = `
+  # a name for the service being polled
+  name = "webserver_stats"
+
+  # URL of each server in the service's cluster
+  servers = [
+    "http://localhost:9999/stats/",
+    "http://localhost:9998/stats/",
+  ]
+
+  # HTTP method to use (case-sensitive)
+  method = "GET"
+
+  # List of tag names to extract from top-level of JSON server response
+  # tag_keys = [
+  #   "my_tag_1",
+  #   "my_tag_2"
+  # ]
+
+  # HTTP parameters (all values must be strings)
+  [plugins.httpjson.parameters]
+    event_type = "cpu_spike"
+    threshold = "0.75"
+`
+
+func (h *HttpJson) SampleConfig() string {
+	return sampleConfig
+}
+
+func (h *HttpJson) Description() string {
+	return "Read flattened metrics from one or more JSON HTTP endpoints"
+}
+
+// Gathers data for all servers.
+func (h *HttpJson) Gather(acc plugins.Accumulator) error {
+	var wg sync.WaitGroup
+
+	errorChannel := make(chan error, len(h.Servers))
+
+	for _, server := range h.Servers {
+		wg.Add(1)
+		go func(server string) {
+			defer wg.Done()
+			if err := h.gatherServer(acc, server); err != nil {
+				errorChannel <- err
+			}
+		}(server)
+	}
+
+	wg.Wait()
+	close(errorChannel)
+
+	// Get all errors and return them as one giant error
+	errorStrings := []string{}
+	for err := range errorChannel {
+		errorStrings = append(errorStrings, err.Error())
+	}
+
+	if len(errorStrings) == 0 {
+		return nil
+	}
+	return errors.New(strings.Join(errorStrings, "\n"))
+}
+
+// Gathers data from a particular server
+// Parameters:
+//     acc      : The telegraf Accumulator to use
+//     serverURL: endpoint to send request to
+//     service  : the service being queried
+//
+// Returns:
+//     error: Any error that may have occurred
+func (h *HttpJson) gatherServer(
+	acc plugins.Accumulator,
+	serverURL string,
+) error {
+	resp, err := h.sendRequest(serverURL)
+	if err != nil {
+		return err
+	}
+
+	var jsonOut map[string]interface{}
+	if err = json.Unmarshal([]byte(resp), &jsonOut); err != nil {
+		return errors.New("Error decoding JSON response")
+	}
+
+	tags := map[string]string{
+		"server": serverURL,
+	}
+
+	for _, tag := range h.TagKeys {
+		switch v := jsonOut[tag].(type) {
+		case string:
+			tags[tag] = v
+		}
+		delete(jsonOut, tag)
+	}
+
+	f := internal.JSONFlattener{}
+	err = f.FlattenJSON("", jsonOut)
+	if err != nil {
+		return err
+	}
+
+	var msrmnt_name string
+	if h.Name == "" {
+		msrmnt_name = "httpjson"
+	} else {
+		msrmnt_name = "httpjson_" + h.Name
+	}
+	acc.AddFields(msrmnt_name, f.Fields, nil)
+	return nil
+}
+
+// Sends an HTTP request to the server using the HttpJson object's HTTPClient
+// Parameters:
+//     serverURL: endpoint to send request to
+//
+// Returns:
+//     string: body of the response
+//     error : Any error that may have occurred
+func (h *HttpJson) sendRequest(serverURL string) (string, error) {
+	// Prepare URL
+	requestURL, err := url.Parse(serverURL)
+	if err != nil {
+		return "", fmt.Errorf("Invalid server URL \"%s\"", serverURL)
+	}
+
+	params := url.Values{}
+	for k, v := range h.Parameters {
+		params.Add(k, v)
+	}
+	requestURL.RawQuery = params.Encode()
+
+	// Create + send request
+	req, err := http.NewRequest(h.Method, requestURL.String(), nil)
+	if err != nil {
+		return "", err
+	}
+	defer req.Body.Close()
+
+	resp, err := h.client.MakeRequest(req)
+	if err != nil {
+		return "", err
+	}
+	defer resp.Body.Close()
+
+	defer resp.Body.Close()
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return string(body), err
+	}
+
+	// Process response
+	if resp.StatusCode != http.StatusOK {
+		err = fmt.Errorf("Response from url \"%s\" has status code %d (%s), expected %d (%s)",
+			requestURL.String(),
+			resp.StatusCode,
+			http.StatusText(resp.StatusCode),
+			http.StatusOK,
+			http.StatusText(http.StatusOK))
+		return string(body), err
+	}
+
+	return string(body), err
+}
+
+func init() {
+	plugins.Add("httpjson", func() plugins.Plugin {
+		return &HttpJson{client: RealHTTPClient{client: &http.Client{}}}
+	})
+}

plugins/httpjson/httpjson_test.go

@@ -0,0 +1,223 @@
+package httpjson
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const validJSON = `
+	{
+		"parent": {
+			"child": 3,
+			"ignored_child": "hi"
+		},
+		"ignored_null": null,
+		"integer": 4,
+		"ignored_list": [3, 4],
+		"ignored_parent": {
+			"another_ignored_list": [4],
+			"another_ignored_null": null,
+			"ignored_string": "hello, world!"
+		}
+	}`
+
+const validJSONTags = `
+	{
+		"value": 15,
+		"role": "master",
+		"build": "123"
+	}`
+
+const invalidJSON = "I don't think this is JSON"
+
+const empty = ""
+
+type mockHTTPClient struct {
+	responseBody string
+	statusCode   int
+}
+
+// Mock implementation of MakeRequest. Usually returns an http.Response with
+// hard-coded responseBody and statusCode. However, if the request uses a
+// nonstandard method, it uses status code 405 (method not allowed)
+func (c mockHTTPClient) MakeRequest(req *http.Request) (*http.Response, error) {
+	resp := http.Response{}
+	resp.StatusCode = c.statusCode
+
+	// basic error checking on request method
+	allowedMethods := []string{"GET", "HEAD", "POST", "PUT", "DELETE", "TRACE", "CONNECT"}
+	methodValid := false
+	for _, method := range allowedMethods {
+		if req.Method == method {
+			methodValid = true
+			break
+		}
+	}
+
+	if !methodValid {
+		resp.StatusCode = 405 // Method not allowed
+	}
+
+	resp.Body = ioutil.NopCloser(strings.NewReader(c.responseBody))
+	return &resp, nil
+}
+
+// Generates a pointer to an HttpJson object that uses a mock HTTP client.
+// Parameters:
+//     response  : Body of the response that the mock HTTP client should return
+//     statusCode: HTTP status code the mock HTTP client should return
+//
+// Returns:
+//     *HttpJson: Pointer to an HttpJson object that uses the generated mock HTTP client
+func genMockHttpJson(response string, statusCode int) *HttpJson {
+	return &HttpJson{
+		client: mockHTTPClient{responseBody: response, statusCode: statusCode},
+		Services: []Service{
+			Service{
+				Servers: []string{
+					"http://server1.example.com/metrics/",
+					"http://server2.example.com/metrics/",
+				},
+				Name:   "my_webapp",
+				Method: "GET",
+				Parameters: map[string]string{
+					"httpParam1": "12",
+					"httpParam2": "the second parameter",
+				},
+			},
+			Service{
+				Servers: []string{
+					"http://server3.example.com/metrics/",
+					"http://server4.example.com/metrics/",
+				},
+				Name:   "other_webapp",
+				Method: "POST",
+				Parameters: map[string]string{
+					"httpParam1": "12",
+					"httpParam2": "the second parameter",
+				},
+				TagKeys: []string{
+					"role",
+					"build",
+				},
+			},
+		},
+	}
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJson200(t *testing.T) {
+	httpjson := genMockHttpJson(validJSON, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+	require.NoError(t, err)
+
+	assert.Equal(t, 8, len(acc.Points))
+
+	for _, service := range httpjson.Services {
+		for _, srv := range service.Servers {
+			require.NoError(t,
+				acc.ValidateTaggedValue(
+					fmt.Sprintf("%s_parent_child", service.Name),
+					3.0,
+					map[string]string{"server": srv},
+				),
+			)
+			require.NoError(t,
+				acc.ValidateTaggedValue(
+					fmt.Sprintf("%s_integer", service.Name),
+					4.0,
+					map[string]string{"server": srv},
+				),
+			)
+		}
+	}
+}
+
+// Test response to HTTP 500
+func TestHttpJson500(t *testing.T) {
+	httpjson := genMockHttpJson(validJSON, 500)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 4 error lines for (2 urls) * (2 services)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 4)
+	assert.Equal(t, 0, len(acc.Points))
+}
+
+// Test response to HTTP 405
+func TestHttpJsonBadMethod(t *testing.T) {
+	httpjson := genMockHttpJson(validJSON, 200)
+	httpjson.Services[0].Method = "NOT_A_REAL_METHOD"
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 2 error lines for (2 urls) * (1 falied service)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 2)
+
+	// (2 measurements) * (2 servers) * (1 successful service)
+	assert.Equal(t, 4, len(acc.Points))
+}
+
+// Test response to malformed JSON
+func TestHttpJsonBadJson(t *testing.T) {
+	httpjson := genMockHttpJson(invalidJSON, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 4 error lines for (2 urls) * (2 services)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 4)
+	assert.Equal(t, 0, len(acc.Points))
+}
+
+// Test response to empty string as response objectgT
+func TestHttpJsonEmptyResponse(t *testing.T) {
+	httpjson := genMockHttpJson(empty, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 4 error lines for (2 urls) * (2 services)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 4)
+	assert.Equal(t, 0, len(acc.Points))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJson200Tags(t *testing.T) {
+	httpjson := genMockHttpJson(validJSONTags, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+	require.NoError(t, err)
+
+	assert.Equal(t, 4, len(acc.Points))
+
+	for _, service := range httpjson.Services {
+		if service.Name == "other_webapp" {
+			for _, srv := range service.Servers {
+				require.NoError(t,
+					acc.ValidateTaggedValue(
+						fmt.Sprintf("%s_value", service.Name),
+						15.0,
+						map[string]string{"server": srv, "role": "master", "build": "123"},
+					),
+				)
+			}
+		}
+	}
+}

plugins/influxdb/README.md

@@ -0,0 +1,72 @@
+# influxdb plugin
+
+The influxdb plugin collects InfluxDB-formatted data from JSON endpoints.
+
+With a configuration of:
+
+```toml
+[[plugins.influxdb]]
+  urls = [
+    "http://127.0.0.1:8086/debug/vars",
+    "http://192.168.2.1:8086/debug/vars"
+  ]
+```
+
+And if 127.0.0.1 responds with this JSON:
+
+```json
+{
+  "k1": {
+    "name": "fruit",
+    "tags": {
+      "kind": "apple"
+    },
+    "values": {
+      "inventory": 371,
+      "sold": 112
+    }
+  },
+  "k2": {
+    "name": "fruit",
+    "tags": {
+      "kind": "banana"
+    },
+    "values": {
+      "inventory": 1000,
+      "sold": 403
+    }
+  }
+}
+```
+
+And if 192.168.2.1 responds like so:
+
+```json
+{
+  "k3": {
+    "name": "transactions",
+    "tags": {},
+    "values": {
+      "total": 100,
+      "balance": 184.75
+    }
+  }
+}
+```
+
+Then the collected metrics will be:
+
+```
+influxdb_fruit,url='http://127.0.0.1:8086/debug/vars',kind='apple' inventory=371.0,sold=112.0
+influxdb_fruit,url='http://127.0.0.1:8086/debug/vars',kind='banana' inventory=1000.0,sold=403.0
+
+influxdb_transactions,url='http://192.168.2.1:8086/debug/vars' total=100.0,balance=184.75
+```
+
+There are two important details to note about the collected metrics:
+
+1. Even though the values in JSON are being displayed as integers, the metrics are reported as floats.
+JSON encoders usually don't print the fractional part for round floats.
+Because you cannot change the type of an existing field in InfluxDB, we assume all numbers are floats.
+
+2. The top-level keys' names (in the example above, `"k1"`, `"k2"`, and `"k3"`) are not considered when recording the metrics.

plugins/influxdb/influxdb.go

@@ -0,0 +1,146 @@
+package influxdb
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type InfluxDB struct {
+	URLs []string `toml:"urls"`
+}
+
+func (*InfluxDB) Description() string {
+	return "Read InfluxDB-formatted JSON metrics from one or more HTTP endpoints"
+}
+
+func (*InfluxDB) SampleConfig() string {
+	return `
+  # Works with InfluxDB debug endpoints out of the box,
+  # but other services can use this format too.
+  # See the influxdb plugin's README for more details.
+
+  # Multiple URLs from which to read InfluxDB-formatted JSON
+  urls = [
+    "http://localhost:8086/debug/vars"
+  ]
+`
+}
+
+func (i *InfluxDB) Gather(acc plugins.Accumulator) error {
+	errorChannel := make(chan error, len(i.URLs))
+
+	var wg sync.WaitGroup
+	for _, u := range i.URLs {
+		wg.Add(1)
+		go func(url string) {
+			defer wg.Done()
+			if err := i.gatherURL(acc, url); err != nil {
+				errorChannel <- fmt.Errorf("[url=%s]: %s", url, err)
+			}
+		}(u)
+	}
+
+	wg.Wait()
+	close(errorChannel)
+
+	// If there weren't any errors, we can return nil now.
+	if len(errorChannel) == 0 {
+		return nil
+	}
+
+	// There were errors, so join them all together as one big error.
+	errorStrings := make([]string, 0, len(errorChannel))
+	for err := range errorChannel {
+		errorStrings = append(errorStrings, err.Error())
+	}
+
+	return errors.New(strings.Join(errorStrings, "\n"))
+}
+
+type point struct {
+	Name   string                 `json:"name"`
+	Tags   map[string]string      `json:"tags"`
+	Values map[string]interface{} `json:"values"`
+}
+
+// Gathers data from a particular URL
+// Parameters:
+//     acc    : The telegraf Accumulator to use
+//     url    : endpoint to send request to
+//
+// Returns:
+//     error: Any error that may have occurred
+func (i *InfluxDB) gatherURL(
+	acc plugins.Accumulator,
+	url string,
+) error {
+	resp, err := http.Get(url)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+
+	// It would be nice to be able to decode into a map[string]point, but
+	// we'll get a decoder error like:
+	// `json: cannot unmarshal array into Go value of type influxdb.point`
+	// if any of the values aren't objects.
+	// To avoid that error, we decode by hand.
+	dec := json.NewDecoder(resp.Body)
+
+	// Parse beginning of object
+	if t, err := dec.Token(); err != nil {
+		return err
+	} else if t != json.Delim('{') {
+		return errors.New("document root must be a JSON object")
+	}
+
+	// Loop through rest of object
+	for {
+		// Nothing left in this object, we're done
+		if !dec.More() {
+			break
+		}
+
+		// Read in a string key. We don't do anything with the top-level keys, so it's discarded.
+		_, err := dec.Token()
+		if err != nil {
+			return err
+		}
+
+		// Attempt to parse a whole object into a point.
+		// It might be a non-object, like a string or array.
+		// If we fail to decode it into a point, ignore it and move on.
+		var p point
+		if err := dec.Decode(&p); err != nil {
+			continue
+		}
+
+		// If the object was a point, but was not fully initialized, ignore it and move on.
+		if p.Name == "" || p.Tags == nil || p.Values == nil || len(p.Values) == 0 {
+			continue
+		}
+
+		// Add a tag to indicate the source of the data.
+		p.Tags["url"] = url
+
+		acc.AddFields(
+			p.Name,
+			p.Values,
+			p.Tags,
+		)
+	}
+
+	return nil
+}
+
+func init() {
+	plugins.Add("influxdb", func() plugins.Plugin {
+		return &InfluxDB{}
+	})
+}

plugins/influxdb/influxdb_test.go

@@ -0,0 +1,100 @@
+package influxdb_test
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/influxdb/telegraf/plugins/influxdb"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestBasic(t *testing.T) {
+	js := `
+{
+  "_1": {
+    "name": "foo",
+    "tags": {
+      "id": "ex1"
+    },
+    "values": {
+      "i": -1,
+      "f": 0.5,
+      "b": true,
+      "s": "string"
+    }
+  },
+  "ignored": {
+    "willBeRecorded": false
+  },
+  "ignoredAndNested": {
+    "hash": {
+      "is": "nested"
+    }
+  },
+  "array": [
+   "makes parsing more difficult than necessary"
+  ],
+  "string": "makes parsing more difficult than necessary",
+  "_2": {
+    "name": "bar",
+    "tags": {
+      "id": "ex2"
+    },
+    "values": {
+      "x": "x"
+    }
+  },
+  "pointWithoutFields_willNotBeIncluded": {
+    "name": "asdf",
+    "tags": {
+      "id": "ex3"
+    },
+    "values": {}
+  }
+}
+`
+	fakeServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/endpoint" {
+			_, _ = w.Write([]byte(js))
+		} else {
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	defer fakeServer.Close()
+
+	plugin := &influxdb.InfluxDB{
+		URLs: []string{fakeServer.URL + "/endpoint"},
+	}
+
+	var acc testutil.Accumulator
+	require.NoError(t, plugin.Gather(&acc))
+
+	require.Len(t, acc.Points, 2)
+	require.NoError(t, acc.ValidateTaggedFieldsValue(
+		"foo",
+		map[string]interface{}{
+			// JSON will truncate floats to integer representations.
+			// Since there's no distinction in JSON, we can't assume it's an int.
+			"i": -1.0,
+			"f": 0.5,
+			"b": true,
+			"s": "string",
+		},
+		map[string]string{
+			"id":  "ex1",
+			"url": fakeServer.URL + "/endpoint",
+		},
+	))
+	require.NoError(t, acc.ValidateTaggedFieldsValue(
+		"bar",
+		map[string]interface{}{
+			"x": "x",
+		},
+		map[string]string{
+			"id":  "ex2",
+			"url": fakeServer.URL + "/endpoint",
+		},
+	))
+}

plugins/jolokia/README.md

@@ -0,0 +1,51 @@
+# Telegraf plugin: Jolokia
+
+#### Plugin arguments:
+- **context** string: Context root used of jolokia url
+- **servers** []Server: List of servers
+  + **name** string: Server's logical name
+  + **host** string: Server's ip address or hostname
+  + **port** string: Server's listening port
+- **metrics** []Metric
+  + **name** string: Name of the measure
+  + **jmx** string: Jmx path that identifies mbeans attributes
+  + **pass** []string: Attributes to retain when collecting values
+  + **drop** []string: Attributes to drop when collecting values
+
+#### Description
+
+The Jolokia plugin collects JVM metrics exposed as MBean's attributes through jolokia REST endpoint. All metrics
+are collected for each server configured.
+
+See: https://jolokia.org/
+
+# Measurements:
+Jolokia plugin produces one measure for each metric configured, adding Server's `name`, `host` and `port` as tags.
+
+Given a configuration like:
+
+```ini
+[jolokia]
+
+[[jolokia.servers]]
+  name = "as-service-1"
+  host = "127.0.0.1"
+  port = "8080"
+
+[[jolokia.servers]]
+  name = "as-service-2"
+  host = "127.0.0.1"
+  port = "8180"
+
+[[jolokia.metrics]]
+  name = "heap_memory_usage"
+  jmx  = "/java.lang:type=Memory/HeapMemoryUsage"
+  pass = ["used", "max"]
+```
+
+The collected metrics will be:
+
+```
+jolokia_heap_memory_usage name=as-service-1,host=127.0.0.1,port=8080 used=xxx,max=yyy
+jolokia_heap_memory_usage name=as-service-2,host=127.0.0.1,port=8180 used=vvv,max=zzz
+```

plugins/jolokia/jolokia.go

@@ -0,0 +1,164 @@
+package jolokia
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/url"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Server struct {
+	Name     string
+	Host     string
+	Username string
+	Password string
+	Port     string
+}
+
+type Metric struct {
+	Name string
+	Jmx  string
+}
+
+type JolokiaClient interface {
+	MakeRequest(req *http.Request) (*http.Response, error)
+}
+
+type JolokiaClientImpl struct {
+	client *http.Client
+}
+
+func (c JolokiaClientImpl) MakeRequest(req *http.Request) (*http.Response, error) {
+	return c.client.Do(req)
+}
+
+type Jolokia struct {
+	jClient JolokiaClient
+	Context string
+	Servers []Server
+	Metrics []Metric
+}
+
+func (j *Jolokia) SampleConfig() string {
+	return `
+  # This is the context root used to compose the jolokia url
+  context = "/jolokia/read"
+
+  # List of servers exposing jolokia read service
+  [[plugins.jolokia.servers]]
+    name = "stable"
+    host = "192.168.103.2"
+    port = "8180"
+    # username = "myuser"
+    # password = "mypassword"
+
+  # List of metrics collected on above servers
+  # Each metric consists in a name, a jmx path and either a pass or drop slice attributes
+  # This collect all heap memory usage metrics
+  [[plugins.jolokia.metrics]]
+    name = "heap_memory_usage"
+    jmx  = "/java.lang:type=Memory/HeapMemoryUsage"
+`
+}
+
+func (j *Jolokia) Description() string {
+	return "Read JMX metrics through Jolokia"
+}
+
+func (j *Jolokia) getAttr(requestUrl *url.URL) (map[string]interface{}, error) {
+	// Create + send request
+	req, err := http.NewRequest("GET", requestUrl.String(), nil)
+	if err != nil {
+		return nil, err
+	}
+	defer req.Body.Close()
+
+	resp, err := j.jClient.MakeRequest(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	// Process response
+	if resp.StatusCode != http.StatusOK {
+		err = fmt.Errorf("Response from url \"%s\" has status code %d (%s), expected %d (%s)",
+			requestUrl,
+			resp.StatusCode,
+			http.StatusText(resp.StatusCode),
+			http.StatusOK,
+			http.StatusText(http.StatusOK))
+		return nil, err
+	}
+
+	// read body
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	// Unmarshal json
+	var jsonOut map[string]interface{}
+	if err = json.Unmarshal([]byte(body), &jsonOut); err != nil {
+		return nil, errors.New("Error decoding JSON response")
+	}
+
+	return jsonOut, nil
+}
+
+func (j *Jolokia) Gather(acc plugins.Accumulator) error {
+	context := j.Context //"/jolokia/read"
+	servers := j.Servers
+	metrics := j.Metrics
+	tags := make(map[string]string)
+
+	for _, server := range servers {
+		tags["server"] = server.Name
+		tags["port"] = server.Port
+		tags["host"] = server.Host
+		fields := make(map[string]interface{})
+		for _, metric := range metrics {
+
+			measurement := metric.Name
+			jmxPath := metric.Jmx
+
+			// Prepare URL
+			requestUrl, err := url.Parse("http://" + server.Host + ":" +
+				server.Port + context + jmxPath)
+			if err != nil {
+				return err
+			}
+			if server.Username != "" || server.Password != "" {
+				requestUrl.User = url.UserPassword(server.Username, server.Password)
+			}
+
+			out, _ := j.getAttr(requestUrl)
+
+			if values, ok := out["value"]; ok {
+				switch t := values.(type) {
+				case map[string]interface{}:
+					for k, v := range t {
+						fields[measurement+"_"+k] = v
+					}
+				case interface{}:
+					fields[measurement] = t
+				}
+			} else {
+				fmt.Printf("Missing key 'value' in '%s' output response\n",
+					requestUrl.String())
+			}
+		}
+		acc.AddFields("jolokia", fields, tags)
+	}
+
+	return nil
+}
+
+func init() {
+	plugins.Add("jolokia", func() plugins.Plugin {
+		return &Jolokia{jClient: &JolokiaClientImpl{client: &http.Client{}}}
+	})
+}

plugins/jolokia/jolokia_test.go

@@ -0,0 +1,147 @@
+package jolokia
+
+import (
+	_ "fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	_ "github.com/stretchr/testify/require"
+)
+
+const validMultiValueJSON = `
+{
+  "request":{
+    "mbean":"java.lang:type=Memory",
+    "attribute":"HeapMemoryUsage",
+    "type":"read"
+  },
+  "value":{
+    "init":67108864,
+    "committed":456130560,
+    "max":477626368,
+    "used":203288528
+  },
+  "timestamp":1446129191,
+  "status":200
+}`
+
+const validSingleValueJSON = `
+{
+  "request":{
+    "path":"used",
+    "mbean":"java.lang:type=Memory",
+    "attribute":"HeapMemoryUsage",
+    "type":"read"
+  },
+  "value":209274376,
+  "timestamp":1446129256,
+  "status":200
+}`
+
+const invalidJSON = "I don't think this is JSON"
+
+const empty = ""
+
+var Servers = []Server{Server{Name: "as1", Host: "127.0.0.1", Port: "8080"}}
+var HeapMetric = Metric{Name: "heap_memory_usage", Jmx: "/java.lang:type=Memory/HeapMemoryUsage"}
+var UsedHeapMetric = Metric{Name: "heap_memory_usage", Jmx: "/java.lang:type=Memory/HeapMemoryUsage", Pass: []string{"used"}}
+
+type jolokiaClientStub struct {
+	responseBody string
+	statusCode   int
+}
+
+func (c jolokiaClientStub) MakeRequest(req *http.Request) (*http.Response, error) {
+	resp := http.Response{}
+	resp.StatusCode = c.statusCode
+	resp.Body = ioutil.NopCloser(strings.NewReader(c.responseBody))
+	return &resp, nil
+}
+
+// Generates a pointer to an HttpJson object that uses a mock HTTP client.
+// Parameters:
+//     response  : Body of the response that the mock HTTP client should return
+//     statusCode: HTTP status code the mock HTTP client should return
+//
+// Returns:
+//     *HttpJson: Pointer to an HttpJson object that uses the generated mock HTTP client
+func genJolokiaClientStub(response string, statusCode int, servers []Server, metrics []Metric) *Jolokia {
+	return &Jolokia{
+		jClient: jolokiaClientStub{responseBody: response, statusCode: statusCode},
+		Servers: servers,
+		Metrics: metrics,
+	}
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonMultiValue(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 200, Servers, []Metric{HeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+
+	assert.True(t, acc.CheckFieldsValue("heap_memory_usage", map[string]interface{}{"init": 67108864.0,
+		"committed": 456130560.0,
+		"max":       477626368.0,
+		"used":      203288528.0}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonMultiValueWithPass(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 200, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+
+	assert.True(t, acc.CheckFieldsValue("heap_memory_usage", map[string]interface{}{"used": 203288528.0}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonMultiValueTags(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 200, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+	assert.NoError(t, acc.ValidateTaggedFieldsValue("heap_memory_usage", map[string]interface{}{"used": 203288528.0}, map[string]string{"host": "127.0.0.1", "port": "8080", "server": "as1"}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonSingleValueTags(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validSingleValueJSON, 200, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+	assert.NoError(t, acc.ValidateTaggedFieldsValue("heap_memory_usage", map[string]interface{}{"value": 209274376.0}, map[string]string{"host": "127.0.0.1", "port": "8080", "server": "as1"}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonOn404(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 404, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 0, len(acc.Points))
+}

plugins/kafka_consumer/README.md

@@ -0,0 +1,24 @@
+# Kafka Consumer
+
+The [Kafka](http://kafka.apache.org/) consumer plugin polls a specified Kafka
+topic and adds messages to InfluxDB. The plugin assumes messages follow the
+line protocol. [Consumer Group](http://godoc.org/github.com/wvanbergen/kafka/consumergroup)
+is used to talk to the Kafka cluster so multiple instances of telegraf can read
+from the same topic in parallel.
+
+## Testing
+
+Running integration tests requires running Zookeeper & Kafka. The following
+commands assume you're on OS X & using [boot2docker](http://boot2docker.io/) or docker-machine through [Docker Toolbox](https://www.docker.com/docker-toolbox).
+
+To start Kafka & Zookeeper:
+
+```
+docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`boot2docker ip || docker-machine ip <your_machine_name>` --env ADVERTISED_PORT=9092 spotify/kafka
+```
+
+To run tests:
+
+```
+go test
+```

plugins/kafka_consumer/kafka_consumer.go

@@ -1,153 +1,166 @@
 package kafka_consumer
 
 import (
-	"os"
-	"os/signal"
-	"time"
+	"log"
+	"strings"
+	"sync"
 
-	"github.com/influxdb/influxdb/tsdb"
+	"github.com/influxdb/influxdb/models"
 	"github.com/influxdb/telegraf/plugins"
+
+	"github.com/Shopify/sarama"
 	"github.com/wvanbergen/kafka/consumergroup"
-	"gopkg.in/Shopify/sarama.v1"
 )
 
 type Kafka struct {
-	ConsumerGroupName string
-	Topic             string
-	ZookeeperPeers    []string
-	Consumer          *consumergroup.ConsumerGroup
-	BatchSize         int
+	ConsumerGroup  string
+	Topics         []string
+	ZookeeperPeers []string
+	Consumer       *consumergroup.ConsumerGroup
+	PointBuffer    int
+	Offset         string
+
+	sync.Mutex
+
+	// channel for all incoming kafka messages
+	in <-chan *sarama.ConsumerMessage
+	// channel for all kafka consumer errors
+	errs <-chan *sarama.ConsumerError
+	// channel for all incoming parsed kafka points
+	pointChan chan models.Point
+	done      chan struct{}
+
+	// doNotCommitMsgs tells the parser not to call CommitUpTo on the consumer
+	// this is mostly for test purposes, but there may be a use-case for it later.
+	doNotCommitMsgs bool
 }
 
 var sampleConfig = `
-# topic to consume
-topic = "topic_with_metrics"
-
-# the name of the consumer group
-consumerGroupName = "telegraf_metrics_consumers"
-
-# an array of Zookeeper connection strings
-zookeeperPeers = ["localhost:2181"]
-
-# Batch size of points sent to InfluxDB
-batchSize = 1000`
+  # topic(s) to consume
+  topics = ["telegraf"]
+  # an array of Zookeeper connection strings
+  zookeeper_peers = ["localhost:2181"]
+  # the name of the consumer group
+  consumer_group = "telegraf_metrics_consumers"
+  # Maximum number of points to buffer between collection intervals
+  point_buffer = 100000
+  # Offset (must be either "oldest" or "newest")
+  offset = "oldest"
+`
 
 func (k *Kafka) SampleConfig() string {
 	return sampleConfig
 }
 
 func (k *Kafka) Description() string {
-	return "read metrics from a Kafka topic"
+	return "Read line-protocol metrics from Kafka topic(s)"
 }
 
-type Metric struct {
-	Measurement string                 `json:"measurement"`
-	Values      map[string]interface{} `json:"values"`
-	Tags        map[string]string      `json:"tags"`
-	Time        time.Time              `json:"time"`
-}
-
-func (k *Kafka) Gather(acc plugins.Accumulator) error {
+func (k *Kafka) Start() error {
+	k.Lock()
+	defer k.Unlock()
 	var consumerErr error
-	metricQueue := make(chan []byte, 200)
 
-	if k.Consumer == nil {
+	config := consumergroup.NewConfig()
+	switch strings.ToLower(k.Offset) {
+	case "oldest", "":
+		config.Offsets.Initial = sarama.OffsetOldest
+	case "newest":
+		config.Offsets.Initial = sarama.OffsetNewest
+	default:
+		log.Printf("WARNING: Kafka consumer invalid offset '%s', using 'oldest'\n",
+			k.Offset)
+		config.Offsets.Initial = sarama.OffsetOldest
+	}
+
+	if k.Consumer == nil || k.Consumer.Closed() {
 		k.Consumer, consumerErr = consumergroup.JoinConsumerGroup(
-			k.ConsumerGroupName,
-			[]string{k.Topic},
+			k.ConsumerGroup,
+			k.Topics,
 			k.ZookeeperPeers,
-			nil,
+			config,
 		)
-
 		if consumerErr != nil {
 			return consumerErr
 		}
 
-		c := make(chan os.Signal, 1)
-		halt := make(chan bool, 1)
-		signal.Notify(c, os.Interrupt)
-		go func() {
-			<-c
-			halt <- true
-			emitMetrics(k, acc, metricQueue)
-			k.Consumer.Close()
-		}()
-
-		go readFromKafka(k.Consumer.Messages(), metricQueue, k.BatchSize, k.Consumer.CommitUpto, halt)
+		// Setup message and error channels
+		k.in = k.Consumer.Messages()
+		k.errs = k.Consumer.Errors()
 	}
 
-	return emitMetrics(k, acc, metricQueue)
+	k.done = make(chan struct{})
+	if k.PointBuffer == 0 {
+		k.PointBuffer = 100000
+	}
+	k.pointChan = make(chan models.Point, k.PointBuffer)
+
+	// Start the kafka message reader
+	go k.parser()
+	log.Printf("Started the kafka consumer service, peers: %v, topics: %v\n",
+		k.ZookeeperPeers, k.Topics)
+	return nil
 }
 
-func emitMetrics(k *Kafka, acc plugins.Accumulator, metricConsumer <-chan []byte) error {
-	timeout := time.After(1 * time.Second)
-
+// parser() reads all incoming messages from the consumer, and parses them into
+// influxdb metric points.
+func (k *Kafka) parser() {
 	for {
 		select {
-		case batch := <-metricConsumer:
-			var points []tsdb.Point
-			var err error
-			if points, err = tsdb.ParsePoints(batch); err != nil {
-				return err
+		case <-k.done:
+			return
+		case err := <-k.errs:
+			log.Printf("Kafka Consumer Error: %s\n", err.Error())
+		case msg := <-k.in:
+			points, err := models.ParsePoints(msg.Value)
+			if err != nil {
+				log.Printf("Could not parse kafka message: %s, error: %s",
+					string(msg.Value), err.Error())
 			}
 
 			for _, point := range points {
-				acc.AddValuesWithTime(point.Name(), point.Fields(), point.Tags(), point.Time())
+				select {
+				case k.pointChan <- point:
+					continue
+				default:
+					log.Printf("Kafka Consumer buffer is full, dropping a point." +
+						" You may want to increase the point_buffer setting")
+				}
+			}
+
+			if !k.doNotCommitMsgs {
+				// TODO(cam) this locking can be removed if this PR gets merged:
+				// https://github.com/wvanbergen/kafka/pull/84
+				k.Lock()
+				k.Consumer.CommitUpto(msg)
+				k.Unlock()
 			}
-		case <-timeout:
-			return nil
 		}
 	}
 }
 
-const millisecond = 1000000 * time.Nanosecond
-
-type ack func(*sarama.ConsumerMessage) error
-
-func readFromKafka(kafkaMsgs <-chan *sarama.ConsumerMessage, metricProducer chan<- []byte, maxBatchSize int, ackMsg ack, halt <-chan bool) {
-	batch := make([]byte, 0)
-	currentBatchSize := 0
-	timeout := time.After(500 * millisecond)
-	var msg *sarama.ConsumerMessage
-
-	for {
-		select {
-		case msg = <-kafkaMsgs:
-			if currentBatchSize != 0 {
-				batch = append(batch, '\n')
-			}
-
-			batch = append(batch, msg.Value...)
-			currentBatchSize++
-
-			if currentBatchSize == maxBatchSize {
-				metricProducer <- batch
-				currentBatchSize = 0
-				batch = make([]byte, 0)
-				ackMsg(msg)
-			}
-		case <-timeout:
-			if currentBatchSize != 0 {
-				metricProducer <- batch
-				currentBatchSize = 0
-				batch = make([]byte, 0)
-				ackMsg(msg)
-			}
-
-			timeout = time.After(500 * millisecond)
-		case <-halt:
-			if currentBatchSize != 0 {
-				metricProducer <- batch
-				ackMsg(msg)
-			}
-
-			return
-		}
+func (k *Kafka) Stop() {
+	k.Lock()
+	defer k.Unlock()
+	close(k.done)
+	if err := k.Consumer.Close(); err != nil {
+		log.Printf("Error closing kafka consumer: %s\n", err.Error())
 	}
 }
 
+func (k *Kafka) Gather(acc plugins.Accumulator) error {
+	k.Lock()
+	defer k.Unlock()
+	npoints := len(k.pointChan)
+	for i := 0; i < npoints; i++ {
+		point := <-k.pointChan
+		acc.AddFields(point.Name(), point.Fields(), point.Tags(), point.Time())
+	}
+	return nil
+}
+
 func init() {
-	plugins.Add("kafka", func() plugins.Plugin {
+	plugins.Add("kafka_consumer", func() plugins.Plugin {
 		return &Kafka{}
 	})
 }

plugins/kafka_consumer/kafka_consumer_integration_test.go

@@ -2,8 +2,6 @@ package kafka_consumer
 
 import (
 	"fmt"
-	"os"
-	"strings"
 	"testing"
 	"time"
 
@@ -14,49 +12,80 @@ import (
 )
 
 func TestReadsMetricsFromKafka(t *testing.T) {
-	var zkPeers, brokerPeers []string
-
-	if len(os.Getenv("ZOOKEEPER_PEERS")) == 0 {
-		zkPeers = []string{"localhost:2181"}
-	} else {
-		zkPeers = strings.Split(os.Getenv("ZOOKEEPER_PEERS"), ",")
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
 	}
 
-	if len(os.Getenv("KAFKA_PEERS")) == 0 {
-		brokerPeers = []string{"localhost:9092"}
-	} else {
-		brokerPeers = strings.Split(os.Getenv("KAFKA_PEERS"), ",")
-	}
-
-	k := &Kafka{
-		ConsumerGroupName: "telegraf_test_consumers",
-		Topic:             fmt.Sprintf("telegraf_test_topic_%d", time.Now().Unix()),
-		ZookeeperPeers:    zkPeers,
-	}
+	brokerPeers := []string{testutil.GetLocalHost() + ":9092"}
+	zkPeers := []string{testutil.GetLocalHost() + ":2181"}
+	testTopic := fmt.Sprintf("telegraf_test_topic_%d", time.Now().Unix())
 
+	// Send a Kafka message to the kafka host
 	msg := "cpu_load_short,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257"
 	producer, err := sarama.NewSyncProducer(brokerPeers, nil)
 	require.NoError(t, err)
-	_, _, err = producer.SendMessage(&sarama.ProducerMessage{Topic: k.Topic, Value: sarama.StringEncoder(msg)})
-	producer.Close()
+	_, _, err = producer.SendMessage(
+		&sarama.ProducerMessage{
+			Topic: testTopic,
+			Value: sarama.StringEncoder(msg),
+		})
+	require.NoError(t, err)
+	defer producer.Close()
 
+	// Start the Kafka Consumer
+	k := &Kafka{
+		ConsumerGroup:  "telegraf_test_consumers",
+		Topics:         []string{testTopic},
+		ZookeeperPeers: zkPeers,
+		PointBuffer:    100000,
+		Offset:         "oldest",
+	}
+	if err := k.Start(); err != nil {
+		t.Fatal(err.Error())
+	} else {
+		defer k.Stop()
+	}
+
+	waitForPoint(k, t)
+
+	// Verify that we can now gather the sent message
 	var acc testutil.Accumulator
-
 	// Sanity check
-	assert.Equal(t, 0, len(acc.Points), "there should not be any points")
+	assert.Equal(t, 0, len(acc.Points), "There should not be any points")
 
+	// Gather points
 	err = k.Gather(&acc)
 	require.NoError(t, err)
-
-	assert.Equal(t, 1, len(acc.Points), "there should be a single point")
-
-	point := acc.Points[0]
-	assert.Equal(t, "cpu_load_short", point.Measurement)
-	assert.Equal(t, map[string]interface{}{"value": 23422.0}, point.Values)
-	assert.Equal(t, map[string]string{
-		"host":      "server01",
-		"direction": "in",
-		"region":    "us-west",
-	}, point.Tags)
-	assert.Equal(t, time.Unix(0, 1422568543702900257), point.Time)
+	if len(acc.Points) == 1 {
+		point := acc.Points[0]
+		assert.Equal(t, "cpu_load_short", point.Measurement)
+		assert.Equal(t, map[string]interface{}{"value": 23422.0}, point.Fields)
+		assert.Equal(t, map[string]string{
+			"host":      "server01",
+			"direction": "in",
+			"region":    "us-west",
+		}, point.Tags)
+		assert.Equal(t, time.Unix(0, 1422568543702900257).Unix(), point.Time.Unix())
+	} else {
+		t.Errorf("No points found in accumulator, expected 1")
+	}
+}
+
+// Waits for the metric that was sent to the kafka broker to arrive at the kafka
+// consumer
+func waitForPoint(k *Kafka, t *testing.T) {
+	// Give the kafka container up to 2 seconds to get the point to the consumer
+	ticker := time.NewTicker(5 * time.Millisecond)
+	counter := 0
+	for {
+		select {
+		case <-ticker.C:
+			counter++
+			if counter > 1000 {
+				t.Fatal("Waited for 5s, point never arrived to consumer")
+			} else if len(k.pointChan) == 1 {
+				return
+			}
+		}
+	}
 }

plugins/kafka_consumer/kafka_consumer_test.go

@@ -1,88 +1,91 @@
 package kafka_consumer
 
 import (
-	"strings"
 	"testing"
 	"time"
 
+	"github.com/influxdb/influxdb/models"
 	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/Shopify/sarama"
 	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"gopkg.in/Shopify/sarama.v1"
 )
 
-const testMsg = "cpu_load_short,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257"
+const (
+	testMsg     = "cpu_load_short,host=server01 value=23422.0 1422568543702900257"
+	invalidMsg  = "cpu_load_short,host=server01 1422568543702900257"
+	pointBuffer = 5
+)
 
-func TestReadFromKafkaBatchesMsgsOnBatchSize(t *testing.T) {
-	halt := make(chan bool, 1)
-	metricChan := make(chan []byte, 1)
-	kafkaChan := make(chan *sarama.ConsumerMessage, 10)
-	for i := 0; i < 10; i++ {
-		kafkaChan <- saramaMsg(testMsg)
+func NewTestKafka() (*Kafka, chan *sarama.ConsumerMessage) {
+	in := make(chan *sarama.ConsumerMessage, pointBuffer)
+	k := Kafka{
+		ConsumerGroup:   "test",
+		Topics:          []string{"telegraf"},
+		ZookeeperPeers:  []string{"localhost:2181"},
+		PointBuffer:     pointBuffer,
+		Offset:          "oldest",
+		in:              in,
+		doNotCommitMsgs: true,
+		errs:            make(chan *sarama.ConsumerError, pointBuffer),
+		done:            make(chan struct{}),
+		pointChan:       make(chan models.Point, pointBuffer),
 	}
-
-	expectedBatch := strings.Repeat(testMsg+"\n", 9) + testMsg
-	readFromKafka(kafkaChan, metricChan, 10, func(msg *sarama.ConsumerMessage) error {
-		batch := <-metricChan
-		assert.Equal(t, expectedBatch, string(batch))
-
-		halt <- true
-
-		return nil
-	}, halt)
+	return &k, in
 }
 
-func TestReadFromKafkaBatchesMsgsOnTimeout(t *testing.T) {
-	halt := make(chan bool, 1)
-	metricChan := make(chan []byte, 1)
-	kafkaChan := make(chan *sarama.ConsumerMessage, 10)
-	for i := 0; i < 3; i++ {
-		kafkaChan <- saramaMsg(testMsg)
+// Test that the parser parses kafka messages into points
+func TestRunParser(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	in <- saramaMsg(testMsg)
+	time.Sleep(time.Millisecond)
+
+	assert.Equal(t, len(k.pointChan), 1)
+}
+
+// Test that the parser ignores invalid messages
+func TestRunParserInvalidMsg(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	in <- saramaMsg(invalidMsg)
+	time.Sleep(time.Millisecond)
+
+	assert.Equal(t, len(k.pointChan), 0)
+}
+
+// Test that points are dropped when we hit the buffer limit
+func TestRunParserRespectsBuffer(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	for i := 0; i < pointBuffer+1; i++ {
+		in <- saramaMsg(testMsg)
 	}
+	time.Sleep(time.Millisecond)
 
-	expectedBatch := strings.Repeat(testMsg+"\n", 2) + testMsg
-	readFromKafka(kafkaChan, metricChan, 10, func(msg *sarama.ConsumerMessage) error {
-		batch := <-metricChan
-		assert.Equal(t, expectedBatch, string(batch))
-
-		halt <- true
-
-		return nil
-	}, halt)
+	assert.Equal(t, len(k.pointChan), 5)
 }
 
-func TestEmitMetricsSendMetricsToAcc(t *testing.T) {
-	k := &Kafka{}
-	var acc testutil.Accumulator
-	testChan := make(chan []byte, 1)
-	testChan <- []byte(testMsg)
+// Test that the parser parses kafka messages into points
+func TestRunParserAndGather(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
 
-	err := emitMetrics(k, &acc, testChan)
-	require.NoError(t, err)
+	go k.parser()
+	in <- saramaMsg(testMsg)
+	time.Sleep(time.Millisecond)
 
-	assert.Equal(t, 1, len(acc.Points), "there should be a single point")
+	acc := testutil.Accumulator{}
+	k.Gather(&acc)
 
-	point := acc.Points[0]
-	assert.Equal(t, "cpu_load_short", point.Measurement)
-	assert.Equal(t, map[string]interface{}{"value": 23422.0}, point.Values)
-	assert.Equal(t, map[string]string{
-		"host":      "server01",
-		"direction": "in",
-		"region":    "us-west",
-	}, point.Tags)
-
-	assert.Equal(t, time.Unix(0, 1422568543702900257), point.Time)
-}
-
-func TestEmitMetricsTimesOut(t *testing.T) {
-	k := &Kafka{}
-	var acc testutil.Accumulator
-	testChan := make(chan []byte)
-
-	err := emitMetrics(k, &acc, testChan)
-	require.NoError(t, err)
-
-	assert.Equal(t, 0, len(acc.Points), "there should not be a any points")
+	assert.Equal(t, len(acc.Points), 1)
+	assert.True(t, acc.CheckValue("cpu_load_short", 23422.0))
 }
 
 func saramaMsg(val string) *sarama.ConsumerMessage {

plugins/leofs/leofs.go

@@ -0,0 +1,231 @@
+package leofs
+
+import (
+	"bufio"
+	"fmt"
+	"github.com/influxdb/telegraf/plugins"
+	"net/url"
+	"os/exec"
+	"strconv"
+	"strings"
+	"sync"
+)
+
+const oid = ".1.3.6.1.4.1.35450"
+
+// For Manager Master
+const defaultEndpoint = "127.0.0.1:4020"
+
+type ServerType int
+
+const (
+	ServerTypeManagerMaster ServerType = iota
+	ServerTypeManagerSlave
+	ServerTypeStorage
+	ServerTypeGateway
+)
+
+type LeoFS struct {
+	Servers []string
+}
+
+var KeyMapping = map[ServerType][]string{
+	ServerTypeManagerMaster: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+	ServerTypeManagerSlave: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+	ServerTypeStorage: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"num_of_writes",
+		"num_of_reads",
+		"num_of_deletes",
+		"num_of_writes_5min",
+		"num_of_reads_5min",
+		"num_of_deletes_5min",
+		"num_of_active_objects",
+		"total_objects",
+		"total_size_of_active_objects",
+		"total_size",
+		"num_of_replication_messages",
+		"num_of_sync-vnode_messages",
+		"num_of_rebalance_messages",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+	ServerTypeGateway: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"num_of_writes",
+		"num_of_reads",
+		"num_of_deletes",
+		"num_of_writes_5min",
+		"num_of_reads_5min",
+		"num_of_deletes_5min",
+		"count_of_cache-hit",
+		"count_of_cache-miss",
+		"total_of_files",
+		"total_cached_size",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+}
+
+var serverTypeMapping = map[string]ServerType{
+	"4020": ServerTypeManagerMaster,
+	"4021": ServerTypeManagerSlave,
+	"4010": ServerTypeStorage,
+	"4011": ServerTypeStorage,
+	"4012": ServerTypeStorage,
+	"4013": ServerTypeStorage,
+	"4000": ServerTypeGateway,
+	"4001": ServerTypeGateway,
+}
+
+var sampleConfig = `
+  # An array of URI to gather stats about LeoFS.
+  # Specify an ip or hostname with port. ie 127.0.0.1:4020
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 4020 as the port.
+  servers = ["127.0.0.1:4021"]
+`
+
+func (l *LeoFS) SampleConfig() string {
+	return sampleConfig
+}
+
+func (l *LeoFS) Description() string {
+	return "Read metrics from a LeoFS Server via SNMP"
+}
+
+func (l *LeoFS) Gather(acc plugins.Accumulator) error {
+	if len(l.Servers) == 0 {
+		l.gatherServer(defaultEndpoint, ServerTypeManagerMaster, acc)
+		return nil
+	}
+	var wg sync.WaitGroup
+	var outerr error
+	for _, endpoint := range l.Servers {
+		_, err := url.Parse(endpoint)
+		if err != nil {
+			return fmt.Errorf("Unable to parse the address:%s, err:%s", endpoint, err)
+		}
+		port, err := retrieveTokenAfterColon(endpoint)
+		if err != nil {
+			return err
+		}
+		st, ok := serverTypeMapping[port]
+		if !ok {
+			st = ServerTypeStorage
+		}
+		wg.Add(1)
+		go func(endpoint string, st ServerType) {
+			defer wg.Done()
+			outerr = l.gatherServer(endpoint, st, acc)
+		}(endpoint, st)
+	}
+	wg.Wait()
+	return outerr
+}
+
+func (l *LeoFS) gatherServer(endpoint string, serverType ServerType, acc plugins.Accumulator) error {
+	cmd := exec.Command("snmpwalk", "-v2c", "-cpublic", endpoint, oid)
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return err
+	}
+	cmd.Start()
+	defer cmd.Wait()
+	scanner := bufio.NewScanner(stdout)
+	if !scanner.Scan() {
+		return fmt.Errorf("Unable to retrieve the node name")
+	}
+	nodeName, err := retrieveTokenAfterColon(scanner.Text())
+	if err != nil {
+		return err
+	}
+	nodeNameTrimmed := strings.Trim(nodeName, "\"")
+	tags := map[string]string{
+		"node": nodeNameTrimmed,
+	}
+	i := 0
+
+	fields := make(map[string]interface{})
+	for scanner.Scan() {
+		key := KeyMapping[serverType][i]
+		val, err := retrieveTokenAfterColon(scanner.Text())
+		if err != nil {
+			return err
+		}
+		fVal, err := strconv.ParseFloat(val, 64)
+		if err != nil {
+			return fmt.Errorf("Unable to parse the value:%s, err:%s", val, err)
+		}
+		fields[key] = fVal
+		i++
+	}
+	acc.AddFields("leofs", fields, tags)
+	return nil
+}
+
+func retrieveTokenAfterColon(line string) (string, error) {
+	tokens := strings.Split(line, ":")
+	if len(tokens) != 2 {
+		return "", fmt.Errorf("':' not found in the line:%s", line)
+	}
+	return strings.TrimSpace(tokens[1]), nil
+}
+
+func init() {
+	plugins.Add("leofs", func() plugins.Plugin {
+		return &LeoFS{}
+	})
+}

plugins/leofs/leofs_test.go

@@ -0,0 +1,173 @@
+package leofs
+
+import (
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"io/ioutil"
+	"log"
+	"os"
+	"os/exec"
+	"testing"
+)
+
+var fakeSNMP4Manager = `
+package main
+
+import "fmt"
+
+const output = ` + "`" + `iso.3.6.1.4.1.35450.15.1.0 = STRING: "manager_888@127.0.0.1"
+iso.3.6.1.4.1.35450.15.2.0 = Gauge32: 186
+iso.3.6.1.4.1.35450.15.3.0 = Gauge32: 46235519
+iso.3.6.1.4.1.35450.15.4.0 = Gauge32: 32168525
+iso.3.6.1.4.1.35450.15.5.0 = Gauge32: 14066068
+iso.3.6.1.4.1.35450.15.6.0 = Gauge32: 5512968
+iso.3.6.1.4.1.35450.15.7.0 = Gauge32: 186
+iso.3.6.1.4.1.35450.15.8.0 = Gauge32: 46269006
+iso.3.6.1.4.1.35450.15.9.0 = Gauge32: 32202867
+iso.3.6.1.4.1.35450.15.10.0 = Gauge32: 14064995
+iso.3.6.1.4.1.35450.15.11.0 = Gauge32: 5492634
+iso.3.6.1.4.1.35450.15.12.0 = Gauge32: 60
+iso.3.6.1.4.1.35450.15.13.0 = Gauge32: 43515904
+iso.3.6.1.4.1.35450.15.14.0 = Gauge32: 60
+iso.3.6.1.4.1.35450.15.15.0 = Gauge32: 43533983` + "`" +
+	`
+func main() {
+	fmt.Println(output)
+}
+`
+
+var fakeSNMP4Storage = `
+package main
+
+import "fmt"
+
+const output = ` + "`" + `iso.3.6.1.4.1.35450.34.1.0 = STRING: "storage_0@127.0.0.1"
+iso.3.6.1.4.1.35450.34.2.0 = Gauge32: 512
+iso.3.6.1.4.1.35450.34.3.0 = Gauge32: 38126307
+iso.3.6.1.4.1.35450.34.4.0 = Gauge32: 22308716
+iso.3.6.1.4.1.35450.34.5.0 = Gauge32: 15816448
+iso.3.6.1.4.1.35450.34.6.0 = Gauge32: 5232008
+iso.3.6.1.4.1.35450.34.7.0 = Gauge32: 512
+iso.3.6.1.4.1.35450.34.8.0 = Gauge32: 38113176
+iso.3.6.1.4.1.35450.34.9.0 = Gauge32: 22313398
+iso.3.6.1.4.1.35450.34.10.0 = Gauge32: 15798779
+iso.3.6.1.4.1.35450.34.11.0 = Gauge32: 5237315
+iso.3.6.1.4.1.35450.34.12.0 = Gauge32: 191
+iso.3.6.1.4.1.35450.34.13.0 = Gauge32: 824
+iso.3.6.1.4.1.35450.34.14.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.15.0 = Gauge32: 50105
+iso.3.6.1.4.1.35450.34.16.0 = Gauge32: 196654
+iso.3.6.1.4.1.35450.34.17.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.18.0 = Gauge32: 2052
+iso.3.6.1.4.1.35450.34.19.0 = Gauge32: 50296
+iso.3.6.1.4.1.35450.34.20.0 = Gauge32: 35
+iso.3.6.1.4.1.35450.34.21.0 = Gauge32: 898
+iso.3.6.1.4.1.35450.34.22.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.23.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.24.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.31.0 = Gauge32: 51
+iso.3.6.1.4.1.35450.34.32.0 = Gauge32: 53219328
+iso.3.6.1.4.1.35450.34.33.0 = Gauge32: 51
+iso.3.6.1.4.1.35450.34.34.0 = Gauge32: 53351083` + "`" +
+	`
+func main() {
+	fmt.Println(output)
+}
+`
+
+var fakeSNMP4Gateway = `
+package main
+
+import "fmt"
+
+const output = ` + "`" + `iso.3.6.1.4.1.35450.34.1.0 = STRING: "gateway_0@127.0.0.1"
+iso.3.6.1.4.1.35450.34.2.0 = Gauge32: 465
+iso.3.6.1.4.1.35450.34.3.0 = Gauge32: 61676335
+iso.3.6.1.4.1.35450.34.4.0 = Gauge32: 46890415
+iso.3.6.1.4.1.35450.34.5.0 = Gauge32: 14785011
+iso.3.6.1.4.1.35450.34.6.0 = Gauge32: 5578855
+iso.3.6.1.4.1.35450.34.7.0 = Gauge32: 465
+iso.3.6.1.4.1.35450.34.8.0 = Gauge32: 61644426
+iso.3.6.1.4.1.35450.34.9.0 = Gauge32: 46880358
+iso.3.6.1.4.1.35450.34.10.0 = Gauge32: 14763002
+iso.3.6.1.4.1.35450.34.11.0 = Gauge32: 5582125
+iso.3.6.1.4.1.35450.34.12.0 = Gauge32: 191
+iso.3.6.1.4.1.35450.34.13.0 = Gauge32: 827
+iso.3.6.1.4.1.35450.34.14.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.15.0 = Gauge32: 50105
+iso.3.6.1.4.1.35450.34.16.0 = Gauge32: 196650
+iso.3.6.1.4.1.35450.34.17.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.18.0 = Gauge32: 30256
+iso.3.6.1.4.1.35450.34.19.0 = Gauge32: 532158
+iso.3.6.1.4.1.35450.34.20.0 = Gauge32: 34
+iso.3.6.1.4.1.35450.34.21.0 = Gauge32: 1
+iso.3.6.1.4.1.35450.34.31.0 = Gauge32: 53
+iso.3.6.1.4.1.35450.34.32.0 = Gauge32: 55050240
+iso.3.6.1.4.1.35450.34.33.0 = Gauge32: 53
+iso.3.6.1.4.1.35450.34.34.0 = Gauge32: 55186538` + "`" +
+	`
+func main() {
+	fmt.Println(output)
+}
+`
+
+func makeFakeSNMPSrc(code string) string {
+	path := os.TempDir() + "/test.go"
+	err := ioutil.WriteFile(path, []byte(code), 0600)
+	if err != nil {
+		log.Fatalln(err)
+	}
+	return path
+}
+
+func buildFakeSNMPCmd(src string) {
+	err := exec.Command("go", "build", "-o", "snmpwalk", src).Run()
+	if err != nil {
+		log.Fatalln(err)
+	}
+}
+
+func testMain(t *testing.T, code string, endpoint string, serverType ServerType) {
+
+	// Build the fake snmpwalk for test
+	src := makeFakeSNMPSrc(code)
+	defer os.Remove(src)
+	buildFakeSNMPCmd(src)
+	defer os.Remove("./snmpwalk")
+	envPathOrigin := os.Getenv("PATH")
+	// Refer to the fake snmpwalk
+	os.Setenv("PATH", ".")
+	defer os.Setenv("PATH", envPathOrigin)
+
+	l := &LeoFS{
+		Servers: []string{endpoint},
+	}
+
+	var acc testutil.Accumulator
+
+	err := l.Gather(&acc)
+	require.NoError(t, err)
+
+	floatMetrics := KeyMapping[serverType]
+
+	for _, metric := range floatMetrics {
+		assert.True(t, acc.HasFloatValue(metric), metric)
+	}
+}
+
+func TestLeoFSManagerMasterMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Manager, "localhost:4020", ServerTypeManagerMaster)
+}
+
+func TestLeoFSManagerSlaveMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Manager, "localhost:4021", ServerTypeManagerSlave)
+}
+
+func TestLeoFSStorageMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Storage, "localhost:4010", ServerTypeStorage)
+}
+
+func TestLeoFSGatewayMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Gateway, "localhost:4000", ServerTypeGateway)
+}

plugins/lustre2/lustre2.go

@@ -0,0 +1,235 @@
+/*
+Lustre 2.x telegraf plugin
+
+Lustre (http://lustre.org/) is an open-source, parallel file system
+for HPC environments. It stores statistics about its activity in
+/proc
+
+*/
+package lustre2
+
+import (
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+// Lustre proc files can change between versions, so we want to future-proof
+// by letting people choose what to look at.
+type Lustre2 struct {
+	Ost_procfiles []string
+	Mds_procfiles []string
+}
+
+var sampleConfig = `
+  # An array of /proc globs to search for Lustre stats
+  # If not specified, the default will work on Lustre 2.5.x
+  #
+  # ost_procfiles = ["/proc/fs/lustre/obdfilter/*/stats", "/proc/fs/lustre/osd-ldiskfs/*/stats"]
+  # mds_procfiles = ["/proc/fs/lustre/mdt/*/md_stats"]
+`
+
+/* The wanted fields would be a []string if not for the
+lines that start with read_bytes/write_bytes and contain
+   both the byte count and the function call count
+*/
+type mapping struct {
+	inProc   string // What to look for at the start of a line in /proc/fs/lustre/*
+	field    uint32 // which field to extract from that line
+	reportAs string // What measurement name to use
+	tag      string // Additional tag to add for this metric
+}
+
+var wanted_ost_fields = []*mapping{
+	{
+		inProc:   "write_bytes",
+		field:    6,
+		reportAs: "write_bytes",
+	},
+	{ // line starts with 'write_bytes', but value write_calls is in second column
+		inProc:   "write_bytes",
+		field:    1,
+		reportAs: "write_calls",
+	},
+	{
+		inProc:   "read_bytes",
+		field:    6,
+		reportAs: "read_bytes",
+	},
+	{ // line starts with 'read_bytes', but value read_calls is in second column
+		inProc:   "read_bytes",
+		field:    1,
+		reportAs: "read_calls",
+	},
+	{
+		inProc: "cache_hit",
+	},
+	{
+		inProc: "cache_miss",
+	},
+	{
+		inProc: "cache_access",
+	},
+}
+
+var wanted_mds_fields = []*mapping{
+	{
+		inProc: "open",
+	},
+	{
+		inProc: "close",
+	},
+	{
+		inProc: "mknod",
+	},
+	{
+		inProc: "link",
+	},
+	{
+		inProc: "unlink",
+	},
+	{
+		inProc: "mkdir",
+	},
+	{
+		inProc: "rmdir",
+	},
+	{
+		inProc: "rename",
+	},
+	{
+		inProc: "getattr",
+	},
+	{
+		inProc: "setattr",
+	},
+	{
+		inProc: "getxattr",
+	},
+	{
+		inProc: "setxattr",
+	},
+	{
+		inProc: "statfs",
+	},
+	{
+		inProc: "sync",
+	},
+	{
+		inProc: "samedir_rename",
+	},
+	{
+		inProc: "crossdir_rename",
+	},
+}
+
+func (l *Lustre2) GetLustreProcStats(fileglob string, wanted_fields []*mapping, acc plugins.Accumulator) error {
+	files, err := filepath.Glob(fileglob)
+	if err != nil {
+		return err
+	}
+
+	for _, file := range files {
+		/* Turn /proc/fs/lustre/obdfilter/<ost_name>/stats and similar
+		 * into just the object store target name
+		 * Assumpion: the target name is always second to last,
+		 * which is true in Lustre 2.1->2.5
+		 */
+		path := strings.Split(file, "/")
+		name := path[len(path)-2]
+		tags := map[string]string{
+			"name": name,
+		}
+
+		lines, err := internal.ReadLines(file)
+		if err != nil {
+			return err
+		}
+
+		fields := make(map[string]interface{})
+		for _, line := range lines {
+			parts := strings.Fields(line)
+			for _, wanted := range wanted_fields {
+				var data uint64
+				if parts[0] == wanted.inProc {
+					wanted_field := wanted.field
+					// if not set, assume field[1]. Shouldn't be field[0], as
+					// that's a string
+					if wanted_field == 0 {
+						wanted_field = 1
+					}
+					data, err = strconv.ParseUint((parts[wanted_field]), 10, 64)
+					if err != nil {
+						return err
+					}
+					report_name := wanted.inProc
+					if wanted.reportAs != "" {
+						report_name = wanted.reportAs
+					}
+					fields[report_name] = data
+				}
+			}
+		}
+		acc.AddFields("lustre2", fields, tags)
+	}
+	return nil
+}
+
+// SampleConfig returns sample configuration message
+func (l *Lustre2) SampleConfig() string {
+	return sampleConfig
+}
+
+// Description returns description of Lustre2 plugin
+func (l *Lustre2) Description() string {
+	return "Read metrics from local Lustre service on OST, MDS"
+}
+
+// Gather reads stats from all lustre targets
+func (l *Lustre2) Gather(acc plugins.Accumulator) error {
+
+	if len(l.Ost_procfiles) == 0 {
+		// read/write bytes are in obdfilter/<ost_name>/stats
+		err := l.GetLustreProcStats("/proc/fs/lustre/obdfilter/*/stats", wanted_ost_fields, acc)
+		if err != nil {
+			return err
+		}
+		// cache counters are in osd-ldiskfs/<ost_name>/stats
+		err = l.GetLustreProcStats("/proc/fs/lustre/osd-ldiskfs/*/stats", wanted_ost_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+
+	if len(l.Mds_procfiles) == 0 {
+		// Metadata server stats
+		err := l.GetLustreProcStats("/proc/fs/lustre/mdt/*/md_stats", wanted_mds_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+
+	for _, procfile := range l.Ost_procfiles {
+		err := l.GetLustreProcStats(procfile, wanted_ost_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+	for _, procfile := range l.Mds_procfiles {
+		err := l.GetLustreProcStats(procfile, wanted_mds_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func init() {
+	plugins.Add("lustre2", func() plugins.Plugin {
+		return &Lustre2{}
+	})
+}

plugins/lustre2/lustre2_test.go

@@ -0,0 +1,144 @@
+package lustre2
+
+import (
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Set config file variables to point to fake directory structure instead of /proc?
+
+const obdfilterProcContents = `snapshot_time             1438693064.430544 secs.usecs
+read_bytes                203238095 samples [bytes] 4096 1048576 78026117632000
+write_bytes               71893382 samples [bytes] 1 1048576 15201500833981
+get_info                  1182008495 samples [reqs]
+set_info_async            2 samples [reqs]
+connect                   1117 samples [reqs]
+reconnect                 1160 samples [reqs]
+disconnect                1084 samples [reqs]
+statfs                    3575885 samples [reqs]
+create                    698 samples [reqs]
+destroy                   3190060 samples [reqs]
+setattr                   605647 samples [reqs]
+punch                     805187 samples [reqs]
+sync                      6608753 samples [reqs]
+preprw                    275131477 samples [reqs]
+commitrw                  275131477 samples [reqs]
+quotactl                  229231 samples [reqs]
+ping                      78020757 samples [reqs]
+`
+
+const osdldiskfsProcContents = `snapshot_time             1438693135.640551 secs.usecs
+get_page                  275132812 samples [usec] 0 3147 1320420955 22041662259
+cache_access              19047063027 samples [pages] 1 1 19047063027
+cache_hit                 7393729777 samples [pages] 1 1 7393729777
+cache_miss                11653333250 samples [pages] 1 1 11653333250
+`
+
+const mdtProcContents = `snapshot_time             1438693238.20113 secs.usecs
+open                      1024577037 samples [reqs]
+close                     873243496 samples [reqs]
+mknod                     349042 samples [reqs]
+link                      445 samples [reqs]
+unlink                    3549417 samples [reqs]
+mkdir                     705499 samples [reqs]
+rmdir                     227434 samples [reqs]
+rename                    629196 samples [reqs]
+getattr                   1503663097 samples [reqs]
+setattr                   1898364 samples [reqs]
+getxattr                  6145349681 samples [reqs]
+setxattr                  83969 samples [reqs]
+statfs                    2916320 samples [reqs]
+sync                      434081 samples [reqs]
+samedir_rename            259625 samples [reqs]
+crossdir_rename           369571 samples [reqs]
+`
+
+type metrics struct {
+	name  string
+	value uint64
+}
+
+func TestLustre2GeneratesMetrics(t *testing.T) {
+
+	tempdir := os.TempDir() + "/telegraf/proc/fs/lustre/"
+	ost_name := "OST0001"
+
+	mdtdir := tempdir + "/mdt/"
+	err := os.MkdirAll(mdtdir+"/"+ost_name, 0755)
+	require.NoError(t, err)
+
+	osddir := tempdir + "/osd-ldiskfs/"
+	err = os.MkdirAll(osddir+"/"+ost_name, 0755)
+	require.NoError(t, err)
+
+	obddir := tempdir + "/obdfilter/"
+	err = os.MkdirAll(obddir+"/"+ost_name, 0755)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(mdtdir+"/"+ost_name+"/md_stats", []byte(mdtProcContents), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(osddir+"/"+ost_name+"/stats", []byte(osdldiskfsProcContents), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(obddir+"/"+ost_name+"/stats", []byte(obdfilterProcContents), 0644)
+	require.NoError(t, err)
+
+	m := &Lustre2{
+		Ost_procfiles: []string{obddir + "/*/stats", osddir + "/*/stats"},
+		Mds_procfiles: []string{mdtdir + "/*/md_stats"},
+	}
+
+	var acc testutil.Accumulator
+
+	err = m.Gather(&acc)
+	require.NoError(t, err)
+
+	tags := map[string]string{
+		"name": ost_name,
+	}
+
+	intMetrics := []*metrics{
+		{
+			name:  "write_bytes",
+			value: 15201500833981,
+		},
+		{
+			name:  "read_bytes",
+			value: 78026117632000,
+		},
+		{
+			name:  "write_calls",
+			value: 71893382,
+		},
+		{
+			name:  "read_calls",
+			value: 203238095,
+		},
+		{
+			name:  "cache_hit",
+			value: 7393729777,
+		},
+		{
+			name:  "cache_access",
+			value: 19047063027,
+		},
+		{
+			name:  "cache_miss",
+			value: 11653333250,
+		},
+	}
+
+	for _, metric := range intMetrics {
+		assert.True(t, acc.HasUIntValue(metric.name), metric.name)
+		assert.True(t, acc.CheckTaggedValue(metric.name, metric.value, tags))
+	}
+
+	err = os.RemoveAll(os.TempDir() + "/telegraf")
+	require.NoError(t, err)
+}