0.3.0 Removing internal parallelism: twemproxy and rabbitmq

closes #449

.gitignore

@@ -1,3 +1,4 @@
-pkg/
 tivan
 .vagrant
+telegraf
+.idea

CHANGELOG.md

@@ -0,0 +1,338 @@
+## v0.3.0 [unreleased]
+
+### Release Notes
+- **breaking change** the `io` plugin has been renamed `diskio`
+- **breaking change** Plugin measurements aggregated into a single measurement.
+- **breaking change** `jolokia` plugin: must use global tag/drop/pass parameters
+for configuration.
+- `twemproxy` plugin: `prefix` option removed.
+- `procstat` cpu measurements are now prepended with `cpu_time_` instead of
+only `cpu_`
+- The prometheus plugin schema has not been changed (measurements have not been
+aggregated).
+
+### Features
+- Plugin measurements aggregated into a single measurement.
+- Added ability to specify per-plugin tags
+- Added ability to specify per-plugin measurement suffix and prefix.
+(`name_prefix` and `name_suffix`)
+- Added ability to override base plugin name. (`name_override`)
+
+### Bugfixes
+
+## v0.2.5 [unreleased]
+
+### Features
+- [#427](https://github.com/influxdb/telegraf/pull/427): zfs plugin: pool stats added. Thanks @allenpetersen!
+- [#428](https://github.com/influxdb/telegraf/pull/428): Amazon Kinesis output. Thanks @jimmystewpot!
+- [#449](https://github.com/influxdb/telegraf/pull/449): influxdb plugin, thanks @mark-rushakoff
+
+### Bugfixes
+- [#430](https://github.com/influxdb/telegraf/issues/430): Network statistics removed in elasticsearch 2.1. Thanks @jipperinbham!
+- [#452](https://github.com/influxdb/telegraf/issues/452): Elasticsearch open file handles error. Thanks @jipperinbham!
+
+## v0.2.4 [2015-12-08]
+
+### Features
+- [#412](https://github.com/influxdb/telegraf/pull/412): Additional memcached stats. Thanks @mgresser!
+- [#410](https://github.com/influxdb/telegraf/pull/410): Additional redis metrics. Thanks @vlaadbrain!
+- [#414](https://github.com/influxdb/telegraf/issues/414): Jolokia plugin auth parameters
+- [#415](https://github.com/influxdb/telegraf/issues/415): memcached plugin: support unix sockets
+- [#418](https://github.com/influxdb/telegraf/pull/418): memcached plugin additional unit tests.
+- [#408](https://github.com/influxdb/telegraf/pull/408): MailChimp plugin.
+- [#382](https://github.com/influxdb/telegraf/pull/382): Add system wide network protocol stats to `net` plugin.
+- [#401](https://github.com/influxdb/telegraf/pull/401): Support pass/drop/tagpass/tagdrop for outputs. Thanks @oldmantaiter!
+
+### Bugfixes
+- [#405](https://github.com/influxdb/telegraf/issues/405): Prometheus output cardinality issue
+- [#388](https://github.com/influxdb/telegraf/issues/388): Fix collection hangup when cpu times decrement.
+
+## v0.2.3 [2015-11-30]
+
+### Release Notes
+- **breaking change** The `kafka` plugin has been renamed to `kafka_consumer`.
+and most of the config option names have changed.
+This only affects the kafka consumer _plugin_ (not the
+output). There were a number of problems with the kafka plugin that led to it
+only collecting data once at startup, so the kafka plugin was basically non-
+functional.
+- Plugins can now be specified as a list, and multiple plugin instances of the
+same type can be specified, like this:
+
+```
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  drop = ["cpu_time"]
+```
+
+- Riemann output added
+- Aerospike plugin: tag changed from `host` -> `aerospike_host`
+
+### Features
+- [#379](https://github.com/influxdb/telegraf/pull/379): Riemann output, thanks @allenj!
+- [#375](https://github.com/influxdb/telegraf/pull/375): kafka_consumer service plugin.
+- [#392](https://github.com/influxdb/telegraf/pull/392): Procstat plugin can now accept pgrep -f pattern, thanks @ecarreras!
+- [#383](https://github.com/influxdb/telegraf/pull/383): Specify plugins as a list.
+- [#354](https://github.com/influxdb/telegraf/pull/354): Add ability to specify multiple metrics in one statsd line. Thanks @MerlinDMC!
+
+### Bugfixes
+- [#371](https://github.com/influxdb/telegraf/issues/371): Kafka consumer plugin not functioning.
+- [#389](https://github.com/influxdb/telegraf/issues/389): NaN value panic
+
+## v0.2.2 [2015-11-18]
+
+### Release Notes
+- 0.2.1 has a bug where all lists within plugins get duplicated, this includes
+lists of servers/URLs. 0.2.2 is being released solely to fix that bug
+
+### Bugfixes
+- [#377](https://github.com/influxdb/telegraf/pull/377): Fix for duplicate slices in plugins.
+
+## v0.2.1 [2015-11-16]
+
+### Release Notes
+- Telegraf will no longer use docker-compose for "long" unit test, it has been
+changed to just run docker commands in the Makefile. See `make docker-run` and
+`make docker-kill`. `make test` will still run all unit tests with docker.
+- Long unit tests are now run in CircleCI, with docker & race detector
+- Redis plugin tag has changed from `host` to `server`
+- HAProxy plugin tag has changed from `host` to `server`
+- UDP output now supported
+- Telegraf will now compile on FreeBSD
+- Users can now specify outputs as lists, specifying multiple outputs of the
+same type.
+
+### Features
+- [#325](https://github.com/influxdb/telegraf/pull/325): NSQ output. Thanks @jrxFive!
+- [#318](https://github.com/influxdb/telegraf/pull/318): Prometheus output. Thanks @oldmantaiter!
+- [#338](https://github.com/influxdb/telegraf/pull/338): Restart Telegraf on package upgrade. Thanks @linsomniac!
+- [#337](https://github.com/influxdb/telegraf/pull/337): Jolokia plugin, thanks @saiello!
+- [#350](https://github.com/influxdb/telegraf/pull/350): Amon output.
+- [#365](https://github.com/influxdb/telegraf/pull/365): Twemproxy plugin by @codeb2cc
+- [#317](https://github.com/influxdb/telegraf/issues/317): ZFS plugin, thanks @cornerot!
+- [#364](https://github.com/influxdb/telegraf/pull/364): Support InfluxDB UDP output.
+- [#370](https://github.com/influxdb/telegraf/pull/370): Support specifying multiple outputs, as lists.
+- [#372](https://github.com/influxdb/telegraf/pull/372): Remove gosigar and update go-dockerclient for FreeBSD support. Thanks @MerlinDMC!
+
+### Bugfixes
+- [#331](https://github.com/influxdb/telegraf/pull/331): Dont overwrite host tag in redis plugin.
+- [#336](https://github.com/influxdb/telegraf/pull/336): Mongodb plugin should take 2 measurements.
+- [#351](https://github.com/influxdb/telegraf/issues/317): Fix continual "CREATE DATABASE" in writes
+- [#360](https://github.com/influxdb/telegraf/pull/360): Apply prefix before ShouldPass check. Thanks @sotfo!
+
+## v0.2.0 [2015-10-27]
+
+### Release Notes
+- The -test flag will now only output 2 collections for plugins that need it
+- There is a new agent configuration option: `flush_interval`. This option tells
+Telegraf how often to flush data to InfluxDB and other output sinks. For example,
+users can set `interval = "2s"` and `flush_interval = "60s"` for Telegraf to
+collect data every 2 seconds, and flush every 60 seconds.
+- `precision` and `utc` are no longer valid agent config values. `precision` has
+moved to the `influxdb` output config, where it will continue to default to "s"
+- debug and test output will now print the raw line-protocol string
+- Telegraf will now, by default, round the collection interval to the nearest
+even interval. This means that `interval="10s"` will collect every :00, :10, etc.
+To ease scale concerns, flushing will be "jittered" by a random amount so that
+all Telegraf instances do not flush at the same time. Both of these options can
+be controlled via the `round_interval` and `flush_jitter` config options.
+- Telegraf will now retry metric flushes twice
+
+### Features
+- [#205](https://github.com/influxdb/telegraf/issues/205): Include per-db redis keyspace info
+- [#226](https://github.com/influxdb/telegraf/pull/226): Add timestamps to points in Kafka/AMQP outputs. Thanks @ekini
+- [#90](https://github.com/influxdb/telegraf/issues/90): Add Docker labels to tags in docker plugin
+- [#223](https://github.com/influxdb/telegraf/pull/223): Add port tag to nginx plugin. Thanks @neezgee!
+- [#227](https://github.com/influxdb/telegraf/pull/227): Add command intervals to exec plugin. Thanks @jpalay!
+- [#241](https://github.com/influxdb/telegraf/pull/241): MQTT Output. Thanks @shirou!
+- Memory plugin: cached and buffered measurements re-added
+- Logging: additional logging for each collection interval, track the number
+of metrics collected and from how many plugins.
+- [#240](https://github.com/influxdb/telegraf/pull/240): procstat plugin, thanks @ranjib!
+- [#244](https://github.com/influxdb/telegraf/pull/244): netstat plugin, thanks @shirou!
+- [#262](https://github.com/influxdb/telegraf/pull/262): zookeeper plugin, thanks @jrxFive!
+- [#237](https://github.com/influxdb/telegraf/pull/237): statsd service plugin, thanks @sparrc
+- [#273](https://github.com/influxdb/telegraf/pull/273): puppet agent plugin, thats @jrxFive!
+- [#280](https://github.com/influxdb/telegraf/issues/280): Use InfluxDB client v2.
+- [#281](https://github.com/influxdb/telegraf/issues/281): Eliminate need to deep copy Batch Points.
+- [#286](https://github.com/influxdb/telegraf/issues/286): bcache plugin, thanks @cornerot!
+- [#287](https://github.com/influxdb/telegraf/issues/287): Batch AMQP output, thanks @ekini!
+- [#301](https://github.com/influxdb/telegraf/issues/301): Collect on even intervals
+- [#298](https://github.com/influxdb/telegraf/pull/298): Support retrying output writes
+- [#300](https://github.com/influxdb/telegraf/issues/300): aerospike plugin. Thanks @oldmantaiter!
+- [#322](https://github.com/influxdb/telegraf/issues/322): Librato output. Thanks @jipperinbham!
+
+### Bugfixes
+- [#228](https://github.com/influxdb/telegraf/pull/228): New version of package will replace old one. Thanks @ekini!
+- [#232](https://github.com/influxdb/telegraf/pull/232): Fix bashism run during deb package installation. Thanks @yankcrime!
+- [#261](https://github.com/influxdb/telegraf/issues/260): RabbitMQ panics if wrong credentials given. Thanks @ekini!
+- [#245](https://github.com/influxdb/telegraf/issues/245): Document Exec plugin example. Thanks @ekini!
+- [#264](https://github.com/influxdb/telegraf/issues/264): logrotate config file fixes. Thanks @linsomniac!
+- [#290](https://github.com/influxdb/telegraf/issues/290): Fix some plugins sending their values as strings.
+- [#289](https://github.com/influxdb/telegraf/issues/289): Fix accumulator panic on nil tags.
+- [#302](https://github.com/influxdb/telegraf/issues/302): Fix `[tags]` getting applied, thanks @gotyaoi!
+
+## v0.1.9 [2015-09-22]
+
+### Release Notes
+- InfluxDB output config change: `url` is now `urls`, and is a list. Config files
+will still be backwards compatible if only `url` is specified.
+- The -test flag will now output two metric collections
+- Support for filtering telegraf outputs on the CLI -- Telegraf will now
+allow filtering of output sinks on the command-line using the `-outputfilter`
+flag, much like how the `-filter` flag works for plugins.
+- Support for filtering on config-file creation -- Telegraf now supports
+filtering to -sample-config command. You can now run
+`telegraf -sample-config -filter cpu -outputfilter influxdb` to get a config
+file with only the cpu plugin defined, and the influxdb output defined.
+- **Breaking Change**: The CPU collection plugin has been refactored to fix some
+bugs and outdated dependency issues. At the same time, I also decided to fix
+a naming consistency issue, so cpu_percentageIdle will become cpu_usage_idle.
+Also, all CPU time measurements now have it indicated in their name, so cpu_idle
+will become cpu_time_idle. Additionally, cpu_time measurements are going to be
+dropped in the default config.
+- **Breaking Change**: The memory plugin has been refactored and some measurements
+have been renamed for consistency. Some measurements have also been removed from being outputted. They are still being collected by gopsutil, and could easily be
+re-added in a "verbose" mode if there is demand for it.
+
+### Features
+- [#143](https://github.com/influxdb/telegraf/issues/143): InfluxDB clustering support
+- [#181](https://github.com/influxdb/telegraf/issues/181): Makefile GOBIN support. Thanks @Vye!
+- [#203](https://github.com/influxdb/telegraf/pull/200): AMQP output. Thanks @ekini!
+- [#182](https://github.com/influxdb/telegraf/pull/182): OpenTSDB output. Thanks @rplessl!
+- [#187](https://github.com/influxdb/telegraf/pull/187): Retry output sink connections on startup.
+- [#220](https://github.com/influxdb/telegraf/pull/220): Add port tag to apache plugin. Thanks @neezgee!
+- [#217](https://github.com/influxdb/telegraf/pull/217): Add filtering for output sinks
+and filtering when specifying a config file.
+
+### Bugfixes
+- [#170](https://github.com/influxdb/telegraf/issues/170): Systemd support
+- [#175](https://github.com/influxdb/telegraf/issues/175): Set write precision before gathering metrics
+- [#178](https://github.com/influxdb/telegraf/issues/178): redis plugin, multiple server thread hang bug
+- Fix net plugin on darwin
+- [#84](https://github.com/influxdb/telegraf/issues/84): Fix docker plugin on CentOS. Thanks @neezgee!
+- [#189](https://github.com/influxdb/telegraf/pull/189): Fix mem_used_perc. Thanks @mced!
+- [#192](https://github.com/influxdb/telegraf/issues/192): Increase compatibility of postgresql plugin. Now supports versions 8.1+
+- [#203](https://github.com/influxdb/telegraf/issues/203): EL5 rpm support. Thanks @ekini!
+- [#206](https://github.com/influxdb/telegraf/issues/206): CPU steal/guest values wrong on linux.
+- [#212](https://github.com/influxdb/telegraf/issues/212): Add hashbang to postinstall script. Thanks @ekini!
+- [#212](https://github.com/influxdb/telegraf/issues/212): Fix makefile warning. Thanks @ekini!
+
+## v0.1.8 [2015-09-04]
+
+### Release Notes
+- Telegraf will now write data in UTC at second precision by default
+- Now using Go 1.5 to build telegraf
+
+### Features
+- [#150](https://github.com/influxdb/telegraf/pull/150): Add Host Uptime metric to system plugin
+- [#158](https://github.com/influxdb/telegraf/pull/158): Apache Plugin. Thanks @KPACHbIuLLIAnO4
+- [#159](https://github.com/influxdb/telegraf/pull/159): Use second precision for InfluxDB writes
+- [#165](https://github.com/influxdb/telegraf/pull/165): Add additional metrics to mysql plugin. Thanks @nickscript0
+- [#162](https://github.com/influxdb/telegraf/pull/162): Write UTC by default, provide option
+- [#166](https://github.com/influxdb/telegraf/pull/166): Upload binaries to S3
+- [#169](https://github.com/influxdb/telegraf/pull/169): Ping plugin
+
+### Bugfixes
+
+## v0.1.7 [2015-08-28]
+
+### Features
+- [#38](https://github.com/influxdb/telegraf/pull/38): Kafka output producer.
+- [#133](https://github.com/influxdb/telegraf/pull/133): Add plugin.Gather error logging. Thanks @nickscript0!
+- [#136](https://github.com/influxdb/telegraf/issues/136): Add a -usage flag for printing usage of a single plugin.
+- [#137](https://github.com/influxdb/telegraf/issues/137): Memcached: fix when a value contains a space
+- [#138](https://github.com/influxdb/telegraf/issues/138): MySQL server address tag.
+- [#142](https://github.com/influxdb/telegraf/pull/142): Add Description and SampleConfig funcs to output interface
+- Indent the toml config file for readability
+
+### Bugfixes
+- [#128](https://github.com/influxdb/telegraf/issues/128): system_load measurement missing.
+- [#129](https://github.com/influxdb/telegraf/issues/129): Latest pkg url fix.
+- [#131](https://github.com/influxdb/telegraf/issues/131): Fix memory reporting on linux & darwin. Thanks @subhachandrachandra!
+- [#140](https://github.com/influxdb/telegraf/issues/140): Memory plugin prec->perc typo fix. Thanks @brunoqc!
+
+## v0.1.6 [2015-08-20]
+
+### Features
+- [#112](https://github.com/influxdb/telegraf/pull/112): Datadog output. Thanks @jipperinbham!
+- [#116](https://github.com/influxdb/telegraf/pull/116): Use godep to vendor all dependencies
+- [#120](https://github.com/influxdb/telegraf/pull/120): Httpjson plugin. Thanks @jpalay & @alvaromorales!
+
+### Bugfixes
+- [#113](https://github.com/influxdb/telegraf/issues/113): Update README with Telegraf/InfluxDB compatibility
+- [#118](https://github.com/influxdb/telegraf/pull/118): Fix for disk usage stats in Windows. Thanks @srfraser!
+- [#122](https://github.com/influxdb/telegraf/issues/122): Fix for DiskUsage segv fault. Thanks @srfraser!
+- [#126](https://github.com/influxdb/telegraf/issues/126): Nginx plugin not catching net.SplitHostPort error
+
+## v0.1.5 [2015-08-13]
+
+### Features
+- [#54](https://github.com/influxdb/telegraf/pull/54): MongoDB plugin. Thanks @jipperinbham!
+- [#55](https://github.com/influxdb/telegraf/pull/55): Elasticsearch plugin. Thanks @brocaar!
+- [#71](https://github.com/influxdb/telegraf/pull/71): HAProxy plugin. Thanks @kureikain!
+- [#72](https://github.com/influxdb/telegraf/pull/72): Adding TokuDB metrics to MySQL. Thanks vadimtk!
+- [#73](https://github.com/influxdb/telegraf/pull/73): RabbitMQ plugin. Thanks @ianunruh!
+- [#77](https://github.com/influxdb/telegraf/issues/77): Automatically create database.
+- [#79](https://github.com/influxdb/telegraf/pull/56): Nginx plugin. Thanks @codeb2cc!
+- [#86](https://github.com/influxdb/telegraf/pull/86): Lustre2 plugin. Thanks srfraser!
+- [#91](https://github.com/influxdb/telegraf/pull/91): Unit testing
+- [#92](https://github.com/influxdb/telegraf/pull/92): Exec plugin. Thanks @alvaromorales!
+- [#98](https://github.com/influxdb/telegraf/pull/98): LeoFS plugin. Thanks @mocchira!
+- [#103](https://github.com/influxdb/telegraf/pull/103): Filter by metric tags. Thanks @srfraser!
+- [#106](https://github.com/influxdb/telegraf/pull/106): Options to filter plugins on startup. Thanks @zepouet!
+- [#107](https://github.com/influxdb/telegraf/pull/107): Multiple outputs beyong influxdb. Thanks @jipperinbham!
+- [#108](https://github.com/influxdb/telegraf/issues/108): Support setting per-CPU and total-CPU gathering.
+- [#111](https://github.com/influxdb/telegraf/pull/111): Report CPU Usage in cpu plugin. Thanks @jpalay!
+
+### Bugfixes
+- [#85](https://github.com/influxdb/telegraf/pull/85): Fix GetLocalHost testutil function for mac users
+- [#89](https://github.com/influxdb/telegraf/pull/89): go fmt fixes
+- [#94](https://github.com/influxdb/telegraf/pull/94): Fix for issue #93, explicitly call sarama.v1 -> sarama
+- [#101](https://github.com/influxdb/telegraf/issues/101): switch back from master branch if building locally
+- [#99](https://github.com/influxdb/telegraf/issues/99): update integer output to new InfluxDB line protocol format
+
+## v0.1.4 [2015-07-09]
+
+### Features
+- [#56](https://github.com/influxdb/telegraf/pull/56): Update README for Kafka plugin. Thanks @EmilS!
+
+### Bugfixes
+- [#50](https://github.com/influxdb/telegraf/pull/50): Fix init.sh script to use telegraf directory. Thanks @jseriff!
+- [#52](https://github.com/influxdb/telegraf/pull/52): Update CHANGELOG to reference updated directory. Thanks @benfb!
+
+## v0.1.3 [2015-07-05]
+
+### Features
+- [#35](https://github.com/influxdb/telegraf/pull/35): Add Kafka plugin. Thanks @EmilS!
+- [#47](https://github.com/influxdb/telegraf/pull/47): Add RethinkDB plugin. Thanks @jipperinbham!
+
+### Bugfixes
+- [#45](https://github.com/influxdb/telegraf/pull/45): Skip disk tags that don't have a value. Thanks @jhofeditz!
+- [#43](https://github.com/influxdb/telegraf/pull/43): Fix bug in MySQL plugin. Thanks @marcosnils!
+
+## v0.1.2 [2015-07-01]
+
+### Features
+- [#12](https://github.com/influxdb/telegraf/pull/12): Add Linux/ARM to the list of built binaries. Thanks @voxxit!
+- [#14](https://github.com/influxdb/telegraf/pull/14): Clarify the S3 buckets that Telegraf is pushed to.
+- [#16](https://github.com/influxdb/telegraf/pull/16): Convert Redis to use URI, support Redis AUTH. Thanks @jipperinbham!
+- [#21](https://github.com/influxdb/telegraf/pull/21): Add memcached plugin. Thanks @Yukki!
+
+### Bugfixes
+- [#13](https://github.com/influxdb/telegraf/pull/13): Fix the packaging script.
+- [#19](https://github.com/influxdb/telegraf/pull/19): Add host name to metric tags. Thanks @sherifzain!
+- [#20](https://github.com/influxdb/telegraf/pull/20): Fix race condition with accumulator mutex. Thanks @nkatsaros!
+- [#23](https://github.com/influxdb/telegraf/pull/23): Change name of folder for packages. Thanks @colinrymer!
+- [#32](https://github.com/influxdb/telegraf/pull/32): Fix spelling of memoory -> memory. Thanks @tylernisonoff!
+
+## v0.1.1 [2015-06-19]
+
+### Release Notes
+
+This is the initial release of Telegraf.

CONFIGURATION.md

@@ -0,0 +1,177 @@
+# Telegraf Configuration
+
+## Plugin Configuration
+
+There are some configuration options that are configurable per plugin:
+
+* **name_override**: Override the base name of the measurement.
+(Default is the name of the plugin).
+* **name_prefix**: Specifies a prefix to attach to the measurement name.
+* **name_suffix**: Specifies a suffix to attach to the measurement name.
+* **tags**: A map of tags to apply to a specific plugin's measurements.
+
+### Plugin Filters
+
+There are also filters that can be configured per plugin:
+
+* **pass**: An array of strings that is used to filter metrics generated by the
+current plugin. Each string in the array is tested as a glob match against field names
+and if it matches, the field is emitted.
+* **drop**: The inverse of pass, if a field name matches, it is not emitted.
+* **tagpass**: tag names and arrays of strings that are used to filter
+measurements by the current plugin. Each string in the array is tested as a glob
+match against the tag name, and if it matches the measurement is emitted.
+* **tagdrop**: The inverse of tagpass. If a tag matches, the measurement is not emitted.
+This is tested on measurements that have passed the tagpass test.
+* **interval**: How often to gather this metric. Normal plugins use a single
+global interval, but if one particular plugin should be run less or more often,
+you can configure that here.
+
+### Plugin Configuration Examples
+
+This is a full working config that will output CPU data to an InfluxDB instance
+at 192.168.59.103:8086, tagging measurements with dc="denver-1". It will output
+measurements at a 10s interval and will collect per-cpu data, dropping any
+fields which begin with `time_`.
+
+```toml
+[tags]
+  dc = "denver-1"
+
+[agent]
+  interval = "10s"
+
+# OUTPUTS
+[outputs]
+[[outputs.influxdb]]
+  url = "http://192.168.59.103:8086" # required.
+  database = "telegraf" # required.
+  precision = "s"
+
+# PLUGINS
+[plugins]
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  # filter all fields beginning with 'time_'
+  drop = ["time_*"]
+```
+
+### Plugin Config: tagpass and tagdrop
+
+```toml
+[plugins]
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  drop = ["cpu_time"]
+  # Don't collect CPU data for cpu6 & cpu7
+  [plugins.cpu.tagdrop]
+    cpu = [ "cpu6", "cpu7" ]
+
+[[plugins.disk]]
+  [plugins.disk.tagpass]
+    # tagpass conditions are OR, not AND.
+    # If the (filesystem is ext4 or xfs) OR (the path is /opt or /home)
+    # then the metric passes
+    fstype = [ "ext4", "xfs" ]
+    # Globs can also be used on the tag values
+    path = [ "/opt", "/home*" ]
+```
+
+### Plugin Config: pass and drop
+
+```toml
+# Drop all metrics for guest & steal CPU usage
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+  drop = ["usage_guest", "usage_steal"]
+
+# Only store inode related metrics for disks
+[[plugins.disk]]
+  pass = ["inodes*"]
+```
+
+### Plugin config: prefix, suffix, and override
+
+This plugin will emit measurements with the name `cpu_total`
+
+```toml
+[[plugins.cpu]]
+  name_suffix = "_total"
+  percpu = false
+  totalcpu = true
+```
+
+This will emit measurements with the name `foobar`
+
+```toml
+[[plugins.cpu]]
+  name_override = "foobar"
+  percpu = false
+  totalcpu = true
+```
+
+### Plugin config: tags
+
+This plugin will emit measurements with two additional tags: `tag1=foo` and
+`tag2=bar`
+
+```toml
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+  [plugins.cpu.tags]
+    tag1 = "foo"
+    tag2 = "bar"
+```
+
+### Multiple plugins of the same type
+
+Additional plugins (or outputs) of the same type can be specified,
+just define more instances in the config file:
+
+```toml
+[[plugins.cpu]]
+  percpu = false
+  totalcpu = true
+
+[[plugins.cpu]]
+  percpu = true
+  totalcpu = false
+  drop = ["cpu_time*"]
+```
+
+## Output Configuration
+
+Telegraf also supports specifying multiple output sinks to send data to,
+configuring each output sink is different, but examples can be
+found by running `telegraf -sample-config`.
+
+Outputs also support the same configurable options as plugins
+(pass, drop, tagpass, tagdrop), added in 0.2.4
+
+```toml
+[[outputs.influxdb]]
+  urls = [ "http://localhost:8086" ]
+  database = "telegraf"
+  precision = "s"
+  # Drop all measurements that start with "aerospike"
+  drop = ["aerospike*"]
+
+[[outputs.influxdb]]
+  urls = [ "http://localhost:8086" ]
+  database = "telegraf-aerospike-data"
+  precision = "s"
+  # Only accept aerospike data:
+  pass = ["aerospike*"]
+
+[[outputs.influxdb]]
+  urls = [ "http://localhost:8086" ]
+  database = "telegraf-cpu0-data"
+  precision = "s"
+  # Only store measurements where the tag "cpu" matches the value "cpu0"
+  [outputs.influxdb.tagpass]
+    cpu = ["cpu0"]
+```

CONTRIBUTING.md

@@ -0,0 +1,293 @@
+## Sign the CLA
+
+Before we can merge a pull request, you will need to sign the CLA,
+which can be found [on our website](http://influxdb.com/community/cla.html)
+
+## Plugins
+
+This section is for developers who want to create new collection plugins.
+Telegraf is entirely plugin driven. This interface allows for operators to
+pick and chose what is gathered as well as makes it easy for developers
+to create new ways of generating metrics.
+
+Plugin authorship is kept as simple as possible to promote people to develop
+and submit new plugins.
+
+### Plugin Guidelines
+
+* A plugin must conform to the `plugins.Plugin` interface.
+* Each generated metric automatically has the name of the plugin that generated
+it prepended. This is to keep plugins honest.
+* Plugins should call `plugins.Add` in their `init` function to register themselves.
+See below for a quick example.
+* To be available within Telegraf itself, plugins must add themselves to the
+`github.com/influxdb/telegraf/plugins/all/all.go` file.
+* The `SampleConfig` function should return valid toml that describes how the
+plugin can be configured. This is include in `telegraf -sample-config`.
+* The `Description` function should say in one line what this plugin does.
+
+### Plugin interface
+
+```go
+type Plugin interface {
+    SampleConfig() string
+    Description() string
+    Gather(Accumulator) error
+}
+
+type Accumulator interface {
+    Add(measurement string,
+        value interface{},
+        tags map[string]string,
+        timestamp ...time.Time)
+    AddFields(measurement string,
+        fields map[string]interface{},
+        tags map[string]string,
+        timestamp ...time.Time)
+}
+```
+
+### Accumulator
+
+The way that a plugin emits metrics is by interacting with the Accumulator.
+
+The `Add` function takes 3 arguments:
+* **measurement**: A string description of the metric. For instance `bytes_read` or `faults`.
+* **value**: A value for the metric. This accepts 5 different types of value:
+  * **int**: The most common type. All int types are accepted but favor using `int64`
+  Useful for counters, etc.
+  * **float**: Favor `float64`, useful for gauges, percentages, etc.
+  * **bool**: `true` or `false`, useful to indicate the presence of a state. `light_on`, etc.
+  * **string**: Typically used to indicate a message, or some kind of freeform information.
+  * **time.Time**: Useful for indicating when a state last occurred, for instance `light_on_since`.
+* **tags**: This is a map of strings to strings to describe the where or who
+about the metric. For instance, the `net` plugin adds a tag named `"interface"`
+set to the name of the network interface, like `"eth0"`.
+
+The `AddFieldsWithTime` allows multiple values for a point to be passed. The values
+used are the same type profile as **value** above. The **timestamp** argument
+allows a point to be registered as having occurred at an arbitrary time.
+
+Let's say you've written a plugin that emits metrics about processes on the current host.
+
+```go
+
+type Process struct {
+    CPUTime float64
+    MemoryBytes int64
+    PID int
+}
+
+func Gather(acc plugins.Accumulator) error {
+    for _, process := range system.Processes() {
+        tags := map[string]string {
+            "pid": fmt.Sprintf("%d", process.Pid),
+        }
+
+        acc.Add("cpu", process.CPUTime, tags, time.Now())
+        acc.Add("memory", process.MemoryBytes, tags, time.Now())
+    }
+}
+```
+
+### Plugin Example
+
+```go
+package simple
+
+// simple.go
+
+import "github.com/influxdb/telegraf/plugins"
+
+type Simple struct {
+    Ok bool
+}
+
+func (s *Simple) Description() string {
+    return "a demo plugin"
+}
+
+func (s *Simple) SampleConfig() string {
+    return "ok = true # indicate if everything is fine"
+}
+
+func (s *Simple) Gather(acc plugins.Accumulator) error {
+    if s.Ok {
+        acc.Add("state", "pretty good", nil)
+    } else {
+        acc.Add("state", "not great", nil)
+    }
+
+    return nil
+}
+
+func init() {
+    plugins.Add("simple", func() plugins.Plugin { return &Simple{} })
+}
+```
+
+## Service Plugins
+
+This section is for developers who want to create new "service" collection
+plugins. A service plugin differs from a regular plugin in that it operates
+a background service while Telegraf is running. One example would be the `statsd`
+plugin, which operates a statsd server.
+
+Service Plugins are substantially more complicated than a regular plugin, as they
+will require threads and locks to verify data integrity. Service Plugins should
+be avoided unless there is no way to create their behavior with a regular plugin.
+
+Their interface is quite similar to a regular plugin, with the addition of `Start()`
+and `Stop()` methods.
+
+### Service Plugin Guidelines
+
+* Same as the `Plugin` guidelines, except that they must conform to the
+`plugins.ServicePlugin` interface.
+
+### Service Plugin interface
+
+```go
+type ServicePlugin interface {
+    SampleConfig() string
+    Description() string
+    Gather(Accumulator) error
+    Start() error
+    Stop()
+}
+```
+
+## Outputs
+
+This section is for developers who want to create a new output sink. Outputs
+are created in a similar manner as collection plugins, and their interface has
+similar constructs.
+
+### Output Guidelines
+
+* An output must conform to the `outputs.Output` interface.
+* Outputs should call `outputs.Add` in their `init` function to register themselves.
+See below for a quick example.
+* To be available within Telegraf itself, plugins must add themselves to the
+`github.com/influxdb/telegraf/outputs/all/all.go` file.
+* The `SampleConfig` function should return valid toml that describes how the
+output can be configured. This is include in `telegraf -sample-config`.
+* The `Description` function should say in one line what this output does.
+
+### Output interface
+
+```go
+type Output interface {
+    Connect() error
+    Close() error
+    Description() string
+    SampleConfig() string
+    Write(points []*client.Point) error
+}
+```
+
+### Output Example
+
+```go
+package simpleoutput
+
+// simpleoutput.go
+
+import "github.com/influxdb/telegraf/outputs"
+
+type Simple struct {
+    Ok bool
+}
+
+func (s *Simple) Description() string {
+    return "a demo output"
+}
+
+func (s *Simple) SampleConfig() string {
+    return "url = localhost"
+}
+
+func (s *Simple) Connect() error {
+    // Make a connection to the URL here
+    return nil
+}
+
+func (s *Simple) Close() error {
+    // Close connection to the URL here
+    return nil
+}
+
+func (s *Simple) Write(points []*client.Point) error {
+    for _, pt := range points {
+        // write `pt` to the output sink here
+    }
+    return nil
+}
+
+func init() {
+    outputs.Add("simpleoutput", func() outputs.Output { return &Simple{} })
+}
+
+```
+
+## Service Outputs
+
+This section is for developers who want to create new "service" output. A
+service output differs from a regular output in that it operates a background service
+while Telegraf is running. One example would be the `prometheus_client` output,
+which operates an HTTP server.
+
+Their interface is quite similar to a regular output, with the addition of `Start()`
+and `Stop()` methods.
+
+### Service Output Guidelines
+
+* Same as the `Output` guidelines, except that they must conform to the
+`plugins.ServiceOutput` interface.
+
+### Service Output interface
+
+```go
+type ServiceOutput interface {
+    Connect() error
+    Close() error
+    Description() string
+    SampleConfig() string
+    Write(points []*client.Point) error
+    Start() error
+    Stop()
+}
+```
+
+## Unit Tests
+
+### Execute short tests
+
+execute `make test-short`
+
+### Execute long tests
+
+As Telegraf collects metrics from several third-party services it becomes a
+difficult task to mock each service as some of them have complicated protocols
+which would take some time to replicate.
+
+To overcome this situation we've decided to use docker containers to provide a
+fast and reproducible environment to test those services which require it.
+For other situations
+(i.e: https://github.com/influxdb/telegraf/blob/master/plugins/redis/redis_test.go )
+a simple mock will suffice.
+
+To execute Telegraf tests follow these simple steps:
+
+- Install docker following [these](https://docs.docker.com/installation/)
+instructions
+- execute `make test`
+
+**OSX users**: you will need to install `boot2docker` or `docker-machine`.
+The Makefile will assume that you have a `docker-machine` box called `default` to
+get the IP address.
+
+### Unit test troubleshooting
+
+Try cleaning up your test environment by executing `make docker-kill` and
+re-running

Godeps

@@ -0,0 +1,52 @@
+git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git dbd8d5c40a582eb9adacde36b47932b3a3ad0034
+github.com/Shopify/sarama 159e9990b0796511607dd0d7aaa3eb37d1829d16
+github.com/Sirupsen/logrus 446d1c146faa8ed3f4218f056fcd165f6bcfda81
+github.com/amir/raidman 6a8e089bbe32e6b907feae5ba688841974b3c339
+github.com/armon/go-metrics 06b60999766278efd6d2b5d8418a58c3d5b99e87
+github.com/aws/aws-sdk-go 999b1591218c36d5050d1ba7266eba956e65965f
+github.com/beorn7/perks b965b613227fddccbfffe13eae360ed3fa822f8d
+github.com/boltdb/bolt b34b35ea8d06bb9ae69d9a349119252e4c1d8ee0
+github.com/cenkalti/backoff 4dc77674aceaabba2c7e3da25d4c823edfb73f99
+github.com/dancannon/gorethink a124c9663325ed9f7fb669d17c69961b59151e6e
+github.com/davecgh/go-spew 5215b55f46b2b919f50a1df0eaa5886afe4e3b3d
+github.com/eapache/go-resiliency f341fb4dca45128e4aa86389fa6a675d55fe25e1
+github.com/eapache/queue ded5959c0d4e360646dc9e9908cff48666781367
+github.com/fsouza/go-dockerclient 7177a9e3543b0891a5d91dbf7051e0f71455c8ef
+github.com/go-ini/ini 9314fb0ef64171d6a3d0a4fa570dfa33441cba05
+github.com/go-sql-driver/mysql d512f204a577a4ab037a1816604c48c9c13210be
+github.com/gogo/protobuf e492fd34b12d0230755c45aa5fb1e1eea6a84aa9
+github.com/golang/protobuf 68415e7123da32b07eab49c96d2c4d6158360e9b
+github.com/golang/snappy 723cc1e459b8eea2dea4583200fd60757d40097a
+github.com/gonuts/go-shellquote e842a11b24c6abfb3dd27af69a17f482e4b483c2
+github.com/hailocab/go-hostpool 0637eae892be221164aff5fcbccc57171aea6406
+github.com/hashicorp/go-msgpack fa3f63826f7c23912c15263591e65d54d080b458
+github.com/hashicorp/raft d136cd15dfb7876fd7c89cad1995bc4f19ceb294
+github.com/hashicorp/raft-boltdb d1e82c1ec3f15ee991f7cc7ffd5b67ff6f5bbaee
+github.com/influxdb/influxdb 69a7664f2d4b75aec300b7cbfc7e57c971721f04
+github.com/jmespath/go-jmespath c01cf91b011868172fdcd9f41838e80c9d716264
+github.com/klauspost/crc32 0aff1ea9c20474c3901672b5b6ead0ac611156de
+github.com/lib/pq 11fc39a580a008f1f39bb3d11d984fb34ed778d9
+github.com/matttproud/golang_protobuf_extensions d0c3fe89de86839aecf2e0579c40ba3bb336a453
+github.com/mreiferson/go-snappystream 028eae7ab5c4c9e2d1cb4c4ca1e53259bbe7e504
+github.com/naoina/go-stringutil 6b638e95a32d0c1131db0e7fe83775cbea4a0d0b
+github.com/naoina/toml 751171607256bb66e64c9f0220c00662420c38e9
+github.com/nsqio/go-nsq 2118015c120962edc5d03325c680daf3163a8b5f
+github.com/pborman/uuid cccd189d45f7ac3368a0d127efb7f4d08ae0b655
+github.com/pmezard/go-difflib e8554b8641db39598be7f6342874b958f12ae1d4
+github.com/prometheus/client_golang 67994f177195311c3ea3d4407ed0175e34a4256f
+github.com/prometheus/client_model fa8ad6fec33561be4280a8f0514318c79d7f6cb6
+github.com/prometheus/common 56b90312e937d43b930f06a59bf0d6a4ae1944bc
+github.com/prometheus/procfs 406e5b7bfd8201a36e2bb5f7bdae0b03380c2ce8
+github.com/samuel/go-zookeeper 218e9c81c0dd8b3b18172b2bbfad92cc7d6db55f
+github.com/shirou/gopsutil fc932d9090f13a84fb4b3cb8baa124610cab184c
+github.com/streadway/amqp b4f3ceab0337f013208d31348b578d83c0064744
+github.com/stretchr/objx 1a9d0bb9f541897e62256577b352fdbc1fb4fd94
+github.com/stretchr/testify e3a8ff8ce36581f87a15341206f205b1da467059
+github.com/wvanbergen/kafka 1a8639a45164fcc245d5c7b4bd3ccfbd1a0ffbf3
+github.com/wvanbergen/kazoo-go 0f768712ae6f76454f987c3356177e138df258f8
+golang.org/x/crypto 7b85b097bf7527677d54d3220065e966a0e3b613
+golang.org/x/net 1796f9b8b7178e3c7587dff118d3bb9d37f9b0b3
+gopkg.in/dancannon/gorethink.v1 a124c9663325ed9f7fb669d17c69961b59151e6e
+gopkg.in/fatih/pool.v2 cba550ebf9bce999a02e963296d4bc7a486cb715
+gopkg.in/mgo.v2 e30de8ac9ae3b30df7065f766c71f88bba7d4e49
+gopkg.in/yaml.v2 f7716cbe52baa25d2e9b0d0da546fcf909fc16b4

LICENSE_OF_DEPENDENCIES.md

@@ -0,0 +1,33 @@
+# List
+- github.com/Shopify/sarama [MIT LICENSE](https://github.com/Shopify/sarama/blob/master/MIT-LICENSE)
+- github.com/Sirupsen/logrus [MIT LICENSE](https://github.com/Sirupsen/logrus/blob/master/LICENSE)
+- github.com/armon/go-metrics [MIT LICENSE](https://github.com/armon/go-metrics/blob/master/LICENSE)
+- github.com/boltdb/bolt [MIT LICENSE](https://github.com/boltdb/bolt/blob/master/LICENSE)
+- github.com/cenkalti/backoff [MIT LICENSE](https://github.com/cenkalti/backoff/blob/master/LICENSE)
+- github.com/dancannon/gorethink [APACHE LICENSE](https://github.com/dancannon/gorethink/blob/master/LICENSE)
+- github.com/eapache/go-resiliency [MIT LICENSE](https://github.com/eapache/go-resiliency/blob/master/LICENSE)
+- github.com/eapache/queue [MIT LICENSE](https://github.com/eapache/queue/blob/master/LICENSE)
+- github.com/fsouza/go-dockerclient [BSD LICENSE](https://github.com/fsouza/go-dockerclient/blob/master/LICENSE)
+- github.com/go-sql-driver/mysql [MPL LICENSE](https://github.com/go-sql-driver/mysql/blob/master/LICENSE)
+- github.com/gogo/protobuf [BSD LICENSE](https://github.com/gogo/protobuf/blob/master/LICENSE)
+- github.com/golang/protobuf [BSD LICENSE](https://github.com/golang/protobuf/blob/master/LICENSE)
+- github.com/golang/snappy [BSD LICENSE](https://github.com/golang/snappy/blob/master/LICENSE)
+- github.com/gonuts/go-shellquote (No License, but the project it was forked from https://github.com/kballard/go-shellquote is [MIT](https://github.com/kballard/go-shellquote/blob/master/LICENSE)).
+- github.com/hashicorp/go-msgpack [BSD LICENSE](https://github.com/hashicorp/go-msgpack/blob/master/LICENSE)
+- github.com/hashicorp/raft [MPL LICENSE](https://github.com/hashicorp/raft/blob/master/LICENSE)
+- github.com/hashicorp/raft-boltdb [MPL LICENSE](https://github.com/hashicorp/raft-boltdb/blob/master/LICENSE)
+- github.com/lib/pq [MIT LICENSE](https://github.com/lib/pq/blob/master/LICENSE.md)
+- github.com/matttproud/golang_protobuf_extensions [APACHE LICENSE](https://github.com/matttproud/golang_protobuf_extensions/blob/master/LICENSE)
+- github.com/naoina/go-stringutil [MIT LICENSE](https://github.com/naoina/go-stringutil/blob/master/LICENSE)
+- github.com/naoina/toml [MIT LICENSE](https://github.com/naoina/toml/blob/master/LICENSE)
+- github.com/prometheus/client_golang [APACHE LICENSE](https://github.com/prometheus/client_golang/blob/master/LICENSE)
+- github.com/samuel/go-zookeeper [BSD LICENSE](https://github.com/samuel/go-zookeeper/blob/master/LICENSE)
+- github.com/stretchr/objx [MIT LICENSE](github.com/stretchr/objx)
+- github.com/stretchr/testify [MIT LICENSE](https://github.com/stretchr/testify/blob/master/LICENCE.txt)
+- github.com/wvanbergen/kafka [MIT LICENSE](https://github.com/wvanbergen/kafka/blob/master/LICENSE)
+- github.com/wvanbergen/kazoo-go [MIT LICENSE](https://github.com/wvanbergen/kazoo-go/blob/master/MIT-LICENSE)
+- gopkg.in/dancannon/gorethink.v1 [APACHE LICENSE](https://github.com/dancannon/gorethink/blob/v1.1.2/LICENSE)
+- gopkg.in/mgo.v2 [BSD LICENSE](https://github.com/go-mgo/mgo/blob/v2/LICENSE)
+- golang.org/x/crypto/* [BSD LICENSE](https://github.com/golang/crypto/blob/master/LICENSE)
+- internal Glob function [MIT LICENSE](https://github.com/ryanuber/go-glob/blob/master/LICENSE)
+

Makefile

@@ -0,0 +1,98 @@
+UNAME := $(shell sh -c 'uname')
+VERSION := $(shell sh -c 'git describe --always --tags')
+ifdef GOBIN
+PATH := $(GOBIN):$(PATH)
+else
+PATH := $(subst :,/bin:,$(GOPATH))/bin:$(PATH)
+endif
+
+# Standard Telegraf build
+default: prepare build
+
+# Only run the build (no dependency grabbing)
+build:
+	go build -o telegraf -ldflags \
+		"-X main.Version=$(VERSION)" \
+		./cmd/telegraf/telegraf.go
+
+# Build with race detector
+dev: prepare
+	go build -race -o telegraf -ldflags \
+		"-X main.Version=$(VERSION)" \
+		./cmd/telegraf/telegraf.go
+
+# Build linux 64-bit, 32-bit and arm architectures
+build-linux-bins: prepare
+	GOARCH=amd64 GOOS=linux go build -o telegraf_linux_amd64 \
+								-ldflags "-X main.Version=$(VERSION)" \
+								./cmd/telegraf/telegraf.go
+	GOARCH=386 GOOS=linux go build -o telegraf_linux_386 \
+								-ldflags "-X main.Version=$(VERSION)" \
+								./cmd/telegraf/telegraf.go
+	GOARCH=arm GOOS=linux go build -o telegraf_linux_arm \
+								-ldflags "-X main.Version=$(VERSION)" \
+								./cmd/telegraf/telegraf.go
+
+# Get dependencies and use gdm to checkout changesets
+prepare:
+	go get ./...
+	go get github.com/sparrc/gdm
+	gdm restore
+
+# Run all docker containers necessary for unit tests
+docker-run:
+ifeq ($(UNAME), Darwin)
+	docker run --name kafka \
+		-e ADVERTISED_HOST=$(shell sh -c 'boot2docker ip || docker-machine ip default') \
+		-e ADVERTISED_PORT=9092 \
+		-p "2181:2181" -p "9092:9092" \
+		-d spotify/kafka
+endif
+ifeq ($(UNAME), Linux)
+	docker run --name kafka \
+		-e ADVERTISED_HOST=localhost \
+		-e ADVERTISED_PORT=9092 \
+		-p "2181:2181" -p "9092:9092" \
+		-d spotify/kafka
+endif
+	docker run --name mysql -p "3306:3306" -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -d mysql
+	docker run --name memcached -p "11211:11211" -d memcached
+	docker run --name postgres -p "5432:5432" -d postgres
+	docker run --name rabbitmq -p "15672:15672" -p "5672:5672" -d rabbitmq:3-management
+	docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
+	docker run --name redis -p "6379:6379" -d redis
+	docker run --name aerospike -p "3000:3000" -d aerospike
+	docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
+	docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
+	docker run --name riemann -p "5555:5555" -d blalor/riemann
+
+# Run docker containers necessary for CircleCI unit tests
+docker-run-circle:
+	docker run --name kafka \
+		-e ADVERTISED_HOST=localhost \
+		-e ADVERTISED_PORT=9092 \
+		-p "2181:2181" -p "9092:9092" \
+		-d spotify/kafka
+	docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
+	docker run --name aerospike -p "3000:3000" -d aerospike
+	docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
+	docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
+	docker run --name riemann -p "5555:5555" -d blalor/riemann
+
+# Kill all docker containers, ignore errors
+docker-kill:
+	-docker kill nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann
+	-docker rm nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann
+
+# Run full unit tests using docker containers (includes setup and teardown)
+test: docker-kill docker-run
+	# Sleeping for kafka leadership election, TSDB setup, etc.
+	sleep 60
+	# SUCCESS, running tests
+	go test -race ./...
+
+# Run "short" unit tests
+test-short:
+	go test -short ./...
+
+.PHONY: test

README.md

@@ -1 +1,171 @@
-# tivan
+# Telegraf - A native agent for InfluxDB [![Circle CI](https://circleci.com/gh/influxdb/telegraf.svg?style=svg)](https://circleci.com/gh/influxdb/telegraf)
+
+Telegraf is an agent written in Go for collecting metrics from the system it's
+running on, or from other services, and writing them into InfluxDB.
+
+Design goals are to have a minimal memory footprint with a plugin system so
+that developers in the community can easily add support for collecting metrics
+from well known services (like Hadoop, Postgres, or Redis) and third party
+APIs (like Mailchimp, AWS CloudWatch, or Google Analytics).
+
+We'll eagerly accept pull requests for new plugins and will manage the set of
+plugins that Telegraf supports. See the
+[contributing guide](CONTRIBUTING.md) for instructions on
+writing new plugins.
+
+## Installation:
+
+### Linux deb and rpm packages:
+
+Latest:
+* http://get.influxdb.org/telegraf/telegraf_0.2.4_amd64.deb
+* http://get.influxdb.org/telegraf/telegraf-0.2.4-1.x86_64.rpm
+
+##### Package instructions:
+
+* Telegraf binary is installed in `/opt/telegraf/telegraf`
+* Telegraf daemon configuration file is in `/etc/opt/telegraf/telegraf.conf`
+* On sysv systems, the telegraf daemon can be controlled via
+`service telegraf [action]`
+* On systemd systems (such as Ubuntu 15+), the telegraf daemon can be
+controlled via `systemctl [action] telegraf`
+
+### Linux binaries:
+
+Latest:
+* http://get.influxdb.org/telegraf/telegraf_linux_amd64_0.2.4.tar.gz
+* http://get.influxdb.org/telegraf/telegraf_linux_386_0.2.4.tar.gz
+* http://get.influxdb.org/telegraf/telegraf_linux_arm_0.2.4.tar.gz
+
+##### Binary instructions:
+
+These are standalone binaries that can be unpacked and executed on any linux
+system. They can be unpacked and renamed in a location such as
+`/usr/local/bin` for convenience. A config file will need to be generated,
+see "How to use it" below.
+
+### OSX via Homebrew:
+
+```
+brew update
+brew install telegraf
+```
+
+### From Source:
+
+Telegraf manages dependencies via [gdm](https://github.com/sparrc/gdm),
+which gets installed via the Makefile
+if you don't have it already. You also must build with golang version 1.4+.
+
+1. [Install Go](https://golang.org/doc/install)
+2. [Setup your GOPATH](https://golang.org/doc/code.html#GOPATH)
+3. Run `go get github.com/influxdb/telegraf`
+4. Run `cd $GOPATH/src/github.com/influxdb/telegraf`
+5. Run `make`
+
+### How to use it:
+
+* Run `telegraf -sample-config > telegraf.conf` to create an initial configuration.
+* Or run `telegraf -sample-config -filter cpu:mem -outputfilter influxdb > telegraf.conf`.
+to create a config file with only CPU and memory plugins defined, and InfluxDB
+output defined.
+* Edit the configuration to match your needs.
+* Run `telegraf -config telegraf.conf -test` to output one full measurement
+sample to STDOUT. NOTE: you may want to run as the telegraf user if you are using
+the linux packages `sudo -u telegraf telegraf -config telegraf.conf -test`
+* Run `telegraf -config telegraf.conf` to gather and send metrics to configured outputs.
+* Run `telegraf -config telegraf.conf -filter system:swap`.
+to run telegraf with only the system & swap plugins defined in the config.
+
+## Telegraf Options
+
+Telegraf has a few options you can configure under the `agent` section of the
+config.
+
+* **hostname**: The hostname is passed as a tag. By default this will be
+the value returned by `hostname` on the machine running Telegraf.
+You can override that value here.
+* **interval**: How often to gather metrics. Uses a simple number +
+unit parser, e.g. "10s" for 10 seconds or "5m" for 5 minutes.
+* **debug**: Set to true to gather and send metrics to STDOUT as well as
+InfluxDB.
+
+## Configuration
+
+See the [configuration guide](CONFIGURATION.md) for a rundown of the more advanced
+configuration options.
+
+## Supported Plugins
+
+**You can view usage instructions for each plugin by running**
+`telegraf -usage <pluginname>`.
+
+Telegraf currently has support for collecting metrics from:
+
+* aerospike
+* apache
+* bcache
+* disque
+* elasticsearch
+* exec (generic JSON-emitting executable plugin)
+* haproxy
+* httpjson (generic JSON-emitting http service plugin)
+* influxdb
+* jolokia
+* leofs
+* lustre2
+* mailchimp
+* memcached
+* mongodb
+* mysql
+* nginx
+* phpfpm
+* ping
+* postgresql
+* procstat
+* prometheus
+* puppetagent
+* rabbitmq
+* redis
+* rethinkdb
+* twemproxy
+* zfs
+* zookeeper
+* system
+    * cpu
+    * mem
+    * net
+    * netstat
+    * disk
+    * diskio
+    * swap
+
+## Supported Service Plugins
+
+Telegraf can collect metrics via the following services:
+
+* statsd
+* kafka_consumer
+
+We'll be adding support for many more over the coming months. Read on if you
+want to add support for another service or third-party API.
+
+## Supported Outputs
+
+* influxdb
+* nsq
+* kafka
+* datadog
+* opentsdb
+* amqp (rabbitmq)
+* mqtt
+* librato
+* prometheus
+* amon
+* riemann
+
+## Contributing
+
+Please see the
+[contributing guide](CONTRIBUTING.md)
+for details on contributing a plugin or output to Telegraf.

Vagrantfile

@@ -1,122 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
-VAGRANTFILE_API_VERSION = "2"
-
-Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  # All Vagrant configuration is done here. The most common configuration
-  # options are documented and commented below. For a complete reference,
-  # please see the online documentation at vagrantup.com.
-
-  # Every Vagrant virtual environment requires a box to build off of.
-  config.vm.box = "ubuntu/trusty64"
-
-  # Disable automatic box update checking. If you disable this, then
-  # boxes will only be checked for updates when the user runs
-  # `vagrant box outdated`. This is not recommended.
-  # config.vm.box_check_update = false
-
-  # Create a forwarded port mapping which allows access to a specific port
-  # within the machine from a port on the host machine. In the example below,
-  # accessing "localhost:8080" will access port 80 on the guest machine.
-  # config.vm.network "forwarded_port", guest: 80, host: 8080
-
-  # Create a private network, which allows host-only access to the machine
-  # using a specific IP.
-  # config.vm.network "private_network", ip: "192.168.33.10"
-
-  # Create a public network, which generally matched to bridged network.
-  # Bridged networks make the machine appear as another physical device on
-  # your network.
-  # config.vm.network "public_network"
-
-  # If true, then any SSH connections made will enable agent forwarding.
-  # Default value: false
-  # config.ssh.forward_agent = true
-
-  # Share an additional folder to the guest VM. The first argument is
-  # the path on the host to the actual folder. The second argument is
-  # the path on the guest to mount the folder. And the optional third
-  # argument is a set of non-required options.
-  config.vm.synced_folder "~/go", "/home/vagrant/go"
-
-  # Provider-specific configuration so you can fine-tune various
-  # backing providers for Vagrant. These expose provider-specific options.
-  # Example for VirtualBox:
-  #
-  config.vm.provider "virtualbox" do |vb|
-  #   # Don't boot with headless mode
-  #   vb.gui = true
-  #
-  #   # Use VBoxManage to customize the VM. For example to change memory:
-    vb.customize ["modifyvm", :id, "--memory", "1024"]
-  end
-  #
-  # View the documentation for the provider you're using for more
-  # information on available options.
-
-  # Enable provisioning with CFEngine. CFEngine Community packages are
-  # automatically installed. For example, configure the host as a
-  # policy server and optionally a policy file to run:
-  #
-  # config.vm.provision "cfengine" do |cf|
-  #   cf.am_policy_hub = true
-  #   # cf.run_file = "motd.cf"
-  # end
-  #
-  # You can also configure and bootstrap a client to an existing
-  # policy server:
-  #
-  # config.vm.provision "cfengine" do |cf|
-  #   cf.policy_server_address = "10.0.2.15"
-  # end
-
-  # Enable provisioning with Puppet stand alone.  Puppet manifests
-  # are contained in a directory path relative to this Vagrantfile.
-  # You will need to create the manifests directory and a manifest in
-  # the file default.pp in the manifests_path directory.
-  #
-  # config.vm.provision "puppet" do |puppet|
-  #   puppet.manifests_path = "manifests"
-  #   puppet.manifest_file  = "site.pp"
-  # end
-
-  # Enable provisioning with chef solo, specifying a cookbooks path, roles
-  # path, and data_bags path (all relative to this Vagrantfile), and adding
-  # some recipes and/or roles.
-  #
-  # config.vm.provision "chef_solo" do |chef|
-  #   chef.cookbooks_path = "../my-recipes/cookbooks"
-  #   chef.roles_path = "../my-recipes/roles"
-  #   chef.data_bags_path = "../my-recipes/data_bags"
-  #   chef.add_recipe "mysql"
-  #   chef.add_role "web"
-  #
-  #   # You may also specify custom JSON attributes:
-  #   chef.json = { mysql_password: "foo" }
-  # end
-
-  # Enable provisioning with chef server, specifying the chef server URL,
-  # and the path to the validation key (relative to this Vagrantfile).
-  #
-  # The Opscode Platform uses HTTPS. Substitute your organization for
-  # ORGNAME in the URL and validation key.
-  #
-  # If you have your own Chef Server, use the appropriate URL, which may be
-  # HTTP instead of HTTPS depending on your configuration. Also change the
-  # validation key to validation.pem.
-  #
-  # config.vm.provision "chef_client" do |chef|
-  #   chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
-  #   chef.validation_key_path = "ORGNAME-validator.pem"
-  # end
-  #
-  # If you're using the Opscode platform, your validator client is
-  # ORGNAME-validator, replacing ORGNAME with your organization name.
-  #
-  # If you have your own Chef Server, the default validation client name is
-  # chef-validator, unless you changed the configuration.
-  #
-  #   chef.validation_client_name = "ORGNAME-validator"
-end

accumulator.go

@@ -1,37 +1,184 @@
-package tivan
+package telegraf
 
 import (
 	"fmt"
-	"sort"
-	"strings"
+	"log"
+	"math"
+	"sync"
+	"time"
 
-	"github.com/influxdb/influxdb/client"
+	"github.com/influxdb/telegraf/internal/config"
+
+	"github.com/influxdb/influxdb/client/v2"
 )
 
-type BatchPoints struct {
-	client.BatchPoints
+type Accumulator interface {
+	Add(measurement string, value interface{},
+		tags map[string]string, t ...time.Time)
+	AddFields(measurement string, fields map[string]interface{},
+		tags map[string]string, t ...time.Time)
 
-	Debug bool
+	SetDefaultTags(tags map[string]string)
+	AddDefaultTag(key, value string)
+
+	Prefix() string
+	SetPrefix(prefix string)
+
+	Debug() bool
+	SetDebug(enabled bool)
 }
 
-func (bp *BatchPoints) Add(name string, val interface{}, tags map[string]string) {
-	if bp.Debug {
-		var tg []string
+func NewAccumulator(
+	pluginConfig *config.PluginConfig,
+	points chan *client.Point,
+) Accumulator {
+	acc := accumulator{}
+	acc.points = points
+	acc.pluginConfig = pluginConfig
+	return &acc
+}
 
-		for k, v := range tags {
-			tg = append(tg, fmt.Sprintf("%s=\"%s\"", k, v))
-		}
+type accumulator struct {
+	sync.Mutex
 
-		sort.Strings(tg)
+	points chan *client.Point
 
-		fmt.Printf("> [%s] %s=%v\n", strings.Join(tg, " "), name, val)
+	defaultTags map[string]string
+
+	debug bool
+
+	pluginConfig *config.PluginConfig
+
+	prefix string
+}
+
+func (ac *accumulator) Add(
+	measurement string,
+	value interface{},
+	tags map[string]string,
+	t ...time.Time,
+) {
+	fields := make(map[string]interface{})
+	fields["value"] = value
+	ac.AddFields(measurement, fields, tags, t...)
+}
+
+func (ac *accumulator) AddFields(
+	measurement string,
+	fields map[string]interface{},
+	tags map[string]string,
+	t ...time.Time,
+) {
+	if !ac.pluginConfig.Filter.ShouldTagsPass(tags) {
+		return
 	}
 
-	bp.Points = append(bp.Points, client.Point{
-		Name: name,
-		Tags: tags,
-		Fields: map[string]interface{}{
-			"value": val,
-		},
-	})
+	// Override measurement name if set
+	if len(ac.pluginConfig.NameOverride) != 0 {
+		measurement = ac.pluginConfig.NameOverride
+	}
+	// Apply measurement prefix and suffix if set
+	if len(ac.pluginConfig.MeasurementPrefix) != 0 {
+		measurement = ac.pluginConfig.MeasurementPrefix + measurement
+	}
+	if len(ac.pluginConfig.MeasurementSuffix) != 0 {
+		measurement = measurement + ac.pluginConfig.MeasurementSuffix
+	}
+
+	if tags == nil {
+		tags = make(map[string]string)
+	}
+	// Apply plugin-wide tags if set
+	for k, v := range ac.pluginConfig.Tags {
+		if _, ok := tags[k]; !ok {
+			tags[k] = v
+		}
+	}
+	// Apply daemon-wide tags if set
+	for k, v := range ac.defaultTags {
+		if _, ok := tags[k]; !ok {
+			tags[k] = v
+		}
+	}
+
+	result := make(map[string]interface{})
+	for k, v := range fields {
+		// Filter out any filtered fields
+		if ac.pluginConfig != nil {
+			if !ac.pluginConfig.Filter.ShouldPass(k) {
+				continue
+			}
+		}
+		result[k] = v
+
+		// Validate uint64 and float64 fields
+		switch val := v.(type) {
+		case uint64:
+			// InfluxDB does not support writing uint64
+			if val < uint64(9223372036854775808) {
+				result[k] = int64(val)
+			} else {
+				result[k] = int64(9223372036854775807)
+			}
+		case float64:
+			// NaNs are invalid values in influxdb, skip measurement
+			if math.IsNaN(val) || math.IsInf(val, 0) {
+				if ac.debug {
+					log.Printf("Measurement [%s] field [%s] has a NaN or Inf "+
+						"field, skipping",
+						measurement, k)
+				}
+				continue
+			}
+		}
+	}
+	fields = nil
+	if len(result) == 0 {
+		return
+	}
+
+	var timestamp time.Time
+	if len(t) > 0 {
+		timestamp = t[0]
+	} else {
+		timestamp = time.Now()
+	}
+
+	if ac.prefix != "" {
+		measurement = ac.prefix + measurement
+	}
+
+	pt, err := client.NewPoint(measurement, tags, result, timestamp)
+	if err != nil {
+		log.Printf("Error adding point [%s]: %s\n", measurement, err.Error())
+		return
+	}
+	if ac.debug {
+		fmt.Println("> " + pt.String())
+	}
+	ac.points <- pt
+}
+
+func (ac *accumulator) SetDefaultTags(tags map[string]string) {
+	ac.defaultTags = tags
+}
+
+func (ac *accumulator) AddDefaultTag(key, value string) {
+	ac.defaultTags[key] = value
+}
+
+func (ac *accumulator) Prefix() string {
+	return ac.prefix
+}
+
+func (ac *accumulator) SetPrefix(prefix string) {
+	ac.prefix = prefix
+}
+
+func (ac *accumulator) Debug() bool {
+	return ac.debug
+}
+
+func (ac *accumulator) SetDebug(debug bool) {
+	ac.debug = debug
 }

agent.go

@@ -1,148 +1,390 @@
-package tivan
+package telegraf
 
 import (
+	"crypto/rand"
+	"fmt"
 	"log"
-	"net/url"
+	"math/big"
 	"os"
-	"sort"
+	"sync"
 	"time"
 
-	"github.com/influxdb/influxdb/client"
-	"github.com/influxdb/tivan/plugins"
+	"github.com/influxdb/telegraf/internal/config"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/influxdb/telegraf/plugins"
+
+	"github.com/influxdb/influxdb/client/v2"
 )
 
+// Agent runs telegraf and collects data based on the given config
 type Agent struct {
-	Interval Duration
-	Debug    bool
-	HTTP     string
-	Hostname string
-
-	Config *Config
-
-	plugins []plugins.Plugin
-
-	conn *client.Client
+	Config *config.Config
 }
 
-func NewAgent(config *Config) (*Agent, error) {
-	agent := &Agent{Config: config}
-
-	err := config.Apply("agent", agent)
-	if err != nil {
-		return nil, err
+// NewAgent returns an Agent struct based off the given Config
+func NewAgent(config *config.Config) (*Agent, error) {
+	a := &Agent{
+		Config: config,
 	}
 
-	if agent.Hostname == "" {
+	if a.Config.Agent.Hostname == "" {
 		hostname, err := os.Hostname()
 		if err != nil {
 			return nil, err
 		}
 
-		agent.Hostname = hostname
-
-		if config.Tags == nil {
-			config.Tags = map[string]string{}
-		}
-
-		config.Tags["host"] = agent.Hostname
+		a.Config.Agent.Hostname = hostname
 	}
 
-	return agent, nil
+	config.Tags["host"] = a.Config.Agent.Hostname
+
+	return a, nil
 }
 
-func (agent *Agent) Connect() error {
-	config := agent.Config
+// Connect connects to all configured outputs
+func (a *Agent) Connect() error {
+	for _, o := range a.Config.Outputs {
+		switch ot := o.Output.(type) {
+		case outputs.ServiceOutput:
+			if err := ot.Start(); err != nil {
+				log.Printf("Service for output %s failed to start, exiting\n%s\n",
+					o.Name, err.Error())
+				return err
+			}
+		}
 
-	u, err := url.Parse(config.URL)
-	if err != nil {
-		return err
+		if a.Config.Agent.Debug {
+			log.Printf("Attempting connection to output: %s\n", o.Name)
+		}
+		err := o.Output.Connect()
+		if err != nil {
+			log.Printf("Failed to connect to output %s, retrying in 15s\n", o.Name)
+			time.Sleep(15 * time.Second)
+			err = o.Output.Connect()
+			if err != nil {
+				return err
+			}
+		}
+		if a.Config.Agent.Debug {
+			log.Printf("Successfully connected to output: %s\n", o.Name)
+		}
 	}
-
-	c, err := client.NewClient(client.Config{
-		URL:       *u,
-		Username:  config.Username,
-		Password:  config.Password,
-		UserAgent: config.UserAgent,
-	})
-
-	if err != nil {
-		return err
-	}
-
-	agent.conn = c
-
 	return nil
 }
 
-func (a *Agent) LoadPlugins() ([]string, error) {
-	var names []string
-
-	for name, creator := range plugins.Plugins {
-		plugin := creator()
-
-		err := a.Config.Apply(name, plugin)
-		if err != nil {
-			return nil, err
-		}
-
-		a.plugins = append(a.plugins, plugin)
-		names = append(names, name)
-	}
-
-	sort.Strings(names)
-
-	return names, nil
-}
-
-func (a *Agent) crank() error {
-	var acc BatchPoints
-
-	acc.Debug = a.Debug
-
-	for _, plugin := range a.plugins {
-		err := plugin.Gather(&acc)
-		if err != nil {
-			return err
+// Close closes the connection to all configured outputs
+func (a *Agent) Close() error {
+	var err error
+	for _, o := range a.Config.Outputs {
+		err = o.Output.Close()
+		switch ot := o.Output.(type) {
+		case outputs.ServiceOutput:
+			ot.Stop()
 		}
 	}
-
-	acc.Tags = a.Config.Tags
-	acc.Timestamp = time.Now()
-	acc.Database = a.Config.Database
-
-	_, err := a.conn.Write(acc.BatchPoints)
 	return err
 }
 
-func (a *Agent) Test() error {
-	var acc BatchPoints
+// gatherParallel runs the plugins that are using the same reporting interval
+// as the telegraf agent.
+func (a *Agent) gatherParallel(pointChan chan *client.Point) error {
+	var wg sync.WaitGroup
 
-	acc.Debug = true
-
-	for _, plugin := range a.plugins {
-		err := plugin.Gather(&acc)
-		if err != nil {
-			return err
+	start := time.Now()
+	counter := 0
+	for _, plugin := range a.Config.Plugins {
+		if plugin.Config.Interval != 0 {
+			continue
 		}
+
+		wg.Add(1)
+		counter++
+		go func(plugin *config.RunningPlugin) {
+			defer wg.Done()
+
+			acc := NewAccumulator(plugin.Config, pointChan)
+			acc.SetDebug(a.Config.Agent.Debug)
+			// acc.SetPrefix(plugin.Name + "_")
+			acc.SetDefaultTags(a.Config.Tags)
+
+			if err := plugin.Plugin.Gather(acc); err != nil {
+				log.Printf("Error in plugin [%s]: %s", plugin.Name, err)
+			}
+
+		}(plugin)
 	}
 
+	if counter == 0 {
+		return nil
+	}
+
+	wg.Wait()
+
+	elapsed := time.Since(start)
+	log.Printf("Gathered metrics, (%s interval), from %d plugins in %s\n",
+		a.Config.Agent.Interval, counter, elapsed)
 	return nil
 }
 
-func (a *Agent) Run(shutdown chan struct{}) error {
-	if a.conn == nil {
-		err := a.Connect()
-		if err != nil {
-			return err
-		}
-	}
-
-	ticker := time.NewTicker(a.Interval.Duration)
+// gatherSeparate runs the plugins that have been configured with their own
+// reporting interval.
+func (a *Agent) gatherSeparate(
+	shutdown chan struct{},
+	plugin *config.RunningPlugin,
+	pointChan chan *client.Point,
+) error {
+	ticker := time.NewTicker(plugin.Config.Interval)
 
 	for {
-		err := a.crank()
-		if err != nil {
-			log.Printf("Error in plugins: %s", err)
+		var outerr error
+		start := time.Now()
+
+		acc := NewAccumulator(plugin.Config, pointChan)
+		acc.SetDebug(a.Config.Agent.Debug)
+		// acc.SetPrefix(plugin.Name + "_")
+		acc.SetDefaultTags(a.Config.Tags)
+
+		if err := plugin.Plugin.Gather(acc); err != nil {
+			log.Printf("Error in plugin [%s]: %s", plugin.Name, err)
+		}
+
+		elapsed := time.Since(start)
+		log.Printf("Gathered metrics, (separate %s interval), from %s in %s\n",
+			plugin.Config.Interval, plugin.Name, elapsed)
+
+		if outerr != nil {
+			return outerr
+		}
+
+		select {
+		case <-shutdown:
+			return nil
+		case <-ticker.C:
+			continue
+		}
+	}
+}
+
+// Test verifies that we can 'Gather' from all plugins with their configured
+// Config struct
+func (a *Agent) Test() error {
+	shutdown := make(chan struct{})
+	defer close(shutdown)
+	pointChan := make(chan *client.Point)
+
+	// dummy receiver for the point channel
+	go func() {
+		for {
+			select {
+			case <-pointChan:
+				// do nothing
+			case <-shutdown:
+				return
+			}
+		}
+	}()
+
+	for _, plugin := range a.Config.Plugins {
+		acc := NewAccumulator(plugin.Config, pointChan)
+		acc.SetDebug(true)
+		// acc.SetPrefix(plugin.Name + "_")
+
+		fmt.Printf("* Plugin: %s, Collection 1\n", plugin.Name)
+		if plugin.Config.Interval != 0 {
+			fmt.Printf("* Internal: %s\n", plugin.Config.Interval)
+		}
+
+		if err := plugin.Plugin.Gather(acc); err != nil {
+			return err
+		}
+
+		// Special instructions for some plugins. cpu, for example, needs to be
+		// run twice in order to return cpu usage percentages.
+		switch plugin.Name {
+		case "cpu", "mongodb":
+			time.Sleep(500 * time.Millisecond)
+			fmt.Printf("* Plugin: %s, Collection 2\n", plugin.Name)
+			if err := plugin.Plugin.Gather(acc); err != nil {
+				return err
+			}
+		}
+
+	}
+	return nil
+}
+
+// writeOutput writes a list of points to a single output, with retries.
+// Optionally takes a `done` channel to indicate that it is done writing.
+func (a *Agent) writeOutput(
+	points []*client.Point,
+	ro *config.RunningOutput,
+	shutdown chan struct{},
+	wg *sync.WaitGroup,
+) {
+	defer wg.Done()
+	if len(points) == 0 {
+		return
+	}
+	retry := 0
+	retries := a.Config.Agent.FlushRetries
+	start := time.Now()
+
+	for {
+		filtered := ro.FilterPoints(points)
+		err := ro.Output.Write(filtered)
+		if err == nil {
+			// Write successful
+			elapsed := time.Since(start)
+			log.Printf("Flushed %d metrics to output %s in %s\n",
+				len(filtered), ro.Name, elapsed)
+			return
+		}
+
+		select {
+		case <-shutdown:
+			return
+		default:
+			if retry >= retries {
+				// No more retries
+				msg := "FATAL: Write to output [%s] failed %d times, dropping" +
+					" %d metrics\n"
+				log.Printf(msg, ro.Name, retries+1, len(points))
+				return
+			} else if err != nil {
+				// Sleep for a retry
+				log.Printf("Error in output [%s]: %s, retrying in %s",
+					ro.Name, err.Error(), a.Config.Agent.FlushInterval.Duration)
+				time.Sleep(a.Config.Agent.FlushInterval.Duration)
+			}
+		}
+
+		retry++
+	}
+}
+
+// flush writes a list of points to all configured outputs
+func (a *Agent) flush(
+	points []*client.Point,
+	shutdown chan struct{},
+	wait bool,
+) {
+	var wg sync.WaitGroup
+	for _, o := range a.Config.Outputs {
+		wg.Add(1)
+		go a.writeOutput(points, o, shutdown, &wg)
+	}
+	if wait {
+		wg.Wait()
+	}
+}
+
+// flusher monitors the points input channel and flushes on the minimum interval
+func (a *Agent) flusher(shutdown chan struct{}, pointChan chan *client.Point) error {
+	// Inelegant, but this sleep is to allow the Gather threads to run, so that
+	// the flusher will flush after metrics are collected.
+	time.Sleep(time.Millisecond * 100)
+
+	ticker := time.NewTicker(a.Config.Agent.FlushInterval.Duration)
+	points := make([]*client.Point, 0)
+
+	for {
+		select {
+		case <-shutdown:
+			log.Println("Hang on, flushing any cached points before shutdown")
+			a.flush(points, shutdown, true)
+			return nil
+		case <-ticker.C:
+			a.flush(points, shutdown, false)
+			points = make([]*client.Point, 0)
+		case pt := <-pointChan:
+			points = append(points, pt)
+		}
+	}
+}
+
+// jitterInterval applies the the interval jitter to the flush interval using
+// crypto/rand number generator
+func jitterInterval(ininterval, injitter time.Duration) time.Duration {
+	var jitter int64
+	outinterval := ininterval
+	if injitter.Nanoseconds() != 0 {
+		maxjitter := big.NewInt(injitter.Nanoseconds())
+		if j, err := rand.Int(rand.Reader, maxjitter); err == nil {
+			jitter = j.Int64()
+		}
+		outinterval = time.Duration(jitter + ininterval.Nanoseconds())
+	}
+
+	if outinterval.Nanoseconds() < time.Duration(500*time.Millisecond).Nanoseconds() {
+		log.Printf("Flush interval %s too low, setting to 500ms\n", outinterval)
+		outinterval = time.Duration(500 * time.Millisecond)
+	}
+
+	return outinterval
+}
+
+// Run runs the agent daemon, gathering every Interval
+func (a *Agent) Run(shutdown chan struct{}) error {
+	var wg sync.WaitGroup
+
+	a.Config.Agent.FlushInterval.Duration = jitterInterval(a.Config.Agent.FlushInterval.Duration,
+		a.Config.Agent.FlushJitter.Duration)
+
+	log.Printf("Agent Config: Interval:%s, Debug:%#v, Hostname:%#v, "+
+		"Flush Interval:%s\n",
+		a.Config.Agent.Interval, a.Config.Agent.Debug,
+		a.Config.Agent.Hostname, a.Config.Agent.FlushInterval)
+
+	// channel shared between all plugin threads for accumulating points
+	pointChan := make(chan *client.Point, 1000)
+
+	// Round collection to nearest interval by sleeping
+	if a.Config.Agent.RoundInterval {
+		i := int64(a.Config.Agent.Interval.Duration)
+		time.Sleep(time.Duration(i - (time.Now().UnixNano() % i)))
+	}
+	ticker := time.NewTicker(a.Config.Agent.Interval.Duration)
+
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		if err := a.flusher(shutdown, pointChan); err != nil {
+			log.Printf("Flusher routine failed, exiting: %s\n", err.Error())
+			close(shutdown)
+		}
+	}()
+
+	for _, plugin := range a.Config.Plugins {
+
+		// Start service of any ServicePlugins
+		switch p := plugin.Plugin.(type) {
+		case plugins.ServicePlugin:
+			if err := p.Start(); err != nil {
+				log.Printf("Service for plugin %s failed to start, exiting\n%s\n",
+					plugin.Name, err.Error())
+				return err
+			}
+			defer p.Stop()
+		}
+
+		// Special handling for plugins that have their own collection interval
+		// configured. Default intervals are handled below with gatherParallel
+		if plugin.Config.Interval != 0 {
+			wg.Add(1)
+			go func(plugin *config.RunningPlugin) {
+				defer wg.Done()
+				if err := a.gatherSeparate(shutdown, plugin, pointChan); err != nil {
+					log.Printf(err.Error())
+				}
+			}(plugin)
+		}
+	}
+
+	defer wg.Wait()
+
+	for {
+		if err := a.gatherParallel(pointChan); err != nil {
+			log.Printf(err.Error())
 		}
 
 		select {

agent_test.go

@@ -1,61 +1,156 @@
-package tivan
+package telegraf
 
-/*
-func TestAgent_DrivesMetrics(t *testing.T) {
-	var (
-		plugin  plugins.MockPlugin
-	)
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+	"time"
 
-	defer plugin.AssertExpectations(t)
-	defer metrics.AssertExpectations(t)
+	"github.com/influxdb/telegraf/internal/config"
 
-	a := &Agent{
-		plugins: []plugins.Plugin{&plugin},
-		Config:  &Config{},
-	}
+	// needing to load the plugins
+	_ "github.com/influxdb/telegraf/plugins/all"
+	// needing to load the outputs
+	_ "github.com/influxdb/telegraf/outputs/all"
+)
 
-	plugin.On("Add", "foo", 1.2, nil).Return(nil)
-	plugin.On("Add", "bar", 888, nil).Return(nil)
+func TestAgent_LoadPlugin(t *testing.T) {
+	c := config.NewConfig()
+	c.PluginFilters = []string{"mysql"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ := NewAgent(c)
+	assert.Equal(t, 1, len(a.Config.Plugins))
 
-	err := a.crank()
-	require.NoError(t, err)
+	c = config.NewConfig()
+	c.PluginFilters = []string{"foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 0, len(a.Config.Plugins))
+
+	c = config.NewConfig()
+	c.PluginFilters = []string{"mysql", "foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 1, len(a.Config.Plugins))
+
+	c = config.NewConfig()
+	c.PluginFilters = []string{"mysql", "redis"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Plugins))
+
+	c = config.NewConfig()
+	c.PluginFilters = []string{"mysql", "foo", "redis", "bar"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Plugins))
 }
 
-func TestAgent_AppliesTags(t *testing.T) {
-	var (
-		plugin  plugins.MockPlugin
-		metrics MockMetrics
-	)
+func TestAgent_LoadOutput(t *testing.T) {
+	c := config.NewConfig()
+	c.OutputFilters = []string{"influxdb"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ := NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Outputs))
 
-	defer plugin.AssertExpectations(t)
-	defer metrics.AssertExpectations(t)
+	c = config.NewConfig()
+	c.OutputFilters = []string{}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 3, len(a.Config.Outputs))
 
-	a := &Agent{
-		plugins: []plugins.Plugin{&plugin},
-		metrics: &metrics,
-		Config: &Config{
-			Tags: map[string]string{
-				"dc": "us-west-1",
-			},
-		},
-	}
+	c = config.NewConfig()
+	c.OutputFilters = []string{"foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 0, len(a.Config.Outputs))
 
-	m1 := cypress.Metric()
-	m1.Add("name", "foo")
-	m1.Add("value", 1.2)
+	c = config.NewConfig()
+	c.OutputFilters = []string{"influxdb", "foo"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 2, len(a.Config.Outputs))
 
-	msgs := []*cypress.Message{m1}
+	c = config.NewConfig()
+	c.OutputFilters = []string{"influxdb", "kafka"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 3, len(a.Config.Outputs))
 
-	m2 := cypress.Metric()
-	m2.Timestamp = m1.Timestamp
-	m2.Add("name", "foo")
-	m2.Add("value", 1.2)
-	m2.AddTag("dc", "us-west-1")
-
-	plugin.On("Read").Return(msgs, nil)
-	metrics.On("Receive", m2).Return(nil)
-
-	err := a.crank()
-	require.NoError(t, err)
+	c = config.NewConfig()
+	c.OutputFilters = []string{"influxdb", "foo", "kafka", "bar"}
+	c.LoadConfig("./internal/config/testdata/telegraf-agent.toml")
+	a, _ = NewAgent(c)
+	assert.Equal(t, 3, len(a.Config.Outputs))
+}
+
+func TestAgent_ZeroJitter(t *testing.T) {
+	flushinterval := jitterInterval(time.Duration(10*time.Second),
+		time.Duration(0*time.Second))
+
+	actual := flushinterval.Nanoseconds()
+	exp := time.Duration(10 * time.Second).Nanoseconds()
+
+	if actual != exp {
+		t.Errorf("Actual %v, expected %v", actual, exp)
+	}
+}
+
+func TestAgent_ZeroInterval(t *testing.T) {
+	min := time.Duration(500 * time.Millisecond).Nanoseconds()
+	max := time.Duration(5 * time.Second).Nanoseconds()
+
+	for i := 0; i < 1000; i++ {
+		flushinterval := jitterInterval(time.Duration(0*time.Second),
+			time.Duration(5*time.Second))
+		actual := flushinterval.Nanoseconds()
+
+		if actual > max {
+			t.Errorf("Didn't expect interval %d to be > %d", actual, max)
+			break
+		}
+		if actual < min {
+			t.Errorf("Didn't expect interval %d to be < %d", actual, min)
+			break
+		}
+	}
+}
+
+func TestAgent_ZeroBoth(t *testing.T) {
+	flushinterval := jitterInterval(time.Duration(0*time.Second),
+		time.Duration(0*time.Second))
+
+	actual := flushinterval
+	exp := time.Duration(500 * time.Millisecond)
+
+	if actual != exp {
+		t.Errorf("Actual %v, expected %v", actual, exp)
+	}
+}
+
+func TestAgent_JitterMax(t *testing.T) {
+	max := time.Duration(32 * time.Second).Nanoseconds()
+
+	for i := 0; i < 1000; i++ {
+		flushinterval := jitterInterval(time.Duration(30*time.Second),
+			time.Duration(2*time.Second))
+		actual := flushinterval.Nanoseconds()
+		if actual > max {
+			t.Errorf("Didn't expect interval %d to be > %d", actual, max)
+			break
+		}
+	}
+}
+
+func TestAgent_JitterMin(t *testing.T) {
+	min := time.Duration(30 * time.Second).Nanoseconds()
+
+	for i := 0; i < 1000; i++ {
+		flushinterval := jitterInterval(time.Duration(30*time.Second),
+			time.Duration(2*time.Second))
+		actual := flushinterval.Nanoseconds()
+		if actual < min {
+			t.Errorf("Didn't expect interval %d to be < %d", actual, min)
+			break
+		}
+	}
 }
-*/

circle.yml

@@ -0,0 +1,20 @@
+machine:
+  services:
+    - docker
+  post:
+    - sudo service zookeeper stop
+    - go version
+    - go version | grep 1.5.1 || sudo rm -rf /usr/local/go
+    - wget https://storage.googleapis.com/golang/go1.5.1.linux-amd64.tar.gz
+    - sudo tar -C /usr/local -xzf go1.5.1.linux-amd64.tar.gz
+    - go version
+
+dependencies:
+  cache_directories:
+    - "~/telegraf-build/src"
+  override:
+    - docker info
+
+test:
+  override:
+    - bash scripts/circle-test.sh

cmd/telegraf/telegraf.go

@@ -0,0 +1,152 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+	"os"
+	"os/signal"
+	"strings"
+
+	"github.com/influxdb/telegraf"
+	"github.com/influxdb/telegraf/internal/config"
+	_ "github.com/influxdb/telegraf/outputs/all"
+	_ "github.com/influxdb/telegraf/plugins/all"
+)
+
+var fDebug = flag.Bool("debug", false,
+	"show metrics as they're generated to stdout")
+var fTest = flag.Bool("test", false, "gather metrics, print them out, and exit")
+var fConfig = flag.String("config", "", "configuration file to load")
+var fConfigDirectory = flag.String("configdirectory", "",
+	"directory containing additional *.conf files")
+var fVersion = flag.Bool("version", false, "display the version")
+var fSampleConfig = flag.Bool("sample-config", false,
+	"print out full sample configuration")
+var fPidfile = flag.String("pidfile", "", "file to write our pid to")
+var fPLuginFilters = flag.String("filter", "",
+	"filter the plugins to enable, separator is :")
+var fOutputFilters = flag.String("outputfilter", "",
+	"filter the outputs to enable, separator is :")
+var fUsage = flag.String("usage", "",
+	"print usage for a plugin, ie, 'telegraf -usage mysql'")
+
+// Telegraf version
+//	-ldflags "-X main.Version=`git describe --always --tags`"
+var Version string
+
+func main() {
+	flag.Parse()
+
+	var pluginFilters []string
+	if *fPLuginFilters != "" {
+		pluginsFilter := strings.TrimSpace(*fPLuginFilters)
+		pluginFilters = strings.Split(":"+pluginsFilter+":", ":")
+	}
+
+	var outputFilters []string
+	if *fOutputFilters != "" {
+		outputFilter := strings.TrimSpace(*fOutputFilters)
+		outputFilters = strings.Split(":"+outputFilter+":", ":")
+	}
+
+	if *fVersion {
+		v := fmt.Sprintf("Telegraf - Version %s", Version)
+		fmt.Println(v)
+		return
+	}
+
+	if *fSampleConfig {
+		config.PrintSampleConfig(pluginFilters, outputFilters)
+		return
+	}
+
+	if *fUsage != "" {
+		if err := config.PrintPluginConfig(*fUsage); err != nil {
+			if err2 := config.PrintOutputConfig(*fUsage); err2 != nil {
+				log.Fatalf("%s and %s", err, err2)
+			}
+		}
+		return
+	}
+
+	var (
+		c   *config.Config
+		err error
+	)
+
+	if *fConfig != "" {
+		c = config.NewConfig()
+		c.OutputFilters = outputFilters
+		c.PluginFilters = pluginFilters
+		err = c.LoadConfig(*fConfig)
+		if err != nil {
+			log.Fatal(err)
+		}
+	} else {
+		fmt.Println("Usage: Telegraf")
+		flag.PrintDefaults()
+		return
+	}
+
+	if *fConfigDirectory != "" {
+		err = c.LoadDirectory(*fConfigDirectory)
+		if err != nil {
+			log.Fatal(err)
+		}
+	}
+	if len(c.Outputs) == 0 {
+		log.Fatalf("Error: no outputs found, did you provide a valid config file?")
+	}
+	if len(c.Plugins) == 0 {
+		log.Fatalf("Error: no plugins found, did you provide a valid config file?")
+	}
+
+	ag, err := telegraf.NewAgent(c)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	if *fDebug {
+		ag.Config.Agent.Debug = true
+	}
+
+	if *fTest {
+		err = ag.Test()
+		if err != nil {
+			log.Fatal(err)
+		}
+		return
+	}
+
+	err = ag.Connect()
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	shutdown := make(chan struct{})
+	signals := make(chan os.Signal)
+	signal.Notify(signals, os.Interrupt)
+	go func() {
+		<-signals
+		close(shutdown)
+	}()
+
+	log.Printf("Starting Telegraf (version %s)\n", Version)
+	log.Printf("Loaded outputs: %s", strings.Join(c.OutputNames(), " "))
+	log.Printf("Loaded plugins: %s", strings.Join(c.PluginNames(), " "))
+	log.Printf("Tags enabled: %s", c.ListTags())
+
+	if *fPidfile != "" {
+		f, err := os.Create(*fPidfile)
+		if err != nil {
+			log.Fatalf("Unable to create pidfile: %s", err)
+		}
+
+		fmt.Fprintf(f, "%d\n", os.Getpid())
+
+		f.Close()
+	}
+
+	ag.Run(shutdown)
+}

cmd/tivan/tivan.go

@@ -1,96 +0,0 @@
-package main
-
-import (
-	"flag"
-	"fmt"
-	"log"
-	"os"
-	"os/signal"
-	"strings"
-
-	"github.com/influxdb/tivan"
-	_ "github.com/influxdb/tivan/plugins/all"
-)
-
-var fDebug = flag.Bool("debug", false, "show metrics as they're generated to stdout")
-var fTest = flag.Bool("test", false, "gather metrics, print them out, and exit")
-var fConfig = flag.String("config", "", "configuration file to load")
-var fVersion = flag.Bool("version", false, "display the version")
-
-var Version = "unreleased"
-
-func main() {
-	flag.Parse()
-
-	if *fVersion {
-		fmt.Printf("InfluxDB Tivan agent - Version %s\n", Version)
-		return
-	}
-
-	var (
-		config *tivan.Config
-		err    error
-	)
-
-	if *fConfig != "" {
-		config, err = tivan.LoadConfig(*fConfig)
-		if err != nil {
-			log.Fatal(err)
-		}
-	} else {
-		config = tivan.DefaultConfig()
-	}
-
-	ag, err := tivan.NewAgent(config)
-	if err != nil {
-		log.Fatal(err)
-	}
-
-	if *fDebug {
-		ag.Debug = true
-	}
-
-	plugins, err := ag.LoadPlugins()
-	if err != nil {
-		log.Fatal(err)
-	}
-
-	if *fTest {
-		err = ag.Test()
-		if err != nil {
-			log.Fatal(err)
-		}
-
-		return
-	}
-
-	err = ag.Connect()
-	if err != nil {
-		log.Fatal(err)
-	}
-
-	shutdown := make(chan struct{})
-
-	signals := make(chan os.Signal)
-
-	signal.Notify(signals, os.Interrupt)
-
-	go func() {
-		<-signals
-		close(shutdown)
-	}()
-
-	log.Print("InfluxDB Agent running")
-	log.Printf("Loaded plugins: %s", strings.Join(plugins, " "))
-	if ag.Debug {
-		log.Printf("Debug: enabled")
-		log.Printf("Agent Config: %#v", ag)
-	}
-
-	if config.URL != "" {
-		log.Printf("Sending metrics to: %s", config.URL)
-		log.Printf("Tags enabled: %v", config.ListTags())
-	}
-
-	ag.Run(shutdown)
-}

config.go

@@ -1,103 +0,0 @@
-package tivan
-
-import (
-	"errors"
-	"fmt"
-	"io/ioutil"
-	"sort"
-	"strings"
-	"time"
-
-	"github.com/naoina/toml"
-	"github.com/naoina/toml/ast"
-)
-
-type Duration struct {
-	time.Duration
-}
-
-func (d *Duration) UnmarshalTOML(b []byte) error {
-	dur, err := time.ParseDuration(string(b[1 : len(b)-1]))
-	if err != nil {
-		return err
-	}
-
-	d.Duration = dur
-
-	return nil
-}
-
-type Config struct {
-	URL       string
-	Username  string
-	Password  string
-	Database  string
-	UserAgent string
-	Tags      map[string]string
-
-	plugins map[string]*ast.Table
-}
-
-func (c *Config) Plugins() map[string]*ast.Table {
-	return c.plugins
-}
-
-func (c *Config) Apply(name string, v interface{}) error {
-	if tbl, ok := c.plugins[name]; ok {
-		return toml.UnmarshalTable(tbl, v)
-	}
-
-	return nil
-}
-
-func DefaultConfig() *Config {
-	return &Config{}
-}
-
-var ErrInvalidConfig = errors.New("invalid configuration")
-
-func LoadConfig(path string) (*Config, error) {
-	data, err := ioutil.ReadFile(path)
-	if err != nil {
-		return nil, err
-	}
-
-	tbl, err := toml.Parse(data)
-	if err != nil {
-		return nil, err
-	}
-
-	c := &Config{
-		plugins: make(map[string]*ast.Table),
-	}
-
-	for name, val := range tbl.Fields {
-		subtbl, ok := val.(*ast.Table)
-		if !ok {
-			return nil, ErrInvalidConfig
-		}
-
-		if name == "influxdb" {
-			err := toml.UnmarshalTable(subtbl, c)
-			if err != nil {
-				return nil, err
-			}
-		} else {
-			c.plugins[name] = subtbl
-		}
-	}
-
-	return c, nil
-}
-
-func (c *Config) ListTags() string {
-	var tags []string
-
-	for k, v := range c.Tags {
-		tags = append(tags, fmt.Sprintf("%s=%s", k, v))
-	}
-
-	sort.Strings(tags)
-
-	return strings.Join(tags, " ")
-}

etc/logrotate.d/telegraf

@@ -0,0 +1,11 @@
+/var/log/telegraf/telegraf.log 
+{
+    rotate 6
+    daily
+    missingok
+    dateext
+    copytruncate
+    notifempty
+    compress
+}
+

etc/telegraf.conf

@@ -0,0 +1,123 @@
+# Telegraf configuration
+
+# Telegraf is entirely plugin driven. All metrics are gathered from the
+# declared plugins.
+
+# Even if a plugin has no configuration, it must be declared in here
+# to be active. Declaring a plugin means just specifying the name
+# as a section with no variables. To deactivate a plugin, comment
+# out the name and any variables.
+
+# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
+# file would generate.
+
+# One rule that plugins conform to is wherever a connection string
+# can be passed, the values '' and 'localhost' are treated specially.
+# They indicate to the plugin to use their own builtin configuration to
+# connect to the local system.
+
+# NOTE: The configuration has a few required parameters. They are marked
+# with 'required'. Be sure to edit those to make this configuration work.
+
+# Tags can also be specified via a normal map, but only one form at a time:
+[tags]
+  # dc = "us-east-1"
+
+# Configuration for telegraf agent
+[agent]
+  # Default data collection interval for all plugins
+  interval = "10s"
+  # Rounds collection interval to 'interval'
+  # ie, if interval="10s" then always collect on :00, :10, :20, etc.
+  round_interval = true
+
+  # Default data flushing interval for all outputs. You should not set this below
+  # interval. Maximum flush_interval will be flush_interval + flush_jitter
+  flush_interval = "10s"
+  # Jitter the flush interval by a random amount. This is primarily to avoid
+  # large write spikes for users running a large number of telegraf instances.
+  # ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
+  flush_jitter = "0s"
+
+  # Run telegraf in debug mode
+  debug = false
+  # Override default hostname, if empty use os.Hostname()
+  hostname = ""
+
+
+###############################################################################
+#                                  OUTPUTS                                    #
+###############################################################################
+
+[outputs]
+
+# Configuration for influxdb server to send metrics to
+[[outputs.influxdb]]
+  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
+  # Multiple urls can be specified but it is assumed that they are part of the same
+  # cluster, this means that only ONE of the urls will be written to each interval.
+  # urls = ["udp://localhost:8089"] # UDP endpoint example
+  urls = ["http://localhost:8086"] # required
+  # The target database for metrics (telegraf will create it if not exists)
+  database = "telegraf" # required
+  # Precision of writes, valid values are n, u, ms, s, m, and h
+  # note: using second precision greatly helps InfluxDB compression
+  precision = "s"
+
+  # Connection timeout (for the connection with InfluxDB), formatted as a string.
+  # If not provided, will default to 0 (no timeout)
+  # timeout = "5s"
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
+  # user_agent = "telegraf"
+  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
+  # udp_payload = 512
+
+
+###############################################################################
+#                                  PLUGINS                                    #
+###############################################################################
+
+[plugins]
+
+# Read metrics about cpu usage
+[[plugins.cpu]]
+  # Whether to report per-cpu stats or not
+  percpu = true
+  # Whether to report total system cpu stats or not
+  totalcpu = true
+  # Comment this line if you want the raw CPU time metrics
+  drop = ["cpu_time"]
+
+# Read metrics about disk usage by mount point
+[[plugins.disk]]
+  # By default, telegraf gather stats for all mountpoints.
+  # Setting mountpoints will restrict the stats to the specified mountpoints.
+  # Mountpoints=["/"]
+
+# Read metrics about disk IO by device
+[[plugins.diskio]]
+  # By default, telegraf will gather stats for all devices including
+  # disk partitions.
+  # Setting devices will restrict the stats to the specified devcies.
+  # Devices=["sda","sdb"]
+  # Uncomment the following line if you do not need disk serial numbers.
+  # SkipSerialNumber = true
+
+# Read metrics about memory usage
+[[plugins.mem]]
+  # no configuration
+
+# Read metrics about swap memory usage
+[[plugins.swap]]
+  # no configuration
+
+# Read metrics about system load & uptime
+[[plugins.system]]
+  # no configuration
+
+
+###############################################################################
+#                              SERVICE PLUGINS                                #
+###############################################################################

internal/config/config.go

@@ -0,0 +1,709 @@
+package config
+
+import (
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"path/filepath"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/influxdb/telegraf/plugins"
+
+	"github.com/naoina/toml"
+	"github.com/naoina/toml/ast"
+
+	"github.com/influxdb/influxdb/client/v2"
+)
+
+// Config specifies the URL/user/password for the database that telegraf
+// will be logging to, as well as all the plugins that the user has
+// specified
+type Config struct {
+	Tags          map[string]string
+	PluginFilters []string
+	OutputFilters []string
+
+	Agent   *AgentConfig
+	Plugins []*RunningPlugin
+	Outputs []*RunningOutput
+}
+
+func NewConfig() *Config {
+	c := &Config{
+		// Agent defaults:
+		Agent: &AgentConfig{
+			Interval:      internal.Duration{Duration: 10 * time.Second},
+			RoundInterval: true,
+			FlushInterval: internal.Duration{Duration: 10 * time.Second},
+			FlushRetries:  2,
+			FlushJitter:   internal.Duration{Duration: 5 * time.Second},
+		},
+
+		Tags:          make(map[string]string),
+		Plugins:       make([]*RunningPlugin, 0),
+		Outputs:       make([]*RunningOutput, 0),
+		PluginFilters: make([]string, 0),
+		OutputFilters: make([]string, 0),
+	}
+	return c
+}
+
+type AgentConfig struct {
+	// Interval at which to gather information
+	Interval internal.Duration
+
+	// RoundInterval rounds collection interval to 'interval'.
+	//     ie, if Interval=10s then always collect on :00, :10, :20, etc.
+	RoundInterval bool
+
+	// Interval at which to flush data
+	FlushInterval internal.Duration
+
+	// FlushRetries is the number of times to retry each data flush
+	FlushRetries int
+
+	// FlushJitter tells
+	FlushJitter internal.Duration
+
+	// TODO(cam): Remove UTC and Precision parameters, they are no longer
+	// valid for the agent config. Leaving them here for now for backwards-
+	// compatability
+	UTC       bool `toml:"utc"`
+	Precision string
+
+	// Option for running in debug mode
+	Debug    bool
+	Hostname string
+}
+
+// TagFilter is the name of a tag, and the values on which to filter
+type TagFilter struct {
+	Name   string
+	Filter []string
+}
+
+type RunningOutput struct {
+	Name   string
+	Output outputs.Output
+	Config *OutputConfig
+}
+
+type RunningPlugin struct {
+	Name   string
+	Plugin plugins.Plugin
+	Config *PluginConfig
+}
+
+// Filter containing drop/pass and tagdrop/tagpass rules
+type Filter struct {
+	Drop []string
+	Pass []string
+
+	TagDrop []TagFilter
+	TagPass []TagFilter
+
+	IsActive bool
+}
+
+// PluginConfig containing a name, interval, and filter
+type PluginConfig struct {
+	Name              string
+	NameOverride      string
+	MeasurementPrefix string
+	MeasurementSuffix string
+	Tags              map[string]string
+	Filter            Filter
+	Interval          time.Duration
+}
+
+// OutputConfig containing name and filter
+type OutputConfig struct {
+	Name   string
+	Filter Filter
+}
+
+// Filter returns filtered slice of client.Points based on whether filters
+// are active for this RunningOutput.
+func (ro *RunningOutput) FilterPoints(points []*client.Point) []*client.Point {
+	if !ro.Config.Filter.IsActive {
+		return points
+	}
+
+	var filteredPoints []*client.Point
+	for i := range points {
+		if !ro.Config.Filter.ShouldPass(points[i].Name()) || !ro.Config.Filter.ShouldTagsPass(points[i].Tags()) {
+			continue
+		}
+		filteredPoints = append(filteredPoints, points[i])
+	}
+	return filteredPoints
+}
+
+// ShouldPass returns true if the metric should pass, false if should drop
+// based on the drop/pass filter parameters
+func (f Filter) ShouldPass(fieldkey string) bool {
+	if f.Pass != nil {
+		for _, pat := range f.Pass {
+			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			// Cam, 2015-12-07
+			if strings.HasPrefix(fieldkey, pat) || internal.Glob(pat, fieldkey) {
+				return true
+			}
+		}
+		return false
+	}
+
+	if f.Drop != nil {
+		for _, pat := range f.Drop {
+			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			// Cam, 2015-12-07
+			if strings.HasPrefix(fieldkey, pat) || internal.Glob(pat, fieldkey) {
+				return false
+			}
+		}
+
+		return true
+	}
+	return true
+}
+
+// ShouldTagsPass returns true if the metric should pass, false if should drop
+// based on the tagdrop/tagpass filter parameters
+func (f Filter) ShouldTagsPass(tags map[string]string) bool {
+	if f.TagPass != nil {
+		for _, pat := range f.TagPass {
+			if tagval, ok := tags[pat.Name]; ok {
+				for _, filter := range pat.Filter {
+					if internal.Glob(filter, tagval) {
+						return true
+					}
+				}
+			}
+		}
+		return false
+	}
+
+	if f.TagDrop != nil {
+		for _, pat := range f.TagDrop {
+			if tagval, ok := tags[pat.Name]; ok {
+				for _, filter := range pat.Filter {
+					if internal.Glob(filter, tagval) {
+						return false
+					}
+				}
+			}
+		}
+		return true
+	}
+
+	return true
+}
+
+// Plugins returns a list of strings of the configured plugins.
+func (c *Config) PluginNames() []string {
+	var name []string
+	for _, plugin := range c.Plugins {
+		name = append(name, plugin.Name)
+	}
+	return name
+}
+
+// Outputs returns a list of strings of the configured plugins.
+func (c *Config) OutputNames() []string {
+	var name []string
+	for _, output := range c.Outputs {
+		name = append(name, output.Name)
+	}
+	return name
+}
+
+// ListTags returns a string of tags specified in the config,
+// line-protocol style
+func (c *Config) ListTags() string {
+	var tags []string
+
+	for k, v := range c.Tags {
+		tags = append(tags, fmt.Sprintf("%s=%s", k, v))
+	}
+
+	sort.Strings(tags)
+
+	return strings.Join(tags, " ")
+}
+
+var header = `# Telegraf configuration
+
+# Telegraf is entirely plugin driven. All metrics are gathered from the
+# declared plugins.
+
+# Even if a plugin has no configuration, it must be declared in here
+# to be active. Declaring a plugin means just specifying the name
+# as a section with no variables. To deactivate a plugin, comment
+# out the name and any variables.
+
+# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
+# file would generate.
+
+# One rule that plugins conform to is wherever a connection string
+# can be passed, the values '' and 'localhost' are treated specially.
+# They indicate to the plugin to use their own builtin configuration to
+# connect to the local system.
+
+# NOTE: The configuration has a few required parameters. They are marked
+# with 'required'. Be sure to edit those to make this configuration work.
+
+# Tags can also be specified via a normal map, but only one form at a time:
+[tags]
+  # dc = "us-east-1"
+
+# Configuration for telegraf agent
+[agent]
+  # Default data collection interval for all plugins
+  interval = "10s"
+  # Rounds collection interval to 'interval'
+  # ie, if interval="10s" then always collect on :00, :10, :20, etc.
+  round_interval = true
+
+  # Default data flushing interval for all outputs. You should not set this below
+  # interval. Maximum flush_interval will be flush_interval + flush_jitter
+  flush_interval = "10s"
+  # Jitter the flush interval by a random amount. This is primarily to avoid
+  # large write spikes for users running a large number of telegraf instances.
+  # ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
+  flush_jitter = "0s"
+
+  # Run telegraf in debug mode
+  debug = false
+  # Override default hostname, if empty use os.Hostname()
+  hostname = ""
+
+
+###############################################################################
+#                                  OUTPUTS                                    #
+###############################################################################
+
+[outputs]
+`
+
+var pluginHeader = `
+
+###############################################################################
+#                                  PLUGINS                                    #
+###############################################################################
+
+[plugins]
+`
+
+var servicePluginHeader = `
+
+###############################################################################
+#                              SERVICE PLUGINS                                #
+###############################################################################
+`
+
+// PrintSampleConfig prints the sample config
+func PrintSampleConfig(pluginFilters []string, outputFilters []string) {
+	fmt.Printf(header)
+
+	// Filter outputs
+	var onames []string
+	for oname := range outputs.Outputs {
+		if len(outputFilters) == 0 || sliceContains(oname, outputFilters) {
+			onames = append(onames, oname)
+		}
+	}
+	sort.Strings(onames)
+
+	// Print Outputs
+	for _, oname := range onames {
+		creator := outputs.Outputs[oname]
+		output := creator()
+		printConfig(oname, output, "outputs")
+	}
+
+	// Filter plugins
+	var pnames []string
+	for pname := range plugins.Plugins {
+		if len(pluginFilters) == 0 || sliceContains(pname, pluginFilters) {
+			pnames = append(pnames, pname)
+		}
+	}
+	sort.Strings(pnames)
+
+	// Print Plugins
+	fmt.Printf(pluginHeader)
+	servPlugins := make(map[string]plugins.ServicePlugin)
+	for _, pname := range pnames {
+		creator := plugins.Plugins[pname]
+		plugin := creator()
+
+		switch p := plugin.(type) {
+		case plugins.ServicePlugin:
+			servPlugins[pname] = p
+			continue
+		}
+
+		printConfig(pname, plugin, "plugins")
+	}
+
+	// Print Service Plugins
+	fmt.Printf(servicePluginHeader)
+	for name, plugin := range servPlugins {
+		printConfig(name, plugin, "plugins")
+	}
+}
+
+type printer interface {
+	Description() string
+	SampleConfig() string
+}
+
+func printConfig(name string, p printer, op string) {
+	fmt.Printf("\n# %s\n[[%s.%s]]", p.Description(), op, name)
+	config := p.SampleConfig()
+	if config == "" {
+		fmt.Printf("\n  # no configuration\n")
+	} else {
+		fmt.Printf(config)
+	}
+}
+
+func sliceContains(name string, list []string) bool {
+	for _, b := range list {
+		if b == name {
+			return true
+		}
+	}
+	return false
+}
+
+// PrintPluginConfig prints the config usage of a single plugin.
+func PrintPluginConfig(name string) error {
+	if creator, ok := plugins.Plugins[name]; ok {
+		printConfig(name, creator(), "plugins")
+	} else {
+		return errors.New(fmt.Sprintf("Plugin %s not found", name))
+	}
+	return nil
+}
+
+// PrintOutputConfig prints the config usage of a single output.
+func PrintOutputConfig(name string) error {
+	if creator, ok := outputs.Outputs[name]; ok {
+		printConfig(name, creator(), "outputs")
+	} else {
+		return errors.New(fmt.Sprintf("Output %s not found", name))
+	}
+	return nil
+}
+
+func (c *Config) LoadDirectory(path string) error {
+	directoryEntries, err := ioutil.ReadDir(path)
+	if err != nil {
+		return err
+	}
+	for _, entry := range directoryEntries {
+		if entry.IsDir() {
+			continue
+		}
+		name := entry.Name()
+		if len(name) < 6 || name[len(name)-5:] != ".conf" {
+			continue
+		}
+		err := c.LoadConfig(filepath.Join(path, name))
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// LoadConfig loads the given config file and applies it to c
+func (c *Config) LoadConfig(path string) error {
+	data, err := ioutil.ReadFile(path)
+	if err != nil {
+		return err
+	}
+
+	tbl, err := toml.Parse(data)
+	if err != nil {
+		return err
+	}
+
+	for name, val := range tbl.Fields {
+		subTable, ok := val.(*ast.Table)
+		if !ok {
+			return errors.New("invalid configuration")
+		}
+
+		switch name {
+		case "agent":
+			if err = toml.UnmarshalTable(subTable, c.Agent); err != nil {
+				log.Printf("Could not parse [agent] config\n")
+				return err
+			}
+		case "tags":
+			if err = toml.UnmarshalTable(subTable, c.Tags); err != nil {
+				log.Printf("Could not parse [tags] config\n")
+				return err
+			}
+		case "outputs":
+			for outputName, outputVal := range subTable.Fields {
+				switch outputSubTable := outputVal.(type) {
+				case *ast.Table:
+					if err = c.addOutput(outputName, outputSubTable); err != nil {
+						return err
+					}
+				case []*ast.Table:
+					for _, t := range outputSubTable {
+						if err = c.addOutput(outputName, t); err != nil {
+							return err
+						}
+					}
+				default:
+					return fmt.Errorf("Unsupported config format: %s",
+						outputName)
+				}
+			}
+		case "plugins":
+			for pluginName, pluginVal := range subTable.Fields {
+				switch pluginSubTable := pluginVal.(type) {
+				case *ast.Table:
+					if err = c.addPlugin(pluginName, pluginSubTable); err != nil {
+						return err
+					}
+				case []*ast.Table:
+					for _, t := range pluginSubTable {
+						if err = c.addPlugin(pluginName, t); err != nil {
+							return err
+						}
+					}
+				default:
+					return fmt.Errorf("Unsupported config format: %s",
+						pluginName)
+				}
+			}
+		// Assume it's a plugin for legacy config file support if no other
+		// identifiers are present
+		default:
+			if err = c.addPlugin(name, subTable); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func (c *Config) addOutput(name string, table *ast.Table) error {
+	if len(c.OutputFilters) > 0 && !sliceContains(name, c.OutputFilters) {
+		return nil
+	}
+	creator, ok := outputs.Outputs[name]
+	if !ok {
+		return fmt.Errorf("Undefined but requested output: %s", name)
+	}
+	output := creator()
+
+	outputConfig, err := buildOutput(name, table)
+	if err != nil {
+		return err
+	}
+
+	if err := toml.UnmarshalTable(table, output); err != nil {
+		return err
+	}
+
+	ro := &RunningOutput{
+		Name:   name,
+		Output: output,
+		Config: outputConfig,
+	}
+	c.Outputs = append(c.Outputs, ro)
+	return nil
+}
+
+func (c *Config) addPlugin(name string, table *ast.Table) error {
+	if len(c.PluginFilters) > 0 && !sliceContains(name, c.PluginFilters) {
+		return nil
+	}
+	// Legacy support renaming io plugin to diskio
+	if name == "io" {
+		name = "diskio"
+	}
+
+	creator, ok := plugins.Plugins[name]
+	if !ok {
+		return fmt.Errorf("Undefined but requested plugin: %s", name)
+	}
+	plugin := creator()
+
+	pluginConfig, err := buildPlugin(name, table)
+	if err != nil {
+		return err
+	}
+
+	if err := toml.UnmarshalTable(table, plugin); err != nil {
+		return err
+	}
+
+	rp := &RunningPlugin{
+		Name:   name,
+		Plugin: plugin,
+		Config: pluginConfig,
+	}
+	c.Plugins = append(c.Plugins, rp)
+	return nil
+}
+
+// buildFilter builds a Filter (tagpass/tagdrop/pass/drop) to
+// be inserted into the OutputConfig/PluginConfig to be used for prefix
+// filtering on tags and measurements
+func buildFilter(tbl *ast.Table) Filter {
+	f := Filter{}
+
+	if node, ok := tbl.Fields["pass"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if ary, ok := kv.Value.(*ast.Array); ok {
+				for _, elem := range ary.Value {
+					if str, ok := elem.(*ast.String); ok {
+						f.Pass = append(f.Pass, str.Value)
+						f.IsActive = true
+					}
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["drop"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if ary, ok := kv.Value.(*ast.Array); ok {
+				for _, elem := range ary.Value {
+					if str, ok := elem.(*ast.String); ok {
+						f.Drop = append(f.Drop, str.Value)
+						f.IsActive = true
+					}
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["tagpass"]; ok {
+		if subtbl, ok := node.(*ast.Table); ok {
+			for name, val := range subtbl.Fields {
+				if kv, ok := val.(*ast.KeyValue); ok {
+					tagfilter := &TagFilter{Name: name}
+					if ary, ok := kv.Value.(*ast.Array); ok {
+						for _, elem := range ary.Value {
+							if str, ok := elem.(*ast.String); ok {
+								tagfilter.Filter = append(tagfilter.Filter, str.Value)
+							}
+						}
+					}
+					f.TagPass = append(f.TagPass, *tagfilter)
+					f.IsActive = true
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["tagdrop"]; ok {
+		if subtbl, ok := node.(*ast.Table); ok {
+			for name, val := range subtbl.Fields {
+				if kv, ok := val.(*ast.KeyValue); ok {
+					tagfilter := &TagFilter{Name: name}
+					if ary, ok := kv.Value.(*ast.Array); ok {
+						for _, elem := range ary.Value {
+							if str, ok := elem.(*ast.String); ok {
+								tagfilter.Filter = append(tagfilter.Filter, str.Value)
+							}
+						}
+					}
+					f.TagDrop = append(f.TagDrop, *tagfilter)
+					f.IsActive = true
+				}
+			}
+		}
+	}
+
+	delete(tbl.Fields, "drop")
+	delete(tbl.Fields, "pass")
+	delete(tbl.Fields, "tagdrop")
+	delete(tbl.Fields, "tagpass")
+	return f
+}
+
+// buildPlugin parses plugin specific items from the ast.Table,
+// builds the filter and returns a
+// PluginConfig to be inserted into RunningPlugin
+func buildPlugin(name string, tbl *ast.Table) (*PluginConfig, error) {
+	cp := &PluginConfig{Name: name}
+	if node, ok := tbl.Fields["interval"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				dur, err := time.ParseDuration(str.Value)
+				if err != nil {
+					return nil, err
+				}
+
+				cp.Interval = dur
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["name_prefix"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				cp.MeasurementPrefix = str.Value
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["name_suffix"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				cp.MeasurementSuffix = str.Value
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["name_override"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				cp.NameOverride = str.Value
+			}
+		}
+	}
+
+	cp.Tags = make(map[string]string)
+	if node, ok := tbl.Fields["tags"]; ok {
+		if subtbl, ok := node.(*ast.Table); ok {
+			if err := toml.UnmarshalTable(subtbl, cp.Tags); err != nil {
+				log.Printf("Could not parse tags for plugin %s\n", name)
+			}
+		}
+	}
+
+	delete(tbl.Fields, "name_prefix")
+	delete(tbl.Fields, "name_suffix")
+	delete(tbl.Fields, "name_override")
+	delete(tbl.Fields, "interval")
+	delete(tbl.Fields, "tags")
+	cp.Filter = buildFilter(tbl)
+	return cp, nil
+}
+
+// buildOutput parses output specific items from the ast.Table, builds the filter and returns an
+// OutputConfig to be inserted into RunningPlugin
+// Note: error exists in the return for future calls that might require error
+func buildOutput(name string, tbl *ast.Table) (*OutputConfig, error) {
+	oc := &OutputConfig{
+		Name:   name,
+		Filter: buildFilter(tbl),
+	}
+	return oc, nil
+}

internal/config/config_test.go

@@ -0,0 +1,296 @@
+package config
+
+import (
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/plugins"
+	"github.com/influxdb/telegraf/plugins/exec"
+	"github.com/influxdb/telegraf/plugins/memcached"
+	"github.com/influxdb/telegraf/plugins/procstat"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestConfig_LoadSinglePlugin(t *testing.T) {
+	c := NewConfig()
+	c.LoadConfig("./testdata/single_plugin.toml")
+
+	memcached := plugins.Plugins["memcached"]().(*memcached.Memcached)
+	memcached.Servers = []string{"localhost"}
+
+	mConfig := &PluginConfig{
+		Name: "memcached",
+		Filter: Filter{
+			Drop: []string{"other", "stuff"},
+			Pass: []string{"some", "strings"},
+			TagDrop: []TagFilter{
+				TagFilter{
+					Name:   "badtag",
+					Filter: []string{"othertag"},
+				},
+			},
+			TagPass: []TagFilter{
+				TagFilter{
+					Name:   "goodtag",
+					Filter: []string{"mytag"},
+				},
+			},
+			IsActive: true,
+		},
+		Interval: 5 * time.Second,
+	}
+
+	assert.Equal(t, memcached, c.Plugins[0].Plugin,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Plugins[0].Config,
+		"Testdata did not produce correct memcached metadata.")
+}
+
+func TestConfig_LoadDirectory(t *testing.T) {
+	c := NewConfig()
+	err := c.LoadConfig("./testdata/single_plugin.toml")
+	if err != nil {
+		t.Error(err)
+	}
+	err = c.LoadDirectory("./testdata/subconfig")
+	if err != nil {
+		t.Error(err)
+	}
+
+	memcached := plugins.Plugins["memcached"]().(*memcached.Memcached)
+	memcached.Servers = []string{"localhost"}
+
+	mConfig := &PluginConfig{
+		Name: "memcached",
+		Filter: Filter{
+			Drop: []string{"other", "stuff"},
+			Pass: []string{"some", "strings"},
+			TagDrop: []TagFilter{
+				TagFilter{
+					Name:   "badtag",
+					Filter: []string{"othertag"},
+				},
+			},
+			TagPass: []TagFilter{
+				TagFilter{
+					Name:   "goodtag",
+					Filter: []string{"mytag"},
+				},
+			},
+			IsActive: true,
+		},
+		Interval: 5 * time.Second,
+	}
+	assert.Equal(t, memcached, c.Plugins[0].Plugin,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Plugins[0].Config,
+		"Testdata did not produce correct memcached metadata.")
+
+	ex := plugins.Plugins["exec"]().(*exec.Exec)
+	ex.Commands = []*exec.Command{
+		&exec.Command{
+			Command: "/usr/bin/myothercollector --foo=bar",
+			Name:    "myothercollector",
+		},
+	}
+	eConfig := &PluginConfig{Name: "exec"}
+	assert.Equal(t, ex, c.Plugins[1].Plugin,
+		"Merged Testdata did not produce a correct exec struct.")
+	assert.Equal(t, eConfig, c.Plugins[1].Config,
+		"Merged Testdata did not produce correct exec metadata.")
+
+	memcached.Servers = []string{"192.168.1.1"}
+	assert.Equal(t, memcached, c.Plugins[2].Plugin,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Plugins[2].Config,
+		"Testdata did not produce correct memcached metadata.")
+
+	pstat := plugins.Plugins["procstat"]().(*procstat.Procstat)
+	pstat.Specifications = []*procstat.Specification{
+		&procstat.Specification{
+			PidFile: "/var/run/grafana-server.pid",
+		},
+		&procstat.Specification{
+			PidFile: "/var/run/influxdb/influxd.pid",
+		},
+	}
+
+	pConfig := &PluginConfig{Name: "procstat"}
+
+	assert.Equal(t, pstat, c.Plugins[3].Plugin,
+		"Merged Testdata did not produce a correct procstat struct.")
+	assert.Equal(t, pConfig, c.Plugins[3].Config,
+		"Merged Testdata did not produce correct procstat metadata.")
+}
+
+func TestFilter_Empty(t *testing.T) {
+	f := Filter{}
+
+	measurements := []string{
+		"foo",
+		"bar",
+		"barfoo",
+		"foo_bar",
+		"foo.bar",
+		"foo-bar",
+		"supercalifradjulisticexpialidocious",
+	}
+
+	for _, measurement := range measurements {
+		if !f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to pass", measurement)
+		}
+	}
+}
+
+func TestFilter_Pass(t *testing.T) {
+	f := Filter{
+		Pass: []string{"foo*", "cpu_usage_idle"},
+	}
+
+	passes := []string{
+		"foo",
+		"foo_bar",
+		"foo.bar",
+		"foo-bar",
+		"cpu_usage_idle",
+	}
+
+	drops := []string{
+		"bar",
+		"barfoo",
+		"bar_foo",
+		"cpu_usage_busy",
+	}
+
+	for _, measurement := range passes {
+		if !f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to pass", measurement)
+		}
+	}
+
+	for _, measurement := range drops {
+		if f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to drop", measurement)
+		}
+	}
+}
+
+func TestFilter_Drop(t *testing.T) {
+	f := Filter{
+		Drop: []string{"foo*", "cpu_usage_idle"},
+	}
+
+	drops := []string{
+		"foo",
+		"foo_bar",
+		"foo.bar",
+		"foo-bar",
+		"cpu_usage_idle",
+	}
+
+	passes := []string{
+		"bar",
+		"barfoo",
+		"bar_foo",
+		"cpu_usage_busy",
+	}
+
+	for _, measurement := range passes {
+		if !f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to pass", measurement)
+		}
+	}
+
+	for _, measurement := range drops {
+		if f.ShouldPass(measurement) {
+			t.Errorf("Expected measurement %s to drop", measurement)
+		}
+	}
+}
+
+func TestFilter_TagPass(t *testing.T) {
+	filters := []TagFilter{
+		TagFilter{
+			Name:   "cpu",
+			Filter: []string{"cpu-*"},
+		},
+		TagFilter{
+			Name:   "mem",
+			Filter: []string{"mem_free"},
+		}}
+	f := Filter{
+		TagPass: filters,
+	}
+
+	passes := []map[string]string{
+		{"cpu": "cpu-total"},
+		{"cpu": "cpu-0"},
+		{"cpu": "cpu-1"},
+		{"cpu": "cpu-2"},
+		{"mem": "mem_free"},
+	}
+
+	drops := []map[string]string{
+		{"cpu": "cputotal"},
+		{"cpu": "cpu0"},
+		{"cpu": "cpu1"},
+		{"cpu": "cpu2"},
+		{"mem": "mem_used"},
+	}
+
+	for _, tags := range passes {
+		if !f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to pass", tags)
+		}
+	}
+
+	for _, tags := range drops {
+		if f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to drop", tags)
+		}
+	}
+}
+
+func TestFilter_TagDrop(t *testing.T) {
+	filters := []TagFilter{
+		TagFilter{
+			Name:   "cpu",
+			Filter: []string{"cpu-*"},
+		},
+		TagFilter{
+			Name:   "mem",
+			Filter: []string{"mem_free"},
+		}}
+	f := Filter{
+		TagDrop: filters,
+	}
+
+	drops := []map[string]string{
+		{"cpu": "cpu-total"},
+		{"cpu": "cpu-0"},
+		{"cpu": "cpu-1"},
+		{"cpu": "cpu-2"},
+		{"mem": "mem_free"},
+	}
+
+	passes := []map[string]string{
+		{"cpu": "cputotal"},
+		{"cpu": "cpu0"},
+		{"cpu": "cpu1"},
+		{"cpu": "cpu2"},
+		{"mem": "mem_used"},
+	}
+
+	for _, tags := range passes {
+		if !f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to pass", tags)
+		}
+	}
+
+	for _, tags := range drops {
+		if f.ShouldTagsPass(tags) {
+			t.Errorf("Expected tags %v to drop", tags)
+		}
+	}
+}

internal/config/testdata/single_plugin.toml

@@ -0,0 +1,9 @@
+[[plugins.memcached]]
+  servers = ["localhost"]
+  pass = ["some", "strings"]
+  drop = ["other", "stuff"]
+  interval = "5s"
+  [plugins.memcached.tagpass]
+    goodtag = ["mytag"]
+  [plugins.memcached.tagdrop]
+    badtag = ["othertag"]

internal/config/testdata/subconfig/exec.conf

@@ -0,0 +1,8 @@
+[[plugins.exec]]
+  # specify commands via an array of tables
+  [[plugins.exec.commands]]
+  # the command to run
+  command = "/usr/bin/myothercollector --foo=bar"
+
+  # name of the command (used as a prefix for measurements)
+  name = "myothercollector"

internal/config/testdata/subconfig/memcached.conf

@@ -0,0 +1,9 @@
+[[plugins.memcached]]
+  servers = ["192.168.1.1"]
+  pass = ["some", "strings"]
+  drop = ["other", "stuff"]
+  interval = "5s"
+  [plugins.memcached.tagpass]
+    goodtag = ["mytag"]
+  [plugins.memcached.tagdrop]
+    badtag = ["othertag"]

internal/config/testdata/subconfig/procstat.conf

@@ -0,0 +1,5 @@
+[[plugins.procstat]]
+  [[plugins.procstat.specifications]]
+  pid_file = "/var/run/grafana-server.pid"
+  [[plugins.procstat.specifications]]
+  pid_file = "/var/run/influxdb/influxd.pid"

internal/config/testdata/telegraf-agent.toml

@@ -0,0 +1,334 @@
+# Telegraf configuration
+
+# Telegraf is entirely plugin driven. All metrics are gathered from the
+# declared plugins.
+
+# Even if a plugin has no configuration, it must be declared in here
+# to be active. Declaring a plugin means just specifying the name
+# as a section with no variables. To deactivate a plugin, comment
+# out the name and any variables.
+
+# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
+# file would generate.
+
+# One rule that plugins conform to is wherever a connection string
+# can be passed, the values '' and 'localhost' are treated specially.
+# They indicate to the plugin to use their own builtin configuration to
+# connect to the local system.
+
+# NOTE: The configuration has a few required parameters. They are marked
+# with 'required'. Be sure to edit those to make this configuration work.
+
+# Tags can also be specified via a normal map, but only one form at a time:
+[tags]
+  # dc = "us-east-1"
+
+# Configuration for telegraf agent
+[agent]
+  # Default data collection interval for all plugins
+  interval = "10s"
+
+  # If utc = false, uses local time (utc is highly recommended)
+  utc = true
+
+  # Precision of writes, valid values are n, u, ms, s, m, and h
+  # note: using second precision greatly helps InfluxDB compression
+  precision = "s"
+
+  # run telegraf in debug mode
+  debug = false
+
+  # Override default hostname, if empty use os.Hostname()
+  hostname = ""
+
+
+###############################################################################
+#                                  OUTPUTS                                    #
+###############################################################################
+
+[outputs]
+
+# Configuration for influxdb server to send metrics to
+[[outputs.influxdb]]
+  # The full HTTP endpoint URL for your InfluxDB instance
+  # Multiple urls can be specified for InfluxDB cluster support. Server to
+  # write to will be randomly chosen each interval.
+  urls = ["http://localhost:8086"] # required.
+
+  # The target database for metrics. This database must already exist
+  database = "telegraf" # required.
+
+  # Connection timeout (for the connection with InfluxDB), formatted as a string.
+  # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+  # If not provided, will default to 0 (no timeout)
+  # timeout = "5s"
+
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+
+  # Set the user agent for the POSTs (can be useful for log differentiation)
+  # user_agent = "telegraf"
+
+[[outputs.influxdb]]
+  urls = ["udp://localhost:8089"]
+  database = "udp-telegraf"
+
+# Configuration for the Kafka server to send metrics to
+[[outputs.kafka]]
+  # URLs of kafka brokers
+  brokers = ["localhost:9092"]
+  # Kafka topic for producer messages
+  topic = "telegraf"
+  # Telegraf tag to use as a routing key
+  #  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+
+
+###############################################################################
+#                                  PLUGINS                                    #
+###############################################################################
+
+[plugins]
+
+# Read Apache status information (mod_status)
+[[plugins.apache]]
+# An array of Apache status URI to gather stats.
+urls = ["http://localhost/server-status?auto"]
+
+# Read metrics about cpu usage
+[[plugins.cpu]]
+  # Whether to report per-cpu stats or not
+  percpu = true
+  # Whether to report total system cpu stats or not
+  totalcpu = true
+  # Comment this line if you want the raw CPU time metrics
+  drop = ["cpu_time"]
+
+# Read metrics about disk usage by mount point
+[[plugins.diskio]]
+  # no configuration
+
+# Read metrics from one or many disque servers
+[[plugins.disque]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port and password. ie disque://localhost, disque://10.10.3.33:18832,
+  # 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read stats from one or more Elasticsearch servers or clusters
+[[plugins.elasticsearch]]
+  # specify a list of one or more Elasticsearch servers
+  servers = ["http://localhost:9200"]
+
+  # set local to false when you want to read the indices stats from all nodes
+  # within the cluster
+  local = true
+
+# Read flattened metrics from one or more commands that output JSON to stdout
+[[plugins.exec]]
+  # specify commands via an array of tables
+  [[exec.commands]]
+  # the command to run
+  command = "/usr/bin/mycollector --foo=bar"
+
+  # name of the command (used as a prefix for measurements)
+  name = "mycollector"
+
+# Read metrics of haproxy, via socket or csv stats page
+[[plugins.haproxy]]
+  # An array of address to gather stats about. Specify an ip on hostname
+  # with optional port. ie localhost, 10.10.3.33:1936, etc.
+  #
+  # If no servers are specified, then default to 127.0.0.1:1936
+  servers = ["http://myhaproxy.com:1936", "http://anotherhaproxy.com:1936"]
+  # Or you can also use local socket(not work yet)
+  # servers = ["socket:/run/haproxy/admin.sock"]
+
+# Read flattened metrics from one or more JSON HTTP endpoints
+[[plugins.httpjson]]
+  # Specify services via an array of tables
+  [[httpjson.services]]
+
+    # a name for the service being polled
+    name = "webserver_stats"
+
+    # URL of each server in the service's cluster
+    servers = [
+      "http://localhost:9999/stats/",
+      "http://localhost:9998/stats/",
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "GET"
+
+    # HTTP parameters (all values must be strings)
+    [httpjson.services.parameters]
+      event_type = "cpu_spike"
+      threshold = "0.75"
+
+# Read metrics about disk IO by device
+[[plugins.io]]
+  # no configuration
+
+# read metrics from a Kafka topic
+[[plugins.kafka_consumer]]
+  # topic(s) to consume
+  topics = ["telegraf"]
+  # an array of Zookeeper connection strings
+  zookeeper_peers = ["localhost:2181"]
+  # the name of the consumer group
+  consumer_group = "telegraf_metrics_consumers"
+  # Maximum number of points to buffer between collection intervals
+  point_buffer = 100000
+  # Offset (must be either "oldest" or "newest")
+  offset = "oldest"
+
+# Read metrics from a LeoFS Server via SNMP
+[[plugins.leofs]]
+  # An array of URI to gather stats about LeoFS.
+  # Specify an ip or hostname with port. ie 127.0.0.1:4020
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 4020 as the port.
+  servers = ["127.0.0.1:4021"]
+
+# Read metrics from local Lustre service on OST, MDS
+[[plugins.lustre2]]
+  # An array of /proc globs to search for Lustre stats
+  # If not specified, the default will work on Lustre 2.5.x
+  #
+  # ost_procfiles = ["/proc/fs/lustre/obdfilter/*/stats", "/proc/fs/lustre/osd-ldiskfs/*/stats"]
+  # mds_procfiles = ["/proc/fs/lustre/mdt/*/md_stats"]
+
+# Read metrics about memory usage
+[[plugins.mem]]
+  # no configuration
+
+# Read metrics from one or many memcached servers
+[[plugins.memcached]]
+  # An array of address to gather stats about. Specify an ip on hostname
+  # with optional port. ie localhost, 10.0.0.1:11211, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read metrics from one or many MongoDB servers
+[[plugins.mongodb]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port add password. ie mongodb://user:auth_key@10.10.3.30:27017,
+  # mongodb://10.10.3.33:18832, 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 27107 as the port.
+  servers = ["127.0.0.1:27017"]
+
+# Read metrics from one or many mysql servers
+[[plugins.mysql]]
+  # specify servers via a url matching:
+  #  [username[:password]@][protocol[(address)]]/[?tls=[true|false|skip-verify]]
+  #  e.g.
+  #    servers = ["root:root@http://10.0.0.18/?tls=false"]
+  #    servers = ["root:passwd@tcp(127.0.0.1:3306)/"]
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read metrics about network interface usage
+[[plugins.net]]
+  # By default, telegraf gathers stats from any up interface (excluding loopback)
+  # Setting interfaces will tell it to gather these explicit interfaces,
+  # regardless of status.
+  #
+  # interfaces = ["eth0", ... ]
+
+# Read Nginx's basic status information (ngx_http_stub_status_module)
+[[plugins.nginx]]
+  # An array of Nginx stub_status URI to gather stats.
+  urls = ["http://localhost/status"]
+
+# Ping given url(s) and return statistics
+[[plugins.ping]]
+  # urls to ping
+  urls = ["www.google.com"] # required
+  # number of pings to send (ping -c <COUNT>)
+  count = 1 # required
+  # interval, in s, at which to ping. 0 == default (ping -i <PING_INTERVAL>)
+  ping_interval = 0.0
+  # ping timeout, in s. 0 == no timeout (ping -t <TIMEOUT>)
+  timeout = 0.0
+  # interface to send ping from (ping -I <INTERFACE>)
+  interface = ""
+
+# Read metrics from one or many postgresql servers
+[[plugins.postgresql]]
+  # specify servers via an array of tables
+  [[postgresql.servers]]
+
+  # specify address via a url matching:
+  #   postgres://[pqgotest[:password]]@localhost[/dbname]?sslmode=[disable|verify-ca|verify-full]
+  # or a simple string:
+  #   host=localhost user=pqotest password=... sslmode=... dbname=app_production
+  #
+  # All connection parameters are optional. By default, the host is localhost
+  # and the user is the currently running user. For localhost, we default
+  # to sslmode=disable as well.
+  #
+  # Without the dbname parameter, the driver will default to a database
+  # with the same name as the user. This dbname is just for instantiating a
+  # connection with the server and doesn't restrict the databases we are trying
+  # to grab metrics for.
+  #
+
+  address = "sslmode=disable"
+
+  # A list of databases to pull metrics about. If not specified, metrics for all
+  # databases are gathered.
+
+  # databases = ["app_production", "blah_testing"]
+
+  # [[postgresql.servers]]
+  # address = "influx@remoteserver"
+
+# Read metrics from one or many prometheus clients
+[[plugins.prometheus]]
+  # An array of urls to scrape metrics from.
+  urls = ["http://localhost:9100/metrics"]
+
+# Read metrics from one or many RabbitMQ servers via the management API
+[[plugins.rabbitmq]]
+  # Specify servers via an array of tables
+  [[rabbitmq.servers]]
+  # name = "rmq-server-1" # optional tag
+  # url = "http://localhost:15672"
+  # username = "guest"
+  # password = "guest"
+
+  # A list of nodes to pull metrics about. If not specified, metrics for
+  # all nodes are gathered.
+  # nodes = ["rabbit@node1", "rabbit@node2"]
+
+# Read metrics from one or many redis servers
+[[plugins.redis]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port add password. ie redis://localhost, redis://10.10.3.33:18832,
+  # 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+
+# Read metrics from one or many RethinkDB servers
+[[plugins.rethinkdb]]
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port add password. ie rethinkdb://user:auth_key@10.10.3.30:28105,
+  # rethinkdb://10.10.3.33:18832, 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 28015 as the port.
+  servers = ["127.0.0.1:28015"]
+
+# Read metrics about swap memory usage
+[[plugins.swap]]
+  # no configuration
+
+# Read metrics about system load & uptime
+[[plugins.system]]
+  # no configuration

internal/internal.go

@@ -0,0 +1,153 @@
+package internal
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+	"time"
+)
+
+// Duration just wraps time.Duration
+type Duration struct {
+	Duration time.Duration
+}
+
+// UnmarshalTOML parses the duration from the TOML config file
+func (d *Duration) UnmarshalTOML(b []byte) error {
+	dur, err := time.ParseDuration(string(b[1 : len(b)-1]))
+	if err != nil {
+		return err
+	}
+
+	d.Duration = dur
+
+	return nil
+}
+
+var NotImplementedError = errors.New("not implemented yet")
+
+type JSONFlattener struct {
+	Fields map[string]interface{}
+}
+
+// FlattenJSON flattens nested maps/interfaces into a fields map
+func (f *JSONFlattener) FlattenJSON(
+	fieldname string,
+	v interface{},
+) error {
+	if f.Fields == nil {
+		f.Fields = make(map[string]interface{})
+	}
+	fieldname = strings.Trim(fieldname, "_")
+	switch t := v.(type) {
+	case map[string]interface{}:
+		for k, v := range t {
+			err := f.FlattenJSON(fieldname+"_"+k+"_", v)
+			if err != nil {
+				return err
+			}
+		}
+	case float64:
+		f.Fields[fieldname] = t
+	case bool, string, []interface{}:
+		// ignored types
+		return nil
+	default:
+		return fmt.Errorf("JSON Flattener: got unexpected type %T with value %v (%s)",
+			t, t, fieldname)
+	}
+	return nil
+}
+
+// ReadLines reads contents from a file and splits them by new lines.
+// A convenience wrapper to ReadLinesOffsetN(filename, 0, -1).
+func ReadLines(filename string) ([]string, error) {
+	return ReadLinesOffsetN(filename, 0, -1)
+}
+
+// ReadLines reads contents from file and splits them by new line.
+// The offset tells at which line number to start.
+// The count determines the number of lines to read (starting from offset):
+//   n >= 0: at most n lines
+//   n < 0: whole file
+func ReadLinesOffsetN(filename string, offset uint, n int) ([]string, error) {
+	f, err := os.Open(filename)
+	if err != nil {
+		return []string{""}, err
+	}
+	defer f.Close()
+
+	var ret []string
+
+	r := bufio.NewReader(f)
+	for i := 0; i < n+int(offset) || n < 0; i++ {
+		line, err := r.ReadString('\n')
+		if err != nil {
+			break
+		}
+		if i < int(offset) {
+			continue
+		}
+		ret = append(ret, strings.Trim(line, "\n"))
+	}
+
+	return ret, nil
+}
+
+// Glob will test a string pattern, potentially containing globs, against a
+// subject string. The result is a simple true/false, determining whether or
+// not the glob pattern matched the subject text.
+//
+// Adapted from https://github.com/ryanuber/go-glob/blob/master/glob.go
+// thanks Ryan Uber!
+func Glob(pattern, measurement string) bool {
+	// Empty pattern can only match empty subject
+	if pattern == "" {
+		return measurement == pattern
+	}
+
+	// If the pattern _is_ a glob, it matches everything
+	if pattern == "*" {
+		return true
+	}
+
+	parts := strings.Split(pattern, "*")
+
+	if len(parts) == 1 {
+		// No globs in pattern, so test for match
+		return pattern == measurement
+	}
+
+	leadingGlob := strings.HasPrefix(pattern, "*")
+	trailingGlob := strings.HasSuffix(pattern, "*")
+	end := len(parts) - 1
+
+	for i, part := range parts {
+		switch i {
+		case 0:
+			if leadingGlob {
+				continue
+			}
+			if !strings.HasPrefix(measurement, part) {
+				return false
+			}
+		case end:
+			if len(measurement) > 0 {
+				return trailingGlob || strings.HasSuffix(measurement, part)
+			}
+		default:
+			if !strings.Contains(measurement, part) {
+				return false
+			}
+		}
+
+		// Trim evaluated text from measurement as we loop over the pattern.
+		idx := strings.Index(measurement, part) + len(part)
+		measurement = measurement[idx:]
+	}
+
+	// All parts of the pattern matched
+	return true
+}

internal/internal_test.go

@@ -0,0 +1,44 @@
+package internal
+
+import "testing"
+
+func testGlobMatch(t *testing.T, pattern, subj string) {
+	if !Glob(pattern, subj) {
+		t.Errorf("%s should match %s", pattern, subj)
+	}
+}
+
+func testGlobNoMatch(t *testing.T, pattern, subj string) {
+	if Glob(pattern, subj) {
+		t.Errorf("%s should not match %s", pattern, subj)
+	}
+}
+
+func TestEmptyPattern(t *testing.T) {
+	testGlobMatch(t, "", "")
+	testGlobNoMatch(t, "", "test")
+}
+
+func TestPatternWithoutGlobs(t *testing.T) {
+	testGlobMatch(t, "test", "test")
+}
+
+func TestGlob(t *testing.T) {
+	for _, pattern := range []string{
+		"*test",           // Leading glob
+		"this*",           // Trailing glob
+		"*is*a*",          // Lots of globs
+		"**test**",        // Double glob characters
+		"**is**a***test*", // Varying number of globs
+	} {
+		testGlobMatch(t, pattern, "this_is_a_test")
+	}
+
+	for _, pattern := range []string{
+		"test*", // Implicit substring match should fail
+		"*is",   // Partial match should fail
+		"*no*",  // Globs without a match between them should fail
+	} {
+		testGlobNoMatch(t, pattern, "this_is_a_test")
+	}
+}

outputs/all/all.go

@@ -0,0 +1,16 @@
+package all
+
+import (
+	_ "github.com/influxdb/telegraf/outputs/amon"
+	_ "github.com/influxdb/telegraf/outputs/amqp"
+	_ "github.com/influxdb/telegraf/outputs/datadog"
+	_ "github.com/influxdb/telegraf/outputs/influxdb"
+	_ "github.com/influxdb/telegraf/outputs/kafka"
+	_ "github.com/influxdb/telegraf/outputs/kinesis"
+	_ "github.com/influxdb/telegraf/outputs/librato"
+	_ "github.com/influxdb/telegraf/outputs/mqtt"
+	_ "github.com/influxdb/telegraf/outputs/nsq"
+	_ "github.com/influxdb/telegraf/outputs/opentsdb"
+	_ "github.com/influxdb/telegraf/outputs/prometheus_client"
+	_ "github.com/influxdb/telegraf/outputs/riemann"
+)

outputs/amon/README.md

@@ -0,0 +1,9 @@
+# Amon Output Plugin
+
+This plugin writes to [Amon](https://www.amon.cx)
+and requires an `serverkey` and `amoninstance` URL which can be obtained [here](https://www.amon.cx/docs/monitoring/)
+for the account.
+
+If the point value being sent cannot be converted to a float64, the metric is skipped.
+
+Metrics are grouped by converting any `_` characters to `.` in the Point Name.

outputs/amon/amon.go

@@ -0,0 +1,157 @@
+package amon
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+	"strings"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Amon struct {
+	ServerKey    string
+	AmonInstance string
+	Timeout      internal.Duration
+
+	client *http.Client
+}
+
+var sampleConfig = `
+  # Amon Server Key
+  server_key = "my-server-key" # required.
+
+  # Amon Instance URL
+  amon_instance = "https://youramoninstance" # required
+
+  # Connection timeout.
+  # timeout = "5s"
+`
+
+type TimeSeries struct {
+	Series []*Metric `json:"series"`
+}
+
+type Metric struct {
+	Metric string   `json:"metric"`
+	Points [1]Point `json:"points"`
+}
+
+type Point [2]float64
+
+func (a *Amon) Connect() error {
+	if a.ServerKey == "" || a.AmonInstance == "" {
+		return fmt.Errorf("serverkey and amon_instance are required fields for amon output")
+	}
+	a.client = &http.Client{
+		Timeout: a.Timeout.Duration,
+	}
+	return nil
+}
+
+func (a *Amon) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	ts := TimeSeries{}
+	tempSeries := []*Metric{}
+	metricCounter := 0
+
+	for _, pt := range points {
+		mname := strings.Replace(pt.Name(), "_", ".", -1)
+		if amonPts, err := buildPoints(pt); err == nil {
+			for fieldName, amonPt := range amonPts {
+				metric := &Metric{
+					Metric: mname + "_" + strings.Replace(fieldName, "_", ".", -1),
+				}
+				metric.Points[0] = amonPt
+				tempSeries = append(tempSeries, metric)
+				metricCounter++
+			}
+		} else {
+			log.Printf("unable to build Metric for %s, skipping\n", pt.Name())
+		}
+	}
+
+	ts.Series = make([]*Metric, metricCounter)
+	copy(ts.Series, tempSeries[0:])
+	tsBytes, err := json.Marshal(ts)
+	if err != nil {
+		return fmt.Errorf("unable to marshal TimeSeries, %s\n", err.Error())
+	}
+	req, err := http.NewRequest("POST", a.authenticatedUrl(), bytes.NewBuffer(tsBytes))
+	if err != nil {
+		return fmt.Errorf("unable to create http.Request, %s\n", err.Error())
+	}
+	req.Header.Add("Content-Type", "application/json")
+
+	resp, err := a.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("error POSTing metrics, %s\n", err.Error())
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode < 200 || resp.StatusCode > 209 {
+		return fmt.Errorf("received bad status code, %d\n", resp.StatusCode)
+	}
+
+	return nil
+}
+
+func (a *Amon) SampleConfig() string {
+	return sampleConfig
+}
+
+func (a *Amon) Description() string {
+	return "Configuration for Amon Server to send metrics to."
+}
+
+func (a *Amon) authenticatedUrl() string {
+
+	return fmt.Sprintf("%s/api/system/%s", a.AmonInstance, a.ServerKey)
+}
+
+func buildPoints(pt *client.Point) (map[string]Point, error) {
+	pts := make(map[string]Point)
+	for k, v := range pt.Fields() {
+		var p Point
+		if err := p.setValue(v); err != nil {
+			return pts, fmt.Errorf("unable to extract value from Fields, %s", err.Error())
+		}
+		p[0] = float64(pt.Time().Unix())
+		pts[k] = p
+	}
+	return pts, nil
+}
+
+func (p *Point) setValue(v interface{}) error {
+	switch d := v.(type) {
+	case int:
+		p[1] = float64(int(d))
+	case int32:
+		p[1] = float64(int32(d))
+	case int64:
+		p[1] = float64(int64(d))
+	case float32:
+		p[1] = float64(d)
+	case float64:
+		p[1] = float64(d)
+	default:
+		return fmt.Errorf("undeterminable type")
+	}
+	return nil
+}
+
+func (a *Amon) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("amon", func() outputs.Output {
+		return &Amon{}
+	})
+}

outputs/amon/amon_test.go

@@ -0,0 +1,89 @@
+package amon
+
+import (
+	"fmt"
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/influxdb/influxdb/client/v2"
+)
+
+func TestBuildPoint(t *testing.T) {
+	var tagtests = []struct {
+		ptIn  *client.Point
+		outPt Point
+		err   error
+	}{
+		{
+			testutil.TestPoint(float64(0.0)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				0.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float64(1.0)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				1.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int(10)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				10.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int32(112345)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int64(112345)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float32(11234.5)),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint("11234.5"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			fmt.Errorf("unable to extract value from Fields, undeterminable type"),
+		},
+	}
+	for _, tt := range tagtests {
+		pt, err := buildPoint(tt.ptIn)
+		if err != nil && tt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", tt.ptIn.Name(), err)
+		}
+		if tt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", tt.ptIn.Name(), tt.err.Error())
+		}
+		if !reflect.DeepEqual(pt, tt.outPt) && tt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", tt.ptIn.Name(), tt.outPt, pt)
+		}
+	}
+}

outputs/amqp/README.md

@@ -0,0 +1,9 @@
+# AMQP Output Plugin
+
+This plugin writes to a AMQP exchange using tag, defined in configuration file
+as RoutingTag, as a routing key.
+
+If RoutingTag is empty, then empty routing key will be used.
+Metrics are grouped in batches by RoutingTag.
+
+This plugin doesn't bind exchange to a queue, so it should be done by consumer.

outputs/amqp/amqp.go

@@ -0,0 +1,161 @@
+package amqp
+
+import (
+	"bytes"
+	"fmt"
+	"log"
+	"sync"
+	"time"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/streadway/amqp"
+)
+
+type AMQP struct {
+	// AMQP brokers to send metrics to
+	URL string
+	// AMQP exchange
+	Exchange string
+	// Routing Key Tag
+	RoutingTag string `toml:"routing_tag"`
+	// InfluxDB database
+	Database string
+	// InfluxDB retention policy
+	RetentionPolicy string
+	// InfluxDB precision
+	Precision string
+
+	channel *amqp.Channel
+	sync.Mutex
+	headers amqp.Table
+}
+
+const (
+	DefaultRetentionPolicy = "default"
+	DefaultDatabase        = "telegraf"
+	DefaultPrecision       = "s"
+)
+
+var sampleConfig = `
+  # AMQP url
+  url = "amqp://localhost:5672/influxdb"
+  # AMQP exchange
+  exchange = "telegraf"
+  # Telegraf tag to use as a routing key
+  #  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+
+  # InfluxDB retention policy
+  #retention_policy = "default"
+  # InfluxDB database
+  #database = "telegraf"
+  # InfluxDB precision
+  #precision = "s"
+`
+
+func (q *AMQP) Connect() error {
+	q.Lock()
+	defer q.Unlock()
+
+	q.headers = amqp.Table{
+		"precision":        q.Precision,
+		"database":         q.Database,
+		"retention_policy": q.RetentionPolicy,
+	}
+
+	connection, err := amqp.Dial(q.URL)
+	if err != nil {
+		return err
+	}
+	channel, err := connection.Channel()
+	if err != nil {
+		return fmt.Errorf("Failed to open a channel: %s", err)
+	}
+
+	err = channel.ExchangeDeclare(
+		q.Exchange, // name
+		"topic",    // type
+		true,       // durable
+		false,      // delete when unused
+		false,      // internal
+		false,      // no-wait
+		nil,        // arguments
+	)
+	if err != nil {
+		return fmt.Errorf("Failed to declare an exchange: %s", err)
+	}
+	q.channel = channel
+	go func() {
+		log.Printf("Closing: %s", <-connection.NotifyClose(make(chan *amqp.Error)))
+		log.Printf("Trying to reconnect")
+		for err := q.Connect(); err != nil; err = q.Connect() {
+			log.Println(err)
+			time.Sleep(10 * time.Second)
+		}
+
+	}()
+	return nil
+}
+
+func (q *AMQP) Close() error {
+	return q.channel.Close()
+}
+
+func (q *AMQP) SampleConfig() string {
+	return sampleConfig
+}
+
+func (q *AMQP) Description() string {
+	return "Configuration for the AMQP server to send metrics to"
+}
+
+func (q *AMQP) Write(points []*client.Point) error {
+	q.Lock()
+	defer q.Unlock()
+	if len(points) == 0 {
+		return nil
+	}
+	var outbuf = make(map[string][][]byte)
+
+	for _, p := range points {
+		// Combine tags from Point and BatchPoints and grab the resulting
+		// line-protocol output string to write to AMQP
+		var value, key string
+		value = p.String()
+
+		if q.RoutingTag != "" {
+			if h, ok := p.Tags()[q.RoutingTag]; ok {
+				key = h
+			}
+		}
+		outbuf[key] = append(outbuf[key], []byte(value))
+
+	}
+	for key, buf := range outbuf {
+		err := q.channel.Publish(
+			q.Exchange, // exchange
+			key,        // routing key
+			false,      // mandatory
+			false,      // immediate
+			amqp.Publishing{
+				Headers:     q.headers,
+				ContentType: "text/plain",
+				Body:        bytes.Join(buf, []byte("\n")),
+			})
+		if err != nil {
+			return fmt.Errorf("FAILED to send amqp message: %s", err)
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("amqp", func() outputs.Output {
+		return &AMQP{
+			Database:        DefaultDatabase,
+			Precision:       DefaultPrecision,
+			RetentionPolicy: DefaultRetentionPolicy,
+		}
+	})
+}

outputs/amqp/amqp_test.go

@@ -0,0 +1,28 @@
+package amqp
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	var url = "amqp://" + testutil.GetLocalHost() + ":5672/"
+	q := &AMQP{
+		URL:      url,
+		Exchange: "telegraf_test",
+	}
+
+	// Verify that we can connect to the AMQP broker
+	err := q.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the amqp broker
+	err = q.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/datadog/README.md

@@ -0,0 +1,9 @@
+# Datadog Output Plugin
+
+This plugin writes to the [Datadog Metrics API](http://docs.datadoghq.com/api/#metrics)
+and requires an `apikey` which can be obtained [here](https://app.datadoghq.com/account/settings#api)
+for the account.
+
+If the point value being sent cannot be converted to a float64, the metric is skipped.
+
+Metrics are grouped by converting any `_` characters to `.` in the Point Name.

outputs/datadog/datadog.go

@@ -0,0 +1,179 @@
+package datadog
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+	"net/url"
+	"sort"
+	"strings"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Datadog struct {
+	Apikey  string
+	Timeout internal.Duration
+
+	apiUrl string
+	client *http.Client
+}
+
+var sampleConfig = `
+  # Datadog API key
+  apikey = "my-secret-key" # required.
+
+  # Connection timeout.
+  # timeout = "5s"
+`
+
+type TimeSeries struct {
+	Series []*Metric `json:"series"`
+}
+
+type Metric struct {
+	Metric string   `json:"metric"`
+	Points [1]Point `json:"points"`
+	Host   string   `json:"host"`
+	Tags   []string `json:"tags,omitempty"`
+}
+
+type Point [2]float64
+
+const datadog_api = "https://app.datadoghq.com/api/v1/series"
+
+func NewDatadog(apiUrl string) *Datadog {
+	return &Datadog{
+		apiUrl: apiUrl,
+	}
+}
+
+func (d *Datadog) Connect() error {
+	if d.Apikey == "" {
+		return fmt.Errorf("apikey is a required field for datadog output")
+	}
+	d.client = &http.Client{
+		Timeout: d.Timeout.Duration,
+	}
+	return nil
+}
+
+func (d *Datadog) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	ts := TimeSeries{}
+	tempSeries := []*Metric{}
+	metricCounter := 0
+
+	for _, pt := range points {
+		mname := strings.Replace(pt.Name(), "_", ".", -1)
+		if amonPts, err := buildPoints(pt); err == nil {
+			for fieldName, amonPt := range amonPts {
+				metric := &Metric{
+					Metric: mname + strings.Replace(fieldName, "_", ".", -1),
+				}
+				metric.Points[0] = amonPt
+				tempSeries = append(tempSeries, metric)
+				metricCounter++
+			}
+		} else {
+			log.Printf("unable to build Metric for %s, skipping\n", pt.Name())
+		}
+	}
+
+	ts.Series = make([]*Metric, metricCounter)
+	copy(ts.Series, tempSeries[0:])
+	tsBytes, err := json.Marshal(ts)
+	if err != nil {
+		return fmt.Errorf("unable to marshal TimeSeries, %s\n", err.Error())
+	}
+	req, err := http.NewRequest("POST", d.authenticatedUrl(), bytes.NewBuffer(tsBytes))
+	if err != nil {
+		return fmt.Errorf("unable to create http.Request, %s\n", err.Error())
+	}
+	req.Header.Add("Content-Type", "application/json")
+
+	resp, err := d.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("error POSTing metrics, %s\n", err.Error())
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode < 200 || resp.StatusCode > 209 {
+		return fmt.Errorf("received bad status code, %d\n", resp.StatusCode)
+	}
+
+	return nil
+}
+
+func (d *Datadog) SampleConfig() string {
+	return sampleConfig
+}
+
+func (d *Datadog) Description() string {
+	return "Configuration for DataDog API to send metrics to."
+}
+
+func (d *Datadog) authenticatedUrl() string {
+	q := url.Values{
+		"api_key": []string{d.Apikey},
+	}
+	return fmt.Sprintf("%s?%s", d.apiUrl, q.Encode())
+}
+
+func buildPoints(pt *client.Point) (map[string]Point, error) {
+	pts := make(map[string]Point)
+	for k, v := range pt.Fields() {
+		var p Point
+		if err := p.setValue(v); err != nil {
+			return pts, fmt.Errorf("unable to extract value from Fields, %s", err.Error())
+		}
+		p[0] = float64(pt.Time().Unix())
+		pts[k] = p
+	}
+	return pts, nil
+}
+
+func buildTags(ptTags map[string]string) []string {
+	tags := make([]string, len(ptTags))
+	index := 0
+	for k, v := range ptTags {
+		tags[index] = fmt.Sprintf("%s:%s", k, v)
+		index += 1
+	}
+	sort.Strings(tags)
+	return tags
+}
+
+func (p *Point) setValue(v interface{}) error {
+	switch d := v.(type) {
+	case int:
+		p[1] = float64(int(d))
+	case int32:
+		p[1] = float64(int32(d))
+	case int64:
+		p[1] = float64(int64(d))
+	case float32:
+		p[1] = float64(d)
+	case float64:
+		p[1] = float64(d)
+	default:
+		return fmt.Errorf("undeterminable type")
+	}
+	return nil
+}
+
+func (d *Datadog) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("datadog", func() outputs.Output {
+		return NewDatadog(datadog_api)
+	})
+}

outputs/datadog/datadog_test.go

@@ -0,0 +1,176 @@
+package datadog
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var (
+	fakeUrl    = "http://test.datadog.com"
+	fakeApiKey = "123456"
+)
+
+func fakeDatadog() *Datadog {
+	d := NewDatadog(fakeUrl)
+	d.Apikey = fakeApiKey
+	return d
+}
+
+func TestUriOverride(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		json.NewEncoder(w).Encode(`{"status":"ok"}`)
+	}))
+	defer ts.Close()
+
+	d := NewDatadog(ts.URL)
+	d.Apikey = "123456"
+	err := d.Connect()
+	require.NoError(t, err)
+	err = d.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}
+
+func TestBadStatusCode(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusInternalServerError)
+		json.NewEncoder(w).Encode(`{ 'errors': [
+    	'Something bad happened to the server.',
+    	'Your query made the server very sad.'
+  		]
+		}`)
+	}))
+	defer ts.Close()
+
+	d := NewDatadog(ts.URL)
+	d.Apikey = "123456"
+	err := d.Connect()
+	require.NoError(t, err)
+	err = d.Write(testutil.MockBatchPoints().Points())
+	if err == nil {
+		t.Errorf("error expected but none returned")
+	} else {
+		require.EqualError(t, fmt.Errorf("received bad status code, 500\n"), err.Error())
+	}
+}
+
+func TestAuthenticatedUrl(t *testing.T) {
+	d := fakeDatadog()
+
+	authUrl := d.authenticatedUrl()
+	assert.EqualValues(t, fmt.Sprintf("%s?api_key=%s", fakeUrl, fakeApiKey), authUrl)
+}
+
+func TestBuildTags(t *testing.T) {
+	var tagtests = []struct {
+		ptIn    map[string]string
+		outTags []string
+	}{
+		{
+			map[string]string{"one": "two", "three": "four"},
+			[]string{"one:two", "three:four"},
+		},
+		{
+			map[string]string{"aaa": "bbb"},
+			[]string{"aaa:bbb"},
+		},
+		{
+			map[string]string{},
+			[]string{},
+		},
+	}
+	for _, tt := range tagtests {
+		tags := buildTags(tt.ptIn)
+		if !reflect.DeepEqual(tags, tt.outTags) {
+			t.Errorf("\nexpected %+v\ngot %+v\n", tt.outTags, tags)
+		}
+	}
+}
+
+func TestBuildPoint(t *testing.T) {
+	var tagtests = []struct {
+		ptIn  *client.Point
+		outPt Point
+		err   error
+	}{
+		{
+			testutil.TestPoint(0.0, "test1"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				0.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(1.0, "test2"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				1.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(10, "test3"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				10.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int32(112345), "test4"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int64(112345), "test5"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float32(11234.5), "test6"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint("11234.5", "test7"),
+			Point{
+				float64(time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix()),
+				11234.5,
+			},
+			fmt.Errorf("unable to extract value from Fields, undeterminable type"),
+		},
+	}
+	for _, tt := range tagtests {
+		pt, err := buildPoint(tt.ptIn)
+		if err != nil && tt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", tt.ptIn.Name(), err)
+		}
+		if tt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", tt.ptIn.Name(), tt.err.Error())
+		}
+		if !reflect.DeepEqual(pt, tt.outPt) && tt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", tt.ptIn.Name(), tt.outPt, pt)
+		}
+	}
+}

outputs/influxdb/README.md

@@ -0,0 +1,12 @@
+# InfluxDB Output Plugin
+
+This plugin writes to [InfluxDB](https://www.influxdb.com) via HTTP or UDP.
+
+Required parameters:
+
+* `urls`: List of strings, this is for InfluxDB clustering
+support. On each flush interval, Telegraf will randomly choose one of the urls
+to write to. Each URL should start with either `http://` or `udp://`
+* `database`: The name of the database to write to.
+
+

outputs/influxdb/influxdb.go

@@ -0,0 +1,162 @@
+package influxdb
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"math/rand"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type InfluxDB struct {
+	// URL is only for backwards compatability
+	URL        string
+	URLs       []string `toml:"urls"`
+	Username   string
+	Password   string
+	Database   string
+	UserAgent  string
+	Precision  string
+	Timeout    internal.Duration
+	UDPPayload int `toml:"udp_payload"`
+
+	conns []client.Client
+}
+
+var sampleConfig = `
+  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
+  # Multiple urls can be specified but it is assumed that they are part of the same
+  # cluster, this means that only ONE of the urls will be written to each interval.
+  # urls = ["udp://localhost:8089"] # UDP endpoint example
+  urls = ["http://localhost:8086"] # required
+  # The target database for metrics (telegraf will create it if not exists)
+  database = "telegraf" # required
+  # Precision of writes, valid values are n, u, ms, s, m, and h
+  # note: using second precision greatly helps InfluxDB compression
+  precision = "s"
+
+  # Connection timeout (for the connection with InfluxDB), formatted as a string.
+  # If not provided, will default to 0 (no timeout)
+  # timeout = "5s"
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
+  # user_agent = "telegraf"
+  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
+  # udp_payload = 512
+`
+
+func (i *InfluxDB) Connect() error {
+	var urls []string
+	for _, u := range i.URLs {
+		urls = append(urls, u)
+	}
+
+	// Backward-compatability with single Influx URL config files
+	// This could eventually be removed in favor of specifying the urls as a list
+	if i.URL != "" {
+		urls = append(urls, i.URL)
+	}
+
+	var conns []client.Client
+	for _, u := range urls {
+		switch {
+		case strings.HasPrefix(u, "udp"):
+			parsed_url, err := url.Parse(u)
+			if err != nil {
+				return err
+			}
+
+			if i.UDPPayload == 0 {
+				i.UDPPayload = client.UDPPayloadSize
+			}
+			c, err := client.NewUDPClient(client.UDPConfig{
+				Addr:        parsed_url.Host,
+				PayloadSize: i.UDPPayload,
+			})
+			if err != nil {
+				return err
+			}
+			conns = append(conns, c)
+		default:
+			// If URL doesn't start with "udp", assume HTTP client
+			c, err := client.NewHTTPClient(client.HTTPConfig{
+				Addr:      u,
+				Username:  i.Username,
+				Password:  i.Password,
+				UserAgent: i.UserAgent,
+				Timeout:   i.Timeout.Duration,
+			})
+			if err != nil {
+				return err
+			}
+
+			// Create Database if it doesn't exist
+			_, e := c.Query(client.Query{
+				Command: fmt.Sprintf("CREATE DATABASE IF NOT EXISTS %s", i.Database),
+			})
+
+			if e != nil {
+				log.Println("Database creation failed: " + e.Error())
+			}
+
+			conns = append(conns, c)
+		}
+	}
+
+	i.conns = conns
+	rand.Seed(time.Now().UnixNano())
+	return nil
+}
+
+func (i *InfluxDB) Close() error {
+	// InfluxDB client does not provide a Close() function
+	return nil
+}
+
+func (i *InfluxDB) SampleConfig() string {
+	return sampleConfig
+}
+
+func (i *InfluxDB) Description() string {
+	return "Configuration for influxdb server to send metrics to"
+}
+
+// Choose a random server in the cluster to write to until a successful write
+// occurs, logging each unsuccessful. If all servers fail, return error.
+func (i *InfluxDB) Write(points []*client.Point) error {
+	bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
+		Database:  i.Database,
+		Precision: i.Precision,
+	})
+
+	for _, point := range points {
+		bp.AddPoint(point)
+	}
+
+	// This will get set to nil if a successful write occurs
+	err := errors.New("Could not write to any InfluxDB server in cluster")
+
+	p := rand.Perm(len(i.conns))
+	for _, n := range p {
+		if e := i.conns[n].Write(bp); e != nil {
+			log.Println("ERROR: " + e.Error())
+		} else {
+			err = nil
+			break
+		}
+	}
+	return err
+}
+
+func init() {
+	outputs.Add("influxdb", func() outputs.Output {
+		return &InfluxDB{}
+	})
+}

outputs/influxdb/influxdb_test.go

@@ -0,0 +1,41 @@
+package influxdb
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestUDPInflux(t *testing.T) {
+	i := InfluxDB{
+		URLs: []string{"udp://localhost:8089"},
+	}
+
+	err := i.Connect()
+	require.NoError(t, err)
+	err = i.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}
+
+func TestHTTPInflux(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprintln(w, `{"results":[{}]}`)
+	}))
+	defer ts.Close()
+
+	i := InfluxDB{
+		URLs: []string{ts.URL},
+	}
+
+	err := i.Connect()
+	require.NoError(t, err)
+	err = i.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/kafka/kafka.go

@@ -0,0 +1,85 @@
+package kafka
+
+import (
+	"errors"
+	"fmt"
+
+	"github.com/Shopify/sarama"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Kafka struct {
+	// Kafka brokers to send metrics to
+	Brokers []string
+	// Kafka topic
+	Topic string
+	// Routing Key Tag
+	RoutingTag string `toml:"routing_tag"`
+
+	producer sarama.SyncProducer
+}
+
+var sampleConfig = `
+  # URLs of kafka brokers
+  brokers = ["localhost:9092"]
+  # Kafka topic for producer messages
+  topic = "telegraf"
+  # Telegraf tag to use as a routing key
+  #  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+`
+
+func (k *Kafka) Connect() error {
+	producer, err := sarama.NewSyncProducer(k.Brokers, nil)
+	if err != nil {
+		return err
+	}
+	k.producer = producer
+	return nil
+}
+
+func (k *Kafka) Close() error {
+	return k.producer.Close()
+}
+
+func (k *Kafka) SampleConfig() string {
+	return sampleConfig
+}
+
+func (k *Kafka) Description() string {
+	return "Configuration for the Kafka server to send metrics to"
+}
+
+func (k *Kafka) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	for _, p := range points {
+		// Combine tags from Point and BatchPoints and grab the resulting
+		// line-protocol output string to write to Kafka
+		value := p.String()
+
+		m := &sarama.ProducerMessage{
+			Topic: k.Topic,
+			Value: sarama.StringEncoder(value),
+		}
+		if h, ok := p.Tags()[k.RoutingTag]; ok {
+			m.Key = sarama.StringEncoder(h)
+		}
+
+		_, _, err := k.producer.SendMessage(m)
+		if err != nil {
+			return errors.New(fmt.Sprintf("FAILED to send kafka message: %s\n",
+				err))
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("kafka", func() outputs.Output {
+		return &Kafka{}
+	})
+}

outputs/kafka/kafka_test.go

@@ -0,0 +1,28 @@
+package kafka
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	brokers := []string{testutil.GetLocalHost() + ":9092"}
+	k := &Kafka{
+		Brokers: brokers,
+		Topic:   "Test",
+	}
+
+	// Verify that we can connect to the Kafka broker
+	err := k.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the kafka broker
+	err = k.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/kinesis/README.md

@@ -0,0 +1,61 @@
+## Amazon Kinesis Output for Telegraf
+
+This is an experimental plugin that is still in the early stages of development. It will batch up all of the Points
+in one Put request to Kinesis. This should save the number of API requests by a considerable level.
+
+## About Kinesis
+
+This is not the place to document all of the various Kinesis terms however it
+maybe useful for users to review Amazons official documentation which is available
+[here](http://docs.aws.amazon.com/kinesis/latest/dev/key-concepts.html).
+
+## Amazon Authentication
+
+This plugin uses a credential chain for Authentication with the Kinesis API endpoint. In the following order the plugin
+will attempt to authenticate.
+1. [IAMS Role](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
+2. [Environment Variables](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk)
+3. [Shared Credentials](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk)
+
+
+## Config
+
+For this output plugin to function correctly the following variables must be configured.
+
+* region
+* streamname
+* partitionkey
+
+### region
+
+The region is the Amazon region that you wish to connect to. Examples include but are not limited to
+* us-west-1
+* us-west-2
+* us-east-1
+* ap-southeast-1
+* ap-southeast-2
+
+### streamname
+
+The streamname is used by the plugin to ensure that data is sent to the correct Kinesis stream. It is important to
+note that the stream *MUST* be pre-configured for this plugin to function correctly. If the stream does not exist the
+plugin will result in telegraf exiting with an exit code of 1.
+
+### partitionkey
+
+This is used to group data within a stream. Currently this plugin only supports a single partitionkey.
+Manually configuring different hosts, or groups of hosts with manually selected partitionkeys might be a workable
+solution to scale out.
+
+### format
+
+The format configuration value has been designated to allow people to change the format of the Point as written to
+Kinesis. Right now there are two supported formats string and custom.
+
+#### string
+
+String is defined using the default Point.String() value and translated to []byte for the Kinesis stream.
+
+#### custom
+
+Custom is a string defined by a number of values in the FormatMetric() function.

outputs/kinesis/kinesis.go

@@ -0,0 +1,179 @@
+package kinesis
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"os"
+	"sync/atomic"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
+	"github.com/aws/aws-sdk-go/aws/ec2metadata"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/kinesis"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type KinesisOutput struct {
+	Region       string `toml:"region"`
+	StreamName   string `toml:"streamname"`
+	PartitionKey string `toml:"partitionkey"`
+	Format       string `toml:"format"`
+	Debug        bool   `toml:"debug"`
+	svc          *kinesis.Kinesis
+}
+
+var sampleConfig = `
+  # Amazon REGION of kinesis endpoint.
+  region = "ap-southeast-2"
+  # Kinesis StreamName must exist prior to starting telegraf.
+  streamname = "StreamName"
+  # PartitionKey as used for sharding data.
+  partitionkey = "PartitionKey"
+  # format of the Data payload in the kinesis PutRecord, supported
+  # String and Custom.
+  format = "string"
+  # debug will show upstream aws messages.
+  debug = false
+`
+
+func (k *KinesisOutput) SampleConfig() string {
+	return sampleConfig
+}
+
+func (k *KinesisOutput) Description() string {
+	return "Configuration for the AWS Kinesis output."
+}
+
+func checkstream(l []*string, s string) bool {
+	// Check if the StreamName exists in the slice returned from the ListStreams API request.
+	for _, stream := range l {
+		if *stream == s {
+			return true
+		}
+	}
+	return false
+}
+
+func (k *KinesisOutput) Connect() error {
+	// We attempt first to create a session to Kinesis using an IAMS role, if that fails it will fall through to using
+	// environment variables, and then Shared Credentials.
+	if k.Debug {
+		log.Printf("kinesis: Establishing a connection to Kinesis in %+v", k.Region)
+	}
+	Config := &aws.Config{
+		Region: aws.String(k.Region),
+		Credentials: credentials.NewChainCredentials(
+			[]credentials.Provider{
+				&ec2rolecreds.EC2RoleProvider{Client: ec2metadata.New(session.New())},
+				&credentials.EnvProvider{},
+				&credentials.SharedCredentialsProvider{},
+			}),
+	}
+	svc := kinesis.New(session.New(Config))
+
+	KinesisParams := &kinesis.ListStreamsInput{
+		Limit: aws.Int64(100),
+	}
+
+	resp, err := svc.ListStreams(KinesisParams)
+
+	if err != nil {
+		log.Printf("kinesis: Error in ListSteams API call : %+v \n", err)
+	}
+
+	if checkstream(resp.StreamNames, k.StreamName) {
+		if k.Debug {
+			log.Printf("kinesis: Stream Exists")
+		}
+		k.svc = svc
+		return nil
+	} else {
+		log.Printf("kinesis : You have configured a StreamName %+v which does not exist. exiting.", k.StreamName)
+		os.Exit(1)
+	}
+	return err
+}
+
+func (k *KinesisOutput) Close() error {
+	return errors.New("Error")
+}
+
+func FormatMetric(k *KinesisOutput, point *client.Point) (string, error) {
+	if k.Format == "string" {
+		return point.String(), nil
+	} else {
+		m := fmt.Sprintf("%+v,%+v,%+v",
+			point.Name(),
+			point.Tags(),
+			point.String())
+		return m, nil
+	}
+}
+
+func writekinesis(k *KinesisOutput, r []*kinesis.PutRecordsRequestEntry) time.Duration {
+	start := time.Now()
+	payload := &kinesis.PutRecordsInput{
+		Records:    r,
+		StreamName: aws.String(k.StreamName),
+	}
+
+	if k.Debug {
+		resp, err := k.svc.PutRecords(payload)
+		if err != nil {
+			log.Printf("kinesis: Unable to write to Kinesis : %+v \n", err.Error())
+		}
+		log.Printf("%+v \n", resp)
+
+	} else {
+		_, err := k.svc.PutRecords(payload)
+		if err != nil {
+			log.Printf("kinesis: Unable to write to Kinesis : %+v \n", err.Error())
+		}
+	}
+	return time.Since(start)
+}
+
+func (k *KinesisOutput) Write(points []*client.Point) error {
+	var sz uint32 = 0
+
+	if len(points) == 0 {
+		return nil
+	}
+
+	r := []*kinesis.PutRecordsRequestEntry{}
+
+	for _, p := range points {
+		atomic.AddUint32(&sz, 1)
+
+		metric, _ := FormatMetric(k, p)
+		d := kinesis.PutRecordsRequestEntry{
+			Data:         []byte(metric),
+			PartitionKey: aws.String(k.PartitionKey),
+		}
+		r = append(r, &d)
+
+		if sz == 500 {
+			// Max Messages Per PutRecordRequest is 500
+			elapsed := writekinesis(k, r)
+			log.Printf("Wrote a %+v point batch to Kinesis in %+v.\n", sz, elapsed)
+			atomic.StoreUint32(&sz, 0)
+			r = nil
+		}
+	}
+
+	writekinesis(k, r)
+
+	return nil
+}
+
+func init() {
+	outputs.Add("kinesis", func() outputs.Output {
+		return &KinesisOutput{}
+	})
+}

outputs/kinesis/kinesis_test.go

@@ -0,0 +1,39 @@
+package kinesis
+
+import (
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+	"testing"
+)
+
+func TestFormatMetric(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	k := &KinesisOutput{
+		Format: "string",
+	}
+
+	p := testutil.MockBatchPoints().Points()[0]
+
+	valid_string := "test1,tag1=value1 value=1 1257894000000000000"
+	func_string, err := FormatMetric(k, p)
+
+	if func_string != valid_string {
+		t.Error("Expected ", valid_string)
+	}
+	require.NoError(t, err)
+
+	k = &KinesisOutput{
+		Format: "custom",
+	}
+
+	valid_custom := "test1,map[tag1:value1],test1,tag1=value1 value=1 1257894000000000000"
+	func_custom, err := FormatMetric(k, p)
+
+	if func_custom != valid_custom {
+		t.Error("Expected ", valid_custom)
+	}
+	require.NoError(t, err)
+}

outputs/librato/README.md

@@ -0,0 +1,12 @@
+# Librato Output Plugin
+
+This plugin writes to the [Librato Metrics API](http://dev.librato.com/v1/metrics#metrics)
+and requires an `api_user` and `api_token` which can be obtained [here](https://metrics.librato.com/account/api_tokens)
+for the account.
+
+The `source_tag` option in the Configuration file is used to send contextual information from
+Point Tags to the API.
+
+If the point value being sent cannot be converted to a float64, the metric is skipped.
+
+Currently, the plugin does not send any associated Point Tags.

outputs/librato/librato.go

@@ -0,0 +1,175 @@
+package librato
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Librato struct {
+	ApiUser   string
+	ApiToken  string
+	SourceTag string
+	Timeout   internal.Duration
+
+	apiUrl string
+	client *http.Client
+}
+
+var sampleConfig = `
+  # Librator API Docs
+  # http://dev.librato.com/v1/metrics-authentication
+
+  # Librato API user
+  api_user = "telegraf@influxdb.com" # required.
+
+  # Librato API token
+  api_token = "my-secret-token" # required.
+
+  # Tag Field to populate source attribute (optional)
+  # This is typically the _hostname_ from which the metric was obtained.
+  source_tag = "hostname"
+
+  # Connection timeout.
+  # timeout = "5s"
+`
+
+type Metrics struct {
+	Gauges []*Gauge `json:"gauges"`
+}
+
+type Gauge struct {
+	Name        string  `json:"name"`
+	Value       float64 `json:"value"`
+	Source      string  `json:"source"`
+	MeasureTime int64   `json:"measure_time"`
+}
+
+const librato_api = "https://metrics-api.librato.com/v1/metrics"
+
+func NewLibrato(apiUrl string) *Librato {
+	return &Librato{
+		apiUrl: apiUrl,
+	}
+}
+
+func (l *Librato) Connect() error {
+	if l.ApiUser == "" || l.ApiToken == "" {
+		return fmt.Errorf("api_user and api_token are required fields for librato output")
+	}
+	l.client = &http.Client{
+		Timeout: l.Timeout.Duration,
+	}
+	return nil
+}
+
+func (l *Librato) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	metrics := Metrics{}
+	tempGauges := []*Gauge{}
+	metricCounter := 0
+
+	for _, pt := range points {
+		if gauges, err := l.buildGauges(pt); err == nil {
+			for _, gauge := range gauges {
+				tempGauges = append(tempGauges, gauge)
+				metricCounter++
+			}
+		} else {
+			log.Printf("unable to build Gauge for %s, skipping\n", pt.Name())
+		}
+	}
+
+	metrics.Gauges = make([]*Gauge, metricCounter)
+	copy(metrics.Gauges, tempGauges[0:])
+	metricsBytes, err := json.Marshal(metrics)
+	if err != nil {
+		return fmt.Errorf("unable to marshal Metrics, %s\n", err.Error())
+	}
+	req, err := http.NewRequest("POST", l.apiUrl, bytes.NewBuffer(metricsBytes))
+	if err != nil {
+		return fmt.Errorf("unable to create http.Request, %s\n", err.Error())
+	}
+	req.Header.Add("Content-Type", "application/json")
+	req.SetBasicAuth(l.ApiUser, l.ApiToken)
+
+	resp, err := l.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("error POSTing metrics, %s\n", err.Error())
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != 200 {
+		return fmt.Errorf("received bad status code, %d\n", resp.StatusCode)
+	}
+
+	return nil
+}
+
+func (l *Librato) SampleConfig() string {
+	return sampleConfig
+}
+
+func (l *Librato) Description() string {
+	return "Configuration for Librato API to send metrics to."
+}
+
+func (l *Librato) buildGauges(pt *client.Point) ([]*Gauge, error) {
+	gauges := []*Gauge{}
+	for fieldName, value := range pt.Fields() {
+		gauge := &Gauge{
+			Name:        pt.Name() + "_" + fieldName,
+			MeasureTime: pt.Time().Unix(),
+		}
+		if err := gauge.setValue(value); err != nil {
+			return gauges, fmt.Errorf("unable to extract value from Fields, %s\n",
+				err.Error())
+		}
+		if l.SourceTag != "" {
+			if source, ok := pt.Tags()[l.SourceTag]; ok {
+				gauge.Source = source
+			} else {
+				return gauges,
+					fmt.Errorf("undeterminable Source type from Field, %s\n",
+						l.SourceTag)
+			}
+		}
+	}
+	return gauges, nil
+}
+
+func (g *Gauge) setValue(v interface{}) error {
+	switch d := v.(type) {
+	case int:
+		g.Value = float64(int(d))
+	case int32:
+		g.Value = float64(int32(d))
+	case int64:
+		g.Value = float64(int64(d))
+	case float32:
+		g.Value = float64(d)
+	case float64:
+		g.Value = float64(d)
+	default:
+		return fmt.Errorf("undeterminable type %+v", d)
+	}
+	return nil
+}
+
+func (l *Librato) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("librato", func() outputs.Output {
+		return NewLibrato(librato_api)
+	})
+}

outputs/librato/librato_test.go

@@ -0,0 +1,212 @@
+package librato
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/stretchr/testify/require"
+)
+
+var (
+	fakeUrl   = "http://test.librato.com"
+	fakeUser  = "telegraf@influxdb.com"
+	fakeToken = "123456"
+)
+
+func fakeLibrato() *Librato {
+	l := NewLibrato(fakeUrl)
+	l.ApiUser = fakeUser
+	l.ApiToken = fakeToken
+	return l
+}
+
+func TestUriOverride(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer ts.Close()
+
+	l := NewLibrato(ts.URL)
+	l.ApiUser = "telegraf@influxdb.com"
+	l.ApiToken = "123456"
+	err := l.Connect()
+	require.NoError(t, err)
+	err = l.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}
+
+func TestBadStatusCode(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusServiceUnavailable)
+		json.NewEncoder(w).Encode(`{
+      "errors": {
+        "system": [
+          "The API is currently down for maintenance. It'll be back shortly."
+        ]
+      }
+    }`)
+	}))
+	defer ts.Close()
+
+	l := NewLibrato(ts.URL)
+	l.ApiUser = "telegraf@influxdb.com"
+	l.ApiToken = "123456"
+	err := l.Connect()
+	require.NoError(t, err)
+	err = l.Write(testutil.MockBatchPoints().Points())
+	if err == nil {
+		t.Errorf("error expected but none returned")
+	} else {
+		require.EqualError(t, fmt.Errorf("received bad status code, 503\n"), err.Error())
+	}
+}
+
+func TestBuildGauge(t *testing.T) {
+	var gaugeTests = []struct {
+		ptIn     *client.Point
+		outGauge *Gauge
+		err      error
+	}{
+		{
+			testutil.TestPoint(0.0, "test1"),
+			&Gauge{
+				Name:        "test1",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       0.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(1.0, "test2"),
+			&Gauge{
+				Name:        "test2",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       1.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(10, "test3"),
+			&Gauge{
+				Name:        "test3",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       10.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int32(112345), "test4"),
+			&Gauge{
+				Name:        "test4",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(int64(112345), "test5"),
+			&Gauge{
+				Name:        "test5",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       112345.0,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint(float32(11234.5), "test6"),
+			&Gauge{
+				Name:        "test6",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       11234.5,
+			},
+			nil,
+		},
+		{
+			testutil.TestPoint("11234.5", "test7"),
+			&Gauge{
+				Name:        "test7",
+				MeasureTime: time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       11234.5,
+			},
+			fmt.Errorf("unable to extract value from Fields, undeterminable type"),
+		},
+	}
+
+	l := NewLibrato(fakeUrl)
+	for _, gt := range gaugeTests {
+		gauge, err := l.buildGauge(gt.ptIn)
+		if err != nil && gt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", gt.ptIn.Name(), err)
+		}
+		if gt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", gt.ptIn.Name(), gt.err.Error())
+		}
+		if !reflect.DeepEqual(gauge, gt.outGauge) && gt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", gt.ptIn.Name(), gt.outGauge, gauge)
+		}
+	}
+}
+
+func TestBuildGaugeWithSource(t *testing.T) {
+	pt1, _ := client.NewPoint(
+		"test1",
+		map[string]string{"hostname": "192.168.0.1"},
+		map[string]interface{}{"value": 0.0},
+		time.Date(2010, time.November, 10, 23, 0, 0, 0, time.UTC),
+	)
+	pt2, _ := client.NewPoint(
+		"test2",
+		map[string]string{"hostnam": "192.168.0.1"},
+		map[string]interface{}{"value": 1.0},
+		time.Date(2010, time.December, 10, 23, 0, 0, 0, time.UTC),
+	)
+	var gaugeTests = []struct {
+		ptIn     *client.Point
+		outGauge *Gauge
+		err      error
+	}{
+
+		{
+			pt1,
+			&Gauge{
+				Name:        "test1",
+				MeasureTime: time.Date(2010, time.November, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       0.0,
+				Source:      "192.168.0.1",
+			},
+			nil,
+		},
+		{
+			pt2,
+			&Gauge{
+				Name:        "test2",
+				MeasureTime: time.Date(2010, time.December, 10, 23, 0, 0, 0, time.UTC).Unix(),
+				Value:       1.0,
+			},
+			fmt.Errorf("undeterminable Source type from Field, hostname"),
+		},
+	}
+
+	l := NewLibrato(fakeUrl)
+	l.SourceTag = "hostname"
+	for _, gt := range gaugeTests {
+		gauge, err := l.buildGauge(gt.ptIn)
+		if err != nil && gt.err == nil {
+			t.Errorf("%s: unexpected error, %+v\n", gt.ptIn.Name(), err)
+		}
+		if gt.err != nil && err == nil {
+			t.Errorf("%s: expected an error (%s) but none returned", gt.ptIn.Name(), gt.err.Error())
+		}
+		if !reflect.DeepEqual(gauge, gt.outGauge) && gt.err == nil {
+			t.Errorf("%s: \nexpected %+v\ngot %+v\n", gt.ptIn.Name(), gt.outGauge, gauge)
+		}
+	}
+}

outputs/mqtt/mqtt.go

@@ -0,0 +1,190 @@
+package mqtt
+
+import (
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"io/ioutil"
+	"strings"
+	"sync"
+
+	paho "git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+const MaxClientIdLen = 8
+const MaxRetryCount = 3
+const ClientIdPrefix = "telegraf"
+
+type MQTT struct {
+	Servers     []string `toml:"servers"`
+	Username    string
+	Password    string
+	Database    string
+	Timeout     internal.Duration
+	TopicPrefix string
+
+	Client *paho.Client
+	Opts   *paho.ClientOptions
+	sync.Mutex
+}
+
+var sampleConfig = `
+  servers = ["localhost:1883"] # required.
+
+  # MQTT outputs send metrics to this topic format
+  #    "<topic_prefix>/host/<hostname>/<pluginname>/"
+  #   ex: prefix/host/web01.example.com/mem/available
+  # topic_prefix = "prefix"
+
+  # username and password to connect MQTT server.
+  # username = "telegraf"
+  # password = "metricsmetricsmetricsmetrics"
+`
+
+func (m *MQTT) Connect() error {
+	var err error
+	m.Lock()
+	defer m.Unlock()
+
+	m.Opts, err = m.CreateOpts()
+	if err != nil {
+		return err
+	}
+
+	m.Client = paho.NewClient(m.Opts)
+	if token := m.Client.Connect(); token.Wait() && token.Error() != nil {
+		return token.Error()
+	}
+
+	return nil
+}
+
+func (m *MQTT) Close() error {
+	if m.Client.IsConnected() {
+		m.Client.Disconnect(20)
+	}
+	return nil
+}
+
+func (m *MQTT) SampleConfig() string {
+	return sampleConfig
+}
+
+func (m *MQTT) Description() string {
+	return "Configuration for MQTT server to send metrics to"
+}
+
+func (m *MQTT) Write(points []*client.Point) error {
+	m.Lock()
+	defer m.Unlock()
+	if len(points) == 0 {
+		return nil
+	}
+	hostname, ok := points[0].Tags()["host"]
+	if !ok {
+		hostname = ""
+	}
+
+	for _, p := range points {
+		var t []string
+		if m.TopicPrefix != "" {
+			t = append(t, m.TopicPrefix)
+		}
+		tm := strings.Split(p.Name(), "_")
+		if len(tm) < 2 {
+			tm = []string{p.Name(), "stat"}
+		}
+
+		t = append(t, "host", hostname, tm[0], tm[1])
+		topic := strings.Join(t, "/")
+
+		value := p.String()
+		err := m.publish(topic, value)
+		if err != nil {
+			return fmt.Errorf("Could not write to MQTT server, %s", err)
+		}
+	}
+
+	return nil
+}
+
+func (m *MQTT) publish(topic, body string) error {
+	token := m.Client.Publish(topic, 0, false, body)
+	token.Wait()
+	if token.Error() != nil {
+		return token.Error()
+	}
+	return nil
+}
+
+func (m *MQTT) CreateOpts() (*paho.ClientOptions, error) {
+	opts := paho.NewClientOptions()
+
+	clientId := getRandomClientId()
+	opts.SetClientID(clientId)
+
+	TLSConfig := &tls.Config{InsecureSkipVerify: false}
+	ca := "" // TODO
+	scheme := "tcp"
+	if ca != "" {
+		scheme = "ssl"
+		certPool, err := getCertPool(ca)
+		if err != nil {
+			return nil, err
+		}
+		TLSConfig.RootCAs = certPool
+	}
+	TLSConfig.InsecureSkipVerify = true // TODO
+	opts.SetTLSConfig(TLSConfig)
+
+	user := m.Username
+	if user == "" {
+		opts.SetUsername(user)
+	}
+	password := m.Password
+	if password != "" {
+		opts.SetPassword(password)
+	}
+
+	if len(m.Servers) == 0 {
+		return opts, fmt.Errorf("could not get host infomations")
+	}
+	for _, host := range m.Servers {
+		server := fmt.Sprintf("%s://%s", scheme, host)
+
+		opts.AddBroker(server)
+	}
+	opts.SetAutoReconnect(true)
+	return opts, nil
+}
+
+func getRandomClientId() string {
+	const alphanum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+	var bytes = make([]byte, MaxClientIdLen)
+	rand.Read(bytes)
+	for i, b := range bytes {
+		bytes[i] = alphanum[b%byte(len(alphanum))]
+	}
+	return ClientIdPrefix + "-" + string(bytes)
+}
+
+func getCertPool(pemPath string) (*x509.CertPool, error) {
+	certs := x509.NewCertPool()
+
+	pemData, err := ioutil.ReadFile(pemPath)
+	if err != nil {
+		return nil, err
+	}
+	certs.AppendCertsFromPEM(pemData)
+	return certs, nil
+}
+
+func init() {
+	outputs.Add("mqtt", func() outputs.Output {
+		return &MQTT{}
+	})
+}

outputs/mqtt/mqtt_test.go

@@ -0,0 +1,27 @@
+package mqtt
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	var url = testutil.GetLocalHost() + ":1883"
+	m := &MQTT{
+		Servers: []string{url},
+	}
+
+	// Verify that we can connect to the MQTT broker
+	err := m.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the mqtt broker
+	err = m.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/nsq/README.md

@@ -0,0 +1,4 @@
+# NSQ Output Plugin
+
+This plugin writes to a specified NSQD instance, usually local to the producer. It requires
+a `server` name and a `topic` name.

outputs/nsq/nsq.go

@@ -0,0 +1,71 @@
+package nsq
+
+import (
+	"fmt"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/nsqio/go-nsq"
+)
+
+type NSQ struct {
+	Server   string
+	Topic    string
+	producer *nsq.Producer
+}
+
+var sampleConfig = `
+  # Location of nsqd instance listening on TCP
+  server = "localhost:4150"
+  # NSQ topic for producer messages
+  topic = "telegraf"
+`
+
+func (n *NSQ) Connect() error {
+	config := nsq.NewConfig()
+	producer, err := nsq.NewProducer(n.Server, config)
+
+	if err != nil {
+		return err
+	}
+
+	n.producer = producer
+	return nil
+}
+
+func (n *NSQ) Close() error {
+	n.producer.Stop()
+	return nil
+}
+
+func (n *NSQ) SampleConfig() string {
+	return sampleConfig
+}
+
+func (n *NSQ) Description() string {
+	return "Send telegraf measurements to NSQD"
+}
+
+func (n *NSQ) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	for _, p := range points {
+		// Combine tags from Point and BatchPoints and grab the resulting
+		// line-protocol output string to write to NSQ
+		value := p.String()
+
+		err := n.producer.Publish(n.Topic, []byte(value))
+
+		if err != nil {
+			return fmt.Errorf("FAILED to send NSQD message: %s", err)
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("nsq", func() outputs.Output {
+		return &NSQ{}
+	})
+}

outputs/nsq/nsq_test.go

@@ -0,0 +1,28 @@
+package nsq
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	server := []string{testutil.GetLocalHost() + ":4150"}
+	n := &NSQ{
+		Server: server[0],
+		Topic:  "telegraf",
+	}
+
+	// Verify that we can connect to the NSQ daemon
+	err := n.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to the NSQ daemon
+	err = n.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

outputs/opentsdb/README.md

@@ -0,0 +1,78 @@
+# OpenTSDB Output Plugin
+
+This plugin writes to a OpenTSDB instance using the "telnet" mode
+
+## Transfer "Protocol" in the telnet mode
+
+The expected input from OpenTSDB is specified in the following way:
+
+```
+put <metric> <timestamp> <value> <tagk1=tagv1[ tagk2=tagv2 ...tagkN=tagvN]>
+```
+
+The telegraf output plugin adds an optional prefix to the metric keys so 
+that a subamount can be selected.
+
+```
+put <[prefix.]metric> <timestamp> <value> <tagk1=tagv1[ tagk2=tagv2 ...tagkN=tagvN]>
+```
+
+### Example 
+
+```
+put nine.telegraf.system_load1 1441910356 0.430000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_load5 1441910356 0.580000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_load15 1441910356 0.730000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_uptime 1441910356 3655970.000000 dc=homeoffice host=irimame scope=green
+put nine.telegraf.system_uptime_format 1441910356  dc=homeoffice host=irimame scope=green
+put nine.telegraf.mem_total 1441910356 4145426432 dc=homeoffice host=irimame scope=green
+...
+put nine.telegraf.io_write_bytes 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.io_read_time 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.io_write_time 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.io_io_time 1441910366 0 dc=homeoffice host=irimame name=vda2 scope=green
+put nine.telegraf.ping_packets_transmitted 1441910366  dc=homeoffice host=irimame scope=green url=www.google.com
+put nine.telegraf.ping_packets_received 1441910366  dc=homeoffice host=irimame scope=green url=www.google.com
+put nine.telegraf.ping_percent_packet_loss 1441910366 0.000000 dc=homeoffice host=irimame scope=green url=www.google.com
+put nine.telegraf.ping_average_response_ms 1441910366 24.006000 dc=homeoffice host=irimame scope=green url=www.google.com
+...
+```
+
+## 
+
+The OpenTSDB interface can be simulated with this reader:
+
+```
+// opentsdb_telnet_mode_mock.go 
+package main
+
+import (
+	"io"
+	"log"
+	"net"
+	"os"
+)
+
+func main() {
+	l, err := net.Listen("tcp", "localhost:4242")
+	if err != nil {
+		log.Fatal(err)
+	}
+	defer l.Close()
+	for {
+		conn, err := l.Accept()
+		if err != nil {
+			log.Fatal(err)
+		}
+		go func(c net.Conn) {
+			defer c.Close()
+			io.Copy(os.Stdout, c)
+		}(conn)
+	}
+}
+
+```
+
+## Allowed values for metrics
+
+OpenTSDB allows `integers` and `floats` as input values

outputs/opentsdb/opentsdb.go

@@ -0,0 +1,168 @@
+package opentsdb
+
+import (
+	"fmt"
+	"net"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type OpenTSDB struct {
+	Prefix string
+
+	Host string
+	Port int
+
+	Debug bool
+}
+
+var sampleConfig = `
+  # prefix for metrics keys
+  prefix = "my.specific.prefix."
+
+  ## Telnet Mode ##
+  # DNS name of the OpenTSDB server in telnet mode
+  host = "opentsdb.example.com"
+
+  # Port of the OpenTSDB server in telnet mode
+  port = 4242
+
+  # Debug true - Prints OpenTSDB communication
+  debug = false
+`
+
+type MetricLine struct {
+	Metric    string
+	Timestamp int64
+	Value     string
+	Tags      string
+}
+
+func (o *OpenTSDB) Connect() error {
+	// Test Connection to OpenTSDB Server
+	uri := fmt.Sprintf("%s:%d", o.Host, o.Port)
+	tcpAddr, err := net.ResolveTCPAddr("tcp", uri)
+	if err != nil {
+		return fmt.Errorf("OpenTSDB: TCP address cannot be resolved")
+	}
+	connection, err := net.DialTCP("tcp", nil, tcpAddr)
+	if err != nil {
+		return fmt.Errorf("OpenTSDB: Telnet connect fail")
+	}
+	defer connection.Close()
+	return nil
+}
+
+func (o *OpenTSDB) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+	now := time.Now()
+
+	// Send Data with telnet / socket communication
+	uri := fmt.Sprintf("%s:%d", o.Host, o.Port)
+	tcpAddr, _ := net.ResolveTCPAddr("tcp", uri)
+	connection, err := net.DialTCP("tcp", nil, tcpAddr)
+	if err != nil {
+		return fmt.Errorf("OpenTSDB: Telnet connect fail")
+	}
+	defer connection.Close()
+
+	for _, pt := range points {
+		for _, metric := range buildMetrics(pt, now, o.Prefix) {
+			messageLine := fmt.Sprintf("put %s %v %s %s\n",
+				metric.Metric, metric.Timestamp, metric.Value, metric.Tags)
+			if o.Debug {
+				fmt.Print(messageLine)
+			}
+			_, err := connection.Write([]byte(messageLine))
+			if err != nil {
+				return fmt.Errorf("OpenTSDB: Telnet writing error %s", err.Error())
+			}
+		}
+	}
+
+	return nil
+}
+
+func buildTags(ptTags map[string]string) []string {
+	tags := make([]string, len(ptTags))
+	index := 0
+	for k, v := range ptTags {
+		tags[index] = fmt.Sprintf("%s=%s", k, v)
+		index += 1
+	}
+	sort.Strings(tags)
+	return tags
+}
+
+func buildMetrics(pt *client.Point, now time.Time, prefix string) []*MetricLine {
+	ret := []*MetricLine{}
+	for fieldName, value := range pt.Fields() {
+		metric := &MetricLine{
+			Metric:    fmt.Sprintf("%s%s_%s", prefix, pt.Name(), fieldName),
+			Timestamp: now.Unix(),
+		}
+
+		metricValue, buildError := buildValue(value)
+		if buildError != nil {
+			fmt.Printf("OpenTSDB: %s\n", buildError.Error())
+			continue
+		}
+		metric.Value = metricValue
+		tagsSlice := buildTags(pt.Tags())
+		metric.Tags = fmt.Sprint(strings.Join(tagsSlice, " "))
+		ret = append(ret, metric)
+	}
+	return ret
+}
+
+func buildValue(v interface{}) (string, error) {
+	var retv string
+	switch p := v.(type) {
+	case int64:
+		retv = IntToString(int64(p))
+	case uint64:
+		retv = UIntToString(uint64(p))
+	case float64:
+		retv = FloatToString(float64(p))
+	default:
+		return retv, fmt.Errorf("unexpected type %T with value %v for OpenTSDB", v, v)
+	}
+	return retv, nil
+}
+
+func IntToString(input_num int64) string {
+	return strconv.FormatInt(input_num, 10)
+}
+
+func UIntToString(input_num uint64) string {
+	return strconv.FormatUint(input_num, 10)
+}
+
+func FloatToString(input_num float64) string {
+	return strconv.FormatFloat(input_num, 'f', 6, 64)
+}
+
+func (o *OpenTSDB) SampleConfig() string {
+	return sampleConfig
+}
+
+func (o *OpenTSDB) Description() string {
+	return "Configuration for OpenTSDB server to send metrics to"
+}
+
+func (o *OpenTSDB) Close() error {
+	return nil
+}
+
+func init() {
+	outputs.Add("opentsdb", func() outputs.Output {
+		return &OpenTSDB{}
+	})
+}

outputs/opentsdb/opentsdb_test.go

@@ -0,0 +1,71 @@
+package opentsdb
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestBuildTagsTelnet(t *testing.T) {
+	var tagtests = []struct {
+		ptIn    map[string]string
+		outTags []string
+	}{
+		{
+			map[string]string{"one": "two", "three": "four"},
+			[]string{"one=two", "three=four"},
+		},
+		{
+			map[string]string{"aaa": "bbb"},
+			[]string{"aaa=bbb"},
+		},
+		{
+			map[string]string{"one": "two", "aaa": "bbb"},
+			[]string{"aaa=bbb", "one=two"},
+		},
+		{
+			map[string]string{},
+			[]string{},
+		},
+	}
+	for _, tt := range tagtests {
+		tags := buildTags(tt.ptIn)
+		if !reflect.DeepEqual(tags, tt.outTags) {
+			t.Errorf("\nexpected %+v\ngot %+v\n", tt.outTags, tags)
+		}
+	}
+}
+
+func TestWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	o := &OpenTSDB{
+		Host:   testutil.GetLocalHost(),
+		Port:   4242,
+		Prefix: "prefix.test.",
+	}
+
+	// Verify that we can connect to the OpenTSDB instance
+	err := o.Connect()
+	require.NoError(t, err)
+
+	// Verify that we can successfully write data to OpenTSDB
+	err = o.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+
+	// Verify postive and negative test cases of writing data
+	bp := testutil.MockBatchPoints()
+	bp.AddPoint(testutil.TestPoint(float64(1.0), "justametric.float"))
+	bp.AddPoint(testutil.TestPoint(int64(123456789), "justametric.int"))
+	bp.AddPoint(testutil.TestPoint(uint64(123456789012345), "justametric.uint"))
+	bp.AddPoint(testutil.TestPoint("Lorem Ipsum", "justametric.string"))
+	bp.AddPoint(testutil.TestPoint(float64(42.0), "justametric.anotherfloat"))
+
+	err = o.Write(bp.Points())
+	require.NoError(t, err)
+
+}

outputs/prometheus_client/README.md

@@ -0,0 +1,6 @@
+# Prometheus Client Service Output Plugin
+
+This plugin starts a Prometheus Client, listening on a port defined in the
+configuration file.
+
+It exposes all metrics on `/metrics` to be polled by a Prometheus server.

outputs/prometheus_client/prometheus_client.go

@@ -0,0 +1,125 @@
+package prometheus_client
+
+import (
+	"fmt"
+	"log"
+	"net/http"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+type PrometheusClient struct {
+	Listen  string
+	metrics map[string]*prometheus.UntypedVec
+}
+
+var sampleConfig = `
+  # Address to listen on
+  # listen = ":9126"
+`
+
+func (p *PrometheusClient) Start() error {
+	if p.Listen == "" {
+		p.Listen = "localhost:9126"
+	}
+
+	http.Handle("/metrics", prometheus.Handler())
+	server := &http.Server{
+		Addr: p.Listen,
+	}
+
+	p.metrics = make(map[string]*prometheus.UntypedVec)
+	go server.ListenAndServe()
+	return nil
+}
+
+func (p *PrometheusClient) Stop() {
+	// TODO: Use a listener for http.Server that counts active connections
+	//       that can be stopped and closed gracefully
+}
+
+func (p *PrometheusClient) Connect() error {
+	// This service output does not need to make any further connections
+	return nil
+}
+
+func (p *PrometheusClient) Close() error {
+	// This service output does not need to close any of its connections
+	return nil
+}
+
+func (p *PrometheusClient) SampleConfig() string {
+	return sampleConfig
+}
+
+func (p *PrometheusClient) Description() string {
+	return "Configuration for the Prometheus client to spawn"
+}
+
+func (p *PrometheusClient) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	for _, point := range points {
+		var labels []string
+		key := point.Name()
+
+		for k, _ := range point.Tags() {
+			if len(k) > 0 {
+				labels = append(labels, k)
+			}
+		}
+
+		if _, ok := p.metrics[key]; !ok {
+			p.metrics[key] = prometheus.NewUntypedVec(
+				prometheus.UntypedOpts{
+					Name: key,
+					Help: fmt.Sprintf("Telegraf collected point '%s'", key),
+				},
+				labels,
+			)
+			prometheus.MustRegister(p.metrics[key])
+		}
+
+		l := prometheus.Labels{}
+		for tk, tv := range point.Tags() {
+			l[tk] = tv
+		}
+
+		for _, val := range point.Fields() {
+			switch val := val.(type) {
+			default:
+				log.Printf("Prometheus output, unsupported type. key: %s, type: %T\n",
+					key, val)
+			case int64:
+				m, err := p.metrics[key].GetMetricWith(l)
+				if err != nil {
+					log.Printf("ERROR Getting metric in Prometheus output, "+
+						"key: %s, labels: %v,\nerr: %s\n",
+						key, l, err.Error())
+					continue
+				}
+				m.Set(float64(val))
+			case float64:
+				m, err := p.metrics[key].GetMetricWith(l)
+				if err != nil {
+					log.Printf("ERROR Getting metric in Prometheus output, "+
+						"key: %s, labels: %v,\nerr: %s\n",
+						key, l, err.Error())
+					continue
+				}
+				m.Set(val)
+			}
+		}
+	}
+	return nil
+}
+
+func init() {
+	outputs.Add("prometheus_client", func() outputs.Output {
+		return &PrometheusClient{}
+	})
+}

outputs/prometheus_client/prometheus_client_test.go

@@ -0,0 +1,98 @@
+package prometheus_client
+
+import (
+	"testing"
+
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/plugins/prometheus"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var pTesting *PrometheusClient
+
+func TestPrometheusWritePointEmptyTag(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	p := &prometheus.Prometheus{
+		Urls: []string{"http://localhost:9126/metrics"},
+	}
+	tags := make(map[string]string)
+	pt1, _ := client.NewPoint(
+		"test_point_1",
+		tags,
+		map[string]interface{}{"value": 0.0})
+	pt2, _ := client.NewPoint(
+		"test_point_2",
+		tags,
+		map[string]interface{}{"value": 1.0})
+	var points = []*client.Point{
+		pt1,
+		pt2,
+	}
+	require.NoError(t, pTesting.Write(points))
+
+	expected := []struct {
+		name  string
+		value float64
+		tags  map[string]string
+	}{
+		{"test_point_1", 0.0, tags},
+		{"test_point_2", 1.0, tags},
+	}
+
+	var acc testutil.Accumulator
+
+	require.NoError(t, p.Gather(&acc))
+	for _, e := range expected {
+		assert.NoError(t, acc.ValidateValue(e.name, e.value))
+	}
+}
+
+func TestPrometheusWritePointTag(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	p := &prometheus.Prometheus{
+		Urls: []string{"http://localhost:9126/metrics"},
+	}
+	tags := make(map[string]string)
+	tags["testtag"] = "testvalue"
+	pt1, _ := client.NewPoint(
+		"test_point_3",
+		tags,
+		map[string]interface{}{"value": 0.0})
+	pt2, _ := client.NewPoint(
+		"test_point_4",
+		tags,
+		map[string]interface{}{"value": 1.0})
+	var points = []*client.Point{
+		pt1,
+		pt2,
+	}
+	require.NoError(t, pTesting.Write(points))
+
+	expected := []struct {
+		name  string
+		value float64
+	}{
+		{"test_point_3", 0.0},
+		{"test_point_4", 1.0},
+	}
+
+	var acc testutil.Accumulator
+
+	require.NoError(t, p.Gather(&acc))
+	for _, e := range expected {
+		assert.True(t, acc.CheckTaggedValue(e.name, e.value, tags))
+	}
+}
+
+func init() {
+	pTesting = &PrometheusClient{Listen: "localhost:9126"}
+	pTesting.Start()
+}

outputs/registry.go

@@ -0,0 +1,43 @@
+package outputs
+
+import (
+	"github.com/influxdb/influxdb/client/v2"
+)
+
+type Output interface {
+	// Connect to the Output
+	Connect() error
+	// Close any connections to the Output
+	Close() error
+	// Description returns a one-sentence description on the Output
+	Description() string
+	// SampleConfig returns the default configuration of the Output
+	SampleConfig() string
+	// Write takes in group of points to be written to the Output
+	Write(points []*client.Point) error
+}
+
+type ServiceOutput interface {
+	// Connect to the Output
+	Connect() error
+	// Close any connections to the Output
+	Close() error
+	// Description returns a one-sentence description on the Output
+	Description() string
+	// SampleConfig returns the default configuration of the Output
+	SampleConfig() string
+	// Write takes in group of points to be written to the Output
+	Write(points []*client.Point) error
+	// Start the "service" that will provide an Output
+	Start() error
+	// Stop the "service" that will provide an Output
+	Stop()
+}
+
+type Creator func() Output
+
+var Outputs = map[string]Creator{}
+
+func Add(name string, creator Creator) {
+	Outputs[name] = creator
+}

outputs/riemann/riemann.go

@@ -0,0 +1,101 @@
+package riemann
+
+import (
+	"errors"
+	"fmt"
+	"os"
+
+	"github.com/amir/raidman"
+	"github.com/influxdb/influxdb/client/v2"
+	"github.com/influxdb/telegraf/outputs"
+)
+
+type Riemann struct {
+	URL       string
+	Transport string
+
+	client *raidman.Client
+}
+
+var sampleConfig = `
+  # URL of server
+  url = "localhost:5555"
+  # transport protocol to use either tcp or udp
+  transport = "tcp"
+`
+
+func (r *Riemann) Connect() error {
+	c, err := raidman.Dial(r.Transport, r.URL)
+
+	if err != nil {
+		return err
+	}
+
+	r.client = c
+	return nil
+}
+
+func (r *Riemann) Close() error {
+	r.client.Close()
+	return nil
+}
+
+func (r *Riemann) SampleConfig() string {
+	return sampleConfig
+}
+
+func (r *Riemann) Description() string {
+	return "Configuration for the Riemann server to send metrics to"
+}
+
+func (r *Riemann) Write(points []*client.Point) error {
+	if len(points) == 0 {
+		return nil
+	}
+
+	var events []*raidman.Event
+	for _, p := range points {
+		evs := buildEvents(p)
+		for _, ev := range evs {
+			events = append(events, ev)
+		}
+	}
+
+	var senderr = r.client.SendMulti(events)
+	if senderr != nil {
+		return errors.New(fmt.Sprintf("FAILED to send riemann message: %s\n",
+			senderr))
+	}
+
+	return nil
+}
+
+func buildEvents(p *client.Point) []*raidman.Event {
+	events := []*raidman.Event{}
+	for fieldName, value := range p.Fields() {
+		host, ok := p.Tags()["host"]
+		if !ok {
+			hostname, err := os.Hostname()
+			if err != nil {
+				host = "unknown"
+			} else {
+				host = hostname
+			}
+		}
+
+		event := &raidman.Event{
+			Host:    host,
+			Service: p.Name() + "_" + fieldName,
+			Metric:  value,
+		}
+		events = append(events, event)
+	}
+
+	return events
+}
+
+func init() {
+	outputs.Add("riemann", func() outputs.Output {
+		return &Riemann{}
+	})
+}

outputs/riemann/riemann_test.go

@@ -0,0 +1,27 @@
+package riemann
+
+import (
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestConnectAndWrite(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	url := testutil.GetLocalHost() + ":5555"
+
+	r := &Riemann{
+		URL:       url,
+		Transport: "tcp",
+	}
+
+	err := r.Connect()
+	require.NoError(t, err)
+
+	err = r.Write(testutil.MockBatchPoints().Points())
+	require.NoError(t, err)
+}

plugins/aerospike/README.md

@@ -0,0 +1,265 @@
+## Telegraf Plugin: Aerospike
+
+#### Plugin arguments:
+- **servers** string array: List of aerospike servers to query (def: 127.0.0.1:3000)
+
+#### Description
+
+The aerospike plugin queries aerospike server(s) and get node statistics. It also collects stats for
+all the configured namespaces.
+
+For what the measurements mean, please consult the [Aerospike Metrics Reference Docs](http://www.aerospike.com/docs/reference/metrics).
+
+The metric names, to make it less complicated in querying, have replaced all `-` with `_` as Aerospike metrics come in both forms (no idea why).
+
+# Measurements:
+#### Aerospike Statistics [values]:
+
+Meta:
+- units: Integer
+
+Measurement names:
+- batch_index_queue
+- batch_index_unused_buffers
+- batch_queue
+- batch_tree_count
+- client_connections
+- data_used_bytes_memory
+- index_used_bytes_memory
+- info_queue
+- migrate_progress_recv
+- migrate_progress_send
+- migrate_rx_objs
+- migrate_tx_objs
+- objects
+- ongoing_write_reqs
+- partition_absent
+- partition_actual
+- partition_desync
+- partition_object_count
+- partition_ref_count
+- partition_replica
+- proxy_in_progress
+- query_agg_avg_rec_count
+- query_avg_rec_count
+- query_lookup_avg_rec_count
+- queue
+- record_locks
+- record_refs
+- sindex_used_bytes_memory
+- sindex_gc_garbage_cleaned
+- system_free_mem_pct
+- total_bytes_disk
+- total_bytes_memory
+- tree_count
+- scans_active
+- uptime
+- used_bytes_disk
+- used_bytes_memory
+- cluster_size
+- waiting_transactions
+
+#### Aerospike Statistics [cumulative]:
+
+Meta:
+- units: Integer
+
+Measurement names:
+- batch_errors
+- batch_index_complete
+- batch_index_errors
+- batch_index_initiate
+- batch_index_timeout
+- batch_initiate
+- batch_timeout
+- err_duplicate_proxy_request
+- err_out_of_space
+- err_replica_non_null_node
+- err_replica_null_node
+- err_rw_cant_put_unique
+- err_rw_pending_limit
+- err_rw_request_not_found
+- err_storage_queue_full
+- err_sync_copy_null_master
+- err_sync_copy_null_node
+- err_tsvc_requests
+- err_write_fail_bin_exists
+- err_write_fail_generation
+- err_write_fail_generation_xdr
+- err_write_fail_incompatible_type
+- err_write_fail_key_exists
+- err_write_fail_key_mismatch
+- err_write_fail_not_found
+- err_write_fail_noxdr
+- err_write_fail_parameter
+- err_write_fail_prole_delete
+- err_write_fail_prole_generation
+- err_write_fail_prole_unknown
+- err_write_fail_unknown
+- fabric_msgs_rcvd
+- fabric_msgs_sent
+- heartbeat_received_foreign
+- heartbeat_received_self
+- migrate_msgs_recv
+- migrate_msgs_sent
+- migrate_num_incoming_accepted
+- migrate_num_incoming_refused
+- proxy_action
+- proxy_initiate
+- proxy_retry
+- proxy_retry_new_dest
+- proxy_retry_q_full
+- proxy_retry_same_dest
+- proxy_unproxy
+- query_abort
+- query_agg
+- query_agg_abort
+- query_agg_err
+- query_agg_success
+- query_bad_records
+- query_fail
+- query_long_queue_full
+- query_long_running
+- query_lookup_abort
+- query_lookup_err
+- query_lookups
+- query_lookup_success
+- query_reqs
+- query_short_queue_full
+- query_short_running
+- query_success
+- query_tracked
+- read_dup_prole
+- reaped_fds
+- rw_err_ack_badnode
+- rw_err_ack_internal
+- rw_err_ack_nomatch
+- rw_err_dup_cluster_key
+- rw_err_dup_internal
+- rw_err_dup_send
+- rw_err_write_cluster_key
+- rw_err_write_internal
+- rw_err_write_send
+- sindex_ucgarbage_found
+- sindex_gc_locktimedout
+- sindex_gc_inactivity_dur
+- sindex_gc_activity_dur
+- sindex_gc_list_creation_time
+- sindex_gc_list_deletion_time
+- sindex_gc_objects_validated
+- sindex_gc_garbage_found
+- stat_cluster_key_err_ack_dup_trans_reenqueue
+- stat_cluster_key_err_ack_rw_trans_reenqueue
+- stat_cluster_key_prole_retry
+- stat_cluster_key_regular_processed
+- stat_cluster_key_trans_to_proxy_retry
+- stat_deleted_set_object
+- stat_delete_success
+- stat_duplicate_operation
+- stat_evicted_objects
+- stat_evicted_objects_time
+- stat_evicted_set_objects
+- stat_expired_objects
+- stat_nsup_deletes_not_shipped
+- stat_proxy_errs
+- stat_proxy_reqs
+- stat_proxy_reqs_xdr
+- stat_proxy_success
+- stat_read_errs_notfound
+- stat_read_errs_other
+- stat_read_reqs
+- stat_read_reqs_xdr
+- stat_read_success
+- stat_rw_timeout
+- stat_slow_trans_queue_batch_pop
+- stat_slow_trans_queue_pop
+- stat_slow_trans_queue_push
+- stat_write_errs
+- stat_write_errs_notfound
+- stat_write_errs_other
+- stat_write_reqs
+- stat_write_reqs_xdr
+- stat_write_success
+- stat_xdr_pipe_miss
+- stat_xdr_pipe_writes
+- stat_zero_bin_records
+- storage_defrag_corrupt_record
+- storage_defrag_wait
+- transactions
+- basic_scans_succeeded
+- basic_scans_failed
+- aggr_scans_succeeded
+- aggr_scans_failed
+- udf_bg_scans_succeeded
+- udf_bg_scans_failed
+- udf_delete_err_others
+- udf_delete_reqs
+- udf_delete_success
+- udf_lua_errs
+- udf_query_rec_reqs
+- udf_read_errs_other
+- udf_read_reqs
+- udf_read_success
+- udf_replica_writes
+- udf_scan_rec_reqs
+- udf_write_err_others
+- udf_write_reqs
+- udf_write_success
+- write_master
+- write_prole
+
+#### Aerospike Statistics [percentage]:
+
+Meta:
+- units: percent (out of 100)
+
+Measurement names:
+- free_pct_disk
+- free_pct_memory
+
+# Measurements:
+#### Aerospike Namespace Statistics [values]:
+
+Meta:
+- units: Integer
+- tags: `namespace=<namespace>`
+
+Measurement names:
+- available_bin_names
+- available_pct
+- current_time
+- data_used_bytes_memory
+- index_used_bytes_memory
+- master_objects
+- max_evicted_ttl
+- max_void_time
+- non_expirable_objects
+- objects
+- prole_objects
+- sindex_used_bytes_memory
+- total_bytes_disk
+- total_bytes_memory
+- used_bytes_disk
+- used_bytes_memory
+
+#### Aerospike Namespace Statistics [cumulative]:
+
+Meta:
+- units: Integer
+- tags: `namespace=<namespace>`
+
+Measurement names:
+- evicted_objects
+- expired_objects
+- set_deleted_objects
+- set_evicted_objects
+
+#### Aerospike Namespace Statistics [percentage]:
+
+Meta:
+- units: percent (out of 100)
+- tags: `namespace=<namespace>`
+
+Measurement names:
+- free_pct_disk
+- free_pct_memory

plugins/aerospike/aerospike.go

@@ -0,0 +1,342 @@
+package aerospike
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"github.com/influxdb/telegraf/plugins"
+	"net"
+	"strconv"
+	"strings"
+	"sync"
+)
+
+const (
+	MSG_HEADER_SIZE = 8
+	MSG_TYPE        = 1 // Info is 1
+	MSG_VERSION     = 2
+)
+
+var (
+	STATISTICS_COMMAND = []byte("statistics\n")
+	NAMESPACES_COMMAND = []byte("namespaces\n")
+)
+
+type aerospikeMessageHeader struct {
+	Version uint8
+	Type    uint8
+	DataLen [6]byte
+}
+
+type aerospikeMessage struct {
+	aerospikeMessageHeader
+	Data []byte
+}
+
+// Taken from aerospike-client-go/types/message.go
+func (msg *aerospikeMessage) Serialize() []byte {
+	msg.DataLen = msgLenToBytes(int64(len(msg.Data)))
+	buf := bytes.NewBuffer([]byte{})
+	binary.Write(buf, binary.BigEndian, msg.aerospikeMessageHeader)
+	binary.Write(buf, binary.BigEndian, msg.Data[:])
+	return buf.Bytes()
+}
+
+type aerospikeInfoCommand struct {
+	msg *aerospikeMessage
+}
+
+// Taken from aerospike-client-go/info.go
+func (nfo *aerospikeInfoCommand) parseMultiResponse() (map[string]string, error) {
+	responses := make(map[string]string)
+	offset := int64(0)
+	begin := int64(0)
+
+	dataLen := int64(len(nfo.msg.Data))
+
+	// Create reusable StringBuilder for performance.
+	for offset < dataLen {
+		b := nfo.msg.Data[offset]
+
+		if b == '\t' {
+			name := nfo.msg.Data[begin:offset]
+			offset++
+			begin = offset
+
+			// Parse field value.
+			for offset < dataLen {
+				if nfo.msg.Data[offset] == '\n' {
+					break
+				}
+				offset++
+			}
+
+			if offset > begin {
+				value := nfo.msg.Data[begin:offset]
+				responses[string(name)] = string(value)
+			} else {
+				responses[string(name)] = ""
+			}
+			offset++
+			begin = offset
+		} else if b == '\n' {
+			if offset > begin {
+				name := nfo.msg.Data[begin:offset]
+				responses[string(name)] = ""
+			}
+			offset++
+			begin = offset
+		} else {
+			offset++
+		}
+	}
+
+	if offset > begin {
+		name := nfo.msg.Data[begin:offset]
+		responses[string(name)] = ""
+	}
+	return responses, nil
+}
+
+type Aerospike struct {
+	Servers []string
+}
+
+var sampleConfig = `
+  # Aerospike servers to connect to (with port)
+  # Default: servers = ["localhost:3000"]
+  #
+  # This plugin will query all namespaces the aerospike
+  # server has configured and get stats for them.
+  servers = ["localhost:3000"]
+ `
+
+func (a *Aerospike) SampleConfig() string {
+	return sampleConfig
+}
+
+func (a *Aerospike) Description() string {
+	return "Read stats from an aerospike server"
+}
+
+func (a *Aerospike) Gather(acc plugins.Accumulator) error {
+	if len(a.Servers) == 0 {
+		return a.gatherServer("127.0.0.1:3000", acc)
+	}
+
+	var wg sync.WaitGroup
+
+	var outerr error
+
+	for _, server := range a.Servers {
+		wg.Add(1)
+		go func(server string) {
+			defer wg.Done()
+			outerr = a.gatherServer(server, acc)
+		}(server)
+	}
+
+	wg.Wait()
+	return outerr
+}
+
+func (a *Aerospike) gatherServer(host string, acc plugins.Accumulator) error {
+	aerospikeInfo, err := getMap(STATISTICS_COMMAND, host)
+	if err != nil {
+		return fmt.Errorf("Aerospike info failed: %s", err)
+	}
+	readAerospikeStats(aerospikeInfo, acc, host, "")
+	namespaces, err := getList(NAMESPACES_COMMAND, host)
+	if err != nil {
+		return fmt.Errorf("Aerospike namespace list failed: %s", err)
+	}
+	for ix := range namespaces {
+		nsInfo, err := getMap([]byte("namespace/"+namespaces[ix]+"\n"), host)
+		if err != nil {
+			return fmt.Errorf("Aerospike namespace '%s' query failed: %s", namespaces[ix], err)
+		}
+		readAerospikeStats(nsInfo, acc, host, namespaces[ix])
+	}
+	return nil
+}
+
+func getMap(key []byte, host string) (map[string]string, error) {
+	data, err := get(key, host)
+	if err != nil {
+		return nil, fmt.Errorf("Failed to get data: %s", err)
+	}
+	parsed, err := unmarshalMapInfo(data, string(key))
+	if err != nil {
+		return nil, fmt.Errorf("Failed to unmarshal data: %s", err)
+	}
+
+	return parsed, nil
+}
+
+func getList(key []byte, host string) ([]string, error) {
+	data, err := get(key, host)
+	if err != nil {
+		return nil, fmt.Errorf("Failed to get data: %s", err)
+	}
+	parsed, err := unmarshalListInfo(data, string(key))
+	if err != nil {
+		return nil, fmt.Errorf("Failed to unmarshal data: %s", err)
+	}
+
+	return parsed, nil
+}
+
+func get(key []byte, host string) (map[string]string, error) {
+	var err error
+	var data map[string]string
+
+	asInfo := &aerospikeInfoCommand{
+		msg: &aerospikeMessage{
+			aerospikeMessageHeader: aerospikeMessageHeader{
+				Version: uint8(MSG_VERSION),
+				Type:    uint8(MSG_TYPE),
+				DataLen: msgLenToBytes(int64(len(key))),
+			},
+			Data: key,
+		},
+	}
+
+	cmd := asInfo.msg.Serialize()
+	addr, err := net.ResolveTCPAddr("tcp", host)
+	if err != nil {
+		return data, fmt.Errorf("Lookup failed for '%s': %s", host, err)
+	}
+
+	conn, err := net.DialTCP("tcp", nil, addr)
+	if err != nil {
+		return data, fmt.Errorf("Connection failed for '%s': %s", host, err)
+	}
+	defer conn.Close()
+
+	_, err = conn.Write(cmd)
+	if err != nil {
+		return data, fmt.Errorf("Failed to send to '%s': %s", host, err)
+	}
+
+	msgHeader := bytes.NewBuffer(make([]byte, MSG_HEADER_SIZE))
+	_, err = readLenFromConn(conn, msgHeader.Bytes(), MSG_HEADER_SIZE)
+	if err != nil {
+		return data, fmt.Errorf("Failed to read header: %s", err)
+	}
+	err = binary.Read(msgHeader, binary.BigEndian, &asInfo.msg.aerospikeMessageHeader)
+	if err != nil {
+		return data, fmt.Errorf("Failed to unmarshal header: %s", err)
+	}
+
+	msgLen := msgLenFromBytes(asInfo.msg.aerospikeMessageHeader.DataLen)
+
+	if int64(len(asInfo.msg.Data)) != msgLen {
+		asInfo.msg.Data = make([]byte, msgLen)
+	}
+
+	_, err = readLenFromConn(conn, asInfo.msg.Data, len(asInfo.msg.Data))
+	if err != nil {
+		return data, fmt.Errorf("Failed to read from connection to '%s': %s", host, err)
+	}
+
+	data, err = asInfo.parseMultiResponse()
+	if err != nil {
+		return data, fmt.Errorf("Failed to parse response from '%s': %s", host, err)
+	}
+
+	return data, err
+}
+
+func readAerospikeStats(
+	stats map[string]string,
+	acc plugins.Accumulator,
+	host string,
+	namespace string,
+) {
+	fields := make(map[string]interface{})
+	tags := map[string]string{
+		"aerospike_host": host,
+		"namespace":      "_service",
+	}
+
+	if namespace != "" {
+		tags["namespace"] = namespace
+	}
+	for key, value := range stats {
+		// We are going to ignore all string based keys
+		val, err := strconv.ParseInt(value, 10, 64)
+		if err == nil {
+			if strings.Contains(key, "-") {
+				key = strings.Replace(key, "-", "_", -1)
+			}
+			fields[key] = val
+		}
+	}
+	acc.AddFields("aerospike", fields, tags)
+}
+
+func unmarshalMapInfo(infoMap map[string]string, key string) (map[string]string, error) {
+	key = strings.TrimSuffix(key, "\n")
+	res := map[string]string{}
+
+	v, exists := infoMap[key]
+	if !exists {
+		return res, fmt.Errorf("Key '%s' missing from info", key)
+	}
+
+	values := strings.Split(v, ";")
+	for i := range values {
+		kv := strings.Split(values[i], "=")
+		if len(kv) > 1 {
+			res[kv[0]] = kv[1]
+		}
+	}
+
+	return res, nil
+}
+
+func unmarshalListInfo(infoMap map[string]string, key string) ([]string, error) {
+	key = strings.TrimSuffix(key, "\n")
+
+	v, exists := infoMap[key]
+	if !exists {
+		return []string{}, fmt.Errorf("Key '%s' missing from info", key)
+	}
+
+	values := strings.Split(v, ";")
+	return values, nil
+}
+
+func readLenFromConn(c net.Conn, buffer []byte, length int) (total int, err error) {
+	var r int
+	for total < length {
+		r, err = c.Read(buffer[total:length])
+		total += r
+		if err != nil {
+			break
+		}
+	}
+	return
+}
+
+// Taken from aerospike-client-go/types/message.go
+func msgLenToBytes(DataLen int64) [6]byte {
+	b := make([]byte, 8)
+	binary.BigEndian.PutUint64(b, uint64(DataLen))
+	res := [6]byte{}
+	copy(res[:], b[2:])
+	return res
+}
+
+// Taken from aerospike-client-go/types/message.go
+func msgLenFromBytes(buf [6]byte) int64 {
+	nbytes := append([]byte{0, 0}, buf[:]...)
+	DataLen := binary.BigEndian.Uint64(nbytes)
+	return int64(DataLen)
+}
+
+func init() {
+	plugins.Add("aerospike", func() plugins.Plugin {
+		return &Aerospike{}
+	})
+}

plugins/aerospike/aerospike_test.go

@@ -0,0 +1,112 @@
+package aerospike
+
+import (
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"reflect"
+	"testing"
+)
+
+func TestAerospikeStatistics(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	a := &Aerospike{
+		Servers: []string{testutil.GetLocalHost() + ":3000"},
+	}
+
+	var acc testutil.Accumulator
+
+	err := a.Gather(&acc)
+	require.NoError(t, err)
+
+	// Only use a few of the metrics
+	asMetrics := []string{
+		"transactions",
+		"stat_write_errs",
+		"stat_read_reqs",
+		"stat_write_reqs",
+	}
+
+	for _, metric := range asMetrics {
+		assert.True(t, acc.HasIntValue(metric), metric)
+	}
+
+}
+
+func TestAerospikeMsgLenFromToBytes(t *testing.T) {
+	var i int64 = 8
+	assert.True(t, i == msgLenFromBytes(msgLenToBytes(i)))
+}
+
+func TestReadAerospikeStatsNoNamespace(t *testing.T) {
+	// Also test for re-writing
+	var acc testutil.Accumulator
+	stats := map[string]string{
+		"stat-write-errs": "12345",
+		"stat_read_reqs":  "12345",
+	}
+	readAerospikeStats(stats, &acc, "host1", "")
+	for k := range stats {
+		if k == "stat-write-errs" {
+			k = "stat_write_errs"
+		}
+		assert.True(t, acc.HasMeasurement(k))
+		assert.True(t, acc.CheckValue(k, int64(12345)))
+	}
+}
+
+func TestReadAerospikeStatsNamespace(t *testing.T) {
+	var acc testutil.Accumulator
+	stats := map[string]string{
+		"stat_write_errs": "12345",
+		"stat_read_reqs":  "12345",
+	}
+	readAerospikeStats(stats, &acc, "host1", "test")
+
+	tags := map[string]string{
+		"aerospike_host": "host1",
+		"namespace":      "test",
+	}
+	for k := range stats {
+		assert.True(t, acc.ValidateTaggedValue(k, int64(12345), tags) == nil)
+	}
+}
+
+func TestAerospikeUnmarshalList(t *testing.T) {
+	i := map[string]string{
+		"test": "one;two;three",
+	}
+
+	expected := []string{"one", "two", "three"}
+
+	list, err := unmarshalListInfo(i, "test2")
+	assert.True(t, err != nil)
+
+	list, err = unmarshalListInfo(i, "test")
+	assert.True(t, err == nil)
+	equal := true
+	for ix := range expected {
+		if list[ix] != expected[ix] {
+			equal = false
+			break
+		}
+	}
+	assert.True(t, equal)
+}
+
+func TestAerospikeUnmarshalMap(t *testing.T) {
+	i := map[string]string{
+		"test": "key1=value1;key2=value2",
+	}
+
+	expected := map[string]string{
+		"key1": "value1",
+		"key2": "value2",
+	}
+	m, err := unmarshalMapInfo(i, "test")
+	assert.True(t, err == nil)
+	assert.True(t, reflect.DeepEqual(m, expected))
+}

plugins/all/all.go

@@ -1,6 +1,37 @@
 package all
 
 import (
-	_ "github.com/influxdb/tivan/plugins/redis"
-	_ "github.com/influxdb/tivan/plugins/system"
+	_ "github.com/influxdb/telegraf/plugins/aerospike"
+	_ "github.com/influxdb/telegraf/plugins/apache"
+	_ "github.com/influxdb/telegraf/plugins/bcache"
+	_ "github.com/influxdb/telegraf/plugins/disque"
+	_ "github.com/influxdb/telegraf/plugins/elasticsearch"
+	_ "github.com/influxdb/telegraf/plugins/exec"
+	_ "github.com/influxdb/telegraf/plugins/haproxy"
+	_ "github.com/influxdb/telegraf/plugins/httpjson"
+	_ "github.com/influxdb/telegraf/plugins/influxdb"
+	_ "github.com/influxdb/telegraf/plugins/jolokia"
+	_ "github.com/influxdb/telegraf/plugins/kafka_consumer"
+	_ "github.com/influxdb/telegraf/plugins/leofs"
+	_ "github.com/influxdb/telegraf/plugins/lustre2"
+	_ "github.com/influxdb/telegraf/plugins/mailchimp"
+	_ "github.com/influxdb/telegraf/plugins/memcached"
+	_ "github.com/influxdb/telegraf/plugins/mongodb"
+	_ "github.com/influxdb/telegraf/plugins/mysql"
+	_ "github.com/influxdb/telegraf/plugins/nginx"
+	_ "github.com/influxdb/telegraf/plugins/phpfpm"
+	_ "github.com/influxdb/telegraf/plugins/ping"
+	_ "github.com/influxdb/telegraf/plugins/postgresql"
+	_ "github.com/influxdb/telegraf/plugins/procstat"
+	_ "github.com/influxdb/telegraf/plugins/prometheus"
+	_ "github.com/influxdb/telegraf/plugins/puppetagent"
+	_ "github.com/influxdb/telegraf/plugins/rabbitmq"
+	_ "github.com/influxdb/telegraf/plugins/redis"
+	_ "github.com/influxdb/telegraf/plugins/rethinkdb"
+	_ "github.com/influxdb/telegraf/plugins/statsd"
+	_ "github.com/influxdb/telegraf/plugins/system"
+	_ "github.com/influxdb/telegraf/plugins/trig"
+	_ "github.com/influxdb/telegraf/plugins/twemproxy"
+	_ "github.com/influxdb/telegraf/plugins/zfs"
+	_ "github.com/influxdb/telegraf/plugins/zookeeper"
 )

plugins/apache/README.md

@@ -0,0 +1,45 @@
+# Telegraf plugin: Apache
+
+#### Plugin arguments:
+- **urls** []string: List of apache-status URLs to collect from.
+
+#### Description
+
+The Apache plugin collects from the /server-status?auto URL. See
+[apache.org/server-status?auto](http://www.apache.org/server-status?auto) for an
+example. And
+[here](http://httpd.apache.org/docs/2.2/mod/mod_status.html) for the apache
+mod_status documentation.
+
+# Measurements:
+
+Meta:
+- tags: `port=<port>`, `server=url`
+
+- apache_TotalAccesses
+- apache_TotalkBytes
+- apache_CPULoad
+- apache_Uptime
+- apache_ReqPerSec
+- apache_BytesPerSec
+- apache_BytesPerReq
+- apache_BusyWorkers
+- apache_IdleWorkers
+- apache_ConnsTotal
+- apache_ConnsAsyncWriting
+- apache_ConnsAsyncKeepAlive
+- apache_ConnsAsyncClosing
+
+### Scoreboard measurements
+
+- apache_scboard_waiting
+- apache_scboard_starting
+- apache_scboard_reading
+- apache_scboard_sending
+- apache_scboard_keepalive
+- apache_scboard_dnslookup
+- apache_scboard_closing
+- apache_scboard_logging
+- apache_scboard_finishing
+- apache_scboard_idle_cleanup
+- apache_scboard_open

plugins/apache/apache.go

@@ -0,0 +1,170 @@
+package apache
+
+import (
+	"bufio"
+	"fmt"
+	"net"
+	"net/http"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Apache struct {
+	Urls []string
+}
+
+var sampleConfig = `
+  # An array of Apache status URI to gather stats.
+  urls = ["http://localhost/server-status?auto"]
+`
+
+func (n *Apache) SampleConfig() string {
+	return sampleConfig
+}
+
+func (n *Apache) Description() string {
+	return "Read Apache status information (mod_status)"
+}
+
+func (n *Apache) Gather(acc plugins.Accumulator) error {
+	var wg sync.WaitGroup
+	var outerr error
+
+	for _, u := range n.Urls {
+		addr, err := url.Parse(u)
+		if err != nil {
+			return fmt.Errorf("Unable to parse address '%s': %s", u, err)
+		}
+
+		wg.Add(1)
+		go func(addr *url.URL) {
+			defer wg.Done()
+			outerr = n.gatherUrl(addr, acc)
+		}(addr)
+	}
+
+	wg.Wait()
+
+	return outerr
+}
+
+var tr = &http.Transport{
+	ResponseHeaderTimeout: time.Duration(3 * time.Second),
+}
+
+var client = &http.Client{Transport: tr}
+
+func (n *Apache) gatherUrl(addr *url.URL, acc plugins.Accumulator) error {
+	resp, err := client.Get(addr.String())
+	if err != nil {
+		return fmt.Errorf("error making HTTP request to %s: %s", addr.String(), err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("%s returned HTTP status %s", addr.String(), resp.Status)
+	}
+
+	tags := getTags(addr)
+
+	sc := bufio.NewScanner(resp.Body)
+	fields := make(map[string]interface{})
+	for sc.Scan() {
+		line := sc.Text()
+		if strings.Contains(line, ":") {
+			parts := strings.SplitN(line, ":", 2)
+			key, part := strings.Replace(parts[0], " ", "", -1), strings.TrimSpace(parts[1])
+
+			switch key {
+			case "Scoreboard":
+				for field, value := range n.gatherScores(part) {
+					fields[field] = value
+				}
+			default:
+				value, err := strconv.ParseFloat(part, 64)
+				if err != nil {
+					continue
+				}
+				fields[key] = value
+			}
+		}
+	}
+	acc.AddFields("apache", fields, tags)
+
+	return nil
+}
+
+func (n *Apache) gatherScores(data string) map[string]interface{} {
+	var waiting, open int = 0, 0
+	var S, R, W, K, D, C, L, G, I int = 0, 0, 0, 0, 0, 0, 0, 0, 0
+
+	for _, s := range strings.Split(data, "") {
+
+		switch s {
+		case "_":
+			waiting++
+		case "S":
+			S++
+		case "R":
+			R++
+		case "W":
+			W++
+		case "K":
+			K++
+		case "D":
+			D++
+		case "C":
+			C++
+		case "L":
+			L++
+		case "G":
+			G++
+		case "I":
+			I++
+		case ".":
+			open++
+		}
+	}
+
+	fields := map[string]interface{}{
+		"scboard_waiting":      float64(waiting),
+		"scboard_starting":     float64(S),
+		"scboard_reading":      float64(R),
+		"scboard_sending":      float64(W),
+		"scboard_keepalive":    float64(K),
+		"scboard_dnslookup":    float64(D),
+		"scboard_closing":      float64(C),
+		"scboard_logging":      float64(L),
+		"scboard_finishing":    float64(G),
+		"scboard_idle_cleanup": float64(I),
+		"scboard_open":         float64(open),
+	}
+	return fields
+}
+
+// Get tag(s) for the apache plugin
+func getTags(addr *url.URL) map[string]string {
+	h := addr.Host
+	host, port, err := net.SplitHostPort(h)
+	if err != nil {
+		host = addr.Host
+		if addr.Scheme == "http" {
+			port = "80"
+		} else if addr.Scheme == "https" {
+			port = "443"
+		} else {
+			port = ""
+		}
+	}
+	return map[string]string{"server": host, "port": port}
+}
+
+func init() {
+	plugins.Add("apache", func() plugins.Plugin {
+		return &Apache{}
+	})
+}

plugins/apache/apache_test.go

@@ -0,0 +1,80 @@
+package apache
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+var apacheStatus = `
+Total Accesses: 129811861
+Total kBytes: 5213701865
+CPULoad: 6.51929
+Uptime: 941553
+ReqPerSec: 137.87
+BytesPerSec: 5670240
+BytesPerReq: 41127.4
+BusyWorkers: 270
+IdleWorkers: 630
+ConnsTotal: 1451
+ConnsAsyncWriting: 32
+ConnsAsyncKeepAlive: 945
+ConnsAsyncClosing: 205
+Scoreboard: WW_____W_RW_R_W__RRR____WR_W___WW________W_WW_W_____R__R_WR__WRWR_RRRW___R_RWW__WWWRW__R_RW___RR_RW_R__W__WR_WWW______WWR__R___R_WR_W___RW______RR________________W______R__RR______W________________R____R__________________________RW_W____R_____W_R_________________R____RR__W___R_R____RW______R____W______W_W_R_R______R__R_R__________R____W_______WW____W____RR__W_____W_R_______W__________W___W____________W_______WRR_R_W____W_____R____W_WW_R____RRW__W............................................................................................................................................................................................................................................................................................................WRRWR____WR__RR_R___RWR_________W_R____RWRRR____R_R__RW_R___WWW_RW__WR_RRR____W___R____WW_R__R___RR_W_W_RRRRWR__RRWR__RRW_W_RRRW_R_RR_W__RR_RWRR_R__R___RR_RR______R__RR____R_____W_R_R_R__R__R__________W____WW_R___R_R___R_________RR__RR____RWWWW___W_R________R_R____R_W___W___R___W_WRRWW_______R__W_RW_______R________RR__R________W_______________________W_W______________RW_________WR__R___R__R_______________WR_R_________W___RW_____R____________W____......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
+`
+
+func TestHTTPApache(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		fmt.Fprintln(w, apacheStatus)
+	}))
+	defer ts.Close()
+
+	a := Apache{
+		Urls: []string{ts.URL},
+	}
+
+	var acc testutil.Accumulator
+	err := a.Gather(&acc)
+	require.NoError(t, err)
+
+	testInt := []struct {
+		measurement string
+		value       float64
+	}{
+		{"TotalAccesses", 1.29811861e+08},
+		{"TotalkBytes", 5.213701865e+09},
+		{"CPULoad", 6.51929},
+		{"Uptime", 941553},
+		{"ReqPerSec", 137.87},
+		{"BytesPerSec", 5.67024e+06},
+		{"BytesPerReq", 41127.4},
+		{"BusyWorkers", 270},
+		{"IdleWorkers", 630},
+		{"ConnsTotal", 1451},
+		{"ConnsAsyncWriting", 32},
+		{"ConnsAsyncKeepAlive", 945},
+		{"ConnsAsyncClosing", 205},
+		{"scboard_waiting", 630},
+		{"scboard_starting", 0},
+		{"scboard_reading", 157},
+		{"scboard_sending", 113},
+		{"scboard_keepalive", 0},
+		{"scboard_dnslookup", 0},
+		{"scboard_closing", 0},
+		{"scboard_logging", 0},
+		{"scboard_finishing", 0},
+		{"scboard_idle_cleanup", 0},
+		{"scboard_open", 2850},
+	}
+
+	for _, test := range testInt {
+		assert.True(t, acc.CheckValue(test.measurement, test.value))
+	}
+}

plugins/bcache/README.md

@@ -0,0 +1,89 @@
+# Telegraf plugin: bcache
+
+Get bcache stat from stats_total directory and dirty_data file.
+
+# Measurements
+
+Meta:
+
+- tags: `backing_dev=dev bcache_dev=dev`
+
+Measurement names:
+
+- dirty_data
+- bypassed
+- cache_bypass_hits
+- cache_bypass_misses
+- cache_hit_ratio
+- cache_hits
+- cache_miss_collisions
+- cache_misses
+- cache_readaheads
+
+### Description
+
+```
+dirty_data
+  Amount of dirty data for this backing device in the cache. Continuously
+  updated unlike the cache set's version, but may be slightly off.
+  
+bypassed
+  Amount of IO (both reads and writes) that has bypassed the cache
+  
+
+cache_bypass_hits
+cache_bypass_misses
+  Hits and misses for IO that is intended to skip the cache are still counted,
+  but broken out here.
+  
+cache_hits
+cache_misses
+cache_hit_ratio
+  Hits and misses are counted per individual IO as bcache sees them; a
+  partial hit is counted as a miss.
+  
+cache_miss_collisions
+  Counts instances where data was going to be inserted into the cache from a
+  cache miss, but raced with a write and data was already present (usually 0
+  since the synchronization for cache misses was rewritten)
+  
+cache_readaheads
+  Count of times readahead occurred.
+```
+
+# Example output
+
+Using this configuration:
+
+```
+[bcache]
+  # Bcache sets path
+  # If not specified, then default is:
+  # bcachePath = "/sys/fs/bcache"
+  #
+  # By default, telegraf gather stats for all bcache devices
+  # Setting devices will restrict the stats to the specified
+  # bcache devices.
+  # bcacheDevs = ["bcache0", ...]
+```
+
+When run with:
+
+```
+./telegraf -config telegraf.conf -filter bcache -test
+```
+
+It produces:
+
+```
+* Plugin: bcache, Collection 1
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_dirty_data value=11639194
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_bypassed value=5167704440832
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_bypass_hits value=146270986
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_bypass_misses value=0
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_hit_ratio value=90
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_hits value=511941651
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_miss_collisions value=157678
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_misses value=50647396
+> [backing_dev="md10" bcache_dev="bcache0"] bcache_cache_readaheads value=0
+```

plugins/bcache/bcache.go

@@ -0,0 +1,141 @@
+package bcache
+
+import (
+	"errors"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Bcache struct {
+	BcachePath string
+	BcacheDevs []string
+}
+
+var sampleConfig = `
+  # Bcache sets path
+  # If not specified, then default is:
+  # bcachePath = "/sys/fs/bcache"
+  #
+  # By default, telegraf gather stats for all bcache devices
+  # Setting devices will restrict the stats to the specified
+  # bcache devices.
+  # bcacheDevs = ["bcache0", ...]
+`
+
+func (b *Bcache) SampleConfig() string {
+	return sampleConfig
+}
+
+func (b *Bcache) Description() string {
+	return "Read metrics of bcache from stats_total and dirty_data"
+}
+
+func getTags(bdev string) map[string]string {
+	backingDevFile, _ := os.Readlink(bdev)
+	backingDevPath := strings.Split(backingDevFile, "/")
+	backingDev := backingDevPath[len(backingDevPath)-2]
+
+	bcacheDevFile, _ := os.Readlink(bdev + "/dev")
+	bcacheDevPath := strings.Split(bcacheDevFile, "/")
+	bcacheDev := bcacheDevPath[len(bcacheDevPath)-1]
+
+	return map[string]string{"backing_dev": backingDev, "bcache_dev": bcacheDev}
+}
+
+func prettyToBytes(v string) uint64 {
+	var factors = map[string]uint64{
+		"k": 1 << 10,
+		"M": 1 << 20,
+		"G": 1 << 30,
+		"T": 1 << 40,
+		"P": 1 << 50,
+		"E": 1 << 60,
+	}
+	var factor uint64
+	factor = 1
+	prefix := v[len(v)-1 : len(v)]
+	if factors[prefix] != 0 {
+		v = v[:len(v)-1]
+		factor = factors[prefix]
+	}
+	result, _ := strconv.ParseFloat(v, 32)
+	result = result * float64(factor)
+
+	return uint64(result)
+}
+
+func (b *Bcache) gatherBcache(bdev string, acc plugins.Accumulator) error {
+	tags := getTags(bdev)
+	metrics, err := filepath.Glob(bdev + "/stats_total/*")
+	if len(metrics) < 0 {
+		return errors.New("Can't read any stats file")
+	}
+	file, err := ioutil.ReadFile(bdev + "/dirty_data")
+	if err != nil {
+		return err
+	}
+	rawValue := strings.TrimSpace(string(file))
+	value := prettyToBytes(rawValue)
+
+	fields := make(map[string]interface{})
+	fields["dirty_data"] = value
+
+	for _, path := range metrics {
+		key := filepath.Base(path)
+		file, err := ioutil.ReadFile(path)
+		rawValue := strings.TrimSpace(string(file))
+		if err != nil {
+			return err
+		}
+		if key == "bypassed" {
+			value := prettyToBytes(rawValue)
+			fields[key] = value
+		} else {
+			value, _ := strconv.ParseUint(rawValue, 10, 64)
+			fields[key] = value
+		}
+	}
+	acc.AddFields("bcache", fields, tags)
+	return nil
+}
+
+func (b *Bcache) Gather(acc plugins.Accumulator) error {
+	bcacheDevsChecked := make(map[string]bool)
+	var restrictDevs bool
+	if len(b.BcacheDevs) != 0 {
+		restrictDevs = true
+		for _, bcacheDev := range b.BcacheDevs {
+			bcacheDevsChecked[bcacheDev] = true
+		}
+	}
+
+	bcachePath := b.BcachePath
+	if len(bcachePath) == 0 {
+		bcachePath = "/sys/fs/bcache"
+	}
+	bdevs, _ := filepath.Glob(bcachePath + "/*/bdev*")
+	if len(bdevs) < 1 {
+		return errors.New("Can't find any bcache device")
+	}
+	for _, bdev := range bdevs {
+		if restrictDevs {
+			bcacheDev := getTags(bdev)["bcache_dev"]
+			if !bcacheDevsChecked[bcacheDev] {
+				continue
+			}
+		}
+		b.gatherBcache(bdev, acc)
+	}
+	return nil
+}
+
+func init() {
+	plugins.Add("bcache", func() plugins.Plugin {
+		return &Bcache{}
+	})
+}

plugins/bcache/bcache_test.go

@@ -0,0 +1,153 @@
+package bcache
+
+import (
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	dirty_data            = "1.5G"
+	bypassed              = "4.7T"
+	cache_bypass_hits     = "146155333"
+	cache_bypass_misses   = "0"
+	cache_hit_ratio       = "90"
+	cache_hits            = "511469583"
+	cache_miss_collisions = "157567"
+	cache_misses          = "50616331"
+	cache_readaheads      = "2"
+)
+
+var (
+	testBcachePath           = os.TempDir() + "/telegraf/sys/fs/bcache"
+	testBcacheUuidPath       = testBcachePath + "/663955a3-765a-4737-a9fd-8250a7a78411"
+	testBcacheDevPath        = os.TempDir() + "/telegraf/sys/devices/virtual/block/bcache0"
+	testBcacheBackingDevPath = os.TempDir() + "/telegraf/sys/devices/virtual/block/md10"
+)
+
+type metrics struct {
+	name  string
+	value uint64
+}
+
+func TestBcacheGeneratesMetrics(t *testing.T) {
+	err := os.MkdirAll(testBcacheUuidPath, 0755)
+	require.NoError(t, err)
+
+	err = os.MkdirAll(testBcacheDevPath, 0755)
+	require.NoError(t, err)
+
+	err = os.MkdirAll(testBcacheBackingDevPath+"/bcache", 0755)
+	require.NoError(t, err)
+
+	err = os.Symlink(testBcacheBackingDevPath+"/bcache", testBcacheUuidPath+"/bdev0")
+	require.NoError(t, err)
+
+	err = os.Symlink(testBcacheDevPath, testBcacheUuidPath+"/bdev0/dev")
+	require.NoError(t, err)
+
+	err = os.MkdirAll(testBcacheUuidPath+"/bdev0/stats_total", 0755)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/dirty_data", []byte(dirty_data), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/bypassed", []byte(bypassed), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_bypass_hits", []byte(cache_bypass_hits), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_bypass_misses", []byte(cache_bypass_misses), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_hit_ratio", []byte(cache_hit_ratio), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_hits", []byte(cache_hits), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_miss_collisions", []byte(cache_miss_collisions), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_misses", []byte(cache_misses), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(testBcacheUuidPath+"/bdev0/stats_total/cache_readaheads", []byte(cache_readaheads), 0644)
+	require.NoError(t, err)
+
+	intMetrics := []*metrics{
+		{
+			name:  "dirty_data",
+			value: 1610612736,
+		},
+		{
+			name:  "bypassed",
+			value: 5167704440832,
+		},
+		{
+			name:  "cache_bypass_hits",
+			value: 146155333,
+		},
+		{
+			name:  "cache_bypass_misses",
+			value: 0,
+		},
+		{
+			name:  "cache_hit_ratio",
+			value: 90,
+		},
+		{
+			name:  "cache_hits",
+			value: 511469583,
+		},
+		{
+			name:  "cache_miss_collisions",
+			value: 157567,
+		},
+		{
+			name:  "cache_misses",
+			value: 50616331,
+		},
+		{
+			name:  "cache_readaheads",
+			value: 2,
+		},
+	}
+
+	tags := map[string]string{
+		"backing_dev": "md10",
+		"bcache_dev":  "bcache0",
+	}
+
+	var acc testutil.Accumulator
+
+	//all devs
+	b := &Bcache{BcachePath: testBcachePath}
+
+	err = b.Gather(&acc)
+	require.NoError(t, err)
+
+	for _, metric := range intMetrics {
+		assert.True(t, acc.HasUIntValue(metric.name), metric.name)
+		assert.True(t, acc.CheckTaggedValue(metric.name, metric.value, tags))
+	}
+
+	//one exist dev
+	b = &Bcache{BcachePath: testBcachePath, BcacheDevs: []string{"bcache0"}}
+
+	err = b.Gather(&acc)
+	require.NoError(t, err)
+
+	for _, metric := range intMetrics {
+		assert.True(t, acc.HasUIntValue(metric.name), metric.name)
+		assert.True(t, acc.CheckTaggedValue(metric.name, metric.value, tags))
+	}
+
+	err = os.RemoveAll(os.TempDir() + "/telegraf")
+	require.NoError(t, err)
+}

plugins/disque/disque.go

@@ -0,0 +1,204 @@
+package disque
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Disque struct {
+	Servers []string
+
+	c   net.Conn
+	buf []byte
+}
+
+var sampleConfig = `
+  # An array of URI to gather stats about. Specify an ip or hostname
+  # with optional port and password. ie disque://localhost, disque://10.10.3.33:18832,
+  # 10.0.0.1:10000, etc.
+  #
+  # If no servers are specified, then localhost is used as the host.
+  servers = ["localhost"]
+`
+
+func (r *Disque) SampleConfig() string {
+	return sampleConfig
+}
+
+func (r *Disque) Description() string {
+	return "Read metrics from one or many disque servers"
+}
+
+var Tracking = map[string]string{
+	"uptime_in_seconds":          "uptime",
+	"connected_clients":          "clients",
+	"blocked_clients":            "blocked_clients",
+	"used_memory":                "used_memory",
+	"used_memory_rss":            "used_memory_rss",
+	"used_memory_peak":           "used_memory_peak",
+	"total_connections_received": "total_connections_received",
+	"total_commands_processed":   "total_commands_processed",
+	"instantaneous_ops_per_sec":  "instantaneous_ops_per_sec",
+	"latest_fork_usec":           "latest_fork_usec",
+	"mem_fragmentation_ratio":    "mem_fragmentation_ratio",
+	"used_cpu_sys":               "used_cpu_sys",
+	"used_cpu_user":              "used_cpu_user",
+	"used_cpu_sys_children":      "used_cpu_sys_children",
+	"used_cpu_user_children":     "used_cpu_user_children",
+	"registered_jobs":            "registered_jobs",
+	"registered_queues":          "registered_queues",
+}
+
+var ErrProtocolError = errors.New("disque protocol error")
+
+// Reads stats from all configured servers accumulates stats.
+// Returns one of the errors encountered while gather stats (if any).
+func (g *Disque) Gather(acc plugins.Accumulator) error {
+	if len(g.Servers) == 0 {
+		url := &url.URL{
+			Host: ":7711",
+		}
+		g.gatherServer(url, acc)
+		return nil
+	}
+
+	var wg sync.WaitGroup
+
+	var outerr error
+
+	for _, serv := range g.Servers {
+		u, err := url.Parse(serv)
+		if err != nil {
+			return fmt.Errorf("Unable to parse to address '%s': %s", serv, err)
+		} else if u.Scheme == "" {
+			// fallback to simple string based address (i.e. "10.0.0.1:10000")
+			u.Scheme = "tcp"
+			u.Host = serv
+			u.Path = ""
+		}
+		wg.Add(1)
+		go func(serv string) {
+			defer wg.Done()
+			outerr = g.gatherServer(u, acc)
+		}(serv)
+	}
+
+	wg.Wait()
+
+	return outerr
+}
+
+const defaultPort = "7711"
+
+func (g *Disque) gatherServer(addr *url.URL, acc plugins.Accumulator) error {
+	if g.c == nil {
+
+		_, _, err := net.SplitHostPort(addr.Host)
+		if err != nil {
+			addr.Host = addr.Host + ":" + defaultPort
+		}
+
+		c, err := net.Dial("tcp", addr.Host)
+		if err != nil {
+			return fmt.Errorf("Unable to connect to disque server '%s': %s", addr.Host, err)
+		}
+
+		if addr.User != nil {
+			pwd, set := addr.User.Password()
+			if set && pwd != "" {
+				c.Write([]byte(fmt.Sprintf("AUTH %s\r\n", pwd)))
+
+				r := bufio.NewReader(c)
+
+				line, err := r.ReadString('\n')
+				if err != nil {
+					return err
+				}
+				if line[0] != '+' {
+					return fmt.Errorf("%s", strings.TrimSpace(line)[1:])
+				}
+			}
+		}
+
+		g.c = c
+	}
+
+	g.c.Write([]byte("info\r\n"))
+
+	r := bufio.NewReader(g.c)
+
+	line, err := r.ReadString('\n')
+	if err != nil {
+		return err
+	}
+
+	if line[0] != '$' {
+		return fmt.Errorf("bad line start: %s", ErrProtocolError)
+	}
+
+	line = strings.TrimSpace(line)
+
+	szStr := line[1:]
+
+	sz, err := strconv.Atoi(szStr)
+	if err != nil {
+		return fmt.Errorf("bad size string <<%s>>: %s", szStr, ErrProtocolError)
+	}
+
+	var read int
+
+	fields := make(map[string]interface{})
+	tags := map[string]string{"host": addr.String()}
+	for read < sz {
+		line, err := r.ReadString('\n')
+		if err != nil {
+			return err
+		}
+
+		read += len(line)
+
+		if len(line) == 1 || line[0] == '#' {
+			continue
+		}
+
+		parts := strings.SplitN(line, ":", 2)
+
+		name := string(parts[0])
+
+		metric, ok := Tracking[name]
+		if !ok {
+			continue
+		}
+
+		val := strings.TrimSpace(parts[1])
+
+		ival, err := strconv.ParseUint(val, 10, 64)
+		if err == nil {
+			fields[metric] = ival
+			continue
+		}
+
+		fval, err := strconv.ParseFloat(val, 64)
+		if err != nil {
+			return err
+		}
+
+		fields[metric] = fval
+	}
+	acc.AddFields("disque", fields, tags)
+	return nil
+}
+
+func init() {
+	plugins.Add("disque", func() plugins.Plugin {
+		return &Disque{}
+	})
+}

plugins/disque/disque_test.go

@@ -0,0 +1,250 @@
+package disque
+
+import (
+	"bufio"
+	"fmt"
+	"net"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestDisqueGeneratesMetrics(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	l, err := net.Listen("tcp", "localhost:0")
+	require.NoError(t, err)
+
+	defer l.Close()
+
+	go func() {
+		c, err := l.Accept()
+		if err != nil {
+			return
+		}
+
+		buf := bufio.NewReader(c)
+
+		for {
+			line, err := buf.ReadString('\n')
+			if err != nil {
+				return
+			}
+
+			if line != "info\r\n" {
+				return
+			}
+
+			fmt.Fprintf(c, "$%d\n", len(testOutput))
+			c.Write([]byte(testOutput))
+		}
+	}()
+
+	addr := fmt.Sprintf("disque://%s", l.Addr().String())
+
+	r := &Disque{
+		Servers: []string{addr},
+	}
+
+	var acc testutil.Accumulator
+
+	err = r.Gather(&acc)
+	require.NoError(t, err)
+
+	checkInt := []struct {
+		name  string
+		value uint64
+	}{
+		{"uptime", 1452705},
+		{"clients", 31},
+		{"blocked_clients", 13},
+		{"used_memory", 1840104},
+		{"used_memory_rss", 3227648},
+		{"used_memory_peak", 89603656},
+		{"total_connections_received", 5062777},
+		{"total_commands_processed", 12308396},
+		{"instantaneous_ops_per_sec", 18},
+		{"latest_fork_usec", 1644},
+		{"registered_jobs", 360},
+		{"registered_queues", 12},
+	}
+
+	for _, c := range checkInt {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mem_fragmentation_ratio", 1.75},
+		{"used_cpu_sys", 19585.73},
+		{"used_cpu_user", 11255.96},
+		{"used_cpu_sys_children", 1.75},
+		{"used_cpu_user_children", 1.91},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+}
+
+func TestDisqueCanPullStatsFromMultipleServers(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	l, err := net.Listen("tcp", "localhost:0")
+	require.NoError(t, err)
+
+	defer l.Close()
+
+	go func() {
+		c, err := l.Accept()
+		if err != nil {
+			return
+		}
+
+		buf := bufio.NewReader(c)
+
+		for {
+			line, err := buf.ReadString('\n')
+			if err != nil {
+				return
+			}
+
+			if line != "info\r\n" {
+				return
+			}
+
+			fmt.Fprintf(c, "$%d\n", len(testOutput))
+			c.Write([]byte(testOutput))
+		}
+	}()
+
+	addr := fmt.Sprintf("disque://%s", l.Addr().String())
+
+	r := &Disque{
+		Servers: []string{addr},
+	}
+
+	var acc testutil.Accumulator
+
+	err = r.Gather(&acc)
+	require.NoError(t, err)
+
+	checkInt := []struct {
+		name  string
+		value uint64
+	}{
+		{"uptime", 1452705},
+		{"clients", 31},
+		{"blocked_clients", 13},
+		{"used_memory", 1840104},
+		{"used_memory_rss", 3227648},
+		{"used_memory_peak", 89603656},
+		{"total_connections_received", 5062777},
+		{"total_commands_processed", 12308396},
+		{"instantaneous_ops_per_sec", 18},
+		{"latest_fork_usec", 1644},
+		{"registered_jobs", 360},
+		{"registered_queues", 12},
+	}
+
+	for _, c := range checkInt {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mem_fragmentation_ratio", 1.75},
+		{"used_cpu_sys", 19585.73},
+		{"used_cpu_user", 11255.96},
+		{"used_cpu_sys_children", 1.75},
+		{"used_cpu_user_children", 1.91},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+}
+
+const testOutput = `# Server
+disque_version:0.0.1
+disque_git_sha1:b5247598
+disque_git_dirty:0
+disque_build_id:379fda78983a60c6
+os:Linux 3.13.0-44-generic x86_64
+arch_bits:64
+multiplexing_api:epoll
+gcc_version:4.8.2
+process_id:32420
+run_id:1cfdfa4c6bc3f285182db5427522a8a4c16e42e4
+tcp_port:7711
+uptime_in_seconds:1452705
+uptime_in_days:16
+hz:10
+config_file:/usr/local/etc/disque/disque.conf
+
+# Clients
+connected_clients:31
+client_longest_output_list:0
+client_biggest_input_buf:0
+blocked_clients:13
+
+# Memory
+used_memory:1840104
+used_memory_human:1.75M
+used_memory_rss:3227648
+used_memory_peak:89603656
+used_memory_peak_human:85.45M
+mem_fragmentation_ratio:1.75
+mem_allocator:jemalloc-3.6.0
+
+# Jobs
+registered_jobs:360
+
+# Queues
+registered_queues:12
+
+# Persistence
+loading:0
+aof_enabled:1
+aof_state:on
+aof_rewrite_in_progress:0
+aof_rewrite_scheduled:0
+aof_last_rewrite_time_sec:0
+aof_current_rewrite_time_sec:-1
+aof_last_bgrewrite_status:ok
+aof_last_write_status:ok
+aof_current_size:41952430
+aof_base_size:9808
+aof_pending_rewrite:0
+aof_buffer_length:0
+aof_rewrite_buffer_length:0
+aof_pending_bio_fsync:0
+aof_delayed_fsync:1
+
+# Stats
+total_connections_received:5062777
+total_commands_processed:12308396
+instantaneous_ops_per_sec:18
+total_net_input_bytes:1346996528
+total_net_output_bytes:1967551763
+instantaneous_input_kbps:1.38
+instantaneous_output_kbps:1.78
+rejected_connections:0
+latest_fork_usec:1644
+
+# CPU
+used_cpu_sys:19585.73
+used_cpu_user:11255.96
+used_cpu_sys_children:1.75
+used_cpu_user_children:1.91
+`

plugins/elasticsearch/README.md

@@ -0,0 +1,320 @@
+# Elasticsearch plugin
+
+#### Plugin arguments:
+- **servers** []string: list of one or more Elasticsearch servers
+- **local** boolean: If false, it will read the indices stats from all nodes
+- **cluster_health** boolean: If true, it will also obtain cluster level stats
+
+#### Description
+
+The [elasticsearch](https://www.elastic.co/) plugin queries endpoints to obtain
+[node](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html)
+and optionally [cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) stats.
+
+Example:
+
+```
+    [elasticsearch]
+
+    servers = ["http://localhost:9200"]
+
+    local = true
+
+    cluster_health = true
+```
+
+# Measurements
+#### cluster measurements (utilizes fields instead of single values):
+
+contains `status`, `timed_out`, `number_of_nodes`, `number_of_data_nodes`,
+`active_primary_shards`, `active_shards`, `relocating_shards`,
+`initializing_shards`, `unassigned_shards` fields
+- elasticsearch_cluster_health
+
+contains `status`, `number_of_shards`, `number_of_replicas`,
+`active_primary_shards`, `active_shards`, `relocating_shards`,
+`initializing_shards`, `unassigned_shards` fields
+- elasticsearch_indices
+
+#### node measurements:
+
+field data circuit breaker measurement names:
+- elasticsearch_breakers_fielddata_estimated_size_in_bytes value=0
+- elasticsearch_breakers_fielddata_overhead value=1.03
+- elasticsearch_breakers_fielddata_tripped value=0
+- elasticsearch_breakers_fielddata_limit_size_in_bytes value=623326003
+- elasticsearch_breakers_request_estimated_size_in_bytes value=0
+- elasticsearch_breakers_request_overhead value=1.0
+- elasticsearch_breakers_request_tripped value=0
+- elasticsearch_breakers_request_limit_size_in_bytes value=415550668
+- elasticsearch_breakers_parent_overhead value=1.0
+- elasticsearch_breakers_parent_tripped value=0
+- elasticsearch_breakers_parent_limit_size_in_bytes value=727213670
+- elasticsearch_breakers_parent_estimated_size_in_bytes value=0
+
+File system information, data path, free disk space, read/write measurement names:
+- elasticsearch_fs_timestamp value=1436460392946
+- elasticsearch_fs_total_free_in_bytes value=16909316096
+- elasticsearch_fs_total_available_in_bytes value=15894814720
+- elasticsearch_fs_total_total_in_bytes value=19507089408
+
+indices size, document count, indexing and deletion times, search times,
+field cache size, merges and flushes measurement names:
+- elasticsearch_indices_id_cache_memory_size_in_bytes value=0
+- elasticsearch_indices_completion_size_in_bytes value=0
+- elasticsearch_indices_suggest_total value=0
+- elasticsearch_indices_suggest_time_in_millis value=0
+- elasticsearch_indices_suggest_current value=0
+- elasticsearch_indices_query_cache_memory_size_in_bytes value=0
+- elasticsearch_indices_query_cache_evictions value=0
+- elasticsearch_indices_query_cache_hit_count value=0
+- elasticsearch_indices_query_cache_miss_count value=0
+- elasticsearch_indices_store_size_in_bytes value=37715234
+- elasticsearch_indices_store_throttle_time_in_millis value=215
+- elasticsearch_indices_merges_current_docs value=0
+- elasticsearch_indices_merges_current_size_in_bytes value=0
+- elasticsearch_indices_merges_total value=133
+- elasticsearch_indices_merges_total_time_in_millis value=21060
+- elasticsearch_indices_merges_total_docs value=203672
+- elasticsearch_indices_merges_total_size_in_bytes value=142900226
+- elasticsearch_indices_merges_current value=0
+- elasticsearch_indices_filter_cache_memory_size_in_bytes value=7384
+- elasticsearch_indices_filter_cache_evictions value=0
+- elasticsearch_indices_indexing_index_total value=84790
+- elasticsearch_indices_indexing_index_time_in_millis value=29680
+- elasticsearch_indices_indexing_index_current value=0
+- elasticsearch_indices_indexing_noop_update_total value=0
+- elasticsearch_indices_indexing_throttle_time_in_millis value=0
+- elasticsearch_indices_indexing_delete_tota value=13879
+- elasticsearch_indices_indexing_delete_time_in_millis value=1139
+- elasticsearch_indices_indexing_delete_current value=0
+- elasticsearch_indices_get_exists_time_in_millis value=0
+- elasticsearch_indices_get_missing_total value=1
+- elasticsearch_indices_get_missing_time_in_millis value=2
+- elasticsearch_indices_get_current value=0
+- elasticsearch_indices_get_total value=1
+- elasticsearch_indices_get_time_in_millis value=2
+- elasticsearch_indices_get_exists_total value=0
+- elasticsearch_indices_refresh_total value=1076
+- elasticsearch_indices_refresh_total_time_in_millis value=20078
+- elasticsearch_indices_percolate_current value=0
+- elasticsearch_indices_percolate_memory_size_in_bytes value=-1
+- elasticsearch_indices_percolate_queries value=0
+- elasticsearch_indices_percolate_total value=0
+- elasticsearch_indices_percolate_time_in_millis value=0
+- elasticsearch_indices_translog_operations value=17702
+- elasticsearch_indices_translog_size_in_bytes value=17
+- elasticsearch_indices_recovery_current_as_source value=0
+- elasticsearch_indices_recovery_current_as_target value=0
+- elasticsearch_indices_recovery_throttle_time_in_millis value=0
+- elasticsearch_indices_docs_count value=29652
+- elasticsearch_indices_docs_deleted value=5229
+- elasticsearch_indices_flush_total_time_in_millis value=2401
+- elasticsearch_indices_flush_total value=115
+- elasticsearch_indices_fielddata_memory_size_in_bytes value=12996
+- elasticsearch_indices_fielddata_evictions value=0
+- elasticsearch_indices_search_fetch_current value=0
+- elasticsearch_indices_search_open_contexts value=0
+- elasticsearch_indices_search_query_total value=1452
+- elasticsearch_indices_search_query_time_in_millis value=5695
+- elasticsearch_indices_search_query_current value=0
+- elasticsearch_indices_search_fetch_total value=414
+- elasticsearch_indices_search_fetch_time_in_millis value=146
+- elasticsearch_indices_warmer_current value=0
+- elasticsearch_indices_warmer_total value=2319
+- elasticsearch_indices_warmer_total_time_in_millis value=448
+- elasticsearch_indices_segments_count value=134
+- elasticsearch_indices_segments_memory_in_bytes value=1285212
+- elasticsearch_indices_segments_index_writer_memory_in_bytes value=0
+- elasticsearch_indices_segments_index_writer_max_memory_in_bytes value=172368955
+- elasticsearch_indices_segments_version_map_memory_in_bytes value=611844
+- elasticsearch_indices_segments_fixed_bit_set_memory_in_bytes value=0
+
+HTTP connection measurement names:
+- elasticsearch_http_current_open value=3
+- elasticsearch_http_total_opened value=3
+
+JVM stats, memory pool information, garbage collection, buffer pools measurement names:
+- elasticsearch_jvm_timestamp value=1436460392945
+- elasticsearch_jvm_uptime_in_millis value=202245
+- elasticsearch_jvm_mem_non_heap_used_in_bytes value=39634576
+- elasticsearch_jvm_mem_non_heap_committed_in_bytes value=40841216
+- elasticsearch_jvm_mem_pools_young_max_in_bytes value=279183360
+- elasticsearch_jvm_mem_pools_young_peak_used_in_bytes value=71630848
+- elasticsearch_jvm_mem_pools_young_peak_max_in_bytes value=279183360
+- elasticsearch_jvm_mem_pools_young_used_in_bytes value=32685760
+- elasticsearch_jvm_mem_pools_survivor_peak_used_in_bytes value=8912888
+- elasticsearch_jvm_mem_pools_survivor_peak_max_in_bytes value=34865152
+- elasticsearch_jvm_mem_pools_survivor_used_in_bytes value=8912880
+- elasticsearch_jvm_mem_pools_survivor_max_in_bytes value=34865152
+- elasticsearch_jvm_mem_pools_old_peak_max_in_bytes value=724828160
+- elasticsearch_jvm_mem_pools_old_used_in_bytes value=11110928
+- elasticsearch_jvm_mem_pools_old_max_in_bytes value=724828160
+- elasticsearch_jvm_mem_pools_old_peak_used_in_bytes value=14354608
+- elasticsearch_jvm_mem_heap_used_in_bytes value=52709568
+- elasticsearch_jvm_mem_heap_used_percent value=5
+- elasticsearch_jvm_mem_heap_committed_in_bytes value=259522560
+- elasticsearch_jvm_mem_heap_max_in_bytes value=1038876672
+- elasticsearch_jvm_threads_peak_count value=45
+- elasticsearch_jvm_threads_count value=44
+- elasticsearch_jvm_gc_collectors_young_collection_count value=2
+- elasticsearch_jvm_gc_collectors_young_collection_time_in_millis value=98
+- elasticsearch_jvm_gc_collectors_old_collection_count value=1
+- elasticsearch_jvm_gc_collectors_old_collection_time_in_millis value=24
+- elasticsearch_jvm_buffer_pools_direct_count value=40
+- elasticsearch_jvm_buffer_pools_direct_used_in_bytes value=6304239
+- elasticsearch_jvm_buffer_pools_direct_total_capacity_in_bytes value=6304239
+- elasticsearch_jvm_buffer_pools_mapped_count value=0
+- elasticsearch_jvm_buffer_pools_mapped_used_in_bytes value=0
+- elasticsearch_jvm_buffer_pools_mapped_total_capacity_in_bytes value=0
+
+TCP information measurement names:
+- elasticsearch_network_tcp_in_errs value=0
+- elasticsearch_network_tcp_passive_opens value=16
+- elasticsearch_network_tcp_curr_estab value=29
+- elasticsearch_network_tcp_in_segs value=113
+- elasticsearch_network_tcp_out_segs value=97
+- elasticsearch_network_tcp_retrans_segs value=0
+- elasticsearch_network_tcp_attempt_fails value=0
+- elasticsearch_network_tcp_active_opens value=13
+- elasticsearch_network_tcp_estab_resets value=0
+- elasticsearch_network_tcp_out_rsts value=0
+
+Operating system stats, load average, cpu, mem, swap measurement names:
+- elasticsearch_os_swap_used_in_bytes value=0
+- elasticsearch_os_swap_free_in_bytes value=487997440
+- elasticsearch_os_timestamp value=1436460392944
+- elasticsearch_os_uptime_in_millis value=25092
+- elasticsearch_os_cpu_sys value=0
+- elasticsearch_os_cpu_user value=0
+- elasticsearch_os_cpu_idle value=99
+- elasticsearch_os_cpu_usage value=0
+- elasticsearch_os_cpu_stolen value=0
+- elasticsearch_os_mem_free_percent value=74
+- elasticsearch_os_mem_used_percent value=25
+- elasticsearch_os_mem_actual_free_in_bytes value=1565470720
+- elasticsearch_os_mem_actual_used_in_bytes value=534159360
+- elasticsearch_os_mem_free_in_bytes value=477761536
+- elasticsearch_os_mem_used_in_bytes value=1621868544
+
+Process statistics, memory consumption, cpu usage, open file descriptors measurement names:
+- elasticsearch_process_mem_resident_in_bytes value=246382592
+- elasticsearch_process_mem_share_in_bytes value=18747392
+- elasticsearch_process_mem_total_virtual_in_bytes value=4747890688
+- elasticsearch_process_timestamp value=1436460392945
+- elasticsearch_process_open_file_descriptors value=160
+- elasticsearch_process_cpu_total_in_millis value=15480
+- elasticsearch_process_cpu_percent value=2
+- elasticsearch_process_cpu_sys_in_millis value=1870
+- elasticsearch_process_cpu_user_in_millis value=13610
+
+Statistics about each thread pool, including current size, queue and rejected tasks measurement names:
+- elasticsearch_thread_pool_merge_threads value=6
+- elasticsearch_thread_pool_merge_queue value=4
+- elasticsearch_thread_pool_merge_active value=5
+- elasticsearch_thread_pool_merge_rejected value=2
+- elasticsearch_thread_pool_merge_largest value=5
+- elasticsearch_thread_pool_merge_completed value=1
+- elasticsearch_thread_pool_bulk_threads value=4
+- elasticsearch_thread_pool_bulk_queue value=5
+- elasticsearch_thread_pool_bulk_active value=7
+- elasticsearch_thread_pool_bulk_rejected value=3
+- elasticsearch_thread_pool_bulk_largest value=1
+- elasticsearch_thread_pool_bulk_completed value=4
+- elasticsearch_thread_pool_warmer_threads value=2
+- elasticsearch_thread_pool_warmer_queue value=7
+- elasticsearch_thread_pool_warmer_active value=3
+- elasticsearch_thread_pool_warmer_rejected value=2
+- elasticsearch_thread_pool_warmer_largest value=3
+- elasticsearch_thread_pool_warmer_completed value=1
+- elasticsearch_thread_pool_get_largest value=2
+- elasticsearch_thread_pool_get_completed value=1
+- elasticsearch_thread_pool_get_threads value=1
+- elasticsearch_thread_pool_get_queue value=8
+- elasticsearch_thread_pool_get_active value=4
+- elasticsearch_thread_pool_get_rejected value=3
+- elasticsearch_thread_pool_index_threads value=6
+- elasticsearch_thread_pool_index_queue value=8
+- elasticsearch_thread_pool_index_active value=4
+- elasticsearch_thread_pool_index_rejected value=2
+- elasticsearch_thread_pool_index_largest value=3
+- elasticsearch_thread_pool_index_completed value=6
+- elasticsearch_thread_pool_suggest_threads value=2
+- elasticsearch_thread_pool_suggest_queue value=7
+- elasticsearch_thread_pool_suggest_active value=2
+- elasticsearch_thread_pool_suggest_rejected value=1
+- elasticsearch_thread_pool_suggest_largest value=8
+- elasticsearch_thread_pool_suggest_completed value=3
+- elasticsearch_thread_pool_fetch_shard_store_queue value=7
+- elasticsearch_thread_pool_fetch_shard_store_active value=4
+- elasticsearch_thread_pool_fetch_shard_store_rejected value=2
+- elasticsearch_thread_pool_fetch_shard_store_largest value=4
+- elasticsearch_thread_pool_fetch_shard_store_completed value=1
+- elasticsearch_thread_pool_fetch_shard_store_threads value=1
+- elasticsearch_thread_pool_management_threads value=2
+- elasticsearch_thread_pool_management_queue value=3
+- elasticsearch_thread_pool_management_active value=1
+- elasticsearch_thread_pool_management_rejected value=6
+- elasticsearch_thread_pool_management_largest value=2
+- elasticsearch_thread_pool_management_completed value=22
+- elasticsearch_thread_pool_percolate_queue value=23
+- elasticsearch_thread_pool_percolate_active value=13
+- elasticsearch_thread_pool_percolate_rejected value=235
+- elasticsearch_thread_pool_percolate_largest value=23
+- elasticsearch_thread_pool_percolate_completed value=33
+- elasticsearch_thread_pool_percolate_threads value=123
+- elasticsearch_thread_pool_listener_active value=4
+- elasticsearch_thread_pool_listener_rejected value=8
+- elasticsearch_thread_pool_listener_largest value=1
+- elasticsearch_thread_pool_listener_completed value=1
+- elasticsearch_thread_pool_listener_threads value=1
+- elasticsearch_thread_pool_listener_queue value=2
+- elasticsearch_thread_pool_search_rejected value=7
+- elasticsearch_thread_pool_search_largest value=2
+- elasticsearch_thread_pool_search_completed value=4
+- elasticsearch_thread_pool_search_threads value=5
+- elasticsearch_thread_pool_search_queue value=7
+- elasticsearch_thread_pool_search_active value=2
+- elasticsearch_thread_pool_fetch_shard_started_threads value=3
+- elasticsearch_thread_pool_fetch_shard_started_queue value=1
+- elasticsearch_thread_pool_fetch_shard_started_active value=5
+- elasticsearch_thread_pool_fetch_shard_started_rejected value=6
+- elasticsearch_thread_pool_fetch_shard_started_largest value=4
+- elasticsearch_thread_pool_fetch_shard_started_completed value=54
+- elasticsearch_thread_pool_refresh_rejected value=4
+- elasticsearch_thread_pool_refresh_largest value=8
+- elasticsearch_thread_pool_refresh_completed value=3
+- elasticsearch_thread_pool_refresh_threads value=23
+- elasticsearch_thread_pool_refresh_queue value=7
+- elasticsearch_thread_pool_refresh_active value=3
+- elasticsearch_thread_pool_optimize_threads value=3
+- elasticsearch_thread_pool_optimize_queue value=4
+- elasticsearch_thread_pool_optimize_active value=1
+- elasticsearch_thread_pool_optimize_rejected value=2
+- elasticsearch_thread_pool_optimize_largest value=7
+- elasticsearch_thread_pool_optimize_completed value=3
+- elasticsearch_thread_pool_snapshot_largest value=1
+- elasticsearch_thread_pool_snapshot_completed value=0
+- elasticsearch_thread_pool_snapshot_threads value=8
+- elasticsearch_thread_pool_snapshot_queue value=5
+- elasticsearch_thread_pool_snapshot_active value=6
+- elasticsearch_thread_pool_snapshot_rejected value=2
+- elasticsearch_thread_pool_generic_threads value=1
+- elasticsearch_thread_pool_generic_queue value=4
+- elasticsearch_thread_pool_generic_active value=6
+- elasticsearch_thread_pool_generic_rejected value=3
+- elasticsearch_thread_pool_generic_largest value=2
+- elasticsearch_thread_pool_generic_completed value=27
+- elasticsearch_thread_pool_flush_threads value=3
+- elasticsearch_thread_pool_flush_queue value=8
+- elasticsearch_thread_pool_flush_active value=0
+- elasticsearch_thread_pool_flush_rejected value=1
+- elasticsearch_thread_pool_flush_largest value=5
+- elasticsearch_thread_pool_flush_completed value=3
+
+Transport statistics about sent and received bytes in cluster communication measurement names:
+- elasticsearch_transport_server_open value=13
+- elasticsearch_transport_rx_count value=6
+- elasticsearch_transport_rx_size_in_bytes value=1380
+- elasticsearch_transport_tx_count value=6
+- elasticsearch_transport_tx_size_in_bytes value=1380

plugins/elasticsearch/elasticsearch.go

@@ -0,0 +1,226 @@
+package elasticsearch
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+const statsPath = "/_nodes/stats"
+const statsPathLocal = "/_nodes/_local/stats"
+const healthPath = "/_cluster/health"
+
+type node struct {
+	Host       string            `json:"host"`
+	Name       string            `json:"name"`
+	Attributes map[string]string `json:"attributes"`
+	Indices    interface{}       `json:"indices"`
+	OS         interface{}       `json:"os"`
+	Process    interface{}       `json:"process"`
+	JVM        interface{}       `json:"jvm"`
+	ThreadPool interface{}       `json:"thread_pool"`
+	FS         interface{}       `json:"fs"`
+	Transport  interface{}       `json:"transport"`
+	HTTP       interface{}       `json:"http"`
+	Breakers   interface{}       `json:"breakers"`
+}
+
+type clusterHealth struct {
+	ClusterName         string                 `json:"cluster_name"`
+	Status              string                 `json:"status"`
+	TimedOut            bool                   `json:"timed_out"`
+	NumberOfNodes       int                    `json:"number_of_nodes"`
+	NumberOfDataNodes   int                    `json:"number_of_data_nodes"`
+	ActivePrimaryShards int                    `json:"active_primary_shards"`
+	ActiveShards        int                    `json:"active_shards"`
+	RelocatingShards    int                    `json:"relocating_shards"`
+	InitializingShards  int                    `json:"initializing_shards"`
+	UnassignedShards    int                    `json:"unassigned_shards"`
+	Indices             map[string]indexHealth `json:"indices"`
+}
+
+type indexHealth struct {
+	Status              string `json:"status"`
+	NumberOfShards      int    `json:"number_of_shards"`
+	NumberOfReplicas    int    `json:"number_of_replicas"`
+	ActivePrimaryShards int    `json:"active_primary_shards"`
+	ActiveShards        int    `json:"active_shards"`
+	RelocatingShards    int    `json:"relocating_shards"`
+	InitializingShards  int    `json:"initializing_shards"`
+	UnassignedShards    int    `json:"unassigned_shards"`
+}
+
+const sampleConfig = `
+  # specify a list of one or more Elasticsearch servers
+  servers = ["http://localhost:9200"]
+
+  # set local to false when you want to read the indices stats from all nodes
+  # within the cluster
+  local = true
+
+  # set cluster_health to true when you want to also obtain cluster level stats
+  cluster_health = false
+`
+
+// Elasticsearch is a plugin to read stats from one or many Elasticsearch
+// servers.
+type Elasticsearch struct {
+	Local         bool
+	Servers       []string
+	ClusterHealth bool
+	client        *http.Client
+}
+
+// NewElasticsearch return a new instance of Elasticsearch
+func NewElasticsearch() *Elasticsearch {
+	return &Elasticsearch{client: http.DefaultClient}
+}
+
+// SampleConfig returns sample configuration for this plugin.
+func (e *Elasticsearch) SampleConfig() string {
+	return sampleConfig
+}
+
+// Description returns the plugin description.
+func (e *Elasticsearch) Description() string {
+	return "Read stats from one or more Elasticsearch servers or clusters"
+}
+
+// Gather reads the stats from Elasticsearch and writes it to the
+// Accumulator.
+func (e *Elasticsearch) Gather(acc plugins.Accumulator) error {
+	for _, serv := range e.Servers {
+		var url string
+		if e.Local {
+			url = serv + statsPathLocal
+		} else {
+			url = serv + statsPath
+		}
+		if err := e.gatherNodeStats(url, acc); err != nil {
+			return err
+		}
+		if e.ClusterHealth {
+			e.gatherClusterStats(fmt.Sprintf("%s/_cluster/health?level=indices", serv), acc)
+		}
+	}
+	return nil
+}
+
+func (e *Elasticsearch) gatherNodeStats(url string, acc plugins.Accumulator) error {
+	nodeStats := &struct {
+		ClusterName string           `json:"cluster_name"`
+		Nodes       map[string]*node `json:"nodes"`
+	}{}
+	if err := e.gatherData(url, nodeStats); err != nil {
+		return err
+	}
+	for id, n := range nodeStats.Nodes {
+		tags := map[string]string{
+			"node_id":      id,
+			"node_host":    n.Host,
+			"node_name":    n.Name,
+			"cluster_name": nodeStats.ClusterName,
+		}
+
+		for k, v := range n.Attributes {
+			tags["node_attribute_"+k] = v
+		}
+
+		stats := map[string]interface{}{
+			"indices":     n.Indices,
+			"os":          n.OS,
+			"process":     n.Process,
+			"jvm":         n.JVM,
+			"thread_pool": n.ThreadPool,
+			"fs":          n.FS,
+			"transport":   n.Transport,
+			"http":        n.HTTP,
+			"breakers":    n.Breakers,
+		}
+
+		now := time.Now()
+		for p, s := range stats {
+			f := internal.JSONFlattener{}
+			err := f.FlattenJSON("", s)
+			if err != nil {
+				return err
+			}
+			acc.AddFields("elasticsearch_"+p, f.Fields, tags, now)
+		}
+	}
+	return nil
+}
+
+func (e *Elasticsearch) gatherClusterStats(url string, acc plugins.Accumulator) error {
+	clusterStats := &clusterHealth{}
+	if err := e.gatherData(url, clusterStats); err != nil {
+		return err
+	}
+	measurementTime := time.Now()
+	clusterFields := map[string]interface{}{
+		"status":                clusterStats.Status,
+		"timed_out":             clusterStats.TimedOut,
+		"number_of_nodes":       clusterStats.NumberOfNodes,
+		"number_of_data_nodes":  clusterStats.NumberOfDataNodes,
+		"active_primary_shards": clusterStats.ActivePrimaryShards,
+		"active_shards":         clusterStats.ActiveShards,
+		"relocating_shards":     clusterStats.RelocatingShards,
+		"initializing_shards":   clusterStats.InitializingShards,
+		"unassigned_shards":     clusterStats.UnassignedShards,
+	}
+	acc.AddFields(
+		"elasticsearch_cluster_health",
+		clusterFields,
+		map[string]string{"name": clusterStats.ClusterName},
+		measurementTime,
+	)
+
+	for name, health := range clusterStats.Indices {
+		indexFields := map[string]interface{}{
+			"status":                health.Status,
+			"number_of_shards":      health.NumberOfShards,
+			"number_of_replicas":    health.NumberOfReplicas,
+			"active_primary_shards": health.ActivePrimaryShards,
+			"active_shards":         health.ActiveShards,
+			"relocating_shards":     health.RelocatingShards,
+			"initializing_shards":   health.InitializingShards,
+			"unassigned_shards":     health.UnassignedShards,
+		}
+		acc.AddFields(
+			"elasticsearch_indices",
+			indexFields,
+			map[string]string{"index": name},
+			measurementTime,
+		)
+	}
+	return nil
+}
+
+func (e *Elasticsearch) gatherData(url string, v interface{}) error {
+	r, err := e.client.Get(url)
+	if err != nil {
+		return err
+	}
+	defer r.Body.Close()
+	if r.StatusCode != http.StatusOK {
+		// NOTE: we are not going to read/discard r.Body under the assumption we'd prefer
+		// to let the underlying transport close the connection and re-establish a new one for
+		// future calls.
+		return fmt.Errorf("elasticsearch: API responded with status-code %d, expected %d",
+			r.StatusCode, http.StatusOK)
+	}
+	if err = json.NewDecoder(r.Body).Decode(v); err != nil {
+		return err
+	}
+	return nil
+}
+
+func init() {
+	plugins.Add("elasticsearch", func() plugins.Plugin {
+		return NewElasticsearch()
+	})
+}

plugins/elasticsearch/elasticsearch_test.go

@@ -0,0 +1,108 @@
+package elasticsearch
+
+import (
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type transportMock struct {
+	statusCode int
+	body       string
+}
+
+func newTransportMock(statusCode int, body string) http.RoundTripper {
+	return &transportMock{
+		statusCode: statusCode,
+		body:       body,
+	}
+}
+
+func (t *transportMock) RoundTrip(r *http.Request) (*http.Response, error) {
+	res := &http.Response{
+		Header:     make(http.Header),
+		Request:    r,
+		StatusCode: t.statusCode,
+	}
+	res.Header.Set("Content-Type", "application/json")
+	res.Body = ioutil.NopCloser(strings.NewReader(t.body))
+	return res, nil
+}
+
+func TestElasticsearch(t *testing.T) {
+	es := NewElasticsearch()
+	es.Servers = []string{"http://example.com:9200"}
+	es.client.Transport = newTransportMock(http.StatusOK, statsResponse)
+
+	var acc testutil.Accumulator
+	if err := es.Gather(&acc); err != nil {
+		t.Fatal(err)
+	}
+
+	tags := map[string]string{
+		"cluster_name":          "es-testcluster",
+		"node_attribute_master": "true",
+		"node_id":               "SDFsfSDFsdfFSDSDfSFDSDF",
+		"node_name":             "test.host.com",
+		"node_host":             "test",
+	}
+
+	testTables := []map[string]float64{
+		indicesExpected,
+		osExpected,
+		processExpected,
+		jvmExpected,
+		threadPoolExpected,
+		fsExpected,
+		transportExpected,
+		httpExpected,
+		breakersExpected,
+	}
+
+	for _, testTable := range testTables {
+		for k, v := range testTable {
+			assert.NoError(t, acc.ValidateTaggedValue(k, v, tags))
+		}
+	}
+}
+
+func TestGatherClusterStats(t *testing.T) {
+	es := NewElasticsearch()
+	es.Servers = []string{"http://example.com:9200"}
+	es.ClusterHealth = true
+	es.client.Transport = newTransportMock(http.StatusOK, clusterResponse)
+
+	var acc testutil.Accumulator
+	require.NoError(t, es.Gather(&acc))
+
+	var clusterHealthTests = []struct {
+		measurement string
+		fields      map[string]interface{}
+		tags        map[string]string
+	}{
+		{
+			"cluster_health",
+			clusterHealthExpected,
+			map[string]string{"name": "elasticsearch_telegraf"},
+		},
+		{
+			"indices",
+			v1IndexExpected,
+			map[string]string{"index": "v1"},
+		},
+		{
+			"indices",
+			v2IndexExpected,
+			map[string]string{"index": "v2"},
+		},
+	}
+
+	for _, exp := range clusterHealthTests {
+		assert.NoError(t, acc.ValidateTaggedFields(exp.measurement, exp.fields, exp.tags))
+	}
+}

plugins/elasticsearch/testdata_test.go

@@ -0,0 +1,759 @@
+package elasticsearch
+
+const clusterResponse = `
+{
+   "cluster_name": "elasticsearch_telegraf",
+   "status": "green",
+   "timed_out": false,
+   "number_of_nodes": 3,
+   "number_of_data_nodes": 3,
+   "active_primary_shards": 5,
+   "active_shards": 15,
+   "relocating_shards": 0,
+   "initializing_shards": 0,
+   "unassigned_shards": 0,
+   "indices": {
+      "v1": {
+         "status": "green",
+         "number_of_shards": 10,
+         "number_of_replicas": 1,
+         "active_primary_shards": 10,
+         "active_shards": 20,
+         "relocating_shards": 0,
+         "initializing_shards": 0,
+         "unassigned_shards": 0
+      },
+      "v2": {
+         "status": "red",
+         "number_of_shards": 10,
+         "number_of_replicas": 1,
+         "active_primary_shards": 0,
+         "active_shards": 0,
+         "relocating_shards": 0,
+         "initializing_shards": 0,
+         "unassigned_shards": 20
+      }
+   }
+}
+`
+
+var clusterHealthExpected = map[string]interface{}{
+	"status":                "green",
+	"timed_out":             false,
+	"number_of_nodes":       3,
+	"number_of_data_nodes":  3,
+	"active_primary_shards": 5,
+	"active_shards":         15,
+	"relocating_shards":     0,
+	"initializing_shards":   0,
+	"unassigned_shards":     0,
+}
+
+var v1IndexExpected = map[string]interface{}{
+	"status":                "green",
+	"number_of_shards":      10,
+	"number_of_replicas":    1,
+	"active_primary_shards": 10,
+	"active_shards":         20,
+	"relocating_shards":     0,
+	"initializing_shards":   0,
+	"unassigned_shards":     0,
+}
+
+var v2IndexExpected = map[string]interface{}{
+	"status":                "red",
+	"number_of_shards":      10,
+	"number_of_replicas":    1,
+	"active_primary_shards": 0,
+	"active_shards":         0,
+	"relocating_shards":     0,
+	"initializing_shards":   0,
+	"unassigned_shards":     20,
+}
+
+const statsResponse = `
+{
+  "cluster_name": "es-testcluster",
+  "nodes": {
+    "SDFsfSDFsdfFSDSDfSFDSDF": {
+      "timestamp": 1436365550135,
+      "name": "test.host.com",
+      "transport_address": "inet[/127.0.0.1:9300]",
+      "host": "test",
+      "ip": [
+        "inet[/127.0.0.1:9300]",
+        "NONE"
+      ],
+      "attributes": {
+        "master": "true"
+      },
+      "indices": {
+        "docs": {
+          "count": 29652,
+          "deleted": 5229
+        },
+        "store": {
+          "size_in_bytes": 37715234,
+          "throttle_time_in_millis": 215
+        },
+        "indexing": {
+          "index_total": 84790,
+          "index_time_in_millis": 29680,
+          "index_current": 0,
+          "delete_total": 13879,
+          "delete_time_in_millis": 1139,
+          "delete_current": 0,
+          "noop_update_total": 0,
+          "is_throttled": false,
+          "throttle_time_in_millis": 0
+        },
+        "get": {
+          "total": 1,
+          "time_in_millis": 2,
+          "exists_total": 0,
+          "exists_time_in_millis": 0,
+          "missing_total": 1,
+          "missing_time_in_millis": 2,
+          "current": 0
+        },
+        "search": {
+          "open_contexts": 0,
+          "query_total": 1452,
+          "query_time_in_millis": 5695,
+          "query_current": 0,
+          "fetch_total": 414,
+          "fetch_time_in_millis": 146,
+          "fetch_current": 0
+        },
+        "merges": {
+          "current": 0,
+          "current_docs": 0,
+          "current_size_in_bytes": 0,
+          "total": 133,
+          "total_time_in_millis": 21060,
+          "total_docs": 203672,
+          "total_size_in_bytes": 142900226
+        },
+        "refresh": {
+          "total": 1076,
+          "total_time_in_millis": 20078
+        },
+        "flush": {
+          "total": 115,
+          "total_time_in_millis": 2401
+        },
+        "warmer": {
+          "current": 0,
+          "total": 2319,
+          "total_time_in_millis": 448
+        },
+        "filter_cache": {
+          "memory_size_in_bytes": 7384,
+          "evictions": 0
+        },
+        "id_cache": {
+          "memory_size_in_bytes": 0
+        },
+        "fielddata": {
+          "memory_size_in_bytes": 12996,
+          "evictions": 0
+        },
+        "percolate": {
+          "total": 0,
+          "time_in_millis": 0,
+          "current": 0,
+          "memory_size_in_bytes": -1,
+          "memory_size": "-1b",
+          "queries": 0
+        },
+        "completion": {
+          "size_in_bytes": 0
+        },
+        "segments": {
+          "count": 134,
+          "memory_in_bytes": 1285212,
+          "index_writer_memory_in_bytes": 0,
+          "index_writer_max_memory_in_bytes": 172368955,
+          "version_map_memory_in_bytes": 611844,
+          "fixed_bit_set_memory_in_bytes": 0
+        },
+        "translog": {
+          "operations": 17702,
+          "size_in_bytes": 17
+        },
+        "suggest": {
+          "total": 0,
+          "time_in_millis": 0,
+          "current": 0
+        },
+        "query_cache": {
+          "memory_size_in_bytes": 0,
+          "evictions": 0,
+          "hit_count": 0,
+          "miss_count": 0
+        },
+        "recovery": {
+          "current_as_source": 0,
+          "current_as_target": 0,
+          "throttle_time_in_millis": 0
+        }
+      },
+      "os": {
+        "timestamp": 1436460392944,
+        "load_average": [
+          0.01,
+          0.04,
+          0.05
+        ],
+        "mem": {
+          "free_in_bytes": 477761536,
+          "used_in_bytes": 1621868544,
+          "free_percent": 74,
+          "used_percent": 25,
+          "actual_free_in_bytes": 1565470720,
+          "actual_used_in_bytes": 534159360
+        },
+        "swap": {
+          "used_in_bytes": 0,
+          "free_in_bytes": 487997440
+        }
+      },
+      "process": {
+        "timestamp": 1436460392945,
+        "open_file_descriptors": 160,
+        "cpu": {
+          "percent": 2,
+          "sys_in_millis": 1870,
+          "user_in_millis": 13610,
+          "total_in_millis": 15480
+        },
+        "mem": {
+          "total_virtual_in_bytes": 4747890688
+        }
+      },
+      "jvm": {
+        "timestamp": 1436460392945,
+        "uptime_in_millis": 202245,
+        "mem": {
+          "heap_used_in_bytes": 52709568,
+          "heap_used_percent": 5,
+          "heap_committed_in_bytes": 259522560,
+          "heap_max_in_bytes": 1038876672,
+          "non_heap_used_in_bytes": 39634576,
+          "non_heap_committed_in_bytes": 40841216,
+          "pools": {
+            "young": {
+              "used_in_bytes": 32685760,
+              "max_in_bytes": 279183360,
+              "peak_used_in_bytes": 71630848,
+              "peak_max_in_bytes": 279183360
+            },
+            "survivor": {
+              "used_in_bytes": 8912880,
+              "max_in_bytes": 34865152,
+              "peak_used_in_bytes": 8912888,
+              "peak_max_in_bytes": 34865152
+            },
+            "old": {
+              "used_in_bytes": 11110928,
+              "max_in_bytes": 724828160,
+              "peak_used_in_bytes": 14354608,
+              "peak_max_in_bytes": 724828160
+            }
+          }
+        },
+        "threads": {
+          "count": 44,
+          "peak_count": 45
+        },
+        "gc": {
+          "collectors": {
+            "young": {
+              "collection_count": 2,
+              "collection_time_in_millis": 98
+            },
+            "old": {
+              "collection_count": 1,
+              "collection_time_in_millis": 24
+            }
+          }
+        },
+        "buffer_pools": {
+          "direct": {
+            "count": 40,
+            "used_in_bytes": 6304239,
+            "total_capacity_in_bytes": 6304239
+          },
+          "mapped": {
+            "count": 0,
+            "used_in_bytes": 0,
+            "total_capacity_in_bytes": 0
+          }
+        }
+      },
+      "thread_pool": {
+        "percolate": {
+          "threads": 123,
+          "queue": 23,
+          "active": 13,
+          "rejected": 235,
+          "largest": 23,
+          "completed": 33
+        },
+        "fetch_shard_started": {
+          "threads": 3,
+          "queue": 1,
+          "active": 5,
+          "rejected": 6,
+          "largest": 4,
+          "completed": 54
+        },
+        "listener": {
+          "threads": 1,
+          "queue": 2,
+          "active": 4,
+          "rejected": 8,
+          "largest": 1,
+          "completed": 1
+        },
+        "index": {
+          "threads": 6,
+          "queue": 8,
+          "active": 4,
+          "rejected": 2,
+          "largest": 3,
+          "completed": 6
+        },
+        "refresh": {
+          "threads": 23,
+          "queue": 7,
+          "active": 3,
+          "rejected": 4,
+          "largest": 8,
+          "completed": 3
+        },
+        "suggest": {
+          "threads": 2,
+          "queue": 7,
+          "active": 2,
+          "rejected": 1,
+          "largest": 8,
+          "completed": 3
+        },
+        "generic": {
+          "threads": 1,
+          "queue": 4,
+          "active": 6,
+          "rejected": 3,
+          "largest": 2,
+          "completed": 27
+        },
+        "warmer": {
+          "threads": 2,
+          "queue": 7,
+          "active": 3,
+          "rejected": 2,
+          "largest": 3,
+          "completed": 1
+        },
+        "search": {
+          "threads": 5,
+          "queue": 7,
+          "active": 2,
+          "rejected": 7,
+          "largest": 2,
+          "completed": 4
+        },
+        "flush": {
+          "threads": 3,
+          "queue": 8,
+          "active": 0,
+          "rejected": 1,
+          "largest": 5,
+          "completed": 3
+        },
+        "optimize": {
+          "threads": 3,
+          "queue": 4,
+          "active": 1,
+          "rejected": 2,
+          "largest": 7,
+          "completed": 3
+        },
+        "fetch_shard_store": {
+          "threads": 1,
+          "queue": 7,
+          "active": 4,
+          "rejected": 2,
+          "largest": 4,
+          "completed": 1
+        },
+        "management": {
+          "threads": 2,
+          "queue": 3,
+          "active": 1,
+          "rejected": 6,
+          "largest": 2,
+          "completed": 22
+        },
+        "get": {
+          "threads": 1,
+          "queue": 8,
+          "active": 4,
+          "rejected": 3,
+          "largest": 2,
+          "completed": 1
+        },
+        "merge": {
+          "threads": 6,
+          "queue": 4,
+          "active": 5,
+          "rejected": 2,
+          "largest": 5,
+          "completed": 1
+        },
+        "bulk": {
+          "threads": 4,
+          "queue": 5,
+          "active": 7,
+          "rejected": 3,
+          "largest": 1,
+          "completed": 4
+        },
+        "snapshot": {
+          "threads": 8,
+          "queue": 5,
+          "active": 6,
+          "rejected": 2,
+          "largest": 1,
+          "completed": 0
+        }
+      },
+      "fs": {
+        "timestamp": 1436460392946,
+        "total": {
+          "total_in_bytes": 19507089408,
+          "free_in_bytes": 16909316096,
+          "available_in_bytes": 15894814720
+        },
+        "data": [
+          {
+            "path": "/usr/share/elasticsearch/data/elasticsearch/nodes/0",
+            "mount": "/usr/share/elasticsearch/data",
+            "type": "ext4",
+            "total_in_bytes": 19507089408,
+            "free_in_bytes": 16909316096,
+            "available_in_bytes": 15894814720
+          }
+        ]
+      },
+      "transport": {
+        "server_open": 13,
+        "rx_count": 6,
+        "rx_size_in_bytes": 1380,
+        "tx_count": 6,
+        "tx_size_in_bytes": 1380
+      },
+      "http": {
+        "current_open": 3,
+        "total_opened": 3
+      },
+      "breakers": {
+        "fielddata": {
+          "limit_size_in_bytes": 623326003,
+          "limit_size": "594.4mb",
+          "estimated_size_in_bytes": 0,
+          "estimated_size": "0b",
+          "overhead": 1.03,
+          "tripped": 0
+        },
+        "request": {
+          "limit_size_in_bytes": 415550668,
+          "limit_size": "396.2mb",
+          "estimated_size_in_bytes": 0,
+          "estimated_size": "0b",
+          "overhead": 1.0,
+          "tripped": 0
+        },
+        "parent": {
+          "limit_size_in_bytes": 727213670,
+          "limit_size": "693.5mb",
+          "estimated_size_in_bytes": 0,
+          "estimated_size": "0b",
+          "overhead": 1.0,
+          "tripped": 0
+        }
+      }
+    }
+  }
+}
+`
+
+var indicesExpected = map[string]float64{
+	"indices_id_cache_memory_size_in_bytes":             0,
+	"indices_completion_size_in_bytes":                  0,
+	"indices_suggest_total":                             0,
+	"indices_suggest_time_in_millis":                    0,
+	"indices_suggest_current":                           0,
+	"indices_query_cache_memory_size_in_bytes":          0,
+	"indices_query_cache_evictions":                     0,
+	"indices_query_cache_hit_count":                     0,
+	"indices_query_cache_miss_count":                    0,
+	"indices_store_size_in_bytes":                       37715234,
+	"indices_store_throttle_time_in_millis":             215,
+	"indices_merges_current_docs":                       0,
+	"indices_merges_current_size_in_bytes":              0,
+	"indices_merges_total":                              133,
+	"indices_merges_total_time_in_millis":               21060,
+	"indices_merges_total_docs":                         203672,
+	"indices_merges_total_size_in_bytes":                142900226,
+	"indices_merges_current":                            0,
+	"indices_filter_cache_memory_size_in_bytes":         7384,
+	"indices_filter_cache_evictions":                    0,
+	"indices_indexing_index_total":                      84790,
+	"indices_indexing_index_time_in_millis":             29680,
+	"indices_indexing_index_current":                    0,
+	"indices_indexing_noop_update_total":                0,
+	"indices_indexing_throttle_time_in_millis":          0,
+	"indices_indexing_delete_total":                     13879,
+	"indices_indexing_delete_time_in_millis":            1139,
+	"indices_indexing_delete_current":                   0,
+	"indices_get_exists_time_in_millis":                 0,
+	"indices_get_missing_total":                         1,
+	"indices_get_missing_time_in_millis":                2,
+	"indices_get_current":                               0,
+	"indices_get_total":                                 1,
+	"indices_get_time_in_millis":                        2,
+	"indices_get_exists_total":                          0,
+	"indices_refresh_total":                             1076,
+	"indices_refresh_total_time_in_millis":              20078,
+	"indices_percolate_current":                         0,
+	"indices_percolate_memory_size_in_bytes":            -1,
+	"indices_percolate_queries":                         0,
+	"indices_percolate_total":                           0,
+	"indices_percolate_time_in_millis":                  0,
+	"indices_translog_operations":                       17702,
+	"indices_translog_size_in_bytes":                    17,
+	"indices_recovery_current_as_source":                0,
+	"indices_recovery_current_as_target":                0,
+	"indices_recovery_throttle_time_in_millis":          0,
+	"indices_docs_count":                                29652,
+	"indices_docs_deleted":                              5229,
+	"indices_flush_total_time_in_millis":                2401,
+	"indices_flush_total":                               115,
+	"indices_fielddata_memory_size_in_bytes":            12996,
+	"indices_fielddata_evictions":                       0,
+	"indices_search_fetch_current":                      0,
+	"indices_search_open_contexts":                      0,
+	"indices_search_query_total":                        1452,
+	"indices_search_query_time_in_millis":               5695,
+	"indices_search_query_current":                      0,
+	"indices_search_fetch_total":                        414,
+	"indices_search_fetch_time_in_millis":               146,
+	"indices_warmer_current":                            0,
+	"indices_warmer_total":                              2319,
+	"indices_warmer_total_time_in_millis":               448,
+	"indices_segments_count":                            134,
+	"indices_segments_memory_in_bytes":                  1285212,
+	"indices_segments_index_writer_memory_in_bytes":     0,
+	"indices_segments_index_writer_max_memory_in_bytes": 172368955,
+	"indices_segments_version_map_memory_in_bytes":      611844,
+	"indices_segments_fixed_bit_set_memory_in_bytes":    0,
+}
+
+var osExpected = map[string]float64{
+	"os_swap_used_in_bytes":       0,
+	"os_swap_free_in_bytes":       487997440,
+	"os_timestamp":                1436460392944,
+	"os_mem_free_percent":         74,
+	"os_mem_used_percent":         25,
+	"os_mem_actual_free_in_bytes": 1565470720,
+	"os_mem_actual_used_in_bytes": 534159360,
+	"os_mem_free_in_bytes":        477761536,
+	"os_mem_used_in_bytes":        1621868544,
+}
+
+var processExpected = map[string]float64{
+	"process_mem_total_virtual_in_bytes": 4747890688,
+	"process_timestamp":                  1436460392945,
+	"process_open_file_descriptors":      160,
+	"process_cpu_total_in_millis":        15480,
+	"process_cpu_percent":                2,
+	"process_cpu_sys_in_millis":          1870,
+	"process_cpu_user_in_millis":         13610,
+}
+
+var jvmExpected = map[string]float64{
+	"jvm_timestamp":                                     1436460392945,
+	"jvm_uptime_in_millis":                              202245,
+	"jvm_mem_non_heap_used_in_bytes":                    39634576,
+	"jvm_mem_non_heap_committed_in_bytes":               40841216,
+	"jvm_mem_pools_young_max_in_bytes":                  279183360,
+	"jvm_mem_pools_young_peak_used_in_bytes":            71630848,
+	"jvm_mem_pools_young_peak_max_in_bytes":             279183360,
+	"jvm_mem_pools_young_used_in_bytes":                 32685760,
+	"jvm_mem_pools_survivor_peak_used_in_bytes":         8912888,
+	"jvm_mem_pools_survivor_peak_max_in_bytes":          34865152,
+	"jvm_mem_pools_survivor_used_in_bytes":              8912880,
+	"jvm_mem_pools_survivor_max_in_bytes":               34865152,
+	"jvm_mem_pools_old_peak_max_in_bytes":               724828160,
+	"jvm_mem_pools_old_used_in_bytes":                   11110928,
+	"jvm_mem_pools_old_max_in_bytes":                    724828160,
+	"jvm_mem_pools_old_peak_used_in_bytes":              14354608,
+	"jvm_mem_heap_used_in_bytes":                        52709568,
+	"jvm_mem_heap_used_percent":                         5,
+	"jvm_mem_heap_committed_in_bytes":                   259522560,
+	"jvm_mem_heap_max_in_bytes":                         1038876672,
+	"jvm_threads_peak_count":                            45,
+	"jvm_threads_count":                                 44,
+	"jvm_gc_collectors_young_collection_count":          2,
+	"jvm_gc_collectors_young_collection_time_in_millis": 98,
+	"jvm_gc_collectors_old_collection_count":            1,
+	"jvm_gc_collectors_old_collection_time_in_millis":   24,
+	"jvm_buffer_pools_direct_count":                     40,
+	"jvm_buffer_pools_direct_used_in_bytes":             6304239,
+	"jvm_buffer_pools_direct_total_capacity_in_bytes":   6304239,
+	"jvm_buffer_pools_mapped_count":                     0,
+	"jvm_buffer_pools_mapped_used_in_bytes":             0,
+	"jvm_buffer_pools_mapped_total_capacity_in_bytes":   0,
+}
+
+var threadPoolExpected = map[string]float64{
+	"thread_pool_merge_threads":                 6,
+	"thread_pool_merge_queue":                   4,
+	"thread_pool_merge_active":                  5,
+	"thread_pool_merge_rejected":                2,
+	"thread_pool_merge_largest":                 5,
+	"thread_pool_merge_completed":               1,
+	"thread_pool_bulk_threads":                  4,
+	"thread_pool_bulk_queue":                    5,
+	"thread_pool_bulk_active":                   7,
+	"thread_pool_bulk_rejected":                 3,
+	"thread_pool_bulk_largest":                  1,
+	"thread_pool_bulk_completed":                4,
+	"thread_pool_warmer_threads":                2,
+	"thread_pool_warmer_queue":                  7,
+	"thread_pool_warmer_active":                 3,
+	"thread_pool_warmer_rejected":               2,
+	"thread_pool_warmer_largest":                3,
+	"thread_pool_warmer_completed":              1,
+	"thread_pool_get_largest":                   2,
+	"thread_pool_get_completed":                 1,
+	"thread_pool_get_threads":                   1,
+	"thread_pool_get_queue":                     8,
+	"thread_pool_get_active":                    4,
+	"thread_pool_get_rejected":                  3,
+	"thread_pool_index_threads":                 6,
+	"thread_pool_index_queue":                   8,
+	"thread_pool_index_active":                  4,
+	"thread_pool_index_rejected":                2,
+	"thread_pool_index_largest":                 3,
+	"thread_pool_index_completed":               6,
+	"thread_pool_suggest_threads":               2,
+	"thread_pool_suggest_queue":                 7,
+	"thread_pool_suggest_active":                2,
+	"thread_pool_suggest_rejected":              1,
+	"thread_pool_suggest_largest":               8,
+	"thread_pool_suggest_completed":             3,
+	"thread_pool_fetch_shard_store_queue":       7,
+	"thread_pool_fetch_shard_store_active":      4,
+	"thread_pool_fetch_shard_store_rejected":    2,
+	"thread_pool_fetch_shard_store_largest":     4,
+	"thread_pool_fetch_shard_store_completed":   1,
+	"thread_pool_fetch_shard_store_threads":     1,
+	"thread_pool_management_threads":            2,
+	"thread_pool_management_queue":              3,
+	"thread_pool_management_active":             1,
+	"thread_pool_management_rejected":           6,
+	"thread_pool_management_largest":            2,
+	"thread_pool_management_completed":          22,
+	"thread_pool_percolate_queue":               23,
+	"thread_pool_percolate_active":              13,
+	"thread_pool_percolate_rejected":            235,
+	"thread_pool_percolate_largest":             23,
+	"thread_pool_percolate_completed":           33,
+	"thread_pool_percolate_threads":             123,
+	"thread_pool_listener_active":               4,
+	"thread_pool_listener_rejected":             8,
+	"thread_pool_listener_largest":              1,
+	"thread_pool_listener_completed":            1,
+	"thread_pool_listener_threads":              1,
+	"thread_pool_listener_queue":                2,
+	"thread_pool_search_rejected":               7,
+	"thread_pool_search_largest":                2,
+	"thread_pool_search_completed":              4,
+	"thread_pool_search_threads":                5,
+	"thread_pool_search_queue":                  7,
+	"thread_pool_search_active":                 2,
+	"thread_pool_fetch_shard_started_threads":   3,
+	"thread_pool_fetch_shard_started_queue":     1,
+	"thread_pool_fetch_shard_started_active":    5,
+	"thread_pool_fetch_shard_started_rejected":  6,
+	"thread_pool_fetch_shard_started_largest":   4,
+	"thread_pool_fetch_shard_started_completed": 54,
+	"thread_pool_refresh_rejected":              4,
+	"thread_pool_refresh_largest":               8,
+	"thread_pool_refresh_completed":             3,
+	"thread_pool_refresh_threads":               23,
+	"thread_pool_refresh_queue":                 7,
+	"thread_pool_refresh_active":                3,
+	"thread_pool_optimize_threads":              3,
+	"thread_pool_optimize_queue":                4,
+	"thread_pool_optimize_active":               1,
+	"thread_pool_optimize_rejected":             2,
+	"thread_pool_optimize_largest":              7,
+	"thread_pool_optimize_completed":            3,
+	"thread_pool_snapshot_largest":              1,
+	"thread_pool_snapshot_completed":            0,
+	"thread_pool_snapshot_threads":              8,
+	"thread_pool_snapshot_queue":                5,
+	"thread_pool_snapshot_active":               6,
+	"thread_pool_snapshot_rejected":             2,
+	"thread_pool_generic_threads":               1,
+	"thread_pool_generic_queue":                 4,
+	"thread_pool_generic_active":                6,
+	"thread_pool_generic_rejected":              3,
+	"thread_pool_generic_largest":               2,
+	"thread_pool_generic_completed":             27,
+	"thread_pool_flush_threads":                 3,
+	"thread_pool_flush_queue":                   8,
+	"thread_pool_flush_active":                  0,
+	"thread_pool_flush_rejected":                1,
+	"thread_pool_flush_largest":                 5,
+	"thread_pool_flush_completed":               3,
+}
+
+var fsExpected = map[string]float64{
+	"fs_timestamp":                1436460392946,
+	"fs_total_free_in_bytes":      16909316096,
+	"fs_total_available_in_bytes": 15894814720,
+	"fs_total_total_in_bytes":     19507089408,
+}
+
+var transportExpected = map[string]float64{
+	"transport_server_open":      13,
+	"transport_rx_count":         6,
+	"transport_rx_size_in_bytes": 1380,
+	"transport_tx_count":         6,
+	"transport_tx_size_in_bytes": 1380,
+}
+
+var httpExpected = map[string]float64{
+	"http_current_open": 3,
+	"http_total_opened": 3,
+}
+
+var breakersExpected = map[string]float64{
+	"breakers_fielddata_estimated_size_in_bytes": 0,
+	"breakers_fielddata_overhead":                1.03,
+	"breakers_fielddata_tripped":                 0,
+	"breakers_fielddata_limit_size_in_bytes":     623326003,
+	"breakers_request_estimated_size_in_bytes":   0,
+	"breakers_request_overhead":                  1.0,
+	"breakers_request_tripped":                   0,
+	"breakers_request_limit_size_in_bytes":       415550668,
+	"breakers_parent_overhead":                   1.0,
+	"breakers_parent_tripped":                    0,
+	"breakers_parent_limit_size_in_bytes":        727213670,
+	"breakers_parent_estimated_size_in_bytes":    0,
+}

plugins/exec/README.md

@@ -0,0 +1,42 @@
+# Exec Plugin
+
+The exec plugin can execute arbitrary commands which output JSON. Then it flattens JSON and finds
+all numeric values, treating them as floats.
+
+For example, if you have a json-returning command called mycollector, you could
+setup the exec plugin with:
+
+```
+[[exec.commands]]
+command = "/usr/bin/mycollector --output=json"
+name = "mycollector"
+interval = 10
+```
+
+The name is used as a prefix for the measurements.
+
+The interval is used to determine how often a particular command should be run. Each
+time the exec plugin runs, it will only run a particular command if it has been at least
+`interval` seconds since the exec plugin last ran the command.
+
+
+# Sample
+
+Let's say that we have a command named "mycollector", which gives the following output:
+```json
+{
+    "a": 0.5,
+    "b": {
+        "c": "some text",
+        "d": 0.1,
+        "e": 5
+    }
+}
+```
+
+The collected metrics will be:
+```
+exec_mycollector_a value=0.5
+exec_mycollector_b_d value=0.1
+exec_mycollector_b_e value=5
+```

plugins/exec/exec.go

@@ -0,0 +1,98 @@
+package exec
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"os/exec"
+
+	"github.com/gonuts/go-shellquote"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+const sampleConfig = `
+  # the command to run
+  command = "/usr/bin/mycollector --foo=bar"
+
+  # name of the command (used as a prefix for measurements)
+  name = "mycollector"
+`
+
+type Exec struct {
+	Command string
+	Name    string
+
+	runner Runner
+}
+
+type Runner interface {
+	Run(*Exec) ([]byte, error)
+}
+
+type CommandRunner struct{}
+
+func (c CommandRunner) Run(e *Exec) ([]byte, error) {
+	split_cmd, err := shellquote.Split(e.Command)
+	if err != nil || len(split_cmd) == 0 {
+		return nil, fmt.Errorf("exec: unable to parse command, %s", err)
+	}
+
+	cmd := exec.Command(split_cmd[0], split_cmd[1:]...)
+	var out bytes.Buffer
+	cmd.Stdout = &out
+
+	if err := cmd.Run(); err != nil {
+		return nil, fmt.Errorf("exec: %s for command '%s'", err, e.Command)
+	}
+
+	return out.Bytes(), nil
+}
+
+func NewExec() *Exec {
+	return &Exec{runner: CommandRunner{}}
+}
+
+func (e *Exec) SampleConfig() string {
+	return sampleConfig
+}
+
+func (e *Exec) Description() string {
+	return "Read flattened metrics from one or more commands that output JSON to stdout"
+}
+
+func (e *Exec) Gather(acc plugins.Accumulator) error {
+	out, err := e.runner.Run(e)
+	if err != nil {
+		return err
+	}
+
+	var jsonOut interface{}
+	err = json.Unmarshal(out, &jsonOut)
+	if err != nil {
+		return fmt.Errorf("exec: unable to parse output of '%s' as JSON, %s",
+			e.Command, err)
+	}
+
+	f := internal.JSONFlattener{}
+	err = f.FlattenJSON("", jsonOut)
+	if err != nil {
+		return err
+	}
+
+	var msrmnt_name string
+	if e.Name == "" {
+		msrmnt_name = "exec"
+	} else {
+		msrmnt_name = "exec_" + e.Name
+	}
+	acc.AddFields(msrmnt_name, f.Fields, nil)
+	return nil
+}
+
+func init() {
+	plugins.Add("exec", func() plugins.Plugin {
+		return NewExec()
+	})
+}

plugins/exec/exec_test.go

@@ -0,0 +1,262 @@
+package exec
+
+import (
+	"fmt"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"math"
+	"testing"
+	"time"
+)
+
+// Midnight 9/22/2015
+const baseTimeSeconds = 1442905200
+
+const validJson = `
+{
+    "status": "green",
+    "num_processes": 82,
+    "cpu": {
+        "status": "red",
+        "nil_status": null,
+        "used": 8234,
+        "free": 32
+    },
+    "percent": 0.81,
+    "users": [0, 1, 2, 3]
+}`
+
+const malformedJson = `
+{
+    "status": "green",
+`
+
+type runnerMock struct {
+	out []byte
+	err error
+}
+
+type clockMock struct {
+	now time.Time
+}
+
+func newRunnerMock(out []byte, err error) Runner {
+	return &runnerMock{
+		out: out,
+		err: err,
+	}
+}
+
+func (r runnerMock) Run(command *Command) ([]byte, error) {
+	if r.err != nil {
+		return nil, r.err
+	}
+	return r.out, nil
+}
+
+func newClockMock(now time.Time) Clock {
+	return &clockMock{now: now}
+}
+
+func (c clockMock) Now() time.Time {
+	return c.now
+}
+
+func TestExec(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+20, 0))
+	command := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mycollector_num_processes", 82},
+		{"mycollector_cpu_used", 8234},
+		{"mycollector_cpu_free", 32},
+		{"mycollector_percent", 0.81},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	assert.Equal(t, deltaPoints, 4, "non-numeric measurements should be ignored")
+}
+
+func TestExecMalformed(t *testing.T) {
+	runner := newRunnerMock([]byte(malformedJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+20, 0))
+	command := Command{
+		Command:   "badcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.Error(t, err)
+
+	assert.Equal(t, deltaPoints, 0, "No new points should have been added")
+}
+
+func TestCommandError(t *testing.T) {
+	runner := newRunnerMock(nil, fmt.Errorf("exit status code 1"))
+	clock := newClockMock(time.Unix(baseTimeSeconds+20, 0))
+	command := Command{
+		Command:   "badcommand",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.Error(t, err)
+
+	assert.Equal(t, deltaPoints, 0, "No new points should have been added")
+}
+
+func TestExecNotEnoughTime(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+5, 0))
+	command := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	assert.Equal(t, deltaPoints, 0, "No new points should have been added")
+}
+
+func TestExecUninitializedLastRunAt(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds, 0))
+	command := Command{
+		Command:  "testcommand arg1",
+		Name:     "mycollector",
+		Interval: math.MaxInt32,
+		// Uninitialized lastRunAt should default to time.Unix(0, 0), so this should
+		// run no matter what the interval is
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&command},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mycollector_num_processes", 82},
+		{"mycollector_cpu_used", 8234},
+		{"mycollector_cpu_free", 32},
+		{"mycollector_percent", 0.81},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	assert.Equal(t, deltaPoints, 4, "non-numeric measurements should be ignored")
+}
+func TestExecOneNotEnoughTimeAndOneEnoughTime(t *testing.T) {
+	runner := newRunnerMock([]byte(validJson), nil)
+	clock := newClockMock(time.Unix(baseTimeSeconds+5, 0))
+	notEnoughTimeCommand := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  10,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+	enoughTimeCommand := Command{
+		Command:   "testcommand arg1",
+		Name:      "mycollector",
+		Interval:  3,
+		lastRunAt: time.Unix(baseTimeSeconds, 0),
+	}
+
+	e := &Exec{
+		runner:   runner,
+		clock:    clock,
+		Commands: []*Command{&notEnoughTimeCommand, &enoughTimeCommand},
+	}
+
+	var acc testutil.Accumulator
+	initialPoints := len(acc.Points)
+	err := e.Gather(&acc)
+	deltaPoints := len(acc.Points) - initialPoints
+	require.NoError(t, err)
+
+	checkFloat := []struct {
+		name  string
+		value float64
+	}{
+		{"mycollector_num_processes", 82},
+		{"mycollector_cpu_used", 8234},
+		{"mycollector_cpu_free", 32},
+		{"mycollector_percent", 0.81},
+	}
+
+	for _, c := range checkFloat {
+		assert.True(t, acc.CheckValue(c.name, c.value))
+	}
+
+	assert.Equal(t, deltaPoints, 4, "Only one command should have been run")
+}

plugins/haproxy/haproxy.go

@@ -0,0 +1,364 @@
+package haproxy
+
+import (
+	"encoding/csv"
+	"fmt"
+	"github.com/influxdb/telegraf/plugins"
+	"io"
+	"net/http"
+	"net/url"
+	"strconv"
+	"sync"
+	"time"
+)
+
+//CSV format: https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.1
+const (
+	HF_PXNAME         = 0  // 0. pxname [LFBS]: proxy name
+	HF_SVNAME         = 1  // 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend, any name for server/listener)
+	HF_QCUR           = 2  //2. qcur [..BS]: current queued requests. For the backend this reports the number queued without a server assigned.
+	HF_QMAX           = 3  //3. qmax [..BS]: max value of qcur
+	HF_SCUR           = 4  // 4. scur [LFBS]: current sessions
+	HF_SMAX           = 5  //5. smax [LFBS]: max sessions
+	HF_SLIM           = 6  //6. slim [LFBS]: configured session limit
+	HF_STOT           = 7  //7. stot [LFBS]: cumulative number of connections
+	HF_BIN            = 8  //8. bin [LFBS]: bytes in
+	HF_BOUT           = 9  //9. bout [LFBS]: bytes out
+	HF_DREQ           = 10 //10. dreq [LFB.]: requests denied because of security concerns.
+	HF_DRESP          = 11 //11. dresp [LFBS]: responses denied because of security concerns.
+	HF_EREQ           = 12 //12. ereq [LF..]: request errors. Some of the possible causes are:
+	HF_ECON           = 13 //13. econ [..BS]: number of requests that encountered an error trying to
+	HF_ERESP          = 14 //14. eresp [..BS]: response errors. srv_abrt will be counted here also.  Some other errors are: - write error on the client socket (won't be counted for the server stat) - failure applying filters to the response.
+	HF_WRETR          = 15 //15. wretr [..BS]: number of times a connection to a server was retried.
+	HF_WREDIS         = 16 //16. wredis [..BS]: number of times a request was redispatched to another server. The server value counts the number of times that server was switched away from.
+	HF_STATUS         = 17 //17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)...)
+	HF_WEIGHT         = 18 //18. weight [..BS]: total weight (backend), server weight (server)
+	HF_ACT            = 19 //19. act [..BS]: number of active servers (backend), server is active (server)
+	HF_BCK            = 20 //20. bck [..BS]: number of backup servers (backend), server is backup (server)
+	HF_CHKFAIL        = 21 //21. chkfail [...S]: number of failed checks. (Only counts checks failed when the server is up.)
+	HF_CHKDOWN        = 22 //22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts transitions to the whole backend being down, rather than the sum of the counters for each server.
+	HF_LASTCHG        = 23 //23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
+	HF_DOWNTIME       = 24 //24. downtime [..BS]: total downtime (in seconds). The value for the backend is the downtime for the whole backend, not the sum of the server downtime.
+	HF_QLIMIT         = 25 //25. qlimit [...S]: configured maxqueue for the server, or nothing in the value is 0 (default, meaning no limit)
+	HF_PID            = 26 //26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
+	HF_IID            = 27 //27. iid [LFBS]: unique proxy id
+	HF_SID            = 28 //28. sid [L..S]: server id (unique inside a proxy)
+	HF_THROTTLE       = 29 //29. throttle [...S]: current throttle percentage for the server, when slowstart is active, or no value if not in slowstart.
+	HF_LBTOT          = 30 //30. lbtot [..BS]: total number of times a server was selected, either for new sessions, or when re-dispatching. The server counter is the number of times that server was selected.
+	HF_TRACKED        = 31 //31. tracked [...S]: id of proxy/server if tracking is enabled.
+	HF_TYPE           = 32 //32. type [LFBS]: (0                                                                                                                                                                                                  = frontend, 1 = backend, 2 = server, 3 = socket/listener)
+	HF_RATE           = 33 //33. rate [.FBS]: number of sessions per second over last elapsed second
+	HF_RATE_LIM       = 34 //34. rate_lim [.F..]: configured limit on new sessions per second
+	HF_RATE_MAX       = 35 //35. rate_max [.FBS]: max number of new sessions per second
+	HF_CHECK_STATUS   = 36 //36. check_status [...S]: status of last health check, one of:
+	HF_CHECK_CODE     = 37 //37. check_code [...S]: layer5-7 code, if available
+	HF_CHECK_DURATION = 38 //38. check_duration [...S]: time in ms took to finish last health check
+	HF_HRSP_1xx       = 39 //39. hrsp_1xx [.FBS]: http responses with 1xx code
+	HF_HRSP_2xx       = 40 //40. hrsp_2xx [.FBS]: http responses with 2xx code
+	HF_HRSP_3xx       = 41 //41. hrsp_3xx [.FBS]: http responses with 3xx code
+	HF_HRSP_4xx       = 42 //42. hrsp_4xx [.FBS]: http responses with 4xx code
+	HF_HRSP_5xx       = 43 //43. hrsp_5xx [.FBS]: http responses with 5xx code
+	HF_HRSP_OTHER     = 44 //44. hrsp_other [.FBS]: http responses with other codes (protocol error)
+	HF_HANAFAIL       = 45 //45. hanafail [...S]: failed health checks details
+	HF_REQ_RATE       = 46 //46. req_rate [.F..]: HTTP requests per second over last elapsed second
+	HF_REQ_RATE_MAX   = 47 //47. req_rate_max [.F..]: max number of HTTP requests per second observed
+	HF_REQ_TOT        = 48 //48. req_tot [.F..]: total number of HTTP requests received
+	HF_CLI_ABRT       = 49 //49. cli_abrt [..BS]: number of data transfers aborted by the client
+	HF_SRV_ABRT       = 50 //50. srv_abrt [..BS]: number of data transfers aborted by the server (inc. in eresp)
+	HF_COMP_IN        = 51 //51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
+	HF_COMP_OUT       = 52 //52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
+	HF_COMP_BYP       = 53 //53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor (CPU/BW limit)
+	HF_COMP_RSP       = 54 //54. comp_rsp [.FB.]: number of HTTP responses that were compressed
+	HF_LASTSESS       = 55 //55. lastsess [..BS]: number of seconds since last session assigned to server/backend
+	HF_LAST_CHK       = 56 //56. last_chk [...S]: last health check contents or textual error
+	HF_LAST_AGT       = 57 //57. last_agt [...S]: last agent check contents or textual error
+	HF_QTIME          = 58 //58. qtime [..BS]:
+	HF_CTIME          = 59 //59. ctime [..BS]:
+	HF_RTIME          = 60 //60. rtime [..BS]: (0 for TCP)
+	HF_TTIME          = 61 //61. ttime [..BS]: the average total session time in ms over the 1024 last requests
+)
+
+type haproxy struct {
+	Servers []string
+
+	client *http.Client
+}
+
+var sampleConfig = `
+  # An array of address to gather stats about. Specify an ip on hostname
+  # with optional port. ie localhost, 10.10.3.33:1936, etc.
+  #
+  # If no servers are specified, then default to 127.0.0.1:1936
+  servers = ["http://myhaproxy.com:1936", "http://anotherhaproxy.com:1936"]
+  # Or you can also use local socket(not work yet)
+  # servers = ["socket:/run/haproxy/admin.sock"]
+`
+
+func (r *haproxy) SampleConfig() string {
+	return sampleConfig
+}
+
+func (r *haproxy) Description() string {
+	return "Read metrics of haproxy, via socket or csv stats page"
+}
+
+// Reads stats from all configured servers accumulates stats.
+// Returns one of the errors encountered while gather stats (if any).
+func (g *haproxy) Gather(acc plugins.Accumulator) error {
+	if len(g.Servers) == 0 {
+		return g.gatherServer("http://127.0.0.1:1936", acc)
+	}
+
+	var wg sync.WaitGroup
+
+	var outerr error
+
+	for _, serv := range g.Servers {
+		wg.Add(1)
+		go func(serv string) {
+			defer wg.Done()
+			outerr = g.gatherServer(serv, acc)
+		}(serv)
+	}
+
+	wg.Wait()
+
+	return outerr
+}
+
+func (g *haproxy) gatherServer(addr string, acc plugins.Accumulator) error {
+	if g.client == nil {
+
+		client := &http.Client{}
+		g.client = client
+	}
+
+	u, err := url.Parse(addr)
+	if err != nil {
+		return fmt.Errorf("Unable parse server address '%s': %s", addr, err)
+	}
+
+	req, err := http.NewRequest("GET", fmt.Sprintf("%s://%s%s/;csv", u.Scheme, u.Host, u.Path), nil)
+	if u.User != nil {
+		p, _ := u.User.Password()
+		req.SetBasicAuth(u.User.Username(), p)
+	}
+
+	res, err := g.client.Do(req)
+	if err != nil {
+		return fmt.Errorf("Unable to connect to haproxy server '%s': %s", addr, err)
+	}
+
+	if res.StatusCode != 200 {
+		return fmt.Errorf("Unable to get valid stat result from '%s': %s", addr, err)
+	}
+
+	return importCsvResult(res.Body, acc, u.Host)
+}
+
+func importCsvResult(r io.Reader, acc plugins.Accumulator, host string) error {
+	csv := csv.NewReader(r)
+	result, err := csv.ReadAll()
+	now := time.Now()
+
+	for _, row := range result {
+		fields := make(map[string]interface{})
+		tags := map[string]string{
+			"server": host,
+			"proxy":  row[HF_PXNAME],
+			"sv":     row[HF_SVNAME],
+		}
+		for field, v := range row {
+			switch field {
+			case HF_QCUR:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["qcur"] = ival
+				}
+			case HF_QMAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["qmax"] = ival
+				}
+			case HF_SCUR:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["scur"] = ival
+				}
+			case HF_SMAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["smax"] = ival
+				}
+			case HF_STOT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["stot"] = ival
+				}
+			case HF_BIN:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["bin"] = ival
+				}
+			case HF_BOUT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["bout"] = ival
+				}
+			case HF_DREQ:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["dreq"] = ival
+				}
+			case HF_DRESP:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["dresp"] = ival
+				}
+			case HF_EREQ:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["ereq"] = ival
+				}
+			case HF_ECON:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["econ"] = ival
+				}
+			case HF_ERESP:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["eresp"] = ival
+				}
+			case HF_WRETR:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["wretr"] = ival
+				}
+			case HF_WREDIS:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["wredis"] = ival
+				}
+			case HF_ACT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["active_servers"] = ival
+				}
+			case HF_BCK:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["backup_servers"] = ival
+				}
+			case HF_DOWNTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["downtime"] = ival
+				}
+			case HF_THROTTLE:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["throttle"] = ival
+				}
+			case HF_LBTOT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["lbtot"] = ival
+				}
+			case HF_RATE:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["rate"] = ival
+				}
+			case HF_RATE_MAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["rate_max"] = ival
+				}
+			case HF_CHECK_DURATION:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["check_duration"] = ival
+				}
+			case HF_HRSP_1xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.1xx"] = ival
+				}
+			case HF_HRSP_2xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.2xx"] = ival
+				}
+			case HF_HRSP_3xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.3xx"] = ival
+				}
+			case HF_HRSP_4xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.4xx"] = ival
+				}
+			case HF_HRSP_5xx:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["http_response.5xx"] = ival
+				}
+			case HF_REQ_RATE:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["req_rate"] = ival
+				}
+			case HF_REQ_RATE_MAX:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["req_rate_max"] = ival
+				}
+			case HF_REQ_TOT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["req_tot"] = ival
+				}
+			case HF_CLI_ABRT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["cli_abort"] = ival
+				}
+			case HF_SRV_ABRT:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["srv_abort"] = ival
+				}
+			case HF_QTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["qtime"] = ival
+				}
+			case HF_CTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["ctime"] = ival
+				}
+			case HF_RTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["rtime"] = ival
+				}
+			case HF_TTIME:
+				ival, err := strconv.ParseUint(v, 10, 64)
+				if err == nil {
+					fields["ttime"] = ival
+				}
+			}
+		}
+		acc.AddFields("haproxy", fields, tags, now)
+	}
+	return err
+}
+
+func init() {
+	plugins.Add("haproxy", func() plugins.Plugin {
+		return &haproxy{}
+	})
+}

plugins/haproxy/haproxy_test.go

@@ -0,0 +1,163 @@
+package haproxy
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"net/http"
+	"net/http/httptest"
+)
+
+func TestHaproxyGeneratesMetricsWithAuthentication(t *testing.T) {
+	//We create a fake server to return test data
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		username, password, ok := r.BasicAuth()
+		if !ok {
+			w.WriteHeader(http.StatusNotFound)
+			fmt.Fprint(w, "Unauthorized")
+			return
+		}
+
+		if username == "user" && password == "password" {
+			fmt.Fprint(w, csvOutputSample)
+		} else {
+			w.WriteHeader(http.StatusNotFound)
+			fmt.Fprint(w, "Unauthorized")
+		}
+	}))
+	defer ts.Close()
+
+	//Now we tested again above server, with our authentication data
+	r := &haproxy{
+		Servers: []string{strings.Replace(ts.URL, "http://", "http://user:password@", 1)},
+	}
+
+	var acc testutil.Accumulator
+
+	err := r.Gather(&acc)
+	require.NoError(t, err)
+
+	tags := map[string]string{
+		"server": ts.Listener.Addr().String(),
+		"proxy":  "be_app",
+		"sv":     "host0",
+	}
+
+	assert.NoError(t, acc.ValidateTaggedValue("stot", uint64(171014), tags))
+
+	checkInt := []struct {
+		name  string
+		value uint64
+	}{
+
+		{"qmax", 81},
+		{"scur", 288},
+		{"smax", 713},
+		{"bin", 5557055817},
+		{"bout", 24096715169},
+		{"dreq", 1102},
+		{"dresp", 80},
+		{"ereq", 95740},
+		{"econ", 0},
+		{"eresp", 0},
+		{"wretr", 17},
+		{"wredis", 19},
+		{"active_servers", 1},
+		{"backup_servers", 0},
+		{"downtime", 0},
+		{"throttle", 13},
+		{"lbtot", 114},
+		{"rate", 18},
+		{"rate_max", 102},
+		{"check_duration", 1},
+		{"http_response.1xx", 0},
+		{"http_response.2xx", 1314093},
+		{"http_response.3xx", 537036},
+		{"http_response.4xx", 123452},
+		{"http_response.5xx", 11966},
+		{"req_rate", 35},
+		{"req_rate_max", 140},
+		{"req_tot", 1987928},
+		{"cli_abort", 0},
+		{"srv_abort", 0},
+		{"qtime", 0},
+		{"ctime", 2},
+		{"rtime", 23},
+		{"ttime", 545},
+	}
+
+	for _, c := range checkInt {
+		assert.Equal(t, true, acc.CheckValue(c.name, c.value))
+	}
+
+	//Here, we should get error because we don't pass authentication data
+	r = &haproxy{
+		Servers: []string{ts.URL},
+	}
+
+	err = r.Gather(&acc)
+	require.Error(t, err)
+}
+
+func TestHaproxyGeneratesMetricsWithoutAuthentication(t *testing.T) {
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		fmt.Fprint(w, csvOutputSample)
+	}))
+	defer ts.Close()
+
+	r := &haproxy{
+		Servers: []string{ts.URL},
+	}
+
+	var acc testutil.Accumulator
+
+	err := r.Gather(&acc)
+	require.NoError(t, err)
+
+	tags := map[string]string{
+		"proxy":  "be_app",
+		"server": ts.Listener.Addr().String(),
+		"sv":     "host0",
+	}
+
+	assert.NoError(t, acc.ValidateTaggedValue("stot", uint64(171014), tags))
+	assert.NoError(t, acc.ValidateTaggedValue("scur", uint64(1), tags))
+	assert.NoError(t, acc.ValidateTaggedValue("rate", uint64(3), tags))
+	assert.Equal(t, true, acc.CheckValue("bin", uint64(5557055817)))
+}
+
+//When not passing server config, we default to localhost
+//We just want to make sure we did request stat from localhost
+func TestHaproxyDefaultGetFromLocalhost(t *testing.T) {
+	r := &haproxy{}
+
+	var acc testutil.Accumulator
+
+	err := r.Gather(&acc)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "127.0.0.1:1936/;csv")
+}
+
+const csvOutputSample = `
+# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
+fe_app,FRONTEND,,81,288,713,2000,1094063,5557055817,24096715169,1102,80,95740,,,17,19,OPEN,,,,,,,,,2,16,113,13,114,,0,18,0,102,,,,0,1314093,537036,123452,11966,1360,,35,140,1987928,,,0,0,0,0,,,,,,,,
+be_static,host0,0,0,0,3,,3209,1141294,17389596,,0,,0,0,0,0,no check,1,1,0,,,,,,2,17,1,,3209,,2,0,,7,,,,0,218,1497,1494,0,0,0,,,,0,0,,,,,2,,,0,2,23,545,
+be_static,BACKEND,0,0,0,3,200,3209,1141294,17389596,0,0,,0,0,0,0,UP,1,1,0,,0,70698,0,,2,17,0,,3209,,1,0,,7,,,,0,218,1497,1494,0,0,,,,,0,0,0,0,0,0,2,,,0,2,23,545,
+be_static,host0,0,0,0,1,,28,17313,466003,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,1,,28,,2,0,,1,L4OK,,1,0,17,6,5,0,0,0,,,,0,0,,,,,2103,,,0,1,1,36,
+be_static,host4,0,0,0,1,,28,15358,1281073,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,2,,28,,2,0,,1,L4OK,,1,0,20,5,3,0,0,0,,,,0,0,,,,,2076,,,0,1,1,54,
+be_static,host5,0,0,0,1,,28,17547,1970404,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,3,,28,,2,0,,1,L4OK,,0,0,20,5,3,0,0,0,,,,0,0,,,,,1495,,,0,1,1,53,
+be_static,host6,0,0,0,1,,28,14105,1328679,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,4,,28,,2,0,,1,L4OK,,0,0,18,8,2,0,0,0,,,,0,0,,,,,1418,,,0,0,1,49,
+be_static,host7,0,0,0,1,,28,15258,1965185,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,5,,28,,2,0,,1,L4OK,,0,0,17,8,3,0,0,0,,,,0,0,,,,,935,,,0,0,1,28,
+be_static,host8,0,0,0,1,,28,12934,1034779,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,6,,28,,2,0,,1,L4OK,,0,0,17,9,2,0,0,0,,,,0,0,,,,,582,,,0,1,1,66,
+be_static,host9,0,0,0,1,,28,13434,134063,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,7,,28,,2,0,,1,L4OK,,0,0,17,8,3,0,0,0,,,,0,0,,,,,539,,,0,0,1,80,
+be_static,host1,0,0,0,1,,28,7873,1209688,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,8,,28,,2,0,,1,L4OK,,0,0,22,6,0,0,0,0,,,,0,0,,,,,487,,,0,0,1,36,
+be_static,host2,0,0,0,1,,28,13830,1085929,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,9,,28,,2,0,,1,L4OK,,0,0,19,6,3,0,0,0,,,,0,0,,,,,338,,,0,1,1,38,
+be_static,host3,0,0,0,1,,28,17959,1259760,,0,,0,0,0,0,UP,1,1,0,0,0,70698,0,,2,18,10,,28,,2,0,,1,L4OK,,1,0,20,6,2,0,0,0,,,,0,0,,,,,92,,,0,1,1,17,
+be_static,BACKEND,0,0,0,2,200,307,160276,13322728,0,0,,0,0,0,0,UP,11,11,0,,0,70698,0,,2,18,0,,307,,1,0,,4,,,,0,205,73,29,0,0,,,,,0,0,0,0,0,0,92,,,0,1,3,381,
+be_app,host0,0,0,1,32,,171014,510913516,2193856571,,0,,0,1,1,0,UP,100,1,0,1,0,70698,0,,2,19,1,,171013,,2,3,,12,L7OK,301,10,0,119534,48051,2345,1056,0,0,,,,73,1,,,,,0,Moved Permanently,,0,2,312,2341,
+be_app,host4,0,0,2,29,,171013,499318742,2195595896,12,34,,0,2,0,0,UP,100,1,0,2,0,70698,0,,2,19,2,,171013,,2,3,,12,L7OK,301,12,0,119572,47882,2441,1088,0,0,,,,84,2,,,,,0,Moved Permanently,,0,2,316,2355,
+`

plugins/httpjson/README.md

@@ -0,0 +1,148 @@
+# HTTP JSON Plugin
+
+The httpjson plugin can collect data from remote URLs which respond with JSON. Then it flattens JSON and finds all numeric values, treating them as floats.
+
+For example, if you have a service called _mycollector_, which has HTTP endpoint for gathering stats at http://my.service.com/_stats, you would configure the HTTP JSON
+plugin like this:
+
+```
+[[httpjson.services]]
+  name = "mycollector"
+
+  servers = [
+    "http://my.service.com/_stats"
+  ]
+
+  # HTTP method to use (case-sensitive)
+  method = "GET"
+```
+
+`name` is used as a prefix for the measurements.
+
+`method` specifies HTTP method to use for requests.
+
+You can also specify which keys from server response should be considered tags:
+
+```
+[[httpjson.services]]
+  ...
+
+  tag_keys = [
+    "role",
+    "version"
+  ]
+```
+
+You can also specify additional request parameters for the service:
+
+```
+[[httpjson.services]]
+  ...
+
+ [httpjson.services.parameters]
+    event_type = "cpu_spike"
+    threshold = "0.75"
+
+```
+
+
+# Example:
+
+Let's say that we have a service named "mycollector" configured like this:
+
+```
+[httpjson]
+  [[httpjson.services]]
+    name = "mycollector"
+
+    servers = [
+      "http://my.service.com/_stats"
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "GET"
+
+    tag_keys = ["service"]
+```
+
+which responds with the following JSON:
+
+```json
+{
+    "service": "service01",
+    "a": 0.5,
+    "b": {
+        "c": "some text",
+        "d": 0.1,
+        "e": 5
+    }
+}
+```
+
+The collected metrics will be:
+```
+httpjson_mycollector_a,service='service01',server='http://my.service.com/_stats' value=0.5
+httpjson_mycollector_b_d,service='service01',server='http://my.service.com/_stats' value=0.1
+httpjson_mycollector_b_e,service='service01',server='http://my.service.com/_stats' value=5
+```
+
+# Example 2, Multiple Services:
+
+There is also the option to collect JSON from multiple services, here is an
+example doing that.
+
+```
+[httpjson]
+  [[httpjson.services]]
+    name = "mycollector1"
+
+    servers = [
+      "http://my.service1.com/_stats"
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "GET"
+
+  [[httpjson.services]]
+    name = "mycollector2"
+
+    servers = [
+      "http://service.net/json/stats"
+    ]
+
+    # HTTP method to use (case-sensitive)
+    method = "POST"
+```
+
+The services respond with the following JSON:
+
+mycollector1:
+```json
+{
+    "a": 0.5,
+    "b": {
+        "c": "some text",
+        "d": 0.1,
+        "e": 5
+    }
+}
+```
+
+mycollector2:
+```json
+{
+    "load": 100,
+    "users": 1335
+}
+```
+
+The collected metrics will be:
+
+```
+httpjson_mycollector1_a,server='http://my.service.com/_stats' value=0.5
+httpjson_mycollector1_b_d,server='http://my.service.com/_stats' value=0.1
+httpjson_mycollector1_b_e,server='http://my.service.com/_stats' value=5
+
+httpjson_mycollector2_load,server='http://service.net/json/stats' value=100
+httpjson_mycollector2_users,server='http://service.net/json/stats' value=1335
+```

plugins/httpjson/httpjson.go

@@ -0,0 +1,217 @@
+package httpjson
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/url"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type HttpJson struct {
+	Name       string
+	Servers    []string
+	Method     string
+	TagKeys    []string
+	Parameters map[string]string
+	client     HTTPClient
+}
+
+type HTTPClient interface {
+	// Returns the result of an http request
+	//
+	// Parameters:
+	// req: HTTP request object
+	//
+	// Returns:
+	// http.Response:  HTTP respons object
+	// error        :  Any error that may have occurred
+	MakeRequest(req *http.Request) (*http.Response, error)
+}
+
+type RealHTTPClient struct {
+	client *http.Client
+}
+
+func (c RealHTTPClient) MakeRequest(req *http.Request) (*http.Response, error) {
+	return c.client.Do(req)
+}
+
+var sampleConfig = `
+  # a name for the service being polled
+  name = "webserver_stats"
+
+  # URL of each server in the service's cluster
+  servers = [
+    "http://localhost:9999/stats/",
+    "http://localhost:9998/stats/",
+  ]
+
+  # HTTP method to use (case-sensitive)
+  method = "GET"
+
+  # List of tag names to extract from top-level of JSON server response
+  # tag_keys = [
+  #   "my_tag_1",
+  #   "my_tag_2"
+  # ]
+
+  # HTTP parameters (all values must be strings)
+  [plugins.httpjson.parameters]
+    event_type = "cpu_spike"
+    threshold = "0.75"
+`
+
+func (h *HttpJson) SampleConfig() string {
+	return sampleConfig
+}
+
+func (h *HttpJson) Description() string {
+	return "Read flattened metrics from one or more JSON HTTP endpoints"
+}
+
+// Gathers data for all servers.
+func (h *HttpJson) Gather(acc plugins.Accumulator) error {
+	var wg sync.WaitGroup
+
+	errorChannel := make(chan error, len(h.Servers))
+
+	for _, server := range h.Servers {
+		wg.Add(1)
+		go func(server string) {
+			defer wg.Done()
+			if err := h.gatherServer(acc, server); err != nil {
+				errorChannel <- err
+			}
+		}(server)
+	}
+
+	wg.Wait()
+	close(errorChannel)
+
+	// Get all errors and return them as one giant error
+	errorStrings := []string{}
+	for err := range errorChannel {
+		errorStrings = append(errorStrings, err.Error())
+	}
+
+	if len(errorStrings) == 0 {
+		return nil
+	}
+	return errors.New(strings.Join(errorStrings, "\n"))
+}
+
+// Gathers data from a particular server
+// Parameters:
+//     acc      : The telegraf Accumulator to use
+//     serverURL: endpoint to send request to
+//     service  : the service being queried
+//
+// Returns:
+//     error: Any error that may have occurred
+func (h *HttpJson) gatherServer(
+	acc plugins.Accumulator,
+	serverURL string,
+) error {
+	resp, err := h.sendRequest(serverURL)
+	if err != nil {
+		return err
+	}
+
+	var jsonOut map[string]interface{}
+	if err = json.Unmarshal([]byte(resp), &jsonOut); err != nil {
+		return errors.New("Error decoding JSON response")
+	}
+
+	tags := map[string]string{
+		"server": serverURL,
+	}
+
+	for _, tag := range h.TagKeys {
+		switch v := jsonOut[tag].(type) {
+		case string:
+			tags[tag] = v
+		}
+		delete(jsonOut, tag)
+	}
+
+	f := internal.JSONFlattener{}
+	err = f.FlattenJSON("", jsonOut)
+	if err != nil {
+		return err
+	}
+
+	var msrmnt_name string
+	if h.Name == "" {
+		msrmnt_name = "httpjson"
+	} else {
+		msrmnt_name = "httpjson_" + h.Name
+	}
+	acc.AddFields(msrmnt_name, f.Fields, nil)
+	return nil
+}
+
+// Sends an HTTP request to the server using the HttpJson object's HTTPClient
+// Parameters:
+//     serverURL: endpoint to send request to
+//
+// Returns:
+//     string: body of the response
+//     error : Any error that may have occurred
+func (h *HttpJson) sendRequest(serverURL string) (string, error) {
+	// Prepare URL
+	requestURL, err := url.Parse(serverURL)
+	if err != nil {
+		return "", fmt.Errorf("Invalid server URL \"%s\"", serverURL)
+	}
+
+	params := url.Values{}
+	for k, v := range h.Parameters {
+		params.Add(k, v)
+	}
+	requestURL.RawQuery = params.Encode()
+
+	// Create + send request
+	req, err := http.NewRequest(h.Method, requestURL.String(), nil)
+	if err != nil {
+		return "", err
+	}
+	defer req.Body.Close()
+
+	resp, err := h.client.MakeRequest(req)
+	if err != nil {
+		return "", err
+	}
+	defer resp.Body.Close()
+
+	defer resp.Body.Close()
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return string(body), err
+	}
+
+	// Process response
+	if resp.StatusCode != http.StatusOK {
+		err = fmt.Errorf("Response from url \"%s\" has status code %d (%s), expected %d (%s)",
+			requestURL.String(),
+			resp.StatusCode,
+			http.StatusText(resp.StatusCode),
+			http.StatusOK,
+			http.StatusText(http.StatusOK))
+		return string(body), err
+	}
+
+	return string(body), err
+}
+
+func init() {
+	plugins.Add("httpjson", func() plugins.Plugin {
+		return &HttpJson{client: RealHTTPClient{client: &http.Client{}}}
+	})
+}

plugins/httpjson/httpjson_test.go

@@ -0,0 +1,223 @@
+package httpjson
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const validJSON = `
+	{
+		"parent": {
+			"child": 3,
+			"ignored_child": "hi"
+		},
+		"ignored_null": null,
+		"integer": 4,
+		"ignored_list": [3, 4],
+		"ignored_parent": {
+			"another_ignored_list": [4],
+			"another_ignored_null": null,
+			"ignored_string": "hello, world!"
+		}
+	}`
+
+const validJSONTags = `
+	{
+		"value": 15,
+		"role": "master",
+		"build": "123"
+	}`
+
+const invalidJSON = "I don't think this is JSON"
+
+const empty = ""
+
+type mockHTTPClient struct {
+	responseBody string
+	statusCode   int
+}
+
+// Mock implementation of MakeRequest. Usually returns an http.Response with
+// hard-coded responseBody and statusCode. However, if the request uses a
+// nonstandard method, it uses status code 405 (method not allowed)
+func (c mockHTTPClient) MakeRequest(req *http.Request) (*http.Response, error) {
+	resp := http.Response{}
+	resp.StatusCode = c.statusCode
+
+	// basic error checking on request method
+	allowedMethods := []string{"GET", "HEAD", "POST", "PUT", "DELETE", "TRACE", "CONNECT"}
+	methodValid := false
+	for _, method := range allowedMethods {
+		if req.Method == method {
+			methodValid = true
+			break
+		}
+	}
+
+	if !methodValid {
+		resp.StatusCode = 405 // Method not allowed
+	}
+
+	resp.Body = ioutil.NopCloser(strings.NewReader(c.responseBody))
+	return &resp, nil
+}
+
+// Generates a pointer to an HttpJson object that uses a mock HTTP client.
+// Parameters:
+//     response  : Body of the response that the mock HTTP client should return
+//     statusCode: HTTP status code the mock HTTP client should return
+//
+// Returns:
+//     *HttpJson: Pointer to an HttpJson object that uses the generated mock HTTP client
+func genMockHttpJson(response string, statusCode int) *HttpJson {
+	return &HttpJson{
+		client: mockHTTPClient{responseBody: response, statusCode: statusCode},
+		Services: []Service{
+			Service{
+				Servers: []string{
+					"http://server1.example.com/metrics/",
+					"http://server2.example.com/metrics/",
+				},
+				Name:   "my_webapp",
+				Method: "GET",
+				Parameters: map[string]string{
+					"httpParam1": "12",
+					"httpParam2": "the second parameter",
+				},
+			},
+			Service{
+				Servers: []string{
+					"http://server3.example.com/metrics/",
+					"http://server4.example.com/metrics/",
+				},
+				Name:   "other_webapp",
+				Method: "POST",
+				Parameters: map[string]string{
+					"httpParam1": "12",
+					"httpParam2": "the second parameter",
+				},
+				TagKeys: []string{
+					"role",
+					"build",
+				},
+			},
+		},
+	}
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJson200(t *testing.T) {
+	httpjson := genMockHttpJson(validJSON, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+	require.NoError(t, err)
+
+	assert.Equal(t, 8, len(acc.Points))
+
+	for _, service := range httpjson.Services {
+		for _, srv := range service.Servers {
+			require.NoError(t,
+				acc.ValidateTaggedValue(
+					fmt.Sprintf("%s_parent_child", service.Name),
+					3.0,
+					map[string]string{"server": srv},
+				),
+			)
+			require.NoError(t,
+				acc.ValidateTaggedValue(
+					fmt.Sprintf("%s_integer", service.Name),
+					4.0,
+					map[string]string{"server": srv},
+				),
+			)
+		}
+	}
+}
+
+// Test response to HTTP 500
+func TestHttpJson500(t *testing.T) {
+	httpjson := genMockHttpJson(validJSON, 500)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 4 error lines for (2 urls) * (2 services)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 4)
+	assert.Equal(t, 0, len(acc.Points))
+}
+
+// Test response to HTTP 405
+func TestHttpJsonBadMethod(t *testing.T) {
+	httpjson := genMockHttpJson(validJSON, 200)
+	httpjson.Services[0].Method = "NOT_A_REAL_METHOD"
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 2 error lines for (2 urls) * (1 falied service)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 2)
+
+	// (2 measurements) * (2 servers) * (1 successful service)
+	assert.Equal(t, 4, len(acc.Points))
+}
+
+// Test response to malformed JSON
+func TestHttpJsonBadJson(t *testing.T) {
+	httpjson := genMockHttpJson(invalidJSON, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 4 error lines for (2 urls) * (2 services)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 4)
+	assert.Equal(t, 0, len(acc.Points))
+}
+
+// Test response to empty string as response objectgT
+func TestHttpJsonEmptyResponse(t *testing.T) {
+	httpjson := genMockHttpJson(empty, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+
+	assert.NotNil(t, err)
+	// 4 error lines for (2 urls) * (2 services)
+	assert.Equal(t, len(strings.Split(err.Error(), "\n")), 4)
+	assert.Equal(t, 0, len(acc.Points))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJson200Tags(t *testing.T) {
+	httpjson := genMockHttpJson(validJSONTags, 200)
+
+	var acc testutil.Accumulator
+	err := httpjson.Gather(&acc)
+	require.NoError(t, err)
+
+	assert.Equal(t, 4, len(acc.Points))
+
+	for _, service := range httpjson.Services {
+		if service.Name == "other_webapp" {
+			for _, srv := range service.Servers {
+				require.NoError(t,
+					acc.ValidateTaggedValue(
+						fmt.Sprintf("%s_value", service.Name),
+						15.0,
+						map[string]string{"server": srv, "role": "master", "build": "123"},
+					),
+				)
+			}
+		}
+	}
+}

plugins/influxdb/README.md

@@ -0,0 +1,72 @@
+# influxdb plugin
+
+The influxdb plugin collects InfluxDB-formatted data from JSON endpoints.
+
+With a configuration of:
+
+```toml
+[[plugins.influxdb]]
+  urls = [
+    "http://127.0.0.1:8086/debug/vars",
+    "http://192.168.2.1:8086/debug/vars"
+  ]
+```
+
+And if 127.0.0.1 responds with this JSON:
+
+```json
+{
+  "k1": {
+    "name": "fruit",
+    "tags": {
+      "kind": "apple"
+    },
+    "values": {
+      "inventory": 371,
+      "sold": 112
+    }
+  },
+  "k2": {
+    "name": "fruit",
+    "tags": {
+      "kind": "banana"
+    },
+    "values": {
+      "inventory": 1000,
+      "sold": 403
+    }
+  }
+}
+```
+
+And if 192.168.2.1 responds like so:
+
+```json
+{
+  "k3": {
+    "name": "transactions",
+    "tags": {},
+    "values": {
+      "total": 100,
+      "balance": 184.75
+    }
+  }
+}
+```
+
+Then the collected metrics will be:
+
+```
+influxdb_fruit,url='http://127.0.0.1:8086/debug/vars',kind='apple' inventory=371.0,sold=112.0
+influxdb_fruit,url='http://127.0.0.1:8086/debug/vars',kind='banana' inventory=1000.0,sold=403.0
+
+influxdb_transactions,url='http://192.168.2.1:8086/debug/vars' total=100.0,balance=184.75
+```
+
+There are two important details to note about the collected metrics:
+
+1. Even though the values in JSON are being displayed as integers, the metrics are reported as floats.
+JSON encoders usually don't print the fractional part for round floats.
+Because you cannot change the type of an existing field in InfluxDB, we assume all numbers are floats.
+
+2. The top-level keys' names (in the example above, `"k1"`, `"k2"`, and `"k3"`) are not considered when recording the metrics.

plugins/influxdb/influxdb.go

@@ -0,0 +1,146 @@
+package influxdb
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type InfluxDB struct {
+	URLs []string `toml:"urls"`
+}
+
+func (*InfluxDB) Description() string {
+	return "Read InfluxDB-formatted JSON metrics from one or more HTTP endpoints"
+}
+
+func (*InfluxDB) SampleConfig() string {
+	return `
+  # Works with InfluxDB debug endpoints out of the box,
+  # but other services can use this format too.
+  # See the influxdb plugin's README for more details.
+
+  # Multiple URLs from which to read InfluxDB-formatted JSON
+  urls = [
+    "http://localhost:8086/debug/vars"
+  ]
+`
+}
+
+func (i *InfluxDB) Gather(acc plugins.Accumulator) error {
+	errorChannel := make(chan error, len(i.URLs))
+
+	var wg sync.WaitGroup
+	for _, u := range i.URLs {
+		wg.Add(1)
+		go func(url string) {
+			defer wg.Done()
+			if err := i.gatherURL(acc, url); err != nil {
+				errorChannel <- fmt.Errorf("[url=%s]: %s", url, err)
+			}
+		}(u)
+	}
+
+	wg.Wait()
+	close(errorChannel)
+
+	// If there weren't any errors, we can return nil now.
+	if len(errorChannel) == 0 {
+		return nil
+	}
+
+	// There were errors, so join them all together as one big error.
+	errorStrings := make([]string, 0, len(errorChannel))
+	for err := range errorChannel {
+		errorStrings = append(errorStrings, err.Error())
+	}
+
+	return errors.New(strings.Join(errorStrings, "\n"))
+}
+
+type point struct {
+	Name   string                 `json:"name"`
+	Tags   map[string]string      `json:"tags"`
+	Values map[string]interface{} `json:"values"`
+}
+
+// Gathers data from a particular URL
+// Parameters:
+//     acc    : The telegraf Accumulator to use
+//     url    : endpoint to send request to
+//
+// Returns:
+//     error: Any error that may have occurred
+func (i *InfluxDB) gatherURL(
+	acc plugins.Accumulator,
+	url string,
+) error {
+	resp, err := http.Get(url)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+
+	// It would be nice to be able to decode into a map[string]point, but
+	// we'll get a decoder error like:
+	// `json: cannot unmarshal array into Go value of type influxdb.point`
+	// if any of the values aren't objects.
+	// To avoid that error, we decode by hand.
+	dec := json.NewDecoder(resp.Body)
+
+	// Parse beginning of object
+	if t, err := dec.Token(); err != nil {
+		return err
+	} else if t != json.Delim('{') {
+		return errors.New("document root must be a JSON object")
+	}
+
+	// Loop through rest of object
+	for {
+		// Nothing left in this object, we're done
+		if !dec.More() {
+			break
+		}
+
+		// Read in a string key. We don't do anything with the top-level keys, so it's discarded.
+		_, err := dec.Token()
+		if err != nil {
+			return err
+		}
+
+		// Attempt to parse a whole object into a point.
+		// It might be a non-object, like a string or array.
+		// If we fail to decode it into a point, ignore it and move on.
+		var p point
+		if err := dec.Decode(&p); err != nil {
+			continue
+		}
+
+		// If the object was a point, but was not fully initialized, ignore it and move on.
+		if p.Name == "" || p.Tags == nil || p.Values == nil || len(p.Values) == 0 {
+			continue
+		}
+
+		// Add a tag to indicate the source of the data.
+		p.Tags["url"] = url
+
+		acc.AddFields(
+			p.Name,
+			p.Values,
+			p.Tags,
+		)
+	}
+
+	return nil
+}
+
+func init() {
+	plugins.Add("influxdb", func() plugins.Plugin {
+		return &InfluxDB{}
+	})
+}

plugins/influxdb/influxdb_test.go

@@ -0,0 +1,100 @@
+package influxdb_test
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/influxdb/telegraf/plugins/influxdb"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/require"
+)
+
+func TestBasic(t *testing.T) {
+	js := `
+{
+  "_1": {
+    "name": "foo",
+    "tags": {
+      "id": "ex1"
+    },
+    "values": {
+      "i": -1,
+      "f": 0.5,
+      "b": true,
+      "s": "string"
+    }
+  },
+  "ignored": {
+    "willBeRecorded": false
+  },
+  "ignoredAndNested": {
+    "hash": {
+      "is": "nested"
+    }
+  },
+  "array": [
+   "makes parsing more difficult than necessary"
+  ],
+  "string": "makes parsing more difficult than necessary",
+  "_2": {
+    "name": "bar",
+    "tags": {
+      "id": "ex2"
+    },
+    "values": {
+      "x": "x"
+    }
+  },
+  "pointWithoutFields_willNotBeIncluded": {
+    "name": "asdf",
+    "tags": {
+      "id": "ex3"
+    },
+    "values": {}
+  }
+}
+`
+	fakeServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/endpoint" {
+			_, _ = w.Write([]byte(js))
+		} else {
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	defer fakeServer.Close()
+
+	plugin := &influxdb.InfluxDB{
+		URLs: []string{fakeServer.URL + "/endpoint"},
+	}
+
+	var acc testutil.Accumulator
+	require.NoError(t, plugin.Gather(&acc))
+
+	require.Len(t, acc.Points, 2)
+	require.NoError(t, acc.ValidateTaggedFieldsValue(
+		"foo",
+		map[string]interface{}{
+			// JSON will truncate floats to integer representations.
+			// Since there's no distinction in JSON, we can't assume it's an int.
+			"i": -1.0,
+			"f": 0.5,
+			"b": true,
+			"s": "string",
+		},
+		map[string]string{
+			"id":  "ex1",
+			"url": fakeServer.URL + "/endpoint",
+		},
+	))
+	require.NoError(t, acc.ValidateTaggedFieldsValue(
+		"bar",
+		map[string]interface{}{
+			"x": "x",
+		},
+		map[string]string{
+			"id":  "ex2",
+			"url": fakeServer.URL + "/endpoint",
+		},
+	))
+}

plugins/jolokia/README.md

@@ -0,0 +1,51 @@
+# Telegraf plugin: Jolokia
+
+#### Plugin arguments:
+- **context** string: Context root used of jolokia url
+- **servers** []Server: List of servers
+  + **name** string: Server's logical name
+  + **host** string: Server's ip address or hostname
+  + **port** string: Server's listening port
+- **metrics** []Metric
+  + **name** string: Name of the measure
+  + **jmx** string: Jmx path that identifies mbeans attributes
+  + **pass** []string: Attributes to retain when collecting values
+  + **drop** []string: Attributes to drop when collecting values
+
+#### Description
+
+The Jolokia plugin collects JVM metrics exposed as MBean's attributes through jolokia REST endpoint. All metrics
+are collected for each server configured.
+
+See: https://jolokia.org/
+
+# Measurements:
+Jolokia plugin produces one measure for each metric configured, adding Server's `name`, `host` and `port` as tags.
+
+Given a configuration like:
+
+```ini
+[jolokia]
+
+[[jolokia.servers]]
+  name = "as-service-1"
+  host = "127.0.0.1"
+  port = "8080"
+
+[[jolokia.servers]]
+  name = "as-service-2"
+  host = "127.0.0.1"
+  port = "8180"
+
+[[jolokia.metrics]]
+  name = "heap_memory_usage"
+  jmx  = "/java.lang:type=Memory/HeapMemoryUsage"
+  pass = ["used", "max"]
+```
+
+The collected metrics will be:
+
+```
+jolokia_heap_memory_usage name=as-service-1,host=127.0.0.1,port=8080 used=xxx,max=yyy
+jolokia_heap_memory_usage name=as-service-2,host=127.0.0.1,port=8180 used=vvv,max=zzz
+```

plugins/jolokia/jolokia.go

@@ -0,0 +1,164 @@
+package jolokia
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/url"
+
+	"github.com/influxdb/telegraf/plugins"
+)
+
+type Server struct {
+	Name     string
+	Host     string
+	Username string
+	Password string
+	Port     string
+}
+
+type Metric struct {
+	Name string
+	Jmx  string
+}
+
+type JolokiaClient interface {
+	MakeRequest(req *http.Request) (*http.Response, error)
+}
+
+type JolokiaClientImpl struct {
+	client *http.Client
+}
+
+func (c JolokiaClientImpl) MakeRequest(req *http.Request) (*http.Response, error) {
+	return c.client.Do(req)
+}
+
+type Jolokia struct {
+	jClient JolokiaClient
+	Context string
+	Servers []Server
+	Metrics []Metric
+}
+
+func (j *Jolokia) SampleConfig() string {
+	return `
+  # This is the context root used to compose the jolokia url
+  context = "/jolokia/read"
+
+  # List of servers exposing jolokia read service
+  [[plugins.jolokia.servers]]
+    name = "stable"
+    host = "192.168.103.2"
+    port = "8180"
+    # username = "myuser"
+    # password = "mypassword"
+
+  # List of metrics collected on above servers
+  # Each metric consists in a name, a jmx path and either a pass or drop slice attributes
+  # This collect all heap memory usage metrics
+  [[plugins.jolokia.metrics]]
+    name = "heap_memory_usage"
+    jmx  = "/java.lang:type=Memory/HeapMemoryUsage"
+`
+}
+
+func (j *Jolokia) Description() string {
+	return "Read JMX metrics through Jolokia"
+}
+
+func (j *Jolokia) getAttr(requestUrl *url.URL) (map[string]interface{}, error) {
+	// Create + send request
+	req, err := http.NewRequest("GET", requestUrl.String(), nil)
+	if err != nil {
+		return nil, err
+	}
+	defer req.Body.Close()
+
+	resp, err := j.jClient.MakeRequest(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	// Process response
+	if resp.StatusCode != http.StatusOK {
+		err = fmt.Errorf("Response from url \"%s\" has status code %d (%s), expected %d (%s)",
+			requestUrl,
+			resp.StatusCode,
+			http.StatusText(resp.StatusCode),
+			http.StatusOK,
+			http.StatusText(http.StatusOK))
+		return nil, err
+	}
+
+	// read body
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	// Unmarshal json
+	var jsonOut map[string]interface{}
+	if err = json.Unmarshal([]byte(body), &jsonOut); err != nil {
+		return nil, errors.New("Error decoding JSON response")
+	}
+
+	return jsonOut, nil
+}
+
+func (j *Jolokia) Gather(acc plugins.Accumulator) error {
+	context := j.Context //"/jolokia/read"
+	servers := j.Servers
+	metrics := j.Metrics
+	tags := make(map[string]string)
+
+	for _, server := range servers {
+		tags["server"] = server.Name
+		tags["port"] = server.Port
+		tags["host"] = server.Host
+		fields := make(map[string]interface{})
+		for _, metric := range metrics {
+
+			measurement := metric.Name
+			jmxPath := metric.Jmx
+
+			// Prepare URL
+			requestUrl, err := url.Parse("http://" + server.Host + ":" +
+				server.Port + context + jmxPath)
+			if err != nil {
+				return err
+			}
+			if server.Username != "" || server.Password != "" {
+				requestUrl.User = url.UserPassword(server.Username, server.Password)
+			}
+
+			out, _ := j.getAttr(requestUrl)
+
+			if values, ok := out["value"]; ok {
+				switch t := values.(type) {
+				case map[string]interface{}:
+					for k, v := range t {
+						fields[measurement+"_"+k] = v
+					}
+				case interface{}:
+					fields[measurement] = t
+				}
+			} else {
+				fmt.Printf("Missing key 'value' in '%s' output response\n",
+					requestUrl.String())
+			}
+		}
+		acc.AddFields("jolokia", fields, tags)
+	}
+
+	return nil
+}
+
+func init() {
+	plugins.Add("jolokia", func() plugins.Plugin {
+		return &Jolokia{jClient: &JolokiaClientImpl{client: &http.Client{}}}
+	})
+}

plugins/jolokia/jolokia_test.go

@@ -0,0 +1,147 @@
+package jolokia
+
+import (
+	_ "fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	_ "github.com/stretchr/testify/require"
+)
+
+const validMultiValueJSON = `
+{
+  "request":{
+    "mbean":"java.lang:type=Memory",
+    "attribute":"HeapMemoryUsage",
+    "type":"read"
+  },
+  "value":{
+    "init":67108864,
+    "committed":456130560,
+    "max":477626368,
+    "used":203288528
+  },
+  "timestamp":1446129191,
+  "status":200
+}`
+
+const validSingleValueJSON = `
+{
+  "request":{
+    "path":"used",
+    "mbean":"java.lang:type=Memory",
+    "attribute":"HeapMemoryUsage",
+    "type":"read"
+  },
+  "value":209274376,
+  "timestamp":1446129256,
+  "status":200
+}`
+
+const invalidJSON = "I don't think this is JSON"
+
+const empty = ""
+
+var Servers = []Server{Server{Name: "as1", Host: "127.0.0.1", Port: "8080"}}
+var HeapMetric = Metric{Name: "heap_memory_usage", Jmx: "/java.lang:type=Memory/HeapMemoryUsage"}
+var UsedHeapMetric = Metric{Name: "heap_memory_usage", Jmx: "/java.lang:type=Memory/HeapMemoryUsage", Pass: []string{"used"}}
+
+type jolokiaClientStub struct {
+	responseBody string
+	statusCode   int
+}
+
+func (c jolokiaClientStub) MakeRequest(req *http.Request) (*http.Response, error) {
+	resp := http.Response{}
+	resp.StatusCode = c.statusCode
+	resp.Body = ioutil.NopCloser(strings.NewReader(c.responseBody))
+	return &resp, nil
+}
+
+// Generates a pointer to an HttpJson object that uses a mock HTTP client.
+// Parameters:
+//     response  : Body of the response that the mock HTTP client should return
+//     statusCode: HTTP status code the mock HTTP client should return
+//
+// Returns:
+//     *HttpJson: Pointer to an HttpJson object that uses the generated mock HTTP client
+func genJolokiaClientStub(response string, statusCode int, servers []Server, metrics []Metric) *Jolokia {
+	return &Jolokia{
+		jClient: jolokiaClientStub{responseBody: response, statusCode: statusCode},
+		Servers: servers,
+		Metrics: metrics,
+	}
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonMultiValue(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 200, Servers, []Metric{HeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+
+	assert.True(t, acc.CheckFieldsValue("heap_memory_usage", map[string]interface{}{"init": 67108864.0,
+		"committed": 456130560.0,
+		"max":       477626368.0,
+		"used":      203288528.0}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonMultiValueWithPass(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 200, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+
+	assert.True(t, acc.CheckFieldsValue("heap_memory_usage", map[string]interface{}{"used": 203288528.0}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonMultiValueTags(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 200, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+	assert.NoError(t, acc.ValidateTaggedFieldsValue("heap_memory_usage", map[string]interface{}{"used": 203288528.0}, map[string]string{"host": "127.0.0.1", "port": "8080", "server": "as1"}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonSingleValueTags(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validSingleValueJSON, 200, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Points))
+	assert.NoError(t, acc.ValidateTaggedFieldsValue("heap_memory_usage", map[string]interface{}{"value": 209274376.0}, map[string]string{"host": "127.0.0.1", "port": "8080", "server": "as1"}))
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonOn404(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validMultiValueJSON, 404, Servers, []Metric{UsedHeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 0, len(acc.Points))
+}

plugins/kafka_consumer/README.md

@@ -0,0 +1,24 @@
+# Kafka Consumer
+
+The [Kafka](http://kafka.apache.org/) consumer plugin polls a specified Kafka
+topic and adds messages to InfluxDB. The plugin assumes messages follow the
+line protocol. [Consumer Group](http://godoc.org/github.com/wvanbergen/kafka/consumergroup)
+is used to talk to the Kafka cluster so multiple instances of telegraf can read
+from the same topic in parallel.
+
+## Testing
+
+Running integration tests requires running Zookeeper & Kafka. The following
+commands assume you're on OS X & using [boot2docker](http://boot2docker.io/) or docker-machine through [Docker Toolbox](https://www.docker.com/docker-toolbox).
+
+To start Kafka & Zookeeper:
+
+```
+docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`boot2docker ip || docker-machine ip <your_machine_name>` --env ADVERTISED_PORT=9092 spotify/kafka
+```
+
+To run tests:
+
+```
+go test
+```

plugins/kafka_consumer/kafka_consumer.go

@@ -0,0 +1,166 @@
+package kafka_consumer
+
+import (
+	"log"
+	"strings"
+	"sync"
+
+	"github.com/influxdb/influxdb/models"
+	"github.com/influxdb/telegraf/plugins"
+
+	"github.com/Shopify/sarama"
+	"github.com/wvanbergen/kafka/consumergroup"
+)
+
+type Kafka struct {
+	ConsumerGroup  string
+	Topics         []string
+	ZookeeperPeers []string
+	Consumer       *consumergroup.ConsumerGroup
+	PointBuffer    int
+	Offset         string
+
+	sync.Mutex
+
+	// channel for all incoming kafka messages
+	in <-chan *sarama.ConsumerMessage
+	// channel for all kafka consumer errors
+	errs <-chan *sarama.ConsumerError
+	// channel for all incoming parsed kafka points
+	pointChan chan models.Point
+	done      chan struct{}
+
+	// doNotCommitMsgs tells the parser not to call CommitUpTo on the consumer
+	// this is mostly for test purposes, but there may be a use-case for it later.
+	doNotCommitMsgs bool
+}
+
+var sampleConfig = `
+  # topic(s) to consume
+  topics = ["telegraf"]
+  # an array of Zookeeper connection strings
+  zookeeper_peers = ["localhost:2181"]
+  # the name of the consumer group
+  consumer_group = "telegraf_metrics_consumers"
+  # Maximum number of points to buffer between collection intervals
+  point_buffer = 100000
+  # Offset (must be either "oldest" or "newest")
+  offset = "oldest"
+`
+
+func (k *Kafka) SampleConfig() string {
+	return sampleConfig
+}
+
+func (k *Kafka) Description() string {
+	return "Read line-protocol metrics from Kafka topic(s)"
+}
+
+func (k *Kafka) Start() error {
+	k.Lock()
+	defer k.Unlock()
+	var consumerErr error
+
+	config := consumergroup.NewConfig()
+	switch strings.ToLower(k.Offset) {
+	case "oldest", "":
+		config.Offsets.Initial = sarama.OffsetOldest
+	case "newest":
+		config.Offsets.Initial = sarama.OffsetNewest
+	default:
+		log.Printf("WARNING: Kafka consumer invalid offset '%s', using 'oldest'\n",
+			k.Offset)
+		config.Offsets.Initial = sarama.OffsetOldest
+	}
+
+	if k.Consumer == nil || k.Consumer.Closed() {
+		k.Consumer, consumerErr = consumergroup.JoinConsumerGroup(
+			k.ConsumerGroup,
+			k.Topics,
+			k.ZookeeperPeers,
+			config,
+		)
+		if consumerErr != nil {
+			return consumerErr
+		}
+
+		// Setup message and error channels
+		k.in = k.Consumer.Messages()
+		k.errs = k.Consumer.Errors()
+	}
+
+	k.done = make(chan struct{})
+	if k.PointBuffer == 0 {
+		k.PointBuffer = 100000
+	}
+	k.pointChan = make(chan models.Point, k.PointBuffer)
+
+	// Start the kafka message reader
+	go k.parser()
+	log.Printf("Started the kafka consumer service, peers: %v, topics: %v\n",
+		k.ZookeeperPeers, k.Topics)
+	return nil
+}
+
+// parser() reads all incoming messages from the consumer, and parses them into
+// influxdb metric points.
+func (k *Kafka) parser() {
+	for {
+		select {
+		case <-k.done:
+			return
+		case err := <-k.errs:
+			log.Printf("Kafka Consumer Error: %s\n", err.Error())
+		case msg := <-k.in:
+			points, err := models.ParsePoints(msg.Value)
+			if err != nil {
+				log.Printf("Could not parse kafka message: %s, error: %s",
+					string(msg.Value), err.Error())
+			}
+
+			for _, point := range points {
+				select {
+				case k.pointChan <- point:
+					continue
+				default:
+					log.Printf("Kafka Consumer buffer is full, dropping a point." +
+						" You may want to increase the point_buffer setting")
+				}
+			}
+
+			if !k.doNotCommitMsgs {
+				// TODO(cam) this locking can be removed if this PR gets merged:
+				// https://github.com/wvanbergen/kafka/pull/84
+				k.Lock()
+				k.Consumer.CommitUpto(msg)
+				k.Unlock()
+			}
+		}
+	}
+}
+
+func (k *Kafka) Stop() {
+	k.Lock()
+	defer k.Unlock()
+	close(k.done)
+	if err := k.Consumer.Close(); err != nil {
+		log.Printf("Error closing kafka consumer: %s\n", err.Error())
+	}
+}
+
+func (k *Kafka) Gather(acc plugins.Accumulator) error {
+	k.Lock()
+	defer k.Unlock()
+	npoints := len(k.pointChan)
+	for i := 0; i < npoints; i++ {
+		point := <-k.pointChan
+		acc.AddFields(point.Name(), point.Fields(), point.Tags(), point.Time())
+	}
+	return nil
+}
+
+func init() {
+	plugins.Add("kafka_consumer", func() plugins.Plugin {
+		return &Kafka{}
+	})
+}

plugins/kafka_consumer/kafka_consumer_integration_test.go

@@ -0,0 +1,91 @@
+package kafka_consumer
+
+import (
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/Shopify/sarama"
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestReadsMetricsFromKafka(t *testing.T) {
+	if testing.Short() {
+		t.Skip("Skipping integration test in short mode")
+	}
+
+	brokerPeers := []string{testutil.GetLocalHost() + ":9092"}
+	zkPeers := []string{testutil.GetLocalHost() + ":2181"}
+	testTopic := fmt.Sprintf("telegraf_test_topic_%d", time.Now().Unix())
+
+	// Send a Kafka message to the kafka host
+	msg := "cpu_load_short,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257"
+	producer, err := sarama.NewSyncProducer(brokerPeers, nil)
+	require.NoError(t, err)
+	_, _, err = producer.SendMessage(
+		&sarama.ProducerMessage{
+			Topic: testTopic,
+			Value: sarama.StringEncoder(msg),
+		})
+	require.NoError(t, err)
+	defer producer.Close()
+
+	// Start the Kafka Consumer
+	k := &Kafka{
+		ConsumerGroup:  "telegraf_test_consumers",
+		Topics:         []string{testTopic},
+		ZookeeperPeers: zkPeers,
+		PointBuffer:    100000,
+		Offset:         "oldest",
+	}
+	if err := k.Start(); err != nil {
+		t.Fatal(err.Error())
+	} else {
+		defer k.Stop()
+	}
+
+	waitForPoint(k, t)
+
+	// Verify that we can now gather the sent message
+	var acc testutil.Accumulator
+	// Sanity check
+	assert.Equal(t, 0, len(acc.Points), "There should not be any points")
+
+	// Gather points
+	err = k.Gather(&acc)
+	require.NoError(t, err)
+	if len(acc.Points) == 1 {
+		point := acc.Points[0]
+		assert.Equal(t, "cpu_load_short", point.Measurement)
+		assert.Equal(t, map[string]interface{}{"value": 23422.0}, point.Fields)
+		assert.Equal(t, map[string]string{
+			"host":      "server01",
+			"direction": "in",
+			"region":    "us-west",
+		}, point.Tags)
+		assert.Equal(t, time.Unix(0, 1422568543702900257).Unix(), point.Time.Unix())
+	} else {
+		t.Errorf("No points found in accumulator, expected 1")
+	}
+}
+
+// Waits for the metric that was sent to the kafka broker to arrive at the kafka
+// consumer
+func waitForPoint(k *Kafka, t *testing.T) {
+	// Give the kafka container up to 2 seconds to get the point to the consumer
+	ticker := time.NewTicker(5 * time.Millisecond)
+	counter := 0
+	for {
+		select {
+		case <-ticker.C:
+			counter++
+			if counter > 1000 {
+				t.Fatal("Waited for 5s, point never arrived to consumer")
+			} else if len(k.pointChan) == 1 {
+				return
+			}
+		}
+	}
+}

plugins/kafka_consumer/kafka_consumer_test.go

@@ -0,0 +1,98 @@
+package kafka_consumer
+
+import (
+	"testing"
+	"time"
+
+	"github.com/influxdb/influxdb/models"
+	"github.com/influxdb/telegraf/testutil"
+
+	"github.com/Shopify/sarama"
+	"github.com/stretchr/testify/assert"
+)
+
+const (
+	testMsg     = "cpu_load_short,host=server01 value=23422.0 1422568543702900257"
+	invalidMsg  = "cpu_load_short,host=server01 1422568543702900257"
+	pointBuffer = 5
+)
+
+func NewTestKafka() (*Kafka, chan *sarama.ConsumerMessage) {
+	in := make(chan *sarama.ConsumerMessage, pointBuffer)
+	k := Kafka{
+		ConsumerGroup:   "test",
+		Topics:          []string{"telegraf"},
+		ZookeeperPeers:  []string{"localhost:2181"},
+		PointBuffer:     pointBuffer,
+		Offset:          "oldest",
+		in:              in,
+		doNotCommitMsgs: true,
+		errs:            make(chan *sarama.ConsumerError, pointBuffer),
+		done:            make(chan struct{}),
+		pointChan:       make(chan models.Point, pointBuffer),
+	}
+	return &k, in
+}
+
+// Test that the parser parses kafka messages into points
+func TestRunParser(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	in <- saramaMsg(testMsg)
+	time.Sleep(time.Millisecond)
+
+	assert.Equal(t, len(k.pointChan), 1)
+}
+
+// Test that the parser ignores invalid messages
+func TestRunParserInvalidMsg(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	in <- saramaMsg(invalidMsg)
+	time.Sleep(time.Millisecond)
+
+	assert.Equal(t, len(k.pointChan), 0)
+}
+
+// Test that points are dropped when we hit the buffer limit
+func TestRunParserRespectsBuffer(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	for i := 0; i < pointBuffer+1; i++ {
+		in <- saramaMsg(testMsg)
+	}
+	time.Sleep(time.Millisecond)
+
+	assert.Equal(t, len(k.pointChan), 5)
+}
+
+// Test that the parser parses kafka messages into points
+func TestRunParserAndGather(t *testing.T) {
+	k, in := NewTestKafka()
+	defer close(k.done)
+
+	go k.parser()
+	in <- saramaMsg(testMsg)
+	time.Sleep(time.Millisecond)
+
+	acc := testutil.Accumulator{}
+	k.Gather(&acc)
+
+	assert.Equal(t, len(acc.Points), 1)
+	assert.True(t, acc.CheckValue("cpu_load_short", 23422.0))
+}
+
+func saramaMsg(val string) *sarama.ConsumerMessage {
+	return &sarama.ConsumerMessage{
+		Key:       nil,
+		Value:     []byte(val),
+		Offset:    0,
+		Partition: 0,
+	}
+}

plugins/leofs/leofs.go

@@ -0,0 +1,231 @@
+package leofs
+
+import (
+	"bufio"
+	"fmt"
+	"github.com/influxdb/telegraf/plugins"
+	"net/url"
+	"os/exec"
+	"strconv"
+	"strings"
+	"sync"
+)
+
+const oid = ".1.3.6.1.4.1.35450"
+
+// For Manager Master
+const defaultEndpoint = "127.0.0.1:4020"
+
+type ServerType int
+
+const (
+	ServerTypeManagerMaster ServerType = iota
+	ServerTypeManagerSlave
+	ServerTypeStorage
+	ServerTypeGateway
+)
+
+type LeoFS struct {
+	Servers []string
+}
+
+var KeyMapping = map[ServerType][]string{
+	ServerTypeManagerMaster: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+	ServerTypeManagerSlave: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+	ServerTypeStorage: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"num_of_writes",
+		"num_of_reads",
+		"num_of_deletes",
+		"num_of_writes_5min",
+		"num_of_reads_5min",
+		"num_of_deletes_5min",
+		"num_of_active_objects",
+		"total_objects",
+		"total_size_of_active_objects",
+		"total_size",
+		"num_of_replication_messages",
+		"num_of_sync-vnode_messages",
+		"num_of_rebalance_messages",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+	ServerTypeGateway: {
+		"num_of_processes",
+		"total_memory_usage",
+		"system_memory_usage",
+		"processes_memory_usage",
+		"ets_memory_usage",
+		"num_of_processes_5min",
+		"total_memory_usage_5min",
+		"system_memory_usage_5min",
+		"processes_memory_usage_5min",
+		"ets_memory_usage_5min",
+		"num_of_writes",
+		"num_of_reads",
+		"num_of_deletes",
+		"num_of_writes_5min",
+		"num_of_reads_5min",
+		"num_of_deletes_5min",
+		"count_of_cache-hit",
+		"count_of_cache-miss",
+		"total_of_files",
+		"total_cached_size",
+		"used_allocated_memory",
+		"allocated_memory",
+		"used_allocated_memory_5min",
+		"allocated_memory_5min",
+	},
+}
+
+var serverTypeMapping = map[string]ServerType{
+	"4020": ServerTypeManagerMaster,
+	"4021": ServerTypeManagerSlave,
+	"4010": ServerTypeStorage,
+	"4011": ServerTypeStorage,
+	"4012": ServerTypeStorage,
+	"4013": ServerTypeStorage,
+	"4000": ServerTypeGateway,
+	"4001": ServerTypeGateway,
+}
+
+var sampleConfig = `
+  # An array of URI to gather stats about LeoFS.
+  # Specify an ip or hostname with port. ie 127.0.0.1:4020
+  #
+  # If no servers are specified, then 127.0.0.1 is used as the host and 4020 as the port.
+  servers = ["127.0.0.1:4021"]
+`
+
+func (l *LeoFS) SampleConfig() string {
+	return sampleConfig
+}
+
+func (l *LeoFS) Description() string {
+	return "Read metrics from a LeoFS Server via SNMP"
+}
+
+func (l *LeoFS) Gather(acc plugins.Accumulator) error {
+	if len(l.Servers) == 0 {
+		l.gatherServer(defaultEndpoint, ServerTypeManagerMaster, acc)
+		return nil
+	}
+	var wg sync.WaitGroup
+	var outerr error
+	for _, endpoint := range l.Servers {
+		_, err := url.Parse(endpoint)
+		if err != nil {
+			return fmt.Errorf("Unable to parse the address:%s, err:%s", endpoint, err)
+		}
+		port, err := retrieveTokenAfterColon(endpoint)
+		if err != nil {
+			return err
+		}
+		st, ok := serverTypeMapping[port]
+		if !ok {
+			st = ServerTypeStorage
+		}
+		wg.Add(1)
+		go func(endpoint string, st ServerType) {
+			defer wg.Done()
+			outerr = l.gatherServer(endpoint, st, acc)
+		}(endpoint, st)
+	}
+	wg.Wait()
+	return outerr
+}
+
+func (l *LeoFS) gatherServer(endpoint string, serverType ServerType, acc plugins.Accumulator) error {
+	cmd := exec.Command("snmpwalk", "-v2c", "-cpublic", endpoint, oid)
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return err
+	}
+	cmd.Start()
+	defer cmd.Wait()
+	scanner := bufio.NewScanner(stdout)
+	if !scanner.Scan() {
+		return fmt.Errorf("Unable to retrieve the node name")
+	}
+	nodeName, err := retrieveTokenAfterColon(scanner.Text())
+	if err != nil {
+		return err
+	}
+	nodeNameTrimmed := strings.Trim(nodeName, "\"")
+	tags := map[string]string{
+		"node": nodeNameTrimmed,
+	}
+	i := 0
+
+	fields := make(map[string]interface{})
+	for scanner.Scan() {
+		key := KeyMapping[serverType][i]
+		val, err := retrieveTokenAfterColon(scanner.Text())
+		if err != nil {
+			return err
+		}
+		fVal, err := strconv.ParseFloat(val, 64)
+		if err != nil {
+			return fmt.Errorf("Unable to parse the value:%s, err:%s", val, err)
+		}
+		fields[key] = fVal
+		i++
+	}
+	acc.AddFields("leofs", fields, tags)
+	return nil
+}
+
+func retrieveTokenAfterColon(line string) (string, error) {
+	tokens := strings.Split(line, ":")
+	if len(tokens) != 2 {
+		return "", fmt.Errorf("':' not found in the line:%s", line)
+	}
+	return strings.TrimSpace(tokens[1]), nil
+}
+
+func init() {
+	plugins.Add("leofs", func() plugins.Plugin {
+		return &LeoFS{}
+	})
+}

plugins/leofs/leofs_test.go

@@ -0,0 +1,173 @@
+package leofs
+
+import (
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"io/ioutil"
+	"log"
+	"os"
+	"os/exec"
+	"testing"
+)
+
+var fakeSNMP4Manager = `
+package main
+
+import "fmt"
+
+const output = ` + "`" + `iso.3.6.1.4.1.35450.15.1.0 = STRING: "manager_888@127.0.0.1"
+iso.3.6.1.4.1.35450.15.2.0 = Gauge32: 186
+iso.3.6.1.4.1.35450.15.3.0 = Gauge32: 46235519
+iso.3.6.1.4.1.35450.15.4.0 = Gauge32: 32168525
+iso.3.6.1.4.1.35450.15.5.0 = Gauge32: 14066068
+iso.3.6.1.4.1.35450.15.6.0 = Gauge32: 5512968
+iso.3.6.1.4.1.35450.15.7.0 = Gauge32: 186
+iso.3.6.1.4.1.35450.15.8.0 = Gauge32: 46269006
+iso.3.6.1.4.1.35450.15.9.0 = Gauge32: 32202867
+iso.3.6.1.4.1.35450.15.10.0 = Gauge32: 14064995
+iso.3.6.1.4.1.35450.15.11.0 = Gauge32: 5492634
+iso.3.6.1.4.1.35450.15.12.0 = Gauge32: 60
+iso.3.6.1.4.1.35450.15.13.0 = Gauge32: 43515904
+iso.3.6.1.4.1.35450.15.14.0 = Gauge32: 60
+iso.3.6.1.4.1.35450.15.15.0 = Gauge32: 43533983` + "`" +
+	`
+func main() {
+	fmt.Println(output)
+}
+`
+
+var fakeSNMP4Storage = `
+package main
+
+import "fmt"
+
+const output = ` + "`" + `iso.3.6.1.4.1.35450.34.1.0 = STRING: "storage_0@127.0.0.1"
+iso.3.6.1.4.1.35450.34.2.0 = Gauge32: 512
+iso.3.6.1.4.1.35450.34.3.0 = Gauge32: 38126307
+iso.3.6.1.4.1.35450.34.4.0 = Gauge32: 22308716
+iso.3.6.1.4.1.35450.34.5.0 = Gauge32: 15816448
+iso.3.6.1.4.1.35450.34.6.0 = Gauge32: 5232008
+iso.3.6.1.4.1.35450.34.7.0 = Gauge32: 512
+iso.3.6.1.4.1.35450.34.8.0 = Gauge32: 38113176
+iso.3.6.1.4.1.35450.34.9.0 = Gauge32: 22313398
+iso.3.6.1.4.1.35450.34.10.0 = Gauge32: 15798779
+iso.3.6.1.4.1.35450.34.11.0 = Gauge32: 5237315
+iso.3.6.1.4.1.35450.34.12.0 = Gauge32: 191
+iso.3.6.1.4.1.35450.34.13.0 = Gauge32: 824
+iso.3.6.1.4.1.35450.34.14.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.15.0 = Gauge32: 50105
+iso.3.6.1.4.1.35450.34.16.0 = Gauge32: 196654
+iso.3.6.1.4.1.35450.34.17.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.18.0 = Gauge32: 2052
+iso.3.6.1.4.1.35450.34.19.0 = Gauge32: 50296
+iso.3.6.1.4.1.35450.34.20.0 = Gauge32: 35
+iso.3.6.1.4.1.35450.34.21.0 = Gauge32: 898
+iso.3.6.1.4.1.35450.34.22.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.23.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.24.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.31.0 = Gauge32: 51
+iso.3.6.1.4.1.35450.34.32.0 = Gauge32: 53219328
+iso.3.6.1.4.1.35450.34.33.0 = Gauge32: 51
+iso.3.6.1.4.1.35450.34.34.0 = Gauge32: 53351083` + "`" +
+	`
+func main() {
+	fmt.Println(output)
+}
+`
+
+var fakeSNMP4Gateway = `
+package main
+
+import "fmt"
+
+const output = ` + "`" + `iso.3.6.1.4.1.35450.34.1.0 = STRING: "gateway_0@127.0.0.1"
+iso.3.6.1.4.1.35450.34.2.0 = Gauge32: 465
+iso.3.6.1.4.1.35450.34.3.0 = Gauge32: 61676335
+iso.3.6.1.4.1.35450.34.4.0 = Gauge32: 46890415
+iso.3.6.1.4.1.35450.34.5.0 = Gauge32: 14785011
+iso.3.6.1.4.1.35450.34.6.0 = Gauge32: 5578855
+iso.3.6.1.4.1.35450.34.7.0 = Gauge32: 465
+iso.3.6.1.4.1.35450.34.8.0 = Gauge32: 61644426
+iso.3.6.1.4.1.35450.34.9.0 = Gauge32: 46880358
+iso.3.6.1.4.1.35450.34.10.0 = Gauge32: 14763002
+iso.3.6.1.4.1.35450.34.11.0 = Gauge32: 5582125
+iso.3.6.1.4.1.35450.34.12.0 = Gauge32: 191
+iso.3.6.1.4.1.35450.34.13.0 = Gauge32: 827
+iso.3.6.1.4.1.35450.34.14.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.15.0 = Gauge32: 50105
+iso.3.6.1.4.1.35450.34.16.0 = Gauge32: 196650
+iso.3.6.1.4.1.35450.34.17.0 = Gauge32: 0
+iso.3.6.1.4.1.35450.34.18.0 = Gauge32: 30256
+iso.3.6.1.4.1.35450.34.19.0 = Gauge32: 532158
+iso.3.6.1.4.1.35450.34.20.0 = Gauge32: 34
+iso.3.6.1.4.1.35450.34.21.0 = Gauge32: 1
+iso.3.6.1.4.1.35450.34.31.0 = Gauge32: 53
+iso.3.6.1.4.1.35450.34.32.0 = Gauge32: 55050240
+iso.3.6.1.4.1.35450.34.33.0 = Gauge32: 53
+iso.3.6.1.4.1.35450.34.34.0 = Gauge32: 55186538` + "`" +
+	`
+func main() {
+	fmt.Println(output)
+}
+`
+
+func makeFakeSNMPSrc(code string) string {
+	path := os.TempDir() + "/test.go"
+	err := ioutil.WriteFile(path, []byte(code), 0600)
+	if err != nil {
+		log.Fatalln(err)
+	}
+	return path
+}
+
+func buildFakeSNMPCmd(src string) {
+	err := exec.Command("go", "build", "-o", "snmpwalk", src).Run()
+	if err != nil {
+		log.Fatalln(err)
+	}
+}
+
+func testMain(t *testing.T, code string, endpoint string, serverType ServerType) {
+
+	// Build the fake snmpwalk for test
+	src := makeFakeSNMPSrc(code)
+	defer os.Remove(src)
+	buildFakeSNMPCmd(src)
+	defer os.Remove("./snmpwalk")
+	envPathOrigin := os.Getenv("PATH")
+	// Refer to the fake snmpwalk
+	os.Setenv("PATH", ".")
+	defer os.Setenv("PATH", envPathOrigin)
+
+	l := &LeoFS{
+		Servers: []string{endpoint},
+	}
+
+	var acc testutil.Accumulator
+
+	err := l.Gather(&acc)
+	require.NoError(t, err)
+
+	floatMetrics := KeyMapping[serverType]
+
+	for _, metric := range floatMetrics {
+		assert.True(t, acc.HasFloatValue(metric), metric)
+	}
+}
+
+func TestLeoFSManagerMasterMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Manager, "localhost:4020", ServerTypeManagerMaster)
+}
+
+func TestLeoFSManagerSlaveMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Manager, "localhost:4021", ServerTypeManagerSlave)
+}
+
+func TestLeoFSStorageMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Storage, "localhost:4010", ServerTypeStorage)
+}
+
+func TestLeoFSGatewayMetrics(t *testing.T) {
+	testMain(t, fakeSNMP4Gateway, "localhost:4000", ServerTypeGateway)
+}

plugins/lustre2/lustre2.go

@@ -0,0 +1,235 @@
+/*
+Lustre 2.x telegraf plugin
+
+Lustre (http://lustre.org/) is an open-source, parallel file system
+for HPC environments. It stores statistics about its activity in
+/proc
+
+*/
+package lustre2
+
+import (
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/influxdb/telegraf/internal"
+	"github.com/influxdb/telegraf/plugins"
+)
+
+// Lustre proc files can change between versions, so we want to future-proof
+// by letting people choose what to look at.
+type Lustre2 struct {
+	Ost_procfiles []string
+	Mds_procfiles []string
+}
+
+var sampleConfig = `
+  # An array of /proc globs to search for Lustre stats
+  # If not specified, the default will work on Lustre 2.5.x
+  #
+  # ost_procfiles = ["/proc/fs/lustre/obdfilter/*/stats", "/proc/fs/lustre/osd-ldiskfs/*/stats"]
+  # mds_procfiles = ["/proc/fs/lustre/mdt/*/md_stats"]
+`
+
+/* The wanted fields would be a []string if not for the
+lines that start with read_bytes/write_bytes and contain
+   both the byte count and the function call count
+*/
+type mapping struct {
+	inProc   string // What to look for at the start of a line in /proc/fs/lustre/*
+	field    uint32 // which field to extract from that line
+	reportAs string // What measurement name to use
+	tag      string // Additional tag to add for this metric
+}
+
+var wanted_ost_fields = []*mapping{
+	{
+		inProc:   "write_bytes",
+		field:    6,
+		reportAs: "write_bytes",
+	},
+	{ // line starts with 'write_bytes', but value write_calls is in second column
+		inProc:   "write_bytes",
+		field:    1,
+		reportAs: "write_calls",
+	},
+	{
+		inProc:   "read_bytes",
+		field:    6,
+		reportAs: "read_bytes",
+	},
+	{ // line starts with 'read_bytes', but value read_calls is in second column
+		inProc:   "read_bytes",
+		field:    1,
+		reportAs: "read_calls",
+	},
+	{
+		inProc: "cache_hit",
+	},
+	{
+		inProc: "cache_miss",
+	},
+	{
+		inProc: "cache_access",
+	},
+}
+
+var wanted_mds_fields = []*mapping{
+	{
+		inProc: "open",
+	},
+	{
+		inProc: "close",
+	},
+	{
+		inProc: "mknod",
+	},
+	{
+		inProc: "link",
+	},
+	{
+		inProc: "unlink",
+	},
+	{
+		inProc: "mkdir",
+	},
+	{
+		inProc: "rmdir",
+	},
+	{
+		inProc: "rename",
+	},
+	{
+		inProc: "getattr",
+	},
+	{
+		inProc: "setattr",
+	},
+	{
+		inProc: "getxattr",
+	},
+	{
+		inProc: "setxattr",
+	},
+	{
+		inProc: "statfs",
+	},
+	{
+		inProc: "sync",
+	},
+	{
+		inProc: "samedir_rename",
+	},
+	{
+		inProc: "crossdir_rename",
+	},
+}
+
+func (l *Lustre2) GetLustreProcStats(fileglob string, wanted_fields []*mapping, acc plugins.Accumulator) error {
+	files, err := filepath.Glob(fileglob)
+	if err != nil {
+		return err
+	}
+
+	for _, file := range files {
+		/* Turn /proc/fs/lustre/obdfilter/<ost_name>/stats and similar
+		 * into just the object store target name
+		 * Assumpion: the target name is always second to last,
+		 * which is true in Lustre 2.1->2.5
+		 */
+		path := strings.Split(file, "/")
+		name := path[len(path)-2]
+		tags := map[string]string{
+			"name": name,
+		}
+
+		lines, err := internal.ReadLines(file)
+		if err != nil {
+			return err
+		}
+
+		fields := make(map[string]interface{})
+		for _, line := range lines {
+			parts := strings.Fields(line)
+			for _, wanted := range wanted_fields {
+				var data uint64
+				if parts[0] == wanted.inProc {
+					wanted_field := wanted.field
+					// if not set, assume field[1]. Shouldn't be field[0], as
+					// that's a string
+					if wanted_field == 0 {
+						wanted_field = 1
+					}
+					data, err = strconv.ParseUint((parts[wanted_field]), 10, 64)
+					if err != nil {
+						return err
+					}
+					report_name := wanted.inProc
+					if wanted.reportAs != "" {
+						report_name = wanted.reportAs
+					}
+					fields[report_name] = data
+				}
+			}
+		}
+		acc.AddFields("lustre2", fields, tags)
+	}
+	return nil
+}
+
+// SampleConfig returns sample configuration message
+func (l *Lustre2) SampleConfig() string {
+	return sampleConfig
+}
+
+// Description returns description of Lustre2 plugin
+func (l *Lustre2) Description() string {
+	return "Read metrics from local Lustre service on OST, MDS"
+}
+
+// Gather reads stats from all lustre targets
+func (l *Lustre2) Gather(acc plugins.Accumulator) error {
+
+	if len(l.Ost_procfiles) == 0 {
+		// read/write bytes are in obdfilter/<ost_name>/stats
+		err := l.GetLustreProcStats("/proc/fs/lustre/obdfilter/*/stats", wanted_ost_fields, acc)
+		if err != nil {
+			return err
+		}
+		// cache counters are in osd-ldiskfs/<ost_name>/stats
+		err = l.GetLustreProcStats("/proc/fs/lustre/osd-ldiskfs/*/stats", wanted_ost_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+
+	if len(l.Mds_procfiles) == 0 {
+		// Metadata server stats
+		err := l.GetLustreProcStats("/proc/fs/lustre/mdt/*/md_stats", wanted_mds_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+
+	for _, procfile := range l.Ost_procfiles {
+		err := l.GetLustreProcStats(procfile, wanted_ost_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+	for _, procfile := range l.Mds_procfiles {
+		err := l.GetLustreProcStats(procfile, wanted_mds_fields, acc)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func init() {
+	plugins.Add("lustre2", func() plugins.Plugin {
+		return &Lustre2{}
+	})
+}

plugins/lustre2/lustre2_test.go

@@ -0,0 +1,144 @@
+package lustre2
+
+import (
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/influxdb/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Set config file variables to point to fake directory structure instead of /proc?
+
+const obdfilterProcContents = `snapshot_time             1438693064.430544 secs.usecs
+read_bytes                203238095 samples [bytes] 4096 1048576 78026117632000
+write_bytes               71893382 samples [bytes] 1 1048576 15201500833981
+get_info                  1182008495 samples [reqs]
+set_info_async            2 samples [reqs]
+connect                   1117 samples [reqs]
+reconnect                 1160 samples [reqs]
+disconnect                1084 samples [reqs]
+statfs                    3575885 samples [reqs]
+create                    698 samples [reqs]
+destroy                   3190060 samples [reqs]
+setattr                   605647 samples [reqs]
+punch                     805187 samples [reqs]
+sync                      6608753 samples [reqs]
+preprw                    275131477 samples [reqs]
+commitrw                  275131477 samples [reqs]
+quotactl                  229231 samples [reqs]
+ping                      78020757 samples [reqs]
+`
+
+const osdldiskfsProcContents = `snapshot_time             1438693135.640551 secs.usecs
+get_page                  275132812 samples [usec] 0 3147 1320420955 22041662259
+cache_access              19047063027 samples [pages] 1 1 19047063027
+cache_hit                 7393729777 samples [pages] 1 1 7393729777
+cache_miss                11653333250 samples [pages] 1 1 11653333250
+`
+
+const mdtProcContents = `snapshot_time             1438693238.20113 secs.usecs
+open                      1024577037 samples [reqs]
+close                     873243496 samples [reqs]
+mknod                     349042 samples [reqs]
+link                      445 samples [reqs]
+unlink                    3549417 samples [reqs]
+mkdir                     705499 samples [reqs]
+rmdir                     227434 samples [reqs]
+rename                    629196 samples [reqs]
+getattr                   1503663097 samples [reqs]
+setattr                   1898364 samples [reqs]
+getxattr                  6145349681 samples [reqs]
+setxattr                  83969 samples [reqs]
+statfs                    2916320 samples [reqs]
+sync                      434081 samples [reqs]
+samedir_rename            259625 samples [reqs]
+crossdir_rename           369571 samples [reqs]
+`
+
+type metrics struct {
+	name  string
+	value uint64
+}
+
+func TestLustre2GeneratesMetrics(t *testing.T) {
+
+	tempdir := os.TempDir() + "/telegraf/proc/fs/lustre/"
+	ost_name := "OST0001"
+
+	mdtdir := tempdir + "/mdt/"
+	err := os.MkdirAll(mdtdir+"/"+ost_name, 0755)
+	require.NoError(t, err)
+
+	osddir := tempdir + "/osd-ldiskfs/"
+	err = os.MkdirAll(osddir+"/"+ost_name, 0755)
+	require.NoError(t, err)
+
+	obddir := tempdir + "/obdfilter/"
+	err = os.MkdirAll(obddir+"/"+ost_name, 0755)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(mdtdir+"/"+ost_name+"/md_stats", []byte(mdtProcContents), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(osddir+"/"+ost_name+"/stats", []byte(osdldiskfsProcContents), 0644)
+	require.NoError(t, err)
+
+	err = ioutil.WriteFile(obddir+"/"+ost_name+"/stats", []byte(obdfilterProcContents), 0644)
+	require.NoError(t, err)
+
+	m := &Lustre2{
+		Ost_procfiles: []string{obddir + "/*/stats", osddir + "/*/stats"},
+		Mds_procfiles: []string{mdtdir + "/*/md_stats"},
+	}
+
+	var acc testutil.Accumulator
+
+	err = m.Gather(&acc)
+	require.NoError(t, err)
+
+	tags := map[string]string{
+		"name": ost_name,
+	}
+
+	intMetrics := []*metrics{
+		{
+			name:  "write_bytes",
+			value: 15201500833981,
+		},
+		{
+			name:  "read_bytes",
+			value: 78026117632000,
+		},
+		{
+			name:  "write_calls",
+			value: 71893382,
+		},
+		{
+			name:  "read_calls",
+			value: 203238095,
+		},
+		{
+			name:  "cache_hit",
+			value: 7393729777,
+		},
+		{
+			name:  "cache_access",
+			value: 19047063027,
+		},
+		{
+			name:  "cache_miss",
+			value: 11653333250,
+		},
+	}
+
+	for _, metric := range intMetrics {
+		assert.True(t, acc.HasUIntValue(metric.name), metric.name)
+		assert.True(t, acc.CheckTaggedValue(metric.name, metric.value, tags))
+	}
+
+	err = os.RemoveAll(os.TempDir() + "/telegraf")
+	require.NoError(t, err)
+}