Telegraf v1.0 beta 3

this may address, or at least log issue #1462

.gitattributes

@@ -0,0 +1,4 @@
+CHANGELOG.md merge=union
+README.md merge=union
+plugins/inputs/all/all.go merge=union
+plugins/outputs/all/all.go merge=union

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,44 @@
+## Directions
+
+GitHub Issues are reserved for actionable bug reports and feature requests.
+General questions should be sent to the [InfluxDB mailing list](https://groups.google.com/forum/#!forum/influxdb).
+
+Before opening an issue, search for similar bug reports or feature requests on GitHub Issues.
+If no similar issue can be found, fill out either the "Bug Report" or the "Feature Request" section below.
+Erase the other section and everything on and above this line.
+
+*Please note, the quickest way to fix a bug is to open a Pull Request.*
+
+## Bug report
+
+### Relevant telegraf.conf:
+
+### System info:
+
+[Include Telegraf version, operating system name, and other relevant details]
+
+### Steps to reproduce:
+
+1. ...
+2. ...
+
+### Expected behavior:
+
+### Actual behavior:
+
+### Additional info:
+
+[Include gist of relevant config, logs, etc.]
+
+
+## Feature Request
+
+Opening a feature request kicks off a discussion.
+
+### Proposal:
+
+### Current behavior:
+
+### Desired behavior:
+
+### Use case: [Why is this important (helps with prioritizing requests)]

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,5 @@
+### Required for all PRs:
+
+- [ ] CHANGELOG.md updated
+- [ ] Sign [CLA](https://influxdata.com/community/cla/) (if not already signed)
+- [ ] README.md updated (if adding a new plugin)

.gitignore

@@ -1,3 +1,4 @@
+build
 tivan
 .vagrant
 /telegraf

CHANGELOG.md

@@ -1,4 +1,352 @@
-## v0.11.1 [unreleased]
+## v1.0 [unreleased]
+
+## v1.0 beta 3 [2016-07-18]
+
+### Release Notes
+
+**Breaking Change**: Aerospike main server node measurements have been renamed
+aerospike_node. Aerospike namespace measurements have been renamed to
+aerospike_namespace. They will also now be tagged with the node_name
+that they correspond to. This has been done to differentiate measurements
+that pertain to node vs. namespace statistics.
+
+**Breaking Change**: users of github_webhooks must change to the new
+`[[inputs.webhooks]]` plugin.
+
+This means that the default github_webhooks config:
+
+```
+# A Github Webhook Event collector
+[[inputs.github_webhooks]]
+  ## Address and port to host Webhook listener on
+  service_address = ":1618"
+```
+
+should now look like:
+
+```
+# A Webhooks Event collector
+[[inputs.webhooks]]
+  ## Address and port to host Webhook listener on
+  service_address = ":1618"
+
+  [inputs.webhooks.github]
+    path = "/"
+```
+
+### Features
+
+- [#1289](https://github.com/influxdata/telegraf/pull/1289): webhooks input plugin. Thanks @francois2metz and @cduez!
+- [#1247](https://github.com/influxdata/telegraf/pull/1247): rollbar webhook plugin.
+- [#1408](https://github.com/influxdata/telegraf/pull/1408): mandrill webhook plugin.
+- [#1402](https://github.com/influxdata/telegraf/pull/1402): docker-machine/boot2docker no longer required for unit tests.
+- [#1350](https://github.com/influxdata/telegraf/pull/1350): cgroup input plugin.
+- [#1369](https://github.com/influxdata/telegraf/pull/1369): Add input plugin for consuming metrics from NSQD.
+- [#1369](https://github.com/influxdata/telegraf/pull/1480): add ability to read redis from a socket.
+- [#1387](https://github.com/influxdata/telegraf/pull/1387): **Breaking Change** - Redis `role` tag renamed to `replication_role` to avoid global_tags override
+- [#1437](https://github.com/influxdata/telegraf/pull/1437): Fetching Galera status metrics in MySQL
+- [#1500](https://github.com/influxdata/telegraf/pull/1500): Aerospike plugin refactored to use official client lib.
+- [#1434](https://github.com/influxdata/telegraf/pull/1434): Add measurement name arg to logparser plugin.
+- [#1479](https://github.com/influxdata/telegraf/pull/1479): logparser: change resp_code from a field to a tag.
+
+### Bugfixes
+
+- [#1472](https://github.com/influxdata/telegraf/pull/1472): diskio input plugin: set 'skip_serial_number = true' by default to avoid high cardinality.
+- [#1426](https://github.com/influxdata/telegraf/pull/1426): nil metrics panic fix.
+- [#1384](https://github.com/influxdata/telegraf/pull/1384): Fix datarace in apache input plugin.
+- [#1399](https://github.com/influxdata/telegraf/issues/1399): Add `read_repairs` statistics to riak plugin.
+- [#1405](https://github.com/influxdata/telegraf/issues/1405): Fix memory/connection leak in prometheus input plugin.
+- [#1378](https://github.com/influxdata/telegraf/issues/1378): Trim BOM from config file for Windows support.
+- [#1339](https://github.com/influxdata/telegraf/issues/1339): Prometheus client output panic on service reload.
+- [#1461](https://github.com/influxdata/telegraf/pull/1461): Prometheus parser, protobuf format header fix.
+- [#1334](https://github.com/influxdata/telegraf/issues/1334): Prometheus output, metric refresh and caching fixes.
+- [#1432](https://github.com/influxdata/telegraf/issues/1432): Panic fix for multiple graphite outputs under very high load.
+- [#1412](https://github.com/influxdata/telegraf/pull/1412): Instrumental output has better reconnect behavior
+- [#1460](https://github.com/influxdata/telegraf/issues/1460): Remove PID from procstat plugin to fix cardinality issues.
+- [#1427](https://github.com/influxdata/telegraf/issues/1427): Cassandra input: version 2.x "column family" fix.
+- [#1463](https://github.com/influxdata/telegraf/issues/1463): Shared WaitGroup in Exec plugin
+- [#1436](https://github.com/influxdata/telegraf/issues/1436): logparser: honor modifiers in "pattern" config.
+- [#1418](https://github.com/influxdata/telegraf/issues/1418): logparser: error and exit on file permissions/missing errors.
+
+## v1.0 beta 2 [2016-06-21]
+
+### Features
+
+- [#1340](https://github.com/influxdata/telegraf/issues/1340): statsd: do not log every dropped metric.
+- [#1368](https://github.com/influxdata/telegraf/pull/1368): Add precision rounding to all metrics on collection.
+- [#1390](https://github.com/influxdata/telegraf/pull/1390): Add support for Tengine
+- [#1320](https://github.com/influxdata/telegraf/pull/1320): Logparser input plugin for parsing grok-style log patterns.
+- [#1397](https://github.com/influxdata/telegraf/issues/1397): ElasticSearch: now supports connecting to ElasticSearch via SSL
+
+### Bugfixes
+
+- [#1330](https://github.com/influxdata/telegraf/issues/1330): Fix exec plugin panic when using single binary.
+- [#1336](https://github.com/influxdata/telegraf/issues/1336): Fixed incorrect prometheus metrics source selection.
+- [#1112](https://github.com/influxdata/telegraf/issues/1112): Set default Zookeeper chroot to empty string.
+- [#1335](https://github.com/influxdata/telegraf/issues/1335): Fix overall ping timeout to be calculated based on per-ping timeout.
+- [#1374](https://github.com/influxdata/telegraf/pull/1374): Change "default" retention policy to "".
+- [#1377](https://github.com/influxdata/telegraf/issues/1377): Graphite output mangling '%' character.
+- [#1396](https://github.com/influxdata/telegraf/pull/1396): Prometheus input plugin now supports x509 certs authentication
+
+## v1.0 beta 1 [2016-06-07]
+
+### Release Notes
+
+- `flush_jitter` behavior has been changed. The random jitter will now be
+evaluated at every flush interval, rather than once at startup. This makes it
+consistent with the behavior of `collection_jitter`.
+
+- All AWS plugins now utilize a standard mechanism for evaluating credentials.
+This allows all AWS plugins to support environment variables, shared credential
+files & profiles, and role assumptions. See the specific plugin README for
+details.
+
+- The AWS CloudWatch input plugin can now declare a wildcard value for a metric
+dimension. This causes the plugin to read all metrics that contain the specified
+dimension key regardless of value. This is used to export collections of metrics
+without having to know the dimension values ahead of time.
+
+- The AWS CloudWatch input plugin can now be configured with the `cache_ttl`
+attribute. This configures the TTL of the internal metric cache. This is useful
+in conjunction with wildcard dimension values as it will control the amount of
+time before a new metric is included by the plugin.
+
+### Features
+
+- [#1262](https://github.com/influxdata/telegraf/pull/1261): Add graylog input pluging.
+- [#1294](https://github.com/influxdata/telegraf/pull/1294): consul input plugin. Thanks @harnash
+- [#1164](https://github.com/influxdata/telegraf/pull/1164): conntrack input plugin. Thanks @robinpercy!
+- [#1165](https://github.com/influxdata/telegraf/pull/1165): vmstat input plugin. Thanks @jshim-xm!
+- [#1208](https://github.com/influxdata/telegraf/pull/1208): Standardized AWS credentials evaluation & wildcard CloudWatch dimensions. Thanks @johnrengelman!
+- [#1264](https://github.com/influxdata/telegraf/pull/1264): Add SSL config options to http_response plugin.
+- [#1272](https://github.com/influxdata/telegraf/pull/1272): graphite parser: add ability to specify multiple tag keys, for consistency with influxdb parser.
+- [#1265](https://github.com/influxdata/telegraf/pull/1265): Make dns lookups for chrony configurable. Thanks @zbindenren!
+- [#1275](https://github.com/influxdata/telegraf/pull/1275): Allow wildcard filtering of varnish stats.
+- [#1142](https://github.com/influxdata/telegraf/pull/1142): Support for glob patterns in exec plugin commands configuration.
+- [#1278](https://github.com/influxdata/telegraf/pull/1278): RabbitMQ input: made url parameter optional by using DefaultURL (http://localhost:15672) if not specified
+- [#1197](https://github.com/influxdata/telegraf/pull/1197): Limit AWS GetMetricStatistics requests to 10 per second.
+- [#1278](https://github.com/influxdata/telegraf/pull/1278) & [#1288](https://github.com/influxdata/telegraf/pull/1288) & [#1295](https://github.com/influxdata/telegraf/pull/1295): RabbitMQ/Apache/InfluxDB inputs: made url(s) parameter optional by using reasonable input defaults if not specified
+- [#1296](https://github.com/influxdata/telegraf/issues/1296): Refactor of flush_jitter argument.
+- [#1213](https://github.com/influxdata/telegraf/issues/1213): Add inactive & active memory to mem plugin.
+
+### Bugfixes
+
+- [#1252](https://github.com/influxdata/telegraf/pull/1252) & [#1279](https://github.com/influxdata/telegraf/pull/1279): Fix systemd service. Thanks @zbindenren & @PierreF!
+- [#1221](https://github.com/influxdata/telegraf/pull/1221): Fix influxdb n_shards counter.
+- [#1258](https://github.com/influxdata/telegraf/pull/1258): Fix potential kernel plugin integer parse error.
+- [#1268](https://github.com/influxdata/telegraf/pull/1268): Fix potential influxdb input type assertion panic.
+- [#1283](https://github.com/influxdata/telegraf/pull/1283): Still send processes metrics if a process exited during metric collection.
+- [#1297](https://github.com/influxdata/telegraf/issues/1297): disk plugin panic when usage grab fails.
+- [#1316](https://github.com/influxdata/telegraf/pull/1316): Removed leaked "database" tag on redis metrics. Thanks @PierreF!
+- [#1323](https://github.com/influxdata/telegraf/issues/1323): Processes plugin: fix potential error with /proc/net/stat directory.
+- [#1322](https://github.com/influxdata/telegraf/issues/1322): Fix rare RHEL 5.2 panic in gopsutil diskio gathering function.
+
+## v0.13.1 [2016-05-24]
+
+### Release Notes
+
+- net_response and http_response plugins timeouts will now accept duration
+strings, ie, "2s" or "500ms".
+- Input plugin Gathers will no longer be logged by default, but a Gather for
+_each_ plugin will be logged in Debug mode.
+- Debug mode will no longer print every point added to the accumulator. This
+functionality can be duplicated using the `file` output plugin and printing
+to "stdout".
+
+### Features
+
+- [#1173](https://github.com/influxdata/telegraf/pull/1173): varnish input plugin. Thanks @sfox-xmatters!
+- [#1138](https://github.com/influxdata/telegraf/pull/1138): nstat input plugin. Thanks @Maksadbek!
+- [#1139](https://github.com/influxdata/telegraf/pull/1139): instrumental output plugin. Thanks @jasonroelofs!
+- [#1172](https://github.com/influxdata/telegraf/pull/1172): Ceph storage stats. Thanks @robinpercy!
+- [#1233](https://github.com/influxdata/telegraf/pull/1233): Updated golint gopsutil dependency.
+- [#1238](https://github.com/influxdata/telegraf/pull/1238): chrony input plugin. Thanks @zbindenren!
+- [#479](https://github.com/influxdata/telegraf/issues/479): per-plugin execution time added to debug output.
+- [#1249](https://github.com/influxdata/telegraf/issues/1249): influxdb output: added write_consistency argument.
+
+### Bugfixes
+
+- [#1195](https://github.com/influxdata/telegraf/pull/1195): Docker panic on timeout. Thanks @zstyblik!
+- [#1211](https://github.com/influxdata/telegraf/pull/1211): mongodb input. Fix possible panic. Thanks @kols!
+- [#1215](https://github.com/influxdata/telegraf/pull/1215): Fix for possible gopsutil-dependent plugin hangs.
+- [#1228](https://github.com/influxdata/telegraf/pull/1228): Fix service plugin host tag overwrite.
+- [#1198](https://github.com/influxdata/telegraf/pull/1198): http_response: override request Host header properly
+- [#1230](https://github.com/influxdata/telegraf/issues/1230): Fix Telegraf process hangup due to a single plugin hanging.
+- [#1214](https://github.com/influxdata/telegraf/issues/1214): Use TCP timeout argument in net_response plugin.
+- [#1243](https://github.com/influxdata/telegraf/pull/1243): Logfile not created on systemd.
+
+## v0.13 [2016-05-11]
+
+### Release Notes
+
+- **Breaking change** in jolokia plugin. See
+https://github.com/influxdata/telegraf/blob/master/plugins/inputs/jolokia/README.md
+for updated configuration. The plugin will now support proxy mode and will make
+POST requests.
+
+- New [agent] configuration option: `metric_batch_size`. This option tells
+telegraf the maximum batch size to allow to accumulate before sending a flush
+to the configured outputs. `metric_buffer_limit` now refers to the absolute
+maximum number of metrics that will accumulate before metrics are dropped.
+
+- There is no longer an option to
+`flush_buffer_when_full`, this is now the default and only behavior of telegraf.
+
+- **Breaking Change**: docker plugin tags. The cont_id tag no longer exists, it
+will now be a field, and be called container_id. Additionally, cont_image and
+cont_name are being renamed to container_image and container_name.
+
+- **Breaking Change**: docker plugin measurements. The `docker_cpu`, `docker_mem`,
+`docker_blkio` and `docker_net` measurements are being renamed to
+`docker_container_cpu`, `docker_container_mem`, `docker_container_blkio` and
+`docker_container_net`. Why? Because these metrics are
+specifically tracking per-container stats. The problem with per-container stats,
+in some use-cases, is that if containers are short-lived AND names are not
+kept consistent, then the series cardinality will balloon very quickly.
+So adding "container" to each metric will:
+(1) make it more clear that these metrics are per-container, and
+(2) allow users to easily drop per-container metrics if cardinality is an
+issue (`namedrop = ["docker_container_*"]`)
+
+- `tagexclude` and `taginclude` are now available, which can be used to remove
+tags from measurements on inputs and outputs. See
+[the configuration doc](https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md)
+for more details.
+
+- **Measurement filtering:** All measurement filters now match based on glob
+only. Previously there was an undocumented behavior where filters would match
+based on _prefix_ in addition to globs. This means that a filter like
+`fielddrop = ["time_"]` will need to be changed to `fielddrop = ["time_*"]`
+
+- **datadog**: measurement and field names will no longer have `_` replaced by `.`
+
+- The following plugins have changed their tags to _not_ overwrite the host tag:
+  - cassandra: `host -> cassandra_host`
+  - disque: `host -> disque_host`
+  - rethinkdb: `host -> rethinkdb_host`
+
+- **Breaking Change**: The `win_perf_counters` input has been changed to
+sanitize field names, replacing `/Sec` and `/sec` with `_persec`, as well as
+spaces with underscores. This is needed because Graphite doesn't like slashes
+and spaces, and was failing to accept metrics that had them.
+The `/[sS]ec` -> `_persec` is just to make things clearer and uniform.
+
+- **Breaking Change**: snmp plugin. The `host` tag of the snmp plugin has been
+changed to the `snmp_host` tag.
+
+- The `disk` input plugin can now be configured with the `HOST_MOUNT_PREFIX` environment variable.
+This value is prepended to any mountpaths discovered before retrieving stats.
+It is not included on the report path. This is necessary for reporting host disk stats when running from within a container.
+
+### Features
+
+- [#1031](https://github.com/influxdata/telegraf/pull/1031): Jolokia plugin proxy mode. Thanks @saiello!
+- [#1017](https://github.com/influxdata/telegraf/pull/1017): taginclude and tagexclude arguments.
+- [#1015](https://github.com/influxdata/telegraf/pull/1015): Docker plugin schema refactor.
+- [#889](https://github.com/influxdata/telegraf/pull/889): Improved MySQL plugin. Thanks @maksadbek!
+- [#1060](https://github.com/influxdata/telegraf/pull/1060): TTL metrics added to MongoDB input plugin
+- [#1056](https://github.com/influxdata/telegraf/pull/1056): Don't allow inputs to overwrite host tags.
+- [#1035](https://github.com/influxdata/telegraf/issues/1035): Add `user`, `exe`, `pidfile` tags to procstat plugin.
+- [#1041](https://github.com/influxdata/telegraf/issues/1041): Add `n_cpus` field to the system plugin.
+- [#1072](https://github.com/influxdata/telegraf/pull/1072): New Input Plugin: filestat.
+- [#1066](https://github.com/influxdata/telegraf/pull/1066): Replication lag metrics for MongoDB input plugin
+- [#1086](https://github.com/influxdata/telegraf/pull/1086): Ability to specify AWS keys in config file. Thanks @johnrengleman!
+- [#1096](https://github.com/influxdata/telegraf/pull/1096): Performance refactor of running output buffers.
+- [#967](https://github.com/influxdata/telegraf/issues/967): Buffer logging improvements.
+- [#1107](https://github.com/influxdata/telegraf/issues/1107): Support lustre2 job stats. Thanks @hanleyja!
+- [#1122](https://github.com/influxdata/telegraf/pull/1122): Support setting config path through env variable and default paths.
+- [#1128](https://github.com/influxdata/telegraf/pull/1128): MongoDB jumbo chunks metric for MongoDB input plugin
+- [#1146](https://github.com/influxdata/telegraf/pull/1146): HAProxy socket support. Thanks weshmashian!
+
+### Bugfixes
+
+- [#1050](https://github.com/influxdata/telegraf/issues/1050): jolokia plugin - do not overwrite host tag. Thanks @saiello!
+- [#921](https://github.com/influxdata/telegraf/pull/921): mqtt_consumer stops gathering metrics. Thanks @chaton78!
+- [#1013](https://github.com/influxdata/telegraf/pull/1013): Close dead riemann output connections. Thanks @echupriyanov!
+- [#1012](https://github.com/influxdata/telegraf/pull/1012): Set default tags in test accumulator.
+- [#1024](https://github.com/influxdata/telegraf/issues/1024): Don't replace `.` with `_` in datadog output.
+- [#1058](https://github.com/influxdata/telegraf/issues/1058): Fix possible leaky TCP connections in influxdb output.
+- [#1044](https://github.com/influxdata/telegraf/pull/1044): Fix SNMP OID possible collisions. Thanks @relip
+- [#1022](https://github.com/influxdata/telegraf/issues/1022): Dont error deb/rpm install on systemd errors.
+- [#1078](https://github.com/influxdata/telegraf/issues/1078): Use default AWS credential chain.
+- [#1070](https://github.com/influxdata/telegraf/issues/1070): SQL Server input. Fix datatype conversion.
+- [#1089](https://github.com/influxdata/telegraf/issues/1089): Fix leaky TCP connections in phpfpm plugin.
+- [#914](https://github.com/influxdata/telegraf/issues/914): Telegraf can drop metrics on full buffers.
+- [#1098](https://github.com/influxdata/telegraf/issues/1098): Sanitize invalid OpenTSDB characters.
+- [#1110](https://github.com/influxdata/telegraf/pull/1110): Sanitize * to - in graphite serializer. Thanks @goodeggs!
+- [#1118](https://github.com/influxdata/telegraf/pull/1118): Sanitize Counter names for `win_perf_counters` input.
+- [#1125](https://github.com/influxdata/telegraf/pull/1125): Wrap all exec command runners with a timeout, so hung os processes don't halt Telegraf.
+- [#1113](https://github.com/influxdata/telegraf/pull/1113): Set MaxRetry and RequiredAcks defaults in Kafka output.
+- [#1090](https://github.com/influxdata/telegraf/issues/1090): [agent] and [global_tags] config sometimes not getting applied.
+- [#1133](https://github.com/influxdata/telegraf/issues/1133): Use a timeout for docker list & stat cmds.
+- [#1052](https://github.com/influxdata/telegraf/issues/1052): Docker panic fix when decode fails.
+- [#1136](https://github.com/influxdata/telegraf/pull/1136): "DELAYED" Inserts were deprecated in MySQL 5.6.6. Thanks @PierreF
+
+## v0.12.1 [2016-04-14]
+
+### Release Notes
+- Breaking change in the dovecot input plugin. See Features section below.
+- Graphite output templates are now supported. See
+https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#graphite
+- Possible breaking change for the librato and graphite outputs. Telegraf will
+no longer insert field names when the field is simply named `value`. This is
+because the `value` field is redundant in the graphite/librato context.
+
+### Features
+- [#1009](https://github.com/influxdata/telegraf/pull/1009): Cassandra input plugin. Thanks @subhachandrachandra!
+- [#976](https://github.com/influxdata/telegraf/pull/976): Reduce allocations in the UDP and statsd inputs.
+- [#979](https://github.com/influxdata/telegraf/pull/979): Reduce allocations in the TCP listener.
+- [#992](https://github.com/influxdata/telegraf/pull/992): Refactor allocations in TCP/UDP listeners.
+- [#935](https://github.com/influxdata/telegraf/pull/935): AWS Cloudwatch input plugin. Thanks @joshhardy & @ljosa!
+- [#943](https://github.com/influxdata/telegraf/pull/943): http_response input plugin. Thanks @Lswith!
+- [#939](https://github.com/influxdata/telegraf/pull/939): sysstat input plugin. Thanks @zbindenren!
+- [#998](https://github.com/influxdata/telegraf/pull/998): **breaking change** enabled global, user and ip queries in dovecot plugin. Thanks @mikif70!
+- [#1001](https://github.com/influxdata/telegraf/pull/1001): Graphite serializer templates.
+- [#1008](https://github.com/influxdata/telegraf/pull/1008): Adding memstats metrics to the influxdb plugin.
+
+### Bugfixes
+- [#968](https://github.com/influxdata/telegraf/issues/968): Processes plugin gets unknown state when spaces are in (command name)
+- [#969](https://github.com/influxdata/telegraf/pull/969): ipmi_sensors: allow : in password. Thanks @awaw!
+- [#972](https://github.com/influxdata/telegraf/pull/972): dovecot: remove extra newline in dovecot command. Thanks @mrannanj!
+- [#645](https://github.com/influxdata/telegraf/issues/645): docker plugin i/o error on closed pipe. Thanks @tripledes!
+
+## v0.12.0 [2016-04-05]
+
+### Features
+- [#951](https://github.com/influxdata/telegraf/pull/951): Parse environment variables in the config file.
+- [#948](https://github.com/influxdata/telegraf/pull/948): Cleanup config file and make default package version include all plugins (but commented).
+- [#927](https://github.com/influxdata/telegraf/pull/927): Adds parsing of tags to the statsd input when using DataDog's dogstatsd extension
+- [#863](https://github.com/influxdata/telegraf/pull/863): AMQP output: allow external auth. Thanks @ekini!
+- [#707](https://github.com/influxdata/telegraf/pull/707): Improved prometheus plugin. Thanks @titilambert!
+- [#878](https://github.com/influxdata/telegraf/pull/878): Added json serializer. Thanks @ch3lo!
+- [#880](https://github.com/influxdata/telegraf/pull/880): Add the ability to specify the bearer token to the prometheus plugin. Thanks @jchauncey!
+- [#882](https://github.com/influxdata/telegraf/pull/882): Fixed SQL Server Plugin issues
+- [#849](https://github.com/influxdata/telegraf/issues/849): Adding ability to parse single values as an input data type.
+- [#844](https://github.com/influxdata/telegraf/pull/844): postgres_extensible plugin added. Thanks @menardorama!
+- [#866](https://github.com/influxdata/telegraf/pull/866): couchbase input plugin. Thanks @ljosa!
+- [#789](https://github.com/influxdata/telegraf/pull/789): Support multiple field specification and `field*` in graphite templates. Thanks @chrusty!
+- [#762](https://github.com/influxdata/telegraf/pull/762): Nagios parser for the exec plugin. Thanks @titilambert!
+- [#848](https://github.com/influxdata/telegraf/issues/848): Provide option to omit host tag from telegraf agent.
+- [#928](https://github.com/influxdata/telegraf/pull/928): Deprecating the statsd "convert_names" options, expose separator config.
+- [#919](https://github.com/influxdata/telegraf/pull/919): ipmi_sensor input plugin. Thanks @ebookbug!
+- [#945](https://github.com/influxdata/telegraf/pull/945): KAFKA output: codec, acks, and retry configuration. Thanks @framiere!
+
+### Bugfixes
+- [#890](https://github.com/influxdata/telegraf/issues/890): Create TLS config even if only ssl_ca is provided.
+- [#884](https://github.com/influxdata/telegraf/issues/884): Do not call write method if there are 0 metrics to write.
+- [#898](https://github.com/influxdata/telegraf/issues/898): Put database name in quotes, fixes special characters in the database name.
+- [#656](https://github.com/influxdata/telegraf/issues/656): No longer run `lsof` on linux to get netstat data, fixes permissions issue.
+- [#907](https://github.com/influxdata/telegraf/issues/907): Fix prometheus invalid label/measurement name key.
+- [#841](https://github.com/influxdata/telegraf/issues/841): Fix memcached unix socket panic.
+- [#873](https://github.com/influxdata/telegraf/issues/873): Fix SNMP plugin sometimes not returning metrics. Thanks @titiliambert!
+- [#934](https://github.com/influxdata/telegraf/pull/934): phpfpm: Fix fcgi uri path. Thanks @rudenkovk!
+- [#805](https://github.com/influxdata/telegraf/issues/805): Kafka consumer stops gathering after i/o timeout.
+- [#959](https://github.com/influxdata/telegraf/pull/959): reduce mongodb & prometheus collection timeouts. Thanks @PierreF!
+
+## v0.11.1 [2016-03-17]
+
+### Release Notes
+- Primarily this release was cut to fix [#859](https://github.com/influxdata/telegraf/issues/859)
 
 ### Features
 - [#747](https://github.com/influxdata/telegraf/pull/747): Start telegraf on install & remove on uninstall. Thanks @pierref!

CONTRIBUTING.md

@@ -114,7 +114,7 @@ creating the `Parser` object.
 You should also add the following to your SampleConfig() return:
 
 ```toml
-  ## Data format to consume. This can be "json", "influx" or "graphite"
+  ## Data format to consume.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
@@ -212,8 +212,8 @@ func (s *Simple) Close() error {
 }
 
 func (s *Simple) Write(metrics []telegraf.Metric) error {
-    for _, pt := range points {
+    for _, metric := range metrics {
-        // write `pt` to the output sink here
+        // write `metric` to the output sink here
     }
     return nil
 }
@@ -244,7 +244,7 @@ instantiating and creating the `Serializer` object.
 You should also add the following to your SampleConfig() return:
 
 ```toml
-  ## Data format to output. This can be "influx" or "graphite"
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
@@ -290,10 +290,6 @@ To execute Telegraf tests follow these simple steps:
 instructions
 - execute `make test`
 
-**OSX users**: you will need to install `boot2docker` or `docker-machine`.
-The Makefile will assume that you have a `docker-machine` box called `default` to
-get the IP address.
-
 ### Unit test troubleshooting
 
 Try cleaning up your test environment by executing `make docker-kill` and

Godeps

@@ -1,27 +1,34 @@
-git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git 617c801af238c3af2d9e72c5d4a0f02edad03ce5
 github.com/Shopify/sarama 8aadb476e66ca998f2f6bb3c993e9a2daa3666b9
 github.com/Sirupsen/logrus 219c8cb75c258c552e999735be6df753ffc7afdc
+github.com/aerospike/aerospike-client-go 45863b7fd8640dc12f7fdd397104d97e1986f25a
 github.com/amir/raidman 53c1b967405155bfc8758557863bf2e14f814687
 github.com/aws/aws-sdk-go 13a12060f716145019378a10e2806c174356b857
 github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4
 github.com/cenkalti/backoff 4dc77674aceaabba2c7e3da25d4c823edfb73f99
+github.com/couchbase/go-couchbase cb664315a324d87d19c879d9cc67fda6be8c2ac1
+github.com/couchbase/gomemcached a5ea6356f648fec6ab89add00edd09151455b4b2
+github.com/couchbase/goutils 5823a0cbaaa9008406021dc5daf80125ea30bba6
 github.com/dancannon/gorethink e7cac92ea2bc52638791a021f212145acfedb1fc
 github.com/davecgh/go-spew 5215b55f46b2b919f50a1df0eaa5886afe4e3b3d
+github.com/docker/engine-api 8924d6900370b4c7e7984be5adc61f50a80d7537
+github.com/docker/go-connections f549a9393d05688dff0992ef3efd8bbe6c628aeb
+github.com/docker/go-units 5d2041e26a699eaca682e2ea41c8f891e1060444
 github.com/eapache/go-resiliency b86b1ec0dd4209a588dc1285cdd471e73525c0b3
 github.com/eapache/queue ded5959c0d4e360646dc9e9908cff48666781367
-github.com/fsouza/go-dockerclient a49c8269a6899cae30da1f8a4b82e0ce945f9967
+github.com/eclipse/paho.mqtt.golang 0f7a459f04f13a41b7ed752d47944528d4bf9a86
-github.com/go-ini/ini 776aa739ce9373377cd16f526cdf06cb4c89b40f
 github.com/go-sql-driver/mysql 1fca743146605a172a266e1654e01e5cd5669bee
+github.com/gobwas/glob 49571a1557cd20e6a2410adc6421f85b66c730b5
 github.com/golang/protobuf 552c7b9542c194800fd493123b3798ef0a832032
 github.com/golang/snappy 427fb6fc07997f43afa32f35e850833760e489a7
 github.com/gonuts/go-shellquote e842a11b24c6abfb3dd27af69a17f482e4b483c2
 github.com/gorilla/context 1ea25387ff6f684839d82767c1733ff4d4d15d0a
 github.com/gorilla/mux c9e326e2bdec29039a3761c07bece13133863e1e
 github.com/hailocab/go-hostpool e80d13ce29ede4452c43dea11e79b9bc8a15b478
+github.com/hashicorp/consul 5aa90455ce78d4d41578bafc86305e6e6b28d7d2
+github.com/hpcloud/tail b2940955ab8b26e19d43a43c4da0475dd81bdb56
 github.com/influxdata/config b79f6829346b8d6e78ba73544b1e1038f1f1c9da
-github.com/influxdata/influxdb e3fef5593c21644f2b43af55d6e17e70910b0e48
+github.com/influxdata/influxdb e094138084855d444195b252314dfee9eae34cab
 github.com/influxdata/toml af4df43894b16e3fd2b788d01bd27ad0776ef2d0
-github.com/jmespath/go-jmespath 0b12d6b521d83fc7f755e7cfc1b1fbdd35a01a74
 github.com/klauspost/crc32 19b0b332c9e4516a6370a0456e6182c3b5036720
 github.com/lib/pq e182dc4027e2ded4b19396d638610f2653295f36
 github.com/matttproud/golang_protobuf_extensions d0c3fe89de86839aecf2e0579c40ba3bb336a453
@@ -31,19 +38,21 @@ github.com/naoina/go-stringutil 6b638e95a32d0c1131db0e7fe83775cbea4a0d0b
 github.com/nats-io/nats b13fc9d12b0b123ebc374e6b808c6228ae4234a3
 github.com/nats-io/nuid 4f84f5f3b2786224e336af2e13dba0a0a80b76fa
 github.com/nsqio/go-nsq 0b80d6f05e15ca1930e0c5e1d540ed627e299980
-github.com/pmezard/go-difflib 792786c7400a136282c1664665ae0a8db921c6c2
+github.com/opencontainers/runc 89ab7f2ccc1e45ddf6485eaa802c35dcf321dfc8
 github.com/prometheus/client_golang 18acf9993a863f4c4b40612e19cdd243e7c86831
 github.com/prometheus/client_model fa8ad6fec33561be4280a8f0514318c79d7f6cb6
 github.com/prometheus/common e8eabff8812b05acf522b45fdcd725a785188e37
 github.com/prometheus/procfs 406e5b7bfd8201a36e2bb5f7bdae0b03380c2ce8
 github.com/samuel/go-zookeeper 218e9c81c0dd8b3b18172b2bbfad92cc7d6db55f
-github.com/shirou/gopsutil 1de1357e7737a536c7f4ff6be7bd27977db4d2cb
+github.com/shirou/gopsutil 586bb697f3ec9f8ec08ffefe18f521a64534037c
 github.com/soniah/gosnmp b1b4f885b12c5dcbd021c5cee1c904110de6db7d
+github.com/sparrc/aerospike-client-go d4bb42d2c2d39dae68e054116f4538af189e05d5
 github.com/streadway/amqp b4f3ceab0337f013208d31348b578d83c0064744
-github.com/stretchr/objx 1a9d0bb9f541897e62256577b352fdbc1fb4fd94
 github.com/stretchr/testify 1f4a1643a57e798696635ea4c126e9127adb7d3c
-github.com/wvanbergen/kafka 1a8639a45164fcc245d5c7b4bd3ccfbd1a0ffbf3
+github.com/vjeantet/grok 83bfdfdfd1a8146795b28e547a8e3c8b28a466c2
+github.com/wvanbergen/kafka 46f9a1cf3f670edec492029fadded9c2d9e18866
 github.com/wvanbergen/kazoo-go 0f768712ae6f76454f987c3356177e138df258f8
+github.com/yuin/gopher-lua bf3808abd44b1e55143a2d7f08571aaa80db1808
 github.com/zensqlmonitor/go-mssqldb ffe5510c6fa5e15e6d983210ab501c815b56b363
 golang.org/x/crypto 5dc8cb4b8a8eb076cbb5a06bc3b8682c15bdbbd3
 golang.org/x/net 6acef71eb69611914f7a30939ea9f6e194c78172

Godeps_windows

@@ -1,4 +1,4 @@
-git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git 617c801af238c3af2d9e72c5d4a0f02edad03ce5
+github.com/Microsoft/go-winio 9f57cbbcbcb41dea496528872a4f0e37a4f7ae98
 github.com/Shopify/sarama 8aadb476e66ca998f2f6bb3c993e9a2daa3666b9
 github.com/Sirupsen/logrus 219c8cb75c258c552e999735be6df753ffc7afdc
 github.com/StackExchange/wmi f3e2bae1e0cb5aef83e319133eabfee30013a4a5
@@ -6,11 +6,17 @@ github.com/amir/raidman 53c1b967405155bfc8758557863bf2e14f814687
 github.com/aws/aws-sdk-go 13a12060f716145019378a10e2806c174356b857
 github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4
 github.com/cenkalti/backoff 4dc77674aceaabba2c7e3da25d4c823edfb73f99
+github.com/couchbase/go-couchbase cb664315a324d87d19c879d9cc67fda6be8c2ac1
+github.com/couchbase/gomemcached a5ea6356f648fec6ab89add00edd09151455b4b2
+github.com/couchbase/goutils 5823a0cbaaa9008406021dc5daf80125ea30bba6
 github.com/dancannon/gorethink e7cac92ea2bc52638791a021f212145acfedb1fc
 github.com/davecgh/go-spew 5215b55f46b2b919f50a1df0eaa5886afe4e3b3d
+github.com/docker/engine-api 8924d6900370b4c7e7984be5adc61f50a80d7537
+github.com/docker/go-connections f549a9393d05688dff0992ef3efd8bbe6c628aeb
+github.com/docker/go-units 5d2041e26a699eaca682e2ea41c8f891e1060444
 github.com/eapache/go-resiliency b86b1ec0dd4209a588dc1285cdd471e73525c0b3
 github.com/eapache/queue ded5959c0d4e360646dc9e9908cff48666781367
-github.com/fsouza/go-dockerclient a49c8269a6899cae30da1f8a4b82e0ce945f9967
+github.com/eclipse/paho.mqtt.golang 0f7a459f04f13a41b7ed752d47944528d4bf9a86
 github.com/go-ole/go-ole 50055884d646dd9434f16bbb5c9801749b9bafe4
 github.com/go-sql-driver/mysql 1fca743146605a172a266e1654e01e5cd5669bee
 github.com/golang/protobuf 552c7b9542c194800fd493123b3798ef0a832032
@@ -37,12 +43,12 @@ github.com/prometheus/client_model fa8ad6fec33561be4280a8f0514318c79d7f6cb6
 github.com/prometheus/common e8eabff8812b05acf522b45fdcd725a785188e37
 github.com/prometheus/procfs 406e5b7bfd8201a36e2bb5f7bdae0b03380c2ce8
 github.com/samuel/go-zookeeper 218e9c81c0dd8b3b18172b2bbfad92cc7d6db55f
-github.com/shirou/gopsutil 1de1357e7737a536c7f4ff6be7bd27977db4d2cb
+github.com/shirou/gopsutil 1f32ce1bb380845be7f5d174ac641a2c592c0c42
 github.com/shirou/w32 ada3ba68f000aa1b58580e45c9d308fe0b7fc5c5
 github.com/soniah/gosnmp b1b4f885b12c5dcbd021c5cee1c904110de6db7d
 github.com/streadway/amqp b4f3ceab0337f013208d31348b578d83c0064744
 github.com/stretchr/testify 1f4a1643a57e798696635ea4c126e9127adb7d3c
-github.com/wvanbergen/kafka 1a8639a45164fcc245d5c7b4bd3ccfbd1a0ffbf3
+github.com/wvanbergen/kafka 46f9a1cf3f670edec492029fadded9c2d9e18866
 github.com/wvanbergen/kazoo-go 0f768712ae6f76454f987c3356177e138df258f8
 github.com/zensqlmonitor/go-mssqldb ffe5510c6fa5e15e6d983210ab501c815b56b363
 golang.org/x/net 6acef71eb69611914f7a30939ea9f6e194c78172

Makefile

@@ -1,4 +1,3 @@
-UNAME := $(shell sh -c 'uname')
 VERSION := $(shell sh -c 'git describe --always --tags')
 ifdef GOBIN
 PATH := $(GOBIN):$(PATH)
@@ -14,22 +13,18 @@ windows: prepare-windows build-windows
 
 # Only run the build (no dependency grabbing)
 build:
-	go install -ldflags "-X main.Version=$(VERSION)" ./...
+	go install -ldflags "-X main.version=$(VERSION)" ./...
 
 build-windows:
 	go build -o telegraf.exe -ldflags \
-		"-X main.Version=$(VERSION)" \
+		"-X main.version=$(VERSION)" \
 		./cmd/telegraf/telegraf.go
 
 build-for-docker:
 	CGO_ENABLED=0 GOOS=linux go build -installsuffix cgo -o telegraf -ldflags \
-					"-s -X main.Version=$(VERSION)" \
+					"-s -X main.version=$(VERSION)" \
 					./cmd/telegraf/telegraf.go
 
-# Build with race detector
-dev: prepare
-	go build -race -ldflags "-X main.Version=$(VERSION)" ./...
-
 # run package script
 package:
 	./scripts/build.py --package --version="$(VERSION)" --platform=linux --arch=all --upload
@@ -46,27 +41,17 @@ prepare-windows:
 
 # Run all docker containers necessary for unit tests
 docker-run:
-ifeq ($(UNAME), Darwin)
-	docker run --name kafka \
-		-e ADVERTISED_HOST=$(shell sh -c 'boot2docker ip || docker-machine ip default') \
-		-e ADVERTISED_PORT=9092 \
-		-p "2181:2181" -p "9092:9092" \
-		-d spotify/kafka
-endif
-ifeq ($(UNAME), Linux)
 	docker run --name kafka \
 		-e ADVERTISED_HOST=localhost \
 		-e ADVERTISED_PORT=9092 \
 		-p "2181:2181" -p "9092:9092" \
 		-d spotify/kafka
-endif
 	docker run --name mysql -p "3306:3306" -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -d mysql
 	docker run --name memcached -p "11211:11211" -d memcached
 	docker run --name postgres -p "5432:5432" -d postgres
 	docker run --name rabbitmq -p "15672:15672" -p "5672:5672" -d rabbitmq:3-management
-	docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
 	docker run --name redis -p "6379:6379" -d redis
-	docker run --name aerospike -p "3000:3000" -d aerospike
+	docker run --name aerospike -p "3000:3000" -d aerospike/aerospike-server
 	docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
 	docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
 	docker run --name riemann -p "5555:5555" -d blalor/riemann
@@ -79,8 +64,7 @@ docker-run-circle:
 		-e ADVERTISED_PORT=9092 \
 		-p "2181:2181" -p "9092:9092" \
 		-d spotify/kafka
-	docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
+	docker run --name aerospike -p "3000:3000" -d aerospike/aerospike-server
-	docker run --name aerospike -p "3000:3000" -d aerospike
 	docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
 	docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
 	docker run --name riemann -p "5555:5555" -d blalor/riemann
@@ -88,8 +72,8 @@ docker-run-circle:
 
 # Kill all docker containers, ignore errors
 docker-kill:
-	-docker kill nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann snmp
+	-docker kill nsq aerospike redis rabbitmq postgres memcached mysql kafka mqtt riemann snmp
-	-docker rm nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann snmp
+	-docker rm nsq aerospike redis rabbitmq postgres memcached mysql kafka mqtt riemann snmp
 
 # Run full unit tests using docker containers (includes setup and teardown)
 test: vet docker-kill docker-run

README.md

@@ -1,4 +1,4 @@
-# Telegraf [![Circle CI](https://circleci.com/gh/influxdata/telegraf.svg?style=svg)](https://circleci.com/gh/influxdata/telegraf)
+# Telegraf [![Circle CI](https://circleci.com/gh/influxdata/telegraf.svg?style=svg)](https://circleci.com/gh/influxdata/telegraf) [![Docker pulls](https://img.shields.io/docker/pulls/library/telegraf.svg)](https://hub.docker.com/_/telegraf/)
 
 Telegraf is an agent written in Go for collecting metrics from the system it's
 running on, or from other services, and writing them into InfluxDB or other
@@ -17,26 +17,15 @@ new plugins.
 
 ## Installation:
 
-NOTE: Telegraf 0.10.x is **not** backwards-compatible with previous versions
-of telegraf, both in the database layout and the configuration file. 0.2.x
-will continue to be supported, see below for download links.
-
-For more details on the differences between Telegraf 0.2.x and 0.10.x, see
-the [release blog post](https://influxdata.com/blog/announcing-telegraf-0-10-0/).
-
 ### Linux deb and rpm Packages:
 
 Latest:
-* http://get.influxdb.org/telegraf/telegraf_0.11.1-1_amd64.deb
+* https://dl.influxdata.com/telegraf/releases/telegraf_1.0.0-beta3_amd64.deb
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1.x86_64.rpm
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0_beta3.x86_64.rpm
 
 Latest (arm):
-* http://get.influxdb.org/telegraf/telegraf_0.11.1-1_armhf.deb
+* https://dl.influxdata.com/telegraf/releases/telegraf_1.0.0-beta3_armhf.deb
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1.armhf.rpm
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0_beta3.armhf.rpm
-
-0.2.x:
-* http://get.influxdb.org/telegraf/telegraf_0.2.4_amd64.deb
-* http://get.influxdb.org/telegraf/telegraf-0.2.4-1.x86_64.rpm
 
 ##### Package Instructions:
 
@@ -50,34 +39,21 @@ controlled via `systemctl [action] telegraf`
 ### yum/apt Repositories:
 
 There is a yum/apt repo available for the whole InfluxData stack, see
-[here](https://docs.influxdata.com/influxdb/v0.9/introduction/installation/#installation)
+[here](https://docs.influxdata.com/influxdb/v0.10/introduction/installation/#installation)
-for instructions, replacing the `influxdb` package name with `telegraf`.
+for instructions on setting up the repo. Once it is configured, you will be able
+to use this repo to install & update telegraf.
 
 ### Linux tarballs:
 
 Latest:
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1_linux_amd64.tar.gz
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta3_linux_amd64.tar.gz
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1_linux_i386.tar.gz
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta3_linux_i386.tar.gz
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1_linux_armhf.tar.gz
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta3_linux_armhf.tar.gz
 
-0.2.x:
+### FreeBSD tarball:
-* http://get.influxdb.org/telegraf/telegraf_linux_amd64_0.2.4.tar.gz
-* http://get.influxdb.org/telegraf/telegraf_linux_386_0.2.4.tar.gz
-* http://get.influxdb.org/telegraf/telegraf_linux_arm_0.2.4.tar.gz
 
-##### tarball Instructions:
+Latest:
-
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta3_freebsd_amd64.tar.gz
-To install the full directory structure with config file, run:
-
-```
-sudo tar -C / -zxvf ./telegraf-0.11.1-1_linux_amd64.tar.gz
-```
-
-To extract only the binary, run:
-
-```
-tar -zxvf telegraf-0.11.1-1_linux_amd64.tar.gz --strip-components=3 ./usr/bin/telegraf
-```
 
 ### Ansible Role:
 
@@ -93,8 +69,7 @@ brew install telegraf
 ### Windows Binaries (EXPERIMENTAL)
 
 Latest:
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1_windows_amd64.zip
+* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta3_windows_amd64.zip
-* http://get.influxdb.org/telegraf/telegraf-0.11.1-1_windows_i386.zip
 
 ### From Source:
 
@@ -162,52 +137,66 @@ more information on each, please look at the directory of the same name in
 
 Currently implemented sources:
 
-* aerospike
+* [aws cloudwatch](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cloudwatch)
-* apache
+* [aerospike](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/aerospike)
-* bcache
+* [apache](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/apache)
-* couchdb
+* [bcache](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/bcache)
-* disque
+* [cassandra](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cassandra)
-* dns query time
+* [ceph](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph)
-* docker
+* [chrony](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/chrony)
-* dovecot
+* [consul](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/consul)
-* elasticsearch
+* [conntrack](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/conntrack)
-* exec (generic executable plugin, support JSON, influx and graphite)
+* [couchbase](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/couchbase)
-* haproxy
+* [couchdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/couchdb)
-* httpjson (generic JSON-emitting http service plugin)
+* [disque](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/disque)
-* influxdb
+* [dns query time](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/dns_query)
-* jolokia
+* [docker](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/docker)
-* leofs
+* [dovecot](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/dovecot)
-* lustre2
+* [elasticsearch](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/elasticsearch)
-* mailchimp
+* [exec](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec) (generic executable plugin, support JSON, influx, graphite and nagios)
-* memcached
+* [filestat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/filestat)
-* mesos
+* [haproxy](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/haproxy)
-* mongodb
+* [http_response](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/http_response)
-* mysql
+* [httpjson](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/httpjson) (generic JSON-emitting http service plugin)
-* net_response
+* [influxdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/influxdb)
-* nginx
+* [ipmi_sensor](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ipmi_sensor)
-* nsq
+* [jolokia](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/jolokia)
-* ntpq
+* [leofs](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/leofs)
-* phpfpm
+* [lustre2](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/lustre2)
-* phusion passenger
+* [mailchimp](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mailchimp)
-* ping
+* [memcached](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/memcached)
-* postgresql
+* [mesos](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mesos)
-* powerdns
+* [mongodb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mongodb)
-* procstat
+* [mysql](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mysql)
-* prometheus
+* [net_response](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/net_response)
-* puppetagent
+* [nginx](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nginx)
-* rabbitmq
+* [nsq](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nsq)
-* raindrops
+* [nstat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nstat)
-* redis
+* [ntpq](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ntpq)
-* rethinkdb
+* [phpfpm](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/phpfpm)
-* riak
+* [phusion passenger](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/passenger)
-* sensors (only available if built from source)
+* [ping](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ping)
-* snmp
+* [postgresql](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/postgresql)
-* sql server (microsoft)
+* [postgresql_extensible](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/postgresql_extensible)
-* twemproxy
+* [powerdns](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/powerdns)
-* zfs
+* [procstat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/procstat)
-* zookeeper
+* [prometheus](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/prometheus)
-* win_perf_counters (windows performance counters)
+* [puppetagent](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/puppetagent)
-* system
+* [rabbitmq](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/rabbitmq)
+* [raindrops](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/raindrops)
+* [redis](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/redis)
+* [rethinkdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/rethinkdb)
+* [riak](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/riak)
+* [sensors ](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sensors) (only available if built from source)
+* [snmp](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/snmp)
+* [sql server](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver) (microsoft)
+* [twemproxy](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/twemproxy)
+* [varnish](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/varnish)
+* [zfs](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/zfs)
+* [zookeeper](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/zookeeper)
+* [win_perf_counters ](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/win_perf_counters) (windows performance counters)
+* [sysstat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sysstat)
+* [system](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/system)
     * cpu
     * mem
     * net
@@ -217,36 +206,45 @@ Currently implemented sources:
     * swap
     * processes
     * kernel (/proc/stat)
+    * kernel (/proc/vmstat)
 
 Telegraf can also collect metrics via the following service plugins:
 
-* statsd
+* [statsd](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd)
-* udp_listener
+* [tail](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tail)
-* tcp_listener
+* [udp_listener](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/udp_listener)
-* mqtt_consumer
+* [tcp_listener](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tcp_listener)
-* kafka_consumer
+* [mqtt_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer)
-* nats_consumer
+* [kafka_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/kafka_consumer)
-* github_webhooks
+* [nats_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nats_consumer)
+* [webhooks](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks)
+  * [github](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/github)
+  * [mandrill](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/mandrill)
+  * [rollbar](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/rollbar)
+* [nsq_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nsq_consumer)
 
 We'll be adding support for many more over the coming months. Read on if you
 want to add support for another service or third-party API.
 
 ## Supported Output Plugins
 
-* influxdb
+* [influxdb](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/influxdb)
-* amon
+* [amon](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/amon)
-* amqp
+* [amqp](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/amqp)
-* aws kinesis
+* [aws kinesis](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kinesis)
-* aws cloudwatch
+* [aws cloudwatch](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/cloudwatch)
-* datadog
+* [datadog](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/datadog)
-* graphite
+* [file](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/file)
-* kafka
+* [graphite](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/graphite)
-* librato
+* [graylog](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/graylog)
-* mqtt
+* [instrumental](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/instrumental)
-* nsq
+* [kafka](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kafka)
-* opentsdb
+* [librato](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/librato)
-* prometheus
+* [mqtt](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/mqtt)
-* riemann
+* [nsq](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/nsq)
+* [opentsdb](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/opentsdb)
+* [prometheus](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/prometheus_client)
+* [riemann](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/riemann)
 
 ## Contributing
 

accumulator.go

@@ -18,4 +18,8 @@ type Accumulator interface {
 
 	Debug() bool
 	SetDebug(enabled bool)
+
+	SetPrecision(precision, interval time.Duration)
+
+	DisablePrecision()
 }

agent/accumulator.go

@@ -4,7 +4,6 @@ import (
 	"fmt"
 	"log"
 	"math"
-	"sync"
 	"time"
 
 	"github.com/influxdata/telegraf"
@@ -18,21 +17,24 @@ func NewAccumulator(
 	acc := accumulator{}
 	acc.metrics = metrics
 	acc.inputConfig = inputConfig
+	acc.precision = time.Nanosecond
 	return &acc
 }
 
 type accumulator struct {
-	sync.Mutex
-
 	metrics chan telegraf.Metric
 
 	defaultTags map[string]string
 
 	debug bool
+	// print every point added to the accumulator
+	trace bool
 
 	inputConfig *internal_models.InputConfig
 
 	prefix string
+
+	precision time.Duration
 }
 
 func (ac *accumulator) Add(
@@ -96,6 +98,7 @@ func (ac *accumulator) AddFields(
 			tags[k] = v
 		}
 	}
+	ac.inputConfig.Filter.FilterTags(tags)
 
 	result := make(map[string]interface{})
 	for k, v := range fields {
@@ -141,6 +144,7 @@ func (ac *accumulator) AddFields(
 	} else {
 		timestamp = time.Now()
 	}
+	timestamp = timestamp.Round(ac.precision)
 
 	if ac.prefix != "" {
 		measurement = ac.prefix + measurement
@@ -151,7 +155,7 @@ func (ac *accumulator) AddFields(
 		log.Printf("Error adding point [%s]: %s\n", measurement, err.Error())
 		return
 	}
-	if ac.debug {
+	if ac.trace {
 		fmt.Println("> " + m.String())
 	}
 	ac.metrics <- m
@@ -165,6 +169,39 @@ func (ac *accumulator) SetDebug(debug bool) {
 	ac.debug = debug
 }
 
+func (ac *accumulator) Trace() bool {
+	return ac.trace
+}
+
+func (ac *accumulator) SetTrace(trace bool) {
+	ac.trace = trace
+}
+
+// SetPrecision takes two time.Duration objects. If the first is non-zero,
+// it sets that as the precision. Otherwise, it takes the second argument
+// as the order of time that the metrics should be rounded to, with the
+// maximum being 1s.
+func (ac *accumulator) SetPrecision(precision, interval time.Duration) {
+	if precision > 0 {
+		ac.precision = precision
+		return
+	}
+	switch {
+	case interval >= time.Second:
+		ac.precision = time.Second
+	case interval >= time.Millisecond:
+		ac.precision = time.Millisecond
+	case interval >= time.Microsecond:
+		ac.precision = time.Microsecond
+	default:
+		ac.precision = time.Nanosecond
+	}
+}
+
+func (ac *accumulator) DisablePrecision() {
+	ac.precision = time.Nanosecond
+}
+
 func (ac *accumulator) setDefaultTags(tags map[string]string) {
 	ac.defaultTags = tags
 }

agent/accumulator_test.go

@@ -38,6 +38,128 @@ func TestAdd(t *testing.T) {
 		actual)
 }
 
+func TestAddNoPrecisionWithInterval(t *testing.T) {
+	a := accumulator{}
+	now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
+	a.metrics = make(chan telegraf.Metric, 10)
+	defer close(a.metrics)
+	a.inputConfig = &internal_models.InputConfig{}
+
+	a.SetPrecision(0, time.Second)
+	a.Add("acctest", float64(101), map[string]string{})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+
+	testm := <-a.metrics
+	actual := testm.String()
+	assert.Contains(t, actual, "acctest value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Contains(t, actual, "acctest,acc=test value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800000000000)),
+		actual)
+}
+
+func TestAddNoIntervalWithPrecision(t *testing.T) {
+	a := accumulator{}
+	now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
+	a.metrics = make(chan telegraf.Metric, 10)
+	defer close(a.metrics)
+	a.inputConfig = &internal_models.InputConfig{}
+
+	a.SetPrecision(time.Second, time.Millisecond)
+	a.Add("acctest", float64(101), map[string]string{})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+
+	testm := <-a.metrics
+	actual := testm.String()
+	assert.Contains(t, actual, "acctest value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Contains(t, actual, "acctest,acc=test value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800000000000)),
+		actual)
+}
+
+func TestAddDisablePrecision(t *testing.T) {
+	a := accumulator{}
+	now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
+	a.metrics = make(chan telegraf.Metric, 10)
+	defer close(a.metrics)
+	a.inputConfig = &internal_models.InputConfig{}
+
+	a.SetPrecision(time.Second, time.Millisecond)
+	a.DisablePrecision()
+	a.Add("acctest", float64(101), map[string]string{})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+
+	testm := <-a.metrics
+	actual := testm.String()
+	assert.Contains(t, actual, "acctest value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Contains(t, actual, "acctest,acc=test value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800082912748)),
+		actual)
+}
+
+func TestDifferentPrecisions(t *testing.T) {
+	a := accumulator{}
+	now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
+	a.metrics = make(chan telegraf.Metric, 10)
+	defer close(a.metrics)
+	a.inputConfig = &internal_models.InputConfig{}
+
+	a.SetPrecision(0, time.Second)
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+	testm := <-a.metrics
+	actual := testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800000000000)),
+		actual)
+
+	a.SetPrecision(0, time.Millisecond)
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800083000000)),
+		actual)
+
+	a.SetPrecision(0, time.Microsecond)
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800082913000)),
+		actual)
+
+	a.SetPrecision(0, time.Nanosecond)
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800082912748)),
+		actual)
+}
+
 func TestAddDefaultTags(t *testing.T) {
 	a := accumulator{}
 	a.addDefaultTag("default", "tag")
@@ -300,3 +422,35 @@ func TestAddBools(t *testing.T) {
 		fmt.Sprintf("acctest,acc=test,default=tag value=false %d", now.UnixNano()),
 		actual)
 }
+
+// Test that tag filters get applied to metrics.
+func TestAccFilterTags(t *testing.T) {
+	a := accumulator{}
+	now := time.Now()
+	a.metrics = make(chan telegraf.Metric, 10)
+	defer close(a.metrics)
+	filter := internal_models.Filter{
+		TagExclude: []string{"acc"},
+	}
+	assert.NoError(t, filter.CompileFilter())
+	a.inputConfig = &internal_models.InputConfig{}
+	a.inputConfig.Filter = filter
+
+	a.Add("acctest", float64(101), map[string]string{})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"})
+	a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
+
+	testm := <-a.metrics
+	actual := testm.String()
+	assert.Contains(t, actual, "acctest value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Contains(t, actual, "acctest value=101")
+
+	testm = <-a.metrics
+	actual = testm.String()
+	assert.Equal(t,
+		fmt.Sprintf("acctest value=101 %d", now.UnixNano()),
+		actual)
+}

agent/agent.go

@@ -1,17 +1,15 @@
 package agent
 
 import (
-	cryptorand "crypto/rand"
 	"fmt"
 	"log"
-	"math/big"
-	"math/rand"
 	"os"
 	"runtime"
 	"sync"
 	"time"
 
 	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/internal"
 	"github.com/influxdata/telegraf/internal/config"
 	"github.com/influxdata/telegraf/internal/models"
 )
@@ -27,17 +25,19 @@ func NewAgent(config *config.Config) (*Agent, error) {
 		Config: config,
 	}
 
-	if a.Config.Agent.Hostname == "" {
+	if !a.Config.Agent.OmitHostname {
-		hostname, err := os.Hostname()
+		if a.Config.Agent.Hostname == "" {
-		if err != nil {
+			hostname, err := os.Hostname()
-			return nil, err
+			if err != nil {
+				return nil, err
+			}
+
+			a.Config.Agent.Hostname = hostname
 		}
 
-		a.Config.Agent.Hostname = hostname
+		config.Tags["host"] = a.Config.Agent.Hostname
 	}
 
-	config.Tags["host"] = a.Config.Agent.Hostname
-
 	return a, nil
 }
 
@@ -100,93 +100,41 @@ func panicRecover(input *internal_models.RunningInput) {
 	}
 }
 
-// gatherParallel runs the inputs that are using the same reporting interval
+// gatherer runs the inputs that have been configured with their own
-// as the telegraf agent.
-func (a *Agent) gatherParallel(metricC chan telegraf.Metric) error {
-	var wg sync.WaitGroup
-
-	start := time.Now()
-	counter := 0
-	jitter := a.Config.Agent.CollectionJitter.Duration.Nanoseconds()
-	for _, input := range a.Config.Inputs {
-		if input.Config.Interval != 0 {
-			continue
-		}
-
-		wg.Add(1)
-		counter++
-		go func(input *internal_models.RunningInput) {
-			defer panicRecover(input)
-			defer wg.Done()
-
-			acc := NewAccumulator(input.Config, metricC)
-			acc.SetDebug(a.Config.Agent.Debug)
-			acc.setDefaultTags(a.Config.Tags)
-
-			if jitter != 0 {
-				nanoSleep := rand.Int63n(jitter)
-				d, err := time.ParseDuration(fmt.Sprintf("%dns", nanoSleep))
-				if err != nil {
-					log.Printf("Jittering collection interval failed for plugin %s",
-						input.Name)
-				} else {
-					time.Sleep(d)
-				}
-			}
-
-			if err := input.Input.Gather(acc); err != nil {
-				log.Printf("Error in input [%s]: %s", input.Name, err)
-			}
-
-		}(input)
-	}
-
-	if counter == 0 {
-		return nil
-	}
-
-	wg.Wait()
-
-	elapsed := time.Since(start)
-	if !a.Config.Agent.Quiet {
-		log.Printf("Gathered metrics, (%s interval), from %d inputs in %s\n",
-			a.Config.Agent.Interval.Duration, counter, elapsed)
-	}
-	return nil
-}
-
-// gatherSeparate runs the inputs that have been configured with their own
 // reporting interval.
-func (a *Agent) gatherSeparate(
+func (a *Agent) gatherer(
 	shutdown chan struct{},
 	input *internal_models.RunningInput,
+	interval time.Duration,
 	metricC chan telegraf.Metric,
 ) error {
 	defer panicRecover(input)
 
-	ticker := time.NewTicker(input.Config.Interval)
+	ticker := time.NewTicker(interval)
+	defer ticker.Stop()
 
 	for {
 		var outerr error
-		start := time.Now()
 
 		acc := NewAccumulator(input.Config, metricC)
 		acc.SetDebug(a.Config.Agent.Debug)
+		acc.SetPrecision(a.Config.Agent.Precision.Duration,
+			a.Config.Agent.Interval.Duration)
 		acc.setDefaultTags(a.Config.Tags)
 
-		if err := input.Input.Gather(acc); err != nil {
+		internal.RandomSleep(a.Config.Agent.CollectionJitter.Duration, shutdown)
-			log.Printf("Error in input [%s]: %s", input.Name, err)
-		}
 
+		start := time.Now()
+		gatherWithTimeout(shutdown, input, acc, interval)
 		elapsed := time.Since(start)
-		if !a.Config.Agent.Quiet {
-			log.Printf("Gathered metrics, (separate %s interval), from %s in %s\n",
-				input.Config.Interval, input.Name, elapsed)
-		}
 
 		if outerr != nil {
 			return outerr
 		}
+		if a.Config.Agent.Debug {
+			log.Printf("Input [%s] gathered metrics, (%s interval) in %s\n",
+				input.Name, interval, elapsed)
+		}
 
 		select {
 		case <-shutdown:
@@ -197,6 +145,42 @@ func (a *Agent) gatherSeparate(
 	}
 }
 
+// gatherWithTimeout gathers from the given input, with the given timeout.
+//   when the given timeout is reached, gatherWithTimeout logs an error message
+//   but continues waiting for it to return. This is to avoid leaving behind
+//   hung processes, and to prevent re-calling the same hung process over and
+//   over.
+func gatherWithTimeout(
+	shutdown chan struct{},
+	input *internal_models.RunningInput,
+	acc *accumulator,
+	timeout time.Duration,
+) {
+	ticker := time.NewTicker(timeout)
+	defer ticker.Stop()
+	done := make(chan error)
+	go func() {
+		done <- input.Input.Gather(acc)
+	}()
+
+	for {
+		select {
+		case err := <-done:
+			if err != nil {
+				log.Printf("ERROR in input [%s]: %s", input.Name, err)
+			}
+			return
+		case <-ticker.C:
+			log.Printf("ERROR: input [%s] took longer to collect than "+
+				"collection interval (%s)",
+				input.Name, timeout)
+			continue
+		case <-shutdown:
+			return
+		}
+	}
+}
+
 // Test verifies that we can 'Gather' from all inputs with their configured
 // Config struct
 func (a *Agent) Test() error {
@@ -218,7 +202,10 @@ func (a *Agent) Test() error {
 
 	for _, input := range a.Config.Inputs {
 		acc := NewAccumulator(input.Config, metricC)
-		acc.SetDebug(true)
+		acc.SetTrace(true)
+		acc.SetPrecision(a.Config.Agent.Precision.Duration,
+			a.Config.Agent.Interval.Duration)
+		acc.setDefaultTags(a.Config.Tags)
 
 		fmt.Printf("* Plugin: %s, Collection 1\n", input.Name)
 		if input.Config.Interval != 0 {
@@ -278,44 +265,40 @@ func (a *Agent) flusher(shutdown chan struct{}, metricC chan telegraf.Metric) er
 			a.flush()
 			return nil
 		case <-ticker.C:
+			internal.RandomSleep(a.Config.Agent.FlushJitter.Duration, shutdown)
 			a.flush()
 		case m := <-metricC:
-			for _, o := range a.Config.Outputs {
+			for i, o := range a.Config.Outputs {
-				o.AddMetric(m)
+				if i == len(a.Config.Outputs)-1 {
+					o.AddMetric(m)
+				} else {
+					o.AddMetric(copyMetric(m))
+				}
 			}
 		}
 	}
 }
 
-// jitterInterval applies the the interval jitter to the flush interval using
+func copyMetric(m telegraf.Metric) telegraf.Metric {
-// crypto/rand number generator
+	t := time.Time(m.Time())
-func jitterInterval(ininterval, injitter time.Duration) time.Duration {
+
-	var jitter int64
+	tags := make(map[string]string)
-	outinterval := ininterval
+	fields := make(map[string]interface{})
-	if injitter.Nanoseconds() != 0 {
+	for k, v := range m.Tags() {
-		maxjitter := big.NewInt(injitter.Nanoseconds())
+		tags[k] = v
-		if j, err := cryptorand.Int(cryptorand.Reader, maxjitter); err == nil {
+	}
-			jitter = j.Int64()
+	for k, v := range m.Fields() {
-		}
+		fields[k] = v
-		outinterval = time.Duration(jitter + ininterval.Nanoseconds())
 	}
 
-	if outinterval.Nanoseconds() < time.Duration(500*time.Millisecond).Nanoseconds() {
+	out, _ := telegraf.NewMetric(m.Name(), tags, fields, t)
-		log.Printf("Flush interval %s too low, setting to 500ms\n", outinterval)
+	return out
-		outinterval = time.Duration(500 * time.Millisecond)
-	}
-
-	return outinterval
 }
 
 // Run runs the agent daemon, gathering every Interval
 func (a *Agent) Run(shutdown chan struct{}) error {
 	var wg sync.WaitGroup
 
-	a.Config.Agent.FlushInterval.Duration = jitterInterval(
-		a.Config.Agent.FlushInterval.Duration,
-		a.Config.Agent.FlushJitter.Duration)
-
 	log.Printf("Agent Config: Interval:%s, Debug:%#v, Quiet:%#v, Hostname:%#v, "+
 		"Flush Interval:%s \n",
 		a.Config.Agent.Interval.Duration, a.Config.Agent.Debug, a.Config.Agent.Quiet,
@@ -330,6 +313,9 @@ func (a *Agent) Run(shutdown chan struct{}) error {
 		case telegraf.ServiceInput:
 			acc := NewAccumulator(input.Config, metricC)
 			acc.SetDebug(a.Config.Agent.Debug)
+			// Service input plugins should set their own precision of their
+			// metrics.
+			acc.DisablePrecision()
 			acc.setDefaultTags(a.Config.Tags)
 			if err := p.Start(acc); err != nil {
 				log.Printf("Service for input %s failed to start, exiting\n%s\n",
@@ -345,7 +331,6 @@ func (a *Agent) Run(shutdown chan struct{}) error {
 		i := int64(a.Config.Agent.Interval.Duration)
 		time.Sleep(time.Duration(i - (time.Now().UnixNano() % i)))
 	}
-	ticker := time.NewTicker(a.Config.Agent.Interval.Duration)
 
 	wg.Add(1)
 	go func() {
@@ -356,32 +341,21 @@ func (a *Agent) Run(shutdown chan struct{}) error {
 		}
 	}()
 
+	wg.Add(len(a.Config.Inputs))
 	for _, input := range a.Config.Inputs {
-		// Special handling for inputs that have their own collection interval
+		interval := a.Config.Agent.Interval.Duration
-		// configured. Default intervals are handled below with gatherParallel
+		// overwrite global interval if this plugin has it's own.
 		if input.Config.Interval != 0 {
-			wg.Add(1)
+			interval = input.Config.Interval
-			go func(input *internal_models.RunningInput) {
-				defer wg.Done()
-				if err := a.gatherSeparate(shutdown, input, metricC); err != nil {
-					log.Printf(err.Error())
-				}
-			}(input)
 		}
+		go func(in *internal_models.RunningInput, interv time.Duration) {
+			defer wg.Done()
+			if err := a.gatherer(shutdown, in, interv, metricC); err != nil {
+				log.Printf(err.Error())
+			}
+		}(input, interval)
 	}
 
-	defer wg.Wait()
+	wg.Wait()
-
+	return nil
-	for {
-		if err := a.gatherParallel(metricC); err != nil {
-			log.Printf(err.Error())
-		}
-
-		select {
-		case <-shutdown:
-			return nil
-		case <-ticker.C:
-			continue
-		}
-	}
 }

agent/agent_test.go

@@ -1,9 +1,7 @@
 package agent
 
 import (
-	"github.com/stretchr/testify/assert"
 	"testing"
-	"time"
 
 	"github.com/influxdata/telegraf/internal/config"
 
@@ -11,8 +9,18 @@ import (
 	_ "github.com/influxdata/telegraf/plugins/inputs/all"
 	// needing to load the outputs
 	_ "github.com/influxdata/telegraf/plugins/outputs/all"
+
+	"github.com/stretchr/testify/assert"
 )
 
+func TestAgent_OmitHostname(t *testing.T) {
+	c := config.NewConfig()
+	c.Agent.OmitHostname = true
+	_, err := NewAgent(c)
+	assert.NoError(t, err)
+	assert.NotContains(t, c.Tags, "host")
+}
+
 func TestAgent_LoadPlugin(t *testing.T) {
 	c := config.NewConfig()
 	c.InputFilters = []string{"mysql"}
@@ -101,75 +109,3 @@ func TestAgent_LoadOutput(t *testing.T) {
 	a, _ = NewAgent(c)
 	assert.Equal(t, 3, len(a.Config.Outputs))
 }
-
-func TestAgent_ZeroJitter(t *testing.T) {
-	flushinterval := jitterInterval(time.Duration(10*time.Second),
-		time.Duration(0*time.Second))
-
-	actual := flushinterval.Nanoseconds()
-	exp := time.Duration(10 * time.Second).Nanoseconds()
-
-	if actual != exp {
-		t.Errorf("Actual %v, expected %v", actual, exp)
-	}
-}
-
-func TestAgent_ZeroInterval(t *testing.T) {
-	min := time.Duration(500 * time.Millisecond).Nanoseconds()
-	max := time.Duration(5 * time.Second).Nanoseconds()
-
-	for i := 0; i < 1000; i++ {
-		flushinterval := jitterInterval(time.Duration(0*time.Second),
-			time.Duration(5*time.Second))
-		actual := flushinterval.Nanoseconds()
-
-		if actual > max {
-			t.Errorf("Didn't expect interval %d to be > %d", actual, max)
-			break
-		}
-		if actual < min {
-			t.Errorf("Didn't expect interval %d to be < %d", actual, min)
-			break
-		}
-	}
-}
-
-func TestAgent_ZeroBoth(t *testing.T) {
-	flushinterval := jitterInterval(time.Duration(0*time.Second),
-		time.Duration(0*time.Second))
-
-	actual := flushinterval
-	exp := time.Duration(500 * time.Millisecond)
-
-	if actual != exp {
-		t.Errorf("Actual %v, expected %v", actual, exp)
-	}
-}
-
-func TestAgent_JitterMax(t *testing.T) {
-	max := time.Duration(32 * time.Second).Nanoseconds()
-
-	for i := 0; i < 1000; i++ {
-		flushinterval := jitterInterval(time.Duration(30*time.Second),
-			time.Duration(2*time.Second))
-		actual := flushinterval.Nanoseconds()
-		if actual > max {
-			t.Errorf("Didn't expect interval %d to be > %d", actual, max)
-			break
-		}
-	}
-}
-
-func TestAgent_JitterMin(t *testing.T) {
-	min := time.Duration(30 * time.Second).Nanoseconds()
-
-	for i := 0; i < 1000; i++ {
-		flushinterval := jitterInterval(time.Duration(30*time.Second),
-			time.Duration(2*time.Second))
-		actual := flushinterval.Nanoseconds()
-		if actual < min {
-			t.Errorf("Didn't expect interval %d to be < %d", actual, min)
-			break
-		}
-	}
-}

circle.yml

@@ -4,9 +4,9 @@ machine:
   post:
     - sudo service zookeeper stop
     - go version
-    - go version | grep 1.5.3 || sudo rm -rf /usr/local/go
+    - go version | grep 1.6.2 || sudo rm -rf /usr/local/go
-    - wget https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz
+    - wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
-    - sudo tar -C /usr/local -xzf go1.5.3.linux-amd64.tar.gz
+    - sudo tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz
     - go version
 
 dependencies:

cmd/telegraf/telegraf.go

@@ -32,7 +32,7 @@ var fPidfile = flag.String("pidfile", "", "file to write our pid to")
 var fInputFilters = flag.String("input-filter", "",
 	"filter the inputs to enable, separator is :")
 var fInputList = flag.Bool("input-list", false,
-	"print available output plugins.")
+	"print available input plugins.")
 var fOutputFilters = flag.String("output-filter", "",
 	"filter the outputs to enable, separator is :")
 var fOutputList = flag.Bool("output-list", false,
@@ -46,9 +46,13 @@ var fOutputFiltersLegacy = flag.String("outputfilter", "",
 var fConfigDirectoryLegacy = flag.String("configdirectory", "",
 	"directory containing additional *.conf files")
 
-// Telegraf version
+// Telegraf version, populated linker.
-//	-ldflags "-X main.Version=`git describe --always --tags`"
+//   ie, -ldflags "-X main.version=`git describe --always --tags`"
-var Version string
+var (
+	version string
+	commit  string
+	branch  string
+)
 
 const usage = `Telegraf, The plugin-driven server agent for collecting and reporting metrics.
 
@@ -71,6 +75,13 @@ The flags are:
   -quiet             run in quiet mode
   -version           print the version to stdout
 
+In addition to the -config flag, telegraf will also load the config file from
+an environment variable or default location. Precedence is:
+  1. -config flag
+  2. $TELEGRAF_CONFIG_PATH environment variable
+  3. $HOME/.telegraf/telegraf.conf
+  4. /etc/telegraf/telegraf.conf
+
 Examples:
 
   # generate a telegraf config file:
@@ -98,12 +109,10 @@ func main() {
 		flag.Parse()
 		args := flag.Args()
 
-		if flag.NFlag() == 0 && len(args) == 0 {
-			usageExit(0)
-		}
-
 		var inputFilters []string
 		if *fInputFiltersLegacy != "" {
+			fmt.Printf("WARNING '--filter' flag is deprecated, please use" +
+				" '--input-filter'")
 			inputFilter := strings.TrimSpace(*fInputFiltersLegacy)
 			inputFilters = strings.Split(":"+inputFilter+":", ":")
 		}
@@ -114,6 +123,8 @@ func main() {
 
 		var outputFilters []string
 		if *fOutputFiltersLegacy != "" {
+			fmt.Printf("WARNING '--outputfilter' flag is deprecated, please use" +
+				" '--output-filter'")
 			outputFilter := strings.TrimSpace(*fOutputFiltersLegacy)
 			outputFilters = strings.Split(":"+outputFilter+":", ":")
 		}
@@ -125,7 +136,7 @@ func main() {
 		if len(args) > 0 {
 			switch args[0] {
 			case "version":
-				v := fmt.Sprintf("Telegraf - Version %s", Version)
+				v := fmt.Sprintf("Telegraf - version %s", version)
 				fmt.Println(v)
 				return
 			case "config":
@@ -151,7 +162,7 @@ func main() {
 		}
 
 		if *fVersion {
-			v := fmt.Sprintf("Telegraf - Version %s", Version)
+			v := fmt.Sprintf("Telegraf - version %s", version)
 			fmt.Println(v)
 			return
 		}
@@ -170,25 +181,19 @@ func main() {
 			return
 		}
 
-		var (
+		// If no other options are specified, load the config file and run.
-			c   *config.Config
+		c := config.NewConfig()
-			err error
+		c.OutputFilters = outputFilters
-		)
+		c.InputFilters = inputFilters
-
+		err := c.LoadConfig(*fConfig)
-		if *fConfig != "" {
+		if err != nil {
-			c = config.NewConfig()
+			fmt.Println(err)
-			c.OutputFilters = outputFilters
-			c.InputFilters = inputFilters
-			err = c.LoadConfig(*fConfig)
-			if err != nil {
-				log.Fatal(err)
-			}
-		} else {
-			fmt.Println("You must specify a config file. See telegraf --help")
 			os.Exit(1)
 		}
 
 		if *fConfigDirectoryLegacy != "" {
+			fmt.Printf("WARNING '--configdirectory' flag is deprecated, please use" +
+				" '--config-directory'")
 			err = c.LoadDirectory(*fConfigDirectoryLegacy)
 			if err != nil {
 				log.Fatal(err)
@@ -250,7 +255,7 @@ func main() {
 			}
 		}()
 
-		log.Printf("Starting Telegraf (version %s)\n", Version)
+		log.Printf("Starting Telegraf (version %s)\n", version)
 		log.Printf("Loaded outputs: %s", strings.Join(c.OutputNames(), " "))
 		log.Printf("Loaded inputs: %s", strings.Join(c.InputNames(), " "))
 		log.Printf("Tags enabled: %s", c.ListTags())

docs/CONFIGURATION.md

@@ -3,16 +3,31 @@
 ## Generating a Configuration File
 
 A default Telegraf config file can be generated using the -sample-config flag:
-`telegraf -sample-config > telegraf.conf`
+
+```
+telegraf -sample-config > telegraf.conf
+```
 
 To generate a file with specific inputs and outputs, you can use the
 -input-filter and -output-filter flags:
-`telegraf -sample-config -input-filter cpu:mem:net:swap -output-filter influxdb:kafka`
+
+```
+telegraf -sample-config -input-filter cpu:mem:net:swap -output-filter influxdb:kafka
+```
+
+You can see the latest config file with all available plugins here:
+[telegraf.conf](https://github.com/influxdata/telegraf/blob/master/etc/telegraf.conf)
+
+## Environment Variables
+
+Environment variables can be used anywhere in the config file, simply prepend
+them with $. For strings the variable must be within quotes (ie, "$STR_VAR"),
+for numbers and booleans they should be plain (ie, $INT_VAR, $BOOL_VAR)
 
 ## `[global_tags]` Configuration
 
-Global tags can be specific in the `[global_tags]` section of the config file in
+Global tags can be specified in the `[global_tags]` section of the config file
-key="value" format. All metrics being gathered on this host will be tagged
+in key="value" format. All metrics being gathered on this host will be tagged
 with the tags specified here.
 
 ## `[agent]` Configuration
@@ -23,8 +38,12 @@ config.
 * **interval**: Default data collection interval for all inputs
 * **round_interval**: Rounds collection interval to 'interval'
 ie, if interval="10s" then always collect on :00, :10, :20, etc.
+* **metric_batch_size**: Telegraf will send metrics to output in batch of at
+most metric_batch_size metrics.
 * **metric_buffer_limit**: Telegraf will cache metric_buffer_limit metrics
 for each output, and will flush this buffer on a successful write.
+This should be a multiple of metric_batch_size and could not be less
+than 2 times metric_batch_size.
 * **collection_jitter**: Collection jitter is used to jitter
 the collection by a random amount.
 Each plugin will sleep for a random time within jitter before collecting.
@@ -41,9 +60,35 @@ ie, a jitter of 5s and flush_interval 10s means flushes will happen every 10-15s
 * **quiet**: Run telegraf in quiet mode.
 * **hostname**: Override default hostname, if empty use os.Hostname().
 
-## `[inputs.xxx]` Configuration
+#### Measurement Filtering
 
-There are some configuration options that are configurable per input:
+Filters can be configured per input or output, see below for examples.
+
+* **namepass**: An array of strings that is used to filter metrics generated by the
+current input. Each string in the array is tested as a glob match against
+measurement names and if it matches, the field is emitted.
+* **namedrop**: The inverse of pass, if a measurement name matches, it is not emitted.
+* **fieldpass**: An array of strings that is used to filter metrics generated by the
+current input. Each string in the array is tested as a glob match against field names
+and if it matches, the field is emitted. fieldpass is not available for outputs.
+* **fielddrop**: The inverse of pass, if a field name matches, it is not emitted.
+fielddrop is not available for outputs.
+* **tagpass**: tag names and arrays of strings that are used to filter
+measurements by the current input. Each string in the array is tested as a glob
+match against the tag name, and if it matches the measurement is emitted.
+* **tagdrop**: The inverse of tagpass. If a tag matches, the measurement is not
+emitted. This is tested on measurements that have passed the tagpass test.
+* **tagexclude**: tagexclude can be used to exclude a tag from measurement(s).
+As opposed to tagdrop, which will drop an entire measurement based on it's
+tags, tagexclude simply strips the given tag keys from the measurement. This
+can be used on inputs & outputs, but it is _recommended_ to be used on inputs,
+as it is more efficient to filter out tags at the ingestion point.
+* **taginclude**: taginclude is the inverse of tagexclude. It will only include
+the tag keys in the final measurement.
+
+## Input Configuration
+
+Some configuration options are configurable per input:
 
 * **name_override**: Override the base name of the measurement.
 (Default is the name of the input).
@@ -54,24 +99,6 @@ There are some configuration options that are configurable per input:
 global interval, but if one particular input should be run less or more often,
 you can configure that here.
 
-#### Input Filters
-
-There are also filters that can be configured per input:
-
-* **namepass**: An array of strings that is used to filter metrics generated by the
-current input. Each string in the array is tested as a glob match against
-measurement names and if it matches, the field is emitted.
-* **namedrop**: The inverse of pass, if a measurement name matches, it is not emitted.
-* **fieldpass**: An array of strings that is used to filter metrics generated by the
-current input. Each string in the array is tested as a glob match against field names
-and if it matches, the field is emitted.
-* **fielddrop**: The inverse of pass, if a field name matches, it is not emitted.
-* **tagpass**: tag names and arrays of strings that are used to filter
-measurements by the current input. Each string in the array is tested as a glob
-match against the tag name, and if it matches the measurement is emitted.
-* **tagdrop**: The inverse of tagpass. If a tag matches, the measurement is not
-emitted. This is tested on measurements that have passed the tagpass test.
-
 #### Input Configuration Examples
 
 This is a full working config that will output CPU data to an InfluxDB instance
@@ -141,12 +168,26 @@ fields which begin with `time_`.
 # Drop all metrics about containers for kubelet
 [[inputs.prometheus]]
   urls = ["http://kube-node-1:4194/metrics"]
-  namedrop = ["container_"]
+  namedrop = ["container_*"]
 
 # Only store rest client related metrics for kubelet
 [[inputs.prometheus]]
   urls = ["http://kube-node-1:4194/metrics"]
-  namepass = ["rest_client_"]
+  namepass = ["rest_client_*"]
+```
+
+#### Input Config: taginclude and tagexclude
+
+```toml
+# Only include the "cpu" tag in the measurements for the cpu plugin.
+[[inputs.cpu]]
+  percpu = true
+  totalcpu = true
+  taginclude = ["cpu"]
+
+# Exclude the "fstype" tag from the measurements for the disk plugin.
+[[inputs.disk]]
+  tagexclude = ["fstype"]
 ```
 
 #### Input config: prefix, suffix, and override
@@ -174,6 +215,9 @@ This will emit measurements with the name `foobar`
 This plugin will emit measurements with two additional tags: `tag1=foo` and
 `tag2=bar`
 
+NOTE: Order matters, the `[inputs.cpu.tags]` table must be at the _end_ of the
+plugin definition.
+
 ```toml
 [[inputs.cpu]]
   percpu = false
@@ -202,15 +246,12 @@ to avoid measurement collisions:
   fielddrop = ["cpu_time*"]
 ```
 
-## `[outputs.xxx]` Configuration
+## Output Configuration
 
 Telegraf also supports specifying multiple output sinks to send data to,
 configuring each output sink is different, but examples can be
 found by running `telegraf -sample-config`.
 
-Outputs also support the same configurable options as inputs
-(namepass, namedrop, tagpass, tagdrop)
-
 ```toml
 [[outputs.influxdb]]
   urls = [ "http://localhost:8086" ]

docs/DATA_FORMATS_INPUT.md

@@ -1,5 +1,13 @@
 # Telegraf Input Data Formats
 
+Telegraf is able to parse the following input data formats into metrics:
+
+1. [InfluxDB Line Protocol](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#influx)
+1. [JSON](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#json)
+1. [Graphite](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#graphite)
+1. [Value](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#value), ie: 45 or "booyah"
+1. [Nagios](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#nagios) (exec input only)
+
 Telegraf metrics, like InfluxDB
 [points](https://docs.influxdata.com/influxdb/v0.10/write_protocols/line/),
 are a combination of four basic parts:
@@ -31,7 +39,7 @@ example, in the exec plugin:
   ## measurement name suffix (for separating different commands)
   name_suffix = "_mycollector"
 
-  ## Data format to consume. This can be "json", "influx" or "graphite"
+  ## Data format to consume.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
@@ -43,7 +51,7 @@ example, in the exec plugin:
 Each data_format has an additional set of configuration options available, which
 I'll go over below.
 
-## Influx:
+# Influx:
 
 There are no additional configuration options for InfluxDB line-protocol. The
 metrics are parsed directly into Telegraf metrics.
@@ -58,23 +66,28 @@ metrics are parsed directly into Telegraf metrics.
   ## measurement name suffix (for separating different commands)
   name_suffix = "_mycollector"
 
-  ## Data format to consume. This can be "json", "influx" or "graphite"
+  ## Data format to consume.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
   data_format = "influx"
 ```
 
-## JSON:
+# JSON:
 
-The JSON data format flattens JSON into metric _fields_. For example, this JSON:
+The JSON data format flattens JSON into metric _fields_.
+NOTE: Only numerical values are converted to fields, and they are converted
+into a float. strings are ignored unless specified as a tag_key (see below).
+
+So for example, this JSON:
 
 ```json
 {
     "a": 5,
     "b": {
         "c": 6
-    }
+    },
+    "ignored": "I'm a string"
 }
 ```
 
@@ -103,7 +116,7 @@ For example, if you had this configuration:
   ## measurement name suffix (for separating different commands)
   name_suffix = "_mycollector"
 
-  ## Data format to consume. This can be "json", "influx" or "graphite"
+  ## Data format to consume.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
@@ -134,62 +147,125 @@ Your Telegraf metrics would get tagged with "my_tag_1"
 exec_mycollector,my_tag_1=foo a=5,b_c=6
 ```
 
-## Graphite:
+# Value:
+
+The "value" data format translates single values into Telegraf metrics. This
+is done by assigning a measurement name and setting a single field ("value")
+as the parsed metric.
+
+#### Value Configuration:
+
+You **must** tell Telegraf what type of metric to collect by using the
+`data_type` configuration option. Available options are:
+
+1. integer
+2. float or long
+3. string
+4. boolean
+
+**Note:** It is also recommended that you set `name_override` to a measurement
+name that makes sense for your metric, otherwise it will just be set to the
+name of the plugin.
+
+```toml
+[[inputs.exec]]
+  ## Commands array
+  commands = ["cat /proc/sys/kernel/random/entropy_avail"]
+
+  ## override the default metric name of "exec"
+  name_override = "entropy_available"
+
+  ## Data format to consume.
+  ## Each data format has it's own unique set of configuration options, read
+  ## more about them here:
+  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
+  data_format = "value"
+  data_type = "integer" # required
+```
+
+# Graphite:
 
 The Graphite data format translates graphite _dot_ buckets directly into
-telegraf measurement names, with a single value field, and without any tags. For
+telegraf measurement names, with a single value field, and without any tags.
-more advanced options, Telegraf supports specifying "templates" to translate
+By default, the separator is left as ".", but this can be changed using the
+"separator" argument. For more advanced options,
+Telegraf supports specifying "templates" to translate
 graphite buckets into Telegraf metrics.
 
-#### Separator:
+Templates are of the form:
-
-You can specify a separator to use for the parsed metrics.
-By default, it will leave the metrics with a "." separator.
-Setting `separator = "_"` will translate:
 
 ```
-cpu.usage.idle 99
+"host.mytag.mytag.measurement.measurement.field*"
-=> cpu_usage_idle value=99
 ```
 
-#### Measurement/Tag Templates:
+Where the following keywords exist:
+
+1. `measurement`: specifies that this section of the graphite bucket corresponds
+to the measurement name. This can be specified multiple times.
+2. `field`: specifies that this section of the graphite bucket corresponds
+to the field name. This can be specified multiple times.
+3. `measurement*`: specifies that all remaining elements of the graphite bucket
+correspond to the measurement name.
+4. `field*`: specifies that all remaining elements of the graphite bucket
+correspond to the field name.
+
+Any part of the template that is not a keyword is treated as a tag key. This
+can also be specified multiple times.
+
+NOTE: `field*` cannot be used in conjunction with `measurement*`!
+
+#### Measurement & Tag Templates:
 
 The most basic template is to specify a single transformation to apply to all
-incoming metrics. _measurement_ is a special keyword that tells Telegraf which
+incoming metrics. So the following template:
-parts of the graphite bucket to combine into the measurement name. It can have a
-trailing `*` to indicate that the remainder of the metric should be used.
-Other words are considered tag keys. So the following template:
 
 ```toml
 templates = [
-    "region.measurement*"
+    "region.region.measurement*"
 ]
 ```
 
 would result in the following Graphite -> Telegraf transformation.
 
 ```
-us-west.cpu.load 100
+us.west.cpu.load 100
-=> cpu.load,region=us-west value=100
+=> cpu.load,region=us.west value=100
 ```
 
 #### Field Templates:
 
-There is also a _field_ keyword, which can only be specified once.
 The field keyword tells Telegraf to give the metric that field name.
 So the following template:
 
 ```toml
+separator = "_"
 templates = [
-    "measurement.measurement.field.region"
+    "measurement.measurement.field.field.region"
 ]
 ```
 
 would result in the following Graphite -> Telegraf transformation.
 
 ```
-cpu.usage.idle.us-west 100
+cpu.usage.idle.percent.eu-east 100
-=> cpu_usage,region=us-west idle=100
+=> cpu_usage,region=eu-east idle_percent=100
+```
+
+The field key can also be derived from all remaining elements of the graphite
+bucket by specifying `field*`:
+
+```toml
+separator = "_"
+templates = [
+    "measurement.measurement.region.field*"
+]
+```
+
+which would result in the following Graphite -> Telegraf transformation.
+
+```
+cpu.usage.eu-east.idle.percentage 100
+=> cpu_usage,region=eu-east idle_percentage=100
 ```
 
 #### Filter Templates:
@@ -207,8 +283,8 @@ templates = [
 which would result in the following transformation:
 
 ```
-cpu.load.us-west 100
+cpu.load.eu-east 100
-=> cpu_load,region=us-west value=100
+=> cpu_load,region=eu-east value=100
 
 mem.cached.localhost 256
 => mem_cached,host=localhost value=256
@@ -230,8 +306,8 @@ templates = [
 would result in the following Graphite -> Telegraf transformation.
 
 ```
-cpu.usage.idle.us-west 100
+cpu.usage.idle.eu-east 100
-=> cpu_usage,region=us-west,datacenter=1a idle=100
+=> cpu_usage,region=eu-east,datacenter=1a idle=100
 ```
 
 There are many more options available,
@@ -247,7 +323,7 @@ There are many more options available,
   ## measurement name suffix (for separating different commands)
   name_suffix = "_mycollector"
 
-  ## Data format to consume. This can be "json", "influx" or "graphite" (line-protocol)
+  ## Data format to consume.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
@@ -262,13 +338,37 @@ There are many more options available,
   ## similar to the line protocol format. There can be only one default template.
   ## Templates support below format:
   ## 1. filter + template
-  ## 2. filter + template + extra tag
+  ## 2. filter + template + extra tag(s)
   ## 3. filter + template with field key
   ## 4. default template
   templates = [
     "*.app env.service.resource.measurement",
-    "stats.* .host.measurement* region=us-west,agent=sensu",
+    "stats.* .host.measurement* region=eu-east,agent=sensu",
     "stats2.* .host.measurement.field",
     "measurement*"
   ]
 ```
+
+# Nagios:
+
+There are no additional configuration options for Nagios line-protocol. The
+metrics are parsed directly into Telegraf metrics.
+
+Note: Nagios Input Data Formats is only supported in `exec` input plugin.
+
+#### Nagios Configuration:
+
+```toml
+[[inputs.exec]]
+  ## Commands array
+  commands = ["/usr/lib/nagios/plugins/check_load", "-w 5,6,7 -c 7,8,9"]
+
+  ## measurement name suffix (for separating different commands)
+  name_suffix = "_mycollector"
+
+  ## Data format to consume.
+  ## Each data format has it's own unique set of configuration options, read
+  ## more about them here:
+  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
+  data_format = "nagios"
+```

docs/DATA_FORMATS_OUTPUT.md

@@ -1,5 +1,11 @@
 # Telegraf Output Data Formats
 
+Telegraf is able to serialize metrics into the following output data formats:
+
+1. [InfluxDB Line Protocol](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#influx)
+1. [JSON](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#json)
+1. [Graphite](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#graphite)
+
 Telegraf metrics, like InfluxDB
 [points](https://docs.influxdata.com/influxdb/v0.10/write_protocols/line/),
 are a combination of four basic parts:
@@ -29,7 +35,7 @@ config option, for example, in the `file` output plugin:
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout"]
 
-  ## Data format to output. This can be "influx" or "graphite"
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
@@ -41,34 +47,46 @@ config option, for example, in the `file` output plugin:
 Each data_format has an additional set of configuration options available, which
 I'll go over below.
 
-## Influx:
+# Influx:
 
 There are no additional configuration options for InfluxDB line-protocol. The
 metrics are serialized directly into InfluxDB line-protocol.
 
-#### Influx Configuration:
+### Influx Configuration:
 
 ```toml
 [[outputs.file]]
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/tmp/metrics.out"]
 
-  ## Data format to output. This can be "influx" or "graphite"
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
   data_format = "influx"
 ```
 
-## Graphite:
+# Graphite:
 
-The Graphite data format translates Telegraf metrics into _dot_ buckets.
+The Graphite data format translates Telegraf metrics into _dot_ buckets. A
-The format is:
+template can be specified for the output of Telegraf metrics into Graphite
+buckets. The default template is:
 
 ```
-[prefix].[host tag].[all tags (alphabetical)].[measurement name].[field name] value timestamp
+template = "host.tags.measurement.field"
 ```
 
+In the above template, we have four parts:
+
+1. _host_ is a tag key. This can be any tag key that is in the Telegraf
+metric(s). If the key doesn't exist, it will be ignored. If it does exist, the
+tag value will be filled in.
+1. _tags_ is a special keyword that outputs all remaining tag values, separated
+by dots and in alphabetical order (by tag key). These will be filled after all
+tag keys are filled.
+1. _measurement_ is a special keyword that outputs the measurement name.
+1. _field_ is a special keyword that outputs the field name.
+
 Which means the following influx metric -> graphite conversion would happen:
 
 ```
@@ -78,20 +96,55 @@ tars.cpu-total.us-east-1.cpu.usage_user 0.89 1455320690
 tars.cpu-total.us-east-1.cpu.usage_idle 98.09 1455320690
 ```
 
-`prefix` is a configuration option when using the graphite output data format.
+### Graphite Configuration:
-
-#### Graphite Configuration:
 
 ```toml
 [[outputs.file]]
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/tmp/metrics.out"]
 
-  ## Data format to output. This can be "influx" or "graphite"
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
-  data_format = "influx"
+  data_format = "graphite"
 
+  # prefix each graphite bucket
   prefix = "telegraf"
+  # graphite template
+  template = "host.tags.measurement.field"
+```
+
+# JSON:
+
+The JSON data format serialized Telegraf metrics in json format. The format is:
+
+```json
+{
+   "fields":{
+      "field_1":30,
+      "field_2":4,
+      "field_N":59,
+      "n_images":660
+   },
+   "name":"docker",
+   "tags":{
+      "host":"raynor"
+   },
+   "timestamp":1458229140
+}
+```
+
+### JSON Configuration:
+
+```toml
+[[outputs.file]]
+  ## Files to write to, "stdout" is a specially handled file.
+  files = ["stdout", "/tmp/metrics.out"]
+
+  ## Data format to output.
+  ## Each data format has it's own unique set of configuration options, read
+  ## more about them here:
+  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
+  data_format = "json"
 ```

docs/LICENSE_OF_DEPENDENCIES.md

@@ -28,6 +28,5 @@
 - github.com/wvanbergen/kazoo-go [MIT LICENSE](https://github.com/wvanbergen/kazoo-go/blob/master/MIT-LICENSE)
 - gopkg.in/dancannon/gorethink.v1 [APACHE LICENSE](https://github.com/dancannon/gorethink/blob/v1.1.2/LICENSE)
 - gopkg.in/mgo.v2 [BSD LICENSE](https://github.com/go-mgo/mgo/blob/v2/LICENSE)
-- golang.org/x/crypto/* [BSD LICENSE](https://github.com/golang/crypto/blob/master/LICENSE)
+- golang.org/x/crypto/ [BSD LICENSE](https://github.com/golang/crypto/blob/master/LICENSE)
-- internal Glob function [MIT LICENSE](https://github.com/ryanuber/go-glob/blob/master/LICENSE)
 

docs/WINDOWS_SERVICE.md

@@ -0,0 +1,36 @@
+# Running Telegraf as a Windows Service
+
+If you have tried to install Go binaries as Windows Services with the **sc.exe**
+tool you may have seen that the service errors and stops running after a while.
+
+**NSSM** (the Non-Sucking Service Manager) is a tool that helps you in a 
+[number of scenarios](http://nssm.cc/scenarios) including running Go binaries
+that were not specifically designed to run only in Windows platforms.
+
+## NSSM Installation via Chocolatey
+
+You can install [Chocolatey](https://chocolatey.org/) and [NSSM](http://nssm.cc/) 
+with these commands
+
+```powershell
+iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
+choco install -y nssm
+```
+
+## Installing Telegraf as a Windows Service with NSSM
+
+You can download the latest Telegraf Windows binaries (still Experimental at 
+the moment) from [the Telegraf Github repo](https://github.com/influxdata/telegraf).
+
+Then you can create a C:\telegraf folder, unzip the binary there and modify the 
+**telegraf.conf** sample to allocate the metrics you want to send to **InfluxDB**.
+
+Once you have NSSM installed in your system, the process is quite straightforward.
+You only need to type this command in your Windows shell
+
+```powershell
+nssm install Telegraf c:\telegraf\telegraf.exe -config c:\telegraf\telegraf.config
+```
+
+And now your service will be installed in Windows and you will be able to start and
+stop it gracefully

filter/filter.go

@@ -0,0 +1,79 @@
+package filter
+
+import (
+	"strings"
+
+	"github.com/gobwas/glob"
+)
+
+type Filter interface {
+	Match(string) bool
+}
+
+// CompileFilter takes a list of string filters and returns a Filter interface
+// for matching a given string against the filter list. The filter list
+// supports glob matching too, ie:
+//
+//   f, _ := CompileFilter([]string{"cpu", "mem", "net*"})
+//   f.Match("cpu")     // true
+//   f.Match("network") // true
+//   f.Match("memory")  // false
+//
+func CompileFilter(filters []string) (Filter, error) {
+	// return if there is nothing to compile
+	if len(filters) == 0 {
+		return nil, nil
+	}
+
+	// check if we can compile a non-glob filter
+	noGlob := true
+	for _, filter := range filters {
+		if hasMeta(filter) {
+			noGlob = false
+			break
+		}
+	}
+
+	switch {
+	case noGlob:
+		// return non-globbing filter if not needed.
+		return compileFilterNoGlob(filters), nil
+	case len(filters) == 1:
+		return glob.Compile(filters[0])
+	default:
+		return glob.Compile("{" + strings.Join(filters, ",") + "}")
+	}
+}
+
+// hasMeta reports whether path contains any magic glob characters.
+func hasMeta(s string) bool {
+	return strings.IndexAny(s, "*?[") >= 0
+}
+
+type filter struct {
+	m map[string]struct{}
+}
+
+func (f *filter) Match(s string) bool {
+	_, ok := f.m[s]
+	return ok
+}
+
+type filtersingle struct {
+	s string
+}
+
+func (f *filtersingle) Match(s string) bool {
+	return f.s == s
+}
+
+func compileFilterNoGlob(filters []string) Filter {
+	if len(filters) == 1 {
+		return &filtersingle{s: filters[0]}
+	}
+	out := filter{m: make(map[string]struct{})}
+	for _, filter := range filters {
+		out.m[filter] = struct{}{}
+	}
+	return &out
+}

filter/filter_test.go

@@ -0,0 +1,96 @@
+package filter
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestCompileFilter(t *testing.T) {
+	f, err := CompileFilter([]string{})
+	assert.NoError(t, err)
+	assert.Nil(t, f)
+
+	f, err = CompileFilter([]string{"cpu"})
+	assert.NoError(t, err)
+	assert.True(t, f.Match("cpu"))
+	assert.False(t, f.Match("cpu0"))
+	assert.False(t, f.Match("mem"))
+
+	f, err = CompileFilter([]string{"cpu*"})
+	assert.NoError(t, err)
+	assert.True(t, f.Match("cpu"))
+	assert.True(t, f.Match("cpu0"))
+	assert.False(t, f.Match("mem"))
+
+	f, err = CompileFilter([]string{"cpu", "mem"})
+	assert.NoError(t, err)
+	assert.True(t, f.Match("cpu"))
+	assert.False(t, f.Match("cpu0"))
+	assert.True(t, f.Match("mem"))
+
+	f, err = CompileFilter([]string{"cpu", "mem", "net*"})
+	assert.NoError(t, err)
+	assert.True(t, f.Match("cpu"))
+	assert.False(t, f.Match("cpu0"))
+	assert.True(t, f.Match("mem"))
+	assert.True(t, f.Match("network"))
+}
+
+var benchbool bool
+
+func BenchmarkFilterSingleNoGlobFalse(b *testing.B) {
+	f, _ := CompileFilter([]string{"cpu"})
+	var tmp bool
+	for n := 0; n < b.N; n++ {
+		tmp = f.Match("network")
+	}
+	benchbool = tmp
+}
+
+func BenchmarkFilterSingleNoGlobTrue(b *testing.B) {
+	f, _ := CompileFilter([]string{"cpu"})
+	var tmp bool
+	for n := 0; n < b.N; n++ {
+		tmp = f.Match("cpu")
+	}
+	benchbool = tmp
+}
+
+func BenchmarkFilter(b *testing.B) {
+	f, _ := CompileFilter([]string{"cpu", "mem", "net*"})
+	var tmp bool
+	for n := 0; n < b.N; n++ {
+		tmp = f.Match("network")
+	}
+	benchbool = tmp
+}
+
+func BenchmarkFilterNoGlob(b *testing.B) {
+	f, _ := CompileFilter([]string{"cpu", "mem", "net"})
+	var tmp bool
+	for n := 0; n < b.N; n++ {
+		tmp = f.Match("net")
+	}
+	benchbool = tmp
+}
+
+func BenchmarkFilter2(b *testing.B) {
+	f, _ := CompileFilter([]string{"aa", "bb", "c", "ad", "ar", "at", "aq",
+		"aw", "az", "axxx", "ab", "cpu", "mem", "net*"})
+	var tmp bool
+	for n := 0; n < b.N; n++ {
+		tmp = f.Match("network")
+	}
+	benchbool = tmp
+}
+
+func BenchmarkFilter2NoGlob(b *testing.B) {
+	f, _ := CompileFilter([]string{"aa", "bb", "c", "ad", "ar", "at", "aq",
+		"aw", "az", "axxx", "ab", "cpu", "mem", "net"})
+	var tmp bool
+	for n := 0; n < b.N; n++ {
+		tmp = f.Match("net")
+	}
+	benchbool = tmp
+}

internal/buffer/buffer.go

@@ -0,0 +1,77 @@
+package buffer
+
+import (
+	"github.com/influxdata/telegraf"
+)
+
+// Buffer is an object for storing metrics in a circular buffer.
+type Buffer struct {
+	buf chan telegraf.Metric
+	// total dropped metrics
+	drops int
+	// total metrics added
+	total int
+}
+
+// NewBuffer returns a Buffer
+//   size is the maximum number of metrics that Buffer will cache. If Add is
+//   called when the buffer is full, then the oldest metric(s) will be dropped.
+func NewBuffer(size int) *Buffer {
+	return &Buffer{
+		buf: make(chan telegraf.Metric, size),
+	}
+}
+
+// IsEmpty returns true if Buffer is empty.
+func (b *Buffer) IsEmpty() bool {
+	return len(b.buf) == 0
+}
+
+// Len returns the current length of the buffer.
+func (b *Buffer) Len() int {
+	return len(b.buf)
+}
+
+// Drops returns the total number of dropped metrics that have occured in this
+// buffer since instantiation.
+func (b *Buffer) Drops() int {
+	return b.drops
+}
+
+// Total returns the total number of metrics that have been added to this buffer.
+func (b *Buffer) Total() int {
+	return b.total
+}
+
+// Add adds metrics to the buffer.
+func (b *Buffer) Add(metrics ...telegraf.Metric) {
+	for i, _ := range metrics {
+		b.total++
+		select {
+		case b.buf <- metrics[i]:
+		default:
+			b.drops++
+			<-b.buf
+			b.buf <- metrics[i]
+		}
+	}
+}
+
+// Batch returns a batch of metrics of size batchSize.
+// the batch will be of maximum length batchSize. It can be less than batchSize,
+// if the length of Buffer is less than batchSize.
+func (b *Buffer) Batch(batchSize int) []telegraf.Metric {
+	n := min(len(b.buf), batchSize)
+	out := make([]telegraf.Metric, n)
+	for i := 0; i < n; i++ {
+		out[i] = <-b.buf
+	}
+	return out
+}
+
+func min(a, b int) int {
+	if b < a {
+		return b
+	}
+	return a
+}

internal/buffer/buffer_test.go

@@ -0,0 +1,94 @@
+package buffer
+
+import (
+	"testing"
+
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/testutil"
+
+	"github.com/stretchr/testify/assert"
+)
+
+var metricList = []telegraf.Metric{
+	testutil.TestMetric(2, "mymetric1"),
+	testutil.TestMetric(1, "mymetric2"),
+	testutil.TestMetric(11, "mymetric3"),
+	testutil.TestMetric(15, "mymetric4"),
+	testutil.TestMetric(8, "mymetric5"),
+}
+
+func BenchmarkAddMetrics(b *testing.B) {
+	buf := NewBuffer(10000)
+	m := testutil.TestMetric(1, "mymetric")
+	for n := 0; n < b.N; n++ {
+		buf.Add(m)
+	}
+}
+
+func TestNewBufferBasicFuncs(t *testing.T) {
+	b := NewBuffer(10)
+
+	assert.True(t, b.IsEmpty())
+	assert.Zero(t, b.Len())
+	assert.Zero(t, b.Drops())
+	assert.Zero(t, b.Total())
+
+	m := testutil.TestMetric(1, "mymetric")
+	b.Add(m)
+	assert.False(t, b.IsEmpty())
+	assert.Equal(t, b.Len(), 1)
+	assert.Equal(t, b.Drops(), 0)
+	assert.Equal(t, b.Total(), 1)
+
+	b.Add(metricList...)
+	assert.False(t, b.IsEmpty())
+	assert.Equal(t, b.Len(), 6)
+	assert.Equal(t, b.Drops(), 0)
+	assert.Equal(t, b.Total(), 6)
+}
+
+func TestDroppingMetrics(t *testing.T) {
+	b := NewBuffer(10)
+
+	// Add up to the size of the buffer
+	b.Add(metricList...)
+	b.Add(metricList...)
+	assert.False(t, b.IsEmpty())
+	assert.Equal(t, b.Len(), 10)
+	assert.Equal(t, b.Drops(), 0)
+	assert.Equal(t, b.Total(), 10)
+
+	// Add 5 more and verify they were dropped
+	b.Add(metricList...)
+	assert.False(t, b.IsEmpty())
+	assert.Equal(t, b.Len(), 10)
+	assert.Equal(t, b.Drops(), 5)
+	assert.Equal(t, b.Total(), 15)
+}
+
+func TestGettingBatches(t *testing.T) {
+	b := NewBuffer(20)
+
+	// Verify that the buffer returned is smaller than requested when there are
+	// not as many items as requested.
+	b.Add(metricList...)
+	batch := b.Batch(10)
+	assert.Len(t, batch, 5)
+
+	// Verify that the buffer is now empty
+	assert.True(t, b.IsEmpty())
+	assert.Zero(t, b.Len())
+	assert.Zero(t, b.Drops())
+	assert.Equal(t, b.Total(), 5)
+
+	// Verify that the buffer returned is not more than the size requested
+	b.Add(metricList...)
+	batch = b.Batch(3)
+	assert.Len(t, batch, 3)
+
+	// Verify that buffer is not empty
+	assert.False(t, b.IsEmpty())
+	assert.Equal(t, b.Len(), 2)
+	assert.Equal(t, b.Drops(), 0)
+	assert.Equal(t, b.Total(), 10)
+}

internal/config/aws/credentials.go

@@ -0,0 +1,49 @@
+package aws
+
+import (
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/client"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/credentials/stscreds"
+	"github.com/aws/aws-sdk-go/aws/session"
+)
+
+type CredentialConfig struct {
+	Region    string
+	AccessKey string
+	SecretKey string
+	RoleARN   string
+	Profile   string
+	Filename  string
+	Token     string
+}
+
+func (c *CredentialConfig) Credentials() client.ConfigProvider {
+	if c.RoleARN != "" {
+		return c.assumeCredentials()
+	} else {
+		return c.rootCredentials()
+	}
+}
+
+func (c *CredentialConfig) rootCredentials() client.ConfigProvider {
+	config := &aws.Config{
+		Region: aws.String(c.Region),
+	}
+	if c.AccessKey != "" || c.SecretKey != "" {
+		config.Credentials = credentials.NewStaticCredentials(c.AccessKey, c.SecretKey, c.Token)
+	} else if c.Profile != "" || c.Filename != "" {
+		config.Credentials = credentials.NewSharedCredentials(c.Filename, c.Profile)
+	}
+
+	return session.New(config)
+}
+
+func (c *CredentialConfig) assumeCredentials() client.ConfigProvider {
+	rootCredentials := c.rootCredentials()
+	config := &aws.Config{
+		Region: aws.String(c.Region),
+	}
+	config.Credentials = stscreds.NewCredentials(rootCredentials, c.RoleARN)
+	return session.New(config)
+}

internal/config/config.go

@@ -1,11 +1,14 @@
 package config
 
 import (
+	"bytes"
 	"errors"
 	"fmt"
 	"io/ioutil"
 	"log"
+	"os"
 	"path/filepath"
+	"regexp"
 	"sort"
 	"strings"
 	"time"
@@ -19,9 +22,22 @@ import (
 	"github.com/influxdata/telegraf/plugins/serializers"
 
 	"github.com/influxdata/config"
+	"github.com/influxdata/toml"
 	"github.com/influxdata/toml/ast"
 )
 
+var (
+	// Default input plugins
+	inputDefaults = []string{"cpu", "mem", "swap", "system", "kernel",
+		"processes", "disk", "diskio"}
+
+	// Default output plugins
+	outputDefaults = []string{"influxdb"}
+
+	// envVarRe is a regex to find environment variables in the config file
+	envVarRe = regexp.MustCompile(`\$\w+`)
+)
+
 // Config specifies the URL/user/password for the database that telegraf
 // will be logging to, as well as all the plugins that the user has
 // specified
@@ -42,7 +58,6 @@ func NewConfig() *Config {
 			Interval:      internal.Duration{Duration: 10 * time.Second},
 			RoundInterval: true,
 			FlushInterval: internal.Duration{Duration: 10 * time.Second},
-			FlushJitter:   internal.Duration{Duration: 5 * time.Second},
 		},
 
 		Tags:          make(map[string]string),
@@ -62,6 +77,14 @@ type AgentConfig struct {
 	//     ie, if Interval=10s then always collect on :00, :10, :20, etc.
 	RoundInterval bool
 
+	// By default, precision will be set to the same timestamp order as the
+	// collection interval, with the maximum being 1s.
+	//   ie, when interval = "10s", precision will be "1s"
+	//       when interval = "250ms", precision will be "1ms"
+	// Precision will NOT be used for service inputs. It is up to each individual
+	// service input to set the timestamp at the appropriate precision.
+	Precision internal.Duration
+
 	// CollectionJitter is used to jitter the collection by a random amount.
 	// Each plugin will sleep for a random time within jitter before collecting.
 	// This can be used to avoid many plugins querying things like sysfs at the
@@ -77,9 +100,15 @@ type AgentConfig struct {
 	// ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
 	FlushJitter internal.Duration
 
+	// MetricBatchSize is the maximum number of metrics that is wrote to an
+	// output plugin in one call.
+	MetricBatchSize int
+
 	// MetricBufferLimit is the max number of metrics that each output plugin
 	// will cache. The buffer is cleared when a successful write occurs. When
-	// full, the oldest metrics will be overwritten.
+	// full, the oldest metrics will be overwritten. This number should be a
+	// multiple of MetricBatchSize. Due to current implementation, this could
+	// not be less than 2 times MetricBatchSize.
 	MetricBufferLimit int
 
 	// FlushBufferWhenFull tells Telegraf to flush the metric buffer whenever
@@ -87,18 +116,18 @@ type AgentConfig struct {
 	// does _not_ deactivate FlushInterval.
 	FlushBufferWhenFull bool
 
-	// TODO(cam): Remove UTC and Precision parameters, they are no longer
+	// TODO(cam): Remove UTC and parameter, they are no longer
 	// valid for the agent config. Leaving them here for now for backwards-
 	// compatability
-	UTC       bool `toml:"utc"`
+	UTC bool `toml:"utc"`
-	Precision string
 
 	// Debug is the option for running in debug mode
 	Debug bool
 
 	// Quiet is the option for running in quiet mode
-	Quiet    bool
+	Quiet        bool
-	Hostname string
+	Hostname     string
+	OmitHostname bool
 }
 
 // Inputs returns a list of strings of the configured inputs.
@@ -134,20 +163,28 @@ func (c *Config) ListTags() string {
 }
 
 var header = `# Telegraf Configuration
-
+#
 # Telegraf is entirely plugin driven. All metrics are gathered from the
 # declared inputs, and sent to the declared outputs.
-
+#
 # Plugins must be declared in here to be active.
 # To deactivate a plugin, comment out the name and any variables.
-
+#
 # Use 'telegraf -config telegraf.conf -test' to see what metrics a config
 # file would generate.
+#
+# Environment variables can be used anywhere in this config file, simply prepend
+# them with $. For strings the variable must be within quotes (ie, "$STR_VAR"),
+# for numbers and booleans they should be plain (ie, $INT_VAR, $BOOL_VAR)
+
 
 # Global tags can be specified here in key="value" format.
 [global_tags]
   # dc = "us-east-1" # will tag all metrics with dc=us-east-1
   # rack = "1a"
+  ## Environment variables can be used as tags, and throughout the config file
+  # user = "$USER"
+
 
 # Configuration for telegraf agent
 [agent]
@@ -157,11 +194,13 @@ var header = `# Telegraf Configuration
   ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
   round_interval = true
 
-  ## Telegraf will cache metric_buffer_limit metrics for each output, and will
+  ## Telegraf will send metrics to outputs in batches of at
-  ## flush this buffer on a successful write.
+  ## most metric_batch_size metrics.
-  metric_buffer_limit = 1000
+  metric_batch_size = 1000
-  ## Flush the buffer whenever full, regardless of flush_interval.
+  ## For failed writes, telegraf will cache metric_buffer_limit metrics for each
-  flush_buffer_when_full = true
+  ## output, and will flush this buffer on a successful write. Oldest metrics
+  ## are dropped first when this buffer fills.
+  metric_buffer_limit = 10000
 
   ## Collection jitter is used to jitter the collection by a random amount.
   ## Each plugin will sleep for a random time within jitter before collecting.
@@ -177,40 +216,122 @@ var header = `# Telegraf Configuration
   ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
   flush_jitter = "0s"
 
+  ## By default, precision will be set to the same timestamp order as the
+  ## collection interval, with the maximum being 1s.
+  ## Precision will NOT be used for service inputs, such as logparser and statsd.
+  ## Valid values are "Nns", "Nus" (or "Nµs"), "Nms", "Ns".
+  precision = ""
   ## Run telegraf in debug mode
   debug = false
   ## Run telegraf in quiet mode
   quiet = false
   ## Override default hostname, if empty use os.Hostname()
   hostname = ""
+  ## If set to true, do no set the "host" tag in the telegraf agent.
+  omit_hostname = false
 
 
-#
+###############################################################################
-# OUTPUTS:
+#                            OUTPUT PLUGINS                                   #
-#
+###############################################################################
-
 `
 
-var pluginHeader = `
+var inputHeader = `
-#
+
-# INPUTS:
+###############################################################################
-#
+#                            INPUT PLUGINS                                    #
+###############################################################################
 `
 
 var serviceInputHeader = `
-#
+
-# SERVICE INPUTS:
+###############################################################################
-#
+#                            SERVICE INPUT PLUGINS                            #
+###############################################################################
 `
 
 // PrintSampleConfig prints the sample config
-func PrintSampleConfig(pluginFilters []string, outputFilters []string) {
+func PrintSampleConfig(inputFilters []string, outputFilters []string) {
 	fmt.Printf(header)
 
+	if len(outputFilters) != 0 {
+		printFilteredOutputs(outputFilters, false)
+	} else {
+		printFilteredOutputs(outputDefaults, false)
+		// Print non-default outputs, commented
+		var pnames []string
+		for pname := range outputs.Outputs {
+			if !sliceContains(pname, outputDefaults) {
+				pnames = append(pnames, pname)
+			}
+		}
+		sort.Strings(pnames)
+		printFilteredOutputs(pnames, true)
+	}
+
+	fmt.Printf(inputHeader)
+	if len(inputFilters) != 0 {
+		printFilteredInputs(inputFilters, false)
+	} else {
+		printFilteredInputs(inputDefaults, false)
+		// Print non-default inputs, commented
+		var pnames []string
+		for pname := range inputs.Inputs {
+			if !sliceContains(pname, inputDefaults) {
+				pnames = append(pnames, pname)
+			}
+		}
+		sort.Strings(pnames)
+		printFilteredInputs(pnames, true)
+	}
+}
+
+func printFilteredInputs(inputFilters []string, commented bool) {
+	// Filter inputs
+	var pnames []string
+	for pname := range inputs.Inputs {
+		if sliceContains(pname, inputFilters) {
+			pnames = append(pnames, pname)
+		}
+	}
+	sort.Strings(pnames)
+
+	// cache service inputs to print them at the end
+	servInputs := make(map[string]telegraf.ServiceInput)
+	// for alphabetical looping:
+	servInputNames := []string{}
+
+	// Print Inputs
+	for _, pname := range pnames {
+		creator := inputs.Inputs[pname]
+		input := creator()
+
+		switch p := input.(type) {
+		case telegraf.ServiceInput:
+			servInputs[pname] = p
+			servInputNames = append(servInputNames, pname)
+			continue
+		}
+
+		printConfig(pname, input, "inputs", commented)
+	}
+
+	// Print Service Inputs
+	if len(servInputs) == 0 {
+		return
+	}
+	sort.Strings(servInputNames)
+	fmt.Printf(serviceInputHeader)
+	for _, name := range servInputNames {
+		printConfig(name, servInputs[name], "inputs", commented)
+	}
+}
+
+func printFilteredOutputs(outputFilters []string, commented bool) {
 	// Filter outputs
 	var onames []string
 	for oname := range outputs.Outputs {
-		if len(outputFilters) == 0 || sliceContains(oname, outputFilters) {
+		if sliceContains(oname, outputFilters) {
 			onames = append(onames, oname)
 		}
 	}
@@ -220,38 +341,7 @@ func PrintSampleConfig(pluginFilters []string, outputFilters []string) {
 	for _, oname := range onames {
 		creator := outputs.Outputs[oname]
 		output := creator()
-		printConfig(oname, output, "outputs")
+		printConfig(oname, output, "outputs", commented)
-	}
-
-	// Filter inputs
-	var pnames []string
-	for pname := range inputs.Inputs {
-		if len(pluginFilters) == 0 || sliceContains(pname, pluginFilters) {
-			pnames = append(pnames, pname)
-		}
-	}
-	sort.Strings(pnames)
-
-	// Print Inputs
-	fmt.Printf(pluginHeader)
-	servInputs := make(map[string]telegraf.ServiceInput)
-	for _, pname := range pnames {
-		creator := inputs.Inputs[pname]
-		input := creator()
-
-		switch p := input.(type) {
-		case telegraf.ServiceInput:
-			servInputs[pname] = p
-			continue
-		}
-
-		printConfig(pname, input, "inputs")
-	}
-
-	// Print Service Inputs
-	fmt.Printf(serviceInputHeader)
-	for name, input := range servInputs {
-		printConfig(name, input, "inputs")
 	}
 }
 
@@ -260,13 +350,26 @@ type printer interface {
 	SampleConfig() string
 }
 
-func printConfig(name string, p printer, op string) {
+func printConfig(name string, p printer, op string, commented bool) {
-	fmt.Printf("\n# %s\n[[%s.%s]]", p.Description(), op, name)
+	comment := ""
+	if commented {
+		comment = "# "
+	}
+	fmt.Printf("\n%s# %s\n%s[[%s.%s]]", comment, p.Description(), comment,
+		op, name)
+
 	config := p.SampleConfig()
 	if config == "" {
-		fmt.Printf("\n  # no configuration\n")
+		fmt.Printf("\n%s  # no configuration\n\n", comment)
 	} else {
-		fmt.Printf(config)
+		lines := strings.Split(config, "\n")
+		for i, line := range lines {
+			if i == 0 || i == len(lines)-1 {
+				fmt.Print("\n")
+				continue
+			}
+			fmt.Print(strings.TrimRight(comment+line, " ") + "\n")
+		}
 	}
 }
 
@@ -282,7 +385,7 @@ func sliceContains(name string, list []string) bool {
 // PrintInputConfig prints the config usage of a single input.
 func PrintInputConfig(name string) error {
 	if creator, ok := inputs.Inputs[name]; ok {
-		printConfig(name, creator(), "inputs")
+		printConfig(name, creator(), "inputs", false)
 	} else {
 		return errors.New(fmt.Sprintf("Input %s not found", name))
 	}
@@ -292,7 +395,7 @@ func PrintInputConfig(name string) error {
 // PrintOutputConfig prints the config usage of a single output.
 func PrintOutputConfig(name string) error {
 	if creator, ok := outputs.Outputs[name]; ok {
-		printConfig(name, creator(), "outputs")
+		printConfig(name, creator(), "outputs", false)
 	} else {
 		return errors.New(fmt.Sprintf("Output %s not found", name))
 	}
@@ -320,46 +423,91 @@ func (c *Config) LoadDirectory(path string) error {
 	return nil
 }
 
-// LoadConfig loads the given config file and applies it to c
+// Try to find a default config file at these locations (in order):
-func (c *Config) LoadConfig(path string) error {
+//   1. $TELEGRAF_CONFIG_PATH
-	tbl, err := config.ParseFile(path)
+//   2. $HOME/.telegraf/telegraf.conf
-	if err != nil {
+//   3. /etc/telegraf/telegraf.conf
-		return err
+//
+func getDefaultConfigPath() (string, error) {
+	envfile := os.Getenv("TELEGRAF_CONFIG_PATH")
+	homefile := os.ExpandEnv("${HOME}/.telegraf/telegraf.conf")
+	etcfile := "/etc/telegraf/telegraf.conf"
+	for _, path := range []string{envfile, homefile, etcfile} {
+		if _, err := os.Stat(path); err == nil {
+			log.Printf("Using config file: %s", path)
+			return path, nil
+		}
 	}
 
+	// if we got here, we didn't find a file in a default location
+	return "", fmt.Errorf("No config file specified, and could not find one"+
+		" in $TELEGRAF_CONFIG_PATH, %s, or %s", homefile, etcfile)
+}
+
+// LoadConfig loads the given config file and applies it to c
+func (c *Config) LoadConfig(path string) error {
+	var err error
+	if path == "" {
+		if path, err = getDefaultConfigPath(); err != nil {
+			return err
+		}
+	}
+	tbl, err := parseFile(path)
+	if err != nil {
+		return fmt.Errorf("Error parsing %s, %s", path, err)
+	}
+
+	// Parse tags tables first:
+	for _, tableName := range []string{"tags", "global_tags"} {
+		if val, ok := tbl.Fields[tableName]; ok {
+			subTable, ok := val.(*ast.Table)
+			if !ok {
+				return fmt.Errorf("%s: invalid configuration", path)
+			}
+			if err = config.UnmarshalTable(subTable, c.Tags); err != nil {
+				log.Printf("Could not parse [global_tags] config\n")
+				return fmt.Errorf("Error parsing %s, %s", path, err)
+			}
+		}
+	}
+
+	// Parse agent table:
+	if val, ok := tbl.Fields["agent"]; ok {
+		subTable, ok := val.(*ast.Table)
+		if !ok {
+			return fmt.Errorf("%s: invalid configuration", path)
+		}
+		if err = config.UnmarshalTable(subTable, c.Agent); err != nil {
+			log.Printf("Could not parse [agent] config\n")
+			return fmt.Errorf("Error parsing %s, %s", path, err)
+		}
+	}
+
+	// Parse all the rest of the plugins:
 	for name, val := range tbl.Fields {
 		subTable, ok := val.(*ast.Table)
 		if !ok {
-			return errors.New("invalid configuration")
+			return fmt.Errorf("%s: invalid configuration", path)
 		}
 
 		switch name {
-		case "agent":
+		case "agent", "global_tags", "tags":
-			if err = config.UnmarshalTable(subTable, c.Agent); err != nil {
-				log.Printf("Could not parse [agent] config\n")
-				return err
-			}
-		case "global_tags", "tags":
-			if err = config.UnmarshalTable(subTable, c.Tags); err != nil {
-				log.Printf("Could not parse [global_tags] config\n")
-				return err
-			}
 		case "outputs":
 			for pluginName, pluginVal := range subTable.Fields {
 				switch pluginSubTable := pluginVal.(type) {
 				case *ast.Table:
 					if err = c.addOutput(pluginName, pluginSubTable); err != nil {
-						return err
+						return fmt.Errorf("Error parsing %s, %s", path, err)
 					}
 				case []*ast.Table:
 					for _, t := range pluginSubTable {
 						if err = c.addOutput(pluginName, t); err != nil {
-							return err
+							return fmt.Errorf("Error parsing %s, %s", path, err)
 						}
 					}
 				default:
-					return fmt.Errorf("Unsupported config format: %s",
+					return fmt.Errorf("Unsupported config format: %s, file %s",
-						pluginName)
+						pluginName, path)
 				}
 			}
 		case "inputs", "plugins":
@@ -367,30 +515,59 @@ func (c *Config) LoadConfig(path string) error {
 				switch pluginSubTable := pluginVal.(type) {
 				case *ast.Table:
 					if err = c.addInput(pluginName, pluginSubTable); err != nil {
-						return err
+						return fmt.Errorf("Error parsing %s, %s", path, err)
 					}
 				case []*ast.Table:
 					for _, t := range pluginSubTable {
 						if err = c.addInput(pluginName, t); err != nil {
-							return err
+							return fmt.Errorf("Error parsing %s, %s", path, err)
 						}
 					}
 				default:
-					return fmt.Errorf("Unsupported config format: %s",
+					return fmt.Errorf("Unsupported config format: %s, file %s",
-						pluginName)
+						pluginName, path)
 				}
 			}
 		// Assume it's an input input for legacy config file support if no other
 		// identifiers are present
 		default:
 			if err = c.addInput(name, subTable); err != nil {
-				return err
+				return fmt.Errorf("Error parsing %s, %s", path, err)
 			}
 		}
 	}
 	return nil
 }
 
+// trimBOM trims the Byte-Order-Marks from the beginning of the file.
+// this is for Windows compatability only.
+// see https://github.com/influxdata/telegraf/issues/1378
+func trimBOM(f []byte) []byte {
+	return bytes.TrimPrefix(f, []byte("\xef\xbb\xbf"))
+}
+
+// parseFile loads a TOML configuration from a provided path and
+// returns the AST produced from the TOML parser. When loading the file, it
+// will find environment variables and replace them.
+func parseFile(fpath string) (*ast.Table, error) {
+	contents, err := ioutil.ReadFile(fpath)
+	if err != nil {
+		return nil, err
+	}
+	// ugh windows why
+	contents = trimBOM(contents)
+
+	env_vars := envVarRe.FindAll(contents, -1)
+	for _, env_var := range env_vars {
+		env_val := os.Getenv(strings.TrimPrefix(string(env_var), "$"))
+		if env_val != "" {
+			contents = bytes.Replace(contents, env_var, []byte(env_val), 1)
+		}
+	}
+
+	return toml.Parse(contents)
+}
+
 func (c *Config) addOutput(name string, table *ast.Table) error {
 	if len(c.OutputFilters) > 0 && !sliceContains(name, c.OutputFilters) {
 		return nil
@@ -421,11 +598,8 @@ func (c *Config) addOutput(name string, table *ast.Table) error {
 		return err
 	}
 
-	ro := internal_models.NewRunningOutput(name, output, outputConfig)
+	ro := internal_models.NewRunningOutput(name, output, outputConfig,
-	if c.Agent.MetricBufferLimit > 0 {
+		c.Agent.MetricBatchSize, c.Agent.MetricBufferLimit)
-		ro.MetricBufferLimit = c.Agent.MetricBufferLimit
-	}
-	ro.FlushBufferWhenFull = c.Agent.FlushBufferWhenFull
 	c.Outputs = append(c.Outputs, ro)
 	return nil
 }
@@ -476,9 +650,9 @@ func (c *Config) addInput(name string, table *ast.Table) error {
 
 // buildFilter builds a Filter
 // (tagpass/tagdrop/namepass/namedrop/fieldpass/fielddrop) to
-// be inserted into the internal_models.OutputConfig/internal_models.InputConfig to be used for prefix
+// be inserted into the internal_models.OutputConfig/internal_models.InputConfig
-// filtering on tags and measurements
+// to be used for glob filtering on tags and measurements
-func buildFilter(tbl *ast.Table) internal_models.Filter {
+func buildFilter(tbl *ast.Table) (internal_models.Filter, error) {
 	f := internal_models.Filter{}
 
 	if node, ok := tbl.Fields["namepass"]; ok {
@@ -577,6 +751,33 @@ func buildFilter(tbl *ast.Table) internal_models.Filter {
 		}
 	}
 
+	if node, ok := tbl.Fields["tagexclude"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if ary, ok := kv.Value.(*ast.Array); ok {
+				for _, elem := range ary.Value {
+					if str, ok := elem.(*ast.String); ok {
+						f.TagExclude = append(f.TagExclude, str.Value)
+					}
+				}
+			}
+		}
+	}
+
+	if node, ok := tbl.Fields["taginclude"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if ary, ok := kv.Value.(*ast.Array); ok {
+				for _, elem := range ary.Value {
+					if str, ok := elem.(*ast.String); ok {
+						f.TagInclude = append(f.TagInclude, str.Value)
+					}
+				}
+			}
+		}
+	}
+	if err := f.CompileFilter(); err != nil {
+		return f, err
+	}
+
 	delete(tbl.Fields, "namedrop")
 	delete(tbl.Fields, "namepass")
 	delete(tbl.Fields, "fielddrop")
@@ -585,7 +786,9 @@ func buildFilter(tbl *ast.Table) internal_models.Filter {
 	delete(tbl.Fields, "pass")
 	delete(tbl.Fields, "tagdrop")
 	delete(tbl.Fields, "tagpass")
-	return f
+	delete(tbl.Fields, "tagexclude")
+	delete(tbl.Fields, "taginclude")
+	return f, nil
 }
 
 // buildInput parses input specific items from the ast.Table,
@@ -644,7 +847,11 @@ func buildInput(name string, tbl *ast.Table) (*internal_models.InputConfig, erro
 	delete(tbl.Fields, "name_override")
 	delete(tbl.Fields, "interval")
 	delete(tbl.Fields, "tags")
-	cp.Filter = buildFilter(tbl)
+	var err error
+	cp.Filter, err = buildFilter(tbl)
+	if err != nil {
+		return cp, err
+	}
 	return cp, nil
 }
 
@@ -701,12 +908,21 @@ func buildParser(name string, tbl *ast.Table) (parsers.Parser, error) {
 		}
 	}
 
+	if node, ok := tbl.Fields["data_type"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				c.DataType = str.Value
+			}
+		}
+	}
+
 	c.MetricName = name
 
 	delete(tbl.Fields, "data_format")
 	delete(tbl.Fields, "separator")
 	delete(tbl.Fields, "templates")
 	delete(tbl.Fields, "tag_keys")
+	delete(tbl.Fields, "data_type")
 
 	return parsers.NewParser(c)
 }
@@ -737,18 +953,32 @@ func buildSerializer(name string, tbl *ast.Table) (serializers.Serializer, error
 		}
 	}
 
+	if node, ok := tbl.Fields["template"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				c.Template = str.Value
+			}
+		}
+	}
+
 	delete(tbl.Fields, "data_format")
 	delete(tbl.Fields, "prefix")
+	delete(tbl.Fields, "template")
 	return serializers.NewSerializer(c)
 }
 
-// buildOutput parses output specific items from the ast.Table, builds the filter and returns an
+// buildOutput parses output specific items from the ast.Table,
+// builds the filter and returns an
 // internal_models.OutputConfig to be inserted into internal_models.RunningInput
 // Note: error exists in the return for future calls that might require error
 func buildOutput(name string, tbl *ast.Table) (*internal_models.OutputConfig, error) {
+	filter, err := buildFilter(tbl)
+	if err != nil {
+		return nil, err
+	}
 	oc := &internal_models.OutputConfig{
 		Name:   name,
-		Filter: buildFilter(tbl),
+		Filter: filter,
 	}
 	// Outputs don't support FieldDrop/FieldPass, so set to NameDrop/NamePass
 	if len(oc.Filter.FieldDrop) > 0 {

internal/config/config_test.go

@@ -1,6 +1,7 @@
 package config
 
 import (
+	"os"
 	"testing"
 	"time"
 
@@ -10,9 +11,54 @@ import (
 	"github.com/influxdata/telegraf/plugins/inputs/memcached"
 	"github.com/influxdata/telegraf/plugins/inputs/procstat"
 	"github.com/influxdata/telegraf/plugins/parsers"
+
 	"github.com/stretchr/testify/assert"
 )
 
+func TestConfig_LoadSingleInputWithEnvVars(t *testing.T) {
+	c := NewConfig()
+	err := os.Setenv("MY_TEST_SERVER", "192.168.1.1")
+	assert.NoError(t, err)
+	err = os.Setenv("TEST_INTERVAL", "10s")
+	assert.NoError(t, err)
+	c.LoadConfig("./testdata/single_plugin_env_vars.toml")
+
+	memcached := inputs.Inputs["memcached"]().(*memcached.Memcached)
+	memcached.Servers = []string{"192.168.1.1"}
+
+	filter := internal_models.Filter{
+		NameDrop:  []string{"metricname2"},
+		NamePass:  []string{"metricname1"},
+		FieldDrop: []string{"other", "stuff"},
+		FieldPass: []string{"some", "strings"},
+		TagDrop: []internal_models.TagFilter{
+			internal_models.TagFilter{
+				Name:   "badtag",
+				Filter: []string{"othertag"},
+			},
+		},
+		TagPass: []internal_models.TagFilter{
+			internal_models.TagFilter{
+				Name:   "goodtag",
+				Filter: []string{"mytag"},
+			},
+		},
+		IsActive: true,
+	}
+	assert.NoError(t, filter.CompileFilter())
+	mConfig := &internal_models.InputConfig{
+		Name:     "memcached",
+		Filter:   filter,
+		Interval: 10 * time.Second,
+	}
+	mConfig.Tags = make(map[string]string)
+
+	assert.Equal(t, memcached, c.Inputs[0].Input,
+		"Testdata did not produce a correct memcached struct.")
+	assert.Equal(t, mConfig, c.Inputs[0].Config,
+		"Testdata did not produce correct memcached metadata.")
+}
+
 func TestConfig_LoadSingleInput(t *testing.T) {
 	c := NewConfig()
 	c.LoadConfig("./testdata/single_plugin.toml")
@@ -20,27 +66,29 @@ func TestConfig_LoadSingleInput(t *testing.T) {
 	memcached := inputs.Inputs["memcached"]().(*memcached.Memcached)
 	memcached.Servers = []string{"localhost"}
 
-	mConfig := &internal_models.InputConfig{
+	filter := internal_models.Filter{
-		Name: "memcached",
+		NameDrop:  []string{"metricname2"},
-		Filter: internal_models.Filter{
+		NamePass:  []string{"metricname1"},
-			NameDrop:  []string{"metricname2"},
+		FieldDrop: []string{"other", "stuff"},
-			NamePass:  []string{"metricname1"},
+		FieldPass: []string{"some", "strings"},
-			FieldDrop: []string{"other", "stuff"},
+		TagDrop: []internal_models.TagFilter{
-			FieldPass: []string{"some", "strings"},
+			internal_models.TagFilter{
-			TagDrop: []internal_models.TagFilter{
+				Name:   "badtag",
-				internal_models.TagFilter{
+				Filter: []string{"othertag"},
-					Name:   "badtag",
-					Filter: []string{"othertag"},
-				},
 			},
-			TagPass: []internal_models.TagFilter{
-				internal_models.TagFilter{
-					Name:   "goodtag",
-					Filter: []string{"mytag"},
-				},
-			},
-			IsActive: true,
 		},
+		TagPass: []internal_models.TagFilter{
+			internal_models.TagFilter{
+				Name:   "goodtag",
+				Filter: []string{"mytag"},
+			},
+		},
+		IsActive: true,
+	}
+	assert.NoError(t, filter.CompileFilter())
+	mConfig := &internal_models.InputConfig{
+		Name:     "memcached",
+		Filter:   filter,
 		Interval: 5 * time.Second,
 	}
 	mConfig.Tags = make(map[string]string)
@@ -65,27 +113,29 @@ func TestConfig_LoadDirectory(t *testing.T) {
 	memcached := inputs.Inputs["memcached"]().(*memcached.Memcached)
 	memcached.Servers = []string{"localhost"}
 
-	mConfig := &internal_models.InputConfig{
+	filter := internal_models.Filter{
-		Name: "memcached",
+		NameDrop:  []string{"metricname2"},
-		Filter: internal_models.Filter{
+		NamePass:  []string{"metricname1"},
-			NameDrop:  []string{"metricname2"},
+		FieldDrop: []string{"other", "stuff"},
-			NamePass:  []string{"metricname1"},
+		FieldPass: []string{"some", "strings"},
-			FieldDrop: []string{"other", "stuff"},
+		TagDrop: []internal_models.TagFilter{
-			FieldPass: []string{"some", "strings"},
+			internal_models.TagFilter{
-			TagDrop: []internal_models.TagFilter{
+				Name:   "badtag",
-				internal_models.TagFilter{
+				Filter: []string{"othertag"},
-					Name:   "badtag",
-					Filter: []string{"othertag"},
-				},
 			},
-			TagPass: []internal_models.TagFilter{
-				internal_models.TagFilter{
-					Name:   "goodtag",
-					Filter: []string{"mytag"},
-				},
-			},
-			IsActive: true,
 		},
+		TagPass: []internal_models.TagFilter{
+			internal_models.TagFilter{
+				Name:   "goodtag",
+				Filter: []string{"mytag"},
+			},
+		},
+		IsActive: true,
+	}
+	assert.NoError(t, filter.CompileFilter())
+	mConfig := &internal_models.InputConfig{
+		Name:     "memcached",
+		Filter:   filter,
 		Interval: 5 * time.Second,
 	}
 	mConfig.Tags = make(map[string]string)

internal/config/testdata/single_plugin_env_vars.toml

@@ -0,0 +1,11 @@
+[[inputs.memcached]]
+  servers = ["$MY_TEST_SERVER"]
+  namepass = ["metricname1"]
+  namedrop = ["metricname2"]
+  fieldpass = ["some", "strings"]
+  fielddrop = ["other", "stuff"]
+  interval = "$TEST_INTERVAL"
+  [inputs.memcached.tagpass]
+    goodtag = ["mytag"]
+  [inputs.memcached.tagdrop]
+    badtag = ["othertag"]

internal/errchan/errchan.go

@@ -0,0 +1,37 @@
+package errchan
+
+import (
+	"fmt"
+	"strings"
+)
+
+type ErrChan struct {
+	C chan error
+}
+
+// New returns an error channel of max length 'n'
+// errors can be sent to the ErrChan.C channel, and will be returned when
+// ErrChan.Error() is called.
+func New(n int) *ErrChan {
+	return &ErrChan{
+		C: make(chan error, n),
+	}
+}
+
+// Error closes the ErrChan.C channel and returns an error if there are any
+// non-nil errors, otherwise returns nil.
+func (e *ErrChan) Error() error {
+	close(e.C)
+
+	var out string
+	for err := range e.C {
+		if err != nil {
+			out += "[" + err.Error() + "], "
+		}
+	}
+
+	if out != "" {
+		return fmt.Errorf("Errors encountered: " + strings.TrimRight(out, ", "))
+	}
+	return nil
+}

internal/globpath/globpath.go

@@ -0,0 +1,98 @@
+package globpath
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/gobwas/glob"
+)
+
+var sepStr = fmt.Sprintf("%v", string(os.PathSeparator))
+
+type GlobPath struct {
+	path    string
+	hasMeta bool
+	g       glob.Glob
+	root    string
+}
+
+func Compile(path string) (*GlobPath, error) {
+	out := GlobPath{
+		hasMeta: hasMeta(path),
+		path:    path,
+	}
+
+	// if there are no glob meta characters in the path, don't bother compiling
+	// a glob object or finding the root directory. (see short-circuit in Match)
+	if !out.hasMeta {
+		return &out, nil
+	}
+
+	var err error
+	if out.g, err = glob.Compile(path, os.PathSeparator); err != nil {
+		return nil, err
+	}
+	// Get the root directory for this filepath
+	out.root = findRootDir(path)
+	return &out, nil
+}
+
+func (g *GlobPath) Match() map[string]os.FileInfo {
+	if !g.hasMeta {
+		out := make(map[string]os.FileInfo)
+		info, err := os.Stat(g.path)
+		if !os.IsNotExist(err) {
+			out[g.path] = info
+		}
+		return out
+	}
+	return walkFilePath(g.root, g.g)
+}
+
+// walk the filepath from the given root and return a list of files that match
+// the given glob.
+func walkFilePath(root string, g glob.Glob) map[string]os.FileInfo {
+	matchedFiles := make(map[string]os.FileInfo)
+	walkfn := func(path string, info os.FileInfo, _ error) error {
+		if g.Match(path) {
+			matchedFiles[path] = info
+		}
+		return nil
+	}
+	filepath.Walk(root, walkfn)
+	return matchedFiles
+}
+
+// find the root dir of the given path (could include globs).
+// ie:
+//   /var/log/telegraf.conf -> /var/log
+//   /home/** ->               /home
+//   /home/*/** ->             /home
+//   /lib/share/*/*/**.txt ->  /lib/share
+func findRootDir(path string) string {
+	pathItems := strings.Split(path, sepStr)
+	out := sepStr
+	for i, item := range pathItems {
+		if i == len(pathItems)-1 {
+			break
+		}
+		if item == "" {
+			continue
+		}
+		if hasMeta(item) {
+			break
+		}
+		out += item + sepStr
+	}
+	if out != "/" {
+		out = strings.TrimSuffix(out, "/")
+	}
+	return out
+}
+
+// hasMeta reports whether path contains any magic glob characters.
+func hasMeta(path string) bool {
+	return strings.IndexAny(path, "*?[") >= 0
+}

internal/globpath/globpath_test.go

@@ -0,0 +1,62 @@
+package globpath
+
+import (
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestCompileAndMatch(t *testing.T) {
+	dir := getTestdataDir()
+	// test super asterisk
+	g1, err := Compile(dir + "/**")
+	require.NoError(t, err)
+	// test single asterisk
+	g2, err := Compile(dir + "/*.log")
+	require.NoError(t, err)
+	// test no meta characters (file exists)
+	g3, err := Compile(dir + "/log1.log")
+	require.NoError(t, err)
+	// test file that doesn't exist
+	g4, err := Compile(dir + "/i_dont_exist.log")
+	require.NoError(t, err)
+	// test super asterisk that doesn't exist
+	g5, err := Compile(dir + "/dir_doesnt_exist/**")
+	require.NoError(t, err)
+
+	matches := g1.Match()
+	assert.Len(t, matches, 3)
+	matches = g2.Match()
+	assert.Len(t, matches, 2)
+	matches = g3.Match()
+	assert.Len(t, matches, 1)
+	matches = g4.Match()
+	assert.Len(t, matches, 0)
+	matches = g5.Match()
+	assert.Len(t, matches, 0)
+}
+
+func TestFindRootDir(t *testing.T) {
+	tests := []struct {
+		input  string
+		output string
+	}{
+		{"/var/log/telegraf.conf", "/var/log"},
+		{"/home/**", "/home"},
+		{"/home/*/**", "/home"},
+		{"/lib/share/*/*/**.txt", "/lib/share"},
+	}
+
+	for _, test := range tests {
+		actual := findRootDir(test.input)
+		assert.Equal(t, test.output, actual)
+	}
+}
+
+func getTestdataDir() string {
+	_, filename, _, _ := runtime.Caller(1)
+	return strings.Replace(filename, "globpath_test.go", "testdata", 1)
+}

internal/globpath/testdata/test.conf

@@ -0,0 +1,5 @@
+# this is a fake testing config file
+# for testing the filestat plugin
+
+option1 = "foo"
+option2 = "bar"

internal/internal.go

@@ -2,19 +2,31 @@ package internal
 
 import (
 	"bufio"
+	"bytes"
 	"crypto/rand"
 	"crypto/tls"
 	"crypto/x509"
 	"errors"
 	"fmt"
 	"io/ioutil"
+	"log"
+	"math/big"
 	"os"
+	"os/exec"
+	"strconv"
 	"strings"
 	"time"
+	"unicode"
 )
 
 const alphanum string = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
 
+var (
+	TimeoutErr = errors.New("Command timed out.")
+
+	NotImplementedError = errors.New("not implemented yet")
+)
+
 // Duration just wraps time.Duration
 type Duration struct {
 	Duration time.Duration
@@ -22,18 +34,29 @@ type Duration struct {
 
 // UnmarshalTOML parses the duration from the TOML config file
 func (d *Duration) UnmarshalTOML(b []byte) error {
-	dur, err := time.ParseDuration(string(b[1 : len(b)-1]))
+	var err error
-	if err != nil {
+	// Parse string duration, ie, "1s"
-		return err
+	d.Duration, err = time.ParseDuration(string(b[1 : len(b)-1]))
+	if err == nil {
+		return nil
 	}
 
-	d.Duration = dur
+	// First try parsing as integer seconds
+	sI, err := strconv.ParseInt(string(b), 10, 64)
+	if err == nil {
+		d.Duration = time.Second * time.Duration(sI)
+		return nil
+	}
+	// Second try parsing as float seconds
+	sF, err := strconv.ParseFloat(string(b), 64)
+	if err == nil {
+		d.Duration = time.Second * time.Duration(sF)
+		return nil
+	}
 
 	return nil
 }
 
-var NotImplementedError = errors.New("not implemented yet")
-
 // ReadLines reads contents from a file and splits them by new lines.
 // A convenience wrapper to ReadLinesOffsetN(filename, 0, -1).
 func ReadLines(filename string) ([]string, error) {
@@ -86,15 +109,15 @@ func GetTLSConfig(
 	SSLCert, SSLKey, SSLCA string,
 	InsecureSkipVerify bool,
 ) (*tls.Config, error) {
-	t := &tls.Config{}
+	if SSLCert == "" && SSLKey == "" && SSLCA == "" && !InsecureSkipVerify {
-	if SSLCert != "" && SSLKey != "" && SSLCA != "" {
+		return nil, nil
-		cert, err := tls.LoadX509KeyPair(SSLCert, SSLKey)
+	}
-		if err != nil {
-			return nil, errors.New(fmt.Sprintf(
-				"Could not load TLS client key/certificate: %s",
-				err))
-		}
 
+	t := &tls.Config{
+		InsecureSkipVerify: InsecureSkipVerify,
+	}
+
+	if SSLCA != "" {
 		caCert, err := ioutil.ReadFile(SSLCA)
 		if err != nil {
 			return nil, errors.New(fmt.Sprintf("Could not load TLS CA: %s",
@@ -103,75 +126,107 @@ func GetTLSConfig(
 
 		caCertPool := x509.NewCertPool()
 		caCertPool.AppendCertsFromPEM(caCert)
-
+		t.RootCAs = caCertPool
-		t = &tls.Config{
-			Certificates:       []tls.Certificate{cert},
-			RootCAs:            caCertPool,
-			InsecureSkipVerify: InsecureSkipVerify,
-		}
-	} else {
-		if InsecureSkipVerify {
-			t.InsecureSkipVerify = true
-		} else {
-			return nil, nil
-		}
 	}
+
+	if SSLCert != "" && SSLKey != "" {
+		cert, err := tls.LoadX509KeyPair(SSLCert, SSLKey)
+		if err != nil {
+			return nil, errors.New(fmt.Sprintf(
+				"Could not load TLS client key/certificate from %s:%s: %s",
+				SSLKey, SSLCert, err))
+		}
+
+		t.Certificates = []tls.Certificate{cert}
+		t.BuildNameToCertificate()
+	}
+
 	// will be nil by default if nothing is provided
 	return t, nil
 }
 
-// Glob will test a string pattern, potentially containing globs, against a
+// SnakeCase converts the given string to snake case following the Golang format:
-// subject string. The result is a simple true/false, determining whether or
+// acronyms are converted to lower-case and preceded by an underscore.
-// not the glob pattern matched the subject text.
+func SnakeCase(in string) string {
-//
+	runes := []rune(in)
-// Adapted from https://github.com/ryanuber/go-glob/blob/master/glob.go
+	length := len(runes)
-// thanks Ryan Uber!
-func Glob(pattern, measurement string) bool {
-	// Empty pattern can only match empty subject
-	if pattern == "" {
-		return measurement == pattern
-	}
 
-	// If the pattern _is_ a glob, it matches everything
+	var out []rune
-	if pattern == "*" {
+	for i := 0; i < length; i++ {
-		return true
+		if i > 0 && unicode.IsUpper(runes[i]) && ((i+1 < length && unicode.IsLower(runes[i+1])) || unicode.IsLower(runes[i-1])) {
-	}
+			out = append(out, '_')
-
-	parts := strings.Split(pattern, "*")
-
-	if len(parts) == 1 {
-		// No globs in pattern, so test for match
-		return pattern == measurement
-	}
-
-	leadingGlob := strings.HasPrefix(pattern, "*")
-	trailingGlob := strings.HasSuffix(pattern, "*")
-	end := len(parts) - 1
-
-	for i, part := range parts {
-		switch i {
-		case 0:
-			if leadingGlob {
-				continue
-			}
-			if !strings.HasPrefix(measurement, part) {
-				return false
-			}
-		case end:
-			if len(measurement) > 0 {
-				return trailingGlob || strings.HasSuffix(measurement, part)
-			}
-		default:
-			if !strings.Contains(measurement, part) {
-				return false
-			}
 		}
-
+		out = append(out, unicode.ToLower(runes[i]))
-		// Trim evaluated text from measurement as we loop over the pattern.
-		idx := strings.Index(measurement, part) + len(part)
-		measurement = measurement[idx:]
 	}
 
-	// All parts of the pattern matched
+	return string(out)
-	return true
+}
+
+// CombinedOutputTimeout runs the given command with the given timeout and
+// returns the combined output of stdout and stderr.
+// If the command times out, it attempts to kill the process.
+func CombinedOutputTimeout(c *exec.Cmd, timeout time.Duration) ([]byte, error) {
+	var b bytes.Buffer
+	c.Stdout = &b
+	c.Stderr = &b
+	if err := c.Start(); err != nil {
+		return nil, err
+	}
+	err := WaitTimeout(c, timeout)
+	return b.Bytes(), err
+}
+
+// RunTimeout runs the given command with the given timeout.
+// If the command times out, it attempts to kill the process.
+func RunTimeout(c *exec.Cmd, timeout time.Duration) error {
+	if err := c.Start(); err != nil {
+		return err
+	}
+	return WaitTimeout(c, timeout)
+}
+
+// WaitTimeout waits for the given command to finish with a timeout.
+// It assumes the command has already been started.
+// If the command times out, it attempts to kill the process.
+func WaitTimeout(c *exec.Cmd, timeout time.Duration) error {
+	timer := time.NewTimer(timeout)
+	done := make(chan error)
+	go func() { done <- c.Wait() }()
+	select {
+	case err := <-done:
+		timer.Stop()
+		return err
+	case <-timer.C:
+		if err := c.Process.Kill(); err != nil {
+			log.Printf("FATAL error killing process: %s", err)
+			return err
+		}
+		// wait for the command to return after killing it
+		<-done
+		return TimeoutErr
+	}
+}
+
+// RandomSleep will sleep for a random amount of time up to max.
+// If the shutdown channel is closed, it will return before it has finished
+// sleeping.
+func RandomSleep(max time.Duration, shutdown chan struct{}) {
+	if max == 0 {
+		return
+	}
+	maxSleep := big.NewInt(max.Nanoseconds())
+
+	var sleepns int64
+	if j, err := rand.Int(rand.Reader, maxSleep); err == nil {
+		sleepns = j.Int64()
+	}
+
+	t := time.NewTimer(time.Nanosecond * time.Duration(sleepns))
+	select {
+	case <-t.C:
+		return
+	case <-shutdown:
+		t.Stop()
+		return
+	}
 }

internal/internal_test.go

@@ -1,44 +1,133 @@
 package internal
 
-import "testing"
+import (
+	"os/exec"
+	"testing"
+	"time"
 
-func testGlobMatch(t *testing.T, pattern, subj string) {
+	"github.com/stretchr/testify/assert"
-	if !Glob(pattern, subj) {
+)
-		t.Errorf("%s should match %s", pattern, subj)
+
+type SnakeTest struct {
+	input  string
+	output string
+}
+
+var tests = []SnakeTest{
+	{"a", "a"},
+	{"snake", "snake"},
+	{"A", "a"},
+	{"ID", "id"},
+	{"MOTD", "motd"},
+	{"Snake", "snake"},
+	{"SnakeTest", "snake_test"},
+	{"APIResponse", "api_response"},
+	{"SnakeID", "snake_id"},
+	{"SnakeIDGoogle", "snake_id_google"},
+	{"LinuxMOTD", "linux_motd"},
+	{"OMGWTFBBQ", "omgwtfbbq"},
+	{"omg_wtf_bbq", "omg_wtf_bbq"},
+}
+
+func TestSnakeCase(t *testing.T) {
+	for _, test := range tests {
+		if SnakeCase(test.input) != test.output {
+			t.Errorf(`SnakeCase("%s"), wanted "%s", got \%s"`, test.input, test.output, SnakeCase(test.input))
+		}
 	}
 }
 
-func testGlobNoMatch(t *testing.T, pattern, subj string) {
+var (
-	if Glob(pattern, subj) {
+	sleepbin, _ = exec.LookPath("sleep")
-		t.Errorf("%s should not match %s", pattern, subj)
+	echobin, _  = exec.LookPath("echo")
+)
+
+func TestRunTimeout(t *testing.T) {
+	if sleepbin == "" {
+		t.Skip("'sleep' binary not available on OS, skipping.")
 	}
+	cmd := exec.Command(sleepbin, "10")
+	start := time.Now()
+	err := RunTimeout(cmd, time.Millisecond*20)
+	elapsed := time.Since(start)
+
+	assert.Equal(t, TimeoutErr, err)
+	// Verify that command gets killed in 20ms, with some breathing room
+	assert.True(t, elapsed < time.Millisecond*75)
 }
 
-func TestEmptyPattern(t *testing.T) {
+func TestCombinedOutputTimeout(t *testing.T) {
-	testGlobMatch(t, "", "")
+	if sleepbin == "" {
-	testGlobNoMatch(t, "", "test")
+		t.Skip("'sleep' binary not available on OS, skipping.")
-}
-
-func TestPatternWithoutGlobs(t *testing.T) {
-	testGlobMatch(t, "test", "test")
-}
-
-func TestGlob(t *testing.T) {
-	for _, pattern := range []string{
-		"*test",           // Leading glob
-		"this*",           // Trailing glob
-		"*is*a*",          // Lots of globs
-		"**test**",        // Double glob characters
-		"**is**a***test*", // Varying number of globs
-	} {
-		testGlobMatch(t, pattern, "this_is_a_test")
 	}
+	cmd := exec.Command(sleepbin, "10")
+	start := time.Now()
+	_, err := CombinedOutputTimeout(cmd, time.Millisecond*20)
+	elapsed := time.Since(start)
 
-	for _, pattern := range []string{
+	assert.Equal(t, TimeoutErr, err)
-		"test*", // Implicit substring match should fail
+	// Verify that command gets killed in 20ms, with some breathing room
-		"*is",   // Partial match should fail
+	assert.True(t, elapsed < time.Millisecond*75)
-		"*no*",  // Globs without a match between them should fail
+}
-	} {
+
-		testGlobNoMatch(t, pattern, "this_is_a_test")
+func TestCombinedOutput(t *testing.T) {
-	}
+	if echobin == "" {
+		t.Skip("'echo' binary not available on OS, skipping.")
+	}
+	cmd := exec.Command(echobin, "foo")
+	out, err := CombinedOutputTimeout(cmd, time.Second)
+
+	assert.NoError(t, err)
+	assert.Equal(t, "foo\n", string(out))
+}
+
+// test that CombinedOutputTimeout and exec.Cmd.CombinedOutput return
+// the same output from a failed command.
+func TestCombinedOutputError(t *testing.T) {
+	if sleepbin == "" {
+		t.Skip("'sleep' binary not available on OS, skipping.")
+	}
+	cmd := exec.Command(sleepbin, "foo")
+	expected, err := cmd.CombinedOutput()
+
+	cmd2 := exec.Command(sleepbin, "foo")
+	actual, err := CombinedOutputTimeout(cmd2, time.Second)
+
+	assert.Error(t, err)
+	assert.Equal(t, expected, actual)
+}
+
+func TestRunError(t *testing.T) {
+	if sleepbin == "" {
+		t.Skip("'sleep' binary not available on OS, skipping.")
+	}
+	cmd := exec.Command(sleepbin, "foo")
+	err := RunTimeout(cmd, time.Second)
+
+	assert.Error(t, err)
+}
+
+func TestRandomSleep(t *testing.T) {
+	// test that zero max returns immediately
+	s := time.Now()
+	RandomSleep(time.Duration(0), make(chan struct{}))
+	elapsed := time.Since(s)
+	assert.True(t, elapsed < time.Millisecond)
+
+	// test that max sleep is respected
+	s = time.Now()
+	RandomSleep(time.Millisecond*50, make(chan struct{}))
+	elapsed = time.Since(s)
+	assert.True(t, elapsed < time.Millisecond*50)
+
+	// test that shutdown is respected
+	s = time.Now()
+	shutdown := make(chan struct{})
+	go func() {
+		time.Sleep(time.Millisecond * 100)
+		close(shutdown)
+	}()
+	RandomSleep(time.Second, shutdown)
+	elapsed = time.Since(s)
+	assert.True(t, elapsed < time.Millisecond*150)
 }

internal/limiter/limiter.go

@@ -0,0 +1,59 @@
+package limiter
+
+import (
+	"sync"
+	"time"
+)
+
+// NewRateLimiter returns a rate limiter that will will emit from the C
+// channel only 'n' times every 'rate' seconds.
+func NewRateLimiter(n int, rate time.Duration) *rateLimiter {
+	r := &rateLimiter{
+		C:        make(chan bool),
+		rate:     rate,
+		n:        n,
+		shutdown: make(chan bool),
+	}
+	r.wg.Add(1)
+	go r.limiter()
+	return r
+}
+
+type rateLimiter struct {
+	C    chan bool
+	rate time.Duration
+	n    int
+
+	shutdown chan bool
+	wg       sync.WaitGroup
+}
+
+func (r *rateLimiter) Stop() {
+	close(r.shutdown)
+	r.wg.Wait()
+	close(r.C)
+}
+
+func (r *rateLimiter) limiter() {
+	defer r.wg.Done()
+	ticker := time.NewTicker(r.rate)
+	defer ticker.Stop()
+	counter := 0
+	for {
+		select {
+		case <-r.shutdown:
+			return
+		case <-ticker.C:
+			counter = 0
+		default:
+			if counter < r.n {
+				select {
+				case r.C <- true:
+					counter++
+				case <-r.shutdown:
+					return
+				}
+			}
+		}
+	}
+}

internal/limiter/limiter_test.go

@@ -0,0 +1,54 @@
+package limiter
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestRateLimiter(t *testing.T) {
+	r := NewRateLimiter(5, time.Second)
+	ticker := time.NewTicker(time.Millisecond * 75)
+
+	// test that we can only get 5 receives from the rate limiter
+	counter := 0
+outer:
+	for {
+		select {
+		case <-r.C:
+			counter++
+		case <-ticker.C:
+			break outer
+		}
+	}
+
+	assert.Equal(t, 5, counter)
+	r.Stop()
+	// verify that the Stop function closes the channel.
+	_, ok := <-r.C
+	assert.False(t, ok)
+}
+
+func TestRateLimiterMultipleIterations(t *testing.T) {
+	r := NewRateLimiter(5, time.Millisecond*50)
+	ticker := time.NewTicker(time.Millisecond * 250)
+
+	// test that we can get 15 receives from the rate limiter
+	counter := 0
+outer:
+	for {
+		select {
+		case <-ticker.C:
+			break outer
+		case <-r.C:
+			counter++
+		}
+	}
+
+	assert.True(t, counter > 10)
+	r.Stop()
+	// verify that the Stop function closes the channel.
+	_, ok := <-r.C
+	assert.False(t, ok)
+}

internal/models/filter.go

@@ -1,33 +1,88 @@
 package internal_models
 
 import (
-	"strings"
+	"fmt"
 
 	"github.com/influxdata/telegraf"
-	"github.com/influxdata/telegraf/internal"
+	"github.com/influxdata/telegraf/filter"
 )
 
 // TagFilter is the name of a tag, and the values on which to filter
 type TagFilter struct {
 	Name   string
 	Filter []string
+	filter filter.Filter
 }
 
 // Filter containing drop/pass and tagdrop/tagpass rules
 type Filter struct {
 	NameDrop []string
+	nameDrop filter.Filter
 	NamePass []string
+	namePass filter.Filter
 
 	FieldDrop []string
+	fieldDrop filter.Filter
 	FieldPass []string
+	fieldPass filter.Filter
 
 	TagDrop []TagFilter
 	TagPass []TagFilter
 
+	TagExclude []string
+	tagExclude filter.Filter
+	TagInclude []string
+	tagInclude filter.Filter
+
 	IsActive bool
 }
 
-func (f Filter) ShouldMetricPass(metric telegraf.Metric) bool {
+// Compile all Filter lists into filter.Filter objects.
+func (f *Filter) CompileFilter() error {
+	var err error
+	f.nameDrop, err = filter.CompileFilter(f.NameDrop)
+	if err != nil {
+		return fmt.Errorf("Error compiling 'namedrop', %s", err)
+	}
+	f.namePass, err = filter.CompileFilter(f.NamePass)
+	if err != nil {
+		return fmt.Errorf("Error compiling 'namepass', %s", err)
+	}
+
+	f.fieldDrop, err = filter.CompileFilter(f.FieldDrop)
+	if err != nil {
+		return fmt.Errorf("Error compiling 'fielddrop', %s", err)
+	}
+	f.fieldPass, err = filter.CompileFilter(f.FieldPass)
+	if err != nil {
+		return fmt.Errorf("Error compiling 'fieldpass', %s", err)
+	}
+
+	f.tagExclude, err = filter.CompileFilter(f.TagExclude)
+	if err != nil {
+		return fmt.Errorf("Error compiling 'tagexclude', %s", err)
+	}
+	f.tagInclude, err = filter.CompileFilter(f.TagInclude)
+	if err != nil {
+		return fmt.Errorf("Error compiling 'taginclude', %s", err)
+	}
+
+	for i, _ := range f.TagDrop {
+		f.TagDrop[i].filter, err = filter.CompileFilter(f.TagDrop[i].Filter)
+		if err != nil {
+			return fmt.Errorf("Error compiling 'tagdrop', %s", err)
+		}
+	}
+	for i, _ := range f.TagPass {
+		f.TagPass[i].filter, err = filter.CompileFilter(f.TagPass[i].Filter)
+		if err != nil {
+			return fmt.Errorf("Error compiling 'tagpass', %s", err)
+		}
+	}
+	return nil
+}
+
+func (f *Filter) ShouldMetricPass(metric telegraf.Metric) bool {
 	if f.ShouldNamePass(metric.Name()) && f.ShouldTagsPass(metric.Tags()) {
 		return true
 	}
@@ -36,70 +91,51 @@ func (f Filter) ShouldMetricPass(metric telegraf.Metric) bool {
 
 // ShouldFieldsPass returns true if the metric should pass, false if should drop
 // based on the drop/pass filter parameters
-func (f Filter) ShouldNamePass(key string) bool {
+func (f *Filter) ShouldNamePass(key string) bool {
-	if f.NamePass != nil {
+	if f.namePass != nil {
-		for _, pat := range f.NamePass {
+		if f.namePass.Match(key) {
-			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			return true
-			// Cam, 2015-12-07
-			if strings.HasPrefix(key, pat) || internal.Glob(pat, key) {
-				return true
-			}
 		}
 		return false
 	}
 
-	if f.NameDrop != nil {
+	if f.nameDrop != nil {
-		for _, pat := range f.NameDrop {
+		if f.nameDrop.Match(key) {
-			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			return false
-			// Cam, 2015-12-07
-			if strings.HasPrefix(key, pat) || internal.Glob(pat, key) {
-				return false
-			}
 		}
-
-		return true
 	}
 	return true
 }
 
 // ShouldFieldsPass returns true if the metric should pass, false if should drop
 // based on the drop/pass filter parameters
-func (f Filter) ShouldFieldsPass(key string) bool {
+func (f *Filter) ShouldFieldsPass(key string) bool {
-	if f.FieldPass != nil {
+	if f.fieldPass != nil {
-		for _, pat := range f.FieldPass {
+		if f.fieldPass.Match(key) {
-			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			return true
-			// Cam, 2015-12-07
-			if strings.HasPrefix(key, pat) || internal.Glob(pat, key) {
-				return true
-			}
 		}
 		return false
 	}
 
-	if f.FieldDrop != nil {
+	if f.fieldDrop != nil {
-		for _, pat := range f.FieldDrop {
+		if f.fieldDrop.Match(key) {
-			// TODO remove HasPrefix check, leaving it for now for legacy support.
+			return false
-			// Cam, 2015-12-07
-			if strings.HasPrefix(key, pat) || internal.Glob(pat, key) {
-				return false
-			}
 		}
-
-		return true
 	}
 	return true
 }
 
 // ShouldTagsPass returns true if the metric should pass, false if should drop
 // based on the tagdrop/tagpass filter parameters
-func (f Filter) ShouldTagsPass(tags map[string]string) bool {
+func (f *Filter) ShouldTagsPass(tags map[string]string) bool {
 	if f.TagPass != nil {
 		for _, pat := range f.TagPass {
+			if pat.filter == nil {
+				continue
+			}
 			if tagval, ok := tags[pat.Name]; ok {
-				for _, filter := range pat.Filter {
+				if pat.filter.Match(tagval) {
-					if internal.Glob(filter, tagval) {
+					return true
-						return true
-					}
 				}
 			}
 		}
@@ -108,11 +144,12 @@ func (f Filter) ShouldTagsPass(tags map[string]string) bool {
 
 	if f.TagDrop != nil {
 		for _, pat := range f.TagDrop {
+			if pat.filter == nil {
+				continue
+			}
 			if tagval, ok := tags[pat.Name]; ok {
-				for _, filter := range pat.Filter {
+				if pat.filter.Match(tagval) {
-					if internal.Glob(filter, tagval) {
+					return false
-						return false
-					}
 				}
 			}
 		}
@@ -121,3 +158,23 @@ func (f Filter) ShouldTagsPass(tags map[string]string) bool {
 
 	return true
 }
+
+// Apply TagInclude and TagExclude filters.
+// modifies the tags map in-place.
+func (f *Filter) FilterTags(tags map[string]string) {
+	if f.tagInclude != nil {
+		for k, _ := range tags {
+			if !f.tagInclude.Match(k) {
+				delete(tags, k)
+			}
+		}
+	}
+
+	if f.tagExclude != nil {
+		for k, _ := range tags {
+			if f.tagExclude.Match(k) {
+				delete(tags, k)
+			}
+		}
+	}
+}

internal/models/filter_test.go

@@ -2,6 +2,11 @@ package internal_models
 
 import (
 	"testing"
+
+	"github.com/influxdata/telegraf/testutil"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
 func TestFilter_Empty(t *testing.T) {
@@ -28,6 +33,7 @@ func TestFilter_NamePass(t *testing.T) {
 	f := Filter{
 		NamePass: []string{"foo*", "cpu_usage_idle"},
 	}
+	require.NoError(t, f.CompileFilter())
 
 	passes := []string{
 		"foo",
@@ -61,6 +67,7 @@ func TestFilter_NameDrop(t *testing.T) {
 	f := Filter{
 		NameDrop: []string{"foo*", "cpu_usage_idle"},
 	}
+	require.NoError(t, f.CompileFilter())
 
 	drops := []string{
 		"foo",
@@ -94,6 +101,7 @@ func TestFilter_FieldPass(t *testing.T) {
 	f := Filter{
 		FieldPass: []string{"foo*", "cpu_usage_idle"},
 	}
+	require.NoError(t, f.CompileFilter())
 
 	passes := []string{
 		"foo",
@@ -127,6 +135,7 @@ func TestFilter_FieldDrop(t *testing.T) {
 	f := Filter{
 		FieldDrop: []string{"foo*", "cpu_usage_idle"},
 	}
+	require.NoError(t, f.CompileFilter())
 
 	drops := []string{
 		"foo",
@@ -169,6 +178,7 @@ func TestFilter_TagPass(t *testing.T) {
 	f := Filter{
 		TagPass: filters,
 	}
+	require.NoError(t, f.CompileFilter())
 
 	passes := []map[string]string{
 		{"cpu": "cpu-total"},
@@ -212,6 +222,7 @@ func TestFilter_TagDrop(t *testing.T) {
 	f := Filter{
 		TagDrop: filters,
 	}
+	require.NoError(t, f.CompileFilter())
 
 	drops := []map[string]string{
 		{"cpu": "cpu-total"},
@@ -241,3 +252,70 @@ func TestFilter_TagDrop(t *testing.T) {
 		}
 	}
 }
+
+func TestFilter_ShouldMetricsPass(t *testing.T) {
+	m := testutil.TestMetric(1, "testmetric")
+	f := Filter{
+		NameDrop: []string{"foobar"},
+	}
+	require.NoError(t, f.CompileFilter())
+	require.True(t, f.ShouldMetricPass(m))
+
+	m = testutil.TestMetric(1, "foobar")
+	require.False(t, f.ShouldMetricPass(m))
+}
+
+func TestFilter_FilterTagsNoMatches(t *testing.T) {
+	pretags := map[string]string{
+		"host":  "localhost",
+		"mytag": "foobar",
+	}
+	f := Filter{
+		TagExclude: []string{"nomatch"},
+	}
+	require.NoError(t, f.CompileFilter())
+
+	f.FilterTags(pretags)
+	assert.Equal(t, map[string]string{
+		"host":  "localhost",
+		"mytag": "foobar",
+	}, pretags)
+
+	f = Filter{
+		TagInclude: []string{"nomatch"},
+	}
+	require.NoError(t, f.CompileFilter())
+
+	f.FilterTags(pretags)
+	assert.Equal(t, map[string]string{}, pretags)
+}
+
+func TestFilter_FilterTagsMatches(t *testing.T) {
+	pretags := map[string]string{
+		"host":  "localhost",
+		"mytag": "foobar",
+	}
+	f := Filter{
+		TagExclude: []string{"ho*"},
+	}
+	require.NoError(t, f.CompileFilter())
+
+	f.FilterTags(pretags)
+	assert.Equal(t, map[string]string{
+		"mytag": "foobar",
+	}, pretags)
+
+	pretags = map[string]string{
+		"host":  "localhost",
+		"mytag": "foobar",
+	}
+	f = Filter{
+		TagInclude: []string{"my*"},
+	}
+	require.NoError(t, f.CompileFilter())
+
+	f.FilterTags(pretags)
+	assert.Equal(t, map[string]string{
+		"mytag": "foobar",
+	}, pretags)
+}

internal/models/running_output.go

@@ -2,48 +2,54 @@ package internal_models
 
 import (
 	"log"
-	"sync"
 	"time"
 
 	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/internal/buffer"
 )
 
 const (
-	// Default number of metrics kept between flushes.
+	// Default size of metrics batch size.
-	DEFAULT_METRIC_BUFFER_LIMIT = 1000
+	DEFAULT_METRIC_BATCH_SIZE = 1000
 
-	// Limit how many full metric buffers are kept due to failed writes.
+	// Default number of metrics kept. It should be a multiple of batch size.
-	FULL_METRIC_BUFFERS_LIMIT = 100
+	DEFAULT_METRIC_BUFFER_LIMIT = 10000
 )
 
+// RunningOutput contains the output configuration
 type RunningOutput struct {
-	Name                string
+	Name              string
-	Output              telegraf.Output
+	Output            telegraf.Output
-	Config              *OutputConfig
+	Config            *OutputConfig
-	Quiet               bool
+	Quiet             bool
-	MetricBufferLimit   int
+	MetricBufferLimit int
-	FlushBufferWhenFull bool
+	MetricBatchSize   int
 
-	metrics    []telegraf.Metric
+	metrics     *buffer.Buffer
-	tmpmetrics map[int][]telegraf.Metric
+	failMetrics *buffer.Buffer
-	overwriteI int
-	mapI       int
-
-	sync.Mutex
 }
 
 func NewRunningOutput(
 	name string,
 	output telegraf.Output,
 	conf *OutputConfig,
+	batchSize int,
+	bufferLimit int,
 ) *RunningOutput {
+	if bufferLimit == 0 {
+		bufferLimit = DEFAULT_METRIC_BUFFER_LIMIT
+	}
+	if batchSize == 0 {
+		batchSize = DEFAULT_METRIC_BATCH_SIZE
+	}
 	ro := &RunningOutput{
 		Name:              name,
-		metrics:           make([]telegraf.Metric, 0),
+		metrics:           buffer.NewBuffer(batchSize),
-		tmpmetrics:        make(map[int][]telegraf.Metric),
+		failMetrics:       buffer.NewBuffer(bufferLimit),
 		Output:            output,
 		Config:            conf,
-		MetricBufferLimit: DEFAULT_METRIC_BUFFER_LIMIT,
+		MetricBufferLimit: bufferLimit,
+		MetricBatchSize:   batchSize,
 	}
 	return ro
 }
@@ -56,78 +62,92 @@ func (ro *RunningOutput) AddMetric(metric telegraf.Metric) {
 			return
 		}
 	}
-	ro.Lock()
-	defer ro.Unlock()
 
-	if len(ro.metrics) < ro.MetricBufferLimit {
+	// Filter any tagexclude/taginclude parameters before adding metric
-		ro.metrics = append(ro.metrics, metric)
+	if len(ro.Config.Filter.TagExclude) != 0 || len(ro.Config.Filter.TagInclude) != 0 {
-	} else {
+		// In order to filter out tags, we need to create a new metric, since
-		if ro.FlushBufferWhenFull {
+		// metrics are immutable once created.
-			ro.metrics = append(ro.metrics, metric)
+		tags := metric.Tags()
-			tmpmetrics := make([]telegraf.Metric, len(ro.metrics))
+		fields := metric.Fields()
-			copy(tmpmetrics, ro.metrics)
+		t := metric.Time()
-			ro.metrics = make([]telegraf.Metric, 0)
+		name := metric.Name()
-			err := ro.write(tmpmetrics)
+		ro.Config.Filter.FilterTags(tags)
-			if err != nil {
+		// error is not possible if creating from another metric, so ignore.
-				log.Printf("ERROR writing full metric buffer to output %s, %s",
+		metric, _ = telegraf.NewMetric(name, tags, fields, t)
-					ro.Name, err)
+	}
-				if len(ro.tmpmetrics) == FULL_METRIC_BUFFERS_LIMIT {
+
-					ro.mapI = 0
+	ro.metrics.Add(metric)
-					// overwrite one
+	if ro.metrics.Len() == ro.MetricBatchSize {
-					ro.tmpmetrics[ro.mapI] = tmpmetrics
+		batch := ro.metrics.Batch(ro.MetricBatchSize)
-					ro.mapI++
+		err := ro.write(batch)
-				} else {
+		if err != nil {
-					ro.tmpmetrics[ro.mapI] = tmpmetrics
+			ro.failMetrics.Add(batch...)
-					ro.mapI++
-				}
-			}
-		} else {
-			if ro.overwriteI == 0 {
-				log.Printf("WARNING: overwriting cached metrics, you may want to " +
-					"increase the metric_buffer_limit setting in your [agent] " +
-					"config if you do not wish to overwrite metrics.\n")
-			}
-			if ro.overwriteI == len(ro.metrics) {
-				ro.overwriteI = 0
-			}
-			ro.metrics[ro.overwriteI] = metric
-			ro.overwriteI++
 		}
 	}
 }
 
 // Write writes all cached points to this output.
 func (ro *RunningOutput) Write() error {
-	ro.Lock()
+	if !ro.Quiet {
-	defer ro.Unlock()
+		log.Printf("Output [%s] buffer fullness: %d / %d metrics. "+
-	err := ro.write(ro.metrics)
+			"Total gathered metrics: %d. Total dropped metrics: %d.",
-	if err != nil {
+			ro.Name,
-		return err
+			ro.failMetrics.Len()+ro.metrics.Len(),
-	} else {
+			ro.MetricBufferLimit,
-		ro.metrics = make([]telegraf.Metric, 0)
+			ro.metrics.Total(),
-		ro.overwriteI = 0
+			ro.metrics.Drops()+ro.failMetrics.Drops())
 	}
 
-	// Write any cached metric buffers that failed previously
+	var err error
-	for i, tmpmetrics := range ro.tmpmetrics {
+	if !ro.failMetrics.IsEmpty() {
-		if err := ro.write(tmpmetrics); err != nil {
+		bufLen := ro.failMetrics.Len()
-			return err
+		// how many batches of failed writes we need to write.
-		} else {
+		nBatches := bufLen/ro.MetricBatchSize + 1
-			delete(ro.tmpmetrics, i)
+		batchSize := ro.MetricBatchSize
+
+		for i := 0; i < nBatches; i++ {
+			// If it's the last batch, only grab the metrics that have not had
+			// a write attempt already (this is primarily to preserve order).
+			if i == nBatches-1 {
+				batchSize = bufLen % ro.MetricBatchSize
+			}
+			batch := ro.failMetrics.Batch(batchSize)
+			// If we've already failed previous writes, don't bother trying to
+			// write to this output again. We are not exiting the loop just so
+			// that we can rotate the metrics to preserve order.
+			if err == nil {
+				err = ro.write(batch)
+			}
+			if err != nil {
+				ro.failMetrics.Add(batch...)
+			}
 		}
 	}
 
+	batch := ro.metrics.Batch(ro.MetricBatchSize)
+	// see comment above about not trying to write to an already failed output.
+	// if ro.failMetrics is empty then err will always be nil at this point.
+	if err == nil {
+		err = ro.write(batch)
+	}
+	if err != nil {
+		ro.failMetrics.Add(batch...)
+		return err
+	}
 	return nil
 }
 
 func (ro *RunningOutput) write(metrics []telegraf.Metric) error {
+	if metrics == nil || len(metrics) == 0 {
+		return nil
+	}
 	start := time.Now()
 	err := ro.Output.Write(metrics)
 	elapsed := time.Since(start)
 	if err == nil {
 		if !ro.Quiet {
-			log.Printf("Wrote %d metrics to output %s in %s\n",
+			log.Printf("Output [%s] wrote batch of %d metrics in %s\n",
-				len(metrics), ro.Name, elapsed)
+				ro.Name, len(metrics), elapsed)
 		}
 	}
 	return err

internal/models/running_output_test.go

@@ -2,7 +2,6 @@ package internal_models
 
 import (
 	"fmt"
-	"sort"
 	"sync"
 	"testing"
 
@@ -29,16 +28,100 @@ var next5 = []telegraf.Metric{
 	testutil.TestMetric(101, "metric10"),
 }
 
-// Test that we can write metrics with simple default setup.
+// Benchmark adding metrics.
-func TestRunningOutputDefault(t *testing.T) {
+func BenchmarkRunningOutputAddWrite(b *testing.B) {
 	conf := &OutputConfig{
 		Filter: Filter{
 			IsActive: false,
 		},
 	}
 
+	m := &perfOutput{}
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+	ro.Quiet = true
+
+	for n := 0; n < b.N; n++ {
+		ro.AddMetric(first5[0])
+		ro.Write()
+	}
+}
+
+// Benchmark adding metrics.
+func BenchmarkRunningOutputAddWriteEvery100(b *testing.B) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: false,
+		},
+	}
+
+	m := &perfOutput{}
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+	ro.Quiet = true
+
+	for n := 0; n < b.N; n++ {
+		ro.AddMetric(first5[0])
+		if n%100 == 0 {
+			ro.Write()
+		}
+	}
+}
+
+// Benchmark adding metrics.
+func BenchmarkRunningOutputAddFailWrites(b *testing.B) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: false,
+		},
+	}
+
+	m := &perfOutput{}
+	m.failWrite = true
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+	ro.Quiet = true
+
+	for n := 0; n < b.N; n++ {
+		ro.AddMetric(first5[0])
+	}
+}
+
+// Test that NameDrop filters ger properly applied.
+func TestRunningOutput_DropFilter(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: true,
+			NameDrop: []string{"metric1", "metric2"},
+		},
+	}
+	assert.NoError(t, conf.Filter.CompileFilter())
+
 	m := &mockOutput{}
-	ro := NewRunningOutput("test", m, conf)
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+
+	for _, metric := range first5 {
+		ro.AddMetric(metric)
+	}
+	for _, metric := range next5 {
+		ro.AddMetric(metric)
+	}
+	assert.Len(t, m.Metrics(), 0)
+
+	err := ro.Write()
+	assert.NoError(t, err)
+	assert.Len(t, m.Metrics(), 8)
+}
+
+// Test that NameDrop filters without a match do nothing.
+func TestRunningOutput_PassFilter(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: true,
+			NameDrop: []string{"metric1000", "foo*"},
+		},
+	}
+	assert.NoError(t, conf.Filter.CompileFilter())
+
+	m := &mockOutput{}
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
 
 	for _, metric := range first5 {
 		ro.AddMetric(metric)
@@ -53,41 +136,96 @@ func TestRunningOutputDefault(t *testing.T) {
 	assert.Len(t, m.Metrics(), 10)
 }
 
-// Test that the first metric gets overwritten if there is a buffer overflow.
+// Test that tags are properly included
-func TestRunningOutputOverwrite(t *testing.T) {
+func TestRunningOutput_TagIncludeNoMatch(t *testing.T) {
 	conf := &OutputConfig{
 		Filter: Filter{
-			IsActive: false,
+			IsActive:   true,
+			TagInclude: []string{"nothing*"},
 		},
 	}
+	assert.NoError(t, conf.Filter.CompileFilter())
 
 	m := &mockOutput{}
-	ro := NewRunningOutput("test", m, conf)
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
-	ro.MetricBufferLimit = 4
 
-	for _, metric := range first5 {
+	ro.AddMetric(first5[0])
-		ro.AddMetric(metric)
+	assert.Len(t, m.Metrics(), 0)
-	}
-	require.Len(t, m.Metrics(), 0)
 
 	err := ro.Write()
-	require.NoError(t, err)
+	assert.NoError(t, err)
-	require.Len(t, m.Metrics(), 4)
+	assert.Len(t, m.Metrics(), 1)
-
+	assert.Empty(t, m.Metrics()[0].Tags())
-	var expected, actual []string
-	for i, exp := range first5[1:] {
-		expected = append(expected, exp.String())
-		actual = append(actual, m.Metrics()[i].String())
-	}
-
-	sort.Strings(expected)
-	sort.Strings(actual)
-
-	assert.Equal(t, expected, actual)
 }
 
-// Test that multiple buffer overflows are handled properly.
+// Test that tags are properly excluded
-func TestRunningOutputMultiOverwrite(t *testing.T) {
+func TestRunningOutput_TagExcludeMatch(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive:   true,
+			TagExclude: []string{"tag*"},
+		},
+	}
+	assert.NoError(t, conf.Filter.CompileFilter())
+
+	m := &mockOutput{}
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+
+	ro.AddMetric(first5[0])
+	assert.Len(t, m.Metrics(), 0)
+
+	err := ro.Write()
+	assert.NoError(t, err)
+	assert.Len(t, m.Metrics(), 1)
+	assert.Len(t, m.Metrics()[0].Tags(), 0)
+}
+
+// Test that tags are properly Excluded
+func TestRunningOutput_TagExcludeNoMatch(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive:   true,
+			TagExclude: []string{"nothing*"},
+		},
+	}
+	assert.NoError(t, conf.Filter.CompileFilter())
+
+	m := &mockOutput{}
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+
+	ro.AddMetric(first5[0])
+	assert.Len(t, m.Metrics(), 0)
+
+	err := ro.Write()
+	assert.NoError(t, err)
+	assert.Len(t, m.Metrics(), 1)
+	assert.Len(t, m.Metrics()[0].Tags(), 1)
+}
+
+// Test that tags are properly included
+func TestRunningOutput_TagIncludeMatch(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive:   true,
+			TagInclude: []string{"tag*"},
+		},
+	}
+	assert.NoError(t, conf.Filter.CompileFilter())
+
+	m := &mockOutput{}
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
+
+	ro.AddMetric(first5[0])
+	assert.Len(t, m.Metrics(), 0)
+
+	err := ro.Write()
+	assert.NoError(t, err)
+	assert.Len(t, m.Metrics(), 1)
+	assert.Len(t, m.Metrics()[0].Tags(), 1)
+}
+
+// Test that we can write metrics with simple default setup.
+func TestRunningOutputDefault(t *testing.T) {
 	conf := &OutputConfig{
 		Filter: Filter{
 			IsActive: false,
@@ -95,8 +233,7 @@ func TestRunningOutputMultiOverwrite(t *testing.T) {
 	}
 
 	m := &mockOutput{}
-	ro := NewRunningOutput("test", m, conf)
+	ro := NewRunningOutput("test", m, conf, 1000, 10000)
-	ro.MetricBufferLimit = 3
 
 	for _, metric := range first5 {
 		ro.AddMetric(metric)
@@ -104,22 +241,11 @@ func TestRunningOutputMultiOverwrite(t *testing.T) {
 	for _, metric := range next5 {
 		ro.AddMetric(metric)
 	}
-	require.Len(t, m.Metrics(), 0)
+	assert.Len(t, m.Metrics(), 0)
 
 	err := ro.Write()
-	require.NoError(t, err)
+	assert.NoError(t, err)
-	require.Len(t, m.Metrics(), 3)
+	assert.Len(t, m.Metrics(), 10)
-
-	var expected, actual []string
-	for i, exp := range next5[2:] {
-		expected = append(expected, exp.String())
-		actual = append(actual, m.Metrics()[i].String())
-	}
-
-	sort.Strings(expected)
-	sort.Strings(actual)
-
-	assert.Equal(t, expected, actual)
 }
 
 // Test that running output doesn't flush until it's full when
@@ -132,11 +258,9 @@ func TestRunningOutputFlushWhenFull(t *testing.T) {
 	}
 
 	m := &mockOutput{}
-	ro := NewRunningOutput("test", m, conf)
+	ro := NewRunningOutput("test", m, conf, 6, 10)
-	ro.FlushBufferWhenFull = true
-	ro.MetricBufferLimit = 5
 
-	// Fill buffer to limit
+	// Fill buffer to 1 under limit
 	for _, metric := range first5 {
 		ro.AddMetric(metric)
 	}
@@ -165,9 +289,7 @@ func TestRunningOutputMultiFlushWhenFull(t *testing.T) {
 	}
 
 	m := &mockOutput{}
-	ro := NewRunningOutput("test", m, conf)
+	ro := NewRunningOutput("test", m, conf, 4, 12)
-	ro.FlushBufferWhenFull = true
-	ro.MetricBufferLimit = 4
 
 	// Fill buffer past limit twive
 	for _, metric := range first5 {
@@ -177,7 +299,7 @@ func TestRunningOutputMultiFlushWhenFull(t *testing.T) {
 		ro.AddMetric(metric)
 	}
 	// flushed twice
-	assert.Len(t, m.Metrics(), 10)
+	assert.Len(t, m.Metrics(), 8)
 }
 
 func TestRunningOutputWriteFail(t *testing.T) {
@@ -189,11 +311,9 @@ func TestRunningOutputWriteFail(t *testing.T) {
 
 	m := &mockOutput{}
 	m.failWrite = true
-	ro := NewRunningOutput("test", m, conf)
+	ro := NewRunningOutput("test", m, conf, 4, 12)
-	ro.FlushBufferWhenFull = true
-	ro.MetricBufferLimit = 4
 
-	// Fill buffer past limit twice
+	// Fill buffer to limit twice
 	for _, metric := range first5 {
 		ro.AddMetric(metric)
 	}
@@ -216,6 +336,161 @@ func TestRunningOutputWriteFail(t *testing.T) {
 	assert.Len(t, m.Metrics(), 10)
 }
 
+// Verify that the order of points is preserved during a write failure.
+func TestRunningOutputWriteFailOrder(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: false,
+		},
+	}
+
+	m := &mockOutput{}
+	m.failWrite = true
+	ro := NewRunningOutput("test", m, conf, 100, 1000)
+
+	// add 5 metrics
+	for _, metric := range first5 {
+		ro.AddMetric(metric)
+	}
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	// Write fails
+	err := ro.Write()
+	require.Error(t, err)
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	m.failWrite = false
+	// add 5 more metrics
+	for _, metric := range next5 {
+		ro.AddMetric(metric)
+	}
+	err = ro.Write()
+	require.NoError(t, err)
+
+	// Verify that 10 metrics were written
+	assert.Len(t, m.Metrics(), 10)
+	// Verify that they are in order
+	expected := append(first5, next5...)
+	assert.Equal(t, expected, m.Metrics())
+}
+
+// Verify that the order of points is preserved during many write failures.
+func TestRunningOutputWriteFailOrder2(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: false,
+		},
+	}
+
+	m := &mockOutput{}
+	m.failWrite = true
+	ro := NewRunningOutput("test", m, conf, 5, 100)
+
+	// add 5 metrics
+	for _, metric := range first5 {
+		ro.AddMetric(metric)
+	}
+	// Write fails
+	err := ro.Write()
+	require.Error(t, err)
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	// add 5 metrics
+	for _, metric := range next5 {
+		ro.AddMetric(metric)
+	}
+	// Write fails
+	err = ro.Write()
+	require.Error(t, err)
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	// add 5 metrics
+	for _, metric := range first5 {
+		ro.AddMetric(metric)
+	}
+	// Write fails
+	err = ro.Write()
+	require.Error(t, err)
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	// add 5 metrics
+	for _, metric := range next5 {
+		ro.AddMetric(metric)
+	}
+	// Write fails
+	err = ro.Write()
+	require.Error(t, err)
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	m.failWrite = false
+	err = ro.Write()
+	require.NoError(t, err)
+
+	// Verify that 10 metrics were written
+	assert.Len(t, m.Metrics(), 20)
+	// Verify that they are in order
+	expected := append(first5, next5...)
+	expected = append(expected, first5...)
+	expected = append(expected, next5...)
+	assert.Equal(t, expected, m.Metrics())
+}
+
+// Verify that the order of points is preserved when there is a remainder
+// of points for the batch.
+//
+// ie, with a batch size of 5:
+//
+//     1 2 3 4 5 6 <-- order, failed points
+//     6 1 2 3 4 5 <-- order, after 1st write failure (1 2 3 4 5 was batch)
+//     1 2 3 4 5 6 <-- order, after 2nd write failure, (6 was batch)
+//
+func TestRunningOutputWriteFailOrder3(t *testing.T) {
+	conf := &OutputConfig{
+		Filter: Filter{
+			IsActive: false,
+		},
+	}
+
+	m := &mockOutput{}
+	m.failWrite = true
+	ro := NewRunningOutput("test", m, conf, 5, 1000)
+
+	// add 5 metrics
+	for _, metric := range first5 {
+		ro.AddMetric(metric)
+	}
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	// Write fails
+	err := ro.Write()
+	require.Error(t, err)
+	// no successful flush yet
+	assert.Len(t, m.Metrics(), 0)
+
+	// add and attempt to write a single metric:
+	ro.AddMetric(next5[0])
+	err = ro.Write()
+	require.Error(t, err)
+
+	// unset fail and write metrics
+	m.failWrite = false
+	err = ro.Write()
+	require.NoError(t, err)
+
+	// Verify that 6 metrics were written
+	assert.Len(t, m.Metrics(), 6)
+	// Verify that they are in order
+	expected := append(first5, next5[0])
+	assert.Equal(t, expected, m.Metrics())
+}
+
 type mockOutput struct {
 	sync.Mutex
 
@@ -263,3 +538,31 @@ func (m *mockOutput) Metrics() []telegraf.Metric {
 	defer m.Unlock()
 	return m.metrics
 }
+
+type perfOutput struct {
+	// if true, mock a write failure
+	failWrite bool
+}
+
+func (m *perfOutput) Connect() error {
+	return nil
+}
+
+func (m *perfOutput) Close() error {
+	return nil
+}
+
+func (m *perfOutput) Description() string {
+	return ""
+}
+
+func (m *perfOutput) SampleConfig() string {
+	return ""
+}
+
+func (m *perfOutput) Write(metrics []telegraf.Metric) error {
+	if m.failWrite {
+		return fmt.Errorf("Failed Write!")
+	}
+	return nil
+}

metric.go

@@ -45,14 +45,9 @@ func NewMetric(
 	name string,
 	tags map[string]string,
 	fields map[string]interface{},
-	t ...time.Time,
+	t time.Time,
 ) (Metric, error) {
-	var T time.Time
+	pt, err := client.NewPoint(name, tags, fields, t)
-	if len(t) > 0 {
-		T = t[0]
-	}
-
-	pt, err := client.NewPoint(name, tags, fields, T)
 	if err != nil {
 		return nil, err
 	}

metric_test.go

@@ -51,23 +51,6 @@ func TestNewMetricString(t *testing.T) {
 	assert.Equal(t, lineProtoPrecision, m.PrecisionString("s"))
 }
 
-func TestNewMetricStringNoTime(t *testing.T) {
-	tags := map[string]string{
-		"host": "localhost",
-	}
-	fields := map[string]interface{}{
-		"usage_idle": float64(99),
-	}
-	m, err := NewMetric("cpu", tags, fields)
-	assert.NoError(t, err)
-
-	lineProto := fmt.Sprintf("cpu,host=localhost usage_idle=99")
-	assert.Equal(t, lineProto, m.String())
-
-	lineProtoPrecision := fmt.Sprintf("cpu,host=localhost usage_idle=99")
-	assert.Equal(t, lineProtoPrecision, m.PrecisionString("s"))
-}
-
 func TestNewMetricFailNaN(t *testing.T) {
 	now := time.Now()
 

plugins/inputs/aerospike/aerospike.go

@@ -1,104 +1,19 @@
 package aerospike
 
 import (
-	"bytes"
-	"encoding/binary"
-	"fmt"
-	"github.com/influxdata/telegraf"
-	"github.com/influxdata/telegraf/plugins/inputs"
 	"net"
 	"strconv"
 	"strings"
 	"sync"
+	"time"
+
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/internal/errchan"
+	"github.com/influxdata/telegraf/plugins/inputs"
+
+	as "github.com/sparrc/aerospike-client-go"
 )
 
-const (
-	MSG_HEADER_SIZE = 8
-	MSG_TYPE        = 1 // Info is 1
-	MSG_VERSION     = 2
-)
-
-var (
-	STATISTICS_COMMAND = []byte("statistics\n")
-	NAMESPACES_COMMAND = []byte("namespaces\n")
-)
-
-type aerospikeMessageHeader struct {
-	Version uint8
-	Type    uint8
-	DataLen [6]byte
-}
-
-type aerospikeMessage struct {
-	aerospikeMessageHeader
-	Data []byte
-}
-
-// Taken from aerospike-client-go/types/message.go
-func (msg *aerospikeMessage) Serialize() []byte {
-	msg.DataLen = msgLenToBytes(int64(len(msg.Data)))
-	buf := bytes.NewBuffer([]byte{})
-	binary.Write(buf, binary.BigEndian, msg.aerospikeMessageHeader)
-	binary.Write(buf, binary.BigEndian, msg.Data[:])
-	return buf.Bytes()
-}
-
-type aerospikeInfoCommand struct {
-	msg *aerospikeMessage
-}
-
-// Taken from aerospike-client-go/info.go
-func (nfo *aerospikeInfoCommand) parseMultiResponse() (map[string]string, error) {
-	responses := make(map[string]string)
-	offset := int64(0)
-	begin := int64(0)
-
-	dataLen := int64(len(nfo.msg.Data))
-
-	// Create reusable StringBuilder for performance.
-	for offset < dataLen {
-		b := nfo.msg.Data[offset]
-
-		if b == '\t' {
-			name := nfo.msg.Data[begin:offset]
-			offset++
-			begin = offset
-
-			// Parse field value.
-			for offset < dataLen {
-				if nfo.msg.Data[offset] == '\n' {
-					break
-				}
-				offset++
-			}
-
-			if offset > begin {
-				value := nfo.msg.Data[begin:offset]
-				responses[string(name)] = string(value)
-			} else {
-				responses[string(name)] = ""
-			}
-			offset++
-			begin = offset
-		} else if b == '\n' {
-			if offset > begin {
-				name := nfo.msg.Data[begin:offset]
-				responses[string(name)] = ""
-			}
-			offset++
-			begin = offset
-		} else {
-			offset++
-		}
-	}
-
-	if offset > begin {
-		name := nfo.msg.Data[begin:offset]
-		responses[string(name)] = ""
-	}
-	return responses, nil
-}
-
 type Aerospike struct {
 	Servers []string
 }
@@ -115,7 +30,7 @@ func (a *Aerospike) SampleConfig() string {
 }
 
 func (a *Aerospike) Description() string {
-	return "Read stats from an aerospike server"
+	return "Read stats from aerospike server(s)"
 }
 
 func (a *Aerospike) Gather(acc telegraf.Accumulator) error {
@@ -124,214 +39,90 @@ func (a *Aerospike) Gather(acc telegraf.Accumulator) error {
 	}
 
 	var wg sync.WaitGroup
-
+	errChan := errchan.New(len(a.Servers))
-	var outerr error
+	wg.Add(len(a.Servers))
-
 	for _, server := range a.Servers {
-		wg.Add(1)
+		go func(serv string) {
-		go func(server string) {
 			defer wg.Done()
-			outerr = a.gatherServer(server, acc)
+			errChan.C <- a.gatherServer(serv, acc)
 		}(server)
 	}
 
 	wg.Wait()
-	return outerr
+	return errChan.Error()
 }
 
-func (a *Aerospike) gatherServer(host string, acc telegraf.Accumulator) error {
+func (a *Aerospike) gatherServer(hostport string, acc telegraf.Accumulator) error {
-	aerospikeInfo, err := getMap(STATISTICS_COMMAND, host)
+	host, port, err := net.SplitHostPort(hostport)
 	if err != nil {
-		return fmt.Errorf("Aerospike info failed: %s", err)
+		return err
 	}
-	readAerospikeStats(aerospikeInfo, acc, host, "")
+
-	namespaces, err := getList(NAMESPACES_COMMAND, host)
+	iport, err := strconv.Atoi(port)
 	if err != nil {
-		return fmt.Errorf("Aerospike namespace list failed: %s", err)
+		iport = 3000
 	}
-	for ix := range namespaces {
+
-		nsInfo, err := getMap([]byte("namespace/"+namespaces[ix]+"\n"), host)
+	c, err := as.NewClient(host, iport)
-		if err != nil {
+	if err != nil {
-			return fmt.Errorf("Aerospike namespace '%s' query failed: %s", namespaces[ix], err)
+		return err
+	}
+	defer c.Close()
+
+	nodes := c.GetNodes()
+	for _, n := range nodes {
+		tags := map[string]string{
+			"node_name":      n.GetName(),
+			"aerospike_host": hostport,
+		}
+		fields := make(map[string]interface{})
+		stats, err := as.RequestNodeStats(n)
+		if err != nil {
+			return err
+		}
+		for k, v := range stats {
+			if iv, err := strconv.ParseInt(v, 10, 64); err == nil {
+				fields[strings.Replace(k, "-", "_", -1)] = iv
+			}
+		}
+		acc.AddFields("aerospike_node", fields, tags, time.Now())
+
+		info, err := as.RequestNodeInfo(n, "namespaces")
+		if err != nil {
+			return err
+		}
+		namespaces := strings.Split(info["namespaces"], ";")
+
+		for _, namespace := range namespaces {
+			nTags := copyTags(tags)
+			nTags["namespace"] = namespace
+			nFields := make(map[string]interface{})
+			info, err := as.RequestNodeInfo(n, "namespace/"+namespace)
+			if err != nil {
+				continue
+			}
+			stats := strings.Split(info["namespace/"+namespace], ";")
+			for _, stat := range stats {
+				parts := strings.Split(stat, "=")
+				if len(parts) < 2 {
+					continue
+				}
+				if iv, err := strconv.ParseInt(parts[1], 10, 64); err == nil {
+					nFields[strings.Replace(parts[0], "-", "_", -1)] = iv
+				}
+			}
+			acc.AddFields("aerospike_namespace", nFields, nTags, time.Now())
 		}
-		readAerospikeStats(nsInfo, acc, host, namespaces[ix])
 	}
 	return nil
 }
 
-func getMap(key []byte, host string) (map[string]string, error) {
+func copyTags(m map[string]string) map[string]string {
-	data, err := get(key, host)
+	out := make(map[string]string)
-	if err != nil {
+	for k, v := range m {
-		return nil, fmt.Errorf("Failed to get data: %s", err)
+		out[k] = v
 	}
-	parsed, err := unmarshalMapInfo(data, string(key))
+	return out
-	if err != nil {
-		return nil, fmt.Errorf("Failed to unmarshal data: %s", err)
-	}
-
-	return parsed, nil
-}
-
-func getList(key []byte, host string) ([]string, error) {
-	data, err := get(key, host)
-	if err != nil {
-		return nil, fmt.Errorf("Failed to get data: %s", err)
-	}
-	parsed, err := unmarshalListInfo(data, string(key))
-	if err != nil {
-		return nil, fmt.Errorf("Failed to unmarshal data: %s", err)
-	}
-
-	return parsed, nil
-}
-
-func get(key []byte, host string) (map[string]string, error) {
-	var err error
-	var data map[string]string
-
-	asInfo := &aerospikeInfoCommand{
-		msg: &aerospikeMessage{
-			aerospikeMessageHeader: aerospikeMessageHeader{
-				Version: uint8(MSG_VERSION),
-				Type:    uint8(MSG_TYPE),
-				DataLen: msgLenToBytes(int64(len(key))),
-			},
-			Data: key,
-		},
-	}
-
-	cmd := asInfo.msg.Serialize()
-	addr, err := net.ResolveTCPAddr("tcp", host)
-	if err != nil {
-		return data, fmt.Errorf("Lookup failed for '%s': %s", host, err)
-	}
-
-	conn, err := net.DialTCP("tcp", nil, addr)
-	if err != nil {
-		return data, fmt.Errorf("Connection failed for '%s': %s", host, err)
-	}
-	defer conn.Close()
-
-	_, err = conn.Write(cmd)
-	if err != nil {
-		return data, fmt.Errorf("Failed to send to '%s': %s", host, err)
-	}
-
-	msgHeader := bytes.NewBuffer(make([]byte, MSG_HEADER_SIZE))
-	_, err = readLenFromConn(conn, msgHeader.Bytes(), MSG_HEADER_SIZE)
-	if err != nil {
-		return data, fmt.Errorf("Failed to read header: %s", err)
-	}
-	err = binary.Read(msgHeader, binary.BigEndian, &asInfo.msg.aerospikeMessageHeader)
-	if err != nil {
-		return data, fmt.Errorf("Failed to unmarshal header: %s", err)
-	}
-
-	msgLen := msgLenFromBytes(asInfo.msg.aerospikeMessageHeader.DataLen)
-
-	if int64(len(asInfo.msg.Data)) != msgLen {
-		asInfo.msg.Data = make([]byte, msgLen)
-	}
-
-	_, err = readLenFromConn(conn, asInfo.msg.Data, len(asInfo.msg.Data))
-	if err != nil {
-		return data, fmt.Errorf("Failed to read from connection to '%s': %s", host, err)
-	}
-
-	data, err = asInfo.parseMultiResponse()
-	if err != nil {
-		return data, fmt.Errorf("Failed to parse response from '%s': %s", host, err)
-	}
-
-	return data, err
-}
-
-func readAerospikeStats(
-	stats map[string]string,
-	acc telegraf.Accumulator,
-	host string,
-	namespace string,
-) {
-	fields := make(map[string]interface{})
-	tags := map[string]string{
-		"aerospike_host": host,
-		"namespace":      "_service",
-	}
-
-	if namespace != "" {
-		tags["namespace"] = namespace
-	}
-	for key, value := range stats {
-		// We are going to ignore all string based keys
-		val, err := strconv.ParseInt(value, 10, 64)
-		if err == nil {
-			if strings.Contains(key, "-") {
-				key = strings.Replace(key, "-", "_", -1)
-			}
-			fields[key] = val
-		}
-	}
-	acc.AddFields("aerospike", fields, tags)
-}
-
-func unmarshalMapInfo(infoMap map[string]string, key string) (map[string]string, error) {
-	key = strings.TrimSuffix(key, "\n")
-	res := map[string]string{}
-
-	v, exists := infoMap[key]
-	if !exists {
-		return res, fmt.Errorf("Key '%s' missing from info", key)
-	}
-
-	values := strings.Split(v, ";")
-	for i := range values {
-		kv := strings.Split(values[i], "=")
-		if len(kv) > 1 {
-			res[kv[0]] = kv[1]
-		}
-	}
-
-	return res, nil
-}
-
-func unmarshalListInfo(infoMap map[string]string, key string) ([]string, error) {
-	key = strings.TrimSuffix(key, "\n")
-
-	v, exists := infoMap[key]
-	if !exists {
-		return []string{}, fmt.Errorf("Key '%s' missing from info", key)
-	}
-
-	values := strings.Split(v, ";")
-	return values, nil
-}
-
-func readLenFromConn(c net.Conn, buffer []byte, length int) (total int, err error) {
-	var r int
-	for total < length {
-		r, err = c.Read(buffer[total:length])
-		total += r
-		if err != nil {
-			break
-		}
-	}
-	return
-}
-
-// Taken from aerospike-client-go/types/message.go
-func msgLenToBytes(DataLen int64) [6]byte {
-	b := make([]byte, 8)
-	binary.BigEndian.PutUint64(b, uint64(DataLen))
-	res := [6]byte{}
-	copy(res[:], b[2:])
-	return res
-}
-
-// Taken from aerospike-client-go/types/message.go
-func msgLenFromBytes(buf [6]byte) int64 {
-	nbytes := append([]byte{0, 0}, buf[:]...)
-	DataLen := binary.BigEndian.Uint64(nbytes)
-	return int64(DataLen)
 }
 
 func init() {

plugins/inputs/aerospike/aerospike_test.go

@@ -1,7 +1,6 @@
 package aerospike
 
 import (
-	"reflect"
 	"testing"
 
 	"github.com/influxdata/telegraf/testutil"
@@ -23,96 +22,29 @@ func TestAerospikeStatistics(t *testing.T) {
 	err := a.Gather(&acc)
 	require.NoError(t, err)
 
-	// Only use a few of the metrics
+	assert.True(t, acc.HasMeasurement("aerospike_node"))
-	asMetrics := []string{
+	assert.True(t, acc.HasMeasurement("aerospike_namespace"))
-		"transactions",
+	assert.True(t, acc.HasIntField("aerospike_node", "batch_error"))
-		"stat_write_errs",
-		"stat_read_reqs",
-		"stat_write_reqs",
-	}
-
-	for _, metric := range asMetrics {
-		assert.True(t, acc.HasIntField("aerospike", metric), metric)
-	}
-
 }
 
-func TestAerospikeMsgLenFromToBytes(t *testing.T) {
+func TestAerospikeStatisticsPartialErr(t *testing.T) {
-	var i int64 = 8
+	if testing.Short() {
-	assert.True(t, i == msgLenFromBytes(msgLenToBytes(i)))
+		t.Skip("Skipping integration test in short mode")
-}
+	}
+
+	a := &Aerospike{
+		Servers: []string{
+			testutil.GetLocalHost() + ":3000",
+			testutil.GetLocalHost() + ":9999",
+		},
+	}
 
-func TestReadAerospikeStatsNoNamespace(t *testing.T) {
-	// Also test for re-writing
 	var acc testutil.Accumulator
-	stats := map[string]string{
-		"stat-write-errs": "12345",
-		"stat_read_reqs":  "12345",
-	}
-	readAerospikeStats(stats, &acc, "host1", "")
 
-	fields := map[string]interface{}{
+	err := a.Gather(&acc)
-		"stat_write_errs": int64(12345),
+	require.Error(t, err)
-		"stat_read_reqs":  int64(12345),
+
-	}
+	assert.True(t, acc.HasMeasurement("aerospike_node"))
-	tags := map[string]string{
+	assert.True(t, acc.HasMeasurement("aerospike_namespace"))
-		"aerospike_host": "host1",
+	assert.True(t, acc.HasIntField("aerospike_node", "batch_error"))
-		"namespace":      "_service",
-	}
-	acc.AssertContainsTaggedFields(t, "aerospike", fields, tags)
-}
-
-func TestReadAerospikeStatsNamespace(t *testing.T) {
-	var acc testutil.Accumulator
-	stats := map[string]string{
-		"stat_write_errs": "12345",
-		"stat_read_reqs":  "12345",
-	}
-	readAerospikeStats(stats, &acc, "host1", "test")
-
-	fields := map[string]interface{}{
-		"stat_write_errs": int64(12345),
-		"stat_read_reqs":  int64(12345),
-	}
-	tags := map[string]string{
-		"aerospike_host": "host1",
-		"namespace":      "test",
-	}
-	acc.AssertContainsTaggedFields(t, "aerospike", fields, tags)
-}
-
-func TestAerospikeUnmarshalList(t *testing.T) {
-	i := map[string]string{
-		"test": "one;two;three",
-	}
-
-	expected := []string{"one", "two", "three"}
-
-	list, err := unmarshalListInfo(i, "test2")
-	assert.True(t, err != nil)
-
-	list, err = unmarshalListInfo(i, "test")
-	assert.True(t, err == nil)
-	equal := true
-	for ix := range expected {
-		if list[ix] != expected[ix] {
-			equal = false
-			break
-		}
-	}
-	assert.True(t, equal)
-}
-
-func TestAerospikeUnmarshalMap(t *testing.T) {
-	i := map[string]string{
-		"test": "key1=value1;key2=value2",
-	}
-
-	expected := map[string]string{
-		"key1": "value1",
-		"key2": "value2",
-	}
-	m, err := unmarshalMapInfo(i, "test")
-	assert.True(t, err == nil)
-	assert.True(t, reflect.DeepEqual(m, expected))
 }

plugins/inputs/all/all.go

@@ -4,6 +4,14 @@ import (
 	_ "github.com/influxdata/telegraf/plugins/inputs/aerospike"
 	_ "github.com/influxdata/telegraf/plugins/inputs/apache"
 	_ "github.com/influxdata/telegraf/plugins/inputs/bcache"
+	_ "github.com/influxdata/telegraf/plugins/inputs/cassandra"
+	_ "github.com/influxdata/telegraf/plugins/inputs/ceph"
+	_ "github.com/influxdata/telegraf/plugins/inputs/cgroup"
+	_ "github.com/influxdata/telegraf/plugins/inputs/chrony"
+	_ "github.com/influxdata/telegraf/plugins/inputs/cloudwatch"
+	_ "github.com/influxdata/telegraf/plugins/inputs/conntrack"
+	_ "github.com/influxdata/telegraf/plugins/inputs/consul"
+	_ "github.com/influxdata/telegraf/plugins/inputs/couchbase"
 	_ "github.com/influxdata/telegraf/plugins/inputs/couchdb"
 	_ "github.com/influxdata/telegraf/plugins/inputs/disque"
 	_ "github.com/influxdata/telegraf/plugins/inputs/dns_query"
@@ -11,13 +19,17 @@ import (
 	_ "github.com/influxdata/telegraf/plugins/inputs/dovecot"
 	_ "github.com/influxdata/telegraf/plugins/inputs/elasticsearch"
 	_ "github.com/influxdata/telegraf/plugins/inputs/exec"
-	_ "github.com/influxdata/telegraf/plugins/inputs/github_webhooks"
+	_ "github.com/influxdata/telegraf/plugins/inputs/filestat"
+	_ "github.com/influxdata/telegraf/plugins/inputs/graylog"
 	_ "github.com/influxdata/telegraf/plugins/inputs/haproxy"
+	_ "github.com/influxdata/telegraf/plugins/inputs/http_response"
 	_ "github.com/influxdata/telegraf/plugins/inputs/httpjson"
 	_ "github.com/influxdata/telegraf/plugins/inputs/influxdb"
+	_ "github.com/influxdata/telegraf/plugins/inputs/ipmi_sensor"
 	_ "github.com/influxdata/telegraf/plugins/inputs/jolokia"
 	_ "github.com/influxdata/telegraf/plugins/inputs/kafka_consumer"
 	_ "github.com/influxdata/telegraf/plugins/inputs/leofs"
+	_ "github.com/influxdata/telegraf/plugins/inputs/logparser"
 	_ "github.com/influxdata/telegraf/plugins/inputs/lustre2"
 	_ "github.com/influxdata/telegraf/plugins/inputs/mailchimp"
 	_ "github.com/influxdata/telegraf/plugins/inputs/memcached"
@@ -29,11 +41,14 @@ import (
 	_ "github.com/influxdata/telegraf/plugins/inputs/net_response"
 	_ "github.com/influxdata/telegraf/plugins/inputs/nginx"
 	_ "github.com/influxdata/telegraf/plugins/inputs/nsq"
+	_ "github.com/influxdata/telegraf/plugins/inputs/nsq_consumer"
+	_ "github.com/influxdata/telegraf/plugins/inputs/nstat"
 	_ "github.com/influxdata/telegraf/plugins/inputs/ntpq"
 	_ "github.com/influxdata/telegraf/plugins/inputs/passenger"
 	_ "github.com/influxdata/telegraf/plugins/inputs/phpfpm"
 	_ "github.com/influxdata/telegraf/plugins/inputs/ping"
 	_ "github.com/influxdata/telegraf/plugins/inputs/postgresql"
+	_ "github.com/influxdata/telegraf/plugins/inputs/postgresql_extensible"
 	_ "github.com/influxdata/telegraf/plugins/inputs/powerdns"
 	_ "github.com/influxdata/telegraf/plugins/inputs/procstat"
 	_ "github.com/influxdata/telegraf/plugins/inputs/prometheus"
@@ -47,11 +62,15 @@ import (
 	_ "github.com/influxdata/telegraf/plugins/inputs/snmp"
 	_ "github.com/influxdata/telegraf/plugins/inputs/sqlserver"
 	_ "github.com/influxdata/telegraf/plugins/inputs/statsd"
+	_ "github.com/influxdata/telegraf/plugins/inputs/sysstat"
 	_ "github.com/influxdata/telegraf/plugins/inputs/system"
+	_ "github.com/influxdata/telegraf/plugins/inputs/tail"
 	_ "github.com/influxdata/telegraf/plugins/inputs/tcp_listener"
 	_ "github.com/influxdata/telegraf/plugins/inputs/trig"
 	_ "github.com/influxdata/telegraf/plugins/inputs/twemproxy"
 	_ "github.com/influxdata/telegraf/plugins/inputs/udp_listener"
+	_ "github.com/influxdata/telegraf/plugins/inputs/varnish"
+	_ "github.com/influxdata/telegraf/plugins/inputs/webhooks"
 	_ "github.com/influxdata/telegraf/plugins/inputs/win_perf_counters"
 	_ "github.com/influxdata/telegraf/plugins/inputs/zfs"
 	_ "github.com/influxdata/telegraf/plugins/inputs/zookeeper"

plugins/inputs/apache/README.md

@@ -1,7 +1,7 @@
 # Telegraf plugin: Apache
 
 #### Plugin arguments:
-- **urls** []string: List of apache-status URLs to collect from.
+- **urls** []string: List of apache-status URLs to collect from. Default is "http://localhost/server-status?auto".
 
 #### Description
 

plugins/inputs/apache/apache.go

@@ -8,7 +8,6 @@ import (
 	"net/url"
 	"strconv"
 	"strings"
-	"sync"
 	"time"
 
 	"github.com/influxdata/telegraf"
@@ -21,6 +20,7 @@ type Apache struct {
 
 var sampleConfig = `
   ## An array of Apache status URI to gather stats.
+  ## Default is "http://localhost/server-status?auto".
   urls = ["http://localhost/server-status?auto"]
 `
 
@@ -33,8 +33,12 @@ func (n *Apache) Description() string {
 }
 
 func (n *Apache) Gather(acc telegraf.Accumulator) error {
-	var wg sync.WaitGroup
+	if len(n.Urls) == 0 {
+		n.Urls = []string{"http://localhost/server-status?auto"}
+	}
+
 	var outerr error
+	var errch = make(chan error)
 
 	for _, u := range n.Urls {
 		addr, err := url.Parse(u)
@@ -42,14 +46,17 @@ func (n *Apache) Gather(acc telegraf.Accumulator) error {
 			return fmt.Errorf("Unable to parse address '%s': %s", u, err)
 		}
 
-		wg.Add(1)
 		go func(addr *url.URL) {
-			defer wg.Done()
+			errch <- n.gatherUrl(addr, acc)
-			outerr = n.gatherUrl(addr, acc)
 		}(addr)
 	}
 
-	wg.Wait()
+	// Drain channel, waiting for all requests to finish and save last error.
+	for range n.Urls {
+		if err := <-errch; err != nil {
+			outerr = err
+		}
+	}
 
 	return outerr
 }

plugins/inputs/apache/apache_test.go

@@ -36,7 +36,8 @@ func TestHTTPApache(t *testing.T) {
 	defer ts.Close()
 
 	a := Apache{
-		Urls: []string{ts.URL},
+		// Fetch it 2 times to catch possible data races.
+		Urls: []string{ts.URL, ts.URL},
 	}
 
 	var acc testutil.Accumulator

plugins/inputs/cassandra/README.md

@@ -0,0 +1,125 @@
+# Telegraf plugin: Cassandra
+
+#### Plugin arguments:
+- **context** string: Context root used for jolokia url
+- **servers** []string: List of servers with the format "<user:passwd@><host>:port"
+- **metrics** []string: List of Jmx paths that identify mbeans attributes
+
+#### Description
+
+The Cassandra plugin collects Cassandra/JVM metrics exposed as MBean's attributes through jolokia REST endpoint. All metrics are collected for each server configured.
+
+See: https://jolokia.org/ and [Cassandra Documentation](http://docs.datastax.com/en/cassandra/3.x/cassandra/operations/monitoringCassandraTOC.html)
+
+# Measurements:
+Cassandra plugin produces one or more measurements for each metric configured, adding Server's name  as `host` tag. More than one measurement is generated when querying table metrics with a wildcard for the keyspace or table name.
+
+Given a configuration like:
+
+```toml
+[[inputs.cassandra]]
+  context = "/jolokia/read"
+  servers = [":8778"]
+  metrics = ["/java.lang:type=Memory/HeapMemoryUsage"]
+```
+
+The collected metrics will be:
+
+```
+javaMemory,host=myHost,mname=HeapMemoryUsage HeapMemoryUsage_committed=1040187392,HeapMemoryUsage_init=1050673152,HeapMemoryUsage_max=1040187392,HeapMemoryUsage_used=368155000 1459551767230567084
+```
+
+# Useful Metrics:
+
+Here is a list of metrics that might be useful to monitor your cassandra cluster. This was put together from multiple sources on the web.
+
+- [How to monitor Cassandra performance metrics](https://www.datadoghq.com/blog/how-to-monitor-cassandra-performance-metrics)
+- [Cassandra Documentation](http://docs.datastax.com/en/cassandra/3.x/cassandra/operations/monitoringCassandraTOC.html)
+
+####measurement = javaGarbageCollector
+
+- /java.lang:type=GarbageCollector,name=ConcurrentMarkSweep/CollectionTime       
+- /java.lang:type=GarbageCollector,name=ConcurrentMarkSweep/CollectionCount
+- /java.lang:type=GarbageCollector,name=ParNew/CollectionTime          
+- /java.lang:type=GarbageCollector,name=ParNew/CollectionCount
+
+####measurement = javaMemory
+
+- /java.lang:type=Memory/HeapMemoryUsage
+- /java.lang:type=Memory/NonHeapMemoryUsage
+
+####measurement = cassandraCache
+
+- /org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Hit
+- /org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Requests
+- /org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Entries
+- /org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Size 
+- /org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Capacity 
+- /org.apache.cassandra.metrics:type=Cache,scope=RowCache,name=Hit
+- /org.apache.cassandra.metrics:type=Cache,scope=RowCache,name=Requests     
+- /org.apache.cassandra.metrics:type=Cache,scope=RowCache,name=Entries
+- /org.apache.cassandra.metrics:type=Cache,scope=RowCache,name=Size
+- /org.apache.cassandra.metrics:type=Cache,scope=RowCache,name=Capacity
+
+####measurement = cassandraClient
+
+- /org.apache.cassandra.metrics:type=Client,name=connectedNativeClients
+
+####measurement = cassandraClientRequest
+
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=TotalLatency        
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=TotalLatency         
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Latency       
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency        
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Timeouts       
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Timeouts
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Unavailables        
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Unavailables         
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Failures         
+- /org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Failures 
+
+####measurement = cassandraCommitLog
+
+- /org.apache.cassandra.metrics:type=CommitLog,name=PendingTasks        
+- /org.apache.cassandra.metrics:type=CommitLog,name=TotalCommitLogSize
+
+####measurement = cassandraCompaction
+
+- /org.apache.cassandra.metrics:type=Compaction,name=CompletedTask
+- /org.apache.cassandra.metrics:type=Compaction,name=PendingTasks    
+- /org.apache.cassandra.metrics:type=Compaction,name=TotalCompactionsCompleted
+- /org.apache.cassandra.metrics:type=Compaction,name=BytesCompacted
+
+####measurement = cassandraStorage
+
+- /org.apache.cassandra.metrics:type=Storage,name=Load
+-  /org.apache.cassandra.metrics:type=Storage,name=Exceptions 
+
+####measurement = cassandraTable
+Using wildcards for "keyspace" and "scope" can create a lot of series as metrics will be reported for every table and keyspace including internal system tables. Specify a keyspace name and/or a table name to limit them.
+
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=LiveDiskSpaceUsed
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=TotalDiskSpaceUsed
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=ReadLatency
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=CoordinatorReadLatency 
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=WriteLatency 
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=ReadTotalLatency 
+- /org.apache.cassandra.metrics:type=Table,keyspace=\*,scope=\*,name=WriteTotalLatency              
+
+
+####measurement = cassandraThreadPools
+
+- /org.apache.cassandra.metrics:type=ThreadPools,path=internal,scope=CompactionExecutor,name=ActiveTasks          
+- /org.apache.cassandra.metrics:type=ThreadPools,path=internal,scope=AntiEntropyStage,name=ActiveTasks         
+-  /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=CounterMutationStage,name=PendingTasks
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=CounterMutationStage,name=CurrentlyBlockedTasks        
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=MutationStage,name=PendingTasks
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=MutationStage,name=CurrentlyBlockedTasks        
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadRepairStage,name=PendingTasks
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadRepairStage,name=CurrentlyBlockedTasks        
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=PendingTasks
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=CurrentlyBlockedTasks          
+-  /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=RequestResponseStage,name=PendingTasks                  
+- /org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=RequestResponseStage,name=CurrentlyBlockedTasks 
+
+

plugins/inputs/cassandra/cassandra.go

@@ -0,0 +1,309 @@
+package cassandra
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/plugins/inputs"
+	"io/ioutil"
+	"log"
+	"net/http"
+	"net/url"
+	"strings"
+)
+
+type JolokiaClient interface {
+	MakeRequest(req *http.Request) (*http.Response, error)
+}
+
+type JolokiaClientImpl struct {
+	client *http.Client
+}
+
+func (c JolokiaClientImpl) MakeRequest(req *http.Request) (*http.Response, error) {
+	return c.client.Do(req)
+}
+
+type Cassandra struct {
+	jClient JolokiaClient
+	Context string
+	Servers []string
+	Metrics []string
+}
+
+type javaMetric struct {
+	host   string
+	metric string
+	acc    telegraf.Accumulator
+}
+
+type cassandraMetric struct {
+	host   string
+	metric string
+	acc    telegraf.Accumulator
+}
+
+type jmxMetric interface {
+	addTagsFields(out map[string]interface{})
+}
+
+func newJavaMetric(host string, metric string,
+	acc telegraf.Accumulator) *javaMetric {
+	return &javaMetric{host: host, metric: metric, acc: acc}
+}
+
+func newCassandraMetric(host string, metric string,
+	acc telegraf.Accumulator) *cassandraMetric {
+	return &cassandraMetric{host: host, metric: metric, acc: acc}
+}
+
+func addValuesAsFields(values map[string]interface{}, fields map[string]interface{},
+	mname string) {
+	for k, v := range values {
+		if v != nil {
+			fields[mname+"_"+k] = v
+		}
+	}
+}
+
+func parseJmxMetricRequest(mbean string) map[string]string {
+	tokens := make(map[string]string)
+	classAndPairs := strings.Split(mbean, ":")
+	if classAndPairs[0] == "org.apache.cassandra.metrics" {
+		tokens["class"] = "cassandra"
+	} else if classAndPairs[0] == "java.lang" {
+		tokens["class"] = "java"
+	} else {
+		return tokens
+	}
+	pairs := strings.Split(classAndPairs[1], ",")
+	for _, pair := range pairs {
+		p := strings.Split(pair, "=")
+		tokens[p[0]] = p[1]
+	}
+	return tokens
+}
+
+func addTokensToTags(tokens map[string]string, tags map[string]string) {
+	for k, v := range tokens {
+		if k == "name" {
+			tags["mname"] = v // name seems to a reserved word in influxdb
+		} else if k == "class" || k == "type" {
+			continue // class and type are used in the metric name
+		} else {
+			tags[k] = v
+		}
+	}
+}
+
+func (j javaMetric) addTagsFields(out map[string]interface{}) {
+	tags := make(map[string]string)
+	fields := make(map[string]interface{})
+
+	a := out["request"].(map[string]interface{})
+	attribute := a["attribute"].(string)
+	mbean := a["mbean"].(string)
+
+	tokens := parseJmxMetricRequest(mbean)
+	addTokensToTags(tokens, tags)
+	tags["cassandra_host"] = j.host
+
+	if _, ok := tags["mname"]; !ok {
+		//Queries for a single value will not return a "name" tag in the response.
+		tags["mname"] = attribute
+	}
+
+	if values, ok := out["value"]; ok {
+		switch t := values.(type) {
+		case map[string]interface{}:
+			addValuesAsFields(values.(map[string]interface{}), fields, attribute)
+		case interface{}:
+			fields[attribute] = t
+		}
+		j.acc.AddFields(tokens["class"]+tokens["type"], fields, tags)
+	} else {
+		fmt.Printf("Missing key 'value' in '%s' output response\n%v\n",
+			j.metric, out)
+	}
+}
+
+func addCassandraMetric(mbean string, c cassandraMetric,
+	values map[string]interface{}) {
+
+	tags := make(map[string]string)
+	fields := make(map[string]interface{})
+	tokens := parseJmxMetricRequest(mbean)
+	addTokensToTags(tokens, tags)
+	tags["cassandra_host"] = c.host
+	addValuesAsFields(values, fields, tags["mname"])
+	c.acc.AddFields(tokens["class"]+tokens["type"], fields, tags)
+
+}
+
+func (c cassandraMetric) addTagsFields(out map[string]interface{}) {
+
+	r := out["request"]
+
+	tokens := parseJmxMetricRequest(r.(map[string]interface{})["mbean"].(string))
+	// Requests with wildcards for keyspace or table names will return nested
+	// maps in the json response
+	if (tokens["type"] == "Table" || tokens["type"] == "ColumnFamily") && (tokens["keyspace"] == "*" ||
+		tokens["scope"] == "*") {
+		if valuesMap, ok := out["value"]; ok {
+			for k, v := range valuesMap.(map[string]interface{}) {
+				addCassandraMetric(k, c, v.(map[string]interface{}))
+			}
+		} else {
+			fmt.Printf("Missing key 'value' in '%s' output response\n%v\n",
+				c.metric, out)
+			return
+		}
+	} else {
+		if values, ok := out["value"]; ok {
+			addCassandraMetric(r.(map[string]interface{})["mbean"].(string),
+				c, values.(map[string]interface{}))
+		} else {
+			fmt.Printf("Missing key 'value' in '%s' output response\n%v\n",
+				c.metric, out)
+			return
+		}
+	}
+}
+
+func (j *Cassandra) SampleConfig() string {
+	return `
+  # This is the context root used to compose the jolokia url
+  context = "/jolokia/read"
+  ## List of cassandra servers exposing jolokia read service
+  servers = ["myuser:mypassword@10.10.10.1:8778","10.10.10.2:8778",":8778"]
+  ## List of metrics collected on above servers
+  ## Each metric consists of a jmx path.
+  ## This will collect all heap memory usage metrics from the jvm and
+  ## ReadLatency metrics for all keyspaces and tables.
+  ## "type=Table" in the query works with Cassandra3.0. Older versions might
+  ## need to use "type=ColumnFamily"
+  metrics  = [
+    "/java.lang:type=Memory/HeapMemoryUsage",
+    "/org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=ReadLatency"
+  ]
+`
+}
+
+func (j *Cassandra) Description() string {
+	return "Read Cassandra metrics through Jolokia"
+}
+
+func (j *Cassandra) getAttr(requestUrl *url.URL) (map[string]interface{}, error) {
+	// Create + send request
+	req, err := http.NewRequest("GET", requestUrl.String(), nil)
+	if err != nil {
+		return nil, err
+	}
+
+	resp, err := j.jClient.MakeRequest(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	// Process response
+	if resp.StatusCode != http.StatusOK {
+		err = fmt.Errorf("Response from url \"%s\" has status code %d (%s), expected %d (%s)",
+			requestUrl,
+			resp.StatusCode,
+			http.StatusText(resp.StatusCode),
+			http.StatusOK,
+			http.StatusText(http.StatusOK))
+		return nil, err
+	}
+
+	// read body
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	// Unmarshal json
+	var jsonOut map[string]interface{}
+	if err = json.Unmarshal([]byte(body), &jsonOut); err != nil {
+		return nil, errors.New("Error decoding JSON response")
+	}
+
+	return jsonOut, nil
+}
+
+func parseServerTokens(server string) map[string]string {
+	serverTokens := make(map[string]string)
+
+	hostAndUser := strings.Split(server, "@")
+	hostPort := ""
+	userPasswd := ""
+	if len(hostAndUser) == 2 {
+		hostPort = hostAndUser[1]
+		userPasswd = hostAndUser[0]
+	} else {
+		hostPort = hostAndUser[0]
+	}
+	hostTokens := strings.Split(hostPort, ":")
+	serverTokens["host"] = hostTokens[0]
+	serverTokens["port"] = hostTokens[1]
+
+	if userPasswd != "" {
+		userTokens := strings.Split(userPasswd, ":")
+		serverTokens["user"] = userTokens[0]
+		serverTokens["passwd"] = userTokens[1]
+	}
+	return serverTokens
+}
+
+func (c *Cassandra) Gather(acc telegraf.Accumulator) error {
+	context := c.Context
+	servers := c.Servers
+	metrics := c.Metrics
+
+	for _, server := range servers {
+		for _, metric := range metrics {
+			serverTokens := parseServerTokens(server)
+
+			var m jmxMetric
+			if strings.HasPrefix(metric, "/java.lang:") {
+				m = newJavaMetric(serverTokens["host"], metric, acc)
+			} else if strings.HasPrefix(metric,
+				"/org.apache.cassandra.metrics:") {
+				m = newCassandraMetric(serverTokens["host"], metric, acc)
+			} else {
+				// unsupported metric type
+				log.Printf("Unsupported Cassandra metric [%s], skipping",
+					metric)
+				continue
+			}
+
+			// Prepare URL
+			requestUrl, err := url.Parse("http://" + serverTokens["host"] + ":" +
+				serverTokens["port"] + context + metric)
+			if err != nil {
+				return err
+			}
+			if serverTokens["user"] != "" && serverTokens["passwd"] != "" {
+				requestUrl.User = url.UserPassword(serverTokens["user"],
+					serverTokens["passwd"])
+			}
+			fmt.Printf("host %s url %s\n", serverTokens["host"], requestUrl)
+
+			out, err := c.getAttr(requestUrl)
+			if out["status"] != 200.0 {
+				fmt.Printf("URL returned with status %v\n", out["status"])
+				continue
+			}
+			m.addTagsFields(out)
+		}
+	}
+	return nil
+}
+
+func init() {
+	inputs.Add("cassandra", func() telegraf.Input {
+		return &Cassandra{jClient: &JolokiaClientImpl{client: &http.Client{}}}
+	})
+}

plugins/inputs/cassandra/cassandra_test.go

@@ -0,0 +1,286 @@
+package cassandra
+
+import (
+	_ "fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/influxdata/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	_ "github.com/stretchr/testify/require"
+)
+
+const validJavaMultiValueJSON = `
+{
+  "request":{
+    "mbean":"java.lang:type=Memory",
+    "attribute":"HeapMemoryUsage",
+    "type":"read"
+  },
+  "value":{
+    "init":67108864,
+    "committed":456130560,
+    "max":477626368,
+    "used":203288528
+  },
+  "timestamp":1446129191,
+  "status":200
+}`
+
+const validCassandraMultiValueJSON = `
+{
+	"request": {
+		"mbean": "org.apache.cassandra.metrics:keyspace=test_keyspace1,name=ReadLatency,scope=test_table,type=Table",
+		"type": "read"},
+	"status": 200,
+	"timestamp": 1458089229,
+	"value": {
+		"999thPercentile": 20.0,
+		"99thPercentile": 10.0,
+		"Count": 400,
+		"DurationUnit": "microseconds",
+		"Max": 30.0,
+		"Mean": null,
+		"MeanRate": 3.0,
+		"Min": 1.0,
+		"RateUnit": "events/second",
+		"StdDev": null
+	}
+}`
+
+const validCassandraNestedMultiValueJSON = `
+{
+	"request": {
+		"mbean": "org.apache.cassandra.metrics:keyspace=test_keyspace1,name=ReadLatency,scope=*,type=Table",
+		"type": "read"},
+    "status": 200,
+    "timestamp": 1458089184,
+    "value": {
+		"org.apache.cassandra.metrics:keyspace=test_keyspace1,name=ReadLatency,scope=test_table1,type=Table":
+		{  	"999thPercentile": 1.0,
+			"Count": 100,
+			"DurationUnit": "microseconds",
+			"OneMinuteRate": 1.0,
+			"RateUnit": "events/second",
+			"StdDev": null
+		},
+		"org.apache.cassandra.metrics:keyspace=test_keyspace2,name=ReadLatency,scope=test_table2,type=Table":
+		{  	"999thPercentile": 2.0,
+			"Count": 200,
+			"DurationUnit": "microseconds",
+			"OneMinuteRate": 2.0,
+			"RateUnit": "events/second",
+			"StdDev": null
+		}
+	}
+}`
+
+const validSingleValueJSON = `
+{
+  "request":{
+    "path":"used",
+    "mbean":"java.lang:type=Memory",
+    "attribute":"HeapMemoryUsage",
+    "type":"read"
+  },
+  "value":209274376,
+  "timestamp":1446129256,
+  "status":200
+}`
+
+const validJavaMultiTypeJSON = `
+{
+   "request":{
+	   "mbean":"java.lang:name=ConcurrentMarkSweep,type=GarbageCollector",
+	   "attribute":"CollectionCount",
+	   "type":"read"
+   },
+   "value":1,
+   "timestamp":1459316570,
+   "status":200
+}`
+
+const invalidJSON = "I don't think this is JSON"
+
+const empty = ""
+
+var Servers = []string{"10.10.10.10:8778"}
+var AuthServers = []string{"user:passwd@10.10.10.10:8778"}
+var MultipleServers = []string{"10.10.10.10:8778", "10.10.10.11:8778"}
+var HeapMetric = "/java.lang:type=Memory/HeapMemoryUsage"
+var ReadLatencyMetric = "/org.apache.cassandra.metrics:type=Table,keyspace=test_keyspace1,scope=test_table,name=ReadLatency"
+var NestedReadLatencyMetric = "/org.apache.cassandra.metrics:type=Table,keyspace=test_keyspace1,scope=*,name=ReadLatency"
+var GarbageCollectorMetric1 = "/java.lang:type=GarbageCollector,name=ConcurrentMarkSweep/CollectionCount"
+var GarbageCollectorMetric2 = "/java.lang:type=GarbageCollector,name=ConcurrentMarkSweep/CollectionTime"
+var Context = "/jolokia/read"
+
+type jolokiaClientStub struct {
+	responseBody string
+	statusCode   int
+}
+
+func (c jolokiaClientStub) MakeRequest(req *http.Request) (*http.Response, error) {
+	resp := http.Response{}
+	resp.StatusCode = c.statusCode
+	resp.Body = ioutil.NopCloser(strings.NewReader(c.responseBody))
+	return &resp, nil
+}
+
+// Generates a pointer to an HttpJson object that uses a mock HTTP client.
+// Parameters:
+//     response  : Body of the response that the mock HTTP client should return
+//     statusCode: HTTP status code the mock HTTP client should return
+//
+// Returns:
+//     *HttpJson: Pointer to an HttpJson object that uses the generated mock HTTP client
+func genJolokiaClientStub(response string, statusCode int, servers []string, metrics []string) *Cassandra {
+	return &Cassandra{
+		jClient: jolokiaClientStub{responseBody: response, statusCode: statusCode},
+		Context: Context,
+		Servers: servers,
+		Metrics: metrics,
+	}
+}
+
+// Test that the proper values are ignored or collected for class=Java
+func TestHttpJsonJavaMultiValue(t *testing.T) {
+	cassandra := genJolokiaClientStub(validJavaMultiValueJSON, 200,
+		MultipleServers, []string{HeapMetric})
+
+	var acc testutil.Accumulator
+	acc.SetDebug(true)
+	err := cassandra.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 2, len(acc.Metrics))
+
+	fields := map[string]interface{}{
+		"HeapMemoryUsage_init":      67108864.0,
+		"HeapMemoryUsage_committed": 456130560.0,
+		"HeapMemoryUsage_max":       477626368.0,
+		"HeapMemoryUsage_used":      203288528.0,
+	}
+	tags1 := map[string]string{
+		"cassandra_host": "10.10.10.10",
+		"mname":          "HeapMemoryUsage",
+	}
+
+	tags2 := map[string]string{
+		"cassandra_host": "10.10.10.11",
+		"mname":          "HeapMemoryUsage",
+	}
+	acc.AssertContainsTaggedFields(t, "javaMemory", fields, tags1)
+	acc.AssertContainsTaggedFields(t, "javaMemory", fields, tags2)
+}
+
+func TestHttpJsonJavaMultiType(t *testing.T) {
+	cassandra := genJolokiaClientStub(validJavaMultiTypeJSON, 200, AuthServers, []string{GarbageCollectorMetric1, GarbageCollectorMetric2})
+
+	var acc testutil.Accumulator
+	acc.SetDebug(true)
+	err := cassandra.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 2, len(acc.Metrics))
+
+	fields := map[string]interface{}{
+		"CollectionCount": 1.0,
+	}
+
+	tags := map[string]string{
+		"cassandra_host": "10.10.10.10",
+		"mname":          "ConcurrentMarkSweep",
+	}
+	acc.AssertContainsTaggedFields(t, "javaGarbageCollector", fields, tags)
+}
+
+// Test that the proper values are ignored or collected
+func TestHttpJsonOn404(t *testing.T) {
+
+	jolokia := genJolokiaClientStub(validJavaMultiValueJSON, 404, Servers,
+		[]string{HeapMetric})
+
+	var acc testutil.Accumulator
+	err := jolokia.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 0, len(acc.Metrics))
+}
+
+// Test that the proper values are ignored or collected for class=Cassandra
+func TestHttpJsonCassandraMultiValue(t *testing.T) {
+	cassandra := genJolokiaClientStub(validCassandraMultiValueJSON, 200, Servers, []string{ReadLatencyMetric})
+
+	var acc testutil.Accumulator
+	err := cassandra.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 1, len(acc.Metrics))
+
+	fields := map[string]interface{}{
+		"ReadLatency_999thPercentile": 20.0,
+		"ReadLatency_99thPercentile":  10.0,
+		"ReadLatency_Count":           400.0,
+		"ReadLatency_DurationUnit":    "microseconds",
+		"ReadLatency_Max":             30.0,
+		"ReadLatency_MeanRate":        3.0,
+		"ReadLatency_Min":             1.0,
+		"ReadLatency_RateUnit":        "events/second",
+	}
+
+	tags := map[string]string{
+		"cassandra_host": "10.10.10.10",
+		"mname":          "ReadLatency",
+		"keyspace":       "test_keyspace1",
+		"scope":          "test_table",
+	}
+	acc.AssertContainsTaggedFields(t, "cassandraTable", fields, tags)
+}
+
+// Test that the proper values are ignored or collected for class=Cassandra with
+// nested values
+func TestHttpJsonCassandraNestedMultiValue(t *testing.T) {
+	cassandra := genJolokiaClientStub(validCassandraNestedMultiValueJSON, 200, Servers, []string{NestedReadLatencyMetric})
+
+	var acc testutil.Accumulator
+	acc.SetDebug(true)
+	err := cassandra.Gather(&acc)
+
+	assert.Nil(t, err)
+	assert.Equal(t, 2, len(acc.Metrics))
+
+	fields1 := map[string]interface{}{
+		"ReadLatency_999thPercentile": 1.0,
+		"ReadLatency_Count":           100.0,
+		"ReadLatency_DurationUnit":    "microseconds",
+		"ReadLatency_OneMinuteRate":   1.0,
+		"ReadLatency_RateUnit":        "events/second",
+	}
+
+	fields2 := map[string]interface{}{
+		"ReadLatency_999thPercentile": 2.0,
+		"ReadLatency_Count":           200.0,
+		"ReadLatency_DurationUnit":    "microseconds",
+		"ReadLatency_OneMinuteRate":   2.0,
+		"ReadLatency_RateUnit":        "events/second",
+	}
+
+	tags1 := map[string]string{
+		"cassandra_host": "10.10.10.10",
+		"mname":          "ReadLatency",
+		"keyspace":       "test_keyspace1",
+		"scope":          "test_table1",
+	}
+
+	tags2 := map[string]string{
+		"cassandra_host": "10.10.10.10",
+		"mname":          "ReadLatency",
+		"keyspace":       "test_keyspace2",
+		"scope":          "test_table2",
+	}
+
+	acc.AssertContainsTaggedFields(t, "cassandraTable", fields1, tags1)
+	acc.AssertContainsTaggedFields(t, "cassandraTable", fields2, tags2)
+}

plugins/inputs/ceph/README.md

@@ -0,0 +1,109 @@
+# Ceph Storage Input Plugin
+
+Collects performance metrics from the MON and OSD nodes in a Ceph storage cluster.  
+
+The plugin works by scanning the configured SocketDir for OSD and MON socket files.  When it finds
+a MON socket, it runs **ceph --admin-daemon $file perfcounters_dump**. For OSDs it runs **ceph --admin-daemon $file perf dump** 
+
+The resulting JSON is parsed and grouped into collections, based on top-level key.  Top-level keys are
+used as collection tags, and all sub-keys are flattened. For example:
+
+```
+ { 
+   "paxos": { 
+     "refresh": 9363435,
+     "refresh_latency": { 
+       "avgcount": 9363435,
+       "sum": 5378.794002000
+     }
+   }
+ }
+```
+
+Would be parsed into the following metrics, all of which would be tagged with collection=paxos:
+
+ - refresh = 9363435
+ - refresh_latency.avgcount: 9363435
+ - refresh_latency.sum: 5378.794002000
+
+
+### Configuration:
+
+```
+# Collects performance metrics from the MON and OSD nodes in a Ceph storage cluster.
+[[inputs.ceph]]
+  ## All configuration values are optional, defaults are shown below
+
+  ## location of ceph binary
+  ceph_binary = "/usr/bin/ceph"
+
+  ## directory in which to look for socket files
+  socket_dir = "/var/run/ceph"
+
+  ## prefix of MON and OSD socket files, used to determine socket type
+  mon_prefix = "ceph-mon"
+  osd_prefix = "ceph-osd"
+
+  ## suffix used to identify socket files
+  socket_suffix = "asok"
+```
+
+### Measurements & Fields:
+
+All fields are collected under the **ceph** measurement and stored as float64s. For a full list of fields, see the sample perf dumps in ceph_test.go. 
+
+
+### Tags:
+
+All measurements will have the following tags:
+
+- type: either 'osd' or 'mon' to indicate which type of node was queried
+- id: a unique string identifier, parsed from the socket file name for the node
+- collection: the top-level key under which these fields were reported. Possible values are:
+  - for MON nodes:
+    - cluster
+    - leveldb
+    - mon
+    - paxos
+    - throttle-mon_client_bytes
+    - throttle-mon_daemon_bytes
+    - throttle-msgr_dispatch_throttler-mon
+  - for OSD nodes:
+    - WBThrottle
+    - filestore
+    - leveldb
+    - mutex-FileJournal::completions_lock
+    - mutex-FileJournal::finisher_lock
+    - mutex-FileJournal::write_lock
+    - mutex-FileJournal::writeq_lock
+    - mutex-JOS::ApplyManager::apply_lock
+    - mutex-JOS::ApplyManager::com_lock
+    - mutex-JOS::SubmitManager::lock
+    - mutex-WBThrottle::lock
+    - objecter
+    - osd
+    - recoverystate_perf
+    - throttle-filestore_bytes
+    - throttle-filestore_ops
+    - throttle-msgr_dispatch_throttler-client
+    - throttle-msgr_dispatch_throttler-cluster
+    - throttle-msgr_dispatch_throttler-hb_back_server
+    - throttle-msgr_dispatch_throttler-hb_front_serve
+    - throttle-msgr_dispatch_throttler-hbclient
+    - throttle-msgr_dispatch_throttler-ms_objecter
+    - throttle-objecter_bytes
+    - throttle-objecter_ops
+    - throttle-osd_client_bytes
+    - throttle-osd_client_messages
+ 
+
+### Example Output:
+
+<pre>
+telegraf -test -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d  -input-filter ceph
+* Plugin: ceph, Collection 1
+> ceph,collection=paxos, id=node-2,role=openstack,type=mon accept_timeout=0,begin=14931264,begin_bytes.avgcount=14931264,begin_bytes.sum=180309683362,begin_keys.avgcount=0,begin_keys.sum=0,begin_latency.avgcount=14931264,begin_latency.sum=9293.29589,collect=1,collect_bytes.avgcount=1,collect_bytes.sum=24,collect_keys.avgcount=1,collect_keys.sum=1,collect_latency.avgcount=1,collect_latency.sum=0.00028,collect_timeout=0,collect_uncommitted=0,commit=14931264,commit_bytes.avgcount=0,commit_bytes.sum=0,commit_keys.avgcount=0,commit_keys.sum=0,commit_latency.avgcount=0,commit_latency.sum=0,lease_ack_timeout=0,lease_timeout=0,new_pn=0,new_pn_latency.avgcount=0,new_pn_latency.sum=0,refresh=14931264,refresh_latency.avgcount=14931264,refresh_latency.sum=8706.98498,restart=4,share_state=0,share_state_bytes.avgcount=0,share_state_bytes.sum=0,share_state_keys.avgcount=0,share_state_keys.sum=0,start_leader=0,start_peon=1,store_state=14931264,store_state_bytes.avgcount=14931264,store_state_bytes.sum=353119959211,store_state_keys.avgcount=14931264,store_state_keys.sum=289807523,store_state_latency.avgcount=14931264,store_state_latency.sum=10952.835724 1462821234814535148
+> ceph,collection=throttle-mon_client_bytes,id=node-2,type=mon get=1413017,get_or_fail_fail=0,get_or_fail_success=0,get_sum=71211705,max=104857600,put=1413013,put_sum=71211459,take=0,take_sum=0,val=246,wait.avgcount=0,wait.sum=0 1462821234814737219
+> ceph,collection=throttle-mon_daemon_bytes,id=node-2,type=mon get=4058121,get_or_fail_fail=0,get_or_fail_success=0,get_sum=6027348117,max=419430400,put=4058121,put_sum=6027348117,take=0,take_sum=0,val=0,wait.avgcount=0,wait.sum=0 1462821234814815661
+> ceph,collection=throttle-msgr_dispatch_throttler-mon,id=node-2,type=mon get=54276277,get_or_fail_fail=0,get_or_fail_success=0,get_sum=370232877040,max=104857600,put=54276277,put_sum=370232877040,take=0,take_sum=0,val=0,wait.avgcount=0,wait.sum=0 1462821234814872064
+</pre>

plugins/inputs/ceph/ceph.go

@@ -0,0 +1,249 @@
+package ceph
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/plugins/inputs"
+	"io/ioutil"
+	"log"
+	"os/exec"
+	"path/filepath"
+	"strings"
+)
+
+const (
+	measurement = "ceph"
+	typeMon     = "monitor"
+	typeOsd     = "osd"
+	osdPrefix   = "ceph-osd"
+	monPrefix   = "ceph-mon"
+	sockSuffix  = "asok"
+)
+
+type Ceph struct {
+	CephBinary   string
+	OsdPrefix    string
+	MonPrefix    string
+	SocketDir    string
+	SocketSuffix string
+}
+
+func (c *Ceph) setDefaults() {
+	if c.CephBinary == "" {
+		c.CephBinary = "/usr/bin/ceph"
+	}
+
+	if c.OsdPrefix == "" {
+		c.OsdPrefix = osdPrefix
+	}
+
+	if c.MonPrefix == "" {
+		c.MonPrefix = monPrefix
+	}
+
+	if c.SocketDir == "" {
+		c.SocketDir = "/var/run/ceph"
+	}
+
+	if c.SocketSuffix == "" {
+		c.SocketSuffix = sockSuffix
+	}
+}
+
+func (c *Ceph) Description() string {
+	return "Collects performance metrics from the MON and OSD nodes in a Ceph storage cluster."
+}
+
+var sampleConfig = `
+  ## All configuration values are optional, defaults are shown below
+
+  ## location of ceph binary
+  ceph_binary = "/usr/bin/ceph"
+
+  ## directory in which to look for socket files
+  socket_dir = "/var/run/ceph"
+
+  ## prefix of MON and OSD socket files, used to determine socket type
+  mon_prefix = "ceph-mon"
+  osd_prefix = "ceph-osd"
+
+  ## suffix used to identify socket files
+  socket_suffix = "asok"
+`
+
+func (c *Ceph) SampleConfig() string {
+	return sampleConfig
+}
+
+func (c *Ceph) Gather(acc telegraf.Accumulator) error {
+	c.setDefaults()
+	sockets, err := findSockets(c)
+	if err != nil {
+		return fmt.Errorf("failed to find sockets at path '%s': %v", c.SocketDir, err)
+	}
+
+	for _, s := range sockets {
+		dump, err := perfDump(c.CephBinary, s)
+		if err != nil {
+			log.Printf("error reading from socket '%s': %v", s.socket, err)
+			continue
+		}
+		data, err := parseDump(dump)
+		if err != nil {
+			log.Printf("error parsing dump from socket '%s': %v", s.socket, err)
+			continue
+		}
+		for tag, metrics := range *data {
+			acc.AddFields(measurement,
+				map[string]interface{}(metrics),
+				map[string]string{"type": s.sockType, "id": s.sockId, "collection": tag})
+		}
+	}
+	return nil
+}
+
+func init() {
+	inputs.Add(measurement, func() telegraf.Input { return &Ceph{} })
+}
+
+var perfDump = func(binary string, socket *socket) (string, error) {
+	cmdArgs := []string{"--admin-daemon", socket.socket}
+	if socket.sockType == typeOsd {
+		cmdArgs = append(cmdArgs, "perf", "dump")
+	} else if socket.sockType == typeMon {
+		cmdArgs = append(cmdArgs, "perfcounters_dump")
+	} else {
+		return "", fmt.Errorf("ignoring unknown socket type: %s", socket.sockType)
+	}
+
+	cmd := exec.Command(binary, cmdArgs...)
+	var out bytes.Buffer
+	cmd.Stdout = &out
+	err := cmd.Run()
+	if err != nil {
+		return "", fmt.Errorf("error running ceph dump: %s", err)
+	}
+
+	return out.String(), nil
+}
+
+var findSockets = func(c *Ceph) ([]*socket, error) {
+	listing, err := ioutil.ReadDir(c.SocketDir)
+	if err != nil {
+		return []*socket{}, fmt.Errorf("Failed to read socket directory '%s': %v", c.SocketDir, err)
+	}
+	sockets := make([]*socket, 0, len(listing))
+	for _, info := range listing {
+		f := info.Name()
+		var sockType string
+		var sockPrefix string
+		if strings.HasPrefix(f, c.MonPrefix) {
+			sockType = typeMon
+			sockPrefix = monPrefix
+		}
+		if strings.HasPrefix(f, c.OsdPrefix) {
+			sockType = typeOsd
+			sockPrefix = osdPrefix
+
+		}
+		if sockType == typeOsd || sockType == typeMon {
+			path := filepath.Join(c.SocketDir, f)
+			sockets = append(sockets, &socket{parseSockId(f, sockPrefix, c.SocketSuffix), sockType, path})
+		}
+	}
+	return sockets, nil
+}
+
+func parseSockId(fname, prefix, suffix string) string {
+	s := fname
+	s = strings.TrimPrefix(s, prefix)
+	s = strings.TrimSuffix(s, suffix)
+	s = strings.Trim(s, ".-_")
+	return s
+}
+
+type socket struct {
+	sockId   string
+	sockType string
+	socket   string
+}
+
+type metric struct {
+	pathStack []string // lifo stack of name components
+	value     float64
+}
+
+// Pops names of pathStack to build the flattened name for a metric
+func (m *metric) name() string {
+	buf := bytes.Buffer{}
+	for i := len(m.pathStack) - 1; i >= 0; i-- {
+		if buf.Len() > 0 {
+			buf.WriteString(".")
+		}
+		buf.WriteString(m.pathStack[i])
+	}
+	return buf.String()
+}
+
+type metricMap map[string]interface{}
+
+type taggedMetricMap map[string]metricMap
+
+// Parses a raw JSON string into a taggedMetricMap
+// Delegates the actual parsing to newTaggedMetricMap(..)
+func parseDump(dump string) (*taggedMetricMap, error) {
+	data := make(map[string]interface{})
+	err := json.Unmarshal([]byte(dump), &data)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse json: '%s': %v", dump, err)
+	}
+
+	tmm := newTaggedMetricMap(data)
+
+	if err != nil {
+		return nil, fmt.Errorf("failed to tag dataset: '%v': %v", tmm, err)
+	}
+
+	return tmm, nil
+}
+
+// Builds a TaggedMetricMap out of a generic string map.
+// The top-level key is used as a tag and all sub-keys are flattened into metrics
+func newTaggedMetricMap(data map[string]interface{}) *taggedMetricMap {
+	tmm := make(taggedMetricMap)
+	for tag, datapoints := range data {
+		mm := make(metricMap)
+		for _, m := range flatten(datapoints) {
+			mm[m.name()] = m.value
+		}
+		tmm[tag] = mm
+	}
+	return &tmm
+}
+
+// Recursively flattens any k-v hierarchy present in data.
+// Nested keys are flattened into ordered slices associated with a metric value.
+// The key slices are treated as stacks, and are expected to be reversed and concatenated
+// when passed as metrics to the accumulator. (see (*metric).name())
+func flatten(data interface{}) []*metric {
+	var metrics []*metric
+
+	switch val := data.(type) {
+	case float64:
+		metrics = []*metric{&metric{make([]string, 0, 1), val}}
+	case map[string]interface{}:
+		metrics = make([]*metric, 0, len(val))
+		for k, v := range val {
+			for _, m := range flatten(v) {
+				m.pathStack = append(m.pathStack, k)
+				metrics = append(metrics, m)
+			}
+		}
+	default:
+		log.Printf("Ignoring unexpected type '%T' for value %v", val, val)
+	}
+
+	return metrics
+}

plugins/inputs/ceph/ceph_test.go

@@ -0,0 +1,682 @@
+package ceph
+
+import (
+	"fmt"
+	"github.com/influxdata/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"io/ioutil"
+	"os"
+	"path"
+	"strconv"
+	"strings"
+	"testing"
+)
+
+const (
+	epsilon = float64(0.00000001)
+)
+
+func TestParseSockId(t *testing.T) {
+	s := parseSockId(sockFile(osdPrefix, 1), osdPrefix, sockSuffix)
+	assert.Equal(t, s, "1")
+}
+
+func TestParseMonDump(t *testing.T) {
+	dump, err := parseDump(monPerfDump)
+	assert.NoError(t, err)
+	assert.InEpsilon(t, 5678670180, (*dump)["cluster"]["osd_kb_used"], epsilon)
+	assert.InEpsilon(t, 6866.540527000, (*dump)["paxos"]["store_state_latency.sum"], epsilon)
+}
+
+func TestParseOsdDump(t *testing.T) {
+	dump, err := parseDump(osdPerfDump)
+	assert.NoError(t, err)
+	assert.InEpsilon(t, 552132.109360000, (*dump)["filestore"]["commitcycle_interval.sum"], epsilon)
+	assert.Equal(t, float64(0), (*dump)["mutex-FileJournal::finisher_lock"]["wait.avgcount"])
+}
+
+func TestGather(t *testing.T) {
+	saveFind := findSockets
+	saveDump := perfDump
+	defer func() {
+		findSockets = saveFind
+		perfDump = saveDump
+	}()
+
+	findSockets = func(c *Ceph) ([]*socket, error) {
+		return []*socket{&socket{"osd.1", typeOsd, ""}}, nil
+	}
+
+	perfDump = func(binary string, s *socket) (string, error) {
+		return osdPerfDump, nil
+	}
+
+	acc := &testutil.Accumulator{}
+	c := &Ceph{}
+	c.Gather(acc)
+
+}
+
+func TestFindSockets(t *testing.T) {
+	tmpdir, err := ioutil.TempDir("", "socktest")
+	assert.NoError(t, err)
+	defer func() {
+		err := os.Remove(tmpdir)
+		assert.NoError(t, err)
+	}()
+	c := &Ceph{
+		CephBinary: "foo",
+		SocketDir:  tmpdir,
+	}
+
+	c.setDefaults()
+
+	for _, st := range sockTestParams {
+		createTestFiles(tmpdir, st)
+
+		sockets, err := findSockets(c)
+		assert.NoError(t, err)
+
+		for i := 1; i <= st.osds; i++ {
+			assertFoundSocket(t, tmpdir, typeOsd, i, sockets)
+		}
+
+		for i := 1; i <= st.mons; i++ {
+			assertFoundSocket(t, tmpdir, typeMon, i, sockets)
+		}
+		cleanupTestFiles(tmpdir, st)
+	}
+}
+
+func assertFoundSocket(t *testing.T, dir, sockType string, i int, sockets []*socket) {
+	var prefix string
+	if sockType == typeOsd {
+		prefix = osdPrefix
+	} else {
+		prefix = monPrefix
+	}
+	expected := path.Join(dir, sockFile(prefix, i))
+	found := false
+	for _, s := range sockets {
+		fmt.Printf("Checking %s\n", s.socket)
+		if s.socket == expected {
+			found = true
+			assert.Equal(t, s.sockType, sockType, "Unexpected socket type for '%s'", s)
+			assert.Equal(t, s.sockId, strconv.Itoa(i))
+		}
+	}
+	assert.True(t, found, "Did not find socket: %s", expected)
+}
+
+func sockFile(prefix string, i int) string {
+	return strings.Join([]string{prefix, strconv.Itoa(i), sockSuffix}, ".")
+}
+
+func createTestFiles(dir string, st *SockTest) {
+	writeFile := func(prefix string, i int) {
+		f := sockFile(prefix, i)
+		fpath := path.Join(dir, f)
+		ioutil.WriteFile(fpath, []byte(""), 0777)
+	}
+	tstFileApply(st, writeFile)
+}
+
+func cleanupTestFiles(dir string, st *SockTest) {
+	rmFile := func(prefix string, i int) {
+		f := sockFile(prefix, i)
+		fpath := path.Join(dir, f)
+		err := os.Remove(fpath)
+		if err != nil {
+			fmt.Printf("Error removing test file %s: %v\n", fpath, err)
+		}
+	}
+	tstFileApply(st, rmFile)
+}
+
+func tstFileApply(st *SockTest, fn func(prefix string, i int)) {
+	for i := 1; i <= st.osds; i++ {
+		fn(osdPrefix, i)
+	}
+	for i := 1; i <= st.mons; i++ {
+		fn(monPrefix, i)
+	}
+}
+
+type SockTest struct {
+	osds int
+	mons int
+}
+
+var sockTestParams = []*SockTest{
+	&SockTest{
+		osds: 2,
+		mons: 2,
+	},
+	&SockTest{
+		mons: 1,
+	},
+	&SockTest{
+		osds: 1,
+	},
+	&SockTest{},
+}
+
+var monPerfDump = `
+{ "cluster": { "num_mon": 2,
+      "num_mon_quorum": 2,
+      "num_osd": 26,
+      "num_osd_up": 26,
+      "num_osd_in": 26,
+      "osd_epoch": 3306,
+      "osd_kb": 11487846448,
+      "osd_kb_used": 5678670180,
+      "osd_kb_avail": 5809176268,
+      "num_pool": 12,
+      "num_pg": 768,
+      "num_pg_active_clean": 768,
+      "num_pg_active": 768,
+      "num_pg_peering": 0,
+      "num_object": 397616,
+      "num_object_degraded": 0,
+      "num_object_unfound": 0,
+      "num_bytes": 2917848227467,
+      "num_mds_up": 0,
+      "num_mds_in": 0,
+      "num_mds_failed": 0,
+      "mds_epoch": 1},
+  "leveldb": { "leveldb_get": 321950312,
+      "leveldb_transaction": 18729922,
+      "leveldb_compact": 0,
+      "leveldb_compact_range": 74141,
+      "leveldb_compact_queue_merge": 0,
+      "leveldb_compact_queue_len": 0},
+  "mon": {},
+  "paxos": { "start_leader": 0,
+      "start_peon": 1,
+      "restart": 4,
+      "refresh": 9363435,
+      "refresh_latency": { "avgcount": 9363435,
+          "sum": 5378.794002000},
+      "begin": 9363435,
+      "begin_keys": { "avgcount": 0,
+          "sum": 0},
+      "begin_bytes": { "avgcount": 9363435,
+          "sum": 110468605489},
+      "begin_latency": { "avgcount": 9363435,
+          "sum": 5850.060682000},
+      "commit": 9363435,
+      "commit_keys": { "avgcount": 0,
+          "sum": 0},
+      "commit_bytes": { "avgcount": 0,
+          "sum": 0},
+      "commit_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "collect": 1,
+      "collect_keys": { "avgcount": 1,
+          "sum": 1},
+      "collect_bytes": { "avgcount": 1,
+          "sum": 24},
+      "collect_latency": { "avgcount": 1,
+          "sum": 0.000280000},
+      "collect_uncommitted": 0,
+      "collect_timeout": 0,
+      "accept_timeout": 0,
+      "lease_ack_timeout": 0,
+      "lease_timeout": 0,
+      "store_state": 9363435,
+      "store_state_keys": { "avgcount": 9363435,
+          "sum": 176572789},
+      "store_state_bytes": { "avgcount": 9363435,
+          "sum": 216355887217},
+      "store_state_latency": { "avgcount": 9363435,
+          "sum": 6866.540527000},
+      "share_state": 0,
+      "share_state_keys": { "avgcount": 0,
+          "sum": 0},
+      "share_state_bytes": { "avgcount": 0,
+          "sum": 0},
+      "new_pn": 0,
+      "new_pn_latency": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-mon_client_bytes": { "val": 246,
+      "max": 104857600,
+      "get": 896030,
+      "get_sum": 45854374,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 896026,
+      "put_sum": 45854128,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-mon_daemon_bytes": { "val": 0,
+      "max": 419430400,
+      "get": 2773768,
+      "get_sum": 3627676976,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 2773768,
+      "put_sum": 3627676976,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-mon": { "val": 0,
+      "max": 104857600,
+      "get": 34504949,
+      "get_sum": 226860281124,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 34504949,
+      "put_sum": 226860281124,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}}}
+`
+
+var osdPerfDump = `
+{ "WBThrottle": { "bytes_dirtied": 28405539,
+      "bytes_wb": 0,
+      "ios_dirtied": 93,
+      "ios_wb": 0,
+      "inodes_dirtied": 86,
+      "inodes_wb": 0},
+  "filestore": { "journal_queue_max_ops": 0,
+      "journal_queue_ops": 0,
+      "journal_ops": 1108008,
+      "journal_queue_max_bytes": 0,
+      "journal_queue_bytes": 0,
+      "journal_bytes": 73233416196,
+      "journal_latency": { "avgcount": 1108008,
+          "sum": 290.981036000},
+      "journal_wr": 1091866,
+      "journal_wr_bytes": { "avgcount": 1091866,
+          "sum": 74925682688},
+      "journal_full": 0,
+      "committing": 0,
+      "commitcycle": 110389,
+      "commitcycle_interval": { "avgcount": 110389,
+          "sum": 552132.109360000},
+      "commitcycle_latency": { "avgcount": 110389,
+          "sum": 178.657804000},
+      "op_queue_max_ops": 50,
+      "op_queue_ops": 0,
+      "ops": 1108008,
+      "op_queue_max_bytes": 104857600,
+      "op_queue_bytes": 0,
+      "bytes": 73226768148,
+      "apply_latency": { "avgcount": 1108008,
+          "sum": 947.742722000},
+      "queue_transaction_latency_avg": { "avgcount": 1108008,
+          "sum": 0.511327000}},
+  "leveldb": { "leveldb_get": 4361221,
+      "leveldb_transaction": 4351276,
+      "leveldb_compact": 0,
+      "leveldb_compact_range": 0,
+      "leveldb_compact_queue_merge": 0,
+      "leveldb_compact_queue_len": 0},
+  "mutex-FileJournal::completions_lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-FileJournal::finisher_lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-FileJournal::write_lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-FileJournal::writeq_lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-JOS::ApplyManager::apply_lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-JOS::ApplyManager::com_lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-JOS::SubmitManager::lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "mutex-WBThrottle::lock": { "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "objecter": { "op_active": 0,
+      "op_laggy": 0,
+      "op_send": 0,
+      "op_send_bytes": 0,
+      "op_resend": 0,
+      "op_ack": 0,
+      "op_commit": 0,
+      "op": 0,
+      "op_r": 0,
+      "op_w": 0,
+      "op_rmw": 0,
+      "op_pg": 0,
+      "osdop_stat": 0,
+      "osdop_create": 0,
+      "osdop_read": 0,
+      "osdop_write": 0,
+      "osdop_writefull": 0,
+      "osdop_append": 0,
+      "osdop_zero": 0,
+      "osdop_truncate": 0,
+      "osdop_delete": 0,
+      "osdop_mapext": 0,
+      "osdop_sparse_read": 0,
+      "osdop_clonerange": 0,
+      "osdop_getxattr": 0,
+      "osdop_setxattr": 0,
+      "osdop_cmpxattr": 0,
+      "osdop_rmxattr": 0,
+      "osdop_resetxattrs": 0,
+      "osdop_tmap_up": 0,
+      "osdop_tmap_put": 0,
+      "osdop_tmap_get": 0,
+      "osdop_call": 0,
+      "osdop_watch": 0,
+      "osdop_notify": 0,
+      "osdop_src_cmpxattr": 0,
+      "osdop_pgls": 0,
+      "osdop_pgls_filter": 0,
+      "osdop_other": 0,
+      "linger_active": 0,
+      "linger_send": 0,
+      "linger_resend": 0,
+      "poolop_active": 0,
+      "poolop_send": 0,
+      "poolop_resend": 0,
+      "poolstat_active": 0,
+      "poolstat_send": 0,
+      "poolstat_resend": 0,
+      "statfs_active": 0,
+      "statfs_send": 0,
+      "statfs_resend": 0,
+      "command_active": 0,
+      "command_send": 0,
+      "command_resend": 0,
+      "map_epoch": 3300,
+      "map_full": 0,
+      "map_inc": 3293,
+      "osd_sessions": 0,
+      "osd_session_open": 0,
+      "osd_session_close": 0,
+      "osd_laggy": 0},
+  "osd": { "opq": 0,
+      "op_wip": 0,
+      "op": 23939,
+      "op_in_bytes": 1245903961,
+      "op_out_bytes": 29103083856,
+      "op_latency": { "avgcount": 23939,
+          "sum": 440.192015000},
+      "op_process_latency": { "avgcount": 23939,
+          "sum": 30.170685000},
+      "op_r": 23112,
+      "op_r_out_bytes": 29103056146,
+      "op_r_latency": { "avgcount": 23112,
+          "sum": 19.373526000},
+      "op_r_process_latency": { "avgcount": 23112,
+          "sum": 14.625928000},
+      "op_w": 549,
+      "op_w_in_bytes": 1245804358,
+      "op_w_rlat": { "avgcount": 549,
+          "sum": 17.022299000},
+      "op_w_latency": { "avgcount": 549,
+          "sum": 418.494610000},
+      "op_w_process_latency": { "avgcount": 549,
+          "sum": 13.316555000},
+      "op_rw": 278,
+      "op_rw_in_bytes": 99603,
+      "op_rw_out_bytes": 27710,
+      "op_rw_rlat": { "avgcount": 278,
+          "sum": 2.213785000},
+      "op_rw_latency": { "avgcount": 278,
+          "sum": 2.323879000},
+      "op_rw_process_latency": { "avgcount": 278,
+          "sum": 2.228202000},
+      "subop": 1074774,
+      "subop_in_bytes": 26841811636,
+      "subop_latency": { "avgcount": 1074774,
+          "sum": 745.509160000},
+      "subop_w": 0,
+      "subop_w_in_bytes": 26841811636,
+      "subop_w_latency": { "avgcount": 1074774,
+          "sum": 745.509160000},
+      "subop_pull": 0,
+      "subop_pull_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "subop_push": 0,
+      "subop_push_in_bytes": 0,
+      "subop_push_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "pull": 0,
+      "push": 28,
+      "push_out_bytes": 103483392,
+      "push_in": 0,
+      "push_in_bytes": 0,
+      "recovery_ops": 15,
+      "loadavg": 202,
+      "buffer_bytes": 0,
+      "numpg": 18,
+      "numpg_primary": 8,
+      "numpg_replica": 10,
+      "numpg_stray": 0,
+      "heartbeat_to_peers": 10,
+      "heartbeat_from_peers": 0,
+      "map_messages": 7413,
+      "map_message_epochs": 9792,
+      "map_message_epoch_dups": 10105,
+      "messages_delayed_for_map": 83,
+      "stat_bytes": 102123175936,
+      "stat_bytes_used": 49961820160,
+      "stat_bytes_avail": 52161355776,
+      "copyfrom": 0,
+      "tier_promote": 0,
+      "tier_flush": 0,
+      "tier_flush_fail": 0,
+      "tier_try_flush": 0,
+      "tier_try_flush_fail": 0,
+      "tier_evict": 0,
+      "tier_whiteout": 0,
+      "tier_dirty": 230,
+      "tier_clean": 0,
+      "tier_delay": 0,
+      "agent_wake": 0,
+      "agent_skip": 0,
+      "agent_flush": 0,
+      "agent_evict": 0},
+  "recoverystate_perf": { "initial_latency": { "avgcount": 473,
+          "sum": 0.027207000},
+      "started_latency": { "avgcount": 1480,
+          "sum": 9854902.397648000},
+      "reset_latency": { "avgcount": 1953,
+          "sum": 0.096206000},
+      "start_latency": { "avgcount": 1953,
+          "sum": 0.059947000},
+      "primary_latency": { "avgcount": 765,
+          "sum": 4688922.186935000},
+      "peering_latency": { "avgcount": 704,
+          "sum": 1668.652135000},
+      "backfilling_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "waitremotebackfillreserved_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "waitlocalbackfillreserved_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "notbackfilling_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "repnotrecovering_latency": { "avgcount": 462,
+          "sum": 5158922.114600000},
+      "repwaitrecoveryreserved_latency": { "avgcount": 15,
+          "sum": 0.008275000},
+      "repwaitbackfillreserved_latency": { "avgcount": 1,
+          "sum": 0.000095000},
+      "RepRecovering_latency": { "avgcount": 16,
+          "sum": 2274.944727000},
+      "activating_latency": { "avgcount": 514,
+          "sum": 261.008520000},
+      "waitlocalrecoveryreserved_latency": { "avgcount": 20,
+          "sum": 0.175422000},
+      "waitremoterecoveryreserved_latency": { "avgcount": 20,
+          "sum": 0.682778000},
+      "recovering_latency": { "avgcount": 20,
+          "sum": 0.697551000},
+      "recovered_latency": { "avgcount": 511,
+          "sum": 0.011038000},
+      "clean_latency": { "avgcount": 503,
+          "sum": 4686961.154278000},
+      "active_latency": { "avgcount": 506,
+          "sum": 4687223.640464000},
+      "replicaactive_latency": { "avgcount": 446,
+          "sum": 5161197.078966000},
+      "stray_latency": { "avgcount": 794,
+          "sum": 4805.105128000},
+      "getinfo_latency": { "avgcount": 704,
+          "sum": 1138.477937000},
+      "getlog_latency": { "avgcount": 678,
+          "sum": 0.036393000},
+      "waitactingchange_latency": { "avgcount": 69,
+          "sum": 59.172893000},
+      "incomplete_latency": { "avgcount": 0,
+          "sum": 0.000000000},
+      "getmissing_latency": { "avgcount": 609,
+          "sum": 0.012288000},
+      "waitupthru_latency": { "avgcount": 576,
+          "sum": 530.106999000}},
+  "throttle-filestore_bytes": { "val": 0,
+      "max": 0,
+      "get": 0,
+      "get_sum": 0,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 0,
+      "put_sum": 0,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-filestore_ops": { "val": 0,
+      "max": 0,
+      "get": 0,
+      "get_sum": 0,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 0,
+      "put_sum": 0,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-client": { "val": 0,
+      "max": 104857600,
+      "get": 130730,
+      "get_sum": 1246039872,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 130730,
+      "put_sum": 1246039872,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-cluster": { "val": 0,
+      "max": 104857600,
+      "get": 1108033,
+      "get_sum": 71277949992,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 1108033,
+      "put_sum": 71277949992,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-hb_back_server": { "val": 0,
+      "max": 104857600,
+      "get": 18320575,
+      "get_sum": 861067025,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 18320575,
+      "put_sum": 861067025,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-hb_front_server": { "val": 0,
+      "max": 104857600,
+      "get": 18320575,
+      "get_sum": 861067025,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 18320575,
+      "put_sum": 861067025,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-hbclient": { "val": 0,
+      "max": 104857600,
+      "get": 40479394,
+      "get_sum": 1902531518,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 40479394,
+      "put_sum": 1902531518,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-msgr_dispatch_throttler-ms_objecter": { "val": 0,
+      "max": 104857600,
+      "get": 0,
+      "get_sum": 0,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 0,
+      "put_sum": 0,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-objecter_bytes": { "val": 0,
+      "max": 104857600,
+      "get": 0,
+      "get_sum": 0,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 0,
+      "put_sum": 0,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-objecter_ops": { "val": 0,
+      "max": 1024,
+      "get": 0,
+      "get_sum": 0,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 0,
+      "put_sum": 0,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-osd_client_bytes": { "val": 0,
+      "max": 524288000,
+      "get": 24241,
+      "get_sum": 1241992581,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 25958,
+      "put_sum": 1241992581,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}},
+  "throttle-osd_client_messages": { "val": 0,
+      "max": 100,
+      "get": 49214,
+      "get_sum": 49214,
+      "get_or_fail_fail": 0,
+      "get_or_fail_success": 0,
+      "take": 0,
+      "take_sum": 0,
+      "put": 49214,
+      "put_sum": 49214,
+      "wait": { "avgcount": 0,
+          "sum": 0.000000000}}}
+`

plugins/inputs/cgroup/README.md

@@ -0,0 +1,59 @@
+# CGroup Input Plugin For Telegraf Agent
+
+This input plugin will capture specific statistics per cgroup.
+
+Following file formats are supported:
+
+* Single value
+
+```
+VAL\n
+```
+
+* New line separated values
+
+```
+VAL0\n
+VAL1\n
+```
+
+* Space separated values
+
+```
+VAL0 VAL1 ...\n
+```
+
+* New line separated key-space-value's
+
+```
+KEY0 VAL0\n
+KEY1 VAL1\n
+```
+
+
+### Tags:
+
+Measurements don't have any specific tags unless you define them at the telegraf level (defaults). We
+used to have the path listed as a tag, but to keep cardinality in check it's easier to move this 
+value to a field. Thanks @sebito91!
+
+
+### Configuration:
+
+```
+# [[inputs.cgroup]]
+  # paths = [
+  #   "/cgroup/memory",           # root cgroup
+  #   "/cgroup/memory/child1",    # container cgroup
+  #   "/cgroup/memory/child2/*",  # all children cgroups under child2, but not child2 itself
+  # ]
+  # files = ["memory.*usage*", "memory.limit_in_bytes"]
+
+# [[inputs.cgroup]]
+  # paths = [
+  #   "/cgroup/cpu",              # root cgroup
+  #   "/cgroup/cpu/*",            # all container cgroups
+  #   "/cgroup/cpu/*/*",          # all children cgroups under each container cgroup
+  # ]
+  # files = ["cpuacct.usage", "cpu.cfs_period_us", "cpu.cfs_quota_us"]
+```

plugins/inputs/cgroup/cgroup.go

@@ -0,0 +1,35 @@
+package cgroup
+
+import (
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/plugins/inputs"
+)
+
+type CGroup struct {
+	Paths []string `toml:"paths"`
+	Files []string `toml:"files"`
+}
+
+var sampleConfig = `
+    ## Directories in which to look for files, globs are supported.
+	# paths = [
+	#   "/cgroup/memory",
+	#   "/cgroup/memory/child1",
+	#   "/cgroup/memory/child2/*",
+	# ]
+	## cgroup stat fields, as file names, globs are supported.
+	## these file names are appended to each path from above.
+	# files = ["memory.*usage*", "memory.limit_in_bytes"]
+`
+
+func (g *CGroup) SampleConfig() string {
+	return sampleConfig
+}
+
+func (g *CGroup) Description() string {
+	return "Read specific statistics per cgroup"
+}
+
+func init() {
+	inputs.Add("cgroup", func() telegraf.Input { return &CGroup{} })
+}

plugins/inputs/cgroup/cgroup_linux.go

@@ -0,0 +1,243 @@
+// +build linux
+
+package cgroup
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+	"path/filepath"
+	"regexp"
+	"strconv"
+
+	"github.com/influxdata/telegraf"
+)
+
+const metricName = "cgroup"
+
+func (g *CGroup) Gather(acc telegraf.Accumulator) error {
+	list := make(chan pathInfo)
+	go g.generateDirs(list)
+
+	for dir := range list {
+		if dir.err != nil {
+			return dir.err
+		}
+		if err := g.gatherDir(dir.path, acc); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (g *CGroup) gatherDir(dir string, acc telegraf.Accumulator) error {
+	fields := make(map[string]interface{})
+
+	list := make(chan pathInfo)
+	go g.generateFiles(dir, list)
+
+	for file := range list {
+		if file.err != nil {
+			return file.err
+		}
+
+		raw, err := ioutil.ReadFile(file.path)
+		if err != nil {
+			return err
+		}
+		if len(raw) == 0 {
+			continue
+		}
+
+		fd := fileData{data: raw, path: file.path}
+		if err := fd.parse(fields); err != nil {
+			return err
+		}
+	}
+	fields["path"] = dir
+
+	acc.AddFields(metricName, fields, nil)
+
+	return nil
+}
+
+// ======================================================================
+
+type pathInfo struct {
+	path string
+	err  error
+}
+
+func isDir(path string) (bool, error) {
+	result, err := os.Stat(path)
+	if err != nil {
+		return false, err
+	}
+	return result.IsDir(), nil
+}
+
+func (g *CGroup) generateDirs(list chan<- pathInfo) {
+	for _, dir := range g.Paths {
+		// getting all dirs that match the pattern 'dir'
+		items, err := filepath.Glob(dir)
+		if err != nil {
+			list <- pathInfo{err: err}
+			return
+		}
+
+		for _, item := range items {
+			ok, err := isDir(item)
+			if err != nil {
+				list <- pathInfo{err: err}
+				return
+			}
+			// supply only dirs
+			if ok {
+				list <- pathInfo{path: item}
+			}
+		}
+	}
+	close(list)
+}
+
+func (g *CGroup) generateFiles(dir string, list chan<- pathInfo) {
+	for _, file := range g.Files {
+		// getting all file paths that match the pattern 'dir + file'
+		// path.Base make sure that file variable does not contains part of path
+		items, err := filepath.Glob(path.Join(dir, path.Base(file)))
+		if err != nil {
+			list <- pathInfo{err: err}
+			return
+		}
+
+		for _, item := range items {
+			ok, err := isDir(item)
+			if err != nil {
+				list <- pathInfo{err: err}
+				return
+			}
+			// supply only files not dirs
+			if !ok {
+				list <- pathInfo{path: item}
+			}
+		}
+	}
+	close(list)
+}
+
+// ======================================================================
+
+type fileData struct {
+	data []byte
+	path string
+}
+
+func (fd *fileData) format() (*fileFormat, error) {
+	for _, ff := range fileFormats {
+		ok, err := ff.match(fd.data)
+		if err != nil {
+			return nil, err
+		}
+		if ok {
+			return &ff, nil
+		}
+	}
+
+	return nil, fmt.Errorf("%v: unknown file format", fd.path)
+}
+
+func (fd *fileData) parse(fields map[string]interface{}) error {
+	format, err := fd.format()
+	if err != nil {
+		return err
+	}
+
+	format.parser(filepath.Base(fd.path), fields, fd.data)
+	return nil
+}
+
+// ======================================================================
+
+type fileFormat struct {
+	name    string
+	pattern string
+	parser  func(measurement string, fields map[string]interface{}, b []byte)
+}
+
+const keyPattern = "[[:alpha:]_]+"
+const valuePattern = "[\\d-]+"
+
+var fileFormats = [...]fileFormat{
+	// 	VAL\n
+	fileFormat{
+		name:    "Single value",
+		pattern: "^" + valuePattern + "\n$",
+		parser: func(measurement string, fields map[string]interface{}, b []byte) {
+			re := regexp.MustCompile("^(" + valuePattern + ")\n$")
+			matches := re.FindAllStringSubmatch(string(b), -1)
+			fields[measurement] = numberOrString(matches[0][1])
+		},
+	},
+	// 	VAL0\n
+	// 	VAL1\n
+	// 	...
+	fileFormat{
+		name:    "New line separated values",
+		pattern: "^(" + valuePattern + "\n){2,}$",
+		parser: func(measurement string, fields map[string]interface{}, b []byte) {
+			re := regexp.MustCompile("(" + valuePattern + ")\n")
+			matches := re.FindAllStringSubmatch(string(b), -1)
+			for i, v := range matches {
+				fields[measurement+"."+strconv.Itoa(i)] = numberOrString(v[1])
+			}
+		},
+	},
+	// 	VAL0 VAL1 ...\n
+	fileFormat{
+		name:    "Space separated values",
+		pattern: "^(" + valuePattern + " )+\n$",
+		parser: func(measurement string, fields map[string]interface{}, b []byte) {
+			re := regexp.MustCompile("(" + valuePattern + ") ")
+			matches := re.FindAllStringSubmatch(string(b), -1)
+			for i, v := range matches {
+				fields[measurement+"."+strconv.Itoa(i)] = numberOrString(v[1])
+			}
+		},
+	},
+	// 	KEY0 VAL0\n
+	// 	KEY1 VAL1\n
+	// 	...
+	fileFormat{
+		name:    "New line separated key-space-value's",
+		pattern: "^(" + keyPattern + " " + valuePattern + "\n)+$",
+		parser: func(measurement string, fields map[string]interface{}, b []byte) {
+			re := regexp.MustCompile("(" + keyPattern + ") (" + valuePattern + ")\n")
+			matches := re.FindAllStringSubmatch(string(b), -1)
+			for _, v := range matches {
+				fields[measurement+"."+v[1]] = numberOrString(v[2])
+			}
+		},
+	},
+}
+
+func numberOrString(s string) interface{} {
+	i, err := strconv.Atoi(s)
+	if err == nil {
+		return i
+	}
+
+	return s
+}
+
+func (f fileFormat) match(b []byte) (bool, error) {
+	ok, err := regexp.Match(f.pattern, b)
+	if err != nil {
+		return false, err
+	}
+	if ok {
+		return true, nil
+	}
+	return false, nil
+}

plugins/inputs/cgroup/cgroup_notlinux.go

@@ -0,0 +1,11 @@
+// +build !linux
+
+package cgroup
+
+import (
+	"github.com/influxdata/telegraf"
+)
+
+func (g *CGroup) Gather(acc telegraf.Accumulator) error {
+	return nil
+}

plugins/inputs/cgroup/cgroup_test.go

@@ -0,0 +1,194 @@
+// +build linux
+
+package cgroup
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/influxdata/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"reflect"
+)
+
+var cg1 = &CGroup{
+	Paths: []string{"testdata/memory"},
+	Files: []string{
+		"memory.empty",
+		"memory.max_usage_in_bytes",
+		"memory.limit_in_bytes",
+		"memory.stat",
+		"memory.use_hierarchy",
+		"notify_on_release",
+	},
+}
+
+func assertContainsFields(a *testutil.Accumulator, t *testing.T, measurement string, fieldSet []map[string]interface{}) {
+	a.Lock()
+	defer a.Unlock()
+
+	numEquals := 0
+	for _, p := range a.Metrics {
+		if p.Measurement == measurement {
+			for _, fields := range fieldSet {
+				if reflect.DeepEqual(fields, p.Fields) {
+					numEquals++
+				}
+			}
+		}
+	}
+
+	if numEquals != len(fieldSet) {
+		assert.Fail(t, fmt.Sprintf("only %d of %d are equal", numEquals, len(fieldSet)))
+	}
+}
+
+func TestCgroupStatistics_1(t *testing.T) {
+	var acc testutil.Accumulator
+
+	err := cg1.Gather(&acc)
+	require.NoError(t, err)
+
+	fields := map[string]interface{}{
+		"memory.stat.cache":           1739362304123123123,
+		"memory.stat.rss":             1775325184,
+		"memory.stat.rss_huge":        778043392,
+		"memory.stat.mapped_file":     421036032,
+		"memory.stat.dirty":           -307200,
+		"memory.max_usage_in_bytes.0": 0,
+		"memory.max_usage_in_bytes.1": -1,
+		"memory.max_usage_in_bytes.2": 2,
+		"memory.limit_in_bytes":       223372036854771712,
+		"memory.use_hierarchy":        "12-781",
+		"notify_on_release":           0,
+		"path":                        "testdata/memory",
+	}
+	assertContainsFields(&acc, t, "cgroup", []map[string]interface{}{fields})
+}
+
+// ======================================================================
+
+var cg2 = &CGroup{
+	Paths: []string{"testdata/cpu"},
+	Files: []string{"cpuacct.usage_percpu"},
+}
+
+func TestCgroupStatistics_2(t *testing.T) {
+	var acc testutil.Accumulator
+
+	err := cg2.Gather(&acc)
+	require.NoError(t, err)
+
+	fields := map[string]interface{}{
+		"cpuacct.usage_percpu.0": -1452543795404,
+		"cpuacct.usage_percpu.1": 1376681271659,
+		"cpuacct.usage_percpu.2": 1450950799997,
+		"cpuacct.usage_percpu.3": -1473113374257,
+		"path": "testdata/cpu",
+	}
+	assertContainsFields(&acc, t, "cgroup", []map[string]interface{}{fields})
+}
+
+// ======================================================================
+
+var cg3 = &CGroup{
+	Paths: []string{"testdata/memory/*"},
+	Files: []string{"memory.limit_in_bytes"},
+}
+
+func TestCgroupStatistics_3(t *testing.T) {
+	var acc testutil.Accumulator
+
+	err := cg3.Gather(&acc)
+	require.NoError(t, err)
+
+	fields := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_1",
+	}
+
+	fieldsTwo := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_2",
+	}
+	assertContainsFields(&acc, t, "cgroup", []map[string]interface{}{fields, fieldsTwo})
+}
+
+// ======================================================================
+
+var cg4 = &CGroup{
+	Paths: []string{"testdata/memory/*/*", "testdata/memory/group_2"},
+	Files: []string{"memory.limit_in_bytes"},
+}
+
+func TestCgroupStatistics_4(t *testing.T) {
+	var acc testutil.Accumulator
+
+	err := cg4.Gather(&acc)
+	require.NoError(t, err)
+
+	fields := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_1/group_1_1",
+	}
+
+	fieldsTwo := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_1/group_1_2",
+	}
+
+	fieldsThree := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_2",
+	}
+
+	assertContainsFields(&acc, t, "cgroup", []map[string]interface{}{fields, fieldsTwo, fieldsThree})
+}
+
+// ======================================================================
+
+var cg5 = &CGroup{
+	Paths: []string{"testdata/memory/*/group_1_1"},
+	Files: []string{"memory.limit_in_bytes"},
+}
+
+func TestCgroupStatistics_5(t *testing.T) {
+	var acc testutil.Accumulator
+
+	err := cg5.Gather(&acc)
+	require.NoError(t, err)
+
+	fields := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_1/group_1_1",
+	}
+
+	fieldsTwo := map[string]interface{}{
+		"memory.limit_in_bytes": 223372036854771712,
+		"path":                  "testdata/memory/group_2/group_1_1",
+	}
+	assertContainsFields(&acc, t, "cgroup", []map[string]interface{}{fields, fieldsTwo})
+}
+
+// ======================================================================
+
+var cg6 = &CGroup{
+	Paths: []string{"testdata/memory"},
+	Files: []string{"memory.us*", "*/memory.kmem.*"},
+}
+
+func TestCgroupStatistics_6(t *testing.T) {
+	var acc testutil.Accumulator
+
+	err := cg6.Gather(&acc)
+	require.NoError(t, err)
+
+	fields := map[string]interface{}{
+		"memory.usage_in_bytes":      3513667584,
+		"memory.use_hierarchy":       "12-781",
+		"memory.kmem.limit_in_bytes": 9223372036854771712,
+		"path": "testdata/memory",
+	}
+	assertContainsFields(&acc, t, "cgroup", []map[string]interface{}{fields})
+}

plugins/inputs/cgroup/testdata/blkio/blkio.io_serviced

@@ -0,0 +1 @@
+Total 0

plugins/inputs/cgroup/testdata/blkio/blkio.throttle.io_serviced

@@ -0,0 +1,131 @@
+11:0 Read 0
+11:0 Write 0
+11:0 Sync 0
+11:0 Async 0
+11:0 Total 0
+8:0 Read 49134
+8:0 Write 216703
+8:0 Sync 177906
+8:0 Async 87931
+8:0 Total 265837
+7:7 Read 0
+7:7 Write 0
+7:7 Sync 0
+7:7 Async 0
+7:7 Total 0
+7:6 Read 0
+7:6 Write 0
+7:6 Sync 0
+7:6 Async 0
+7:6 Total 0
+7:5 Read 0
+7:5 Write 0
+7:5 Sync 0
+7:5 Async 0
+7:5 Total 0
+7:4 Read 0
+7:4 Write 0
+7:4 Sync 0
+7:4 Async 0
+7:4 Total 0
+7:3 Read 0
+7:3 Write 0
+7:3 Sync 0
+7:3 Async 0
+7:3 Total 0
+7:2 Read 0
+7:2 Write 0
+7:2 Sync 0
+7:2 Async 0
+7:2 Total 0
+7:1 Read 0
+7:1 Write 0
+7:1 Sync 0
+7:1 Async 0
+7:1 Total 0
+7:0 Read 0
+7:0 Write 0
+7:0 Sync 0
+7:0 Async 0
+7:0 Total 0
+1:15 Read 3
+1:15 Write 0
+1:15 Sync 0
+1:15 Async 3
+1:15 Total 3
+1:14 Read 3
+1:14 Write 0
+1:14 Sync 0
+1:14 Async 3
+1:14 Total 3
+1:13 Read 3
+1:13 Write 0
+1:13 Sync 0
+1:13 Async 3
+1:13 Total 3
+1:12 Read 3
+1:12 Write 0
+1:12 Sync 0
+1:12 Async 3
+1:12 Total 3
+1:11 Read 3
+1:11 Write 0
+1:11 Sync 0
+1:11 Async 3
+1:11 Total 3
+1:10 Read 3
+1:10 Write 0
+1:10 Sync 0
+1:10 Async 3
+1:10 Total 3
+1:9 Read 3
+1:9 Write 0
+1:9 Sync 0
+1:9 Async 3
+1:9 Total 3
+1:8 Read 3
+1:8 Write 0
+1:8 Sync 0
+1:8 Async 3
+1:8 Total 3
+1:7 Read 3
+1:7 Write 0
+1:7 Sync 0
+1:7 Async 3
+1:7 Total 3
+1:6 Read 3
+1:6 Write 0
+1:6 Sync 0
+1:6 Async 3
+1:6 Total 3
+1:5 Read 3
+1:5 Write 0
+1:5 Sync 0
+1:5 Async 3
+1:5 Total 3
+1:4 Read 3
+1:4 Write 0
+1:4 Sync 0
+1:4 Async 3
+1:4 Total 3
+1:3 Read 3
+1:3 Write 0
+1:3 Sync 0
+1:3 Async 3
+1:3 Total 3
+1:2 Read 3
+1:2 Write 0
+1:2 Sync 0
+1:2 Async 3
+1:2 Total 3
+1:1 Read 3
+1:1 Write 0
+1:1 Sync 0
+1:1 Async 3
+1:1 Total 3
+1:0 Read 3
+1:0 Write 0
+1:0 Sync 0
+1:0 Async 3
+1:0 Total 3
+Total 265885

plugins/inputs/cgroup/testdata/cpu/cpu.cfs_quota_us

@@ -0,0 +1 @@
+-1

plugins/inputs/cgroup/testdata/cpu/cpuacct.usage_percpu

@@ -0,0 +1 @@
+-1452543795404 1376681271659 1450950799997 -1473113374257 

plugins/inputs/cgroup/testdata/memory/group_1/group_1_1/memory.limit_in_bytes

@@ -0,0 +1 @@
+223372036854771712

plugins/inputs/cgroup/testdata/memory/group_1/group_1_1/memory.stat

@@ -0,0 +1,5 @@
+cache 1739362304123123123
+rss 1775325184
+rss_huge 778043392
+mapped_file 421036032
+dirty -307200

plugins/inputs/cgroup/testdata/memory/group_1/group_1_2/memory.limit_in_bytes

@@ -0,0 +1 @@
+223372036854771712

plugins/inputs/cgroup/testdata/memory/group_1/group_1_2/memory.stat

@@ -0,0 +1,5 @@
+cache 1739362304123123123
+rss 1775325184
+rss_huge 778043392
+mapped_file 421036032
+dirty -307200

plugins/inputs/cgroup/testdata/memory/group_1/memory.kmem.limit_in_bytes

@@ -0,0 +1 @@
+9223372036854771712

plugins/inputs/cgroup/testdata/memory/group_1/memory.kmem.max_usage_in_bytes

@@ -0,0 +1 @@
+0

plugins/inputs/cgroup/testdata/memory/group_1/memory.limit_in_bytes

@@ -0,0 +1 @@
+223372036854771712

plugins/inputs/cgroup/testdata/memory/group_1/memory.stat

@@ -0,0 +1,5 @@
+cache 1739362304123123123
+rss 1775325184
+rss_huge 778043392
+mapped_file 421036032
+dirty -307200

plugins/inputs/cgroup/testdata/memory/group_2/group_1_1/memory.limit_in_bytes

@@ -0,0 +1 @@
+223372036854771712

plugins/inputs/cgroup/testdata/memory/group_2/group_1_1/memory.stat

@@ -0,0 +1,5 @@
+cache 1739362304123123123
+rss 1775325184
+rss_huge 778043392
+mapped_file 421036032
+dirty -307200

plugins/inputs/cgroup/testdata/memory/group_2/memory.limit_in_bytes

@@ -0,0 +1 @@
+223372036854771712

plugins/inputs/cgroup/testdata/memory/group_2/memory.stat

@@ -0,0 +1,5 @@
+cache 1739362304123123123
+rss 1775325184
+rss_huge 778043392
+mapped_file 421036032
+dirty -307200

plugins/inputs/cgroup/testdata/memory/memory.kmem.limit_in_bytes

@@ -0,0 +1 @@
+9223372036854771712

plugins/inputs/cgroup/testdata/memory/memory.limit_in_bytes

@@ -0,0 +1 @@
+223372036854771712

plugins/inputs/cgroup/testdata/memory/memory.max_usage_in_bytes

@@ -0,0 +1,3 @@
+0
+-1
+2

plugins/inputs/cgroup/testdata/memory/memory.numa_stat

@@ -0,0 +1,8 @@
+total=858067 N0=858067
+file=406254 N0=406254
+anon=451792 N0=451792
+unevictable=21 N0=21
+hierarchical_total=858067 N0=858067
+hierarchical_file=406254 N0=406254
+hierarchical_anon=451792 N0=451792
+hierarchical_unevictable=21 N0=21

plugins/inputs/cgroup/testdata/memory/memory.stat

@@ -0,0 +1,5 @@
+cache 1739362304123123123
+rss 1775325184
+rss_huge 778043392
+mapped_file 421036032
+dirty -307200

plugins/inputs/cgroup/testdata/memory/memory.usage_in_bytes

@@ -0,0 +1 @@
+3513667584

plugins/inputs/cgroup/testdata/memory/memory.use_hierarchy

@@ -0,0 +1 @@
+12-781

plugins/inputs/cgroup/testdata/memory/notify_on_release

@@ -0,0 +1 @@
+0

plugins/inputs/chrony/README.md

@@ -0,0 +1,92 @@
+# chrony Input Plugin
+
+Get standard chrony metrics, requires chronyc executable.
+
+Below is the documentation of the various headers returned by `chronyc tracking`.
+
+- Reference ID - This is the refid and name (or IP address) if available, of the
+server to which the computer is currently synchronised. If this is 127.127.1.1
+it means the computer is not synchronised to any external source and that you
+have the ‘local’ mode operating (via the local command in chronyc (see section local),
+or the local directive in the ‘/etc/chrony.conf’ file (see section local)).
+- Stratum - The stratum indicates how many hops away from a computer with an attached
+reference clock we are. Such a computer is a stratum-1 computer, so the computer in the
+example is two hops away (i.e. a.b.c is a stratum-2 and is synchronised from a stratum-1).
+- Ref time - This is the time (UTC) at which the last measurement from the reference
+source was processed.
+- System time - In normal operation, chronyd never steps the system clock, because any
+jump in the timescale can have adverse consequences for certain application programs.
+Instead, any error in the system clock is corrected by slightly speeding up or slowing
+down the system clock until the error has been removed, and then returning to the system
+clock’s normal speed. A consequence of this is that there will be a period when the
+system clock (as read by other programs using the gettimeofday() system call, or by the
+date command in the shell) will be different from chronyd's estimate of the current true
+time (which it reports to NTP clients when it is operating in server mode). The value
+reported on this line is the difference due to this effect.
+- Last offset - This is the estimated local offset on the last clock update.
+- RMS offset - This is a long-term average of the offset value.
+- Frequency - The ‘frequency’ is the rate by which the system’s clock would be
+wrong if chronyd was not correcting it. It is expressed in ppm (parts per million).
+For example, a value of 1ppm would mean that when the system’s clock thinks it has
+advanced 1 second, it has actually advanced by 1.000001 seconds relative to true time.
+- Residual freq - This shows the ‘residual frequency’ for the currently selected
+reference source. This reflects any difference between what the measurements from the
+reference source indicate the frequency should be and the frequency currently being used.
+The reason this is not always zero is that a smoothing procedure is applied to the
+frequency. Each time a measurement from the reference source is obtained and a new
+residual frequency computed, the estimated accuracy of this residual is compared with the
+estimated accuracy (see ‘skew’ next) of the existing frequency value. A weighted average
+is computed for the new frequency, with weights depending on these accuracies. If the
+measurements from the reference source follow a consistent trend, the residual will be
+driven to zero over time.
+- Skew - This is the estimated error bound on the frequency.
+- Root delay - This is the total of the network path delays to the stratum-1 computer
+from which the computer is ultimately synchronised. In certain extreme situations, this
+value can be negative. (This can arise in a symmetric peer arrangement where the computers’
+frequencies are not tracking each other and the network delay is very short relative to the
+turn-around time at each computer.)
+- Root dispersion - This is the total dispersion accumulated through all the computers
+back to the stratum-1 computer from which the computer is ultimately synchronised.
+Dispersion is due to system clock resolution, statistical measurement variations etc.
+- Leap status - This is the leap status, which can be Normal, Insert second,
+Delete second or Not synchronised.
+
+### Configuration:
+
+```toml
+# Get standard chrony metrics, requires chronyc executable.
+[[inputs.chrony]]
+  ## If true, chronyc tries to perform a DNS lookup for the time server.
+  # dns_lookup = false
+```
+
+### Measurements & Fields:
+
+- chrony
+    - last_offset (float, seconds)
+    - rms_offset (float, seconds)
+    - frequency (float, ppm)
+    - residual_freq (float, ppm)
+    - skew (float, ppm)
+    - root_delay (float, seconds)
+    - root_dispersion (float, seconds)
+    - update_interval (float, seconds)
+
+### Tags:
+
+- All measurements have the following tags:
+    - reference_id
+    - stratum
+    - leap_status
+
+### Example Output:
+
+```
+$ telegraf -config telegraf.conf -input-filter chrony -test
+* Plugin: chrony, Collection 1
+> chrony,leap_status=normal,reference_id=192.168.1.1,stratum=3 frequency=-35.657,last_offset=-0.000013616,residual_freq=-0,rms_offset=0.000027073,root_delay=0.000644,root_dispersion=0.003444,skew=0.001,update_interval=1031.2 1463750789687639161
+```
+
+
+
+

plugins/inputs/chrony/chrony.go

@@ -0,0 +1,129 @@
+// +build linux
+
+package chrony
+
+import (
+	"errors"
+	"fmt"
+	"os/exec"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/internal"
+	"github.com/influxdata/telegraf/plugins/inputs"
+)
+
+var (
+	execCommand = exec.Command // execCommand is used to mock commands in tests.
+)
+
+type Chrony struct {
+	DNSLookup bool `toml:"dns_lookup"`
+	path      string
+}
+
+func (*Chrony) Description() string {
+	return "Get standard chrony metrics, requires chronyc executable."
+}
+
+func (*Chrony) SampleConfig() string {
+	return `
+  ## If true, chronyc tries to perform a DNS lookup for the time server.
+  # dns_lookup = false
+  `
+}
+
+func (c *Chrony) Gather(acc telegraf.Accumulator) error {
+	if len(c.path) == 0 {
+		return errors.New("chronyc not found: verify that chrony is installed and that chronyc is in your PATH")
+	}
+
+	flags := []string{}
+	if !c.DNSLookup {
+		flags = append(flags, "-n")
+	}
+	flags = append(flags, "tracking")
+
+	cmd := execCommand(c.path, flags...)
+	out, err := internal.CombinedOutputTimeout(cmd, time.Second*5)
+	if err != nil {
+		return fmt.Errorf("failed to run command %s: %s - %s", strings.Join(cmd.Args, " "), err, string(out))
+	}
+	fields, tags, err := processChronycOutput(string(out))
+	if err != nil {
+		return err
+	}
+	acc.AddFields("chrony", fields, tags)
+	return nil
+}
+
+// processChronycOutput takes in a string output from the chronyc command, like:
+//
+//     Reference ID    : 192.168.1.22 (ntp.example.com)
+//     Stratum         : 3
+//     Ref time (UTC)  : Thu May 12 14:27:07 2016
+//     System time     : 0.000020390 seconds fast of NTP time
+//     Last offset     : +0.000012651 seconds
+//     RMS offset      : 0.000025577 seconds
+//     Frequency       : 16.001 ppm slow
+//     Residual freq   : -0.000 ppm
+//     Skew            : 0.006 ppm
+//     Root delay      : 0.001655 seconds
+//     Root dispersion : 0.003307 seconds
+//     Update interval : 507.2 seconds
+//     Leap status     : Normal
+//
+// The value on the left side of the colon is used as field name, if the first field on
+// the right side is a float. If it cannot be parsed as float, it is a tag name.
+//
+// Ref time is ignored and all names are converted to snake case.
+//
+// It returns (<fields>, <tags>)
+func processChronycOutput(out string) (map[string]interface{}, map[string]string, error) {
+	tags := map[string]string{}
+	fields := map[string]interface{}{}
+	lines := strings.Split(strings.TrimSpace(out), "\n")
+	for _, line := range lines {
+		stats := strings.Split(line, ":")
+		if len(stats) < 2 {
+			return nil, nil, fmt.Errorf("unexpected output from chronyc, expected ':' in %s", out)
+		}
+		name := strings.ToLower(strings.Replace(strings.TrimSpace(stats[0]), " ", "_", -1))
+		// ignore reference time
+		if strings.Contains(name, "time") {
+			continue
+		}
+		valueFields := strings.Fields(stats[1])
+		if len(valueFields) == 0 {
+			return nil, nil, fmt.Errorf("unexpected output from chronyc: %s", out)
+		}
+		if strings.Contains(strings.ToLower(name), "stratum") {
+			tags["stratum"] = valueFields[0]
+			continue
+		}
+		value, err := strconv.ParseFloat(valueFields[0], 64)
+		if err != nil {
+			tags[name] = strings.ToLower(valueFields[0])
+			continue
+		}
+		if strings.Contains(stats[1], "slow") {
+			value = -value
+		}
+		fields[name] = value
+	}
+
+	return fields, tags, nil
+}
+
+func init() {
+	c := Chrony{}
+	path, _ := exec.LookPath("chronyc")
+	if len(path) > 0 {
+		c.path = path
+	}
+	inputs.Add("chrony", func() telegraf.Input {
+		return &c
+	})
+}

plugins/inputs/chrony/chrony_notlinux.go

@@ -0,0 +1,3 @@
+// +build !linux
+
+package chrony

plugins/inputs/chrony/chrony_test.go

@@ -0,0 +1,109 @@
+// +build linux
+
+package chrony
+
+import (
+	"fmt"
+	"os"
+	"os/exec"
+	"testing"
+
+	"github.com/influxdata/telegraf/testutil"
+)
+
+func TestGather(t *testing.T) {
+	c := Chrony{
+		path: "chronyc",
+	}
+	// overwriting exec commands with mock commands
+	execCommand = fakeExecCommand
+	defer func() { execCommand = exec.Command }()
+	var acc testutil.Accumulator
+
+	err := c.Gather(&acc)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	tags := map[string]string{
+		"reference_id": "192.168.1.22",
+		"leap_status":  "normal",
+		"stratum":      "3",
+	}
+	fields := map[string]interface{}{
+		"last_offset":     0.000012651,
+		"rms_offset":      0.000025577,
+		"frequency":       -16.001,
+		"residual_freq":   0.0,
+		"skew":            0.006,
+		"root_delay":      0.001655,
+		"root_dispersion": 0.003307,
+		"update_interval": 507.2,
+	}
+
+	acc.AssertContainsTaggedFields(t, "chrony", fields, tags)
+
+	// test with dns lookup
+	c.DNSLookup = true
+	err = c.Gather(&acc)
+	if err != nil {
+		t.Fatal(err)
+	}
+	acc.AssertContainsTaggedFields(t, "chrony", fields, tags)
+
+}
+
+// fackeExecCommand is a helper function that mock
+// the exec.Command call (and call the test binary)
+func fakeExecCommand(command string, args ...string) *exec.Cmd {
+	cs := []string{"-test.run=TestHelperProcess", "--", command}
+	cs = append(cs, args...)
+	cmd := exec.Command(os.Args[0], cs...)
+	cmd.Env = []string{"GO_WANT_HELPER_PROCESS=1"}
+	return cmd
+}
+
+// TestHelperProcess isn't a real test. It's used to mock exec.Command
+// For example, if you run:
+// GO_WANT_HELPER_PROCESS=1 go test -test.run=TestHelperProcess -- chrony tracking
+// it returns below mockData.
+func TestHelperProcess(t *testing.T) {
+	if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
+		return
+	}
+
+	lookup := "Reference ID    : 192.168.1.22 (ntp.example.com)\n"
+	noLookup := "Reference ID    : 192.168.1.22 (192.168.1.22)\n"
+	mockData := `Stratum         : 3
+Ref time (UTC)  : Thu May 12 14:27:07 2016
+System time     : 0.000020390 seconds fast of NTP time
+Last offset     : +0.000012651 seconds
+RMS offset      : 0.000025577 seconds
+Frequency       : 16.001 ppm slow
+Residual freq   : -0.000 ppm
+Skew            : 0.006 ppm
+Root delay      : 0.001655 seconds
+Root dispersion : 0.003307 seconds
+Update interval : 507.2 seconds
+Leap status     : Normal
+`
+
+	args := os.Args
+
+	// Previous arguments are tests stuff, that looks like :
+	// /tmp/go-build970079519/…/_test/integration.test -test.run=TestHelperProcess --
+	cmd, args := args[3], args[4:]
+
+	if cmd == "chronyc" {
+		if args[0] == "tracking" {
+			fmt.Fprint(os.Stdout, lookup+mockData)
+		} else {
+			fmt.Fprint(os.Stdout, noLookup+mockData)
+		}
+	} else {
+		fmt.Fprint(os.Stdout, "command not found")
+		os.Exit(1)
+
+	}
+	os.Exit(0)
+}

plugins/inputs/cloudwatch/README.md

@@ -0,0 +1,126 @@
+# Amazon CloudWatch Statistics Input
+
+This plugin will pull Metric Statistics from Amazon CloudWatch.
+
+### Amazon Authentication
+
+This plugin uses a credential chain for Authentication with the CloudWatch
+API endpoint. In the following order the plugin will attempt to authenticate.
+1. Assumed credentials via STS if `role_arn` attribute is specified (source credentials are evaluated from subsequent rules)
+2. Explicit credentials from `access_key`, `secret_key`, and `token` attributes
+3. Shared profile from `profile` attribute
+4. [Environment Variables](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk#environment-variables)
+5. [Shared Credentials](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk#shared-credentials-file)
+6. [EC2 Instance Profile](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
+
+### Configuration:
+
+```toml
+[[inputs.cloudwatch]]
+  ## Amazon Region (required)
+  region = 'us-east-1'
+
+  ## Requested CloudWatch aggregation Period (required - must be a multiple of 60s)
+  period = '1m'
+
+  ## Collection Delay (required - must account for metrics availability via CloudWatch API)
+  delay = '1m'
+
+  ## Override global run interval (optional - defaults to global interval)
+  ## Recomended: use metric 'interval' that is a multiple of 'period' to avoid
+  ## gaps or overlap in pulled data
+  interval = '1m'
+
+  ## Metric Statistic Namespace (required)
+  namespace = 'AWS/ELB'
+
+  ## Metrics to Pull (optional)
+  ## Defaults to all Metrics in Namespace if nothing is provided
+  ## Refreshes Namespace available metrics every 1h
+  [[inputs.cloudwatch.metrics]]
+    names = ['Latency', 'RequestCount']
+
+    ## Dimension filters for Metric (optional)
+    [[inputs.cloudwatch.metrics.dimensions]]
+      name = 'LoadBalancerName'
+      value = 'p-example'
+
+    [[inputs.cloudwatch.metrics.dimensions]]
+      name = 'AvailabilityZone'
+      value = '*'
+```
+#### Requirements and Terminology
+
+Plugin Configuration utilizes [CloudWatch concepts](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html) and access pattern to allow monitoring of any CloudWatch Metric.
+
+- `region` must be a valid AWS [Region](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#CloudWatchRegions) value
+- `period` must be a valid CloudWatch [Period](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#CloudWatchPeriods) value
+- `namespace` must be a valid CloudWatch [Namespace](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Namespace) value
+- `names` must be valid CloudWatch [Metric](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Metric) names
+- `dimensions` must be valid CloudWatch [Dimension](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Dimension) name/value pairs
+
+Omitting or specifying a value of `'*'` for a dimension value configures all available metrics that contain a dimension with the specified name
+to be retrieved. If specifying >1 dimension, then the metric must contain *all* the configured dimensions where the the value of the
+wildcard dimension is ignored.
+
+Example:
+```
+[[inputs.cloudwatch.metrics]]
+  names = ['Latency']
+
+  ## Dimension filters for Metric (optional)
+  [[inputs.cloudwatch.metrics.dimensions]]
+    name = 'LoadBalancerName'
+    value = 'p-example'
+
+  [[inputs.cloudwatch.metrics.dimensions]]
+    name = 'AvailabilityZone'
+    value = '*'
+```
+
+If the following ELBs are available:
+- name: `p-example`, availabilityZone: `us-east-1a`
+- name: `p-example`, availabilityZone: `us-east-1b`
+- name: `q-example`, availabilityZone: `us-east-1a`
+- name: `q-example`, availabilityZone: `us-east-1b`
+
+
+Then 2 metrics will be output:
+- name: `p-example`, availabilityZone: `us-east-1a`
+- name: `p-example`, availabilityZone: `us-east-1b`
+
+If the `AvailabilityZone` wildcard dimension was omitted, then a single metric (name: `p-example`)
+would be exported containing the aggregate values of the ELB across availability zones.
+
+#### Restrictions and Limitations
+- CloudWatch metrics are not available instantly via the CloudWatch API. You should adjust your collection `delay` to account for this lag in metrics availability based on your [monitoring subscription level](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html)
+- CloudWatch API usage incurs cost - see [GetMetricStatistics Pricing](https://aws.amazon.com/cloudwatch/pricing/)
+
+### Measurements & Fields:
+
+Each CloudWatch Namespace monitored records a measurement with fields for each available Metric Statistic
+Namespace and Metrics are represented in [snake case](https://en.wikipedia.org/wiki/Snake_case)
+
+- cloudwatch_{namespace}
+  - {metric}_sum         (metric Sum value)
+  - {metric}_average     (metric Average value)
+  - {metric}_minimum     (metric Minimum value)
+  - {metric}_maximum     (metric Maximum value)
+  - {metric}_sample_count (metric SampleCount value)
+
+
+### Tags:
+Each measurement is tagged with the following identifiers to uniquely identify the associated metric
+Tag Dimension names are represented in [snake case](https://en.wikipedia.org/wiki/Snake_case)
+
+- All measurements have the following tags:
+  - region           (CloudWatch Region)
+  - unit             (CloudWatch Metric Unit)
+  - {dimension-name} (Cloudwatch Dimension value - one for each metric dimension)
+
+### Example Output:
+
+```
+$ ./telegraf -config telegraf.conf -input-filter cloudwatch -test
+> cloudwatch_aws_elb,load_balancer_name=p-example,region=us-east-1,unit=seconds latency_average=0.004810798017284538,latency_maximum=0.1100282669067383,latency_minimum=0.0006084442138671875,latency_sample_count=4029,latency_sum=19.382705211639404 1459542420000000000
+```

plugins/inputs/cloudwatch/cloudwatch.go

@@ -0,0 +1,389 @@
+package cloudwatch
+
+import (
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+
+	"github.com/aws/aws-sdk-go/service/cloudwatch"
+
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/internal"
+	internalaws "github.com/influxdata/telegraf/internal/config/aws"
+	"github.com/influxdata/telegraf/internal/errchan"
+	"github.com/influxdata/telegraf/internal/limiter"
+	"github.com/influxdata/telegraf/plugins/inputs"
+)
+
+type (
+	CloudWatch struct {
+		Region    string `toml:"region"`
+		AccessKey string `toml:"access_key"`
+		SecretKey string `toml:"secret_key"`
+		RoleARN   string `toml:"role_arn"`
+		Profile   string `toml:"profile"`
+		Filename  string `toml:"shared_credential_file"`
+		Token     string `toml:"token"`
+
+		Period      internal.Duration `toml:"period"`
+		Delay       internal.Duration `toml:"delay"`
+		Namespace   string            `toml:"namespace"`
+		Metrics     []*Metric         `toml:"metrics"`
+		CacheTTL    internal.Duration `toml:"cache_ttl"`
+		client      cloudwatchClient
+		metricCache *MetricCache
+	}
+
+	Metric struct {
+		MetricNames []string     `toml:"names"`
+		Dimensions  []*Dimension `toml:"dimensions"`
+	}
+
+	Dimension struct {
+		Name  string `toml:"name"`
+		Value string `toml:"value"`
+	}
+
+	MetricCache struct {
+		TTL     time.Duration
+		Fetched time.Time
+		Metrics []*cloudwatch.Metric
+	}
+
+	cloudwatchClient interface {
+		ListMetrics(*cloudwatch.ListMetricsInput) (*cloudwatch.ListMetricsOutput, error)
+		GetMetricStatistics(*cloudwatch.GetMetricStatisticsInput) (*cloudwatch.GetMetricStatisticsOutput, error)
+	}
+)
+
+func (c *CloudWatch) SampleConfig() string {
+	return `
+  ## Amazon Region
+  region = 'us-east-1'
+
+  ## Amazon Credentials
+  ## Credentials are loaded in the following order
+  ## 1) Assumed credentials via STS if role_arn is specified
+  ## 2) explicit credentials from 'access_key' and 'secret_key'
+  ## 3) shared profile from 'profile'
+  ## 4) environment variables
+  ## 5) shared credentials file
+  ## 6) EC2 Instance Profile
+  #access_key = ""
+  #secret_key = ""
+  #token = ""
+  #role_arn = ""
+  #profile = ""
+  #shared_credential_file = ""
+
+  ## Requested CloudWatch aggregation Period (required - must be a multiple of 60s)
+  period = '1m'
+
+  ## Collection Delay (required - must account for metrics availability via CloudWatch API)
+  delay = '1m'
+
+  ## Recomended: use metric 'interval' that is a multiple of 'period' to avoid
+  ## gaps or overlap in pulled data
+  interval = '1m'
+
+  ## Configure the TTL for the internal cache of metrics.
+  ## Defaults to 1 hr if not specified
+  #cache_ttl = '10m'
+
+  ## Metric Statistic Namespace (required)
+  namespace = 'AWS/ELB'
+
+  ## Metrics to Pull (optional)
+  ## Defaults to all Metrics in Namespace if nothing is provided
+  ## Refreshes Namespace available metrics every 1h
+  #[[inputs.cloudwatch.metrics]]
+  #  names = ['Latency', 'RequestCount']
+  #
+  #  ## Dimension filters for Metric (optional)
+  #  [[inputs.cloudwatch.metrics.dimensions]]
+  #    name = 'LoadBalancerName'
+  #    value = 'p-example'
+`
+}
+
+func (c *CloudWatch) Description() string {
+	return "Pull Metric Statistics from Amazon CloudWatch"
+}
+
+func (c *CloudWatch) Gather(acc telegraf.Accumulator) error {
+	if c.client == nil {
+		c.initializeCloudWatch()
+	}
+
+	var metrics []*cloudwatch.Metric
+
+	// check for provided metric filter
+	if c.Metrics != nil {
+		metrics = []*cloudwatch.Metric{}
+		for _, m := range c.Metrics {
+			if !hasWilcard(m.Dimensions) {
+				dimensions := make([]*cloudwatch.Dimension, len(m.Dimensions))
+				for k, d := range m.Dimensions {
+					fmt.Printf("Dimension [%s]:[%s]\n", d.Name, d.Value)
+					dimensions[k] = &cloudwatch.Dimension{
+						Name:  aws.String(d.Name),
+						Value: aws.String(d.Value),
+					}
+				}
+				for _, name := range m.MetricNames {
+					metrics = append(metrics, &cloudwatch.Metric{
+						Namespace:  aws.String(c.Namespace),
+						MetricName: aws.String(name),
+						Dimensions: dimensions,
+					})
+				}
+			} else {
+				allMetrics, err := c.fetchNamespaceMetrics()
+				if err != nil {
+					return err
+				}
+				for _, name := range m.MetricNames {
+					for _, metric := range allMetrics {
+						if isSelected(metric, m.Dimensions) {
+							metrics = append(metrics, &cloudwatch.Metric{
+								Namespace:  aws.String(c.Namespace),
+								MetricName: aws.String(name),
+								Dimensions: metric.Dimensions,
+							})
+						}
+					}
+				}
+			}
+
+		}
+	} else {
+		var err error
+		metrics, err = c.fetchNamespaceMetrics()
+		if err != nil {
+			return err
+		}
+	}
+
+	metricCount := len(metrics)
+	errChan := errchan.New(metricCount)
+
+	now := time.Now()
+
+	// limit concurrency or we can easily exhaust user connection limit
+	// see cloudwatch API request limits:
+	// http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_limits.html
+	lmtr := limiter.NewRateLimiter(10, time.Second)
+	defer lmtr.Stop()
+	var wg sync.WaitGroup
+	wg.Add(len(metrics))
+	for _, m := range metrics {
+		<-lmtr.C
+		go func(inm *cloudwatch.Metric) {
+			defer wg.Done()
+			c.gatherMetric(acc, inm, now, errChan.C)
+		}(m)
+	}
+	wg.Wait()
+
+	return errChan.Error()
+}
+
+func init() {
+	inputs.Add("cloudwatch", func() telegraf.Input {
+		ttl, _ := time.ParseDuration("1hr")
+		return &CloudWatch{
+			CacheTTL: internal.Duration{Duration: ttl},
+		}
+	})
+}
+
+/*
+ * Initialize CloudWatch client
+ */
+func (c *CloudWatch) initializeCloudWatch() error {
+	credentialConfig := &internalaws.CredentialConfig{
+		Region:    c.Region,
+		AccessKey: c.AccessKey,
+		SecretKey: c.SecretKey,
+		RoleARN:   c.RoleARN,
+		Profile:   c.Profile,
+		Filename:  c.Filename,
+		Token:     c.Token,
+	}
+	configProvider := credentialConfig.Credentials()
+
+	c.client = cloudwatch.New(configProvider)
+	return nil
+}
+
+/*
+ * Fetch available metrics for given CloudWatch Namespace
+ */
+func (c *CloudWatch) fetchNamespaceMetrics() (metrics []*cloudwatch.Metric, err error) {
+	if c.metricCache != nil && c.metricCache.IsValid() {
+		metrics = c.metricCache.Metrics
+		return
+	}
+
+	metrics = []*cloudwatch.Metric{}
+
+	var token *string
+	for more := true; more; {
+		params := &cloudwatch.ListMetricsInput{
+			Namespace:  aws.String(c.Namespace),
+			Dimensions: []*cloudwatch.DimensionFilter{},
+			NextToken:  token,
+			MetricName: nil,
+		}
+
+		resp, err := c.client.ListMetrics(params)
+		if err != nil {
+			return nil, err
+		}
+
+		metrics = append(metrics, resp.Metrics...)
+
+		token = resp.NextToken
+		more = token != nil
+	}
+
+	c.metricCache = &MetricCache{
+		Metrics: metrics,
+		Fetched: time.Now(),
+		TTL:     c.CacheTTL.Duration,
+	}
+
+	return
+}
+
+/*
+ * Gather given Metric and emit any error
+ */
+func (c *CloudWatch) gatherMetric(
+	acc telegraf.Accumulator,
+	metric *cloudwatch.Metric,
+	now time.Time,
+	errChan chan error,
+) {
+	params := c.getStatisticsInput(metric, now)
+	resp, err := c.client.GetMetricStatistics(params)
+	if err != nil {
+		errChan <- err
+		return
+	}
+
+	for _, point := range resp.Datapoints {
+		tags := map[string]string{
+			"region": c.Region,
+			"unit":   snakeCase(*point.Unit),
+		}
+
+		for _, d := range metric.Dimensions {
+			tags[snakeCase(*d.Name)] = *d.Value
+		}
+
+		// record field for each statistic
+		fields := map[string]interface{}{}
+
+		if point.Average != nil {
+			fields[formatField(*metric.MetricName, cloudwatch.StatisticAverage)] = *point.Average
+		}
+		if point.Maximum != nil {
+			fields[formatField(*metric.MetricName, cloudwatch.StatisticMaximum)] = *point.Maximum
+		}
+		if point.Minimum != nil {
+			fields[formatField(*metric.MetricName, cloudwatch.StatisticMinimum)] = *point.Minimum
+		}
+		if point.SampleCount != nil {
+			fields[formatField(*metric.MetricName, cloudwatch.StatisticSampleCount)] = *point.SampleCount
+		}
+		if point.Sum != nil {
+			fields[formatField(*metric.MetricName, cloudwatch.StatisticSum)] = *point.Sum
+		}
+
+		acc.AddFields(formatMeasurement(c.Namespace), fields, tags, *point.Timestamp)
+	}
+
+	errChan <- nil
+}
+
+/*
+ * Formatting helpers
+ */
+func formatField(metricName string, statistic string) string {
+	return fmt.Sprintf("%s_%s", snakeCase(metricName), snakeCase(statistic))
+}
+
+func formatMeasurement(namespace string) string {
+	namespace = strings.Replace(namespace, "/", "_", -1)
+	namespace = snakeCase(namespace)
+	return fmt.Sprintf("cloudwatch_%s", namespace)
+}
+
+func snakeCase(s string) string {
+	s = internal.SnakeCase(s)
+	s = strings.Replace(s, "__", "_", -1)
+	return s
+}
+
+/*
+ * Map Metric to *cloudwatch.GetMetricStatisticsInput for given timeframe
+ */
+func (c *CloudWatch) getStatisticsInput(metric *cloudwatch.Metric, now time.Time) *cloudwatch.GetMetricStatisticsInput {
+	end := now.Add(-c.Delay.Duration)
+
+	input := &cloudwatch.GetMetricStatisticsInput{
+		StartTime:  aws.Time(end.Add(-c.Period.Duration)),
+		EndTime:    aws.Time(end),
+		MetricName: metric.MetricName,
+		Namespace:  metric.Namespace,
+		Period:     aws.Int64(int64(c.Period.Duration.Seconds())),
+		Dimensions: metric.Dimensions,
+		Statistics: []*string{
+			aws.String(cloudwatch.StatisticAverage),
+			aws.String(cloudwatch.StatisticMaximum),
+			aws.String(cloudwatch.StatisticMinimum),
+			aws.String(cloudwatch.StatisticSum),
+			aws.String(cloudwatch.StatisticSampleCount)},
+	}
+	return input
+}
+
+/*
+ * Check Metric Cache validity
+ */
+func (c *MetricCache) IsValid() bool {
+	return c.Metrics != nil && time.Since(c.Fetched) < c.TTL
+}
+
+func hasWilcard(dimensions []*Dimension) bool {
+	for _, d := range dimensions {
+		if d.Value == "" || d.Value == "*" {
+			return true
+		}
+	}
+	return false
+}
+
+func isSelected(metric *cloudwatch.Metric, dimensions []*Dimension) bool {
+	if len(metric.Dimensions) != len(dimensions) {
+		return false
+	}
+	for _, d := range dimensions {
+		selected := false
+		for _, d2 := range metric.Dimensions {
+			if d.Name == *d2.Name {
+				if d.Value == "" || d.Value == "*" || d.Value == *d2.Value {
+					selected = true
+				}
+			}
+		}
+		if !selected {
+			return false
+		}
+	}
+	return true
+}

plugins/inputs/cloudwatch/cloudwatch_test.go

@@ -0,0 +1,131 @@
+package cloudwatch
+
+import (
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/cloudwatch"
+	"github.com/influxdata/telegraf/internal"
+	"github.com/influxdata/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+)
+
+type mockCloudWatchClient struct{}
+
+func (m *mockCloudWatchClient) ListMetrics(params *cloudwatch.ListMetricsInput) (*cloudwatch.ListMetricsOutput, error) {
+	metric := &cloudwatch.Metric{
+		Namespace:  params.Namespace,
+		MetricName: aws.String("Latency"),
+		Dimensions: []*cloudwatch.Dimension{
+			&cloudwatch.Dimension{
+				Name:  aws.String("LoadBalancerName"),
+				Value: aws.String("p-example"),
+			},
+		},
+	}
+
+	result := &cloudwatch.ListMetricsOutput{
+		Metrics: []*cloudwatch.Metric{metric},
+	}
+	return result, nil
+}
+
+func (m *mockCloudWatchClient) GetMetricStatistics(params *cloudwatch.GetMetricStatisticsInput) (*cloudwatch.GetMetricStatisticsOutput, error) {
+	dataPoint := &cloudwatch.Datapoint{
+		Timestamp:   params.EndTime,
+		Minimum:     aws.Float64(0.1),
+		Maximum:     aws.Float64(0.3),
+		Average:     aws.Float64(0.2),
+		Sum:         aws.Float64(123),
+		SampleCount: aws.Float64(100),
+		Unit:        aws.String("Seconds"),
+	}
+	result := &cloudwatch.GetMetricStatisticsOutput{
+		Label:      aws.String("Latency"),
+		Datapoints: []*cloudwatch.Datapoint{dataPoint},
+	}
+	return result, nil
+}
+
+func TestGather(t *testing.T) {
+	duration, _ := time.ParseDuration("1m")
+	internalDuration := internal.Duration{
+		Duration: duration,
+	}
+	c := &CloudWatch{
+		Region:    "us-east-1",
+		Namespace: "AWS/ELB",
+		Delay:     internalDuration,
+		Period:    internalDuration,
+	}
+
+	var acc testutil.Accumulator
+	c.client = &mockCloudWatchClient{}
+
+	c.Gather(&acc)
+
+	fields := map[string]interface{}{}
+	fields["latency_minimum"] = 0.1
+	fields["latency_maximum"] = 0.3
+	fields["latency_average"] = 0.2
+	fields["latency_sum"] = 123.0
+	fields["latency_sample_count"] = 100.0
+
+	tags := map[string]string{}
+	tags["unit"] = "seconds"
+	tags["region"] = "us-east-1"
+	tags["load_balancer_name"] = "p-example"
+
+	assert.True(t, acc.HasMeasurement("cloudwatch_aws_elb"))
+	acc.AssertContainsTaggedFields(t, "cloudwatch_aws_elb", fields, tags)
+
+}
+
+func TestGenerateStatisticsInputParams(t *testing.T) {
+	d := &cloudwatch.Dimension{
+		Name:  aws.String("LoadBalancerName"),
+		Value: aws.String("p-example"),
+	}
+
+	m := &cloudwatch.Metric{
+		MetricName: aws.String("Latency"),
+		Dimensions: []*cloudwatch.Dimension{d},
+	}
+
+	duration, _ := time.ParseDuration("1m")
+	internalDuration := internal.Duration{
+		Duration: duration,
+	}
+
+	c := &CloudWatch{
+		Namespace: "AWS/ELB",
+		Delay:     internalDuration,
+		Period:    internalDuration,
+	}
+
+	c.initializeCloudWatch()
+
+	now := time.Now()
+
+	params := c.getStatisticsInput(m, now)
+
+	assert.EqualValues(t, *params.EndTime, now.Add(-c.Delay.Duration))
+	assert.EqualValues(t, *params.StartTime, now.Add(-c.Period.Duration).Add(-c.Delay.Duration))
+	assert.Len(t, params.Dimensions, 1)
+	assert.Len(t, params.Statistics, 5)
+	assert.EqualValues(t, *params.Period, 60)
+}
+
+func TestMetricsCacheTimeout(t *testing.T) {
+	ttl, _ := time.ParseDuration("5ms")
+	cache := &MetricCache{
+		Metrics: []*cloudwatch.Metric{},
+		Fetched: time.Now(),
+		TTL:     ttl,
+	}
+
+	assert.True(t, cache.IsValid())
+	time.Sleep(ttl)
+	assert.False(t, cache.IsValid())
+}

plugins/inputs/conntrack/README.md

@@ -0,0 +1,56 @@
+# Conntrack Plugin
+
+Collects stats from Netfilter's conntrack-tools.
+
+The conntrack-tools provide a mechanism for tracking various aspects of
+network connections as they are processed by netfilter. At runtime, 
+conntrack exposes many of those connection statistics within /proc/sys/net.
+Depending on your kernel version, these files can be found in either
+/proc/sys/net/ipv4/netfilter or /proc/sys/net/netfilter and will be
+prefixed with either ip_ or nf_.  This plugin reads the files specified 
+in its configuration and publishes each one as a field, with the prefix
+normalized to ip_.  
+
+In order to simplify configuration in a heterogeneous environment, a superset
+of directory and filenames can be specified.  Any locations that don't exist
+will be ignored.
+
+For more information on conntrack-tools, see the 
+[Netfilter Documentation](http://conntrack-tools.netfilter.org/).
+
+
+### Configuration:
+
+```toml
+ # Collects conntrack stats from the configured directories and files.
+ [[inputs.conntrack]]
+   ## The following defaults would work with multiple versions of conntrack.
+   ## Note the nf_ and ip_ filename prefixes are mutually exclusive across
+   ## kernel versions, as are the directory locations.
+
+   ## Superset of filenames to look for within the conntrack dirs.
+   ## Missing files will be ignored.
+   files = ["ip_conntrack_count","ip_conntrack_max",
+            "nf_conntrack_count","nf_conntrack_max"]
+
+   ## Directories to search within for the conntrack files above.
+   ## Missing directrories will be ignored.
+   dirs = ["/proc/sys/net/ipv4/netfilter","/proc/sys/net/netfilter"]
+```
+
+### Measurements & Fields:
+
+- conntrack
+    - ip_conntrack_count (int, count): the number of entries in the conntrack table 
+    - ip_conntrack_max (int, size): the max capacity of the conntrack table
+
+### Tags:
+
+This input does not use tags.
+
+### Example Output:
+
+```
+$ ./telegraf -config telegraf.conf -input-filter conntrack -test
+conntrack,host=myhost ip_conntrack_count=2,ip_conntrack_max=262144 1461620427667995735
+```

plugins/inputs/conntrack/conntrack.go

@@ -0,0 +1,119 @@
+// +build linux
+
+package conntrack
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/plugins/inputs"
+	"log"
+	"path/filepath"
+)
+
+type Conntrack struct {
+	Path  string
+	Dirs  []string
+	Files []string
+}
+
+const (
+	inputName = "conntrack"
+)
+
+var dfltDirs = []string{
+	"/proc/sys/net/ipv4/netfilter",
+	"/proc/sys/net/netfilter",
+}
+
+var dfltFiles = []string{
+	"ip_conntrack_count",
+	"ip_conntrack_max",
+	"nf_conntrack_count",
+	"nf_conntrack_max",
+}
+
+func (c *Conntrack) setDefaults() {
+	if len(c.Dirs) == 0 {
+		c.Dirs = dfltDirs
+	}
+
+	if len(c.Files) == 0 {
+		c.Files = dfltFiles
+	}
+}
+
+func (c *Conntrack) Description() string {
+	return "Collects conntrack stats from the configured directories and files."
+}
+
+var sampleConfig = `
+   ## The following defaults would work with multiple versions of conntrack.
+   ## Note the nf_ and ip_ filename prefixes are mutually exclusive across
+   ## kernel versions, as are the directory locations.
+
+   ## Superset of filenames to look for within the conntrack dirs.
+   ## Missing files will be ignored.
+   files = ["ip_conntrack_count","ip_conntrack_max",
+            "nf_conntrack_count","nf_conntrack_max"]
+
+   ## Directories to search within for the conntrack files above.
+   ## Missing directrories will be ignored.
+   dirs = ["/proc/sys/net/ipv4/netfilter","/proc/sys/net/netfilter"]
+`
+
+func (c *Conntrack) SampleConfig() string {
+	return sampleConfig
+}
+
+func (c *Conntrack) Gather(acc telegraf.Accumulator) error {
+	c.setDefaults()
+
+	var metricKey string
+	fields := make(map[string]interface{})
+
+	for _, dir := range c.Dirs {
+		for _, file := range c.Files {
+			// NOTE: no system will have both nf_ and ip_ prefixes,
+			// so we're safe to branch on suffix only.
+			parts := strings.SplitN(file, "_", 2)
+			if len(parts) < 2 {
+				continue
+			}
+			metricKey = "ip_" + parts[1]
+
+			fName := filepath.Join(dir, file)
+			if _, err := os.Stat(fName); err != nil {
+				continue
+			}
+
+			contents, err := ioutil.ReadFile(fName)
+			if err != nil {
+				log.Printf("failed to read file '%s': %v", fName, err)
+			}
+
+			v := strings.TrimSpace(string(contents))
+			fields[metricKey], err = strconv.ParseFloat(v, 64)
+			if err != nil {
+				log.Printf("failed to parse metric, expected number but "+
+					" found '%s': %v", v, err)
+			}
+		}
+	}
+
+	if len(fields) == 0 {
+		return fmt.Errorf("Conntrack input failed to collect metrics. " +
+			"Is the conntrack kernel module loaded?")
+	}
+
+	acc.AddFields(inputName, fields, nil)
+	return nil
+}
+
+func init() {
+	inputs.Add(inputName, func() telegraf.Input { return &Conntrack{} })
+}

plugins/inputs/conntrack/conntrack_notlinux.go

@@ -0,0 +1,3 @@
+// +build !linux
+
+package conntrack

plugins/inputs/conntrack/conntrack_test.go

@@ -0,0 +1,90 @@
+// +build linux
+
+package conntrack
+
+import (
+	"github.com/influxdata/telegraf/testutil"
+	"github.com/stretchr/testify/assert"
+	"io/ioutil"
+	"os"
+	"path"
+	"strconv"
+	"strings"
+	"testing"
+)
+
+func restoreDflts(savedFiles, savedDirs []string) {
+	dfltFiles = savedFiles
+	dfltDirs = savedDirs
+}
+
+func TestNoFilesFound(t *testing.T) {
+	defer restoreDflts(dfltFiles, dfltDirs)
+
+	dfltFiles = []string{"baz.txt"}
+	dfltDirs = []string{"./foo/bar"}
+	c := &Conntrack{}
+	acc := &testutil.Accumulator{}
+	err := c.Gather(acc)
+
+	assert.EqualError(t, err, "Conntrack input failed to collect metrics. "+
+		"Is the conntrack kernel module loaded?")
+}
+
+func TestDefaultsUsed(t *testing.T) {
+	defer restoreDflts(dfltFiles, dfltDirs)
+	tmpdir, err := ioutil.TempDir("", "tmp1")
+	assert.NoError(t, err)
+	defer os.Remove(tmpdir)
+
+	tmpFile, err := ioutil.TempFile(tmpdir, "ip_conntrack_count")
+	assert.NoError(t, err)
+
+	dfltDirs = []string{tmpdir}
+	fname := path.Base(tmpFile.Name())
+	dfltFiles = []string{fname}
+
+	count := 1234321
+	ioutil.WriteFile(tmpFile.Name(), []byte(strconv.Itoa(count)), 0660)
+	c := &Conntrack{}
+	acc := &testutil.Accumulator{}
+
+	c.Gather(acc)
+	acc.AssertContainsFields(t, inputName, map[string]interface{}{
+		fname: float64(count)})
+}
+
+func TestConfigsUsed(t *testing.T) {
+	defer restoreDflts(dfltFiles, dfltDirs)
+	tmpdir, err := ioutil.TempDir("", "tmp1")
+	assert.NoError(t, err)
+	defer os.Remove(tmpdir)
+
+	cntFile, err := ioutil.TempFile(tmpdir, "nf_conntrack_count")
+	maxFile, err := ioutil.TempFile(tmpdir, "nf_conntrack_max")
+	assert.NoError(t, err)
+
+	dfltDirs = []string{tmpdir}
+	cntFname := path.Base(cntFile.Name())
+	maxFname := path.Base(maxFile.Name())
+	dfltFiles = []string{cntFname, maxFname}
+
+	count := 1234321
+	max := 9999999
+	ioutil.WriteFile(cntFile.Name(), []byte(strconv.Itoa(count)), 0660)
+	ioutil.WriteFile(maxFile.Name(), []byte(strconv.Itoa(max)), 0660)
+	c := &Conntrack{}
+	acc := &testutil.Accumulator{}
+
+	c.Gather(acc)
+
+	fix := func(s string) string {
+		return strings.Replace(s, "nf_", "ip_", 1)
+	}
+
+	acc.AssertContainsFields(t, inputName,
+		map[string]interface{}{
+			fix(cntFname): float64(count),
+			fix(maxFname): float64(max),
+		})
+}