From 1c7715780ebc82bd97c2f58292381f421c99ed0f Mon Sep 17 00:00:00 2001 From: Cameron Sparr Date: Fri, 4 Nov 2016 14:12:22 +0000 Subject: [PATCH] Documentation improvements - fully document aggregator and processor plugins - improve readme.md closes #1989 --- README.md | 229 ++++++++++++++------------- docs/AGGREGATORS_AND_PROCESSORS.md | 59 +++++++ plugins/aggregators/minmax/README.md | 42 +++++ plugins/processors/printer/README.md | 14 ++ 4 files changed, 233 insertions(+), 111 deletions(-) create mode 100644 docs/AGGREGATORS_AND_PROCESSORS.md create mode 100644 plugins/aggregators/minmax/README.md create mode 100644 plugins/processors/printer/README.md diff --git a/README.md b/README.md index efba9e6c7..92ebf19ee 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,23 @@ # Telegraf [![Circle CI](https://circleci.com/gh/influxdata/telegraf.svg?style=svg)](https://circleci.com/gh/influxdata/telegraf) [![Docker pulls](https://img.shields.io/docker/pulls/library/telegraf.svg)](https://hub.docker.com/_/telegraf/) -Telegraf is an agent written in Go for collecting metrics from the system it's -running on, or from other services, and writing them into InfluxDB or other -[outputs](https://github.com/influxdata/telegraf#supported-output-plugins). +Telegraf is an agent written in Go for collecting, processing, aggregating, +and writing metrics. Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics from well known services (like Hadoop, Postgres, or Redis) and third party APIs (like Mailchimp, AWS CloudWatch, or Google Analytics). -New input and output plugins are designed to be easy to contribute, +Telegraf is plugin-driven and has the concept of 4 distinct plugins: + +1. [Input Plugins](#input-plugins) collect metrics from the system, services, or 3rd party APIs +2. [Processor Plugins](#processor-plugins) transform, decorate, and/or filter metrics +3. [Aggregator Plugins](#aggregator-plugins) create aggregate metrics (e.g. mean, min, max, quantiles, etc.) +4. [Output Plugins](#output-plugins) write metrics to various destinations + +For more information on Processor and Aggregator plugins please [read this](./docs/AGGREGATORS_AND_PROCESSORS.md). + +New plugins are designed to be easy to contribute, we'll eagerly accept pull requests and will manage the set of plugins that Telegraf supports. See the [contributing guide](CONTRIBUTING.md) for instructions on writing @@ -39,7 +47,7 @@ controlled via `systemctl [action] telegraf` ### yum/apt Repositories: There is a yum/apt repo available for the whole InfluxData stack, see -[here](https://docs.influxdata.com/influxdb/v0.10/introduction/installation/#installation) +[here](https://docs.influxdata.com/influxdb/latest/introduction/installation/#installation) for instructions on setting up the repo. Once it is configured, you will be able to use this repo to install & update telegraf. @@ -127,77 +135,71 @@ telegraf --config telegraf.conf -input-filter cpu:mem -output-filter influxdb See the [configuration guide](docs/CONFIGURATION.md) for a rundown of the more advanced configuration options. -## Supported Input Plugins +## Input Plugins -Telegraf currently has support for collecting metrics from many sources. For -more information on each, please look at the directory of the same name in -`plugins/inputs`. - -Currently implemented sources: - -* [aws cloudwatch](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cloudwatch) -* [aerospike](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/aerospike) -* [apache](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/apache) -* [bcache](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/bcache) -* [cassandra](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cassandra) -* [ceph](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) -* [chrony](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/chrony) -* [consul](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/consul) -* [conntrack](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/conntrack) -* [couchbase](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/couchbase) -* [couchdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/couchdb) -* [disque](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/disque) -* [dns query time](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/dns_query) -* [docker](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/docker) -* [dovecot](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/dovecot) -* [elasticsearch](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/elasticsearch) -* [exec](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec) (generic executable plugin, support JSON, influx, graphite and nagios) -* [filestat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/filestat) -* [haproxy](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/haproxy) -* [hddtemp](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/hddtemp) -* [http_response](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/http_response) -* [httpjson](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/httpjson) (generic JSON-emitting http service plugin) -* [influxdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/influxdb) -* [ipmi_sensor](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ipmi_sensor) -* [iptables](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/iptables) -* [jolokia](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/jolokia) -* [leofs](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/leofs) -* [lustre2](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/lustre2) -* [mailchimp](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mailchimp) -* [memcached](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/memcached) -* [mesos](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mesos) -* [mongodb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mongodb) -* [mysql](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mysql) -* [net_response](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/net_response) -* [nginx](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nginx) -* [nsq](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nsq) -* [nstat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nstat) -* [ntpq](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ntpq) -* [phpfpm](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/phpfpm) -* [phusion passenger](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/passenger) -* [ping](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ping) -* [postgresql](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/postgresql) -* [postgresql_extensible](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/postgresql_extensible) -* [powerdns](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/powerdns) -* [procstat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/procstat) -* [prometheus](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/prometheus) -* [puppetagent](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/puppetagent) -* [rabbitmq](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/rabbitmq) -* [raindrops](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/raindrops) -* [redis](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/redis) -* [rethinkdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/rethinkdb) -* [riak](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/riak) -* [sensors](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sensors) -* [snmp](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/snmp) -* [snmp_legacy](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/snmp_legacy) -* [sql server](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver) (microsoft) -* [twemproxy](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/twemproxy) -* [varnish](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/varnish) -* [zfs](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/zfs) -* [zookeeper](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/zookeeper) -* [win_perf_counters ](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/win_perf_counters) (windows performance counters) -* [sysstat](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sysstat) -* [system](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/system) +* [aws cloudwatch](./plugins/inputs/cloudwatch) +* [aerospike](./plugins/inputs/aerospike) +* [apache](./plugins/inputs/apache) +* [bcache](./plugins/inputs/bcache) +* [cassandra](./plugins/inputs/cassandra) +* [ceph](./plugins/inputs/ceph) +* [chrony](./plugins/inputs/chrony) +* [consul](./plugins/inputs/consul) +* [conntrack](./plugins/inputs/conntrack) +* [couchbase](./plugins/inputs/couchbase) +* [couchdb](./plugins/inputs/couchdb) +* [disque](./plugins/inputs/disque) +* [dns query time](./plugins/inputs/dns_query) +* [docker](./plugins/inputs/docker) +* [dovecot](./plugins/inputs/dovecot) +* [elasticsearch](./plugins/inputs/elasticsearch) +* [exec](./plugins/inputs/exec) (generic executable plugin, support JSON, influx, graphite and nagios) +* [filestat](./plugins/inputs/filestat) +* [haproxy](./plugins/inputs/haproxy) +* [hddtemp](./plugins/inputs/hddtemp) +* [http_response](./plugins/inputs/http_response) +* [httpjson](./plugins/inputs/httpjson) (generic JSON-emitting http service plugin) +* [influxdb](./plugins/inputs/influxdb) +* [ipmi_sensor](./plugins/inputs/ipmi_sensor) +* [iptables](./plugins/inputs/iptables) +* [jolokia](./plugins/inputs/jolokia) +* [leofs](./plugins/inputs/leofs) +* [lustre2](./plugins/inputs/lustre2) +* [mailchimp](./plugins/inputs/mailchimp) +* [memcached](./plugins/inputs/memcached) +* [mesos](./plugins/inputs/mesos) +* [mongodb](./plugins/inputs/mongodb) +* [mysql](./plugins/inputs/mysql) +* [net_response](./plugins/inputs/net_response) +* [nginx](./plugins/inputs/nginx) +* [nsq](./plugins/inputs/nsq) +* [nstat](./plugins/inputs/nstat) +* [ntpq](./plugins/inputs/ntpq) +* [phpfpm](./plugins/inputs/phpfpm) +* [phusion passenger](./plugins/inputs/passenger) +* [ping](./plugins/inputs/ping) +* [postgresql](./plugins/inputs/postgresql) +* [postgresql_extensible](./plugins/inputs/postgresql_extensible) +* [powerdns](./plugins/inputs/powerdns) +* [procstat](./plugins/inputs/procstat) +* [prometheus](./plugins/inputs/prometheus) +* [puppetagent](./plugins/inputs/puppetagent) +* [rabbitmq](./plugins/inputs/rabbitmq) +* [raindrops](./plugins/inputs/raindrops) +* [redis](./plugins/inputs/redis) +* [rethinkdb](./plugins/inputs/rethinkdb) +* [riak](./plugins/inputs/riak) +* [sensors](./plugins/inputs/sensors) +* [snmp](./plugins/inputs/snmp) +* [snmp_legacy](./plugins/inputs/snmp_legacy) +* [sql server](./plugins/inputs/sqlserver) (microsoft) +* [twemproxy](./plugins/inputs/twemproxy) +* [varnish](./plugins/inputs/varnish) +* [zfs](./plugins/inputs/zfs) +* [zookeeper](./plugins/inputs/zookeeper) +* [win_perf_counters ](./plugins/inputs/win_perf_counters) (windows performance counters) +* [sysstat](./plugins/inputs/sysstat) +* [system](./plugins/inputs/system) * cpu * mem * net @@ -211,45 +213,50 @@ Currently implemented sources: Telegraf can also collect metrics via the following service plugins: -* [http_listener](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/http_listener) -* [kafka_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/kafka_consumer) -* [mqtt_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer) -* [nats_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nats_consumer) -* [nsq_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nsq_consumer) -* [logparser](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/logparser) -* [statsd](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd) -* [tail](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tail) -* [tcp_listener](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tcp_listener) -* [udp_listener](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/udp_listener) -* [webhooks](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks) - * [filestack](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/filestack) - * [github](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/github) - * [mandrill](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/mandrill) - * [rollbar](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/rollbar) +* [http_listener](./plugins/inputs/http_listener) +* [kafka_consumer](./plugins/inputs/kafka_consumer) +* [mqtt_consumer](./plugins/inputs/mqtt_consumer) +* [nats_consumer](./plugins/inputs/nats_consumer) +* [nsq_consumer](./plugins/inputs/nsq_consumer) +* [logparser](./plugins/inputs/logparser) +* [statsd](./plugins/inputs/statsd) +* [tail](./plugins/inputs/tail) +* [tcp_listener](./plugins/inputs/tcp_listener) +* [udp_listener](./plugins/inputs/udp_listener) +* [webhooks](./plugins/inputs/webhooks) + * [filestack](./plugins/inputs/webhooks/filestack) + * [github](./plugins/inputs/webhooks/github) + * [mandrill](./plugins/inputs/webhooks/mandrill) + * [rollbar](./plugins/inputs/webhooks/rollbar) -We'll be adding support for many more over the coming months. Read on if you -want to add support for another service or third-party API. +## Processor Plugins -## Supported Output Plugins +* [printer](./plugins/processors/printer) -* [influxdb](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/influxdb) -* [amon](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/amon) -* [amqp](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/amqp) -* [aws kinesis](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kinesis) -* [aws cloudwatch](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/cloudwatch) -* [datadog](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/datadog) -* [file](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/file) -* [graphite](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/graphite) -* [graylog](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/graylog) -* [instrumental](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/instrumental) -* [kafka](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kafka) -* [librato](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/librato) -* [mqtt](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/mqtt) -* [nats](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/nats) -* [nsq](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/nsq) -* [opentsdb](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/opentsdb) -* [prometheus](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/prometheus_client) -* [riemann](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/riemann) +## Aggregator Plugins + +* [minmax](./plugins/aggregators/minmax) + +## Output Plugins + +* [influxdb](./plugins/outputs/influxdb) +* [amon](./plugins/outputs/amon) +* [amqp](./plugins/outputs/amqp) +* [aws kinesis](./plugins/outputs/kinesis) +* [aws cloudwatch](./plugins/outputs/cloudwatch) +* [datadog](./plugins/outputs/datadog) +* [file](./plugins/outputs/file) +* [graphite](./plugins/outputs/graphite) +* [graylog](./plugins/outputs/graylog) +* [instrumental](./plugins/outputs/instrumental) +* [kafka](./plugins/outputs/kafka) +* [librato](./plugins/outputs/librato) +* [mqtt](./plugins/outputs/mqtt) +* [nats](./plugins/outputs/nats) +* [nsq](./plugins/outputs/nsq) +* [opentsdb](./plugins/outputs/opentsdb) +* [prometheus](./plugins/outputs/prometheus_client) +* [riemann](./plugins/outputs/riemann) ## Contributing diff --git a/docs/AGGREGATORS_AND_PROCESSORS.md b/docs/AGGREGATORS_AND_PROCESSORS.md new file mode 100644 index 000000000..af4a0e6d0 --- /dev/null +++ b/docs/AGGREGATORS_AND_PROCESSORS.md @@ -0,0 +1,59 @@ +# Telegraf Aggregator & Processor Plugins + +As of release 1.1.0, Telegraf has the concept of Aggregator and Processor Plugins. + +These plugins sit in-between Input & Output plugins, aggregating and processing +metrics as they pass through Telegraf: + +``` +┌───────────┐ +│ │ +│ CPU │───┐ +│ │ │ +└───────────┘ │ + │ +┌───────────┐ │ ┌───────────┐ +│ │ │ │ │ +│ Memory │───┤ ┌──▶│ InfluxDB │ +│ │ │ │ │ │ +└───────────┘ │ ┌─────────────┐ ┌─────────────┐ │ └───────────┘ + │ │ │ │Aggregate │ │ +┌───────────┐ │ │Process │ │ - mean │ │ ┌───────────┐ +│ │ │ │ - transform │ │ - quantiles │ │ │ │ +│ MySQL │───┼───▶│ - decorate │────▶│ - min/max │───┼──▶│ File │ +│ │ │ │ - filter │ │ - count │ │ │ │ +└───────────┘ │ │ │ │ │ │ └───────────┘ + │ └─────────────┘ └─────────────┘ │ +┌───────────┐ │ │ ┌───────────┐ +│ │ │ │ │ │ +│ SNMP │───┤ └──▶│ Kafka │ +│ │ │ │ │ +└───────────┘ │ └───────────┘ + │ +┌───────────┐ │ +│ │ │ +│ Docker │───┘ +│ │ +└───────────┘ +``` + +Both Aggregators and Processors analyze metrics as they pass through Telegraf. + +**Processor** plugins process metrics as they pass through and immediately emit +results based on the values they process. For example, this could be printing +all metrics or adding a tag to all metrics that pass through. + +**Aggregator** plugins, on the other hand, are a bit more complicated. Aggregators +are typically for emitting new _aggregate_ metrics, such as a running mean, +minimum, maximum, quantiles, or standard deviation. For this reason, all _aggregator_ +plugins are configured with a `period`. The `period` is the size of the window +of metrics that each _aggregate_ represents. In other words, the emitted +_aggregate_ metric will be the aggregated value of the past `period` seconds. +Since many users will only care about their aggregates and not every single metric +gathered, there is also a `drop_original` argument, which tells Telegraf to only +emit the aggregates and not the original metrics. + +**NOTE** That since aggregators only aggregate metrics within their period, that +historical data is not supported. In other words, if your metric timestamp is more +than `now() - period` in the past, it will not be aggregated. If this is a feature +that you need, please comment on this [github issue](https://github.com/influxdata/telegraf/issues/1992) diff --git a/plugins/aggregators/minmax/README.md b/plugins/aggregators/minmax/README.md new file mode 100644 index 000000000..f7405b78c --- /dev/null +++ b/plugins/aggregators/minmax/README.md @@ -0,0 +1,42 @@ +# MinMax Aggregator Plugin + +The minmax aggregator plugin aggregates min & max values of each field it sees, +emitting the aggrate every `period` seconds. + +### Configuration: + +```toml +# Keep the aggregate min/max of each metric passing through. +[[aggregators.minmax]] + ## General Aggregator Arguments: + ## The period on which to flush & clear the aggregator. + period = "30s" + ## If true, the original metric will be dropped by the + ## aggregator and will not get sent to the output plugins. + drop_original = false +``` + +### Measurements & Fields: + +- measurement1 + - field1_max + - field1_min + +### Tags: + +No tags are applied by this aggregator. + +### Example Output: + +``` +$ telegraf --config telegraf.conf --quiet +system,host=tars load1=1.72 1475583980000000000 +system,host=tars load1=1.6 1475583990000000000 +system,host=tars load1=1.66 1475584000000000000 +system,host=tars load1=1.63 1475584010000000000 +system,host=tars load1_max=1.72,load1_min=1.6 1475584010000000000 +system,host=tars load1=1.46 1475584020000000000 +system,host=tars load1=1.39 1475584030000000000 +system,host=tars load1=1.41 1475584040000000000 +system,host=tars load1_max=1.46,load1_min=1.39 1475584040000000000 +``` diff --git a/plugins/processors/printer/README.md b/plugins/processors/printer/README.md new file mode 100644 index 000000000..9a79e16fd --- /dev/null +++ b/plugins/processors/printer/README.md @@ -0,0 +1,14 @@ +# Printer Processor Plugin + +The printer processor plugin simple prints every metric passing through it. + +### Configuration: + +```toml +# Print all metrics that pass through this filter. +[[processors.printer]] +``` + +### Tags: + +No tags are applied by this processor.