dom
/
telegraf
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Update histogram aggregator documentation (#3133)

This commit is contained in:
Daniel Nelson 2017-08-18 13:24:05 -07:00 committed by GitHub
parent d5bfc683fe
commit 580fd73468
3 changed files with 81 additions and 112 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
etc/telegraf.conf
View File

@ -602,30 +602,30 @@
# AGGREGATOR PLUGINS # # AGGREGATOR PLUGINS #
############################################################################### ###############################################################################
# # Keep the aggregate histogram of each metric passing through. # # Create aggregate histograms.
# [[aggregators.histogram]] # [[aggregators.histogram]]
# ## General Aggregator Arguments: # ## The period in which to flush the aggregator.
# ## The period on which to flush & clear the aggregator.
# period = "30s" # period = "30s"
#
# ## If true, the original metric will be dropped by the # ## If true, the original metric will be dropped by the
# ## aggregator and will not get sent to the output plugins. # ## aggregator and will not get sent to the output plugins.
# drop_original = false # drop_original = false
# #
# ## The example of config to aggregate histogram for all fields of specified metric. # ## Example config that aggregates all fields of the metric.
# [[aggregators.histogram.config]] # # [[aggregators.histogram.config]]
# ## The set of buckets. # # ## The set of buckets.
# buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0] # # buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0]
# ## The name of metric. # # ## The name of metric.
# metric_name = "cpu" # # measurement_name = "cpu"
# #
# ## The example of config to aggregate for specified fields of metric. # ## Example config that aggregates only specific fields of the metric.
# [[aggregators.histogram.config]] # # [[aggregators.histogram.config]]
# ## The set of buckets. # # ## The set of buckets.
# buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0] # # buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]
# ## The name of metric. # # ## The name of metric.
# metric_name = "diskio" # # measurement_name = "diskio"
# ## The concrete fields of metric # # ## The concrete fields of metric
# metric_fields = ["io_time", "read_time", "write_time"] # # fields = ["io_time", "read_time", "write_time"]
# # Keep the aggregate min/max of each metric passing through. # # Keep the aggregate min/max of each metric passing through.

121
plugins/aggregators/histogram/README.md
View File

@ -1,38 +1,25 @@
# Histogram Aggregator Plugin # Histogram Aggregator Plugin
#### Goal The histogram aggregator plugin creates histograms containing the counts of
field values within a range.
This plugin was added for ability to build histograms. Values added to a bucket are also added to the larger buckets in the
distribution. This creates a [cumulative histogram](https://en.wikipedia.org/wiki/Histogram#/media/File:Cumulative_vs_normal_histogram.svg).
#### Description Like other Telegraf aggregators, the metric is emitted every `period` seconds.
Bucket counts however are not reset between periods and will be non-strictly
increasing while Telegraf is running.
The histogram aggregator plugin aggregates values of specified metric's #### Design
fields. The metric is emitted every `period` seconds. All you need to do
is to specify borders of histogram buckets and fields, for which you want
to aggregate histogram.
#### How it works Each metric is passed to the aggregator and this aggregator searches
The each metric is passed to the aggregator and this aggregator searches
histogram buckets for those fields, which have been specified in the histogram buckets for those fields, which have been specified in the
config. If buckets are found, the aggregator will put +1 to appropriate config. If buckets are found, the aggregator will increment +1 to the appropriate
bucket. Otherwise, nothing will happen. Every `period` seconds these data bucket otherwise it will be added to the `+Inf` bucket. Every `period`
will be pushed to output. seconds this data will be forwarded to the outputs.
Note, that the all hits of current bucket will be also added to all next The algorithm of hit counting to buckets was implemented on the base
buckets in final result of distribution. Why does it work this way? In of the algorithm which is implemented in the Prometheus
configuration you define right borders for each bucket in a ascending
sequence. Internally buckets are presented as ranges with borders
(0..bucketBorder]: 0..1, 0..10, 0..50, …, 0..+Inf. So the value "+1" will be
put into those buckets, in which the metric value fell with such ranges of
buckets.
This plugin creates cumulative histograms. It means, that the hits in the
buckets will always increase from the moment of telegraf start. But if you
restart telegraf, all hits in the buckets will be reset to 0.
Also, the algorithm of hit counting to buckets was implemented on the base
of the algorithm, which is implemented in the Prometheus
[client](https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go). [client](https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go).
### Configuration ### Configuration
@ -40,61 +27,44 @@ of the algorithm, which is implemented in the Prometheus
```toml ```toml
# Configuration for aggregate histogram metrics # Configuration for aggregate histogram metrics
[[aggregators.histogram]] [[aggregators.histogram]]
## General Aggregator Arguments: ## The period in which to flush the aggregator.
## The period on which to flush & clear the aggregator.
period = "30s" period = "30s"
## If true, the original metric will be dropped by the ## If true, the original metric will be dropped by the
## aggregator and will not get sent to the output plugins. ## aggregator and will not get sent to the output plugins.
drop_original = false drop_original = false
## The example of config to aggregate histogram for all fields of specified metric. ## Example config that aggregates all fields of the metric.
[[aggregators.histogram.config]] # [[aggregators.histogram.config]]
## The set of buckets. # ## The set of buckets.
buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0] # buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0]
## The name of metric. # ## The name of metric.
metric_name = "cpu" # measurement_name = "cpu"
## The example of config to aggregate histogram for concrete fields of specified metric. ## Example config that aggregates only specific fields of the metric.
[[aggregators.histogram.config]] # [[aggregators.histogram.config]]
## The set of buckets. # ## The set of buckets.
buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0] # buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]
## The name of metric. # ## The name of metric.
metric_name = "diskio" # measurement_name = "diskio"
## The concrete fields of metric. # ## The concrete fields of metric
metric_fields = ["io_time", "read_time", "write_time"] # fields = ["io_time", "read_time", "write_time"]
``` ```
#### Explanation The user is responsible for defining the bounds of the histogram bucket as
well as the measurement name and fields to aggregate.
The field `metric_fields` is the list of metric fields. For example, the Each histogram config section must contain a `buckets` and `measurement_name`
metric `cpu` has the following fields: usage_user, usage_system, option. Optionally, if `fields` is set only the fields listed will be
usage_idle, usage_nice, usage_iowait, usage_irq, usage_softirq, usage_steal, aggregated. If `fields` is not set all fields are aggregated.
usage_guest, usage_guest_nice.
Note that histogram metrics will be pushed every `period` seconds. The `buckets` option contains a list of floats which specify the bucket
As you know telegraf calls aggregator `Reset()` func each `period` seconds. boundaries. Each float value defines the inclusive upper bound of the bucket.
Histogram aggregator ignores `Reset()` and continues to count hits. The `+Inf` bucket is added automatically and does not need to be defined.
#### Use cases
You can specify fields using two cases:
1. The specifying only metric name. In this case all fields of metric
will be aggregated.
2. The specifying metric name and concrete field.
#### Some rules
- The setting of each histogram must be in separate section with title
`aggregators.histogram.config`.
- The each value of bucket must be float value.
- Don\`t include the border bucket `+Inf`. It will be done automatically.
### Measurements & Fields: ### Measurements & Fields:
The postfix `bucket` will be added to each field. The postfix `bucket` will be added to each field key.
- measurement1 - measurement1
- field1_bucket - field1_bucket
@ -102,16 +72,15 @@ The postfix `bucket` will be added to each field.
### Tags: ### Tags:
All measurements have tag `le`. This tag has the border value of bucket. It All measurements are given the tag `le`. This tag has the border value of
means that the metric value is less or equal to the value of this tag. For bucket. It means that the metric value is less than or equal to the value of
example, let assume that we have the metric value 10 and the following this tag. For example, let assume that we have the metric value 10 and the
buckets: [5, 10, 30, 70, 100]. Then the tag `le` will have the value 10, following buckets: [5, 10, 30, 70, 100]. Then the tag `le` will have the value
because the metrics value is passed into bucket with right border value `10`. 10, because the metrics value is passed into bucket with right border value
`10`.
### Example Output: ### Example Output:
The following output will return to the Prometheus client.
``` ```
cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=0i 1486998330000000000 cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=0i 1486998330000000000
cpu,cpu=cpu1,host=localhost,le=10.0 usage_idle_bucket=0i 1486998330000000000 cpu,cpu=cpu1,host=localhost,le=10.0 usage_idle_bucket=0i 1486998330000000000

38
plugins/aggregators/histogram/histogram.go
View File

@ -24,8 +24,8 @@ type HistogramAggregator struct {
// config is the config, which contains name, field of metric and histogram buckets. // config is the config, which contains name, field of metric and histogram buckets.
type config struct { type config struct {
Metric string `toml:"metric_name"` Metric string `toml:"measurement_name"`
Fields []string `toml:"metric_fields"` Fields []string `toml:"fields"`
Buckets buckets `toml:"buckets"` Buckets buckets `toml:"buckets"`
} }
@ -65,28 +65,28 @@ func NewHistogramAggregator() telegraf.Aggregator {
} }
var sampleConfig = ` var sampleConfig = `
## General Aggregator Arguments: ## The period in which to flush the aggregator.
## The period on which to flush & clear the aggregator.
period = "30s" period = "30s"
## If true, the original metric will be dropped by the ## If true, the original metric will be dropped by the
## aggregator and will not get sent to the output plugins. ## aggregator and will not get sent to the output plugins.
drop_original = false drop_original = false
## The example of config to aggregate histogram for all fields of specified metric. ## Example config that aggregates all fields of the metric.
[[aggregators.histogram.config]] # [[aggregators.histogram.config]]
## The set of buckets. # ## The set of buckets.
buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0] # buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0]
## The name of metric. # ## The name of metric.
metric_name = "cpu" # measurement_name = "cpu"
## The example of config to aggregate for specified fields of metric. ## Example config that aggregates only specific fields of the metric.
[[aggregators.histogram.config]] # [[aggregators.histogram.config]]
## The set of buckets. # ## The set of buckets.
buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0] # buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]
## The name of metric. # ## The name of metric.
metric_name = "diskio" # measurement_name = "diskio"
## The concrete fields of metric # ## The concrete fields of metric
metric_fields = ["io_time", "read_time", "write_time"] # fields = ["io_time", "read_time", "write_time"]
` `
// SampleConfig returns sample of config // SampleConfig returns sample of config
@ -96,7 +96,7 @@ func (h *HistogramAggregator) SampleConfig() string {
// Description returns description of aggregator plugin // Description returns description of aggregator plugin
func (h *HistogramAggregator) Description() string { func (h *HistogramAggregator) Description() string {
return "Keep the aggregate histogram of each metric passing through." return "Create aggregate histograms."
} }
// Add adds new hit to the buckets // Add adds new hit to the buckets