Compare commits

..

1 Commits

Author SHA1 Message Date
Cameron Sparr
359d617b8f Release 0.13.2 2016-08-02 18:26:47 +01:00
125 changed files with 1452 additions and 6817 deletions

4
.gitattributes vendored
View File

@@ -1,4 +1,2 @@
CHANGELOG.md merge=union
README.md merge=union
plugins/inputs/all/all.go merge=union
plugins/outputs/all/all.go merge=union

View File

@@ -11,8 +11,6 @@ Erase the other section and everything on and above this line.
## Bug report
### Relevant telegraf.conf:
### System info:
[Include Telegraf version, operating system name, and other relevant details]

View File

@@ -1,79 +1,4 @@
## v1.0
### Features
### Bugfixes
## v1.0 beta 2 [2016-06-21]
### Features
- [#1340](https://github.com/influxdata/telegraf/issues/1340): statsd: do not log every dropped metric.
- [#1368](https://github.com/influxdata/telegraf/pull/1368): Add precision rounding to all metrics on collection.
- [#1390](https://github.com/influxdata/telegraf/pull/1390): Add support for Tengine
- [#1320](https://github.com/influxdata/telegraf/pull/1320): Logparser input plugin for parsing grok-style log patterns.
### Bugfixes
- [#1330](https://github.com/influxdata/telegraf/issues/1330): Fix exec plugin panic when using single binary.
- [#1336](https://github.com/influxdata/telegraf/issues/1336): Fixed incorrect prometheus metrics source selection.
- [#1112](https://github.com/influxdata/telegraf/issues/1112): Set default Zookeeper chroot to empty string.
- [#1335](https://github.com/influxdata/telegraf/issues/1335): Fix overall ping timeout to be calculated based on per-ping timeout.
- [#1374](https://github.com/influxdata/telegraf/pull/1374): Change "default" retention policy to "".
- [#1377](https://github.com/influxdata/telegraf/issues/1377): Graphite output mangling '%' character.
## v1.0 beta 1 [2016-06-07]
### Release Notes
- `flush_jitter` behavior has been changed. The random jitter will now be
evaluated at every flush interval, rather than once at startup. This makes it
consistent with the behavior of `collection_jitter`.
- All AWS plugins now utilize a standard mechanism for evaluating credentials.
This allows all AWS plugins to support environment variables, shared credential
files & profiles, and role assumptions. See the specific plugin README for
details.
- The AWS CloudWatch input plugin can now declare a wildcard value for a metric
dimension. This causes the plugin to read all metrics that contain the specified
dimension key regardless of value. This is used to export collections of metrics
without having to know the dimension values ahead of time.
- The AWS CloudWatch input plugin can now be configured with the `cache_ttl`
attribute. This configures the TTL of the internal metric cache. This is useful
in conjunction with wildcard dimension values as it will control the amount of
time before a new metric is included by the plugin.
### Features
- [#1262](https://github.com/influxdata/telegraf/pull/1261): Add graylog input pluging.
- [#1294](https://github.com/influxdata/telegraf/pull/1294): consul input plugin. Thanks @harnash
- [#1164](https://github.com/influxdata/telegraf/pull/1164): conntrack input plugin. Thanks @robinpercy!
- [#1165](https://github.com/influxdata/telegraf/pull/1165): vmstat input plugin. Thanks @jshim-xm!
- [#1247](https://github.com/influxdata/telegraf/pull/1247): rollbar input plugin. Thanks @francois2metz and @cduez!
- [#1208](https://github.com/influxdata/telegraf/pull/1208): Standardized AWS credentials evaluation & wildcard CloudWatch dimensions. Thanks @johnrengelman!
- [#1264](https://github.com/influxdata/telegraf/pull/1264): Add SSL config options to http_response plugin.
- [#1272](https://github.com/influxdata/telegraf/pull/1272): graphite parser: add ability to specify multiple tag keys, for consistency with influxdb parser.
- [#1265](https://github.com/influxdata/telegraf/pull/1265): Make dns lookups for chrony configurable. Thanks @zbindenren!
- [#1275](https://github.com/influxdata/telegraf/pull/1275): Allow wildcard filtering of varnish stats.
- [#1142](https://github.com/influxdata/telegraf/pull/1142): Support for glob patterns in exec plugin commands configuration.
- [#1278](https://github.com/influxdata/telegraf/pull/1278): RabbitMQ input: made url parameter optional by using DefaultURL (http://localhost:15672) if not specified
- [#1197](https://github.com/influxdata/telegraf/pull/1197): Limit AWS GetMetricStatistics requests to 10 per second.
- [#1278](https://github.com/influxdata/telegraf/pull/1278) & [#1288](https://github.com/influxdata/telegraf/pull/1288) & [#1295](https://github.com/influxdata/telegraf/pull/1295): RabbitMQ/Apache/InfluxDB inputs: made url(s) parameter optional by using reasonable input defaults if not specified
- [#1296](https://github.com/influxdata/telegraf/issues/1296): Refactor of flush_jitter argument.
- [#1213](https://github.com/influxdata/telegraf/issues/1213): Add inactive & active memory to mem plugin.
### Bugfixes
- [#1252](https://github.com/influxdata/telegraf/pull/1252) & [#1279](https://github.com/influxdata/telegraf/pull/1279): Fix systemd service. Thanks @zbindenren & @PierreF!
- [#1221](https://github.com/influxdata/telegraf/pull/1221): Fix influxdb n_shards counter.
- [#1258](https://github.com/influxdata/telegraf/pull/1258): Fix potential kernel plugin integer parse error.
- [#1268](https://github.com/influxdata/telegraf/pull/1268): Fix potential influxdb input type assertion panic.
- [#1283](https://github.com/influxdata/telegraf/pull/1283): Still send processes metrics if a process exited during metric collection.
- [#1297](https://github.com/influxdata/telegraf/issues/1297): disk plugin panic when usage grab fails.
- [#1316](https://github.com/influxdata/telegraf/pull/1316): Removed leaked "database" tag on redis metrics. Thanks @PierreF!
- [#1323](https://github.com/influxdata/telegraf/issues/1323): Processes plugin: fix potential error with /proc/net/stat directory.
- [#1322](https://github.com/influxdata/telegraf/issues/1322): Fix rare RHEL 5.2 panic in gopsutil diskio gathering function.
## v1.0 [unreleased]
## v0.13.1 [2016-05-24]

View File

@@ -212,8 +212,8 @@ func (s *Simple) Close() error {
}
func (s *Simple) Write(metrics []telegraf.Metric) error {
for _, metric := range metrics {
// write `metric` to the output sink here
for _, pt := range points {
// write `pt` to the output sink here
}
return nil
}

6
Godeps
View File

@@ -16,14 +16,13 @@ github.com/eapache/go-resiliency b86b1ec0dd4209a588dc1285cdd471e73525c0b3
github.com/eapache/queue ded5959c0d4e360646dc9e9908cff48666781367
github.com/eclipse/paho.mqtt.golang 0f7a459f04f13a41b7ed752d47944528d4bf9a86
github.com/go-sql-driver/mysql 1fca743146605a172a266e1654e01e5cd5669bee
github.com/gobwas/glob 49571a1557cd20e6a2410adc6421f85b66c730b5
github.com/gobwas/glob d877f6352135181470c40c73ebb81aefa22115fa
github.com/golang/protobuf 552c7b9542c194800fd493123b3798ef0a832032
github.com/golang/snappy 427fb6fc07997f43afa32f35e850833760e489a7
github.com/gonuts/go-shellquote e842a11b24c6abfb3dd27af69a17f482e4b483c2
github.com/gorilla/context 1ea25387ff6f684839d82767c1733ff4d4d15d0a
github.com/gorilla/mux c9e326e2bdec29039a3761c07bece13133863e1e
github.com/hailocab/go-hostpool e80d13ce29ede4452c43dea11e79b9bc8a15b478
github.com/hashicorp/consul 5aa90455ce78d4d41578bafc86305e6e6b28d7d2
github.com/hpcloud/tail b2940955ab8b26e19d43a43c4da0475dd81bdb56
github.com/influxdata/config b79f6829346b8d6e78ba73544b1e1038f1f1c9da
github.com/influxdata/influxdb e094138084855d444195b252314dfee9eae34cab
@@ -43,11 +42,10 @@ github.com/prometheus/client_model fa8ad6fec33561be4280a8f0514318c79d7f6cb6
github.com/prometheus/common e8eabff8812b05acf522b45fdcd725a785188e37
github.com/prometheus/procfs 406e5b7bfd8201a36e2bb5f7bdae0b03380c2ce8
github.com/samuel/go-zookeeper 218e9c81c0dd8b3b18172b2bbfad92cc7d6db55f
github.com/shirou/gopsutil 586bb697f3ec9f8ec08ffefe18f521a64534037c
github.com/shirou/gopsutil 83c6e72cbdef6e8ada934549abf700ff0ba96776
github.com/soniah/gosnmp b1b4f885b12c5dcbd021c5cee1c904110de6db7d
github.com/streadway/amqp b4f3ceab0337f013208d31348b578d83c0064744
github.com/stretchr/testify 1f4a1643a57e798696635ea4c126e9127adb7d3c
github.com/vjeantet/grok 83bfdfdfd1a8146795b28e547a8e3c8b28a466c2
github.com/wvanbergen/kafka 46f9a1cf3f670edec492029fadded9c2d9e18866
github.com/wvanbergen/kazoo-go 0f768712ae6f76454f987c3356177e138df258f8
github.com/zensqlmonitor/go-mssqldb ffe5510c6fa5e15e6d983210ab501c815b56b363

View File

@@ -64,6 +64,7 @@ endif
docker run --name memcached -p "11211:11211" -d memcached
docker run --name postgres -p "5432:5432" -d postgres
docker run --name rabbitmq -p "15672:15672" -p "5672:5672" -d rabbitmq:3-management
docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
docker run --name redis -p "6379:6379" -d redis
docker run --name aerospike -p "3000:3000" -d aerospike
docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
@@ -78,6 +79,7 @@ docker-run-circle:
-e ADVERTISED_PORT=9092 \
-p "2181:2181" -p "9092:9092" \
-d spotify/kafka
docker run --name opentsdb -p "4242:4242" -d petergrace/opentsdb-docker
docker run --name aerospike -p "3000:3000" -d aerospike
docker run --name nsq -p "4150:4150" -d nsqio/nsq /nsqd
docker run --name mqtt -p "1883:1883" -d ncarlier/mqtt
@@ -86,8 +88,8 @@ docker-run-circle:
# Kill all docker containers, ignore errors
docker-kill:
-docker kill nsq aerospike redis rabbitmq postgres memcached mysql kafka mqtt riemann snmp
-docker rm nsq aerospike redis rabbitmq postgres memcached mysql kafka mqtt riemann snmp
-docker kill nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann snmp
-docker rm nsq aerospike redis opentsdb rabbitmq postgres memcached mysql kafka mqtt riemann snmp
# Run full unit tests using docker containers (includes setup and teardown)
test: vet docker-kill docker-run

View File

@@ -20,12 +20,12 @@ new plugins.
### Linux deb and rpm Packages:
Latest:
* https://dl.influxdata.com/telegraf/releases/telegraf_1.0.0-beta2_amd64.deb
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0_beta2.x86_64.rpm
* https://dl.influxdata.com/telegraf/releases/telegraf_0.13.2_amd64.deb
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2.x86_64.rpm
Latest (arm):
* https://dl.influxdata.com/telegraf/releases/telegraf_1.0.0-beta2_armhf.deb
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0_beta2.armhf.rpm
* https://dl.influxdata.com/telegraf/releases/telegraf_0.13.2_armhf.deb
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2.armhf.rpm
##### Package Instructions:
@@ -46,14 +46,14 @@ to use this repo to install & update telegraf.
### Linux tarballs:
Latest:
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta2_linux_amd64.tar.gz
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta2_linux_i386.tar.gz
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta2_linux_armhf.tar.gz
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2_linux_amd64.tar.gz
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2_linux_i386.tar.gz
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2_linux_armhf.tar.gz
### FreeBSD tarball:
Latest:
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta2_freebsd_amd64.tar.gz
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2_freebsd_amd64.tar.gz
### Ansible Role:
@@ -69,7 +69,8 @@ brew install telegraf
### Windows Binaries (EXPERIMENTAL)
Latest:
* https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta2_windows_amd64.zip
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2_windows_amd64.zip
* https://dl.influxdata.com/telegraf/releases/telegraf-0.13.2_windows_i386.zip
### From Source:
@@ -144,8 +145,6 @@ Currently implemented sources:
* [cassandra](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cassandra)
* [ceph](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph)
* [chrony](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/chrony)
* [consul](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/consul)
* [conntrack](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/conntrack)
* [couchbase](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/couchbase)
* [couchdb](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/couchdb)
* [disque](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/disque)
@@ -206,7 +205,6 @@ Currently implemented sources:
* swap
* processes
* kernel (/proc/stat)
* kernel (/proc/vmstat)
Telegraf can also collect metrics via the following service plugins:
@@ -218,7 +216,6 @@ Telegraf can also collect metrics via the following service plugins:
* [kafka_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/kafka_consumer)
* [nats_consumer](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nats_consumer)
* [github_webhooks](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/github_webhooks)
* [rollbar_webhooks](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/rollbar_webhooks)
We'll be adding support for many more over the coming months. Read on if you
want to add support for another service or third-party API.
@@ -233,7 +230,6 @@ want to add support for another service or third-party API.
* [datadog](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/datadog)
* [file](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/file)
* [graphite](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/graphite)
* [graylog](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/graylog)
* [instrumental](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/instrumental)
* [kafka](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kafka)
* [librato](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/librato)

View File

@@ -18,8 +18,4 @@ type Accumulator interface {
Debug() bool
SetDebug(enabled bool)
SetPrecision(precision, interval time.Duration)
DisablePrecision()
}

View File

@@ -17,7 +17,6 @@ func NewAccumulator(
acc := accumulator{}
acc.metrics = metrics
acc.inputConfig = inputConfig
acc.precision = time.Nanosecond
return &acc
}
@@ -33,8 +32,6 @@ type accumulator struct {
inputConfig *internal_models.InputConfig
prefix string
precision time.Duration
}
func (ac *accumulator) Add(
@@ -144,7 +141,6 @@ func (ac *accumulator) AddFields(
} else {
timestamp = time.Now()
}
timestamp = timestamp.Round(ac.precision)
if ac.prefix != "" {
measurement = ac.prefix + measurement
@@ -177,31 +173,6 @@ func (ac *accumulator) SetTrace(trace bool) {
ac.trace = trace
}
// SetPrecision takes two time.Duration objects. If the first is non-zero,
// it sets that as the precision. Otherwise, it takes the second argument
// as the order of time that the metrics should be rounded to, with the
// maximum being 1s.
func (ac *accumulator) SetPrecision(precision, interval time.Duration) {
if precision > 0 {
ac.precision = precision
return
}
switch {
case interval >= time.Second:
ac.precision = time.Second
case interval >= time.Millisecond:
ac.precision = time.Millisecond
case interval >= time.Microsecond:
ac.precision = time.Microsecond
default:
ac.precision = time.Nanosecond
}
}
func (ac *accumulator) DisablePrecision() {
ac.precision = time.Nanosecond
}
func (ac *accumulator) setDefaultTags(tags map[string]string) {
ac.defaultTags = tags
}

View File

@@ -38,128 +38,6 @@ func TestAdd(t *testing.T) {
actual)
}
func TestAddNoPrecisionWithInterval(t *testing.T) {
a := accumulator{}
now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
a.metrics = make(chan telegraf.Metric, 10)
defer close(a.metrics)
a.inputConfig = &internal_models.InputConfig{}
a.SetPrecision(0, time.Second)
a.Add("acctest", float64(101), map[string]string{})
a.Add("acctest", float64(101), map[string]string{"acc": "test"})
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm := <-a.metrics
actual := testm.String()
assert.Contains(t, actual, "acctest value=101")
testm = <-a.metrics
actual = testm.String()
assert.Contains(t, actual, "acctest,acc=test value=101")
testm = <-a.metrics
actual = testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800000000000)),
actual)
}
func TestAddNoIntervalWithPrecision(t *testing.T) {
a := accumulator{}
now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
a.metrics = make(chan telegraf.Metric, 10)
defer close(a.metrics)
a.inputConfig = &internal_models.InputConfig{}
a.SetPrecision(time.Second, time.Millisecond)
a.Add("acctest", float64(101), map[string]string{})
a.Add("acctest", float64(101), map[string]string{"acc": "test"})
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm := <-a.metrics
actual := testm.String()
assert.Contains(t, actual, "acctest value=101")
testm = <-a.metrics
actual = testm.String()
assert.Contains(t, actual, "acctest,acc=test value=101")
testm = <-a.metrics
actual = testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800000000000)),
actual)
}
func TestAddDisablePrecision(t *testing.T) {
a := accumulator{}
now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
a.metrics = make(chan telegraf.Metric, 10)
defer close(a.metrics)
a.inputConfig = &internal_models.InputConfig{}
a.SetPrecision(time.Second, time.Millisecond)
a.DisablePrecision()
a.Add("acctest", float64(101), map[string]string{})
a.Add("acctest", float64(101), map[string]string{"acc": "test"})
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm := <-a.metrics
actual := testm.String()
assert.Contains(t, actual, "acctest value=101")
testm = <-a.metrics
actual = testm.String()
assert.Contains(t, actual, "acctest,acc=test value=101")
testm = <-a.metrics
actual = testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800082912748)),
actual)
}
func TestDifferentPrecisions(t *testing.T) {
a := accumulator{}
now := time.Date(2006, time.February, 10, 12, 0, 0, 82912748, time.UTC)
a.metrics = make(chan telegraf.Metric, 10)
defer close(a.metrics)
a.inputConfig = &internal_models.InputConfig{}
a.SetPrecision(0, time.Second)
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm := <-a.metrics
actual := testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800000000000)),
actual)
a.SetPrecision(0, time.Millisecond)
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm = <-a.metrics
actual = testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800083000000)),
actual)
a.SetPrecision(0, time.Microsecond)
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm = <-a.metrics
actual = testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800082913000)),
actual)
a.SetPrecision(0, time.Nanosecond)
a.Add("acctest", float64(101), map[string]string{"acc": "test"}, now)
testm = <-a.metrics
actual = testm.String()
assert.Equal(t,
fmt.Sprintf("acctest,acc=test value=101 %d", int64(1139572800082912748)),
actual)
}
func TestAddDefaultTags(t *testing.T) {
a := accumulator{}
a.addDefaultTag("default", "tag")

View File

@@ -1,15 +1,17 @@
package agent
import (
cryptorand "crypto/rand"
"fmt"
"log"
"math/big"
"math/rand"
"os"
"runtime"
"sync"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/internal/config"
"github.com/influxdata/telegraf/internal/models"
)
@@ -113,18 +115,27 @@ func (a *Agent) gatherer(
ticker := time.NewTicker(interval)
defer ticker.Stop()
jitter := a.Config.Agent.CollectionJitter.Duration.Nanoseconds()
for {
var outerr error
start := time.Now()
acc := NewAccumulator(input.Config, metricC)
acc.SetDebug(a.Config.Agent.Debug)
acc.SetPrecision(a.Config.Agent.Precision.Duration,
a.Config.Agent.Interval.Duration)
acc.setDefaultTags(a.Config.Tags)
internal.RandomSleep(a.Config.Agent.CollectionJitter.Duration, shutdown)
if jitter != 0 {
nanoSleep := rand.Int63n(jitter)
d, err := time.ParseDuration(fmt.Sprintf("%dns", nanoSleep))
if err != nil {
log.Printf("Jittering collection interval failed for plugin %s",
input.Name)
} else {
time.Sleep(d)
}
}
start := time.Now()
gatherWithTimeout(shutdown, input, acc, interval)
elapsed := time.Since(start)
@@ -203,8 +214,6 @@ func (a *Agent) Test() error {
for _, input := range a.Config.Inputs {
acc := NewAccumulator(input.Config, metricC)
acc.SetTrace(true)
acc.SetPrecision(a.Config.Agent.Precision.Duration,
a.Config.Agent.Interval.Duration)
acc.setDefaultTags(a.Config.Tags)
fmt.Printf("* Plugin: %s, Collection 1\n", input.Name)
@@ -265,7 +274,6 @@ func (a *Agent) flusher(shutdown chan struct{}, metricC chan telegraf.Metric) er
a.flush()
return nil
case <-ticker.C:
internal.RandomSleep(a.Config.Agent.FlushJitter.Duration, shutdown)
a.flush()
case m := <-metricC:
for _, o := range a.Config.Outputs {
@@ -275,10 +283,35 @@ func (a *Agent) flusher(shutdown chan struct{}, metricC chan telegraf.Metric) er
}
}
// jitterInterval applies the the interval jitter to the flush interval using
// crypto/rand number generator
func jitterInterval(ininterval, injitter time.Duration) time.Duration {
var jitter int64
outinterval := ininterval
if injitter.Nanoseconds() != 0 {
maxjitter := big.NewInt(injitter.Nanoseconds())
if j, err := cryptorand.Int(cryptorand.Reader, maxjitter); err == nil {
jitter = j.Int64()
}
outinterval = time.Duration(jitter + ininterval.Nanoseconds())
}
if outinterval.Nanoseconds() < time.Duration(500*time.Millisecond).Nanoseconds() {
log.Printf("Flush interval %s too low, setting to 500ms\n", outinterval)
outinterval = time.Duration(500 * time.Millisecond)
}
return outinterval
}
// Run runs the agent daemon, gathering every Interval
func (a *Agent) Run(shutdown chan struct{}) error {
var wg sync.WaitGroup
a.Config.Agent.FlushInterval.Duration = jitterInterval(
a.Config.Agent.FlushInterval.Duration,
a.Config.Agent.FlushJitter.Duration)
log.Printf("Agent Config: Interval:%s, Debug:%#v, Quiet:%#v, Hostname:%#v, "+
"Flush Interval:%s \n",
a.Config.Agent.Interval.Duration, a.Config.Agent.Debug, a.Config.Agent.Quiet,
@@ -293,9 +326,6 @@ func (a *Agent) Run(shutdown chan struct{}) error {
case telegraf.ServiceInput:
acc := NewAccumulator(input.Config, metricC)
acc.SetDebug(a.Config.Agent.Debug)
// Service input plugins should set their own precision of their
// metrics.
acc.DisablePrecision()
acc.setDefaultTags(a.Config.Tags)
if err := p.Start(acc); err != nil {
log.Printf("Service for input %s failed to start, exiting\n%s\n",

View File

@@ -2,6 +2,7 @@ package agent
import (
"testing"
"time"
"github.com/influxdata/telegraf/internal/config"
@@ -109,3 +110,75 @@ func TestAgent_LoadOutput(t *testing.T) {
a, _ = NewAgent(c)
assert.Equal(t, 3, len(a.Config.Outputs))
}
func TestAgent_ZeroJitter(t *testing.T) {
flushinterval := jitterInterval(time.Duration(10*time.Second),
time.Duration(0*time.Second))
actual := flushinterval.Nanoseconds()
exp := time.Duration(10 * time.Second).Nanoseconds()
if actual != exp {
t.Errorf("Actual %v, expected %v", actual, exp)
}
}
func TestAgent_ZeroInterval(t *testing.T) {
min := time.Duration(500 * time.Millisecond).Nanoseconds()
max := time.Duration(5 * time.Second).Nanoseconds()
for i := 0; i < 1000; i++ {
flushinterval := jitterInterval(time.Duration(0*time.Second),
time.Duration(5*time.Second))
actual := flushinterval.Nanoseconds()
if actual > max {
t.Errorf("Didn't expect interval %d to be > %d", actual, max)
break
}
if actual < min {
t.Errorf("Didn't expect interval %d to be < %d", actual, min)
break
}
}
}
func TestAgent_ZeroBoth(t *testing.T) {
flushinterval := jitterInterval(time.Duration(0*time.Second),
time.Duration(0*time.Second))
actual := flushinterval
exp := time.Duration(500 * time.Millisecond)
if actual != exp {
t.Errorf("Actual %v, expected %v", actual, exp)
}
}
func TestAgent_JitterMax(t *testing.T) {
max := time.Duration(32 * time.Second).Nanoseconds()
for i := 0; i < 1000; i++ {
flushinterval := jitterInterval(time.Duration(30*time.Second),
time.Duration(2*time.Second))
actual := flushinterval.Nanoseconds()
if actual > max {
t.Errorf("Didn't expect interval %d to be > %d", actual, max)
break
}
}
}
func TestAgent_JitterMin(t *testing.T) {
min := time.Duration(30 * time.Second).Nanoseconds()
for i := 0; i < 1000; i++ {
flushinterval := jitterInterval(time.Duration(30*time.Second),
time.Duration(2*time.Second))
actual := flushinterval.Nanoseconds()
if actual < min {
t.Errorf("Didn't expect interval %d to be < %d", actual, min)
break
}
}
}

View File

@@ -186,59 +186,49 @@ name of the plugin.
# Graphite:
The Graphite data format translates graphite _dot_ buckets directly into
telegraf measurement names, with a single value field, and without any tags.
By default, the separator is left as ".", but this can be changed using the
"separator" argument. For more advanced options,
Telegraf supports specifying "templates" to translate
telegraf measurement names, with a single value field, and without any tags. For
more advanced options, Telegraf supports specifying "templates" to translate
graphite buckets into Telegraf metrics.
Templates are of the form:
#### Separator:
You can specify a separator to use for the parsed metrics.
By default, it will leave the metrics with a "." separator.
Setting `separator = "_"` will translate:
```
"host.mytag.mytag.measurement.measurement.field*"
cpu.usage.idle 99
=> cpu_usage_idle value=99
```
Where the following keywords exist:
1. `measurement`: specifies that this section of the graphite bucket corresponds
to the measurement name. This can be specified multiple times.
2. `field`: specifies that this section of the graphite bucket corresponds
to the field name. This can be specified multiple times.
3. `measurement*`: specifies that all remaining elements of the graphite bucket
correspond to the measurement name.
4. `field*`: specifies that all remaining elements of the graphite bucket
correspond to the field name.
Any part of the template that is not a keyword is treated as a tag key. This
can also be specified multiple times.
NOTE: `field*` cannot be used in conjunction with `measurement*`!
#### Measurement & Tag Templates:
#### Measurement/Tag Templates:
The most basic template is to specify a single transformation to apply to all
incoming metrics. So the following template:
incoming metrics. _measurement_ is a special keyword that tells Telegraf which
parts of the graphite bucket to combine into the measurement name. It can have a
trailing `*` to indicate that the remainder of the metric should be used.
Other words are considered tag keys. So the following template:
```toml
templates = [
"region.region.measurement*"
"region.measurement*"
]
```
would result in the following Graphite -> Telegraf transformation.
```
us.west.cpu.load 100
=> cpu.load,region=us.west value=100
us-west.cpu.load 100
=> cpu.load,region=us-west value=100
```
#### Field Templates:
There is also a _field_ keyword, which can only be specified once.
The field keyword tells Telegraf to give the metric that field name.
So the following template:
```toml
separator = "_"
templates = [
"measurement.measurement.field.field.region"
]
@@ -247,26 +237,24 @@ templates = [
would result in the following Graphite -> Telegraf transformation.
```
cpu.usage.idle.percent.eu-east 100
=> cpu_usage,region=eu-east idle_percent=100
cpu.usage.idle.percent.us-west 100
=> cpu_usage,region=us-west idle_percent=100
```
The field key can also be derived from all remaining elements of the graphite
bucket by specifying `field*`:
The field key can also be derived from the second "half" of the input metric-name by specifying ```field*```:
```toml
separator = "_"
templates = [
"measurement.measurement.region.field*"
]
```
which would result in the following Graphite -> Telegraf transformation.
would result in the following Graphite -> Telegraf transformation.
```
cpu.usage.eu-east.idle.percentage 100
=> cpu_usage,region=eu-east idle_percentage=100
cpu.usage.us-west.idle.percentage 100
=> cpu_usage,region=us-west idle_percentage=100
```
(This cannot be used in conjunction with "measurement*"!)
#### Filter Templates:
@@ -283,8 +271,8 @@ templates = [
which would result in the following transformation:
```
cpu.load.eu-east 100
=> cpu_load,region=eu-east value=100
cpu.load.us-west 100
=> cpu_load,region=us-west value=100
mem.cached.localhost 256
=> mem_cached,host=localhost value=256
@@ -306,8 +294,8 @@ templates = [
would result in the following Graphite -> Telegraf transformation.
```
cpu.usage.idle.eu-east 100
=> cpu_usage,region=eu-east,datacenter=1a idle=100
cpu.usage.idle.us-west 100
=> cpu_usage,region=us-west,datacenter=1a idle=100
```
There are many more options available,
@@ -338,12 +326,12 @@ There are many more options available,
## similar to the line protocol format. There can be only one default template.
## Templates support below format:
## 1. filter + template
## 2. filter + template + extra tag(s)
## 2. filter + template + extra tag
## 3. filter + template with field key
## 4. default template
templates = [
"*.app env.service.resource.measurement",
"stats.* .host.measurement* region=eu-east,agent=sensu",
"stats.* .host.measurement* region=us-west,agent=sensu",
"stats2.* .host.measurement.field",
"measurement*"
]

View File

@@ -52,11 +52,6 @@
## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
flush_jitter = "0s"
## By default, precision will be set to the same timestamp order as the
## collection interval, with the maximum being 1s.
## Precision will NOT be used for service inputs, such as logparser and statsd.
## Valid values are "Nns", "Nus" (or "Nµs"), "Nms", "Ns".
precision = ""
## Run telegraf in debug mode
debug = false
## Run telegraf in quiet mode
@@ -80,9 +75,12 @@
urls = ["http://localhost:8086"] # required
## The target database for metrics (telegraf will create it if not exists).
database = "telegraf" # required
## Precision of writes, valid values are "ns", "us" (or "µs"), "ms", "s", "m", "h".
## note: using "s" precision greatly improves InfluxDB compression.
precision = "s"
## Retention policy to write to. Empty string writes to the default rp.
retention_policy = ""
## Retention policy to write to.
retention_policy = "default"
## Write consistency (clusters only), can be: "any", "one", "quorom", "all"
write_consistency = "any"
@@ -108,10 +106,10 @@
# [[outputs.amon]]
# ## Amon Server Key
# server_key = "my-server-key" # required.
#
#
# ## Amon Instance URL
# amon_instance = "https://youramoninstance" # required
#
#
# ## Connection timeout.
# # timeout = "5s"
@@ -127,21 +125,21 @@
# ## Telegraf tag to use as a routing key
# ## ie, if this tag exists, it's value will be used as the routing key
# routing_tag = "host"
#
#
# ## InfluxDB retention policy
# # retention_policy = "default"
# ## InfluxDB database
# # database = "telegraf"
# ## InfluxDB precision
# # precision = "s"
#
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
# # ssl_key = "/etc/telegraf/key.pem"
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
#
#
# ## Data format to output.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -153,22 +151,16 @@
# [[outputs.cloudwatch]]
# ## Amazon REGION
# region = 'us-east-1'
#
#
# ## Amazon Credentials
# ## Credentials are loaded in the following order
# ## 1) Assumed credentials via STS if role_arn is specified
# ## 2) explicit credentials from 'access_key' and 'secret_key'
# ## 3) shared profile from 'profile'
# ## 4) environment variables
# ## 5) shared credentials file
# ## 6) EC2 Instance Profile
# ## 1) explicit credentials from 'access_key' and 'secret_key'
# ## 2) environment variables
# ## 3) shared credentials file
# ## 4) EC2 Instance Profile
# #access_key = ""
# #secret_key = ""
# #token = ""
# #role_arn = ""
# #profile = ""
# #shared_credential_file = ""
#
#
# ## Namespace for the CloudWatch MetricDatums
# namespace = 'InfluxData/Telegraf'
@@ -177,7 +169,7 @@
# [[outputs.datadog]]
# ## Datadog API key
# apikey = "my-secret-key" # required.
#
#
# ## Connection timeout.
# # timeout = "5s"
@@ -186,7 +178,7 @@
# [[outputs.file]]
# ## Files to write to, "stdout" is a specially handled file.
# files = ["stdout", "/tmp/metrics.out"]
#
#
# ## Data format to output.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -207,12 +199,6 @@
# timeout = 2
# # Send telegraf metrics to graylog(s)
# [[outputs.graylog]]
# ## Udp endpoint for your graylog instance.
# servers = ["127.0.0.1:12201", "192.168.1.1:12201"]
# # Configuration for sending metrics to an Instrumental project
# [[outputs.instrumental]]
# ## Project API Token (required)
@@ -237,14 +223,14 @@
# ## Telegraf tag to use as a routing key
# ## ie, if this tag exists, it's value will be used as the routing key
# routing_tag = "host"
#
#
# ## CompressionCodec represents the various compression codecs recognized by
# ## Kafka in messages.
# ## 0 : No compression
# ## 1 : Gzip compression
# ## 2 : Snappy compression
# compression_codec = 0
#
#
# ## RequiredAcks is used in Produce Requests to tell the broker how many
# ## replica acknowledgements it must see before responding
# ## 0 : the producer never waits for an acknowledgement from the broker.
@@ -260,17 +246,17 @@
# ## guarantee that no messages will be lost as long as at least one in
# ## sync replica remains.
# required_acks = -1
#
#
# ## The total number of times to retry sending a message
# max_retry = 3
#
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
# # ssl_key = "/etc/telegraf/key.pem"
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
#
#
# ## Data format to output.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -282,22 +268,16 @@
# [[outputs.kinesis]]
# ## Amazon REGION of kinesis endpoint.
# region = "ap-southeast-2"
#
#
# ## Amazon Credentials
# ## Credentials are loaded in the following order
# ## 1) Assumed credentials via STS if role_arn is specified
# ## 2) explicit credentials from 'access_key' and 'secret_key'
# ## 3) shared profile from 'profile'
# ## 4) environment variables
# ## 5) shared credentials file
# ## 6) EC2 Instance Profile
# ## 1) explicit credentials from 'access_key' and 'secret_key'
# ## 2) environment variables
# ## 3) shared credentials file
# ## 4) EC2 Instance Profile
# #access_key = ""
# #secret_key = ""
# #token = ""
# #role_arn = ""
# #profile = ""
# #shared_credential_file = ""
#
#
# ## Kinesis StreamName must exist prior to starting telegraf.
# streamname = "StreamName"
# ## PartitionKey as used for sharding data.
@@ -332,23 +312,23 @@
# # Configuration for MQTT server to send metrics to
# [[outputs.mqtt]]
# servers = ["localhost:1883"] # required.
#
#
# ## MQTT outputs send metrics to this topic format
# ## "<topic_prefix>/<hostname>/<pluginname>/"
# ## ex: prefix/web01.example.com/mem
# topic_prefix = "telegraf"
#
#
# ## username and password to connect MQTT server.
# # username = "telegraf"
# # password = "metricsmetricsmetricsmetrics"
#
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
# # ssl_key = "/etc/telegraf/key.pem"
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
#
#
# ## Data format to output.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -362,7 +342,7 @@
# server = "localhost:4150"
# ## NSQ topic for producer messages
# topic = "telegraf"
#
#
# ## Data format to output.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -374,14 +354,14 @@
# [[outputs.opentsdb]]
# ## prefix for metrics keys
# prefix = "my.specific.prefix."
#
#
# ## Telnet Mode ##
# ## DNS name of the OpenTSDB server in telnet mode
# host = "opentsdb.example.com"
#
#
# ## Port of the OpenTSDB server in telnet mode
# port = 4242
#
#
# ## Debug true - Prints OpenTSDB communication
# debug = false
@@ -474,7 +454,6 @@
# # Read Apache status information (mod_status)
# [[inputs.apache]]
# ## An array of Apache status URI to gather stats.
# ## Default is "http://localhost/server-status?auto".
# urls = ["http://localhost/server-status?auto"]
@@ -483,7 +462,7 @@
# ## Bcache sets path
# ## If not specified, then default is:
# bcachePath = "/sys/fs/bcache"
#
#
# ## By default, telegraf gather stats for all bcache devices
# ## Setting devices will restrict the stats to the specified
# ## bcache devices.
@@ -511,17 +490,17 @@
# # Collects performance metrics from the MON and OSD nodes in a Ceph storage cluster.
# [[inputs.ceph]]
# ## All configuration values are optional, defaults are shown below
#
#
# ## location of ceph binary
# ceph_binary = "/usr/bin/ceph"
#
#
# ## directory in which to look for socket files
# socket_dir = "/var/run/ceph"
#
#
# ## prefix of MON and OSD socket files, used to determine socket type
# mon_prefix = "ceph-mon"
# osd_prefix = "ceph-osd"
#
#
# ## suffix used to identify socket files
# socket_suffix = "asok"
@@ -530,39 +509,29 @@
# [[inputs.cloudwatch]]
# ## Amazon Region
# region = 'us-east-1'
#
#
# ## Amazon Credentials
# ## Credentials are loaded in the following order
# ## 1) Assumed credentials via STS if role_arn is specified
# ## 2) explicit credentials from 'access_key' and 'secret_key'
# ## 3) shared profile from 'profile'
# ## 4) environment variables
# ## 5) shared credentials file
# ## 6) EC2 Instance Profile
# ## 1) explicit credentials from 'access_key' and 'secret_key'
# ## 2) environment variables
# ## 3) shared credentials file
# ## 4) EC2 Instance Profile
# #access_key = ""
# #secret_key = ""
# #token = ""
# #role_arn = ""
# #profile = ""
# #shared_credential_file = ""
#
#
# ## Requested CloudWatch aggregation Period (required - must be a multiple of 60s)
# period = '1m'
#
#
# ## Collection Delay (required - must account for metrics availability via CloudWatch API)
# delay = '1m'
#
#
# ## Recomended: use metric 'interval' that is a multiple of 'period' to avoid
# ## gaps or overlap in pulled data
# interval = '1m'
#
# ## Configure the TTL for the internal cache of metrics.
# ## Defaults to 1 hr if not specified
# #cache_ttl = '10m'
#
#
# ## Metric Statistic Namespace (required)
# namespace = 'AWS/ELB'
#
#
# ## Metrics to Pull (optional)
# ## Defaults to all Metrics in Namespace if nothing is provided
# ## Refreshes Namespace available metrics every 1h
@@ -575,23 +544,6 @@
# # value = 'p-example'
# # Gather health check statuses from services registered in Consul
# [[inputs.consul]]
# ## Most of these values defaults to the one configured on a Consul's agent level.
# ## Optional Consul server address (default: "localhost")
# # address = "localhost"
# ## Optional URI scheme for the Consul server (default: "http")
# # scheme = "http"
# ## Optional ACL token used in every request (default: "")
# # token = ""
# ## Optional username used for request HTTP Basic Authentication (default: "")
# # username = ""
# ## Optional password used for HTTP Basic Authentication (default: "")
# # password = ""
# ## Optional data centre to query the health checks from (default: "")
# # datacentre = ""
# # Read metrics from one or many couchbase clusters
# [[inputs.couchbase]]
# ## specify servers via a url matching:
@@ -626,17 +578,17 @@
# [[inputs.dns_query]]
# ## servers to query
# servers = ["8.8.8.8"] # required
#
#
# ## Domains or subdomains to query. "."(root) is default
# domains = ["."] # optional
#
#
# ## Query record type. Default is "A"
# ## Posible values: A, AAAA, CNAME, MX, NS, PTR, TXT, SOA, SPF, SRV.
# record_type = "A" # optional
#
#
# ## Dns server port. 53 is default
# port = 53 # optional
#
#
# ## Query timeout in seconds. Default is 2 seconds
# timeout = 2 # optional
@@ -672,11 +624,11 @@
# [[inputs.elasticsearch]]
# ## specify a list of one or more Elasticsearch servers
# servers = ["http://localhost:9200"]
#
#
# ## set local to false when you want to read the indices stats from all nodes
# ## within the cluster
# local = true
#
#
# ## set cluster_health to true when you want to also obtain cluster level stats
# cluster_health = false
@@ -684,18 +636,14 @@
# # Read metrics from one or more commands that can output to stdout
# [[inputs.exec]]
# ## Commands array
# commands = [
# "/tmp/test.sh",
# "/usr/bin/mycollector --foo=bar",
# "/tmp/collect_*.sh"
# ]
#
# commands = ["/tmp/test.sh", "/usr/bin/mycollector --foo=bar"]
#
# ## Timeout for each command to complete.
# timeout = "5s"
#
#
# ## measurement name suffix (for separating different commands)
# name_suffix = "_mycollector"
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -719,48 +667,11 @@
# md5 = false
# # Read flattened metrics from one or more GrayLog HTTP endpoints
# [[inputs.graylog]]
# ## API endpoint, currently supported API:
# ##
# ## - multiple (Ex http://<host>:12900/system/metrics/multiple)
# ## - namespace (Ex http://<host>:12900/system/metrics/namespace/{namespace})
# ##
# ## For namespace endpoint, the metrics array will be ignored for that call.
# ## Endpoint can contain namespace and multiple type calls.
# ##
# ## Please check http://[graylog-server-ip]:12900/api-browser for full list
# ## of endpoints
# servers = [
# "http://[graylog-server-ip]:12900/system/metrics/multiple",
# ]
#
# ## Metrics list
# ## List of metrics can be found on Graylog webservice documentation.
# ## Or by hitting the the web service api at:
# ## http://[graylog-host]:12900/system/metrics
# metrics = [
# "jvm.cl.loaded",
# "jvm.memory.pools.Metaspace.committed"
# ]
#
# ## Username and password
# username = ""
# password = ""
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
# # ssl_key = "/etc/telegraf/key.pem"
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
# # Read metrics of haproxy, via socket or csv stats page
# [[inputs.haproxy]]
# ## An array of address to gather stats about. Specify an ip on hostname
# ## with optional port. ie localhost, 10.10.3.33:1936, etc.
#
#
# ## If no servers are specified, then default to 127.0.0.1:1936
# servers = ["http://myhaproxy.com:1936", "http://anotherhaproxy.com:1936"]
# ## Or you can also use local socket
@@ -784,48 +695,41 @@
# # body = '''
# # {'fake':'data'}
# # '''
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
# # ssl_key = "/etc/telegraf/key.pem"
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
# # Read flattened metrics from one or more JSON HTTP endpoints
# [[inputs.httpjson]]
# ## NOTE This plugin only reads numerical measurements, strings and booleans
# ## will be ignored.
#
#
# ## a name for the service being polled
# name = "webserver_stats"
#
#
# ## URL of each server in the service's cluster
# servers = [
# "http://localhost:9999/stats/",
# "http://localhost:9998/stats/",
# ]
#
#
# ## HTTP method to use: GET or POST (case-sensitive)
# method = "GET"
#
#
# ## List of tag names to extract from top-level of JSON server response
# # tag_keys = [
# # "my_tag_1",
# # "my_tag_2"
# # ]
#
#
# ## HTTP parameters (all values must be strings)
# [inputs.httpjson.parameters]
# event_type = "cpu_spike"
# threshold = "0.75"
#
#
# ## HTTP Header parameters (all values must be strings)
# # [inputs.httpjson.headers]
# # X-Auth-Token = "my-xauth-token"
# # apiVersion = "v1"
#
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
@@ -839,9 +743,8 @@
# ## Works with InfluxDB debug endpoints out of the box,
# ## but other services can use this format too.
# ## See the influxdb plugin's README for more details.
#
#
# ## Multiple URLs from which to read InfluxDB-formatted JSON
# ## Default is "http://localhost:8086/debug/vars".
# urls = [
# "http://localhost:8086/debug/vars"
# ]
@@ -861,7 +764,7 @@
# [[inputs.jolokia]]
# ## This is the context root used to compose the jolokia url
# context = "/jolokia"
#
#
# ## This specifies the mode used
# # mode = "proxy"
# #
@@ -871,8 +774,8 @@
# # [inputs.jolokia.proxy]
# # host = "127.0.0.1"
# # port = "8080"
#
#
#
#
# ## List of servers exposing jolokia read service
# [[inputs.jolokia.servers]]
# name = "as-server-01"
@@ -880,7 +783,7 @@
# port = "8080"
# # username = "myuser"
# # password = "mypassword"
#
#
# ## List of metrics collected on above servers
# ## Each metric consists in a name, a jmx path and either
# ## a pass or drop slice attribute.
@@ -889,13 +792,13 @@
# name = "heap_memory_usage"
# mbean = "java.lang:type=Memory"
# attribute = "HeapMemoryUsage"
#
#
# ## This collect thread counts metrics.
# [[inputs.jolokia.metrics]]
# name = "thread_count"
# mbean = "java.lang:type=Threading"
# attribute = "TotalStartedThreadCount,ThreadCount,DaemonThreadCount,PeakThreadCount"
#
#
# ## This collect number of class loaded/unloaded counts metrics.
# [[inputs.jolokia.metrics]]
# name = "class_count"
@@ -1048,7 +951,7 @@
# address = "github.com:80"
# ## Set timeout
# timeout = "1s"
#
#
# ## Optional string sent to the server
# # send = "ssh"
# ## Optional expected string in answer
@@ -1140,7 +1043,7 @@
# count = 1 # required
# ## interval, in s, at which to ping. 0 == default (ping -i <PING_INTERVAL>)
# ping_interval = 0.0
# ## per-ping timeout, in s. 0 == no timeout (ping -W <TIMEOUT>)
# ## ping timeout, in s. 0 == no timeout (ping -W <TIMEOUT>)
# timeout = 1.0
# ## interface to send ping from (ping -I <INTERFACE>)
# interface = ""
@@ -1162,7 +1065,7 @@
# ## to grab metrics for.
# ##
# address = "host=localhost user=postgres sslmode=disable"
#
#
# ## A list of databases to pull metrics about. If not specified, metrics for all
# ## databases are gathered.
# # databases = ["app_production", "testing"]
@@ -1244,7 +1147,7 @@
# # pattern = "nginx"
# ## user as argument for pgrep (ie, pgrep -u <user>)
# # user = "nginx"
#
#
# ## override for process_name
# ## This is optional; default is sourced from /proc/<pid>/status
# # process_name = "bar"
@@ -1258,7 +1161,7 @@
# [[inputs.prometheus]]
# ## An array of urls to scrape metrics from.
# urls = ["http://localhost:9100/metrics"]
#
#
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
# ## Use bearer token for authorization
@@ -1273,11 +1176,11 @@
# # Read metrics from one or many RabbitMQ servers via the management API
# [[inputs.rabbitmq]]
# # url = "http://localhost:15672"
# url = "http://localhost:15672" # required
# # name = "rmq-server-1" # optional tag
# # username = "guest"
# # password = "guest"
#
#
# ## A list of nodes to pull metrics about. If not specified, metrics for
# ## all nodes are gathered.
# # nodes = ["rabbit@node1", "rabbit@node2"]
@@ -1341,7 +1244,7 @@
# collect = ["mybulk", "sysservices", "sysdescr"]
# # Simple list of OIDs to get, in addition to "collect"
# get_oids = []
#
#
# [[inputs.snmp.host]]
# address = "192.168.2.3:161"
# community = "public"
@@ -1353,31 +1256,31 @@
# "ifNumber",
# ".1.3.6.1.2.1.1.3.0",
# ]
#
#
# [[inputs.snmp.get]]
# name = "ifnumber"
# oid = "ifNumber"
#
#
# [[inputs.snmp.get]]
# name = "interface_speed"
# oid = "ifSpeed"
# instance = "0"
#
#
# [[inputs.snmp.get]]
# name = "sysuptime"
# oid = ".1.3.6.1.2.1.1.3.0"
# unit = "second"
#
#
# [[inputs.snmp.bulk]]
# name = "mybulk"
# max_repetition = 127
# oid = ".1.3.6.1.2.1.1"
#
#
# [[inputs.snmp.bulk]]
# name = "ifoutoctets"
# max_repetition = 127
# oid = "ifOutOctets"
#
#
# [[inputs.snmp.host]]
# address = "192.168.2.13:161"
# #address = "127.0.0.1:161"
@@ -1390,19 +1293,19 @@
# [[inputs.snmp.host.table]]
# name = "iftable3"
# include_instances = ["enp5s0", "eth1"]
#
#
# # SNMP TABLEs
# # table without mapping neither subtables
# [[inputs.snmp.table]]
# name = "iftable1"
# oid = ".1.3.6.1.2.1.31.1.1.1"
#
#
# # table without mapping but with subtables
# [[inputs.snmp.table]]
# name = "iftable2"
# oid = ".1.3.6.1.2.1.31.1.1.1"
# sub_tables = [".1.3.6.1.2.1.2.2.1.13"]
#
#
# # table with mapping but without subtables
# [[inputs.snmp.table]]
# name = "iftable3"
@@ -1410,7 +1313,7 @@
# # if empty. get all instances
# mapping_table = ".1.3.6.1.2.1.31.1.1.1.1"
# # if empty, get all subtables
#
#
# # table with both mapping and subtables
# [[inputs.snmp.table]]
# name = "iftable4"
@@ -1453,33 +1356,32 @@
# [[inputs.varnish]]
# ## The default location of the varnishstat binary can be overridden with:
# binary = "/usr/bin/varnishstat"
#
#
# ## By default, telegraf gather stats for 3 metric points.
# ## Setting stats will override the defaults shown below.
# ## Glob matching can be used, ie, stats = ["MAIN.*"]
# ## stats may also be set to ["*"], which will collect all stats
# ## stats may also be set to ["all"], which will collect all stats
# stats = ["MAIN.cache_hit", "MAIN.cache_miss", "MAIN.uptime"]
# # Read metrics of ZFS from arcstats, zfetchstats, vdev_cache_stats, and pools
# # Read metrics of ZFS from arcstats, zfetchstats and vdev_cache_stats
# [[inputs.zfs]]
# ## ZFS kstat path. Ignored on FreeBSD
# ## ZFS kstat path
# ## If not specified, then default is:
# # kstatPath = "/proc/spl/kstat/zfs"
#
# kstatPath = "/proc/spl/kstat/zfs"
#
# ## By default, telegraf gather all zfs stats
# ## If not specified, then default is:
# # kstatMetrics = ["arcstats", "zfetchstats", "vdev_cache_stats"]
#
# kstatMetrics = ["arcstats", "zfetchstats", "vdev_cache_stats"]
#
# ## By default, don't gather zpool stats
# # poolMetrics = false
# poolMetrics = false
# # Reads 'mntr' stats from one or many zookeeper servers
# [[inputs.zookeeper]]
# ## An array of address to gather stats about. Specify an ip or hostname
# ## with port. ie localhost:2181, 10.0.0.1:2181, etc.
#
#
# ## If no servers are specified, then localhost is used as the host.
# ## If no port is specified, 2181 is used
# servers = [":2181"]
@@ -1503,12 +1405,12 @@
# ## an array of Zookeeper connection strings
# zookeeper_peers = ["localhost:2181"]
# ## Zookeeper Chroot
# zookeeper_chroot = ""
# zookeeper_chroot = "/"
# ## the name of the consumer group
# consumer_group = "telegraf_metrics_consumers"
# ## Offset (must be either "oldest" or "newest")
# offset = "oldest"
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -1516,66 +1418,37 @@
# data_format = "influx"
# # Stream and parse log file(s).
# [[inputs.logparser]]
# ## Log files to parse.
# ## These accept standard unix glob matching rules, but with the addition of
# ## ** as a "super asterisk". ie:
# ## /var/log/**.log -> recursively find all .log files in /var/log
# ## /var/log/*/*.log -> find all .log files with a parent dir in /var/log
# ## /var/log/apache.log -> only tail the apache log file
# files = ["/var/log/influxdb/influxdb.log"]
# ## Read file from beginning.
# from_beginning = false
#
# ## Parse logstash-style "grok" patterns:
# ## Telegraf built-in parsing patterns: https://goo.gl/dkay10
# [inputs.logparser.grok]
# ## This is a list of patterns to check the given log file(s) for.
# ## Note that adding patterns here increases processing time. The most
# ## efficient configuration is to have one pattern per logparser.
# ## Other common built-in patterns are:
# ## %{COMMON_LOG_FORMAT} (plain apache & nginx access logs)
# ## %{COMBINED_LOG_FORMAT} (access logs + referrer & agent)
# patterns = ["%{INFLUXDB_HTTPD_LOG}"]
# ## Full path(s) to custom pattern files.
# custom_pattern_files = []
# ## Custom patterns can also be defined here. Put one pattern per line.
# custom_patterns = '''
# '''
# # Read metrics from MQTT topic(s)
# [[inputs.mqtt_consumer]]
# servers = ["localhost:1883"]
# ## MQTT QoS, must be 0, 1, or 2
# qos = 0
#
#
# ## Topics to subscribe to
# topics = [
# "telegraf/host01/cpu",
# "telegraf/+/mem",
# "sensors/#",
# ]
#
#
# # if true, messages that can't be delivered while the subscriber is offline
# # will be delivered when it comes back (such as on service restart).
# # NOTE: if true, client_id MUST be set
# persistent_session = false
# # If empty, a random client ID will be generated.
# client_id = ""
#
#
# ## username and password to connect MQTT server.
# # username = "telegraf"
# # password = "metricsmetricsmetricsmetrics"
#
#
# ## Optional SSL Config
# # ssl_ca = "/etc/telegraf/ca.pem"
# # ssl_cert = "/etc/telegraf/cert.pem"
# # ssl_key = "/etc/telegraf/key.pem"
# ## Use SSL but skip chain & host verification
# # insecure_skip_verify = false
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -1593,7 +1466,7 @@
# subjects = ["telegraf"]
# ## name a queue group
# queue_group = "telegraf_consumers"
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -1601,12 +1474,6 @@
# data_format = "influx"
# # A Rollbar Webhook Event collector
# [[inputs.rollbar_webhooks]]
# ## Address and port to host Webhook listener on
# service_address = ":1619"
# # Statsd Server
# [[inputs.statsd]]
# ## Address and port to host UDP listener on
@@ -1621,24 +1488,24 @@
# delete_timings = true
# ## Percentiles to calculate for timing & histogram stats
# percentiles = [90]
#
#
# ## separator to use between elements of a statsd metric
# metric_separator = "_"
#
#
# ## Parses tags in the datadog statsd format
# ## http://docs.datadoghq.com/guides/dogstatsd/
# parse_data_dog_tags = false
#
#
# ## Statsd data translation templates, more info can be read here:
# ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#graphite
# # templates = [
# # "cpu.* measurement*"
# # ]
#
#
# ## Number of UDP messages allowed to queue up, once filled,
# ## the statsd server will start dropping packets
# allowed_pending_messages = 10000
#
#
# ## Number of timing/histogram values to track per-measurement in the
# ## calculation of percentiles. Raising this limit increases the accuracy
# ## of percentiles but also increases the memory usage and cpu time.
@@ -1659,7 +1526,7 @@
# files = ["/var/mymetrics.out"]
# ## Read file from beginning.
# from_beginning = false
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -1671,14 +1538,14 @@
# [[inputs.tcp_listener]]
# ## Address and port to host TCP listener on
# service_address = ":8094"
#
#
# ## Number of TCP messages allowed to queue up. Once filled, the
# ## TCP listener will start dropping packets.
# allowed_pending_messages = 10000
#
#
# ## Maximum number of concurrent TCP connections to allow
# max_tcp_connections = 250
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:
@@ -1690,11 +1557,11 @@
# [[inputs.udp_listener]]
# ## Address and port to host UDP listener on
# service_address = ":8092"
#
#
# ## Number of UDP messages allowed to queue up. Once filled, the
# ## UDP listener will start dropping packets.
# allowed_pending_messages = 10000
#
#
# ## Data format to consume.
# ## Each data format has it's own unique set of configuration options, read
# ## more about them here:

View File

@@ -1,79 +0,0 @@
package filter
import (
"strings"
"github.com/gobwas/glob"
)
type Filter interface {
Match(string) bool
}
// CompileFilter takes a list of string filters and returns a Filter interface
// for matching a given string against the filter list. The filter list
// supports glob matching too, ie:
//
// f, _ := CompileFilter([]string{"cpu", "mem", "net*"})
// f.Match("cpu") // true
// f.Match("network") // true
// f.Match("memory") // false
//
func CompileFilter(filters []string) (Filter, error) {
// return if there is nothing to compile
if len(filters) == 0 {
return nil, nil
}
// check if we can compile a non-glob filter
noGlob := true
for _, filter := range filters {
if hasMeta(filter) {
noGlob = false
break
}
}
switch {
case noGlob:
// return non-globbing filter if not needed.
return compileFilterNoGlob(filters), nil
case len(filters) == 1:
return glob.Compile(filters[0])
default:
return glob.Compile("{" + strings.Join(filters, ",") + "}")
}
}
// hasMeta reports whether path contains any magic glob characters.
func hasMeta(s string) bool {
return strings.IndexAny(s, "*?[") >= 0
}
type filter struct {
m map[string]struct{}
}
func (f *filter) Match(s string) bool {
_, ok := f.m[s]
return ok
}
type filtersingle struct {
s string
}
func (f *filtersingle) Match(s string) bool {
return f.s == s
}
func compileFilterNoGlob(filters []string) Filter {
if len(filters) == 1 {
return &filtersingle{s: filters[0]}
}
out := filter{m: make(map[string]struct{})}
for _, filter := range filters {
out.m[filter] = struct{}{}
}
return &out
}

View File

@@ -1,96 +0,0 @@
package filter
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestCompileFilter(t *testing.T) {
f, err := CompileFilter([]string{})
assert.NoError(t, err)
assert.Nil(t, f)
f, err = CompileFilter([]string{"cpu"})
assert.NoError(t, err)
assert.True(t, f.Match("cpu"))
assert.False(t, f.Match("cpu0"))
assert.False(t, f.Match("mem"))
f, err = CompileFilter([]string{"cpu*"})
assert.NoError(t, err)
assert.True(t, f.Match("cpu"))
assert.True(t, f.Match("cpu0"))
assert.False(t, f.Match("mem"))
f, err = CompileFilter([]string{"cpu", "mem"})
assert.NoError(t, err)
assert.True(t, f.Match("cpu"))
assert.False(t, f.Match("cpu0"))
assert.True(t, f.Match("mem"))
f, err = CompileFilter([]string{"cpu", "mem", "net*"})
assert.NoError(t, err)
assert.True(t, f.Match("cpu"))
assert.False(t, f.Match("cpu0"))
assert.True(t, f.Match("mem"))
assert.True(t, f.Match("network"))
}
var benchbool bool
func BenchmarkFilterSingleNoGlobFalse(b *testing.B) {
f, _ := CompileFilter([]string{"cpu"})
var tmp bool
for n := 0; n < b.N; n++ {
tmp = f.Match("network")
}
benchbool = tmp
}
func BenchmarkFilterSingleNoGlobTrue(b *testing.B) {
f, _ := CompileFilter([]string{"cpu"})
var tmp bool
for n := 0; n < b.N; n++ {
tmp = f.Match("cpu")
}
benchbool = tmp
}
func BenchmarkFilter(b *testing.B) {
f, _ := CompileFilter([]string{"cpu", "mem", "net*"})
var tmp bool
for n := 0; n < b.N; n++ {
tmp = f.Match("network")
}
benchbool = tmp
}
func BenchmarkFilterNoGlob(b *testing.B) {
f, _ := CompileFilter([]string{"cpu", "mem", "net"})
var tmp bool
for n := 0; n < b.N; n++ {
tmp = f.Match("net")
}
benchbool = tmp
}
func BenchmarkFilter2(b *testing.B) {
f, _ := CompileFilter([]string{"aa", "bb", "c", "ad", "ar", "at", "aq",
"aw", "az", "axxx", "ab", "cpu", "mem", "net*"})
var tmp bool
for n := 0; n < b.N; n++ {
tmp = f.Match("network")
}
benchbool = tmp
}
func BenchmarkFilter2NoGlob(b *testing.B) {
f, _ := CompileFilter([]string{"aa", "bb", "c", "ad", "ar", "at", "aq",
"aw", "az", "axxx", "ab", "cpu", "mem", "net"})
var tmp bool
for n := 0; n < b.N; n++ {
tmp = f.Match("net")
}
benchbool = tmp
}

View File

@@ -1,49 +0,0 @@
package aws
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/client"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/stscreds"
"github.com/aws/aws-sdk-go/aws/session"
)
type CredentialConfig struct {
Region string
AccessKey string
SecretKey string
RoleARN string
Profile string
Filename string
Token string
}
func (c *CredentialConfig) Credentials() client.ConfigProvider {
if c.RoleARN != "" {
return c.assumeCredentials()
} else {
return c.rootCredentials()
}
}
func (c *CredentialConfig) rootCredentials() client.ConfigProvider {
config := &aws.Config{
Region: aws.String(c.Region),
}
if c.AccessKey != "" || c.SecretKey != "" {
config.Credentials = credentials.NewStaticCredentials(c.AccessKey, c.SecretKey, c.Token)
} else if c.Profile != "" || c.Filename != "" {
config.Credentials = credentials.NewSharedCredentials(c.Filename, c.Profile)
}
return session.New(config)
}
func (c *CredentialConfig) assumeCredentials() client.ConfigProvider {
rootCredentials := c.rootCredentials()
config := &aws.Config{
Region: aws.String(c.Region),
}
config.Credentials = stscreds.NewCredentials(rootCredentials, c.RoleARN)
return session.New(config)
}

View File

@@ -58,6 +58,7 @@ func NewConfig() *Config {
Interval: internal.Duration{Duration: 10 * time.Second},
RoundInterval: true,
FlushInterval: internal.Duration{Duration: 10 * time.Second},
FlushJitter: internal.Duration{Duration: 5 * time.Second},
},
Tags: make(map[string]string),
@@ -77,14 +78,6 @@ type AgentConfig struct {
// ie, if Interval=10s then always collect on :00, :10, :20, etc.
RoundInterval bool
// By default, precision will be set to the same timestamp order as the
// collection interval, with the maximum being 1s.
// ie, when interval = "10s", precision will be "1s"
// when interval = "250ms", precision will be "1ms"
// Precision will NOT be used for service inputs. It is up to each individual
// service input to set the timestamp at the appropriate precision.
Precision internal.Duration
// CollectionJitter is used to jitter the collection by a random amount.
// Each plugin will sleep for a random time within jitter before collecting.
// This can be used to avoid many plugins querying things like sysfs at the
@@ -116,10 +109,11 @@ type AgentConfig struct {
// does _not_ deactivate FlushInterval.
FlushBufferWhenFull bool
// TODO(cam): Remove UTC and parameter, they are no longer
// TODO(cam): Remove UTC and Precision parameters, they are no longer
// valid for the agent config. Leaving them here for now for backwards-
// compatability
UTC bool `toml:"utc"`
UTC bool `toml:"utc"`
Precision string
// Debug is the option for running in debug mode
Debug bool
@@ -216,11 +210,6 @@ var header = `# Telegraf Configuration
## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
flush_jitter = "0s"
## By default, precision will be set to the same timestamp order as the
## collection interval, with the maximum being 1s.
## Precision will NOT be used for service inputs, such as logparser and statsd.
## Valid values are "Nns", "Nus" (or "Nµs"), "Nms", "Ns".
precision = ""
## Run telegraf in debug mode
debug = false
## Run telegraf in quiet mode
@@ -368,7 +357,7 @@ func printConfig(name string, p printer, op string, commented bool) {
fmt.Print("\n")
continue
}
fmt.Print(strings.TrimRight(comment+line, " ") + "\n")
fmt.Print(comment + line + "\n")
}
}
}

View File

@@ -1,37 +0,0 @@
package errchan
import (
"fmt"
"strings"
)
type ErrChan struct {
C chan error
}
// New returns an error channel of max length 'n'
// errors can be sent to the ErrChan.C channel, and will be returned when
// ErrChan.Error() is called.
func New(n int) *ErrChan {
return &ErrChan{
C: make(chan error, n),
}
}
// Error closes the ErrChan.C channel and returns an error if there are any
// non-nil errors, otherwise returns nil.
func (e *ErrChan) Error() error {
close(e.C)
var out string
for err := range e.C {
if err != nil {
out += "[" + err.Error() + "], "
}
}
if out != "" {
return fmt.Errorf("Errors encountered: " + strings.TrimRight(out, ", "))
}
return nil
}

View File

@@ -10,7 +10,6 @@ import (
"fmt"
"io/ioutil"
"log"
"math/big"
"os"
"os/exec"
"strconv"
@@ -206,27 +205,3 @@ func WaitTimeout(c *exec.Cmd, timeout time.Duration) error {
return TimeoutErr
}
}
// RandomSleep will sleep for a random amount of time up to max.
// If the shutdown channel is closed, it will return before it has finished
// sleeping.
func RandomSleep(max time.Duration, shutdown chan struct{}) {
if max == 0 {
return
}
maxSleep := big.NewInt(max.Nanoseconds())
var sleepns int64
if j, err := rand.Int(rand.Reader, maxSleep); err == nil {
sleepns = j.Int64()
}
t := time.NewTimer(time.Nanosecond * time.Duration(sleepns))
select {
case <-t.C:
return
case <-shutdown:
t.Stop()
return
}
}

View File

@@ -106,28 +106,3 @@ func TestRunError(t *testing.T) {
assert.Error(t, err)
}
func TestRandomSleep(t *testing.T) {
// test that zero max returns immediately
s := time.Now()
RandomSleep(time.Duration(0), make(chan struct{}))
elapsed := time.Since(s)
assert.True(t, elapsed < time.Millisecond)
// test that max sleep is respected
s = time.Now()
RandomSleep(time.Millisecond*50, make(chan struct{}))
elapsed = time.Since(s)
assert.True(t, elapsed < time.Millisecond*50)
// test that shutdown is respected
s = time.Now()
shutdown := make(chan struct{})
go func() {
time.Sleep(time.Millisecond * 100)
close(shutdown)
}()
RandomSleep(time.Second, shutdown)
elapsed = time.Since(s)
assert.True(t, elapsed < time.Millisecond*150)
}

View File

@@ -1,59 +0,0 @@
package limiter
import (
"sync"
"time"
)
// NewRateLimiter returns a rate limiter that will will emit from the C
// channel only 'n' times every 'rate' seconds.
func NewRateLimiter(n int, rate time.Duration) *rateLimiter {
r := &rateLimiter{
C: make(chan bool),
rate: rate,
n: n,
shutdown: make(chan bool),
}
r.wg.Add(1)
go r.limiter()
return r
}
type rateLimiter struct {
C chan bool
rate time.Duration
n int
shutdown chan bool
wg sync.WaitGroup
}
func (r *rateLimiter) Stop() {
close(r.shutdown)
r.wg.Wait()
close(r.C)
}
func (r *rateLimiter) limiter() {
defer r.wg.Done()
ticker := time.NewTicker(r.rate)
defer ticker.Stop()
counter := 0
for {
select {
case <-r.shutdown:
return
case <-ticker.C:
counter = 0
default:
if counter < r.n {
select {
case r.C <- true:
counter++
case <-r.shutdown:
return
}
}
}
}
}

View File

@@ -1,54 +0,0 @@
package limiter
import (
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestRateLimiter(t *testing.T) {
r := NewRateLimiter(5, time.Second)
ticker := time.NewTicker(time.Millisecond * 75)
// test that we can only get 5 receives from the rate limiter
counter := 0
outer:
for {
select {
case <-r.C:
counter++
case <-ticker.C:
break outer
}
}
assert.Equal(t, 5, counter)
r.Stop()
// verify that the Stop function closes the channel.
_, ok := <-r.C
assert.False(t, ok)
}
func TestRateLimiterMultipleIterations(t *testing.T) {
r := NewRateLimiter(5, time.Millisecond*50)
ticker := time.NewTicker(time.Millisecond * 250)
// test that we can get 15 receives from the rate limiter
counter := 0
outer:
for {
select {
case <-ticker.C:
break outer
case <-r.C:
counter++
}
}
assert.True(t, counter > 10)
r.Stop()
// verify that the Stop function closes the channel.
_, ok := <-r.C
assert.False(t, ok)
}

View File

@@ -2,79 +2,81 @@ package internal_models
import (
"fmt"
"strings"
"github.com/gobwas/glob"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/filter"
)
// TagFilter is the name of a tag, and the values on which to filter
type TagFilter struct {
Name string
Filter []string
filter filter.Filter
filter glob.Glob
}
// Filter containing drop/pass and tagdrop/tagpass rules
type Filter struct {
NameDrop []string
nameDrop filter.Filter
nameDrop glob.Glob
NamePass []string
namePass filter.Filter
namePass glob.Glob
FieldDrop []string
fieldDrop filter.Filter
fieldDrop glob.Glob
FieldPass []string
fieldPass filter.Filter
fieldPass glob.Glob
TagDrop []TagFilter
TagPass []TagFilter
TagExclude []string
tagExclude filter.Filter
tagExclude glob.Glob
TagInclude []string
tagInclude filter.Filter
tagInclude glob.Glob
IsActive bool
}
// Compile all Filter lists into filter.Filter objects.
// Compile all Filter lists into glob.Glob objects.
func (f *Filter) CompileFilter() error {
var err error
f.nameDrop, err = filter.CompileFilter(f.NameDrop)
f.nameDrop, err = compileFilter(f.NameDrop)
if err != nil {
return fmt.Errorf("Error compiling 'namedrop', %s", err)
}
f.namePass, err = filter.CompileFilter(f.NamePass)
f.namePass, err = compileFilter(f.NamePass)
if err != nil {
return fmt.Errorf("Error compiling 'namepass', %s", err)
}
f.fieldDrop, err = filter.CompileFilter(f.FieldDrop)
f.fieldDrop, err = compileFilter(f.FieldDrop)
if err != nil {
return fmt.Errorf("Error compiling 'fielddrop', %s", err)
}
f.fieldPass, err = filter.CompileFilter(f.FieldPass)
f.fieldPass, err = compileFilter(f.FieldPass)
if err != nil {
return fmt.Errorf("Error compiling 'fieldpass', %s", err)
}
f.tagExclude, err = filter.CompileFilter(f.TagExclude)
f.tagExclude, err = compileFilter(f.TagExclude)
if err != nil {
return fmt.Errorf("Error compiling 'tagexclude', %s", err)
}
f.tagInclude, err = filter.CompileFilter(f.TagInclude)
f.tagInclude, err = compileFilter(f.TagInclude)
if err != nil {
return fmt.Errorf("Error compiling 'taginclude', %s", err)
}
for i, _ := range f.TagDrop {
f.TagDrop[i].filter, err = filter.CompileFilter(f.TagDrop[i].Filter)
f.TagDrop[i].filter, err = compileFilter(f.TagDrop[i].Filter)
if err != nil {
return fmt.Errorf("Error compiling 'tagdrop', %s", err)
}
}
for i, _ := range f.TagPass {
f.TagPass[i].filter, err = filter.CompileFilter(f.TagPass[i].Filter)
f.TagPass[i].filter, err = compileFilter(f.TagPass[i].Filter)
if err != nil {
return fmt.Errorf("Error compiling 'tagpass', %s", err)
}
@@ -82,6 +84,20 @@ func (f *Filter) CompileFilter() error {
return nil
}
func compileFilter(filter []string) (glob.Glob, error) {
if len(filter) == 0 {
return nil, nil
}
var g glob.Glob
var err error
if len(filter) == 1 {
g, err = glob.Compile(filter[0])
} else {
g, err = glob.Compile("{" + strings.Join(filter, ",") + "}")
}
return g, err
}
func (f *Filter) ShouldMetricPass(metric telegraf.Metric) bool {
if f.ShouldNamePass(metric.Name()) && f.ShouldTagsPass(metric.Tags()) {
return true

View File

@@ -253,6 +253,51 @@ func TestFilter_TagDrop(t *testing.T) {
}
}
func TestFilter_CompileFilterError(t *testing.T) {
f := Filter{
NameDrop: []string{"", ""},
}
assert.Error(t, f.CompileFilter())
f = Filter{
NamePass: []string{"", ""},
}
assert.Error(t, f.CompileFilter())
f = Filter{
FieldDrop: []string{"", ""},
}
assert.Error(t, f.CompileFilter())
f = Filter{
FieldPass: []string{"", ""},
}
assert.Error(t, f.CompileFilter())
f = Filter{
TagExclude: []string{"", ""},
}
assert.Error(t, f.CompileFilter())
f = Filter{
TagInclude: []string{"", ""},
}
assert.Error(t, f.CompileFilter())
filters := []TagFilter{
TagFilter{
Name: "cpu",
Filter: []string{"{foobar}"},
}}
f = Filter{
TagDrop: filters,
}
require.Error(t, f.CompileFilter())
filters = []TagFilter{
TagFilter{
Name: "cpu",
Filter: []string{"{foobar}"},
}}
f = Filter{
TagPass: filters,
}
require.Error(t, f.CompileFilter())
}
func TestFilter_ShouldMetricsPass(t *testing.T) {
m := testutil.TestMetric(1, "testmetric")
f := Filter{

View File

@@ -45,9 +45,14 @@ func NewMetric(
name string,
tags map[string]string,
fields map[string]interface{},
t time.Time,
t ...time.Time,
) (Metric, error) {
pt, err := client.NewPoint(name, tags, fields, t)
var T time.Time
if len(t) > 0 {
T = t[0]
}
pt, err := client.NewPoint(name, tags, fields, T)
if err != nil {
return nil, err
}

View File

@@ -51,6 +51,23 @@ func TestNewMetricString(t *testing.T) {
assert.Equal(t, lineProtoPrecision, m.PrecisionString("s"))
}
func TestNewMetricStringNoTime(t *testing.T) {
tags := map[string]string{
"host": "localhost",
}
fields := map[string]interface{}{
"usage_idle": float64(99),
}
m, err := NewMetric("cpu", tags, fields)
assert.NoError(t, err)
lineProto := fmt.Sprintf("cpu,host=localhost usage_idle=99")
assert.Equal(t, lineProto, m.String())
lineProtoPrecision := fmt.Sprintf("cpu,host=localhost usage_idle=99")
assert.Equal(t, lineProtoPrecision, m.PrecisionString("s"))
}
func TestNewMetricFailNaN(t *testing.T) {
now := time.Now()

View File

@@ -8,8 +8,6 @@ import (
_ "github.com/influxdata/telegraf/plugins/inputs/ceph"
_ "github.com/influxdata/telegraf/plugins/inputs/chrony"
_ "github.com/influxdata/telegraf/plugins/inputs/cloudwatch"
_ "github.com/influxdata/telegraf/plugins/inputs/conntrack"
_ "github.com/influxdata/telegraf/plugins/inputs/consul"
_ "github.com/influxdata/telegraf/plugins/inputs/couchbase"
_ "github.com/influxdata/telegraf/plugins/inputs/couchdb"
_ "github.com/influxdata/telegraf/plugins/inputs/disque"
@@ -20,7 +18,6 @@ import (
_ "github.com/influxdata/telegraf/plugins/inputs/exec"
_ "github.com/influxdata/telegraf/plugins/inputs/filestat"
_ "github.com/influxdata/telegraf/plugins/inputs/github_webhooks"
_ "github.com/influxdata/telegraf/plugins/inputs/graylog"
_ "github.com/influxdata/telegraf/plugins/inputs/haproxy"
_ "github.com/influxdata/telegraf/plugins/inputs/http_response"
_ "github.com/influxdata/telegraf/plugins/inputs/httpjson"
@@ -29,7 +26,6 @@ import (
_ "github.com/influxdata/telegraf/plugins/inputs/jolokia"
_ "github.com/influxdata/telegraf/plugins/inputs/kafka_consumer"
_ "github.com/influxdata/telegraf/plugins/inputs/leofs"
_ "github.com/influxdata/telegraf/plugins/inputs/logparser"
_ "github.com/influxdata/telegraf/plugins/inputs/lustre2"
_ "github.com/influxdata/telegraf/plugins/inputs/mailchimp"
_ "github.com/influxdata/telegraf/plugins/inputs/memcached"
@@ -57,7 +53,6 @@ import (
_ "github.com/influxdata/telegraf/plugins/inputs/redis"
_ "github.com/influxdata/telegraf/plugins/inputs/rethinkdb"
_ "github.com/influxdata/telegraf/plugins/inputs/riak"
_ "github.com/influxdata/telegraf/plugins/inputs/rollbar_webhooks"
_ "github.com/influxdata/telegraf/plugins/inputs/sensors"
_ "github.com/influxdata/telegraf/plugins/inputs/snmp"
_ "github.com/influxdata/telegraf/plugins/inputs/sqlserver"

View File

@@ -1,7 +1,7 @@
# Telegraf plugin: Apache
#### Plugin arguments:
- **urls** []string: List of apache-status URLs to collect from. Default is "http://localhost/server-status?auto".
- **urls** []string: List of apache-status URLs to collect from.
#### Description

View File

@@ -21,7 +21,6 @@ type Apache struct {
var sampleConfig = `
## An array of Apache status URI to gather stats.
## Default is "http://localhost/server-status?auto".
urls = ["http://localhost/server-status?auto"]
`
@@ -34,10 +33,6 @@ func (n *Apache) Description() string {
}
func (n *Apache) Gather(acc telegraf.Accumulator) error {
if len(n.Urls) == 0 {
n.Urls = []string{"http://localhost/server-status?auto"}
}
var wg sync.WaitGroup
var outerr error

View File

@@ -40,7 +40,7 @@ is computed for the new frequency, with weights depending on these accuracies. I
measurements from the reference source follow a consistent trend, the residual will be
driven to zero over time.
- Skew - This is the estimated error bound on the frequency.
- Root delay - This is the total of the network path delays to the stratum-1 computer
- Root delay -This is the total of the network path delays to the stratum-1 computer
from which the computer is ultimately synchronised. In certain extreme situations, this
value can be negative. (This can arise in a symmetric peer arrangement where the computers
frequencies are not tracking each other and the network delay is very short relative to the
@@ -56,8 +56,7 @@ Delete second or Not synchronised.
```toml
# Get standard chrony metrics, requires chronyc executable.
[[inputs.chrony]]
## If true, chronyc tries to perform a DNS lookup for the time server.
# dns_lookup = false
# no configuration
```
### Measurements & Fields:

View File

@@ -20,8 +20,7 @@ var (
)
type Chrony struct {
DNSLookup bool `toml:"dns_lookup"`
path string
path string
}
func (*Chrony) Description() string {
@@ -29,24 +28,14 @@ func (*Chrony) Description() string {
}
func (*Chrony) SampleConfig() string {
return `
## If true, chronyc tries to perform a DNS lookup for the time server.
# dns_lookup = false
`
return ""
}
func (c *Chrony) Gather(acc telegraf.Accumulator) error {
if len(c.path) == 0 {
return errors.New("chronyc not found: verify that chrony is installed and that chronyc is in your PATH")
}
flags := []string{}
if !c.DNSLookup {
flags = append(flags, "-n")
}
flags = append(flags, "tracking")
cmd := execCommand(c.path, flags...)
cmd := execCommand(c.path, "tracking")
out, err := internal.CombinedOutputTimeout(cmd, time.Second*5)
if err != nil {
return fmt.Errorf("failed to run command %s: %s - %s", strings.Join(cmd.Args, " "), err, string(out))

View File

@@ -42,15 +42,6 @@ func TestGather(t *testing.T) {
}
acc.AssertContainsTaggedFields(t, "chrony", fields, tags)
// test with dns lookup
c.DNSLookup = true
err = c.Gather(&acc)
if err != nil {
t.Fatal(err)
}
acc.AssertContainsTaggedFields(t, "chrony", fields, tags)
}
// fackeExecCommand is a helper function that mock
@@ -72,9 +63,8 @@ func TestHelperProcess(t *testing.T) {
return
}
lookup := "Reference ID : 192.168.1.22 (ntp.example.com)\n"
noLookup := "Reference ID : 192.168.1.22 (192.168.1.22)\n"
mockData := `Stratum : 3
mockData := `Reference ID : 192.168.1.22 (ntp.example.com)
Stratum : 3
Ref time (UTC) : Thu May 12 14:27:07 2016
System time : 0.000020390 seconds fast of NTP time
Last offset : +0.000012651 seconds
@@ -94,12 +84,8 @@ Leap status : Normal
// /tmp/go-build970079519/…/_test/integration.test -test.run=TestHelperProcess --
cmd, args := args[3], args[4:]
if cmd == "chronyc" {
if args[0] == "tracking" {
fmt.Fprint(os.Stdout, lookup+mockData)
} else {
fmt.Fprint(os.Stdout, noLookup+mockData)
}
if cmd == "chronyc" && args[0] == "tracking" {
fmt.Fprint(os.Stdout, mockData)
} else {
fmt.Fprint(os.Stdout, "command not found")
os.Exit(1)

View File

@@ -6,12 +6,9 @@ This plugin will pull Metric Statistics from Amazon CloudWatch.
This plugin uses a credential chain for Authentication with the CloudWatch
API endpoint. In the following order the plugin will attempt to authenticate.
1. Assumed credentials via STS if `role_arn` attribute is specified (source credentials are evaluated from subsequent rules)
2. Explicit credentials from `access_key`, `secret_key`, and `token` attributes
3. Shared profile from `profile` attribute
4. [Environment Variables](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk#environment-variables)
5. [Shared Credentials](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk#shared-credentials-file)
6. [EC2 Instance Profile](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
1. [IAMS Role](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
2. [Environment Variables](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk#environment-variables)
3. [Shared Credentials](https://github.com/aws/aws-sdk-go/wiki/configuring-sdk#shared-credentials-file)
### Configuration:
@@ -27,7 +24,7 @@ API endpoint. In the following order the plugin will attempt to authenticate.
delay = '1m'
## Override global run interval (optional - defaults to global interval)
## Recomended: use metric 'interval' that is a multiple of 'period' to avoid
## Recomended: use metric 'interval' that is a multiple of 'period' to avoid
## gaps or overlap in pulled data
interval = '1m'
@@ -39,15 +36,11 @@ API endpoint. In the following order the plugin will attempt to authenticate.
## Refreshes Namespace available metrics every 1h
[[inputs.cloudwatch.metrics]]
names = ['Latency', 'RequestCount']
## Dimension filters for Metric (optional)
[[inputs.cloudwatch.metrics.dimensions]]
name = 'LoadBalancerName'
value = 'p-example'
[[inputs.cloudwatch.metrics.dimensions]]
name = 'AvailabilityZone'
value = '*'
```
#### Requirements and Terminology
@@ -59,39 +52,6 @@ Plugin Configuration utilizes [CloudWatch concepts](http://docs.aws.amazon.com/A
- `names` must be valid CloudWatch [Metric](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Metric) names
- `dimensions` must be valid CloudWatch [Dimension](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Dimension) name/value pairs
Omitting or specifying a value of `'*'` for a dimension value configures all available metrics that contain a dimension with the specified name
to be retrieved. If specifying >1 dimension, then the metric must contain *all* the configured dimensions where the the value of the
wildcard dimension is ignored.
Example:
```
[[inputs.cloudwatch.metrics]]
names = ['Latency']
## Dimension filters for Metric (optional)
[[inputs.cloudwatch.metrics.dimensions]]
name = 'LoadBalancerName'
value = 'p-example'
[[inputs.cloudwatch.metrics.dimensions]]
name = 'AvailabilityZone'
value = '*'
```
If the following ELBs are available:
- name: `p-example`, availabilityZone: `us-east-1a`
- name: `p-example`, availabilityZone: `us-east-1b`
- name: `q-example`, availabilityZone: `us-east-1a`
- name: `q-example`, availabilityZone: `us-east-1b`
Then 2 metrics will be output:
- name: `p-example`, availabilityZone: `us-east-1a`
- name: `p-example`, availabilityZone: `us-east-1b`
If the `AvailabilityZone` wildcard dimension was omitted, then a single metric (name: `p-example`)
would be exported containing the aggregate values of the ELB across availability zones.
#### Restrictions and Limitations
- CloudWatch metrics are not available instantly via the CloudWatch API. You should adjust your collection `delay` to account for this lag in metrics availability based on your [monitoring subscription level](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html)
- CloudWatch API usage incurs cost - see [GetMetricStatistics Pricing](https://aws.amazon.com/cloudwatch/pricing/)

View File

@@ -3,36 +3,28 @@ package cloudwatch
import (
"fmt"
"strings"
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/cloudwatch"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
internalaws "github.com/influxdata/telegraf/internal/config/aws"
"github.com/influxdata/telegraf/internal/errchan"
"github.com/influxdata/telegraf/internal/limiter"
"github.com/influxdata/telegraf/plugins/inputs"
)
type (
CloudWatch struct {
Region string `toml:"region"`
AccessKey string `toml:"access_key"`
SecretKey string `toml:"secret_key"`
RoleARN string `toml:"role_arn"`
Profile string `toml:"profile"`
Filename string `toml:"shared_credential_file"`
Token string `toml:"token"`
Region string `toml:"region"`
AccessKey string `toml:"access_key"`
SecretKey string `toml:"secret_key"`
Period internal.Duration `toml:"period"`
Delay internal.Duration `toml:"delay"`
Namespace string `toml:"namespace"`
Metrics []*Metric `toml:"metrics"`
CacheTTL internal.Duration `toml:"cache_ttl"`
client cloudwatchClient
metricCache *MetricCache
}
@@ -66,18 +58,12 @@ func (c *CloudWatch) SampleConfig() string {
## Amazon Credentials
## Credentials are loaded in the following order
## 1) Assumed credentials via STS if role_arn is specified
## 2) explicit credentials from 'access_key' and 'secret_key'
## 3) shared profile from 'profile'
## 4) environment variables
## 5) shared credentials file
## 6) EC2 Instance Profile
## 1) explicit credentials from 'access_key' and 'secret_key'
## 2) environment variables
## 3) shared credentials file
## 4) EC2 Instance Profile
#access_key = ""
#secret_key = ""
#token = ""
#role_arn = ""
#profile = ""
#shared_credential_file = ""
## Requested CloudWatch aggregation Period (required - must be a multiple of 60s)
period = '1m'
@@ -89,10 +75,6 @@ func (c *CloudWatch) SampleConfig() string {
## gaps or overlap in pulled data
interval = '1m'
## Configure the TTL for the internal cache of metrics.
## Defaults to 1 hr if not specified
#cache_ttl = '10m'
## Metric Statistic Namespace (required)
namespace = 'AWS/ELB'
@@ -124,40 +106,20 @@ func (c *CloudWatch) Gather(acc telegraf.Accumulator) error {
if c.Metrics != nil {
metrics = []*cloudwatch.Metric{}
for _, m := range c.Metrics {
if !hasWilcard(m.Dimensions) {
dimensions := make([]*cloudwatch.Dimension, len(m.Dimensions))
for k, d := range m.Dimensions {
fmt.Printf("Dimension [%s]:[%s]\n", d.Name, d.Value)
dimensions[k] = &cloudwatch.Dimension{
Name: aws.String(d.Name),
Value: aws.String(d.Value),
}
}
for _, name := range m.MetricNames {
metrics = append(metrics, &cloudwatch.Metric{
Namespace: aws.String(c.Namespace),
MetricName: aws.String(name),
Dimensions: dimensions,
})
}
} else {
allMetrics, err := c.fetchNamespaceMetrics()
if err != nil {
return err
}
for _, name := range m.MetricNames {
for _, metric := range allMetrics {
if isSelected(metric, m.Dimensions) {
metrics = append(metrics, &cloudwatch.Metric{
Namespace: aws.String(c.Namespace),
MetricName: aws.String(name),
Dimensions: metric.Dimensions,
})
}
}
dimensions := make([]*cloudwatch.Dimension, len(m.Dimensions))
for k, d := range m.Dimensions {
dimensions[k] = &cloudwatch.Dimension{
Name: aws.String(d.Name),
Value: aws.String(d.Value),
}
}
for _, name := range m.MetricNames {
metrics = append(metrics, &cloudwatch.Metric{
Namespace: aws.String(c.Namespace),
MetricName: aws.String(name),
Dimensions: dimensions,
})
}
}
} else {
var err error
@@ -168,35 +130,30 @@ func (c *CloudWatch) Gather(acc telegraf.Accumulator) error {
}
metricCount := len(metrics)
errChan := errchan.New(metricCount)
var errChan = make(chan error, metricCount)
now := time.Now()
// limit concurrency or we can easily exhaust user connection limit
// see cloudwatch API request limits:
// http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_limits.html
lmtr := limiter.NewRateLimiter(10, time.Second)
defer lmtr.Stop()
var wg sync.WaitGroup
wg.Add(len(metrics))
for _, m := range metrics {
<-lmtr.C
go func(inm *cloudwatch.Metric) {
defer wg.Done()
c.gatherMetric(acc, inm, now, errChan.C)
}(m)
}
wg.Wait()
semaphore := make(chan byte, 64)
return errChan.Error()
for _, m := range metrics {
semaphore <- 0x1
go c.gatherMetric(acc, m, now, semaphore, errChan)
}
for i := 1; i <= metricCount; i++ {
err := <-errChan
if err != nil {
return err
}
}
return nil
}
func init() {
inputs.Add("cloudwatch", func() telegraf.Input {
ttl, _ := time.ParseDuration("1hr")
return &CloudWatch{
CacheTTL: internal.Duration{Duration: ttl},
}
return &CloudWatch{}
})
}
@@ -204,18 +161,14 @@ func init() {
* Initialize CloudWatch client
*/
func (c *CloudWatch) initializeCloudWatch() error {
credentialConfig := &internalaws.CredentialConfig{
Region: c.Region,
AccessKey: c.AccessKey,
SecretKey: c.SecretKey,
RoleARN: c.RoleARN,
Profile: c.Profile,
Filename: c.Filename,
Token: c.Token,
config := &aws.Config{
Region: aws.String(c.Region),
}
if c.AccessKey != "" || c.SecretKey != "" {
config.Credentials = credentials.NewStaticCredentials(c.AccessKey, c.SecretKey, "")
}
configProvider := credentialConfig.Credentials()
c.client = cloudwatch.New(configProvider)
c.client = cloudwatch.New(session.New(config))
return nil
}
@@ -250,10 +203,11 @@ func (c *CloudWatch) fetchNamespaceMetrics() (metrics []*cloudwatch.Metric, err
more = token != nil
}
cacheTTL, _ := time.ParseDuration("1hr")
c.metricCache = &MetricCache{
Metrics: metrics,
Fetched: time.Now(),
TTL: c.CacheTTL.Duration,
TTL: cacheTTL,
}
return
@@ -262,16 +216,12 @@ func (c *CloudWatch) fetchNamespaceMetrics() (metrics []*cloudwatch.Metric, err
/*
* Gather given Metric and emit any error
*/
func (c *CloudWatch) gatherMetric(
acc telegraf.Accumulator,
metric *cloudwatch.Metric,
now time.Time,
errChan chan error,
) {
func (c *CloudWatch) gatherMetric(acc telegraf.Accumulator, metric *cloudwatch.Metric, now time.Time, semaphore chan byte, errChan chan error) {
params := c.getStatisticsInput(metric, now)
resp, err := c.client.GetMetricStatistics(params)
if err != nil {
errChan <- err
<-semaphore
return
}
@@ -308,6 +258,7 @@ func (c *CloudWatch) gatherMetric(
}
errChan <- nil
<-semaphore
}
/*
@@ -358,32 +309,3 @@ func (c *CloudWatch) getStatisticsInput(metric *cloudwatch.Metric, now time.Time
func (c *MetricCache) IsValid() bool {
return c.Metrics != nil && time.Since(c.Fetched) < c.TTL
}
func hasWilcard(dimensions []*Dimension) bool {
for _, d := range dimensions {
if d.Value == "" || d.Value == "*" {
return true
}
}
return false
}
func isSelected(metric *cloudwatch.Metric, dimensions []*Dimension) bool {
if len(metric.Dimensions) != len(dimensions) {
return false
}
for _, d := range dimensions {
selected := false
for _, d2 := range metric.Dimensions {
if d.Name == *d2.Name {
if d.Value == "" || d.Value == "*" || d.Value == *d2.Value {
selected = true
}
}
}
if !selected {
return false
}
}
return true
}

View File

@@ -1,56 +0,0 @@
# Conntrack Plugin
Collects stats from Netfilter's conntrack-tools.
The conntrack-tools provide a mechanism for tracking various aspects of
network connections as they are processed by netfilter. At runtime,
conntrack exposes many of those connection statistics within /proc/sys/net.
Depending on your kernel version, these files can be found in either
/proc/sys/net/ipv4/netfilter or /proc/sys/net/netfilter and will be
prefixed with either ip_ or nf_. This plugin reads the files specified
in its configuration and publishes each one as a field, with the prefix
normalized to ip_.
In order to simplify configuration in a heterogeneous environment, a superset
of directory and filenames can be specified. Any locations that don't exist
will be ignored.
For more information on conntrack-tools, see the
[Netfilter Documentation](http://conntrack-tools.netfilter.org/).
### Configuration:
```toml
# Collects conntrack stats from the configured directories and files.
[[inputs.conntrack]]
## The following defaults would work with multiple versions of conntrack.
## Note the nf_ and ip_ filename prefixes are mutually exclusive across
## kernel versions, as are the directory locations.
## Superset of filenames to look for within the conntrack dirs.
## Missing files will be ignored.
files = ["ip_conntrack_count","ip_conntrack_max",
"nf_conntrack_count","nf_conntrack_max"]
## Directories to search within for the conntrack files above.
## Missing directrories will be ignored.
dirs = ["/proc/sys/net/ipv4/netfilter","/proc/sys/net/netfilter"]
```
### Measurements & Fields:
- conntrack
- ip_conntrack_count (int, count): the number of entries in the conntrack table
- ip_conntrack_max (int, size): the max capacity of the conntrack table
### Tags:
This input does not use tags.
### Example Output:
```
$ ./telegraf -config telegraf.conf -input-filter conntrack -test
conntrack,host=myhost ip_conntrack_count=2,ip_conntrack_max=262144 1461620427667995735
```

View File

@@ -1,119 +0,0 @@
// +build linux
package conntrack
import (
"fmt"
"io/ioutil"
"os"
"strconv"
"strings"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/inputs"
"log"
"path/filepath"
)
type Conntrack struct {
Path string
Dirs []string
Files []string
}
const (
inputName = "conntrack"
)
var dfltDirs = []string{
"/proc/sys/net/ipv4/netfilter",
"/proc/sys/net/netfilter",
}
var dfltFiles = []string{
"ip_conntrack_count",
"ip_conntrack_max",
"nf_conntrack_count",
"nf_conntrack_max",
}
func (c *Conntrack) setDefaults() {
if len(c.Dirs) == 0 {
c.Dirs = dfltDirs
}
if len(c.Files) == 0 {
c.Files = dfltFiles
}
}
func (c *Conntrack) Description() string {
return "Collects conntrack stats from the configured directories and files."
}
var sampleConfig = `
## The following defaults would work with multiple versions of conntrack.
## Note the nf_ and ip_ filename prefixes are mutually exclusive across
## kernel versions, as are the directory locations.
## Superset of filenames to look for within the conntrack dirs.
## Missing files will be ignored.
files = ["ip_conntrack_count","ip_conntrack_max",
"nf_conntrack_count","nf_conntrack_max"]
## Directories to search within for the conntrack files above.
## Missing directrories will be ignored.
dirs = ["/proc/sys/net/ipv4/netfilter","/proc/sys/net/netfilter"]
`
func (c *Conntrack) SampleConfig() string {
return sampleConfig
}
func (c *Conntrack) Gather(acc telegraf.Accumulator) error {
c.setDefaults()
var metricKey string
fields := make(map[string]interface{})
for _, dir := range c.Dirs {
for _, file := range c.Files {
// NOTE: no system will have both nf_ and ip_ prefixes,
// so we're safe to branch on suffix only.
parts := strings.SplitN(file, "_", 2)
if len(parts) < 2 {
continue
}
metricKey = "ip_" + parts[1]
fName := filepath.Join(dir, file)
if _, err := os.Stat(fName); err != nil {
continue
}
contents, err := ioutil.ReadFile(fName)
if err != nil {
log.Printf("failed to read file '%s': %v", fName, err)
}
v := strings.TrimSpace(string(contents))
fields[metricKey], err = strconv.ParseFloat(v, 64)
if err != nil {
log.Printf("failed to parse metric, expected number but "+
" found '%s': %v", v, err)
}
}
}
if len(fields) == 0 {
return fmt.Errorf("Conntrack input failed to collect metrics. " +
"Is the conntrack kernel module loaded?")
}
acc.AddFields(inputName, fields, nil)
return nil
}
func init() {
inputs.Add(inputName, func() telegraf.Input { return &Conntrack{} })
}

View File

@@ -1,3 +0,0 @@
// +build !linux
package conntrack

View File

@@ -1,90 +0,0 @@
// +build linux
package conntrack
import (
"github.com/influxdata/telegraf/testutil"
"github.com/stretchr/testify/assert"
"io/ioutil"
"os"
"path"
"strconv"
"strings"
"testing"
)
func restoreDflts(savedFiles, savedDirs []string) {
dfltFiles = savedFiles
dfltDirs = savedDirs
}
func TestNoFilesFound(t *testing.T) {
defer restoreDflts(dfltFiles, dfltDirs)
dfltFiles = []string{"baz.txt"}
dfltDirs = []string{"./foo/bar"}
c := &Conntrack{}
acc := &testutil.Accumulator{}
err := c.Gather(acc)
assert.EqualError(t, err, "Conntrack input failed to collect metrics. "+
"Is the conntrack kernel module loaded?")
}
func TestDefaultsUsed(t *testing.T) {
defer restoreDflts(dfltFiles, dfltDirs)
tmpdir, err := ioutil.TempDir("", "tmp1")
assert.NoError(t, err)
defer os.Remove(tmpdir)
tmpFile, err := ioutil.TempFile(tmpdir, "ip_conntrack_count")
assert.NoError(t, err)
dfltDirs = []string{tmpdir}
fname := path.Base(tmpFile.Name())
dfltFiles = []string{fname}
count := 1234321
ioutil.WriteFile(tmpFile.Name(), []byte(strconv.Itoa(count)), 0660)
c := &Conntrack{}
acc := &testutil.Accumulator{}
c.Gather(acc)
acc.AssertContainsFields(t, inputName, map[string]interface{}{
fname: float64(count)})
}
func TestConfigsUsed(t *testing.T) {
defer restoreDflts(dfltFiles, dfltDirs)
tmpdir, err := ioutil.TempDir("", "tmp1")
assert.NoError(t, err)
defer os.Remove(tmpdir)
cntFile, err := ioutil.TempFile(tmpdir, "nf_conntrack_count")
maxFile, err := ioutil.TempFile(tmpdir, "nf_conntrack_max")
assert.NoError(t, err)
dfltDirs = []string{tmpdir}
cntFname := path.Base(cntFile.Name())
maxFname := path.Base(maxFile.Name())
dfltFiles = []string{cntFname, maxFname}
count := 1234321
max := 9999999
ioutil.WriteFile(cntFile.Name(), []byte(strconv.Itoa(count)), 0660)
ioutil.WriteFile(maxFile.Name(), []byte(strconv.Itoa(max)), 0660)
c := &Conntrack{}
acc := &testutil.Accumulator{}
c.Gather(acc)
fix := func(s string) string {
return strings.Replace(s, "nf_", "ip_", 1)
}
acc.AssertContainsFields(t, inputName,
map[string]interface{}{
fix(cntFname): float64(count),
fix(maxFname): float64(max),
})
}

View File

@@ -1,46 +0,0 @@
# Telegraf Input Plugin: Consul
This plugin will collect statistics about all helath checks registered in the Consul. It uses [Consul API](https://www.consul.io/docs/agent/http/health.html#health_state)
to query the data. It will not report the [telemetry](https://www.consul.io/docs/agent/telemetry.html) but Consul can report those stats already using StatsD protocol if needed.
## Configuration:
```
# Gather health check statuses from services registered in Consul
[[inputs.consul]]
## Most of these values defaults to the one configured on a Consul's agent level.
## Optional Consul server address (default: "")
# address = ""
## Optional URI scheme for the Consul server (default: "")
# scheme = ""
## Optional ACL token used in every request (default: "")
# token = ""
## Optional username used for request HTTP Basic Authentication (default: "")
# username = ""
## Optional password used for HTTP Basic Authentication (default: "")
# password = ""
## Optional data centre to query the health checks from (default: "")
# datacentre = ""
```
## Measurements:
### Consul:
Tags:
- node: on which node check/service is registered on
- service_name: name of the service (this is the service name not the service ID)
Fields:
- check_id
- check_name
- service_id
- status
## Example output
```
$ telegraf --config ./telegraf.conf -input-filter consul -test
* Plugin: consul, Collection 1
> consul_health_checks,host=wolfpit,node=consul-server-node check_id="serfHealth",check_name="Serf Health Status",service_id="",status="passing" 1464698464486439902
> consul_health_checks,host=wolfpit,node=consul-server-node,service_name=www.example.com check_id="service:www-example-com.test01",check_name="Service 'www.example.com' check",service_id="www-example-com.test01",status="critical" 1464698464486519036
```

View File

@@ -1,136 +0,0 @@
package consul
import (
"net/http"
"github.com/hashicorp/consul/api"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/plugins/inputs"
)
type Consul struct {
Address string
Scheme string
Token string
Username string
Password string
Datacentre string
// Path to CA file
SSLCA string `toml:"ssl_ca"`
// Path to host cert file
SSLCert string `toml:"ssl_cert"`
// Path to cert key file
SSLKey string `toml:"ssl_key"`
// Use SSL but skip chain & host verification
InsecureSkipVerify bool
// client used to connect to Consul agnet
client *api.Client
}
var sampleConfig = `
## Most of these values defaults to the one configured on a Consul's agent level.
## Optional Consul server address (default: "localhost")
# address = "localhost"
## Optional URI scheme for the Consul server (default: "http")
# scheme = "http"
## Optional ACL token used in every request (default: "")
# token = ""
## Optional username used for request HTTP Basic Authentication (default: "")
# username = ""
## Optional password used for HTTP Basic Authentication (default: "")
# password = ""
## Optional data centre to query the health checks from (default: "")
# datacentre = ""
`
func (c *Consul) Description() string {
return "Gather health check statuses from services registered in Consul"
}
func (c *Consul) SampleConfig() string {
return sampleConfig
}
func (c *Consul) createAPIClient() (*api.Client, error) {
config := api.DefaultConfig()
if c.Address != "" {
config.Address = c.Address
}
if c.Scheme != "" {
config.Scheme = c.Scheme
}
if c.Datacentre != "" {
config.Datacenter = c.Datacentre
}
if c.Username != "" {
config.HttpAuth = &api.HttpBasicAuth{
Username: c.Username,
Password: c.Password,
}
}
tlsCfg, err := internal.GetTLSConfig(
c.SSLCert, c.SSLKey, c.SSLCA, c.InsecureSkipVerify)
if err != nil {
return nil, err
}
config.HttpClient.Transport = &http.Transport{
TLSClientConfig: tlsCfg,
}
return api.NewClient(config)
}
func (c *Consul) GatherHealthCheck(acc telegraf.Accumulator, checks []*api.HealthCheck) {
for _, check := range checks {
record := make(map[string]interface{})
tags := make(map[string]string)
record["check_id"] = check.CheckID
record["check_name"] = check.Name
record["service_id"] = check.ServiceID
record["status"] = check.Status
tags["node"] = check.Node
tags["service_name"] = check.ServiceName
acc.AddFields("consul_health_checks", record, tags)
}
}
func (c *Consul) Gather(acc telegraf.Accumulator) error {
if c.client == nil {
newClient, err := c.createAPIClient()
if err != nil {
return err
}
c.client = newClient
}
checks, _, err := c.client.Health().State("any", nil)
if err != nil {
return err
}
c.GatherHealthCheck(acc, checks)
return nil
}
func init() {
inputs.Add("consul", func() telegraf.Input {
return &Consul{}
})
}

View File

@@ -1,42 +0,0 @@
package consul
import (
"testing"
"github.com/hashicorp/consul/api"
"github.com/influxdata/telegraf/testutil"
)
var sampleChecks = []*api.HealthCheck{
&api.HealthCheck{
Node: "localhost",
CheckID: "foo.health123",
Name: "foo.health",
Status: "passing",
Notes: "lorem ipsum",
Output: "OK",
ServiceID: "foo.123",
ServiceName: "foo",
},
}
func TestGatherHealtCheck(t *testing.T) {
expectedFields := map[string]interface{}{
"check_id": "foo.health123",
"check_name": "foo.health",
"status": "passing",
"service_id": "foo.123",
}
expectedTags := map[string]string{
"node": "localhost",
"service_name": "foo",
}
var acc testutil.Accumulator
consul := &Consul{}
consul.GatherHealthCheck(&acc, sampleChecks)
acc.AssertContainsTaggedFields(t, "consul_health_checks", expectedFields, expectedTags)
}

View File

@@ -2,13 +2,14 @@ package elasticsearch
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal/errchan"
"github.com/influxdata/telegraf/plugins/inputs"
jsonparser "github.com/influxdata/telegraf/plugins/parsers/json"
)
@@ -101,7 +102,7 @@ func (e *Elasticsearch) Description() string {
// Gather reads the stats from Elasticsearch and writes it to the
// Accumulator.
func (e *Elasticsearch) Gather(acc telegraf.Accumulator) error {
errChan := errchan.New(len(e.Servers))
errChan := make(chan error, len(e.Servers))
var wg sync.WaitGroup
wg.Add(len(e.Servers))
@@ -115,7 +116,7 @@ func (e *Elasticsearch) Gather(acc telegraf.Accumulator) error {
url = s + statsPath
}
if err := e.gatherNodeStats(url, acc); err != nil {
errChan.C <- err
errChan <- err
return
}
if e.ClusterHealth {
@@ -125,7 +126,17 @@ func (e *Elasticsearch) Gather(acc telegraf.Accumulator) error {
}
wg.Wait()
return errChan.Error()
close(errChan)
// Get all errors and return them as one giant error
errStrings := []string{}
for err := range errChan {
errStrings = append(errStrings, err.Error())
}
if len(errStrings) == 0 {
return nil
}
return errors.New(strings.Join(errStrings, "\n"))
}
func (e *Elasticsearch) gatherNodeStats(url string, acc telegraf.Accumulator) error {

View File

@@ -6,20 +6,14 @@ Please also see: [Telegraf Input Data Formats](https://github.com/influxdata/tel
#### Configuration
In this example a script called ```/tmp/test.sh```, a script called ```/tmp/test2.sh```, and
all scripts matching glob pattern ```/tmp/collect_*.sh``` are configured for ```[[inputs.exec]]```
in JSON format. Glob patterns are matched on every run, so adding new scripts that match the pattern
will cause them to be picked up immediately.
In this example a script called ```/tmp/test.sh``` and a script called ```/tmp/test2.sh```
are configured for ```[[inputs.exec]]``` in JSON format.
```toml
```
# Read flattened metrics from one or more commands that output JSON to stdout
[[inputs.exec]]
# Shell/commands array
# Full command line to executable with parameters, or a glob pattern to run all matching files.
commands = ["/tmp/test.sh", "/tmp/test2.sh", "/tmp/collect_*.sh"]
## Timeout for each command to complete.
timeout = "5s"
commands = ["/tmp/test.sh", "/tmp/test2.sh"]
# Data format to consume.
# NOTE json only reads numerical measurements, strings and booleans are ignored.
@@ -27,6 +21,26 @@ will cause them to be picked up immediately.
# measurement name suffix (for separating different commands)
name_suffix = "_mycollector"
## Below configuration will be used for data_format = "graphite", can be ignored for other data_format
## If matching multiple measurement files, this string will be used to join the matched values.
#separator = "."
## Each template line requires a template pattern. It can have an optional
## filter before the template and separated by spaces. It can also have optional extra
## tags following the template. Multiple tags should be separated by commas and no spaces
## similar to the line protocol format. The can be only one default template.
## Templates support below format:
## 1. filter + template
## 2. filter + template + extra tag
## 3. filter + template with field key
## 4. default template
#templates = [
# "*.app env.service.resource.measurement",
# "stats.* .host.measurement* region=us-west,agent=sensu",
# "stats2.* .host.measurement.field",
# "measurement*"
#]
```
Other options for modifying the measurement names are:
@@ -65,7 +79,7 @@ in influx line-protocol format.
#### Configuration
```toml
```
[[inputs.exec]]
# Shell/commands array
# compatible with old version
@@ -73,9 +87,6 @@ in influx line-protocol format.
# command = "/usr/bin/line_protocol_collector"
commands = ["/usr/bin/line_protocol_collector","/tmp/test2.sh"]
## Timeout for each command to complete.
timeout = "5s"
# Data format to consume.
# NOTE json only reads numerical measurements, strings and booleans are ignored.
data_format = "influx"
@@ -109,16 +120,12 @@ We can also change the data_format to "graphite" to use the metrics collecting s
In this example a script called /tmp/test.sh and a script called /tmp/test2.sh are configured for [[inputs.exec]] in graphite format.
#### Configuration
```toml
```
# Read flattened metrics from one or more commands that output JSON to stdout
[[inputs.exec]]
# Shell/commands array
commands = ["/tmp/test.sh","/tmp/test2.sh"]
## Timeout for each command to complete.
timeout = "5s"
# Data format to consume.
# NOTE json only reads numerical measurements, strings and booleans are ignored.
data_format = "graphite"
@@ -173,3 +180,4 @@ sensu.metric.net.server0.eth0.rx_dropped 0 1444234982
The templates configuration will be used to parse the graphite metrics to support influxdb/opentsdb tagging store engines.
More detail information about templates, please refer to [The graphite Input](https://github.com/influxdata/influxdb/blob/master/services/graphite/README.md)

View File

@@ -4,8 +4,6 @@ import (
"bytes"
"fmt"
"os/exec"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
@@ -14,7 +12,6 @@ import (
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/internal/errchan"
"github.com/influxdata/telegraf/plugins/inputs"
"github.com/influxdata/telegraf/plugins/parsers"
"github.com/influxdata/telegraf/plugins/parsers/nagios"
@@ -22,11 +19,7 @@ import (
const sampleConfig = `
## Commands array
commands = [
"/tmp/test.sh",
"/usr/bin/mycollector --foo=bar",
"/tmp/collect_*.sh"
]
commands = ["/tmp/test.sh", "/usr/bin/mycollector --foo=bar"]
## Timeout for each command to complete.
timeout = "5s"
@@ -157,45 +150,23 @@ func (e *Exec) Gather(acc telegraf.Accumulator) error {
e.Command = ""
}
commands := make([]string, 0, len(e.Commands))
for _, pattern := range e.Commands {
cmdAndArgs := strings.SplitN(pattern, " ", 2)
if len(cmdAndArgs) == 0 {
continue
}
e.errChan = make(chan error, len(e.Commands))
matches, err := filepath.Glob(cmdAndArgs[0])
if err != nil {
return err
}
if len(matches) == 0 {
// There were no matches with the glob pattern, so let's assume
// that the command is in PATH and just run it as it is
commands = append(commands, pattern)
} else {
// There were matches, so we'll append each match together with
// the arguments to the commands slice
for _, match := range matches {
if len(cmdAndArgs) == 1 {
commands = append(commands, match)
} else {
commands = append(commands,
strings.Join([]string{match, cmdAndArgs[1]}, " "))
}
}
}
}
errChan := errchan.New(len(commands))
e.errChan = errChan.C
e.wg.Add(len(commands))
for _, command := range commands {
e.wg.Add(len(e.Commands))
for _, command := range e.Commands {
go e.ProcessCommand(command, acc)
}
e.wg.Wait()
return errChan.Error()
select {
default:
close(e.errChan)
return nil
case err := <-e.errChan:
close(e.errChan)
return err
}
}
func init() {

View File

@@ -169,51 +169,3 @@ func TestLineProtocolParseMultiple(t *testing.T) {
acc.AssertContainsTaggedFields(t, "cpu", fields, tags)
}
}
func TestExecCommandWithGlob(t *testing.T) {
parser, _ := parsers.NewValueParser("metric", "string", nil)
e := NewExec()
e.Commands = []string{"/bin/ech* metric_value"}
e.SetParser(parser)
var acc testutil.Accumulator
err := e.Gather(&acc)
require.NoError(t, err)
fields := map[string]interface{}{
"value": "metric_value",
}
acc.AssertContainsFields(t, "metric", fields)
}
func TestExecCommandWithoutGlob(t *testing.T) {
parser, _ := parsers.NewValueParser("metric", "string", nil)
e := NewExec()
e.Commands = []string{"/bin/echo metric_value"}
e.SetParser(parser)
var acc testutil.Accumulator
err := e.Gather(&acc)
require.NoError(t, err)
fields := map[string]interface{}{
"value": "metric_value",
}
acc.AssertContainsFields(t, "metric", fields)
}
func TestExecCommandWithoutGlobAndPath(t *testing.T) {
parser, _ := parsers.NewValueParser("metric", "string", nil)
e := NewExec()
e.Commands = []string{"echo metric_value"}
e.SetParser(parser)
var acc testutil.Accumulator
err := e.Gather(&acc)
require.NoError(t, err)
fields := map[string]interface{}{
"value": "metric_value",
}
acc.AssertContainsFields(t, "metric", fields)
}

View File

@@ -1,55 +0,0 @@
# GrayLog plugin
The Graylog plugin can collect data from remote Graylog service URLs.
Plugin currently support two type of end points:-
- multiple (Ex http://[graylog-server-ip]:12900/system/metrics/multiple)
- namespace (Ex http://[graylog-server-ip]:12900/system/metrics/namespace/{namespace})
End Point can be a mixe of one multiple end point and several namespaces end points
Note: if namespace end point specified metrics array will be ignored for that call.
### Configuration:
```toml
# Read flattened metrics from one or more GrayLog HTTP endpoints
[[inputs.graylog]]
## API endpoint, currently supported API:
##
## - multiple (Ex http://<host>:12900/system/metrics/multiple)
## - namespace (Ex http://<host>:12900/system/metrics/namespace/{namespace})
##
## For namespace endpoint, the metrics array will be ignored for that call.
## Endpoint can contain namespace and multiple type calls.
##
## Please check http://[graylog-server-ip]:12900/api-browser for full list
## of endpoints
servers = [
"http://[graylog-server-ip]:12900/system/metrics/multiple",
]
## Metrics list
## List of metrics can be found on Graylog webservice documentation.
## Or by hitting the the web service api at:
## http://[graylog-host]:12900/system/metrics
metrics = [
"jvm.cl.loaded",
"jvm.memory.pools.Metaspace.committed"
]
## Username and password
username = ""
password = ""
## Optional SSL Config
# ssl_ca = "/etc/telegraf/ca.pem"
# ssl_cert = "/etc/telegraf/cert.pem"
# ssl_key = "/etc/telegraf/key.pem"
## Use SSL but skip chain & host verification
# insecure_skip_verify = false
```
Please refer to GrayLog metrics api browser for full metric end points http://host:12900/api-browser

View File

@@ -1,312 +0,0 @@
package graylog
import (
"bytes"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net"
"net/http"
"net/url"
"strings"
"sync"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/plugins/inputs"
)
type ResponseMetrics struct {
total int
Metrics []Metric `json:"metrics"`
}
type Metric struct {
FullName string `json:"full_name"`
Name string `json:"name"`
Type string `json:"type"`
Fields map[string]interface{} `json:"metric"`
}
type GrayLog struct {
Servers []string
Metrics []string
Username string
Password string
// Path to CA file
SSLCA string `toml:"ssl_ca"`
// Path to host cert file
SSLCert string `toml:"ssl_cert"`
// Path to cert key file
SSLKey string `toml:"ssl_key"`
// Use SSL but skip chain & host verification
InsecureSkipVerify bool
client HTTPClient
}
type HTTPClient interface {
// Returns the result of an http request
//
// Parameters:
// req: HTTP request object
//
// Returns:
// http.Response: HTTP respons object
// error : Any error that may have occurred
MakeRequest(req *http.Request) (*http.Response, error)
SetHTTPClient(client *http.Client)
HTTPClient() *http.Client
}
type Messagebody struct {
Metrics []string `json:"metrics"`
}
type RealHTTPClient struct {
client *http.Client
}
func (c *RealHTTPClient) MakeRequest(req *http.Request) (*http.Response, error) {
return c.client.Do(req)
}
func (c *RealHTTPClient) SetHTTPClient(client *http.Client) {
c.client = client
}
func (c *RealHTTPClient) HTTPClient() *http.Client {
return c.client
}
var sampleConfig = `
## API endpoint, currently supported API:
##
## - multiple (Ex http://<host>:12900/system/metrics/multiple)
## - namespace (Ex http://<host>:12900/system/metrics/namespace/{namespace})
##
## For namespace endpoint, the metrics array will be ignored for that call.
## Endpoint can contain namespace and multiple type calls.
##
## Please check http://[graylog-server-ip]:12900/api-browser for full list
## of endpoints
servers = [
"http://[graylog-server-ip]:12900/system/metrics/multiple",
]
## Metrics list
## List of metrics can be found on Graylog webservice documentation.
## Or by hitting the the web service api at:
## http://[graylog-host]:12900/system/metrics
metrics = [
"jvm.cl.loaded",
"jvm.memory.pools.Metaspace.committed"
]
## Username and password
username = ""
password = ""
## Optional SSL Config
# ssl_ca = "/etc/telegraf/ca.pem"
# ssl_cert = "/etc/telegraf/cert.pem"
# ssl_key = "/etc/telegraf/key.pem"
## Use SSL but skip chain & host verification
# insecure_skip_verify = false
`
func (h *GrayLog) SampleConfig() string {
return sampleConfig
}
func (h *GrayLog) Description() string {
return "Read flattened metrics from one or more GrayLog HTTP endpoints"
}
// Gathers data for all servers.
func (h *GrayLog) Gather(acc telegraf.Accumulator) error {
var wg sync.WaitGroup
if h.client.HTTPClient() == nil {
tlsCfg, err := internal.GetTLSConfig(
h.SSLCert, h.SSLKey, h.SSLCA, h.InsecureSkipVerify)
if err != nil {
return err
}
tr := &http.Transport{
ResponseHeaderTimeout: time.Duration(3 * time.Second),
TLSClientConfig: tlsCfg,
}
client := &http.Client{
Transport: tr,
Timeout: time.Duration(4 * time.Second),
}
h.client.SetHTTPClient(client)
}
errorChannel := make(chan error, len(h.Servers))
for _, server := range h.Servers {
wg.Add(1)
go func(server string) {
defer wg.Done()
if err := h.gatherServer(acc, server); err != nil {
errorChannel <- err
}
}(server)
}
wg.Wait()
close(errorChannel)
// Get all errors and return them as one giant error
errorStrings := []string{}
for err := range errorChannel {
errorStrings = append(errorStrings, err.Error())
}
if len(errorStrings) == 0 {
return nil
}
return errors.New(strings.Join(errorStrings, "\n"))
}
// Gathers data from a particular server
// Parameters:
// acc : The telegraf Accumulator to use
// serverURL: endpoint to send request to
// service : the service being queried
//
// Returns:
// error: Any error that may have occurred
func (h *GrayLog) gatherServer(
acc telegraf.Accumulator,
serverURL string,
) error {
resp, _, err := h.sendRequest(serverURL)
if err != nil {
return err
}
requestURL, err := url.Parse(serverURL)
host, port, _ := net.SplitHostPort(requestURL.Host)
var dat ResponseMetrics
if err != nil {
return err
}
if err := json.Unmarshal([]byte(resp), &dat); err != nil {
return err
}
for _, m_item := range dat.Metrics {
fields := make(map[string]interface{})
tags := map[string]string{
"server": host,
"port": port,
"name": m_item.Name,
"type": m_item.Type,
}
h.flatten(m_item.Fields, fields, "")
acc.AddFields(m_item.FullName, fields, tags)
}
return nil
}
// Flatten JSON hierarchy to produce field name and field value
// Parameters:
// item: Item map to flatten
// fields: Map to store generated fields.
// id: Prefix for top level metric (empty string "")
// Returns:
// void
func (h *GrayLog) flatten(item map[string]interface{}, fields map[string]interface{}, id string) {
if id != "" {
id = id + "_"
}
for k, i := range item {
switch i.(type) {
case int:
fields[id+k] = i.(float64)
case float64:
fields[id+k] = i.(float64)
case map[string]interface{}:
h.flatten(i.(map[string]interface{}), fields, id+k)
default:
}
}
}
// Sends an HTTP request to the server using the GrayLog object's HTTPClient.
// Parameters:
// serverURL: endpoint to send request to
//
// Returns:
// string: body of the response
// error : Any error that may have occurred
func (h *GrayLog) sendRequest(serverURL string) (string, float64, error) {
headers := map[string]string{
"Content-Type": "application/json",
"Accept": "application/json",
}
method := "GET"
content := bytes.NewBufferString("")
headers["Authorization"] = "Basic " + base64.URLEncoding.EncodeToString([]byte(h.Username+":"+h.Password))
// Prepare URL
requestURL, err := url.Parse(serverURL)
if err != nil {
return "", -1, fmt.Errorf("Invalid server URL \"%s\"", serverURL)
}
if strings.Contains(requestURL.String(), "multiple") {
m := &Messagebody{Metrics: h.Metrics}
http_body, err := json.Marshal(m)
if err != nil {
return "", -1, fmt.Errorf("Invalid list of Metrics %s", h.Metrics)
}
method = "POST"
content = bytes.NewBuffer(http_body)
}
req, err := http.NewRequest(method, requestURL.String(), content)
if err != nil {
return "", -1, err
}
// Add header parameters
for k, v := range headers {
req.Header.Add(k, v)
}
start := time.Now()
resp, err := h.client.MakeRequest(req)
if err != nil {
return "", -1, err
}
defer resp.Body.Close()
responseTime := time.Since(start).Seconds()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return string(body), responseTime, err
}
// Process response
if resp.StatusCode != http.StatusOK {
err = fmt.Errorf("Response from url \"%s\" has status code %d (%s), expected %d (%s)",
requestURL.String(),
resp.StatusCode,
http.StatusText(resp.StatusCode),
http.StatusOK,
http.StatusText(http.StatusOK))
return string(body), responseTime, err
}
return string(body), responseTime, err
}
func init() {
inputs.Add("graylog", func() telegraf.Input {
return &GrayLog{
client: &RealHTTPClient{},
}
})
}

View File

@@ -1,199 +0,0 @@
package graylog
import (
"io/ioutil"
"net/http"
"strings"
"testing"
"github.com/influxdata/telegraf/testutil"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const validJSON = `
{
"total": 3,
"metrics": [
{
"full_name": "jvm.cl.loaded",
"metric": {
"value": 18910
},
"name": "loaded",
"type": "gauge"
},
{
"full_name": "jvm.memory.pools.Metaspace.committed",
"metric": {
"value": 108040192
},
"name": "committed",
"type": "gauge"
},
{
"full_name": "org.graylog2.shared.journal.KafkaJournal.writeTime",
"metric": {
"time": {
"min": 99
},
"rate": {
"total": 10,
"mean": 2
},
"duration_unit": "microseconds",
"rate_unit": "events/second"
},
"name": "writeTime",
"type": "hdrtimer"
}
]
}`
var validTags = map[string]map[string]string{
"jvm.cl.loaded": {
"name": "loaded",
"type": "gauge",
"port": "12900",
"server": "localhost",
},
"jvm.memory.pools.Metaspace.committed": {
"name": "committed",
"type": "gauge",
"port": "12900",
"server": "localhost",
},
"org.graylog2.shared.journal.KafkaJournal.writeTime": {
"name": "writeTime",
"type": "hdrtimer",
"port": "12900",
"server": "localhost",
},
}
var expectedFields = map[string]map[string]interface{}{
"jvm.cl.loaded": {
"value": float64(18910),
},
"jvm.memory.pools.Metaspace.committed": {
"value": float64(108040192),
},
"org.graylog2.shared.journal.KafkaJournal.writeTime": {
"time_min": float64(99),
"rate_total": float64(10),
"rate_mean": float64(2),
},
}
const invalidJSON = "I don't think this is JSON"
const empty = ""
type mockHTTPClient struct {
responseBody string
statusCode int
}
// Mock implementation of MakeRequest. Usually returns an http.Response with
// hard-coded responseBody and statusCode. However, if the request uses a
// nonstandard method, it uses status code 405 (method not allowed)
func (c *mockHTTPClient) MakeRequest(req *http.Request) (*http.Response, error) {
resp := http.Response{}
resp.StatusCode = c.statusCode
// basic error checking on request method
allowedMethods := []string{"GET", "HEAD", "POST", "PUT", "DELETE", "TRACE", "CONNECT"}
methodValid := false
for _, method := range allowedMethods {
if req.Method == method {
methodValid = true
break
}
}
if !methodValid {
resp.StatusCode = 405 // Method not allowed
}
resp.Body = ioutil.NopCloser(strings.NewReader(c.responseBody))
return &resp, nil
}
func (c *mockHTTPClient) SetHTTPClient(_ *http.Client) {
}
func (c *mockHTTPClient) HTTPClient() *http.Client {
return nil
}
// Generates a pointer to an HttpJson object that uses a mock HTTP client.
// Parameters:
// response : Body of the response that the mock HTTP client should return
// statusCode: HTTP status code the mock HTTP client should return
//
// Returns:
// *HttpJson: Pointer to an HttpJson object that uses the generated mock HTTP client
func genMockGrayLog(response string, statusCode int) []*GrayLog {
return []*GrayLog{
&GrayLog{
client: &mockHTTPClient{responseBody: response, statusCode: statusCode},
Servers: []string{
"http://localhost:12900/system/metrics/multiple",
},
Metrics: []string{
"jvm.memory.pools.Metaspace.committed",
"jvm.cl.loaded",
"org.graylog2.shared.journal.KafkaJournal.writeTime",
},
Username: "test",
Password: "test",
},
}
}
// Test that the proper values are ignored or collected
func TestNormalResponse(t *testing.T) {
graylog := genMockGrayLog(validJSON, 200)
for _, service := range graylog {
var acc testutil.Accumulator
err := service.Gather(&acc)
require.NoError(t, err)
for k, v := range expectedFields {
acc.AssertContainsTaggedFields(t, k, v, validTags[k])
}
}
}
// Test response to HTTP 500
func TestHttpJson500(t *testing.T) {
graylog := genMockGrayLog(validJSON, 500)
var acc testutil.Accumulator
err := graylog[0].Gather(&acc)
assert.NotNil(t, err)
assert.Equal(t, 0, acc.NFields())
}
// Test response to malformed JSON
func TestHttpJsonBadJson(t *testing.T) {
graylog := genMockGrayLog(invalidJSON, 200)
var acc testutil.Accumulator
err := graylog[0].Gather(&acc)
assert.NotNil(t, err)
assert.Equal(t, 0, acc.NFields())
}
// Test response to empty string as response objectgT
func TestHttpJsonEmptyResponse(t *testing.T) {
graylog := genMockGrayLog(empty, 200)
var acc testutil.Accumulator
err := graylog[0].Gather(&acc)
assert.NotNil(t, err)
assert.Equal(t, 0, acc.NFields())
}

View File

@@ -3,6 +3,8 @@ package haproxy
import (
"encoding/csv"
"fmt"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/inputs"
"io"
"net"
"net/http"
@@ -11,10 +13,6 @@ import (
"strings"
"sync"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal/errchan"
"github.com/influxdata/telegraf/plugins/inputs"
)
//CSV format: https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.1
@@ -115,17 +113,20 @@ func (g *haproxy) Gather(acc telegraf.Accumulator) error {
}
var wg sync.WaitGroup
errChan := errchan.New(len(g.Servers))
wg.Add(len(g.Servers))
for _, server := range g.Servers {
var outerr error
for _, serv := range g.Servers {
wg.Add(1)
go func(serv string) {
defer wg.Done()
errChan.C <- g.gatherServer(serv, acc)
}(server)
outerr = g.gatherServer(serv, acc)
}(serv)
}
wg.Wait()
return errChan.Error()
return outerr
}
func (g *haproxy) gatherServerSocket(addr string, acc telegraf.Accumulator) error {

View File

@@ -22,13 +22,6 @@ This input plugin will test HTTP/HTTPS connections.
# body = '''
# {'fake':'data'}
# '''
## Optional SSL Config
# ssl_ca = "/etc/telegraf/ca.pem"
# ssl_cert = "/etc/telegraf/cert.pem"
# ssl_key = "/etc/telegraf/key.pem"
## Use SSL but skip chain & host verification
# insecure_skip_verify = false
```
### Measurements & Fields:

View File

@@ -21,15 +21,6 @@ type HTTPResponse struct {
ResponseTimeout internal.Duration
Headers map[string]string
FollowRedirects bool
// Path to CA file
SSLCA string `toml:"ssl_ca"`
// Path to host cert file
SSLCert string `toml:"ssl_cert"`
// Path to cert key file
SSLKey string `toml:"ssl_key"`
// Use SSL but skip chain & host verification
InsecureSkipVerify bool
}
// Description returns the plugin Description
@@ -53,13 +44,6 @@ var sampleConfig = `
# body = '''
# {'fake':'data'}
# '''
## Optional SSL Config
# ssl_ca = "/etc/telegraf/ca.pem"
# ssl_cert = "/etc/telegraf/cert.pem"
# ssl_key = "/etc/telegraf/key.pem"
## Use SSL but skip chain & host verification
# insecure_skip_verify = false
`
// SampleConfig returns the plugin SampleConfig
@@ -72,27 +56,17 @@ var ErrRedirectAttempted = errors.New("redirect")
// CreateHttpClient creates an http client which will timeout at the specified
// timeout period and can follow redirects if specified
func (h *HTTPResponse) createHttpClient() (*http.Client, error) {
tlsCfg, err := internal.GetTLSConfig(
h.SSLCert, h.SSLKey, h.SSLCA, h.InsecureSkipVerify)
if err != nil {
return nil, err
}
tr := &http.Transport{
ResponseHeaderTimeout: h.ResponseTimeout.Duration,
TLSClientConfig: tlsCfg,
}
func CreateHttpClient(followRedirects bool, ResponseTimeout time.Duration) *http.Client {
client := &http.Client{
Transport: tr,
Timeout: h.ResponseTimeout.Duration,
Timeout: ResponseTimeout,
}
if h.FollowRedirects == false {
if followRedirects == false {
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return ErrRedirectAttempted
}
}
return client, nil
return client
}
// HTTPGather gathers all fields and returns any errors it encounters
@@ -100,10 +74,7 @@ func (h *HTTPResponse) HTTPGather() (map[string]interface{}, error) {
// Prepare fields
fields := make(map[string]interface{})
client, err := h.createHttpClient()
if err != nil {
return nil, err
}
client := CreateHttpClient(h.FollowRedirects, h.ResponseTimeout.Duration)
var body io.Reader
if h.Body != "" {

View File

@@ -15,7 +15,6 @@ InfluxDB-formatted endpoints. See below for more information.
## See the influxdb plugin's README for more details.
## Multiple URLs from which to read InfluxDB-formatted JSON
## Default is "http://localhost:8086/debug/vars".
urls = [
"http://localhost:8086/debug/vars"
]

View File

@@ -28,7 +28,6 @@ func (*InfluxDB) SampleConfig() string {
## See the influxdb plugin's README for more details.
## Multiple URLs from which to read InfluxDB-formatted JSON
## Default is "http://localhost:8086/debug/vars".
urls = [
"http://localhost:8086/debug/vars"
]
@@ -36,9 +35,6 @@ func (*InfluxDB) SampleConfig() string {
}
func (i *InfluxDB) Gather(acc telegraf.Accumulator) error {
if len(i.URLs) == 0 {
i.URLs = []string{"http://localhost:8086/debug/vars"}
}
errorChannel := make(chan error, len(i.URLs))
var wg sync.WaitGroup
@@ -161,45 +157,43 @@ func (i *InfluxDB) gatherURL(
return err
}
if keyStr, ok := key.(string); ok {
if keyStr == "memstats" {
var m memstats
if err := dec.Decode(&m); err != nil {
continue
}
acc.AddFields("influxdb_memstats",
map[string]interface{}{
"alloc": m.Alloc,
"total_alloc": m.TotalAlloc,
"sys": m.Sys,
"lookups": m.Lookups,
"mallocs": m.Mallocs,
"frees": m.Frees,
"heap_alloc": m.HeapAlloc,
"heap_sys": m.HeapSys,
"heap_idle": m.HeapIdle,
"heap_inuse": m.HeapInuse,
"heap_released": m.HeapReleased,
"heap_objects": m.HeapObjects,
"stack_inuse": m.StackInuse,
"stack_sys": m.StackSys,
"mspan_inuse": m.MSpanInuse,
"mspan_sys": m.MSpanSys,
"mcache_inuse": m.MCacheInuse,
"mcache_sys": m.MCacheSys,
"buck_hash_sys": m.BuckHashSys,
"gc_sys": m.GCSys,
"other_sys": m.OtherSys,
"next_gc": m.NextGC,
"last_gc": m.LastGC,
"pause_total_ns": m.PauseTotalNs,
"num_gc": m.NumGC,
"gcc_pu_fraction": m.GCCPUFraction,
},
map[string]string{
"url": url,
})
if key.(string) == "memstats" {
var m memstats
if err := dec.Decode(&m); err != nil {
continue
}
acc.AddFields("influxdb_memstats",
map[string]interface{}{
"alloc": m.Alloc,
"total_alloc": m.TotalAlloc,
"sys": m.Sys,
"lookups": m.Lookups,
"mallocs": m.Mallocs,
"frees": m.Frees,
"heap_alloc": m.HeapAlloc,
"heap_sys": m.HeapSys,
"heap_idle": m.HeapIdle,
"heap_inuse": m.HeapInuse,
"heap_released": m.HeapReleased,
"heap_objects": m.HeapObjects,
"stack_inuse": m.StackInuse,
"stack_sys": m.StackSys,
"mspan_inuse": m.MSpanInuse,
"mspan_sys": m.MSpanSys,
"mcache_inuse": m.MCacheInuse,
"mcache_sys": m.MCacheSys,
"buck_hash_sys": m.BuckHashSys,
"gc_sys": m.GCSys,
"other_sys": m.OtherSys,
"next_gc": m.NextGC,
"last_gc": m.LastGC,
"pause_total_ns": m.PauseTotalNs,
"num_gc": m.NumGC,
"gcc_pu_fraction": m.GCCPUFraction,
},
map[string]string{
"url": url,
})
}
// Attempt to parse a whole object into a point.
@@ -210,16 +204,16 @@ func (i *InfluxDB) gatherURL(
continue
}
if p.Name == "shard" {
shardCounter++
}
// If the object was a point, but was not fully initialized,
// ignore it and move on.
if p.Name == "" || p.Tags == nil || p.Values == nil || len(p.Values) == 0 {
continue
}
if p.Name == "shard" {
shardCounter++
}
// Add a tag to indicate the source of the data.
p.Tags["url"] = url

View File

@@ -112,7 +112,7 @@ func TestInfluxDB(t *testing.T) {
acc.AssertContainsTaggedFields(t, "influxdb",
map[string]interface{}{
"n_shards": 1,
"n_shards": 2,
}, map[string]string{})
}

View File

@@ -50,7 +50,7 @@ var sampleConfig = `
## an array of Zookeeper connection strings
zookeeper_peers = ["localhost:2181"]
## Zookeeper Chroot
zookeeper_chroot = ""
zookeeper_chroot = "/"
## the name of the consumer group
consumer_group = "telegraf_metrics_consumers"
## Offset (must be either "oldest" or "newest")

View File

@@ -1,89 +0,0 @@
# logparser Input Plugin
The logparser plugin streams and parses the given logfiles. Currently it only
has the capability of parsing "grok" patterns from logfiles, which also supports
regex patterns.
### Configuration:
```toml
[[inputs.logparser]]
## Log files to parse.
## These accept standard unix glob matching rules, but with the addition of
## ** as a "super asterisk". ie:
## /var/log/**.log -> recursively find all .log files in /var/log
## /var/log/*/*.log -> find all .log files with a parent dir in /var/log
## /var/log/apache.log -> only tail the apache log file
files = ["/var/log/influxdb/influxdb.log"]
## Read file from beginning.
from_beginning = false
## Parse logstash-style "grok" patterns:
## Telegraf builtin parsing patterns: https://goo.gl/dkay10
[inputs.logparser.grok]
## This is a list of patterns to check the given log file(s) for.
## Note that adding patterns here increases processing time. The most
## efficient configuration is to have one file & pattern per logparser.
patterns = ["%{INFLUXDB_HTTPD_LOG}"]
## Full path(s) to custom pattern files.
custom_pattern_files = []
## Custom patterns can also be defined here. Put one pattern per line.
custom_patterns = '''
'''
```
## Grok Parser
The grok parser uses a slightly modified version of logstash "grok" patterns,
with the format `%{<capture_syntax>[:<semantic_name>][:<modifier>]}`
Telegraf has many of it's own
[built-in patterns](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/logparser/grok/patterns/influx-patterns),
as well as supporting
[logstash's builtin patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns).
The best way to get acquainted with grok patterns is to read the logstash docs,
which are available here:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
If you need help building patterns to match your logs,
you will find the http://grokdebug.herokuapp.com application quite useful!
By default all named captures are converted into string fields.
Modifiers can be used to convert captures to other types or tags.
Timestamp modifiers can be used to convert captures to the timestamp of the
parsed metric.
- Available modifiers:
- string (default if nothing is specified)
- int
- float
- duration (ie, 5.23ms gets converted to int nanoseconds)
- tag (converts the field into a tag)
- drop (drops the field completely)
- Timestamp modifiers:
- ts-ansic ("Mon Jan _2 15:04:05 2006")
- ts-unix ("Mon Jan _2 15:04:05 MST 2006")
- ts-ruby ("Mon Jan 02 15:04:05 -0700 2006")
- ts-rfc822 ("02 Jan 06 15:04 MST")
- ts-rfc822z ("02 Jan 06 15:04 -0700")
- ts-rfc850 ("Monday, 02-Jan-06 15:04:05 MST")
- ts-rfc1123 ("Mon, 02 Jan 2006 15:04:05 MST")
- ts-rfc1123z ("Mon, 02 Jan 2006 15:04:05 -0700")
- ts-rfc3339 ("2006-01-02T15:04:05Z07:00")
- ts-rfc3339nano ("2006-01-02T15:04:05.999999999Z07:00")
- ts-httpd ("02/Jan/2006:15:04:05 -0700")
- ts-epoch (seconds since unix epoch)
- ts-epochnano (nanoseconds since unix epoch)
- ts-"CUSTOM"
CUSTOM time layouts must be within quotes and be the representation of the
"reference time", which is `Mon Jan 2 15:04:05 -0700 MST 2006`
See https://golang.org/pkg/time/#Parse for more details.

View File

@@ -1,373 +0,0 @@
package grok
import (
"bufio"
"fmt"
"log"
"os"
"regexp"
"strconv"
"strings"
"time"
"github.com/vjeantet/grok"
"github.com/influxdata/telegraf"
)
var timeFormats = map[string]string{
"ts-ansic": "Mon Jan _2 15:04:05 2006",
"ts-unix": "Mon Jan _2 15:04:05 MST 2006",
"ts-ruby": "Mon Jan 02 15:04:05 -0700 2006",
"ts-rfc822": "02 Jan 06 15:04 MST",
"ts-rfc822z": "02 Jan 06 15:04 -0700", // RFC822 with numeric zone
"ts-rfc850": "Monday, 02-Jan-06 15:04:05 MST",
"ts-rfc1123": "Mon, 02 Jan 2006 15:04:05 MST",
"ts-rfc1123z": "Mon, 02 Jan 2006 15:04:05 -0700", // RFC1123 with numeric zone
"ts-rfc3339": "2006-01-02T15:04:05Z07:00",
"ts-rfc3339nano": "2006-01-02T15:04:05.999999999Z07:00",
"ts-httpd": "02/Jan/2006:15:04:05 -0700",
"ts-epoch": "EPOCH",
"ts-epochnano": "EPOCH_NANO",
}
const (
INT = "int"
TAG = "tag"
FLOAT = "float"
STRING = "string"
DURATION = "duration"
DROP = "drop"
)
var (
// matches named captures that contain a type.
// ie,
// %{NUMBER:bytes:int}
// %{IPORHOST:clientip:tag}
// %{HTTPDATE:ts1:ts-http}
// %{HTTPDATE:ts2:ts-"02 Jan 06 15:04"}
typedRe = regexp.MustCompile(`%{\w+:(\w+):(ts-".+"|t?s?-?\w+)}`)
// matches a plain pattern name. ie, %{NUMBER}
patternOnlyRe = regexp.MustCompile(`%{(\w+)}`)
)
type Parser struct {
Patterns []string
CustomPatterns string
CustomPatternFiles []string
// typeMap is a map of patterns -> capture name -> modifier,
// ie, {
// "%{TESTLOG}":
// {
// "bytes": "int",
// "clientip": "tag"
// }
// }
typeMap map[string]map[string]string
// tsMap is a map of patterns -> capture name -> timestamp layout.
// ie, {
// "%{TESTLOG}":
// {
// "httptime": "02/Jan/2006:15:04:05 -0700"
// }
// }
tsMap map[string]map[string]string
// patterns is a map of all of the parsed patterns from CustomPatterns
// and CustomPatternFiles.
// ie, {
// "DURATION": "%{NUMBER}[nuµm]?s"
// "RESPONSE_CODE": "%{NUMBER:rc:tag}"
// }
patterns map[string]string
g *grok.Grok
tsModder *tsModder
}
func (p *Parser) Compile() error {
p.typeMap = make(map[string]map[string]string)
p.tsMap = make(map[string]map[string]string)
p.patterns = make(map[string]string)
p.tsModder = &tsModder{}
var err error
p.g, err = grok.NewWithConfig(&grok.Config{NamedCapturesOnly: true})
if err != nil {
return err
}
p.CustomPatterns = DEFAULT_PATTERNS + p.CustomPatterns
if len(p.CustomPatterns) != 0 {
scanner := bufio.NewScanner(strings.NewReader(p.CustomPatterns))
p.addCustomPatterns(scanner)
}
for _, filename := range p.CustomPatternFiles {
file, err := os.Open(filename)
if err != nil {
return err
}
scanner := bufio.NewScanner(bufio.NewReader(file))
p.addCustomPatterns(scanner)
}
return p.compileCustomPatterns()
}
func (p *Parser) ParseLine(line string) (telegraf.Metric, error) {
var err error
var values map[string]string
// the matching pattern string
var patternName string
for _, pattern := range p.Patterns {
if values, err = p.g.Parse(pattern, line); err != nil {
return nil, err
}
if len(values) != 0 {
patternName = pattern
break
}
}
if len(values) == 0 {
return nil, nil
}
fields := make(map[string]interface{})
tags := make(map[string]string)
timestamp := time.Now()
for k, v := range values {
if k == "" || v == "" {
continue
}
var t string
// check if pattern has some modifiers
if types, ok := p.typeMap[patternName]; ok {
t = types[k]
}
// if we didn't find a modifier, check if we have a timestamp layout
if t == "" {
if ts, ok := p.tsMap[patternName]; ok {
// check if the modifier is a timestamp layout
if layout, ok := ts[k]; ok {
t = layout
}
}
}
// if we didn't find a type OR timestamp modifier, assume string
if t == "" {
t = STRING
}
switch t {
case INT:
iv, err := strconv.ParseInt(v, 10, 64)
if err != nil {
log.Printf("ERROR parsing %s to int: %s", v, err)
} else {
fields[k] = iv
}
case FLOAT:
fv, err := strconv.ParseFloat(v, 64)
if err != nil {
log.Printf("ERROR parsing %s to float: %s", v, err)
} else {
fields[k] = fv
}
case DURATION:
d, err := time.ParseDuration(v)
if err != nil {
log.Printf("ERROR parsing %s to duration: %s", v, err)
} else {
fields[k] = int64(d)
}
case TAG:
tags[k] = v
case STRING:
fields[k] = strings.Trim(v, `"`)
case "EPOCH":
iv, err := strconv.ParseInt(v, 10, 64)
if err != nil {
log.Printf("ERROR parsing %s to int: %s", v, err)
} else {
timestamp = time.Unix(iv, 0)
}
case "EPOCH_NANO":
iv, err := strconv.ParseInt(v, 10, 64)
if err != nil {
log.Printf("ERROR parsing %s to int: %s", v, err)
} else {
timestamp = time.Unix(0, iv)
}
case DROP:
// goodbye!
default:
ts, err := time.Parse(t, v)
if err == nil {
timestamp = ts
} else {
log.Printf("ERROR parsing %s to time layout [%s]: %s", v, t, err)
}
}
}
return telegraf.NewMetric("logparser_grok", tags, fields, p.tsModder.tsMod(timestamp))
}
func (p *Parser) addCustomPatterns(scanner *bufio.Scanner) {
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if len(line) > 0 && line[0] != '#' {
names := strings.SplitN(line, " ", 2)
p.patterns[names[0]] = names[1]
}
}
}
func (p *Parser) compileCustomPatterns() error {
var err error
// check if the pattern contains a subpattern that is already defined
// replace it with the subpattern for modifier inheritance.
for i := 0; i < 2; i++ {
for name, pattern := range p.patterns {
subNames := patternOnlyRe.FindAllStringSubmatch(pattern, -1)
for _, subName := range subNames {
if subPattern, ok := p.patterns[subName[1]]; ok {
pattern = strings.Replace(pattern, subName[0], subPattern, 1)
}
}
p.patterns[name] = pattern
}
}
// check if pattern contains modifiers. Parse them out if it does.
for name, pattern := range p.patterns {
if typedRe.MatchString(pattern) {
// this pattern has modifiers, so parse out the modifiers
pattern, err = p.parseTypedCaptures(name, pattern)
if err != nil {
return err
}
p.patterns[name] = pattern
}
}
return p.g.AddPatternsFromMap(p.patterns)
}
// parseTypedCaptures parses the capture types, and then deletes the type from
// the line so that it is a valid "grok" pattern again.
// ie,
// %{NUMBER:bytes:int} => %{NUMBER:bytes} (stores %{NUMBER}->bytes->int)
// %{IPORHOST:clientip:tag} => %{IPORHOST:clientip} (stores %{IPORHOST}->clientip->tag)
func (p *Parser) parseTypedCaptures(name, pattern string) (string, error) {
matches := typedRe.FindAllStringSubmatch(pattern, -1)
// grab the name of the capture pattern
patternName := "%{" + name + "}"
// create type map for this pattern
p.typeMap[patternName] = make(map[string]string)
p.tsMap[patternName] = make(map[string]string)
// boolean to verify that each pattern only has a single ts- data type.
hasTimestamp := false
for _, match := range matches {
// regex capture 1 is the name of the capture
// regex capture 2 is the type of the capture
if strings.HasPrefix(match[2], "ts-") {
if hasTimestamp {
return pattern, fmt.Errorf("logparser pattern compile error: "+
"Each pattern is allowed only one named "+
"timestamp data type. pattern: %s", pattern)
}
if f, ok := timeFormats[match[2]]; ok {
p.tsMap[patternName][match[1]] = f
} else {
p.tsMap[patternName][match[1]] = strings.TrimSuffix(strings.TrimPrefix(match[2], `ts-"`), `"`)
}
hasTimestamp = true
} else {
p.typeMap[patternName][match[1]] = match[2]
}
// the modifier is not a valid part of a "grok" pattern, so remove it
// from the pattern.
pattern = strings.Replace(pattern, ":"+match[2]+"}", "}", 1)
}
return pattern, nil
}
// tsModder is a struct for incrementing identical timestamps of log lines
// so that we don't push identical metrics that will get overwritten.
type tsModder struct {
dupe time.Time
last time.Time
incr time.Duration
incrn time.Duration
rollover time.Duration
}
// tsMod increments the given timestamp one unit more from the previous
// duplicate timestamp.
// the increment unit is determined as the next smallest time unit below the
// most significant time unit of ts.
// ie, if the input is at ms precision, it will increment it 1µs.
func (t *tsModder) tsMod(ts time.Time) time.Time {
defer func() { t.last = ts }()
// don't mod the time if we don't need to
if t.last.IsZero() || ts.IsZero() {
t.incrn = 0
t.rollover = 0
return ts
}
if !ts.Equal(t.last) && !ts.Equal(t.dupe) {
t.incr = 0
t.incrn = 0
t.rollover = 0
return ts
}
if ts.Equal(t.last) {
t.dupe = ts
}
if ts.Equal(t.dupe) && t.incr == time.Duration(0) {
tsNano := ts.UnixNano()
d := int64(10)
counter := 1
for {
a := tsNano % d
if a > 0 {
break
}
d = d * 10
counter++
}
switch {
case counter <= 6:
t.incr = time.Nanosecond
case counter <= 9:
t.incr = time.Microsecond
case counter > 9:
t.incr = time.Millisecond
}
}
t.incrn++
if t.incrn == 999 && t.incr > time.Nanosecond {
t.rollover = t.incr * t.incrn
t.incrn = 1
t.incr = t.incr / 1000
if t.incr < time.Nanosecond {
t.incr = time.Nanosecond
}
}
return ts.Add(t.incr*t.incrn + t.rollover)
}

View File

@@ -1,508 +0,0 @@
package grok
import (
"testing"
"time"
"github.com/influxdata/telegraf"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
var benchM telegraf.Metric
func Benchmark_ParseLine_CommonLogFormat(b *testing.B) {
p := &Parser{
Patterns: []string{"%{COMMON_LOG_FORMAT}"},
}
p.Compile()
var m telegraf.Metric
for n := 0; n < b.N; n++ {
m, _ = p.ParseLine(`127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326`)
}
benchM = m
}
func Benchmark_ParseLine_CombinedLogFormat(b *testing.B) {
p := &Parser{
Patterns: []string{"%{COMBINED_LOG_FORMAT}"},
}
p.Compile()
var m telegraf.Metric
for n := 0; n < b.N; n++ {
m, _ = p.ParseLine(`127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "-" "Mozilla"`)
}
benchM = m
}
func Benchmark_ParseLine_InfluxLog(b *testing.B) {
p := &Parser{
Patterns: []string{"%{INFLUXDB_HTTPD_LOG}"},
}
p.Compile()
var m telegraf.Metric
for n := 0; n < b.N; n++ {
m, _ = p.ParseLine(`[httpd] 192.168.1.1 - - [14/Jun/2016:11:33:29 +0100] "POST /write?consistency=any&db=telegraf&precision=ns&rp= HTTP/1.1" 204 0 "-" "InfluxDBClient" 6f61bc44-321b-11e6-8050-000000000000 2513`)
}
benchM = m
}
func Benchmark_ParseLine_InfluxLog_NoMatch(b *testing.B) {
p := &Parser{
Patterns: []string{"%{INFLUXDB_HTTPD_LOG}"},
}
p.Compile()
var m telegraf.Metric
for n := 0; n < b.N; n++ {
m, _ = p.ParseLine(`[retention] 2016/06/14 14:38:24 retention policy shard deletion check commencing`)
}
benchM = m
}
func Benchmark_ParseLine_CustomPattern(b *testing.B) {
p := &Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatterns: `
DURATION %{NUMBER}[nuµm]?s
RESPONSE_CODE %{NUMBER:response_code:tag}
RESPONSE_TIME %{DURATION:response_time:duration}
TEST_LOG_A %{NUMBER:myfloat:float} %{RESPONSE_CODE} %{IPORHOST:clientip} %{RESPONSE_TIME}
`,
}
p.Compile()
var m telegraf.Metric
for n := 0; n < b.N; n++ {
m, _ = p.ParseLine(`[04/Jun/2016:12:41:45 +0100] 1.25 200 192.168.1.1 5.432µs 101`)
}
benchM = m
}
func TestBuiltinInfluxdbHttpd(t *testing.T) {
p := &Parser{
Patterns: []string{"%{INFLUXDB_HTTPD_LOG}"},
}
assert.NoError(t, p.Compile())
// Parse an influxdb POST request
m, err := p.ParseLine(`[httpd] ::1 - - [14/Jun/2016:11:33:29 +0100] "POST /write?consistency=any&db=telegraf&precision=ns&rp= HTTP/1.1" 204 0 "-" "InfluxDBClient" 6f61bc44-321b-11e6-8050-000000000000 2513`)
require.NotNil(t, m)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"resp_bytes": int64(0),
"auth": "-",
"client_ip": "::1",
"resp_code": int64(204),
"http_version": float64(1.1),
"ident": "-",
"referrer": "-",
"request": "/write?consistency=any&db=telegraf&precision=ns&rp=",
"response_time_us": int64(2513),
"agent": "InfluxDBClient",
},
m.Fields())
assert.Equal(t, map[string]string{"verb": "POST"}, m.Tags())
// Parse an influxdb GET request
m, err = p.ParseLine(`[httpd] ::1 - - [14/Jun/2016:12:10:02 +0100] "GET /query?db=telegraf&q=SELECT+bytes%2Cresponse_time_us+FROM+logparser_grok+WHERE+http_method+%3D+%27GET%27+AND+response_time_us+%3E+0+AND+time+%3E+now%28%29+-+1h HTTP/1.1" 200 578 "http://localhost:8083/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36" 8a3806f1-3220-11e6-8006-000000000000 988`)
require.NotNil(t, m)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"resp_bytes": int64(578),
"auth": "-",
"client_ip": "::1",
"resp_code": int64(200),
"http_version": float64(1.1),
"ident": "-",
"referrer": "http://localhost:8083/",
"request": "/query?db=telegraf&q=SELECT+bytes%2Cresponse_time_us+FROM+logparser_grok+WHERE+http_method+%3D+%27GET%27+AND+response_time_us+%3E+0+AND+time+%3E+now%28%29+-+1h",
"response_time_us": int64(988),
"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36",
},
m.Fields())
assert.Equal(t, map[string]string{"verb": "GET"}, m.Tags())
}
// common log format
// 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
func TestBuiltinCommonLogFormat(t *testing.T) {
p := &Parser{
Patterns: []string{"%{COMMON_LOG_FORMAT}"},
}
assert.NoError(t, p.Compile())
// Parse an influxdb POST request
m, err := p.ParseLine(`127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326`)
require.NotNil(t, m)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"resp_bytes": int64(2326),
"auth": "frank",
"client_ip": "127.0.0.1",
"resp_code": int64(200),
"http_version": float64(1.0),
"ident": "user-identifier",
"request": "/apache_pb.gif",
},
m.Fields())
assert.Equal(t, map[string]string{"verb": "GET"}, m.Tags())
}
// combined log format
// 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "-" "Mozilla"
func TestBuiltinCombinedLogFormat(t *testing.T) {
p := &Parser{
Patterns: []string{"%{COMBINED_LOG_FORMAT}"},
}
assert.NoError(t, p.Compile())
// Parse an influxdb POST request
m, err := p.ParseLine(`127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "-" "Mozilla"`)
require.NotNil(t, m)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"resp_bytes": int64(2326),
"auth": "frank",
"client_ip": "127.0.0.1",
"resp_code": int64(200),
"http_version": float64(1.0),
"ident": "user-identifier",
"request": "/apache_pb.gif",
"referrer": "-",
"agent": "Mozilla",
},
m.Fields())
assert.Equal(t, map[string]string{"verb": "GET"}, m.Tags())
}
func TestCompileStringAndParse(t *testing.T) {
p := &Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatterns: `
DURATION %{NUMBER}[nuµm]?s
RESPONSE_CODE %{NUMBER:response_code:tag}
RESPONSE_TIME %{DURATION:response_time:duration}
TEST_LOG_A %{NUMBER:myfloat:float} %{RESPONSE_CODE} %{IPORHOST:clientip} %{RESPONSE_TIME}
`,
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`1.25 200 192.168.1.1 5.432µs`)
require.NotNil(t, metricA)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"clientip": "192.168.1.1",
"myfloat": float64(1.25),
"response_time": int64(5432),
},
metricA.Fields())
assert.Equal(t, map[string]string{"response_code": "200"}, metricA.Tags())
}
func TestParseEpochNano(t *testing.T) {
p := &Parser{
Patterns: []string{"%{MYAPP}"},
CustomPatterns: `
MYAPP %{POSINT:ts:ts-epochnano} response_time=%{POSINT:response_time:int} mymetric=%{NUMBER:metric:float}
`,
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`1466004605359052000 response_time=20821 mymetric=10890.645`)
require.NotNil(t, metricA)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"response_time": int64(20821),
"metric": float64(10890.645),
},
metricA.Fields())
assert.Equal(t, map[string]string{}, metricA.Tags())
assert.Equal(t, time.Unix(0, 1466004605359052000), metricA.Time())
}
func TestParseEpoch(t *testing.T) {
p := &Parser{
Patterns: []string{"%{MYAPP}"},
CustomPatterns: `
MYAPP %{POSINT:ts:ts-epoch} response_time=%{POSINT:response_time:int} mymetric=%{NUMBER:metric:float}
`,
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`1466004605 response_time=20821 mymetric=10890.645`)
require.NotNil(t, metricA)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"response_time": int64(20821),
"metric": float64(10890.645),
},
metricA.Fields())
assert.Equal(t, map[string]string{}, metricA.Tags())
assert.Equal(t, time.Unix(1466004605, 0), metricA.Time())
}
func TestParseEpochErrors(t *testing.T) {
p := &Parser{
Patterns: []string{"%{MYAPP}"},
CustomPatterns: `
MYAPP %{WORD:ts:ts-epoch} response_time=%{POSINT:response_time:int} mymetric=%{NUMBER:metric:float}
`,
}
assert.NoError(t, p.Compile())
_, err := p.ParseLine(`foobar response_time=20821 mymetric=10890.645`)
assert.NoError(t, err)
p = &Parser{
Patterns: []string{"%{MYAPP}"},
CustomPatterns: `
MYAPP %{WORD:ts:ts-epochnano} response_time=%{POSINT:response_time:int} mymetric=%{NUMBER:metric:float}
`,
}
assert.NoError(t, p.Compile())
_, err = p.ParseLine(`foobar response_time=20821 mymetric=10890.645`)
assert.NoError(t, err)
}
func TestCompileFileAndParse(t *testing.T) {
p := &Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatternFiles: []string{"./testdata/test-patterns"},
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`[04/Jun/2016:12:41:45 +0100] 1.25 200 192.168.1.1 5.432µs 101`)
require.NotNil(t, metricA)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"clientip": "192.168.1.1",
"myfloat": float64(1.25),
"response_time": int64(5432),
"myint": int64(101),
},
metricA.Fields())
assert.Equal(t, map[string]string{"response_code": "200"}, metricA.Tags())
assert.Equal(t,
time.Date(2016, time.June, 4, 12, 41, 45, 0, time.FixedZone("foo", 60*60)).Nanosecond(),
metricA.Time().Nanosecond())
metricB, err := p.ParseLine(`[04/06/2016--12:41:45] 1.25 mystring dropme nomodifier`)
require.NotNil(t, metricB)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"myfloat": 1.25,
"mystring": "mystring",
"nomodifier": "nomodifier",
},
metricB.Fields())
assert.Equal(t, map[string]string{}, metricB.Tags())
assert.Equal(t,
time.Date(2016, time.June, 4, 12, 41, 45, 0, time.FixedZone("foo", 60*60)).Nanosecond(),
metricB.Time().Nanosecond())
}
func TestCompileNoModifiersAndParse(t *testing.T) {
p := &Parser{
Patterns: []string{"%{TEST_LOG_C}"},
CustomPatterns: `
DURATION %{NUMBER}[nuµm]?s
TEST_LOG_C %{NUMBER:myfloat} %{NUMBER} %{IPORHOST:clientip} %{DURATION:rt}
`,
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`1.25 200 192.168.1.1 5.432µs`)
require.NotNil(t, metricA)
assert.NoError(t, err)
assert.Equal(t,
map[string]interface{}{
"clientip": "192.168.1.1",
"myfloat": "1.25",
"rt": "5.432µs",
},
metricA.Fields())
assert.Equal(t, map[string]string{}, metricA.Tags())
}
func TestCompileNoNamesAndParse(t *testing.T) {
p := &Parser{
Patterns: []string{"%{TEST_LOG_C}"},
CustomPatterns: `
DURATION %{NUMBER}[nuµm]?s
TEST_LOG_C %{NUMBER} %{NUMBER} %{IPORHOST} %{DURATION}
`,
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`1.25 200 192.168.1.1 5.432µs`)
require.Nil(t, metricA)
assert.NoError(t, err)
}
func TestParseNoMatch(t *testing.T) {
p := &Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatternFiles: []string{"./testdata/test-patterns"},
}
assert.NoError(t, p.Compile())
metricA, err := p.ParseLine(`[04/Jun/2016:12:41:45 +0100] notnumber 200 192.168.1.1 5.432µs 101`)
assert.NoError(t, err)
assert.Nil(t, metricA)
}
func TestCompileErrors(t *testing.T) {
// Compile fails because there are multiple timestamps:
p := &Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatterns: `
TEST_LOG_A %{HTTPDATE:ts1:ts-httpd} %{HTTPDATE:ts2:ts-httpd} %{NUMBER:mynum:int}
`,
}
assert.Error(t, p.Compile())
// Compile fails because file doesn't exist:
p = &Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatternFiles: []string{"/tmp/foo/bar/baz"},
}
assert.Error(t, p.Compile())
}
func TestParseErrors(t *testing.T) {
// Parse fails because the pattern doesn't exist
p := &Parser{
Patterns: []string{"%{TEST_LOG_B}"},
CustomPatterns: `
TEST_LOG_A %{HTTPDATE:ts:ts-httpd} %{WORD:myword:int} %{}
`,
}
assert.NoError(t, p.Compile())
_, err := p.ParseLine(`[04/Jun/2016:12:41:45 +0100] notnumber 200 192.168.1.1 5.432µs 101`)
assert.Error(t, err)
// Parse fails because myword is not an int
p = &Parser{
Patterns: []string{"%{TEST_LOG_A}"},
CustomPatterns: `
TEST_LOG_A %{HTTPDATE:ts:ts-httpd} %{WORD:myword:int}
`,
}
assert.NoError(t, p.Compile())
_, err = p.ParseLine(`04/Jun/2016:12:41:45 +0100 notnumber`)
assert.Error(t, err)
// Parse fails because myword is not a float
p = &Parser{
Patterns: []string{"%{TEST_LOG_A}"},
CustomPatterns: `
TEST_LOG_A %{HTTPDATE:ts:ts-httpd} %{WORD:myword:float}
`,
}
assert.NoError(t, p.Compile())
_, err = p.ParseLine(`04/Jun/2016:12:41:45 +0100 notnumber`)
assert.Error(t, err)
// Parse fails because myword is not a duration
p = &Parser{
Patterns: []string{"%{TEST_LOG_A}"},
CustomPatterns: `
TEST_LOG_A %{HTTPDATE:ts:ts-httpd} %{WORD:myword:duration}
`,
}
assert.NoError(t, p.Compile())
_, err = p.ParseLine(`04/Jun/2016:12:41:45 +0100 notnumber`)
assert.Error(t, err)
// Parse fails because the time layout is wrong.
p = &Parser{
Patterns: []string{"%{TEST_LOG_A}"},
CustomPatterns: `
TEST_LOG_A %{HTTPDATE:ts:ts-unix} %{WORD:myword:duration}
`,
}
assert.NoError(t, p.Compile())
_, err = p.ParseLine(`04/Jun/2016:12:41:45 +0100 notnumber`)
assert.Error(t, err)
}
func TestTsModder(t *testing.T) {
tsm := &tsModder{}
reftime := time.Date(2006, time.December, 1, 1, 1, 1, int(time.Millisecond), time.UTC)
modt := tsm.tsMod(reftime)
assert.Equal(t, reftime, modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Microsecond*1), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Microsecond*2), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Microsecond*3), modt)
reftime = time.Date(2006, time.December, 1, 1, 1, 1, int(time.Microsecond), time.UTC)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime, modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Nanosecond*1), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Nanosecond*2), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Nanosecond*3), modt)
reftime = time.Date(2006, time.December, 1, 1, 1, 1, int(time.Microsecond)*999, time.UTC)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime, modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Nanosecond*1), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Nanosecond*2), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Nanosecond*3), modt)
reftime = time.Date(2006, time.December, 1, 1, 1, 1, 0, time.UTC)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime, modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Millisecond*1), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Millisecond*2), modt)
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime.Add(time.Millisecond*3), modt)
reftime = time.Time{}
modt = tsm.tsMod(reftime)
assert.Equal(t, reftime, modt)
}
func TestTsModder_Rollover(t *testing.T) {
tsm := &tsModder{}
reftime := time.Date(2006, time.December, 1, 1, 1, 1, int(time.Millisecond), time.UTC)
modt := tsm.tsMod(reftime)
for i := 1; i < 1000; i++ {
modt = tsm.tsMod(reftime)
}
assert.Equal(t, reftime.Add(time.Microsecond*999+time.Nanosecond), modt)
reftime = time.Date(2006, time.December, 1, 1, 1, 1, int(time.Microsecond), time.UTC)
modt = tsm.tsMod(reftime)
for i := 1; i < 1001; i++ {
modt = tsm.tsMod(reftime)
}
assert.Equal(t, reftime.Add(time.Nanosecond*1000), modt)
}

View File

@@ -1,80 +0,0 @@
package grok
// THIS SHOULD BE KEPT IN-SYNC WITH patterns/influx-patterns
const DEFAULT_PATTERNS = `
# Captures are a slightly modified version of logstash "grok" patterns, with
# the format %{<capture syntax>[:<semantic name>][:<modifier>]}
# By default all named captures are converted into string fields.
# Modifiers can be used to convert captures to other types or tags.
# Timestamp modifiers can be used to convert captures to the timestamp of the
# parsed metric.
# View logstash grok pattern docs here:
# https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
# All default logstash patterns are supported, these can be viewed here:
# https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
# Available modifiers:
# string (default if nothing is specified)
# int
# float
# duration (ie, 5.23ms gets converted to int nanoseconds)
# tag (converts the field into a tag)
# drop (drops the field completely)
# Timestamp modifiers:
# ts-ansic ("Mon Jan _2 15:04:05 2006")
# ts-unix ("Mon Jan _2 15:04:05 MST 2006")
# ts-ruby ("Mon Jan 02 15:04:05 -0700 2006")
# ts-rfc822 ("02 Jan 06 15:04 MST")
# ts-rfc822z ("02 Jan 06 15:04 -0700")
# ts-rfc850 ("Monday, 02-Jan-06 15:04:05 MST")
# ts-rfc1123 ("Mon, 02 Jan 2006 15:04:05 MST")
# ts-rfc1123z ("Mon, 02 Jan 2006 15:04:05 -0700")
# ts-rfc3339 ("2006-01-02T15:04:05Z07:00")
# ts-rfc3339nano ("2006-01-02T15:04:05.999999999Z07:00")
# ts-httpd ("02/Jan/2006:15:04:05 -0700")
# ts-epoch (seconds since unix epoch)
# ts-epochnano (nanoseconds since unix epoch)
# ts-"CUSTOM"
# CUSTOM time layouts must be within quotes and be the representation of the
# "reference time", which is Mon Jan 2 15:04:05 -0700 MST 2006
# See https://golang.org/pkg/time/#Parse for more details.
# Example log file pattern, example log looks like this:
# [04/Jun/2016:12:41:45 +0100] 1.25 200 192.168.1.1 5.432µs
# Breakdown of the DURATION pattern below:
# NUMBER is a builtin logstash grok pattern matching float & int numbers.
# [nuµm]? is a regex specifying 0 or 1 of the characters within brackets.
# s is also regex, this pattern must end in "s".
# so DURATION will match something like '5.324ms' or '6.1µs' or '10s'
DURATION %{NUMBER}[nuµm]?s
RESPONSE_CODE %{NUMBER:response_code:tag}
RESPONSE_TIME %{DURATION:response_time_ns:duration}
EXAMPLE_LOG \[%{HTTPDATE:ts:ts-httpd}\] %{NUMBER:myfloat:float} %{RESPONSE_CODE} %{IPORHOST:clientip} %{RESPONSE_TIME}
# Wider-ranging username matching vs. logstash built-in %{USER}
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
##
## COMMON LOG PATTERNS
##
# InfluxDB log patterns
CLIENT (?:%{IPORHOST}|%{HOSTPORT}|::1)
INFLUXDB_HTTPD_LOG \[httpd\] %{COMBINED_LOG_FORMAT} %{UUID:uuid:drop} %{NUMBER:response_time_us:int}
# apache & nginx logs, this is also known as the "common log format"
# see https://en.wikipedia.org/wiki/Common_Log_Format
COMMON_LOG_FORMAT %{CLIENT:client_ip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:ts:ts-httpd}\] "(?:%{WORD:verb:tag} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version:float})?|%{DATA})" %{NUMBER:resp_code:int} (?:%{NUMBER:resp_bytes:int}|-)
# Combined log format is the same as the common log format but with the addition
# of two quoted strings at the end for "referrer" and "agent"
# See Examples at http://httpd.apache.org/docs/current/mod/mod_log_config.html
COMBINED_LOG_FORMAT %{COMMON_LOG_FORMAT} %{QS:referrer} %{QS:agent}
# HTTPD log formats
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel:tag}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel:tag}\] \[pid %{POSINT:pid:int}:tid %{NUMBER:tid:int}\]( \(%{POSINT:proxy_errorcode:int}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
`

View File

@@ -1,75 +0,0 @@
# Captures are a slightly modified version of logstash "grok" patterns, with
# the format %{<capture syntax>[:<semantic name>][:<modifier>]}
# By default all named captures are converted into string fields.
# Modifiers can be used to convert captures to other types or tags.
# Timestamp modifiers can be used to convert captures to the timestamp of the
# parsed metric.
# View logstash grok pattern docs here:
# https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
# All default logstash patterns are supported, these can be viewed here:
# https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
# Available modifiers:
# string (default if nothing is specified)
# int
# float
# duration (ie, 5.23ms gets converted to int nanoseconds)
# tag (converts the field into a tag)
# drop (drops the field completely)
# Timestamp modifiers:
# ts-ansic ("Mon Jan _2 15:04:05 2006")
# ts-unix ("Mon Jan _2 15:04:05 MST 2006")
# ts-ruby ("Mon Jan 02 15:04:05 -0700 2006")
# ts-rfc822 ("02 Jan 06 15:04 MST")
# ts-rfc822z ("02 Jan 06 15:04 -0700")
# ts-rfc850 ("Monday, 02-Jan-06 15:04:05 MST")
# ts-rfc1123 ("Mon, 02 Jan 2006 15:04:05 MST")
# ts-rfc1123z ("Mon, 02 Jan 2006 15:04:05 -0700")
# ts-rfc3339 ("2006-01-02T15:04:05Z07:00")
# ts-rfc3339nano ("2006-01-02T15:04:05.999999999Z07:00")
# ts-httpd ("02/Jan/2006:15:04:05 -0700")
# ts-epoch (seconds since unix epoch)
# ts-epochnano (nanoseconds since unix epoch)
# ts-"CUSTOM"
# CUSTOM time layouts must be within quotes and be the representation of the
# "reference time", which is Mon Jan 2 15:04:05 -0700 MST 2006
# See https://golang.org/pkg/time/#Parse for more details.
# Example log file pattern, example log looks like this:
# [04/Jun/2016:12:41:45 +0100] 1.25 200 192.168.1.1 5.432µs
# Breakdown of the DURATION pattern below:
# NUMBER is a builtin logstash grok pattern matching float & int numbers.
# [nuµm]? is a regex specifying 0 or 1 of the characters within brackets.
# s is also regex, this pattern must end in "s".
# so DURATION will match something like '5.324ms' or '6.1µs' or '10s'
DURATION %{NUMBER}[nuµm]?s
RESPONSE_CODE %{NUMBER:response_code:tag}
RESPONSE_TIME %{DURATION:response_time_ns:duration}
EXAMPLE_LOG \[%{HTTPDATE:ts:ts-httpd}\] %{NUMBER:myfloat:float} %{RESPONSE_CODE} %{IPORHOST:clientip} %{RESPONSE_TIME}
# Wider-ranging username matching vs. logstash built-in %{USER}
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
##
## COMMON LOG PATTERNS
##
# InfluxDB log patterns
CLIENT (?:%{IPORHOST}|%{HOSTPORT}|::1)
INFLUXDB_HTTPD_LOG \[httpd\] %{COMBINED_LOG_FORMAT} %{UUID:uuid:drop} %{NUMBER:response_time_us:int}
# apache & nginx logs, this is also known as the "common log format"
# see https://en.wikipedia.org/wiki/Common_Log_Format
COMMON_LOG_FORMAT %{CLIENT:client_ip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:ts:ts-httpd}\] "(?:%{WORD:verb:tag} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version:float})?|%{DATA})" %{NUMBER:resp_code:int} (?:%{NUMBER:resp_bytes:int}|-)
# Combined log format is the same as the common log format but with the addition
# of two quoted strings at the end for "referrer" and "agent"
# See Examples at http://httpd.apache.org/docs/current/mod/mod_log_config.html
COMBINED_LOG_FORMAT %{COMMON_LOG_FORMAT} %{QS:referrer} %{QS:agent}
# HTTPD log formats
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel:tag}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel:tag}\] \[pid %{POSINT:pid:int}:tid %{NUMBER:tid:int}\]( \(%{POSINT:proxy_errorcode:int}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

View File

@@ -1,14 +0,0 @@
# Test A log line:
# [04/Jun/2016:12:41:45 +0100] 1.25 200 192.168.1.1 5.432µs 101
DURATION %{NUMBER}[nuµm]?s
RESPONSE_CODE %{NUMBER:response_code:tag}
RESPONSE_TIME %{DURATION:response_time:duration}
TEST_LOG_A \[%{HTTPDATE:timestamp:ts-httpd}\] %{NUMBER:myfloat:float} %{RESPONSE_CODE} %{IPORHOST:clientip} %{RESPONSE_TIME} %{NUMBER:myint:int}
# Test B log line:
# [04/06/2016--12:41:45] 1.25 mystring dropme nomodifier
TEST_TIMESTAMP %{MONTHDAY}/%{MONTHNUM}/%{YEAR}--%{TIME}
TEST_LOG_B \[%{TEST_TIMESTAMP:timestamp:ts-"02/01/2006--15:04:05"}\] %{NUMBER:myfloat:float} %{WORD:mystring:string} %{WORD:dropme:drop} %{WORD:nomodifier}
TEST_TIMESTAMP %{MONTHDAY}/%{MONTHNUM}/%{YEAR}--%{TIME}
TEST_LOG_BAD \[%{TEST_TIMESTAMP:timestamp:ts-"02/01/2006--15:04:05"}\] %{NUMBER:myfloat:float} %{WORD:mystring:int} %{WORD:dropme:drop} %{WORD:nomodifier}

View File

@@ -1 +0,0 @@
[04/Jun/2016:12:41:45 +0100] 1.25 200 192.168.1.1 5.432µs 101

View File

@@ -1 +0,0 @@
[04/06/2016--12:41:45] 1.25 mystring dropme nomodifier

View File

@@ -1,228 +0,0 @@
package logparser
import (
"fmt"
"log"
"reflect"
"sync"
"github.com/hpcloud/tail"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal/globpath"
"github.com/influxdata/telegraf/plugins/inputs"
// Parsers
"github.com/influxdata/telegraf/plugins/inputs/logparser/grok"
)
type LogParser interface {
ParseLine(line string) (telegraf.Metric, error)
Compile() error
}
type LogParserPlugin struct {
Files []string
FromBeginning bool
tailers []*tail.Tail
lines chan string
done chan struct{}
wg sync.WaitGroup
acc telegraf.Accumulator
parsers []LogParser
sync.Mutex
GrokParser *grok.Parser `toml:"grok"`
}
const sampleConfig = `
## Log files to parse.
## These accept standard unix glob matching rules, but with the addition of
## ** as a "super asterisk". ie:
## /var/log/**.log -> recursively find all .log files in /var/log
## /var/log/*/*.log -> find all .log files with a parent dir in /var/log
## /var/log/apache.log -> only tail the apache log file
files = ["/var/log/influxdb/influxdb.log"]
## Read file from beginning.
from_beginning = false
## Parse logstash-style "grok" patterns:
## Telegraf built-in parsing patterns: https://goo.gl/dkay10
[inputs.logparser.grok]
## This is a list of patterns to check the given log file(s) for.
## Note that adding patterns here increases processing time. The most
## efficient configuration is to have one pattern per logparser.
## Other common built-in patterns are:
## %{COMMON_LOG_FORMAT} (plain apache & nginx access logs)
## %{COMBINED_LOG_FORMAT} (access logs + referrer & agent)
patterns = ["%{INFLUXDB_HTTPD_LOG}"]
## Full path(s) to custom pattern files.
custom_pattern_files = []
## Custom patterns can also be defined here. Put one pattern per line.
custom_patterns = '''
'''
`
func (l *LogParserPlugin) SampleConfig() string {
return sampleConfig
}
func (l *LogParserPlugin) Description() string {
return "Stream and parse log file(s)."
}
func (l *LogParserPlugin) Gather(acc telegraf.Accumulator) error {
return nil
}
func (l *LogParserPlugin) Start(acc telegraf.Accumulator) error {
l.Lock()
defer l.Unlock()
l.acc = acc
l.lines = make(chan string, 1000)
l.done = make(chan struct{})
// Looks for fields which implement LogParser interface
l.parsers = []LogParser{}
s := reflect.ValueOf(l).Elem()
for i := 0; i < s.NumField(); i++ {
f := s.Field(i)
if !f.CanInterface() {
continue
}
if lpPlugin, ok := f.Interface().(LogParser); ok {
if reflect.ValueOf(lpPlugin).IsNil() {
continue
}
l.parsers = append(l.parsers, lpPlugin)
}
}
if len(l.parsers) == 0 {
return fmt.Errorf("ERROR: logparser input plugin: no parser defined.")
}
// compile log parser patterns:
for _, parser := range l.parsers {
if err := parser.Compile(); err != nil {
return err
}
}
var seek tail.SeekInfo
if !l.FromBeginning {
seek.Whence = 2
seek.Offset = 0
}
l.wg.Add(1)
go l.parser()
var errS string
// Create a "tailer" for each file
for _, filepath := range l.Files {
g, err := globpath.Compile(filepath)
if err != nil {
log.Printf("ERROR Glob %s failed to compile, %s", filepath, err)
}
for file, _ := range g.Match() {
tailer, err := tail.TailFile(file,
tail.Config{
ReOpen: true,
Follow: true,
Location: &seek,
})
if err != nil {
errS += err.Error() + " "
continue
}
// create a goroutine for each "tailer"
l.wg.Add(1)
go l.receiver(tailer)
l.tailers = append(l.tailers, tailer)
}
}
if errS != "" {
return fmt.Errorf(errS)
}
return nil
}
// receiver is launched as a goroutine to continuously watch a tailed logfile
// for changes and send any log lines down the l.lines channel.
func (l *LogParserPlugin) receiver(tailer *tail.Tail) {
defer l.wg.Done()
var line *tail.Line
for line = range tailer.Lines {
if line.Err != nil {
log.Printf("ERROR tailing file %s, Error: %s\n",
tailer.Filename, line.Err)
continue
}
select {
case <-l.done:
case l.lines <- line.Text:
}
}
}
// parser is launched as a goroutine to watch the l.lines channel.
// when a line is available, parser parses it and adds the metric(s) to the
// accumulator.
func (l *LogParserPlugin) parser() {
defer l.wg.Done()
var m telegraf.Metric
var err error
var line string
for {
select {
case <-l.done:
return
case line = <-l.lines:
if line == "" || line == "\n" {
continue
}
}
for _, parser := range l.parsers {
m, err = parser.ParseLine(line)
if err == nil {
if m != nil {
l.acc.AddFields(m.Name(), m.Fields(), m.Tags(), m.Time())
}
} else {
log.Printf("Malformed log line in [%s], Error: %s\n", line, err)
}
}
}
}
func (l *LogParserPlugin) Stop() {
l.Lock()
defer l.Unlock()
for _, t := range l.tailers {
err := t.Stop()
if err != nil {
log.Printf("ERROR stopping tail on file %s\n", t.Filename)
}
t.Cleanup()
}
close(l.done)
l.wg.Wait()
}
func init() {
inputs.Add("logparser", func() telegraf.Input {
return &LogParserPlugin{}
})
}

View File

@@ -1,116 +0,0 @@
package logparser
import (
"runtime"
"strings"
"testing"
"time"
"github.com/influxdata/telegraf/testutil"
"github.com/influxdata/telegraf/plugins/inputs/logparser/grok"
"github.com/stretchr/testify/assert"
)
func TestStartNoParsers(t *testing.T) {
logparser := &LogParserPlugin{
FromBeginning: true,
Files: []string{"grok/testdata/*.log"},
}
acc := testutil.Accumulator{}
assert.Error(t, logparser.Start(&acc))
}
func TestGrokParseLogFilesNonExistPattern(t *testing.T) {
thisdir := getCurrentDir()
p := &grok.Parser{
Patterns: []string{"%{FOOBAR}"},
CustomPatternFiles: []string{thisdir + "grok/testdata/test-patterns"},
}
logparser := &LogParserPlugin{
FromBeginning: true,
Files: []string{thisdir + "grok/testdata/*.log"},
GrokParser: p,
}
acc := testutil.Accumulator{}
assert.NoError(t, logparser.Start(&acc))
time.Sleep(time.Millisecond * 500)
logparser.Stop()
}
func TestGrokParseLogFiles(t *testing.T) {
thisdir := getCurrentDir()
p := &grok.Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_B}"},
CustomPatternFiles: []string{thisdir + "grok/testdata/test-patterns"},
}
logparser := &LogParserPlugin{
FromBeginning: true,
Files: []string{thisdir + "grok/testdata/*.log"},
GrokParser: p,
}
acc := testutil.Accumulator{}
assert.NoError(t, logparser.Start(&acc))
time.Sleep(time.Millisecond * 500)
logparser.Stop()
acc.AssertContainsTaggedFields(t, "logparser_grok",
map[string]interface{}{
"clientip": "192.168.1.1",
"myfloat": float64(1.25),
"response_time": int64(5432),
"myint": int64(101),
},
map[string]string{"response_code": "200"})
acc.AssertContainsTaggedFields(t, "logparser_grok",
map[string]interface{}{
"myfloat": 1.25,
"mystring": "mystring",
"nomodifier": "nomodifier",
},
map[string]string{})
}
func TestGrokParseLogFilesOneBad(t *testing.T) {
thisdir := getCurrentDir()
p := &grok.Parser{
Patterns: []string{"%{TEST_LOG_A}", "%{TEST_LOG_BAD}"},
CustomPatternFiles: []string{thisdir + "grok/testdata/test-patterns"},
}
assert.NoError(t, p.Compile())
logparser := &LogParserPlugin{
FromBeginning: true,
Files: []string{thisdir + "grok/testdata/*.log"},
GrokParser: p,
}
acc := testutil.Accumulator{}
assert.NoError(t, logparser.Start(&acc))
time.Sleep(time.Millisecond * 500)
logparser.Stop()
acc.AssertContainsTaggedFields(t, "logparser_grok",
map[string]interface{}{
"clientip": "192.168.1.1",
"myfloat": float64(1.25),
"response_time": int64(5432),
"myint": int64(101),
},
map[string]string{"response_code": "200"})
}
func getCurrentDir() string {
_, filename, _, _ := runtime.Caller(1)
return strings.Replace(filename, "logparser_test.go", "", 1)
}

View File

@@ -33,7 +33,7 @@ func (s *Server) gatherData(acc telegraf.Accumulator) error {
result_repl := &ReplSetStatus{}
err = s.Session.DB("admin").Run(bson.D{{"replSetGetStatus", 1}}, result_repl)
if err != nil {
log.Println("Not gathering replica set status, member not in replica set (" + err.Error() + ")")
log.Println("Not gathering replica set status, member not in replica set")
}
jumbo_chunks, _ := s.Session.DB("config").C("chunks").Find(bson.M{"jumbo": true}).Count()

View File

@@ -78,9 +78,9 @@ type ReplSetStatus struct {
// ReplSetMember stores information related to a replica set member
type ReplSetMember struct {
Name string `bson:"name"`
State int64 `bson:"state"`
OptimeDate *bson.MongoTimestamp `bson:"optimeDate"`
Name string `bson:"name"`
State int64 `bson:"state"`
Optime *bson.MongoTimestamp `bson:"optime"`
}
// WiredTiger stores information related to the WiredTiger storage engine.
@@ -663,9 +663,9 @@ func NewStatLine(oldMongo, newMongo MongoStatus, key string, all bool, sampleSec
}
}
if me.OptimeDate != nil && master.OptimeDate != nil && me.State == 2 {
if me.Optime != nil && master.Optime != nil && me.State == 2 {
// MongoTimestamp type is int64 where the first 32bits are the unix timestamp
lag := int64(*master.OptimeDate>>32 - *me.OptimeDate>>32)
lag := int64(*master.Optime>>32 - *me.Optime>>32)
if lag < 0 {
returnVal.ReplLag = 0
} else {

View File

@@ -53,13 +53,13 @@ This plugin gathers the statistic data from MySQL server
## gather metrics from SHOW BINARY LOGS command output
gather_binary_logs = false
#
## gather metrics from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMARY_BY_TABLE
## gather metrics from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMART_BY_TABLE
gather_table_io_waits = false
#
## gather metrics from PERFORMANCE_SCHEMA.TABLE_LOCK_WAITS
gather_table_lock_waits = false
#
## gather metrics from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMARY_BY_INDEX_USAGE
## gather metrics from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMART_BY_INDEX_USAGE
gather_index_io_waits = false
#
## gather metrics from PERFORMANCE_SCHEMA.EVENT_WAITS

View File

@@ -97,12 +97,11 @@ func (n *Nginx) gatherUrl(addr *url.URL, acc telegraf.Accumulator) error {
if err != nil {
return err
}
data := strings.Fields(line)
data := strings.SplitN(strings.TrimSpace(line), " ", 3)
accepts, err := strconv.ParseUint(data[0], 10, 64)
if err != nil {
return err
}
handled, err := strconv.ParseUint(data[1], 10, 64)
if err != nil {
return err
@@ -117,7 +116,7 @@ func (n *Nginx) gatherUrl(addr *url.URL, acc telegraf.Accumulator) error {
if err != nil {
return err
}
data = strings.Fields(line)
data = strings.SplitN(strings.TrimSpace(line), " ", 6)
reading, err := strconv.ParseUint(data[1], 10, 64)
if err != nil {
return err

View File

@@ -13,18 +13,12 @@ import (
"github.com/stretchr/testify/require"
)
const nginxSampleResponse = `
const sampleResponse = `
Active connections: 585
server accepts handled requests
85340 85340 35085
Reading: 4 Writing: 135 Waiting: 446
`
const tengineSampleResponse = `
Active connections: 403
server accepts handled requests request_time
853 8533 3502 1546565864
Reading: 8 Writing: 125 Waiting: 946
`
// Verify that nginx tags are properly parsed based on the server
func TestNginxTags(t *testing.T) {
@@ -42,9 +36,7 @@ func TestNginxGeneratesMetrics(t *testing.T) {
var rsp string
if r.URL.Path == "/stub_status" {
rsp = nginxSampleResponse
} else if r.URL.Path == "/tengine_status" {
rsp = tengineSampleResponse
rsp = sampleResponse
} else {
panic("Cannot handle request")
}
@@ -57,20 +49,12 @@ func TestNginxGeneratesMetrics(t *testing.T) {
Urls: []string{fmt.Sprintf("%s/stub_status", ts.URL)},
}
nt := &Nginx{
Urls: []string{fmt.Sprintf("%s/tengine_status", ts.URL)},
}
var acc testutil.Accumulator
var acc_nginx testutil.Accumulator
var acc_tengine testutil.Accumulator
err := n.Gather(&acc)
require.NoError(t, err)
err_nginx := n.Gather(&acc_nginx)
err_tengine := nt.Gather(&acc_tengine)
require.NoError(t, err_nginx)
require.NoError(t, err_tengine)
fields_nginx := map[string]interface{}{
fields := map[string]interface{}{
"active": uint64(585),
"accepts": uint64(85340),
"handled": uint64(85340),
@@ -79,17 +63,6 @@ func TestNginxGeneratesMetrics(t *testing.T) {
"writing": uint64(135),
"waiting": uint64(446),
}
fields_tengine := map[string]interface{}{
"active": uint64(403),
"accepts": uint64(853),
"handled": uint64(8533),
"requests": uint64(3502),
"reading": uint64(8),
"writing": uint64(125),
"waiting": uint64(946),
}
addr, err := url.Parse(ts.URL)
if err != nil {
panic(err)
@@ -108,6 +81,5 @@ func TestNginxGeneratesMetrics(t *testing.T) {
}
tags := map[string]string{"server": host, "port": port}
acc_nginx.AssertContainsTaggedFields(t, "nginx", fields_nginx, tags)
acc_tengine.AssertContainsTaggedFields(t, "nginx", fields_tengine, tags)
acc.AssertContainsTaggedFields(t, "nginx", fields, tags)
}

View File

@@ -28,7 +28,7 @@ type Ping struct {
// Number of pings to send (ping -c <COUNT>)
Count int
// Ping timeout, in seconds. 0 means no timeout (ping -W <TIMEOUT>)
// Ping timeout, in seconds. 0 means no timeout (ping -t <TIMEOUT>)
Timeout float64
// Interface to send ping from (ping -I <INTERFACE>)
@@ -55,7 +55,7 @@ const sampleConfig = `
count = 1 # required
## interval, in s, at which to ping. 0 == default (ping -i <PING_INTERVAL>)
ping_interval = 0.0
## per-ping timeout, in s. 0 == no timeout (ping -W <TIMEOUT>)
## ping timeout, in s. 0 == no timeout (ping -W <TIMEOUT>)
timeout = 1.0
## interface to send ping from (ping -I <INTERFACE>)
interface = ""
@@ -76,8 +76,7 @@ func (p *Ping) Gather(acc telegraf.Accumulator) error {
go func(u string) {
defer wg.Done()
args := p.args(u)
totalTimeout := float64(p.Count)*p.Timeout + float64(p.Count-1)*p.PingInterval
out, err := p.pingHost(totalTimeout, args...)
out, err := p.pingHost(p.Timeout, args...)
if err != nil {
// Combine go err + stderr output
errorChannel <- errors.New(
@@ -139,8 +138,8 @@ func (p *Ping) args(url string) []string {
}
if p.Timeout > 0 {
switch runtime.GOOS {
case "darwin":
args = append(args, "-W", strconv.FormatFloat(p.Timeout/1000, 'f', 1, 64))
case "darwin", "freebsd":
args = append(args, "-t", strconv.FormatFloat(p.Timeout, 'f', 1, 64))
case "linux":
args = append(args, "-W", strconv.FormatFloat(p.Timeout, 'f', 1, 64))
default:

View File

@@ -10,7 +10,6 @@ import (
"io"
"math"
"mime"
"time"
"github.com/influxdata/telegraf"
@@ -75,13 +74,13 @@ func (p *PrometheusParser) Parse(buf []byte) ([]telegraf.Metric, error) {
if mf.GetType() == dto.MetricType_SUMMARY {
// summary metric
fields = makeQuantiles(m)
fields["count"] = float64(m.GetSummary().GetSampleCount())
fields["count"] = float64(m.GetHistogram().GetSampleCount())
fields["sum"] = float64(m.GetSummary().GetSampleSum())
} else if mf.GetType() == dto.MetricType_HISTOGRAM {
// historgram metric
fields = makeBuckets(m)
fields["count"] = float64(m.GetHistogram().GetSampleCount())
fields["sum"] = float64(m.GetHistogram().GetSampleSum())
fields["sum"] = float64(m.GetSummary().GetSampleSum())
} else {
// standard metric
@@ -89,13 +88,7 @@ func (p *PrometheusParser) Parse(buf []byte) ([]telegraf.Metric, error) {
}
// converting to telegraf metric
if len(fields) > 0 {
var t time.Time
if m.TimestampMs != nil && *m.TimestampMs > 0 {
t = time.Unix(0, *m.TimestampMs*1000000)
} else {
t = time.Now()
}
metric, err := telegraf.NewMetric(metricName, tags, fields, t)
metric, err := telegraf.NewMetric(metricName, tags, fields)
if err == nil {
metrics = append(metrics, metric)
}

View File

@@ -138,7 +138,7 @@ func TestParseValidPrometheus(t *testing.T) {
"0.5": 552048.506,
"0.9": 5.876804288e+06,
"0.99": 5.876804288e+06,
"count": 9.0,
"count": 0.0,
"sum": 1.8909097205e+07,
}, metrics[0].Fields())
assert.Equal(t, map[string]string{"handler": "prometheus"}, metrics[0].Tags())
@@ -151,7 +151,7 @@ func TestParseValidPrometheus(t *testing.T) {
assert.Equal(t, map[string]interface{}{
"500000": 2000.0,
"count": 2025.0,
"sum": 1.02726334e+08,
"sum": 0.0,
"250000": 1997.0,
"2e+06": 2012.0,
"4e+06": 2017.0,

View File

@@ -5,11 +5,9 @@ import (
"fmt"
"net/http"
"strconv"
"sync"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal/errchan"
"github.com/influxdata/telegraf/plugins/inputs"
)
@@ -104,7 +102,7 @@ type gatherFunc func(r *RabbitMQ, acc telegraf.Accumulator, errChan chan error)
var gatherFunctions = []gatherFunc{gatherOverview, gatherNodes, gatherQueues}
var sampleConfig = `
# url = "http://localhost:15672"
url = "http://localhost:15672" # required
# name = "rmq-server-1" # optional tag
# username = "guest"
# password = "guest"
@@ -131,24 +129,23 @@ func (r *RabbitMQ) Gather(acc telegraf.Accumulator) error {
}
}
var wg sync.WaitGroup
wg.Add(len(gatherFunctions))
errChan := errchan.New(len(gatherFunctions))
for _, f := range gatherFunctions {
go func(gf gatherFunc) {
defer wg.Done()
gf(r, acc, errChan.C)
}(f)
}
wg.Wait()
var errChan = make(chan error, len(gatherFunctions))
return errChan.Error()
for _, f := range gatherFunctions {
go f(r, acc, errChan)
}
for i := 1; i <= len(gatherFunctions); i++ {
err := <-errChan
if err != nil {
return err
}
}
return nil
}
func (r *RabbitMQ) requestJSON(u string, target interface{}) error {
if r.URL == "" {
r.URL = DefaultURL
}
u = fmt.Sprintf("%s%s", r.URL, u)
req, err := http.NewRequest("GET", u, nil)

View File

@@ -241,14 +241,10 @@ func gatherKeyspaceLine(
name string,
line string,
acc telegraf.Accumulator,
global_tags map[string]string,
tags map[string]string,
) {
if strings.Contains(line, "keys=") {
fields := make(map[string]interface{})
tags := make(map[string]string)
for k, v := range global_tags {
tags[k] = v
}
tags["database"] = name
dbparts := strings.Split(line, ",")
for _, dbp := range dbparts {

View File

@@ -35,7 +35,6 @@ func TestRedis_ParseMetrics(t *testing.T) {
err := gatherInfoOutput(rdr, &acc, tags)
require.NoError(t, err)
tags = map[string]string{"host": "redis.net", "role": "master"}
fields := map[string]interface{}{
"uptime": uint64(238),
"clients": uint64(1),
@@ -71,14 +70,13 @@ func TestRedis_ParseMetrics(t *testing.T) {
"used_cpu_user_children": float64(0.00),
"keyspace_hitrate": float64(0.50),
}
keyspaceTags := map[string]string{"host": "redis.net", "role": "master", "database": "db0"}
keyspaceFields := map[string]interface{}{
"avg_ttl": uint64(0),
"expires": uint64(0),
"keys": uint64(2),
}
acc.AssertContainsTaggedFields(t, "redis", fields, tags)
acc.AssertContainsTaggedFields(t, "redis_keyspace", keyspaceFields, keyspaceTags)
acc.AssertContainsTaggedFields(t, "redis_keyspace", keyspaceFields, tags)
}
const testOutput = `# Server

View File

@@ -1,47 +0,0 @@
# rollbar_webhooks
This is a Telegraf service plugin that listens for events kicked off by Rollbar Webhooks service and persists data from them into configured outputs. To set up the listener first generate the proper configuration:
```sh
$ telegraf -sample-config -input-filter rollbar_webhooks -output-filter influxdb > config.conf.new
```
Change the config file to point to the InfluxDB server you are using and adjust the settings to match your environment. Once that is complete:
```sh
$ cp config.conf.new /etc/telegraf/telegraf.conf
$ sudo service telegraf start
```
Once the server is running you should configure your Rollbar's Webhooks to point at the `rollbar_webhooks` service. To do this go to `rollbar.com/` and click `Settings > Notifications > Webhook`. In the resulting page set `URL` to `http://<my_ip>:1619`, and click on `Enable Webhook Integration`.
## Events
The titles of the following sections are links to the full payloads and details for each event. The body contains what information from the event is persisted. The format is as follows:
```
# TAGS
* 'tagKey' = `tagValue` type
# FIELDS
* 'fieldKey' = `fieldValue` type
```
The tag values and field values show the place on the incoming JSON object where the data is sourced from.
See [webhook doc](https://rollbar.com/docs/webhooks/)
#### `new_item` event
**Tags:**
* 'event' = `event.event_name` string
* 'environment' = `event.data.item.environment` string
* 'project_id = `event.data.item.project_id` int
* 'language' = `event.data.item.last_occurence.language` string
* 'level' = `event.data.item.last_occurence.level` string
**Fields:**
* 'id' = `event.data.item.id` int
#### `deploy` event
**Tags:**
* 'event' = `event.event_name` string
* 'environment' = `event.data.deploy.environment` string
* 'project_id = `event.data.deploy.project_id` int
**Fields:**
* 'id' = `event.data.item.id` int

View File

@@ -1,119 +0,0 @@
package rollbar_webhooks
import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
"time"
"github.com/gorilla/mux"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/inputs"
)
func init() {
inputs.Add("rollbar_webhooks", func() telegraf.Input { return NewRollbarWebhooks() })
}
type RollbarWebhooks struct {
ServiceAddress string
// Lock for the struct
sync.Mutex
// Events buffer to store events between Gather calls
events []Event
}
func NewRollbarWebhooks() *RollbarWebhooks {
return &RollbarWebhooks{}
}
func (rb *RollbarWebhooks) SampleConfig() string {
return `
## Address and port to host Webhook listener on
service_address = ":1619"
`
}
func (rb *RollbarWebhooks) Description() string {
return "A Rollbar Webhook Event collector"
}
func (rb *RollbarWebhooks) Gather(acc telegraf.Accumulator) error {
rb.Lock()
defer rb.Unlock()
for _, event := range rb.events {
acc.AddFields("rollbar_webhooks", event.Fields(), event.Tags(), time.Now())
}
rb.events = make([]Event, 0)
return nil
}
func (rb *RollbarWebhooks) Listen() {
r := mux.NewRouter()
r.HandleFunc("/", rb.eventHandler).Methods("POST")
err := http.ListenAndServe(fmt.Sprintf("%s", rb.ServiceAddress), r)
if err != nil {
log.Printf("Error starting server: %v", err)
}
}
func (rb *RollbarWebhooks) Start(_ telegraf.Accumulator) error {
go rb.Listen()
log.Printf("Started the rollbar_webhooks service on %s\n", rb.ServiceAddress)
return nil
}
func (rb *RollbarWebhooks) Stop() {
log.Println("Stopping the rbWebhooks service")
}
func (rb *RollbarWebhooks) eventHandler(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
data, err := ioutil.ReadAll(r.Body)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
dummyEvent := &DummyEvent{}
err = json.Unmarshal(data, dummyEvent)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
event, err := NewEvent(dummyEvent, data)
if err != nil {
w.WriteHeader(http.StatusOK)
return
}
rb.Lock()
rb.events = append(rb.events, event)
rb.Unlock()
w.WriteHeader(http.StatusOK)
}
func generateEvent(event Event, data []byte) (Event, error) {
err := json.Unmarshal(data, event)
if err != nil {
return nil, err
}
return event, nil
}
func NewEvent(dummyEvent *DummyEvent, data []byte) (Event, error) {
switch dummyEvent.EventName {
case "new_item":
return generateEvent(&NewItem{}, data)
case "deploy":
return generateEvent(&Deploy{}, data)
default:
return nil, errors.New("Not implemented type: " + dummyEvent.EventName)
}
}

View File

@@ -1,78 +0,0 @@
package rollbar_webhooks
import "strconv"
type Event interface {
Tags() map[string]string
Fields() map[string]interface{}
}
type DummyEvent struct {
EventName string `json:"event_name"`
}
type NewItemDataItemLastOccurence struct {
Language string `json:"language"`
Level string `json:"level"`
}
type NewItemDataItem struct {
Id int `json:"id"`
Environment string `json:"environment"`
ProjectId int `json:"project_id"`
LastOccurence NewItemDataItemLastOccurence `json:"last_occurrence"`
}
type NewItemData struct {
Item NewItemDataItem `json:"item"`
}
type NewItem struct {
EventName string `json:"event_name"`
Data NewItemData `json:"data"`
}
func (ni *NewItem) Tags() map[string]string {
return map[string]string{
"event": ni.EventName,
"environment": ni.Data.Item.Environment,
"project_id": strconv.Itoa(ni.Data.Item.ProjectId),
"language": ni.Data.Item.LastOccurence.Language,
"level": ni.Data.Item.LastOccurence.Level,
}
}
func (ni *NewItem) Fields() map[string]interface{} {
return map[string]interface{}{
"id": ni.Data.Item.Id,
}
}
type DeployDataDeploy struct {
Id int `json:"id"`
Environment string `json:"environment"`
ProjectId int `json:"project_id"`
}
type DeployData struct {
Deploy DeployDataDeploy `json:"deploy"`
}
type Deploy struct {
EventName string `json:"event_name"`
Data DeployData `json:"data"`
}
func (ni *Deploy) Tags() map[string]string {
return map[string]string{
"event": ni.EventName,
"environment": ni.Data.Deploy.Environment,
"project_id": strconv.Itoa(ni.Data.Deploy.ProjectId),
}
}
func (ni *Deploy) Fields() map[string]interface{} {
return map[string]interface{}{
"id": ni.Data.Deploy.Id,
}
}

View File

@@ -1,96 +0,0 @@
package rollbar_webhooks
func NewItemJSON() string {
return `
{
"event_name": "new_item",
"data": {
"item": {
"public_item_id": null,
"integrations_data": {},
"last_activated_timestamp": 1382655421,
"unique_occurrences": null,
"id": 272716944,
"environment": "production",
"title": "testing aobg98wrwe",
"last_occurrence_id": 481761639,
"last_occurrence_timestamp": 1382655421,
"platform": 0,
"first_occurrence_timestamp": 1382655421,
"project_id": 90,
"resolved_in_version": null,
"status": 1,
"hash": "c595b2ae0af9b397bb6bdafd57104ac4d5f6b382",
"last_occurrence": {
"body": {
"message": {
"body": "testing aobg98wrwe"
}
},
"uuid": "d2036647-e0b7-4cad-bc98-934831b9b6d1",
"language": "python",
"level": "error",
"timestamp": 1382655421,
"server": {
"host": "dev",
"argv": [
""
]
},
"environment": "production",
"framework": "unknown",
"notifier": {
"version": "0.5.12",
"name": "pyrollbar"
},
"metadata": {
"access_token": "",
"debug": {
"routes": {
"start_time": 1382212080401,
"counters": {
"post_item": 3274122
}
}
},
"customer_timestamp": 1382655421,
"api_server_hostname": "web6"
}
},
"framework": 0,
"total_occurrences": 1,
"level": 40,
"counter": 4,
"first_occurrence_id": 481761639,
"activating_occurrence_id": 481761639
}
}
}`
}
func DeployJSON() string {
return `
{
"event_name": "deploy",
"data": {
"deploy": {
"comment": "deploying webs",
"user_id": 1,
"finish_time": 1382656039,
"start_time": 1382656038,
"id": 187585,
"environment": "production",
"project_id": 90,
"local_username": "brian",
"revision": "e4b9b7db860b2e5ac799f8c06b9498b71ab270bb"
}
}
}`
}
func UnknowJSON() string {
return `
{
"event_name": "roger"
}`
}

View File

@@ -1,74 +0,0 @@
package rollbar_webhooks
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/influxdata/telegraf/testutil"
)
func postWebhooks(rb *RollbarWebhooks, eventBody string) *httptest.ResponseRecorder {
req, _ := http.NewRequest("POST", "/", strings.NewReader(eventBody))
w := httptest.NewRecorder()
w.Code = 500
rb.eventHandler(w, req)
return w
}
func TestNewItem(t *testing.T) {
var acc testutil.Accumulator
rb := NewRollbarWebhooks()
resp := postWebhooks(rb, NewItemJSON())
if resp.Code != http.StatusOK {
t.Errorf("POST new_item returned HTTP status code %v.\nExpected %v", resp.Code, http.StatusOK)
}
rb.Gather(&acc)
fields := map[string]interface{}{
"id": 272716944,
}
tags := map[string]string{
"event": "new_item",
"environment": "production",
"project_id": "90",
"language": "python",
"level": "error",
}
acc.AssertContainsTaggedFields(t, "rollbar_webhooks", fields, tags)
}
func TestDeploy(t *testing.T) {
var acc testutil.Accumulator
rb := NewRollbarWebhooks()
resp := postWebhooks(rb, DeployJSON())
if resp.Code != http.StatusOK {
t.Errorf("POST deploy returned HTTP status code %v.\nExpected %v", resp.Code, http.StatusOK)
}
rb.Gather(&acc)
fields := map[string]interface{}{
"id": 187585,
}
tags := map[string]string{
"event": "deploy",
"environment": "production",
"project_id": "90",
}
acc.AssertContainsTaggedFields(t, "rollbar_webhooks", fields, tags)
}
func TestUnknowItem(t *testing.T) {
rb := NewRollbarWebhooks()
resp := postWebhooks(rb, UnknowJSON())
if resp.Code != http.StatusOK {
t.Errorf("POST unknow returned HTTP status code %v.\nExpected %v", resp.Code, http.StatusOK)
}
}

View File

@@ -27,8 +27,7 @@ const (
defaultSeparator = "_"
)
var dropwarn = "ERROR: statsd message queue full. " +
"We have dropped %d messages so far. " +
var dropwarn = "ERROR: Message queue full. Discarding line [%s] " +
"You may want to increase allowed_pending_messages in the config\n"
var prevInstance *Statsd
@@ -66,8 +65,6 @@ type Statsd struct {
sync.Mutex
wg sync.WaitGroup
// drops tracks the number of dropped metrics.
drops int
// Channel for all incoming statsd packets
in chan []byte
@@ -294,10 +291,7 @@ func (s *Statsd) udpListen() error {
select {
case s.in <- bufCopy:
default:
s.drops++
if s.drops == 1 || s.drops%s.AllowedPendingMessages == 0 {
log.Printf(dropwarn, s.drops)
}
log.Printf(dropwarn, string(buf[:n]))
}
}
}

View File

@@ -1,225 +0,0 @@
# Kernel VMStat Input Plugin
The kernel_vmstat plugin gathers virtual memory statistics
by reading /proc/vmstat. For a full list of available fields see the
/proc/vmstat section of the [proc man page](http://man7.org/linux/man-pages/man5/proc.5.html).
For a better idea of what each field represents, see the
[vmstat man page](http://linux.die.net/man/8/vmstat).
```
/proc/vmstat
kernel/system statistics. Common entries include (from http://www.linuxinsight.com/proc_vmstat.html):
Number of pages that are dirty, under writeback or unstable:
nr_dirty 1550
nr_writeback 0
nr_unstable 0
Number of pages allocated to page tables, mapped by files or allocated by the kernel slab allocator:
nr_page_table_pages 699
nr_mapped 139596
nr_slab 42723
Number of pageins and pageouts (since the last boot):
pgpgin 33754195
pgpgout 38985992
Number of swapins and swapouts (since the last boot):
pswpin 2473
pswpout 2995
Number of page allocations per zone (since the last boot):
pgalloc_high 0
pgalloc_normal 110123213
pgalloc_dma32 0
pgalloc_dma 415219
Number of page frees, activations and deactivations (since the last boot):
pgfree 110549163
pgactivate 4509729
pgdeactivate 2136215
Number of minor and major page faults (since the last boot):
pgfault 80663722
pgmajfault 49813
Number of page refills (per zone, since the last boot):
pgrefill_high 0
pgrefill_normal 5817500
pgrefill_dma32 0
pgrefill_dma 149176
Number of page steals (per zone, since the last boot):
pgsteal_high 0
pgsteal_normal 10421346
pgsteal_dma32 0
pgsteal_dma 142196
Number of pages scanned by the kswapd daemon (per zone, since the last boot):
pgscan_kswapd_high 0
pgscan_kswapd_normal 10491424
pgscan_kswapd_dma32 0
pgscan_kswapd_dma 156130
Number of pages reclaimed directly (per zone, since the last boot):
pgscan_direct_high 0
pgscan_direct_normal 11904
pgscan_direct_dma32 0
pgscan_direct_dma 225
Number of pages reclaimed via inode freeing (since the last boot):
pginodesteal 11
Number of slab objects scanned (since the last boot):
slabs_scanned 8926976
Number of pages reclaimed by kswapd (since the last boot):
kswapd_steal 10551674
Number of pages reclaimed by kswapd via inode freeing (since the last boot):
kswapd_inodesteal 338730
Number of kswapd's calls to page reclaim (since the last boot):
pageoutrun 181908
Number of direct reclaim calls (since the last boot):
allocstall 160
Miscellaneous statistics:
pgrotated 3781
nr_bounce 0
```
### Configuration:
```toml
# Get kernel statistics from /proc/vmstat
[[inputs.kernel_vmstat]]
# no configuration
```
### Measurements & Fields:
- kernel_vmstat
- nr_free_pages (integer, `nr_free_pages`)
- nr_inactive_anon (integer, `nr_inactive_anon`)
- nr_active_anon (integer, `nr_active_anon`)
- nr_inactive_file (integer, `nr_inactive_file`)
- nr_active_file (integer, `nr_active_file`)
- nr_unevictable (integer, `nr_unevictable`)
- nr_mlock (integer, `nr_mlock`)
- nr_anon_pages (integer, `nr_anon_pages`)
- nr_mapped (integer, `nr_mapped`)
- nr_file_pages (integer, `nr_file_pages`)
- nr_dirty (integer, `nr_dirty`)
- nr_writeback (integer, `nr_writeback`)
- nr_slab_reclaimable (integer, `nr_slab_reclaimable`)
- nr_slab_unreclaimable (integer, `nr_slab_unreclaimable`)
- nr_page_table_pages (integer, `nr_page_table_pages`)
- nr_kernel_stack (integer, `nr_kernel_stack`)
- nr_unstable (integer, `nr_unstable`)
- nr_bounce (integer, `nr_bounce`)
- nr_vmscan_write (integer, `nr_vmscan_write`)
- nr_writeback_temp (integer, `nr_writeback_temp`)
- nr_isolated_anon (integer, `nr_isolated_anon`)
- nr_isolated_file (integer, `nr_isolated_file`)
- nr_shmem (integer, `nr_shmem`)
- numa_hit (integer, `numa_hit`)
- numa_miss (integer, `numa_miss`)
- numa_foreign (integer, `numa_foreign`)
- numa_interleave (integer, `numa_interleave`)
- numa_local (integer, `numa_local`)
- numa_other (integer, `numa_other`)
- nr_anon_transparent_hugepages (integer, `nr_anon_transparent_hugepages`)
- pgpgin (integer, `pgpgin`)
- pgpgout (integer, `pgpgout`)
- pswpin (integer, `pswpin`)
- pswpout (integer, `pswpout`)
- pgalloc_dma (integer, `pgalloc_dma`)
- pgalloc_dma32 (integer, `pgalloc_dma32`)
- pgalloc_normal (integer, `pgalloc_normal`)
- pgalloc_movable (integer, `pgalloc_movable`)
- pgfree (integer, `pgfree`)
- pgactivate (integer, `pgactivate`)
- pgdeactivate (integer, `pgdeactivate`)
- pgfault (integer, `pgfault`)
- pgmajfault (integer, `pgmajfault`)
- pgrefill_dma (integer, `pgrefill_dma`)
- pgrefill_dma32 (integer, `pgrefill_dma32`)
- pgrefill_normal (integer, `pgrefill_normal`)
- pgrefill_movable (integer, `pgrefill_movable`)
- pgsteal_dma (integer, `pgsteal_dma`)
- pgsteal_dma32 (integer, `pgsteal_dma32`)
- pgsteal_normal (integer, `pgsteal_normal`)
- pgsteal_movable (integer, `pgsteal_movable`)
- pgscan_kswapd_dma (integer, `pgscan_kswapd_dma`)
- pgscan_kswapd_dma32 (integer, `pgscan_kswapd_dma32`)
- pgscan_kswapd_normal (integer, `pgscan_kswapd_normal`)
- pgscan_kswapd_movable (integer, `pgscan_kswapd_movable`)
- pgscan_direct_dma (integer, `pgscan_direct_dma`)
- pgscan_direct_dma32 (integer, `pgscan_direct_dma32`)
- pgscan_direct_normal (integer, `pgscan_direct_normal`)
- pgscan_direct_movable (integer, `pgscan_direct_movable`)
- zone_reclaim_failed (integer, `zone_reclaim_failed`)
- pginodesteal (integer, `pginodesteal`)
- slabs_scanned (integer, `slabs_scanned`)
- kswapd_steal (integer, `kswapd_steal`)
- kswapd_inodesteal (integer, `kswapd_inodesteal`)
- kswapd_low_wmark_hit_quickly (integer, `kswapd_low_wmark_hit_quickly`)
- kswapd_high_wmark_hit_quickly (integer, `kswapd_high_wmark_hit_quickly`)
- kswapd_skip_congestion_wait (integer, `kswapd_skip_congestion_wait`)
- pageoutrun (integer, `pageoutrun`)
- allocstall (integer, `allocstall`)
- pgrotated (integer, `pgrotated`)
- compact_blocks_moved (integer, `compact_blocks_moved`)
- compact_pages_moved (integer, `compact_pages_moved`)
- compact_pagemigrate_failed (integer, `compact_pagemigrate_failed`)
- compact_stall (integer, `compact_stall`)
- compact_fail (integer, `compact_fail`)
- compact_success (integer, `compact_success`)
- htlb_buddy_alloc_success (integer, `htlb_buddy_alloc_success`)
- htlb_buddy_alloc_fail (integer, `htlb_buddy_alloc_fail`)
- unevictable_pgs_culled (integer, `unevictable_pgs_culled`)
- unevictable_pgs_scanned (integer, `unevictable_pgs_scanned`)
- unevictable_pgs_rescued (integer, `unevictable_pgs_rescued`)
- unevictable_pgs_mlocked (integer, `unevictable_pgs_mlocked`)
- unevictable_pgs_munlocked (integer, `unevictable_pgs_munlocked`)
- unevictable_pgs_cleared (integer, `unevictable_pgs_cleared`)
- unevictable_pgs_stranded (integer, `unevictable_pgs_stranded`)
- unevictable_pgs_mlockfreed (integer, `unevictable_pgs_mlockfreed`)
- thp_fault_alloc (integer, `thp_fault_alloc`)
- thp_fault_fallback (integer, `thp_fault_fallback`)
- thp_collapse_alloc (integer, `thp_collapse_alloc`)
- thp_collapse_alloc_failed (integer, `thp_collapse_alloc_failed`)
- thp_split (integer, `thp_split`)
### Tags:
None
### Example Output:
```
$ telegraf -config ~/ws/telegraf.conf -input-filter kernel_vmstat -test
* Plugin: kernel_vmstat, Collection 1
> kernel_vmstat allocstall=81496i,compact_blocks_moved=238196i,compact_fail=135220i,compact_pagemigrate_failed=0i,compact_pages_moved=6370588i,compact_stall=142092i,compact_success=6872i,htlb_buddy_alloc_fail=0i,htlb_buddy_alloc_success=0i,kswapd_high_wmark_hit_quickly=25439i,kswapd_inodesteal=29770874i,kswapd_low_wmark_hit_quickly=8756i,kswapd_skip_congestion_wait=0i,kswapd_steal=291534428i,nr_active_anon=2515657i,nr_active_file=2244914i,nr_anon_pages=1358675i,nr_anon_transparent_hugepages=2034i,nr_bounce=0i,nr_dirty=5690i,nr_file_pages=5153546i,nr_free_pages=78730i,nr_inactive_anon=426259i,nr_inactive_file=2366791i,nr_isolated_anon=0i,nr_isolated_file=0i,nr_kernel_stack=579i,nr_mapped=558821i,nr_mlock=0i,nr_page_table_pages=11115i,nr_shmem=541689i,nr_slab_reclaimable=459806i,nr_slab_unreclaimable=47859i,nr_unevictable=0i,nr_unstable=0i,nr_vmscan_write=6206i,nr_writeback=0i,nr_writeback_temp=0i,numa_foreign=0i,numa_hit=5113399878i,numa_interleave=35793i,numa_local=5113399878i,numa_miss=0i,numa_other=0i,pageoutrun=505006i,pgactivate=375664931i,pgalloc_dma=0i,pgalloc_dma32=122480220i,pgalloc_movable=0i,pgalloc_normal=5233176719i,pgdeactivate=122735906i,pgfault=8699921410i,pgfree=5359765021i,pginodesteal=9188431i,pgmajfault=122210i,pgpgin=219717626i,pgpgout=3495885510i,pgrefill_dma=0i,pgrefill_dma32=1180010i,pgrefill_movable=0i,pgrefill_normal=119866676i,pgrotated=60620i,pgscan_direct_dma=0i,pgscan_direct_dma32=12256i,pgscan_direct_movable=0i,pgscan_direct_normal=31501600i,pgscan_kswapd_dma=0i,pgscan_kswapd_dma32=4480608i,pgscan_kswapd_movable=0i,pgscan_kswapd_normal=287857984i,pgsteal_dma=0i,pgsteal_dma32=4466436i,pgsteal_movable=0i,pgsteal_normal=318463755i,pswpin=2092i,pswpout=6206i,slabs_scanned=93775616i,thp_collapse_alloc=24857i,thp_collapse_alloc_failed=102214i,thp_fault_alloc=346219i,thp_fault_fallback=895453i,thp_split=9817i,unevictable_pgs_cleared=0i,unevictable_pgs_culled=1531i,unevictable_pgs_mlocked=6988i,unevictable_pgs_mlockfreed=0i,unevictable_pgs_munlocked=6988i,unevictable_pgs_rescued=5426i,unevictable_pgs_scanned=0i,unevictable_pgs_stranded=0i,zone_reclaim_failed=0i 1459455200071462843
```

View File

@@ -18,7 +18,7 @@ It is supposed to be used to monitor actual memory usage in a cross platform fas
designed for informational purposes only.
- **free**: memory not being used at all (zeroed) that is readily available; note
that this doesn't reflect the actual memory available (use 'available' instead).
- **used_percent**: the percentage usage calculated as `used / total * 100`
- **used_percent**: the percentage usage calculated as `(total - used) / total * 100`
## Measurements:
#### Raw Memory measurements:

View File

@@ -44,35 +44,35 @@ func (k *Kernel) Gather(acc telegraf.Accumulator) error {
for i, field := range dataFields {
switch {
case bytes.Equal(field, interrupts):
m, err := strconv.ParseInt(string(dataFields[i+1]), 10, 64)
m, err := strconv.Atoi(string(dataFields[i+1]))
if err != nil {
return err
}
fields["interrupts"] = int64(m)
case bytes.Equal(field, context_switches):
m, err := strconv.ParseInt(string(dataFields[i+1]), 10, 64)
m, err := strconv.Atoi(string(dataFields[i+1]))
if err != nil {
return err
}
fields["context_switches"] = int64(m)
case bytes.Equal(field, processes_forked):
m, err := strconv.ParseInt(string(dataFields[i+1]), 10, 64)
m, err := strconv.Atoi(string(dataFields[i+1]))
if err != nil {
return err
}
fields["processes_forked"] = int64(m)
case bytes.Equal(field, boot_time):
m, err := strconv.ParseInt(string(dataFields[i+1]), 10, 64)
m, err := strconv.Atoi(string(dataFields[i+1]))
if err != nil {
return err
}
fields["boot_time"] = int64(m)
case bytes.Equal(field, disk_pages):
in, err := strconv.ParseInt(string(dataFields[i+1]), 10, 64)
in, err := strconv.Atoi(string(dataFields[i+1]))
if err != nil {
return err
}
out, err := strconv.ParseInt(string(dataFields[i+2]), 10, 64)
out, err := strconv.Atoi(string(dataFields[i+2]))
if err != nil {
return err
}

View File

@@ -1,78 +0,0 @@
// +build linux
package system
import (
"bytes"
"fmt"
"io/ioutil"
"os"
"strconv"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/inputs"
)
type KernelVmstat struct {
statFile string
}
func (k *KernelVmstat) Description() string {
return "Get kernel statistics from /proc/vmstat"
}
func (k *KernelVmstat) SampleConfig() string {
return ""
}
func (k *KernelVmstat) Gather(acc telegraf.Accumulator) error {
data, err := k.getProcVmstat()
if err != nil {
return err
}
fields := make(map[string]interface{})
dataFields := bytes.Fields(data)
for i, field := range dataFields {
// dataFields is an array of {"stat1_name", "stat1_value", "stat2_name",
// "stat2_value", ...}
// We only want the even number index as that contain the stat name.
if i%2 == 0 {
// Convert the stat value into an integer.
m, err := strconv.Atoi(string(dataFields[i+1]))
if err != nil {
return err
}
fields[string(field)] = int64(m)
}
}
acc.AddFields("kernel_vmstat", fields, map[string]string{})
return nil
}
func (k *KernelVmstat) getProcVmstat() ([]byte, error) {
if _, err := os.Stat(k.statFile); os.IsNotExist(err) {
return nil, fmt.Errorf("kernel_vmstat: %s does not exist!", k.statFile)
} else if err != nil {
return nil, err
}
data, err := ioutil.ReadFile(k.statFile)
if err != nil {
return nil, err
}
return data, nil
}
func init() {
inputs.Add("kernel_vmstat", func() telegraf.Input {
return &KernelVmstat{
statFile: "/proc/vmstat",
}
})
}

View File

@@ -1,315 +0,0 @@
// +build linux
package system
import (
"io/ioutil"
"os"
"testing"
"github.com/influxdata/telegraf/testutil"
"github.com/stretchr/testify/assert"
)
func TestFullVmStatProcFile(t *testing.T) {
tmpfile := makeFakeStatFile([]byte(vmStatFile_Full))
defer os.Remove(tmpfile)
k := KernelVmstat{
statFile: tmpfile,
}
acc := testutil.Accumulator{}
err := k.Gather(&acc)
assert.NoError(t, err)
fields := map[string]interface{}{
"nr_free_pages": int64(78730),
"nr_inactive_anon": int64(426259),
"nr_active_anon": int64(2515657),
"nr_inactive_file": int64(2366791),
"nr_active_file": int64(2244914),
"nr_unevictable": int64(0),
"nr_mlock": int64(0),
"nr_anon_pages": int64(1358675),
"nr_mapped": int64(558821),
"nr_file_pages": int64(5153546),
"nr_dirty": int64(5690),
"nr_writeback": int64(0),
"nr_slab_reclaimable": int64(459806),
"nr_slab_unreclaimable": int64(47859),
"nr_page_table_pages": int64(11115),
"nr_kernel_stack": int64(579),
"nr_unstable": int64(0),
"nr_bounce": int64(0),
"nr_vmscan_write": int64(6206),
"nr_writeback_temp": int64(0),
"nr_isolated_anon": int64(0),
"nr_isolated_file": int64(0),
"nr_shmem": int64(541689),
"numa_hit": int64(5113399878),
"numa_miss": int64(0),
"numa_foreign": int64(0),
"numa_interleave": int64(35793),
"numa_local": int64(5113399878),
"numa_other": int64(0),
"nr_anon_transparent_hugepages": int64(2034),
"pgpgin": int64(219717626),
"pgpgout": int64(3495885510),
"pswpin": int64(2092),
"pswpout": int64(6206),
"pgalloc_dma": int64(0),
"pgalloc_dma32": int64(122480220),
"pgalloc_normal": int64(5233176719),
"pgalloc_movable": int64(0),
"pgfree": int64(5359765021),
"pgactivate": int64(375664931),
"pgdeactivate": int64(122735906),
"pgfault": int64(8699921410),
"pgmajfault": int64(122210),
"pgrefill_dma": int64(0),
"pgrefill_dma32": int64(1180010),
"pgrefill_normal": int64(119866676),
"pgrefill_movable": int64(0),
"pgsteal_dma": int64(0),
"pgsteal_dma32": int64(4466436),
"pgsteal_normal": int64(318463755),
"pgsteal_movable": int64(0),
"pgscan_kswapd_dma": int64(0),
"pgscan_kswapd_dma32": int64(4480608),
"pgscan_kswapd_normal": int64(287857984),
"pgscan_kswapd_movable": int64(0),
"pgscan_direct_dma": int64(0),
"pgscan_direct_dma32": int64(12256),
"pgscan_direct_normal": int64(31501600),
"pgscan_direct_movable": int64(0),
"zone_reclaim_failed": int64(0),
"pginodesteal": int64(9188431),
"slabs_scanned": int64(93775616),
"kswapd_steal": int64(291534428),
"kswapd_inodesteal": int64(29770874),
"kswapd_low_wmark_hit_quickly": int64(8756),
"kswapd_high_wmark_hit_quickly": int64(25439),
"kswapd_skip_congestion_wait": int64(0),
"pageoutrun": int64(505006),
"allocstall": int64(81496),
"pgrotated": int64(60620),
"compact_blocks_moved": int64(238196),
"compact_pages_moved": int64(6370588),
"compact_pagemigrate_failed": int64(0),
"compact_stall": int64(142092),
"compact_fail": int64(135220),
"compact_success": int64(6872),
"htlb_buddy_alloc_success": int64(0),
"htlb_buddy_alloc_fail": int64(0),
"unevictable_pgs_culled": int64(1531),
"unevictable_pgs_scanned": int64(0),
"unevictable_pgs_rescued": int64(5426),
"unevictable_pgs_mlocked": int64(6988),
"unevictable_pgs_munlocked": int64(6988),
"unevictable_pgs_cleared": int64(0),
"unevictable_pgs_stranded": int64(0),
"unevictable_pgs_mlockfreed": int64(0),
"thp_fault_alloc": int64(346219),
"thp_fault_fallback": int64(895453),
"thp_collapse_alloc": int64(24857),
"thp_collapse_alloc_failed": int64(102214),
"thp_split": int64(9817),
}
acc.AssertContainsFields(t, "kernel_vmstat", fields)
}
func TestPartialVmStatProcFile(t *testing.T) {
tmpfile := makeFakeStatFile([]byte(vmStatFile_Partial))
defer os.Remove(tmpfile)
k := KernelVmstat{
statFile: tmpfile,
}
acc := testutil.Accumulator{}
err := k.Gather(&acc)
assert.NoError(t, err)
fields := map[string]interface{}{
"unevictable_pgs_culled": int64(1531),
"unevictable_pgs_scanned": int64(0),
"unevictable_pgs_rescued": int64(5426),
"unevictable_pgs_mlocked": int64(6988),
"unevictable_pgs_munlocked": int64(6988),
"unevictable_pgs_cleared": int64(0),
"unevictable_pgs_stranded": int64(0),
"unevictable_pgs_mlockfreed": int64(0),
"thp_fault_alloc": int64(346219),
"thp_fault_fallback": int64(895453),
"thp_collapse_alloc": int64(24857),
"thp_collapse_alloc_failed": int64(102214),
"thp_split": int64(9817),
}
acc.AssertContainsFields(t, "kernel_vmstat", fields)
}
func TestInvalidVmStatProcFile1(t *testing.T) {
tmpfile := makeFakeStatFile([]byte(vmStatFile_Invalid))
defer os.Remove(tmpfile)
k := KernelVmstat{
statFile: tmpfile,
}
acc := testutil.Accumulator{}
err := k.Gather(&acc)
assert.Error(t, err)
}
func TestNoVmStatProcFile(t *testing.T) {
tmpfile := makeFakeStatFile([]byte(vmStatFile_Invalid))
os.Remove(tmpfile)
k := KernelVmstat{
statFile: tmpfile,
}
acc := testutil.Accumulator{}
err := k.Gather(&acc)
assert.Error(t, err)
assert.Contains(t, err.Error(), "does not exist")
}
const vmStatFile_Full = `nr_free_pages 78730
nr_inactive_anon 426259
nr_active_anon 2515657
nr_inactive_file 2366791
nr_active_file 2244914
nr_unevictable 0
nr_mlock 0
nr_anon_pages 1358675
nr_mapped 558821
nr_file_pages 5153546
nr_dirty 5690
nr_writeback 0
nr_slab_reclaimable 459806
nr_slab_unreclaimable 47859
nr_page_table_pages 11115
nr_kernel_stack 579
nr_unstable 0
nr_bounce 0
nr_vmscan_write 6206
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 541689
numa_hit 5113399878
numa_miss 0
numa_foreign 0
numa_interleave 35793
numa_local 5113399878
numa_other 0
nr_anon_transparent_hugepages 2034
pgpgin 219717626
pgpgout 3495885510
pswpin 2092
pswpout 6206
pgalloc_dma 0
pgalloc_dma32 122480220
pgalloc_normal 5233176719
pgalloc_movable 0
pgfree 5359765021
pgactivate 375664931
pgdeactivate 122735906
pgfault 8699921410
pgmajfault 122210
pgrefill_dma 0
pgrefill_dma32 1180010
pgrefill_normal 119866676
pgrefill_movable 0
pgsteal_dma 0
pgsteal_dma32 4466436
pgsteal_normal 318463755
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_dma32 4480608
pgscan_kswapd_normal 287857984
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 12256
pgscan_direct_normal 31501600
pgscan_direct_movable 0
zone_reclaim_failed 0
pginodesteal 9188431
slabs_scanned 93775616
kswapd_steal 291534428
kswapd_inodesteal 29770874
kswapd_low_wmark_hit_quickly 8756
kswapd_high_wmark_hit_quickly 25439
kswapd_skip_congestion_wait 0
pageoutrun 505006
allocstall 81496
pgrotated 60620
compact_blocks_moved 238196
compact_pages_moved 6370588
compact_pagemigrate_failed 0
compact_stall 142092
compact_fail 135220
compact_success 6872
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 1531
unevictable_pgs_scanned 0
unevictable_pgs_rescued 5426
unevictable_pgs_mlocked 6988
unevictable_pgs_munlocked 6988
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0
thp_fault_alloc 346219
thp_fault_fallback 895453
thp_collapse_alloc 24857
thp_collapse_alloc_failed 102214
thp_split 9817`
const vmStatFile_Partial = `unevictable_pgs_culled 1531
unevictable_pgs_scanned 0
unevictable_pgs_rescued 5426
unevictable_pgs_mlocked 6988
unevictable_pgs_munlocked 6988
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0
thp_fault_alloc 346219
thp_fault_fallback 895453
thp_collapse_alloc 24857
thp_collapse_alloc_failed 102214
thp_split 9817`
// invalid thp_split measurement
const vmStatFile_Invalid = `unevictable_pgs_culled 1531
unevictable_pgs_scanned 0
unevictable_pgs_rescued 5426
unevictable_pgs_mlocked 6988
unevictable_pgs_munlocked 6988
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0
thp_fault_alloc 346219
thp_fault_fallback 895453
thp_collapse_alloc 24857
thp_collapse_alloc_failed 102214
thp_split abcd`
func makeFakeVmStatFile(content []byte) string {
tmpfile, err := ioutil.TempFile("", "kernel_vmstat_test")
if err != nil {
panic(err)
}
if _, err := tmpfile.Write(content); err != nil {
panic(err)
}
if err := tmpfile.Close(); err != nil {
panic(err)
}
return tmpfile.Name()
}

View File

@@ -30,8 +30,6 @@ func (s *MemStats) Gather(acc telegraf.Accumulator) error {
"free": vm.Free,
"cached": vm.Cached,
"buffered": vm.Buffers,
"active": vm.Active,
"inactive": vm.Inactive,
"used_percent": 100 * float64(vm.Used) / float64(vm.Total),
"available_percent": 100 * float64(vm.Available) / float64(vm.Total),
}

View File

@@ -19,8 +19,8 @@ func TestMemStats(t *testing.T) {
Available: 7600,
Used: 5000,
Free: 1235,
Active: 8134,
Inactive: 1124,
// Active: 8134,
// Inactive: 1124,
// Buffers: 771,
// Cached: 4312,
// Wired: 134,
@@ -52,8 +52,6 @@ func TestMemStats(t *testing.T) {
"free": uint64(1235),
"cached": uint64(0),
"buffered": uint64(0),
"active": uint64(8134),
"inactive": uint64(1124),
}
acc.AssertContainsTaggedFields(t, "mem", memfields, make(map[string]string))

View File

@@ -9,7 +9,7 @@ import (
"log"
"os"
"os/exec"
"path/filepath"
"path"
"runtime"
"strconv"
@@ -19,7 +19,7 @@ import (
type Processes struct {
execPS func() ([]byte, error)
readProcFile func(filename string) ([]byte, error)
readProcFile func(statFile string) ([]byte, error)
forcePS bool
forceProc bool
@@ -128,15 +128,18 @@ func (p *Processes) gatherFromPS(fields map[string]interface{}) error {
// get process states from /proc/(pid)/stat files
func (p *Processes) gatherFromProc(fields map[string]interface{}) error {
filenames, err := filepath.Glob("/proc/[0-9]*/stat")
files, err := ioutil.ReadDir("/proc")
if err != nil {
return err
}
for _, filename := range filenames {
_, err := os.Stat(filename)
for _, file := range files {
if !file.IsDir() {
continue
}
data, err := p.readProcFile(filename)
statFile := path.Join("/proc", file.Name(), "stat")
data, err := p.readProcFile(statFile)
if err != nil {
return err
}
@@ -153,7 +156,7 @@ func (p *Processes) gatherFromProc(fields map[string]interface{}) error {
stats := bytes.Fields(data)
if len(stats) < 3 {
return fmt.Errorf("Something is terribly wrong with %s", filename)
return fmt.Errorf("Something is terribly wrong with %s", statFile)
}
switch stats[0][0] {
case 'R':
@@ -170,7 +173,7 @@ func (p *Processes) gatherFromProc(fields map[string]interface{}) error {
fields["paging"] = fields["paging"].(int64) + int64(1)
default:
log.Printf("processes: Unknown state [ %s ] in file %s",
string(stats[0][0]), filename)
string(stats[0][0]), statFile)
}
fields["total"] = fields["total"].(int64) + int64(1)
@@ -184,12 +187,15 @@ func (p *Processes) gatherFromProc(fields map[string]interface{}) error {
return nil
}
func readProcFile(filename string) ([]byte, error) {
data, err := ioutil.ReadFile(filename)
func readProcFile(statFile string) ([]byte, error) {
if _, err := os.Stat(statFile); os.IsNotExist(err) {
return nil, nil
} else if err != nil {
return nil, err
}
data, err := ioutil.ReadFile(statFile)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}

View File

@@ -84,10 +84,10 @@ func (s *systemPS) DiskUsage(
mountpoint := os.Getenv("HOST_MOUNT_PREFIX") + p.Mountpoint
if _, err := os.Stat(mountpoint); err == nil {
du, err := disk.Usage(mountpoint)
du.Path = p.Mountpoint
if err != nil {
return nil, err
}
du.Path = p.Mountpoint
// If the mount point is a member of the exclude set,
// don't gather info on it.
_, ok := fstypeExcludeSet[p.Fstype]

View File

@@ -95,7 +95,6 @@ func (t *Tail) Start(acc telegraf.Accumulator) error {
continue
}
// create a goroutine for each "tailer"
t.wg.Add(1)
go t.receiver(tailer)
t.tailers = append(t.tailers, tailer)
}
@@ -110,6 +109,7 @@ func (t *Tail) Start(acc telegraf.Accumulator) error {
// this is launched as a goroutine to continuously watch a tailed logfile
// for changes, parse any incoming msgs, and add to the accumulator.
func (t *Tail) receiver(tailer *tail.Tail) {
t.wg.Add(1)
defer t.wg.Done()
var m telegraf.Metric

View File

@@ -32,8 +32,7 @@ func TestTailFromBeginning(t *testing.T) {
_, err = tmpfile.WriteString("cpu,mytag=foo usage_idle=100\n")
require.NoError(t, err)
require.NoError(t, tt.Gather(&acc))
// arbitrary sleep to wait for message to show up
time.Sleep(time.Millisecond * 250)
time.Sleep(time.Millisecond * 50)
acc.AssertContainsTaggedFields(t, "cpu",
map[string]interface{}{

View File

@@ -29,8 +29,6 @@ type TcpListener struct {
// is an available bool in accept, then we are below the maximum and can
// accept the connection
accept chan bool
// drops tracks the number of dropped metrics.
drops int
// track the listener here so we can close it in Stop()
listener *net.TCPListener
@@ -41,8 +39,7 @@ type TcpListener struct {
acc telegraf.Accumulator
}
var dropwarn = "ERROR: tcp_listener message queue full. " +
"We have dropped %d messages so far. " +
var dropwarn = "ERROR: Message queue full. Discarding metric [%s], " +
"You may want to increase allowed_pending_messages in the config\n"
const sampleConfig = `
@@ -215,10 +212,7 @@ func (t *TcpListener) handler(conn *net.TCPConn, id string) {
select {
case t.in <- bufCopy:
default:
t.drops++
if t.drops == 1 || t.drops%t.AllowedPendingMessages == 0 {
log.Printf(dropwarn, t.drops)
}
log.Printf(dropwarn, scanner.Text())
}
}
}

View File

@@ -25,8 +25,6 @@ type UdpListener struct {
in chan []byte
done chan struct{}
// drops tracks the number of dropped metrics.
drops int
parser parsers.Parser
@@ -40,8 +38,7 @@ type UdpListener struct {
// https://en.wikipedia.org/wiki/User_Datagram_Protocol#Packet_structure
const UDP_MAX_PACKET_SIZE int = 64 * 1024
var dropwarn = "ERROR: udp_listener message queue full. " +
"We have dropped %d messages so far. " +
var dropwarn = "ERROR: Message queue full. Discarding line [%s] " +
"You may want to increase allowed_pending_messages in the config\n"
const sampleConfig = `
@@ -128,10 +125,7 @@ func (u *UdpListener) udpListen() error {
select {
case u.in <- bufCopy:
default:
u.drops++
if u.drops == 1 || u.drops%u.AllowedPendingMessages == 0 {
log.Printf(dropwarn, u.drops)
}
log.Printf(dropwarn, string(bufCopy))
}
}
}

View File

@@ -22,299 +22,299 @@ This is the full list of stats provided by varnish. Stats will be grouped by the
MEMPOOL, etc). In the output, the prefix will be used as a tag, and removed from field names.
- varnish
- MAIN.uptime (uint64, count, Child process uptime)
- MAIN.sess_conn (uint64, count, Sessions accepted)
- MAIN.sess_drop (uint64, count, Sessions dropped)
- MAIN.sess_fail (uint64, count, Session accept failures)
- MAIN.sess_pipe_overflow (uint64, count, Session pipe overflow)
- MAIN.client_req_400 (uint64, count, Client requests received,)
- MAIN.client_req_411 (uint64, count, Client requests received,)
- MAIN.client_req_413 (uint64, count, Client requests received,)
- MAIN.client_req_417 (uint64, count, Client requests received,)
- MAIN.client_req (uint64, count, Good client requests)
- MAIN.cache_hit (uint64, count, Cache hits)
- MAIN.cache_hitpass (uint64, count, Cache hits for)
- MAIN.cache_miss (uint64, count, Cache misses)
- MAIN.backend_conn (uint64, count, Backend conn. success)
- MAIN.backend_unhealthy (uint64, count, Backend conn. not)
- MAIN.backend_busy (uint64, count, Backend conn. too)
- MAIN.backend_fail (uint64, count, Backend conn. failures)
- MAIN.backend_reuse (uint64, count, Backend conn. reuses)
- MAIN.backend_toolate (uint64, count, Backend conn. was)
- MAIN.backend_recycle (uint64, count, Backend conn. recycles)
- MAIN.backend_retry (uint64, count, Backend conn. retry)
- MAIN.fetch_head (uint64, count, Fetch no body)
- MAIN.fetch_length (uint64, count, Fetch with Length)
- MAIN.fetch_chunked (uint64, count, Fetch chunked)
- MAIN.fetch_eof (uint64, count, Fetch EOF)
- MAIN.fetch_bad (uint64, count, Fetch bad T- E)
- MAIN.fetch_close (uint64, count, Fetch wanted close)
- MAIN.fetch_oldhttp (uint64, count, Fetch pre HTTP/1.1)
- MAIN.fetch_zero (uint64, count, Fetch zero len)
- MAIN.fetch_1xx (uint64, count, Fetch no body)
- MAIN.fetch_204 (uint64, count, Fetch no body)
- MAIN.fetch_304 (uint64, count, Fetch no body)
- MAIN.fetch_failed (uint64, count, Fetch failed (all)
- MAIN.fetch_no_thread (uint64, count, Fetch failed (no)
- MAIN.pools (uint64, count, Number of thread)
- MAIN.threads (uint64, count, Total number of)
- MAIN.threads_limited (uint64, count, Threads hit max)
- MAIN.threads_created (uint64, count, Threads created)
- MAIN.threads_destroyed (uint64, count, Threads destroyed)
- MAIN.threads_failed (uint64, count, Thread creation failed)
- MAIN.thread_queue_len (uint64, count, Length of session)
- MAIN.busy_sleep (uint64, count, Number of requests)
- MAIN.busy_wakeup (uint64, count, Number of requests)
- MAIN.sess_queued (uint64, count, Sessions queued for)
- MAIN.sess_dropped (uint64, count, Sessions dropped for)
- MAIN.n_object (uint64, count, object structs made)
- MAIN.n_vampireobject (uint64, count, unresurrected objects)
- MAIN.n_objectcore (uint64, count, objectcore structs made)
- MAIN.n_objecthead (uint64, count, objecthead structs made)
- MAIN.n_waitinglist (uint64, count, waitinglist structs made)
- MAIN.n_backend (uint64, count, Number of backends)
- MAIN.n_expired (uint64, count, Number of expired)
- MAIN.n_lru_nuked (uint64, count, Number of LRU)
- MAIN.n_lru_moved (uint64, count, Number of LRU)
- MAIN.losthdr (uint64, count, HTTP header overflows)
- MAIN.s_sess (uint64, count, Total sessions seen)
- MAIN.s_req (uint64, count, Total requests seen)
- MAIN.s_pipe (uint64, count, Total pipe sessions)
- MAIN.s_pass (uint64, count, Total pass- ed requests)
- MAIN.s_fetch (uint64, count, Total backend fetches)
- MAIN.s_synth (uint64, count, Total synthethic responses)
- MAIN.s_req_hdrbytes (uint64, count, Request header bytes)
- MAIN.s_req_bodybytes (uint64, count, Request body bytes)
- MAIN.s_resp_hdrbytes (uint64, count, Response header bytes)
- MAIN.s_resp_bodybytes (uint64, count, Response body bytes)
- MAIN.s_pipe_hdrbytes (uint64, count, Pipe request header)
- MAIN.s_pipe_in (uint64, count, Piped bytes from)
- MAIN.s_pipe_out (uint64, count, Piped bytes to)
- MAIN.sess_closed (uint64, count, Session Closed)
- MAIN.sess_pipeline (uint64, count, Session Pipeline)
- MAIN.sess_readahead (uint64, count, Session Read Ahead)
- MAIN.sess_herd (uint64, count, Session herd)
- MAIN.shm_records (uint64, count, SHM records)
- MAIN.shm_writes (uint64, count, SHM writes)
- MAIN.shm_flushes (uint64, count, SHM flushes due)
- MAIN.shm_cont (uint64, count, SHM MTX contention)
- MAIN.shm_cycles (uint64, count, SHM cycles through)
- MAIN.sms_nreq (uint64, count, SMS allocator requests)
- MAIN.sms_nobj (uint64, count, SMS outstanding allocations)
- MAIN.sms_nbytes (uint64, count, SMS outstanding bytes)
- MAIN.sms_balloc (uint64, count, SMS bytes allocated)
- MAIN.sms_bfree (uint64, count, SMS bytes freed)
- MAIN.backend_req (uint64, count, Backend requests made)
- MAIN.n_vcl (uint64, count, Number of loaded)
- MAIN.n_vcl_avail (uint64, count, Number of VCLs)
- MAIN.n_vcl_discard (uint64, count, Number of discarded)
- MAIN.bans (uint64, count, Count of bans)
- MAIN.bans_completed (uint64, count, Number of bans)
- MAIN.bans_obj (uint64, count, Number of bans)
- MAIN.bans_req (uint64, count, Number of bans)
- MAIN.bans_added (uint64, count, Bans added)
- MAIN.bans_deleted (uint64, count, Bans deleted)
- MAIN.bans_tested (uint64, count, Bans tested against)
- MAIN.bans_obj_killed (uint64, count, Objects killed by)
- MAIN.bans_lurker_tested (uint64, count, Bans tested against)
- MAIN.bans_tests_tested (uint64, count, Ban tests tested)
- MAIN.bans_lurker_tests_tested (uint64, count, Ban tests tested)
- MAIN.bans_lurker_obj_killed (uint64, count, Objects killed by)
- MAIN.bans_dups (uint64, count, Bans superseded by)
- MAIN.bans_lurker_contention (uint64, count, Lurker gave way)
- MAIN.bans_persisted_bytes (uint64, count, Bytes used by)
- MAIN.bans_persisted_fragmentation (uint64, count, Extra bytes in)
- MAIN.n_purges (uint64, count, Number of purge)
- MAIN.n_obj_purged (uint64, count, Number of purged)
- MAIN.exp_mailed (uint64, count, Number of objects)
- MAIN.exp_received (uint64, count, Number of objects)
- MAIN.hcb_nolock (uint64, count, HCB Lookups without)
- MAIN.hcb_lock (uint64, count, HCB Lookups with)
- MAIN.hcb_insert (uint64, count, HCB Inserts)
- MAIN.esi_errors (uint64, count, ESI parse errors)
- MAIN.esi_warnings (uint64, count, ESI parse warnings)
- MAIN.vmods (uint64, count, Loaded VMODs)
- MAIN.n_gzip (uint64, count, Gzip operations)
- MAIN.n_gunzip (uint64, count, Gunzip operations)
- MAIN.vsm_free (uint64, count, Free VSM space)
- MAIN.vsm_used (uint64, count, Used VSM space)
- MAIN.vsm_cooling (uint64, count, Cooling VSM space)
- MAIN.vsm_overflow (uint64, count, Overflow VSM space)
- MAIN.vsm_overflowed (uint64, count, Overflowed VSM space)
- MGT.uptime (uint64, count, Management process uptime)
- MGT.child_start (uint64, count, Child process started)
- MGT.child_exit (uint64, count, Child process normal)
- MGT.child_stop (uint64, count, Child process unexpected)
- MGT.child_died (uint64, count, Child process died)
- MGT.child_dump (uint64, count, Child process core)
- MGT.child_panic (uint64, count, Child process panic)
- MEMPOOL.vbc.live (uint64, count, In use)
- MEMPOOL.vbc.pool (uint64, count, In Pool)
- MEMPOOL.vbc.sz_wanted (uint64, count, Size requested)
- MEMPOOL.vbc.sz_needed (uint64, count, Size allocated)
- MEMPOOL.vbc.allocs (uint64, count, Allocations )
- MEMPOOL.vbc.frees (uint64, count, Frees )
- MEMPOOL.vbc.recycle (uint64, count, Recycled from pool)
- MEMPOOL.vbc.timeout (uint64, count, Timed out from)
- MEMPOOL.vbc.toosmall (uint64, count, Too small to)
- MEMPOOL.vbc.surplus (uint64, count, Too many for)
- MEMPOOL.vbc.randry (uint64, count, Pool ran dry)
- MEMPOOL.busyobj.live (uint64, count, In use)
- MEMPOOL.busyobj.pool (uint64, count, In Pool)
- MEMPOOL.busyobj.sz_wanted (uint64, count, Size requested)
- MEMPOOL.busyobj.sz_needed (uint64, count, Size allocated)
- MEMPOOL.busyobj.allocs (uint64, count, Allocations )
- MEMPOOL.busyobj.frees (uint64, count, Frees )
- MEMPOOL.busyobj.recycle (uint64, count, Recycled from pool)
- MEMPOOL.busyobj.timeout (uint64, count, Timed out from)
- MEMPOOL.busyobj.toosmall (uint64, count, Too small to)
- MEMPOOL.busyobj.surplus (uint64, count, Too many for)
- MEMPOOL.busyobj.randry (uint64, count, Pool ran dry)
- MEMPOOL.req0.live (uint64, count, In use)
- MEMPOOL.req0.pool (uint64, count, In Pool)
- MEMPOOL.req0.sz_wanted (uint64, count, Size requested)
- MEMPOOL.req0.sz_needed (uint64, count, Size allocated)
- MEMPOOL.req0.allocs (uint64, count, Allocations )
- MEMPOOL.req0.frees (uint64, count, Frees )
- MEMPOOL.req0.recycle (uint64, count, Recycled from pool)
- MEMPOOL.req0.timeout (uint64, count, Timed out from)
- MEMPOOL.req0.toosmall (uint64, count, Too small to)
- MEMPOOL.req0.surplus (uint64, count, Too many for)
- MEMPOOL.req0.randry (uint64, count, Pool ran dry)
- MEMPOOL.sess0.live (uint64, count, In use)
- MEMPOOL.sess0.pool (uint64, count, In Pool)
- MEMPOOL.sess0.sz_wanted (uint64, count, Size requested)
- MEMPOOL.sess0.sz_needed (uint64, count, Size allocated)
- MEMPOOL.sess0.allocs (uint64, count, Allocations )
- MEMPOOL.sess0.frees (uint64, count, Frees )
- MEMPOOL.sess0.recycle (uint64, count, Recycled from pool)
- MEMPOOL.sess0.timeout (uint64, count, Timed out from)
- MEMPOOL.sess0.toosmall (uint64, count, Too small to)
- MEMPOOL.sess0.surplus (uint64, count, Too many for)
- MEMPOOL.sess0.randry (uint64, count, Pool ran dry)
- MEMPOOL.req1.live (uint64, count, In use)
- MEMPOOL.req1.pool (uint64, count, In Pool)
- MEMPOOL.req1.sz_wanted (uint64, count, Size requested)
- MEMPOOL.req1.sz_needed (uint64, count, Size allocated)
- MEMPOOL.req1.allocs (uint64, count, Allocations )
- MEMPOOL.req1.frees (uint64, count, Frees )
- MEMPOOL.req1.recycle (uint64, count, Recycled from pool)
- MEMPOOL.req1.timeout (uint64, count, Timed out from)
- MEMPOOL.req1.toosmall (uint64, count, Too small to)
- MEMPOOL.req1.surplus (uint64, count, Too many for)
- MEMPOOL.req1.randry (uint64, count, Pool ran dry)
- MEMPOOL.sess1.live (uint64, count, In use)
- MEMPOOL.sess1.pool (uint64, count, In Pool)
- MEMPOOL.sess1.sz_wanted (uint64, count, Size requested)
- MEMPOOL.sess1.sz_needed (uint64, count, Size allocated)
- MEMPOOL.sess1.allocs (uint64, count, Allocations )
- MEMPOOL.sess1.frees (uint64, count, Frees )
- MEMPOOL.sess1.recycle (uint64, count, Recycled from pool)
- MEMPOOL.sess1.timeout (uint64, count, Timed out from)
- MEMPOOL.sess1.toosmall (uint64, count, Too small to)
- MEMPOOL.sess1.surplus (uint64, count, Too many for)
- MEMPOOL.sess1.randry (uint64, count, Pool ran dry)
- SMA.s0.c_req (uint64, count, Allocator requests)
- SMA.s0.c_fail (uint64, count, Allocator failures)
- SMA.s0.c_bytes (uint64, count, Bytes allocated)
- SMA.s0.c_freed (uint64, count, Bytes freed)
- SMA.s0.g_alloc (uint64, count, Allocations outstanding)
- SMA.s0.g_bytes (uint64, count, Bytes outstanding)
- SMA.s0.g_space (uint64, count, Bytes available)
- SMA.Transient.c_req (uint64, count, Allocator requests)
- SMA.Transient.c_fail (uint64, count, Allocator failures)
- SMA.Transient.c_bytes (uint64, count, Bytes allocated)
- SMA.Transient.c_freed (uint64, count, Bytes freed)
- SMA.Transient.g_alloc (uint64, count, Allocations outstanding)
- SMA.Transient.g_bytes (uint64, count, Bytes outstanding)
- SMA.Transient.g_space (uint64, count, Bytes available)
- VBE.default(127.0.0.1,,8080).vcls (uint64, count, VCL references)
- VBE.default(127.0.0.1,,8080).happy (uint64, count, Happy health probes)
- VBE.default(127.0.0.1,,8080).bereq_hdrbytes (uint64, count, Request header bytes)
- VBE.default(127.0.0.1,,8080).bereq_bodybytes (uint64, count, Request body bytes)
- VBE.default(127.0.0.1,,8080).beresp_hdrbytes (uint64, count, Response header bytes)
- VBE.default(127.0.0.1,,8080).beresp_bodybytes (uint64, count, Response body bytes)
- VBE.default(127.0.0.1,,8080).pipe_hdrbytes (uint64, count, Pipe request header)
- VBE.default(127.0.0.1,,8080).pipe_out (uint64, count, Piped bytes to)
- VBE.default(127.0.0.1,,8080).pipe_in (uint64, count, Piped bytes from)
- LCK.sms.creat (uint64, count, Created locks)
- LCK.sms.destroy (uint64, count, Destroyed locks)
- LCK.sms.locks (uint64, count, Lock Operations)
- LCK.smp.creat (uint64, count, Created locks)
- LCK.smp.destroy (uint64, count, Destroyed locks)
- LCK.smp.locks (uint64, count, Lock Operations)
- LCK.sma.creat (uint64, count, Created locks)
- LCK.sma.destroy (uint64, count, Destroyed locks)
- LCK.sma.locks (uint64, count, Lock Operations)
- LCK.smf.creat (uint64, count, Created locks)
- LCK.smf.destroy (uint64, count, Destroyed locks)
- LCK.smf.locks (uint64, count, Lock Operations)
- LCK.hsl.creat (uint64, count, Created locks)
- LCK.hsl.destroy (uint64, count, Destroyed locks)
- LCK.hsl.locks (uint64, count, Lock Operations)
- LCK.hcb.creat (uint64, count, Created locks)
- LCK.hcb.destroy (uint64, count, Destroyed locks)
- LCK.hcb.locks (uint64, count, Lock Operations)
- LCK.hcl.creat (uint64, count, Created locks)
- LCK.hcl.destroy (uint64, count, Destroyed locks)
- LCK.hcl.locks (uint64, count, Lock Operations)
- LCK.vcl.creat (uint64, count, Created locks)
- LCK.vcl.destroy (uint64, count, Destroyed locks)
- LCK.vcl.locks (uint64, count, Lock Operations)
- LCK.sessmem.creat (uint64, count, Created locks)
- LCK.sessmem.destroy (uint64, count, Destroyed locks)
- LCK.sessmem.locks (uint64, count, Lock Operations)
- LCK.sess.creat (uint64, count, Created locks)
- LCK.sess.destroy (uint64, count, Destroyed locks)
- LCK.sess.locks (uint64, count, Lock Operations)
- LCK.wstat.creat (uint64, count, Created locks)
- LCK.wstat.destroy (uint64, count, Destroyed locks)
- LCK.wstat.locks (uint64, count, Lock Operations)
- LCK.herder.creat (uint64, count, Created locks)
- LCK.herder.destroy (uint64, count, Destroyed locks)
- LCK.herder.locks (uint64, count, Lock Operations)
- LCK.wq.creat (uint64, count, Created locks)
- LCK.wq.destroy (uint64, count, Destroyed locks)
- LCK.wq.locks (uint64, count, Lock Operations)
- LCK.objhdr.creat (uint64, count, Created locks)
- LCK.objhdr.destroy (uint64, count, Destroyed locks)
- LCK.objhdr.locks (uint64, count, Lock Operations)
- LCK.exp.creat (uint64, count, Created locks)
- LCK.exp.destroy (uint64, count, Destroyed locks)
- LCK.exp.locks (uint64, count, Lock Operations)
- LCK.lru.creat (uint64, count, Created locks)
- LCK.lru.destroy (uint64, count, Destroyed locks)
- LCK.lru.locks (uint64, count, Lock Operations)
- LCK.cli.creat (uint64, count, Created locks)
- LCK.cli.destroy (uint64, count, Destroyed locks)
- LCK.cli.locks (uint64, count, Lock Operations)
- LCK.ban.creat (uint64, count, Created locks)
- LCK.ban.destroy (uint64, count, Destroyed locks)
- LCK.ban.locks (uint64, count, Lock Operations)
- LCK.vbp.creat (uint64, count, Created locks)
- LCK.vbp.destroy (uint64, count, Destroyed locks)
- LCK.vbp.locks (uint64, count, Lock Operations)
- LCK.backend.creat (uint64, count, Created locks)
- LCK.backend.destroy (uint64, count, Destroyed locks)
- LCK.backend.locks (uint64, count, Lock Operations)
- LCK.vcapace.creat (uint64, count, Created locks)
- LCK.vcapace.destroy (uint64, count, Destroyed locks)
- LCK.vcapace.locks (uint64, count, Lock Operations)
- LCK.nbusyobj.creat (uint64, count, Created locks)
- LCK.nbusyobj.destroy (uint64, count, Destroyed locks)
- LCK.nbusyobj.locks (uint64, count, Lock Operations)
- LCK.busyobj.creat (uint64, count, Created locks)
- LCK.busyobj.destroy (uint64, count, Destroyed locks)
- LCK.busyobj.locks (uint64, count, Lock Operations)
- LCK.mempool.creat (uint64, count, Created locks)
- LCK.mempool.destroy (uint64, count, Destroyed locks)
- LCK.mempool.locks (uint64, count, Lock Operations)
- LCK.vxid.creat (uint64, count, Created locks)
- LCK.vxid.destroy (uint64, count, Destroyed locks)
- LCK.vxid.locks (uint64, count, Lock Operations)
- LCK.pipestat.creat (uint64, count, Created locks)
- LCK.pipestat.destroy (uint64, count, Destroyed locks)
- LCK.pipestat.locks (uint64, count, Lock Operations)
- MAIN.uptime (int, count, Child process uptime)
- MAIN.sess_conn (int, count, Sessions accepted)
- MAIN.sess_drop (int, count, Sessions dropped)
- MAIN.sess_fail (int, count, Session accept failures)
- MAIN.sess_pipe_overflow (int, count, Session pipe overflow)
- MAIN.client_req_400 (int, count, Client requests received,)
- MAIN.client_req_411 (int, count, Client requests received,)
- MAIN.client_req_413 (int, count, Client requests received,)
- MAIN.client_req_417 (int, count, Client requests received,)
- MAIN.client_req (int, count, Good client requests)
- MAIN.cache_hit (int, count, Cache hits)
- MAIN.cache_hitpass (int, count, Cache hits for)
- MAIN.cache_miss (int, count, Cache misses)
- MAIN.backend_conn (int, count, Backend conn. success)
- MAIN.backend_unhealthy (int, count, Backend conn. not)
- MAIN.backend_busy (int, count, Backend conn. too)
- MAIN.backend_fail (int, count, Backend conn. failures)
- MAIN.backend_reuse (int, count, Backend conn. reuses)
- MAIN.backend_toolate (int, count, Backend conn. was)
- MAIN.backend_recycle (int, count, Backend conn. recycles)
- MAIN.backend_retry (int, count, Backend conn. retry)
- MAIN.fetch_head (int, count, Fetch no body)
- MAIN.fetch_length (int, count, Fetch with Length)
- MAIN.fetch_chunked (int, count, Fetch chunked)
- MAIN.fetch_eof (int, count, Fetch EOF)
- MAIN.fetch_bad (int, count, Fetch bad T- E)
- MAIN.fetch_close (int, count, Fetch wanted close)
- MAIN.fetch_oldhttp (int, count, Fetch pre HTTP/1.1)
- MAIN.fetch_zero (int, count, Fetch zero len)
- MAIN.fetch_1xx (int, count, Fetch no body)
- MAIN.fetch_204 (int, count, Fetch no body)
- MAIN.fetch_304 (int, count, Fetch no body)
- MAIN.fetch_failed (int, count, Fetch failed (all)
- MAIN.fetch_no_thread (int, count, Fetch failed (no)
- MAIN.pools (int, count, Number of thread)
- MAIN.threads (int, count, Total number of)
- MAIN.threads_limited (int, count, Threads hit max)
- MAIN.threads_created (int, count, Threads created)
- MAIN.threads_destroyed (int, count, Threads destroyed)
- MAIN.threads_failed (int, count, Thread creation failed)
- MAIN.thread_queue_len (int, count, Length of session)
- MAIN.busy_sleep (int, count, Number of requests)
- MAIN.busy_wakeup (int, count, Number of requests)
- MAIN.sess_queued (int, count, Sessions queued for)
- MAIN.sess_dropped (int, count, Sessions dropped for)
- MAIN.n_object (int, count, object structs made)
- MAIN.n_vampireobject (int, count, unresurrected objects)
- MAIN.n_objectcore (int, count, objectcore structs made)
- MAIN.n_objecthead (int, count, objecthead structs made)
- MAIN.n_waitinglist (int, count, waitinglist structs made)
- MAIN.n_backend (int, count, Number of backends)
- MAIN.n_expired (int, count, Number of expired)
- MAIN.n_lru_nuked (int, count, Number of LRU)
- MAIN.n_lru_moved (int, count, Number of LRU)
- MAIN.losthdr (int, count, HTTP header overflows)
- MAIN.s_sess (int, count, Total sessions seen)
- MAIN.s_req (int, count, Total requests seen)
- MAIN.s_pipe (int, count, Total pipe sessions)
- MAIN.s_pass (int, count, Total pass- ed requests)
- MAIN.s_fetch (int, count, Total backend fetches)
- MAIN.s_synth (int, count, Total synthethic responses)
- MAIN.s_req_hdrbytes (int, count, Request header bytes)
- MAIN.s_req_bodybytes (int, count, Request body bytes)
- MAIN.s_resp_hdrbytes (int, count, Response header bytes)
- MAIN.s_resp_bodybytes (int, count, Response body bytes)
- MAIN.s_pipe_hdrbytes (int, count, Pipe request header)
- MAIN.s_pipe_in (int, count, Piped bytes from)
- MAIN.s_pipe_out (int, count, Piped bytes to)
- MAIN.sess_closed (int, count, Session Closed)
- MAIN.sess_pipeline (int, count, Session Pipeline)
- MAIN.sess_readahead (int, count, Session Read Ahead)
- MAIN.sess_herd (int, count, Session herd)
- MAIN.shm_records (int, count, SHM records)
- MAIN.shm_writes (int, count, SHM writes)
- MAIN.shm_flushes (int, count, SHM flushes due)
- MAIN.shm_cont (int, count, SHM MTX contention)
- MAIN.shm_cycles (int, count, SHM cycles through)
- MAIN.sms_nreq (int, count, SMS allocator requests)
- MAIN.sms_nobj (int, count, SMS outstanding allocations)
- MAIN.sms_nbytes (int, count, SMS outstanding bytes)
- MAIN.sms_balloc (int, count, SMS bytes allocated)
- MAIN.sms_bfree (int, count, SMS bytes freed)
- MAIN.backend_req (int, count, Backend requests made)
- MAIN.n_vcl (int, count, Number of loaded)
- MAIN.n_vcl_avail (int, count, Number of VCLs)
- MAIN.n_vcl_discard (int, count, Number of discarded)
- MAIN.bans (int, count, Count of bans)
- MAIN.bans_completed (int, count, Number of bans)
- MAIN.bans_obj (int, count, Number of bans)
- MAIN.bans_req (int, count, Number of bans)
- MAIN.bans_added (int, count, Bans added)
- MAIN.bans_deleted (int, count, Bans deleted)
- MAIN.bans_tested (int, count, Bans tested against)
- MAIN.bans_obj_killed (int, count, Objects killed by)
- MAIN.bans_lurker_tested (int, count, Bans tested against)
- MAIN.bans_tests_tested (int, count, Ban tests tested)
- MAIN.bans_lurker_tests_tested (int, count, Ban tests tested)
- MAIN.bans_lurker_obj_killed (int, count, Objects killed by)
- MAIN.bans_dups (int, count, Bans superseded by)
- MAIN.bans_lurker_contention (int, count, Lurker gave way)
- MAIN.bans_persisted_bytes (int, count, Bytes used by)
- MAIN.bans_persisted_fragmentation (int, count, Extra bytes in)
- MAIN.n_purges (int, count, Number of purge)
- MAIN.n_obj_purged (int, count, Number of purged)
- MAIN.exp_mailed (int, count, Number of objects)
- MAIN.exp_received (int, count, Number of objects)
- MAIN.hcb_nolock (int, count, HCB Lookups without)
- MAIN.hcb_lock (int, count, HCB Lookups with)
- MAIN.hcb_insert (int, count, HCB Inserts)
- MAIN.esi_errors (int, count, ESI parse errors)
- MAIN.esi_warnings (int, count, ESI parse warnings)
- MAIN.vmods (int, count, Loaded VMODs)
- MAIN.n_gzip (int, count, Gzip operations)
- MAIN.n_gunzip (int, count, Gunzip operations)
- MAIN.vsm_free (int, count, Free VSM space)
- MAIN.vsm_used (int, count, Used VSM space)
- MAIN.vsm_cooling (int, count, Cooling VSM space)
- MAIN.vsm_overflow (int, count, Overflow VSM space)
- MAIN.vsm_overflowed (int, count, Overflowed VSM space)
- MGT.uptime (int, count, Management process uptime)
- MGT.child_start (int, count, Child process started)
- MGT.child_exit (int, count, Child process normal)
- MGT.child_stop (int, count, Child process unexpected)
- MGT.child_died (int, count, Child process died)
- MGT.child_dump (int, count, Child process core)
- MGT.child_panic (int, count, Child process panic)
- MEMPOOL.vbc.live (int, count, In use)
- MEMPOOL.vbc.pool (int, count, In Pool)
- MEMPOOL.vbc.sz_wanted (int, count, Size requested)
- MEMPOOL.vbc.sz_needed (int, count, Size allocated)
- MEMPOOL.vbc.allocs (int, count, Allocations )
- MEMPOOL.vbc.frees (int, count, Frees )
- MEMPOOL.vbc.recycle (int, count, Recycled from pool)
- MEMPOOL.vbc.timeout (int, count, Timed out from)
- MEMPOOL.vbc.toosmall (int, count, Too small to)
- MEMPOOL.vbc.surplus (int, count, Too many for)
- MEMPOOL.vbc.randry (int, count, Pool ran dry)
- MEMPOOL.busyobj.live (int, count, In use)
- MEMPOOL.busyobj.pool (int, count, In Pool)
- MEMPOOL.busyobj.sz_wanted (int, count, Size requested)
- MEMPOOL.busyobj.sz_needed (int, count, Size allocated)
- MEMPOOL.busyobj.allocs (int, count, Allocations )
- MEMPOOL.busyobj.frees (int, count, Frees )
- MEMPOOL.busyobj.recycle (int, count, Recycled from pool)
- MEMPOOL.busyobj.timeout (int, count, Timed out from)
- MEMPOOL.busyobj.toosmall (int, count, Too small to)
- MEMPOOL.busyobj.surplus (int, count, Too many for)
- MEMPOOL.busyobj.randry (int, count, Pool ran dry)
- MEMPOOL.req0.live (int, count, In use)
- MEMPOOL.req0.pool (int, count, In Pool)
- MEMPOOL.req0.sz_wanted (int, count, Size requested)
- MEMPOOL.req0.sz_needed (int, count, Size allocated)
- MEMPOOL.req0.allocs (int, count, Allocations )
- MEMPOOL.req0.frees (int, count, Frees )
- MEMPOOL.req0.recycle (int, count, Recycled from pool)
- MEMPOOL.req0.timeout (int, count, Timed out from)
- MEMPOOL.req0.toosmall (int, count, Too small to)
- MEMPOOL.req0.surplus (int, count, Too many for)
- MEMPOOL.req0.randry (int, count, Pool ran dry)
- MEMPOOL.sess0.live (int, count, In use)
- MEMPOOL.sess0.pool (int, count, In Pool)
- MEMPOOL.sess0.sz_wanted (int, count, Size requested)
- MEMPOOL.sess0.sz_needed (int, count, Size allocated)
- MEMPOOL.sess0.allocs (int, count, Allocations )
- MEMPOOL.sess0.frees (int, count, Frees )
- MEMPOOL.sess0.recycle (int, count, Recycled from pool)
- MEMPOOL.sess0.timeout (int, count, Timed out from)
- MEMPOOL.sess0.toosmall (int, count, Too small to)
- MEMPOOL.sess0.surplus (int, count, Too many for)
- MEMPOOL.sess0.randry (int, count, Pool ran dry)
- MEMPOOL.req1.live (int, count, In use)
- MEMPOOL.req1.pool (int, count, In Pool)
- MEMPOOL.req1.sz_wanted (int, count, Size requested)
- MEMPOOL.req1.sz_needed (int, count, Size allocated)
- MEMPOOL.req1.allocs (int, count, Allocations )
- MEMPOOL.req1.frees (int, count, Frees )
- MEMPOOL.req1.recycle (int, count, Recycled from pool)
- MEMPOOL.req1.timeout (int, count, Timed out from)
- MEMPOOL.req1.toosmall (int, count, Too small to)
- MEMPOOL.req1.surplus (int, count, Too many for)
- MEMPOOL.req1.randry (int, count, Pool ran dry)
- MEMPOOL.sess1.live (int, count, In use)
- MEMPOOL.sess1.pool (int, count, In Pool)
- MEMPOOL.sess1.sz_wanted (int, count, Size requested)
- MEMPOOL.sess1.sz_needed (int, count, Size allocated)
- MEMPOOL.sess1.allocs (int, count, Allocations )
- MEMPOOL.sess1.frees (int, count, Frees )
- MEMPOOL.sess1.recycle (int, count, Recycled from pool)
- MEMPOOL.sess1.timeout (int, count, Timed out from)
- MEMPOOL.sess1.toosmall (int, count, Too small to)
- MEMPOOL.sess1.surplus (int, count, Too many for)
- MEMPOOL.sess1.randry (int, count, Pool ran dry)
- SMA.s0.c_req (int, count, Allocator requests)
- SMA.s0.c_fail (int, count, Allocator failures)
- SMA.s0.c_bytes (int, count, Bytes allocated)
- SMA.s0.c_freed (int, count, Bytes freed)
- SMA.s0.g_alloc (int, count, Allocations outstanding)
- SMA.s0.g_bytes (int, count, Bytes outstanding)
- SMA.s0.g_space (int, count, Bytes available)
- SMA.Transient.c_req (int, count, Allocator requests)
- SMA.Transient.c_fail (int, count, Allocator failures)
- SMA.Transient.c_bytes (int, count, Bytes allocated)
- SMA.Transient.c_freed (int, count, Bytes freed)
- SMA.Transient.g_alloc (int, count, Allocations outstanding)
- SMA.Transient.g_bytes (int, count, Bytes outstanding)
- SMA.Transient.g_space (int, count, Bytes available)
- VBE.default(127.0.0.1,,8080).vcls (int, count, VCL references)
- VBE.default(127.0.0.1,,8080).happy (int, count, Happy health probes)
- VBE.default(127.0.0.1,,8080).bereq_hdrbytes (int, count, Request header bytes)
- VBE.default(127.0.0.1,,8080).bereq_bodybytes (int, count, Request body bytes)
- VBE.default(127.0.0.1,,8080).beresp_hdrbytes (int, count, Response header bytes)
- VBE.default(127.0.0.1,,8080).beresp_bodybytes (int, count, Response body bytes)
- VBE.default(127.0.0.1,,8080).pipe_hdrbytes (int, count, Pipe request header)
- VBE.default(127.0.0.1,,8080).pipe_out (int, count, Piped bytes to)
- VBE.default(127.0.0.1,,8080).pipe_in (int, count, Piped bytes from)
- LCK.sms.creat (int, count, Created locks)
- LCK.sms.destroy (int, count, Destroyed locks)
- LCK.sms.locks (int, count, Lock Operations)
- LCK.smp.creat (int, count, Created locks)
- LCK.smp.destroy (int, count, Destroyed locks)
- LCK.smp.locks (int, count, Lock Operations)
- LCK.sma.creat (int, count, Created locks)
- LCK.sma.destroy (int, count, Destroyed locks)
- LCK.sma.locks (int, count, Lock Operations)
- LCK.smf.creat (int, count, Created locks)
- LCK.smf.destroy (int, count, Destroyed locks)
- LCK.smf.locks (int, count, Lock Operations)
- LCK.hsl.creat (int, count, Created locks)
- LCK.hsl.destroy (int, count, Destroyed locks)
- LCK.hsl.locks (int, count, Lock Operations)
- LCK.hcb.creat (int, count, Created locks)
- LCK.hcb.destroy (int, count, Destroyed locks)
- LCK.hcb.locks (int, count, Lock Operations)
- LCK.hcl.creat (int, count, Created locks)
- LCK.hcl.destroy (int, count, Destroyed locks)
- LCK.hcl.locks (int, count, Lock Operations)
- LCK.vcl.creat (int, count, Created locks)
- LCK.vcl.destroy (int, count, Destroyed locks)
- LCK.vcl.locks (int, count, Lock Operations)
- LCK.sessmem.creat (int, count, Created locks)
- LCK.sessmem.destroy (int, count, Destroyed locks)
- LCK.sessmem.locks (int, count, Lock Operations)
- LCK.sess.creat (int, count, Created locks)
- LCK.sess.destroy (int, count, Destroyed locks)
- LCK.sess.locks (int, count, Lock Operations)
- LCK.wstat.creat (int, count, Created locks)
- LCK.wstat.destroy (int, count, Destroyed locks)
- LCK.wstat.locks (int, count, Lock Operations)
- LCK.herder.creat (int, count, Created locks)
- LCK.herder.destroy (int, count, Destroyed locks)
- LCK.herder.locks (int, count, Lock Operations)
- LCK.wq.creat (int, count, Created locks)
- LCK.wq.destroy (int, count, Destroyed locks)
- LCK.wq.locks (int, count, Lock Operations)
- LCK.objhdr.creat (int, count, Created locks)
- LCK.objhdr.destroy (int, count, Destroyed locks)
- LCK.objhdr.locks (int, count, Lock Operations)
- LCK.exp.creat (int, count, Created locks)
- LCK.exp.destroy (int, count, Destroyed locks)
- LCK.exp.locks (int, count, Lock Operations)
- LCK.lru.creat (int, count, Created locks)
- LCK.lru.destroy (int, count, Destroyed locks)
- LCK.lru.locks (int, count, Lock Operations)
- LCK.cli.creat (int, count, Created locks)
- LCK.cli.destroy (int, count, Destroyed locks)
- LCK.cli.locks (int, count, Lock Operations)
- LCK.ban.creat (int, count, Created locks)
- LCK.ban.destroy (int, count, Destroyed locks)
- LCK.ban.locks (int, count, Lock Operations)
- LCK.vbp.creat (int, count, Created locks)
- LCK.vbp.destroy (int, count, Destroyed locks)
- LCK.vbp.locks (int, count, Lock Operations)
- LCK.backend.creat (int, count, Created locks)
- LCK.backend.destroy (int, count, Destroyed locks)
- LCK.backend.locks (int, count, Lock Operations)
- LCK.vcapace.creat (int, count, Created locks)
- LCK.vcapace.destroy (int, count, Destroyed locks)
- LCK.vcapace.locks (int, count, Lock Operations)
- LCK.nbusyobj.creat (int, count, Created locks)
- LCK.nbusyobj.destroy (int, count, Destroyed locks)
- LCK.nbusyobj.locks (int, count, Lock Operations)
- LCK.busyobj.creat (int, count, Created locks)
- LCK.busyobj.destroy (int, count, Destroyed locks)
- LCK.busyobj.locks (int, count, Lock Operations)
- LCK.mempool.creat (int, count, Created locks)
- LCK.mempool.destroy (int, count, Destroyed locks)
- LCK.mempool.locks (int, count, Lock Operations)
- LCK.vxid.creat (int, count, Created locks)
- LCK.vxid.destroy (int, count, Destroyed locks)
- LCK.vxid.locks (int, count, Lock Operations)
- LCK.pipestat.creat (int, count, Created locks)
- LCK.pipestat.destroy (int, count, Destroyed locks)
- LCK.pipestat.locks (int, count, Lock Operations)
### Tags:

View File

@@ -10,36 +10,33 @@ import (
"os/exec"
"strconv"
"strings"
"time"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/filter"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/plugins/inputs"
"time"
)
type runner func(cmdName string) (*bytes.Buffer, error)
const (
kwAll = "all"
)
// Varnish is used to store configuration values
type Varnish struct {
Stats []string
Binary string
filter filter.Filter
run runner
}
var defaultStats = []string{"MAIN.cache_hit", "MAIN.cache_miss", "MAIN.uptime"}
var defaultBinary = "/usr/bin/varnishstat"
var sampleConfig = `
var varnishSampleConfig = `
## The default location of the varnishstat binary can be overridden with:
binary = "/usr/bin/varnishstat"
## By default, telegraf gather stats for 3 metric points.
## Setting stats will override the defaults shown below.
## Glob matching can be used, ie, stats = ["MAIN.*"]
## stats may also be set to ["*"], which will collect all stats
## stats may also be set to ["all"], which will collect all stats
stats = ["MAIN.cache_hit", "MAIN.cache_miss", "MAIN.uptime"]
`
@@ -49,11 +46,44 @@ func (s *Varnish) Description() string {
// SampleConfig displays configuration instructions
func (s *Varnish) SampleConfig() string {
return sampleConfig
return fmt.Sprintf(varnishSampleConfig, strings.Join(defaultStats, "\",\""))
}
func (s *Varnish) setDefaults() {
if len(s.Stats) == 0 {
s.Stats = defaultStats
}
if s.Binary == "" {
s.Binary = defaultBinary
}
}
// Builds a filter function that will indicate whether a given stat should
// be reported
func (s *Varnish) statsFilter() func(string) bool {
s.setDefaults()
// Build a set for constant-time lookup of whether stats should be included
filter := make(map[string]struct{})
for _, s := range s.Stats {
filter[s] = struct{}{}
}
// Create a function that respects the kwAll by always returning true
// if it is set
return func(stat string) bool {
if s.Stats[0] == kwAll {
return true
}
_, found := filter[stat]
return found
}
}
// Shell out to varnish_stat and return the output
func varnishRunner(cmdName string) (*bytes.Buffer, error) {
var varnishStat = func(cmdName string) (*bytes.Buffer, error) {
cmdArgs := []string{"-1"}
cmd := exec.Command(cmdName, cmdArgs...)
@@ -74,27 +104,13 @@ func varnishRunner(cmdName string) (*bytes.Buffer, error) {
// 'section' tag and all stats that share that prefix will be reported as fields
// with that tag
func (s *Varnish) Gather(acc telegraf.Accumulator) error {
if s.filter == nil {
var err error
if len(s.Stats) == 0 {
s.filter, err = filter.CompileFilter(defaultStats)
} else {
// legacy support, change "all" -> "*":
if s.Stats[0] == "all" {
s.Stats[0] = "*"
}
s.filter, err = filter.CompileFilter(s.Stats)
}
if err != nil {
return err
}
}
out, err := s.run(s.Binary)
s.setDefaults()
out, err := varnishStat(s.Binary)
if err != nil {
return fmt.Errorf("error gathering metrics: %s", err)
}
statsFilter := s.statsFilter()
sectionMap := make(map[string]map[string]interface{})
scanner := bufio.NewScanner(out)
for scanner.Scan() {
@@ -109,7 +125,7 @@ func (s *Varnish) Gather(acc telegraf.Accumulator) error {
stat := cols[0]
value := cols[1]
if s.filter != nil && !s.filter.Match(stat) {
if !statsFilter(stat) {
continue
}
@@ -122,7 +138,7 @@ func (s *Varnish) Gather(acc telegraf.Accumulator) error {
sectionMap[section] = make(map[string]interface{})
}
sectionMap[section][field], err = strconv.ParseUint(value, 10, 64)
sectionMap[section][field], err = strconv.Atoi(value)
if err != nil {
fmt.Fprintf(os.Stderr, "Expected a numeric value for %s = %v\n",
stat, value)
@@ -144,9 +160,5 @@ func (s *Varnish) Gather(acc telegraf.Accumulator) error {
}
func init() {
inputs.Add("varnish", func() telegraf.Input {
return &Varnish{
run: varnishRunner,
}
})
inputs.Add("varnish", func() telegraf.Input { return &Varnish{} })
}

View File

@@ -17,12 +17,38 @@ func fakeVarnishStat(output string) func(string) (*bytes.Buffer, error) {
}
}
func TestGather(t *testing.T) {
acc := &testutil.Accumulator{}
v := &Varnish{
run: fakeVarnishStat(smOutput),
Stats: []string{"*"},
func TestConfigsUsed(t *testing.T) {
saved := varnishStat
defer func() {
varnishStat = saved
}()
expecations := map[string]string{
"": defaultBinary,
"/foo/bar/baz": "/foo/bar/baz",
}
for in, expected := range expecations {
varnishStat = func(actual string) (*bytes.Buffer, error) {
assert.Equal(t, expected, actual)
return &bytes.Buffer{}, nil
}
acc := &testutil.Accumulator{}
v := &Varnish{Binary: in}
v.Gather(acc)
}
}
func TestGather(t *testing.T) {
saved := varnishStat
defer func() {
varnishStat = saved
}()
varnishStat = fakeVarnishStat(smOutput)
acc := &testutil.Accumulator{}
v := &Varnish{Stats: []string{"all"}}
v.Gather(acc)
acc.HasMeasurement("varnish")
@@ -34,11 +60,14 @@ func TestGather(t *testing.T) {
}
func TestParseFullOutput(t *testing.T) {
saved := varnishStat
defer func() {
varnishStat = saved
}()
varnishStat = fakeVarnishStat(fullOutput)
acc := &testutil.Accumulator{}
v := &Varnish{
run: fakeVarnishStat(fullOutput),
Stats: []string{"*"},
}
v := &Varnish{Stats: []string{"all"}}
err := v.Gather(acc)
assert.NoError(t, err)
@@ -48,24 +77,15 @@ func TestParseFullOutput(t *testing.T) {
assert.Equal(t, 293, len(flat))
}
func TestFilterSomeStats(t *testing.T) {
acc := &testutil.Accumulator{}
v := &Varnish{
run: fakeVarnishStat(fullOutput),
Stats: []string{"MGT.*", "VBE.*"},
}
err := v.Gather(acc)
assert.NoError(t, err)
acc.HasMeasurement("varnish")
flat := flatten(acc.Metrics)
assert.Len(t, acc.Metrics, 2)
assert.Equal(t, 16, len(flat))
}
func TestFieldConfig(t *testing.T) {
saved := varnishStat
defer func() {
varnishStat = saved
}()
varnishStat = fakeVarnishStat(fullOutput)
expect := map[string]int{
"*": 293,
"all": 293,
"": 0, // default
"MAIN.uptime": 1,
"MEMPOOL.req0.sz_needed,MAIN.fetch_bad": 2,
@@ -73,10 +93,7 @@ func TestFieldConfig(t *testing.T) {
for fieldCfg, expected := range expect {
acc := &testutil.Accumulator{}
v := &Varnish{
run: fakeVarnishStat(fullOutput),
Stats: strings.Split(fieldCfg, ","),
}
v := &Varnish{Stats: strings.Split(fieldCfg, ",")}
err := v.Gather(acc)
assert.NoError(t, err)
@@ -113,18 +130,18 @@ MEMPOOL.vbc.sz_wanted 88 . Size requested
var parsedSmOutput = map[string]map[string]interface{}{
"MAIN": map[string]interface{}{
"uptime": uint64(895),
"cache_hit": uint64(95),
"cache_miss": uint64(5),
"uptime": 895,
"cache_hit": 95,
"cache_miss": 5,
},
"MGT": map[string]interface{}{
"uptime": uint64(896),
"child_start": uint64(1),
"uptime": 896,
"child_start": 1,
},
"MEMPOOL": map[string]interface{}{
"vbc.live": uint64(0),
"vbc.pool": uint64(10),
"vbc.sz_wanted": uint64(88),
"vbc.live": 0,
"vbc.pool": 10,
"vbc.sz_wanted": 88,
},
}

View File

@@ -1,294 +1,227 @@
# ZFS plugin
# Telegraf plugin: zfs
This ZFS plugin provides metrics from your ZFS filesystems. It supports ZFS on
Linux and FreeBSD. It gets ZFS stat from `/proc/spl/kstat/zfs` on Linux and
from `sysctl` and `zpool` on FreeBSD.
Get ZFS stat from /proc/spl/kstat/zfs
### Configuration:
# Measurements
```toml
[[inputs.zfs]]
## ZFS kstat path. Ignored on FreeBSD
## If not specified, then default is:
# kstatPath = "/proc/spl/kstat/zfs"
Meta:
## By default, telegraf gather all zfs stats
## If not specified, then default is:
# kstatMetrics = ["arcstats", "zfetchstats", "vdev_cache_stats"]
- tags: `pools=POOL1::POOL2`
## By default, don't gather zpool stats
# poolMetrics = false
```
Measurement names:
### Measurements & Fields:
By default this plugin collects metrics about **Arc**, **Zfetch**, and
**Vdev cache**. All these metrics are either counters or measure sizes
in bytes. These metrics will be in the `zfs` measurement with the field
names listed bellow.
If `poolMetrics` is enabled then additional metrics will be gathered for
each pool.
- zfs
With fields listed bellow.
#### Arc Stats
- arcstats_allocated (FreeBSD only)
- arcstats_anon_evict_data (Linux only)
- arcstats_anon_evict_metadata (Linux only)
- arcstats_anon_evictable_data (FreeBSD only)
- arcstats_anon_evictable_metadata (FreeBSD only)
- arcstats_anon_size
- arcstats_arc_loaned_bytes (Linux only)
- arcstats_arc_meta_limit
- arcstats_arc_meta_max
- arcstats_arc_meta_min (FreeBSD only)
- arcstats_arc_meta_used
- arcstats_arc_no_grow (Linux only)
- arcstats_arc_prune (Linux only)
- arcstats_arc_tempreserve (Linux only)
- arcstats_c
- arcstats_c_max
- arcstats_c_min
- arcstats_data_size
- arcstats_deleted
- arcstats_hits
- arcstats_misses
- arcstats_demand_data_hits
- arcstats_demand_data_misses
- arcstats_demand_hit_predictive_prefetch (FreeBSD only)
- arcstats_demand_metadata_hits
- arcstats_demand_metadata_misses
- arcstats_duplicate_buffers
- arcstats_duplicate_buffers_size
- arcstats_duplicate_reads
- arcstats_evict_l2_cached
- arcstats_evict_l2_eligible
- arcstats_evict_l2_ineligible
- arcstats_evict_l2_skip (FreeBSD only)
- arcstats_evict_not_enough (FreeBSD only)
- arcstats_evict_skip
- arcstats_hash_chain_max
- arcstats_hash_chains
- arcstats_hash_collisions
- arcstats_hash_elements
- arcstats_hash_elements_max
- arcstats_hdr_size
- arcstats_hits
- arcstats_l2_abort_lowmem
- arcstats_l2_asize
- arcstats_l2_cdata_free_on_write
- arcstats_l2_cksum_bad
- arcstats_l2_compress_failures
- arcstats_l2_compress_successes
- arcstats_l2_compress_zeros
- arcstats_l2_evict_l1cached (FreeBSD only)
- arcstats_l2_evict_lock_retry
- arcstats_l2_evict_reading
- arcstats_l2_feeds
- arcstats_l2_free_on_write
- arcstats_l2_hdr_size
- arcstats_l2_hits
- arcstats_l2_io_error
- arcstats_l2_misses
- arcstats_l2_read_bytes
- arcstats_l2_rw_clash
- arcstats_l2_size
- arcstats_l2_write_buffer_bytes_scanned (FreeBSD only)
- arcstats_l2_write_buffer_iter (FreeBSD only)
- arcstats_l2_write_buffer_list_iter (FreeBSD only)
- arcstats_l2_write_buffer_list_null_iter (FreeBSD only)
- arcstats_l2_write_bytes
- arcstats_l2_write_full (FreeBSD only)
- arcstats_l2_write_in_l2 (FreeBSD only)
- arcstats_l2_write_io_in_progress (FreeBSD only)
- arcstats_l2_write_not_cacheable (FreeBSD only)
- arcstats_l2_write_passed_headroom (FreeBSD only)
- arcstats_l2_write_pios (FreeBSD only)
- arcstats_l2_write_spa_mismatch (FreeBSD only)
- arcstats_l2_write_trylock_fail (FreeBSD only)
- arcstats_l2_writes_done
- arcstats_l2_writes_error
- arcstats_l2_writes_hdr_miss (Linux only)
- arcstats_l2_writes_lock_retry (FreeBSD only)
- arcstats_l2_writes_sent
- arcstats_memory_direct_count (Linux only)
- arcstats_memory_indirect_count (Linux only)
- arcstats_memory_throttle_count
- arcstats_meta_size (Linux only)
- arcstats_mfu_evict_data (Linux only)
- arcstats_mfu_evict_metadata (Linux only)
- arcstats_mfu_ghost_evict_data (Linux only)
- arcstats_mfu_ghost_evict_metadata (Linux only)
- arcstats_metadata_size (FreeBSD only)
- arcstats_mfu_evictable_data (FreeBSD only)
- arcstats_mfu_evictable_metadata (FreeBSD only)
- arcstats_mfu_ghost_evictable_data (FreeBSD only)
- arcstats_mfu_ghost_evictable_metadata (FreeBSD only)
- arcstats_mfu_ghost_hits
- arcstats_mfu_ghost_size
- arcstats_mfu_hits
- arcstats_mfu_size
- arcstats_misses
- arcstats_mru_evict_data (Linux only)
- arcstats_mru_evict_metadata (Linux only)
- arcstats_mru_ghost_evict_data (Linux only)
- arcstats_mru_ghost_evict_metadata (Linux only)
- arcstats_mru_evictable_data (FreeBSD only)
- arcstats_mru_evictable_metadata (FreeBSD only)
- arcstats_mru_ghost_evictable_data (FreeBSD only)
- arcstats_mru_ghost_evictable_metadata (FreeBSD only)
- arcstats_mru_ghost_hits
- arcstats_mru_ghost_size
- arcstats_mru_hits
- arcstats_mru_size
- arcstats_mutex_miss
- arcstats_other_size
- arcstats_p
- arcstats_prefetch_data_hits
- arcstats_prefetch_data_misses
- arcstats_prefetch_metadata_hits
- arcstats_prefetch_metadata_misses
- arcstats_recycle_miss (Linux only)
- arcstats_mru_hits
- arcstats_mru_ghost_hits
- arcstats_mfu_hits
- arcstats_mfu_ghost_hits
- arcstats_deleted
- arcstats_recycle_miss
- arcstats_mutex_miss
- arcstats_evict_skip
- arcstats_evict_l2_cached
- arcstats_evict_l2_eligible
- arcstats_evict_l2_ineligible
- arcstats_hash_elements
- arcstats_hash_elements_max
- arcstats_hash_collisions
- arcstats_hash_chains
- arcstats_hash_chain_max
- arcstats_p
- arcstats_c
- arcstats_c_min
- arcstats_c_max
- arcstats_size
- arcstats_sync_wait_for_async (FreeBSD only)
#### Zfetch Stats
- zfetchstats_bogus_streams (Linux only)
- zfetchstats_colinear_hits (Linux only)
- zfetchstats_colinear_misses (Linux only)
- arcstats_hdr_size
- arcstats_data_size
- arcstats_meta_size
- arcstats_other_size
- arcstats_anon_size
- arcstats_anon_evict_data
- arcstats_anon_evict_metadata
- arcstats_mru_size
- arcstats_mru_evict_data
- arcstats_mru_evict_metadata
- arcstats_mru_ghost_size
- arcstats_mru_ghost_evict_data
- arcstats_mru_ghost_evict_metadata
- arcstats_mfu_size
- arcstats_mfu_evict_data
- arcstats_mfu_evict_metadata
- arcstats_mfu_ghost_size
- arcstats_mfu_ghost_evict_data
- arcstats_mfu_ghost_evict_metadata
- arcstats_l2_hits
- arcstats_l2_misses
- arcstats_l2_feeds
- arcstats_l2_rw_clash
- arcstats_l2_read_bytes
- arcstats_l2_write_bytes
- arcstats_l2_writes_sent
- arcstats_l2_writes_done
- arcstats_l2_writes_error
- arcstats_l2_writes_hdr_miss
- arcstats_l2_evict_lock_retry
- arcstats_l2_evict_reading
- arcstats_l2_free_on_write
- arcstats_l2_cdata_free_on_write
- arcstats_l2_abort_lowmem
- arcstats_l2_cksum_bad
- arcstats_l2_io_error
- arcstats_l2_size
- arcstats_l2_asize
- arcstats_l2_hdr_size
- arcstats_l2_compress_successes
- arcstats_l2_compress_zeros
- arcstats_l2_compress_failures
- arcstats_memory_throttle_count
- arcstats_duplicate_buffers
- arcstats_duplicate_buffers_size
- arcstats_duplicate_reads
- arcstats_memory_direct_count
- arcstats_memory_indirect_count
- arcstats_arc_no_grow
- arcstats_arc_tempreserve
- arcstats_arc_loaned_bytes
- arcstats_arc_prune
- arcstats_arc_meta_used
- arcstats_arc_meta_limit
- arcstats_arc_meta_max
- zfetchstats_hits
- zfetchstats_max_streams (FreeBSD only)
- zfetchstats_misses
- zfetchstats_reclaim_failures (Linux only)
- zfetchstats_reclaim_successes (Linux only)
- zfetchstats_streams_noresets (Linux only)
- zfetchstats_streams_resets (Linux only)
- zfetchstats_stride_hits (Linux only)
- zfetchstats_stride_misses (Linux only)
#### Vdev Cache Stats
- zfetchstats_colinear_hits
- zfetchstats_colinear_misses
- zfetchstats_stride_hits
- zfetchstats_stride_misses
- zfetchstats_reclaim_successes
- zfetchstats_reclaim_failures
- zfetchstats_streams_resets
- zfetchstats_streams_noresets
- zfetchstats_bogus_streams
- vdev_cache_stats_delegations
- vdev_cache_stats_hits
- vdev_cache_stats_misses
#### Pool Metrics (optional)
On Linux:
- zfs_pool
- nread (integer, )
- nwritten (integer, )
- reads (integer, )
- writes (integer, )
- wtime (integer, )
- wlentime (integer, )
- wupdate (integer, )
- rtime (integer, )
- rlentime (integer, )
- rupdate (integer, )
- wcnt (integer, )
- rcnt (integer, )
On FreeBSD:
- zfs_pool
- allocated (integer, bytes)
- capacity (integer, bytes)
- dedupratio (float, ratio)
- free (integer, bytes)
- size (integer, bytes)
- fragmentation (integer, percent)
### Tags:
- ZFS stats (`zfs`) will have the following tag:
- pools - A `::` concatenated list of all ZFS pools on the machine.
- Pool metrics (`zfs_pool`) will have the following tag:
- pool - with the name of the pool which the metrics are for.
- health - the health status of the pool. (FreeBSD only)
### Example Output:
```
$ ./telegraf -config telegraf.conf -input-filter zfs -test
* Plugin: zfs, Collection 1
> zfs_pool,health=ONLINE,pool=zroot allocated=1578590208i,capacity=2i,dedupratio=1,fragmentation=1i,free=64456531968i,size=66035122176i 1464473103625653908
> zfs,pools=zroot arcstats_allocated=4167764i,arcstats_anon_evictable_data=0i,arcstats_anon_evictable_metadata=0i,arcstats_anon_size=16896i,arcstats_arc_meta_limit=10485760i,arcstats_arc_meta_max=115269568i,arcstats_arc_meta_min=8388608i,arcstats_arc_meta_used=51977456i,arcstats_c=16777216i,arcstats_c_max=41943040i,arcstats_c_min=16777216i,arcstats_data_size=0i,arcstats_deleted=1699340i,arcstats_demand_data_hits=14836131i,arcstats_demand_data_misses=2842945i,arcstats_demand_hit_predictive_prefetch=0i,arcstats_demand_metadata_hits=1655006i,arcstats_demand_metadata_misses=830074i,arcstats_duplicate_buffers=0i,arcstats_duplicate_buffers_size=0i,arcstats_duplicate_reads=123i,arcstats_evict_l2_cached=0i,arcstats_evict_l2_eligible=332172623872i,arcstats_evict_l2_ineligible=6168576i,arcstats_evict_l2_skip=0i,arcstats_evict_not_enough=12189444i,arcstats_evict_skip=195190764i,arcstats_hash_chain_max=2i,arcstats_hash_chains=10i,arcstats_hash_collisions=43134i,arcstats_hash_elements=2268i,arcstats_hash_elements_max=6136i,arcstats_hdr_size=565632i,arcstats_hits=16515778i,arcstats_l2_abort_lowmem=0i,arcstats_l2_asize=0i,arcstats_l2_cdata_free_on_write=0i,arcstats_l2_cksum_bad=0i,arcstats_l2_compress_failures=0i,arcstats_l2_compress_successes=0i,arcstats_l2_compress_zeros=0i,arcstats_l2_evict_l1cached=0i,arcstats_l2_evict_lock_retry=0i,arcstats_l2_evict_reading=0i,arcstats_l2_feeds=0i,arcstats_l2_free_on_write=0i,arcstats_l2_hdr_size=0i,arcstats_l2_hits=0i,arcstats_l2_io_error=0i,arcstats_l2_misses=0i,arcstats_l2_read_bytes=0i,arcstats_l2_rw_clash=0i,arcstats_l2_size=0i,arcstats_l2_write_buffer_bytes_scanned=0i,arcstats_l2_write_buffer_iter=0i,arcstats_l2_write_buffer_list_iter=0i,arcstats_l2_write_buffer_list_null_iter=0i,arcstats_l2_write_bytes=0i,arcstats_l2_write_full=0i,arcstats_l2_write_in_l2=0i,arcstats_l2_write_io_in_progress=0i,arcstats_l2_write_not_cacheable=380i,arcstats_l2_write_passed_headroom=0i,arcstats_l2_write_pios=0i,arcstats_l2_write_spa_mismatch=0i,arcstats_l2_write_trylock_fail=0i,arcstats_l2_writes_done=0i,arcstats_l2_writes_error=0i,arcstats_l2_writes_lock_retry=0i,arcstats_l2_writes_sent=0i,arcstats_memory_throttle_count=0i,arcstats_metadata_size=17014784i,arcstats_mfu_evictable_data=0i,arcstats_mfu_evictable_metadata=16384i,arcstats_mfu_ghost_evictable_data=5723648i,arcstats_mfu_ghost_evictable_metadata=10709504i,arcstats_mfu_ghost_hits=1315619i,arcstats_mfu_ghost_size=16433152i,arcstats_mfu_hits=7646611i,arcstats_mfu_size=305152i,arcstats_misses=3676993i,arcstats_mru_evictable_data=0i,arcstats_mru_evictable_metadata=0i,arcstats_mru_ghost_evictable_data=0i,arcstats_mru_ghost_evictable_metadata=80896i,arcstats_mru_ghost_hits=324250i,arcstats_mru_ghost_size=80896i,arcstats_mru_hits=8844526i,arcstats_mru_size=16693248i,arcstats_mutex_miss=354023i,arcstats_other_size=34397040i,arcstats_p=4172800i,arcstats_prefetch_data_hits=0i,arcstats_prefetch_data_misses=0i,arcstats_prefetch_metadata_hits=24641i,arcstats_prefetch_metadata_misses=3974i,arcstats_size=51977456i,arcstats_sync_wait_for_async=0i,vdev_cache_stats_delegations=779i,vdev_cache_stats_hits=323123i,vdev_cache_stats_misses=59929i,zfetchstats_hits=0i,zfetchstats_max_streams=0i,zfetchstats_misses=0i 1464473103634124908
```
### Description
A short description for some of the metrics.
```
arcstats_hits
Total amount of cache hits in the arc.
#### Arc Stats
arcstats_misses
Total amount of cache misses in the arc.
`arcstats_hits` Total amount of cache hits in the arc.
arcstats_demand_data_hits
Amount of cache hits for demand data, this is what matters (is good) for your application/share.
`arcstats_misses` Total amount of cache misses in the arc.
arcstats_demand_data_misses
Amount of cache misses for demand data, this is what matters (is bad) for your application/share.
`arcstats_demand_data_hits` Amount of cache hits for demand data, this is what matters (is good) for your application/share.
arcstats_demand_metadata_hits
Ammount of cache hits for demand metadata, this matters (is good) for getting filesystem data (ls,find,…)
`arcstats_demand_data_misses` Amount of cache misses for demand data, this is what matters (is bad) for your application/share.
arcstats_demand_metadata_misses
Ammount of cache misses for demand metadata, this matters (is bad) for getting filesystem data (ls,find,…)
`arcstats_demand_metadata_hits` Amount of cache hits for demand metadata, this matters (is good) for getting filesystem data (ls,find,…)
arcstats_prefetch_data_hits
The zfs prefetcher tried to prefetch somethin, but it was allready cached (boring)
`arcstats_demand_metadata_misses` Amount of cache misses for demand metadata, this matters (is bad) for getting filesystem data (ls,find,…)
arcstats_prefetch_data_misses
The zfs prefetcher prefetched something which was not in the cache (good job, could become a demand hit in the future)
`arcstats_prefetch_data_hits` The zfs prefetcher tried to prefetch something, but it was already cached (boring)
arcstats_prefetch_metadata_hits
Same as above, but for metadata
`arcstats_prefetch_data_misses` The zfs prefetcher prefetched something which was not in the cache (good job, could become a demand hit in the future)
arcstats_prefetch_metadata_misses
Same as above, but for metadata
`arcstats_prefetch_metadata_hits` Same as above, but for metadata
arcstats_mru_hits
Cache hit in the “most recently used cache”, we move this to the mfu cache.
`arcstats_prefetch_metadata_misses` Same as above, but for metadata
arcstats_mru_ghost_hits
Cache hit in the “most recently used ghost list” we had this item in the cache, but evicted it, maybe we should increase the mru cache size.
`arcstats_mru_hits` Cache hit in the “most recently used cache”, we move this to the mfu cache.
arcstats_mfu_hits
Cache hit in the “most freqently used cache” we move this to the begining of the mfu cache.
`arcstats_mru_ghost_hits` Cache hit in the “most recently used ghost list” we had this item in the cache, but evicted it, maybe we should increase the mru cache size.
arcstats_mfu_ghost_hits
Cache hit in the “most frequently used ghost list” we had this item in the cache, but evicted it, maybe we should increase the mfu cache size.
`arcstats_mfu_hits` Cache hit in the “most frequently used cache” we move this to the beginning of the mfu cache.
arcstats_allocated
New data is written to the cache.
`arcstats_mfu_ghost_hits` Cache hit in the “most frequently used ghost list” we had this item in the cache, but evicted it, maybe we should increase the mfu cache size.
arcstats_deleted
Old data is evicted (deleted) from the cache.
`arcstats_allocated` New data is written to the cache.
arcstats_evict_l2_cached
We evicted something from the arc, but its still cached in the l2 if we need it.
`arcstats_deleted` Old data is evicted (deleted) from the cache.
arcstats_evict_l2_eligible
We evicted something from the arc, and its not in the l2 this is sad. (maybe we hadnt had enough time to store it there)
`arcstats_evict_l2_cached` We evicted something from the arc, but its still cached in the l2 if we need it.
arcstats_evict_l2_ineligible
We evicted something which cannot be stored in the l2.
Reasons could be:
We have multiple pools, we evicted something from a pool whithot an l2 device.
The zfs property secondarycache.
`arcstats_evict_l2_eligible` We evicted something from the arc, and its not in the l2 this is sad. (maybe we hadnt had enough time to store it there)
arcstats_c
Arc target size, this is the size the system thinks the arc should have.
`arcstats_evict_l2_ineligible` We evicted something which cannot be stored in the l2.
Reasons could be:
- We have multiple pools, we evicted something from a pool whithout an l2 device.
- The zfs property secondary cache.
arcstats_size
Total size of the arc.
`arcstats_c` Arc target size, this is the size the system thinks the arc should have.
arcstats_l2_hits
Hits to the L2 cache. (It was not in the arc, but in the l2 cache)
`arcstats_size` Total size of the arc.
arcstats_l2_misses
Miss to the L2 cache. (It was not in the arc, and not in the l2 cache)
`arcstats_l2_hits` Hits to the L2 cache. (It was not in the arc, but in the l2 cache)
arcstats_l2_size
Size of the l2 cache.
`arcstats_l2_misses` Miss to the L2 cache. (It was not in the arc, and not in the l2 cache)
arcstats_l2_hdr_size
Size of the metadata in the arc (ram) used to manage (lookup if someting is in the l2) the l2 cache.
`arcstats_l2_size` Size of the l2 cache.
`arcstats_l2_hdr_size` Size of the metadata in the arc (ram) used to manage (lookup if something is in the l2) the l2 cache.
#### Zfetch Stats
zfetchstats_hits
Counts the number of cache hits, to items wich are in the cache because of the prefetcher.
`zfetchstats_hits` Counts the number of cache hits, to items which are in the cache because of the prefetcher.
zfetchstats_colinear_hits
Counts the number of cache hits, to items wich are in the cache because of the prefetcher (prefetched linear reads)
`zfetchstats_colinear_hits` Counts the number of cache hits, to items which are in the cache because of the prefetcher (prefetched linear reads)
zfetchstats_stride_hits
Counts the number of cache hits, to items wich are in the cache because of the prefetcher (prefetched stride reads)
`zfetchstats_stride_hits` Counts the number of cache hits, to items which are in the cache because of the prefetcher (prefetched stride reads)
#### Vdev Cache Stats
`vdev_cache_stats_hits` Hits to the vdev (device level) cache.
vdev_cache_stats_hits
Hits to the vdev (device level) cache.
vdev_cache_stats_misses
Misses to the vdev (device level) cache.
```
# Default config
```
[zfs]
# ZFS kstat path
# If not specified, then default is:
# kstatPath = "/proc/spl/kstat/zfs"
#
# By default, telegraf gather all zfs stats
# If not specified, then default is:
# kstatMetrics = ["arcstats", "zfetchstats", "vdev_cache_stats"]
```
`vdev_cache_stats_misses` Misses to the vdev (device level) cache.

Some files were not shown because too many files have changed in this diff Show More