Commit Graph

236 Commits

Author SHA1 Message Date
jvassev 2c19d74829 Prevent loading config twice in K8S (#3999)
When config dir is mounted from configmap, filepath.Walk() finds the same
.conf file twice as 20-acme.conf is a link to ..data/20-acme.conf for example.

This patch skips all folder names starting with '..' which is pretty
uncommon and mainly used by Kubernetes mounts.
2018-04-11 16:51:19 -07:00
Daniel Nelson beeab2c509
Sort field names when running --test 2018-04-04 16:30:22 -07:00
Daniel Nelson b99cd14129
Add influx uint support as a runtime option (#3948) 2018-03-29 13:31:43 -07:00
Daniel Nelson 1c0f63a90d
Add new line protocol parser and serializer, influxdb output (#3924) 2018-03-27 17:30:51 -07:00
Daniel Nelson 8147d60973
Fix metric buffer limit in internal plugin after reload 2018-02-19 20:55:28 -08:00
Paul Myjavec 12d62e60b3 Use CircleCI 2.0 for builds (#3731) 2018-02-01 15:05:39 -08:00
Piotr Popieluch 601dc99606 Align aggregator period with internal ticker to avoid skipping metrics (#3693)
By the time the aggregator.run() was called about 600ms already passed since setting now which was skewing up the aggregation intervals and skipping metrics.
2018-01-18 17:37:53 -08:00
atzoum 317de40ac4 Add support for dropwizard input format (#2846) 2018-01-08 15:11:36 -08:00
Daniel Nelson b0c2bb870e
Escape environment variables during config toml parsing (#3637) 2018-01-04 15:28:00 -08:00
Maximilien Richer 5f215c22fe Fix typos in comments (#3415) 2017-10-31 17:00:06 -07:00
Daniel Nelson d0b690f040 Fix short tests on darwin (#3099) 2017-09-08 13:03:37 -07:00
Daniel Nelson 28d16188b3 Fix panic when handling string fields with escapes (#3188) 2017-08-30 21:16:37 -07:00
Daniel Nelson 573bd4aa32 Start first aggregator period at startup time (#3050)
Fixes issue where metrics collected immediately after startup would not
be aggregated.
2017-07-24 18:25:05 -07:00
Daniel Nelson 31ce98fa91 Don't match pattern on any error (#3040)
This prevents a pattern with no wildcards from matching in case
permissions is denied.
2017-07-21 14:28:14 -07:00
DanKans d903a9142d Fix filtering when both pass and drop match an item (#3036)
Adjust logic in functions responsible for passing metrics in order to be able
to process them correctly in case where pass and drop are defined together.
2017-07-21 10:53:57 -07:00
DanKans 34c042c7dc Fix combined tagdrop/tagpass filtering (#3031) 2017-07-19 13:08:40 -07:00
Daniel Nelson b165ce4cd5 Prevent possible deadlock when using aggregators (#3016)
Looping the metrics back through the same channel could result in a
deadlock, by using a new channel and locking the processor we can ensure
that all stages can make continual progress.
2017-07-13 15:34:21 -07:00
Daniel Nelson 5d2c093105 Prevent Write from being called concurrently (#3011) 2017-07-12 12:03:23 -07:00
Daniel Nelson 1388e2cf92 Do not allow metrics with trailing slashes (#3007)
It is not possible to encode a measurement, tag, or field whose last
character is a backslash due to it being an unescapable character.
Because the tight coupling between line protocol and the internal metric
model, prevent metrics like this from being created.

Measurements with a trailing slash are not allowed and the point will be
dropped.  Tags and fields with a trailing a slash will be dropped from
the point.
2017-07-11 15:54:38 -07:00
Seuf 2092443cd7 Add Kafka 0.9+ consumer support (#2487) 2017-06-07 18:22:28 -07:00
Daniel Nelson 052e88ad5e
Fix grammar 2017-04-27 14:59:18 -07:00
Patrick Hemmer 06baf7cf78 use AddError everywhere (#2372) 2017-04-24 11:13:26 -07:00
Daniel Nelson 360b10c4de Clarify precision documentation (#2655) 2017-04-12 10:42:11 -07:00
Daniel Nelson 2c98e5ae66 Add collectd parser (#2654) 2017-04-12 10:41:26 -07:00
Daniel Nelson 540f98e228 Fix possible deadlock when output cannot write. (#2610) 2017-03-31 12:45:28 -07:00
tjmcs fb1c7d0154 Adds a new json_timestamp_units configuration parameter (#2587) 2017-03-29 17:12:29 -07:00
Cameron Sparr 9e810ac463
Handle nil os.FileInfo in filepath.Walk
closes #2466
2017-02-28 17:51:03 +00:00
Cameron Sparr 06176ef410
Only set the buffer size once
fixes #2380
2017-02-17 14:11:15 +00:00
Cameron Sparr 21cf79163c
don't use influxdata/config, just use influxdata/toml 2017-02-10 17:27:18 +00:00
Cameron Sparr f7d551a807
Add more nested globpath tests 2017-02-01 23:44:35 +00:00
Cameron Sparr e87ce22af9
running output: Drop nil metrics
fixes #2317
2017-02-01 08:55:22 +00:00
Cameron Sparr 7279018cfe
readme fixup & test output fixup 2017-01-09 12:28:13 +00:00
Cameron Sparr 12db3b9120
Check if metric is nil before calling SetAggregate
fixes #2146
2016-12-13 12:27:10 +00:00
Cameron Sparr d71a42cd1b Implement telegraf collecting stats on itself
closes #1348
2016-12-05 18:56:54 +00:00
Cameron Sparr e5c7a71d8e Fix unit tests for new metric implementation 2016-12-01 18:17:02 +00:00
Cameron Sparr db7a4b24b6 Implement telegraf's own full metric type
main reasons behind this:
- make adding/removing tags cheap
- make adding/removing fields cheap
- make parsing cheaper
- make parse -> decorate -> write out bytes metric flow much faster

Refactor serializer to use byte buffer
2016-12-01 18:17:02 +00:00
Cameron Sparr 0ecbf9e349 Fix single quote parsing of TOML durations
closes #2023
2016-11-10 09:47:46 +00:00
Cameron Sparr f729fa990d Unit testing for internal.Duration Unmarshal
closes #1926
2016-10-25 13:11:32 +01:00
Alex Zorin 662db7a944 Fix panic in internal.Duration UnmarshalTOML 2016-10-25 18:30:01 +11:00
Cameron Sparr c849b58de9 http_listener input unit tests 2016-10-24 18:17:49 +01:00
Cameron Sparr 097b1e09db http listener refactor
in this commit:

- chunks out the http request body to avoid making very large
  allocations.
- establishes a limit for the maximum http request body size that the
  listener will accept.
- utilizes a pool of byte buffers to reduce GC pressure.
2016-10-24 18:17:49 +01:00
Cameron Sparr b00ad65b08 Log config file parsing errors properly
closes #1344
2016-10-12 16:50:22 +01:00
Cameron Sparr a84ce5d5cb drop metrics outside of the aggregators period 2016-10-12 14:56:03 +01:00
Cameron Sparr 8ca4a50c18 delete nil fields in the metric maker.
closes #1771
2016-10-12 14:50:19 +01:00
Cameron Sparr 03b2984ac2 Fixup some code based on feedback from @dgnorton 2016-10-12 14:50:19 +01:00
Cameron Sparr 9540a6532f Update influxdb dependency for new models.Tags 2016-10-12 14:50:19 +01:00
Cameron Sparr 78f544c0aa Support --aggregator-filter & --processor-filter 2016-10-12 14:50:19 +01:00
Cameron Sparr b03c1d9691 Support ordering of processor plugins 2016-10-12 14:50:19 +01:00
Cameron Sparr fead80844e Refactor handling of MinMax functionality into RunningAggregator
allows for easier addition of a sliding window at a later time.

Also makes `period` be a generic argument for all aggregator plugins.
2016-10-12 14:50:19 +01:00
Cameron Sparr ef885eda62 Change minmax aggregator to store float64 2016-10-12 14:50:19 +01:00
Cameron Sparr 64a71263a1 Support Processor & Aggregator Plugins
closes #1726
2016-10-12 14:50:19 +01:00
Cameron Sparr c7834209d2 Major Logging Overhaul
in this commit:

- centralize logging output handler.
- set global Info/Debug/Error log levels based on config file or flags.
- remove per-plugin debug arg handling.
- add a I!, D!, or E! to every log message.
- add configuration option to specify where to send logs.

closes #1786
2016-10-03 17:13:03 +01:00
Cameron Sparr e19845c202 Load config directory using filepath.Walk
closes #1137
2016-09-28 16:01:52 +01:00
Cameron Sparr 52134555d6 globpath: only walk tree if ** is defined
closes #1517
2016-09-28 15:44:29 +01:00
Cameron Sparr 2fe427b3b3 mongodb input: fix version 2.2 panic
closes #1628
2016-09-06 11:58:06 +01:00
Cameron Sparr 50ef3282b6 Refactor and code cleanup of filtering
started working on this with the idea of fixing #1623, although I
realized that this was actually just a documentation issue around
a toml eccentricity.

closes #1623
2016-09-05 16:30:18 +01:00
Cameron Sparr 8e22526756 Adding c:\program files\telegraf\telegraf.conf
this will now be the default config file location on windows, basically
it is the windows equivalent of /etc/telegraf/telegraf.conf

also updating the changelog

closes #1543
2016-08-08 23:17:27 +01:00
Cameron Sparr d3bb1e7010 Rename internal_models package to models 2016-08-08 14:41:40 +01:00
Cameron Sparr 49988b15a3 Default config typo fix 2016-08-06 07:40:28 +01:00
Jason Gardner 986735234b Fix output config typo. (#1527) 2016-07-22 16:05:53 +01:00
Cameron Sparr 2d4864e126 nil metric list panic fix 2016-06-29 12:08:36 +02:00
Cameron Sparr 755b2ec953 fixup: BOM Trim -> TrimPrefix 2016-06-24 08:47:31 +01:00
Cameron Sparr 5ddd61d2e2 Trim BOM from config file for windows support
closes #1378
2016-06-23 09:00:51 +01:00
Mike Glazer e3448153e1 Allow for TLS connections to ElasticSearch (#1398)
* Allow for TLS connections to ElasticSearch

Extremely similar implementation to the HTTP JSON module's
implementation of the same code.

* Changelog update
2016-06-22 16:23:49 +01:00
Cameron Sparr cb3c54a1ae logparser input plugin
closes #102
closes #328
2016-06-21 14:23:01 +01:00
Cameron Sparr d7efb7a71d Add precision rounding to accumulator
Adding precision rounding to the accumulator. This means that now every
input metric will get rounded at collection, rather than at write (and
only for the influxdb output).

This feature is disabled for service inputs, because service inputs
should be in control of their own timestamps & precisions.
2016-06-14 00:36:39 +01:00
Cameron Sparr 2c448e22e1 New object: ErrChan for concurrent err handling 2016-06-02 13:29:37 +01:00
Cameron Sparr bd1282eddf Don't print config with trailing whitespace 2016-05-31 16:25:02 +01:00
Cameron Sparr 892abec025 Refactor collection_jitter and flush_jitter
use a common function between collection_jitter and flush_jitter. which
creates the same behavior between the two options.

going forward, both jitters will be random sleeps that get re-evaluated
at runtime for every interval (previously only collection_jitter did
this)

also fixes behavior so that both jitters will exit in the event of a
process exit.

closes #1296
2016-05-31 14:24:32 +01:00
Cameron Sparr 9ff536d94d Limit GetMetricStatistics to 10 per second
closes #1197
2016-05-31 11:26:52 +01:00
Cameron Sparr 9bbdb2d562 Allow wildcard filtering of varnish stats
closes #1275
2016-05-26 10:42:34 +01:00
John Engelman 5f3a91bffd Consolidate AWS credentials (#1208)
* Use shared AWS credential configuration.

*  Cloudwatch dimension wilcards 

* Allow configuring cache_ttl for cloudwatch metrics.

* Allow for wildcard in dimension values to select all available metrics.

* Use internal.Duration for CacheTTL and go fmt

* Refactor to not use embedded structs for config.

* Update AWS plugin READMEs with credentials details, update Changelog.

* Fix changelog after rebasing to master and 0.13.1 release.

* Fix changelog after rebase.
2016-05-25 12:30:39 +01:00
Cameron Sparr c44ecf54a5 Utilize timeout in net_response plugin.
Also changing the net_response and http_response plugins to only accept
duration strings for their timeout parameters. This is a breaking config
file change.

closes #1214
2016-05-23 15:59:23 +01:00
Cameron Sparr 4e9798d0e6 agent and tags configs sometimes not applied
closes #1090
2016-04-29 19:44:01 -06:00
Cameron Sparr 3f807a9432 Implement timeouts for all exec command runners
First is to write an internal CombinedOutput and Run function with a
timeout.

Second, the following instances of command runners need to have timeouts:

    plugins/inputs/ping/ping.go
    125:	out, err := c.CombinedOutput()

    plugins/inputs/exec/exec.go
    91:	if err := cmd.Run(); err != nil {

    plugins/inputs/ipmi_sensor/command.go
    31:	err := cmd.Run()

    plugins/inputs/sysstat/sysstat.go
    194:	out, err := cmd.CombinedOutput()

    plugins/inputs/leofs/leofs.go
    185:	defer cmd.Wait()

    plugins/inputs/sysstat/sysstat.go
    282:	if err := cmd.Wait(); err != nil {

closes #1067
2016-04-29 12:06:22 -06:00
Cameron Sparr cbe32c7482 Support default config paths
precedence will be:

 1. --config command-line option
 2. $TELEGRAF_CONFIG_PATH
 3. $HOME/.telegraf/telegraf.conf
 4. /etc/telegraf/telegraf.conf
2016-04-28 17:48:24 -06:00
Cameron Sparr 4de75ce621 Performance refactor of running_output buffers
closes #914
closes #967
2016-04-27 13:40:05 -06:00
Cameron Sparr c7719ac365 buffers: fix bug when Write called before AddMetric 2016-04-26 10:25:04 -06:00
Pierre Fersing c159460b2c Refactor running_output buffering
closes #1087
2016-04-25 16:32:41 -06:00
Cameron Sparr 07728d7425 Refactor globpath pkg to return a map
this is so that we don't call os.Stat twice for every file matched
by Match(). Also changing the behavior to _not_ return the name of a
file that doesn't exist if it's not a glob.
2016-04-24 14:37:44 -06:00
Cameron Sparr d3a25e4dc1 globpath refactor into pkg separate from filestat 2016-04-23 11:56:33 -06:00
Cameron Sparr 81d0a64d46 Adds support for removing/keeping tags from metrics
closes #706
2016-04-16 15:13:38 -06:00
Cameron Sparr f5878eafb9 Create a template system for the graphite serializer
closes #925
closes #879
2016-04-11 16:30:18 -06:00
Cameron Sparr 8e041420cd config: parse environment variables in the config file
closes #663
2016-04-01 16:06:19 -06:00
Cameron Sparr e436b2d720 Cleanup & standardize config file
changes:

- -sample-config will now comment out all but a few default plugins.
- config file parse errors will output path to bad conf file.
- cleanup 80-char line-length and some other style issues.
- default package conf file will now have all plugins, but commented
  out.

closes #199
closes #944
2016-04-01 10:59:53 -06:00
Cameron Sparr 21ece2d76d Convert ipmi stats/tags to underscore and lowercase
closes #888
2016-03-29 11:39:57 -06:00
Cameron Sparr f543dbb42f Allow users to tell telegraf Agent not to include host tag
closes #848
2016-03-21 15:51:10 -06:00
Cameron Sparr 77dcbe95c0 Do not write metrics if there are 0 to write
closes #884
2016-03-18 10:54:51 -06:00
Cameron Sparr 061b749041 TLS config: if only given ssl_ca, create tls config anyways
fixes #890
2016-03-18 10:53:55 -06:00
Cameron Sparr 5c1b635229 Value parser, for parsing a single value into a metric
closes #849
2016-03-17 20:08:21 -06:00
Eugene Dementiev 530b4f3bee [amqp output] Allow external auth (cert-based tls auth)
closes #863
2016-03-16 19:03:41 -06:00
Cameron Sparr 845abcdd77 Only log the overwritten metric warning on 1st overwrite per buffer
see #807
2016-03-09 14:44:32 +01:00
Pierre Fersing 29d1cbb673 Reduce metric_buffer_limit to 1000
closes #780
2016-03-03 15:32:44 +00:00
Cameron Sparr 9af8d6912a Remove naoina/toml dependency, use influxdata/toml
closes #745
2016-03-01 10:17:02 +00:00
Cameron Sparr 2a6ff16819 Fix up config panic points for naoina/toml support
closes #736
2016-02-22 14:44:33 -07:00
Cameron Sparr 5aef725c13 Change pass/drop to namepass/namedrop for outputs
closes #730
2016-02-22 13:35:06 -07:00
Thibault Cohen d00550c45f Add metric pass/drop filter 2016-02-22 12:11:33 -07:00
Cameron Sparr 9ce8d78835 Set running output quiet mode in agent connect func
closes #701
2016-02-22 11:42:02 -07:00
Cameron Sparr 187d1b853d Update Makefile to 'go install' rather than 'go build' 2016-02-18 16:48:59 -07:00
Cameron Sparr 8d2e5f0bda Seems to be a toml parse bug around triple pounds 2016-02-18 14:36:03 -07:00
Sergio Jimenez 963c51f473 fix(config): Made sample config consistent.
closes #682
2016-02-18 10:01:03 -07:00
Sergio Jimenez 38ac9d2ecf List mesos in main README
And on the test configuration file
2016-02-18 09:00:27 +01:00
Cameron Sparr 88c83277c6 Write unit tests for RunningOutput 2016-02-17 17:06:34 -07:00
Cameron Sparr d0734b105b Start service plugins immediately, fix off-by-one bug 2016-02-17 15:10:32 -07:00
Cameron Sparr ee468be696 Flush based on buffer size rather than time
this includes:
- Add Accumulator to the Start() function of service inputs
- For message consumer plugins, use the Accumulator to constantly add
  metrics and make Gather a dummy function
- rework unit tests to match this new behavior.
- make "flush_buffer_when_full" a config option that defaults to true

closes #666
2016-02-16 22:25:22 -07:00
Cameron Sparr a9c135488e Add Serializer plugins, and 'file' output plugin 2016-02-12 14:13:49 -07:00
Cameron Sparr 28664fedb2 Support exec input plugin legacy behavior 2016-02-10 13:26:02 -07:00
Cameron Sparr fc7fa4b6c5 Cleanup comments and indentation in config file 2016-02-09 11:01:50 -07:00
Cameron Sparr e619493ece Implementing generic parser plugins and documentation
This constitutes a large change in how we will parse different data
formats going forward (for the plugins that support it)

This is working off @henrypfhu's changes.
2016-02-08 21:08:44 -07:00
Henry Hu 1449c8b887 Add Graphite line protocol parsing to exec plugin
closes #637
2016-02-08 17:12:28 -07:00
Cameron Sparr 6b06a23102 Change [tags] to [global_tags] to deal with toml bug
closes #662
2016-02-08 16:20:47 -07:00
Cameron Sparr bd9c5b6995 mqtt output: cleanup, implement TLS
Also normalize TLS config across all output plugins and normalize
comment strings as well.
2016-02-04 10:44:37 -07:00
Cameron Sparr 962325cc40 Warn when metrics are being overwritten
closes #601
2016-01-28 14:00:14 -07:00
Cameron Sparr c549ab907a Throughout telegraf, use telegraf.Metric rather than client.Point
closes #599
2016-01-27 23:47:32 -07:00
Cameron Sparr 9c0d14bb60 Create public models for telegraf metrics, accumlator, plugins
This will basically make the root directory a place for storing the
major telegraf interfaces, which will make telegraf's godoc looks quite
a bit nicer. And make it easier for contributors to lookup the few data
types that they actually care about.

closes #564
2016-01-27 15:42:50 -07:00
Cameron Sparr 6ebb6bc7ee Fix SNMP unit tests on OSX, improve tag config doc
closes #592
2016-01-27 11:27:10 -07:00
Tim Raymond 40d0da404e Change configuration package to influxdata/config
We are unifying the way that we handle configuration across the products
into the influxdata/config package. This provides the same API as
naoina/toml that was used previously, but provides some additional
features such as support for documenting generated TOML configs as well
as support for handling default options. This replaces all usage of
naoina/toml with influxdata/config.
2016-01-25 12:02:44 -07:00
Cameron Sparr 5349a3b6d1 Implement a per-output fixed size metric buffer
Also moved some objects out of config.go and put them in their own
package, internal/models

fixes #568
closes #285
2016-01-22 16:29:02 -07:00
Jack Zampolin 0cdf1b07e9 Fix issue 524 2016-01-20 10:57:35 -08:00
Cameron Sparr d3a5cca1bc Collection interval random jittering
closes #460
2016-01-19 13:22:54 -07:00
Cameron Sparr f60c090e4c Add a quiet mode to telegraf
closes #514
2016-01-15 13:31:04 -07:00
Philip Silva 3c89847489 internal: FlattenJSON, flatten arrays as well
With HTTP JSON or Elasticsearch, one can also process values nested in arrays.
2016-01-14 09:52:58 -08:00
Cameron Sparr 6496d185ab add backwards-compatability for 'plugins', remove [inputs] and [outputs] headers 2016-01-08 12:49:50 -07:00
Cameron Sparr 9c5db1057d renaming plugins -> inputs 2016-01-07 15:04:30 -07:00
Cameron Sparr 64b98a9b61 0.3.0 unit tests: agent and prometheus 2016-01-07 10:52:46 -07:00
Cameron Sparr 4fdcb136bc 0.3.0 unit tests: internal 2016-01-07 10:23:38 -07:00
Cameron Sparr 524fddedb4 0.3.0 unit tests: exec, httpjson, and haproxy 2016-01-06 16:11:16 -07:00
Cameron Sparr a34418d724 backwards compatability for io->diskio change 2016-01-05 23:48:59 -07:00
Cameron Sparr 3be111a160 Breakout JSON flattening into internal package, exec & elasticsearch aggregation 2016-01-05 23:48:59 -07:00
Cameron Sparr 50fc3ec974 Updating system plugins for 0.3.0 2016-01-05 23:48:59 -07:00
Cameron Sparr 03e66d5b87 Implement Glob matching for pass/drop filters 2015-12-07 16:58:31 -07:00
Tait Clarridge 22afc99f1e Add support for pass/drop/tagpass/tagdrop for outputs
Reuses same logic as the plugins for filtering points, should be only
a marginal performance decrease to check all the points before writing
to the output.

Added examples to the README as well (for generic pass/drop as well as
output pass/drop/tagpass/tagdrop).

X-Github-Closes #398

closes #398
closes #401
2015-12-04 15:51:53 -07:00
Cameron Sparr e6517d4140 Update gopsutil godep dependency. Dont use godep go build anymore
godep seems to have a problem when dependencies have `internal`
packages. So removing `godep go build` and `godep go test` from the
build process in favor of just checking out the correct revisions using
`godep restore` into the regular GOPATH.

This basically means that we are not actually using anything within the
Godeps directory except Godeps.json. I should probably make a separate
go dependency management system that does this.
2015-12-04 12:22:16 -07:00
Cameron Sparr a5f2d5ff21 Put Agent Config into the config package 2015-11-30 10:31:31 -07:00
Cameron Sparr 979e5f193a Overhaul config <-> agent coupling. Put config in it's own package. 2015-11-25 19:07:04 -07:00
Cameron Sparr acf9c1141a Change duration -> internal and implement private gopsutil methods 2015-11-11 15:38:22 -07:00