Merge remote-tracking branch 'influxdata/master'

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ### Release Notes
 
+- Telegraf now supports being installed as an official windows service,
+which can be installed via
+`> C:\Program Files\Telegraf\telegraf.exe --service install`
+
 **Breaking Change**: Aerospike main server node measurements have been renamed
 aerospike_node. Aerospike namespace measurements have been renamed to
 aerospike_namespace. They will also now be tagged with the node_name
@@ -75,6 +79,7 @@ consistent with the behavior of `collection_jitter`.
 - [#1278](https://github.com/influxdata/telegraf/pull/1278) & [#1288](https://github.com/influxdata/telegraf/pull/1288) & [#1295](https://github.com/influxdata/telegraf/pull/1295): RabbitMQ/Apache/InfluxDB inputs: made url(s) parameter optional by using reasonable input defaults if not specified
 - [#1296](https://github.com/influxdata/telegraf/issues/1296): Refactor of flush_jitter argument.
 - [#1213](https://github.com/influxdata/telegraf/issues/1213): Add inactive & active memory to mem plugin.
+- [#1543](https://github.com/influxdata/telegraf/pull/1543): Official Windows service.
 
 ### Bugfixes
 

CONTRIBUTING.md

@@ -11,6 +11,7 @@ Output plugins READMEs are less structured,
 but any information you can provide on how the data will look is appreciated.
 See the [OpenTSDB output](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/opentsdb)
 for a good example.
+1. **Optional:** Help users of your plugin by including example queries for populating dashboards. Include these sample queries in the `README.md` for the plugin.
 1. **Optional:** Write a [tickscript](https://docs.influxdata.com/kapacitor/v1.0/tick/syntax/) for your plugin and add it to [Kapacitor](https://github.com/influxdata/kapacitor/tree/master/examples/telegraf). Or mention @jackzampolin in a PR comment with some common queries that you would want to alert on and he will write one for you.
 
 ## GoDoc

Godeps

@@ -30,6 +30,8 @@ github.com/influxdata/config b79f6829346b8d6e78ba73544b1e1038f1f1c9da
 github.com/influxdata/influxdb e094138084855d444195b252314dfee9eae34cab
 github.com/influxdata/telegraf 59e981d971ee6b1f97427b7cf2f15783234d59ca
 github.com/influxdata/toml af4df43894b16e3fd2b788d01bd27ad0776ef2d0
+github.com/kardianos/osext 29ae4ffbc9a6fe9fb2bc5029050ce6996ea1d3bc
+github.com/kardianos/service 5e335590050d6d00f3aa270217d288dda1c94d0a
 github.com/klauspost/crc32 19b0b332c9e4516a6370a0456e6182c3b5036720
 github.com/lib/pq e182dc4027e2ded4b19396d638610f2653295f36
 github.com/matttproud/golang_protobuf_extensions d0c3fe89de86839aecf2e0579c40ba3bb336a453

Godeps_windows

@@ -1,59 +1,6 @@
-github.com/Microsoft/go-winio 9f57cbbcbcb41dea496528872a4f0e37a4f7ae98
+github.com/Microsoft/go-winio ce2922f643c8fd76b46cadc7f404a06282678b34
-github.com/Shopify/sarama 8aadb476e66ca998f2f6bb3c993e9a2daa3666b9
-github.com/Sirupsen/logrus 219c8cb75c258c552e999735be6df753ffc7afdc
 github.com/StackExchange/wmi f3e2bae1e0cb5aef83e319133eabfee30013a4a5
-github.com/amir/raidman 53c1b967405155bfc8758557863bf2e14f814687
+github.com/go-ole/go-ole be49f7c07711fcb603cff39e1de7c67926dc0ba7
-github.com/aws/aws-sdk-go 13a12060f716145019378a10e2806c174356b857
+github.com/lxn/win 950a0e81e7678e63d8e6cd32412bdecb325ccd88
-github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4
+github.com/shirou/w32 3c9377fc6748f222729a8270fe2775d149a249ad
-github.com/cenkalti/backoff 4dc77674aceaabba2c7e3da25d4c823edfb73f99
+golang.org/x/sys a646d33e2ee3172a661fc09bca23bb4889a41bc8
-github.com/couchbase/go-couchbase cb664315a324d87d19c879d9cc67fda6be8c2ac1
-github.com/couchbase/gomemcached a5ea6356f648fec6ab89add00edd09151455b4b2
-github.com/couchbase/goutils 5823a0cbaaa9008406021dc5daf80125ea30bba6
-github.com/dancannon/gorethink e7cac92ea2bc52638791a021f212145acfedb1fc
-github.com/davecgh/go-spew 5215b55f46b2b919f50a1df0eaa5886afe4e3b3d
-github.com/docker/engine-api 8924d6900370b4c7e7984be5adc61f50a80d7537
-github.com/docker/go-connections f549a9393d05688dff0992ef3efd8bbe6c628aeb
-github.com/docker/go-units 5d2041e26a699eaca682e2ea41c8f891e1060444
-github.com/eapache/go-resiliency b86b1ec0dd4209a588dc1285cdd471e73525c0b3
-github.com/eapache/queue ded5959c0d4e360646dc9e9908cff48666781367
-github.com/eclipse/paho.mqtt.golang 0f7a459f04f13a41b7ed752d47944528d4bf9a86
-github.com/go-ole/go-ole 50055884d646dd9434f16bbb5c9801749b9bafe4
-github.com/go-sql-driver/mysql 1fca743146605a172a266e1654e01e5cd5669bee
-github.com/golang/protobuf 552c7b9542c194800fd493123b3798ef0a832032
-github.com/golang/snappy 427fb6fc07997f43afa32f35e850833760e489a7
-github.com/gonuts/go-shellquote e842a11b24c6abfb3dd27af69a17f482e4b483c2
-github.com/gorilla/context 1ea25387ff6f684839d82767c1733ff4d4d15d0a
-github.com/gorilla/mux c9e326e2bdec29039a3761c07bece13133863e1e
-github.com/hailocab/go-hostpool e80d13ce29ede4452c43dea11e79b9bc8a15b478
-github.com/influxdata/config b79f6829346b8d6e78ba73544b1e1038f1f1c9da
-github.com/influxdata/influxdb e3fef5593c21644f2b43af55d6e17e70910b0e48
-github.com/influxdata/toml af4df43894b16e3fd2b788d01bd27ad0776ef2d0
-github.com/klauspost/crc32 19b0b332c9e4516a6370a0456e6182c3b5036720
-github.com/lib/pq e182dc4027e2ded4b19396d638610f2653295f36
-github.com/lxn/win 9a7734ea4db26bc593d52f6a8a957afdad39c5c1
-github.com/matttproud/golang_protobuf_extensions d0c3fe89de86839aecf2e0579c40ba3bb336a453
-github.com/miekg/dns cce6c130cdb92c752850880fd285bea1d64439dd
-github.com/mreiferson/go-snappystream 028eae7ab5c4c9e2d1cb4c4ca1e53259bbe7e504
-github.com/naoina/go-stringutil 6b638e95a32d0c1131db0e7fe83775cbea4a0d0b
-github.com/nats-io/nats b13fc9d12b0b123ebc374e6b808c6228ae4234a3
-github.com/nats-io/nuid 4f84f5f3b2786224e336af2e13dba0a0a80b76fa
-github.com/nsqio/go-nsq 0b80d6f05e15ca1930e0c5e1d540ed627e299980
-github.com/prometheus/client_golang 18acf9993a863f4c4b40612e19cdd243e7c86831
-github.com/prometheus/client_model fa8ad6fec33561be4280a8f0514318c79d7f6cb6
-github.com/prometheus/common e8eabff8812b05acf522b45fdcd725a785188e37
-github.com/prometheus/procfs 406e5b7bfd8201a36e2bb5f7bdae0b03380c2ce8
-github.com/samuel/go-zookeeper 218e9c81c0dd8b3b18172b2bbfad92cc7d6db55f
-github.com/shirou/gopsutil 1f32ce1bb380845be7f5d174ac641a2c592c0c42
-github.com/shirou/w32 ada3ba68f000aa1b58580e45c9d308fe0b7fc5c5
-github.com/soniah/gosnmp b1b4f885b12c5dcbd021c5cee1c904110de6db7d
-github.com/streadway/amqp b4f3ceab0337f013208d31348b578d83c0064744
-github.com/stretchr/testify 1f4a1643a57e798696635ea4c126e9127adb7d3c
-github.com/wvanbergen/kafka 46f9a1cf3f670edec492029fadded9c2d9e18866
-github.com/wvanbergen/kazoo-go 0f768712ae6f76454f987c3356177e138df258f8
-github.com/zensqlmonitor/go-mssqldb ffe5510c6fa5e15e6d983210ab501c815b56b363
-golang.org/x/net 6acef71eb69611914f7a30939ea9f6e194c78172
-golang.org/x/text a71fd10341b064c10f4a81ceac72bcf70f26ea34
-gopkg.in/dancannon/gorethink.v1 7d1af5be49cb5ecc7b177bf387d232050299d6ef
-gopkg.in/fatih/pool.v2 cba550ebf9bce999a02e963296d4bc7a486cb715
-gopkg.in/mgo.v2 d90005c5262a3463800497ea5a89aed5fe22c886
-gopkg.in/yaml.v2 a83829b6f1293c91addabc89d0571c246397bbf4

Makefile

@@ -37,6 +37,7 @@ prepare:
 # Use the windows godeps file to prepare dependencies
 prepare-windows:
 	go get github.com/sparrc/gdm
+	gdm restore
 	gdm restore -f Godeps_windows
 
 # Run all docker containers necessary for unit tests

cmd/telegraf/telegraf.go

@@ -6,6 +6,7 @@ import (
 	"log"
 	"os"
 	"os/signal"
+	"runtime"
 	"strings"
 	"syscall"
 
@@ -15,6 +16,7 @@ import (
 	_ "github.com/influxdata/telegraf/plugins/inputs/all"
 	"github.com/influxdata/telegraf/plugins/outputs"
 	_ "github.com/influxdata/telegraf/plugins/outputs/all"
+	"github.com/kardianos/service"
 )
 
 var fDebug = flag.Bool("debug", false,
@@ -39,6 +41,8 @@ var fOutputList = flag.Bool("output-list", false,
 	"print available output plugins.")
 var fUsage = flag.String("usage", "",
 	"print usage for a plugin, ie, 'telegraf -usage mysql'")
+var fService = flag.String("service", "",
+	"operate on the service")
 
 // Telegraf version, populated linker.
 //   ie, -ldflags "-X main.version=`git describe --always --tags`"
@@ -68,6 +72,7 @@ The flags are:
   -debug             print metrics as they're generated to stdout
   -quiet             run in quiet mode
   -version           print the version to stdout
+  -service           Control the service, ie, 'telegraf -service install (windows only)'
 
 In addition to the -config flag, telegraf will also load the config file from
 an environment variable or default location. Precedence is:
@@ -94,7 +99,22 @@ Examples:
   telegraf -config telegraf.conf -input-filter cpu:mem -output-filter influxdb
 `
 
-func main() {
+var logger service.Logger
+
+var stop chan struct{}
+
+var srvc service.Service
+var svcConfig *service.Config
+
+type program struct{}
+
+func reloadLoop(stop chan struct{}, s service.Service) {
+	defer func() {
+		if service.Interactive() {
+			os.Exit(0)
+		}
+		return
+	}()
 	reload := make(chan bool, 1)
 	reload <- true
 	for <-reload {
@@ -154,6 +174,15 @@ func main() {
 				}
 			}
 			return
+		case *fService != "" && runtime.GOOS == "windows":
+			if *fConfig != "" {
+				(*svcConfig).Arguments = []string{"-config", *fConfig}
+			}
+			err := service.Control(s, *fService)
+			if err != nil {
+				log.Fatal(err)
+			}
+			return
 		}
 
 		// If no other options are specified, load the config file and run.
@@ -209,14 +238,18 @@ func main() {
 		signals := make(chan os.Signal)
 		signal.Notify(signals, os.Interrupt, syscall.SIGHUP)
 		go func() {
-			sig := <-signals
+			select {
-			if sig == os.Interrupt {
+			case sig := <-signals:
-				close(shutdown)
+				if sig == os.Interrupt {
-			}
+					close(shutdown)
-			if sig == syscall.SIGHUP {
+				}
-				log.Printf("Reloading Telegraf config\n")
+				if sig == syscall.SIGHUP {
-				<-reload
+					log.Printf("Reloading Telegraf config\n")
-				reload <- true
+					<-reload
+					reload <- true
+					close(shutdown)
+				}
+			case <-stop:
 				close(shutdown)
 			}
 		}()
@@ -245,3 +278,46 @@ func usageExit(rc int) {
 	fmt.Println(usage)
 	os.Exit(rc)
 }
+
+func (p *program) Start(s service.Service) error {
+	srvc = s
+	go p.run()
+	return nil
+}
+func (p *program) run() {
+	stop = make(chan struct{})
+	reloadLoop(stop, srvc)
+}
+func (p *program) Stop(s service.Service) error {
+	close(stop)
+	return nil
+}
+
+func main() {
+	if runtime.GOOS == "windows" {
+		svcConfig = &service.Config{
+			Name:        "telegraf",
+			DisplayName: "Telegraf Data Collector Service",
+			Description: "Collects data using a series of plugins and publishes it to" +
+				"another series of plugins.",
+			Arguments: []string{"-config", "C:\\Program Files\\Telegraf\\telegraf.conf"},
+		}
+
+		prg := &program{}
+		s, err := service.New(prg, svcConfig)
+		if err != nil {
+			log.Fatal(err)
+		}
+		logger, err = s.Logger(nil)
+		if err != nil {
+			log.Fatal(err)
+		}
+		err = s.Run()
+		if err != nil {
+			logger.Error(err)
+		}
+	} else {
+		stop = make(chan struct{})
+		reloadLoop(stop, nil)
+	}
+}

docs/LICENSE_OF_DEPENDENCIES.md

@@ -16,6 +16,7 @@
 - github.com/hashicorp/go-msgpack [BSD LICENSE](https://github.com/hashicorp/go-msgpack/blob/master/LICENSE)
 - github.com/hashicorp/raft [MPL LICENSE](https://github.com/hashicorp/raft/blob/master/LICENSE)
 - github.com/hashicorp/raft-boltdb [MPL LICENSE](https://github.com/hashicorp/raft-boltdb/blob/master/LICENSE)
+- github.com/kardianos/service [ZLIB LICENSE](https://github.com/kardianos/service/blob/master/LICENSE) (License not named but matches word for word with ZLib)
 - github.com/lib/pq [MIT LICENSE](https://github.com/lib/pq/blob/master/LICENSE.md)
 - github.com/matttproud/golang_protobuf_extensions [APACHE LICENSE](https://github.com/matttproud/golang_protobuf_extensions/blob/master/LICENSE)
 - github.com/naoina/go-stringutil [MIT LICENSE](https://github.com/naoina/go-stringutil/blob/master/LICENSE)

docs/WINDOWS_SERVICE.md

@@ -1,36 +1,40 @@
 # Running Telegraf as a Windows Service
 
-If you have tried to install Go binaries as Windows Services with the **sc.exe**
+Telegraf natively supports running as a Windows Service. Outlined below is are
-tool you may have seen that the service errors and stops running after a while.
+the general steps to set it up.
 
-**NSSM** (the Non-Sucking Service Manager) is a tool that helps you in a 
+1. Obtain the telegraf windows distribution
-[number of scenarios](http://nssm.cc/scenarios) including running Go binaries
+2. Create the directory `C:\Program Files\Telegraf` (if you install in a different
-that were not specifically designed to run only in Windows platforms.
+   location simply specify the `-config` parameter with the desired location)
+3. Place the telegraf.exe and the config file into `C:\Program Files\Telegraf`
+4. To install the service into the Windows Service Manager, run (as an
+   administrator):
 
-## NSSM Installation via Chocolatey
+   ```
+   > C:\Program Files\Telegraf\telegraf.exe --service install
+   ```
 
-You can install [Chocolatey](https://chocolatey.org/) and [NSSM](http://nssm.cc/) 
+5. Edit the configuration file to meet your needs
-with these commands
+6. To check that it works, run:
 
-```powershell
+   ```
-iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
+   > C:\Program Files\Telegraf\telegraf.exe --config C:\Program Files\Telegraf\telegraf.conf --test
-choco install -y nssm
+   ```
-```
 
-## Installing Telegraf as a Windows Service with NSSM
+7. To start collecting data, run:
 
-You can download the latest Telegraf Windows binaries (still Experimental at 
+   ```
-the moment) from [the Telegraf Github repo](https://github.com/influxdata/telegraf).
+   > net start telegraf
+   ```
 
-Then you can create a C:\telegraf folder, unzip the binary there and modify the 
+## Other supported operations
-**telegraf.conf** sample to allocate the metrics you want to send to **InfluxDB**.
 
-Once you have NSSM installed in your system, the process is quite straightforward.
+Telegraf can manage its own service through the --service flag:
-You only need to type this command in your Windows shell
 
-```powershell
+| Command                            | Effect                        |
-nssm install Telegraf c:\telegraf\telegraf.exe -config c:\telegraf\telegraf.config
+|------------------------------------|-------------------------------|
-```
+| `telegraf.exe --service install`   | Install telegraf as a service |
+| `telegraf.exe --service uninstall` | Remove the telegraf service   |
+| `telegraf.exe --service start`     | Start the telegraf service    |
+| `telegraf.exe --service stop`      | Stop the telegraf service     |
 
-And now your service will be installed in Windows and you will be able to start and
-stop it gracefully

internal/config/config.go

@@ -9,6 +9,7 @@ import (
 	"os"
 	"path/filepath"
 	"regexp"
+	"runtime"
 	"sort"
 	"strings"
 	"time"
@@ -432,6 +433,9 @@ func getDefaultConfigPath() (string, error) {
 	envfile := os.Getenv("TELEGRAF_CONFIG_PATH")
 	homefile := os.ExpandEnv("${HOME}/.telegraf/telegraf.conf")
 	etcfile := "/etc/telegraf/telegraf.conf"
+	if runtime.GOOS == "windows" {
+		etcfile = `C:\Program Files\Telegraf\telegraf.conf`
+	}
 	for _, path := range []string{envfile, homefile, etcfile} {
 		if _, err := os.Stat(path); err == nil {
 			log.Printf("Using config file: %s", path)

plugins/inputs/EXAMPLE_README.md

@@ -27,6 +27,14 @@ The example plugin gathers metrics about example things
     - tag2
 - measurement2 has the following tags:
     - tag3
+    
+### Sample Queries:
+
+These are some useful queries (to generate dashboards or other) to run against data from this plugin:
+
+```
+SELECT max(field1), mean(field1), min(field1) FROM measurement1 WHERE tag1=bar AND time > now() - 1h GROUP BY tag
+```
 
 ### Example Output:
 

plugins/inputs/mock_Plugin.go

@@ -6,10 +6,22 @@ import (
 	"github.com/stretchr/testify/mock"
 )
 
+// MockPlugin struct should be named the same as the Plugin
 type MockPlugin struct {
 	mock.Mock
 }
 
+// Description will appear directly above the plugin definition in the config file
+func (m *MockPlugin) Description() string {
+	return `This is an example plugin`
+}
+
+// SampleConfig will populate the sample configuration portion of the plugin's configuration
+func (m *MockPlugin) SampleConfig() string {
+	return `  sampleVar = 'foo'`
+}
+
+// Gather defines what data the plugin will gather.
 func (m *MockPlugin) Gather(_a0 telegraf.Accumulator) error {
 	ret := m.Called(_a0)
 

plugins/outputs/kafka/README.md

@@ -0,0 +1,67 @@
+# Kafka Producer Output Plugin
+
+This plugin writes to a [Kafka Broker](http://kafka.apache.org/07/quickstart.html) acting a Kafka Producer.
+
+```
+[[outputs.kafka]]
+  ## URLs of kafka brokers
+  brokers = ["localhost:9092"]
+  ## Kafka topic for producer messages
+  topic = "telegraf"
+  ## Telegraf tag to use as a routing key
+  ##  ie, if this tag exists, it's value will be used as the routing key
+  routing_tag = "host"
+
+  ## CompressionCodec represents the various compression codecs recognized by
+  ## Kafka in messages.
+  ##  0 : No compression
+  ##  1 : Gzip compression
+  ##  2 : Snappy compression
+  compression_codec = 0
+
+  ##  RequiredAcks is used in Produce Requests to tell the broker how many
+  ##  replica acknowledgements it must see before responding
+  ##   0 : the producer never waits for an acknowledgement from the broker.
+  ##       This option provides the lowest latency but the weakest durability
+  ##       guarantees (some data will be lost when a server fails).
+  ##   1 : the producer gets an acknowledgement after the leader replica has
+  ##       received the data. This option provides better durability as the
+  ##       client waits until the server acknowledges the request as successful
+  ##       (only messages that were written to the now-dead leader but not yet
+  ##       replicated will be lost).
+  ##   -1: the producer gets an acknowledgement after all in-sync replicas have
+  ##       received the data. This option provides the best durability, we
+  ##       guarantee that no messages will be lost as long as at least one in
+  ##       sync replica remains.
+  required_acks = -1
+
+  ##  The total number of times to retry sending a message
+  max_retry = 3
+
+  ## Optional SSL Config
+  # ssl_ca = "/etc/telegraf/ca.pem"
+  # ssl_cert = "/etc/telegraf/cert.pem"
+  # ssl_key = "/etc/telegraf/key.pem"
+  ## Use SSL but skip chain & host verification
+  # insecure_skip_verify = false
+
+  data_format = "influx"
+```
+
+### Required parameters:
+
+* `brokers`: List of strings, this is for speaking to a cluster of `kafka` brokers. On each flush interval, Telegraf will randomly choose one of the urls to write to. Each URL should just include host and port e.g. -> `["{host}:{port}","{host2}:{port2}"]`
+* `topic`: The `kafka` topic to publish to.
+
+
+### Optional parameters:
+
+* `routing_tag`:  if this tag exists, it's value will be used as the routing key
+* `compression_codec`: What level of compression to use: `0` -> no compression, `1` -> gzip compression, `2` -> snappy compression
+* `required_acks`: a setting for how may `acks` required from the `kafka` broker cluster.
+* `max_retry`: Max number of times to retry failed write
+* `ssl_ca`: SSL CA
+* `ssl_cert`: SSL CERT
+* `ssl_key`: SSL key
+* `insecure_skip_verify`: Use SSL but skip chain & host verification (default: false)
+* `data_format`: [About Telegraf data formats](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md)