Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
768e701539 | ||
|
|
b2c2a64da9 |
69
CHANGELOG.md
69
CHANGELOG.md
@@ -1,72 +1,3 @@
|
||||
## v0.1.9 [unreleased]
|
||||
|
||||
### Release Notes
|
||||
- InfluxDB output config change: `url` is now `urls`, and is a list. Config files
|
||||
will still be backwards compatible if only `url` is specified.
|
||||
- The -test flag will now output two metric collections
|
||||
- **Breaking Change**: The CPU collection plugin has been refactored to fix some
|
||||
bugs and outdated dependency issues. At the same time, I also decided to fix
|
||||
a naming consistency issue, so cpu_percentageIdle will become cpu_usage_idle.
|
||||
Also, all CPU time measurements now have it indicated in their name, so cpu_idle will become cpu_time_idle. Additionally, cpu_time measurements are going to be dropped in the default config.
|
||||
- **Breaking Change**: The memory plugin has been refactored and some measurements
|
||||
have been renamed for consistency. Some measurements have also been removed from being outputted. They are still being collected by gopsutil, and could easily be
|
||||
re-added in a "verbose" mode if there is demand for it.
|
||||
|
||||
### Features
|
||||
- [#143](https://github.com/influxdb/telegraf/issues/143): InfluxDB clustering support
|
||||
- [#181](https://github.com/influxdb/telegraf/issues/181): Makefile GOBIN support. Thanks @Vye!
|
||||
- [#203](https://github.com/influxdb/telegraf/pull/200): AMQP output. Thanks @ekini!
|
||||
- [#182](https://github.com/influxdb/telegraf/pull/182): OpenTSDB output. Thanks @rplessl!
|
||||
- [#187](https://github.com/influxdb/telegraf/pull/187): Retry output sink connections on startup.
|
||||
- [#220](https://github.com/influxdb/telegraf/pull/220): Add port tag to apache plugin. Thanks @neezgee!
|
||||
|
||||
### Bugfixes
|
||||
- [#170](https://github.com/influxdb/telegraf/issues/170): Systemd support
|
||||
- [#175](https://github.com/influxdb/telegraf/issues/175): Set write precision before gathering metrics
|
||||
- [#178](https://github.com/influxdb/telegraf/issues/178): redis plugin, multiple server thread hang bug
|
||||
- Fix net plugin on darwin
|
||||
- [#84](https://github.com/influxdb/telegraf/issues/84): Fix docker plugin on CentOS. Thanks @neezgee!
|
||||
- [#189](https://github.com/influxdb/telegraf/pull/189): Fix mem_used_perc. Thanks @mced!
|
||||
- [#192](https://github.com/influxdb/telegraf/issues/192): Increase compatibility of postgresql plugin. Now supports versions 8.1+
|
||||
- [#203](https://github.com/influxdb/telegraf/issues/203): EL5 rpm support. Thanks @ekini!
|
||||
- [#206](https://github.com/influxdb/telegraf/issues/206): CPU steal/guest values wrong on linux.
|
||||
- [#212](https://github.com/influxdb/telegraf/issues/212): Add hashbang to postinstall script. Thanks @ekini!
|
||||
- [#212](https://github.com/influxdb/telegraf/issues/212): Fix makefile warning. Thanks @ekini!
|
||||
|
||||
## v0.1.8 [2015-09-04]
|
||||
|
||||
### Release Notes
|
||||
- Telegraf will now write data in UTC at second precision by default
|
||||
- Now using Go 1.5 to build telegraf
|
||||
|
||||
### Features
|
||||
- [#150](https://github.com/influxdb/telegraf/pull/150): Add Host Uptime metric to system plugin
|
||||
- [#158](https://github.com/influxdb/telegraf/pull/158): Apache Plugin. Thanks @KPACHbIuLLIAnO4
|
||||
- [#159](https://github.com/influxdb/telegraf/pull/159): Use second precision for InfluxDB writes
|
||||
- [#165](https://github.com/influxdb/telegraf/pull/165): Add additional metrics to mysql plugin. Thanks @nickscript0
|
||||
- [#162](https://github.com/influxdb/telegraf/pull/162): Write UTC by default, provide option
|
||||
- [#166](https://github.com/influxdb/telegraf/pull/166): Upload binaries to S3
|
||||
- [#169](https://github.com/influxdb/telegraf/pull/169): Ping plugin
|
||||
|
||||
### Bugfixes
|
||||
|
||||
## v0.1.7 [2015-08-28]
|
||||
|
||||
### Features
|
||||
- [#38](https://github.com/influxdb/telegraf/pull/38): Kafka output producer.
|
||||
- [#133](https://github.com/influxdb/telegraf/pull/133): Add plugin.Gather error logging. Thanks @nickscript0!
|
||||
- [#136](https://github.com/influxdb/telegraf/issues/136): Add a -usage flag for printing usage of a single plugin.
|
||||
- [#137](https://github.com/influxdb/telegraf/issues/137): Memcached: fix when a value contains a space
|
||||
- [#138](https://github.com/influxdb/telegraf/issues/138): MySQL server address tag.
|
||||
- [#142](https://github.com/influxdb/telegraf/pull/142): Add Description and SampleConfig funcs to output interface
|
||||
- Indent the toml config file for readability
|
||||
|
||||
### Bugfixes
|
||||
- [#128](https://github.com/influxdb/telegraf/issues/128): system_load measurement missing.
|
||||
- [#129](https://github.com/influxdb/telegraf/issues/129): Latest pkg url fix.
|
||||
- [#131](https://github.com/influxdb/telegraf/issues/131): Fix memory reporting on linux & darwin. Thanks @subhachandrachandra!
|
||||
- [#140](https://github.com/influxdb/telegraf/issues/140): Memory plugin prec->perc typo fix. Thanks @brunoqc!
|
||||
|
||||
## v0.1.6 [2015-08-20]
|
||||
|
||||
### Features
|
||||
|
||||
163
CONTRIBUTING.md
163
CONTRIBUTING.md
@@ -1,163 +0,0 @@
|
||||
## Sign the CLA
|
||||
|
||||
Before we can merge a pull request, you will need to sign the CLA,
|
||||
which can be found [on our website](http://influxdb.com/community/cla.html)
|
||||
|
||||
## Plugins
|
||||
|
||||
This section is for developers that want to create new collection plugins.
|
||||
Telegraf is entirely plugin driven. This interface allows for operators to
|
||||
pick and chose what is gathered as well as makes it easy for developers
|
||||
to create new ways of generating metrics.
|
||||
|
||||
Plugin authorship is kept as simple as possible to promote people to develop
|
||||
and submit new plugins.
|
||||
|
||||
### Plugin Guidelines
|
||||
|
||||
* A plugin must conform to the `plugins.Plugin` interface.
|
||||
* Telegraf promises to run each plugin's Gather function serially. This means
|
||||
developers don't have to worry about thread safety within these functions.
|
||||
* Each generated metric automatically has the name of the plugin that generated
|
||||
it prepended. This is to keep plugins honest.
|
||||
* Plugins should call `plugins.Add` in their `init` function to register themselves.
|
||||
See below for a quick example.
|
||||
* To be available within Telegraf itself, plugins must add themselves to the
|
||||
`github.com/influxdb/telegraf/plugins/all/all.go` file.
|
||||
* The `SampleConfig` function should return valid toml that describes how the
|
||||
plugin can be configured. This is include in `telegraf -sample-config`.
|
||||
* The `Description` function should say in one line what this plugin does.
|
||||
|
||||
### Plugin interface
|
||||
|
||||
```go
|
||||
type Plugin interface {
|
||||
SampleConfig() string
|
||||
Description() string
|
||||
Gather(Accumulator) error
|
||||
}
|
||||
|
||||
type Accumulator interface {
|
||||
Add(measurement string, value interface{}, tags map[string]string)
|
||||
AddFieldsWithTime(measurement string,
|
||||
values map[string]interface{},
|
||||
tags map[string]string,
|
||||
timestamp time.Time)
|
||||
}
|
||||
```
|
||||
|
||||
### Accumulator
|
||||
|
||||
The way that a plugin emits metrics is by interacting with the Accumulator.
|
||||
|
||||
The `Add` function takes 3 arguments:
|
||||
* **measurement**: A string description of the metric. For instance `bytes_read` or `faults`.
|
||||
* **value**: A value for the metric. This accepts 5 different types of value:
|
||||
* **int**: The most common type. All int types are accepted but favor using `int64`
|
||||
Useful for counters, etc.
|
||||
* **float**: Favor `float64`, useful for gauges, percentages, etc.
|
||||
* **bool**: `true` or `false`, useful to indicate the presence of a state. `light_on`, etc.
|
||||
* **string**: Typically used to indicate a message, or some kind of freeform information.
|
||||
* **time.Time**: Useful for indicating when a state last occurred, for instance `light_on_since`.
|
||||
* **tags**: This is a map of strings to strings to describe the where or who
|
||||
about the metric. For instance, the `net` plugin adds a tag named `"interface"`
|
||||
set to the name of the network interface, like `"eth0"`.
|
||||
|
||||
The `AddFieldsWithTime` allows multiple values for a point to be passed. The values
|
||||
used are the same type profile as **value** above. The **timestamp** argument
|
||||
allows a point to be registered as having occurred at an arbitrary time.
|
||||
|
||||
Let's say you've written a plugin that emits metrics about processes on the current host.
|
||||
|
||||
```go
|
||||
|
||||
type Process struct {
|
||||
CPUTime float64
|
||||
MemoryBytes int64
|
||||
PID int
|
||||
}
|
||||
|
||||
func Gather(acc plugins.Accumulator) error {
|
||||
for _, process := range system.Processes() {
|
||||
tags := map[string]string {
|
||||
"pid": fmt.Sprintf("%d", process.Pid),
|
||||
}
|
||||
|
||||
acc.Add("cpu", process.CPUTime, tags)
|
||||
acc.Add("memory", process.MemoryBytes, tags)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example
|
||||
|
||||
```go
|
||||
package simple
|
||||
|
||||
// simple.go
|
||||
|
||||
import "github.com/influxdb/telegraf/plugins"
|
||||
|
||||
type Simple struct {
|
||||
Ok bool
|
||||
}
|
||||
|
||||
func (s *Simple) Description() string {
|
||||
return "a demo plugin"
|
||||
}
|
||||
|
||||
func (s *Simple) SampleConfig() string {
|
||||
return "ok = true # indicate if everything is fine"
|
||||
}
|
||||
|
||||
func (s *Simple) Gather(acc plugins.Accumulator) error {
|
||||
if s.Ok {
|
||||
acc.Add("state", "pretty good", nil)
|
||||
} else {
|
||||
acc.Add("state", "not great", nil)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func init() {
|
||||
plugins.Add("simple", func() plugins.Plugin { return &Simple{} })
|
||||
}
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
TODO: this section will describe requirements for contributing an output
|
||||
|
||||
## Unit Tests
|
||||
|
||||
### Execute short tests
|
||||
|
||||
execute `make test-short`
|
||||
|
||||
### Execute long tests
|
||||
|
||||
As Telegraf collects metrics from several third-party services it becomes a
|
||||
difficult task to mock each service as some of them have complicated protocols
|
||||
which would take some time to replicate.
|
||||
|
||||
To overcome this situation we've decided to use docker containers to provide a
|
||||
fast and reproducible environment to test those services which require it.
|
||||
For other situations
|
||||
(i.e: https://github.com/influxdb/telegraf/blob/master/plugins/redis/redis_test.go )
|
||||
a simple mock will suffice.
|
||||
|
||||
To execute Telegraf tests follow these simple steps:
|
||||
|
||||
- Install docker compose following [these](https://docs.docker.com/compose/install/)
|
||||
instructions
|
||||
- execute `make test`
|
||||
|
||||
**OSX users**: you will need to install `boot2docker` or `docker-machine`.
|
||||
The Makefile will assume that you have a `docker-machine` box called `default` to
|
||||
get the IP address.
|
||||
|
||||
### Unit test troubleshooting
|
||||
|
||||
Try cleaning up your test environment by executing `make test-cleanup` and
|
||||
re-running
|
||||
77
Godeps/Godeps.json
generated
77
Godeps/Godeps.json
generated
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/telegraf",
|
||||
"GoVersion": "go1.5",
|
||||
"GoVersion": "go1.4.2",
|
||||
"Packages": [
|
||||
"./..."
|
||||
],
|
||||
@@ -28,11 +28,6 @@
|
||||
"ImportPath": "github.com/cenkalti/backoff",
|
||||
"Rev": "4dc77674aceaabba2c7e3da25d4c823edfb73f99"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/cloudfoundry/gosigar",
|
||||
"Comment": "scotty_09012012-27-g3ed7c74",
|
||||
"Rev": "3ed7c74352dae6dc00bdc8c74045375352e3ec05"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/dancannon/gorethink/encoding",
|
||||
"Comment": "v1.x.x-1-g786f12a",
|
||||
@@ -96,9 +91,34 @@
|
||||
"Rev": "d1e82c1ec3f15ee991f7cc7ffd5b67ff6f5bbaee"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/influxdb",
|
||||
"Comment": "v0.9.3",
|
||||
"Rev": "5d42b212fca8facfe9db0c83822f09b88be643ec"
|
||||
"ImportPath": "github.com/influxdb/influxdb/client",
|
||||
"Comment": "v0.9.3-rc1",
|
||||
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/influxdb/influxql",
|
||||
"Comment": "v0.9.3-rc1",
|
||||
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/influxdb/meta",
|
||||
"Comment": "v0.9.3-rc1",
|
||||
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/influxdb/snapshot",
|
||||
"Comment": "v0.9.3-rc1",
|
||||
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/influxdb/toml",
|
||||
"Comment": "v0.9.3-rc1",
|
||||
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/influxdb/influxdb/tsdb",
|
||||
"Comment": "v0.9.3-rc1",
|
||||
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/lib/pq",
|
||||
@@ -141,45 +161,6 @@
|
||||
"ImportPath": "github.com/samuel/go-zookeeper/zk",
|
||||
"Rev": "5bb5cfc093ad18a28148c578f8632cfdb4d802e4"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/common",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/cpu",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/disk",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/docker",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/load",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/mem",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/shirou/gopsutil/net",
|
||||
"Comment": "1.0.0-158-g0fd612e",
|
||||
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/streadway/amqp",
|
||||
"Rev": "f4879ba28fffbb576743b03622a9ff20461826b2"
|
||||
},
|
||||
{
|
||||
"ImportPath": "github.com/stretchr/objx",
|
||||
"Rev": "cbeaeb16a013161a98496fad62933b1d21786672"
|
||||
|
||||
1
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/.gitignore
generated
vendored
1
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/.gitignore
generated
vendored
@@ -1 +0,0 @@
|
||||
.vagrant
|
||||
8
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/.travis.yml
generated
vendored
8
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/.travis.yml
generated
vendored
@@ -1,8 +0,0 @@
|
||||
language: go
|
||||
|
||||
go:
|
||||
- 1.2
|
||||
|
||||
install:
|
||||
- 'go install github.com/onsi/ginkgo/ginkgo'
|
||||
script: 'ginkgo -r'
|
||||
201
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/LICENSE
generated
vendored
201
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/LICENSE
generated
vendored
@@ -1,201 +0,0 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
9
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/NOTICE
generated
vendored
9
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/NOTICE
generated
vendored
@@ -1,9 +0,0 @@
|
||||
Copyright (c) [2009-2011] VMware, Inc. All Rights Reserved.
|
||||
|
||||
This product is licensed to you under the Apache License, Version 2.0 (the "License").
|
||||
You may not use this product except in compliance with the License.
|
||||
|
||||
This product includes a number of subcomponents with
|
||||
separate copyright notices and license terms. Your use of these
|
||||
subcomponents is subject to the terms and conditions of the
|
||||
subcomponent's license, as noted in the LICENSE file.
|
||||
22
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/README.md
generated
vendored
22
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/README.md
generated
vendored
@@ -1,22 +0,0 @@
|
||||
# Go sigar
|
||||
|
||||
## Overview
|
||||
|
||||
Go sigar is a golang implementation of the
|
||||
[sigar API](https://github.com/hyperic/sigar). The Go version of
|
||||
sigar has a very similar interface, but is being written from scratch
|
||||
in pure go/cgo, rather than cgo bindings for libsigar.
|
||||
|
||||
## Test drive
|
||||
|
||||
$ go get github.com/cloudfoundry/gosigar
|
||||
$ cd $GOPATH/src/github.com/cloudfoundry/gosigar/examples
|
||||
$ go run uptime.go
|
||||
|
||||
## Supported platforms
|
||||
|
||||
Currently targeting modern flavors of darwin and linux.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
25
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/Vagrantfile
generated
vendored
25
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/Vagrantfile
generated
vendored
@@ -1,25 +0,0 @@
|
||||
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
|
||||
VAGRANTFILE_API_VERSION = "2"
|
||||
|
||||
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
|
||||
config.vm.box = "hashicorp/precise64"
|
||||
config.vm.provision "shell", inline: "mkdir -p /home/vagrant/go"
|
||||
config.vm.synced_folder ".", "/home/vagrant/go/src/github.com/cloudfoundry/gosigar"
|
||||
config.vm.provision "shell", inline: "chown -R vagrant:vagrant /home/vagrant/go"
|
||||
install_go = <<-BASH
|
||||
set -e
|
||||
|
||||
if [ ! -d "/usr/local/go" ]; then
|
||||
cd /tmp && wget https://storage.googleapis.com/golang/go1.3.3.linux-amd64.tar.gz
|
||||
cd /usr/local
|
||||
tar xvzf /tmp/go1.3.3.linux-amd64.tar.gz
|
||||
echo 'export GOPATH=/home/vagrant/go; export PATH=/usr/local/go/bin:$PATH:$GOPATH/bin' >> /home/vagrant/.bashrc
|
||||
fi
|
||||
export GOPATH=/home/vagrant/go
|
||||
export PATH=/usr/local/go/bin:$PATH:$GOPATH/bin
|
||||
/usr/local/go/bin/go get -u github.com/onsi/ginkgo/ginkgo
|
||||
/usr/local/go/bin/go get -u github.com/onsi/gomega;
|
||||
BASH
|
||||
config.vm.provision "shell", inline: 'apt-get install -y git-core'
|
||||
config.vm.provision "shell", inline: install_go
|
||||
end
|
||||
69
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/concrete_sigar.go
generated
vendored
69
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/concrete_sigar.go
generated
vendored
@@ -1,69 +0,0 @@
|
||||
package sigar
|
||||
|
||||
import (
|
||||
"time"
|
||||
)
|
||||
|
||||
type ConcreteSigar struct{}
|
||||
|
||||
func (c *ConcreteSigar) CollectCpuStats(collectionInterval time.Duration) (<-chan Cpu, chan<- struct{}) {
|
||||
// samplesCh is buffered to 1 value to immediately return first CPU sample
|
||||
samplesCh := make(chan Cpu, 1)
|
||||
|
||||
stopCh := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
var cpuUsage Cpu
|
||||
|
||||
// Immediately provide non-delta value.
|
||||
// samplesCh is buffered to 1 value, so it will not block.
|
||||
cpuUsage.Get()
|
||||
samplesCh <- cpuUsage
|
||||
|
||||
ticker := time.NewTicker(collectionInterval)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
previousCpuUsage := cpuUsage
|
||||
|
||||
cpuUsage.Get()
|
||||
|
||||
select {
|
||||
case samplesCh <- cpuUsage.Delta(previousCpuUsage):
|
||||
default:
|
||||
// Include default to avoid channel blocking
|
||||
}
|
||||
|
||||
case <-stopCh:
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return samplesCh, stopCh
|
||||
}
|
||||
|
||||
func (c *ConcreteSigar) GetLoadAverage() (LoadAverage, error) {
|
||||
l := LoadAverage{}
|
||||
err := l.Get()
|
||||
return l, err
|
||||
}
|
||||
|
||||
func (c *ConcreteSigar) GetMem() (Mem, error) {
|
||||
m := Mem{}
|
||||
err := m.Get()
|
||||
return m, err
|
||||
}
|
||||
|
||||
func (c *ConcreteSigar) GetSwap() (Swap, error) {
|
||||
s := Swap{}
|
||||
err := s.Get()
|
||||
return s, err
|
||||
}
|
||||
|
||||
func (c *ConcreteSigar) GetFileSystemUsage(path string) (FileSystemUsage, error) {
|
||||
f := FileSystemUsage{}
|
||||
err := f.Get(path)
|
||||
return f, err
|
||||
}
|
||||
85
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/concrete_sigar_test.go
generated
vendored
85
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/concrete_sigar_test.go
generated
vendored
@@ -1,85 +0,0 @@
|
||||
package sigar_test
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
sigar "github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
var _ = Describe("ConcreteSigar", func() {
|
||||
var concreteSigar *sigar.ConcreteSigar
|
||||
|
||||
BeforeEach(func() {
|
||||
concreteSigar = &sigar.ConcreteSigar{}
|
||||
})
|
||||
|
||||
Describe("CollectCpuStats", func() {
|
||||
It("immediately makes first CPU usage available even though it's not very accurate", func() {
|
||||
samplesCh, stop := concreteSigar.CollectCpuStats(500 * time.Millisecond)
|
||||
|
||||
firstValue := <-samplesCh
|
||||
Expect(firstValue.User).To(BeNumerically(">", 0))
|
||||
|
||||
stop <- struct{}{}
|
||||
})
|
||||
|
||||
It("makes CPU usage delta values available", func() {
|
||||
samplesCh, stop := concreteSigar.CollectCpuStats(500 * time.Millisecond)
|
||||
|
||||
firstValue := <-samplesCh
|
||||
|
||||
secondValue := <-samplesCh
|
||||
Expect(secondValue.User).To(BeNumerically("<", firstValue.User))
|
||||
|
||||
stop <- struct{}{}
|
||||
})
|
||||
|
||||
It("does not block", func() {
|
||||
_, stop := concreteSigar.CollectCpuStats(10 * time.Millisecond)
|
||||
|
||||
// Sleep long enough for samplesCh to fill at least 2 values
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
|
||||
stop <- struct{}{}
|
||||
|
||||
// If CollectCpuStats blocks it will never get here
|
||||
Expect(true).To(BeTrue())
|
||||
})
|
||||
})
|
||||
|
||||
It("GetLoadAverage", func() {
|
||||
avg, err := concreteSigar.GetLoadAverage()
|
||||
Expect(avg.One).ToNot(BeNil())
|
||||
Expect(avg.Five).ToNot(BeNil())
|
||||
Expect(avg.Fifteen).ToNot(BeNil())
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("GetMem", func() {
|
||||
mem, err := concreteSigar.GetMem()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(mem.Total).To(BeNumerically(">", 0))
|
||||
Expect(mem.Used + mem.Free).To(BeNumerically("<=", mem.Total))
|
||||
})
|
||||
|
||||
It("GetSwap", func() {
|
||||
swap, err := concreteSigar.GetSwap()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(swap.Used + swap.Free).To(BeNumerically("<=", swap.Total))
|
||||
})
|
||||
|
||||
It("GetSwap", func() {
|
||||
fsusage, err := concreteSigar.GetFileSystemUsage("/")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(fsusage.Total).ToNot(BeNil())
|
||||
|
||||
fsusage, err = concreteSigar.GetFileSystemUsage("T O T A L L Y B O G U S")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(fsusage.Total).To(Equal(uint64(0)))
|
||||
})
|
||||
})
|
||||
52
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/cputimes.go
generated
vendored
52
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/cputimes.go
generated
vendored
@@ -1,52 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cpus := sigar.CpuList{}
|
||||
cpus.Get()
|
||||
tcpu := getOverallCpu(cpus)
|
||||
|
||||
for i, cpu := range cpus.List {
|
||||
fmt.Printf("CPU%d Ticks: %d\n", i, cpu.Total())
|
||||
}
|
||||
|
||||
fmt.Printf("Total CPU Ticks: %d\n", tcpu.Total())
|
||||
fmt.Printf("Total CPU Time: %d\n", tcpu.Total()/128)
|
||||
fmt.Printf("User CPU Time: %d\n", tcpu.User/128)
|
||||
|
||||
time.Sleep(1 * time.Second)
|
||||
tcpu2 := sigar.Cpu{}
|
||||
tcpu2.Get()
|
||||
|
||||
dcpu := tcpu2.Delta(tcpu)
|
||||
tcpuDelta := tcpu2.Total() - tcpu.Total()
|
||||
iPercentage := 100.0 * float64(dcpu.Idle) / float64(tcpuDelta)
|
||||
fmt.Printf("Idle percentage: %f\n", iPercentage)
|
||||
bPercentage := 100.0 * float64(busy(tcpu2)-busy(tcpu)) / float64(tcpuDelta)
|
||||
fmt.Printf("Busy percentage: %f\n", bPercentage)
|
||||
}
|
||||
|
||||
func busy(c sigar.Cpu) uint64 {
|
||||
return c.Total() - c.Idle
|
||||
}
|
||||
|
||||
func getOverallCpu(cl sigar.CpuList) sigar.Cpu {
|
||||
var overallCpu sigar.Cpu
|
||||
for _, c := range cl.List {
|
||||
overallCpu.User += c.User
|
||||
overallCpu.Nice += c.Nice
|
||||
overallCpu.Sys += c.Sys
|
||||
overallCpu.Idle += c.Idle
|
||||
overallCpu.Wait += c.Wait
|
||||
overallCpu.Irq += c.Irq
|
||||
overallCpu.SoftIrq += c.SoftIrq
|
||||
overallCpu.Stolen += c.Stolen
|
||||
}
|
||||
return overallCpu
|
||||
}
|
||||
39
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/df.go
generated
vendored
39
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/df.go
generated
vendored
@@ -1,39 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/cloudfoundry/gosigar"
|
||||
"os"
|
||||
)
|
||||
|
||||
const output_format = "%-15s %4s %4s %5s %4s %-15s\n"
|
||||
|
||||
func formatSize(size uint64) string {
|
||||
return sigar.FormatSize(size * 1024)
|
||||
}
|
||||
|
||||
func main() {
|
||||
fslist := sigar.FileSystemList{}
|
||||
fslist.Get()
|
||||
|
||||
fmt.Fprintf(os.Stdout, output_format,
|
||||
"Filesystem", "Size", "Used", "Avail", "Use%", "Mounted on")
|
||||
|
||||
for _, fs := range fslist.List {
|
||||
dir_name := fs.DirName
|
||||
|
||||
usage := sigar.FileSystemUsage{}
|
||||
|
||||
usage.Get(dir_name)
|
||||
|
||||
fmt.Fprintf(os.Stdout, output_format,
|
||||
fs.DevName,
|
||||
formatSize(usage.Total),
|
||||
formatSize(usage.Used),
|
||||
formatSize(usage.Avail),
|
||||
sigar.FormatPercent(usage.UsePercent()),
|
||||
dir_name)
|
||||
}
|
||||
}
|
||||
33
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/free.go
generated
vendored
33
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/free.go
generated
vendored
@@ -1,33 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/cloudfoundry/gosigar"
|
||||
"os"
|
||||
)
|
||||
|
||||
func format(val uint64) uint64 {
|
||||
return val / 1024
|
||||
}
|
||||
|
||||
func main() {
|
||||
mem := sigar.Mem{}
|
||||
swap := sigar.Swap{}
|
||||
|
||||
mem.Get()
|
||||
swap.Get()
|
||||
|
||||
fmt.Fprintf(os.Stdout, "%18s %10s %10s\n",
|
||||
"total", "used", "free")
|
||||
|
||||
fmt.Fprintf(os.Stdout, "Mem: %10d %10d %10d\n",
|
||||
format(mem.Total), format(mem.Used), format(mem.Free))
|
||||
|
||||
fmt.Fprintf(os.Stdout, "-/+ buffers/cache: %10d %10d\n",
|
||||
format(mem.ActualUsed), format(mem.ActualFree))
|
||||
|
||||
fmt.Fprintf(os.Stdout, "Swap: %10d %10d %10d\n",
|
||||
format(swap.Total), format(swap.Used), format(swap.Free))
|
||||
}
|
||||
37
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/ps.go
generated
vendored
37
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/ps.go
generated
vendored
@@ -1,37 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
func main() {
|
||||
pids := sigar.ProcList{}
|
||||
pids.Get()
|
||||
|
||||
// ps -eo pid,ppid,stime,time,rss,state,comm
|
||||
fmt.Print(" PID PPID STIME TIME RSS S COMMAND\n")
|
||||
|
||||
for _, pid := range pids.List {
|
||||
state := sigar.ProcState{}
|
||||
mem := sigar.ProcMem{}
|
||||
time := sigar.ProcTime{}
|
||||
|
||||
if err := state.Get(pid); err != nil {
|
||||
continue
|
||||
}
|
||||
if err := mem.Get(pid); err != nil {
|
||||
continue
|
||||
}
|
||||
if err := time.Get(pid); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
fmt.Printf("%5d %5d %s %s %6d %c %s\n",
|
||||
pid, state.Ppid,
|
||||
time.FormatStartTime(), time.FormatTotal(),
|
||||
mem.Resident/1024, state.State, state.Name)
|
||||
}
|
||||
}
|
||||
27
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/uptime.go
generated
vendored
27
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/uptime.go
generated
vendored
@@ -1,27 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/cloudfoundry/gosigar"
|
||||
"os"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
concreteSigar := sigar.ConcreteSigar{}
|
||||
|
||||
uptime := sigar.Uptime{}
|
||||
uptime.Get()
|
||||
avg, err := concreteSigar.GetLoadAverage()
|
||||
if err != nil {
|
||||
fmt.Printf("Failed to get load average")
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stdout, " %s up %s load average: %.2f, %.2f, %.2f\n",
|
||||
time.Now().Format("15:04:05"),
|
||||
uptime.Format(),
|
||||
avg.One, avg.Five, avg.Fifteen)
|
||||
}
|
||||
72
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/fakes/fake_sigar.go
generated
vendored
72
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/fakes/fake_sigar.go
generated
vendored
@@ -1,72 +0,0 @@
|
||||
package fakes
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
sigar "github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
type FakeSigar struct {
|
||||
LoadAverage sigar.LoadAverage
|
||||
LoadAverageErr error
|
||||
|
||||
Mem sigar.Mem
|
||||
MemErr error
|
||||
|
||||
Swap sigar.Swap
|
||||
SwapErr error
|
||||
|
||||
FileSystemUsage sigar.FileSystemUsage
|
||||
FileSystemUsageErr error
|
||||
FileSystemUsagePath string
|
||||
|
||||
CollectCpuStatsCpuCh chan sigar.Cpu
|
||||
CollectCpuStatsStopCh chan struct{}
|
||||
}
|
||||
|
||||
func NewFakeSigar() *FakeSigar {
|
||||
return &FakeSigar{
|
||||
CollectCpuStatsCpuCh: make(chan sigar.Cpu, 1),
|
||||
CollectCpuStatsStopCh: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
func (f *FakeSigar) CollectCpuStats(collectionInterval time.Duration) (<-chan sigar.Cpu, chan<- struct{}) {
|
||||
samplesCh := make(chan sigar.Cpu, 1)
|
||||
stopCh := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case cpuStat := <-f.CollectCpuStatsCpuCh:
|
||||
select {
|
||||
case samplesCh <- cpuStat:
|
||||
default:
|
||||
// Include default to avoid channel blocking
|
||||
}
|
||||
|
||||
case <-f.CollectCpuStatsStopCh:
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return samplesCh, stopCh
|
||||
}
|
||||
|
||||
func (f *FakeSigar) GetLoadAverage() (sigar.LoadAverage, error) {
|
||||
return f.LoadAverage, f.LoadAverageErr
|
||||
}
|
||||
|
||||
func (f *FakeSigar) GetMem() (sigar.Mem, error) {
|
||||
return f.Mem, f.MemErr
|
||||
}
|
||||
|
||||
func (f *FakeSigar) GetSwap() (sigar.Swap, error) {
|
||||
return f.Swap, f.SwapErr
|
||||
}
|
||||
|
||||
func (f *FakeSigar) GetFileSystemUsage(path string) (sigar.FileSystemUsage, error) {
|
||||
f.FileSystemUsagePath = path
|
||||
return f.FileSystemUsage, f.FileSystemUsageErr
|
||||
}
|
||||
50
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/README.md
generated
vendored
50
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/README.md
generated
vendored
@@ -1,50 +0,0 @@
|
||||
# Process notifications for Go
|
||||
|
||||
## Overview
|
||||
|
||||
The psnotify package captures process events from the kernel via
|
||||
kqueue on Darwin/BSD and the netlink connector on Linux.
|
||||
|
||||
The psnotify API is similar to the
|
||||
[fsnotify](https://github.com/howeyc/fsnotify) package.
|
||||
|
||||
Example:
|
||||
```go
|
||||
watcher, err := psnotify.NewWatcher()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Process events
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case ev := <-watcher.Fork:
|
||||
log.Println("fork event:", ev)
|
||||
case ev := <-watcher.Exec:
|
||||
log.Println("exec event:", ev)
|
||||
case ev := <-watcher.Exit:
|
||||
log.Println("exit event:", ev)
|
||||
case err := <-watcher.Error:
|
||||
log.Println("error:", err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
err = watcher.Watch(os.Getpid(), psnotify.PROC_EVENT_ALL)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
/* ... do stuff ... */
|
||||
watcher.Close()
|
||||
```
|
||||
|
||||
## Supported platforms
|
||||
|
||||
Currently targeting modern flavors of Darwin and Linux.
|
||||
Should work on BSD, but untested.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
136
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify.go
generated
vendored
136
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify.go
generated
vendored
@@ -1,136 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package psnotify
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
type ProcEventFork struct {
|
||||
ParentPid int // Pid of the process that called fork()
|
||||
ChildPid int // Child process pid created by fork()
|
||||
}
|
||||
|
||||
type ProcEventExec struct {
|
||||
Pid int // Pid of the process that called exec()
|
||||
}
|
||||
|
||||
type ProcEventExit struct {
|
||||
Pid int // Pid of the process that called exit()
|
||||
}
|
||||
|
||||
type watch struct {
|
||||
flags uint32 // Saved value of Watch() flags param
|
||||
}
|
||||
|
||||
type eventListener interface {
|
||||
close() error // Watch.Close() closes the OS specific listener
|
||||
}
|
||||
|
||||
type Watcher struct {
|
||||
listener eventListener // OS specifics (kqueue or netlink)
|
||||
watches map[int]*watch // Map of watched process ids
|
||||
Error chan error // Errors are sent on this channel
|
||||
Fork chan *ProcEventFork // Fork events are sent on this channel
|
||||
Exec chan *ProcEventExec // Exec events are sent on this channel
|
||||
Exit chan *ProcEventExit // Exit events are sent on this channel
|
||||
done chan bool // Used to stop the readEvents() goroutine
|
||||
isClosed bool // Set to true when Close() is first called
|
||||
}
|
||||
|
||||
// Initialize event listener and channels
|
||||
func NewWatcher() (*Watcher, error) {
|
||||
listener, err := createListener()
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
w := &Watcher{
|
||||
listener: listener,
|
||||
watches: make(map[int]*watch),
|
||||
Fork: make(chan *ProcEventFork),
|
||||
Exec: make(chan *ProcEventExec),
|
||||
Exit: make(chan *ProcEventExit),
|
||||
Error: make(chan error),
|
||||
done: make(chan bool, 1),
|
||||
}
|
||||
|
||||
go w.readEvents()
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// Close event channels when done message is received
|
||||
func (w *Watcher) finish() {
|
||||
close(w.Fork)
|
||||
close(w.Exec)
|
||||
close(w.Exit)
|
||||
close(w.Error)
|
||||
}
|
||||
|
||||
// Closes the OS specific event listener,
|
||||
// removes all watches and closes all event channels.
|
||||
func (w *Watcher) Close() error {
|
||||
if w.isClosed {
|
||||
return nil
|
||||
}
|
||||
w.isClosed = true
|
||||
|
||||
for pid := range w.watches {
|
||||
w.RemoveWatch(pid)
|
||||
}
|
||||
|
||||
w.done <- true
|
||||
|
||||
w.listener.close()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Add pid to the watched process set.
|
||||
// The flags param is a bitmask of process events to capture,
|
||||
// must be one or more of: PROC_EVENT_FORK, PROC_EVENT_EXEC, PROC_EVENT_EXIT
|
||||
func (w *Watcher) Watch(pid int, flags uint32) error {
|
||||
if w.isClosed {
|
||||
return errors.New("psnotify watcher is closed")
|
||||
}
|
||||
|
||||
watchEntry, found := w.watches[pid]
|
||||
|
||||
if found {
|
||||
watchEntry.flags |= flags
|
||||
} else {
|
||||
if err := w.register(pid, flags); err != nil {
|
||||
return err
|
||||
}
|
||||
w.watches[pid] = &watch{flags: flags}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Remove pid from the watched process set.
|
||||
func (w *Watcher) RemoveWatch(pid int) error {
|
||||
_, ok := w.watches[pid]
|
||||
if !ok {
|
||||
msg := fmt.Sprintf("watch for pid=%d does not exist", pid)
|
||||
return errors.New(msg)
|
||||
}
|
||||
delete(w.watches, pid)
|
||||
return w.unregister(pid)
|
||||
}
|
||||
|
||||
// Internal helper to check if there is a message on the "done" channel.
|
||||
// The "done" message is sent by the Close() method; when received here,
|
||||
// the Watcher.finish method is called to close all channels and return
|
||||
// true - in which case the caller should break from the readEvents loop.
|
||||
func (w *Watcher) isDone() bool {
|
||||
var done bool
|
||||
select {
|
||||
case done = <-w.done:
|
||||
w.finish()
|
||||
default:
|
||||
}
|
||||
return done
|
||||
}
|
||||
93
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_bsd.go
generated
vendored
93
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_bsd.go
generated
vendored
@@ -1,93 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
// +build darwin freebsd netbsd openbsd
|
||||
|
||||
// Go interface to BSD kqueue process events.
|
||||
package psnotify
|
||||
|
||||
import (
|
||||
"syscall"
|
||||
)
|
||||
|
||||
const (
|
||||
// Flags (from <sys/event.h>)
|
||||
PROC_EVENT_FORK = syscall.NOTE_FORK // fork() events
|
||||
PROC_EVENT_EXEC = syscall.NOTE_EXEC // exec() events
|
||||
PROC_EVENT_EXIT = syscall.NOTE_EXIT // exit() events
|
||||
|
||||
// Watch for all process events
|
||||
PROC_EVENT_ALL = PROC_EVENT_FORK | PROC_EVENT_EXEC | PROC_EVENT_EXIT
|
||||
)
|
||||
|
||||
type kqueueListener struct {
|
||||
kq int // The syscall.Kqueue() file descriptor
|
||||
buf [1]syscall.Kevent_t // An event buffer for Add/Remove watch
|
||||
}
|
||||
|
||||
// Initialize bsd implementation of the eventListener interface
|
||||
func createListener() (eventListener, error) {
|
||||
listener := &kqueueListener{}
|
||||
kq, err := syscall.Kqueue()
|
||||
listener.kq = kq
|
||||
return listener, err
|
||||
}
|
||||
|
||||
// Initialize Kevent_t fields and propagate changelist for the given pid
|
||||
func (w *Watcher) kevent(pid int, fflags uint32, flags int) error {
|
||||
listener, _ := w.listener.(*kqueueListener)
|
||||
event := &listener.buf[0]
|
||||
|
||||
syscall.SetKevent(event, pid, syscall.EVFILT_PROC, flags)
|
||||
event.Fflags = fflags
|
||||
|
||||
_, err := syscall.Kevent(listener.kq, listener.buf[:], nil, nil)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// Delete filter for given pid from the queue
|
||||
func (w *Watcher) unregister(pid int) error {
|
||||
return w.kevent(pid, 0, syscall.EV_DELETE)
|
||||
}
|
||||
|
||||
// Add and enable filter for given pid in the queue
|
||||
func (w *Watcher) register(pid int, flags uint32) error {
|
||||
return w.kevent(pid, flags, syscall.EV_ADD|syscall.EV_ENABLE)
|
||||
}
|
||||
|
||||
// Poll the kqueue file descriptor and dispatch to the Event channels
|
||||
func (w *Watcher) readEvents() {
|
||||
listener, _ := w.listener.(*kqueueListener)
|
||||
events := make([]syscall.Kevent_t, 10)
|
||||
|
||||
for {
|
||||
if w.isDone() {
|
||||
return
|
||||
}
|
||||
|
||||
n, err := syscall.Kevent(listener.kq, nil, events, nil)
|
||||
if err != nil {
|
||||
w.Error <- err
|
||||
continue
|
||||
}
|
||||
|
||||
for _, ev := range events[:n] {
|
||||
pid := int(ev.Ident)
|
||||
|
||||
switch ev.Fflags {
|
||||
case syscall.NOTE_FORK:
|
||||
w.Fork <- &ProcEventFork{ParentPid: pid}
|
||||
case syscall.NOTE_EXEC:
|
||||
w.Exec <- &ProcEventExec{Pid: pid}
|
||||
case syscall.NOTE_EXIT:
|
||||
w.RemoveWatch(pid)
|
||||
w.Exit <- &ProcEventExit{Pid: pid}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Close our kqueue file descriptor; deletes any remaining filters
|
||||
func (listener *kqueueListener) close() error {
|
||||
return syscall.Close(listener.kq)
|
||||
}
|
||||
253
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_linux.go
generated
vendored
253
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_linux.go
generated
vendored
@@ -1,253 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
// Go interface to the Linux netlink process connector.
|
||||
// See Documentation/connector/connector.txt in the linux kernel source tree.
|
||||
package psnotify
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"os"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
const (
|
||||
// internal flags (from <linux/connector.h>)
|
||||
_CN_IDX_PROC = 0x1
|
||||
_CN_VAL_PROC = 0x1
|
||||
|
||||
// internal flags (from <linux/cn_proc.h>)
|
||||
_PROC_CN_MCAST_LISTEN = 1
|
||||
_PROC_CN_MCAST_IGNORE = 2
|
||||
|
||||
// Flags (from <linux/cn_proc.h>)
|
||||
PROC_EVENT_FORK = 0x00000001 // fork() events
|
||||
PROC_EVENT_EXEC = 0x00000002 // exec() events
|
||||
PROC_EVENT_EXIT = 0x80000000 // exit() events
|
||||
|
||||
// Watch for all process events
|
||||
PROC_EVENT_ALL = PROC_EVENT_FORK | PROC_EVENT_EXEC | PROC_EVENT_EXIT
|
||||
)
|
||||
|
||||
var (
|
||||
byteOrder = binary.LittleEndian
|
||||
)
|
||||
|
||||
// linux/connector.h: struct cb_id
|
||||
type cbId struct {
|
||||
Idx uint32
|
||||
Val uint32
|
||||
}
|
||||
|
||||
// linux/connector.h: struct cb_msg
|
||||
type cnMsg struct {
|
||||
Id cbId
|
||||
Seq uint32
|
||||
Ack uint32
|
||||
Len uint16
|
||||
Flags uint16
|
||||
}
|
||||
|
||||
// linux/cn_proc.h: struct proc_event.{what,cpu,timestamp_ns}
|
||||
type procEventHeader struct {
|
||||
What uint32
|
||||
Cpu uint32
|
||||
Timestamp uint64
|
||||
}
|
||||
|
||||
// linux/cn_proc.h: struct proc_event.fork
|
||||
type forkProcEvent struct {
|
||||
ParentPid uint32
|
||||
ParentTgid uint32
|
||||
ChildPid uint32
|
||||
ChildTgid uint32
|
||||
}
|
||||
|
||||
// linux/cn_proc.h: struct proc_event.exec
|
||||
type execProcEvent struct {
|
||||
ProcessPid uint32
|
||||
ProcessTgid uint32
|
||||
}
|
||||
|
||||
// linux/cn_proc.h: struct proc_event.exit
|
||||
type exitProcEvent struct {
|
||||
ProcessPid uint32
|
||||
ProcessTgid uint32
|
||||
ExitCode uint32
|
||||
ExitSignal uint32
|
||||
}
|
||||
|
||||
// standard netlink header + connector header
|
||||
type netlinkProcMessage struct {
|
||||
Header syscall.NlMsghdr
|
||||
Data cnMsg
|
||||
}
|
||||
|
||||
type netlinkListener struct {
|
||||
addr *syscall.SockaddrNetlink // Netlink socket address
|
||||
sock int // The syscall.Socket() file descriptor
|
||||
seq uint32 // struct cn_msg.seq
|
||||
}
|
||||
|
||||
// Initialize linux implementation of the eventListener interface
|
||||
func createListener() (eventListener, error) {
|
||||
listener := &netlinkListener{}
|
||||
err := listener.bind()
|
||||
return listener, err
|
||||
}
|
||||
|
||||
// noop on linux
|
||||
func (w *Watcher) unregister(pid int) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// noop on linux
|
||||
func (w *Watcher) register(pid int, flags uint32) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Read events from the netlink socket
|
||||
func (w *Watcher) readEvents() {
|
||||
buf := make([]byte, syscall.Getpagesize())
|
||||
|
||||
listener, _ := w.listener.(*netlinkListener)
|
||||
|
||||
for {
|
||||
if w.isDone() {
|
||||
return
|
||||
}
|
||||
|
||||
nr, _, err := syscall.Recvfrom(listener.sock, buf, 0)
|
||||
|
||||
if err != nil {
|
||||
w.Error <- err
|
||||
continue
|
||||
}
|
||||
if nr < syscall.NLMSG_HDRLEN {
|
||||
w.Error <- syscall.EINVAL
|
||||
continue
|
||||
}
|
||||
|
||||
msgs, _ := syscall.ParseNetlinkMessage(buf[:nr])
|
||||
|
||||
for _, m := range msgs {
|
||||
if m.Header.Type == syscall.NLMSG_DONE {
|
||||
w.handleEvent(m.Data)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Internal helper to check if pid && event is being watched
|
||||
func (w *Watcher) isWatching(pid int, event uint32) bool {
|
||||
if watch, ok := w.watches[pid]; ok {
|
||||
return (watch.flags & event) == event
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Dispatch events from the netlink socket to the Event channels.
|
||||
// Unlike bsd kqueue, netlink receives events for all pids,
|
||||
// so we apply filtering based on the watch table via isWatching()
|
||||
func (w *Watcher) handleEvent(data []byte) {
|
||||
buf := bytes.NewBuffer(data)
|
||||
msg := &cnMsg{}
|
||||
hdr := &procEventHeader{}
|
||||
|
||||
binary.Read(buf, byteOrder, msg)
|
||||
binary.Read(buf, byteOrder, hdr)
|
||||
|
||||
switch hdr.What {
|
||||
case PROC_EVENT_FORK:
|
||||
event := &forkProcEvent{}
|
||||
binary.Read(buf, byteOrder, event)
|
||||
ppid := int(event.ParentTgid)
|
||||
pid := int(event.ChildTgid)
|
||||
|
||||
if w.isWatching(ppid, PROC_EVENT_EXEC) {
|
||||
// follow forks
|
||||
watch, _ := w.watches[ppid]
|
||||
w.Watch(pid, watch.flags)
|
||||
}
|
||||
|
||||
if w.isWatching(ppid, PROC_EVENT_FORK) {
|
||||
w.Fork <- &ProcEventFork{ParentPid: ppid, ChildPid: pid}
|
||||
}
|
||||
case PROC_EVENT_EXEC:
|
||||
event := &execProcEvent{}
|
||||
binary.Read(buf, byteOrder, event)
|
||||
pid := int(event.ProcessTgid)
|
||||
|
||||
if w.isWatching(pid, PROC_EVENT_EXEC) {
|
||||
w.Exec <- &ProcEventExec{Pid: pid}
|
||||
}
|
||||
case PROC_EVENT_EXIT:
|
||||
event := &exitProcEvent{}
|
||||
binary.Read(buf, byteOrder, event)
|
||||
pid := int(event.ProcessTgid)
|
||||
|
||||
if w.isWatching(pid, PROC_EVENT_EXIT) {
|
||||
w.RemoveWatch(pid)
|
||||
w.Exit <- &ProcEventExit{Pid: pid}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Bind our netlink socket and
|
||||
// send a listen control message to the connector driver.
|
||||
func (listener *netlinkListener) bind() error {
|
||||
sock, err := syscall.Socket(
|
||||
syscall.AF_NETLINK,
|
||||
syscall.SOCK_DGRAM,
|
||||
syscall.NETLINK_CONNECTOR)
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
listener.sock = sock
|
||||
listener.addr = &syscall.SockaddrNetlink{
|
||||
Family: syscall.AF_NETLINK,
|
||||
Groups: _CN_IDX_PROC,
|
||||
}
|
||||
|
||||
err = syscall.Bind(listener.sock, listener.addr)
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return listener.send(_PROC_CN_MCAST_LISTEN)
|
||||
}
|
||||
|
||||
// Send an ignore control message to the connector driver
|
||||
// and close our netlink socket.
|
||||
func (listener *netlinkListener) close() error {
|
||||
err := listener.send(_PROC_CN_MCAST_IGNORE)
|
||||
syscall.Close(listener.sock)
|
||||
return err
|
||||
}
|
||||
|
||||
// Generic method for sending control messages to the connector
|
||||
// driver; where op is one of PROC_CN_MCAST_{LISTEN,IGNORE}
|
||||
func (listener *netlinkListener) send(op uint32) error {
|
||||
listener.seq++
|
||||
pr := &netlinkProcMessage{}
|
||||
plen := binary.Size(pr.Data) + binary.Size(op)
|
||||
pr.Header.Len = syscall.NLMSG_HDRLEN + uint32(plen)
|
||||
pr.Header.Type = uint16(syscall.NLMSG_DONE)
|
||||
pr.Header.Flags = 0
|
||||
pr.Header.Seq = listener.seq
|
||||
pr.Header.Pid = uint32(os.Getpid())
|
||||
|
||||
pr.Data.Id.Idx = _CN_IDX_PROC
|
||||
pr.Data.Id.Val = _CN_VAL_PROC
|
||||
|
||||
pr.Data.Len = uint16(binary.Size(op))
|
||||
|
||||
buf := bytes.NewBuffer(make([]byte, 0, pr.Header.Len))
|
||||
binary.Write(buf, byteOrder, pr)
|
||||
binary.Write(buf, byteOrder, op)
|
||||
|
||||
return syscall.Sendto(listener.sock, buf.Bytes(), 0, listener.addr)
|
||||
}
|
||||
283
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_test.go
generated
vendored
283
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_test.go
generated
vendored
@@ -1,283 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package psnotify
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type anyEvent struct {
|
||||
exits []int
|
||||
forks []int
|
||||
execs []int
|
||||
errors []error
|
||||
done chan bool
|
||||
}
|
||||
|
||||
type testWatcher struct {
|
||||
t *testing.T
|
||||
watcher *Watcher
|
||||
events *anyEvent
|
||||
}
|
||||
|
||||
// General purpose Watcher wrapper for all tests
|
||||
func newTestWatcher(t *testing.T) *testWatcher {
|
||||
watcher, err := NewWatcher()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
events := &anyEvent{
|
||||
done: make(chan bool, 1),
|
||||
}
|
||||
|
||||
tw := &testWatcher{
|
||||
t: t,
|
||||
watcher: watcher,
|
||||
events: events,
|
||||
}
|
||||
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case <-events.done:
|
||||
return
|
||||
case ev := <-watcher.Fork:
|
||||
events.forks = append(events.forks, ev.ParentPid)
|
||||
case ev := <-watcher.Exec:
|
||||
events.execs = append(events.execs, ev.Pid)
|
||||
case ev := <-watcher.Exit:
|
||||
events.exits = append(events.exits, ev.Pid)
|
||||
case err := <-watcher.Error:
|
||||
events.errors = append(events.errors, err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return tw
|
||||
}
|
||||
|
||||
func (tw *testWatcher) close() {
|
||||
pause := 100 * time.Millisecond
|
||||
time.Sleep(pause)
|
||||
|
||||
tw.events.done <- true
|
||||
|
||||
tw.watcher.Close()
|
||||
|
||||
time.Sleep(pause)
|
||||
}
|
||||
|
||||
func skipTest(t *testing.T) bool {
|
||||
if runtime.GOOS == "linux" && os.Getuid() != 0 {
|
||||
fmt.Println("SKIP: test must be run as root on linux")
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func startSleepCommand(t *testing.T) *exec.Cmd {
|
||||
cmd := exec.Command("sh", "-c", "sleep 100")
|
||||
if err := cmd.Start(); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
|
||||
func runCommand(t *testing.T, name string) *exec.Cmd {
|
||||
cmd := exec.Command(name)
|
||||
if err := cmd.Run(); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
|
||||
func expectEvents(t *testing.T, num int, name string, pids []int) bool {
|
||||
if len(pids) != num {
|
||||
t.Errorf("Expected %d %s events, got=%v", num, name, pids)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func expectEventPid(t *testing.T, name string, expect int, pid int) bool {
|
||||
if expect != pid {
|
||||
t.Errorf("Expected %s pid=%d, received=%d", name, expect, pid)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func TestWatchFork(t *testing.T) {
|
||||
if skipTest(t) {
|
||||
return
|
||||
}
|
||||
|
||||
pid := os.Getpid()
|
||||
|
||||
tw := newTestWatcher(t)
|
||||
|
||||
// no watches added yet, so this fork event will no be captured
|
||||
runCommand(t, "date")
|
||||
|
||||
// watch fork events for this process
|
||||
if err := tw.watcher.Watch(pid, PROC_EVENT_FORK); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
// this fork event will be captured,
|
||||
// the exec and exit events will not be captured
|
||||
runCommand(t, "cal")
|
||||
|
||||
tw.close()
|
||||
|
||||
if expectEvents(t, 1, "forks", tw.events.forks) {
|
||||
expectEventPid(t, "fork", pid, tw.events.forks[0])
|
||||
}
|
||||
|
||||
expectEvents(t, 0, "execs", tw.events.execs)
|
||||
expectEvents(t, 0, "exits", tw.events.exits)
|
||||
}
|
||||
|
||||
func TestWatchExit(t *testing.T) {
|
||||
if skipTest(t) {
|
||||
return
|
||||
}
|
||||
|
||||
tw := newTestWatcher(t)
|
||||
|
||||
cmd := startSleepCommand(t)
|
||||
|
||||
childPid := cmd.Process.Pid
|
||||
|
||||
// watch for exit event of our child process
|
||||
if err := tw.watcher.Watch(childPid, PROC_EVENT_EXIT); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
// kill our child process, triggers exit event
|
||||
syscall.Kill(childPid, syscall.SIGTERM)
|
||||
|
||||
cmd.Wait()
|
||||
|
||||
tw.close()
|
||||
|
||||
expectEvents(t, 0, "forks", tw.events.forks)
|
||||
|
||||
expectEvents(t, 0, "execs", tw.events.execs)
|
||||
|
||||
if expectEvents(t, 1, "exits", tw.events.exits) {
|
||||
expectEventPid(t, "exit", childPid, tw.events.exits[0])
|
||||
}
|
||||
}
|
||||
|
||||
// combined version of TestWatchFork() and TestWatchExit()
|
||||
func TestWatchForkAndExit(t *testing.T) {
|
||||
if skipTest(t) {
|
||||
return
|
||||
}
|
||||
|
||||
pid := os.Getpid()
|
||||
|
||||
tw := newTestWatcher(t)
|
||||
|
||||
if err := tw.watcher.Watch(pid, PROC_EVENT_FORK); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
cmd := startSleepCommand(t)
|
||||
|
||||
childPid := cmd.Process.Pid
|
||||
|
||||
if err := tw.watcher.Watch(childPid, PROC_EVENT_EXIT); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
syscall.Kill(childPid, syscall.SIGTERM)
|
||||
|
||||
cmd.Wait()
|
||||
|
||||
tw.close()
|
||||
|
||||
if expectEvents(t, 1, "forks", tw.events.forks) {
|
||||
expectEventPid(t, "fork", pid, tw.events.forks[0])
|
||||
}
|
||||
|
||||
expectEvents(t, 0, "execs", tw.events.execs)
|
||||
|
||||
if expectEvents(t, 1, "exits", tw.events.exits) {
|
||||
expectEventPid(t, "exit", childPid, tw.events.exits[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatchFollowFork(t *testing.T) {
|
||||
if skipTest(t) {
|
||||
return
|
||||
}
|
||||
|
||||
// Darwin is not able to follow forks, as the kqueue fork event
|
||||
// does not provide the child pid.
|
||||
if runtime.GOOS != "linux" {
|
||||
fmt.Println("SKIP: test follow forks is linux only")
|
||||
return
|
||||
}
|
||||
|
||||
pid := os.Getpid()
|
||||
|
||||
tw := newTestWatcher(t)
|
||||
|
||||
// watch for all process events related to this process
|
||||
if err := tw.watcher.Watch(pid, PROC_EVENT_ALL); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
commands := []string{"date", "cal"}
|
||||
childPids := make([]int, len(commands))
|
||||
|
||||
// triggers fork/exec/exit events for each command
|
||||
for i, name := range commands {
|
||||
cmd := runCommand(t, name)
|
||||
childPids[i] = cmd.Process.Pid
|
||||
}
|
||||
|
||||
// remove watch for this process
|
||||
tw.watcher.RemoveWatch(pid)
|
||||
|
||||
// run commands again to make sure we don't receive any unwanted events
|
||||
for _, name := range commands {
|
||||
runCommand(t, name)
|
||||
}
|
||||
|
||||
tw.close()
|
||||
|
||||
// run commands again to make sure nothing panics after
|
||||
// closing the watcher
|
||||
for _, name := range commands {
|
||||
runCommand(t, name)
|
||||
}
|
||||
|
||||
num := len(commands)
|
||||
if expectEvents(t, num, "forks", tw.events.forks) {
|
||||
for _, epid := range tw.events.forks {
|
||||
expectEventPid(t, "fork", pid, epid)
|
||||
}
|
||||
}
|
||||
|
||||
if expectEvents(t, num, "execs", tw.events.execs) {
|
||||
for i, epid := range tw.events.execs {
|
||||
expectEventPid(t, "exec", childPids[i], epid)
|
||||
}
|
||||
}
|
||||
|
||||
if expectEvents(t, num, "exits", tw.events.exits) {
|
||||
for i, epid := range tw.events.exits {
|
||||
expectEventPid(t, "exit", childPids[i], epid)
|
||||
}
|
||||
}
|
||||
}
|
||||
467
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_darwin.go
generated
vendored
467
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_darwin.go
generated
vendored
@@ -1,467 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package sigar
|
||||
|
||||
/*
|
||||
#include <stdlib.h>
|
||||
#include <sys/sysctl.h>
|
||||
#include <sys/mount.h>
|
||||
#include <mach/mach_init.h>
|
||||
#include <mach/mach_host.h>
|
||||
#include <mach/host_info.h>
|
||||
#include <libproc.h>
|
||||
#include <mach/processor_info.h>
|
||||
#include <mach/vm_map.h>
|
||||
*/
|
||||
import "C"
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"io"
|
||||
"syscall"
|
||||
"time"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
func (self *LoadAverage) Get() error {
|
||||
avg := []C.double{0, 0, 0}
|
||||
|
||||
C.getloadavg(&avg[0], C.int(len(avg)))
|
||||
|
||||
self.One = float64(avg[0])
|
||||
self.Five = float64(avg[1])
|
||||
self.Fifteen = float64(avg[2])
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Uptime) Get() error {
|
||||
tv := syscall.Timeval32{}
|
||||
|
||||
if err := sysctlbyname("kern.boottime", &tv); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Length = time.Since(time.Unix(int64(tv.Sec), int64(tv.Usec)*1000)).Seconds()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Mem) Get() error {
|
||||
var vmstat C.vm_statistics_data_t
|
||||
|
||||
if err := sysctlbyname("hw.memsize", &self.Total); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := vm_info(&vmstat); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
kern := uint64(vmstat.inactive_count) << 12
|
||||
self.Free = uint64(vmstat.free_count) << 12
|
||||
|
||||
self.Used = self.Total - self.Free
|
||||
self.ActualFree = self.Free + kern
|
||||
self.ActualUsed = self.Used - kern
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
type xsw_usage struct {
|
||||
Total, Avail, Used uint64
|
||||
}
|
||||
|
||||
func (self *Swap) Get() error {
|
||||
sw_usage := xsw_usage{}
|
||||
|
||||
if err := sysctlbyname("vm.swapusage", &sw_usage); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Total = sw_usage.Total
|
||||
self.Used = sw_usage.Used
|
||||
self.Free = sw_usage.Avail
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Cpu) Get() error {
|
||||
var count C.mach_msg_type_number_t = C.HOST_CPU_LOAD_INFO_COUNT
|
||||
var cpuload C.host_cpu_load_info_data_t
|
||||
|
||||
status := C.host_statistics(C.host_t(C.mach_host_self()),
|
||||
C.HOST_CPU_LOAD_INFO,
|
||||
C.host_info_t(unsafe.Pointer(&cpuload)),
|
||||
&count)
|
||||
|
||||
if status != C.KERN_SUCCESS {
|
||||
return fmt.Errorf("host_statistics error=%d", status)
|
||||
}
|
||||
|
||||
self.User = uint64(cpuload.cpu_ticks[C.CPU_STATE_USER])
|
||||
self.Sys = uint64(cpuload.cpu_ticks[C.CPU_STATE_SYSTEM])
|
||||
self.Idle = uint64(cpuload.cpu_ticks[C.CPU_STATE_IDLE])
|
||||
self.Nice = uint64(cpuload.cpu_ticks[C.CPU_STATE_NICE])
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *CpuList) Get() error {
|
||||
var count C.mach_msg_type_number_t
|
||||
var cpuload *C.processor_cpu_load_info_data_t
|
||||
var ncpu C.natural_t
|
||||
|
||||
status := C.host_processor_info(C.host_t(C.mach_host_self()),
|
||||
C.PROCESSOR_CPU_LOAD_INFO,
|
||||
&ncpu,
|
||||
(*C.processor_info_array_t)(unsafe.Pointer(&cpuload)),
|
||||
&count)
|
||||
|
||||
if status != C.KERN_SUCCESS {
|
||||
return fmt.Errorf("host_processor_info error=%d", status)
|
||||
}
|
||||
|
||||
// jump through some cgo casting hoops and ensure we properly free
|
||||
// the memory that cpuload points to
|
||||
target := C.vm_map_t(C.mach_task_self_)
|
||||
address := C.vm_address_t(uintptr(unsafe.Pointer(cpuload)))
|
||||
defer C.vm_deallocate(target, address, C.vm_size_t(ncpu))
|
||||
|
||||
// the body of struct processor_cpu_load_info
|
||||
// aka processor_cpu_load_info_data_t
|
||||
var cpu_ticks [C.CPU_STATE_MAX]uint32
|
||||
|
||||
// copy the cpuload array to a []byte buffer
|
||||
// where we can binary.Read the data
|
||||
size := int(ncpu) * binary.Size(cpu_ticks)
|
||||
buf := C.GoBytes(unsafe.Pointer(cpuload), C.int(size))
|
||||
|
||||
bbuf := bytes.NewBuffer(buf)
|
||||
|
||||
self.List = make([]Cpu, 0, ncpu)
|
||||
|
||||
for i := 0; i < int(ncpu); i++ {
|
||||
cpu := Cpu{}
|
||||
|
||||
err := binary.Read(bbuf, binary.LittleEndian, &cpu_ticks)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
cpu.User = uint64(cpu_ticks[C.CPU_STATE_USER])
|
||||
cpu.Sys = uint64(cpu_ticks[C.CPU_STATE_SYSTEM])
|
||||
cpu.Idle = uint64(cpu_ticks[C.CPU_STATE_IDLE])
|
||||
cpu.Nice = uint64(cpu_ticks[C.CPU_STATE_NICE])
|
||||
|
||||
self.List = append(self.List, cpu)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *FileSystemList) Get() error {
|
||||
num, err := getfsstat(nil, C.MNT_NOWAIT)
|
||||
if num < 0 {
|
||||
return err
|
||||
}
|
||||
|
||||
buf := make([]syscall.Statfs_t, num)
|
||||
|
||||
num, err = getfsstat(buf, C.MNT_NOWAIT)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fslist := make([]FileSystem, 0, num)
|
||||
|
||||
for i := 0; i < num; i++ {
|
||||
fs := FileSystem{}
|
||||
|
||||
fs.DirName = bytePtrToString(&buf[i].Mntonname[0])
|
||||
fs.DevName = bytePtrToString(&buf[i].Mntfromname[0])
|
||||
fs.SysTypeName = bytePtrToString(&buf[i].Fstypename[0])
|
||||
|
||||
fslist = append(fslist, fs)
|
||||
}
|
||||
|
||||
self.List = fslist
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
func (self *ProcList) Get() error {
|
||||
n := C.proc_listpids(C.PROC_ALL_PIDS, 0, nil, 0)
|
||||
if n <= 0 {
|
||||
return syscall.EINVAL
|
||||
}
|
||||
buf := make([]byte, n)
|
||||
n = C.proc_listpids(C.PROC_ALL_PIDS, 0, unsafe.Pointer(&buf[0]), n)
|
||||
if n <= 0 {
|
||||
return syscall.ENOMEM
|
||||
}
|
||||
|
||||
var pid int32
|
||||
num := int(n) / binary.Size(pid)
|
||||
list := make([]int, 0, num)
|
||||
bbuf := bytes.NewBuffer(buf)
|
||||
|
||||
for i := 0; i < num; i++ {
|
||||
if err := binary.Read(bbuf, binary.LittleEndian, &pid); err != nil {
|
||||
return err
|
||||
}
|
||||
if pid == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
list = append(list, int(pid))
|
||||
}
|
||||
|
||||
self.List = list
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcState) Get(pid int) error {
|
||||
info := C.struct_proc_taskallinfo{}
|
||||
|
||||
if err := task_info(pid, &info); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Name = C.GoString(&info.pbsd.pbi_comm[0])
|
||||
|
||||
switch info.pbsd.pbi_status {
|
||||
case C.SIDL:
|
||||
self.State = RunStateIdle
|
||||
case C.SRUN:
|
||||
self.State = RunStateRun
|
||||
case C.SSLEEP:
|
||||
self.State = RunStateSleep
|
||||
case C.SSTOP:
|
||||
self.State = RunStateStop
|
||||
case C.SZOMB:
|
||||
self.State = RunStateZombie
|
||||
default:
|
||||
self.State = RunStateUnknown
|
||||
}
|
||||
|
||||
self.Ppid = int(info.pbsd.pbi_ppid)
|
||||
|
||||
self.Tty = int(info.pbsd.e_tdev)
|
||||
|
||||
self.Priority = int(info.ptinfo.pti_priority)
|
||||
|
||||
self.Nice = int(info.pbsd.pbi_nice)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcMem) Get(pid int) error {
|
||||
info := C.struct_proc_taskallinfo{}
|
||||
|
||||
if err := task_info(pid, &info); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Size = uint64(info.ptinfo.pti_virtual_size)
|
||||
self.Resident = uint64(info.ptinfo.pti_resident_size)
|
||||
self.PageFaults = uint64(info.ptinfo.pti_faults)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcTime) Get(pid int) error {
|
||||
info := C.struct_proc_taskallinfo{}
|
||||
|
||||
if err := task_info(pid, &info); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.User =
|
||||
uint64(info.ptinfo.pti_total_user) / uint64(time.Millisecond)
|
||||
|
||||
self.Sys =
|
||||
uint64(info.ptinfo.pti_total_system) / uint64(time.Millisecond)
|
||||
|
||||
self.Total = self.User + self.Sys
|
||||
|
||||
self.StartTime = (uint64(info.pbsd.pbi_start_tvsec) * 1000) +
|
||||
(uint64(info.pbsd.pbi_start_tvusec) / 1000)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcArgs) Get(pid int) error {
|
||||
var args []string
|
||||
|
||||
argv := func(arg string) {
|
||||
args = append(args, arg)
|
||||
}
|
||||
|
||||
err := kern_procargs(pid, nil, argv, nil)
|
||||
|
||||
self.List = args
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
func (self *ProcExe) Get(pid int) error {
|
||||
exe := func(arg string) {
|
||||
self.Name = arg
|
||||
}
|
||||
|
||||
return kern_procargs(pid, exe, nil, nil)
|
||||
}
|
||||
|
||||
// wrapper around sysctl KERN_PROCARGS2
|
||||
// callbacks params are optional,
|
||||
// up to the caller as to which pieces of data they want
|
||||
func kern_procargs(pid int,
|
||||
exe func(string),
|
||||
argv func(string),
|
||||
env func(string, string)) error {
|
||||
|
||||
mib := []C.int{C.CTL_KERN, C.KERN_PROCARGS2, C.int(pid)}
|
||||
argmax := uintptr(C.ARG_MAX)
|
||||
buf := make([]byte, argmax)
|
||||
err := sysctl(mib, &buf[0], &argmax, nil, 0)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
bbuf := bytes.NewBuffer(buf)
|
||||
bbuf.Truncate(int(argmax))
|
||||
|
||||
var argc int32
|
||||
binary.Read(bbuf, binary.LittleEndian, &argc)
|
||||
|
||||
path, err := bbuf.ReadBytes(0)
|
||||
if exe != nil {
|
||||
exe(string(chop(path)))
|
||||
}
|
||||
|
||||
// skip trailing \0's
|
||||
for {
|
||||
c, _ := bbuf.ReadByte()
|
||||
if c != 0 {
|
||||
bbuf.UnreadByte()
|
||||
break // start of argv[0]
|
||||
}
|
||||
}
|
||||
|
||||
for i := 0; i < int(argc); i++ {
|
||||
arg, err := bbuf.ReadBytes(0)
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if argv != nil {
|
||||
argv(string(chop(arg)))
|
||||
}
|
||||
}
|
||||
|
||||
if env == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
delim := []byte{61} // "="
|
||||
|
||||
for {
|
||||
line, err := bbuf.ReadBytes(0)
|
||||
if err == io.EOF || line[0] == 0 {
|
||||
break
|
||||
}
|
||||
pair := bytes.SplitN(chop(line), delim, 2)
|
||||
env(string(pair[0]), string(pair[1]))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// XXX copied from zsyscall_darwin_amd64.go
|
||||
func sysctl(mib []C.int, old *byte, oldlen *uintptr,
|
||||
new *byte, newlen uintptr) (err error) {
|
||||
var p0 unsafe.Pointer
|
||||
p0 = unsafe.Pointer(&mib[0])
|
||||
_, _, e1 := syscall.Syscall6(syscall.SYS___SYSCTL, uintptr(p0),
|
||||
uintptr(len(mib)),
|
||||
uintptr(unsafe.Pointer(old)), uintptr(unsafe.Pointer(oldlen)),
|
||||
uintptr(unsafe.Pointer(new)), uintptr(newlen))
|
||||
if e1 != 0 {
|
||||
err = e1
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func vm_info(vmstat *C.vm_statistics_data_t) error {
|
||||
var count C.mach_msg_type_number_t = C.HOST_VM_INFO_COUNT
|
||||
|
||||
status := C.host_statistics(
|
||||
C.host_t(C.mach_host_self()),
|
||||
C.HOST_VM_INFO,
|
||||
C.host_info_t(unsafe.Pointer(vmstat)),
|
||||
&count)
|
||||
|
||||
if status != C.KERN_SUCCESS {
|
||||
return fmt.Errorf("host_statistics=%d", status)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// generic Sysctl buffer unmarshalling
|
||||
func sysctlbyname(name string, data interface{}) (err error) {
|
||||
val, err := syscall.Sysctl(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
buf := []byte(val)
|
||||
|
||||
switch v := data.(type) {
|
||||
case *uint64:
|
||||
*v = *(*uint64)(unsafe.Pointer(&buf[0]))
|
||||
return
|
||||
}
|
||||
|
||||
bbuf := bytes.NewBuffer([]byte(val))
|
||||
return binary.Read(bbuf, binary.LittleEndian, data)
|
||||
}
|
||||
|
||||
// syscall.Getfsstat() wrapper is broken, roll our own to workaround.
|
||||
func getfsstat(buf []syscall.Statfs_t, flags int) (n int, err error) {
|
||||
var ptr uintptr
|
||||
var size uintptr
|
||||
|
||||
if len(buf) > 0 {
|
||||
ptr = uintptr(unsafe.Pointer(&buf[0]))
|
||||
size = unsafe.Sizeof(buf[0]) * uintptr(len(buf))
|
||||
} else {
|
||||
ptr = uintptr(0)
|
||||
size = uintptr(0)
|
||||
}
|
||||
|
||||
trap := uintptr(syscall.SYS_GETFSSTAT64)
|
||||
ret, _, errno := syscall.Syscall(trap, ptr, size, uintptr(flags))
|
||||
|
||||
n = int(ret)
|
||||
if errno != 0 {
|
||||
err = errno
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
func task_info(pid int, info *C.struct_proc_taskallinfo) error {
|
||||
size := C.int(unsafe.Sizeof(*info))
|
||||
ptr := unsafe.Pointer(info)
|
||||
|
||||
n := C.proc_pidinfo(C.int(pid), C.PROC_PIDTASKALLINFO, 0, ptr, size)
|
||||
if n != size {
|
||||
return syscall.ENOMEM
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
126
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_format.go
generated
vendored
126
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_format.go
generated
vendored
@@ -1,126 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package sigar
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Go version of apr_strfsize
|
||||
func FormatSize(size uint64) string {
|
||||
ord := []string{"K", "M", "G", "T", "P", "E"}
|
||||
o := 0
|
||||
buf := new(bytes.Buffer)
|
||||
w := bufio.NewWriter(buf)
|
||||
|
||||
if size < 973 {
|
||||
fmt.Fprintf(w, "%3d ", size)
|
||||
w.Flush()
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
for {
|
||||
remain := size & 1023
|
||||
size >>= 10
|
||||
|
||||
if size >= 973 {
|
||||
o++
|
||||
continue
|
||||
}
|
||||
|
||||
if size < 9 || (size == 9 && remain < 973) {
|
||||
remain = ((remain * 5) + 256) / 512
|
||||
if remain >= 10 {
|
||||
size++
|
||||
remain = 0
|
||||
}
|
||||
|
||||
fmt.Fprintf(w, "%d.%d%s", size, remain, ord[o])
|
||||
break
|
||||
}
|
||||
|
||||
if remain >= 512 {
|
||||
size++
|
||||
}
|
||||
|
||||
fmt.Fprintf(w, "%3d%s", size, ord[o])
|
||||
break
|
||||
}
|
||||
|
||||
w.Flush()
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
func FormatPercent(percent float64) string {
|
||||
return strconv.FormatFloat(percent, 'f', -1, 64) + "%"
|
||||
}
|
||||
|
||||
func (self *FileSystemUsage) UsePercent() float64 {
|
||||
b_used := (self.Total - self.Free) / 1024
|
||||
b_avail := self.Avail / 1024
|
||||
utotal := b_used + b_avail
|
||||
used := b_used
|
||||
|
||||
if utotal != 0 {
|
||||
u100 := used * 100
|
||||
pct := u100 / utotal
|
||||
if u100%utotal != 0 {
|
||||
pct += 1
|
||||
}
|
||||
return (float64(pct) / float64(100)) * 100.0
|
||||
}
|
||||
|
||||
return 0.0
|
||||
}
|
||||
|
||||
func (self *Uptime) Format() string {
|
||||
buf := new(bytes.Buffer)
|
||||
w := bufio.NewWriter(buf)
|
||||
uptime := uint64(self.Length)
|
||||
|
||||
days := uptime / (60 * 60 * 24)
|
||||
|
||||
if days != 0 {
|
||||
s := ""
|
||||
if days > 1 {
|
||||
s = "s"
|
||||
}
|
||||
fmt.Fprintf(w, "%d day%s, ", days, s)
|
||||
}
|
||||
|
||||
minutes := uptime / 60
|
||||
hours := minutes / 60
|
||||
hours %= 24
|
||||
minutes %= 60
|
||||
|
||||
fmt.Fprintf(w, "%2d:%02d", hours, minutes)
|
||||
|
||||
w.Flush()
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
func (self *ProcTime) FormatStartTime() string {
|
||||
if self.StartTime == 0 {
|
||||
return "00:00"
|
||||
}
|
||||
start := time.Unix(int64(self.StartTime)/1000, 0)
|
||||
format := "Jan02"
|
||||
if time.Since(start).Seconds() < (60 * 60 * 24) {
|
||||
format = "15:04"
|
||||
}
|
||||
return start.Format(format)
|
||||
}
|
||||
|
||||
func (self *ProcTime) FormatTotal() string {
|
||||
t := self.Total / 1000
|
||||
ss := t % 60
|
||||
t /= 60
|
||||
mm := t % 60
|
||||
t /= 60
|
||||
hh := t % 24
|
||||
return fmt.Sprintf("%02d:%02d:%02d", hh, mm, ss)
|
||||
}
|
||||
141
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_interface.go
generated
vendored
141
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_interface.go
generated
vendored
@@ -1,141 +0,0 @@
|
||||
package sigar
|
||||
|
||||
import (
|
||||
"time"
|
||||
)
|
||||
|
||||
type Sigar interface {
|
||||
CollectCpuStats(collectionInterval time.Duration) (<-chan Cpu, chan<- struct{})
|
||||
GetLoadAverage() (LoadAverage, error)
|
||||
GetMem() (Mem, error)
|
||||
GetSwap() (Swap, error)
|
||||
GetFileSystemUsage(string) (FileSystemUsage, error)
|
||||
}
|
||||
|
||||
type Cpu struct {
|
||||
User uint64
|
||||
Nice uint64
|
||||
Sys uint64
|
||||
Idle uint64
|
||||
Wait uint64
|
||||
Irq uint64
|
||||
SoftIrq uint64
|
||||
Stolen uint64
|
||||
}
|
||||
|
||||
func (cpu *Cpu) Total() uint64 {
|
||||
return cpu.User + cpu.Nice + cpu.Sys + cpu.Idle +
|
||||
cpu.Wait + cpu.Irq + cpu.SoftIrq + cpu.Stolen
|
||||
}
|
||||
|
||||
func (cpu Cpu) Delta(other Cpu) Cpu {
|
||||
return Cpu{
|
||||
User: cpu.User - other.User,
|
||||
Nice: cpu.Nice - other.Nice,
|
||||
Sys: cpu.Sys - other.Sys,
|
||||
Idle: cpu.Idle - other.Idle,
|
||||
Wait: cpu.Wait - other.Wait,
|
||||
Irq: cpu.Irq - other.Irq,
|
||||
SoftIrq: cpu.SoftIrq - other.SoftIrq,
|
||||
Stolen: cpu.Stolen - other.Stolen,
|
||||
}
|
||||
}
|
||||
|
||||
type LoadAverage struct {
|
||||
One, Five, Fifteen float64
|
||||
}
|
||||
|
||||
type Uptime struct {
|
||||
Length float64
|
||||
}
|
||||
|
||||
type Mem struct {
|
||||
Total uint64
|
||||
Used uint64
|
||||
Free uint64
|
||||
ActualFree uint64
|
||||
ActualUsed uint64
|
||||
}
|
||||
|
||||
type Swap struct {
|
||||
Total uint64
|
||||
Used uint64
|
||||
Free uint64
|
||||
}
|
||||
|
||||
type CpuList struct {
|
||||
List []Cpu
|
||||
}
|
||||
|
||||
type FileSystem struct {
|
||||
DirName string
|
||||
DevName string
|
||||
TypeName string
|
||||
SysTypeName string
|
||||
Options string
|
||||
Flags uint32
|
||||
}
|
||||
|
||||
type FileSystemList struct {
|
||||
List []FileSystem
|
||||
}
|
||||
|
||||
type FileSystemUsage struct {
|
||||
Total uint64
|
||||
Used uint64
|
||||
Free uint64
|
||||
Avail uint64
|
||||
Files uint64
|
||||
FreeFiles uint64
|
||||
}
|
||||
|
||||
type ProcList struct {
|
||||
List []int
|
||||
}
|
||||
|
||||
type RunState byte
|
||||
|
||||
const (
|
||||
RunStateSleep = 'S'
|
||||
RunStateRun = 'R'
|
||||
RunStateStop = 'T'
|
||||
RunStateZombie = 'Z'
|
||||
RunStateIdle = 'D'
|
||||
RunStateUnknown = '?'
|
||||
)
|
||||
|
||||
type ProcState struct {
|
||||
Name string
|
||||
State RunState
|
||||
Ppid int
|
||||
Tty int
|
||||
Priority int
|
||||
Nice int
|
||||
Processor int
|
||||
}
|
||||
|
||||
type ProcMem struct {
|
||||
Size uint64
|
||||
Resident uint64
|
||||
Share uint64
|
||||
MinorFaults uint64
|
||||
MajorFaults uint64
|
||||
PageFaults uint64
|
||||
}
|
||||
|
||||
type ProcTime struct {
|
||||
StartTime uint64
|
||||
User uint64
|
||||
Sys uint64
|
||||
Total uint64
|
||||
}
|
||||
|
||||
type ProcArgs struct {
|
||||
List []string
|
||||
}
|
||||
|
||||
type ProcExe struct {
|
||||
Name string
|
||||
Cwd string
|
||||
Root string
|
||||
}
|
||||
135
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_interface_test.go
generated
vendored
135
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_interface_test.go
generated
vendored
@@ -1,135 +0,0 @@
|
||||
package sigar_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
|
||||
. "github.com/onsi/ginkgo"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
. "github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
var _ = Describe("Sigar", func() {
|
||||
var invalidPid = 666666
|
||||
|
||||
It("cpu", func() {
|
||||
cpu := Cpu{}
|
||||
err := cpu.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("load average", func() {
|
||||
avg := LoadAverage{}
|
||||
err := avg.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("uptime", func() {
|
||||
uptime := Uptime{}
|
||||
err := uptime.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(uptime.Length).To(BeNumerically(">", 0))
|
||||
})
|
||||
|
||||
It("mem", func() {
|
||||
mem := Mem{}
|
||||
err := mem.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(mem.Total).To(BeNumerically(">", 0))
|
||||
Expect(mem.Used + mem.Free).To(BeNumerically("<=", mem.Total))
|
||||
})
|
||||
|
||||
It("swap", func() {
|
||||
swap := Swap{}
|
||||
err := swap.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(swap.Used + swap.Free).To(BeNumerically("<=", swap.Total))
|
||||
})
|
||||
|
||||
It("cpu list", func() {
|
||||
cpulist := CpuList{}
|
||||
err := cpulist.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
nsigar := len(cpulist.List)
|
||||
numcpu := runtime.NumCPU()
|
||||
Expect(nsigar).To(Equal(numcpu))
|
||||
})
|
||||
|
||||
It("file system list", func() {
|
||||
fslist := FileSystemList{}
|
||||
err := fslist.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(len(fslist.List)).To(BeNumerically(">", 0))
|
||||
})
|
||||
|
||||
It("file system usage", func() {
|
||||
fsusage := FileSystemUsage{}
|
||||
err := fsusage.Get("/")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
err = fsusage.Get("T O T A L L Y B O G U S")
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("proc list", func() {
|
||||
pids := ProcList{}
|
||||
err := pids.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(len(pids.List)).To(BeNumerically(">", 2))
|
||||
|
||||
err = pids.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("proc state", func() {
|
||||
state := ProcState{}
|
||||
err := state.Get(os.Getppid())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect([]RunState{RunStateRun, RunStateSleep}).To(ContainElement(state.State))
|
||||
Expect([]string{"go", "ginkgo"}).To(ContainElement(state.Name))
|
||||
|
||||
err = state.Get(invalidPid)
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("proc mem", func() {
|
||||
mem := ProcMem{}
|
||||
err := mem.Get(os.Getppid())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
err = mem.Get(invalidPid)
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("proc time", func() {
|
||||
time := ProcTime{}
|
||||
err := time.Get(os.Getppid())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
err = time.Get(invalidPid)
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("proc args", func() {
|
||||
args := ProcArgs{}
|
||||
err := args.Get(os.Getppid())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(len(args.List)).To(BeNumerically(">=", 2))
|
||||
})
|
||||
|
||||
It("proc exe", func() {
|
||||
exe := ProcExe{}
|
||||
err := exe.Get(os.Getppid())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect([]string{"go", "ginkgo"}).To(ContainElement(filepath.Base(exe.Name)))
|
||||
})
|
||||
})
|
||||
386
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_linux.go
generated
vendored
386
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_linux.go
generated
vendored
@@ -1,386 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package sigar
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
var system struct {
|
||||
ticks uint64
|
||||
btime uint64
|
||||
}
|
||||
|
||||
var Procd string
|
||||
|
||||
func init() {
|
||||
system.ticks = 100 // C.sysconf(C._SC_CLK_TCK)
|
||||
|
||||
Procd = "/proc"
|
||||
|
||||
// grab system boot time
|
||||
readFile(Procd+"/stat", func(line string) bool {
|
||||
if strings.HasPrefix(line, "btime") {
|
||||
system.btime, _ = strtoull(line[6:])
|
||||
return false // stop reading
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
func (self *LoadAverage) Get() error {
|
||||
line, err := ioutil.ReadFile(Procd + "/loadavg")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
fields := strings.Fields(string(line))
|
||||
|
||||
self.One, _ = strconv.ParseFloat(fields[0], 64)
|
||||
self.Five, _ = strconv.ParseFloat(fields[1], 64)
|
||||
self.Fifteen, _ = strconv.ParseFloat(fields[2], 64)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Uptime) Get() error {
|
||||
sysinfo := syscall.Sysinfo_t{}
|
||||
|
||||
if err := syscall.Sysinfo(&sysinfo); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Length = float64(sysinfo.Uptime)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Mem) Get() error {
|
||||
var buffers, cached uint64
|
||||
table := map[string]*uint64{
|
||||
"MemTotal": &self.Total,
|
||||
"MemFree": &self.Free,
|
||||
"Buffers": &buffers,
|
||||
"Cached": &cached,
|
||||
}
|
||||
|
||||
if err := parseMeminfo(table); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Used = self.Total - self.Free
|
||||
kern := buffers + cached
|
||||
self.ActualFree = self.Free + kern
|
||||
self.ActualUsed = self.Used - kern
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Swap) Get() error {
|
||||
table := map[string]*uint64{
|
||||
"SwapTotal": &self.Total,
|
||||
"SwapFree": &self.Free,
|
||||
}
|
||||
|
||||
if err := parseMeminfo(table); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
self.Used = self.Total - self.Free
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Cpu) Get() error {
|
||||
return readFile(Procd+"/stat", func(line string) bool {
|
||||
if len(line) > 4 && line[0:4] == "cpu " {
|
||||
parseCpuStat(self, line)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
|
||||
})
|
||||
}
|
||||
|
||||
func (self *CpuList) Get() error {
|
||||
capacity := len(self.List)
|
||||
if capacity == 0 {
|
||||
capacity = 4
|
||||
}
|
||||
list := make([]Cpu, 0, capacity)
|
||||
|
||||
err := readFile(Procd+"/stat", func(line string) bool {
|
||||
if len(line) > 3 && line[0:3] == "cpu" && line[3] != ' ' {
|
||||
cpu := Cpu{}
|
||||
parseCpuStat(&cpu, line)
|
||||
list = append(list, cpu)
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
self.List = list
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
func (self *FileSystemList) Get() error {
|
||||
capacity := len(self.List)
|
||||
if capacity == 0 {
|
||||
capacity = 10
|
||||
}
|
||||
fslist := make([]FileSystem, 0, capacity)
|
||||
|
||||
err := readFile("/etc/mtab", func(line string) bool {
|
||||
fields := strings.Fields(line)
|
||||
|
||||
fs := FileSystem{}
|
||||
fs.DevName = fields[0]
|
||||
fs.DirName = fields[1]
|
||||
fs.SysTypeName = fields[2]
|
||||
fs.Options = fields[3]
|
||||
|
||||
fslist = append(fslist, fs)
|
||||
|
||||
return true
|
||||
})
|
||||
|
||||
self.List = fslist
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
func (self *ProcList) Get() error {
|
||||
dir, err := os.Open(Procd)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
|
||||
const readAllDirnames = -1 // see os.File.Readdirnames doc
|
||||
|
||||
names, err := dir.Readdirnames(readAllDirnames)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
capacity := len(names)
|
||||
list := make([]int, 0, capacity)
|
||||
|
||||
for _, name := range names {
|
||||
if name[0] < '0' || name[0] > '9' {
|
||||
continue
|
||||
}
|
||||
pid, err := strconv.Atoi(name)
|
||||
if err == nil {
|
||||
list = append(list, pid)
|
||||
}
|
||||
}
|
||||
|
||||
self.List = list
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcState) Get(pid int) error {
|
||||
contents, err := readProcFile(pid, "stat")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fields := strings.Fields(string(contents))
|
||||
|
||||
self.Name = fields[1][1 : len(fields[1])-1] // strip ()'s
|
||||
|
||||
self.State = RunState(fields[2][0])
|
||||
|
||||
self.Ppid, _ = strconv.Atoi(fields[3])
|
||||
|
||||
self.Tty, _ = strconv.Atoi(fields[6])
|
||||
|
||||
self.Priority, _ = strconv.Atoi(fields[17])
|
||||
|
||||
self.Nice, _ = strconv.Atoi(fields[18])
|
||||
|
||||
self.Processor, _ = strconv.Atoi(fields[38])
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcMem) Get(pid int) error {
|
||||
contents, err := readProcFile(pid, "statm")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fields := strings.Fields(string(contents))
|
||||
|
||||
size, _ := strtoull(fields[0])
|
||||
self.Size = size << 12
|
||||
|
||||
rss, _ := strtoull(fields[1])
|
||||
self.Resident = rss << 12
|
||||
|
||||
share, _ := strtoull(fields[2])
|
||||
self.Share = share << 12
|
||||
|
||||
contents, err = readProcFile(pid, "stat")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fields = strings.Fields(string(contents))
|
||||
|
||||
self.MinorFaults, _ = strtoull(fields[10])
|
||||
self.MajorFaults, _ = strtoull(fields[12])
|
||||
self.PageFaults = self.MinorFaults + self.MajorFaults
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcTime) Get(pid int) error {
|
||||
contents, err := readProcFile(pid, "stat")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fields := strings.Fields(string(contents))
|
||||
|
||||
user, _ := strtoull(fields[13])
|
||||
sys, _ := strtoull(fields[14])
|
||||
// convert to millis
|
||||
self.User = user * (1000 / system.ticks)
|
||||
self.Sys = sys * (1000 / system.ticks)
|
||||
self.Total = self.User + self.Sys
|
||||
|
||||
// convert to millis
|
||||
self.StartTime, _ = strtoull(fields[21])
|
||||
self.StartTime /= system.ticks
|
||||
self.StartTime += system.btime
|
||||
self.StartTime *= 1000
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcArgs) Get(pid int) error {
|
||||
contents, err := readProcFile(pid, "cmdline")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
bbuf := bytes.NewBuffer(contents)
|
||||
|
||||
var args []string
|
||||
|
||||
for {
|
||||
arg, err := bbuf.ReadBytes(0)
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
args = append(args, string(chop(arg)))
|
||||
}
|
||||
|
||||
self.List = args
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ProcExe) Get(pid int) error {
|
||||
fields := map[string]*string{
|
||||
"exe": &self.Name,
|
||||
"cwd": &self.Cwd,
|
||||
"root": &self.Root,
|
||||
}
|
||||
|
||||
for name, field := range fields {
|
||||
val, err := os.Readlink(procFileName(pid, name))
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
*field = val
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseMeminfo(table map[string]*uint64) error {
|
||||
return readFile(Procd+"/meminfo", func(line string) bool {
|
||||
fields := strings.Split(line, ":")
|
||||
|
||||
if ptr := table[fields[0]]; ptr != nil {
|
||||
num := strings.TrimLeft(fields[1], " ")
|
||||
val, err := strtoull(strings.Fields(num)[0])
|
||||
if err == nil {
|
||||
*ptr = val * 1024
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
func parseCpuStat(self *Cpu, line string) error {
|
||||
fields := strings.Fields(line)
|
||||
|
||||
self.User, _ = strtoull(fields[1])
|
||||
self.Nice, _ = strtoull(fields[2])
|
||||
self.Sys, _ = strtoull(fields[3])
|
||||
self.Idle, _ = strtoull(fields[4])
|
||||
self.Wait, _ = strtoull(fields[5])
|
||||
self.Irq, _ = strtoull(fields[6])
|
||||
self.SoftIrq, _ = strtoull(fields[7])
|
||||
self.Stolen, _ = strtoull(fields[8])
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func readFile(file string, handler func(string) bool) error {
|
||||
contents, err := ioutil.ReadFile(file)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
reader := bufio.NewReader(bytes.NewBuffer(contents))
|
||||
|
||||
for {
|
||||
line, _, err := reader.ReadLine()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if !handler(string(line)) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func strtoull(val string) (uint64, error) {
|
||||
return strconv.ParseUint(val, 10, 64)
|
||||
}
|
||||
|
||||
func procFileName(pid int, name string) string {
|
||||
return Procd + "/" + strconv.Itoa(pid) + "/" + name
|
||||
}
|
||||
|
||||
func readProcFile(pid int, name string) ([]byte, error) {
|
||||
path := procFileName(pid, name)
|
||||
contents, err := ioutil.ReadFile(path)
|
||||
|
||||
if err != nil {
|
||||
if perr, ok := err.(*os.PathError); ok {
|
||||
if perr.Err == syscall.ENOENT {
|
||||
return nil, syscall.ESRCH
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return contents, err
|
||||
}
|
||||
225
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_linux_test.go
generated
vendored
225
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_linux_test.go
generated
vendored
@@ -1,225 +0,0 @@
|
||||
package sigar_test
|
||||
|
||||
import (
|
||||
"io/ioutil"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
sigar "github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
var _ = Describe("sigarLinux", func() {
|
||||
var procd string
|
||||
|
||||
BeforeEach(func() {
|
||||
var err error
|
||||
procd, err = ioutil.TempDir("", "sigarTests")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
sigar.Procd = procd
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
sigar.Procd = "/proc"
|
||||
})
|
||||
|
||||
Describe("CPU", func() {
|
||||
var (
|
||||
statFile string
|
||||
cpu sigar.Cpu
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
statFile = procd + "/stat"
|
||||
cpu = sigar.Cpu{}
|
||||
})
|
||||
|
||||
Describe("Get", func() {
|
||||
It("gets CPU usage", func() {
|
||||
statContents := []byte("cpu 25 1 2 3 4 5 6 7")
|
||||
err := ioutil.WriteFile(statFile, statContents, 0644)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
err = cpu.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(cpu.User).To(Equal(uint64(25)))
|
||||
})
|
||||
|
||||
It("ignores empty lines", func() {
|
||||
statContents := []byte("cpu ")
|
||||
err := ioutil.WriteFile(statFile, statContents, 0644)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
err = cpu.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(cpu.User).To(Equal(uint64(0)))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("CollectCpuStats", func() {
|
||||
It("collects CPU usage over time", func() {
|
||||
statContents := []byte("cpu 25 1 2 3 4 5 6 7")
|
||||
err := ioutil.WriteFile(statFile, statContents, 0644)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
concreteSigar := &sigar.ConcreteSigar{}
|
||||
cpuUsages, stop := concreteSigar.CollectCpuStats(500 * time.Millisecond)
|
||||
|
||||
Expect(<-cpuUsages).To(Equal(sigar.Cpu{
|
||||
User: uint64(25),
|
||||
Nice: uint64(1),
|
||||
Sys: uint64(2),
|
||||
Idle: uint64(3),
|
||||
Wait: uint64(4),
|
||||
Irq: uint64(5),
|
||||
SoftIrq: uint64(6),
|
||||
Stolen: uint64(7),
|
||||
}))
|
||||
|
||||
statContents = []byte("cpu 30 3 7 10 25 55 36 65")
|
||||
err = ioutil.WriteFile(statFile, statContents, 0644)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(<-cpuUsages).To(Equal(sigar.Cpu{
|
||||
User: uint64(5),
|
||||
Nice: uint64(2),
|
||||
Sys: uint64(5),
|
||||
Idle: uint64(7),
|
||||
Wait: uint64(21),
|
||||
Irq: uint64(50),
|
||||
SoftIrq: uint64(30),
|
||||
Stolen: uint64(58),
|
||||
}))
|
||||
|
||||
stop <- struct{}{}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("Mem", func() {
|
||||
var meminfoFile string
|
||||
BeforeEach(func() {
|
||||
meminfoFile = procd + "/meminfo"
|
||||
|
||||
meminfoContents := `
|
||||
MemTotal: 374256 kB
|
||||
MemFree: 274460 kB
|
||||
Buffers: 9764 kB
|
||||
Cached: 38648 kB
|
||||
SwapCached: 0 kB
|
||||
Active: 33772 kB
|
||||
Inactive: 31184 kB
|
||||
Active(anon): 16572 kB
|
||||
Inactive(anon): 552 kB
|
||||
Active(file): 17200 kB
|
||||
Inactive(file): 30632 kB
|
||||
Unevictable: 0 kB
|
||||
Mlocked: 0 kB
|
||||
SwapTotal: 786428 kB
|
||||
SwapFree: 786428 kB
|
||||
Dirty: 0 kB
|
||||
Writeback: 0 kB
|
||||
AnonPages: 16564 kB
|
||||
Mapped: 6612 kB
|
||||
Shmem: 584 kB
|
||||
Slab: 19092 kB
|
||||
SReclaimable: 9128 kB
|
||||
SUnreclaim: 9964 kB
|
||||
KernelStack: 672 kB
|
||||
PageTables: 1864 kB
|
||||
NFS_Unstable: 0 kB
|
||||
Bounce: 0 kB
|
||||
WritebackTmp: 0 kB
|
||||
CommitLimit: 973556 kB
|
||||
Committed_AS: 55880 kB
|
||||
VmallocTotal: 34359738367 kB
|
||||
VmallocUsed: 21428 kB
|
||||
VmallocChunk: 34359713596 kB
|
||||
HardwareCorrupted: 0 kB
|
||||
AnonHugePages: 0 kB
|
||||
HugePages_Total: 0
|
||||
HugePages_Free: 0
|
||||
HugePages_Rsvd: 0
|
||||
HugePages_Surp: 0
|
||||
Hugepagesize: 2048 kB
|
||||
DirectMap4k: 59328 kB
|
||||
DirectMap2M: 333824 kB
|
||||
`
|
||||
err := ioutil.WriteFile(meminfoFile, []byte(meminfoContents), 0444)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("returns correct memory info", func() {
|
||||
mem := sigar.Mem{}
|
||||
err := mem.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(mem.Total).To(BeNumerically("==", 374256*1024))
|
||||
Expect(mem.Free).To(BeNumerically("==", 274460*1024))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("Swap", func() {
|
||||
var meminfoFile string
|
||||
BeforeEach(func() {
|
||||
meminfoFile = procd + "/meminfo"
|
||||
|
||||
meminfoContents := `
|
||||
MemTotal: 374256 kB
|
||||
MemFree: 274460 kB
|
||||
Buffers: 9764 kB
|
||||
Cached: 38648 kB
|
||||
SwapCached: 0 kB
|
||||
Active: 33772 kB
|
||||
Inactive: 31184 kB
|
||||
Active(anon): 16572 kB
|
||||
Inactive(anon): 552 kB
|
||||
Active(file): 17200 kB
|
||||
Inactive(file): 30632 kB
|
||||
Unevictable: 0 kB
|
||||
Mlocked: 0 kB
|
||||
SwapTotal: 786428 kB
|
||||
SwapFree: 786428 kB
|
||||
Dirty: 0 kB
|
||||
Writeback: 0 kB
|
||||
AnonPages: 16564 kB
|
||||
Mapped: 6612 kB
|
||||
Shmem: 584 kB
|
||||
Slab: 19092 kB
|
||||
SReclaimable: 9128 kB
|
||||
SUnreclaim: 9964 kB
|
||||
KernelStack: 672 kB
|
||||
PageTables: 1864 kB
|
||||
NFS_Unstable: 0 kB
|
||||
Bounce: 0 kB
|
||||
WritebackTmp: 0 kB
|
||||
CommitLimit: 973556 kB
|
||||
Committed_AS: 55880 kB
|
||||
VmallocTotal: 34359738367 kB
|
||||
VmallocUsed: 21428 kB
|
||||
VmallocChunk: 34359713596 kB
|
||||
HardwareCorrupted: 0 kB
|
||||
AnonHugePages: 0 kB
|
||||
HugePages_Total: 0
|
||||
HugePages_Free: 0
|
||||
HugePages_Rsvd: 0
|
||||
HugePages_Surp: 0
|
||||
Hugepagesize: 2048 kB
|
||||
DirectMap4k: 59328 kB
|
||||
DirectMap2M: 333824 kB
|
||||
`
|
||||
err := ioutil.WriteFile(meminfoFile, []byte(meminfoContents), 0444)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("returns correct memory info", func() {
|
||||
swap := sigar.Swap{}
|
||||
err := swap.Get()
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
Expect(swap.Total).To(BeNumerically("==", 786428*1024))
|
||||
Expect(swap.Free).To(BeNumerically("==", 786428*1024))
|
||||
})
|
||||
})
|
||||
})
|
||||
13
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_suite_test.go
generated
vendored
13
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_suite_test.go
generated
vendored
@@ -1,13 +0,0 @@
|
||||
package sigar_test
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGosigar(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Gosigar Suite")
|
||||
}
|
||||
26
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_unix.go
generated
vendored
26
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_unix.go
generated
vendored
@@ -1,26 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
// +build darwin freebsd linux netbsd openbsd
|
||||
|
||||
package sigar
|
||||
|
||||
import "syscall"
|
||||
|
||||
func (self *FileSystemUsage) Get(path string) error {
|
||||
stat := syscall.Statfs_t{}
|
||||
err := syscall.Statfs(path, &stat)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
bsize := stat.Bsize / 512
|
||||
|
||||
self.Total = (uint64(stat.Blocks) * uint64(bsize)) >> 1
|
||||
self.Free = (uint64(stat.Bfree) * uint64(bsize)) >> 1
|
||||
self.Avail = (uint64(stat.Bavail) * uint64(bsize)) >> 1
|
||||
self.Used = self.Total - self.Free
|
||||
self.Files = stat.Files
|
||||
self.FreeFiles = stat.Ffree
|
||||
|
||||
return nil
|
||||
}
|
||||
22
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_util.go
generated
vendored
22
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_util.go
generated
vendored
@@ -1,22 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package sigar
|
||||
|
||||
import (
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
func bytePtrToString(ptr *int8) string {
|
||||
bytes := (*[10000]byte)(unsafe.Pointer(ptr))
|
||||
|
||||
n := 0
|
||||
for bytes[n] != 0 {
|
||||
n++
|
||||
}
|
||||
|
||||
return string(bytes[0:n])
|
||||
}
|
||||
|
||||
func chop(buf []byte) []byte {
|
||||
return buf[0 : len(buf)-1]
|
||||
}
|
||||
100
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_windows.go
generated
vendored
100
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_windows.go
generated
vendored
@@ -1,100 +0,0 @@
|
||||
// Copyright (c) 2012 VMware, Inc.
|
||||
|
||||
package sigar
|
||||
|
||||
// #include <stdlib.h>
|
||||
// #include <windows.h>
|
||||
import "C"
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
func init() {
|
||||
}
|
||||
|
||||
func (self *LoadAverage) Get() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Uptime) Get() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Mem) Get() error {
|
||||
var statex C.MEMORYSTATUSEX
|
||||
statex.dwLength = C.DWORD(unsafe.Sizeof(statex))
|
||||
|
||||
succeeded := C.GlobalMemoryStatusEx(&statex)
|
||||
if succeeded == C.FALSE {
|
||||
lastError := C.GetLastError()
|
||||
return fmt.Errorf("GlobalMemoryStatusEx failed with error: %d", int(lastError))
|
||||
}
|
||||
|
||||
self.Total = uint64(statex.ullTotalPhys)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *Swap) Get() error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *Cpu) Get() error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *CpuList) Get() error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *FileSystemList) Get() error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *ProcList) Get() error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *ProcState) Get(pid int) error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *ProcMem) Get(pid int) error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *ProcTime) Get(pid int) error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *ProcArgs) Get(pid int) error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *ProcExe) Get(pid int) error {
|
||||
return notImplemented()
|
||||
}
|
||||
|
||||
func (self *FileSystemUsage) Get(path string) error {
|
||||
var availableBytes C.ULARGE_INTEGER
|
||||
var totalBytes C.ULARGE_INTEGER
|
||||
var totalFreeBytes C.ULARGE_INTEGER
|
||||
|
||||
pathChars := C.CString(path)
|
||||
defer C.free(unsafe.Pointer(pathChars))
|
||||
|
||||
succeeded := C.GetDiskFreeSpaceEx((*C.CHAR)(pathChars), &availableBytes, &totalBytes, &totalFreeBytes)
|
||||
if succeeded == C.FALSE {
|
||||
lastError := C.GetLastError()
|
||||
return fmt.Errorf("GetDiskFreeSpaceEx failed with error: %d", int(lastError))
|
||||
}
|
||||
|
||||
self.Total = *(*uint64)(unsafe.Pointer(&totalBytes))
|
||||
return nil
|
||||
}
|
||||
|
||||
func notImplemented() error {
|
||||
panic("Not Implemented")
|
||||
return nil
|
||||
}
|
||||
32
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_windows_test.go
generated
vendored
32
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_windows_test.go
generated
vendored
@@ -1,32 +0,0 @@
|
||||
package sigar_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
. "github.com/onsi/ginkgo"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
sigar "github.com/cloudfoundry/gosigar"
|
||||
)
|
||||
|
||||
var _ = Describe("SigarWindows", func() {
|
||||
Describe("Memory", func() {
|
||||
It("gets the total memory", func() {
|
||||
mem := sigar.Mem{}
|
||||
err := mem.Get()
|
||||
|
||||
Ω(err).ShouldNot(HaveOccurred())
|
||||
Ω(mem.Total).Should(BeNumerically(">", 0))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("Disk", func() {
|
||||
It("gets the total disk space", func() {
|
||||
usage := sigar.FileSystemUsage{}
|
||||
err := usage.Get(os.TempDir())
|
||||
|
||||
Ω(err).ShouldNot(HaveOccurred())
|
||||
Ω(usage.Total).Should(BeNumerically(">", 0))
|
||||
})
|
||||
})
|
||||
})
|
||||
67
Godeps/_workspace/src/github.com/influxdb/influxdb/.gitignore
generated
vendored
67
Godeps/_workspace/src/github.com/influxdb/influxdb/.gitignore
generated
vendored
@@ -1,67 +0,0 @@
|
||||
*~
|
||||
src/
|
||||
|
||||
config.json
|
||||
/bin/
|
||||
|
||||
/pkg/
|
||||
|
||||
TAGS
|
||||
|
||||
# vim temp files
|
||||
*.swp
|
||||
|
||||
*.test
|
||||
/query/a.out*
|
||||
.DS_Store
|
||||
|
||||
# ignore generated files.
|
||||
cmd/influxd/version.go
|
||||
|
||||
# executables
|
||||
|
||||
influxd
|
||||
**/influxd
|
||||
!**/influxd/
|
||||
|
||||
influx
|
||||
**/influx
|
||||
!**/influx/
|
||||
|
||||
influxdb
|
||||
**/influxdb
|
||||
!**/influxdb/
|
||||
|
||||
/benchmark-tool
|
||||
/main
|
||||
/benchmark-storage
|
||||
godef
|
||||
gosym
|
||||
gocode
|
||||
inspect-raft
|
||||
|
||||
# dependencies
|
||||
out_rpm/
|
||||
packages/
|
||||
|
||||
# autconf
|
||||
autom4te.cache/
|
||||
config.log
|
||||
config.status
|
||||
Makefile
|
||||
|
||||
# log file
|
||||
influxdb.log
|
||||
benchmark.log
|
||||
|
||||
# config file
|
||||
config.toml
|
||||
|
||||
# test data files
|
||||
integration/migration_data/
|
||||
|
||||
# goide project files
|
||||
.idea
|
||||
|
||||
# goconvey config files
|
||||
*.goconvey
|
||||
1607
Godeps/_workspace/src/github.com/influxdb/influxdb/CHANGELOG.md
generated
vendored
1607
Godeps/_workspace/src/github.com/influxdb/influxdb/CHANGELOG.md
generated
vendored
File diff suppressed because it is too large
Load Diff
231
Godeps/_workspace/src/github.com/influxdb/influxdb/CONTRIBUTING.md
generated
vendored
231
Godeps/_workspace/src/github.com/influxdb/influxdb/CONTRIBUTING.md
generated
vendored
@@ -1,231 +0,0 @@
|
||||
Contributing to InfluxDB
|
||||
========================
|
||||
|
||||
Bug reports
|
||||
---------------
|
||||
Before you file an issue, please search existing issues in case it has already been filed, or perhaps even fixed. If you file an issue, please include the following.
|
||||
* Full details of your operating system (or distribution) e.g. 64-bit Ubuntu 14.04.
|
||||
* The version of InfluxDB you are running
|
||||
* Whether you installed it using a pre-built package, or built it from source.
|
||||
* A small test case, if applicable, that demonstrates the issues.
|
||||
|
||||
Remember the golden rule of bug reports: **The easier you make it for us to reproduce the problem, the faster it will get fixed.**
|
||||
If you have never written a bug report before, or if you want to brush up on your bug reporting skills, we recommend reading [Simon Tatham's essay "How to Report Bugs Effectively."](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
|
||||
|
||||
Test cases should be in the form of `curl` commands. For example:
|
||||
```
|
||||
# create database
|
||||
curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"
|
||||
|
||||
# create retention policy
|
||||
curl -G http://localhost:8086/query --data-urlencode "q=CREATE RETENTION POLICY myrp ON mydb DURATION 365d REPLICATION 1 DEFAULT"
|
||||
|
||||
# write data
|
||||
curl -X POST http://localhost:8086/write --data-urlencode "db=mydb" --data-binary "cpu,region=useast,host=server_1,service=redis value=61"
|
||||
|
||||
# Delete a Measurement
|
||||
curl -G http://localhost:8086/query --data-urlencode 'db=mydb' --data-urlencode 'q=DROP MEASUREMENT cpu'
|
||||
|
||||
# Query the Measurement
|
||||
# Bug: expected it to return no data, but data comes back.
|
||||
curl -G http://localhost:8086/query --data-urlencode 'db=mydb' --data-urlencode 'q=SELECT * from cpu'
|
||||
```
|
||||
**If you don't include a clear test case like this, your issue may not be investigated, and may even be closed**. If writing the data is too difficult, please zip up your data directory and include a link to it in your bug report.
|
||||
|
||||
Please note that issues are *not the place to file general questions* such as "how do I use collectd with InfluxDB?" Questions of this nature should be sent to the [Google Group](https://groups.google.com/forum/#!forum/influxdb), not filed as issues. Issues like this will be closed.
|
||||
|
||||
Feature requests
|
||||
---------------
|
||||
We really like to receive feature requests, as it helps us prioritize our work. Please be clear about your requirements, as incomplete feature requests may simply be closed if we don't understand what you would like to see added to InfluxDB.
|
||||
|
||||
Contributing to the source code
|
||||
---------------
|
||||
|
||||
InfluxDB follows standard Go project structure. This means that all
|
||||
your go development are done in `$GOPATH/src`. GOPATH can be any
|
||||
directory under which InfluxDB and all its dependencies will be
|
||||
cloned. For more details on recommended go project's structure, see
|
||||
[How to Write Go Code](http://golang.org/doc/code.html) and
|
||||
[Go: Best Practices for Production Environments](http://peter.bourgon.org/go-in-production/), or you can just follow
|
||||
the steps below.
|
||||
|
||||
Submitting a pull request
|
||||
------------
|
||||
To submit a pull request you should fork the InfluxDB repository, and make your change on a feature branch of your fork. Then generate a pull request from your branch against *master* of the InfluxDB repository. Include in your pull request details of your change -- the why *and* the how -- as well as the testing your performed. Also, be sure to run the test suite with your change in place. Changes that cause tests to fail cannot be merged.
|
||||
|
||||
There will usually be some back and forth as we finalize the change, but once that completes it may be merged.
|
||||
|
||||
To assist in review for the PR, please add the following to your pull request comment:
|
||||
|
||||
```md
|
||||
- [ ] CHANGELOG.md updated
|
||||
- [ ] Rebased/mergable
|
||||
- [ ] Tests pass
|
||||
- [ ] Sign [CLA](http://influxdb.com/community/cla.html) (if not already signed)
|
||||
```
|
||||
|
||||
Use of third-party packages
|
||||
------------
|
||||
A third-party package is defined as one that is not part of the standard Go distribution. Generally speaking we prefer to minimize our use of third-party packages, and avoid them unless absolutely necessarly. We'll often write a little bit of code rather than pull in a third-party package. Of course, we do use some third-party packages -- most importantly we use [BoltDB](https://github.com/boltdb/bolt) as the storage engine. So to maximise the chance your change will be accepted by us, use only the standard libaries, or the third-party packages we have decided to use.
|
||||
|
||||
For rationale, check out the post [The Case Against Third Party Libraries](http://blog.gopheracademy.com/advent-2014/case-against-3pl/).
|
||||
|
||||
Signing the CLA
|
||||
---------------
|
||||
|
||||
If you are going to be contributing back to InfluxDB please take a
|
||||
second to sign our CLA, which can be found
|
||||
[on our website](http://influxdb.com/community/cla.html).
|
||||
|
||||
Installing Go
|
||||
-------------
|
||||
InfluxDB requires Go 1.4 or greater.
|
||||
|
||||
At InfluxDB we find gvm, a Go version manager, useful for installing Go. For instructions
|
||||
on how to install it see [the gvm page on github](https://github.com/moovweb/gvm).
|
||||
|
||||
After installing gvm you can install and set the default go version by
|
||||
running the following:
|
||||
|
||||
gvm install go1.4
|
||||
gvm use go1.4 --default
|
||||
|
||||
Revision Control Systems
|
||||
------
|
||||
Go has the ability to import remote packages via revision control systems with the `go get` command. To ensure that you can retrieve any remote package, be sure to install the following rcs software to your system.
|
||||
Currently the project only depends on `git` and `mercurial`.
|
||||
|
||||
* [Install Git](http://git-scm.com/book/en/Getting-Started-Installing-Git)
|
||||
* [Install Mercurial](http://mercurial.selenic.com/wiki/Download)
|
||||
|
||||
Project structure
|
||||
-----------------
|
||||
First you need to setup the project structure:
|
||||
|
||||
export GOPATH=$HOME/gocodez
|
||||
mkdir -p $GOPATH/src/github.com/influxdb
|
||||
cd $GOPATH/src/github.com/influxdb
|
||||
git clone git@github.com:influxdb/influxdb
|
||||
|
||||
You can add the line `export GOPATH=$HOME/gocodez` to your bash/zsh
|
||||
file to be set for every shell instead of having to manually run it
|
||||
everytime.
|
||||
|
||||
We have a pre commit hook to make sure code is formatted properly
|
||||
and vetted before you commit any changes. We strongly recommend using the pre
|
||||
commit hook to guard against accidentally committing unformatted
|
||||
code. To use the pre-commit hook, run the following:
|
||||
|
||||
cd $GOPATH/src/github.com/influxdb/influxdb
|
||||
cp .hooks/pre-commit .git/hooks/
|
||||
|
||||
In case the commit is rejected because it's not formatted you can run
|
||||
the following to format the code:
|
||||
|
||||
```
|
||||
go fmt ./...
|
||||
go vet ./...
|
||||
```
|
||||
|
||||
To install go vet, run the following command:
|
||||
```
|
||||
go get golang.org/x/tools/cmd/vet
|
||||
```
|
||||
|
||||
NOTE: If you have not installed mercurial, the above command will fail. See [Revision Control Systems](#revision-control-systems) above.
|
||||
|
||||
For more information on `go vet`, [read the GoDoc](https://godoc.org/golang.org/x/tools/cmd/vet).
|
||||
|
||||
Build and Test
|
||||
-----
|
||||
|
||||
Make sure you have Go installed and the project structure as shown above. To then build the project, execute the following commands:
|
||||
|
||||
```bash
|
||||
cd $GOPATH/src/github.com/influxdb
|
||||
go get -u -f -t ./...
|
||||
go build ./...
|
||||
```
|
||||
|
||||
To then install the binaries, run the following command. They can be found in `$GOPATH/bin`. Please note that the InfluxDB binary is named `influxd`, not `influxdb`.
|
||||
|
||||
```bash
|
||||
go install ./...
|
||||
```
|
||||
|
||||
To set the version and commit flags during the build pass the following to the build command:
|
||||
|
||||
```bash
|
||||
-ldflags="-X main.version $VERSION -X main.branch $BRANCH -X main.commit $COMMIT"
|
||||
```
|
||||
|
||||
where `$VERSION` is the version, `$BRANCH` is the branch, and `$COMMIT` is the git commit hash.
|
||||
|
||||
To run the tests, execute the following command:
|
||||
|
||||
```bash
|
||||
cd $GOPATH/src/github.com/influxdb/influxdb
|
||||
go test -v ./...
|
||||
|
||||
# run tests that match some pattern
|
||||
go test -run=TestDatabase . -v
|
||||
|
||||
# run tests and show coverage
|
||||
go test -coverprofile /tmp/cover . && go tool cover -html /tmp/cover
|
||||
```
|
||||
|
||||
To install go cover, run the following command:
|
||||
```
|
||||
go get golang.org/x/tools/cmd/cover
|
||||
```
|
||||
|
||||
Generated Google Protobuf code
|
||||
-----------------
|
||||
Most changes to the source do not require that the generated protocol buffer code be changed. But if you need to modify the protocol buffer code, you'll first need to install the protocol buffers toolchain.
|
||||
|
||||
First install the [protocol buffer compiler](https://developers.google.com/protocol-buffers/
|
||||
) 2.6.1 or later for your OS:
|
||||
|
||||
Then install the go plugins:
|
||||
|
||||
```bash
|
||||
go get github.com/gogo/protobuf/proto
|
||||
go get github.com/gogo/protobuf/protoc-gen-gogo
|
||||
go get github.com/gogo/protobuf/gogoproto
|
||||
```
|
||||
|
||||
Finally run, `go generate` after updating any `*.proto` file:
|
||||
|
||||
```bash
|
||||
go generate ./...
|
||||
```
|
||||
**Trouleshooting**
|
||||
|
||||
If generating the protobuf code is failing for you, check each of the following:
|
||||
* Ensure the protobuf library can be found. Make sure that `LD_LIBRRARY_PATH` includes the directory in which the library `libprotoc.so` has been installed.
|
||||
* Ensure the command `protoc-gen-gogo`, found in `GOPATH/bin`, is on your path. This can be done by adding `GOPATH/bin` to `PATH`.
|
||||
|
||||
Profiling
|
||||
-----
|
||||
When troubleshooting problems with CPU or memory the Go toolchain can be helpful. You can start InfluxDB with CPU or memory profiling turned on. For example:
|
||||
|
||||
```sh
|
||||
# start influx with profiling
|
||||
./influxd -cpuprofile influxd.prof
|
||||
# run queries, writes, whatever you're testing
|
||||
# open up pprof
|
||||
go tool pprof influxd influxd.prof
|
||||
# once inside run "web", opens up browser with the CPU graph
|
||||
# can also run "web <function name>" to zoom in. Or "list <function name>" to see specific lines
|
||||
```
|
||||
|
||||
Continuous Integration testing
|
||||
-----
|
||||
InfluxDB uses CirceCI for continuous integration testing. To see how the code is built and tested, check out [this file](https://github.com/influxdb/influxdb/blob/master/circle-test.sh). It closely follows the build and test process outlined above. You can see the exact version of Go InfluxDB uses for testing by consulting that file.
|
||||
|
||||
Useful links
|
||||
------------
|
||||
- [Useful techniques in Go](http://arslan.io/ten-useful-techniques-in-go)
|
||||
- [Go in production](http://peter.bourgon.org/go-in-production/)
|
||||
- [Principles of designing Go APIs with channels](https://inconshreveable.com/07-08-2014/principles-of-designing-go-apis-with-channels/)
|
||||
- [Common mistakes in Golang](http://soryy.com/blog/2014/common-mistakes-with-go-lang/). Especially this section `Loops, Closures, and Local Variables`
|
||||
44
Godeps/_workspace/src/github.com/influxdb/influxdb/DOCKER.md
generated
vendored
44
Godeps/_workspace/src/github.com/influxdb/influxdb/DOCKER.md
generated
vendored
@@ -1,44 +0,0 @@
|
||||
# Docker Setup
|
||||
========================
|
||||
|
||||
This document describes how to build and run a minimal InfluxDB container under Docker. Currently, it has only been tested for local development and assumes that you have a working docker environment.
|
||||
|
||||
## Building Image
|
||||
|
||||
To build a docker image for InfluxDB from your current checkout, run the following:
|
||||
|
||||
```
|
||||
$ ./build-docker.sh
|
||||
```
|
||||
|
||||
This script uses the `golang:1.5` image to build a fully static binary of `influxd` and then adds it to a minimal `scratch` image.
|
||||
|
||||
To build the image using a different version of go:
|
||||
|
||||
```
|
||||
$ GO_VER=1.4.2 ./build-docker.sh
|
||||
```
|
||||
|
||||
Available version can be found [here](https://hub.docker.com/_/golang/).
|
||||
|
||||
## Single Node Container
|
||||
|
||||
This will start an interactive, single-node, that publishes the containers port `8086` and `8088` to the hosts ports `8086` and `8088` respectively. This is identical to starting `influxd` manually.
|
||||
|
||||
```
|
||||
$ docker run -it -p 8086:8086 -p 8088:8088 influxdb
|
||||
```
|
||||
|
||||
## Multi-Node Cluster
|
||||
|
||||
This will create a simple 3-node cluster. The data is stored within the container and will be lost when the container is removed. This is only useful for test clusters.
|
||||
|
||||
The `HOST_IP` env variable should be your host IP if running under linux or the virtualbox VM IP if running under OSX. On OSX, this would be something like: `$(docker-machine ip dev)` or `$(boot2docker ip)` depending on which docker tool you are using.
|
||||
|
||||
```
|
||||
$ export HOST_IP=<your host/VM IP>
|
||||
$ docker run -it -p 8086:8088 -p 8088:8088 influxdb -hostname $HOST_IP:8088
|
||||
$ docker run -it -p 8186:8088 -p 8188:8088 influxdb -hostname $HOST_IP:8188 -join $HOST_IP:8088
|
||||
$ docker run -it -p 8286:8088 -p 8288:8088 influxdb -hostname $HOST_IP:8288 -join $HOST_IP:8088
|
||||
```
|
||||
|
||||
24
Godeps/_workspace/src/github.com/influxdb/influxdb/Dockerfile
generated
vendored
24
Godeps/_workspace/src/github.com/influxdb/influxdb/Dockerfile
generated
vendored
@@ -1,24 +0,0 @@
|
||||
FROM busybox:ubuntu-14.04
|
||||
|
||||
MAINTAINER Jason Wilder "<jason@influxdb.com>"
|
||||
|
||||
# admin, http, udp, cluster, graphite, opentsdb, collectd
|
||||
EXPOSE 8083 8086 8086/udp 8088 2003 4242 25826
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# copy binary into image
|
||||
COPY influxd /app/
|
||||
|
||||
# Add influxd to the PATH
|
||||
ENV PATH=/app:$PATH
|
||||
|
||||
# Generate a default config
|
||||
RUN influxd config > /etc/influxdb.toml
|
||||
|
||||
# Use /data for all disk storage
|
||||
RUN sed -i 's/dir = "\/.*influxdb/dir = "\/data/' /etc/influxdb.toml
|
||||
|
||||
VOLUME ["/data"]
|
||||
|
||||
ENTRYPOINT ["influxd", "--config", "/etc/influxdb.toml"]
|
||||
20
Godeps/_workspace/src/github.com/influxdb/influxdb/LICENSE
generated
vendored
20
Godeps/_workspace/src/github.com/influxdb/influxdb/LICENSE
generated
vendored
@@ -1,20 +0,0 @@
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) 2013-2015 Errplane Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
||||
the Software, and to permit persons to whom the Software is furnished to do so,
|
||||
subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
180
Godeps/_workspace/src/github.com/influxdb/influxdb/QUERIES.md
generated
vendored
180
Godeps/_workspace/src/github.com/influxdb/influxdb/QUERIES.md
generated
vendored
@@ -1,180 +0,0 @@
|
||||
The top level name is called a measurement. These names can contain any characters. Then there are field names, field values, tag keys and tag values, which can also contain any characters. However, if the measurement, field, or tag contains any character other than [A-Z,a-z,0-9,_], or if it starts with a digit, it must be double-quoted. Therefore anywhere a measurement name, field name, field value, tag name, or tag value appears it should be wrapped in double quotes.
|
||||
|
||||
# Databases & retention policies
|
||||
|
||||
```sql
|
||||
-- create a database
|
||||
CREATE DATABASE <name>
|
||||
|
||||
-- create a retention policy
|
||||
CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
|
||||
|
||||
-- alter retention policy
|
||||
ALTER RETENTION POLICY <rp-name> ON <db-name> (DURATION <duration> | REPLICATION <n> | DEFAULT)+
|
||||
|
||||
-- drop a database
|
||||
DROP DATABASE <name>
|
||||
|
||||
-- drop a retention policy
|
||||
DROP RETENTION POLICY <rp-name> ON <db-name>
|
||||
```
|
||||
where `<duration>` is either `INF` for infinite retention, or an integer followed by the desired unit of time: u,ms,s,m,h,d,w for microseconds, milliseconds, seconds, minutes, hours, days, or weeks, respectively. `<replication>` must be an integer.
|
||||
|
||||
If present, `DEFAULT` sets the retention policy as the default retention policy for writes and reads.
|
||||
|
||||
# Users and permissions
|
||||
|
||||
```sql
|
||||
-- create user
|
||||
CREATE USER <name> WITH PASSWORD '<password>'
|
||||
|
||||
-- grant privilege on a database
|
||||
GRANT <privilege> ON <db> TO <user>
|
||||
|
||||
-- grant cluster admin privileges
|
||||
GRANT ALL [PRIVILEGES] TO <user>
|
||||
|
||||
-- revoke privilege
|
||||
REVOKE <privilege> ON <db> FROM <user>
|
||||
|
||||
-- revoke all privileges for a DB
|
||||
REVOKE ALL [PRIVILEGES] ON <db> FROM <user>
|
||||
|
||||
-- revoke all privileges including cluster admin
|
||||
REVOKE ALL [PRIVILEGES] FROM <user>
|
||||
|
||||
-- combine db creation with privilege assignment (user must already exist)
|
||||
CREATE DATABASE <name> GRANT <privilege> TO <user>
|
||||
CREATE DATABASE <name> REVOKE <privilege> FROM <user>
|
||||
|
||||
-- delete a user
|
||||
DROP USER <name>
|
||||
|
||||
|
||||
```
|
||||
where `<privilege> := READ | WRITE | All `.
|
||||
|
||||
Authentication must be enabled in the influxdb.conf file for user permissions to be in effect.
|
||||
|
||||
By default, newly created users have no privileges to any databases.
|
||||
|
||||
Cluster administration privileges automatically grant full read and write permissions to all databases, regardless of subsequent database-specific privilege revocation statements.
|
||||
|
||||
# Select
|
||||
|
||||
```sql
|
||||
SELECT mean(value) from cpu WHERE host = 'serverA' AND time > now() - 4h GROUP BY time(5m)
|
||||
|
||||
SELECT mean(value) from cpu WHERE time > now() - 4h GROUP BY time(5m), region
|
||||
```
|
||||
|
||||
## Group By
|
||||
|
||||
# Delete
|
||||
|
||||
# Series
|
||||
|
||||
## Destroy
|
||||
|
||||
```sql
|
||||
DROP MEASUREMENT <name>
|
||||
DROP MEASUREMENT cpu WHERE region = 'uswest'
|
||||
```
|
||||
|
||||
## Show
|
||||
|
||||
Show series queries are for pulling out individual series from measurement names and tag data. They're useful for discovery.
|
||||
|
||||
```sql
|
||||
-- show all databases
|
||||
SHOW DATABASES
|
||||
|
||||
-- show measurement names
|
||||
SHOW MEASUREMENTS
|
||||
SHOW MEASUREMENTS LIMIT 15
|
||||
SHOW MEASUREMENTS LIMIT 10 OFFSET 40
|
||||
SHOW MEASUREMENTS WHERE service = 'redis'
|
||||
-- LIMIT and OFFSET can be applied to any of the SHOW type queries
|
||||
|
||||
-- show all series across all measurements/tagsets
|
||||
SHOW SERIES
|
||||
|
||||
-- get a show of all series for any measurements where tag key region = tak value 'uswest'
|
||||
SHOW SERIES WHERE region = 'uswest'
|
||||
|
||||
SHOW SERIES FROM cpu_load WHERE region = 'uswest' LIMIT 10
|
||||
|
||||
-- returns the 100 - 109 rows in the result. In the case of SHOW SERIES, which returns
|
||||
-- series split into measurements. Each series counts as a row. So you could see only a
|
||||
-- single measurement returned, but 10 series within it.
|
||||
SHOW SERIES FROM cpu_load WHERE region = 'uswest' LIMIT 10 OFFSET 100
|
||||
|
||||
-- show all retention policies on a database
|
||||
SHOW RETENTION POLICIES ON mydb
|
||||
|
||||
-- get a show of all tag keys across all measurements
|
||||
SHOW TAG KEYS
|
||||
|
||||
-- show all the tag keys for a given measurement
|
||||
SHOW TAG KEYS FROM cpu
|
||||
SHOW TAG KEYS FROM temperature, wind_speed
|
||||
|
||||
-- show all the tag values. note that a single WHERE TAG KEY = '...' clause is required
|
||||
SHOW TAG VALUES WITH TAG KEY = 'region'
|
||||
SHOW TAG VALUES FROM cpu WHERE region = 'uswest' WITH TAG KEY = 'host'
|
||||
|
||||
-- and you can do stuff against fields
|
||||
SHOW FIELD KEYS FROM cpu
|
||||
|
||||
-- but you can't do this
|
||||
SHOW FIELD VALUES
|
||||
-- we don't index field values, so this query should be invalid.
|
||||
|
||||
-- show all users
|
||||
SHOW USERS
|
||||
```
|
||||
|
||||
Note that `FROM` and `WHERE` are optional clauses in most of the show series queries.
|
||||
|
||||
And the show series output looks like this:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "cpu",
|
||||
"columns": ["id", "region", "host"],
|
||||
"values": [
|
||||
1, "uswest", "servera",
|
||||
2, "uswest", "serverb"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "reponse_time",
|
||||
"columns": ["id", "application", "host"],
|
||||
"values": [
|
||||
3, "myRailsApp", "servera"
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
# Continuous Queries
|
||||
|
||||
Continous queries are going to be inspired by MySQL `TRIGGER` syntax:
|
||||
|
||||
http://dev.mysql.com/doc/refman/5.0/en/trigger-syntax.html
|
||||
|
||||
Instead of having automatically-assigned ids, named continuous queries allows for some level of duplication prevention,
|
||||
particularly in the case where creation is scripted.
|
||||
|
||||
## Create
|
||||
|
||||
CREATE CONTINUOUS QUERY <name> AS SELECT ... FROM ...
|
||||
|
||||
## Destroy
|
||||
|
||||
DROP CONTINUOUS QUERY <name>
|
||||
|
||||
## List
|
||||
|
||||
SHOW CONTINUOUS QUERIES
|
||||
71
Godeps/_workspace/src/github.com/influxdb/influxdb/README.md
generated
vendored
71
Godeps/_workspace/src/github.com/influxdb/influxdb/README.md
generated
vendored
@@ -1,71 +0,0 @@
|
||||
# InfluxDB [](https://circleci.com/gh/influxdb/influxdb/tree/master)
|
||||
|
||||
## An Open-Source, Distributed, Time Series Database
|
||||
|
||||
> InfluxDB v0.9.0 is now out. Going forward, the 0.9.x series of releases will not make breaking API changes or breaking changes to the underlying data storage. However, 0.9.0 clustering should be considered an alpha release.
|
||||
|
||||
InfluxDB is an open source **distributed time series database** with
|
||||
**no external dependencies**. It's useful for recording metrics,
|
||||
events, and performing analytics.
|
||||
|
||||
## Features
|
||||
|
||||
* Built-in [HTTP API](http://influxdb.com/docs/v0.9/concepts/reading_and_writing_data.html) so you don't have to write any server side code to get up and running.
|
||||
* Data can be tagged, allowing very flexible querying.
|
||||
* SQL-like query language.
|
||||
* Clustering is supported out of the box, so that you can scale horizontally to handle your data.
|
||||
* Simple to install and manage, and fast to get data in and out.
|
||||
* It aims to answer queries in real-time. That means every data point is
|
||||
indexed as it comes in and is immediately available in queries that
|
||||
should return in < 100ms.
|
||||
|
||||
## Getting Started
|
||||
*The following directions apply only to the 0.9.0 release or building from the source on master.*
|
||||
|
||||
### Building
|
||||
|
||||
You don't need to build the project to use it - you can use any of our
|
||||
[pre-built packages](http://influxdb.com/download/index.html) to install InfluxDB. That's
|
||||
the recommended way to get it running. However, if you want to contribute to the core of InfluxDB, you'll need to build.
|
||||
For those adventurous enough, you can
|
||||
[follow along on our docs](http://github.com/influxdb/influxdb/blob/master/CONTRIBUTING.md).
|
||||
|
||||
### Starting InfluxDB
|
||||
* `service influxdb start` if you have installed InfluxDB using an official Debian or RPM package.
|
||||
* `$GOPATH/bin/influxd` if you have built InfluxDB from source.
|
||||
|
||||
### Creating your first database
|
||||
|
||||
```
|
||||
curl -G 'http://localhost:8086/query' --data-urlencode "q=CREATE DATABASE mydb"
|
||||
```
|
||||
|
||||
### Insert some data
|
||||
```
|
||||
curl -XPOST 'http://localhost:8086/write?db=mydb' \
|
||||
-d 'cpu,host=server01,region=uswest load=42 1434055562000000000'
|
||||
|
||||
curl -XPOST 'http://localhost:8086/write?db=mydb' \
|
||||
-d 'cpu,host=server02,region=uswest load=78 1434055562000000000'
|
||||
|
||||
curl -XPOST 'http://localhost:8086/write?db=mydb' \
|
||||
-d 'cpu,host=server03,region=useast load=15.4 1434055562000000000'
|
||||
```
|
||||
|
||||
### Query for the data
|
||||
```JSON
|
||||
curl -G http://localhost:8086/query?pretty=true --data-urlencode "db=mydb" \
|
||||
--data-urlencode "q=SELECT * FROM cpu WHERE host='server01' AND time < now - 1d"
|
||||
```
|
||||
|
||||
### Analyze the data
|
||||
```JSON
|
||||
curl -G http://localhost:8086/query?pretty=true --data-urlencode "db=mydb" \
|
||||
--data-urlencode "q=SELECT mean(load) FROM cpu WHERE region='uswest'"
|
||||
```
|
||||
|
||||
## Helpful Links
|
||||
|
||||
* Understand the [design goals and motivations of the project](http://influxdb.com/docs/v0.9/introduction/overview.html).
|
||||
* Follow the [getting started guide](http://influxdb.com/docs/v0.9/introduction/getting_started.html) to find out how to install InfluxDB, start writing more data, and issue more queries - in just a few minutes.
|
||||
* See the [HTTP API documentation to start writing a library for your favorite language](http://influxdb.com/docs/v0.9/concepts/reading_and_writing_data.html).
|
||||
78
Godeps/_workspace/src/github.com/influxdb/influxdb/balancer.go
generated
vendored
78
Godeps/_workspace/src/github.com/influxdb/influxdb/balancer.go
generated
vendored
@@ -1,78 +0,0 @@
|
||||
package influxdb
|
||||
|
||||
import (
|
||||
"math/rand"
|
||||
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
)
|
||||
|
||||
// Balancer represents a load-balancing algorithm for a set of nodes
|
||||
type Balancer interface {
|
||||
// Next returns the next Node according to the balancing method
|
||||
// or nil if there are no nodes available
|
||||
Next() *meta.NodeInfo
|
||||
}
|
||||
|
||||
type nodeBalancer struct {
|
||||
nodes []meta.NodeInfo // data nodes to balance between
|
||||
p int // current node index
|
||||
}
|
||||
|
||||
// NewNodeBalancer create a shuffled, round-robin balancer so that
|
||||
// multiple instances will return nodes in randomized order and each
|
||||
// each returned node will be repeated in a cycle
|
||||
func NewNodeBalancer(nodes []meta.NodeInfo) Balancer {
|
||||
// make a copy of the node slice so we can randomize it
|
||||
// without affecting the original instance as well as ensure
|
||||
// that each Balancer returns nodes in a different order
|
||||
b := &nodeBalancer{}
|
||||
|
||||
b.nodes = make([]meta.NodeInfo, len(nodes))
|
||||
copy(b.nodes, nodes)
|
||||
|
||||
b.shuffle()
|
||||
return b
|
||||
}
|
||||
|
||||
// shuffle randomizes the ordering the balancers available nodes
|
||||
func (b *nodeBalancer) shuffle() {
|
||||
for i := range b.nodes {
|
||||
j := rand.Intn(i + 1)
|
||||
b.nodes[i], b.nodes[j] = b.nodes[j], b.nodes[i]
|
||||
}
|
||||
}
|
||||
|
||||
// online returns a slice of the nodes that are online
|
||||
func (b *nodeBalancer) online() []meta.NodeInfo {
|
||||
return b.nodes
|
||||
// now := time.Now().UTC()
|
||||
// up := []meta.NodeInfo{}
|
||||
// for _, n := range b.nodes {
|
||||
// if n.OfflineUntil.After(now) {
|
||||
// continue
|
||||
// }
|
||||
// up = append(up, n)
|
||||
// }
|
||||
// return up
|
||||
}
|
||||
|
||||
// Next returns the next available nodes
|
||||
func (b *nodeBalancer) Next() *meta.NodeInfo {
|
||||
// only use online nodes
|
||||
up := b.online()
|
||||
|
||||
// no nodes online
|
||||
if len(up) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// rollover back to the beginning
|
||||
if b.p >= len(up) {
|
||||
b.p = 0
|
||||
}
|
||||
|
||||
d := &up[b.p]
|
||||
b.p += 1
|
||||
|
||||
return d
|
||||
}
|
||||
115
Godeps/_workspace/src/github.com/influxdb/influxdb/balancer_test.go
generated
vendored
115
Godeps/_workspace/src/github.com/influxdb/influxdb/balancer_test.go
generated
vendored
@@ -1,115 +0,0 @@
|
||||
package influxdb_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/influxdb/influxdb"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
)
|
||||
|
||||
func NewNodes() []meta.NodeInfo {
|
||||
var nodes []meta.NodeInfo
|
||||
for i := 1; i <= 2; i++ {
|
||||
nodes = append(nodes, meta.NodeInfo{
|
||||
ID: uint64(i),
|
||||
Host: fmt.Sprintf("localhost:999%d", i),
|
||||
})
|
||||
}
|
||||
return nodes
|
||||
}
|
||||
|
||||
func TestBalancerEmptyNodes(t *testing.T) {
|
||||
b := influxdb.NewNodeBalancer([]meta.NodeInfo{})
|
||||
got := b.Next()
|
||||
if got != nil {
|
||||
t.Errorf("expected nil, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBalancerUp(t *testing.T) {
|
||||
nodes := NewNodes()
|
||||
b := influxdb.NewNodeBalancer(nodes)
|
||||
|
||||
// First node in randomized round-robin order
|
||||
first := b.Next()
|
||||
if first == nil {
|
||||
t.Errorf("expected datanode, got %v", first)
|
||||
}
|
||||
|
||||
// Second node in randomized round-robin order
|
||||
second := b.Next()
|
||||
if second == nil {
|
||||
t.Errorf("expected datanode, got %v", second)
|
||||
}
|
||||
|
||||
// Should never get the same node in order twice
|
||||
if first.ID == second.ID {
|
||||
t.Errorf("expected first != second. got %v = %v", first.ID, second.ID)
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
func TestBalancerDown(t *testing.T) {
|
||||
nodes := NewNodes()
|
||||
b := influxdb.NewNodeBalancer(nodes)
|
||||
|
||||
nodes[0].Down()
|
||||
|
||||
// First node in randomized round-robin order
|
||||
first := b.Next()
|
||||
if first == nil {
|
||||
t.Errorf("expected datanode, got %v", first)
|
||||
}
|
||||
|
||||
// Second node should rollover to the first up node
|
||||
second := b.Next()
|
||||
if second == nil {
|
||||
t.Errorf("expected datanode, got %v", second)
|
||||
}
|
||||
|
||||
// Health node should be returned each time
|
||||
if first.ID != 2 && first.ID != second.ID {
|
||||
t.Errorf("expected first != second. got %v = %v", first.ID, second.ID)
|
||||
}
|
||||
}
|
||||
*/
|
||||
|
||||
/*
|
||||
func TestBalancerBackUp(t *testing.T) {
|
||||
nodes := newDataNodes()
|
||||
b := influxdb.NewNodeBalancer(nodes)
|
||||
|
||||
nodes[0].Down()
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
got := b.Next()
|
||||
if got == nil {
|
||||
t.Errorf("expected datanode, got %v", got)
|
||||
}
|
||||
|
||||
if exp := uint64(2); got.ID != exp {
|
||||
t.Errorf("wrong node id: exp %v, got %v", exp, got.ID)
|
||||
}
|
||||
}
|
||||
|
||||
nodes[0].Up()
|
||||
|
||||
// First node in randomized round-robin order
|
||||
first := b.Next()
|
||||
if first == nil {
|
||||
t.Errorf("expected datanode, got %v", first)
|
||||
}
|
||||
|
||||
// Second node should rollover to the first up node
|
||||
second := b.Next()
|
||||
if second == nil {
|
||||
t.Errorf("expected datanode, got %v", second)
|
||||
}
|
||||
|
||||
// Should get both nodes returned
|
||||
if first.ID == second.ID {
|
||||
t.Errorf("expected first != second. got %v = %v", first.ID, second.ID)
|
||||
}
|
||||
}
|
||||
*/
|
||||
7
Godeps/_workspace/src/github.com/influxdb/influxdb/build-docker.sh
generated
vendored
7
Godeps/_workspace/src/github.com/influxdb/influxdb/build-docker.sh
generated
vendored
@@ -1,7 +0,0 @@
|
||||
#!/bin/sh -x -e
|
||||
|
||||
GO_VER=${GO_VER:-1.5}
|
||||
|
||||
docker run -it -v "$GOPATH":/gopath -v "$(pwd)":/app -e "GOPATH=/gopath" -w /app golang:$GO_VER sh -c 'CGO_ENABLED=0 go build -a --installsuffix cgo --ldflags="-s" -o influxd ./cmd/influxd'
|
||||
|
||||
docker build -t influxdb .
|
||||
63
Godeps/_workspace/src/github.com/influxdb/influxdb/circle-test.sh
generated
vendored
63
Godeps/_workspace/src/github.com/influxdb/influxdb/circle-test.sh
generated
vendored
@@ -1,63 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# This is the InfluxDB CircleCI test script. Using this script allows total control
|
||||
# the environment in which the build and test is run, and matches the official
|
||||
# build process for InfluxDB.
|
||||
|
||||
BUILD_DIR=$HOME/influxdb-build
|
||||
GO_VERSION=go1.4.2
|
||||
PARALLELISM="-parallel 256"
|
||||
TIMEOUT="-timeout 480s"
|
||||
|
||||
# Executes the given statement, and exits if the command returns a non-zero code.
|
||||
function exit_if_fail {
|
||||
command=$@
|
||||
echo "Executing '$command'"
|
||||
$command
|
||||
rc=$?
|
||||
if [ $rc -ne 0 ]; then
|
||||
echo "'$command' returned $rc."
|
||||
exit $rc
|
||||
fi
|
||||
}
|
||||
|
||||
source $HOME/.gvm/scripts/gvm
|
||||
exit_if_fail gvm use $GO_VERSION
|
||||
|
||||
# Set up the build directory, and then GOPATH.
|
||||
exit_if_fail mkdir $BUILD_DIR
|
||||
export GOPATH=$BUILD_DIR
|
||||
exit_if_fail mkdir -p $GOPATH/src/github.com/influxdb
|
||||
|
||||
# Dump some test config to the log.
|
||||
echo "Test configuration"
|
||||
echo "========================================"
|
||||
echo "\$HOME: $HOME"
|
||||
echo "\$GOPATH: $GOPATH"
|
||||
echo "\$CIRCLE_BRANCH: $CIRCLE_BRANCH"
|
||||
|
||||
# Move the checked-out source to a better location.
|
||||
exit_if_fail mv $HOME/influxdb $GOPATH/src/github.com/influxdb
|
||||
exit_if_fail cd $GOPATH/src/github.com/influxdb/influxdb
|
||||
exit_if_fail git branch --set-upstream-to=origin/$CIRCLE_BRANCH $CIRCLE_BRANCH
|
||||
|
||||
# Install the code.
|
||||
exit_if_fail cd $GOPATH/src/github.com/influxdb/influxdb
|
||||
exit_if_fail go get -t -d -v ./...
|
||||
exit_if_fail git checkout $CIRCLE_BRANCH # 'go get' switches to master. Who knew? Switch back.
|
||||
exit_if_fail go build -v ./...
|
||||
|
||||
# Run the tests.
|
||||
exit_if_fail go tool vet --composites=false .
|
||||
case $CIRCLE_NODE_INDEX in
|
||||
0)
|
||||
go test $PARALLELISM $TIMEOUT -v ./... 2>&1 | tee $CIRCLE_ARTIFACTS/test_logs.txt
|
||||
rc=${PIPESTATUS[0]}
|
||||
;;
|
||||
1)
|
||||
GORACE="halt_on_error=1" go test $PARALLELISM $TIMEOUT -v -race ./... 2>&1 | tee $CIRCLE_ARTIFACTS/test_logs_race.txt
|
||||
rc=${PIPESTATUS[0]}
|
||||
;;
|
||||
esac
|
||||
|
||||
exit $rc
|
||||
12
Godeps/_workspace/src/github.com/influxdb/influxdb/circle.yml
generated
vendored
12
Godeps/_workspace/src/github.com/influxdb/influxdb/circle.yml
generated
vendored
@@ -1,12 +0,0 @@
|
||||
machine:
|
||||
pre:
|
||||
- bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
|
||||
- source $HOME/.gvm/scripts/gvm; gvm install go1.4.2 --binary
|
||||
|
||||
dependencies:
|
||||
override:
|
||||
- echo "Dummy override, so no Circle dependencies execute"
|
||||
test:
|
||||
override:
|
||||
- bash circle-test.sh:
|
||||
parallel: true
|
||||
57
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/client_pool.go
generated
vendored
57
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/client_pool.go
generated
vendored
@@ -1,57 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"net"
|
||||
"sync"
|
||||
|
||||
"gopkg.in/fatih/pool.v2"
|
||||
)
|
||||
|
||||
type clientPool struct {
|
||||
mu sync.RWMutex
|
||||
pool map[uint64]pool.Pool
|
||||
}
|
||||
|
||||
func newClientPool() *clientPool {
|
||||
return &clientPool{
|
||||
pool: make(map[uint64]pool.Pool),
|
||||
}
|
||||
}
|
||||
|
||||
func (c *clientPool) setPool(nodeID uint64, p pool.Pool) {
|
||||
c.mu.Lock()
|
||||
c.pool[nodeID] = p
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
func (c *clientPool) getPool(nodeID uint64) (pool.Pool, bool) {
|
||||
c.mu.RLock()
|
||||
p, ok := c.pool[nodeID]
|
||||
c.mu.RUnlock()
|
||||
return p, ok
|
||||
}
|
||||
|
||||
func (c *clientPool) size() int {
|
||||
c.mu.RLock()
|
||||
var size int
|
||||
for _, p := range c.pool {
|
||||
size += p.Len()
|
||||
}
|
||||
c.mu.RUnlock()
|
||||
return size
|
||||
}
|
||||
|
||||
func (c *clientPool) conn(nodeID uint64) (net.Conn, error) {
|
||||
c.mu.RLock()
|
||||
conn, err := c.pool[nodeID].Get()
|
||||
c.mu.RUnlock()
|
||||
return conn, err
|
||||
}
|
||||
|
||||
func (c *clientPool) close() {
|
||||
c.mu.Lock()
|
||||
for _, p := range c.pool {
|
||||
p.Close()
|
||||
}
|
||||
c.mu.Unlock()
|
||||
}
|
||||
35
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/config.go
generated
vendored
35
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/config.go
generated
vendored
@@ -1,35 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/toml"
|
||||
)
|
||||
|
||||
const (
|
||||
// DefaultWriteTimeout is the default timeout for a complete write to succeed.
|
||||
DefaultWriteTimeout = 5 * time.Second
|
||||
|
||||
// DefaultShardWriterTimeout is the default timeout set on shard writers.
|
||||
DefaultShardWriterTimeout = 5 * time.Second
|
||||
|
||||
// DefaultShardMapperTimeout is the default timeout set on shard mappers.
|
||||
DefaultShardMapperTimeout = 5 * time.Second
|
||||
)
|
||||
|
||||
// Config represents the configuration for the clustering service.
|
||||
type Config struct {
|
||||
ForceRemoteShardMapping bool `toml:"force-remote-mapping"`
|
||||
WriteTimeout toml.Duration `toml:"write-timeout"`
|
||||
ShardWriterTimeout toml.Duration `toml:"shard-writer-timeout"`
|
||||
ShardMapperTimeout toml.Duration `toml:"shard-mapper-timeout"`
|
||||
}
|
||||
|
||||
// NewConfig returns an instance of Config with defaults.
|
||||
func NewConfig() Config {
|
||||
return Config{
|
||||
WriteTimeout: toml.Duration(DefaultWriteTimeout),
|
||||
ShardWriterTimeout: toml.Duration(DefaultShardWriterTimeout),
|
||||
ShardMapperTimeout: toml.Duration(DefaultShardMapperTimeout),
|
||||
}
|
||||
}
|
||||
27
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/config_test.go
generated
vendored
27
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/config_test.go
generated
vendored
@@ -1,27 +0,0 @@
|
||||
package cluster_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
"github.com/influxdb/influxdb/cluster"
|
||||
)
|
||||
|
||||
func TestConfig_Parse(t *testing.T) {
|
||||
// Parse configuration.
|
||||
var c cluster.Config
|
||||
if _, err := toml.Decode(`
|
||||
shard-writer-timeout = "10s"
|
||||
write-timeout = "20s"
|
||||
`, &c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Validate configuration.
|
||||
if time.Duration(c.ShardWriterTimeout) != 10*time.Second {
|
||||
t.Fatalf("unexpected shard-writer timeout: %s", c.ShardWriterTimeout)
|
||||
} else if time.Duration(c.WriteTimeout) != 20*time.Second {
|
||||
t.Fatalf("unexpected write timeout s: %s", c.WriteTimeout)
|
||||
}
|
||||
}
|
||||
286
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/internal/data.pb.go
generated
vendored
286
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/internal/data.pb.go
generated
vendored
@@ -1,286 +0,0 @@
|
||||
// Code generated by protoc-gen-gogo.
|
||||
// source: internal/data.proto
|
||||
// DO NOT EDIT!
|
||||
|
||||
/*
|
||||
Package internal is a generated protocol buffer package.
|
||||
|
||||
It is generated from these files:
|
||||
internal/data.proto
|
||||
|
||||
It has these top-level messages:
|
||||
WriteShardRequest
|
||||
Field
|
||||
Tag
|
||||
Point
|
||||
WriteShardResponse
|
||||
MapShardRequest
|
||||
MapShardResponse
|
||||
*/
|
||||
package internal
|
||||
|
||||
import proto "github.com/gogo/protobuf/proto"
|
||||
import math "math"
|
||||
|
||||
// Reference imports to suppress errors if they are not otherwise used.
|
||||
var _ = proto.Marshal
|
||||
var _ = math.Inf
|
||||
|
||||
type WriteShardRequest struct {
|
||||
ShardID *uint64 `protobuf:"varint,1,req" json:"ShardID,omitempty"`
|
||||
Points []*Point `protobuf:"bytes,2,rep" json:"Points,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *WriteShardRequest) Reset() { *m = WriteShardRequest{} }
|
||||
func (m *WriteShardRequest) String() string { return proto.CompactTextString(m) }
|
||||
func (*WriteShardRequest) ProtoMessage() {}
|
||||
|
||||
func (m *WriteShardRequest) GetShardID() uint64 {
|
||||
if m != nil && m.ShardID != nil {
|
||||
return *m.ShardID
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *WriteShardRequest) GetPoints() []*Point {
|
||||
if m != nil {
|
||||
return m.Points
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type Field struct {
|
||||
Name *string `protobuf:"bytes,1,req" json:"Name,omitempty"`
|
||||
Int32 *int32 `protobuf:"varint,2,opt" json:"Int32,omitempty"`
|
||||
Int64 *int64 `protobuf:"varint,3,opt" json:"Int64,omitempty"`
|
||||
Float64 *float64 `protobuf:"fixed64,4,opt" json:"Float64,omitempty"`
|
||||
Bool *bool `protobuf:"varint,5,opt" json:"Bool,omitempty"`
|
||||
String_ *string `protobuf:"bytes,6,opt" json:"String,omitempty"`
|
||||
Bytes []byte `protobuf:"bytes,7,opt" json:"Bytes,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *Field) Reset() { *m = Field{} }
|
||||
func (m *Field) String() string { return proto.CompactTextString(m) }
|
||||
func (*Field) ProtoMessage() {}
|
||||
|
||||
func (m *Field) GetName() string {
|
||||
if m != nil && m.Name != nil {
|
||||
return *m.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (m *Field) GetInt32() int32 {
|
||||
if m != nil && m.Int32 != nil {
|
||||
return *m.Int32
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *Field) GetInt64() int64 {
|
||||
if m != nil && m.Int64 != nil {
|
||||
return *m.Int64
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *Field) GetFloat64() float64 {
|
||||
if m != nil && m.Float64 != nil {
|
||||
return *m.Float64
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *Field) GetBool() bool {
|
||||
if m != nil && m.Bool != nil {
|
||||
return *m.Bool
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (m *Field) GetString_() string {
|
||||
if m != nil && m.String_ != nil {
|
||||
return *m.String_
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (m *Field) GetBytes() []byte {
|
||||
if m != nil {
|
||||
return m.Bytes
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type Tag struct {
|
||||
Key *string `protobuf:"bytes,1,req" json:"Key,omitempty"`
|
||||
Value *string `protobuf:"bytes,2,req" json:"Value,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *Tag) Reset() { *m = Tag{} }
|
||||
func (m *Tag) String() string { return proto.CompactTextString(m) }
|
||||
func (*Tag) ProtoMessage() {}
|
||||
|
||||
func (m *Tag) GetKey() string {
|
||||
if m != nil && m.Key != nil {
|
||||
return *m.Key
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (m *Tag) GetValue() string {
|
||||
if m != nil && m.Value != nil {
|
||||
return *m.Value
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
type Point struct {
|
||||
Name *string `protobuf:"bytes,1,req" json:"Name,omitempty"`
|
||||
Time *int64 `protobuf:"varint,2,req" json:"Time,omitempty"`
|
||||
Fields []*Field `protobuf:"bytes,3,rep" json:"Fields,omitempty"`
|
||||
Tags []*Tag `protobuf:"bytes,4,rep" json:"Tags,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *Point) Reset() { *m = Point{} }
|
||||
func (m *Point) String() string { return proto.CompactTextString(m) }
|
||||
func (*Point) ProtoMessage() {}
|
||||
|
||||
func (m *Point) GetName() string {
|
||||
if m != nil && m.Name != nil {
|
||||
return *m.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (m *Point) GetTime() int64 {
|
||||
if m != nil && m.Time != nil {
|
||||
return *m.Time
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *Point) GetFields() []*Field {
|
||||
if m != nil {
|
||||
return m.Fields
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Point) GetTags() []*Tag {
|
||||
if m != nil {
|
||||
return m.Tags
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type WriteShardResponse struct {
|
||||
Code *int32 `protobuf:"varint,1,req" json:"Code,omitempty"`
|
||||
Message *string `protobuf:"bytes,2,opt" json:"Message,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *WriteShardResponse) Reset() { *m = WriteShardResponse{} }
|
||||
func (m *WriteShardResponse) String() string { return proto.CompactTextString(m) }
|
||||
func (*WriteShardResponse) ProtoMessage() {}
|
||||
|
||||
func (m *WriteShardResponse) GetCode() int32 {
|
||||
if m != nil && m.Code != nil {
|
||||
return *m.Code
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *WriteShardResponse) GetMessage() string {
|
||||
if m != nil && m.Message != nil {
|
||||
return *m.Message
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
type MapShardRequest struct {
|
||||
ShardID *uint64 `protobuf:"varint,1,req" json:"ShardID,omitempty"`
|
||||
Query *string `protobuf:"bytes,2,req" json:"Query,omitempty"`
|
||||
ChunkSize *int32 `protobuf:"varint,3,req" json:"ChunkSize,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *MapShardRequest) Reset() { *m = MapShardRequest{} }
|
||||
func (m *MapShardRequest) String() string { return proto.CompactTextString(m) }
|
||||
func (*MapShardRequest) ProtoMessage() {}
|
||||
|
||||
func (m *MapShardRequest) GetShardID() uint64 {
|
||||
if m != nil && m.ShardID != nil {
|
||||
return *m.ShardID
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *MapShardRequest) GetQuery() string {
|
||||
if m != nil && m.Query != nil {
|
||||
return *m.Query
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (m *MapShardRequest) GetChunkSize() int32 {
|
||||
if m != nil && m.ChunkSize != nil {
|
||||
return *m.ChunkSize
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
type MapShardResponse struct {
|
||||
Code *int32 `protobuf:"varint,1,req" json:"Code,omitempty"`
|
||||
Message *string `protobuf:"bytes,2,opt" json:"Message,omitempty"`
|
||||
Data []byte `protobuf:"bytes,3,opt" json:"Data,omitempty"`
|
||||
TagSets []string `protobuf:"bytes,4,rep" json:"TagSets,omitempty"`
|
||||
Fields []string `protobuf:"bytes,5,rep" json:"Fields,omitempty"`
|
||||
XXX_unrecognized []byte `json:"-"`
|
||||
}
|
||||
|
||||
func (m *MapShardResponse) Reset() { *m = MapShardResponse{} }
|
||||
func (m *MapShardResponse) String() string { return proto.CompactTextString(m) }
|
||||
func (*MapShardResponse) ProtoMessage() {}
|
||||
|
||||
func (m *MapShardResponse) GetCode() int32 {
|
||||
if m != nil && m.Code != nil {
|
||||
return *m.Code
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (m *MapShardResponse) GetMessage() string {
|
||||
if m != nil && m.Message != nil {
|
||||
return *m.Message
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (m *MapShardResponse) GetData() []byte {
|
||||
if m != nil {
|
||||
return m.Data
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *MapShardResponse) GetTagSets() []string {
|
||||
if m != nil {
|
||||
return m.TagSets
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *MapShardResponse) GetFields() []string {
|
||||
if m != nil {
|
||||
return m.Fields
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func init() {
|
||||
}
|
||||
49
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/internal/data.proto
generated
vendored
49
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/internal/data.proto
generated
vendored
@@ -1,49 +0,0 @@
|
||||
package internal;
|
||||
|
||||
message WriteShardRequest {
|
||||
required uint64 ShardID = 1;
|
||||
repeated Point Points = 2;
|
||||
}
|
||||
|
||||
message Field {
|
||||
required string Name = 1;
|
||||
oneof Value {
|
||||
int32 Int32 = 2;
|
||||
int64 Int64 = 3;
|
||||
double Float64 = 4;
|
||||
bool Bool = 5;
|
||||
string String = 6;
|
||||
bytes Bytes = 7;
|
||||
}
|
||||
}
|
||||
|
||||
message Tag {
|
||||
required string Key = 1;
|
||||
required string Value = 2;
|
||||
}
|
||||
|
||||
message Point {
|
||||
required string Name = 1;
|
||||
required int64 Time = 2;
|
||||
repeated Field Fields = 3;
|
||||
repeated Tag Tags = 4;
|
||||
}
|
||||
|
||||
message WriteShardResponse {
|
||||
required int32 Code = 1;
|
||||
optional string Message = 2;
|
||||
}
|
||||
|
||||
message MapShardRequest {
|
||||
required uint64 ShardID = 1;
|
||||
required string Query = 2;
|
||||
required int32 ChunkSize = 3;
|
||||
}
|
||||
|
||||
message MapShardResponse {
|
||||
required int32 Code = 1;
|
||||
optional string Message = 2;
|
||||
optional bytes Data = 3;
|
||||
repeated string TagSets = 4;
|
||||
repeated string Fields = 5;
|
||||
}
|
||||
314
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/points_writer.go
generated
vendored
314
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/points_writer.go
generated
vendored
@@ -1,314 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// ConsistencyLevel represent a required replication criteria before a write can
|
||||
// be returned as successful
|
||||
type ConsistencyLevel int
|
||||
|
||||
const (
|
||||
// ConsistencyLevelAny allows for hinted hand off, potentially no write happened yet
|
||||
ConsistencyLevelAny ConsistencyLevel = iota
|
||||
|
||||
// ConsistencyLevelOne requires at least one data node acknowledged a write
|
||||
ConsistencyLevelOne
|
||||
|
||||
// ConsistencyLevelOne requires a quorum of data nodes to acknowledge a write
|
||||
ConsistencyLevelQuorum
|
||||
|
||||
// ConsistencyLevelAll requires all data nodes to acknowledge a write
|
||||
ConsistencyLevelAll
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrTimeout is returned when a write times out.
|
||||
ErrTimeout = errors.New("timeout")
|
||||
|
||||
// ErrPartialWrite is returned when a write partially succeeds but does
|
||||
// not meet the requested consistency level.
|
||||
ErrPartialWrite = errors.New("partial write")
|
||||
|
||||
// ErrWriteFailed is returned when no writes succeeded.
|
||||
ErrWriteFailed = errors.New("write failed")
|
||||
|
||||
// ErrInvalidConsistencyLevel is returned when parsing the string version
|
||||
// of a consistency level.
|
||||
ErrInvalidConsistencyLevel = errors.New("invalid consistency level")
|
||||
)
|
||||
|
||||
func ParseConsistencyLevel(level string) (ConsistencyLevel, error) {
|
||||
switch strings.ToLower(level) {
|
||||
case "any":
|
||||
return ConsistencyLevelAny, nil
|
||||
case "one":
|
||||
return ConsistencyLevelOne, nil
|
||||
case "quorum":
|
||||
return ConsistencyLevelQuorum, nil
|
||||
case "all":
|
||||
return ConsistencyLevelAll, nil
|
||||
default:
|
||||
return 0, ErrInvalidConsistencyLevel
|
||||
}
|
||||
}
|
||||
|
||||
// PointsWriter handles writes across multiple local and remote data nodes.
|
||||
type PointsWriter struct {
|
||||
mu sync.RWMutex
|
||||
closing chan struct{}
|
||||
WriteTimeout time.Duration
|
||||
Logger *log.Logger
|
||||
|
||||
MetaStore interface {
|
||||
NodeID() uint64
|
||||
Database(name string) (di *meta.DatabaseInfo, err error)
|
||||
RetentionPolicy(database, policy string) (*meta.RetentionPolicyInfo, error)
|
||||
CreateShardGroupIfNotExists(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error)
|
||||
ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo)
|
||||
}
|
||||
|
||||
TSDBStore interface {
|
||||
CreateShard(database, retentionPolicy string, shardID uint64) error
|
||||
WriteToShard(shardID uint64, points []tsdb.Point) error
|
||||
}
|
||||
|
||||
ShardWriter interface {
|
||||
WriteShard(shardID, ownerID uint64, points []tsdb.Point) error
|
||||
}
|
||||
|
||||
HintedHandoff interface {
|
||||
WriteShard(shardID, ownerID uint64, points []tsdb.Point) error
|
||||
}
|
||||
}
|
||||
|
||||
// NewPointsWriter returns a new instance of PointsWriter for a node.
|
||||
func NewPointsWriter() *PointsWriter {
|
||||
return &PointsWriter{
|
||||
closing: make(chan struct{}),
|
||||
WriteTimeout: DefaultWriteTimeout,
|
||||
Logger: log.New(os.Stderr, "[write] ", log.LstdFlags),
|
||||
}
|
||||
}
|
||||
|
||||
// ShardMapping contains a mapping of a shards to a points.
|
||||
type ShardMapping struct {
|
||||
Points map[uint64][]tsdb.Point // The points associated with a shard ID
|
||||
Shards map[uint64]*meta.ShardInfo // The shards that have been mapped, keyed by shard ID
|
||||
}
|
||||
|
||||
// NewShardMapping creates an empty ShardMapping
|
||||
func NewShardMapping() *ShardMapping {
|
||||
return &ShardMapping{
|
||||
Points: map[uint64][]tsdb.Point{},
|
||||
Shards: map[uint64]*meta.ShardInfo{},
|
||||
}
|
||||
}
|
||||
|
||||
// MapPoint maps a point to shard
|
||||
func (s *ShardMapping) MapPoint(shardInfo *meta.ShardInfo, p tsdb.Point) {
|
||||
points, ok := s.Points[shardInfo.ID]
|
||||
if !ok {
|
||||
s.Points[shardInfo.ID] = []tsdb.Point{p}
|
||||
} else {
|
||||
s.Points[shardInfo.ID] = append(points, p)
|
||||
}
|
||||
s.Shards[shardInfo.ID] = shardInfo
|
||||
}
|
||||
|
||||
func (w *PointsWriter) Open() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closing == nil {
|
||||
w.closing = make(chan struct{})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *PointsWriter) Close() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closing != nil {
|
||||
close(w.closing)
|
||||
w.closing = nil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MapShards maps the points contained in wp to a ShardMapping. If a point
|
||||
// maps to a shard group or shard that does not currently exist, it will be
|
||||
// created before returning the mapping.
|
||||
func (w *PointsWriter) MapShards(wp *WritePointsRequest) (*ShardMapping, error) {
|
||||
|
||||
// holds the start time ranges for required shard groups
|
||||
timeRanges := map[time.Time]*meta.ShardGroupInfo{}
|
||||
|
||||
rp, err := w.MetaStore.RetentionPolicy(wp.Database, wp.RetentionPolicy)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for _, p := range wp.Points {
|
||||
timeRanges[p.Time().Truncate(rp.ShardGroupDuration)] = nil
|
||||
}
|
||||
|
||||
// holds all the shard groups and shards that are required for writes
|
||||
for t := range timeRanges {
|
||||
sg, err := w.MetaStore.CreateShardGroupIfNotExists(wp.Database, wp.RetentionPolicy, t)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
timeRanges[t] = sg
|
||||
}
|
||||
|
||||
mapping := NewShardMapping()
|
||||
for _, p := range wp.Points {
|
||||
sg := timeRanges[p.Time().Truncate(rp.ShardGroupDuration)]
|
||||
sh := sg.ShardFor(p.HashID())
|
||||
mapping.MapPoint(&sh, p)
|
||||
}
|
||||
return mapping, nil
|
||||
}
|
||||
|
||||
// WritePoints writes across multiple local and remote data nodes according the consistency level.
|
||||
func (w *PointsWriter) WritePoints(p *WritePointsRequest) error {
|
||||
if p.RetentionPolicy == "" {
|
||||
db, err := w.MetaStore.Database(p.Database)
|
||||
if err != nil {
|
||||
return err
|
||||
} else if db == nil {
|
||||
return influxdb.ErrDatabaseNotFound(p.Database)
|
||||
}
|
||||
p.RetentionPolicy = db.DefaultRetentionPolicy
|
||||
}
|
||||
|
||||
shardMappings, err := w.MapShards(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Write each shard in it's own goroutine and return as soon
|
||||
// as one fails.
|
||||
ch := make(chan error, len(shardMappings.Points))
|
||||
for shardID, points := range shardMappings.Points {
|
||||
go func(shard *meta.ShardInfo, database, retentionPolicy string, points []tsdb.Point) {
|
||||
ch <- w.writeToShard(shard, p.Database, p.RetentionPolicy, p.ConsistencyLevel, points)
|
||||
}(shardMappings.Shards[shardID], p.Database, p.RetentionPolicy, points)
|
||||
}
|
||||
|
||||
for range shardMappings.Points {
|
||||
select {
|
||||
case <-w.closing:
|
||||
return ErrWriteFailed
|
||||
case err := <-ch:
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeToShards writes points to a shard and ensures a write consistency level has been met. If the write
|
||||
// partially succeeds, ErrPartialWrite is returned.
|
||||
func (w *PointsWriter) writeToShard(shard *meta.ShardInfo, database, retentionPolicy string,
|
||||
consistency ConsistencyLevel, points []tsdb.Point) error {
|
||||
// The required number of writes to achieve the requested consistency level
|
||||
required := len(shard.OwnerIDs)
|
||||
switch consistency {
|
||||
case ConsistencyLevelAny, ConsistencyLevelOne:
|
||||
required = 1
|
||||
case ConsistencyLevelQuorum:
|
||||
required = required/2 + 1
|
||||
}
|
||||
|
||||
// response channel for each shard writer go routine
|
||||
ch := make(chan error, len(shard.OwnerIDs))
|
||||
|
||||
for _, nodeID := range shard.OwnerIDs {
|
||||
go func(shardID, nodeID uint64, points []tsdb.Point) {
|
||||
if w.MetaStore.NodeID() == nodeID {
|
||||
err := w.TSDBStore.WriteToShard(shardID, points)
|
||||
// If we've written to shard that should exist on the current node, but the store has
|
||||
// not actually created this shard, tell it to create it and retry the write
|
||||
if err == tsdb.ErrShardNotFound {
|
||||
err = w.TSDBStore.CreateShard(database, retentionPolicy, shardID)
|
||||
if err != nil {
|
||||
ch <- err
|
||||
return
|
||||
}
|
||||
err = w.TSDBStore.WriteToShard(shardID, points)
|
||||
}
|
||||
ch <- err
|
||||
return
|
||||
}
|
||||
|
||||
err := w.ShardWriter.WriteShard(shardID, nodeID, points)
|
||||
if err != nil && tsdb.IsRetryable(err) {
|
||||
// The remote write failed so queue it via hinted handoff
|
||||
hherr := w.HintedHandoff.WriteShard(shardID, nodeID, points)
|
||||
|
||||
// If the write consistency level is ANY, then a successful hinted handoff can
|
||||
// be considered a successful write so send nil to the response channel
|
||||
// otherwise, let the original error propogate to the response channel
|
||||
if hherr == nil && consistency == ConsistencyLevelAny {
|
||||
ch <- nil
|
||||
return
|
||||
}
|
||||
}
|
||||
ch <- err
|
||||
|
||||
}(shard.ID, nodeID, points)
|
||||
}
|
||||
|
||||
var wrote int
|
||||
timeout := time.After(w.WriteTimeout)
|
||||
var writeError error
|
||||
for _, nodeID := range shard.OwnerIDs {
|
||||
select {
|
||||
case <-w.closing:
|
||||
return ErrWriteFailed
|
||||
case <-timeout:
|
||||
// return timeout error to caller
|
||||
return ErrTimeout
|
||||
case err := <-ch:
|
||||
// If the write returned an error, continue to the next response
|
||||
if err != nil {
|
||||
w.Logger.Printf("write failed for shard %d on node %d: %v", shard.ID, nodeID, err)
|
||||
|
||||
// Keep track of the first error we see to return back to the client
|
||||
if writeError == nil {
|
||||
writeError = err
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
wrote += 1
|
||||
}
|
||||
}
|
||||
|
||||
// We wrote the required consistency level
|
||||
if wrote >= required {
|
||||
return nil
|
||||
}
|
||||
|
||||
if wrote > 0 {
|
||||
return ErrPartialWrite
|
||||
}
|
||||
|
||||
if writeError != nil {
|
||||
return fmt.Errorf("write failed: %v", writeError)
|
||||
}
|
||||
|
||||
return ErrWriteFailed
|
||||
}
|
||||
436
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/points_writer_test.go
generated
vendored
436
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/points_writer_test.go
generated
vendored
@@ -1,436 +0,0 @@
|
||||
package cluster_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cluster"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// Ensures the points writer maps a single point to a single shard.
|
||||
func TestPointsWriter_MapShards_One(t *testing.T) {
|
||||
ms := MetaStore{}
|
||||
rp := NewRetentionPolicy("myp", time.Hour, 3)
|
||||
|
||||
ms.NodeIDFn = func() uint64 { return 1 }
|
||||
ms.RetentionPolicyFn = func(db, retentionPolicy string) (*meta.RetentionPolicyInfo, error) {
|
||||
return rp, nil
|
||||
}
|
||||
|
||||
ms.CreateShardGroupIfNotExistsFn = func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
|
||||
return &rp.ShardGroups[0], nil
|
||||
}
|
||||
|
||||
c := cluster.PointsWriter{MetaStore: ms}
|
||||
pr := &cluster.WritePointsRequest{
|
||||
Database: "mydb",
|
||||
RetentionPolicy: "myrp",
|
||||
ConsistencyLevel: cluster.ConsistencyLevelOne,
|
||||
}
|
||||
pr.AddPoint("cpu", 1.0, time.Now(), nil)
|
||||
|
||||
var (
|
||||
shardMappings *cluster.ShardMapping
|
||||
err error
|
||||
)
|
||||
if shardMappings, err = c.MapShards(pr); err != nil {
|
||||
t.Fatalf("unexpected an error: %v", err)
|
||||
}
|
||||
|
||||
if exp := 1; len(shardMappings.Points) != exp {
|
||||
t.Errorf("MapShards() len mismatch. got %v, exp %v", len(shardMappings.Points), exp)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensures the points writer maps a multiple points across shard group boundaries.
|
||||
func TestPointsWriter_MapShards_Multiple(t *testing.T) {
|
||||
ms := MetaStore{}
|
||||
rp := NewRetentionPolicy("myp", time.Hour, 3)
|
||||
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
|
||||
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
|
||||
|
||||
ms.NodeIDFn = func() uint64 { return 1 }
|
||||
ms.RetentionPolicyFn = func(db, retentionPolicy string) (*meta.RetentionPolicyInfo, error) {
|
||||
return rp, nil
|
||||
}
|
||||
|
||||
ms.CreateShardGroupIfNotExistsFn = func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
|
||||
for i, sg := range rp.ShardGroups {
|
||||
if timestamp.Equal(sg.StartTime) || timestamp.After(sg.StartTime) && timestamp.Before(sg.EndTime) {
|
||||
return &rp.ShardGroups[i], nil
|
||||
}
|
||||
}
|
||||
panic("should not get here")
|
||||
}
|
||||
|
||||
c := cluster.PointsWriter{MetaStore: ms}
|
||||
pr := &cluster.WritePointsRequest{
|
||||
Database: "mydb",
|
||||
RetentionPolicy: "myrp",
|
||||
ConsistencyLevel: cluster.ConsistencyLevelOne,
|
||||
}
|
||||
|
||||
// Three points that range over the shardGroup duration (1h) and should map to two
|
||||
// distinct shards
|
||||
pr.AddPoint("cpu", 1.0, time.Unix(0, 0), nil)
|
||||
pr.AddPoint("cpu", 2.0, time.Unix(0, 0).Add(time.Hour), nil)
|
||||
pr.AddPoint("cpu", 3.0, time.Unix(0, 0).Add(time.Hour+time.Second), nil)
|
||||
|
||||
var (
|
||||
shardMappings *cluster.ShardMapping
|
||||
err error
|
||||
)
|
||||
if shardMappings, err = c.MapShards(pr); err != nil {
|
||||
t.Fatalf("unexpected an error: %v", err)
|
||||
}
|
||||
|
||||
if exp := 2; len(shardMappings.Points) != exp {
|
||||
t.Errorf("MapShards() len mismatch. got %v, exp %v", len(shardMappings.Points), exp)
|
||||
}
|
||||
|
||||
for _, points := range shardMappings.Points {
|
||||
// First shard shoud have 1 point w/ first point added
|
||||
if len(points) == 1 && points[0].Time() != pr.Points[0].Time() {
|
||||
t.Fatalf("MapShards() value mismatch. got %v, exp %v", points[0].Time(), pr.Points[0].Time())
|
||||
}
|
||||
|
||||
// Second shard shoud have the last two points added
|
||||
if len(points) == 2 && points[0].Time() != pr.Points[1].Time() {
|
||||
t.Fatalf("MapShards() value mismatch. got %v, exp %v", points[0].Time(), pr.Points[1].Time())
|
||||
}
|
||||
|
||||
if len(points) == 2 && points[1].Time() != pr.Points[2].Time() {
|
||||
t.Fatalf("MapShards() value mismatch. got %v, exp %v", points[1].Time(), pr.Points[2].Time())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPointsWriter_WritePoints(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
database string
|
||||
retentionPolicy string
|
||||
consistency cluster.ConsistencyLevel
|
||||
|
||||
// the responses returned by each shard write call. node ID 1 = pos 0
|
||||
err []error
|
||||
expErr error
|
||||
}{
|
||||
// Consistency one
|
||||
{
|
||||
name: "write one success",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelOne,
|
||||
err: []error{nil, nil, nil},
|
||||
expErr: nil,
|
||||
},
|
||||
{
|
||||
name: "write one error",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelOne,
|
||||
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), fmt.Errorf("a failure")},
|
||||
expErr: fmt.Errorf("write failed: a failure"),
|
||||
},
|
||||
|
||||
// Consistency any
|
||||
{
|
||||
name: "write any success",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelAny,
|
||||
err: []error{fmt.Errorf("a failure"), nil, fmt.Errorf("a failure")},
|
||||
expErr: nil,
|
||||
},
|
||||
// Consistency all
|
||||
{
|
||||
name: "write all success",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelAll,
|
||||
err: []error{nil, nil, nil},
|
||||
expErr: nil,
|
||||
},
|
||||
{
|
||||
name: "write all, 2/3, partial write",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelAll,
|
||||
err: []error{nil, fmt.Errorf("a failure"), nil},
|
||||
expErr: cluster.ErrPartialWrite,
|
||||
},
|
||||
{
|
||||
name: "write all, 1/3 (failure)",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelAll,
|
||||
err: []error{nil, fmt.Errorf("a failure"), fmt.Errorf("a failure")},
|
||||
expErr: cluster.ErrPartialWrite,
|
||||
},
|
||||
|
||||
// Consistency quorum
|
||||
{
|
||||
name: "write quorum, 1/3 failure",
|
||||
consistency: cluster.ConsistencyLevelQuorum,
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), nil},
|
||||
expErr: cluster.ErrPartialWrite,
|
||||
},
|
||||
{
|
||||
name: "write quorum, 2/3 success",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelQuorum,
|
||||
err: []error{nil, nil, fmt.Errorf("a failure")},
|
||||
expErr: nil,
|
||||
},
|
||||
{
|
||||
name: "write quorum, 3/3 success",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelQuorum,
|
||||
err: []error{nil, nil, nil},
|
||||
expErr: nil,
|
||||
},
|
||||
|
||||
// Error write error
|
||||
{
|
||||
name: "no writes succeed",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelOne,
|
||||
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), fmt.Errorf("a failure")},
|
||||
expErr: fmt.Errorf("write failed: a failure"),
|
||||
},
|
||||
|
||||
// Hinted handoff w/ ANY
|
||||
{
|
||||
name: "hinted handoff write succeed",
|
||||
database: "mydb",
|
||||
retentionPolicy: "myrp",
|
||||
consistency: cluster.ConsistencyLevelAny,
|
||||
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), fmt.Errorf("a failure")},
|
||||
expErr: nil,
|
||||
},
|
||||
|
||||
// Write to non-existant database
|
||||
{
|
||||
name: "write to non-existant database",
|
||||
database: "doesnt_exist",
|
||||
retentionPolicy: "",
|
||||
consistency: cluster.ConsistencyLevelAny,
|
||||
err: []error{nil, nil, nil},
|
||||
expErr: fmt.Errorf("database not found: doesnt_exist"),
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
|
||||
pr := &cluster.WritePointsRequest{
|
||||
Database: test.database,
|
||||
RetentionPolicy: test.retentionPolicy,
|
||||
ConsistencyLevel: test.consistency,
|
||||
}
|
||||
|
||||
// Three points that range over the shardGroup duration (1h) and should map to two
|
||||
// distinct shards
|
||||
pr.AddPoint("cpu", 1.0, time.Unix(0, 0), nil)
|
||||
pr.AddPoint("cpu", 2.0, time.Unix(0, 0).Add(time.Hour), nil)
|
||||
pr.AddPoint("cpu", 3.0, time.Unix(0, 0).Add(time.Hour+time.Second), nil)
|
||||
|
||||
// copy to prevent data race
|
||||
theTest := test
|
||||
sm := cluster.NewShardMapping()
|
||||
sm.MapPoint(
|
||||
&meta.ShardInfo{ID: uint64(1), OwnerIDs: []uint64{uint64(1), uint64(2), uint64(3)}},
|
||||
pr.Points[0])
|
||||
sm.MapPoint(
|
||||
&meta.ShardInfo{ID: uint64(2), OwnerIDs: []uint64{uint64(1), uint64(2), uint64(3)}},
|
||||
pr.Points[1])
|
||||
sm.MapPoint(
|
||||
&meta.ShardInfo{ID: uint64(2), OwnerIDs: []uint64{uint64(1), uint64(2), uint64(3)}},
|
||||
pr.Points[2])
|
||||
|
||||
// Local cluster.Node ShardWriter
|
||||
// lock on the write increment since these functions get called in parallel
|
||||
var mu sync.Mutex
|
||||
sw := &fakeShardWriter{
|
||||
ShardWriteFn: func(shardID, nodeID uint64, points []tsdb.Point) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return theTest.err[int(nodeID)-1]
|
||||
},
|
||||
}
|
||||
|
||||
store := &fakeStore{
|
||||
WriteFn: func(shardID uint64, points []tsdb.Point) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return theTest.err[0]
|
||||
},
|
||||
}
|
||||
|
||||
hh := &fakeShardWriter{
|
||||
ShardWriteFn: func(shardID, nodeID uint64, points []tsdb.Point) error {
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
ms := NewMetaStore()
|
||||
ms.DatabaseFn = func(database string) (*meta.DatabaseInfo, error) {
|
||||
return nil, nil
|
||||
}
|
||||
ms.NodeIDFn = func() uint64 { return 1 }
|
||||
c := cluster.NewPointsWriter()
|
||||
c.MetaStore = ms
|
||||
c.ShardWriter = sw
|
||||
c.TSDBStore = store
|
||||
c.HintedHandoff = hh
|
||||
|
||||
err := c.WritePoints(pr)
|
||||
if err == nil && test.expErr != nil {
|
||||
t.Errorf("PointsWriter.WritePoints(): '%s' error: got %v, exp %v", test.name, err, test.expErr)
|
||||
}
|
||||
|
||||
if err != nil && test.expErr == nil {
|
||||
t.Errorf("PointsWriter.WritePoints(): '%s' error: got %v, exp %v", test.name, err, test.expErr)
|
||||
}
|
||||
if err != nil && test.expErr != nil && err.Error() != test.expErr.Error() {
|
||||
t.Errorf("PointsWriter.WritePoints(): '%s' error: got %v, exp %v", test.name, err, test.expErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var shardID uint64
|
||||
|
||||
type fakeShardWriter struct {
|
||||
ShardWriteFn func(shardID, nodeID uint64, points []tsdb.Point) error
|
||||
}
|
||||
|
||||
func (f *fakeShardWriter) WriteShard(shardID, nodeID uint64, points []tsdb.Point) error {
|
||||
return f.ShardWriteFn(shardID, nodeID, points)
|
||||
}
|
||||
|
||||
type fakeStore struct {
|
||||
WriteFn func(shardID uint64, points []tsdb.Point) error
|
||||
CreateShardfn func(database, retentionPolicy string, shardID uint64) error
|
||||
}
|
||||
|
||||
func (f *fakeStore) WriteToShard(shardID uint64, points []tsdb.Point) error {
|
||||
return f.WriteFn(shardID, points)
|
||||
}
|
||||
|
||||
func (f *fakeStore) CreateShard(database, retentionPolicy string, shardID uint64) error {
|
||||
return f.CreateShardfn(database, retentionPolicy, shardID)
|
||||
}
|
||||
|
||||
func NewMetaStore() *MetaStore {
|
||||
ms := &MetaStore{}
|
||||
rp := NewRetentionPolicy("myp", time.Hour, 3)
|
||||
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
|
||||
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
|
||||
|
||||
ms.RetentionPolicyFn = func(db, retentionPolicy string) (*meta.RetentionPolicyInfo, error) {
|
||||
return rp, nil
|
||||
}
|
||||
|
||||
ms.CreateShardGroupIfNotExistsFn = func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
|
||||
for i, sg := range rp.ShardGroups {
|
||||
if timestamp.Equal(sg.StartTime) || timestamp.After(sg.StartTime) && timestamp.Before(sg.EndTime) {
|
||||
return &rp.ShardGroups[i], nil
|
||||
}
|
||||
}
|
||||
panic("should not get here")
|
||||
}
|
||||
return ms
|
||||
}
|
||||
|
||||
type MetaStore struct {
|
||||
NodeIDFn func() uint64
|
||||
RetentionPolicyFn func(database, name string) (*meta.RetentionPolicyInfo, error)
|
||||
CreateShardGroupIfNotExistsFn func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error)
|
||||
DatabaseFn func(database string) (*meta.DatabaseInfo, error)
|
||||
ShardOwnerFn func(shardID uint64) (string, string, *meta.ShardGroupInfo)
|
||||
}
|
||||
|
||||
func (m MetaStore) NodeID() uint64 { return m.NodeIDFn() }
|
||||
|
||||
func (m MetaStore) RetentionPolicy(database, name string) (*meta.RetentionPolicyInfo, error) {
|
||||
return m.RetentionPolicyFn(database, name)
|
||||
}
|
||||
|
||||
func (m MetaStore) CreateShardGroupIfNotExists(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
|
||||
return m.CreateShardGroupIfNotExistsFn(database, policy, timestamp)
|
||||
}
|
||||
|
||||
func (m MetaStore) Database(database string) (*meta.DatabaseInfo, error) {
|
||||
return m.DatabaseFn(database)
|
||||
}
|
||||
|
||||
func (m MetaStore) ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo) {
|
||||
return m.ShardOwnerFn(shardID)
|
||||
}
|
||||
|
||||
func NewRetentionPolicy(name string, duration time.Duration, nodeCount int) *meta.RetentionPolicyInfo {
|
||||
shards := []meta.ShardInfo{}
|
||||
ownerIDs := []uint64{}
|
||||
for i := 1; i <= nodeCount; i++ {
|
||||
ownerIDs = append(ownerIDs, uint64(i))
|
||||
}
|
||||
|
||||
// each node is fully replicated with each other
|
||||
shards = append(shards, meta.ShardInfo{
|
||||
ID: nextShardID(),
|
||||
OwnerIDs: ownerIDs,
|
||||
})
|
||||
|
||||
rp := &meta.RetentionPolicyInfo{
|
||||
Name: "myrp",
|
||||
ReplicaN: nodeCount,
|
||||
Duration: duration,
|
||||
ShardGroupDuration: duration,
|
||||
ShardGroups: []meta.ShardGroupInfo{
|
||||
meta.ShardGroupInfo{
|
||||
ID: nextShardID(),
|
||||
StartTime: time.Unix(0, 0),
|
||||
EndTime: time.Unix(0, 0).Add(duration).Add(-1),
|
||||
Shards: shards,
|
||||
},
|
||||
},
|
||||
}
|
||||
return rp
|
||||
}
|
||||
|
||||
func AttachShardGroupInfo(rp *meta.RetentionPolicyInfo, ownerIDs []uint64) {
|
||||
var startTime, endTime time.Time
|
||||
if len(rp.ShardGroups) == 0 {
|
||||
startTime = time.Unix(0, 0)
|
||||
} else {
|
||||
startTime = rp.ShardGroups[len(rp.ShardGroups)-1].StartTime.Add(rp.ShardGroupDuration)
|
||||
}
|
||||
endTime = startTime.Add(rp.ShardGroupDuration).Add(-1)
|
||||
|
||||
sh := meta.ShardGroupInfo{
|
||||
ID: uint64(len(rp.ShardGroups) + 1),
|
||||
StartTime: startTime,
|
||||
EndTime: endTime,
|
||||
Shards: []meta.ShardInfo{
|
||||
meta.ShardInfo{
|
||||
ID: nextShardID(),
|
||||
OwnerIDs: ownerIDs,
|
||||
},
|
||||
},
|
||||
}
|
||||
rp.ShardGroups = append(rp.ShardGroups, sh)
|
||||
}
|
||||
|
||||
func nextShardID() uint64 {
|
||||
return atomic.AddUint64(&shardID, 1)
|
||||
}
|
||||
229
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/rpc.go
generated
vendored
229
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/rpc.go
generated
vendored
@@ -1,229 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/gogo/protobuf/proto"
|
||||
"github.com/influxdb/influxdb/cluster/internal"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
//go:generate protoc --gogo_out=. internal/data.proto
|
||||
|
||||
// MapShardRequest represents the request to map a remote shard for a query.
|
||||
type MapShardRequest struct {
|
||||
pb internal.MapShardRequest
|
||||
}
|
||||
|
||||
func (m *MapShardRequest) ShardID() uint64 { return m.pb.GetShardID() }
|
||||
func (m *MapShardRequest) Query() string { return m.pb.GetQuery() }
|
||||
func (m *MapShardRequest) ChunkSize() int32 { return m.pb.GetChunkSize() }
|
||||
|
||||
func (m *MapShardRequest) SetShardID(id uint64) { m.pb.ShardID = &id }
|
||||
func (m *MapShardRequest) SetQuery(query string) { m.pb.Query = &query }
|
||||
func (m *MapShardRequest) SetChunkSize(chunkSize int32) { m.pb.ChunkSize = &chunkSize }
|
||||
|
||||
// MarshalBinary encodes the object to a binary format.
|
||||
func (m *MapShardRequest) MarshalBinary() ([]byte, error) {
|
||||
return proto.Marshal(&m.pb)
|
||||
}
|
||||
|
||||
// UnmarshalBinary populates MapShardRequest from a binary format.
|
||||
func (m *MapShardRequest) UnmarshalBinary(buf []byte) error {
|
||||
if err := proto.Unmarshal(buf, &m.pb); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MapShardResponse represents the response returned from a remote MapShardRequest call
|
||||
type MapShardResponse struct {
|
||||
pb internal.MapShardResponse
|
||||
}
|
||||
|
||||
func NewMapShardResponse(code int, message string) *MapShardResponse {
|
||||
m := &MapShardResponse{}
|
||||
m.SetCode(code)
|
||||
m.SetMessage(message)
|
||||
return m
|
||||
}
|
||||
|
||||
func (r *MapShardResponse) Code() int { return int(r.pb.GetCode()) }
|
||||
func (r *MapShardResponse) Message() string { return r.pb.GetMessage() }
|
||||
func (r *MapShardResponse) TagSets() []string { return r.pb.GetTagSets() }
|
||||
func (r *MapShardResponse) Fields() []string { return r.pb.GetFields() }
|
||||
func (r *MapShardResponse) Data() []byte { return r.pb.GetData() }
|
||||
|
||||
func (r *MapShardResponse) SetCode(code int) { r.pb.Code = proto.Int32(int32(code)) }
|
||||
func (r *MapShardResponse) SetMessage(message string) { r.pb.Message = &message }
|
||||
func (r *MapShardResponse) SetTagSets(tagsets []string) { r.pb.TagSets = tagsets }
|
||||
func (r *MapShardResponse) SetFields(fields []string) { r.pb.Fields = fields }
|
||||
func (r *MapShardResponse) SetData(data []byte) { r.pb.Data = data }
|
||||
|
||||
// MarshalBinary encodes the object to a binary format.
|
||||
func (r *MapShardResponse) MarshalBinary() ([]byte, error) {
|
||||
return proto.Marshal(&r.pb)
|
||||
}
|
||||
|
||||
// UnmarshalBinary populates WritePointRequest from a binary format.
|
||||
func (r *MapShardResponse) UnmarshalBinary(buf []byte) error {
|
||||
if err := proto.Unmarshal(buf, &r.pb); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// WritePointsRequest represents a request to write point data to the cluster
|
||||
type WritePointsRequest struct {
|
||||
Database string
|
||||
RetentionPolicy string
|
||||
ConsistencyLevel ConsistencyLevel
|
||||
Points []tsdb.Point
|
||||
}
|
||||
|
||||
// AddPoint adds a point to the WritePointRequest with field name 'value'
|
||||
func (w *WritePointsRequest) AddPoint(name string, value interface{}, timestamp time.Time, tags map[string]string) {
|
||||
w.Points = append(w.Points, tsdb.NewPoint(
|
||||
name, tags, map[string]interface{}{"value": value}, timestamp,
|
||||
))
|
||||
}
|
||||
|
||||
// WriteShardRequest represents the a request to write a slice of points to a shard
|
||||
type WriteShardRequest struct {
|
||||
pb internal.WriteShardRequest
|
||||
}
|
||||
|
||||
// WriteShardResponse represents the response returned from a remote WriteShardRequest call
|
||||
type WriteShardResponse struct {
|
||||
pb internal.WriteShardResponse
|
||||
}
|
||||
|
||||
func (w *WriteShardRequest) SetShardID(id uint64) { w.pb.ShardID = &id }
|
||||
func (w *WriteShardRequest) ShardID() uint64 { return w.pb.GetShardID() }
|
||||
|
||||
func (w *WriteShardRequest) Points() []tsdb.Point { return w.unmarshalPoints() }
|
||||
|
||||
func (w *WriteShardRequest) AddPoint(name string, value interface{}, timestamp time.Time, tags map[string]string) {
|
||||
w.AddPoints([]tsdb.Point{tsdb.NewPoint(
|
||||
name, tags, map[string]interface{}{"value": value}, timestamp,
|
||||
)})
|
||||
}
|
||||
|
||||
func (w *WriteShardRequest) AddPoints(points []tsdb.Point) {
|
||||
w.pb.Points = append(w.pb.Points, w.marshalPoints(points)...)
|
||||
}
|
||||
|
||||
// MarshalBinary encodes the object to a binary format.
|
||||
func (w *WriteShardRequest) MarshalBinary() ([]byte, error) {
|
||||
return proto.Marshal(&w.pb)
|
||||
}
|
||||
|
||||
func (w *WriteShardRequest) marshalPoints(points []tsdb.Point) []*internal.Point {
|
||||
pts := make([]*internal.Point, len(points))
|
||||
for i, p := range points {
|
||||
fields := []*internal.Field{}
|
||||
for k, v := range p.Fields() {
|
||||
name := k
|
||||
f := &internal.Field{
|
||||
Name: &name,
|
||||
}
|
||||
switch t := v.(type) {
|
||||
case int:
|
||||
f.Int64 = proto.Int64(int64(t))
|
||||
case int32:
|
||||
f.Int32 = proto.Int32(t)
|
||||
case int64:
|
||||
f.Int64 = proto.Int64(t)
|
||||
case float64:
|
||||
f.Float64 = proto.Float64(t)
|
||||
case bool:
|
||||
f.Bool = proto.Bool(t)
|
||||
case string:
|
||||
f.String_ = proto.String(t)
|
||||
case []byte:
|
||||
f.Bytes = t
|
||||
}
|
||||
fields = append(fields, f)
|
||||
}
|
||||
|
||||
tags := []*internal.Tag{}
|
||||
for k, v := range p.Tags() {
|
||||
key := k
|
||||
value := v
|
||||
tags = append(tags, &internal.Tag{
|
||||
Key: &key,
|
||||
Value: &value,
|
||||
})
|
||||
}
|
||||
name := p.Name()
|
||||
pts[i] = &internal.Point{
|
||||
Name: &name,
|
||||
Time: proto.Int64(p.Time().UnixNano()),
|
||||
Fields: fields,
|
||||
Tags: tags,
|
||||
}
|
||||
|
||||
}
|
||||
return pts
|
||||
}
|
||||
|
||||
// UnmarshalBinary populates WritePointRequest from a binary format.
|
||||
func (w *WriteShardRequest) UnmarshalBinary(buf []byte) error {
|
||||
if err := proto.Unmarshal(buf, &w.pb); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *WriteShardRequest) unmarshalPoints() []tsdb.Point {
|
||||
points := make([]tsdb.Point, len(w.pb.GetPoints()))
|
||||
for i, p := range w.pb.GetPoints() {
|
||||
pt := tsdb.NewPoint(
|
||||
p.GetName(), map[string]string{},
|
||||
map[string]interface{}{}, time.Unix(0, p.GetTime()))
|
||||
|
||||
for _, f := range p.GetFields() {
|
||||
n := f.GetName()
|
||||
if f.Int32 != nil {
|
||||
pt.AddField(n, f.GetInt32())
|
||||
} else if f.Int64 != nil {
|
||||
pt.AddField(n, f.GetInt64())
|
||||
} else if f.Float64 != nil {
|
||||
pt.AddField(n, f.GetFloat64())
|
||||
} else if f.Bool != nil {
|
||||
pt.AddField(n, f.GetBool())
|
||||
} else if f.String_ != nil {
|
||||
pt.AddField(n, f.GetString_())
|
||||
} else {
|
||||
pt.AddField(n, f.GetBytes())
|
||||
}
|
||||
}
|
||||
|
||||
tags := tsdb.Tags{}
|
||||
for _, t := range p.GetTags() {
|
||||
tags[t.GetKey()] = t.GetValue()
|
||||
}
|
||||
pt.SetTags(tags)
|
||||
points[i] = pt
|
||||
}
|
||||
return points
|
||||
}
|
||||
|
||||
func (w *WriteShardResponse) SetCode(code int) { w.pb.Code = proto.Int32(int32(code)) }
|
||||
func (w *WriteShardResponse) SetMessage(message string) { w.pb.Message = &message }
|
||||
|
||||
func (w *WriteShardResponse) Code() int { return int(w.pb.GetCode()) }
|
||||
func (w *WriteShardResponse) Message() string { return w.pb.GetMessage() }
|
||||
|
||||
// MarshalBinary encodes the object to a binary format.
|
||||
func (w *WriteShardResponse) MarshalBinary() ([]byte, error) {
|
||||
return proto.Marshal(&w.pb)
|
||||
}
|
||||
|
||||
// UnmarshalBinary populates WritePointRequest from a binary format.
|
||||
func (w *WriteShardResponse) UnmarshalBinary(buf []byte) error {
|
||||
if err := proto.Unmarshal(buf, &w.pb); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
110
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/rpc_test.go
generated
vendored
110
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/rpc_test.go
generated
vendored
@@ -1,110 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestWriteShardRequestBinary(t *testing.T) {
|
||||
sr := &WriteShardRequest{}
|
||||
|
||||
sr.SetShardID(uint64(1))
|
||||
if exp := uint64(1); sr.ShardID() != exp {
|
||||
t.Fatalf("ShardID mismatch: got %v, exp %v", sr.ShardID(), exp)
|
||||
}
|
||||
|
||||
sr.AddPoint("cpu", 1.0, time.Unix(0, 0), map[string]string{"host": "serverA"})
|
||||
sr.AddPoint("cpu", 2.0, time.Unix(0, 0).Add(time.Hour), nil)
|
||||
sr.AddPoint("cpu_load", 3.0, time.Unix(0, 0).Add(time.Hour+time.Second), nil)
|
||||
|
||||
b, err := sr.MarshalBinary()
|
||||
if err != nil {
|
||||
t.Fatalf("WritePointsRequest.MarshalBinary() failed: %v", err)
|
||||
}
|
||||
if len(b) == 0 {
|
||||
t.Fatalf("WritePointsRequest.MarshalBinary() returned 0 bytes")
|
||||
}
|
||||
|
||||
got := &WriteShardRequest{}
|
||||
if err := got.UnmarshalBinary(b); err != nil {
|
||||
t.Fatalf("WritePointsRequest.UnmarshalMarshalBinary() failed: %v", err)
|
||||
}
|
||||
|
||||
if got.ShardID() != sr.ShardID() {
|
||||
t.Errorf("ShardID mismatch: got %v, exp %v", got.ShardID(), sr.ShardID())
|
||||
}
|
||||
|
||||
if len(got.Points()) != len(sr.Points()) {
|
||||
t.Errorf("Points count mismatch: got %v, exp %v", len(got.Points()), len(sr.Points()))
|
||||
}
|
||||
|
||||
srPoints := sr.Points()
|
||||
gotPoints := got.Points()
|
||||
for i, p := range srPoints {
|
||||
g := gotPoints[i]
|
||||
|
||||
if g.Name() != p.Name() {
|
||||
t.Errorf("Point %d name mismatch: got %v, exp %v", i, g.Name(), p.Name())
|
||||
}
|
||||
|
||||
if !g.Time().Equal(p.Time()) {
|
||||
t.Errorf("Point %d time mismatch: got %v, exp %v", i, g.Time(), p.Time())
|
||||
}
|
||||
|
||||
if g.HashID() != p.HashID() {
|
||||
t.Errorf("Point #%d HashID() mismatch: got %v, exp %v", i, g.HashID(), p.HashID())
|
||||
}
|
||||
|
||||
for k, v := range p.Tags() {
|
||||
if g.Tags()[k] != v {
|
||||
t.Errorf("Point #%d tag mismatch: got %v, exp %v", i, k, v)
|
||||
}
|
||||
}
|
||||
|
||||
if len(p.Fields()) != len(g.Fields()) {
|
||||
t.Errorf("Point %d field count mismatch: got %v, exp %v", i, len(g.Fields()), len(p.Fields()))
|
||||
}
|
||||
|
||||
for j, f := range p.Fields() {
|
||||
if g.Fields()[j] != f {
|
||||
t.Errorf("Point %d field mismatch: got %v, exp %v", i, g.Fields()[j], f)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteShardResponseBinary(t *testing.T) {
|
||||
sr := &WriteShardResponse{}
|
||||
sr.SetCode(10)
|
||||
sr.SetMessage("foo")
|
||||
b, err := sr.MarshalBinary()
|
||||
|
||||
if exp := 10; sr.Code() != exp {
|
||||
t.Fatalf("Code mismatch: got %v, exp %v", sr.Code(), exp)
|
||||
}
|
||||
|
||||
if exp := "foo"; sr.Message() != exp {
|
||||
t.Fatalf("Message mismatch: got %v, exp %v", sr.Message(), exp)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("WritePointsResponse.MarshalBinary() failed: %v", err)
|
||||
}
|
||||
if len(b) == 0 {
|
||||
t.Fatalf("WritePointsResponse.MarshalBinary() returned 0 bytes")
|
||||
}
|
||||
|
||||
got := &WriteShardResponse{}
|
||||
if err := got.UnmarshalBinary(b); err != nil {
|
||||
t.Fatalf("WritePointsResponse.UnmarshalMarshalBinary() failed: %v", err)
|
||||
}
|
||||
|
||||
if got.Code() != sr.Code() {
|
||||
t.Errorf("Code mismatch: got %v, exp %v", got.Code(), sr.Code())
|
||||
}
|
||||
|
||||
if got.Message() != sr.Message() {
|
||||
t.Errorf("Message mismatch: got %v, exp %v", got.Message(), sr.Message())
|
||||
}
|
||||
|
||||
}
|
||||
338
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/service.go
generated
vendored
338
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/service.go
generated
vendored
@@ -1,338 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// MaxMessageSize defines how large a message can be before we reject it
|
||||
const MaxMessageSize = 1024 * 1024 * 1024 // 1GB
|
||||
|
||||
// MuxHeader is the header byte used in the TCP mux.
|
||||
const MuxHeader = 2
|
||||
|
||||
// Service processes data received over raw TCP connections.
|
||||
type Service struct {
|
||||
mu sync.RWMutex
|
||||
|
||||
wg sync.WaitGroup
|
||||
closing chan struct{}
|
||||
|
||||
Listener net.Listener
|
||||
|
||||
MetaStore interface {
|
||||
ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo)
|
||||
}
|
||||
|
||||
TSDBStore interface {
|
||||
CreateShard(database, policy string, shardID uint64) error
|
||||
WriteToShard(shardID uint64, points []tsdb.Point) error
|
||||
CreateMapper(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error)
|
||||
}
|
||||
|
||||
Logger *log.Logger
|
||||
}
|
||||
|
||||
// NewService returns a new instance of Service.
|
||||
func NewService(c Config) *Service {
|
||||
return &Service{
|
||||
closing: make(chan struct{}),
|
||||
Logger: log.New(os.Stderr, "[tcp] ", log.LstdFlags),
|
||||
}
|
||||
}
|
||||
|
||||
// Open opens the network listener and begins serving requests.
|
||||
func (s *Service) Open() error {
|
||||
|
||||
s.Logger.Println("Starting cluster service")
|
||||
// Begin serving conections.
|
||||
s.wg.Add(1)
|
||||
go s.serve()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetLogger sets the internal logger to the logger passed in.
|
||||
func (s *Service) SetLogger(l *log.Logger) {
|
||||
s.Logger = l
|
||||
}
|
||||
|
||||
// serve accepts connections from the listener and handles them.
|
||||
func (s *Service) serve() {
|
||||
defer s.wg.Done()
|
||||
|
||||
for {
|
||||
// Check if the service is shutting down.
|
||||
select {
|
||||
case <-s.closing:
|
||||
return
|
||||
default:
|
||||
}
|
||||
|
||||
// Accept the next connection.
|
||||
conn, err := s.Listener.Accept()
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "connection closed") {
|
||||
s.Logger.Printf("cluster service accept error: %s", err)
|
||||
return
|
||||
}
|
||||
s.Logger.Printf("accept error: %s", err)
|
||||
continue
|
||||
}
|
||||
|
||||
// Delegate connection handling to a separate goroutine.
|
||||
s.wg.Add(1)
|
||||
go func() {
|
||||
defer s.wg.Done()
|
||||
s.handleConn(conn)
|
||||
}()
|
||||
}
|
||||
}
|
||||
|
||||
// Close shuts down the listener and waits for all connections to finish.
|
||||
func (s *Service) Close() error {
|
||||
if s.Listener != nil {
|
||||
s.Listener.Close()
|
||||
}
|
||||
|
||||
// Shut down all handlers.
|
||||
close(s.closing)
|
||||
// s.wg.Wait() // FIXME(benbjohnson)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// handleConn services an individual TCP connection.
|
||||
func (s *Service) handleConn(conn net.Conn) {
|
||||
// Ensure connection is closed when service is closed.
|
||||
closing := make(chan struct{})
|
||||
defer close(closing)
|
||||
go func() {
|
||||
select {
|
||||
case <-closing:
|
||||
case <-s.closing:
|
||||
}
|
||||
conn.Close()
|
||||
}()
|
||||
|
||||
s.Logger.Printf("accept remote write connection from %v\n", conn.RemoteAddr())
|
||||
defer func() {
|
||||
s.Logger.Printf("close remote write connection from %v\n", conn.RemoteAddr())
|
||||
}()
|
||||
for {
|
||||
// Read type-length-value.
|
||||
typ, buf, err := ReadTLV(conn)
|
||||
if err != nil {
|
||||
if strings.HasSuffix(err.Error(), "EOF") {
|
||||
return
|
||||
}
|
||||
s.Logger.Printf("unable to read type-length-value %s", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Delegate message processing by type.
|
||||
switch typ {
|
||||
case writeShardRequestMessage:
|
||||
err := s.processWriteShardRequest(buf)
|
||||
if err != nil {
|
||||
s.Logger.Printf("process write shard error: %s", err)
|
||||
}
|
||||
s.writeShardResponse(conn, err)
|
||||
case mapShardRequestMessage:
|
||||
err := s.processMapShardRequest(conn, buf)
|
||||
if err != nil {
|
||||
s.Logger.Printf("process map shard error: %s", err)
|
||||
if err := writeMapShardResponseMessage(conn, NewMapShardResponse(1, err.Error())); err != nil {
|
||||
s.Logger.Printf("process map shard error writing response: %s", err.Error())
|
||||
}
|
||||
}
|
||||
default:
|
||||
s.Logger.Printf("cluster service message type not found: %d", typ)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Service) processWriteShardRequest(buf []byte) error {
|
||||
// Build request
|
||||
var req WriteShardRequest
|
||||
if err := req.UnmarshalBinary(buf); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err := s.TSDBStore.WriteToShard(req.ShardID(), req.Points())
|
||||
|
||||
// We may have received a write for a shard that we don't have locally because the
|
||||
// sending node may have just created the shard (via the metastore) and the write
|
||||
// arrived before the local store could create the shard. In this case, we need
|
||||
// to check the metastore to determine what database and retention policy this
|
||||
// shard should reside within.
|
||||
if err == tsdb.ErrShardNotFound {
|
||||
|
||||
// Query the metastore for the owner of this shard
|
||||
database, retentionPolicy, sgi := s.MetaStore.ShardOwner(req.ShardID())
|
||||
if sgi == nil {
|
||||
// If we can't find it, then we need to drop this request
|
||||
// as it is no longer valid. This could happen if writes were queued via
|
||||
// hinted handoff and delivered after a shard group was deleted.
|
||||
s.Logger.Printf("drop write request: shard=%d", req.ShardID())
|
||||
return nil
|
||||
}
|
||||
|
||||
err = s.TSDBStore.CreateShard(database, retentionPolicy, req.ShardID())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.TSDBStore.WriteToShard(req.ShardID(), req.Points())
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("write shard %d: %s", req.ShardID(), err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Service) writeShardResponse(w io.Writer, e error) {
|
||||
// Build response.
|
||||
var resp WriteShardResponse
|
||||
if e != nil {
|
||||
resp.SetCode(1)
|
||||
resp.SetMessage(e.Error())
|
||||
} else {
|
||||
resp.SetCode(0)
|
||||
}
|
||||
|
||||
// Marshal response to binary.
|
||||
buf, err := resp.MarshalBinary()
|
||||
if err != nil {
|
||||
s.Logger.Printf("error marshalling shard response: %s", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Write to connection.
|
||||
if err := WriteTLV(w, writeShardResponseMessage, buf); err != nil {
|
||||
s.Logger.Printf("write shard response error: %s", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Service) processMapShardRequest(w io.Writer, buf []byte) error {
|
||||
// Decode request
|
||||
var req MapShardRequest
|
||||
if err := req.UnmarshalBinary(buf); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
m, err := s.TSDBStore.CreateMapper(req.ShardID(), req.Query(), int(req.ChunkSize()))
|
||||
if err != nil {
|
||||
return fmt.Errorf("create mapper: %s", err)
|
||||
}
|
||||
if m == nil {
|
||||
return writeMapShardResponseMessage(w, NewMapShardResponse(0, ""))
|
||||
}
|
||||
|
||||
if err := m.Open(); err != nil {
|
||||
return fmt.Errorf("mapper open: %s", err)
|
||||
}
|
||||
defer m.Close()
|
||||
|
||||
var metaSent bool
|
||||
for {
|
||||
var resp MapShardResponse
|
||||
|
||||
if !metaSent {
|
||||
resp.SetTagSets(m.TagSets())
|
||||
resp.SetFields(m.Fields())
|
||||
metaSent = true
|
||||
}
|
||||
|
||||
chunk, err := m.NextChunk()
|
||||
if err != nil {
|
||||
return fmt.Errorf("next chunk: %s", err)
|
||||
}
|
||||
if chunk != nil {
|
||||
b, err := json.Marshal(chunk)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encoding: %s", err)
|
||||
}
|
||||
resp.SetData(b)
|
||||
}
|
||||
|
||||
// Write to connection.
|
||||
resp.SetCode(0)
|
||||
if err := writeMapShardResponseMessage(w, &resp); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if chunk == nil {
|
||||
// All mapper data sent.
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func writeMapShardResponseMessage(w io.Writer, msg *MapShardResponse) error {
|
||||
buf, err := msg.MarshalBinary()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return WriteTLV(w, mapShardResponseMessage, buf)
|
||||
}
|
||||
|
||||
// ReadTLV reads a type-length-value record from r.
|
||||
func ReadTLV(r io.Reader) (byte, []byte, error) {
|
||||
var typ [1]byte
|
||||
if _, err := io.ReadFull(r, typ[:]); err != nil {
|
||||
return 0, nil, fmt.Errorf("read message type: %s", err)
|
||||
}
|
||||
|
||||
// Read the size of the message.
|
||||
var sz int64
|
||||
if err := binary.Read(r, binary.BigEndian, &sz); err != nil {
|
||||
return 0, nil, fmt.Errorf("read message size: %s", err)
|
||||
}
|
||||
|
||||
if sz == 0 {
|
||||
return 0, nil, fmt.Errorf("invalid message size: %d", sz)
|
||||
}
|
||||
|
||||
if sz >= MaxMessageSize {
|
||||
return 0, nil, fmt.Errorf("max message size of %d exceeded: %d", MaxMessageSize, sz)
|
||||
}
|
||||
|
||||
// Read the value.
|
||||
buf := make([]byte, sz)
|
||||
if _, err := io.ReadFull(r, buf); err != nil {
|
||||
return 0, nil, fmt.Errorf("read message value: %s", err)
|
||||
}
|
||||
|
||||
return typ[0], buf, nil
|
||||
}
|
||||
|
||||
// WriteTLV writes a type-length-value record to w.
|
||||
func WriteTLV(w io.Writer, typ byte, buf []byte) error {
|
||||
if _, err := w.Write([]byte{typ}); err != nil {
|
||||
return fmt.Errorf("write message type: %s", err)
|
||||
}
|
||||
|
||||
// Write the size of the message.
|
||||
if err := binary.Write(w, binary.BigEndian, int64(len(buf))); err != nil {
|
||||
return fmt.Errorf("write message size: %s", err)
|
||||
}
|
||||
|
||||
// Write the value.
|
||||
if _, err := w.Write(buf); err != nil {
|
||||
return fmt.Errorf("write message value: %s", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
103
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/service_test.go
generated
vendored
103
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/service_test.go
generated
vendored
@@ -1,103 +0,0 @@
|
||||
package cluster_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cluster"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/tcp"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
type metaStore struct {
|
||||
host string
|
||||
}
|
||||
|
||||
func (m *metaStore) Node(nodeID uint64) (*meta.NodeInfo, error) {
|
||||
return &meta.NodeInfo{
|
||||
ID: nodeID,
|
||||
Host: m.host,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type testService struct {
|
||||
nodeID uint64
|
||||
ln net.Listener
|
||||
muxln net.Listener
|
||||
writeShardFunc func(shardID uint64, points []tsdb.Point) error
|
||||
createShardFunc func(database, policy string, shardID uint64) error
|
||||
createMapperFunc func(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error)
|
||||
}
|
||||
|
||||
func newTestWriteService(f func(shardID uint64, points []tsdb.Point) error) testService {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
mux := tcp.NewMux()
|
||||
muxln := mux.Listen(cluster.MuxHeader)
|
||||
go mux.Serve(ln)
|
||||
|
||||
return testService{
|
||||
writeShardFunc: f,
|
||||
ln: ln,
|
||||
muxln: muxln,
|
||||
}
|
||||
}
|
||||
|
||||
func (ts *testService) Close() {
|
||||
if ts.ln != nil {
|
||||
ts.ln.Close()
|
||||
}
|
||||
}
|
||||
|
||||
type serviceResponses []serviceResponse
|
||||
type serviceResponse struct {
|
||||
shardID uint64
|
||||
ownerID uint64
|
||||
points []tsdb.Point
|
||||
}
|
||||
|
||||
func (t testService) WriteToShard(shardID uint64, points []tsdb.Point) error {
|
||||
return t.writeShardFunc(shardID, points)
|
||||
}
|
||||
|
||||
func (t testService) CreateShard(database, policy string, shardID uint64) error {
|
||||
return t.createShardFunc(database, policy, shardID)
|
||||
}
|
||||
|
||||
func (t testService) CreateMapper(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error) {
|
||||
return t.createMapperFunc(shardID, query, chunkSize)
|
||||
}
|
||||
|
||||
func writeShardSuccess(shardID uint64, points []tsdb.Point) error {
|
||||
responses <- &serviceResponse{
|
||||
shardID: shardID,
|
||||
points: points,
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func writeShardFail(shardID uint64, points []tsdb.Point) error {
|
||||
return fmt.Errorf("failed to write")
|
||||
}
|
||||
|
||||
var responses = make(chan *serviceResponse, 1024)
|
||||
|
||||
func (testService) ResponseN(n int) ([]*serviceResponse, error) {
|
||||
var a []*serviceResponse
|
||||
for {
|
||||
select {
|
||||
case r := <-responses:
|
||||
a = append(a, r)
|
||||
if len(a) == n {
|
||||
return a, nil
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
return a, fmt.Errorf("unexpected response count: expected: %d, actual: %d", n, len(a))
|
||||
}
|
||||
}
|
||||
}
|
||||
207
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_mapper.go
generated
vendored
207
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_mapper.go
generated
vendored
@@ -1,207 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/rand"
|
||||
"net"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
"gopkg.in/fatih/pool.v2"
|
||||
)
|
||||
|
||||
// ShardMapper is responsible for providing mappers for requested shards. It is
|
||||
// responsible for creating those mappers from the local store, or reaching
|
||||
// out to another node on the cluster.
|
||||
type ShardMapper struct {
|
||||
ForceRemoteMapping bool // All shards treated as remote. Useful for testing.
|
||||
|
||||
MetaStore interface {
|
||||
NodeID() uint64
|
||||
Node(id uint64) (ni *meta.NodeInfo, err error)
|
||||
}
|
||||
|
||||
TSDBStore interface {
|
||||
CreateMapper(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error)
|
||||
}
|
||||
|
||||
timeout time.Duration
|
||||
pool *clientPool
|
||||
}
|
||||
|
||||
// NewShardMapper returns a mapper of local and remote shards.
|
||||
func NewShardMapper(timeout time.Duration) *ShardMapper {
|
||||
return &ShardMapper{
|
||||
pool: newClientPool(),
|
||||
timeout: timeout,
|
||||
}
|
||||
}
|
||||
|
||||
// CreateMapper returns a Mapper for the given shard ID.
|
||||
func (s *ShardMapper) CreateMapper(sh meta.ShardInfo, stmt string, chunkSize int) (tsdb.Mapper, error) {
|
||||
var err error
|
||||
var m tsdb.Mapper
|
||||
if sh.OwnedBy(s.MetaStore.NodeID()) && !s.ForceRemoteMapping {
|
||||
m, err = s.TSDBStore.CreateMapper(sh.ID, stmt, chunkSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else {
|
||||
// Pick a node in a pseudo-random manner.
|
||||
conn, err := s.dial(sh.OwnerIDs[rand.Intn(len(sh.OwnerIDs))])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
conn.SetDeadline(time.Now().Add(s.timeout))
|
||||
|
||||
rm := NewRemoteMapper(conn.(*pool.PoolConn), sh.ID, stmt, chunkSize)
|
||||
m = rm
|
||||
}
|
||||
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (s *ShardMapper) dial(nodeID uint64) (net.Conn, error) {
|
||||
// If we don't have a connection pool for that addr yet, create one
|
||||
_, ok := s.pool.getPool(nodeID)
|
||||
if !ok {
|
||||
factory := &connFactory{nodeID: nodeID, clientPool: s.pool, timeout: s.timeout}
|
||||
factory.metaStore = s.MetaStore
|
||||
|
||||
p, err := pool.NewChannelPool(1, 3, factory.dial)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.pool.setPool(nodeID, p)
|
||||
}
|
||||
return s.pool.conn(nodeID)
|
||||
}
|
||||
|
||||
type remoteShardConn interface {
|
||||
io.ReadWriter
|
||||
Close() error
|
||||
MarkUnusable()
|
||||
}
|
||||
|
||||
// RemoteMapper implements the tsdb.Mapper interface. It connects to a remote node,
|
||||
// sends a query, and interprets the stream of data that comes back.
|
||||
type RemoteMapper struct {
|
||||
shardID uint64
|
||||
stmt string
|
||||
chunkSize int
|
||||
|
||||
tagsets []string
|
||||
fields []string
|
||||
|
||||
conn remoteShardConn
|
||||
bufferedResponse *MapShardResponse
|
||||
}
|
||||
|
||||
// NewRemoteMapper returns a new remote mapper using the given connection.
|
||||
func NewRemoteMapper(c remoteShardConn, shardID uint64, stmt string, chunkSize int) *RemoteMapper {
|
||||
return &RemoteMapper{
|
||||
conn: c,
|
||||
shardID: shardID,
|
||||
stmt: stmt,
|
||||
chunkSize: chunkSize,
|
||||
}
|
||||
}
|
||||
|
||||
// Open connects to the remote node and starts receiving data.
|
||||
func (r *RemoteMapper) Open() (err error) {
|
||||
defer func() {
|
||||
if err != nil {
|
||||
r.conn.Close()
|
||||
}
|
||||
}()
|
||||
// Build Map request.
|
||||
var request MapShardRequest
|
||||
request.SetShardID(r.shardID)
|
||||
request.SetQuery(r.stmt)
|
||||
request.SetChunkSize(int32(r.chunkSize))
|
||||
|
||||
// Marshal into protocol buffers.
|
||||
buf, err := request.MarshalBinary()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Write request.
|
||||
if err := WriteTLV(r.conn, mapShardRequestMessage, buf); err != nil {
|
||||
r.conn.MarkUnusable()
|
||||
return err
|
||||
}
|
||||
|
||||
// Read the response.
|
||||
_, buf, err = ReadTLV(r.conn)
|
||||
if err != nil {
|
||||
r.conn.MarkUnusable()
|
||||
return err
|
||||
}
|
||||
|
||||
// Unmarshal response.
|
||||
r.bufferedResponse = &MapShardResponse{}
|
||||
if err := r.bufferedResponse.UnmarshalBinary(buf); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if r.bufferedResponse.Code() != 0 {
|
||||
return fmt.Errorf("error code %d: %s", r.bufferedResponse.Code(), r.bufferedResponse.Message())
|
||||
}
|
||||
|
||||
// Decode the first response to get the TagSets.
|
||||
r.tagsets = r.bufferedResponse.TagSets()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *RemoteMapper) TagSets() []string {
|
||||
return r.tagsets
|
||||
}
|
||||
|
||||
func (r *RemoteMapper) Fields() []string {
|
||||
return r.fields
|
||||
}
|
||||
|
||||
// NextChunk returns the next chunk read from the remote node to the client.
|
||||
func (r *RemoteMapper) NextChunk() (chunk interface{}, err error) {
|
||||
output := &tsdb.MapperOutput{}
|
||||
var response *MapShardResponse
|
||||
|
||||
if r.bufferedResponse != nil {
|
||||
response = r.bufferedResponse
|
||||
r.bufferedResponse = nil
|
||||
} else {
|
||||
response = &MapShardResponse{}
|
||||
|
||||
// Read the response.
|
||||
_, buf, err := ReadTLV(r.conn)
|
||||
if err != nil {
|
||||
r.conn.MarkUnusable()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Unmarshal response.
|
||||
if err := response.UnmarshalBinary(buf); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if response.Code() != 0 {
|
||||
return nil, fmt.Errorf("error code %d: %s", response.Code(), response.Message())
|
||||
}
|
||||
}
|
||||
|
||||
if response.Data() == nil {
|
||||
return nil, nil
|
||||
}
|
||||
err = json.Unmarshal(response.Data(), output)
|
||||
return output, err
|
||||
}
|
||||
|
||||
// Close the Mapper
|
||||
func (r *RemoteMapper) Close() {
|
||||
r.conn.Close()
|
||||
}
|
||||
96
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_mapper_test.go
generated
vendored
96
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_mapper_test.go
generated
vendored
@@ -1,96 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// remoteShardResponder implements the remoteShardConn interface.
|
||||
type remoteShardResponder struct {
|
||||
t *testing.T
|
||||
rxBytes []byte
|
||||
|
||||
buffer *bytes.Buffer
|
||||
}
|
||||
|
||||
func newRemoteShardResponder(outputs []*tsdb.MapperOutput, tagsets []string) *remoteShardResponder {
|
||||
r := &remoteShardResponder{}
|
||||
a := make([]byte, 0, 1024)
|
||||
r.buffer = bytes.NewBuffer(a)
|
||||
|
||||
// Pump the outputs in the buffer for later reading.
|
||||
for _, o := range outputs {
|
||||
resp := &MapShardResponse{}
|
||||
resp.SetCode(0)
|
||||
if o != nil {
|
||||
d, _ := json.Marshal(o)
|
||||
resp.SetData(d)
|
||||
resp.SetTagSets(tagsets)
|
||||
}
|
||||
|
||||
g, _ := resp.MarshalBinary()
|
||||
WriteTLV(r.buffer, mapShardResponseMessage, g)
|
||||
}
|
||||
|
||||
return r
|
||||
}
|
||||
|
||||
func (r remoteShardResponder) MarkUnusable() { return }
|
||||
func (r remoteShardResponder) Close() error { return nil }
|
||||
func (r remoteShardResponder) Read(p []byte) (n int, err error) {
|
||||
return io.ReadFull(r.buffer, p)
|
||||
}
|
||||
|
||||
func (r remoteShardResponder) Write(p []byte) (n int, err error) {
|
||||
if r.rxBytes == nil {
|
||||
r.rxBytes = make([]byte, 0)
|
||||
}
|
||||
r.rxBytes = append(r.rxBytes, p...)
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// Ensure a RemoteMapper can process valid responses from a remote shard.
|
||||
func TestShardWriter_RemoteMapper_Success(t *testing.T) {
|
||||
expTagSets := []string{"tagsetA"}
|
||||
expOutput := &tsdb.MapperOutput{
|
||||
Name: "cpu",
|
||||
Tags: map[string]string{"host": "serverA"},
|
||||
}
|
||||
|
||||
c := newRemoteShardResponder([]*tsdb.MapperOutput{expOutput, nil}, expTagSets)
|
||||
|
||||
r := NewRemoteMapper(c, 1234, "SELECT * FROM CPU", 10)
|
||||
if err := r.Open(); err != nil {
|
||||
t.Fatalf("failed to open remote mapper: %s", err.Error())
|
||||
}
|
||||
|
||||
if r.TagSets()[0] != expTagSets[0] {
|
||||
t.Fatalf("incorrect tagsets received, exp %v, got %v", expTagSets, r.TagSets())
|
||||
}
|
||||
|
||||
// Get first chunk from mapper.
|
||||
chunk, err := r.NextChunk()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to get next chunk from mapper: %s", err.Error())
|
||||
}
|
||||
output, ok := chunk.(*tsdb.MapperOutput)
|
||||
if !ok {
|
||||
t.Fatal("chunk is not of expected type")
|
||||
}
|
||||
if output.Name != "cpu" {
|
||||
t.Fatalf("received output incorrect, exp: %v, got %v", expOutput, output)
|
||||
}
|
||||
|
||||
// Next chunk should be nil, indicating no more data.
|
||||
chunk, err = r.NextChunk()
|
||||
if err != nil {
|
||||
t.Fatalf("failed to get next chunk from mapper: %s", err.Error())
|
||||
}
|
||||
if chunk != nil {
|
||||
t.Fatal("received more chunks when none expected")
|
||||
}
|
||||
}
|
||||
163
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_writer.go
generated
vendored
163
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_writer.go
generated
vendored
@@ -1,163 +0,0 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
"gopkg.in/fatih/pool.v2"
|
||||
)
|
||||
|
||||
const (
|
||||
writeShardRequestMessage byte = iota + 1
|
||||
writeShardResponseMessage
|
||||
mapShardRequestMessage
|
||||
mapShardResponseMessage
|
||||
)
|
||||
|
||||
// ShardWriter writes a set of points to a shard.
|
||||
type ShardWriter struct {
|
||||
pool *clientPool
|
||||
timeout time.Duration
|
||||
|
||||
MetaStore interface {
|
||||
Node(id uint64) (ni *meta.NodeInfo, err error)
|
||||
}
|
||||
}
|
||||
|
||||
// NewShardWriter returns a new instance of ShardWriter.
|
||||
func NewShardWriter(timeout time.Duration) *ShardWriter {
|
||||
return &ShardWriter{
|
||||
pool: newClientPool(),
|
||||
timeout: timeout,
|
||||
}
|
||||
}
|
||||
|
||||
func (w *ShardWriter) WriteShard(shardID, ownerID uint64, points []tsdb.Point) error {
|
||||
c, err := w.dial(ownerID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
conn, ok := c.(*pool.PoolConn)
|
||||
if !ok {
|
||||
panic("wrong connection type")
|
||||
}
|
||||
defer func(conn net.Conn) {
|
||||
conn.Close() // return to pool
|
||||
}(conn)
|
||||
|
||||
// Build write request.
|
||||
var request WriteShardRequest
|
||||
request.SetShardID(shardID)
|
||||
request.AddPoints(points)
|
||||
|
||||
// Marshal into protocol buffers.
|
||||
buf, err := request.MarshalBinary()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Write request.
|
||||
conn.SetWriteDeadline(time.Now().Add(w.timeout))
|
||||
if err := WriteTLV(conn, writeShardRequestMessage, buf); err != nil {
|
||||
conn.MarkUnusable()
|
||||
return err
|
||||
}
|
||||
|
||||
// Read the response.
|
||||
conn.SetReadDeadline(time.Now().Add(w.timeout))
|
||||
_, buf, err = ReadTLV(conn)
|
||||
if err != nil {
|
||||
conn.MarkUnusable()
|
||||
return err
|
||||
}
|
||||
|
||||
// Unmarshal response.
|
||||
var response WriteShardResponse
|
||||
if err := response.UnmarshalBinary(buf); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if response.Code() != 0 {
|
||||
return fmt.Errorf("error code %d: %s", response.Code(), response.Message())
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *ShardWriter) dial(nodeID uint64) (net.Conn, error) {
|
||||
// If we don't have a connection pool for that addr yet, create one
|
||||
_, ok := c.pool.getPool(nodeID)
|
||||
if !ok {
|
||||
factory := &connFactory{nodeID: nodeID, clientPool: c.pool, timeout: c.timeout}
|
||||
factory.metaStore = c.MetaStore
|
||||
|
||||
p, err := pool.NewChannelPool(1, 3, factory.dial)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.pool.setPool(nodeID, p)
|
||||
}
|
||||
return c.pool.conn(nodeID)
|
||||
}
|
||||
|
||||
func (w *ShardWriter) Close() error {
|
||||
if w.pool == nil {
|
||||
return fmt.Errorf("client already closed")
|
||||
}
|
||||
w.pool.close()
|
||||
w.pool = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
const (
|
||||
maxConnections = 500
|
||||
maxRetries = 3
|
||||
)
|
||||
|
||||
var errMaxConnectionsExceeded = fmt.Errorf("can not exceed max connections of %d", maxConnections)
|
||||
|
||||
type connFactory struct {
|
||||
nodeID uint64
|
||||
timeout time.Duration
|
||||
|
||||
clientPool interface {
|
||||
size() int
|
||||
}
|
||||
|
||||
metaStore interface {
|
||||
Node(id uint64) (ni *meta.NodeInfo, err error)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *connFactory) dial() (net.Conn, error) {
|
||||
if c.clientPool.size() > maxConnections {
|
||||
return nil, errMaxConnectionsExceeded
|
||||
}
|
||||
|
||||
ni, err := c.metaStore.Node(c.nodeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if ni == nil {
|
||||
return nil, fmt.Errorf("node %d does not exist", c.nodeID)
|
||||
}
|
||||
|
||||
conn, err := net.DialTimeout("tcp", ni.Host, c.timeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Write a marker byte for cluster messages.
|
||||
_, err = conn.Write([]byte{MuxHeader})
|
||||
if err != nil {
|
||||
conn.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return conn, nil
|
||||
}
|
||||
186
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_writer_test.go
generated
vendored
186
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_writer_test.go
generated
vendored
@@ -1,186 +0,0 @@
|
||||
package cluster_test
|
||||
|
||||
import (
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cluster"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// Ensure the shard writer can successful write a single request.
|
||||
func TestShardWriter_WriteShard_Success(t *testing.T) {
|
||||
ts := newTestWriteService(writeShardSuccess)
|
||||
s := cluster.NewService(cluster.Config{})
|
||||
s.Listener = ts.muxln
|
||||
s.TSDBStore = ts
|
||||
if err := s.Open(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
defer ts.Close()
|
||||
|
||||
w := cluster.NewShardWriter(time.Minute)
|
||||
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
|
||||
|
||||
// Build a single point.
|
||||
now := time.Now()
|
||||
var points []tsdb.Point
|
||||
points = append(points, tsdb.NewPoint("cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now))
|
||||
|
||||
// Write to shard and close.
|
||||
if err := w.WriteShard(1, 2, points); err != nil {
|
||||
t.Fatal(err)
|
||||
} else if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Validate response.
|
||||
responses, err := ts.ResponseN(1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
} else if responses[0].shardID != 1 {
|
||||
t.Fatalf("unexpected shard id: %d", responses[0].shardID)
|
||||
}
|
||||
|
||||
// Validate point.
|
||||
if p := responses[0].points[0]; p.Name() != "cpu" {
|
||||
t.Fatalf("unexpected name: %s", p.Name())
|
||||
} else if p.Fields()["value"] != int64(100) {
|
||||
t.Fatalf("unexpected 'value' field: %d", p.Fields()["value"])
|
||||
} else if p.Tags()["host"] != "server01" {
|
||||
t.Fatalf("unexpected 'host' tag: %s", p.Tags()["host"])
|
||||
} else if p.Time().UnixNano() != now.UnixNano() {
|
||||
t.Fatalf("unexpected time: %s", p.Time())
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the shard writer can successful write a multiple requests.
|
||||
func TestShardWriter_WriteShard_Multiple(t *testing.T) {
|
||||
ts := newTestWriteService(writeShardSuccess)
|
||||
s := cluster.NewService(cluster.Config{})
|
||||
s.Listener = ts.muxln
|
||||
s.TSDBStore = ts
|
||||
if err := s.Open(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
defer ts.Close()
|
||||
|
||||
w := cluster.NewShardWriter(time.Minute)
|
||||
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
|
||||
|
||||
// Build a single point.
|
||||
now := time.Now()
|
||||
var points []tsdb.Point
|
||||
points = append(points, tsdb.NewPoint("cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now))
|
||||
|
||||
// Write to shard twice and close.
|
||||
if err := w.WriteShard(1, 2, points); err != nil {
|
||||
t.Fatal(err)
|
||||
} else if err := w.WriteShard(1, 2, points); err != nil {
|
||||
t.Fatal(err)
|
||||
} else if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Validate response.
|
||||
responses, err := ts.ResponseN(1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
} else if responses[0].shardID != 1 {
|
||||
t.Fatalf("unexpected shard id: %d", responses[0].shardID)
|
||||
}
|
||||
|
||||
// Validate point.
|
||||
if p := responses[0].points[0]; p.Name() != "cpu" {
|
||||
t.Fatalf("unexpected name: %s", p.Name())
|
||||
} else if p.Fields()["value"] != int64(100) {
|
||||
t.Fatalf("unexpected 'value' field: %d", p.Fields()["value"])
|
||||
} else if p.Tags()["host"] != "server01" {
|
||||
t.Fatalf("unexpected 'host' tag: %s", p.Tags()["host"])
|
||||
} else if p.Time().UnixNano() != now.UnixNano() {
|
||||
t.Fatalf("unexpected time: %s", p.Time())
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the shard writer returns an error when the server fails to accept the write.
|
||||
func TestShardWriter_WriteShard_Error(t *testing.T) {
|
||||
ts := newTestWriteService(writeShardFail)
|
||||
s := cluster.NewService(cluster.Config{})
|
||||
s.Listener = ts.muxln
|
||||
s.TSDBStore = ts
|
||||
if err := s.Open(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
defer ts.Close()
|
||||
|
||||
w := cluster.NewShardWriter(time.Minute)
|
||||
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
|
||||
now := time.Now()
|
||||
|
||||
shardID := uint64(1)
|
||||
ownerID := uint64(2)
|
||||
var points []tsdb.Point
|
||||
points = append(points, tsdb.NewPoint(
|
||||
"cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now,
|
||||
))
|
||||
|
||||
if err := w.WriteShard(shardID, ownerID, points); err == nil || err.Error() != "error code 1: write shard 1: failed to write" {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the shard writer returns an error when dialing times out.
|
||||
func TestShardWriter_Write_ErrDialTimeout(t *testing.T) {
|
||||
ts := newTestWriteService(writeShardSuccess)
|
||||
s := cluster.NewService(cluster.Config{})
|
||||
s.Listener = ts.muxln
|
||||
s.TSDBStore = ts
|
||||
if err := s.Open(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
defer ts.Close()
|
||||
|
||||
w := cluster.NewShardWriter(time.Nanosecond)
|
||||
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
|
||||
now := time.Now()
|
||||
|
||||
shardID := uint64(1)
|
||||
ownerID := uint64(2)
|
||||
var points []tsdb.Point
|
||||
points = append(points, tsdb.NewPoint(
|
||||
"cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now,
|
||||
))
|
||||
|
||||
if err, exp := w.WriteShard(shardID, ownerID, points), "i/o timeout"; err == nil || !strings.Contains(err.Error(), exp) {
|
||||
t.Fatalf("expected error %v, to contain %s", err, exp)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the shard writer returns an error when reading times out.
|
||||
func TestShardWriter_Write_ErrReadTimeout(t *testing.T) {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
w := cluster.NewShardWriter(time.Millisecond)
|
||||
w.MetaStore = &metaStore{host: ln.Addr().String()}
|
||||
now := time.Now()
|
||||
|
||||
shardID := uint64(1)
|
||||
ownerID := uint64(2)
|
||||
var points []tsdb.Point
|
||||
points = append(points, tsdb.NewPoint(
|
||||
"cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now,
|
||||
))
|
||||
|
||||
if err := w.WriteShard(shardID, ownerID, points); err == nil || !strings.Contains(err.Error(), "i/o timeout") {
|
||||
t.Fatalf("unexpected error: %s", err)
|
||||
}
|
||||
}
|
||||
724
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx/main.go
generated
vendored
724
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx/main.go
generated
vendored
@@ -1,724 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
|
||||
"github.com/influxdb/influxdb/client"
|
||||
"github.com/influxdb/influxdb/importer/v8"
|
||||
"github.com/peterh/liner"
|
||||
)
|
||||
|
||||
// These variables are populated via the Go linker.
|
||||
var (
|
||||
version string = "0.9"
|
||||
)
|
||||
|
||||
const (
|
||||
// defaultFormat is the default format of the results when issuing queries
|
||||
defaultFormat = "column"
|
||||
|
||||
// defaultPPS is the default points per second that the import will throttle at
|
||||
// by default it's 0, which means it will not throttle
|
||||
defaultPPS = 0
|
||||
)
|
||||
|
||||
type CommandLine struct {
|
||||
Client *client.Client
|
||||
Line *liner.State
|
||||
Host string
|
||||
Port int
|
||||
Username string
|
||||
Password string
|
||||
Database string
|
||||
Ssl bool
|
||||
RetentionPolicy string
|
||||
Version string
|
||||
Pretty bool // controls pretty print for json
|
||||
Format string // controls the output format. Valid values are json, csv, or column
|
||||
Execute string
|
||||
ShowVersion bool
|
||||
Import bool
|
||||
PPS int // Controls how many points per second the import will allow via throttling
|
||||
Path string
|
||||
Compressed bool
|
||||
}
|
||||
|
||||
func main() {
|
||||
c := CommandLine{}
|
||||
|
||||
fs := flag.NewFlagSet("InfluxDB shell version "+version, flag.ExitOnError)
|
||||
fs.StringVar(&c.Host, "host", client.DefaultHost, "Influxdb host to connect to.")
|
||||
fs.IntVar(&c.Port, "port", client.DefaultPort, "Influxdb port to connect to.")
|
||||
fs.StringVar(&c.Username, "username", c.Username, "Username to connect to the server.")
|
||||
fs.StringVar(&c.Password, "password", c.Password, `Password to connect to the server. Leaving blank will prompt for password (--password="").`)
|
||||
fs.StringVar(&c.Database, "database", c.Database, "Database to connect to the server.")
|
||||
fs.BoolVar(&c.Ssl, "ssl", false, "Use https for connecting to cluster.")
|
||||
fs.StringVar(&c.Format, "format", defaultFormat, "Format specifies the format of the server responses: json, csv, or column.")
|
||||
fs.BoolVar(&c.Pretty, "pretty", false, "Turns on pretty print for the json format.")
|
||||
fs.StringVar(&c.Execute, "execute", c.Execute, "Execute command and quit.")
|
||||
fs.BoolVar(&c.ShowVersion, "version", false, "Displays the InfluxDB version.")
|
||||
fs.BoolVar(&c.Import, "import", false, "Import a previous database.")
|
||||
fs.IntVar(&c.PPS, "pps", defaultPPS, "How many points per second the import will allow. By default it is zero and will not throttle importing.")
|
||||
fs.StringVar(&c.Path, "path", "", "path to the file to import")
|
||||
fs.BoolVar(&c.Compressed, "compressed", false, "set to true if the import file is compressed")
|
||||
|
||||
// Define our own custom usage to print
|
||||
fs.Usage = func() {
|
||||
fmt.Println(`Usage of influx:
|
||||
-version
|
||||
Display the version and exit.
|
||||
-host 'host name'
|
||||
Host to connect to.
|
||||
-port 'port #'
|
||||
Port to connect to.
|
||||
-database 'database name'
|
||||
Database to connect to the server.
|
||||
-password 'password'
|
||||
Password to connect to the server. Leaving blank will prompt for password (--password '').
|
||||
-username 'username'
|
||||
Username to connect to the server.
|
||||
-ssl
|
||||
Use https for requests.
|
||||
-execute 'command'
|
||||
Execute command and quit.
|
||||
-format 'json|csv|column'
|
||||
Format specifies the format of the server responses: json, csv, or column.
|
||||
-pretty
|
||||
Turns on pretty print for the json format.
|
||||
-import
|
||||
Import a previous database export from file
|
||||
-pps
|
||||
How many points per second the import will allow. By default it is zero and will not throttle importing.
|
||||
-path
|
||||
Path to file to import
|
||||
-compressed
|
||||
Set to true if the import file is compressed
|
||||
|
||||
Examples:
|
||||
|
||||
# Use influx in a non-interactive mode to query the database "metrics" and pretty print json:
|
||||
$ influx -database 'metrics' -execute 'select * from cpu' -format 'json' -pretty
|
||||
|
||||
# Connect to a specific database on startup and set database context:
|
||||
$ influx -database 'metrics' -host 'localhost' -port '8086'
|
||||
`)
|
||||
}
|
||||
fs.Parse(os.Args[1:])
|
||||
|
||||
if c.ShowVersion {
|
||||
showVersion()
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
var promptForPassword bool
|
||||
// determine if they set the password flag but provided no value
|
||||
for _, v := range os.Args {
|
||||
v = strings.ToLower(v)
|
||||
if (strings.HasPrefix(v, "-password") || strings.HasPrefix(v, "--password")) && c.Password == "" {
|
||||
promptForPassword = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
c.Line = liner.NewLiner()
|
||||
defer c.Line.Close()
|
||||
|
||||
if promptForPassword {
|
||||
p, e := c.Line.PasswordPrompt("password: ")
|
||||
if e != nil {
|
||||
fmt.Println("Unable to parse password.")
|
||||
} else {
|
||||
c.Password = p
|
||||
}
|
||||
}
|
||||
|
||||
if err := c.connect(""); err != nil {
|
||||
|
||||
}
|
||||
if c.Execute == "" && !c.Import {
|
||||
fmt.Printf("Connected to %s version %s\n", c.Client.Addr(), c.Version)
|
||||
}
|
||||
|
||||
if c.Execute != "" {
|
||||
if err := c.ExecuteQuery(c.Execute); err != nil {
|
||||
c.Line.Close()
|
||||
os.Exit(1)
|
||||
}
|
||||
c.Line.Close()
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
if c.Import {
|
||||
path := net.JoinHostPort(c.Host, strconv.Itoa(c.Port))
|
||||
u, e := client.ParseConnectionString(path, c.Ssl)
|
||||
if e != nil {
|
||||
fmt.Println(e)
|
||||
return
|
||||
}
|
||||
|
||||
config := v8.NewConfig()
|
||||
config.Username = c.Username
|
||||
config.Password = c.Password
|
||||
config.Precision = "ns"
|
||||
config.WriteConsistency = "any"
|
||||
config.Path = c.Path
|
||||
config.Version = version
|
||||
config.URL = u
|
||||
config.Compressed = c.Compressed
|
||||
config.PPS = c.PPS
|
||||
|
||||
i := v8.NewImporter(config)
|
||||
if err := i.Import(); err != nil {
|
||||
fmt.Printf("ERROR: %s\n", err)
|
||||
c.Line.Close()
|
||||
os.Exit(1)
|
||||
}
|
||||
c.Line.Close()
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
showVersion()
|
||||
|
||||
var historyFile string
|
||||
usr, err := user.Current()
|
||||
// Only load history if we can get the user
|
||||
if err == nil {
|
||||
historyFile = filepath.Join(usr.HomeDir, ".influx_history")
|
||||
|
||||
if f, err := os.Open(historyFile); err == nil {
|
||||
c.Line.ReadHistory(f)
|
||||
f.Close()
|
||||
}
|
||||
}
|
||||
|
||||
for {
|
||||
l, e := c.Line.Prompt("> ")
|
||||
if e != nil {
|
||||
break
|
||||
}
|
||||
if c.ParseCommand(l) {
|
||||
// write out the history
|
||||
if len(historyFile) > 0 {
|
||||
c.Line.AppendHistory(l)
|
||||
if f, err := os.Create(historyFile); err == nil {
|
||||
c.Line.WriteHistory(f)
|
||||
f.Close()
|
||||
}
|
||||
}
|
||||
} else {
|
||||
break // exit main loop
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func showVersion() {
|
||||
fmt.Println("InfluxDB shell " + version)
|
||||
}
|
||||
|
||||
func (c *CommandLine) ParseCommand(cmd string) bool {
|
||||
lcmd := strings.TrimSpace(strings.ToLower(cmd))
|
||||
switch {
|
||||
case strings.HasPrefix(lcmd, "exit"):
|
||||
// signal the program to exit
|
||||
return false
|
||||
case strings.HasPrefix(lcmd, "gopher"):
|
||||
c.gopher()
|
||||
case strings.HasPrefix(lcmd, "connect"):
|
||||
c.connect(cmd)
|
||||
case strings.HasPrefix(lcmd, "auth"):
|
||||
c.SetAuth(cmd)
|
||||
case strings.HasPrefix(lcmd, "help"):
|
||||
c.help()
|
||||
case strings.HasPrefix(lcmd, "format"):
|
||||
c.SetFormat(cmd)
|
||||
case strings.HasPrefix(lcmd, "settings"):
|
||||
c.Settings()
|
||||
case strings.HasPrefix(lcmd, "pretty"):
|
||||
c.Pretty = !c.Pretty
|
||||
if c.Pretty {
|
||||
fmt.Println("Pretty print enabled")
|
||||
} else {
|
||||
fmt.Println("Pretty print disabled")
|
||||
}
|
||||
case strings.HasPrefix(lcmd, "use"):
|
||||
c.use(cmd)
|
||||
case strings.HasPrefix(lcmd, "insert"):
|
||||
c.Insert(cmd)
|
||||
case lcmd == "":
|
||||
break
|
||||
default:
|
||||
c.ExecuteQuery(cmd)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (c *CommandLine) connect(cmd string) error {
|
||||
var cl *client.Client
|
||||
var u url.URL
|
||||
|
||||
// Remove the "connect" keyword if it exists
|
||||
path := strings.TrimSpace(strings.Replace(cmd, "connect", "", -1))
|
||||
|
||||
// If they didn't provide a connection string, use the current settings
|
||||
if path == "" {
|
||||
path = net.JoinHostPort(c.Host, strconv.Itoa(c.Port))
|
||||
}
|
||||
|
||||
var e error
|
||||
u, e = client.ParseConnectionString(path, c.Ssl)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
|
||||
config := client.NewConfig()
|
||||
config.URL = u
|
||||
config.Username = c.Username
|
||||
config.Password = c.Password
|
||||
config.UserAgent = "InfluxDBShell/" + version
|
||||
cl, err := client.NewClient(config)
|
||||
if err != nil {
|
||||
return fmt.Errorf("Could not create client %s", err)
|
||||
}
|
||||
c.Client = cl
|
||||
if _, v, e := c.Client.Ping(); e != nil {
|
||||
return fmt.Errorf("Failed to connect to %s\n", c.Client.Addr())
|
||||
} else {
|
||||
c.Version = v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *CommandLine) SetAuth(cmd string) {
|
||||
// If they pass in the entire command, we should parse it
|
||||
// auth <username> <password>
|
||||
args := strings.Fields(cmd)
|
||||
if len(args) == 3 {
|
||||
args = args[1:]
|
||||
} else {
|
||||
args = []string{}
|
||||
}
|
||||
|
||||
if len(args) == 2 {
|
||||
c.Username = args[0]
|
||||
c.Password = args[1]
|
||||
} else {
|
||||
u, e := c.Line.Prompt("username: ")
|
||||
if e != nil {
|
||||
fmt.Printf("Unable to process input: %s", e)
|
||||
return
|
||||
}
|
||||
c.Username = strings.TrimSpace(u)
|
||||
p, e := c.Line.PasswordPrompt("password: ")
|
||||
if e != nil {
|
||||
fmt.Printf("Unable to process input: %s", e)
|
||||
return
|
||||
}
|
||||
c.Password = p
|
||||
}
|
||||
|
||||
// Update the client as well
|
||||
c.Client.SetAuth(c.Username, c.Password)
|
||||
}
|
||||
|
||||
func (c *CommandLine) use(cmd string) {
|
||||
args := strings.Split(strings.TrimSuffix(strings.TrimSpace(cmd), ";"), " ")
|
||||
if len(args) != 2 {
|
||||
fmt.Printf("Could not parse database name from %q.\n", cmd)
|
||||
return
|
||||
}
|
||||
d := args[1]
|
||||
c.Database = d
|
||||
fmt.Printf("Using database %s\n", d)
|
||||
}
|
||||
|
||||
func (c *CommandLine) SetFormat(cmd string) {
|
||||
// Remove the "format" keyword if it exists
|
||||
cmd = strings.TrimSpace(strings.Replace(cmd, "format", "", -1))
|
||||
// normalize cmd
|
||||
cmd = strings.ToLower(cmd)
|
||||
|
||||
switch cmd {
|
||||
case "json", "csv", "column":
|
||||
c.Format = cmd
|
||||
default:
|
||||
fmt.Printf("Unknown format %q. Please use json, csv, or column.\n", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// isWhitespace returns true if the rune is a space, tab, or newline.
|
||||
func isWhitespace(ch rune) bool { return ch == ' ' || ch == '\t' || ch == '\n' }
|
||||
|
||||
// isLetter returns true if the rune is a letter.
|
||||
func isLetter(ch rune) bool { return (ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') }
|
||||
|
||||
// isDigit returns true if the rune is a digit.
|
||||
func isDigit(ch rune) bool { return (ch >= '0' && ch <= '9') }
|
||||
|
||||
// isIdentFirstChar returns true if the rune can be used as the first char in an unquoted identifer.
|
||||
func isIdentFirstChar(ch rune) bool { return isLetter(ch) || ch == '_' }
|
||||
|
||||
// isIdentChar returns true if the rune can be used in an unquoted identifier.
|
||||
func isNotIdentChar(ch rune) bool { return !(isLetter(ch) || isDigit(ch) || ch == '_') }
|
||||
|
||||
func parseUnquotedIdentifier(stmt string) (string, string) {
|
||||
if fields := strings.FieldsFunc(stmt, isNotIdentChar); len(fields) > 0 {
|
||||
return fields[0], strings.TrimPrefix(stmt, fields[0])
|
||||
}
|
||||
return "", stmt
|
||||
}
|
||||
|
||||
func parseDoubleQuotedIdentifier(stmt string) (string, string) {
|
||||
escapeNext := false
|
||||
fields := strings.FieldsFunc(stmt, func(ch rune) bool {
|
||||
if ch == '\\' {
|
||||
escapeNext = true
|
||||
} else if ch == '"' {
|
||||
if !escapeNext {
|
||||
return true
|
||||
}
|
||||
escapeNext = false
|
||||
}
|
||||
return false
|
||||
})
|
||||
if len(fields) > 0 {
|
||||
return fields[0], strings.TrimPrefix(stmt, "\""+fields[0]+"\"")
|
||||
}
|
||||
return "", stmt
|
||||
}
|
||||
|
||||
func parseNextIdentifier(stmt string) (ident, remainder string) {
|
||||
if len(stmt) > 0 {
|
||||
switch {
|
||||
case isWhitespace(rune(stmt[0])):
|
||||
return parseNextIdentifier(stmt[1:])
|
||||
case isIdentFirstChar(rune(stmt[0])):
|
||||
return parseUnquotedIdentifier(stmt)
|
||||
case stmt[0] == '"':
|
||||
return parseDoubleQuotedIdentifier(stmt)
|
||||
}
|
||||
}
|
||||
return "", stmt
|
||||
}
|
||||
|
||||
func (c *CommandLine) parseInto(stmt string) string {
|
||||
ident, stmt := parseNextIdentifier(stmt)
|
||||
if strings.HasPrefix(stmt, ".") {
|
||||
c.Database = ident
|
||||
fmt.Printf("Using database %s\n", c.Database)
|
||||
ident, stmt = parseNextIdentifier(stmt[1:])
|
||||
}
|
||||
if strings.HasPrefix(stmt, " ") {
|
||||
c.RetentionPolicy = ident
|
||||
fmt.Printf("Using retention policy %s\n", c.RetentionPolicy)
|
||||
return stmt[1:]
|
||||
}
|
||||
return stmt
|
||||
}
|
||||
|
||||
func (c *CommandLine) Insert(stmt string) error {
|
||||
i, point := parseNextIdentifier(stmt)
|
||||
if !strings.EqualFold(i, "insert") {
|
||||
fmt.Printf("ERR: found %s, expected INSERT\n", i)
|
||||
return nil
|
||||
}
|
||||
if i, r := parseNextIdentifier(point); strings.EqualFold(i, "into") {
|
||||
point = c.parseInto(r)
|
||||
}
|
||||
_, err := c.Client.Write(client.BatchPoints{
|
||||
Points: []client.Point{
|
||||
client.Point{Raw: point},
|
||||
},
|
||||
Database: c.Database,
|
||||
RetentionPolicy: c.RetentionPolicy,
|
||||
Precision: "n",
|
||||
WriteConsistency: client.ConsistencyAny,
|
||||
})
|
||||
if err != nil {
|
||||
fmt.Printf("ERR: %s\n", err)
|
||||
if c.Database == "" {
|
||||
fmt.Println("Note: error may be due to not setting a database or retention policy.")
|
||||
fmt.Println(`Please set a database with the command "use <database>" or`)
|
||||
fmt.Println("INSERT INTO <database>.<retention-policy> <point>")
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *CommandLine) ExecuteQuery(query string) error {
|
||||
response, err := c.Client.Query(client.Query{Command: query, Database: c.Database})
|
||||
if err != nil {
|
||||
fmt.Printf("ERR: %s\n", err)
|
||||
return err
|
||||
}
|
||||
c.FormatResponse(response, os.Stdout)
|
||||
if err := response.Error(); err != nil {
|
||||
fmt.Printf("ERR: %s\n", response.Error())
|
||||
if c.Database == "" {
|
||||
fmt.Println("Warning: It is possible this error is due to not setting a database.")
|
||||
fmt.Println(`Please set a database with the command "use <database>".`)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *CommandLine) FormatResponse(response *client.Response, w io.Writer) {
|
||||
switch c.Format {
|
||||
case "json":
|
||||
c.writeJSON(response, w)
|
||||
case "csv":
|
||||
c.writeCSV(response, w)
|
||||
case "column":
|
||||
c.writeColumns(response, w)
|
||||
default:
|
||||
fmt.Fprintf(w, "Unknown output format %q.\n", c.Format)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *CommandLine) writeJSON(response *client.Response, w io.Writer) {
|
||||
var data []byte
|
||||
var err error
|
||||
if c.Pretty {
|
||||
data, err = json.MarshalIndent(response, "", " ")
|
||||
} else {
|
||||
data, err = json.Marshal(response)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintf(w, "Unable to parse json: %s\n", err)
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(w, string(data))
|
||||
}
|
||||
|
||||
func (c *CommandLine) writeCSV(response *client.Response, w io.Writer) {
|
||||
csvw := csv.NewWriter(w)
|
||||
for _, result := range response.Results {
|
||||
// Create a tabbed writer for each result as they won't always line up
|
||||
rows := c.formatResults(result, "\t")
|
||||
for _, r := range rows {
|
||||
csvw.Write(strings.Split(r, "\t"))
|
||||
}
|
||||
csvw.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
func (c *CommandLine) writeColumns(response *client.Response, w io.Writer) {
|
||||
for _, result := range response.Results {
|
||||
// Create a tabbed writer for each result a they won't always line up
|
||||
w := new(tabwriter.Writer)
|
||||
w.Init(os.Stdout, 0, 8, 1, '\t', 0)
|
||||
csv := c.formatResults(result, "\t")
|
||||
for _, r := range csv {
|
||||
fmt.Fprintln(w, r)
|
||||
}
|
||||
w.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
// formatResults will behave differently if you are formatting for columns or csv
|
||||
func (c *CommandLine) formatResults(result client.Result, separator string) []string {
|
||||
rows := []string{}
|
||||
// Create a tabbed writer for each result a they won't always line up
|
||||
for i, row := range result.Series {
|
||||
// gather tags
|
||||
tags := []string{}
|
||||
for k, v := range row.Tags {
|
||||
tags = append(tags, fmt.Sprintf("%s=%s", k, v))
|
||||
sort.Strings(tags)
|
||||
}
|
||||
|
||||
columnNames := []string{}
|
||||
|
||||
// Only put name/tags in a column if format is csv
|
||||
if c.Format == "csv" {
|
||||
if len(tags) > 0 {
|
||||
columnNames = append([]string{"tags"}, columnNames...)
|
||||
}
|
||||
|
||||
if row.Name != "" {
|
||||
columnNames = append([]string{"name"}, columnNames...)
|
||||
}
|
||||
}
|
||||
|
||||
for _, column := range row.Columns {
|
||||
columnNames = append(columnNames, column)
|
||||
}
|
||||
|
||||
// Output a line separator if we have more than one set or results and format is column
|
||||
if i > 0 && c.Format == "column" {
|
||||
rows = append(rows, "")
|
||||
}
|
||||
|
||||
// If we are column format, we break out the name/tag to seperate lines
|
||||
if c.Format == "column" {
|
||||
if row.Name != "" {
|
||||
n := fmt.Sprintf("name: %s", row.Name)
|
||||
rows = append(rows, n)
|
||||
if len(tags) == 0 {
|
||||
l := strings.Repeat("-", len(n))
|
||||
rows = append(rows, l)
|
||||
}
|
||||
}
|
||||
if len(tags) > 0 {
|
||||
t := fmt.Sprintf("tags: %s", (strings.Join(tags, ", ")))
|
||||
rows = append(rows, t)
|
||||
}
|
||||
}
|
||||
|
||||
rows = append(rows, strings.Join(columnNames, separator))
|
||||
|
||||
// if format is column, break tags to their own line/format
|
||||
if c.Format == "column" && len(tags) > 0 {
|
||||
lines := []string{}
|
||||
for _, columnName := range columnNames {
|
||||
lines = append(lines, strings.Repeat("-", len(columnName)))
|
||||
}
|
||||
rows = append(rows, strings.Join(lines, separator))
|
||||
}
|
||||
|
||||
for _, v := range row.Values {
|
||||
var values []string
|
||||
if c.Format == "csv" {
|
||||
if row.Name != "" {
|
||||
values = append(values, row.Name)
|
||||
}
|
||||
if len(tags) > 0 {
|
||||
values = append(values, strings.Join(tags, ","))
|
||||
}
|
||||
}
|
||||
|
||||
for _, vv := range v {
|
||||
values = append(values, interfaceToString(vv))
|
||||
}
|
||||
rows = append(rows, strings.Join(values, separator))
|
||||
}
|
||||
// Outout a line separator if in column format
|
||||
if c.Format == "column" {
|
||||
rows = append(rows, "")
|
||||
}
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func interfaceToString(v interface{}) string {
|
||||
switch t := v.(type) {
|
||||
case nil:
|
||||
return ""
|
||||
case bool:
|
||||
return fmt.Sprintf("%v", v)
|
||||
case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, uintptr:
|
||||
return fmt.Sprintf("%d", t)
|
||||
case float32, float64:
|
||||
return fmt.Sprintf("%v", t)
|
||||
default:
|
||||
return fmt.Sprintf("%v", t)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *CommandLine) Settings() {
|
||||
w := new(tabwriter.Writer)
|
||||
w.Init(os.Stdout, 0, 8, 1, '\t', 0)
|
||||
if c.Port > 0 {
|
||||
fmt.Fprintf(w, "Host\t%s:%d\n", c.Host, c.Port)
|
||||
} else {
|
||||
fmt.Fprintf(w, "Host\t%s\n", c.Host)
|
||||
}
|
||||
fmt.Fprintf(w, "Username\t%s\n", c.Username)
|
||||
fmt.Fprintf(w, "Database\t%s\n", c.Database)
|
||||
fmt.Fprintf(w, "Pretty\t%v\n", c.Pretty)
|
||||
fmt.Fprintf(w, "Format\t%s\n", c.Format)
|
||||
fmt.Fprintln(w)
|
||||
w.Flush()
|
||||
}
|
||||
|
||||
func (c *CommandLine) help() {
|
||||
fmt.Println(`Usage:
|
||||
connect <host:port> connect to another node
|
||||
auth prompt for username and password
|
||||
pretty toggle pretty print
|
||||
use <db_name> set current databases
|
||||
format <format> set the output format: json, csv, or column
|
||||
settings output the current settings for the shell
|
||||
exit quit the influx shell
|
||||
|
||||
show databases show database names
|
||||
show series show series information
|
||||
show measurements show measurement information
|
||||
show tag keys show tag key information
|
||||
show tag values show tag value information
|
||||
|
||||
a full list of influxql commands can be found at:
|
||||
https://influxdb.com/docs/v0.9/query_language/spec.html
|
||||
`)
|
||||
}
|
||||
|
||||
func (c *CommandLine) gopher() {
|
||||
fmt.Println(`
|
||||
.-::-::://:-::- .:/++/'
|
||||
'://:-''/oo+//++o+/.://o- ./+:
|
||||
.:-. '++- .o/ '+yydhy' o-
|
||||
.:/. .h: :osoys .smMN- :/
|
||||
-/:.' s- /MMMymh. '/y/ s'
|
||||
-+s:'''' d -mMMms// '-/o:
|
||||
-/++/++/////:. o: '... s- :s.
|
||||
:+-+s-' ':/' 's- /+ 'o:
|
||||
'+-'o: /ydhsh. '//. '-o- o-
|
||||
.y. o: .MMMdm+y ':+++:::/+:.' s:
|
||||
.-h/ y- 'sdmds'h -+ydds:::-.' 'h.
|
||||
.//-.d' o: '.' 'dsNMMMNh:.:++' :y
|
||||
+y. 'd 's. .s:mddds: ++ o/
|
||||
'N- odd 'o/. './o-s-' .---+++' o-
|
||||
'N' yNd .://:/:::::. -s -+/s/./s' 'o/'
|
||||
so' .h '''' ////s: '+. .s +y'
|
||||
os/-.y' 's' 'y::+ +d'
|
||||
'.:o/ -+:-:.' so.---.'
|
||||
o' 'd-.''/s'
|
||||
.s' :y.''.y
|
||||
-s mo:::'
|
||||
:: yh
|
||||
// '''' /M'
|
||||
o+ .s///:/. 'N:
|
||||
:+ /: -s' ho
|
||||
's- -/s/:+/.+h' +h
|
||||
ys' ':' '-. -d
|
||||
oh .h
|
||||
/o .s
|
||||
s. .h
|
||||
-y .d
|
||||
m/ -h
|
||||
+d /o
|
||||
'N- y:
|
||||
h: m.
|
||||
s- -d
|
||||
o- s+
|
||||
+- 'm'
|
||||
s/ oo--.
|
||||
y- /s ':+'
|
||||
s' 'od--' .d:
|
||||
-+ ':o: ':+-/+
|
||||
y- .:+- '
|
||||
//o- '.:+/.
|
||||
.-:+/' ''-/+/.
|
||||
./:' ''.:o+/-'
|
||||
.+o:/:/+-' ''.-+ooo/-'
|
||||
o: -h///++////-.
|
||||
/: .o/
|
||||
//+ 'y
|
||||
./sooy.
|
||||
|
||||
`)
|
||||
}
|
||||
194
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx/main_test.go
generated
vendored
194
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx/main_test.go
generated
vendored
@@ -1,194 +0,0 @@
|
||||
package main_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
|
||||
"github.com/influxdb/influxdb/client"
|
||||
main "github.com/influxdb/influxdb/cmd/influx"
|
||||
)
|
||||
|
||||
func TestParseCommand_CommandsExist(t *testing.T) {
|
||||
t.Parallel()
|
||||
c := main.CommandLine{}
|
||||
tests := []struct {
|
||||
cmd string
|
||||
}{
|
||||
{cmd: "gopher"},
|
||||
{cmd: "connect"},
|
||||
{cmd: "help"},
|
||||
{cmd: "pretty"},
|
||||
{cmd: "use"},
|
||||
{cmd: ""}, // test that a blank command just returns
|
||||
}
|
||||
for _, test := range tests {
|
||||
if !c.ParseCommand(test.cmd) {
|
||||
t.Fatalf(`Command failed for %q.`, test.cmd)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCommand_TogglePretty(t *testing.T) {
|
||||
t.Parallel()
|
||||
c := main.CommandLine{}
|
||||
if c.Pretty {
|
||||
t.Fatalf(`Pretty should be false.`)
|
||||
}
|
||||
c.ParseCommand("pretty")
|
||||
if !c.Pretty {
|
||||
t.Fatalf(`Pretty should be true.`)
|
||||
}
|
||||
c.ParseCommand("pretty")
|
||||
if c.Pretty {
|
||||
t.Fatalf(`Pretty should be false.`)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCommand_Exit(t *testing.T) {
|
||||
t.Parallel()
|
||||
c := main.CommandLine{}
|
||||
tests := []struct {
|
||||
cmd string
|
||||
}{
|
||||
{cmd: "exit"},
|
||||
{cmd: " exit"},
|
||||
{cmd: "exit "},
|
||||
{cmd: "Exit "},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
if c.ParseCommand(test.cmd) {
|
||||
t.Fatalf(`Command "exit" failed for %q.`, test.cmd)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCommand_Use(t *testing.T) {
|
||||
t.Parallel()
|
||||
c := main.CommandLine{}
|
||||
tests := []struct {
|
||||
cmd string
|
||||
}{
|
||||
{cmd: "use db"},
|
||||
{cmd: " use db"},
|
||||
{cmd: "use db "},
|
||||
{cmd: "use db;"},
|
||||
{cmd: "use db; "},
|
||||
{cmd: "Use db"},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
if !c.ParseCommand(test.cmd) {
|
||||
t.Fatalf(`Command "use" failed for %q.`, test.cmd)
|
||||
}
|
||||
|
||||
if c.Database != "db" {
|
||||
t.Fatalf(`Command "use" changed database to %q. Expected db`, c.Database)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCommand_Insert(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var data client.Response
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
_ = json.NewEncoder(w).Encode(data)
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
u, _ := url.Parse(ts.URL)
|
||||
config := client.Config{URL: *u}
|
||||
c, err := client.NewClient(config)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
|
||||
}
|
||||
m := main.CommandLine{Client: c}
|
||||
|
||||
tests := []struct {
|
||||
cmd string
|
||||
}{
|
||||
{cmd: "INSERT cpu,host=serverA,region=us-west value=1.0"},
|
||||
{cmd: " INSERT cpu,host=serverA,region=us-west value=1.0"},
|
||||
{cmd: "INSERT cpu,host=serverA,region=us-west value=1.0"},
|
||||
{cmd: "insert cpu,host=serverA,region=us-west value=1.0 "},
|
||||
{cmd: "insert"},
|
||||
{cmd: "Insert "},
|
||||
{cmd: "insert c"},
|
||||
{cmd: "insert int"},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
if !m.ParseCommand(test.cmd) {
|
||||
t.Fatalf(`Command "insert" failed for %q.`, test.cmd)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCommand_InsertInto(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var data client.Response
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
_ = json.NewEncoder(w).Encode(data)
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
u, _ := url.Parse(ts.URL)
|
||||
config := client.Config{URL: *u}
|
||||
c, err := client.NewClient(config)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
|
||||
}
|
||||
m := main.CommandLine{Client: c}
|
||||
|
||||
tests := []struct {
|
||||
cmd, db, rp string
|
||||
}{
|
||||
{
|
||||
cmd: `INSERT INTO test cpu,host=serverA,region=us-west value=1.0`,
|
||||
db: "",
|
||||
rp: "test",
|
||||
},
|
||||
{
|
||||
cmd: ` INSERT INTO .test cpu,host=serverA,region=us-west value=1.0`,
|
||||
db: "",
|
||||
rp: "test",
|
||||
},
|
||||
{
|
||||
cmd: `INSERT INTO "test test" cpu,host=serverA,region=us-west value=1.0`,
|
||||
db: "",
|
||||
rp: "test test",
|
||||
},
|
||||
{
|
||||
cmd: `Insert iNTO test.test cpu,host=serverA,region=us-west value=1.0`,
|
||||
db: "test",
|
||||
rp: "test",
|
||||
},
|
||||
{
|
||||
cmd: `insert into "test test" cpu,host=serverA,region=us-west value=1.0`,
|
||||
db: "test",
|
||||
rp: "test test",
|
||||
},
|
||||
{
|
||||
cmd: `insert into "d b"."test test" cpu,host=serverA,region=us-west value=1.0`,
|
||||
db: "d b",
|
||||
rp: "test test",
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
if !m.ParseCommand(test.cmd) {
|
||||
t.Fatalf(`Command "insert into" failed for %q.`, test.cmd)
|
||||
}
|
||||
if m.Database != test.db {
|
||||
t.Fatalf(`Command "insert into" db parsing failed, expected: %q, actual: %q`, test.db, m.Database)
|
||||
}
|
||||
if m.RetentionPolicy != test.rp {
|
||||
t.Fatalf(`Command "insert into" rp parsing failed, expected: %q, actual: %q`, test.rp, m.RetentionPolicy)
|
||||
}
|
||||
}
|
||||
}
|
||||
154
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx_stress/influx_stress.go
generated
vendored
154
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx_stress/influx_stress.go
generated
vendored
@@ -1,154 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"net/url"
|
||||
"runtime"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/client"
|
||||
)
|
||||
|
||||
var (
|
||||
batchSize = flag.Int("batchsize", 5000, "number of points per batch")
|
||||
seriesCount = flag.Int("series", 100000, "number of unique series to create")
|
||||
pointCount = flag.Int("points", 100, "number of points per series to create")
|
||||
concurrency = flag.Int("concurrency", 10, "number of simultaneous writes to run")
|
||||
batchInterval = flag.Duration("batchinterval", 0*time.Second, "duration between batches")
|
||||
database = flag.String("database", "stress", "name of database")
|
||||
address = flag.String("addr", "localhost:8086", "IP address and port of database (e.g., localhost:8086)")
|
||||
)
|
||||
|
||||
func main() {
|
||||
flag.Parse()
|
||||
runtime.GOMAXPROCS(runtime.NumCPU())
|
||||
|
||||
startTime := time.Now()
|
||||
counter := NewConcurrencyLimiter(*concurrency)
|
||||
|
||||
u, _ := url.Parse(fmt.Sprintf("http://%s", *address))
|
||||
c, err := client.NewClient(client.Config{URL: *u})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
var mu sync.Mutex
|
||||
var wg sync.WaitGroup
|
||||
responseTimes := make([]int, 0)
|
||||
|
||||
totalPoints := 0
|
||||
|
||||
batch := &client.BatchPoints{
|
||||
Database: *database,
|
||||
WriteConsistency: "any",
|
||||
Time: time.Now(),
|
||||
Precision: "n",
|
||||
}
|
||||
for i := 1; i <= *pointCount; i++ {
|
||||
for j := 1; j <= *seriesCount; j++ {
|
||||
p := client.Point{
|
||||
Measurement: "cpu",
|
||||
Tags: map[string]string{"region": "uswest", "host": fmt.Sprintf("host-%d", j)},
|
||||
Fields: map[string]interface{}{"value": rand.Float64()},
|
||||
}
|
||||
batch.Points = append(batch.Points, p)
|
||||
if len(batch.Points) >= *batchSize {
|
||||
wg.Add(1)
|
||||
counter.Increment()
|
||||
totalPoints += len(batch.Points)
|
||||
go func(b *client.BatchPoints, total int) {
|
||||
st := time.Now()
|
||||
if _, err := c.Write(*b); err != nil {
|
||||
fmt.Println("ERROR: ", err.Error())
|
||||
} else {
|
||||
mu.Lock()
|
||||
responseTimes = append(responseTimes, int(time.Since(st).Nanoseconds()))
|
||||
mu.Unlock()
|
||||
}
|
||||
wg.Done()
|
||||
counter.Decrement()
|
||||
if total%500000 == 0 {
|
||||
fmt.Printf("%d total points. %d in %s\n", total, *batchSize, time.Since(st))
|
||||
}
|
||||
}(batch, totalPoints)
|
||||
|
||||
batch = &client.BatchPoints{
|
||||
Database: *database,
|
||||
WriteConsistency: "any",
|
||||
Precision: "n",
|
||||
Time: time.Now(),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
sort.Sort(sort.Reverse(sort.IntSlice(responseTimes)))
|
||||
|
||||
total := int64(0)
|
||||
for _, t := range responseTimes {
|
||||
total += int64(t)
|
||||
}
|
||||
mean := total / int64(len(responseTimes))
|
||||
|
||||
fmt.Printf("Wrote %d points at average rate of %.0f\n", totalPoints, float64(totalPoints)/time.Since(startTime).Seconds())
|
||||
fmt.Println("Average response time: ", time.Duration(mean))
|
||||
fmt.Println("Slowest response times:")
|
||||
for _, r := range responseTimes[:100] {
|
||||
fmt.Println(time.Duration(r))
|
||||
}
|
||||
}
|
||||
|
||||
// ConcurrencyLimiter is a go routine safe struct that can be used to
|
||||
// ensure that no more than a specifid max number of goroutines are
|
||||
// executing.
|
||||
type ConcurrencyLimiter struct {
|
||||
inc chan chan struct{}
|
||||
dec chan struct{}
|
||||
max int
|
||||
count int
|
||||
}
|
||||
|
||||
// NewConcurrencyLimiter returns a configured limiter that will
|
||||
// ensure that calls to Increment will block if the max is hit.
|
||||
func NewConcurrencyLimiter(max int) *ConcurrencyLimiter {
|
||||
c := &ConcurrencyLimiter{
|
||||
inc: make(chan chan struct{}),
|
||||
dec: make(chan struct{}, max),
|
||||
max: max,
|
||||
}
|
||||
go c.handleLimits()
|
||||
return c
|
||||
}
|
||||
|
||||
// Increment will increase the count of running goroutines by 1.
|
||||
// if the number is currently at the max, the call to Increment
|
||||
// will block until another goroutine decrements.
|
||||
func (c *ConcurrencyLimiter) Increment() {
|
||||
r := make(chan struct{})
|
||||
c.inc <- r
|
||||
<-r
|
||||
}
|
||||
|
||||
// Decrement will reduce the count of running goroutines by 1
|
||||
func (c *ConcurrencyLimiter) Decrement() {
|
||||
c.dec <- struct{}{}
|
||||
}
|
||||
|
||||
// handleLimits runs in a goroutine to manage the count of
|
||||
// running goroutines.
|
||||
func (c *ConcurrencyLimiter) handleLimits() {
|
||||
for {
|
||||
r := <-c.inc
|
||||
if c.count >= c.max {
|
||||
<-c.dec
|
||||
c.count--
|
||||
}
|
||||
c.count++
|
||||
r <- struct{}{}
|
||||
}
|
||||
}
|
||||
170
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/backup/backup.go
generated
vendored
170
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/backup/backup.go
generated
vendored
@@ -1,170 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"os"
|
||||
|
||||
"github.com/influxdb/influxdb/services/snapshotter"
|
||||
"github.com/influxdb/influxdb/snapshot"
|
||||
)
|
||||
|
||||
// Suffix is a suffix added to the backup while it's in-process.
|
||||
const Suffix = ".pending"
|
||||
|
||||
// Command represents the program execution for "influxd backup".
|
||||
type Command struct {
|
||||
// The logger passed to the ticker during execution.
|
||||
Logger *log.Logger
|
||||
|
||||
// Standard input/output, overridden for testing.
|
||||
Stderr io.Writer
|
||||
}
|
||||
|
||||
// NewCommand returns a new instance of Command with default settings.
|
||||
func NewCommand() *Command {
|
||||
return &Command{
|
||||
Stderr: os.Stderr,
|
||||
}
|
||||
}
|
||||
|
||||
// Run executes the program.
|
||||
func (cmd *Command) Run(args ...string) error {
|
||||
// Set up logger.
|
||||
cmd.Logger = log.New(cmd.Stderr, "", log.LstdFlags)
|
||||
cmd.Logger.Printf("influxdb backup")
|
||||
|
||||
// Parse command line arguments.
|
||||
host, path, err := cmd.parseFlags(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Retrieve snapshot from local file.
|
||||
m, err := snapshot.ReadFileManifest(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("read file snapshot: %s", err)
|
||||
}
|
||||
|
||||
// Determine temporary path to download to.
|
||||
tmppath := path + Suffix
|
||||
|
||||
// Calculate path of next backup file.
|
||||
// This uses the path if it doesn't exist.
|
||||
// Otherwise it appends an autoincrementing number.
|
||||
path, err = cmd.nextPath(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("next path: %s", err)
|
||||
}
|
||||
|
||||
// Retrieve snapshot.
|
||||
if err := cmd.download(host, m, tmppath); err != nil {
|
||||
return fmt.Errorf("download: %s", err)
|
||||
}
|
||||
|
||||
// Rename temporary file to final path.
|
||||
if err := os.Rename(tmppath, path); err != nil {
|
||||
return fmt.Errorf("rename: %s", err)
|
||||
}
|
||||
|
||||
// TODO: Check file integrity.
|
||||
|
||||
// Notify user of completion.
|
||||
cmd.Logger.Println("backup complete")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseFlags parses and validates the command line arguments.
|
||||
func (cmd *Command) parseFlags(args []string) (host string, path string, err error) {
|
||||
fs := flag.NewFlagSet("", flag.ContinueOnError)
|
||||
fs.StringVar(&host, "host", "localhost:8088", "")
|
||||
fs.SetOutput(cmd.Stderr)
|
||||
fs.Usage = cmd.printUsage
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
|
||||
// Ensure that only one arg is specified.
|
||||
if fs.NArg() == 0 {
|
||||
return "", "", errors.New("snapshot path required")
|
||||
} else if fs.NArg() != 1 {
|
||||
return "", "", errors.New("only one snapshot path allowed")
|
||||
}
|
||||
path = fs.Arg(0)
|
||||
|
||||
return host, path, nil
|
||||
}
|
||||
|
||||
// nextPath returns the next file to write to.
|
||||
func (cmd *Command) nextPath(path string) (string, error) {
|
||||
// Use base path if it doesn't exist.
|
||||
if _, err := os.Stat(path); os.IsNotExist(err) {
|
||||
return path, nil
|
||||
} else if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Otherwise iterate through incremental files until one is available.
|
||||
for i := 0; ; i++ {
|
||||
s := fmt.Sprintf(path+".%d", i)
|
||||
if _, err := os.Stat(s); os.IsNotExist(err) {
|
||||
return s, nil
|
||||
} else if err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// download downloads a snapshot from a host to a given path.
|
||||
func (cmd *Command) download(host string, m *snapshot.Manifest, path string) error {
|
||||
// Create local file to write to.
|
||||
f, err := os.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open temp file: %s", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
// Connect to snapshotter service.
|
||||
conn, err := net.Dial("tcp", host)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
// Send snapshotter marker byte.
|
||||
if _, err := conn.Write([]byte{snapshotter.MuxHeader}); err != nil {
|
||||
return fmt.Errorf("write snapshot header byte: %s", err)
|
||||
}
|
||||
|
||||
// Write the manifest we currently have.
|
||||
if err := json.NewEncoder(conn).Encode(m); err != nil {
|
||||
return fmt.Errorf("encode snapshot manifest: %s", err)
|
||||
}
|
||||
|
||||
// Read snapshot from the connection.
|
||||
if _, err := io.Copy(f, conn); err != nil {
|
||||
return fmt.Errorf("copy snapshot to file: %s", err)
|
||||
}
|
||||
|
||||
// FIXME(benbjohnson): Verify integrity of snapshot.
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// printUsage prints the usage message to STDERR.
|
||||
func (cmd *Command) printUsage() {
|
||||
fmt.Fprintf(cmd.Stderr, `usage: influxd backup [flags] PATH
|
||||
|
||||
backup downloads a snapshot of a data node and saves it to disk.
|
||||
|
||||
-host <host:port>
|
||||
The host to connect to snapshot.
|
||||
Defaults to 127.0.0.1:8088.
|
||||
`)
|
||||
}
|
||||
125
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/backup/backup_test.go
generated
vendored
125
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/backup/backup_test.go
generated
vendored
@@ -1,125 +0,0 @@
|
||||
package backup_test
|
||||
|
||||
/*
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/influxdb/influxdb"
|
||||
"github.com/influxdb/influxdb/cmd/influxd"
|
||||
)
|
||||
|
||||
// Ensure the backup can download from the server and save to disk.
|
||||
func TestBackupCommand(t *testing.T) {
|
||||
// Mock the backup endpoint.
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/data/snapshot" {
|
||||
t.Fatalf("unexpected url path: %s", r.URL.Path)
|
||||
}
|
||||
|
||||
// Write a simple snapshot to the buffer.
|
||||
sw := influxdb.NewSnapshotWriter()
|
||||
sw.Snapshot = &influxdb.Snapshot{Files: []influxdb.SnapshotFile{
|
||||
{Name: "meta", Size: 5, Index: 10},
|
||||
}}
|
||||
sw.FileWriters["meta"] = influxdb.NopWriteToCloser(bytes.NewBufferString("55555"))
|
||||
if _, err := sw.WriteTo(w); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}))
|
||||
defer s.Close()
|
||||
|
||||
// Create a temp path and remove incremental backups at the end.
|
||||
path := tempfile()
|
||||
defer os.Remove(path)
|
||||
defer os.Remove(path + ".0")
|
||||
defer os.Remove(path + ".1")
|
||||
|
||||
// Execute the backup against the mock server.
|
||||
for i := 0; i < 3; i++ {
|
||||
if err := NewBackupCommand().Run("-host", s.URL, path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify snapshot and two incremental snapshots were written.
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("snapshot not found: %s", err)
|
||||
} else if _, err = os.Stat(path + ".0"); err != nil {
|
||||
t.Fatalf("incremental snapshot(0) not found: %s", err)
|
||||
} else if _, err = os.Stat(path + ".1"); err != nil {
|
||||
t.Fatalf("incremental snapshot(1) not found: %s", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the backup command returns an error if flags cannot be parsed.
|
||||
func TestBackupCommand_ErrFlagParse(t *testing.T) {
|
||||
cmd := NewBackupCommand()
|
||||
if err := cmd.Run("-bad-flag"); err == nil || err.Error() != `flag provided but not defined: -bad-flag` {
|
||||
t.Fatal(err)
|
||||
} else if !strings.Contains(cmd.Stderr.String(), "usage") {
|
||||
t.Fatal("usage message not displayed")
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the backup command returns an error if the host cannot be parsed.
|
||||
func TestBackupCommand_ErrInvalidHostURL(t *testing.T) {
|
||||
if err := NewBackupCommand().Run("-host", "http://%f"); err == nil || err.Error() != `parse host url: parse http://%f: hexadecimal escape in host` {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the backup command returns an error if the output path is not specified.
|
||||
func TestBackupCommand_ErrPathRequired(t *testing.T) {
|
||||
if err := NewBackupCommand().Run("-host", "//localhost"); err == nil || err.Error() != `snapshot path required` {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the backup returns an error if it cannot connect to the server.
|
||||
func TestBackupCommand_ErrConnectionRefused(t *testing.T) {
|
||||
// Start and immediately stop a server so we have a dead port.
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
|
||||
s.Close()
|
||||
|
||||
// Execute the backup command.
|
||||
path := tempfile()
|
||||
defer os.Remove(path)
|
||||
if err := NewBackupCommand().Run("-host", s.URL, path); err == nil ||
|
||||
!(strings.Contains(err.Error(), `connection refused`) || strings.Contains(err.Error(), `No connection could be made`)) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the backup returns any non-200 status codes.
|
||||
func TestBackupCommand_ErrServerError(t *testing.T) {
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
}))
|
||||
defer s.Close()
|
||||
|
||||
// Execute the backup command.
|
||||
path := tempfile()
|
||||
defer os.Remove(path)
|
||||
if err := NewBackupCommand().Run("-host", s.URL, path); err == nil || err.Error() != `download: snapshot error: status=500` {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// BackupCommand is a test wrapper for main.BackupCommand.
|
||||
type BackupCommand struct {
|
||||
*main.BackupCommand
|
||||
Stderr bytes.Buffer
|
||||
}
|
||||
|
||||
// NewBackupCommand returns a new instance of BackupCommand.
|
||||
func NewBackupCommand() *BackupCommand {
|
||||
cmd := &BackupCommand{BackupCommand: main.NewBackupCommand()}
|
||||
cmd.BackupCommand.Stderr = &cmd.Stderr
|
||||
return cmd
|
||||
}
|
||||
*/
|
||||
46
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/help/help.go
generated
vendored
46
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/help/help.go
generated
vendored
@@ -1,46 +0,0 @@
|
||||
package help
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Command displays help for command-line sub-commands.
|
||||
type Command struct {
|
||||
Stdout io.Writer
|
||||
}
|
||||
|
||||
// NewCommand returns a new instance of Command.
|
||||
func NewCommand() *Command {
|
||||
return &Command{
|
||||
Stdout: os.Stdout,
|
||||
}
|
||||
}
|
||||
|
||||
// Run executes the command.
|
||||
func (cmd *Command) Run(args ...string) error {
|
||||
fmt.Fprintln(cmd.Stdout, strings.TrimSpace(usage))
|
||||
return nil
|
||||
}
|
||||
|
||||
const usage = `
|
||||
Configure and start an InfluxDB server.
|
||||
|
||||
Usage:
|
||||
|
||||
influxd [[command] [arguments]]
|
||||
|
||||
The commands are:
|
||||
|
||||
backup downloads a snapshot of a data node and saves it to disk
|
||||
config display the default configuration
|
||||
restore uses a snapshot of a data node to rebuild a cluster
|
||||
run run node with existing configuration
|
||||
version displays the InfluxDB version
|
||||
|
||||
"run" is the default command.
|
||||
|
||||
Use "influxd help [command]" for more information about a command.
|
||||
`
|
||||
200
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/main.go
generated
vendored
200
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/main.go
generated
vendored
@@ -1,200 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math/rand"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cmd/influxd/backup"
|
||||
"github.com/influxdb/influxdb/cmd/influxd/help"
|
||||
"github.com/influxdb/influxdb/cmd/influxd/restore"
|
||||
"github.com/influxdb/influxdb/cmd/influxd/run"
|
||||
)
|
||||
|
||||
// These variables are populated via the Go linker.
|
||||
var (
|
||||
version string = "0.9"
|
||||
commit string
|
||||
branch string
|
||||
)
|
||||
|
||||
func init() {
|
||||
// If commit or branch are not set, make that clear.
|
||||
if commit == "" {
|
||||
commit = "unknown"
|
||||
}
|
||||
if branch == "" {
|
||||
branch = "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
rand.Seed(time.Now().UnixNano())
|
||||
|
||||
m := NewMain()
|
||||
if err := m.Run(os.Args[1:]...); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Main represents the program execution.
|
||||
type Main struct {
|
||||
Logger *log.Logger
|
||||
|
||||
Stdin io.Reader
|
||||
Stdout io.Writer
|
||||
Stderr io.Writer
|
||||
}
|
||||
|
||||
// NewMain return a new instance of Main.
|
||||
func NewMain() *Main {
|
||||
return &Main{
|
||||
Logger: log.New(os.Stderr, "[run] ", log.LstdFlags),
|
||||
Stdin: os.Stdin,
|
||||
Stdout: os.Stdout,
|
||||
Stderr: os.Stderr,
|
||||
}
|
||||
}
|
||||
|
||||
// Run determines and runs the command specified by the CLI args.
|
||||
func (m *Main) Run(args ...string) error {
|
||||
name, args := ParseCommandName(args)
|
||||
|
||||
// Extract name from args.
|
||||
switch name {
|
||||
case "", "run":
|
||||
cmd := run.NewCommand()
|
||||
|
||||
// Tell the server the build details.
|
||||
cmd.Version = version
|
||||
cmd.Commit = commit
|
||||
cmd.Branch = branch
|
||||
|
||||
if err := cmd.Run(args...); err != nil {
|
||||
return fmt.Errorf("run: %s", err)
|
||||
}
|
||||
|
||||
signalCh := make(chan os.Signal, 1)
|
||||
signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM)
|
||||
m.Logger.Println("Listening for signals")
|
||||
|
||||
// Block until one of the signals above is received
|
||||
select {
|
||||
case <-signalCh:
|
||||
m.Logger.Println("Signal received, initializing clean shutdown...")
|
||||
go func() {
|
||||
cmd.Close()
|
||||
}()
|
||||
}
|
||||
|
||||
// Block again until another signal is received, a shutdown timeout elapses,
|
||||
// or the Command is gracefully closed
|
||||
m.Logger.Println("Waiting for clean shutdown...")
|
||||
select {
|
||||
case <-signalCh:
|
||||
m.Logger.Println("second signal received, initializing hard shutdown")
|
||||
case <-time.After(time.Second * 30):
|
||||
m.Logger.Println("time limit reached, initializing hard shutdown")
|
||||
case <-cmd.Closed:
|
||||
m.Logger.Println("server shutdown completed")
|
||||
}
|
||||
|
||||
// goodbye.
|
||||
|
||||
case "backup":
|
||||
name := backup.NewCommand()
|
||||
if err := name.Run(args...); err != nil {
|
||||
return fmt.Errorf("backup: %s", err)
|
||||
}
|
||||
case "restore":
|
||||
name := restore.NewCommand()
|
||||
if err := name.Run(args...); err != nil {
|
||||
return fmt.Errorf("restore: %s", err)
|
||||
}
|
||||
case "config":
|
||||
if err := run.NewPrintConfigCommand().Run(args...); err != nil {
|
||||
return fmt.Errorf("config: %s", err)
|
||||
}
|
||||
case "version":
|
||||
if err := NewVersionCommand().Run(args...); err != nil {
|
||||
return fmt.Errorf("version: %s", err)
|
||||
}
|
||||
case "help":
|
||||
if err := help.NewCommand().Run(args...); err != nil {
|
||||
return fmt.Errorf("help: %s", err)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf(`unknown command "%s"`+"\n"+`Run 'influxd help' for usage`+"\n\n", name)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseCommandName extracts the command name and args from the args list.
|
||||
func ParseCommandName(args []string) (string, []string) {
|
||||
// Retrieve command name as first argument.
|
||||
var name string
|
||||
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||
name = args[0]
|
||||
}
|
||||
|
||||
// Special case -h immediately following binary name
|
||||
if len(args) > 0 && args[0] == "-h" {
|
||||
name = "help"
|
||||
}
|
||||
|
||||
// If command is "help" and has an argument then rewrite args to use "-h".
|
||||
if name == "help" && len(args) > 1 {
|
||||
args[0], args[1] = args[1], "-h"
|
||||
name = args[0]
|
||||
}
|
||||
|
||||
// If a named command is specified then return it with its arguments.
|
||||
if name != "" {
|
||||
return name, args[1:]
|
||||
}
|
||||
return "", args
|
||||
}
|
||||
|
||||
// Command represents the command executed by "influxd version".
|
||||
type VersionCommand struct {
|
||||
Stdout io.Writer
|
||||
Stderr io.Writer
|
||||
}
|
||||
|
||||
// NewVersionCommand return a new instance of VersionCommand.
|
||||
func NewVersionCommand() *VersionCommand {
|
||||
return &VersionCommand{
|
||||
Stdout: os.Stdout,
|
||||
Stderr: os.Stderr,
|
||||
}
|
||||
}
|
||||
|
||||
// Run prints the current version and commit info.
|
||||
func (cmd *VersionCommand) Run(args ...string) error {
|
||||
// Parse flags in case -h is specified.
|
||||
fs := flag.NewFlagSet("", flag.ContinueOnError)
|
||||
fs.Usage = func() { fmt.Fprintln(cmd.Stderr, strings.TrimSpace(versionUsage)) }
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Print version info.
|
||||
fmt.Fprintf(cmd.Stdout, "InfluxDB v%s (git: %s %s)\n", version, branch, commit)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
var versionUsage = `
|
||||
usage: version
|
||||
|
||||
version displays the InfluxDB version, build branch and git commit hash
|
||||
`
|
||||
250
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/restore/restore.go
generated
vendored
250
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/restore/restore.go
generated
vendored
@@ -1,250 +0,0 @@
|
||||
package restore
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/snapshot"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// Command represents the program execution for "influxd restore".
|
||||
type Command struct {
|
||||
Stdout io.Writer
|
||||
Stderr io.Writer
|
||||
}
|
||||
|
||||
// NewCommand returns a new instance of Command with default settings.
|
||||
func NewCommand() *Command {
|
||||
return &Command{
|
||||
Stdout: os.Stdout,
|
||||
Stderr: os.Stderr,
|
||||
}
|
||||
}
|
||||
|
||||
// Run executes the program.
|
||||
func (cmd *Command) Run(args ...string) error {
|
||||
config, path, err := cmd.parseFlags(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return cmd.Restore(config, path)
|
||||
}
|
||||
|
||||
func (cmd *Command) Restore(config *Config, path string) error {
|
||||
// Remove meta and data directories.
|
||||
if err := os.RemoveAll(config.Meta.Dir); err != nil {
|
||||
return fmt.Errorf("remove meta dir: %s", err)
|
||||
} else if err := os.RemoveAll(config.Data.Dir); err != nil {
|
||||
return fmt.Errorf("remove data dir: %s", err)
|
||||
}
|
||||
|
||||
// Open snapshot file and all incremental backups.
|
||||
mr, files, err := snapshot.OpenFileMultiReader(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open multireader: %s", err)
|
||||
}
|
||||
defer closeAll(files)
|
||||
|
||||
// Unpack files from archive.
|
||||
if err := cmd.unpack(mr, config); err != nil {
|
||||
return fmt.Errorf("unpack: %s", err)
|
||||
}
|
||||
|
||||
// Notify user of completion.
|
||||
fmt.Fprintf(os.Stdout, "restore complete using %s", path)
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseFlags parses and validates the command line arguments.
|
||||
func (cmd *Command) parseFlags(args []string) (*Config, string, error) {
|
||||
fs := flag.NewFlagSet("", flag.ContinueOnError)
|
||||
configPath := fs.String("config", "", "")
|
||||
fs.SetOutput(cmd.Stderr)
|
||||
fs.Usage = cmd.printUsage
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
// Parse configuration file from disk.
|
||||
if *configPath == "" {
|
||||
return nil, "", fmt.Errorf("config required")
|
||||
}
|
||||
|
||||
// Parse config.
|
||||
config := Config{
|
||||
Meta: meta.NewConfig(),
|
||||
Data: tsdb.NewConfig(),
|
||||
}
|
||||
if _, err := toml.DecodeFile(*configPath, &config); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
// Require output path.
|
||||
path := fs.Arg(0)
|
||||
if path == "" {
|
||||
return nil, "", fmt.Errorf("snapshot path required")
|
||||
}
|
||||
|
||||
return &config, path, nil
|
||||
}
|
||||
|
||||
func closeAll(a []io.Closer) {
|
||||
for _, c := range a {
|
||||
_ = c.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// unpack expands the files in the snapshot archive into a directory.
|
||||
func (cmd *Command) unpack(mr *snapshot.MultiReader, config *Config) error {
|
||||
// Loop over files and extract.
|
||||
for {
|
||||
// Read entry header.
|
||||
sf, err := mr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
} else if err != nil {
|
||||
return fmt.Errorf("next: entry=%s, err=%s", sf.Name, err)
|
||||
}
|
||||
|
||||
// Log progress.
|
||||
fmt.Fprintf(os.Stdout, "unpacking: %s (%d bytes)\n", sf.Name, sf.Size)
|
||||
|
||||
// Handle meta and tsdb files separately.
|
||||
switch sf.Name {
|
||||
case "meta":
|
||||
if err := cmd.unpackMeta(mr, sf, config); err != nil {
|
||||
return fmt.Errorf("meta: %s", err)
|
||||
}
|
||||
default:
|
||||
if err := cmd.unpackData(mr, sf, config); err != nil {
|
||||
return fmt.Errorf("data: %s", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackMeta reads the metadata from the snapshot and initializes a raft
|
||||
// cluster and replaces the root metadata.
|
||||
func (cmd *Command) unpackMeta(mr *snapshot.MultiReader, sf snapshot.File, config *Config) error {
|
||||
// Read meta into buffer.
|
||||
var buf bytes.Buffer
|
||||
if _, err := io.CopyN(&buf, mr, sf.Size); err != nil {
|
||||
return fmt.Errorf("copy: %s", err)
|
||||
}
|
||||
|
||||
// Unpack into metadata.
|
||||
var data meta.Data
|
||||
if err := data.UnmarshalBinary(buf.Bytes()); err != nil {
|
||||
return fmt.Errorf("unmarshal: %s", err)
|
||||
}
|
||||
|
||||
// Copy meta config and remove peers so it starts in single mode.
|
||||
c := config.Meta
|
||||
c.Peers = nil
|
||||
|
||||
// Initialize meta store.
|
||||
store := meta.NewStore(config.Meta)
|
||||
store.RaftListener = newNopListener()
|
||||
store.ExecListener = newNopListener()
|
||||
|
||||
// Determine advertised address.
|
||||
_, port, err := net.SplitHostPort(config.Meta.BindAddress)
|
||||
if err != nil {
|
||||
return fmt.Errorf("split bind address: %s", err)
|
||||
}
|
||||
hostport := net.JoinHostPort(config.Meta.Hostname, port)
|
||||
|
||||
// Resolve address.
|
||||
addr, err := net.ResolveTCPAddr("tcp", hostport)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve tcp: addr=%s, err=%s", hostport, err)
|
||||
}
|
||||
store.Addr = addr
|
||||
|
||||
// Open the meta store.
|
||||
if err := store.Open(); err != nil {
|
||||
return fmt.Errorf("open store: %s", err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
// Wait for the store to be ready or error.
|
||||
select {
|
||||
case <-store.Ready():
|
||||
case err := <-store.Err():
|
||||
return err
|
||||
}
|
||||
|
||||
// Force set the full metadata.
|
||||
if err := store.SetData(&data); err != nil {
|
||||
return fmt.Errorf("set data: %s", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (cmd *Command) unpackData(mr *snapshot.MultiReader, sf snapshot.File, config *Config) error {
|
||||
path := filepath.Join(config.Data.Dir, sf.Name)
|
||||
// Create parent directory for output file.
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0777); err != nil {
|
||||
return fmt.Errorf("mkdir: entry=%s, err=%s", sf.Name, err)
|
||||
}
|
||||
|
||||
// Create output file.
|
||||
f, err := os.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create: entry=%s, err=%s", sf.Name, err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
// Copy contents from reader.
|
||||
if _, err := io.CopyN(f, mr, sf.Size); err != nil {
|
||||
return fmt.Errorf("copy: entry=%s, err=%s", sf.Name, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// printUsage prints the usage message to STDERR.
|
||||
func (cmd *Command) printUsage() {
|
||||
fmt.Fprintf(cmd.Stderr, `usage: influxd restore [flags] PATH
|
||||
|
||||
restore uses a snapshot of a data node to rebuild a cluster.
|
||||
|
||||
-config <path>
|
||||
Set the path to the configuration file.
|
||||
`)
|
||||
}
|
||||
|
||||
// Config represents a partial config for rebuilding the server.
|
||||
type Config struct {
|
||||
Meta *meta.Config `toml:"meta"`
|
||||
Data tsdb.Config `toml:"data"`
|
||||
}
|
||||
|
||||
type nopListener struct {
|
||||
closing chan struct{}
|
||||
}
|
||||
|
||||
func newNopListener() *nopListener {
|
||||
return &nopListener{make(chan struct{})}
|
||||
}
|
||||
|
||||
func (ln *nopListener) Accept() (net.Conn, error) {
|
||||
<-ln.closing
|
||||
return nil, errors.New("listener closing")
|
||||
}
|
||||
|
||||
func (ln *nopListener) Close() error { close(ln.closing); return nil }
|
||||
func (ln *nopListener) Addr() net.Addr { return nil }
|
||||
155
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/restore/restore_test.go
generated
vendored
155
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/restore/restore_test.go
generated
vendored
@@ -1,155 +0,0 @@
|
||||
package restore_test
|
||||
|
||||
/*
|
||||
import (
|
||||
"bytes"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
main "github.com/influxdb/influxdb/cmd/influxd"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
func newConfig(path string, port int) main.Config {
|
||||
config := main.NewConfig()
|
||||
config.Port = port
|
||||
config.Broker.Enabled = true
|
||||
config.Broker.Dir = filepath.Join(path, "broker")
|
||||
|
||||
config.Data.Enabled = true
|
||||
config.Data.Dir = filepath.Join(path, "data")
|
||||
return *config
|
||||
}
|
||||
|
||||
// Ensure the restore command can expand a snapshot and bootstrap a broker.
|
||||
func TestRestoreCommand(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping TestRestoreCommand")
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// Create root path to server.
|
||||
path := tempfile()
|
||||
defer os.Remove(path)
|
||||
|
||||
// Parse configuration.
|
||||
config := newConfig(path, 8900)
|
||||
|
||||
// Start server.
|
||||
cmd := main.NewRunCommand()
|
||||
node := cmd.Open(&config, "")
|
||||
if node.Broker == nil {
|
||||
t.Fatal("cannot run broker")
|
||||
} else if node.DataNode == nil {
|
||||
t.Fatal("cannot run server")
|
||||
}
|
||||
b := node.Broker
|
||||
s := node.DataNode
|
||||
|
||||
// Create data.
|
||||
if err := s.CreateDatabase("db"); err != nil {
|
||||
t.Fatalf("cannot create database: %s", err)
|
||||
}
|
||||
if index, err := s.WriteSeries("db", "default", []tsdb.Point{tsdb.NewPoint("cpu", nil, map[string]interface{}{"value": float64(100)}, now)}); err != nil {
|
||||
t.Fatalf("cannot write series: %s", err)
|
||||
} else if err = s.Sync(1, index); err != nil {
|
||||
t.Fatalf("shard sync: %s", err)
|
||||
}
|
||||
|
||||
// Create snapshot writer.
|
||||
sw, err := s.CreateSnapshotWriter()
|
||||
if err != nil {
|
||||
t.Fatalf("create snapshot writer: %s", err)
|
||||
}
|
||||
|
||||
// Snapshot to file.
|
||||
sspath := tempfile()
|
||||
f, err := os.Create(sspath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sw.WriteTo(f)
|
||||
f.Close()
|
||||
|
||||
// Stop server.
|
||||
node.Close()
|
||||
|
||||
// Remove data & broker directories.
|
||||
if err := os.RemoveAll(path); err != nil {
|
||||
t.Fatalf("remove: %s", err)
|
||||
}
|
||||
|
||||
// Execute the restore.
|
||||
if err := NewRestoreCommand().Restore(&config, sspath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Rewrite config to a new port and re-parse.
|
||||
config = newConfig(path, 8910)
|
||||
|
||||
// Restart server.
|
||||
cmd = main.NewRunCommand()
|
||||
node = cmd.Open(&config, "")
|
||||
if b == nil {
|
||||
t.Fatal("cannot run broker")
|
||||
} else if s == nil {
|
||||
t.Fatal("cannot run server")
|
||||
}
|
||||
b = node.Broker
|
||||
s = node.DataNode
|
||||
|
||||
// Write new data.
|
||||
if err := s.CreateDatabase("newdb"); err != nil {
|
||||
t.Fatalf("cannot create new database: %s", err)
|
||||
}
|
||||
if index, err := s.WriteSeries("newdb", "default", []tsdb.Point{tsdb.NewPoint("mem", nil, map[string]interface{}{"value": float64(1000)}, now)}); err != nil {
|
||||
t.Fatalf("cannot write new series: %s", err)
|
||||
} else if err = s.Sync(2, index); err != nil {
|
||||
t.Fatalf("shard sync: %s", err)
|
||||
}
|
||||
|
||||
// Read series data.
|
||||
if v, err := s.ReadSeries("db", "default", "cpu", nil, now); err != nil {
|
||||
t.Fatal(err)
|
||||
} else if !reflect.DeepEqual(v, map[string]interface{}{"value": float64(100)}) {
|
||||
t.Fatalf("read series(0) mismatch: %#v", v)
|
||||
}
|
||||
|
||||
// Read new series data.
|
||||
if v, err := s.ReadSeries("newdb", "default", "mem", nil, now); err != nil {
|
||||
t.Fatal(err)
|
||||
} else if !reflect.DeepEqual(v, map[string]interface{}{"value": float64(1000)}) {
|
||||
t.Fatalf("read series(1) mismatch: %#v", v)
|
||||
}
|
||||
|
||||
// Stop server.
|
||||
node.Close()
|
||||
}
|
||||
|
||||
// RestoreCommand is a test wrapper for main.RestoreCommand.
|
||||
type RestoreCommand struct {
|
||||
*main.RestoreCommand
|
||||
Stderr bytes.Buffer
|
||||
}
|
||||
|
||||
// NewRestoreCommand returns a new instance of RestoreCommand.
|
||||
func NewRestoreCommand() *RestoreCommand {
|
||||
cmd := &RestoreCommand{RestoreCommand: main.NewRestoreCommand()}
|
||||
cmd.RestoreCommand.Stderr = &cmd.Stderr
|
||||
return cmd
|
||||
}
|
||||
|
||||
// MustReadFile reads data from a file. Panic on error.
|
||||
func MustReadFile(filename string) []byte {
|
||||
b, err := ioutil.ReadFile(filename)
|
||||
if err != nil {
|
||||
panic(err.Error())
|
||||
}
|
||||
return b
|
||||
}
|
||||
*/
|
||||
235
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go
generated
vendored
235
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go
generated
vendored
@@ -1,235 +0,0 @@
|
||||
package run
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
)
|
||||
|
||||
const logo = `
|
||||
8888888 .d888 888 8888888b. 888888b.
|
||||
888 d88P" 888 888 "Y88b 888 "88b
|
||||
888 888 888 888 888 888 .88P
|
||||
888 88888b. 888888 888 888 888 888 888 888 888 8888888K.
|
||||
888 888 "88b 888 888 888 888 Y8bd8P' 888 888 888 "Y88b
|
||||
888 888 888 888 888 888 888 X88K 888 888 888 888
|
||||
888 888 888 888 888 Y88b 888 .d8""8b. 888 .d88P 888 d88P
|
||||
8888888 888 888 888 888 "Y88888 888 888 8888888P" 8888888P"
|
||||
|
||||
`
|
||||
|
||||
// Command represents the command executed by "influxd run".
|
||||
type Command struct {
|
||||
Version string
|
||||
Branch string
|
||||
Commit string
|
||||
|
||||
closing chan struct{}
|
||||
Closed chan struct{}
|
||||
|
||||
Stdin io.Reader
|
||||
Stdout io.Writer
|
||||
Stderr io.Writer
|
||||
|
||||
Server *Server
|
||||
}
|
||||
|
||||
// NewCommand return a new instance of Command.
|
||||
func NewCommand() *Command {
|
||||
return &Command{
|
||||
closing: make(chan struct{}),
|
||||
Closed: make(chan struct{}),
|
||||
Stdin: os.Stdin,
|
||||
Stdout: os.Stdout,
|
||||
Stderr: os.Stderr,
|
||||
}
|
||||
}
|
||||
|
||||
// Run parses the config from args and runs the server.
|
||||
func (cmd *Command) Run(args ...string) error {
|
||||
// Parse the command line flags.
|
||||
options, err := cmd.ParseFlags(args...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Print sweet InfluxDB logo.
|
||||
fmt.Print(logo)
|
||||
|
||||
// Write the PID file.
|
||||
if err := cmd.writePIDFile(options.PIDFile); err != nil {
|
||||
return fmt.Errorf("write pid file: %s", err)
|
||||
}
|
||||
|
||||
// Set parallelism.
|
||||
runtime.GOMAXPROCS(runtime.NumCPU())
|
||||
|
||||
// Turn on block profiling to debug stuck databases
|
||||
runtime.SetBlockProfileRate(int(1 * time.Second))
|
||||
|
||||
// Parse config
|
||||
config, err := cmd.ParseConfig(options.ConfigPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse config: %s", err)
|
||||
}
|
||||
|
||||
// Apply any environment variables on top of the parsed config
|
||||
if err := config.ApplyEnvOverrides(); err != nil {
|
||||
return fmt.Errorf("apply env config: %v", err)
|
||||
}
|
||||
|
||||
// Override config hostname if specified in the command line args.
|
||||
if options.Hostname != "" {
|
||||
config.Meta.Hostname = options.Hostname
|
||||
}
|
||||
|
||||
if options.Join != "" {
|
||||
config.Meta.Peers = strings.Split(options.Join, ",")
|
||||
}
|
||||
|
||||
// Validate the configuration.
|
||||
if err := config.Validate(); err != nil {
|
||||
return fmt.Errorf("%s. To generate a valid configuration file run `influxd config > influxdb.generated.conf`.", err)
|
||||
}
|
||||
|
||||
// Create server from config and start it.
|
||||
s, err := NewServer(config, cmd.Version)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create server: %s", err)
|
||||
}
|
||||
s.CPUProfile = options.CPUProfile
|
||||
s.MemProfile = options.MemProfile
|
||||
if err := s.Open(); err != nil {
|
||||
return fmt.Errorf("open server: %s", err)
|
||||
}
|
||||
cmd.Server = s
|
||||
|
||||
// Mark start-up in log.
|
||||
log.Printf("InfluxDB starting, version %s, branch %s, commit %s", cmd.Version, cmd.Branch, cmd.Commit)
|
||||
log.Println("GOMAXPROCS set to", runtime.GOMAXPROCS(0))
|
||||
|
||||
// Begin monitoring the server's error channel.
|
||||
go cmd.monitorServerErrors()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close shuts down the server.
|
||||
func (cmd *Command) Close() error {
|
||||
defer close(cmd.Closed)
|
||||
close(cmd.closing)
|
||||
if cmd.Server != nil {
|
||||
return cmd.Server.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (cmd *Command) monitorServerErrors() {
|
||||
logger := log.New(cmd.Stderr, "", log.LstdFlags)
|
||||
for {
|
||||
select {
|
||||
case err := <-cmd.Server.Err():
|
||||
logger.Println(err)
|
||||
case <-cmd.closing:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ParseFlags parses the command line flags from args and returns an options set.
|
||||
func (cmd *Command) ParseFlags(args ...string) (Options, error) {
|
||||
var options Options
|
||||
fs := flag.NewFlagSet("", flag.ContinueOnError)
|
||||
fs.StringVar(&options.ConfigPath, "config", "", "")
|
||||
fs.StringVar(&options.PIDFile, "pidfile", "", "")
|
||||
fs.StringVar(&options.Hostname, "hostname", "", "")
|
||||
fs.StringVar(&options.Join, "join", "", "")
|
||||
fs.StringVar(&options.CPUProfile, "cpuprofile", "", "")
|
||||
fs.StringVar(&options.MemProfile, "memprofile", "", "")
|
||||
fs.Usage = func() { fmt.Fprintln(cmd.Stderr, usage) }
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return Options{}, err
|
||||
}
|
||||
return options, nil
|
||||
}
|
||||
|
||||
// writePIDFile writes the process ID to path.
|
||||
func (cmd *Command) writePIDFile(path string) error {
|
||||
// Ignore if path is not set.
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Ensure the required directory structure exists.
|
||||
err := os.MkdirAll(filepath.Dir(path), 0777)
|
||||
if err != nil {
|
||||
return fmt.Errorf("mkdir: %s", err)
|
||||
}
|
||||
|
||||
// Retrieve the PID and write it.
|
||||
pid := strconv.Itoa(os.Getpid())
|
||||
if err := ioutil.WriteFile(path, []byte(pid), 0666); err != nil {
|
||||
return fmt.Errorf("write file: %s", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseConfig parses the config at path.
|
||||
// Returns a demo configuration if path is blank.
|
||||
func (cmd *Command) ParseConfig(path string) (*Config, error) {
|
||||
// Use demo configuration if no config path is specified.
|
||||
if path == "" {
|
||||
fmt.Fprintln(cmd.Stdout, "no configuration provided, using default settings")
|
||||
return NewDemoConfig()
|
||||
}
|
||||
|
||||
fmt.Fprintf(cmd.Stdout, "Using configuration at: %s\n", path)
|
||||
|
||||
config := NewConfig()
|
||||
if _, err := toml.DecodeFile(path, &config); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return config, nil
|
||||
}
|
||||
|
||||
var usage = `usage: run [flags]
|
||||
|
||||
run starts the broker and data node server. If this is the first time running
|
||||
the command then a new cluster will be initialized unless the -join argument
|
||||
is used.
|
||||
|
||||
-config <path>
|
||||
Set the path to the configuration file.
|
||||
|
||||
-hostname <name>
|
||||
Override the hostname, the 'hostname' configuration
|
||||
option will be overridden.
|
||||
|
||||
-join <url>
|
||||
Joins the server to an existing cluster.
|
||||
|
||||
-pidfile <path>
|
||||
Write process ID to a file.
|
||||
`
|
||||
|
||||
// Options represents the command line options that can be parsed.
|
||||
type Options struct {
|
||||
ConfigPath string
|
||||
PIDFile string
|
||||
Hostname string
|
||||
Join string
|
||||
CPUProfile string
|
||||
MemProfile string
|
||||
}
|
||||
227
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config.go
generated
vendored
227
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config.go
generated
vendored
@@ -1,227 +0,0 @@
|
||||
package run
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cluster"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/services/admin"
|
||||
"github.com/influxdb/influxdb/services/collectd"
|
||||
"github.com/influxdb/influxdb/services/continuous_querier"
|
||||
"github.com/influxdb/influxdb/services/graphite"
|
||||
"github.com/influxdb/influxdb/services/hh"
|
||||
"github.com/influxdb/influxdb/services/httpd"
|
||||
"github.com/influxdb/influxdb/services/monitor"
|
||||
"github.com/influxdb/influxdb/services/opentsdb"
|
||||
"github.com/influxdb/influxdb/services/precreator"
|
||||
"github.com/influxdb/influxdb/services/retention"
|
||||
"github.com/influxdb/influxdb/services/udp"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
)
|
||||
|
||||
// Config represents the configuration format for the influxd binary.
|
||||
type Config struct {
|
||||
Meta *meta.Config `toml:"meta"`
|
||||
Data tsdb.Config `toml:"data"`
|
||||
Cluster cluster.Config `toml:"cluster"`
|
||||
Retention retention.Config `toml:"retention"`
|
||||
Precreator precreator.Config `toml:"shard-precreation"`
|
||||
|
||||
Admin admin.Config `toml:"admin"`
|
||||
HTTPD httpd.Config `toml:"http"`
|
||||
Graphites []graphite.Config `toml:"graphite"`
|
||||
Collectd collectd.Config `toml:"collectd"`
|
||||
OpenTSDB opentsdb.Config `toml:"opentsdb"`
|
||||
UDPs []udp.Config `toml:"udp"`
|
||||
|
||||
// Snapshot SnapshotConfig `toml:"snapshot"`
|
||||
Monitoring monitor.Config `toml:"monitoring"`
|
||||
ContinuousQuery continuous_querier.Config `toml:"continuous_queries"`
|
||||
|
||||
HintedHandoff hh.Config `toml:"hinted-handoff"`
|
||||
|
||||
// Server reporting
|
||||
ReportingDisabled bool `toml:"reporting-disabled"`
|
||||
}
|
||||
|
||||
// NewConfig returns an instance of Config with reasonable defaults.
|
||||
func NewConfig() *Config {
|
||||
c := &Config{}
|
||||
c.Meta = meta.NewConfig()
|
||||
c.Data = tsdb.NewConfig()
|
||||
c.Cluster = cluster.NewConfig()
|
||||
c.Precreator = precreator.NewConfig()
|
||||
|
||||
c.Admin = admin.NewConfig()
|
||||
c.HTTPD = httpd.NewConfig()
|
||||
c.Collectd = collectd.NewConfig()
|
||||
c.OpenTSDB = opentsdb.NewConfig()
|
||||
c.Graphites = append(c.Graphites, graphite.NewConfig())
|
||||
|
||||
c.Monitoring = monitor.NewConfig()
|
||||
c.ContinuousQuery = continuous_querier.NewConfig()
|
||||
c.Retention = retention.NewConfig()
|
||||
c.HintedHandoff = hh.NewConfig()
|
||||
|
||||
return c
|
||||
}
|
||||
|
||||
// NewDemoConfig returns the config that runs when no config is specified.
|
||||
func NewDemoConfig() (*Config, error) {
|
||||
c := NewConfig()
|
||||
|
||||
var homeDir string
|
||||
// By default, store meta and data files in current users home directory
|
||||
u, err := user.Current()
|
||||
if err == nil {
|
||||
homeDir = u.HomeDir
|
||||
} else if os.Getenv("HOME") != "" {
|
||||
homeDir = os.Getenv("HOME")
|
||||
} else {
|
||||
return nil, fmt.Errorf("failed to determine current user for storage")
|
||||
}
|
||||
|
||||
c.Meta.Dir = filepath.Join(homeDir, ".influxdb/meta")
|
||||
c.Data.Dir = filepath.Join(homeDir, ".influxdb/data")
|
||||
c.HintedHandoff.Dir = filepath.Join(homeDir, ".influxdb/hh")
|
||||
c.Data.WALDir = filepath.Join(homeDir, ".influxdb/wal")
|
||||
|
||||
c.Admin.Enabled = true
|
||||
c.Monitoring.Enabled = false
|
||||
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Validate returns an error if the config is invalid.
|
||||
func (c *Config) Validate() error {
|
||||
if c.Meta.Dir == "" {
|
||||
return errors.New("Meta.Dir must be specified")
|
||||
} else if c.Data.Dir == "" {
|
||||
return errors.New("Data.Dir must be specified")
|
||||
} else if c.HintedHandoff.Dir == "" {
|
||||
return errors.New("HintedHandoff.Dir must be specified")
|
||||
} else if c.Data.WALDir == "" {
|
||||
return errors.New("Data.WALDir must be specified")
|
||||
}
|
||||
|
||||
for _, g := range c.Graphites {
|
||||
if err := g.Validate(); err != nil {
|
||||
return fmt.Errorf("invalid graphite config: %v", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Config) ApplyEnvOverrides() error {
|
||||
return c.applyEnvOverrides("INFLUXDB", reflect.ValueOf(c))
|
||||
}
|
||||
|
||||
func (c *Config) applyEnvOverrides(prefix string, spec reflect.Value) error {
|
||||
// If we have a pointer, dereference it
|
||||
s := spec
|
||||
if spec.Kind() == reflect.Ptr {
|
||||
s = spec.Elem()
|
||||
}
|
||||
|
||||
// Make sure we have struct
|
||||
if s.Kind() != reflect.Struct {
|
||||
return nil
|
||||
}
|
||||
|
||||
typeOfSpec := s.Type()
|
||||
for i := 0; i < s.NumField(); i++ {
|
||||
f := s.Field(i)
|
||||
// Get the toml tag to determine what env var name to use
|
||||
configName := typeOfSpec.Field(i).Tag.Get("toml")
|
||||
// Replace hyphens with underscores to avoid issues with shells
|
||||
configName = strings.Replace(configName, "-", "_", -1)
|
||||
fieldName := typeOfSpec.Field(i).Name
|
||||
|
||||
// Skip any fields that we cannot set
|
||||
if f.CanSet() || f.Kind() == reflect.Slice {
|
||||
|
||||
// Use the upper-case prefix and toml name for the env var
|
||||
key := strings.ToUpper(configName)
|
||||
if prefix != "" {
|
||||
key = strings.ToUpper(fmt.Sprintf("%s_%s", prefix, configName))
|
||||
}
|
||||
value := os.Getenv(key)
|
||||
|
||||
// If the type is s slice, apply to each using the index as a suffix
|
||||
// e.g. GRAPHITE_0
|
||||
if f.Kind() == reflect.Slice || f.Kind() == reflect.Array {
|
||||
for i := 0; i < f.Len(); i++ {
|
||||
if err := c.applyEnvOverrides(fmt.Sprintf("%s_%d", key, i), f.Index(i)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// If it's a sub-config, recursively apply
|
||||
if f.Kind() == reflect.Struct || f.Kind() == reflect.Ptr {
|
||||
if err := c.applyEnvOverrides(key, f); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Skip any fields we don't have a value to set
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
switch f.Kind() {
|
||||
case reflect.String:
|
||||
f.SetString(value)
|
||||
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
|
||||
|
||||
var intValue int64
|
||||
|
||||
// Handle toml.Duration
|
||||
if f.Type().Name() == "Duration" {
|
||||
dur, err := time.ParseDuration(value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
|
||||
}
|
||||
intValue = dur.Nanoseconds()
|
||||
} else {
|
||||
var err error
|
||||
intValue, err = strconv.ParseInt(value, 0, f.Type().Bits())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
|
||||
}
|
||||
}
|
||||
|
||||
f.SetInt(intValue)
|
||||
case reflect.Bool:
|
||||
boolValue, err := strconv.ParseBool(value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
|
||||
|
||||
}
|
||||
f.SetBool(boolValue)
|
||||
case reflect.Float32, reflect.Float64:
|
||||
floatValue, err := strconv.ParseFloat(value, f.Type().Bits())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
|
||||
|
||||
}
|
||||
f.SetFloat(floatValue)
|
||||
default:
|
||||
if err := c.applyEnvOverrides(key, f); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
73
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config_command.go
generated
vendored
73
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config_command.go
generated
vendored
@@ -1,73 +0,0 @@
|
||||
package run
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
)
|
||||
|
||||
// PrintConfigCommand represents the command executed by "influxd config".
|
||||
type PrintConfigCommand struct {
|
||||
Stdin io.Reader
|
||||
Stdout io.Writer
|
||||
Stderr io.Writer
|
||||
}
|
||||
|
||||
// NewPrintConfigCommand return a new instance of PrintConfigCommand.
|
||||
func NewPrintConfigCommand() *PrintConfigCommand {
|
||||
return &PrintConfigCommand{
|
||||
Stdin: os.Stdin,
|
||||
Stdout: os.Stdout,
|
||||
Stderr: os.Stderr,
|
||||
}
|
||||
}
|
||||
|
||||
// Run parses and prints the current config loaded.
|
||||
func (cmd *PrintConfigCommand) Run(args ...string) error {
|
||||
// Parse command flags.
|
||||
fs := flag.NewFlagSet("", flag.ContinueOnError)
|
||||
configPath := fs.String("config", "", "")
|
||||
hostname := fs.String("hostname", "", "")
|
||||
fs.Usage = func() { fmt.Fprintln(cmd.Stderr, printConfigUsage) }
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Parse config from path.
|
||||
config, err := cmd.parseConfig(*configPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse config: %s", err)
|
||||
}
|
||||
|
||||
// Override config properties.
|
||||
if *hostname != "" {
|
||||
config.Meta.Hostname = *hostname
|
||||
}
|
||||
|
||||
toml.NewEncoder(cmd.Stdout).Encode(config)
|
||||
fmt.Fprint(cmd.Stdout, "\n")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseConfig parses the config at path.
|
||||
// Returns a demo configuration if path is blank.
|
||||
func (cmd *PrintConfigCommand) parseConfig(path string) (*Config, error) {
|
||||
if path == "" {
|
||||
return NewDemoConfig()
|
||||
}
|
||||
|
||||
config := NewConfig()
|
||||
if _, err := toml.DecodeFile(path, &config); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return config, nil
|
||||
}
|
||||
|
||||
var printConfigUsage = `usage: config
|
||||
|
||||
config displays the default configuration
|
||||
`
|
||||
144
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config_test.go
generated
vendored
144
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config_test.go
generated
vendored
@@ -1,144 +0,0 @@
|
||||
package run_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
"github.com/influxdb/influxdb/cmd/influxd/run"
|
||||
)
|
||||
|
||||
// Ensure the configuration can be parsed.
|
||||
func TestConfig_Parse(t *testing.T) {
|
||||
// Parse configuration.
|
||||
var c run.Config
|
||||
if _, err := toml.Decode(`
|
||||
[meta]
|
||||
dir = "/tmp/meta"
|
||||
|
||||
[data]
|
||||
dir = "/tmp/data"
|
||||
|
||||
[cluster]
|
||||
|
||||
[admin]
|
||||
bind-address = ":8083"
|
||||
|
||||
[http]
|
||||
bind-address = ":8087"
|
||||
|
||||
[[graphite]]
|
||||
protocol = "udp"
|
||||
|
||||
[[graphite]]
|
||||
protocol = "tcp"
|
||||
|
||||
[collectd]
|
||||
bind-address = ":1000"
|
||||
|
||||
[opentsdb]
|
||||
bind-address = ":2000"
|
||||
|
||||
[[udp]]
|
||||
bind-address = ":4444"
|
||||
|
||||
[monitoring]
|
||||
enabled = true
|
||||
|
||||
[continuous_queries]
|
||||
enabled = true
|
||||
`, &c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Validate configuration.
|
||||
if c.Meta.Dir != "/tmp/meta" {
|
||||
t.Fatalf("unexpected meta dir: %s", c.Meta.Dir)
|
||||
} else if c.Data.Dir != "/tmp/data" {
|
||||
t.Fatalf("unexpected data dir: %s", c.Data.Dir)
|
||||
} else if c.Admin.BindAddress != ":8083" {
|
||||
t.Fatalf("unexpected admin bind address: %s", c.Admin.BindAddress)
|
||||
} else if c.HTTPD.BindAddress != ":8087" {
|
||||
t.Fatalf("unexpected api bind address: %s", c.HTTPD.BindAddress)
|
||||
} else if len(c.Graphites) != 2 {
|
||||
t.Fatalf("unexpected graphites count: %d", len(c.Graphites))
|
||||
} else if c.Graphites[0].Protocol != "udp" {
|
||||
t.Fatalf("unexpected graphite protocol(0): %s", c.Graphites[0].Protocol)
|
||||
} else if c.Graphites[1].Protocol != "tcp" {
|
||||
t.Fatalf("unexpected graphite protocol(1): %s", c.Graphites[1].Protocol)
|
||||
} else if c.Collectd.BindAddress != ":1000" {
|
||||
t.Fatalf("unexpected collectd bind address: %s", c.Collectd.BindAddress)
|
||||
} else if c.OpenTSDB.BindAddress != ":2000" {
|
||||
t.Fatalf("unexpected opentsdb bind address: %s", c.OpenTSDB.BindAddress)
|
||||
} else if c.UDPs[0].BindAddress != ":4444" {
|
||||
t.Fatalf("unexpected udp bind address: %s", c.UDPs[0].BindAddress)
|
||||
} else if c.Monitoring.Enabled != true {
|
||||
t.Fatalf("unexpected monitoring enabled: %v", c.Monitoring.Enabled)
|
||||
} else if c.ContinuousQuery.Enabled != true {
|
||||
t.Fatalf("unexpected continuous query enabled: %v", c.ContinuousQuery.Enabled)
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure the configuration can be parsed.
|
||||
func TestConfig_Parse_EnvOverride(t *testing.T) {
|
||||
// Parse configuration.
|
||||
var c run.Config
|
||||
if _, err := toml.Decode(`
|
||||
[meta]
|
||||
dir = "/tmp/meta"
|
||||
|
||||
[data]
|
||||
dir = "/tmp/data"
|
||||
|
||||
[cluster]
|
||||
|
||||
[admin]
|
||||
bind-address = ":8083"
|
||||
|
||||
[http]
|
||||
bind-address = ":8087"
|
||||
|
||||
[[graphite]]
|
||||
protocol = "udp"
|
||||
|
||||
[[graphite]]
|
||||
protocol = "tcp"
|
||||
|
||||
[collectd]
|
||||
bind-address = ":1000"
|
||||
|
||||
[opentsdb]
|
||||
bind-address = ":2000"
|
||||
|
||||
[[udp]]
|
||||
bind-address = ":4444"
|
||||
|
||||
[monitoring]
|
||||
enabled = true
|
||||
|
||||
[continuous_queries]
|
||||
enabled = true
|
||||
`, &c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := os.Setenv("INFLUXDB_UDP_BIND_ADDRESS", ":1234"); err != nil {
|
||||
t.Fatalf("failed to set env var: %v", err)
|
||||
}
|
||||
|
||||
if err := os.Setenv("INFLUXDB_GRAPHITE_1_PROTOCOL", "udp"); err != nil {
|
||||
t.Fatalf("failed to set env var: %v", err)
|
||||
}
|
||||
|
||||
if err := c.ApplyEnvOverrides(); err != nil {
|
||||
t.Fatalf("failed to apply env overrides: %v", err)
|
||||
}
|
||||
|
||||
if c.UDPs[0].BindAddress != ":4444" {
|
||||
t.Fatalf("unexpected udp bind address: %s", c.UDPs[0].BindAddress)
|
||||
}
|
||||
|
||||
if c.Graphites[1].Protocol != "udp" {
|
||||
t.Fatalf("unexpected graphite protocol(0): %s", c.Graphites[0].Protocol)
|
||||
}
|
||||
}
|
||||
536
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go
generated
vendored
536
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go
generated
vendored
@@ -1,536 +0,0 @@
|
||||
package run
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"runtime"
|
||||
"runtime/pprof"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cluster"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/services/admin"
|
||||
"github.com/influxdb/influxdb/services/collectd"
|
||||
"github.com/influxdb/influxdb/services/continuous_querier"
|
||||
"github.com/influxdb/influxdb/services/graphite"
|
||||
"github.com/influxdb/influxdb/services/hh"
|
||||
"github.com/influxdb/influxdb/services/httpd"
|
||||
"github.com/influxdb/influxdb/services/opentsdb"
|
||||
"github.com/influxdb/influxdb/services/precreator"
|
||||
"github.com/influxdb/influxdb/services/retention"
|
||||
"github.com/influxdb/influxdb/services/snapshotter"
|
||||
"github.com/influxdb/influxdb/services/udp"
|
||||
"github.com/influxdb/influxdb/tcp"
|
||||
"github.com/influxdb/influxdb/tsdb"
|
||||
_ "github.com/influxdb/influxdb/tsdb/engine"
|
||||
)
|
||||
|
||||
// Server represents a container for the metadata and storage data and services.
|
||||
// It is built using a Config and it manages the startup and shutdown of all
|
||||
// services in the proper order.
|
||||
type Server struct {
|
||||
version string // Build version
|
||||
|
||||
err chan error
|
||||
closing chan struct{}
|
||||
|
||||
Hostname string
|
||||
BindAddress string
|
||||
Listener net.Listener
|
||||
|
||||
MetaStore *meta.Store
|
||||
TSDBStore *tsdb.Store
|
||||
QueryExecutor *tsdb.QueryExecutor
|
||||
PointsWriter *cluster.PointsWriter
|
||||
ShardWriter *cluster.ShardWriter
|
||||
ShardMapper *cluster.ShardMapper
|
||||
HintedHandoff *hh.Service
|
||||
|
||||
Services []Service
|
||||
|
||||
// These references are required for the tcp muxer.
|
||||
ClusterService *cluster.Service
|
||||
SnapshotterService *snapshotter.Service
|
||||
|
||||
// Server reporting
|
||||
reportingDisabled bool
|
||||
|
||||
// Profiling
|
||||
CPUProfile string
|
||||
MemProfile string
|
||||
}
|
||||
|
||||
// NewServer returns a new instance of Server built from a config.
|
||||
func NewServer(c *Config, version string) (*Server, error) {
|
||||
// Construct base meta store and data store.
|
||||
tsdbStore := tsdb.NewStore(c.Data.Dir)
|
||||
tsdbStore.EngineOptions.Config = c.Data
|
||||
|
||||
s := &Server{
|
||||
version: version,
|
||||
err: make(chan error),
|
||||
closing: make(chan struct{}),
|
||||
|
||||
Hostname: c.Meta.Hostname,
|
||||
BindAddress: c.Meta.BindAddress,
|
||||
|
||||
MetaStore: meta.NewStore(c.Meta),
|
||||
TSDBStore: tsdbStore,
|
||||
|
||||
reportingDisabled: c.ReportingDisabled,
|
||||
}
|
||||
|
||||
// Copy TSDB configuration.
|
||||
s.TSDBStore.EngineOptions.MaxWALSize = c.Data.MaxWALSize
|
||||
s.TSDBStore.EngineOptions.WALFlushInterval = time.Duration(c.Data.WALFlushInterval)
|
||||
s.TSDBStore.EngineOptions.WALPartitionFlushDelay = time.Duration(c.Data.WALPartitionFlushDelay)
|
||||
|
||||
// Set the shard mapper
|
||||
s.ShardMapper = cluster.NewShardMapper(time.Duration(c.Cluster.ShardMapperTimeout))
|
||||
s.ShardMapper.ForceRemoteMapping = c.Cluster.ForceRemoteShardMapping
|
||||
s.ShardMapper.MetaStore = s.MetaStore
|
||||
s.ShardMapper.TSDBStore = s.TSDBStore
|
||||
|
||||
// Initialize query executor.
|
||||
s.QueryExecutor = tsdb.NewQueryExecutor(s.TSDBStore)
|
||||
s.QueryExecutor.MetaStore = s.MetaStore
|
||||
s.QueryExecutor.MetaStatementExecutor = &meta.StatementExecutor{Store: s.MetaStore}
|
||||
s.QueryExecutor.ShardMapper = s.ShardMapper
|
||||
|
||||
// Set the shard writer
|
||||
s.ShardWriter = cluster.NewShardWriter(time.Duration(c.Cluster.ShardWriterTimeout))
|
||||
s.ShardWriter.MetaStore = s.MetaStore
|
||||
|
||||
// Create the hinted handoff service
|
||||
s.HintedHandoff = hh.NewService(c.HintedHandoff, s.ShardWriter)
|
||||
|
||||
// Initialize points writer.
|
||||
s.PointsWriter = cluster.NewPointsWriter()
|
||||
s.PointsWriter.WriteTimeout = time.Duration(c.Cluster.WriteTimeout)
|
||||
s.PointsWriter.MetaStore = s.MetaStore
|
||||
s.PointsWriter.TSDBStore = s.TSDBStore
|
||||
s.PointsWriter.ShardWriter = s.ShardWriter
|
||||
s.PointsWriter.HintedHandoff = s.HintedHandoff
|
||||
|
||||
// Append services.
|
||||
s.appendClusterService(c.Cluster)
|
||||
s.appendPrecreatorService(c.Precreator)
|
||||
s.appendSnapshotterService()
|
||||
s.appendAdminService(c.Admin)
|
||||
s.appendContinuousQueryService(c.ContinuousQuery)
|
||||
s.appendHTTPDService(c.HTTPD)
|
||||
s.appendCollectdService(c.Collectd)
|
||||
if err := s.appendOpenTSDBService(c.OpenTSDB); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, g := range c.UDPs {
|
||||
s.appendUDPService(g)
|
||||
}
|
||||
s.appendRetentionPolicyService(c.Retention)
|
||||
for _, g := range c.Graphites {
|
||||
if err := s.appendGraphiteService(g); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func (s *Server) appendClusterService(c cluster.Config) {
|
||||
srv := cluster.NewService(c)
|
||||
srv.TSDBStore = s.TSDBStore
|
||||
srv.MetaStore = s.MetaStore
|
||||
s.Services = append(s.Services, srv)
|
||||
s.ClusterService = srv
|
||||
}
|
||||
|
||||
func (s *Server) appendSnapshotterService() {
|
||||
srv := snapshotter.NewService()
|
||||
srv.TSDBStore = s.TSDBStore
|
||||
srv.MetaStore = s.MetaStore
|
||||
s.Services = append(s.Services, srv)
|
||||
s.SnapshotterService = srv
|
||||
}
|
||||
|
||||
func (s *Server) appendRetentionPolicyService(c retention.Config) {
|
||||
if !c.Enabled {
|
||||
return
|
||||
}
|
||||
srv := retention.NewService(c)
|
||||
srv.MetaStore = s.MetaStore
|
||||
srv.TSDBStore = s.TSDBStore
|
||||
s.Services = append(s.Services, srv)
|
||||
}
|
||||
|
||||
func (s *Server) appendAdminService(c admin.Config) {
|
||||
if !c.Enabled {
|
||||
return
|
||||
}
|
||||
srv := admin.NewService(c)
|
||||
s.Services = append(s.Services, srv)
|
||||
}
|
||||
|
||||
func (s *Server) appendHTTPDService(c httpd.Config) {
|
||||
if !c.Enabled {
|
||||
return
|
||||
}
|
||||
srv := httpd.NewService(c)
|
||||
srv.Handler.MetaStore = s.MetaStore
|
||||
srv.Handler.QueryExecutor = s.QueryExecutor
|
||||
srv.Handler.PointsWriter = s.PointsWriter
|
||||
srv.Handler.Version = s.version
|
||||
|
||||
// If a ContinuousQuerier service has been started, attach it.
|
||||
for _, srvc := range s.Services {
|
||||
if cqsrvc, ok := srvc.(continuous_querier.ContinuousQuerier); ok {
|
||||
srv.Handler.ContinuousQuerier = cqsrvc
|
||||
}
|
||||
}
|
||||
|
||||
s.Services = append(s.Services, srv)
|
||||
}
|
||||
|
||||
func (s *Server) appendCollectdService(c collectd.Config) {
|
||||
if !c.Enabled {
|
||||
return
|
||||
}
|
||||
srv := collectd.NewService(c)
|
||||
srv.MetaStore = s.MetaStore
|
||||
srv.PointsWriter = s.PointsWriter
|
||||
s.Services = append(s.Services, srv)
|
||||
}
|
||||
|
||||
func (s *Server) appendOpenTSDBService(c opentsdb.Config) error {
|
||||
if !c.Enabled {
|
||||
return nil
|
||||
}
|
||||
srv, err := opentsdb.NewService(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
srv.PointsWriter = s.PointsWriter
|
||||
srv.MetaStore = s.MetaStore
|
||||
s.Services = append(s.Services, srv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) appendGraphiteService(c graphite.Config) error {
|
||||
if !c.Enabled {
|
||||
return nil
|
||||
}
|
||||
srv, err := graphite.NewService(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
srv.PointsWriter = s.PointsWriter
|
||||
srv.MetaStore = s.MetaStore
|
||||
s.Services = append(s.Services, srv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) appendPrecreatorService(c precreator.Config) error {
|
||||
if !c.Enabled {
|
||||
return nil
|
||||
}
|
||||
srv, err := precreator.NewService(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
srv.MetaStore = s.MetaStore
|
||||
s.Services = append(s.Services, srv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Server) appendUDPService(c udp.Config) {
|
||||
if !c.Enabled {
|
||||
return
|
||||
}
|
||||
srv := udp.NewService(c)
|
||||
srv.PointsWriter = s.PointsWriter
|
||||
s.Services = append(s.Services, srv)
|
||||
}
|
||||
|
||||
func (s *Server) appendContinuousQueryService(c continuous_querier.Config) {
|
||||
if !c.Enabled {
|
||||
return
|
||||
}
|
||||
srv := continuous_querier.NewService(c)
|
||||
srv.MetaStore = s.MetaStore
|
||||
srv.QueryExecutor = s.QueryExecutor
|
||||
srv.PointsWriter = s.PointsWriter
|
||||
s.Services = append(s.Services, srv)
|
||||
}
|
||||
|
||||
// Err returns an error channel that multiplexes all out of band errors received from all services.
|
||||
func (s *Server) Err() <-chan error { return s.err }
|
||||
|
||||
// Open opens the meta and data store and all services.
|
||||
func (s *Server) Open() error {
|
||||
if err := func() error {
|
||||
// Start profiling, if set.
|
||||
startProfile(s.CPUProfile, s.MemProfile)
|
||||
|
||||
host, port, err := s.hostAddr()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
hostport := net.JoinHostPort(host, port)
|
||||
addr, err := net.ResolveTCPAddr("tcp", hostport)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve tcp: addr=%s, err=%s", hostport, err)
|
||||
}
|
||||
s.MetaStore.Addr = addr
|
||||
s.MetaStore.RemoteAddr = &tcpaddr{hostport}
|
||||
|
||||
// Open shared TCP connection.
|
||||
ln, err := net.Listen("tcp", s.BindAddress)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listen: %s", err)
|
||||
}
|
||||
s.Listener = ln
|
||||
|
||||
// The port 0 is used, we need to retrieve the port assigned by the kernel
|
||||
if strings.HasSuffix(s.BindAddress, ":0") {
|
||||
s.MetaStore.Addr = ln.Addr()
|
||||
}
|
||||
|
||||
// Multiplex listener.
|
||||
mux := tcp.NewMux()
|
||||
s.MetaStore.RaftListener = mux.Listen(meta.MuxRaftHeader)
|
||||
s.MetaStore.ExecListener = mux.Listen(meta.MuxExecHeader)
|
||||
s.MetaStore.RPCListener = mux.Listen(meta.MuxRPCHeader)
|
||||
|
||||
s.ClusterService.Listener = mux.Listen(cluster.MuxHeader)
|
||||
s.SnapshotterService.Listener = mux.Listen(snapshotter.MuxHeader)
|
||||
go mux.Serve(ln)
|
||||
|
||||
// Open meta store.
|
||||
if err := s.MetaStore.Open(); err != nil {
|
||||
return fmt.Errorf("open meta store: %s", err)
|
||||
}
|
||||
go s.monitorErrorChan(s.MetaStore.Err())
|
||||
|
||||
// Wait for the store to initialize.
|
||||
<-s.MetaStore.Ready()
|
||||
|
||||
// Open TSDB store.
|
||||
if err := s.TSDBStore.Open(); err != nil {
|
||||
return fmt.Errorf("open tsdb store: %s", err)
|
||||
}
|
||||
|
||||
// Open the hinted handoff service
|
||||
if err := s.HintedHandoff.Open(); err != nil {
|
||||
return fmt.Errorf("open hinted handoff: %s", err)
|
||||
}
|
||||
|
||||
for _, service := range s.Services {
|
||||
if err := service.Open(); err != nil {
|
||||
return fmt.Errorf("open service: %s", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Start the reporting service, if not disabled.
|
||||
if !s.reportingDisabled {
|
||||
go s.startServerReporting()
|
||||
}
|
||||
|
||||
return nil
|
||||
|
||||
}(); err != nil {
|
||||
s.Close()
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close shuts down the meta and data stores and all services.
|
||||
func (s *Server) Close() error {
|
||||
stopProfile()
|
||||
|
||||
if s.Listener != nil {
|
||||
s.Listener.Close()
|
||||
}
|
||||
if s.MetaStore != nil {
|
||||
s.MetaStore.Close()
|
||||
}
|
||||
if s.TSDBStore != nil {
|
||||
s.TSDBStore.Close()
|
||||
}
|
||||
if s.HintedHandoff != nil {
|
||||
s.HintedHandoff.Close()
|
||||
}
|
||||
for _, service := range s.Services {
|
||||
service.Close()
|
||||
}
|
||||
|
||||
close(s.closing)
|
||||
return nil
|
||||
}
|
||||
|
||||
// startServerReporting starts periodic server reporting.
|
||||
func (s *Server) startServerReporting() {
|
||||
for {
|
||||
select {
|
||||
case <-s.closing:
|
||||
return
|
||||
default:
|
||||
}
|
||||
if err := s.MetaStore.WaitForLeader(30 * time.Second); err != nil {
|
||||
log.Printf("no leader available for reporting: %s", err.Error())
|
||||
time.Sleep(time.Second)
|
||||
continue
|
||||
}
|
||||
s.reportServer()
|
||||
<-time.After(24 * time.Hour)
|
||||
}
|
||||
}
|
||||
|
||||
// reportServer reports anonymous statistics about the system.
|
||||
func (s *Server) reportServer() {
|
||||
dis, err := s.MetaStore.Databases()
|
||||
if err != nil {
|
||||
log.Printf("failed to retrieve databases for reporting: %s", err.Error())
|
||||
return
|
||||
}
|
||||
numDatabases := len(dis)
|
||||
|
||||
numMeasurements := 0
|
||||
numSeries := 0
|
||||
for _, di := range dis {
|
||||
d := s.TSDBStore.DatabaseIndex(di.Name)
|
||||
if d == nil {
|
||||
// No data in this store for this database.
|
||||
continue
|
||||
}
|
||||
m, s := d.MeasurementSeriesCounts()
|
||||
numMeasurements += m
|
||||
numSeries += s
|
||||
}
|
||||
|
||||
clusterID, err := s.MetaStore.ClusterID()
|
||||
if err != nil {
|
||||
log.Printf("failed to retrieve cluster ID for reporting: %s", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
json := fmt.Sprintf(`[{
|
||||
"name":"reports",
|
||||
"columns":["os", "arch", "version", "server_id", "cluster_id", "num_series", "num_measurements", "num_databases"],
|
||||
"points":[["%s", "%s", "%s", "%x", "%x", "%d", "%d", "%d"]]
|
||||
}]`, runtime.GOOS, runtime.GOARCH, s.version, s.MetaStore.NodeID(), clusterID, numSeries, numMeasurements, numDatabases)
|
||||
|
||||
data := bytes.NewBufferString(json)
|
||||
|
||||
log.Printf("Sending anonymous usage statistics to m.influxdb.com")
|
||||
|
||||
client := http.Client{Timeout: time.Duration(5 * time.Second)}
|
||||
go client.Post("http://m.influxdb.com:8086/db/reporting/series?u=reporter&p=influxdb", "application/json", data)
|
||||
}
|
||||
|
||||
// monitorErrorChan reads an error channel and resends it through the server.
|
||||
func (s *Server) monitorErrorChan(ch <-chan error) {
|
||||
for {
|
||||
select {
|
||||
case err, ok := <-ch:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
s.err <- err
|
||||
case <-s.closing:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// hostAddr returns the host and port that remote nodes will use to reach this
|
||||
// node.
|
||||
func (s *Server) hostAddr() (string, string, error) {
|
||||
// Resolve host to address.
|
||||
_, port, err := net.SplitHostPort(s.BindAddress)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("split bind address: %s", err)
|
||||
}
|
||||
|
||||
host := s.Hostname
|
||||
|
||||
// See if we might have a port that will override the BindAddress port
|
||||
if host != "" && host[len(host)-1] >= '0' && host[len(host)-1] <= '9' && strings.Contains(host, ":") {
|
||||
hostArg, portArg, err := net.SplitHostPort(s.Hostname)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
|
||||
if hostArg != "" {
|
||||
host = hostArg
|
||||
}
|
||||
|
||||
if portArg != "" {
|
||||
port = portArg
|
||||
}
|
||||
}
|
||||
return host, port, nil
|
||||
}
|
||||
|
||||
// Service represents a service attached to the server.
|
||||
type Service interface {
|
||||
Open() error
|
||||
Close() error
|
||||
}
|
||||
|
||||
// prof stores the file locations of active profiles.
|
||||
var prof struct {
|
||||
cpu *os.File
|
||||
mem *os.File
|
||||
}
|
||||
|
||||
// StartProfile initializes the cpu and memory profile, if specified.
|
||||
func startProfile(cpuprofile, memprofile string) {
|
||||
if cpuprofile != "" {
|
||||
f, err := os.Create(cpuprofile)
|
||||
if err != nil {
|
||||
log.Fatalf("cpuprofile: %v", err)
|
||||
}
|
||||
log.Printf("writing CPU profile to: %s\n", cpuprofile)
|
||||
prof.cpu = f
|
||||
pprof.StartCPUProfile(prof.cpu)
|
||||
}
|
||||
|
||||
if memprofile != "" {
|
||||
f, err := os.Create(memprofile)
|
||||
if err != nil {
|
||||
log.Fatalf("memprofile: %v", err)
|
||||
}
|
||||
log.Printf("writing mem profile to: %s\n", memprofile)
|
||||
prof.mem = f
|
||||
runtime.MemProfileRate = 4096
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// StopProfile closes the cpu and memory profiles if they are running.
|
||||
func stopProfile() {
|
||||
if prof.cpu != nil {
|
||||
pprof.StopCPUProfile()
|
||||
prof.cpu.Close()
|
||||
log.Println("CPU profile stopped")
|
||||
}
|
||||
if prof.mem != nil {
|
||||
pprof.Lookup("heap").WriteTo(prof.mem, 0)
|
||||
prof.mem.Close()
|
||||
log.Println("mem profile stopped")
|
||||
}
|
||||
}
|
||||
|
||||
type tcpaddr struct{ host string }
|
||||
|
||||
func (a *tcpaddr) Network() string { return "tcp" }
|
||||
func (a *tcpaddr) String() string { return a.host }
|
||||
312
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_helpers_test.go
generated
vendored
312
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_helpers_test.go
generated
vendored
@@ -1,312 +0,0 @@
|
||||
// This package is a set of convenience helpers and structs to make integration testing easier
|
||||
package run_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/cmd/influxd/run"
|
||||
"github.com/influxdb/influxdb/meta"
|
||||
"github.com/influxdb/influxdb/services/httpd"
|
||||
"github.com/influxdb/influxdb/toml"
|
||||
)
|
||||
|
||||
// Server represents a test wrapper for run.Server.
|
||||
type Server struct {
|
||||
*run.Server
|
||||
Config *run.Config
|
||||
}
|
||||
|
||||
// NewServer returns a new instance of Server.
|
||||
func NewServer(c *run.Config) *Server {
|
||||
srv, _ := run.NewServer(c, "testServer")
|
||||
s := Server{
|
||||
Server: srv,
|
||||
Config: c,
|
||||
}
|
||||
s.TSDBStore.EngineOptions.Config = c.Data
|
||||
configureLogging(&s)
|
||||
return &s
|
||||
}
|
||||
|
||||
// OpenServer opens a test server.
|
||||
func OpenServer(c *run.Config, joinURLs string) *Server {
|
||||
s := NewServer(c)
|
||||
configureLogging(s)
|
||||
if err := s.Open(); err != nil {
|
||||
panic(err.Error())
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// OpenServerWithVersion opens a test server with a specific version.
|
||||
func OpenServerWithVersion(c *run.Config, version string) *Server {
|
||||
srv, _ := run.NewServer(c, version)
|
||||
s := Server{
|
||||
Server: srv,
|
||||
Config: c,
|
||||
}
|
||||
configureLogging(&s)
|
||||
if err := s.Open(); err != nil {
|
||||
panic(err.Error())
|
||||
}
|
||||
|
||||
return &s
|
||||
}
|
||||
|
||||
// Close shuts down the server and removes all temporary paths.
|
||||
func (s *Server) Close() {
|
||||
os.RemoveAll(s.Config.Meta.Dir)
|
||||
os.RemoveAll(s.Config.Data.Dir)
|
||||
os.RemoveAll(s.Config.HintedHandoff.Dir)
|
||||
s.Server.Close()
|
||||
}
|
||||
|
||||
// URL returns the base URL for the httpd endpoint.
|
||||
func (s *Server) URL() string {
|
||||
for _, service := range s.Services {
|
||||
if service, ok := service.(*httpd.Service); ok {
|
||||
return "http://" + service.Addr().String()
|
||||
}
|
||||
}
|
||||
panic("httpd server not found in services")
|
||||
}
|
||||
|
||||
// CreateDatabaseAndRetentionPolicy will create the database and retention policy.
|
||||
func (s *Server) CreateDatabaseAndRetentionPolicy(db string, rp *meta.RetentionPolicyInfo) error {
|
||||
if _, err := s.MetaStore.CreateDatabase(db); err != nil {
|
||||
return err
|
||||
} else if _, err := s.MetaStore.CreateRetentionPolicy(db, rp); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Query executes a query against the server and returns the results.
|
||||
func (s *Server) Query(query string) (results string, err error) {
|
||||
return s.QueryWithParams(query, nil)
|
||||
}
|
||||
|
||||
// Query executes a query against the server and returns the results.
|
||||
func (s *Server) QueryWithParams(query string, values url.Values) (results string, err error) {
|
||||
if values == nil {
|
||||
values = url.Values{}
|
||||
}
|
||||
values.Set("q", query)
|
||||
resp, err := http.Get(s.URL() + "/query?" + values.Encode())
|
||||
if err != nil {
|
||||
return "", err
|
||||
//} else if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusBadRequest {
|
||||
}
|
||||
body := string(MustReadAll(resp.Body))
|
||||
switch resp.StatusCode {
|
||||
case http.StatusBadRequest:
|
||||
if !expectPattern(".*error parsing query*.", body) {
|
||||
return "", fmt.Errorf("unexpected status code: code=%d, body=%s", resp.StatusCode, body)
|
||||
}
|
||||
return body, nil
|
||||
case http.StatusOK:
|
||||
return body, nil
|
||||
default:
|
||||
return "", fmt.Errorf("unexpected status code: code=%d, body=%s", resp.StatusCode, body)
|
||||
}
|
||||
}
|
||||
|
||||
// Write executes a write against the server and returns the results.
|
||||
func (s *Server) Write(db, rp, body string, params url.Values) (results string, err error) {
|
||||
if params == nil {
|
||||
params = url.Values{}
|
||||
}
|
||||
if params.Get("db") == "" {
|
||||
params.Set("db", db)
|
||||
}
|
||||
if params.Get("rp") == "" {
|
||||
params.Set("rp", rp)
|
||||
}
|
||||
resp, err := http.Post(s.URL()+"/write?"+params.Encode(), "", strings.NewReader(body))
|
||||
if err != nil {
|
||||
return "", err
|
||||
} else if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return "", fmt.Errorf("invalid status code: code=%d, body=%s", resp.StatusCode, MustReadAll(resp.Body))
|
||||
}
|
||||
return string(MustReadAll(resp.Body)), nil
|
||||
}
|
||||
|
||||
// NewConfig returns the default config with temporary paths.
|
||||
func NewConfig() *run.Config {
|
||||
c := run.NewConfig()
|
||||
c.ReportingDisabled = true
|
||||
c.Meta.Dir = MustTempFile()
|
||||
c.Meta.BindAddress = "127.0.0.1:0"
|
||||
c.Meta.HeartbeatTimeout = toml.Duration(50 * time.Millisecond)
|
||||
c.Meta.ElectionTimeout = toml.Duration(50 * time.Millisecond)
|
||||
c.Meta.LeaderLeaseTimeout = toml.Duration(50 * time.Millisecond)
|
||||
c.Meta.CommitTimeout = toml.Duration(5 * time.Millisecond)
|
||||
|
||||
c.Data.Dir = MustTempFile()
|
||||
c.Data.WALDir = MustTempFile()
|
||||
|
||||
c.HintedHandoff.Dir = MustTempFile()
|
||||
|
||||
c.HTTPD.Enabled = true
|
||||
c.HTTPD.BindAddress = "127.0.0.1:0"
|
||||
c.HTTPD.LogEnabled = testing.Verbose()
|
||||
|
||||
return c
|
||||
}
|
||||
|
||||
func newRetentionPolicyInfo(name string, rf int, duration time.Duration) *meta.RetentionPolicyInfo {
|
||||
return &meta.RetentionPolicyInfo{Name: name, ReplicaN: rf, Duration: duration}
|
||||
}
|
||||
|
||||
func maxFloat64() string {
|
||||
maxFloat64, _ := json.Marshal(math.MaxFloat64)
|
||||
return string(maxFloat64)
|
||||
}
|
||||
|
||||
func maxInt64() string {
|
||||
maxInt64, _ := json.Marshal(^int64(0))
|
||||
return string(maxInt64)
|
||||
}
|
||||
|
||||
func now() time.Time {
|
||||
return time.Now().UTC()
|
||||
}
|
||||
|
||||
func yesterday() time.Time {
|
||||
return now().Add(-1 * time.Hour * 24)
|
||||
}
|
||||
|
||||
func mustParseTime(layout, value string) time.Time {
|
||||
tm, err := time.Parse(layout, value)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return tm
|
||||
}
|
||||
|
||||
// MustReadAll reads r. Panic on error.
|
||||
func MustReadAll(r io.Reader) []byte {
|
||||
b, err := ioutil.ReadAll(r)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// MustTempFile returns a path to a temporary file.
|
||||
func MustTempFile() string {
|
||||
f, err := ioutil.TempFile("", "influxd-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
f.Close()
|
||||
os.Remove(f.Name())
|
||||
return f.Name()
|
||||
}
|
||||
|
||||
func expectPattern(exp, act string) bool {
|
||||
re := regexp.MustCompile(exp)
|
||||
if !re.MatchString(act) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type Query struct {
|
||||
name string
|
||||
command string
|
||||
params url.Values
|
||||
exp, act string
|
||||
pattern bool
|
||||
skip bool
|
||||
}
|
||||
|
||||
// Execute runs the command and returns an err if it fails
|
||||
func (q *Query) Execute(s *Server) (err error) {
|
||||
if q.params == nil {
|
||||
q.act, err = s.Query(q.command)
|
||||
return
|
||||
}
|
||||
q.act, err = s.QueryWithParams(q.command, q.params)
|
||||
return
|
||||
}
|
||||
|
||||
func (q *Query) success() bool {
|
||||
if q.pattern {
|
||||
return expectPattern(q.exp, q.act)
|
||||
}
|
||||
return q.exp == q.act
|
||||
}
|
||||
|
||||
func (q *Query) Error(err error) string {
|
||||
return fmt.Sprintf("%s: %v", q.name, err)
|
||||
}
|
||||
|
||||
func (q *Query) failureMessage() string {
|
||||
return fmt.Sprintf("%s: unexpected results\nquery: %s\nexp: %s\nactual: %s\n", q.name, q.command, q.exp, q.act)
|
||||
}
|
||||
|
||||
type Test struct {
|
||||
initialized bool
|
||||
write string
|
||||
params url.Values
|
||||
db string
|
||||
rp string
|
||||
exp string
|
||||
queries []*Query
|
||||
}
|
||||
|
||||
func NewTest(db, rp string) Test {
|
||||
return Test{
|
||||
db: db,
|
||||
rp: rp,
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Test) addQueries(q ...*Query) {
|
||||
t.queries = append(t.queries, q...)
|
||||
}
|
||||
|
||||
func (t *Test) init(s *Server) error {
|
||||
if t.write == "" || t.initialized {
|
||||
return nil
|
||||
}
|
||||
t.initialized = true
|
||||
if res, err := s.Write(t.db, t.rp, t.write, t.params); err != nil {
|
||||
return err
|
||||
} else if t.exp != res {
|
||||
return fmt.Errorf("unexpected results\nexp: %s\ngot: %s\n", t.exp, res)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func configureLogging(s *Server) {
|
||||
// Set the logger to discard unless verbose is on
|
||||
if !testing.Verbose() {
|
||||
type logSetter interface {
|
||||
SetLogger(*log.Logger)
|
||||
}
|
||||
nullLogger := log.New(ioutil.Discard, "", 0)
|
||||
s.MetaStore.Logger = nullLogger
|
||||
s.TSDBStore.Logger = nullLogger
|
||||
s.HintedHandoff.SetLogger(nullLogger)
|
||||
for _, service := range s.Services {
|
||||
if service, ok := service.(logSetter); ok {
|
||||
service.SetLogger(nullLogger)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
3719
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_test.go
generated
vendored
3719
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_test.go
generated
vendored
File diff suppressed because it is too large
Load Diff
150
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_test.md
generated
vendored
150
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_test.md
generated
vendored
@@ -1,150 +0,0 @@
|
||||
# Server Integration Tests
|
||||
|
||||
Currently, the file `server_test.go` has integration tests for single node scenarios.
|
||||
At some point we'll need to add cluster tests, and may add them in a different file, or
|
||||
rename `server_test.go` to `server_single_node_test.go` or something like that.
|
||||
|
||||
## What is in a test?
|
||||
|
||||
Each test is broken apart effectively into the following areas:
|
||||
|
||||
- Write sample data
|
||||
- Use cases for table driven test, that include a command (typically a query) and an expected result.
|
||||
|
||||
When each test runs it does the following:
|
||||
|
||||
- init: determines if there are any writes and if so, writes them to the in-memory database
|
||||
- queries: iterate through each query, executing the command, and comparing the results to the expected result.
|
||||
|
||||
## Idempotent - Allows for parallel tests
|
||||
|
||||
Each test should be `idempotent`, meaining that its data will not be affected by other tests, or use cases within the table tests themselves.
|
||||
This allows for parallel testing, keeping the test suite total execution time very low.
|
||||
|
||||
### Basic sample test
|
||||
|
||||
```go
|
||||
// Ensure the server can have a database with multiple measurements.
|
||||
func TestServer_Query_Multiple_Measurements(t *testing.T) {
|
||||
t.Parallel()
|
||||
s := OpenServer(NewConfig(), "")
|
||||
defer s.Close()
|
||||
|
||||
if err := s.CreateDatabaseAndRetentionPolicy("db0", newRetentionPolicyInfo("rp0", 1, 1*time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Make sure we do writes for measurements that will span across shards
|
||||
writes := []string{
|
||||
fmt.Sprintf("cpu,host=server01 value=100,core=4 %d", mustParseTime(time.RFC3339Nano, "2000-01-01T00:00:00Z").UnixNano()),
|
||||
fmt.Sprintf("cpu1,host=server02 value=50,core=2 %d", mustParseTime(time.RFC3339Nano, "2015-01-01T00:00:00Z").UnixNano()),
|
||||
}
|
||||
test := NewTest("db0", "rp0")
|
||||
test.write = strings.Join(writes, "\n")
|
||||
|
||||
test.addQueries([]*Query{
|
||||
&Query{
|
||||
name: "measurement in one shard but not another shouldn't panic server",
|
||||
command: `SELECT host,value FROM db0.rp0.cpu`,
|
||||
exp: `{"results":[{"series":[{"name":"cpu","tags":{"host":"server01"},"columns":["time","value"],"values":[["2000-01-01T00:00:00Z",100]]}]}]}`,
|
||||
},
|
||||
}...)
|
||||
|
||||
if err := test.init(s); err != nil {
|
||||
t.Fatalf("test init failed: %s", err)
|
||||
}
|
||||
|
||||
for _, query := range test.queries {
|
||||
if query.skip {
|
||||
t.Logf("SKIP:: %s", query.name)
|
||||
continue
|
||||
}
|
||||
if err := query.Execute(s); err != nil {
|
||||
t.Error(query.Error(err))
|
||||
} else if !query.success() {
|
||||
t.Error(query.failureMessage())
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Let's break this down:
|
||||
|
||||
In this test, we first tell it to run in parallel with the `t.Parallel()` call.
|
||||
|
||||
We then open a new server with:
|
||||
|
||||
```go
|
||||
s := OpenServer(NewConfig(), "")
|
||||
defer s.Close()
|
||||
```
|
||||
|
||||
If needed, we create a database and default retention policy. This is usually needed
|
||||
when inserting and querying data. This is not needed if you are testing commands like `CREATE DATABASE`, `SHOW DIAGNOSTICS`, etc.
|
||||
|
||||
```go
|
||||
if err := s.CreateDatabaseAndRetentionPolicy("db0", newRetentionPolicyInfo("rp0", 1, 1*time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
Next, set up the write data you need:
|
||||
|
||||
```go
|
||||
writes := []string{
|
||||
fmt.Sprintf("cpu,host=server01 value=100,core=4 %d", mustParseTime(time.RFC3339Nano, "2000-01-01T00:00:00Z").UnixNano()),
|
||||
fmt.Sprintf("cpu1,host=server02 value=50,core=2 %d", mustParseTime(time.RFC3339Nano, "2015-01-01T00:00:00Z").UnixNano()),
|
||||
}
|
||||
```
|
||||
Create a new test with the database and retention policy:
|
||||
|
||||
```go
|
||||
test := NewTest("db0", "rp0")
|
||||
```
|
||||
|
||||
Send in the writes:
|
||||
```go
|
||||
test.write = strings.Join(writes, "\n")
|
||||
```
|
||||
|
||||
Add some queries (the second one is mocked out to show how to add more than one):
|
||||
|
||||
```go
|
||||
test.addQueries([]*Query{
|
||||
&Query{
|
||||
name: "measurement in one shard but not another shouldn't panic server",
|
||||
command: `SELECT host,value FROM db0.rp0.cpu`,
|
||||
exp: `{"results":[{"series":[{"name":"cpu","tags":{"host":"server01"},"columns":["time","value"],"values":[["2000-01-01T00:00:00Z",100]]}]}]}`,
|
||||
},
|
||||
&Query{
|
||||
name: "another test here...",
|
||||
command: `Some query command`,
|
||||
exp: `the expected results`,
|
||||
},
|
||||
}...)
|
||||
```
|
||||
|
||||
The rest of the code is boilerplate execution code. It is purposefully not refactored out to a helper
|
||||
to make sure the test failure reports the proper lines for debugging purposes.
|
||||
|
||||
#### Running the tests
|
||||
|
||||
To run the tests:
|
||||
|
||||
```sh
|
||||
go test ./cmd/influxd/run -parallel 500 -timeout 10s
|
||||
```
|
||||
|
||||
#### Running a specific test
|
||||
|
||||
```sh
|
||||
go test ./cmd/influxd/run -parallel 500 -timeout 10s -run TestServer_Query_Fill
|
||||
```
|
||||
|
||||
#### Verbose feedback
|
||||
|
||||
By default, all logs are silenced when testing. If you pass in the `-v` flag, the test suite becomes verbose, and enables all logging in the system
|
||||
|
||||
```sh
|
||||
go test ./cmd/influxd/run -parallel 500 -timeout 10s -run TestServer_Query_Fill -v
|
||||
```
|
||||
143
Godeps/_workspace/src/github.com/influxdb/influxdb/diagnostics.go
generated
vendored
143
Godeps/_workspace/src/github.com/influxdb/influxdb/diagnostics.go
generated
vendored
@@ -1,143 +0,0 @@
|
||||
package influxdb
|
||||
|
||||
import (
|
||||
"os"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/influxql"
|
||||
)
|
||||
|
||||
// GoDiagnostics captures basic information about the runtime.
|
||||
type GoDiagnostics struct {
|
||||
GoMaxProcs int
|
||||
NumGoroutine int
|
||||
Version string
|
||||
}
|
||||
|
||||
// NewGoDiagnostics returns a GoDiagnostics object.
|
||||
func NewGoDiagnostics() *GoDiagnostics {
|
||||
return &GoDiagnostics{
|
||||
GoMaxProcs: runtime.GOMAXPROCS(0),
|
||||
NumGoroutine: runtime.NumGoroutine(),
|
||||
Version: runtime.Version(),
|
||||
}
|
||||
}
|
||||
|
||||
// AsRow returns the GoDiagnostic object as an InfluxQL row.
|
||||
func (g *GoDiagnostics) AsRow(measurement string, tags map[string]string) *influxql.Row {
|
||||
return &influxql.Row{
|
||||
Name: measurement,
|
||||
Columns: []string{"time", "goMaxProcs", "numGoRoutine", "version"},
|
||||
Tags: tags,
|
||||
Values: [][]interface{}{[]interface{}{time.Now().UTC(),
|
||||
g.GoMaxProcs, g.NumGoroutine, g.Version}},
|
||||
}
|
||||
}
|
||||
|
||||
// SystemDiagnostics captures basic machine data.
|
||||
type SystemDiagnostics struct {
|
||||
Hostname string
|
||||
PID int
|
||||
OS string
|
||||
Arch string
|
||||
NumCPU int
|
||||
}
|
||||
|
||||
// NewSystemDiagnostics returns a SystemDiagnostics object.
|
||||
func NewSystemDiagnostics() *SystemDiagnostics {
|
||||
hostname, err := os.Hostname()
|
||||
if err != nil {
|
||||
hostname = "unknown"
|
||||
}
|
||||
|
||||
return &SystemDiagnostics{
|
||||
Hostname: hostname,
|
||||
PID: os.Getpid(),
|
||||
OS: runtime.GOOS,
|
||||
Arch: runtime.GOARCH,
|
||||
NumCPU: runtime.NumCPU(),
|
||||
}
|
||||
}
|
||||
|
||||
// AsRow returns the GoDiagnostic object as an InfluxQL row.
|
||||
func (s *SystemDiagnostics) AsRow(measurement string, tags map[string]string) *influxql.Row {
|
||||
return &influxql.Row{
|
||||
Name: measurement,
|
||||
Columns: []string{"time", "hostname", "pid", "os", "arch", "numCPU"},
|
||||
Tags: tags,
|
||||
Values: [][]interface{}{[]interface{}{time.Now().UTC(),
|
||||
s.Hostname, s.PID, s.OS, s.Arch, s.NumCPU}},
|
||||
}
|
||||
}
|
||||
|
||||
// MemoryDiagnostics captures Go memory stats.
|
||||
type MemoryDiagnostics struct {
|
||||
Alloc int64
|
||||
TotalAlloc int64
|
||||
Sys int64
|
||||
Lookups int64
|
||||
Mallocs int64
|
||||
Frees int64
|
||||
HeapAlloc int64
|
||||
HeapSys int64
|
||||
HeapIdle int64
|
||||
HeapInUse int64
|
||||
HeapReleased int64
|
||||
HeapObjects int64
|
||||
PauseTotalNs int64
|
||||
NumGC int64
|
||||
}
|
||||
|
||||
// NewMemoryDiagnostics returns a MemoryDiagnostics object.
|
||||
func NewMemoryDiagnostics() *MemoryDiagnostics {
|
||||
var m runtime.MemStats
|
||||
runtime.ReadMemStats(&m)
|
||||
|
||||
return &MemoryDiagnostics{
|
||||
Alloc: int64(m.Alloc),
|
||||
TotalAlloc: int64(m.TotalAlloc),
|
||||
Sys: int64(m.Sys),
|
||||
Lookups: int64(m.Lookups),
|
||||
Mallocs: int64(m.Mallocs),
|
||||
Frees: int64(m.Frees),
|
||||
HeapAlloc: int64(m.HeapAlloc),
|
||||
HeapSys: int64(m.HeapSys),
|
||||
HeapIdle: int64(m.HeapIdle),
|
||||
HeapInUse: int64(m.HeapInuse),
|
||||
HeapReleased: int64(m.HeapReleased),
|
||||
HeapObjects: int64(m.HeapObjects),
|
||||
PauseTotalNs: int64(m.PauseTotalNs),
|
||||
NumGC: int64(m.NumGC),
|
||||
}
|
||||
}
|
||||
|
||||
// AsRow returns the MemoryDiagnostics object as an InfluxQL row.
|
||||
func (m *MemoryDiagnostics) AsRow(measurement string, tags map[string]string) *influxql.Row {
|
||||
return &influxql.Row{
|
||||
Name: measurement,
|
||||
Columns: []string{"time", "alloc", "totalAlloc", "sys", "lookups", "mallocs", "frees", "heapAlloc",
|
||||
"heapSys", "heapIdle", "heapInUse", "heapReleased", "heapObjects", "pauseTotalNs", "numGG"},
|
||||
Tags: tags,
|
||||
Values: [][]interface{}{[]interface{}{time.Now().UTC(),
|
||||
m.Alloc, m.TotalAlloc, m.Sys, m.Lookups, m.Mallocs, m.Frees, m.HeapAlloc,
|
||||
m.HeapSys, m.HeapIdle, m.HeapInUse, m.HeapReleased, m.HeapObjects, m.PauseTotalNs, m.NumGC}},
|
||||
}
|
||||
}
|
||||
|
||||
// BuildDiagnostics capture basic build version information.
|
||||
type BuildDiagnostics struct {
|
||||
Version string
|
||||
CommitHash string
|
||||
}
|
||||
|
||||
// AsRow returns the BuildDiagnostics object as an InfluxQL row.
|
||||
func (b *BuildDiagnostics) AsRow(measurement string, tags map[string]string) *influxql.Row {
|
||||
return &influxql.Row{
|
||||
Name: measurement,
|
||||
Columns: []string{"time", "version", "commitHash"},
|
||||
Tags: tags,
|
||||
Values: [][]interface{}{[]interface{}{time.Now().UTC(),
|
||||
b.Version, b.CommitHash}},
|
||||
}
|
||||
}
|
||||
78
Godeps/_workspace/src/github.com/influxdb/influxdb/errors.go
generated
vendored
78
Godeps/_workspace/src/github.com/influxdb/influxdb/errors.go
generated
vendored
@@ -1,78 +0,0 @@
|
||||
package influxdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"runtime"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrFieldsRequired is returned when a point does not any fields.
|
||||
ErrFieldsRequired = errors.New("fields required")
|
||||
|
||||
// ErrFieldTypeConflict is returned when a new field already exists with a different type.
|
||||
ErrFieldTypeConflict = errors.New("field type conflict")
|
||||
)
|
||||
|
||||
func ErrDatabaseNotFound(name string) error { return fmt.Errorf("database not found: %s", name) }
|
||||
|
||||
func ErrMeasurementNotFound(name string) error { return fmt.Errorf("measurement not found: %s", name) }
|
||||
|
||||
func Errorf(format string, a ...interface{}) (err error) {
|
||||
if _, file, line, ok := runtime.Caller(2); ok {
|
||||
a = append(a, file, line)
|
||||
err = fmt.Errorf(format+" (%s:%d)", a...)
|
||||
} else {
|
||||
err = fmt.Errorf(format, a...)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// IsClientError indicates whether an error is a known client error.
|
||||
func IsClientError(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
|
||||
if err == ErrFieldsRequired {
|
||||
return true
|
||||
}
|
||||
if err == ErrFieldTypeConflict {
|
||||
return true
|
||||
}
|
||||
|
||||
if strings.Contains(err.Error(), ErrFieldTypeConflict.Error()) {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// mustMarshal encodes a value to JSON.
|
||||
// This will panic if an error occurs. This should only be used internally when
|
||||
// an invalid marshal will cause corruption and a panic is appropriate.
|
||||
func mustMarshalJSON(v interface{}) []byte {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
panic("marshal: " + err.Error())
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// mustUnmarshalJSON decodes a value from JSON.
|
||||
// This will panic if an error occurs. This should only be used internally when
|
||||
// an invalid unmarshal will cause corruption and a panic is appropriate.
|
||||
func mustUnmarshalJSON(b []byte, v interface{}) {
|
||||
if err := json.Unmarshal(b, v); err != nil {
|
||||
panic("unmarshal: " + err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// assert will panic with a given formatted message if the given condition is false.
|
||||
func assert(condition bool, msg string, v ...interface{}) {
|
||||
if !condition {
|
||||
panic(fmt.Sprintf("assert failed: "+msg, v...))
|
||||
}
|
||||
}
|
||||
1
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/.rvmrc
generated
vendored
1
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/.rvmrc
generated
vendored
@@ -1 +0,0 @@
|
||||
rvm use ruby-2.1.0@burn-in --create
|
||||
4
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/Gemfile
generated
vendored
4
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/Gemfile
generated
vendored
@@ -1,4 +0,0 @@
|
||||
source 'https://rubygems.org'
|
||||
|
||||
gem "colorize"
|
||||
gem "influxdb"
|
||||
14
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/Gemfile.lock
generated
vendored
14
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/Gemfile.lock
generated
vendored
@@ -1,14 +0,0 @@
|
||||
GEM
|
||||
remote: https://rubygems.org/
|
||||
specs:
|
||||
colorize (0.6.0)
|
||||
influxdb (0.0.16)
|
||||
json
|
||||
json (1.8.1)
|
||||
|
||||
PLATFORMS
|
||||
ruby
|
||||
|
||||
DEPENDENCIES
|
||||
colorize
|
||||
influxdb
|
||||
79
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/burn-in.rb
generated
vendored
79
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/burn-in.rb
generated
vendored
@@ -1,79 +0,0 @@
|
||||
require "influxdb"
|
||||
require "colorize"
|
||||
require "benchmark"
|
||||
|
||||
require_relative "log"
|
||||
require_relative "random_gaussian"
|
||||
|
||||
BATCH_SIZE = 10_000
|
||||
|
||||
Log.info "Starting burn-in suite"
|
||||
master = InfluxDB::Client.new
|
||||
master.delete_database("burn-in") rescue nil
|
||||
master.create_database("burn-in")
|
||||
master.create_database_user("burn-in", "user", "pass")
|
||||
|
||||
master.database = "burn-in"
|
||||
# master.query "select * from test1 into test2;"
|
||||
# master.query "select count(value) from test1 group by time(1m) into test2;"
|
||||
|
||||
influxdb = InfluxDB::Client.new "burn-in", username: "user", password: "pass"
|
||||
|
||||
Log.success "Connected to server #{influxdb.host}:#{influxdb.port}"
|
||||
|
||||
Log.log "Creating RandomGaussian(500, 25)"
|
||||
gaussian = RandomGaussian.new(500, 25)
|
||||
point_count = 0
|
||||
|
||||
while true
|
||||
Log.log "Generating 10,000 points.."
|
||||
points = []
|
||||
BATCH_SIZE.times do |n|
|
||||
points << {value: gaussian.rand.to_i.abs}
|
||||
end
|
||||
point_count += points.length
|
||||
|
||||
Log.info "Sending points to server.."
|
||||
begin
|
||||
st = Time.now
|
||||
foo = influxdb.write_point("test1", points)
|
||||
et = Time.now
|
||||
Log.log foo.inspect
|
||||
Log.log "#{et-st} seconds elapsed"
|
||||
Log.success "Write successful."
|
||||
rescue => e
|
||||
Log.failure "Write failed:"
|
||||
Log.log e
|
||||
end
|
||||
sleep 0.5
|
||||
|
||||
Log.info "Checking regular points"
|
||||
st = Time.now
|
||||
response = influxdb.query("select count(value) from test1;")
|
||||
et = Time.now
|
||||
|
||||
Log.log "#{et-st} seconds elapsed"
|
||||
|
||||
response_count = response["test1"].first["count"]
|
||||
if point_count == response_count
|
||||
Log.success "Point counts match: #{point_count} == #{response_count}"
|
||||
else
|
||||
Log.failure "Point counts don't match: #{point_count} != #{response_count}"
|
||||
end
|
||||
|
||||
# Log.info "Checking continuous query points for test2"
|
||||
# st = Time.now
|
||||
# response = influxdb.query("select count(value) from test2;")
|
||||
# et = Time.now
|
||||
|
||||
# Log.log "#{et-st} seconds elapsed"
|
||||
|
||||
# response_count = response["test2"].first["count"]
|
||||
# if point_count == response_count
|
||||
# Log.success "Point counts match: #{point_count} == #{response_count}"
|
||||
# else
|
||||
# Log.failure "Point counts don't match: #{point_count} != #{response_count}"
|
||||
# end
|
||||
end
|
||||
|
||||
|
||||
23
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/log.rb
generated
vendored
23
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/log.rb
generated
vendored
@@ -1,23 +0,0 @@
|
||||
module Log
|
||||
def self.info(msg)
|
||||
print Time.now.strftime("%r") + " | "
|
||||
puts msg.to_s.colorize(:yellow)
|
||||
end
|
||||
|
||||
def self.success(msg)
|
||||
print Time.now.strftime("%r") + " | "
|
||||
puts msg.to_s.colorize(:green)
|
||||
end
|
||||
|
||||
def self.failure(msg)
|
||||
print Time.now.strftime("%r") + " | "
|
||||
puts msg.to_s.colorize(:red)
|
||||
end
|
||||
|
||||
def self.log(msg)
|
||||
print Time.now.strftime("%r") + " | "
|
||||
puts msg.to_s
|
||||
end
|
||||
end
|
||||
|
||||
|
||||
31
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/random_gaussian.rb
generated
vendored
31
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/random_gaussian.rb
generated
vendored
@@ -1,31 +0,0 @@
|
||||
class RandomGaussian
|
||||
def initialize(mean, stddev, rand_helper = lambda { Kernel.rand })
|
||||
@rand_helper = rand_helper
|
||||
@mean = mean
|
||||
@stddev = stddev
|
||||
@valid = false
|
||||
@next = 0
|
||||
end
|
||||
|
||||
def rand
|
||||
if @valid then
|
||||
@valid = false
|
||||
return @next
|
||||
else
|
||||
@valid = true
|
||||
x, y = self.class.gaussian(@mean, @stddev, @rand_helper)
|
||||
@next = y
|
||||
return x
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
def self.gaussian(mean, stddev, rand)
|
||||
theta = 2 * Math::PI * rand.call
|
||||
rho = Math.sqrt(-2 * Math.log(1 - rand.call))
|
||||
scale = stddev * rho
|
||||
x = mean + scale * Math.cos(theta)
|
||||
y = mean + scale * Math.sin(theta)
|
||||
return x, y
|
||||
end
|
||||
end
|
||||
29
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/random_points.rb
generated
vendored
29
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/burn-in/random_points.rb
generated
vendored
@@ -1,29 +0,0 @@
|
||||
require "influxdb"
|
||||
|
||||
ONE_WEEK_IN_SECONDS = 7*24*60*60
|
||||
NUM_POINTS = 10_000
|
||||
BATCHES = 100
|
||||
|
||||
master = InfluxDB::Client.new
|
||||
master.delete_database("ctx") rescue nil
|
||||
master.create_database("ctx")
|
||||
|
||||
influxdb = InfluxDB::Client.new "ctx"
|
||||
influxdb.time_precision = "s"
|
||||
|
||||
names = ["foo", "bar", "baz", "quu", "qux"]
|
||||
|
||||
st = Time.now
|
||||
BATCHES.times do |m|
|
||||
points = []
|
||||
|
||||
puts "Writing #{NUM_POINTS} points, time ##{m}.."
|
||||
NUM_POINTS.times do |n|
|
||||
timestamp = Time.now.to_i - rand(ONE_WEEK_IN_SECONDS)
|
||||
points << {value: names.sample, time: timestamp}
|
||||
end
|
||||
|
||||
influxdb.write_point("ct1", points)
|
||||
end
|
||||
puts st
|
||||
puts Time.now
|
||||
246
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/config.sample.toml
generated
vendored
246
Godeps/_workspace/src/github.com/influxdb/influxdb/etc/config.sample.toml
generated
vendored
@@ -1,246 +0,0 @@
|
||||
### Welcome to the InfluxDB configuration file.
|
||||
|
||||
# Once every 24 hours InfluxDB will report anonymous data to m.influxdb.com
|
||||
# The data includes raft id (random 8 bytes), os, arch, version, and metadata.
|
||||
# We don't track ip addresses of servers reporting. This is only used
|
||||
# to track the number of instances running and the versions, which
|
||||
# is very helpful for us.
|
||||
# Change this option to true to disable reporting.
|
||||
reporting-disabled = false
|
||||
|
||||
###
|
||||
### [meta]
|
||||
###
|
||||
### Controls the parameters for the Raft consensus group that stores metadata
|
||||
### about the InfluxDB cluster.
|
||||
###
|
||||
|
||||
[meta]
|
||||
dir = "/var/opt/influxdb/meta"
|
||||
hostname = "localhost"
|
||||
bind-address = ":8088"
|
||||
retention-autocreate = true
|
||||
election-timeout = "1s"
|
||||
heartbeat-timeout = "1s"
|
||||
leader-lease-timeout = "500ms"
|
||||
commit-timeout = "50ms"
|
||||
|
||||
###
|
||||
### [data]
|
||||
###
|
||||
### Controls where the actual shard data for InfluxDB lives and how it is
|
||||
### flushed from the WAL. "dir" may need to be changed to a suitable place
|
||||
### for your system, but the WAL settings are an advanced configuration. The
|
||||
### defaults should work for most systems.
|
||||
###
|
||||
|
||||
[data]
|
||||
dir = "/var/opt/influxdb/data"
|
||||
|
||||
# The following WAL settings are for the b1 storage engine used in 0.9.2. They won't
|
||||
# apply to any new shards created after upgrading to a version > 0.9.3.
|
||||
max-wal-size = 104857600 # Maximum size the WAL can reach before a flush. Defaults to 100MB.
|
||||
wal-flush-interval = "10m" # Maximum time data can sit in WAL before a flush.
|
||||
wal-partition-flush-delay = "2s" # The delay time between each WAL partition being flushed.
|
||||
|
||||
# These are the WAL settings for the storage engine >= 0.9.3
|
||||
wal-dir = "/var/opt/influxdb/wal"
|
||||
wal-enable-logging = true
|
||||
|
||||
# When a series in the WAL in-memory cache reaches this size in bytes it is marked as ready to
|
||||
# flush to the index
|
||||
# wal-ready-series-size = 25600
|
||||
|
||||
# Flush and compact a partition once this ratio of series are over the ready size
|
||||
# wal-compaction-threshold = 0.6
|
||||
|
||||
# Force a flush and compaction if any series in a partition gets above this size in bytes
|
||||
# wal-max-series-size = 2097152
|
||||
|
||||
# Force a flush of all series and full compaction if there have been no writes in this
|
||||
# amount of time. This is useful for ensuring that shards that are cold for writes don't
|
||||
# keep a bunch of data cached in memory and in the WAL.
|
||||
# wal-flush-cold-interval = "10m"
|
||||
|
||||
# Force a partition to flush its largest series if it reaches this approximate size in
|
||||
# bytes. Remember there are 5 partitions so you'll need at least 5x this amount of memory.
|
||||
# The more memory you have, the bigger this can be.
|
||||
# wal-partition-size-threshold = 20971520
|
||||
|
||||
###
|
||||
### [cluster]
|
||||
###
|
||||
### Controls non-Raft cluster behavior, which generally includes how data is
|
||||
### shared across shards.
|
||||
###
|
||||
|
||||
[cluster]
|
||||
shard-writer-timeout = "5s" # The time within which a shard must respond to write.
|
||||
write-timeout = "5s" # The time within which a write operation must complete on the cluster.
|
||||
|
||||
###
|
||||
### [retention]
|
||||
###
|
||||
### Controls the enforcement of retention policies for evicting old data.
|
||||
###
|
||||
|
||||
[retention]
|
||||
enabled = true
|
||||
check-interval = "10m"
|
||||
|
||||
###
|
||||
### [admin]
|
||||
###
|
||||
### Controls the availability of the built-in, web-based admin interface. If HTTPS is
|
||||
### enabled for the admin interface, HTTPS must also be enabled on the [http] service.
|
||||
###
|
||||
|
||||
[admin]
|
||||
enabled = true
|
||||
bind-address = ":8083"
|
||||
https-enabled = false
|
||||
https-certificate = "/etc/ssl/influxdb.pem"
|
||||
|
||||
###
|
||||
### [http]
|
||||
###
|
||||
### Controls how the HTTP endpoints are configured. These are the primary
|
||||
### mechanism for getting data into and out of InfluxDB.
|
||||
###
|
||||
|
||||
[http]
|
||||
enabled = true
|
||||
bind-address = ":8086"
|
||||
auth-enabled = false
|
||||
log-enabled = true
|
||||
write-tracing = false
|
||||
pprof-enabled = false
|
||||
https-enabled = false
|
||||
https-certificate = "/etc/ssl/influxdb.pem"
|
||||
|
||||
###
|
||||
### [[graphite]]
|
||||
###
|
||||
### Controls one or many listeners for Graphite data.
|
||||
###
|
||||
|
||||
[[graphite]]
|
||||
enabled = false
|
||||
# bind-address = ":2003"
|
||||
# protocol = "tcp"
|
||||
# consistency-level = "one"
|
||||
# name-separator = "."
|
||||
|
||||
# These next lines control how batching works. You should have this enabled
|
||||
# otherwise you could get dropped metrics or poor performance. Batching
|
||||
# will buffer points in memory if you have many coming in.
|
||||
|
||||
# batch-size = 1000 # will flush if this many points get buffered
|
||||
# batch-timeout = "1s" # will flush at least this often even if we haven't hit buffer limit
|
||||
|
||||
## "name-schema" configures tag names for parsing the metric name from graphite protocol;
|
||||
## separated by `name-separator`.
|
||||
## The "measurement" tag is special and the corresponding field will become
|
||||
## the name of the metric.
|
||||
## e.g. "type.host.measurement.device" will parse "server.localhost.cpu.cpu0" as
|
||||
## {
|
||||
## measurement: "cpu",
|
||||
## tags: {
|
||||
## "type": "server",
|
||||
## "host": "localhost,
|
||||
## "device": "cpu0"
|
||||
## }
|
||||
## }
|
||||
# name-schema = "type.host.measurement.device"
|
||||
|
||||
## If set to true, when the input metric name has more fields than `name-schema` specified,
|
||||
## the extra fields will be ignored.
|
||||
## Otherwise an error will be logged and the metric rejected.
|
||||
# ignore-unnamed = true
|
||||
|
||||
###
|
||||
### [collectd]
|
||||
###
|
||||
### Controls the listener for collectd data.
|
||||
###
|
||||
|
||||
[collectd]
|
||||
enabled = false
|
||||
# bind-address = ""
|
||||
# database = ""
|
||||
# typesdb = ""
|
||||
|
||||
# These next lines control how batching works. You should have this enabled
|
||||
# otherwise you could get dropped metrics or poor performance. Batching
|
||||
# will buffer points in memory if you have many coming in.
|
||||
|
||||
# batch-size = 1000 # will flush if this many points get buffered
|
||||
# batch-timeout = "1s" # will flush at least this often even if we haven't hit buffer limit
|
||||
|
||||
###
|
||||
### [opentsdb]
|
||||
###
|
||||
### Controls the listener for OpenTSDB data.
|
||||
###
|
||||
|
||||
[opentsdb]
|
||||
enabled = false
|
||||
# bind-address = ""
|
||||
# database = ""
|
||||
# retention-policy = ""
|
||||
|
||||
###
|
||||
### [[udp]]
|
||||
###
|
||||
### Controls the listeners for InfluxDB line protocol data via UDP.
|
||||
###
|
||||
|
||||
[[udp]]
|
||||
enabled = false
|
||||
# bind-address = ""
|
||||
# database = ""
|
||||
|
||||
# These next lines control how batching works. You should have this enabled
|
||||
# otherwise you could get dropped metrics or poor performance. Batching
|
||||
# will buffer points in memory if you have many coming in.
|
||||
|
||||
# batch-size = 1000 # will flush if this many points get buffered
|
||||
# batch-timeout = "1s" # will flush at least this often even if we haven't hit buffer limit
|
||||
|
||||
###
|
||||
### [monitoring]
|
||||
###
|
||||
|
||||
[monitoring]
|
||||
enabled = true
|
||||
write-interval = "24h"
|
||||
|
||||
###
|
||||
### [continuous_queries]
|
||||
###
|
||||
### Controls how continuous queries are run within InfluxDB.
|
||||
###
|
||||
|
||||
[continuous_queries]
|
||||
log-enabled = true
|
||||
enabled = true
|
||||
recompute-previous-n = 2
|
||||
recompute-no-older-than = "10m"
|
||||
compute-runs-per-interval = 10
|
||||
compute-no-more-than = "2m"
|
||||
|
||||
###
|
||||
### [hinted-handoff]
|
||||
###
|
||||
### Controls the hinted handoff feature, which allows nodes to temporarily
|
||||
### store queued data when one node of a cluster is down for a short period
|
||||
### of time.
|
||||
###
|
||||
|
||||
[hinted-handoff]
|
||||
enabled = true
|
||||
dir = "/var/opt/influxdb/hh"
|
||||
max-size = 1073741824
|
||||
max-age = "168h"
|
||||
retry-rate-limit = 0
|
||||
retry-interval = "1s"
|
||||
186
Godeps/_workspace/src/github.com/influxdb/influxdb/importer/README.md
generated
vendored
186
Godeps/_workspace/src/github.com/influxdb/influxdb/importer/README.md
generated
vendored
@@ -1,186 +0,0 @@
|
||||
# Import/Export
|
||||
|
||||
## Exporting from 0.8.9
|
||||
|
||||
Version `0.8.9` of InfluxDB adds support to export your data to a format that can be imported into `0.9.3` and later.
|
||||
|
||||
### Design
|
||||
|
||||
`0.8.9` exports raw data to a flat file that includes two sections, `DDL` and `DML`. You can choose to export them independently (see below).
|
||||
|
||||
The `DDL` section contains the sql commands to create databases and retention policies. the `DML` section is [line protocol](https://github.com/influxdb/influxdb/blob/master/tsdb/README.md) and can be directly posted to the [http endpoint](https://influxdb.com/docs/v0.9/guides/writing_data.html) in `0.9`. Remember that batching is important and we don't recommend batch sizes over 5k.
|
||||
|
||||
You need to specify a database and shard group when you export.
|
||||
|
||||
To list out your shards, use the following http endpoint:
|
||||
|
||||
`/cluster/shard_spaces`
|
||||
|
||||
example:
|
||||
```sh
|
||||
http://username:password@localhost:8086/cluster/shard_spaces
|
||||
```
|
||||
|
||||
Then, to export a database with then name "metrics" and a shard space with the name "default", issue the following curl command:
|
||||
|
||||
```sh
|
||||
curl -o export http://username:password@http://localhost:8086/export/metrics/default
|
||||
```
|
||||
|
||||
Compression is supported, and will result in a significantly smaller file size.
|
||||
|
||||
Use the following command for compression:
|
||||
```sh
|
||||
curl -o export.gz --compressed http://username:password@http://localhost:8086/export/metrics/default
|
||||
```
|
||||
|
||||
You can also export just the `DDL` with this option:
|
||||
|
||||
```sh
|
||||
curl -o export.ddl http://username:password@http://localhost:8086/export/metrics/default?l=ddl
|
||||
```
|
||||
|
||||
Or just the `DML` with this option:
|
||||
|
||||
```sh
|
||||
curl -o export.dml.gz --compressed http://username:password@http://localhost:8086/export/metrics/default?l=dml
|
||||
```
|
||||
|
||||
### Assumptions
|
||||
|
||||
- Series name mapping follows these [guidelines](https://influxdb.com/docs/v0.8/advanced_topics/schema_design.html)
|
||||
- Database name will map directly from `0.8` to `0.9`
|
||||
- Shard Spaces map to Retention Policies
|
||||
- Shard Space Duration is ignored, as in `0.9` we determine shard size automatically
|
||||
- Regex is used to match the correct series names and only exports that data for the database
|
||||
- Duration becomes the new Retention Policy duration
|
||||
|
||||
- Users are not migrated due to inability to get passwords. Anyone using users will need to manually set these back up in `0.9`
|
||||
|
||||
### Upgrade Recommendations
|
||||
|
||||
It's recommended that you upgrade to `0.9.3` first and have all your writes going there. Then, on the `0.8.X` instances, upgrade to `0.8.9`.
|
||||
|
||||
It is important that when exporting you change your config to allow for the http endpoints not timing out. To do so, make this change in your config:
|
||||
|
||||
```toml
|
||||
# Configure the http api
|
||||
[api]
|
||||
read-timeout = "0s"
|
||||
```
|
||||
|
||||
### Exceptions
|
||||
|
||||
If a series can't be exported to tags based on the guidelines mentioned above,
|
||||
we will insert the entire series name as the measurement name. You can either
|
||||
allow that to import into the new InfluxDB instance, or you can do your own
|
||||
data massage on it prior to importing it.
|
||||
|
||||
For example, if you have the following series name:
|
||||
|
||||
```
|
||||
metric.disk.c.host.server01.single
|
||||
```
|
||||
|
||||
It will export as exactly thta as the measurement name and no tags:
|
||||
|
||||
```
|
||||
metric.disk.c.host.server01.single
|
||||
```
|
||||
|
||||
### Export Metrics
|
||||
|
||||
When you export, you will now get comments inline in the `DML`:
|
||||
|
||||
`# Found 999 Series for export`
|
||||
|
||||
As well as count totals for each series exported:
|
||||
|
||||
`# Series FOO - Points Exported: 999`
|
||||
|
||||
With a total at the bottom:
|
||||
|
||||
`# Points Exported: 999`
|
||||
|
||||
You can grep the file that was exported at the end to get all the export metrics:
|
||||
|
||||
`cat myexport | grep Exported`
|
||||
|
||||
## Importing
|
||||
|
||||
Version `0.9.3` of InfluxDB adds support to import your data from version `0.8.9`.
|
||||
|
||||
## Caveats
|
||||
|
||||
For the export/import to work, all requisites have to be met. For export, all series names in `0.8` should be in the following format:
|
||||
|
||||
```
|
||||
<tagName>.<tagValue>.<tagName>.<tagValue>.<measurement>
|
||||
```
|
||||
for example:
|
||||
```
|
||||
az.us-west-1.host.serverA.cpu
|
||||
```
|
||||
or any number of tags
|
||||
```
|
||||
building.2.temperature
|
||||
```
|
||||
|
||||
Additionally, the fields need to have a consistent type (all float64, int64, etc) for every write in `0.8`. Otherwise they have the potential to fail writes in the import.
|
||||
See below for more information.
|
||||
|
||||
## Running the import command
|
||||
|
||||
To import via the cli, you can specify the following command:
|
||||
|
||||
```sh
|
||||
influx -import -path=metrics-default.gz -compressed
|
||||
```
|
||||
|
||||
If the file is not compressed you can issue it without the `-compressed` flag:
|
||||
|
||||
```sh
|
||||
influx -import -path=metrics-default
|
||||
```
|
||||
|
||||
To redirect failed import lines to another file, run this command:
|
||||
|
||||
```sh
|
||||
influx -import -path=metrics-default.gz -compressed > failures
|
||||
```
|
||||
|
||||
The import will use the line protocol in batches of 5,000 lines per batch when sending data to the server.
|
||||
|
||||
### Throttiling the import
|
||||
|
||||
If you need to throttle the import so the database has time to ingest, you can use the `-pps` flag. This will limit the points per second that will be sent to the server.
|
||||
|
||||
```sh
|
||||
influx -import -path=metrics-default.gz -compressed -pps 50000 > failures
|
||||
```
|
||||
|
||||
Which is stating that you don't want MORE than 50,000 points per second to write to the database. Due to the processing that is taking place however, you will likely never get exactly 50,000 pps, more like 35,000 pps, etc.
|
||||
|
||||
## Understanding the results of the import
|
||||
|
||||
During the import, a status message will write out for every 100,000 points imported and report stats on the progress of the import:
|
||||
|
||||
```
|
||||
2015/08/21 14:48:01 Processed 3100000 lines. Time elapsed: 56.740578415s. Points per second (PPS): 54634
|
||||
```
|
||||
|
||||
The batch will give some basic stats when finished:
|
||||
|
||||
```sh
|
||||
2015/07/29 23:15:20 Processed 2 commands
|
||||
2015/07/29 23:15:20 Processed 70207923 inserts
|
||||
2015/07/29 23:15:20 Failed 29785000 inserts
|
||||
```
|
||||
|
||||
Most inserts fail due to the following types of error:
|
||||
|
||||
```sh
|
||||
2015/07/29 22:18:28 error writing batch: write failed: field type conflict: input field "value" on measurement "metric" is type float64, already exists as type integer
|
||||
```
|
||||
|
||||
This is due to the fact that in `0.8` a field could get created and saved as int or float types for independent writes. In `0.9` the field has to have a consistent type.
|
||||
236
Godeps/_workspace/src/github.com/influxdb/influxdb/importer/v8/importer.go
generated
vendored
236
Godeps/_workspace/src/github.com/influxdb/influxdb/importer/v8/importer.go
generated
vendored
@@ -1,236 +0,0 @@
|
||||
package v8
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"compress/gzip"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/influxdb/influxdb/client"
|
||||
)
|
||||
|
||||
const batchSize = 5000
|
||||
|
||||
// Config is the config used to initialize a Importer importer
|
||||
type Config struct {
|
||||
Username string
|
||||
Password string
|
||||
URL url.URL
|
||||
Precision string
|
||||
WriteConsistency string
|
||||
Path string
|
||||
Version string
|
||||
Compressed bool
|
||||
PPS int
|
||||
}
|
||||
|
||||
// NewConfig returns an initialized *Config
|
||||
func NewConfig() *Config {
|
||||
return &Config{}
|
||||
}
|
||||
|
||||
// Importer is the importer used for importing 0.8 data
|
||||
type Importer struct {
|
||||
client *client.Client
|
||||
database string
|
||||
retentionPolicy string
|
||||
config *Config
|
||||
batch []string
|
||||
totalInserts int
|
||||
failedInserts int
|
||||
totalCommands int
|
||||
throttlePointsWritten int
|
||||
lastWrite time.Time
|
||||
throttle *time.Ticker
|
||||
}
|
||||
|
||||
// NewImporter will return an intialized Importer struct
|
||||
func NewImporter(config *Config) *Importer {
|
||||
return &Importer{
|
||||
config: config,
|
||||
batch: make([]string, 0, batchSize),
|
||||
}
|
||||
}
|
||||
|
||||
// Import processes the specified file in the Config and writes the data to the databases in chunks specified by batchSize
|
||||
func (i *Importer) Import() error {
|
||||
// Create a client and try to connect
|
||||
config := client.NewConfig()
|
||||
config.URL = i.config.URL
|
||||
config.Username = i.config.Username
|
||||
config.Password = i.config.Password
|
||||
config.UserAgent = fmt.Sprintf("influxDB importer/%s", i.config.Version)
|
||||
cl, err := client.NewClient(config)
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not create client %s", err)
|
||||
}
|
||||
i.client = cl
|
||||
if _, _, e := i.client.Ping(); e != nil {
|
||||
return fmt.Errorf("failed to connect to %s\n", i.client.Addr())
|
||||
}
|
||||
|
||||
// Validate args
|
||||
if i.config.Path == "" {
|
||||
return fmt.Errorf("file argument required")
|
||||
}
|
||||
|
||||
defer func() {
|
||||
if i.totalInserts > 0 {
|
||||
log.Printf("Processed %d commands\n", i.totalCommands)
|
||||
log.Printf("Processed %d inserts\n", i.totalInserts)
|
||||
log.Printf("Failed %d inserts\n", i.failedInserts)
|
||||
}
|
||||
}()
|
||||
|
||||
// Open the file
|
||||
f, err := os.Open(i.config.Path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var r io.Reader
|
||||
|
||||
// If gzipped, wrap in a gzip reader
|
||||
if i.config.Compressed {
|
||||
gr, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer gr.Close()
|
||||
// Set the reader to the gzip reader
|
||||
r = gr
|
||||
} else {
|
||||
// Standard text file so our reader can just be the file
|
||||
r = f
|
||||
}
|
||||
|
||||
// Get our reader
|
||||
scanner := bufio.NewScanner(r)
|
||||
|
||||
// Process the DDL
|
||||
i.processDDL(scanner)
|
||||
|
||||
// Set up our throttle channel. Since there is effectively no other activity at this point
|
||||
// the smaller resolution gets us much closer to the requested PPS
|
||||
i.throttle = time.NewTicker(time.Microsecond)
|
||||
defer i.throttle.Stop()
|
||||
|
||||
// Prime the last write
|
||||
i.lastWrite = time.Now()
|
||||
|
||||
// Process the DML
|
||||
i.processDML(scanner)
|
||||
|
||||
// Check if we had any errors scanning the file
|
||||
if err := scanner.Err(); err != nil {
|
||||
return fmt.Errorf("reading standard input: %s", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (i *Importer) processDDL(scanner *bufio.Scanner) {
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
// If we find the DML token, we are done with DDL
|
||||
if strings.HasPrefix(line, "# DML") {
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
i.queryExecutor(line)
|
||||
}
|
||||
}
|
||||
|
||||
func (i *Importer) processDML(scanner *bufio.Scanner) {
|
||||
start := time.Now()
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if strings.HasPrefix(line, "# CONTEXT-DATABASE:") {
|
||||
i.database = strings.TrimSpace(strings.Split(line, ":")[1])
|
||||
}
|
||||
if strings.HasPrefix(line, "# CONTEXT-RETENTION-POLICY:") {
|
||||
i.retentionPolicy = strings.TrimSpace(strings.Split(line, ":")[1])
|
||||
}
|
||||
if strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
i.batchAccumulator(line, start)
|
||||
}
|
||||
}
|
||||
|
||||
func (i *Importer) execute(command string) {
|
||||
response, err := i.client.Query(client.Query{Command: command, Database: i.database})
|
||||
if err != nil {
|
||||
log.Printf("error: %s\n", err)
|
||||
return
|
||||
}
|
||||
if err := response.Error(); err != nil {
|
||||
log.Printf("error: %s\n", response.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func (i *Importer) queryExecutor(command string) {
|
||||
i.totalCommands++
|
||||
i.execute(command)
|
||||
}
|
||||
|
||||
func (i *Importer) batchAccumulator(line string, start time.Time) {
|
||||
i.batch = append(i.batch, line)
|
||||
if len(i.batch) == batchSize {
|
||||
if e := i.batchWrite(); e != nil {
|
||||
log.Println("error writing batch: ", e)
|
||||
// Output failed lines to STDOUT so users can capture lines that failed to import
|
||||
fmt.Println(strings.Join(i.batch, "\n"))
|
||||
i.failedInserts += len(i.batch)
|
||||
} else {
|
||||
i.totalInserts += len(i.batch)
|
||||
}
|
||||
i.batch = i.batch[:0]
|
||||
// Give some status feedback every 100000 lines processed
|
||||
processed := i.totalInserts + i.failedInserts
|
||||
if processed%100000 == 0 {
|
||||
since := time.Since(start)
|
||||
pps := float64(processed) / since.Seconds()
|
||||
log.Printf("Processed %d lines. Time elapsed: %s. Points per second (PPS): %d", processed, since.String(), int64(pps))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (i *Importer) batchWrite() error {
|
||||
// Accumulate the batch size to see how many points we have written this second
|
||||
i.throttlePointsWritten += len(i.batch)
|
||||
|
||||
// Find out when we last wrote data
|
||||
since := time.Since(i.lastWrite)
|
||||
|
||||
// Check to see if we've exceeded our points per second for the current timeframe
|
||||
var currentPPS int
|
||||
if since.Seconds() > 0 {
|
||||
currentPPS = int(float64(i.throttlePointsWritten) / since.Seconds())
|
||||
} else {
|
||||
currentPPS = i.throttlePointsWritten
|
||||
}
|
||||
|
||||
// If our currentPPS is greater than the PPS specified, then we wait and retry
|
||||
if int(currentPPS) > i.config.PPS && i.config.PPS != 0 {
|
||||
// Wait for the next tick
|
||||
<-i.throttle.C
|
||||
|
||||
// Decrement the batch size back out as it is going to get called again
|
||||
i.throttlePointsWritten -= len(i.batch)
|
||||
return i.batchWrite()
|
||||
}
|
||||
|
||||
_, e := i.client.WriteLineProtocol(strings.Join(i.batch, "\n"), i.database, i.retentionPolicy, i.config.Precision, i.config.WriteConsistency)
|
||||
i.throttlePointsWritten = 0
|
||||
i.lastWrite = time.Now()
|
||||
return e
|
||||
}
|
||||
12
Godeps/_workspace/src/github.com/influxdb/influxdb/influxql/ast.go
generated
vendored
12
Godeps/_workspace/src/github.com/influxdb/influxdb/influxql/ast.go
generated
vendored
@@ -985,10 +985,6 @@ func (s *SelectStatement) hasTimeDimensions(node Node) bool {
|
||||
}
|
||||
|
||||
func (s *SelectStatement) validate(tr targetRequirement) error {
|
||||
if err := s.validateFields(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := s.validateDistinct(); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -1012,14 +1008,6 @@ func (s *SelectStatement) validate(tr targetRequirement) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *SelectStatement) validateFields() error {
|
||||
ns := s.NamesInSelect()
|
||||
if len(ns) == 1 && ns[0] == "time" {
|
||||
return fmt.Errorf("at least 1 non-time field must be queried")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *SelectStatement) validateAggregates(tr targetRequirement) error {
|
||||
// First, if 1 field is an aggregate, then all fields must be an aggregate. This is
|
||||
// a explicit limitation of the current system.
|
||||
|
||||
1
Godeps/_workspace/src/github.com/influxdb/influxdb/influxql/parser_test.go
generated
vendored
1
Godeps/_workspace/src/github.com/influxdb/influxdb/influxql/parser_test.go
generated
vendored
@@ -1206,7 +1206,6 @@ func TestParser_ParseStatement(t *testing.T) {
|
||||
// Errors
|
||||
{s: ``, err: `found EOF, expected SELECT, DELETE, SHOW, CREATE, DROP, GRANT, REVOKE, ALTER, SET at line 1, char 1`},
|
||||
{s: `SELECT`, err: `found EOF, expected identifier, string, number, bool at line 1, char 8`},
|
||||
{s: `SELECT time FROM myseries`, err: `at least 1 non-time field must be queried`},
|
||||
{s: `blah blah`, err: `found blah, expected SELECT, DELETE, SHOW, CREATE, DROP, GRANT, REVOKE, ALTER, SET at line 1, char 1`},
|
||||
{s: `SELECT field1 X`, err: `found X, expected FROM at line 1, char 15`},
|
||||
{s: `SELECT field1 FROM "series" WHERE X +;`, err: `found ;, expected identifier, string, number, bool at line 1, char 38`},
|
||||
|
||||
1
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/data.go
generated
vendored
1
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/data.go
generated
vendored
@@ -212,7 +212,6 @@ func (data *Data) UpdateRetentionPolicy(database, name string, rpu *RetentionPol
|
||||
}
|
||||
if rpu.Duration != nil {
|
||||
rpi.Duration = *rpu.Duration
|
||||
rpi.ShardGroupDuration = shardGroupDuration(rpi.Duration)
|
||||
}
|
||||
if rpu.ReplicaN != nil {
|
||||
rpi.ReplicaN = *rpu.ReplicaN
|
||||
|
||||
2
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/data_test.go
generated
vendored
2
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/data_test.go
generated
vendored
@@ -178,7 +178,7 @@ func TestData_UpdateRetentionPolicy(t *testing.T) {
|
||||
if rpi, _ := data.RetentionPolicy("db0", "rp1"); !reflect.DeepEqual(rpi, &meta.RetentionPolicyInfo{
|
||||
Name: "rp1",
|
||||
Duration: 10 * time.Hour,
|
||||
ShardGroupDuration: 3600000000000,
|
||||
ShardGroupDuration: 604800000000000,
|
||||
ReplicaN: 3,
|
||||
}) {
|
||||
t.Fatalf("unexpected policy: %#v", rpi)
|
||||
|
||||
2
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/state.go
generated
vendored
2
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/state.go
generated
vendored
@@ -334,6 +334,8 @@ func (r *localRaft) leader() string {
|
||||
}
|
||||
|
||||
func (r *localRaft) isLeader() bool {
|
||||
r.store.mu.RLock()
|
||||
defer r.store.mu.RUnlock()
|
||||
if r.raft == nil {
|
||||
return false
|
||||
}
|
||||
|
||||
2
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/store_test.go
generated
vendored
2
Godeps/_workspace/src/github.com/influxdb/influxdb/meta/store_test.go
generated
vendored
@@ -361,7 +361,7 @@ func TestStore_UpdateRetentionPolicy(t *testing.T) {
|
||||
} else if !reflect.DeepEqual(rpi, &meta.RetentionPolicyInfo{
|
||||
Name: "rp1",
|
||||
Duration: 10 * time.Hour,
|
||||
ShardGroupDuration: 1 * time.Hour,
|
||||
ShardGroupDuration: 7 * 24 * time.Hour,
|
||||
ReplicaN: 1,
|
||||
}) {
|
||||
t.Fatalf("unexpected policy: %#v", rpi)
|
||||
|
||||
14
Godeps/_workspace/src/github.com/influxdb/influxdb/nightly.sh
generated
vendored
14
Godeps/_workspace/src/github.com/influxdb/influxdb/nightly.sh
generated
vendored
@@ -1,14 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
REPO_DIR=`mktemp -d`
|
||||
echo "Using $REPO_DIR for all work..."
|
||||
|
||||
cd $REPO_DIR
|
||||
export GOPATH=`pwd`
|
||||
mkdir -p $GOPATH/src/github.com/influxdb
|
||||
cd $GOPATH/src/github.com/influxdb
|
||||
git clone https://github.com/influxdb/influxdb.git
|
||||
|
||||
cd $GOPATH/src/github.com/influxdb/influxdb
|
||||
NIGHTLY_BUILD=true ./package.sh 0.9.3-nightly-`git log --pretty=format:'%h' -n 1`
|
||||
rm -rf $REPO_DIR
|
||||
409
Godeps/_workspace/src/github.com/influxdb/influxdb/package.sh
generated
vendored
409
Godeps/_workspace/src/github.com/influxdb/influxdb/package.sh
generated
vendored
@@ -1,409 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
###########################################################################
|
||||
# Packaging script which creates debian and RPM packages. It optionally
|
||||
# tags the repo with the given version.
|
||||
#
|
||||
# Requirements: GOPATH must be set. 'fpm' must be on the path, and the AWS
|
||||
# CLI tools must also be installed.
|
||||
#
|
||||
# https://github.com/jordansissel/fpm
|
||||
# http://aws.amazon.com/cli/
|
||||
#
|
||||
# Packaging process: to package a build, simple execute:
|
||||
#
|
||||
# package.sh <version>
|
||||
#
|
||||
# where <version> is the desired version. If generation of a debian and RPM
|
||||
# package is successful, the script will offer to tag the repo using the
|
||||
# supplied version string.
|
||||
#
|
||||
# AWS upload: the script will also offer to upload the packages to S3. If
|
||||
# this option is selected, the credentials should be present in the file
|
||||
# ~/aws.conf. The contents should be of the form:
|
||||
#
|
||||
# [default]
|
||||
# aws_access_key_id=<access ID>
|
||||
# aws_secret_access_key=<secret key>
|
||||
# region = us-east-1
|
||||
#
|
||||
# Trim the leading spaces when creating the file. The script will exit if
|
||||
# S3 upload is requested, but this file does not exist.
|
||||
|
||||
[ -z $DEBUG ] || set -x
|
||||
|
||||
AWS_FILE=~/aws.conf
|
||||
|
||||
INSTALL_ROOT_DIR=/opt/influxdb
|
||||
INFLUXDB_LOG_DIR=/var/log/influxdb
|
||||
INFLUXDB_DATA_DIR=/var/opt/influxdb
|
||||
CONFIG_ROOT_DIR=/etc/opt/influxdb
|
||||
|
||||
SAMPLE_CONFIGURATION=etc/config.sample.toml
|
||||
INITD_SCRIPT=scripts/init.sh
|
||||
|
||||
TMP_WORK_DIR=`mktemp -d`
|
||||
POST_INSTALL_PATH=`mktemp`
|
||||
ARCH=`uname -i`
|
||||
LICENSE=MIT
|
||||
URL=influxdb.com
|
||||
MAINTAINER=support@influxdb.com
|
||||
VENDOR=Influxdb
|
||||
DESCRIPTION="Distributed time-series database"
|
||||
|
||||
# Allow path to FPM to be set by environment variables. Some execution contexts
|
||||
# like cron don't have PATH set correctly to pick it up.
|
||||
if [ -z "$FPM" ]; then
|
||||
FPM=`which fpm`
|
||||
fi
|
||||
|
||||
GO_VERSION="go1.4.2"
|
||||
GOPATH_INSTALL=
|
||||
BINS=(
|
||||
influxd
|
||||
influx
|
||||
)
|
||||
|
||||
###########################################################################
|
||||
# Helper functions.
|
||||
|
||||
# usage prints simple usage information.
|
||||
usage() {
|
||||
echo -e "$0 [<version>] [-h]\n"
|
||||
cleanup_exit $1
|
||||
}
|
||||
|
||||
# cleanup_exit removes all resources created during the process and exits with
|
||||
# the supplied returned code.
|
||||
cleanup_exit() {
|
||||
rm -r $TMP_WORK_DIR
|
||||
rm $POST_INSTALL_PATH
|
||||
exit $1
|
||||
}
|
||||
|
||||
# current_branch echos the current git branch.
|
||||
current_branch() {
|
||||
echo `git rev-parse --abbrev-ref HEAD`
|
||||
}
|
||||
|
||||
# check_gopath sanity checks the value of the GOPATH env variable, and determines
|
||||
# the path where build artifacts are installed. GOPATH may be a colon-delimited
|
||||
# list of directories.
|
||||
check_gopath() {
|
||||
[ -z "$GOPATH" ] && echo "GOPATH is not set." && cleanup_exit 1
|
||||
GOPATH_INSTALL=`echo $GOPATH | cut -d ':' -f 1`
|
||||
[ ! -d "$GOPATH_INSTALL" ] && echo "GOPATH_INSTALL is not a directory." && cleanup_exit 1
|
||||
echo "GOPATH ($GOPATH) looks sane, using $GOPATH_INSTALL for installation."
|
||||
}
|
||||
|
||||
check_gvm() {
|
||||
if [ -n "$GOPATH" ]; then
|
||||
existing_gopath=$GOPATH
|
||||
fi
|
||||
|
||||
source $HOME/.gvm/scripts/gvm
|
||||
which gvm
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "gvm not found -- aborting."
|
||||
cleanup_exit $1
|
||||
fi
|
||||
gvm use $GO_VERSION
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "gvm cannot find Go version $GO_VERSION -- aborting."
|
||||
cleanup_exit $1
|
||||
fi
|
||||
|
||||
# Keep any existing GOPATH set.
|
||||
if [ -n "$existing_gopath" ]; then
|
||||
GOPATH=$existing_gopath
|
||||
fi
|
||||
}
|
||||
|
||||
# check_clean_tree ensures that no source file is locally modified.
|
||||
check_clean_tree() {
|
||||
modified=$(git ls-files --modified | wc -l)
|
||||
if [ $modified -ne 0 ]; then
|
||||
echo "The source tree is not clean -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "Git tree is clean."
|
||||
}
|
||||
|
||||
# update_tree ensures the tree is in-sync with the repo.
|
||||
update_tree() {
|
||||
git pull origin $TARGET_BRANCH
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to pull latest code -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
git fetch --tags
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to fetch tags -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "Git tree updated successfully."
|
||||
}
|
||||
|
||||
# check_tag_exists checks if the existing release already exists in the tags.
|
||||
check_tag_exists () {
|
||||
version=$1
|
||||
git tag | grep -q "^v$version$"
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "Proposed version $version already exists as a tag -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# make_dir_tree creates the directory structure within the packages.
|
||||
make_dir_tree() {
|
||||
work_dir=$1
|
||||
version=$2
|
||||
mkdir -p $work_dir/$INSTALL_ROOT_DIR/versions/$version/scripts
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to create installation directory -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
mkdir -p $work_dir/$CONFIG_ROOT_DIR
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to create configuration directory -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
# do_build builds the code. The version and commit must be passed in.
|
||||
do_build() {
|
||||
for b in ${BINS[*]}; do
|
||||
rm -f $GOPATH_INSTALL/bin/$b
|
||||
done
|
||||
go get -u -f -d ./...
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "WARNING: failed to 'go get' packages."
|
||||
fi
|
||||
|
||||
git checkout $TARGET_BRANCH # go get switches to master, so ensure we're back.
|
||||
version=$1
|
||||
commit=`git rev-parse HEAD`
|
||||
branch=`current_branch`
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Unable to retrieve current commit -- aborting"
|
||||
cleanup_exit 1
|
||||
fi
|
||||
|
||||
go install -a -ldflags="-X main.version $version -X main.branch $branch -X main.commit $commit" ./...
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Build failed, unable to create package -- aborting"
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "Build completed successfully."
|
||||
}
|
||||
|
||||
# generate_postinstall_script creates the post-install script for the
|
||||
# package. It must be passed the version.
|
||||
generate_postinstall_script() {
|
||||
version=$1
|
||||
cat <<EOF >$POST_INSTALL_PATH
|
||||
rm -f $INSTALL_ROOT_DIR/influxd
|
||||
rm -f $INSTALL_ROOT_DIR/influx
|
||||
rm -f $INSTALL_ROOT_DIR/init.sh
|
||||
ln -s $INSTALL_ROOT_DIR/versions/$version/influxd $INSTALL_ROOT_DIR/influxd
|
||||
ln -s $INSTALL_ROOT_DIR/versions/$version/influx $INSTALL_ROOT_DIR/influx
|
||||
ln -s $INSTALL_ROOT_DIR/versions/$version/scripts/init.sh $INSTALL_ROOT_DIR/init.sh
|
||||
|
||||
rm -f /etc/init.d/influxdb
|
||||
ln -sfn $INSTALL_ROOT_DIR/init.sh /etc/init.d/influxdb
|
||||
chmod +x /etc/init.d/influxdb
|
||||
if which update-rc.d > /dev/null 2>&1 ; then
|
||||
update-rc.d -f influxdb remove
|
||||
update-rc.d influxdb defaults
|
||||
else
|
||||
chkconfig --add influxdb
|
||||
fi
|
||||
|
||||
if ! id influxdb >/dev/null 2>&1; then
|
||||
useradd --system -U -M influxdb
|
||||
fi
|
||||
chown -R -L influxdb:influxdb $INSTALL_ROOT_DIR
|
||||
chmod -R a+rX $INSTALL_ROOT_DIR
|
||||
|
||||
mkdir -p $INFLUXDB_LOG_DIR
|
||||
chown -R -L influxdb:influxdb $INFLUXDB_LOG_DIR
|
||||
mkdir -p $INFLUXDB_DATA_DIR
|
||||
chown -R -L influxdb:influxdb $INFLUXDB_DATA_DIR
|
||||
EOF
|
||||
echo "Post-install script created successfully at $POST_INSTALL_PATH"
|
||||
}
|
||||
|
||||
###########################################################################
|
||||
# Start the packaging process.
|
||||
|
||||
if [ $# -ne 1 ]; then
|
||||
usage 1
|
||||
elif [ $1 == "-h" ]; then
|
||||
usage 0
|
||||
else
|
||||
VERSION=$1
|
||||
VERSION_UNDERSCORED=`echo "$VERSION" | tr - _`
|
||||
fi
|
||||
|
||||
echo -e "\nStarting package process...\n"
|
||||
|
||||
# Ensure the current is correct.
|
||||
TARGET_BRANCH=`current_branch`
|
||||
if [ -z "$NIGHTLY_BUILD" ]; then
|
||||
echo -n "Current branch is $TARGET_BRANCH. Start packaging this branch? [Y/n] "
|
||||
read response
|
||||
response=`echo $response | tr 'A-Z' 'a-z'`
|
||||
if [ "x$response" == "xn" ]; then
|
||||
echo "Packaging aborted."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
check_gvm
|
||||
check_gopath
|
||||
if [ -z "$NIGHTLY_BUILD" ]; then
|
||||
check_clean_tree
|
||||
update_tree
|
||||
check_tag_exists $VERSION
|
||||
fi
|
||||
|
||||
do_build $VERSION
|
||||
make_dir_tree $TMP_WORK_DIR $VERSION
|
||||
|
||||
###########################################################################
|
||||
# Copy the assets to the installation directories.
|
||||
|
||||
for b in ${BINS[*]}; do
|
||||
cp $GOPATH_INSTALL/bin/$b $TMP_WORK_DIR/$INSTALL_ROOT_DIR/versions/$VERSION
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to copy binaries to packaging directory -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
done
|
||||
echo "${BINS[*]} copied to $TMP_WORK_DIR/$INSTALL_ROOT_DIR/versions/$VERSION"
|
||||
|
||||
cp $INITD_SCRIPT $TMP_WORK_DIR/$INSTALL_ROOT_DIR/versions/$VERSION/scripts
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to copy init.d script to packaging directory -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "$INITD_SCRIPT copied to $TMP_WORK_DIR/$INSTALL_ROOT_DIR/versions/$VERSION/scripts"
|
||||
|
||||
cp $SAMPLE_CONFIGURATION $TMP_WORK_DIR/$CONFIG_ROOT_DIR/influxdb.conf
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to copy $SAMPLE_CONFIGURATION to packaging directory -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
|
||||
generate_postinstall_script $VERSION
|
||||
|
||||
###########################################################################
|
||||
# Create the actual packages.
|
||||
|
||||
if [ -z "$NIGHTLY_BUILD" ]; then
|
||||
echo -n "Commence creation of $ARCH packages, version $VERSION? [Y/n] "
|
||||
read response
|
||||
response=`echo $response | tr 'A-Z' 'a-z'`
|
||||
if [ "x$response" == "xn" ]; then
|
||||
echo "Packaging aborted."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ $ARCH == "i386" ]; then
|
||||
rpm_package=influxdb-${VERSION}-1.i686.rpm # RPM packages use 1 for default package release.
|
||||
debian_package=influxdb_${VERSION}_i686.deb
|
||||
deb_args="-a i686"
|
||||
rpm_args="setarch i686"
|
||||
elif [ $ARCH == "arm" ]; then
|
||||
rpm_package=influxdb-${VERSION}-1.armel.rpm
|
||||
debian_package=influxdb_${VERSION}_armel.deb
|
||||
else
|
||||
rpm_package=influxdb-${VERSION}-1.x86_64.rpm
|
||||
debian_package=influxdb_${VERSION}_amd64.deb
|
||||
fi
|
||||
|
||||
COMMON_FPM_ARGS="-C $TMP_WORK_DIR --vendor $VENDOR --url $URL --license $LICENSE --maintainer $MAINTAINER --after-install $POST_INSTALL_PATH --name influxdb --version $VERSION --config-files $CONFIG_ROOT_DIR ."
|
||||
$rpm_args $FPM -s dir -t rpm --description "$DESCRIPTION" $COMMON_FPM_ARGS
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to create RPM package -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "RPM package created successfully."
|
||||
|
||||
$FPM -s dir -t deb $deb_args --description "$DESCRIPTION" $COMMON_FPM_ARGS
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to create Debian package -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "Debian package created successfully."
|
||||
|
||||
$FPM -s dir -t tar --prefix influxdb_${VERSION}_${ARCH} -p influxdb_${VERSION}_${ARCH}.tar.gz --description "$DESCRIPTION" $COMMON_FPM_ARGS
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to create Tar package -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
echo "Tar package created successfully."
|
||||
|
||||
###########################################################################
|
||||
# Offer to tag the repo.
|
||||
|
||||
if [ -z "$NIGHTLY_BUILD" ]; then
|
||||
echo -n "Tag source tree with v$VERSION and push to repo? [y/N] "
|
||||
read response
|
||||
response=`echo $response | tr 'A-Z' 'a-z'`
|
||||
if [ "x$response" == "xy" ]; then
|
||||
echo "Creating tag v$VERSION and pushing to repo"
|
||||
git tag v$VERSION
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to create tag v$VERSION -- aborting"
|
||||
cleanup_exit 1
|
||||
fi
|
||||
git push origin v$VERSION
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to push tag v$VERSION to repo -- aborting"
|
||||
cleanup_exit 1
|
||||
fi
|
||||
else
|
||||
echo "Not creating tag v$VERSION."
|
||||
fi
|
||||
fi
|
||||
|
||||
###########################################################################
|
||||
# Offer to publish the packages.
|
||||
|
||||
if [ -z "$NIGHTLY_BUILD" ]; then
|
||||
echo -n "Publish packages to S3? [y/N] "
|
||||
read response
|
||||
response=`echo $response | tr 'A-Z' 'a-z'`
|
||||
fi
|
||||
|
||||
if [ "x$response" == "xy" -o -n "$NIGHTLY_BUILD" ]; then
|
||||
echo "Publishing packages to S3."
|
||||
if [ ! -e "$AWS_FILE" ]; then
|
||||
echo "$AWS_FILE does not exist -- aborting."
|
||||
cleanup_exit 1
|
||||
fi
|
||||
|
||||
for filepath in `ls *.{deb,rpm,gz}`; do
|
||||
filename=`basename $filepath`
|
||||
if [ -n "$NIGHTLY_BUILD" ]; then
|
||||
filename=`echo $filename | sed s/$VERSION/nightly/`
|
||||
filename=`echo $filename | sed s/$VERSION_UNDERSCORED/nightly/`
|
||||
fi
|
||||
AWS_CONFIG_FILE=$AWS_FILE aws s3 cp $filepath s3://influxdb/$filename --acl public-read --region us-east-1
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Upload failed -- aborting".
|
||||
cleanup_exit 1
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo "Not publishing packages to S3."
|
||||
fi
|
||||
|
||||
###########################################################################
|
||||
# All done.
|
||||
|
||||
echo -e "\nPackaging process complete."
|
||||
cleanup_exit 0
|
||||
16
Godeps/_workspace/src/github.com/influxdb/influxdb/scripts/influxdb.service
generated
vendored
16
Godeps/_workspace/src/github.com/influxdb/influxdb/scripts/influxdb.service
generated
vendored
@@ -1,16 +0,0 @@
|
||||
# If you modify this, please also make sure to edit init.sh
|
||||
|
||||
[Unit]
|
||||
Description=InfluxDB is an open-source, distributed, time series database
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=influxdb
|
||||
Group=influxdb
|
||||
LimitNOFILE=65536
|
||||
EnvironmentFile=-/etc/default/influxdb
|
||||
ExecStart=/opt/influxdb/influxd -config /etc/opt/influxdb/influxdb.conf $INFLUXD_OPTS
|
||||
Restart=on-failure
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
205
Godeps/_workspace/src/github.com/influxdb/influxdb/scripts/init.sh
generated
vendored
205
Godeps/_workspace/src/github.com/influxdb/influxdb/scripts/init.sh
generated
vendored
@@ -1,205 +0,0 @@
|
||||
#! /usr/bin/env bash
|
||||
|
||||
### BEGIN INIT INFO
|
||||
# Provides: influxd
|
||||
# Required-Start: $all
|
||||
# Required-Stop: $remote_fs $syslog
|
||||
# Default-Start: 2 3 4 5
|
||||
# Default-Stop: 0 1 6
|
||||
# Short-Description: Start influxd at boot time
|
||||
### END INIT INFO
|
||||
|
||||
# If you modify this, please make sure to also edit influxdb.service
|
||||
# this init script supports three different variations:
|
||||
# 1. New lsb that define start-stop-daemon
|
||||
# 2. Old lsb that don't have start-stop-daemon but define, log, pidofproc and killproc
|
||||
# 3. Centos installations without lsb-core installed
|
||||
#
|
||||
# In the third case we have to define our own functions which are very dumb
|
||||
# and expect the args to be positioned correctly.
|
||||
|
||||
# Command-line options that can be set in /etc/default/influxdb. These will override
|
||||
# any config file values. Example: "-join http://1.2.3.4:8086"
|
||||
DEFAULT=/etc/default/influxdb
|
||||
|
||||
# Daemon options
|
||||
INFLUXD_OPTS=
|
||||
|
||||
# Process name ( For display )
|
||||
NAME=influxdb
|
||||
|
||||
# User and group
|
||||
USER=influxdb
|
||||
GROUP=influxdb
|
||||
|
||||
# Daemon name, where is the actual executable
|
||||
# If the daemon is not there, then exit.
|
||||
DAEMON=/opt/influxdb/influxd
|
||||
[ -x $DAEMON ] || exit 5
|
||||
|
||||
# Configuration file
|
||||
CONFIG=/etc/opt/influxdb/influxdb.conf
|
||||
|
||||
# PID file for the daemon
|
||||
PIDFILE=/var/run/influxdb/influxd.pid
|
||||
PIDDIR=`dirname $PIDFILE`
|
||||
if [ ! -d "$PIDDIR" ]; then
|
||||
mkdir -p $PIDDIR
|
||||
chown $GROUP:$USER $PIDDIR
|
||||
fi
|
||||
|
||||
# Max open files
|
||||
OPEN_FILE_LIMIT=65536
|
||||
|
||||
if [ -r /lib/lsb/init-functions ]; then
|
||||
source /lib/lsb/init-functions
|
||||
fi
|
||||
|
||||
# Logging
|
||||
if [ -z "$STDOUT" ]; then
|
||||
STDOUT=/dev/null
|
||||
fi
|
||||
|
||||
if [ ! -f "$STDOUT" ]; then
|
||||
mkdir -p $(dirname $STDOUT)
|
||||
fi
|
||||
|
||||
if [ -z "$STDERR" ]; then
|
||||
STDERR=/var/log/influxdb/influxd.log
|
||||
fi
|
||||
|
||||
if [ ! -f "$STDERR" ]; then
|
||||
mkdir -p $(dirname $STDERR)
|
||||
fi
|
||||
|
||||
# Overwrite init script variables with /etc/default/influxdb values
|
||||
if [ -r $DEFAULT ]; then
|
||||
source $DEFAULT
|
||||
fi
|
||||
|
||||
function pidofproc() {
|
||||
if [ $# -ne 3 ]; then
|
||||
echo "Expected three arguments, e.g. $0 -p pidfile daemon-name"
|
||||
fi
|
||||
|
||||
PID=`pgrep -f $3`
|
||||
local PIDFILE=`cat $2`
|
||||
|
||||
if [ "x$PIDFILE" == "x" ]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ "x$PID" != "x" -a "$PIDFILE" == "$PID" ]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
function killproc() {
|
||||
if [ $# -ne 3 ]; then
|
||||
echo "Expected three arguments, e.g. $0 -p pidfile signal"
|
||||
fi
|
||||
|
||||
PID=`cat $2`
|
||||
|
||||
kill -s $3 $PID
|
||||
}
|
||||
|
||||
function log_failure_msg() {
|
||||
echo "$@" "[ FAILED ]"
|
||||
}
|
||||
|
||||
function log_success_msg() {
|
||||
echo "$@" "[ OK ]"
|
||||
}
|
||||
|
||||
case $1 in
|
||||
start)
|
||||
# Check if config file exist
|
||||
if [ ! -r $CONFIG ]; then
|
||||
log_failure_msg "config file doesn't exists"
|
||||
exit 4
|
||||
fi
|
||||
|
||||
# Checked the PID file exists and check the actual status of process
|
||||
if [ -e $PIDFILE ]; then
|
||||
pidofproc -p $PIDFILE $DAEMON > /dev/null 2>&1 && STATUS="0" || STATUS="$?"
|
||||
# If the status is SUCCESS then don't need to start again.
|
||||
if [ "x$STATUS" = "x0" ]; then
|
||||
log_failure_msg "$NAME process is running"
|
||||
exit 0 # Exit
|
||||
fi
|
||||
# if PID file does not exist, check if writable
|
||||
else
|
||||
su -c "touch $PIDFILE" $USER > /dev/null 2>&1
|
||||
if [ $? -ne 0 ]; then
|
||||
log_failure_msg "$PIDFILE not writable, check permissions"
|
||||
exit 5
|
||||
fi
|
||||
fi
|
||||
|
||||
# Bump the file limits, before launching the daemon. These will carry over to
|
||||
# launched processes.
|
||||
ulimit -n $OPEN_FILE_LIMIT
|
||||
if [ $? -ne 0 ]; then
|
||||
log_failure_msg "set open file limit to $OPEN_FILE_LIMIT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_success_msg "Starting the process" "$NAME"
|
||||
if which start-stop-daemon > /dev/null 2>&1; then
|
||||
start-stop-daemon --chuid $GROUP:$USER --start --quiet --pidfile $PIDFILE --exec $DAEMON -- -pidfile $PIDFILE -config $CONFIG $INFLUXD_OPTS >>$STDOUT 2>>$STDERR &
|
||||
else
|
||||
nohup $DAEMON -pidfile $PIDFILE -config $CONFIG $INFLUXD_OPTS >>$STDOUT 2>>$STDERR &
|
||||
fi
|
||||
log_success_msg "$NAME process was started"
|
||||
;;
|
||||
|
||||
stop)
|
||||
# Stop the daemon.
|
||||
if [ -e $PIDFILE ]; then
|
||||
pidofproc -p $PIDFILE $DAEMON > /dev/null 2>&1 && STATUS="0" || STATUS="$?"
|
||||
if [ "$STATUS" = 0 ]; then
|
||||
if killproc -p $PIDFILE SIGTERM && /bin/rm -rf $PIDFILE; then
|
||||
log_success_msg "$NAME process was stopped"
|
||||
else
|
||||
log_failure_msg "$NAME failed to stop service"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
log_failure_msg "$NAME process is not running"
|
||||
fi
|
||||
;;
|
||||
|
||||
restart)
|
||||
# Restart the daemon.
|
||||
$0 stop && sleep 2 && $0 start
|
||||
;;
|
||||
|
||||
status)
|
||||
# Check the status of the process.
|
||||
if [ -e $PIDFILE ]; then
|
||||
if pidofproc -p $PIDFILE $DAEMON > /dev/null; then
|
||||
log_success_msg "$NAME Process is running"
|
||||
exit 0
|
||||
else
|
||||
log_failure_msg "$NAME Process is not running"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
log_failure_msg "$NAME Process is not running"
|
||||
exit 3
|
||||
fi
|
||||
;;
|
||||
|
||||
version)
|
||||
$DAEMON version
|
||||
;;
|
||||
|
||||
*)
|
||||
# For invalid arguments, print the usage message.
|
||||
echo "Usage: $0 {start|stop|restart|status|version}"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user