dom
/
telegraf
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' 

Conflicts:
	Godeps/_workspace/src/github.com/shirou/gopsutil/mem/mem_darwin.go
	Godeps/_workspace/src/github.com/shirou/gopsutil/mem/mem_linux.go
This commit is contained in:
subhachandrachandra 2015-09-25 15:19:11 -07:00
parent 13ee9ff37b 0ffaafd788
commit cd93b9ae0b
433 changed files with 71101 additions and 7392 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

92
CHANGELOG.md
View File

@ -1,9 +1,97 @@
## v0.1.7 [unreleased]
## v0.1.10 [unreleased]
### Release Notes
- The -test flag will now only output 2 collections for plugins that need it
### Features
- [#205](https://github.com/influxdb/telegraf/issues/205): Include per-db redis keyspace info
- [#226](https://github.com/influxdb/telegraf/pull/226): Add timestamps to points in Kafka/AMQP outputs. Thanks @ekini
- [#90](https://github.com/influxdb/telegraf/issues/90): Add Docker labels to tags in docker plugin
- [#223](https://github.com/influxdb/telegraf/pull/223): Add port tag to nginx plugin. Thanks @neezgee!
- [#227](https://github.com/influxdb/telegraf/pull/227): Add command intervals to exec plugin. Thanks @jpalay!
### Bigfixes
### Bugfixes
- [#228](https://github.com/influxdb/telegraf/pull/228): New version of package will replace old one. Thanks @ekini!
## v0.1.9 [2015-09-22]
### Release Notes
- InfluxDB output config change: `url` is now `urls`, and is a list. Config files
will still be backwards compatible if only `url` is specified.
- The -test flag will now output two metric collections
- Support for filtering telegraf outputs on the CLI -- Telegraf will now
allow filtering of output sinks on the command-line using the `-outputfilter`
flag, much like how the `-filter` flag works for plugins.
- Support for filtering on config-file creation -- Telegraf now supports
filtering to -sample-config command. You can now run
`telegraf -sample-config -filter cpu -outputfilter influxdb` to get a config
file with only the cpu plugin defined, and the influxdb output defined.
- **Breaking Change**: The CPU collection plugin has been refactored to fix some
bugs and outdated dependency issues. At the same time, I also decided to fix
a naming consistency issue, so cpu_percentageIdle will become cpu_usage_idle.
Also, all CPU time measurements now have it indicated in their name, so cpu_idle
will become cpu_time_idle. Additionally, cpu_time measurements are going to be
dropped in the default config.
- **Breaking Change**: The memory plugin has been refactored and some measurements
have been renamed for consistency. Some measurements have also been removed from being outputted. They are still being collected by gopsutil, and could easily be
re-added in a "verbose" mode if there is demand for it.
### Features
- [#143](https://github.com/influxdb/telegraf/issues/143): InfluxDB clustering support
- [#181](https://github.com/influxdb/telegraf/issues/181): Makefile GOBIN support. Thanks @Vye!
- [#203](https://github.com/influxdb/telegraf/pull/200): AMQP output. Thanks @ekini!
- [#182](https://github.com/influxdb/telegraf/pull/182): OpenTSDB output. Thanks @rplessl!
- [#187](https://github.com/influxdb/telegraf/pull/187): Retry output sink connections on startup.
- [#220](https://github.com/influxdb/telegraf/pull/220): Add port tag to apache plugin. Thanks @neezgee!
- [#217](https://github.com/influxdb/telegraf/pull/217): Add filtering for output sinks
and filtering when specifying a config file.
### Bugfixes
- [#170](https://github.com/influxdb/telegraf/issues/170): Systemd support
- [#175](https://github.com/influxdb/telegraf/issues/175): Set write precision before gathering metrics
- [#178](https://github.com/influxdb/telegraf/issues/178): redis plugin, multiple server thread hang bug
- Fix net plugin on darwin
- [#84](https://github.com/influxdb/telegraf/issues/84): Fix docker plugin on CentOS. Thanks @neezgee!
- [#189](https://github.com/influxdb/telegraf/pull/189): Fix mem_used_perc. Thanks @mced!
- [#192](https://github.com/influxdb/telegraf/issues/192): Increase compatibility of postgresql plugin. Now supports versions 8.1+
- [#203](https://github.com/influxdb/telegraf/issues/203): EL5 rpm support. Thanks @ekini!
- [#206](https://github.com/influxdb/telegraf/issues/206): CPU steal/guest values wrong on linux.
- [#212](https://github.com/influxdb/telegraf/issues/212): Add hashbang to postinstall script. Thanks @ekini!
- [#212](https://github.com/influxdb/telegraf/issues/212): Fix makefile warning. Thanks @ekini!
## v0.1.8 [2015-09-04]
### Release Notes
- Telegraf will now write data in UTC at second precision by default
- Now using Go 1.5 to build telegraf
### Features
- [#150](https://github.com/influxdb/telegraf/pull/150): Add Host Uptime metric to system plugin
- [#158](https://github.com/influxdb/telegraf/pull/158): Apache Plugin. Thanks @KPACHbIuLLIAnO4
- [#159](https://github.com/influxdb/telegraf/pull/159): Use second precision for InfluxDB writes
- [#165](https://github.com/influxdb/telegraf/pull/165): Add additional metrics to mysql plugin. Thanks @nickscript0
- [#162](https://github.com/influxdb/telegraf/pull/162): Write UTC by default, provide option
- [#166](https://github.com/influxdb/telegraf/pull/166): Upload binaries to S3
- [#169](https://github.com/influxdb/telegraf/pull/169): Ping plugin
### Bugfixes
## v0.1.7 [2015-08-28]
### Features
- [#38](https://github.com/influxdb/telegraf/pull/38): Kafka output producer.
- [#133](https://github.com/influxdb/telegraf/pull/133): Add plugin.Gather error logging. Thanks @nickscript0!
- [#136](https://github.com/influxdb/telegraf/issues/136): Add a -usage flag for printing usage of a single plugin.
- [#137](https://github.com/influxdb/telegraf/issues/137): Memcached: fix when a value contains a space
- [#138](https://github.com/influxdb/telegraf/issues/138): MySQL server address tag.
- [#142](https://github.com/influxdb/telegraf/pull/142): Add Description and SampleConfig funcs to output interface
- Indent the toml config file for readability
### Bugfixes
- [#128](https://github.com/influxdb/telegraf/issues/128): system_load measurement missing.
- [#129](https://github.com/influxdb/telegraf/issues/129): Latest pkg url fix.
- [#131](https://github.com/influxdb/telegraf/issues/131): Fix memory reporting on linux & darwin. Thanks @subhachandrachandra!
- [#140](https://github.com/influxdb/telegraf/issues/140): Memory plugin prec->perc typo fix. Thanks @brunoqc!
## v0.1.6 [2015-08-20]

163
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,163 @@
## Sign the CLA
Before we can merge a pull request, you will need to sign the CLA,
which can be found [on our website](http://influxdb.com/community/cla.html)
## Plugins
This section is for developers that want to create new collection plugins.
Telegraf is entirely plugin driven. This interface allows for operators to
pick and chose what is gathered as well as makes it easy for developers
to create new ways of generating metrics.
Plugin authorship is kept as simple as possible to promote people to develop
and submit new plugins.
### Plugin Guidelines
* A plugin must conform to the `plugins.Plugin` interface.
* Telegraf promises to run each plugin's Gather function serially. This means
developers don't have to worry about thread safety within these functions.
* Each generated metric automatically has the name of the plugin that generated
it prepended. This is to keep plugins honest.
* Plugins should call `plugins.Add` in their `init` function to register themselves.
See below for a quick example.
* To be available within Telegraf itself, plugins must add themselves to the
`github.com/influxdb/telegraf/plugins/all/all.go` file.
* The `SampleConfig` function should return valid toml that describes how the
plugin can be configured. This is include in `telegraf -sample-config`.
* The `Description` function should say in one line what this plugin does.
### Plugin interface
```go
type Plugin interface {
SampleConfig() string
Description() string
Gather(Accumulator) error
}
type Accumulator interface {
Add(measurement string, value interface{}, tags map[string]string)
AddFieldsWithTime(measurement string,
values map[string]interface{},
tags map[string]string,
timestamp time.Time)
}
```
### Accumulator
The way that a plugin emits metrics is by interacting with the Accumulator.
The `Add` function takes 3 arguments:
* **measurement**: A string description of the metric. For instance `bytes_read` or `faults`.
* **value**: A value for the metric. This accepts 5 different types of value:
* **int**: The most common type. All int types are accepted but favor using `int64`
Useful for counters, etc.
* **float**: Favor `float64`, useful for gauges, percentages, etc.
* **bool**: `true` or `false`, useful to indicate the presence of a state. `light_on`, etc.
* **string**: Typically used to indicate a message, or some kind of freeform information.
* **time.Time**: Useful for indicating when a state last occurred, for instance `light_on_since`.
* **tags**: This is a map of strings to strings to describe the where or who
about the metric. For instance, the `net` plugin adds a tag named `"interface"`
set to the name of the network interface, like `"eth0"`.
The `AddFieldsWithTime` allows multiple values for a point to be passed. The values
used are the same type profile as **value** above. The **timestamp** argument
allows a point to be registered as having occurred at an arbitrary time.
Let's say you've written a plugin that emits metrics about processes on the current host.
```go
type Process struct {
CPUTime float64
MemoryBytes int64
PID int
}
func Gather(acc plugins.Accumulator) error {
for _, process := range system.Processes() {
tags := map[string]string {
"pid": fmt.Sprintf("%d", process.Pid),
}
acc.Add("cpu", process.CPUTime, tags)
acc.Add("memory", process.MemoryBytes, tags)
}
}
```
### Example
```go
package simple
// simple.go
import "github.com/influxdb/telegraf/plugins"
type Simple struct {
Ok bool
}
func (s *Simple) Description() string {
return "a demo plugin"
}
func (s *Simple) SampleConfig() string {
return "ok = true # indicate if everything is fine"
}
func (s *Simple) Gather(acc plugins.Accumulator) error {
if s.Ok {
acc.Add("state", "pretty good", nil)
} else {
acc.Add("state", "not great", nil)
}
return nil
}
func init() {
plugins.Add("simple", func() plugins.Plugin { return &Simple{} })
}
```
## Outputs
TODO: this section will describe requirements for contributing an output
## Unit Tests
### Execute short tests
execute `make test-short`
### Execute long tests
As Telegraf collects metrics from several third-party services it becomes a
difficult task to mock each service as some of them have complicated protocols
which would take some time to replicate.
To overcome this situation we've decided to use docker containers to provide a
fast and reproducible environment to test those services which require it.
For other situations
(i.e: https://github.com/influxdb/telegraf/blob/master/plugins/redis/redis_test.go )
a simple mock will suffice.
To execute Telegraf tests follow these simple steps:
- Install docker compose following [these](https://docs.docker.com/compose/install/)
instructions
- execute `make test`
**OSX users**: you will need to install `boot2docker` or `docker-machine`.
The Makefile will assume that you have a `docker-machine` box called `default` to
get the IP address.
### Unit test troubleshooting
Try cleaning up your test environment by executing `make test-cleanup` and
re-running

79
Godeps/Godeps.json generated
View File

@ -1,6 +1,6 @@
{
"ImportPath": "github.com/influxdb/telegraf",
"GoVersion": "go1.4.2",
"GoVersion": "go1.5",
"Packages": [
"./..."
],
@ -28,6 +28,11 @@
"ImportPath": "github.com/cenkalti/backoff",
"Rev": "4dc77674aceaabba2c7e3da25d4c823edfb73f99"
},
{
"ImportPath": "github.com/cloudfoundry/gosigar",
"Comment": "scotty_09012012-27-g3ed7c74",
"Rev": "3ed7c74352dae6dc00bdc8c74045375352e3ec05"
},
{
"ImportPath": "github.com/dancannon/gorethink/encoding",
"Comment": "v1.x.x-1-g786f12a",
@ -55,7 +60,7 @@
},
{
"ImportPath": "github.com/fsouza/go-dockerclient",
"Rev": "42d06e2b125654477366c320dcea99107a86e9c2"
"Rev": "af9789bbd78acf3e279274caa54682185eb7ed33"
},
{
"ImportPath": "github.com/go-sql-driver/mysql",
@ -91,34 +96,9 @@
"Rev": "d1e82c1ec3f15ee991f7cc7ffd5b67ff6f5bbaee"
},
{
"ImportPath": "github.com/influxdb/influxdb/client",
"Comment": "v0.9.3-rc1",
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
},
{
"ImportPath": "github.com/influxdb/influxdb/influxql",
"Comment": "v0.9.3-rc1",
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
},
{
"ImportPath": "github.com/influxdb/influxdb/meta",
"Comment": "v0.9.3-rc1",
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
},
{
"ImportPath": "github.com/influxdb/influxdb/snapshot",
"Comment": "v0.9.3-rc1",
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
},
{
"ImportPath": "github.com/influxdb/influxdb/toml",
"Comment": "v0.9.3-rc1",
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
},
{
"ImportPath": "github.com/influxdb/influxdb/tsdb",
"Comment": "v0.9.3-rc1",
"Rev": "f4077764b2bb2b03241452d88e9db321c62bb560"
"ImportPath": "github.com/influxdb/influxdb",
"Comment": "v0.9.3",
"Rev": "5d42b212fca8facfe9db0c83822f09b88be643ec"
},
{
"ImportPath": "github.com/lib/pq",
@ -161,6 +141,45 @@
"ImportPath": "github.com/samuel/go-zookeeper/zk",
"Rev": "5bb5cfc093ad18a28148c578f8632cfdb4d802e4"
},
{
"ImportPath": "github.com/shirou/gopsutil/common",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/shirou/gopsutil/cpu",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/shirou/gopsutil/disk",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/shirou/gopsutil/docker",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/shirou/gopsutil/load",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/shirou/gopsutil/mem",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/shirou/gopsutil/net",
"Comment": "1.0.0-158-g0fd612e",
"Rev": "0fd612ec7b9079dc624ae4815acadf1903d82011"
},
{
"ImportPath": "github.com/streadway/amqp",
"Rev": "f4879ba28fffbb576743b03622a9ff20461826b2"
},
{
"ImportPath": "github.com/stretchr/objx",
"Rev": "cbeaeb16a013161a98496fad62933b1d21786672"

1
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/.gitignore generated vendored Normal file
View File

@ -0,0 +1 @@
.vagrant

8
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/.travis.yml generated vendored Normal file
View File

@ -0,0 +1,8 @@
language: go
go:
- 1.2
install:
- 'go install github.com/onsi/ginkgo/ginkgo'
script: 'ginkgo -r'

201
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/LICENSE generated vendored Normal file
View File

@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

9
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/NOTICE generated vendored Normal file
View File

@ -0,0 +1,9 @@
Copyright (c) [2009-2011] VMware, Inc. All Rights Reserved.
This product is licensed to you under the Apache License, Version 2.0 (the "License").
You may not use this product except in compliance with the License.
This product includes a number of subcomponents with
separate copyright notices and license terms. Your use of these
subcomponents is subject to the terms and conditions of the
subcomponent's license, as noted in the LICENSE file.

22
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/README.md generated vendored Normal file
View File

@ -0,0 +1,22 @@
# Go sigar
## Overview
Go sigar is a golang implementation of the
[sigar API](https://github.com/hyperic/sigar). The Go version of
sigar has a very similar interface, but is being written from scratch
in pure go/cgo, rather than cgo bindings for libsigar.
## Test drive
$ go get github.com/cloudfoundry/gosigar
$ cd $GOPATH/src/github.com/cloudfoundry/gosigar/examples
$ go run uptime.go
## Supported platforms
Currently targeting modern flavors of darwin and linux.
## License
Apache 2.0

25
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/Vagrantfile generated vendored Normal file
View File

@ -0,0 +1,25 @@
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "hashicorp/precise64"
config.vm.provision "shell", inline: "mkdir -p /home/vagrant/go"
config.vm.synced_folder ".", "/home/vagrant/go/src/github.com/cloudfoundry/gosigar"
config.vm.provision "shell", inline: "chown -R vagrant:vagrant /home/vagrant/go"
install_go = <<-BASH
set -e
if [ ! -d "/usr/local/go" ]; then
cd /tmp && wget https://storage.googleapis.com/golang/go1.3.3.linux-amd64.tar.gz
cd /usr/local
tar xvzf /tmp/go1.3.3.linux-amd64.tar.gz
echo 'export GOPATH=/home/vagrant/go; export PATH=/usr/local/go/bin:$PATH:$GOPATH/bin' >> /home/vagrant/.bashrc
fi
export GOPATH=/home/vagrant/go
export PATH=/usr/local/go/bin:$PATH:$GOPATH/bin
/usr/local/go/bin/go get -u github.com/onsi/ginkgo/ginkgo
/usr/local/go/bin/go get -u github.com/onsi/gomega;
BASH
config.vm.provision "shell", inline: 'apt-get install -y git-core'
config.vm.provision "shell", inline: install_go
end

69
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/concrete_sigar.go generated vendored Normal file
View File

@ -0,0 +1,69 @@
package sigar
import (
"time"
)
type ConcreteSigar struct{}
func (c *ConcreteSigar) CollectCpuStats(collectionInterval time.Duration) (<-chan Cpu, chan<- struct{}) {
// samplesCh is buffered to 1 value to immediately return first CPU sample
samplesCh := make(chan Cpu, 1)
stopCh := make(chan struct{})
go func() {
var cpuUsage Cpu
// Immediately provide non-delta value.
// samplesCh is buffered to 1 value, so it will not block.
cpuUsage.Get()
samplesCh <- cpuUsage
ticker := time.NewTicker(collectionInterval)
for {
select {
case <-ticker.C:
previousCpuUsage := cpuUsage
cpuUsage.Get()
select {
case samplesCh <- cpuUsage.Delta(previousCpuUsage):
default:
// Include default to avoid channel blocking
}
case <-stopCh:
return
}
}
}()
return samplesCh, stopCh
}
func (c *ConcreteSigar) GetLoadAverage() (LoadAverage, error) {
l := LoadAverage{}
err := l.Get()
return l, err
}
func (c *ConcreteSigar) GetMem() (Mem, error) {
m := Mem{}
err := m.Get()
return m, err
}
func (c *ConcreteSigar) GetSwap() (Swap, error) {
s := Swap{}
err := s.Get()
return s, err
}
func (c *ConcreteSigar) GetFileSystemUsage(path string) (FileSystemUsage, error) {
f := FileSystemUsage{}
err := f.Get(path)
return f, err
}

85
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/concrete_sigar_test.go generated vendored Normal file
View File

@ -0,0 +1,85 @@
package sigar_test
import (
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
sigar "github.com/cloudfoundry/gosigar"
)
var _ = Describe("ConcreteSigar", func() {
var concreteSigar *sigar.ConcreteSigar
BeforeEach(func() {
concreteSigar = &sigar.ConcreteSigar{}
})
Describe("CollectCpuStats", func() {
It("immediately makes first CPU usage available even though it's not very accurate", func() {
samplesCh, stop := concreteSigar.CollectCpuStats(500 * time.Millisecond)
firstValue := <-samplesCh
Expect(firstValue.User).To(BeNumerically(">", 0))
stop <- struct{}{}
})
It("makes CPU usage delta values available", func() {
samplesCh, stop := concreteSigar.CollectCpuStats(500 * time.Millisecond)
firstValue := <-samplesCh
secondValue := <-samplesCh
Expect(secondValue.User).To(BeNumerically("<", firstValue.User))
stop <- struct{}{}
})
It("does not block", func() {
_, stop := concreteSigar.CollectCpuStats(10 * time.Millisecond)
// Sleep long enough for samplesCh to fill at least 2 values
time.Sleep(20 * time.Millisecond)
stop <- struct{}{}
// If CollectCpuStats blocks it will never get here
Expect(true).To(BeTrue())
})
})
It("GetLoadAverage", func() {
avg, err := concreteSigar.GetLoadAverage()
Expect(avg.One).ToNot(BeNil())
Expect(avg.Five).ToNot(BeNil())
Expect(avg.Fifteen).ToNot(BeNil())
Expect(err).ToNot(HaveOccurred())
})
It("GetMem", func() {
mem, err := concreteSigar.GetMem()
Expect(err).ToNot(HaveOccurred())
Expect(mem.Total).To(BeNumerically(">", 0))
Expect(mem.Used + mem.Free).To(BeNumerically("<=", mem.Total))
})
It("GetSwap", func() {
swap, err := concreteSigar.GetSwap()
Expect(err).ToNot(HaveOccurred())
Expect(swap.Used + swap.Free).To(BeNumerically("<=", swap.Total))
})
It("GetSwap", func() {
fsusage, err := concreteSigar.GetFileSystemUsage("/")
Expect(err).ToNot(HaveOccurred())
Expect(fsusage.Total).ToNot(BeNil())
fsusage, err = concreteSigar.GetFileSystemUsage("T O T A L L Y B O G U S")
Expect(err).To(HaveOccurred())
Expect(fsusage.Total).To(Equal(uint64(0)))
})
})

52
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/cputimes.go generated vendored Normal file
View File

@ -0,0 +1,52 @@
package main
import (
"fmt"
"time"
"github.com/cloudfoundry/gosigar"
)
func main() {
cpus := sigar.CpuList{}
cpus.Get()
tcpu := getOverallCpu(cpus)
for i, cpu := range cpus.List {
fmt.Printf("CPU%d Ticks: %d\n", i, cpu.Total())
}
fmt.Printf("Total CPU Ticks: %d\n", tcpu.Total())
fmt.Printf("Total CPU Time: %d\n", tcpu.Total()/128)
fmt.Printf("User CPU Time: %d\n", tcpu.User/128)
time.Sleep(1 * time.Second)
tcpu2 := sigar.Cpu{}
tcpu2.Get()
dcpu := tcpu2.Delta(tcpu)
tcpuDelta := tcpu2.Total() - tcpu.Total()
iPercentage := 100.0 * float64(dcpu.Idle) / float64(tcpuDelta)
fmt.Printf("Idle percentage: %f\n", iPercentage)
bPercentage := 100.0 * float64(busy(tcpu2)-busy(tcpu)) / float64(tcpuDelta)
fmt.Printf("Busy percentage: %f\n", bPercentage)
}
func busy(c sigar.Cpu) uint64 {
return c.Total() - c.Idle
}
func getOverallCpu(cl sigar.CpuList) sigar.Cpu {
var overallCpu sigar.Cpu
for _, c := range cl.List {
overallCpu.User += c.User
overallCpu.Nice += c.Nice
overallCpu.Sys += c.Sys
overallCpu.Idle += c.Idle
overallCpu.Wait += c.Wait
overallCpu.Irq += c.Irq
overallCpu.SoftIrq += c.SoftIrq
overallCpu.Stolen += c.Stolen
}
return overallCpu
}

39
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/df.go generated vendored Normal file
View File

@ -0,0 +1,39 @@
// Copyright (c) 2012 VMware, Inc.
package main
import (
"fmt"
"github.com/cloudfoundry/gosigar"
"os"
)
const output_format = "%-15s %4s %4s %5s %4s %-15s\n"
func formatSize(size uint64) string {
return sigar.FormatSize(size * 1024)
}
func main() {
fslist := sigar.FileSystemList{}
fslist.Get()
fmt.Fprintf(os.Stdout, output_format,
"Filesystem", "Size", "Used", "Avail", "Use%", "Mounted on")
for _, fs := range fslist.List {
dir_name := fs.DirName
usage := sigar.FileSystemUsage{}
usage.Get(dir_name)
fmt.Fprintf(os.Stdout, output_format,
fs.DevName,
formatSize(usage.Total),
formatSize(usage.Used),
formatSize(usage.Avail),
sigar.FormatPercent(usage.UsePercent()),
dir_name)
}
}

33
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/free.go generated vendored Normal file
View File

@ -0,0 +1,33 @@
// Copyright (c) 2012 VMware, Inc.
package main
import (
"fmt"
"github.com/cloudfoundry/gosigar"
"os"
)
func format(val uint64) uint64 {
return val / 1024
}
func main() {
mem := sigar.Mem{}
swap := sigar.Swap{}
mem.Get()
swap.Get()
fmt.Fprintf(os.Stdout, "%18s %10s %10s\n",
"total", "used", "free")
fmt.Fprintf(os.Stdout, "Mem: %10d %10d %10d\n",
format(mem.Total), format(mem.Used), format(mem.Free))
fmt.Fprintf(os.Stdout, "-/+ buffers/cache: %10d %10d\n",
format(mem.ActualUsed), format(mem.ActualFree))
fmt.Fprintf(os.Stdout, "Swap: %10d %10d %10d\n",
format(swap.Total), format(swap.Used), format(swap.Free))
}

37
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/ps.go generated vendored Normal file
View File

@ -0,0 +1,37 @@
// Copyright (c) 2012 VMware, Inc.
package main
import (
"fmt"
"github.com/cloudfoundry/gosigar"
)
func main() {
pids := sigar.ProcList{}
pids.Get()
// ps -eo pid,ppid,stime,time,rss,state,comm
fmt.Print(" PID PPID STIME TIME RSS S COMMAND\n")
for _, pid := range pids.List {
state := sigar.ProcState{}
mem := sigar.ProcMem{}
time := sigar.ProcTime{}
if err := state.Get(pid); err != nil {
continue
}
if err := mem.Get(pid); err != nil {
continue
}
if err := time.Get(pid); err != nil {
continue
}
fmt.Printf("%5d %5d %s %s %6d %c %s\n",
pid, state.Ppid,
time.FormatStartTime(), time.FormatTotal(),
mem.Resident/1024, state.State, state.Name)
}
}

27
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/examples/uptime.go generated vendored Normal file
View File

@ -0,0 +1,27 @@
// Copyright (c) 2012 VMware, Inc.
package main
import (
"fmt"
"github.com/cloudfoundry/gosigar"
"os"
"time"
)
func main() {
concreteSigar := sigar.ConcreteSigar{}
uptime := sigar.Uptime{}
uptime.Get()
avg, err := concreteSigar.GetLoadAverage()
if err != nil {
fmt.Printf("Failed to get load average")
return
}
fmt.Fprintf(os.Stdout, " %s up %s load average: %.2f, %.2f, %.2f\n",
time.Now().Format("15:04:05"),
uptime.Format(),
avg.One, avg.Five, avg.Fifteen)
}

72
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/fakes/fake_sigar.go generated vendored Normal file
View File

@ -0,0 +1,72 @@
package fakes
import (
"time"
sigar "github.com/cloudfoundry/gosigar"
)
type FakeSigar struct {
LoadAverage sigar.LoadAverage
LoadAverageErr error
Mem sigar.Mem
MemErr error
Swap sigar.Swap
SwapErr error
FileSystemUsage sigar.FileSystemUsage
FileSystemUsageErr error
FileSystemUsagePath string
CollectCpuStatsCpuCh chan sigar.Cpu
CollectCpuStatsStopCh chan struct{}
}
func NewFakeSigar() *FakeSigar {
return &FakeSigar{
CollectCpuStatsCpuCh: make(chan sigar.Cpu, 1),
CollectCpuStatsStopCh: make(chan struct{}),
}
}
func (f *FakeSigar) CollectCpuStats(collectionInterval time.Duration) (<-chan sigar.Cpu, chan<- struct{}) {
samplesCh := make(chan sigar.Cpu, 1)
stopCh := make(chan struct{})
go func() {
for {
select {
case cpuStat := <-f.CollectCpuStatsCpuCh:
select {
case samplesCh <- cpuStat:
default:
// Include default to avoid channel blocking
}
case <-f.CollectCpuStatsStopCh:
return
}
}
}()
return samplesCh, stopCh
}
func (f *FakeSigar) GetLoadAverage() (sigar.LoadAverage, error) {
return f.LoadAverage, f.LoadAverageErr
}
func (f *FakeSigar) GetMem() (sigar.Mem, error) {
return f.Mem, f.MemErr
}
func (f *FakeSigar) GetSwap() (sigar.Swap, error) {
return f.Swap, f.SwapErr
}
func (f *FakeSigar) GetFileSystemUsage(path string) (sigar.FileSystemUsage, error) {
f.FileSystemUsagePath = path
return f.FileSystemUsage, f.FileSystemUsageErr
}

50
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/README.md generated vendored Normal file
View File

@ -0,0 +1,50 @@
# Process notifications for Go
## Overview
The psnotify package captures process events from the kernel via
kqueue on Darwin/BSD and the netlink connector on Linux.
The psnotify API is similar to the
[fsnotify](https://github.com/howeyc/fsnotify) package.
Example:
```go
watcher, err := psnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
// Process events
go func() {
for {
select {
case ev := <-watcher.Fork:
log.Println("fork event:", ev)
case ev := <-watcher.Exec:
log.Println("exec event:", ev)
case ev := <-watcher.Exit:
log.Println("exit event:", ev)
case err := <-watcher.Error:
log.Println("error:", err)
}
}
}()
err = watcher.Watch(os.Getpid(), psnotify.PROC_EVENT_ALL)
if err != nil {
log.Fatal(err)
}
/* ... do stuff ... */
watcher.Close()
```
## Supported platforms
Currently targeting modern flavors of Darwin and Linux.
Should work on BSD, but untested.
## License
Apache 2.0

136
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify.go generated vendored Normal file
View File

@ -0,0 +1,136 @@
// Copyright (c) 2012 VMware, Inc.
package psnotify
import (
"errors"
"fmt"
)
type ProcEventFork struct {
ParentPid int // Pid of the process that called fork()
ChildPid int // Child process pid created by fork()
}
type ProcEventExec struct {
Pid int // Pid of the process that called exec()
}
type ProcEventExit struct {
Pid int // Pid of the process that called exit()
}
type watch struct {
flags uint32 // Saved value of Watch() flags param
}
type eventListener interface {
close() error // Watch.Close() closes the OS specific listener
}
type Watcher struct {
listener eventListener // OS specifics (kqueue or netlink)
watches map[int]*watch // Map of watched process ids
Error chan error // Errors are sent on this channel
Fork chan *ProcEventFork // Fork events are sent on this channel
Exec chan *ProcEventExec // Exec events are sent on this channel
Exit chan *ProcEventExit // Exit events are sent on this channel
done chan bool // Used to stop the readEvents() goroutine
isClosed bool // Set to true when Close() is first called
}
// Initialize event listener and channels
func NewWatcher() (*Watcher, error) {
listener, err := createListener()
if err != nil {
return nil, err
}
w := &Watcher{
listener: listener,
watches: make(map[int]*watch),
Fork: make(chan *ProcEventFork),
Exec: make(chan *ProcEventExec),
Exit: make(chan *ProcEventExit),
Error: make(chan error),
done: make(chan bool, 1),
}
go w.readEvents()
return w, nil
}
// Close event channels when done message is received
func (w *Watcher) finish() {
close(w.Fork)
close(w.Exec)
close(w.Exit)
close(w.Error)
}
// Closes the OS specific event listener,
// removes all watches and closes all event channels.
func (w *Watcher) Close() error {
if w.isClosed {
return nil
}
w.isClosed = true
for pid := range w.watches {
w.RemoveWatch(pid)
}
w.done <- true
w.listener.close()
return nil
}
// Add pid to the watched process set.
// The flags param is a bitmask of process events to capture,
// must be one or more of: PROC_EVENT_FORK, PROC_EVENT_EXEC, PROC_EVENT_EXIT
func (w *Watcher) Watch(pid int, flags uint32) error {
if w.isClosed {
return errors.New("psnotify watcher is closed")
}
watchEntry, found := w.watches[pid]
if found {
watchEntry.flags |= flags
} else {
if err := w.register(pid, flags); err != nil {
return err
}
w.watches[pid] = &watch{flags: flags}
}
return nil
}
// Remove pid from the watched process set.
func (w *Watcher) RemoveWatch(pid int) error {
_, ok := w.watches[pid]
if !ok {
msg := fmt.Sprintf("watch for pid=%d does not exist", pid)
return errors.New(msg)
}
delete(w.watches, pid)
return w.unregister(pid)
}
// Internal helper to check if there is a message on the "done" channel.
// The "done" message is sent by the Close() method; when received here,
// the Watcher.finish method is called to close all channels and return
// true - in which case the caller should break from the readEvents loop.
func (w *Watcher) isDone() bool {
var done bool
select {
case done = <-w.done:
w.finish()
default:
}
return done
}

93
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_bsd.go generated vendored Normal file
View File

@ -0,0 +1,93 @@
// Copyright (c) 2012 VMware, Inc.
// +build darwin freebsd netbsd openbsd
// Go interface to BSD kqueue process events.
package psnotify
import (
"syscall"
)
const (
// Flags (from <sys/event.h>)
PROC_EVENT_FORK = syscall.NOTE_FORK // fork() events
PROC_EVENT_EXEC = syscall.NOTE_EXEC // exec() events
PROC_EVENT_EXIT = syscall.NOTE_EXIT // exit() events
// Watch for all process events
PROC_EVENT_ALL = PROC_EVENT_FORK | PROC_EVENT_EXEC | PROC_EVENT_EXIT
)
type kqueueListener struct {
kq int // The syscall.Kqueue() file descriptor
buf [1]syscall.Kevent_t // An event buffer for Add/Remove watch
}
// Initialize bsd implementation of the eventListener interface
func createListener() (eventListener, error) {
listener := &kqueueListener{}
kq, err := syscall.Kqueue()
listener.kq = kq
return listener, err
}
// Initialize Kevent_t fields and propagate changelist for the given pid
func (w *Watcher) kevent(pid int, fflags uint32, flags int) error {
listener, _ := w.listener.(*kqueueListener)
event := &listener.buf[0]
syscall.SetKevent(event, pid, syscall.EVFILT_PROC, flags)
event.Fflags = fflags
_, err := syscall.Kevent(listener.kq, listener.buf[:], nil, nil)
return err
}
// Delete filter for given pid from the queue
func (w *Watcher) unregister(pid int) error {
return w.kevent(pid, 0, syscall.EV_DELETE)
}
// Add and enable filter for given pid in the queue
func (w *Watcher) register(pid int, flags uint32) error {
return w.kevent(pid, flags, syscall.EV_ADD|syscall.EV_ENABLE)
}
// Poll the kqueue file descriptor and dispatch to the Event channels
func (w *Watcher) readEvents() {
listener, _ := w.listener.(*kqueueListener)
events := make([]syscall.Kevent_t, 10)
for {
if w.isDone() {
return
}
n, err := syscall.Kevent(listener.kq, nil, events, nil)
if err != nil {
w.Error <- err
continue
}
for _, ev := range events[:n] {
pid := int(ev.Ident)
switch ev.Fflags {
case syscall.NOTE_FORK:
w.Fork <- &ProcEventFork{ParentPid: pid}
case syscall.NOTE_EXEC:
w.Exec <- &ProcEventExec{Pid: pid}
case syscall.NOTE_EXIT:
w.RemoveWatch(pid)
w.Exit <- &ProcEventExit{Pid: pid}
}
}
}
}
// Close our kqueue file descriptor; deletes any remaining filters
func (listener *kqueueListener) close() error {
return syscall.Close(listener.kq)
}

253
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_linux.go generated vendored Normal file
View File

@ -0,0 +1,253 @@
// Copyright (c) 2012 VMware, Inc.
// Go interface to the Linux netlink process connector.
// See Documentation/connector/connector.txt in the linux kernel source tree.
package psnotify
import (
"bytes"
"encoding/binary"
"os"
"syscall"
)
const (
// internal flags (from <linux/connector.h>)
_CN_IDX_PROC = 0x1
_CN_VAL_PROC = 0x1
// internal flags (from <linux/cn_proc.h>)
_PROC_CN_MCAST_LISTEN = 1
_PROC_CN_MCAST_IGNORE = 2
// Flags (from <linux/cn_proc.h>)
PROC_EVENT_FORK = 0x00000001 // fork() events
PROC_EVENT_EXEC = 0x00000002 // exec() events
PROC_EVENT_EXIT = 0x80000000 // exit() events
// Watch for all process events
PROC_EVENT_ALL = PROC_EVENT_FORK | PROC_EVENT_EXEC | PROC_EVENT_EXIT
)
var (
byteOrder = binary.LittleEndian
)
// linux/connector.h: struct cb_id
type cbId struct {
Idx uint32
Val uint32
}
// linux/connector.h: struct cb_msg
type cnMsg struct {
Id cbId
Seq uint32
Ack uint32
Len uint16
Flags uint16
}
// linux/cn_proc.h: struct proc_event.{what,cpu,timestamp_ns}
type procEventHeader struct {
What uint32
Cpu uint32
Timestamp uint64
}
// linux/cn_proc.h: struct proc_event.fork
type forkProcEvent struct {
ParentPid uint32
ParentTgid uint32
ChildPid uint32
ChildTgid uint32
}
// linux/cn_proc.h: struct proc_event.exec
type execProcEvent struct {
ProcessPid uint32
ProcessTgid uint32
}
// linux/cn_proc.h: struct proc_event.exit
type exitProcEvent struct {
ProcessPid uint32
ProcessTgid uint32
ExitCode uint32
ExitSignal uint32
}
// standard netlink header + connector header
type netlinkProcMessage struct {
Header syscall.NlMsghdr
Data cnMsg
}
type netlinkListener struct {
addr *syscall.SockaddrNetlink // Netlink socket address
sock int // The syscall.Socket() file descriptor
seq uint32 // struct cn_msg.seq
}
// Initialize linux implementation of the eventListener interface
func createListener() (eventListener, error) {
listener := &netlinkListener{}
err := listener.bind()
return listener, err
}
// noop on linux
func (w *Watcher) unregister(pid int) error {
return nil
}
// noop on linux
func (w *Watcher) register(pid int, flags uint32) error {
return nil
}
// Read events from the netlink socket
func (w *Watcher) readEvents() {
buf := make([]byte, syscall.Getpagesize())
listener, _ := w.listener.(*netlinkListener)
for {
if w.isDone() {
return
}
nr, _, err := syscall.Recvfrom(listener.sock, buf, 0)
if err != nil {
w.Error <- err
continue
}
if nr < syscall.NLMSG_HDRLEN {
w.Error <- syscall.EINVAL
continue
}
msgs, _ := syscall.ParseNetlinkMessage(buf[:nr])
for _, m := range msgs {
if m.Header.Type == syscall.NLMSG_DONE {
w.handleEvent(m.Data)
}
}
}
}
// Internal helper to check if pid && event is being watched
func (w *Watcher) isWatching(pid int, event uint32) bool {
if watch, ok := w.watches[pid]; ok {
return (watch.flags & event) == event
}
return false
}
// Dispatch events from the netlink socket to the Event channels.
// Unlike bsd kqueue, netlink receives events for all pids,
// so we apply filtering based on the watch table via isWatching()
func (w *Watcher) handleEvent(data []byte) {
buf := bytes.NewBuffer(data)
msg := &cnMsg{}
hdr := &procEventHeader{}
binary.Read(buf, byteOrder, msg)
binary.Read(buf, byteOrder, hdr)
switch hdr.What {
case PROC_EVENT_FORK:
event := &forkProcEvent{}
binary.Read(buf, byteOrder, event)
ppid := int(event.ParentTgid)
pid := int(event.ChildTgid)
if w.isWatching(ppid, PROC_EVENT_EXEC) {
// follow forks
watch, _ := w.watches[ppid]
w.Watch(pid, watch.flags)
}
if w.isWatching(ppid, PROC_EVENT_FORK) {
w.Fork <- &ProcEventFork{ParentPid: ppid, ChildPid: pid}
}
case PROC_EVENT_EXEC:
event := &execProcEvent{}
binary.Read(buf, byteOrder, event)
pid := int(event.ProcessTgid)
if w.isWatching(pid, PROC_EVENT_EXEC) {
w.Exec <- &ProcEventExec{Pid: pid}
}
case PROC_EVENT_EXIT:
event := &exitProcEvent{}
binary.Read(buf, byteOrder, event)
pid := int(event.ProcessTgid)
if w.isWatching(pid, PROC_EVENT_EXIT) {
w.RemoveWatch(pid)
w.Exit <- &ProcEventExit{Pid: pid}
}
}
}
// Bind our netlink socket and
// send a listen control message to the connector driver.
func (listener *netlinkListener) bind() error {
sock, err := syscall.Socket(
syscall.AF_NETLINK,
syscall.SOCK_DGRAM,
syscall.NETLINK_CONNECTOR)
if err != nil {
return err
}
listener.sock = sock
listener.addr = &syscall.SockaddrNetlink{
Family: syscall.AF_NETLINK,
Groups: _CN_IDX_PROC,
}
err = syscall.Bind(listener.sock, listener.addr)
if err != nil {
return err
}
return listener.send(_PROC_CN_MCAST_LISTEN)
}
// Send an ignore control message to the connector driver
// and close our netlink socket.
func (listener *netlinkListener) close() error {
err := listener.send(_PROC_CN_MCAST_IGNORE)
syscall.Close(listener.sock)
return err
}
// Generic method for sending control messages to the connector
// driver; where op is one of PROC_CN_MCAST_{LISTEN,IGNORE}
func (listener *netlinkListener) send(op uint32) error {
listener.seq++
pr := &netlinkProcMessage{}
plen := binary.Size(pr.Data) + binary.Size(op)
pr.Header.Len = syscall.NLMSG_HDRLEN + uint32(plen)
pr.Header.Type = uint16(syscall.NLMSG_DONE)
pr.Header.Flags = 0
pr.Header.Seq = listener.seq
pr.Header.Pid = uint32(os.Getpid())
pr.Data.Id.Idx = _CN_IDX_PROC
pr.Data.Id.Val = _CN_VAL_PROC
pr.Data.Len = uint16(binary.Size(op))
buf := bytes.NewBuffer(make([]byte, 0, pr.Header.Len))
binary.Write(buf, byteOrder, pr)
binary.Write(buf, byteOrder, op)
return syscall.Sendto(listener.sock, buf.Bytes(), 0, listener.addr)
}

283
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/psnotify/psnotify_test.go generated vendored Normal file
View File

@ -0,0 +1,283 @@
// Copyright (c) 2012 VMware, Inc.
package psnotify
import (
"fmt"
"os"
"os/exec"
"runtime"
"syscall"
"testing"
"time"
)
type anyEvent struct {
exits []int
forks []int
execs []int
errors []error
done chan bool
}
type testWatcher struct {
t *testing.T
watcher *Watcher
events *anyEvent
}
// General purpose Watcher wrapper for all tests
func newTestWatcher(t *testing.T) *testWatcher {
watcher, err := NewWatcher()
if err != nil {
t.Fatal(err)
}
events := &anyEvent{
done: make(chan bool, 1),
}
tw := &testWatcher{
t: t,
watcher: watcher,
events: events,
}
go func() {
for {
select {
case <-events.done:
return
case ev := <-watcher.Fork:
events.forks = append(events.forks, ev.ParentPid)
case ev := <-watcher.Exec:
events.execs = append(events.execs, ev.Pid)
case ev := <-watcher.Exit:
events.exits = append(events.exits, ev.Pid)
case err := <-watcher.Error:
events.errors = append(events.errors, err)
}
}
}()
return tw
}
func (tw *testWatcher) close() {
pause := 100 * time.Millisecond
time.Sleep(pause)
tw.events.done <- true
tw.watcher.Close()
time.Sleep(pause)
}
func skipTest(t *testing.T) bool {
if runtime.GOOS == "linux" && os.Getuid() != 0 {
fmt.Println("SKIP: test must be run as root on linux")
return true
}
return false
}
func startSleepCommand(t *testing.T) *exec.Cmd {
cmd := exec.Command("sh", "-c", "sleep 100")
if err := cmd.Start(); err != nil {
t.Error(err)
}
return cmd
}
func runCommand(t *testing.T, name string) *exec.Cmd {
cmd := exec.Command(name)
if err := cmd.Run(); err != nil {
t.Error(err)
}
return cmd
}
func expectEvents(t *testing.T, num int, name string, pids []int) bool {
if len(pids) != num {
t.Errorf("Expected %d %s events, got=%v", num, name, pids)
return false
}
return true
}
func expectEventPid(t *testing.T, name string, expect int, pid int) bool {
if expect != pid {
t.Errorf("Expected %s pid=%d, received=%d", name, expect, pid)
return false
}
return true
}
func TestWatchFork(t *testing.T) {
if skipTest(t) {
return
}
pid := os.Getpid()
tw := newTestWatcher(t)
// no watches added yet, so this fork event will no be captured
runCommand(t, "date")
// watch fork events for this process
if err := tw.watcher.Watch(pid, PROC_EVENT_FORK); err != nil {
t.Error(err)
}
// this fork event will be captured,
// the exec and exit events will not be captured
runCommand(t, "cal")
tw.close()
if expectEvents(t, 1, "forks", tw.events.forks) {
expectEventPid(t, "fork", pid, tw.events.forks[0])
}
expectEvents(t, 0, "execs", tw.events.execs)
expectEvents(t, 0, "exits", tw.events.exits)
}
func TestWatchExit(t *testing.T) {
if skipTest(t) {
return
}
tw := newTestWatcher(t)
cmd := startSleepCommand(t)
childPid := cmd.Process.Pid
// watch for exit event of our child process
if err := tw.watcher.Watch(childPid, PROC_EVENT_EXIT); err != nil {
t.Error(err)
}
// kill our child process, triggers exit event
syscall.Kill(childPid, syscall.SIGTERM)
cmd.Wait()
tw.close()
expectEvents(t, 0, "forks", tw.events.forks)
expectEvents(t, 0, "execs", tw.events.execs)
if expectEvents(t, 1, "exits", tw.events.exits) {
expectEventPid(t, "exit", childPid, tw.events.exits[0])
}
}
// combined version of TestWatchFork() and TestWatchExit()
func TestWatchForkAndExit(t *testing.T) {
if skipTest(t) {
return
}
pid := os.Getpid()
tw := newTestWatcher(t)
if err := tw.watcher.Watch(pid, PROC_EVENT_FORK); err != nil {
t.Error(err)
}
cmd := startSleepCommand(t)
childPid := cmd.Process.Pid
if err := tw.watcher.Watch(childPid, PROC_EVENT_EXIT); err != nil {
t.Error(err)
}
syscall.Kill(childPid, syscall.SIGTERM)
cmd.Wait()
tw.close()
if expectEvents(t, 1, "forks", tw.events.forks) {
expectEventPid(t, "fork", pid, tw.events.forks[0])
}
expectEvents(t, 0, "execs", tw.events.execs)
if expectEvents(t, 1, "exits", tw.events.exits) {
expectEventPid(t, "exit", childPid, tw.events.exits[0])
}
}
func TestWatchFollowFork(t *testing.T) {
if skipTest(t) {
return
}
// Darwin is not able to follow forks, as the kqueue fork event
// does not provide the child pid.
if runtime.GOOS != "linux" {
fmt.Println("SKIP: test follow forks is linux only")
return
}
pid := os.Getpid()
tw := newTestWatcher(t)
// watch for all process events related to this process
if err := tw.watcher.Watch(pid, PROC_EVENT_ALL); err != nil {
t.Error(err)
}
commands := []string{"date", "cal"}
childPids := make([]int, len(commands))
// triggers fork/exec/exit events for each command
for i, name := range commands {
cmd := runCommand(t, name)
childPids[i] = cmd.Process.Pid
}
// remove watch for this process
tw.watcher.RemoveWatch(pid)
// run commands again to make sure we don't receive any unwanted events
for _, name := range commands {
runCommand(t, name)
}
tw.close()
// run commands again to make sure nothing panics after
// closing the watcher
for _, name := range commands {
runCommand(t, name)
}
num := len(commands)
if expectEvents(t, num, "forks", tw.events.forks) {
for _, epid := range tw.events.forks {
expectEventPid(t, "fork", pid, epid)
}
}
if expectEvents(t, num, "execs", tw.events.execs) {
for i, epid := range tw.events.execs {
expectEventPid(t, "exec", childPids[i], epid)
}
}
if expectEvents(t, num, "exits", tw.events.exits) {
for i, epid := range tw.events.exits {
expectEventPid(t, "exit", childPids[i], epid)
}
}
}

467
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_darwin.go generated vendored Normal file
View File

@ -0,0 +1,467 @@
// Copyright (c) 2012 VMware, Inc.
package sigar
/*
#include <stdlib.h>
#include <sys/sysctl.h>
#include <sys/mount.h>
#include <mach/mach_init.h>
#include <mach/mach_host.h>
#include <mach/host_info.h>
#include <libproc.h>
#include <mach/processor_info.h>
#include <mach/vm_map.h>
*/
import "C"
import (
"bytes"
"encoding/binary"
"fmt"
"io"
"syscall"
"time"
"unsafe"
)
func (self *LoadAverage) Get() error {
avg := []C.double{0, 0, 0}
C.getloadavg(&avg[0], C.int(len(avg)))
self.One = float64(avg[0])
self.Five = float64(avg[1])
self.Fifteen = float64(avg[2])
return nil
}
func (self *Uptime) Get() error {
tv := syscall.Timeval32{}
if err := sysctlbyname("kern.boottime", &tv); err != nil {
return err
}
self.Length = time.Since(time.Unix(int64(tv.Sec), int64(tv.Usec)*1000)).Seconds()
return nil
}
func (self *Mem) Get() error {
var vmstat C.vm_statistics_data_t
if err := sysctlbyname("hw.memsize", &self.Total); err != nil {
return err
}
if err := vm_info(&vmstat); err != nil {
return err
}
kern := uint64(vmstat.inactive_count) << 12
self.Free = uint64(vmstat.free_count) << 12
self.Used = self.Total - self.Free
self.ActualFree = self.Free + kern
self.ActualUsed = self.Used - kern
return nil
}
type xsw_usage struct {
Total, Avail, Used uint64
}
func (self *Swap) Get() error {
sw_usage := xsw_usage{}
if err := sysctlbyname("vm.swapusage", &sw_usage); err != nil {
return err
}
self.Total = sw_usage.Total
self.Used = sw_usage.Used
self.Free = sw_usage.Avail
return nil
}
func (self *Cpu) Get() error {
var count C.mach_msg_type_number_t = C.HOST_CPU_LOAD_INFO_COUNT
var cpuload C.host_cpu_load_info_data_t
status := C.host_statistics(C.host_t(C.mach_host_self()),
C.HOST_CPU_LOAD_INFO,
C.host_info_t(unsafe.Pointer(&cpuload)),
&count)
if status != C.KERN_SUCCESS {
return fmt.Errorf("host_statistics error=%d", status)
}
self.User = uint64(cpuload.cpu_ticks[C.CPU_STATE_USER])
self.Sys = uint64(cpuload.cpu_ticks[C.CPU_STATE_SYSTEM])
self.Idle = uint64(cpuload.cpu_ticks[C.CPU_STATE_IDLE])
self.Nice = uint64(cpuload.cpu_ticks[C.CPU_STATE_NICE])
return nil
}
func (self *CpuList) Get() error {
var count C.mach_msg_type_number_t
var cpuload *C.processor_cpu_load_info_data_t
var ncpu C.natural_t
status := C.host_processor_info(C.host_t(C.mach_host_self()),
C.PROCESSOR_CPU_LOAD_INFO,
&ncpu,
(*C.processor_info_array_t)(unsafe.Pointer(&cpuload)),
&count)
if status != C.KERN_SUCCESS {
return fmt.Errorf("host_processor_info error=%d", status)
}
// jump through some cgo casting hoops and ensure we properly free
// the memory that cpuload points to
target := C.vm_map_t(C.mach_task_self_)
address := C.vm_address_t(uintptr(unsafe.Pointer(cpuload)))
defer C.vm_deallocate(target, address, C.vm_size_t(ncpu))
// the body of struct processor_cpu_load_info
// aka processor_cpu_load_info_data_t
var cpu_ticks [C.CPU_STATE_MAX]uint32
// copy the cpuload array to a []byte buffer
// where we can binary.Read the data
size := int(ncpu) * binary.Size(cpu_ticks)
buf := C.GoBytes(unsafe.Pointer(cpuload), C.int(size))
bbuf := bytes.NewBuffer(buf)
self.List = make([]Cpu, 0, ncpu)
for i := 0; i < int(ncpu); i++ {
cpu := Cpu{}
err := binary.Read(bbuf, binary.LittleEndian, &cpu_ticks)
if err != nil {
return err
}
cpu.User = uint64(cpu_ticks[C.CPU_STATE_USER])
cpu.Sys = uint64(cpu_ticks[C.CPU_STATE_SYSTEM])
cpu.Idle = uint64(cpu_ticks[C.CPU_STATE_IDLE])
cpu.Nice = uint64(cpu_ticks[C.CPU_STATE_NICE])
self.List = append(self.List, cpu)
}
return nil
}
func (self *FileSystemList) Get() error {
num, err := getfsstat(nil, C.MNT_NOWAIT)
if num < 0 {
return err
}
buf := make([]syscall.Statfs_t, num)
num, err = getfsstat(buf, C.MNT_NOWAIT)
if err != nil {
return err
}
fslist := make([]FileSystem, 0, num)
for i := 0; i < num; i++ {
fs := FileSystem{}
fs.DirName = bytePtrToString(&buf[i].Mntonname[0])
fs.DevName = bytePtrToString(&buf[i].Mntfromname[0])
fs.SysTypeName = bytePtrToString(&buf[i].Fstypename[0])
fslist = append(fslist, fs)
}
self.List = fslist
return err
}
func (self *ProcList) Get() error {
n := C.proc_listpids(C.PROC_ALL_PIDS, 0, nil, 0)
if n <= 0 {
return syscall.EINVAL
}
buf := make([]byte, n)
n = C.proc_listpids(C.PROC_ALL_PIDS, 0, unsafe.Pointer(&buf[0]), n)
if n <= 0 {
return syscall.ENOMEM
}
var pid int32
num := int(n) / binary.Size(pid)
list := make([]int, 0, num)
bbuf := bytes.NewBuffer(buf)
for i := 0; i < num; i++ {
if err := binary.Read(bbuf, binary.LittleEndian, &pid); err != nil {
return err
}
if pid == 0 {
continue
}
list = append(list, int(pid))
}
self.List = list
return nil
}
func (self *ProcState) Get(pid int) error {
info := C.struct_proc_taskallinfo{}
if err := task_info(pid, &info); err != nil {
return err
}
self.Name = C.GoString(&info.pbsd.pbi_comm[0])
switch info.pbsd.pbi_status {
case C.SIDL:
self.State = RunStateIdle
case C.SRUN:
self.State = RunStateRun
case C.SSLEEP:
self.State = RunStateSleep
case C.SSTOP:
self.State = RunStateStop
case C.SZOMB:
self.State = RunStateZombie
default:
self.State = RunStateUnknown
}
self.Ppid = int(info.pbsd.pbi_ppid)
self.Tty = int(info.pbsd.e_tdev)
self.Priority = int(info.ptinfo.pti_priority)
self.Nice = int(info.pbsd.pbi_nice)
return nil
}
func (self *ProcMem) Get(pid int) error {
info := C.struct_proc_taskallinfo{}
if err := task_info(pid, &info); err != nil {
return err
}
self.Size = uint64(info.ptinfo.pti_virtual_size)
self.Resident = uint64(info.ptinfo.pti_resident_size)
self.PageFaults = uint64(info.ptinfo.pti_faults)
return nil
}
func (self *ProcTime) Get(pid int) error {
info := C.struct_proc_taskallinfo{}
if err := task_info(pid, &info); err != nil {
return err
}
self.User =
uint64(info.ptinfo.pti_total_user) / uint64(time.Millisecond)
self.Sys =
uint64(info.ptinfo.pti_total_system) / uint64(time.Millisecond)
self.Total = self.User + self.Sys
self.StartTime = (uint64(info.pbsd.pbi_start_tvsec) * 1000) +
(uint64(info.pbsd.pbi_start_tvusec) / 1000)
return nil
}
func (self *ProcArgs) Get(pid int) error {
var args []string
argv := func(arg string) {
args = append(args, arg)
}
err := kern_procargs(pid, nil, argv, nil)
self.List = args
return err
}
func (self *ProcExe) Get(pid int) error {
exe := func(arg string) {
self.Name = arg
}
return kern_procargs(pid, exe, nil, nil)
}
// wrapper around sysctl KERN_PROCARGS2
// callbacks params are optional,
// up to the caller as to which pieces of data they want
func kern_procargs(pid int,
exe func(string),
argv func(string),
env func(string, string)) error {
mib := []C.int{C.CTL_KERN, C.KERN_PROCARGS2, C.int(pid)}
argmax := uintptr(C.ARG_MAX)
buf := make([]byte, argmax)
err := sysctl(mib, &buf[0], &argmax, nil, 0)
if err != nil {
return nil
}
bbuf := bytes.NewBuffer(buf)
bbuf.Truncate(int(argmax))
var argc int32
binary.Read(bbuf, binary.LittleEndian, &argc)
path, err := bbuf.ReadBytes(0)
if exe != nil {
exe(string(chop(path)))
}
// skip trailing \0's
for {
c, _ := bbuf.ReadByte()
if c != 0 {
bbuf.UnreadByte()
break // start of argv[0]
}
}
for i := 0; i < int(argc); i++ {
arg, err := bbuf.ReadBytes(0)
if err == io.EOF {
break
}
if argv != nil {
argv(string(chop(arg)))
}
}
if env == nil {
return nil
}
delim := []byte{61} // "="
for {
line, err := bbuf.ReadBytes(0)
if err == io.EOF || line[0] == 0 {
break
}
pair := bytes.SplitN(chop(line), delim, 2)
env(string(pair[0]), string(pair[1]))
}
return nil
}
// XXX copied from zsyscall_darwin_amd64.go
func sysctl(mib []C.int, old *byte, oldlen *uintptr,
new *byte, newlen uintptr) (err error) {
var p0 unsafe.Pointer
p0 = unsafe.Pointer(&mib[0])
_, _, e1 := syscall.Syscall6(syscall.SYS___SYSCTL, uintptr(p0),
uintptr(len(mib)),
uintptr(unsafe.Pointer(old)), uintptr(unsafe.Pointer(oldlen)),
uintptr(unsafe.Pointer(new)), uintptr(newlen))
if e1 != 0 {
err = e1
}
return
}
func vm_info(vmstat *C.vm_statistics_data_t) error {
var count C.mach_msg_type_number_t = C.HOST_VM_INFO_COUNT
status := C.host_statistics(
C.host_t(C.mach_host_self()),
C.HOST_VM_INFO,
C.host_info_t(unsafe.Pointer(vmstat)),
&count)
if status != C.KERN_SUCCESS {
return fmt.Errorf("host_statistics=%d", status)
}
return nil
}
// generic Sysctl buffer unmarshalling
func sysctlbyname(name string, data interface{}) (err error) {
val, err := syscall.Sysctl(name)
if err != nil {
return err
}
buf := []byte(val)
switch v := data.(type) {
case *uint64:
*v = *(*uint64)(unsafe.Pointer(&buf[0]))
return
}
bbuf := bytes.NewBuffer([]byte(val))
return binary.Read(bbuf, binary.LittleEndian, data)
}
// syscall.Getfsstat() wrapper is broken, roll our own to workaround.
func getfsstat(buf []syscall.Statfs_t, flags int) (n int, err error) {
var ptr uintptr
var size uintptr
if len(buf) > 0 {
ptr = uintptr(unsafe.Pointer(&buf[0]))
size = unsafe.Sizeof(buf[0]) * uintptr(len(buf))
} else {
ptr = uintptr(0)
size = uintptr(0)
}
trap := uintptr(syscall.SYS_GETFSSTAT64)
ret, _, errno := syscall.Syscall(trap, ptr, size, uintptr(flags))
n = int(ret)
if errno != 0 {
err = errno
}
return
}
func task_info(pid int, info *C.struct_proc_taskallinfo) error {
size := C.int(unsafe.Sizeof(*info))
ptr := unsafe.Pointer(info)
n := C.proc_pidinfo(C.int(pid), C.PROC_PIDTASKALLINFO, 0, ptr, size)
if n != size {
return syscall.ENOMEM
}
return nil
}

126
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_format.go generated vendored Normal file
View File

@ -0,0 +1,126 @@
// Copyright (c) 2012 VMware, Inc.
package sigar
import (
"bufio"
"bytes"
"fmt"
"strconv"
"time"
)
// Go version of apr_strfsize
func FormatSize(size uint64) string {
ord := []string{"K", "M", "G", "T", "P", "E"}
o := 0
buf := new(bytes.Buffer)
w := bufio.NewWriter(buf)
if size < 973 {
fmt.Fprintf(w, "%3d ", size)
w.Flush()
return buf.String()
}
for {
remain := size & 1023
size >>= 10
if size >= 973 {
o++
continue
}
if size < 9 || (size == 9 && remain < 973) {
remain = ((remain * 5) + 256) / 512
if remain >= 10 {
size++
remain = 0
}
fmt.Fprintf(w, "%d.%d%s", size, remain, ord[o])
break
}
if remain >= 512 {
size++
}
fmt.Fprintf(w, "%3d%s", size, ord[o])
break
}
w.Flush()
return buf.String()
}
func FormatPercent(percent float64) string {
return strconv.FormatFloat(percent, 'f', -1, 64) + "%"
}
func (self *FileSystemUsage) UsePercent() float64 {
b_used := (self.Total - self.Free) / 1024
b_avail := self.Avail / 1024
utotal := b_used + b_avail
used := b_used
if utotal != 0 {
u100 := used * 100
pct := u100 / utotal
if u100%utotal != 0 {
pct += 1
}
return (float64(pct) / float64(100)) * 100.0
}
return 0.0
}
func (self *Uptime) Format() string {
buf := new(bytes.Buffer)
w := bufio.NewWriter(buf)
uptime := uint64(self.Length)
days := uptime / (60 * 60 * 24)
if days != 0 {
s := ""
if days > 1 {
s = "s"
}
fmt.Fprintf(w, "%d day%s, ", days, s)
}
minutes := uptime / 60
hours := minutes / 60
hours %= 24
minutes %= 60
fmt.Fprintf(w, "%2d:%02d", hours, minutes)
w.Flush()
return buf.String()
}
func (self *ProcTime) FormatStartTime() string {
if self.StartTime == 0 {
return "00:00"
}
start := time.Unix(int64(self.StartTime)/1000, 0)
format := "Jan02"
if time.Since(start).Seconds() < (60 * 60 * 24) {
format = "15:04"
}
return start.Format(format)
}
func (self *ProcTime) FormatTotal() string {
t := self.Total / 1000
ss := t % 60
t /= 60
mm := t % 60
t /= 60
hh := t % 24
return fmt.Sprintf("%02d:%02d:%02d", hh, mm, ss)
}

141
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_interface.go generated vendored Normal file
View File

@ -0,0 +1,141 @@
package sigar
import (
"time"
)
type Sigar interface {
CollectCpuStats(collectionInterval time.Duration) (<-chan Cpu, chan<- struct{})
GetLoadAverage() (LoadAverage, error)
GetMem() (Mem, error)
GetSwap() (Swap, error)
GetFileSystemUsage(string) (FileSystemUsage, error)
}
type Cpu struct {
User uint64
Nice uint64
Sys uint64
Idle uint64
Wait uint64
Irq uint64
SoftIrq uint64
Stolen uint64
}
func (cpu *Cpu) Total() uint64 {
return cpu.User + cpu.Nice + cpu.Sys + cpu.Idle +
cpu.Wait + cpu.Irq + cpu.SoftIrq + cpu.Stolen
}
func (cpu Cpu) Delta(other Cpu) Cpu {
return Cpu{
User: cpu.User - other.User,
Nice: cpu.Nice - other.Nice,
Sys: cpu.Sys - other.Sys,
Idle: cpu.Idle - other.Idle,
Wait: cpu.Wait - other.Wait,
Irq: cpu.Irq - other.Irq,
SoftIrq: cpu.SoftIrq - other.SoftIrq,
Stolen: cpu.Stolen - other.Stolen,
}
}
type LoadAverage struct {
One, Five, Fifteen float64
}
type Uptime struct {
Length float64
}
type Mem struct {
Total uint64
Used uint64
Free uint64
ActualFree uint64
ActualUsed uint64
}
type Swap struct {
Total uint64
Used uint64
Free uint64
}
type CpuList struct {
List []Cpu
}
type FileSystem struct {
DirName string
DevName string
TypeName string
SysTypeName string
Options string
Flags uint32
}
type FileSystemList struct {
List []FileSystem
}
type FileSystemUsage struct {
Total uint64
Used uint64
Free uint64
Avail uint64
Files uint64
FreeFiles uint64
}
type ProcList struct {
List []int
}
type RunState byte
const (
RunStateSleep = 'S'
RunStateRun = 'R'
RunStateStop = 'T'
RunStateZombie = 'Z'
RunStateIdle = 'D'
RunStateUnknown = '?'
)
type ProcState struct {
Name string
State RunState
Ppid int
Tty int
Priority int
Nice int
Processor int
}
type ProcMem struct {
Size uint64
Resident uint64
Share uint64
MinorFaults uint64
MajorFaults uint64
PageFaults uint64
}
type ProcTime struct {
StartTime uint64
User uint64
Sys uint64
Total uint64
}
type ProcArgs struct {
List []string
}
type ProcExe struct {
Name string
Cwd string
Root string
}

135
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_interface_test.go generated vendored Normal file
View File

@ -0,0 +1,135 @@
package sigar_test
import (
"os"
"path/filepath"
"runtime"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
. "github.com/cloudfoundry/gosigar"
)
var _ = Describe("Sigar", func() {
var invalidPid = 666666
It("cpu", func() {
cpu := Cpu{}
err := cpu.Get()
Expect(err).ToNot(HaveOccurred())
})
It("load average", func() {
avg := LoadAverage{}
err := avg.Get()
Expect(err).ToNot(HaveOccurred())
})
It("uptime", func() {
uptime := Uptime{}
err := uptime.Get()
Expect(err).ToNot(HaveOccurred())
Expect(uptime.Length).To(BeNumerically(">", 0))
})
It("mem", func() {
mem := Mem{}
err := mem.Get()
Expect(err).ToNot(HaveOccurred())
Expect(mem.Total).To(BeNumerically(">", 0))
Expect(mem.Used + mem.Free).To(BeNumerically("<=", mem.Total))
})
It("swap", func() {
swap := Swap{}
err := swap.Get()
Expect(err).ToNot(HaveOccurred())
Expect(swap.Used + swap.Free).To(BeNumerically("<=", swap.Total))
})
It("cpu list", func() {
cpulist := CpuList{}
err := cpulist.Get()
Expect(err).ToNot(HaveOccurred())
nsigar := len(cpulist.List)
numcpu := runtime.NumCPU()
Expect(nsigar).To(Equal(numcpu))
})
It("file system list", func() {
fslist := FileSystemList{}
err := fslist.Get()
Expect(err).ToNot(HaveOccurred())
Expect(len(fslist.List)).To(BeNumerically(">", 0))
})
It("file system usage", func() {
fsusage := FileSystemUsage{}
err := fsusage.Get("/")
Expect(err).ToNot(HaveOccurred())
err = fsusage.Get("T O T A L L Y B O G U S")
Expect(err).To(HaveOccurred())
})
It("proc list", func() {
pids := ProcList{}
err := pids.Get()
Expect(err).ToNot(HaveOccurred())
Expect(len(pids.List)).To(BeNumerically(">", 2))
err = pids.Get()
Expect(err).ToNot(HaveOccurred())
})
It("proc state", func() {
state := ProcState{}
err := state.Get(os.Getppid())
Expect(err).ToNot(HaveOccurred())
Expect([]RunState{RunStateRun, RunStateSleep}).To(ContainElement(state.State))
Expect([]string{"go", "ginkgo"}).To(ContainElement(state.Name))
err = state.Get(invalidPid)
Expect(err).To(HaveOccurred())
})
It("proc mem", func() {
mem := ProcMem{}
err := mem.Get(os.Getppid())
Expect(err).ToNot(HaveOccurred())
err = mem.Get(invalidPid)
Expect(err).To(HaveOccurred())
})
It("proc time", func() {
time := ProcTime{}
err := time.Get(os.Getppid())
Expect(err).ToNot(HaveOccurred())
err = time.Get(invalidPid)
Expect(err).To(HaveOccurred())
})
It("proc args", func() {
args := ProcArgs{}
err := args.Get(os.Getppid())
Expect(err).ToNot(HaveOccurred())
Expect(len(args.List)).To(BeNumerically(">=", 2))
})
It("proc exe", func() {
exe := ProcExe{}
err := exe.Get(os.Getppid())
Expect(err).ToNot(HaveOccurred())
Expect([]string{"go", "ginkgo"}).To(ContainElement(filepath.Base(exe.Name)))
})
})

386
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_linux.go generated vendored Normal file
View File

@ -0,0 +1,386 @@
// Copyright (c) 2012 VMware, Inc.
package sigar
import (
"bufio"
"bytes"
"io"
"io/ioutil"
"os"
"strconv"
"strings"
"syscall"
)
var system struct {
ticks uint64
btime uint64
}
var Procd string
func init() {
system.ticks = 100 // C.sysconf(C._SC_CLK_TCK)
Procd = "/proc"
// grab system boot time
readFile(Procd+"/stat", func(line string) bool {
if strings.HasPrefix(line, "btime") {
system.btime, _ = strtoull(line[6:])
return false // stop reading
}
return true
})
}
func (self *LoadAverage) Get() error {
line, err := ioutil.ReadFile(Procd + "/loadavg")
if err != nil {
return nil
}
fields := strings.Fields(string(line))
self.One, _ = strconv.ParseFloat(fields[0], 64)
self.Five, _ = strconv.ParseFloat(fields[1], 64)
self.Fifteen, _ = strconv.ParseFloat(fields[2], 64)
return nil
}
func (self *Uptime) Get() error {
sysinfo := syscall.Sysinfo_t{}
if err := syscall.Sysinfo(&sysinfo); err != nil {
return err
}
self.Length = float64(sysinfo.Uptime)
return nil
}
func (self *Mem) Get() error {
var buffers, cached uint64
table := map[string]*uint64{
"MemTotal": &self.Total,
"MemFree": &self.Free,
"Buffers": &buffers,
"Cached": &cached,
}
if err := parseMeminfo(table); err != nil {
return err
}
self.Used = self.Total - self.Free
kern := buffers + cached
self.ActualFree = self.Free + kern
self.ActualUsed = self.Used - kern
return nil
}
func (self *Swap) Get() error {
table := map[string]*uint64{
"SwapTotal": &self.Total,
"SwapFree": &self.Free,
}
if err := parseMeminfo(table); err != nil {
return err
}
self.Used = self.Total - self.Free
return nil
}
func (self *Cpu) Get() error {
return readFile(Procd+"/stat", func(line string) bool {
if len(line) > 4 && line[0:4] == "cpu " {
parseCpuStat(self, line)
return false
}
return true
})
}
func (self *CpuList) Get() error {
capacity := len(self.List)
if capacity == 0 {
capacity = 4
}
list := make([]Cpu, 0, capacity)
err := readFile(Procd+"/stat", func(line string) bool {
if len(line) > 3 && line[0:3] == "cpu" && line[3] != ' ' {
cpu := Cpu{}
parseCpuStat(&cpu, line)
list = append(list, cpu)
}
return true
})
self.List = list
return err
}
func (self *FileSystemList) Get() error {
capacity := len(self.List)
if capacity == 0 {
capacity = 10
}
fslist := make([]FileSystem, 0, capacity)
err := readFile("/etc/mtab", func(line string) bool {
fields := strings.Fields(line)
fs := FileSystem{}
fs.DevName = fields[0]
fs.DirName = fields[1]
fs.SysTypeName = fields[2]
fs.Options = fields[3]
fslist = append(fslist, fs)
return true
})
self.List = fslist
return err
}
func (self *ProcList) Get() error {
dir, err := os.Open(Procd)
if err != nil {
return err
}
defer dir.Close()
const readAllDirnames = -1 // see os.File.Readdirnames doc
names, err := dir.Readdirnames(readAllDirnames)
if err != nil {
return err
}
capacity := len(names)
list := make([]int, 0, capacity)
for _, name := range names {
if name[0] < '0' || name[0] > '9' {
continue
}
pid, err := strconv.Atoi(name)
if err == nil {
list = append(list, pid)
}
}
self.List = list
return nil
}
func (self *ProcState) Get(pid int) error {
contents, err := readProcFile(pid, "stat")
if err != nil {
return err
}
fields := strings.Fields(string(contents))
self.Name = fields[1][1 : len(fields[1])-1] // strip ()'s
self.State = RunState(fields[2][0])
self.Ppid, _ = strconv.Atoi(fields[3])
self.Tty, _ = strconv.Atoi(fields[6])
self.Priority, _ = strconv.Atoi(fields[17])
self.Nice, _ = strconv.Atoi(fields[18])
self.Processor, _ = strconv.Atoi(fields[38])
return nil
}
func (self *ProcMem) Get(pid int) error {
contents, err := readProcFile(pid, "statm")
if err != nil {
return err
}
fields := strings.Fields(string(contents))
size, _ := strtoull(fields[0])
self.Size = size << 12
rss, _ := strtoull(fields[1])
self.Resident = rss << 12
share, _ := strtoull(fields[2])
self.Share = share << 12
contents, err = readProcFile(pid, "stat")
if err != nil {
return err
}
fields = strings.Fields(string(contents))
self.MinorFaults, _ = strtoull(fields[10])
self.MajorFaults, _ = strtoull(fields[12])
self.PageFaults = self.MinorFaults + self.MajorFaults
return nil
}
func (self *ProcTime) Get(pid int) error {
contents, err := readProcFile(pid, "stat")
if err != nil {
return err
}
fields := strings.Fields(string(contents))
user, _ := strtoull(fields[13])
sys, _ := strtoull(fields[14])
// convert to millis
self.User = user * (1000 / system.ticks)
self.Sys = sys * (1000 / system.ticks)
self.Total = self.User + self.Sys
// convert to millis
self.StartTime, _ = strtoull(fields[21])
self.StartTime /= system.ticks
self.StartTime += system.btime
self.StartTime *= 1000
return nil
}
func (self *ProcArgs) Get(pid int) error {
contents, err := readProcFile(pid, "cmdline")
if err != nil {
return err
}
bbuf := bytes.NewBuffer(contents)
var args []string
for {
arg, err := bbuf.ReadBytes(0)
if err == io.EOF {
break
}
args = append(args, string(chop(arg)))
}
self.List = args
return nil
}
func (self *ProcExe) Get(pid int) error {
fields := map[string]*string{
"exe": &self.Name,
"cwd": &self.Cwd,
"root": &self.Root,
}
for name, field := range fields {
val, err := os.Readlink(procFileName(pid, name))
if err != nil {
return err
}
*field = val
}
return nil
}
func parseMeminfo(table map[string]*uint64) error {
return readFile(Procd+"/meminfo", func(line string) bool {
fields := strings.Split(line, ":")
if ptr := table[fields[0]]; ptr != nil {
num := strings.TrimLeft(fields[1], " ")
val, err := strtoull(strings.Fields(num)[0])
if err == nil {
*ptr = val * 1024
}
}
return true
})
}
func parseCpuStat(self *Cpu, line string) error {
fields := strings.Fields(line)
self.User, _ = strtoull(fields[1])
self.Nice, _ = strtoull(fields[2])
self.Sys, _ = strtoull(fields[3])
self.Idle, _ = strtoull(fields[4])
self.Wait, _ = strtoull(fields[5])
self.Irq, _ = strtoull(fields[6])
self.SoftIrq, _ = strtoull(fields[7])
self.Stolen, _ = strtoull(fields[8])
return nil
}
func readFile(file string, handler func(string) bool) error {
contents, err := ioutil.ReadFile(file)
if err != nil {
return err
}
reader := bufio.NewReader(bytes.NewBuffer(contents))
for {
line, _, err := reader.ReadLine()
if err == io.EOF {
break
}
if !handler(string(line)) {
break
}
}
return nil
}
func strtoull(val string) (uint64, error) {
return strconv.ParseUint(val, 10, 64)
}
func procFileName(pid int, name string) string {
return Procd + "/" + strconv.Itoa(pid) + "/" + name
}
func readProcFile(pid int, name string) ([]byte, error) {
path := procFileName(pid, name)
contents, err := ioutil.ReadFile(path)
if err != nil {
if perr, ok := err.(*os.PathError); ok {
if perr.Err == syscall.ENOENT {
return nil, syscall.ESRCH
}
}
}
return contents, err
}

225
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_linux_test.go generated vendored Normal file
View File

@ -0,0 +1,225 @@
package sigar_test
import (
"io/ioutil"
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
sigar "github.com/cloudfoundry/gosigar"
)
var _ = Describe("sigarLinux", func() {
var procd string
BeforeEach(func() {
var err error
procd, err = ioutil.TempDir("", "sigarTests")
Expect(err).ToNot(HaveOccurred())
sigar.Procd = procd
})
AfterEach(func() {
sigar.Procd = "/proc"
})
Describe("CPU", func() {
var (
statFile string
cpu sigar.Cpu
)
BeforeEach(func() {
statFile = procd + "/stat"
cpu = sigar.Cpu{}
})
Describe("Get", func() {
It("gets CPU usage", func() {
statContents := []byte("cpu 25 1 2 3 4 5 6 7")
err := ioutil.WriteFile(statFile, statContents, 0644)
Expect(err).ToNot(HaveOccurred())
err = cpu.Get()
Expect(err).ToNot(HaveOccurred())
Expect(cpu.User).To(Equal(uint64(25)))
})
It("ignores empty lines", func() {
statContents := []byte("cpu ")
err := ioutil.WriteFile(statFile, statContents, 0644)
Expect(err).ToNot(HaveOccurred())
err = cpu.Get()
Expect(err).ToNot(HaveOccurred())
Expect(cpu.User).To(Equal(uint64(0)))
})
})
Describe("CollectCpuStats", func() {
It("collects CPU usage over time", func() {
statContents := []byte("cpu 25 1 2 3 4 5 6 7")
err := ioutil.WriteFile(statFile, statContents, 0644)
Expect(err).ToNot(HaveOccurred())
concreteSigar := &sigar.ConcreteSigar{}
cpuUsages, stop := concreteSigar.CollectCpuStats(500 * time.Millisecond)
Expect(<-cpuUsages).To(Equal(sigar.Cpu{
User: uint64(25),
Nice: uint64(1),
Sys: uint64(2),
Idle: uint64(3),
Wait: uint64(4),
Irq: uint64(5),
SoftIrq: uint64(6),
Stolen: uint64(7),
}))
statContents = []byte("cpu 30 3 7 10 25 55 36 65")
err = ioutil.WriteFile(statFile, statContents, 0644)
Expect(err).ToNot(HaveOccurred())
Expect(<-cpuUsages).To(Equal(sigar.Cpu{
User: uint64(5),
Nice: uint64(2),
Sys: uint64(5),
Idle: uint64(7),
Wait: uint64(21),
Irq: uint64(50),
SoftIrq: uint64(30),
Stolen: uint64(58),
}))
stop <- struct{}{}
})
})
})
Describe("Mem", func() {
var meminfoFile string
BeforeEach(func() {
meminfoFile = procd + "/meminfo"
meminfoContents := `
MemTotal: 374256 kB
MemFree: 274460 kB
Buffers: 9764 kB
Cached: 38648 kB
SwapCached: 0 kB
Active: 33772 kB
Inactive: 31184 kB
Active(anon): 16572 kB
Inactive(anon): 552 kB
Active(file): 17200 kB
Inactive(file): 30632 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 786428 kB
SwapFree: 786428 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 16564 kB
Mapped: 6612 kB
Shmem: 584 kB
Slab: 19092 kB
SReclaimable: 9128 kB
SUnreclaim: 9964 kB
KernelStack: 672 kB
PageTables: 1864 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 973556 kB
Committed_AS: 55880 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 21428 kB
VmallocChunk: 34359713596 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 59328 kB
DirectMap2M: 333824 kB
`
err := ioutil.WriteFile(meminfoFile, []byte(meminfoContents), 0444)
Expect(err).ToNot(HaveOccurred())
})
It("returns correct memory info", func() {
mem := sigar.Mem{}
err := mem.Get()
Expect(err).ToNot(HaveOccurred())
Expect(mem.Total).To(BeNumerically("==", 374256*1024))
Expect(mem.Free).To(BeNumerically("==", 274460*1024))
})
})
Describe("Swap", func() {
var meminfoFile string
BeforeEach(func() {
meminfoFile = procd + "/meminfo"
meminfoContents := `
MemTotal: 374256 kB
MemFree: 274460 kB
Buffers: 9764 kB
Cached: 38648 kB
SwapCached: 0 kB
Active: 33772 kB
Inactive: 31184 kB
Active(anon): 16572 kB
Inactive(anon): 552 kB
Active(file): 17200 kB
Inactive(file): 30632 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 786428 kB
SwapFree: 786428 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 16564 kB
Mapped: 6612 kB
Shmem: 584 kB
Slab: 19092 kB
SReclaimable: 9128 kB
SUnreclaim: 9964 kB
KernelStack: 672 kB
PageTables: 1864 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 973556 kB
Committed_AS: 55880 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 21428 kB
VmallocChunk: 34359713596 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 59328 kB
DirectMap2M: 333824 kB
`
err := ioutil.WriteFile(meminfoFile, []byte(meminfoContents), 0444)
Expect(err).ToNot(HaveOccurred())
})
It("returns correct memory info", func() {
swap := sigar.Swap{}
err := swap.Get()
Expect(err).ToNot(HaveOccurred())
Expect(swap.Total).To(BeNumerically("==", 786428*1024))
Expect(swap.Free).To(BeNumerically("==", 786428*1024))
})
})
})

13
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_suite_test.go generated vendored Normal file
View File

@ -0,0 +1,13 @@
package sigar_test
import (
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"testing"
)
func TestGosigar(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Gosigar Suite")
}

26
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_unix.go generated vendored Normal file
View File

@ -0,0 +1,26 @@
// Copyright (c) 2012 VMware, Inc.
// +build darwin freebsd linux netbsd openbsd
package sigar
import "syscall"
func (self *FileSystemUsage) Get(path string) error {
stat := syscall.Statfs_t{}
err := syscall.Statfs(path, &stat)
if err != nil {
return err
}
bsize := stat.Bsize / 512
self.Total = (uint64(stat.Blocks) * uint64(bsize)) >> 1
self.Free = (uint64(stat.Bfree) * uint64(bsize)) >> 1
self.Avail = (uint64(stat.Bavail) * uint64(bsize)) >> 1
self.Used = self.Total - self.Free
self.Files = stat.Files
self.FreeFiles = stat.Ffree
return nil
}

22
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_util.go generated vendored Normal file
View File

@ -0,0 +1,22 @@
// Copyright (c) 2012 VMware, Inc.
package sigar
import (
"unsafe"
)
func bytePtrToString(ptr *int8) string {
bytes := (*[10000]byte)(unsafe.Pointer(ptr))
n := 0
for bytes[n] != 0 {
n++
}
return string(bytes[0:n])
}
func chop(buf []byte) []byte {
return buf[0 : len(buf)-1]
}

100
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_windows.go generated vendored Normal file
View File

@ -0,0 +1,100 @@
// Copyright (c) 2012 VMware, Inc.
package sigar
// #include <stdlib.h>
// #include <windows.h>
import "C"
import (
"fmt"
"unsafe"
)
func init() {
}
func (self *LoadAverage) Get() error {
return nil
}
func (self *Uptime) Get() error {
return nil
}
func (self *Mem) Get() error {
var statex C.MEMORYSTATUSEX
statex.dwLength = C.DWORD(unsafe.Sizeof(statex))
succeeded := C.GlobalMemoryStatusEx(&statex)
if succeeded == C.FALSE {
lastError := C.GetLastError()
return fmt.Errorf("GlobalMemoryStatusEx failed with error: %d", int(lastError))
}
self.Total = uint64(statex.ullTotalPhys)
return nil
}
func (self *Swap) Get() error {
return notImplemented()
}
func (self *Cpu) Get() error {
return notImplemented()
}
func (self *CpuList) Get() error {
return notImplemented()
}
func (self *FileSystemList) Get() error {
return notImplemented()
}
func (self *ProcList) Get() error {
return notImplemented()
}
func (self *ProcState) Get(pid int) error {
return notImplemented()
}
func (self *ProcMem) Get(pid int) error {
return notImplemented()
}
func (self *ProcTime) Get(pid int) error {
return notImplemented()
}
func (self *ProcArgs) Get(pid int) error {
return notImplemented()
}
func (self *ProcExe) Get(pid int) error {
return notImplemented()
}
func (self *FileSystemUsage) Get(path string) error {
var availableBytes C.ULARGE_INTEGER
var totalBytes C.ULARGE_INTEGER
var totalFreeBytes C.ULARGE_INTEGER
pathChars := C.CString(path)
defer C.free(unsafe.Pointer(pathChars))
succeeded := C.GetDiskFreeSpaceEx((*C.CHAR)(pathChars), &availableBytes, &totalBytes, &totalFreeBytes)
if succeeded == C.FALSE {
lastError := C.GetLastError()
return fmt.Errorf("GetDiskFreeSpaceEx failed with error: %d", int(lastError))
}
self.Total = *(*uint64)(unsafe.Pointer(&totalBytes))
return nil
}
func notImplemented() error {
panic("Not Implemented")
return nil
}

32
Godeps/_workspace/src/github.com/cloudfoundry/gosigar/sigar_windows_test.go generated vendored Normal file
View File

@ -0,0 +1,32 @@
package sigar_test
import (
"os"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
sigar "github.com/cloudfoundry/gosigar"
)
var _ = Describe("SigarWindows", func() {
Describe("Memory", func() {
It("gets the total memory", func() {
mem := sigar.Mem{}
err := mem.Get()
Ω(err).ShouldNot(HaveOccurred())
Ω(mem.Total).Should(BeNumerically(">", 0))
})
})
Describe("Disk", func() {
It("gets the total disk space", func() {
usage := sigar.FileSystemUsage{}
err := usage.Get(os.TempDir())
Ω(err).ShouldNot(HaveOccurred())
Ω(usage.Total).Should(BeNumerically(">", 0))
})
})
})

2
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/.gitignore generated vendored Normal file
View File

@ -0,0 +1,2 @@
# temporary symlink for testing
testing/data/symlink

5
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/.travis.yml generated vendored
View File

@ -1,8 +1,9 @@
language: go
sudo: false
go:
- 1.3.1
- 1.4
- 1.3.3
- 1.4.2
- 1.5.1
- tip
env:
- GOARCH=amd64

8
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/AUTHORS generated vendored
View File

@ -11,8 +11,10 @@ Ben McCann <benmccann.com>
Brendan Fosberry <brendan@codeship.com>
Brian Lalor <blalor@bravo5.org>
Brian Palmer <brianp@instructure.com>
Bryan Boreham <bjboreham@gmail.com>
Burke Libbey <burke@libbey.me>
Carlos Diaz-Padron <cpadron@mozilla.com>
Cesar Wong <cewong@redhat.com>
Cezar Sa Espinola <cezar.sa@corp.globo.com>
Cheah Chu Yeow <chuyeow@gmail.com>
cheneydeng <cheneydeng@qq.com>
@ -27,12 +29,14 @@ David Huie <dahuie@gmail.com>
Dawn Chen <dawnchen@google.com>
Dinesh Subhraveti <dinesh@gemini-systems.net>
Ed <edrocksit@gmail.com>
Erez Horev <erez.horev@elastifile.com>
Eric Anderson <anderson@copperegg.com>
Ewout Prangsma <ewout@prangsma.net>
Fabio Rehm <fgrehm@gmail.com>
Fatih Arslan <ftharsln@gmail.com>
Flavia Missi <flaviamissi@gmail.com>
Francisco Souza <f@souza.cc>
Grégoire Delattre <gregoire.delattre@gmail.com>
Guillermo Álvarez Fernández <guillermo@cientifico.net>
He Simei <hesimei@zju.edu.cn>
Ivan Mikushin <i.mikushin@gmail.com>
@ -43,6 +47,7 @@ Jawher Moussa <jawher.moussa@gmail.com>
Jean-Baptiste Dalido <jeanbaptiste@appgratis.com>
Jeff Mitchell <jeffrey.mitchell@gmail.com>
Jeffrey Hulten <jhulten@gmail.com>
Jen Andre <jandre@gmail.com>
Johan Euphrosine <proppy@google.com>
Kamil Domanski <kamil@domanski.co>
Karan Misra <kidoman@gmail.com>
@ -50,6 +55,7 @@ Kim, Hirokuni <hirokuni.kim@kvh.co.jp>
Kyle Allan <kallan357@gmail.com>
Liron Levin <levinlir@gmail.com>
Liu Peng <vslene@gmail.com>
Lorenz Leutgeb <lorenz.leutgeb@gmail.com>
Lucas Clemente <lucas@clemente.io>
Lucas Weiblen <lucasweiblen@gmail.com>
Mantas Matelis <mmatelis@coursera.org>
@ -66,12 +72,14 @@ Paul Morie <pmorie@gmail.com>
Paul Weil <pweil@redhat.com>
Peter Edge <peter.edge@gmail.com>
Peter Jihoon Kim <raingrove@gmail.com>
Phil Lu <lu@stackengine.com>
Philippe Lafoucrière <philippe.lafoucriere@tech-angels.com>
Rafe Colton <rafael.colton@gmail.com>
Rob Miller <rob@kalistra.com>
Robert Williamson <williamson.robert@gmail.com>
Salvador Gironès <salvadorgirones@gmail.com>
Sam Rijs <srijs@airpost.net>
Samuel Karp <skarp@amazon.com>
Simon Eskildsen <sirup@sirupsen.com>
Simon Menke <simon.menke@gmail.com>
Skolos <skolos@gopherlab.com>

17
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/auth.go generated vendored
View File

@ -16,7 +16,8 @@ import (
"strings"
)
var AuthParseError error = errors.New("Failed to read authentication from dockercfg")
// ErrCannotParseDockercfg is the error returned by NewAuthConfigurations when the dockercfg cannot be parsed.
var ErrCannotParseDockercfg = errors.New("Failed to read authentication from dockercfg")
// AuthConfiguration represents authentication options to use in the PushImage
// method. It represents the authentication in the Docker index server.
@ -33,6 +34,10 @@ type AuthConfigurations struct {
Configs map[string]AuthConfiguration `json:"configs"`
}
// AuthConfigurations119 is used to serialize a set of AuthConfigurations
// for Docker API >= 1.19.
type AuthConfigurations119 map[string]AuthConfiguration
// dockerConfig represents a registry authentation configuration from the
// .dockercfg file.
type dockerConfig struct {
@ -103,7 +108,7 @@ func authConfigs(confs map[string]dockerConfig) (*AuthConfigurations, error) {
}
userpass := strings.Split(string(data), ":")
if len(userpass) != 2 {
return nil, AuthParseError
return nil, ErrCannotParseDockercfg
}
c.Configs[reg] = AuthConfiguration{
Email: conf.Email,
@ -117,17 +122,15 @@ func authConfigs(confs map[string]dockerConfig) (*AuthConfigurations, error) {
// AuthCheck validates the given credentials. It returns nil if successful.
//
// See https://goo.gl/vPoEfJ for more details.
// See https://goo.gl/m2SleN for more details.
func (c *Client) AuthCheck(conf *AuthConfiguration) error {
if conf == nil {
return fmt.Errorf("conf is nil")
}
body, statusCode, err := c.do("POST", "/auth", doOptions{data: conf})
resp, err := c.do("POST", "/auth", doOptions{data: conf})
if err != nil {
return err
}
if statusCode > 400 {
return fmt.Errorf("auth error (%d): %s", statusCode, body)
}
resp.Body.Close()
return nil
}

2
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/auth_test.go generated vendored
View File

@ -41,7 +41,7 @@ func TestAuthBadConfig(t *testing.T) {
auth := base64.StdEncoding.EncodeToString([]byte("userpass"))
read := strings.NewReader(fmt.Sprintf(`{"docker.io":{"auth":"%s","email":"user@example.com"}}`, auth))
ac, err := NewAuthConfigurations(read)
if err != AuthParseError {
if err != ErrCannotParseDockercfg {
t.Errorf("Incorrect error returned %v\n", err)
}
if ac != nil {

10
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/build_test.go generated vendored
View File

@ -35,6 +35,16 @@ func TestBuildImageMultipleContextsError(t *testing.T) {
func TestBuildImageContextDirDockerignoreParsing(t *testing.T) {
fakeRT := &FakeRoundTripper{message: "", status: http.StatusOK}
client := newTestClient(fakeRT)
if err := os.Symlink("doesnotexist", "testing/data/symlink"); err != nil {
t.Errorf("error creating symlink on demand: %s", err)
}
defer func() {
if err := os.Remove("testing/data/symlink"); err != nil {
t.Errorf("error removing symlink on demand: %s", err)
}
}()
var buf bytes.Buffer
opts := BuildImageOptions{
Name: "testImage",

2
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/change.go generated vendored
View File

@ -23,7 +23,7 @@ const (
// Change represents a change in a container.
//
// See http://goo.gl/QkW9sH for more details.
// See https://goo.gl/9GsTIF for more details.
type Change struct {
Path string
Kind ChangeType

171
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/client.go generated vendored
View File

@ -4,7 +4,7 @@
// Package docker provides a client for the Docker remote API.
//
// See http://goo.gl/G3plxW for more details on the remote API.
// See https://goo.gl/G3plxW for more details on the remote API.
package docker
import (
@ -45,6 +45,8 @@ var (
ErrConnectionRefused = errors.New("cannot connect to Docker endpoint")
apiVersion112, _ = NewAPIVersion("1.12")
apiVersion119, _ = NewAPIVersion("1.19")
)
// APIVersion is an internal representation of a version of the Remote API.
@ -128,6 +130,7 @@ type Client struct {
SkipServerVersionCheck bool
HTTPClient *http.Client
TLSConfig *tls.Config
Dialer *net.Dialer
endpoint string
endpointURL *url.URL
@ -135,6 +138,7 @@ type Client struct {
requestedAPIVersion APIVersion
serverAPIVersion APIVersion
expectedAPIVersion APIVersion
unixHTTPClient *http.Client
}
// NewClient returns a Client instance ready for communication with the given
@ -189,6 +193,7 @@ func NewVersionedClient(endpoint string, apiVersionString string) (*Client, erro
}
return &Client{
HTTPClient: http.DefaultClient,
Dialer: &net.Dialer{},
endpoint: endpoint,
endpointURL: u,
eventMonitor: new(eventMonitoringState),
@ -300,6 +305,7 @@ func NewVersionedTLSClientFromBytes(endpoint string, certPEMBlock, keyPEMBlock,
return &Client{
HTTPClient: &http.Client{Transport: tr},
TLSConfig: tlsConfig,
Dialer: &net.Dialer{},
endpoint: endpoint,
endpointURL: u,
eventMonitor: new(eventMonitoringState),
@ -324,32 +330,40 @@ func (c *Client) checkAPIVersion() error {
return nil
}
// Endpoint returns the current endpoint. It's useful for getting the endpoint
// when using functions that get this data from the environment (like
// NewClientFromEnv.
func (c *Client) Endpoint() string {
return c.endpoint
}
// Ping pings the docker server
//
// See http://goo.gl/stJENm for more details.
// See https://goo.gl/kQCfJj for more details.
func (c *Client) Ping() error {
path := "/_ping"
body, status, err := c.do("GET", path, doOptions{})
resp, err := c.do("GET", path, doOptions{})
if err != nil {
return err
}
if status != http.StatusOK {
return newError(status, body)
if resp.StatusCode != http.StatusOK {
return newError(resp)
}
resp.Body.Close()
return nil
}
func (c *Client) getServerAPIVersionString() (version string, err error) {
body, status, err := c.do("GET", "/version", doOptions{})
resp, err := c.do("GET", "/version", doOptions{})
if err != nil {
return "", err
}
if status != http.StatusOK {
return "", fmt.Errorf("Received unexpected status %d while trying to retrieve the server version", status)
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("Received unexpected status %d while trying to retrieve the server version", resp.StatusCode)
}
var versionResponse map[string]interface{}
err = json.Unmarshal(body, &versionResponse)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&versionResponse); err != nil {
return "", err
}
if version, ok := (versionResponse["ApiVersion"]).(string); ok {
@ -363,24 +377,35 @@ type doOptions struct {
forceJSON bool
}
func (c *Client) do(method, path string, doOptions doOptions) ([]byte, int, error) {
func (c *Client) do(method, path string, doOptions doOptions) (*http.Response, error) {
var params io.Reader
if doOptions.data != nil || doOptions.forceJSON {
buf, err := json.Marshal(doOptions.data)
if err != nil {
return nil, -1, err
return nil, err
}
params = bytes.NewBuffer(buf)
}
if path != "/version" && !c.SkipServerVersionCheck && c.expectedAPIVersion == nil {
err := c.checkAPIVersion()
if err != nil {
return nil, -1, err
return nil, err
}
}
req, err := http.NewRequest(method, c.getURL(path), params)
httpClient := c.HTTPClient
protocol := c.endpointURL.Scheme
var u string
if protocol == "unix" {
httpClient = c.unixClient()
u = c.getFakeUnixURL(path)
} else {
u = c.getURL(path)
}
req, err := http.NewRequest(method, u, params)
if err != nil {
return nil, -1, err
return nil, err
}
req.Header.Set("User-Agent", userAgent)
if doOptions.data != nil {
@ -388,40 +413,19 @@ func (c *Client) do(method, path string, doOptions doOptions) ([]byte, int, erro
} else if method == "POST" {
req.Header.Set("Content-Type", "plain/text")
}
var resp *http.Response
protocol := c.endpointURL.Scheme
address := c.endpointURL.Path
if protocol == "unix" {
var dial net.Conn
dial, err = net.Dial(protocol, address)
if err != nil {
return nil, -1, err
}
defer dial.Close()
breader := bufio.NewReader(dial)
err = req.Write(dial)
if err != nil {
return nil, -1, err
}
resp, err = http.ReadResponse(breader, req)
} else {
resp, err = c.HTTPClient.Do(req)
}
resp, err := httpClient.Do(req)
if err != nil {
if strings.Contains(err.Error(), "connection refused") {
return nil, -1, ErrConnectionRefused
return nil, ErrConnectionRefused
}
return nil, -1, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, -1, err
return nil, err
}
if resp.StatusCode < 200 || resp.StatusCode >= 400 {
return nil, resp.StatusCode, newError(resp.StatusCode, body)
return nil, newError(resp)
}
return body, resp.StatusCode, nil
return resp, nil
}
type streamOptions struct {
@ -462,12 +466,16 @@ func (c *Client) stream(method, path string, streamOptions streamOptions) error
address := c.endpointURL.Path
if streamOptions.stdout == nil {
streamOptions.stdout = ioutil.Discard
} else if t, ok := streamOptions.stdout.(io.Closer); ok {
defer t.Close()
}
if streamOptions.stderr == nil {
streamOptions.stderr = ioutil.Discard
} else if t, ok := streamOptions.stderr.(io.Closer); ok {
defer t.Close()
}
if protocol == "unix" {
dial, err := net.Dial(protocol, address)
dial, err := c.Dialer.Dial(protocol, address)
if err != nil {
return err
}
@ -503,11 +511,7 @@ func (c *Client) stream(method, path string, streamOptions streamOptions) error
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 400 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return newError(resp.StatusCode, body)
return newError(resp)
}
if streamOptions.useJSONDecoder || resp.Header.Get("Content-Type") == "application/json" {
// if we want to get raw json stream, just copy it back to output
@ -583,6 +587,8 @@ func (c *Client) hijack(method, path string, hijackOptions hijackOptions) error
return err
}
req.Header.Set("Content-Type", "plain/text")
req.Header.Set("Connection", "Upgrade")
req.Header.Set("Upgrade", "tcp")
protocol := c.endpointURL.Scheme
address := c.endpointURL.Path
if protocol != "unix" {
@ -591,12 +597,12 @@ func (c *Client) hijack(method, path string, hijackOptions hijackOptions) error
}
var dial net.Conn
if c.TLSConfig != nil && protocol != "unix" {
dial, err = tlsDial(protocol, address, c.TLSConfig)
dial, err = tlsDialWithDialer(c.Dialer, protocol, address, c.TLSConfig)
if err != nil {
return err
}
} else {
dial, err = net.Dial(protocol, address)
dial, err = c.Dialer.Dial(protocol, address)
if err != nil {
return err
}
@ -612,13 +618,16 @@ func (c *Client) hijack(method, path string, hijackOptions hijackOptions) error
defer rwc.Close()
errChanOut := make(chan error, 1)
errChanIn := make(chan error, 1)
exit := make(chan bool)
go func() {
defer close(exit)
defer close(errChanOut)
defer func() {
if hijackOptions.in != nil {
if closer, ok := hijackOptions.in.(io.Closer); ok {
closer.Close()
}
}
}()
var err error
if hijackOptions.setRawTerminal {
// When TTY is ON, use regular copy
_, err = io.Copy(hijackOptions.stdout, br)
} else {
_, err = stdcopy.StdCopy(hijackOptions.stdout, hijackOptions.stderr, br)
@ -626,17 +635,15 @@ func (c *Client) hijack(method, path string, hijackOptions hijackOptions) error
errChanOut <- err
}()
go func() {
var err error
if hijackOptions.in != nil {
_, err := io.Copy(rwc, hijackOptions.in)
errChanIn <- err
} else {
errChanIn <- nil
_, err = io.Copy(rwc, hijackOptions.in)
}
errChanIn <- err
rwc.(interface {
CloseWrite() error
}).CloseWrite()
}()
<-exit
errIn := <-errChanIn
errOut := <-errChanOut
if errIn != nil {
@ -657,6 +664,41 @@ func (c *Client) getURL(path string) string {
return fmt.Sprintf("%s%s", urlStr, path)
}
// getFakeUnixURL returns the URL needed to make an HTTP request over a UNIX
// domain socket to the given path.
func (c *Client) getFakeUnixURL(path string) string {
u := *c.endpointURL // Copy.
// Override URL so that net/http will not complain.
u.Scheme = "http"
u.Host = "unix.sock" // Doesn't matter what this is - it's not used.
u.Path = ""
urlStr := strings.TrimRight(u.String(), "/")
if c.requestedAPIVersion != nil {
return fmt.Sprintf("%s/v%s%s", urlStr, c.requestedAPIVersion, path)
}
return fmt.Sprintf("%s%s", urlStr, path)
}
func (c *Client) unixClient() *http.Client {
if c.unixHTTPClient != nil {
return c.unixHTTPClient
}
socketPath := c.endpointURL.Path
c.unixHTTPClient = &http.Client{
Transport: &http.Transport{
Dial: func(network, addr string) (net.Conn, error) {
return c.Dialer.Dial("unix", socketPath)
},
},
}
return c.unixHTTPClient
}
type jsonMessage struct {
Status string `json:"status,omitempty"`
Progress string `json:"progress,omitempty"`
@ -738,8 +780,13 @@ type Error struct {
Message string
}
func newError(status int, body []byte) *Error {
return &Error{Status: status, Message: string(body)}
func newError(resp *http.Response) *Error {
defer resp.Body.Close()
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
return &Error{Status: resp.StatusCode, Message: fmt.Sprintf("cannot read body, err: %v", err)}
}
return &Error{Status: resp.StatusCode, Message: string(data)}
}
func (e *Error) Error() string {

44
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/client_test.go generated vendored
View File

@ -5,6 +5,7 @@
package docker
import (
"bytes"
"fmt"
"io/ioutil"
"net"
@ -161,6 +162,16 @@ func TestNewTLSClient(t *testing.T) {
}
}
func TestEndpoint(t *testing.T) {
client, err := NewVersionedClient("http://localhost:4243", "1.12")
if err != nil {
t.Fatal(err)
}
if endpoint := client.Endpoint(); endpoint != client.endpoint {
t.Errorf("Client.Endpoint(): want %q. Got %q", client.endpoint, endpoint)
}
}
func TestGetURL(t *testing.T) {
var tests = []struct {
endpoint string
@ -185,8 +196,34 @@ func TestGetURL(t *testing.T) {
}
}
func TestGetFakeUnixURL(t *testing.T) {
var tests = []struct {
endpoint string
path string
expected string
}{
{"unix://var/run/docker.sock", "/", "http://unix.sock/"},
{"unix://var/run/docker.socket", "/", "http://unix.sock/"},
{"unix://var/run/docker.sock", "/containers/ps", "http://unix.sock/containers/ps"},
}
for _, tt := range tests {
client, _ := NewClient(tt.endpoint)
client.endpoint = tt.endpoint
client.SkipServerVersionCheck = true
got := client.getFakeUnixURL(tt.path)
if got != tt.expected {
t.Errorf("getURL(%q): Got %s. Want %s.", tt.path, got, tt.expected)
}
}
}
func TestError(t *testing.T) {
err := newError(400, []byte("bad parameter"))
fakeBody := ioutil.NopCloser(bytes.NewBufferString("bad parameter"))
resp := &http.Response{
StatusCode: 400,
Body: fakeBody,
}
err := newError(resp)
expected := Error{Status: 400, Message: "bad parameter"}
if !reflect.DeepEqual(expected, *err) {
t.Errorf("Wrong error type. Want %#v. Got %#v.", expected, *err)
@ -334,7 +371,7 @@ func TestPingErrorWithUnixSocket(t *testing.T) {
}
defer li.Close()
if err != nil {
t.Fatalf("Expected to get listner, but failed: %#v", err)
t.Fatalf("Expected to get listener, but failed: %#v", err)
}
fd, err := li.Accept()
@ -345,7 +382,7 @@ func TestPingErrorWithUnixSocket(t *testing.T) {
buf := make([]byte, 512)
nr, err := fd.Read(buf)
// Create invalid response message to occur error
// Create invalid response message to trigger error.
data := buf[0:nr]
for i := 0; i < 10; i++ {
data[i] = 63
@ -366,6 +403,7 @@ func TestPingErrorWithUnixSocket(t *testing.T) {
u, _ := parseEndpoint(endpoint, false)
client := Client{
HTTPClient: http.DefaultClient,
Dialer: &net.Dialer{},
endpoint: endpoint,
endpointURL: u,
SkipServerVersionCheck: true,

430
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/container.go generated vendored
View File

@ -5,7 +5,6 @@
package docker
import (
"bytes"
"encoding/json"
"errors"
"fmt"
@ -23,7 +22,7 @@ var ErrContainerAlreadyExists = errors.New("container already exists")
// ListContainersOptions specify parameters to the ListContainers function.
//
// See http://goo.gl/6Y4Gz7 for more details.
// See https://goo.gl/47a6tO for more details.
type ListContainersOptions struct {
All bool
Size bool
@ -41,33 +40,33 @@ type APIPort struct {
IP string `json:"IP,omitempty" yaml:"IP,omitempty"`
}
// APIContainers represents a container.
//
// See http://goo.gl/QeFH7U for more details.
// APIContainers represents each container in the list returned by
// ListContainers.
type APIContainers struct {
ID string `json:"Id" yaml:"Id"`
Image string `json:"Image,omitempty" yaml:"Image,omitempty"`
Command string `json:"Command,omitempty" yaml:"Command,omitempty"`
Created int64 `json:"Created,omitempty" yaml:"Created,omitempty"`
Status string `json:"Status,omitempty" yaml:"Status,omitempty"`
Ports []APIPort `json:"Ports,omitempty" yaml:"Ports,omitempty"`
SizeRw int64 `json:"SizeRw,omitempty" yaml:"SizeRw,omitempty"`
SizeRootFs int64 `json:"SizeRootFs,omitempty" yaml:"SizeRootFs,omitempty"`
Names []string `json:"Names,omitempty" yaml:"Names,omitempty"`
ID string `json:"Id" yaml:"Id"`
Image string `json:"Image,omitempty" yaml:"Image,omitempty"`
Command string `json:"Command,omitempty" yaml:"Command,omitempty"`
Created int64 `json:"Created,omitempty" yaml:"Created,omitempty"`
Status string `json:"Status,omitempty" yaml:"Status,omitempty"`
Ports []APIPort `json:"Ports,omitempty" yaml:"Ports,omitempty"`
SizeRw int64 `json:"SizeRw,omitempty" yaml:"SizeRw,omitempty"`
SizeRootFs int64 `json:"SizeRootFs,omitempty" yaml:"SizeRootFs,omitempty"`
Names []string `json:"Names,omitempty" yaml:"Names,omitempty"`
Labels map[string]string `json:"Labels,omitempty" yaml:"Labels, omitempty"`
}
// ListContainers returns a slice of containers matching the given criteria.
//
// See http://goo.gl/6Y4Gz7 for more details.
// See https://goo.gl/47a6tO for more details.
func (c *Client) ListContainers(opts ListContainersOptions) ([]APIContainers, error) {
path := "/containers/json?" + queryString(opts)
body, _, err := c.do("GET", path, doOptions{})
resp, err := c.do("GET", path, doOptions{})
if err != nil {
return nil, err
}
defer resp.Body.Close()
var containers []APIContainers
err = json.Unmarshal(body, &containers)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&containers); err != nil {
return nil, err
}
return containers, nil
@ -206,6 +205,7 @@ type Config struct {
DNS []string `json:"Dns,omitempty" yaml:"Dns,omitempty"` // For Docker API v1.9 and below only
Image string `json:"Image,omitempty" yaml:"Image,omitempty"`
Volumes map[string]struct{} `json:"Volumes,omitempty" yaml:"Volumes,omitempty"`
VolumeDriver string `json:"VolumeDriver,omitempty" yaml:"VolumeDriver,omitempty"`
VolumesFrom string `json:"VolumesFrom,omitempty" yaml:"VolumesFrom,omitempty"`
WorkingDir string `json:"WorkingDir,omitempty" yaml:"WorkingDir,omitempty"`
MacAddress string `json:"MacAddress,omitempty" yaml:"MacAddress,omitempty"`
@ -213,9 +213,21 @@ type Config struct {
NetworkDisabled bool `json:"NetworkDisabled,omitempty" yaml:"NetworkDisabled,omitempty"`
SecurityOpts []string `json:"SecurityOpts,omitempty" yaml:"SecurityOpts,omitempty"`
OnBuild []string `json:"OnBuild,omitempty" yaml:"OnBuild,omitempty"`
Mounts []Mount `json:"Mounts,omitempty" yaml:"Mounts,omitempty"`
Labels map[string]string `json:"Labels,omitempty" yaml:"Labels,omitempty"`
}
// Mount represents a mount point in the container.
//
// It has been added in the version 1.20 of the Docker API, available since
// Docker 1.8.
type Mount struct {
Source string
Destination string
Mode string
RW bool
}
// LogConfig defines the log driver type and the configuration for it.
type LogConfig struct {
Type string `json:"Type,omitempty" yaml:"Type,omitempty"`
@ -259,13 +271,14 @@ type Container struct {
NetworkSettings *NetworkSettings `json:"NetworkSettings,omitempty" yaml:"NetworkSettings,omitempty"`
SysInitPath string `json:"SysInitPath,omitempty" yaml:"SysInitPath,omitempty"`
ResolvConfPath string `json:"ResolvConfPath,omitempty" yaml:"ResolvConfPath,omitempty"`
HostnamePath string `json:"HostnamePath,omitempty" yaml:"HostnamePath,omitempty"`
HostsPath string `json:"HostsPath,omitempty" yaml:"HostsPath,omitempty"`
LogPath string `json:"LogPath,omitempty" yaml:"LogPath,omitempty"`
Name string `json:"Name,omitempty" yaml:"Name,omitempty"`
Driver string `json:"Driver,omitempty" yaml:"Driver,omitempty"`
SysInitPath string `json:"SysInitPath,omitempty" yaml:"SysInitPath,omitempty"`
ResolvConfPath string `json:"ResolvConfPath,omitempty" yaml:"ResolvConfPath,omitempty"`
HostnamePath string `json:"HostnamePath,omitempty" yaml:"HostnamePath,omitempty"`
HostsPath string `json:"HostsPath,omitempty" yaml:"HostsPath,omitempty"`
LogPath string `json:"LogPath,omitempty" yaml:"LogPath,omitempty"`
Name string `json:"Name,omitempty" yaml:"Name,omitempty"`
Driver string `json:"Driver,omitempty" yaml:"Driver,omitempty"`
Mounts []Mount `json:"Mounts,omitempty" yaml:"Mounts,omitempty"`
Volumes map[string]string `json:"Volumes,omitempty" yaml:"Volumes,omitempty"`
VolumesRW map[string]bool `json:"VolumesRW,omitempty" yaml:"VolumesRW,omitempty"`
@ -279,7 +292,7 @@ type Container struct {
// RenameContainerOptions specify parameters to the RenameContainer function.
//
// See http://goo.gl/L00hoj for more details.
// See https://goo.gl/laSOIy for more details.
type RenameContainerOptions struct {
// ID of container to rename
ID string `qs:"-"`
@ -290,27 +303,31 @@ type RenameContainerOptions struct {
// RenameContainer updates and existing containers name
//
// See http://goo.gl/L00hoj for more details.
// See https://goo.gl/laSOIy for more details.
func (c *Client) RenameContainer(opts RenameContainerOptions) error {
_, _, err := c.do("POST", fmt.Sprintf("/containers/"+opts.ID+"/rename?%s", queryString(opts)), doOptions{})
return err
resp, err := c.do("POST", fmt.Sprintf("/containers/"+opts.ID+"/rename?%s", queryString(opts)), doOptions{})
if err != nil {
return err
}
resp.Body.Close()
return nil
}
// InspectContainer returns information about a container by its ID.
//
// See http://goo.gl/CxVuJ5 for more details.
// See https://goo.gl/RdIq0b for more details.
func (c *Client) InspectContainer(id string) (*Container, error) {
path := "/containers/" + id + "/json"
body, status, err := c.do("GET", path, doOptions{})
if status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: id}
}
resp, err := c.do("GET", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: id}
}
return nil, err
}
defer resp.Body.Close()
var container Container
err = json.Unmarshal(body, &container)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&container); err != nil {
return nil, err
}
return &container, nil
@ -318,19 +335,19 @@ func (c *Client) InspectContainer(id string) (*Container, error) {
// ContainerChanges returns changes in the filesystem of the given container.
//
// See http://goo.gl/QkW9sH for more details.
// See https://goo.gl/9GsTIF for more details.
func (c *Client) ContainerChanges(id string) ([]Change, error) {
path := "/containers/" + id + "/changes"
body, status, err := c.do("GET", path, doOptions{})
if status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: id}
}
resp, err := c.do("GET", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: id}
}
return nil, err
}
defer resp.Body.Close()
var changes []Change
err = json.Unmarshal(body, &changes)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&changes); err != nil {
return nil, err
}
return changes, nil
@ -338,7 +355,7 @@ func (c *Client) ContainerChanges(id string) ([]Change, error) {
// CreateContainerOptions specify parameters to the CreateContainer function.
//
// See http://goo.gl/2xxQQK for more details.
// See https://goo.gl/WxQzrr for more details.
type CreateContainerOptions struct {
Name string
Config *Config `qs:"-"`
@ -348,10 +365,10 @@ type CreateContainerOptions struct {
// CreateContainer creates a new container, returning the container instance,
// or an error in case of failure.
//
// See http://goo.gl/mErxNp for more details.
// See https://goo.gl/WxQzrr for more details.
func (c *Client) CreateContainer(opts CreateContainerOptions) (*Container, error) {
path := "/containers/create?" + queryString(opts)
body, status, err := c.do(
resp, err := c.do(
"POST",
path,
doOptions{
@ -365,18 +382,21 @@ func (c *Client) CreateContainer(opts CreateContainerOptions) (*Container, error
},
)
if status == http.StatusNotFound {
return nil, ErrNoSuchImage
}
if status == http.StatusConflict {
return nil, ErrContainerAlreadyExists
if e, ok := err.(*Error); ok {
if e.Status == http.StatusNotFound {
return nil, ErrNoSuchImage
}
if e.Status == http.StatusConflict {
return nil, ErrContainerAlreadyExists
}
}
if err != nil {
return nil, err
}
defer resp.Body.Close()
var container Container
err = json.Unmarshal(body, &container)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&container); err != nil {
return nil, err
}
@ -434,125 +454,135 @@ type Device struct {
// HostConfig contains the container options related to starting a container on
// a given host
type HostConfig struct {
Binds []string `json:"Binds,omitempty" yaml:"Binds,omitempty"`
CapAdd []string `json:"CapAdd,omitempty" yaml:"CapAdd,omitempty"`
CapDrop []string `json:"CapDrop,omitempty" yaml:"CapDrop,omitempty"`
ContainerIDFile string `json:"ContainerIDFile,omitempty" yaml:"ContainerIDFile,omitempty"`
LxcConf []KeyValuePair `json:"LxcConf,omitempty" yaml:"LxcConf,omitempty"`
Privileged bool `json:"Privileged,omitempty" yaml:"Privileged,omitempty"`
PortBindings map[Port][]PortBinding `json:"PortBindings,omitempty" yaml:"PortBindings,omitempty"`
Links []string `json:"Links,omitempty" yaml:"Links,omitempty"`
PublishAllPorts bool `json:"PublishAllPorts,omitempty" yaml:"PublishAllPorts,omitempty"`
DNS []string `json:"Dns,omitempty" yaml:"Dns,omitempty"` // For Docker API v1.10 and above only
DNSSearch []string `json:"DnsSearch,omitempty" yaml:"DnsSearch,omitempty"`
ExtraHosts []string `json:"ExtraHosts,omitempty" yaml:"ExtraHosts,omitempty"`
VolumesFrom []string `json:"VolumesFrom,omitempty" yaml:"VolumesFrom,omitempty"`
NetworkMode string `json:"NetworkMode,omitempty" yaml:"NetworkMode,omitempty"`
IpcMode string `json:"IpcMode,omitempty" yaml:"IpcMode,omitempty"`
PidMode string `json:"PidMode,omitempty" yaml:"PidMode,omitempty"`
UTSMode string `json:"UTSMode,omitempty" yaml:"UTSMode,omitempty"`
RestartPolicy RestartPolicy `json:"RestartPolicy,omitempty" yaml:"RestartPolicy,omitempty"`
Devices []Device `json:"Devices,omitempty" yaml:"Devices,omitempty"`
LogConfig LogConfig `json:"LogConfig,omitempty" yaml:"LogConfig,omitempty"`
ReadonlyRootfs bool `json:"ReadonlyRootfs,omitempty" yaml:"ReadonlyRootfs,omitempty"`
SecurityOpt []string `json:"SecurityOpt,omitempty" yaml:"SecurityOpt,omitempty"`
CgroupParent string `json:"CgroupParent,omitempty" yaml:"CgroupParent,omitempty"`
Memory int64 `json:"Memory,omitempty" yaml:"Memory,omitempty"`
MemorySwap int64 `json:"MemorySwap,omitempty" yaml:"MemorySwap,omitempty"`
CPUShares int64 `json:"CpuShares,omitempty" yaml:"CpuShares,omitempty"`
CPUSet string `json:"Cpuset,omitempty" yaml:"Cpuset,omitempty"`
CPUQuota int64 `json:"CpuQuota,omitempty" yaml:"CpuQuota,omitempty"`
CPUPeriod int64 `json:"CpuPeriod,omitempty" yaml:"CpuPeriod,omitempty"`
Ulimits []ULimit `json:"Ulimits,omitempty" yaml:"Ulimits,omitempty"`
Binds []string `json:"Binds,omitempty" yaml:"Binds,omitempty"`
CapAdd []string `json:"CapAdd,omitempty" yaml:"CapAdd,omitempty"`
CapDrop []string `json:"CapDrop,omitempty" yaml:"CapDrop,omitempty"`
ContainerIDFile string `json:"ContainerIDFile,omitempty" yaml:"ContainerIDFile,omitempty"`
LxcConf []KeyValuePair `json:"LxcConf,omitempty" yaml:"LxcConf,omitempty"`
Privileged bool `json:"Privileged,omitempty" yaml:"Privileged,omitempty"`
PortBindings map[Port][]PortBinding `json:"PortBindings,omitempty" yaml:"PortBindings,omitempty"`
Links []string `json:"Links,omitempty" yaml:"Links,omitempty"`
PublishAllPorts bool `json:"PublishAllPorts,omitempty" yaml:"PublishAllPorts,omitempty"`
DNS []string `json:"Dns,omitempty" yaml:"Dns,omitempty"` // For Docker API v1.10 and above only
DNSSearch []string `json:"DnsSearch,omitempty" yaml:"DnsSearch,omitempty"`
ExtraHosts []string `json:"ExtraHosts,omitempty" yaml:"ExtraHosts,omitempty"`
VolumesFrom []string `json:"VolumesFrom,omitempty" yaml:"VolumesFrom,omitempty"`
NetworkMode string `json:"NetworkMode,omitempty" yaml:"NetworkMode,omitempty"`
IpcMode string `json:"IpcMode,omitempty" yaml:"IpcMode,omitempty"`
PidMode string `json:"PidMode,omitempty" yaml:"PidMode,omitempty"`
UTSMode string `json:"UTSMode,omitempty" yaml:"UTSMode,omitempty"`
RestartPolicy RestartPolicy `json:"RestartPolicy,omitempty" yaml:"RestartPolicy,omitempty"`
Devices []Device `json:"Devices,omitempty" yaml:"Devices,omitempty"`
LogConfig LogConfig `json:"LogConfig,omitempty" yaml:"LogConfig,omitempty"`
ReadonlyRootfs bool `json:"ReadonlyRootfs,omitempty" yaml:"ReadonlyRootfs,omitempty"`
SecurityOpt []string `json:"SecurityOpt,omitempty" yaml:"SecurityOpt,omitempty"`
CgroupParent string `json:"CgroupParent,omitempty" yaml:"CgroupParent,omitempty"`
Memory int64 `json:"Memory,omitempty" yaml:"Memory,omitempty"`
MemorySwap int64 `json:"MemorySwap,omitempty" yaml:"MemorySwap,omitempty"`
MemorySwappiness int64 `json:"MemorySwappiness,omitempty" yaml:"MemorySwappiness,omitempty"`
OOMKillDisable bool `json:"OomKillDisable,omitempty" yaml:"OomKillDisable"`
CPUShares int64 `json:"CpuShares,omitempty" yaml:"CpuShares,omitempty"`
CPUSet string `json:"Cpuset,omitempty" yaml:"Cpuset,omitempty"`
CPUSetCPUs string `json:"CpusetCpus,omitempty" yaml:"CpusetCpus,omitempty"`
CPUSetMEMs string `json:"CpusetMems,omitempty" yaml:"CpusetMems,omitempty"`
CPUQuota int64 `json:"CpuQuota,omitempty" yaml:"CpuQuota,omitempty"`
CPUPeriod int64 `json:"CpuPeriod,omitempty" yaml:"CpuPeriod,omitempty"`
BlkioWeight int64 `json:"BlkioWeight,omitempty" yaml:"BlkioWeight"`
Ulimits []ULimit `json:"Ulimits,omitempty" yaml:"Ulimits,omitempty"`
}
// StartContainer starts a container, returning an error in case of failure.
//
// See http://goo.gl/iM5GYs for more details.
// See https://goo.gl/MrBAJv for more details.
func (c *Client) StartContainer(id string, hostConfig *HostConfig) error {
path := "/containers/" + id + "/start"
_, status, err := c.do("POST", path, doOptions{data: hostConfig, forceJSON: true})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: id, Err: err}
}
if status == http.StatusNotModified {
return &ContainerAlreadyRunning{ID: id}
}
resp, err := c.do("POST", path, doOptions{data: hostConfig, forceJSON: true})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: id, Err: err}
}
return err
}
if resp.StatusCode == http.StatusNotModified {
return &ContainerAlreadyRunning{ID: id}
}
resp.Body.Close()
return nil
}
// StopContainer stops a container, killing it after the given timeout (in
// seconds).
//
// See http://goo.gl/EbcpXt for more details.
// See https://goo.gl/USqsFt for more details.
func (c *Client) StopContainer(id string, timeout uint) error {
path := fmt.Sprintf("/containers/%s/stop?t=%d", id, timeout)
_, status, err := c.do("POST", path, doOptions{})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
if status == http.StatusNotModified {
return &ContainerNotRunning{ID: id}
}
resp, err := c.do("POST", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
return err
}
if resp.StatusCode == http.StatusNotModified {
return &ContainerNotRunning{ID: id}
}
resp.Body.Close()
return nil
}
// RestartContainer stops a container, killing it after the given timeout (in
// seconds), during the stop process.
//
// See http://goo.gl/VOzR2n for more details.
// See https://goo.gl/QzsDnz for more details.
func (c *Client) RestartContainer(id string, timeout uint) error {
path := fmt.Sprintf("/containers/%s/restart?t=%d", id, timeout)
_, status, err := c.do("POST", path, doOptions{})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
resp, err := c.do("POST", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
return err
}
resp.Body.Close()
return nil
}
// PauseContainer pauses the given container.
//
// See http://goo.gl/AM5t42 for more details.
// See https://goo.gl/OF7W9X for more details.
func (c *Client) PauseContainer(id string) error {
path := fmt.Sprintf("/containers/%s/pause", id)
_, status, err := c.do("POST", path, doOptions{})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
resp, err := c.do("POST", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
return err
}
resp.Body.Close()
return nil
}
// UnpauseContainer unpauses the given container.
//
// See http://goo.gl/eBrNSL for more details.
// See https://goo.gl/7dwyPA for more details.
func (c *Client) UnpauseContainer(id string) error {
path := fmt.Sprintf("/containers/%s/unpause", id)
_, status, err := c.do("POST", path, doOptions{})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
resp, err := c.do("POST", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: id}
}
return err
}
resp.Body.Close()
return nil
}
// TopResult represents the list of processes running in a container, as
// returned by /containers/<id>/top.
//
// See http://goo.gl/qu4gse for more details.
// See https://goo.gl/Rb46aY for more details.
type TopResult struct {
Titles []string
Processes [][]string
@ -560,7 +590,7 @@ type TopResult struct {
// TopContainer returns processes running inside a container
//
// See http://goo.gl/qu4gse for more details.
// See https://goo.gl/Rb46aY for more details.
func (c *Client) TopContainer(id string, psArgs string) (TopResult, error) {
var args string
var result TopResult
@ -568,15 +598,15 @@ func (c *Client) TopContainer(id string, psArgs string) (TopResult, error) {
args = fmt.Sprintf("?ps_args=%s", psArgs)
}
path := fmt.Sprintf("/containers/%s/top%s", id, args)
body, status, err := c.do("GET", path, doOptions{})
if status == http.StatusNotFound {
return result, &NoSuchContainer{ID: id}
}
resp, err := c.do("GET", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return result, &NoSuchContainer{ID: id}
}
return result, err
}
err = json.Unmarshal(body, &result)
if err != nil {
defer resp.Body.Close()
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return result, err
}
return result, nil
@ -584,7 +614,7 @@ func (c *Client) TopContainer(id string, psArgs string) (TopResult, error) {
// Stats represents container statistics, returned by /containers/<id>/stats.
//
// See http://goo.gl/DFMiYD for more details.
// See https://goo.gl/GNmLHb for more details.
type Stats struct {
Read time.Time `json:"read,omitempty" yaml:"read,omitempty"`
Network struct {
@ -674,7 +704,7 @@ type BlkioStatsEntry struct {
// StatsOptions specify parameters to the Stats function.
//
// See http://goo.gl/DFMiYD for more details.
// See https://goo.gl/GNmLHb for more details.
type StatsOptions struct {
ID string
Stats chan<- *Stats
@ -690,9 +720,10 @@ type StatsOptions struct {
// This function is blocking, similar to a streaming call for logs, and should be run
// on a separate goroutine from the caller. Note that this function will block until
// the given container is removed, not just exited. When finished, this function
// will close the given channel. Alternatively, function can be stopped by signaling on the Done channel
// will close the given channel. Alternatively, function can be stopped by
// signaling on the Done channel.
//
// See http://goo.gl/DFMiYD for more details.
// See https://goo.gl/GNmLHb for more details.
func (c *Client) Stats(opts StatsOptions) (retErr error) {
errC := make(chan error, 1)
readCloser, writeCloser := io.Pipe()
@ -750,7 +781,7 @@ func (c *Client) Stats(opts StatsOptions) (retErr error) {
decoder := json.NewDecoder(readCloser)
stats := new(Stats)
for err := decoder.Decode(&stats); err != io.EOF; err = decoder.Decode(stats) {
for err := decoder.Decode(stats); err != io.EOF; err = decoder.Decode(stats) {
if err != nil {
return err
}
@ -763,7 +794,7 @@ func (c *Client) Stats(opts StatsOptions) (retErr error) {
// KillContainerOptions represents the set of options that can be used in a
// call to KillContainer.
//
// See http://goo.gl/TFkECx for more details.
// See https://goo.gl/hkS9i8 for more details.
type KillContainerOptions struct {
// The ID of the container.
ID string `qs:"-"`
@ -773,24 +804,26 @@ type KillContainerOptions struct {
Signal Signal
}
// KillContainer kills a container, returning an error in case of failure.
// KillContainer sends a signal to a container, returning an error in case of
// failure.
//
// See http://goo.gl/TFkECx for more details.
// See https://goo.gl/hkS9i8 for more details.
func (c *Client) KillContainer(opts KillContainerOptions) error {
path := "/containers/" + opts.ID + "/kill" + "?" + queryString(opts)
_, status, err := c.do("POST", path, doOptions{})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: opts.ID}
}
resp, err := c.do("POST", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: opts.ID}
}
return err
}
resp.Body.Close()
return nil
}
// RemoveContainerOptions encapsulates options to remove a container.
//
// See http://goo.gl/ZB83ji for more details.
// See https://goo.gl/RQyX62 for more details.
type RemoveContainerOptions struct {
// The ID of the container.
ID string `qs:"-"`
@ -806,64 +839,107 @@ type RemoveContainerOptions struct {
// RemoveContainer removes a container, returning an error in case of failure.
//
// See http://goo.gl/ZB83ji for more details.
// See https://goo.gl/RQyX62 for more details.
func (c *Client) RemoveContainer(opts RemoveContainerOptions) error {
path := "/containers/" + opts.ID + "?" + queryString(opts)
_, status, err := c.do("DELETE", path, doOptions{})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: opts.ID}
}
resp, err := c.do("DELETE", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: opts.ID}
}
return err
}
resp.Body.Close()
return nil
}
// CopyFromContainerOptions is the set of options that can be used when copying
// files or folders from a container.
// UploadToContainerOptions is the set of options that can be used when
// uploading an archive into a container.
//
// See http://goo.gl/rINMlw for more details.
// See https://goo.gl/Ss97HW for more details.
type UploadToContainerOptions struct {
InputStream io.Reader `json:"-" qs:"-"`
Path string `qs:"path"`
NoOverwriteDirNonDir bool `qs:"noOverwriteDirNonDir"`
}
// UploadToContainer uploads a tar archive to be extracted to a path in the
// filesystem of the container.
//
// See https://goo.gl/Ss97HW for more details.
func (c *Client) UploadToContainer(id string, opts UploadToContainerOptions) error {
url := fmt.Sprintf("/containers/%s/archive?", id) + queryString(opts)
return c.stream("PUT", url, streamOptions{
in: opts.InputStream,
})
}
// DownloadFromContainerOptions is the set of options that can be used when
// downloading resources from a container.
//
// See https://goo.gl/KnZJDX for more details.
type DownloadFromContainerOptions struct {
OutputStream io.Writer `json:"-" qs:"-"`
Path string `qs:"path"`
}
// DownloadFromContainer downloads a tar archive of files or folders in a container.
//
// See https://goo.gl/KnZJDX for more details.
func (c *Client) DownloadFromContainer(id string, opts DownloadFromContainerOptions) error {
url := fmt.Sprintf("/containers/%s/archive?", id) + queryString(opts)
return c.stream("GET", url, streamOptions{
setRawTerminal: true,
stdout: opts.OutputStream,
})
}
// CopyFromContainerOptions has been DEPRECATED, please use DownloadFromContainerOptions along with DownloadFromContainer.
//
// See https://goo.gl/R2jevW for more details.
type CopyFromContainerOptions struct {
OutputStream io.Writer `json:"-"`
Container string `json:"-"`
Resource string
}
// CopyFromContainer copy files or folders from a container, using a given
// resource.
// CopyFromContainer has been DEPRECATED, please use DownloadFromContainerOptions along with DownloadFromContainer.
//
// See http://goo.gl/rINMlw for more details.
// See https://goo.gl/R2jevW for more details.
func (c *Client) CopyFromContainer(opts CopyFromContainerOptions) error {
if opts.Container == "" {
return &NoSuchContainer{ID: opts.Container}
}
url := fmt.Sprintf("/containers/%s/copy", opts.Container)
body, status, err := c.do("POST", url, doOptions{data: opts})
if status == http.StatusNotFound {
return &NoSuchContainer{ID: opts.Container}
}
resp, err := c.do("POST", url, doOptions{data: opts})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchContainer{ID: opts.Container}
}
return err
}
_, err = io.Copy(opts.OutputStream, bytes.NewBuffer(body))
defer resp.Body.Close()
_, err = io.Copy(opts.OutputStream, resp.Body)
return err
}
// WaitContainer blocks until the given container stops, return the exit code
// of the container status.
//
// See http://goo.gl/J88DHU for more details.
// See https://goo.gl/Gc1rge for more details.
func (c *Client) WaitContainer(id string) (int, error) {
body, status, err := c.do("POST", "/containers/"+id+"/wait", doOptions{})
if status == http.StatusNotFound {
return 0, &NoSuchContainer{ID: id}
}
resp, err := c.do("POST", "/containers/"+id+"/wait", doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return 0, &NoSuchContainer{ID: id}
}
return 0, err
}
defer resp.Body.Close()
var r struct{ StatusCode int }
err = json.Unmarshal(body, &r)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&r); err != nil {
return 0, err
}
return r.StatusCode, nil
@ -871,7 +947,7 @@ func (c *Client) WaitContainer(id string) (int, error) {
// CommitContainerOptions aggregates parameters to the CommitContainer method.
//
// See http://goo.gl/Jn8pe8 for more details.
// See https://goo.gl/mqfoCw for more details.
type CommitContainerOptions struct {
Container string
Repository string `qs:"repo"`
@ -883,19 +959,19 @@ type CommitContainerOptions struct {
// CommitContainer creates a new image from a container's changes.
//
// See http://goo.gl/Jn8pe8 for more details.
// See https://goo.gl/mqfoCw for more details.
func (c *Client) CommitContainer(opts CommitContainerOptions) (*Image, error) {
path := "/commit?" + queryString(opts)
body, status, err := c.do("POST", path, doOptions{data: opts.Run})
if status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: opts.Container}
}
resp, err := c.do("POST", path, doOptions{data: opts.Run})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: opts.Container}
}
return nil, err
}
defer resp.Body.Close()
var image Image
err = json.Unmarshal(body, &image)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&image); err != nil {
return nil, err
}
return &image, nil
@ -904,7 +980,7 @@ func (c *Client) CommitContainer(opts CommitContainerOptions) (*Image, error) {
// AttachToContainerOptions is the set of options that can be used when
// attaching to a container.
//
// See http://goo.gl/RRAhws for more details.
// See https://goo.gl/NKpkFk for more details.
type AttachToContainerOptions struct {
Container string `qs:"-"`
InputStream io.Reader `qs:"-"`
@ -939,7 +1015,7 @@ type AttachToContainerOptions struct {
// AttachToContainer attaches to a container, using the given options.
//
// See http://goo.gl/RRAhws for more details.
// See https://goo.gl/NKpkFk for more details.
func (c *Client) AttachToContainer(opts AttachToContainerOptions) error {
if opts.Container == "" {
return &NoSuchContainer{ID: opts.Container}
@ -957,7 +1033,7 @@ func (c *Client) AttachToContainer(opts AttachToContainerOptions) error {
// LogsOptions represents the set of options used when getting logs from a
// container.
//
// See http://goo.gl/rLhKSU for more details.
// See https://goo.gl/yl8PGm for more details.
type LogsOptions struct {
Container string `qs:"-"`
OutputStream io.Writer `qs:"-"`
@ -975,7 +1051,7 @@ type LogsOptions struct {
// Logs gets stdout and stderr logs from the specified container.
//
// See http://goo.gl/rLhKSU for more details.
// See https://goo.gl/yl8PGm for more details.
func (c *Client) Logs(opts LogsOptions) error {
if opts.Container == "" {
return &NoSuchContainer{ID: opts.Container}
@ -992,18 +1068,24 @@ func (c *Client) Logs(opts LogsOptions) error {
}
// ResizeContainerTTY resizes the terminal to the given height and width.
//
// See https://goo.gl/xERhCc for more details.
func (c *Client) ResizeContainerTTY(id string, height, width int) error {
params := make(url.Values)
params.Set("h", strconv.Itoa(height))
params.Set("w", strconv.Itoa(width))
_, _, err := c.do("POST", "/containers/"+id+"/resize?"+params.Encode(), doOptions{})
return err
resp, err := c.do("POST", "/containers/"+id+"/resize?"+params.Encode(), doOptions{})
if err != nil {
return err
}
resp.Body.Close()
return nil
}
// ExportContainerOptions is the set of parameters to the ExportContainer
// method.
//
// See http://goo.gl/hnzE62 for more details.
// See https://goo.gl/dOkTyk for more details.
type ExportContainerOptions struct {
ID string
OutputStream io.Writer
@ -1012,7 +1094,7 @@ type ExportContainerOptions struct {
// ExportContainer export the contents of container id as tar archive
// and prints the exported contents to stdout.
//
// See http://goo.gl/hnzE62 for more details.
// See https://goo.gl/dOkTyk for more details.
func (c *Client) ExportContainer(opts ExportContainerOptions) error {
if opts.ID == "" {
return &NoSuchContainer{ID: opts.ID}

56
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/container_test.go generated vendored
View File

@ -1122,10 +1122,7 @@ func TestAttachToContainerRawTerminalFalse(t *testing.T) {
Stream: true,
RawTerminal: false,
}
err := client.AttachToContainer(opts)
if err != nil {
t.Fatal(err)
}
client.AttachToContainer(opts)
expected := map[string][]string{
"stdin": {"1"},
"stdout": {"1"},
@ -1374,6 +1371,7 @@ func TestExportContainerViaUnixSocket(t *testing.T) {
u, _ := parseEndpoint(endpoint, false)
client := Client{
HTTPClient: http.DefaultClient,
Dialer: &net.Dialer{},
endpoint: endpoint,
endpointURL: u,
SkipServerVersionCheck: true,
@ -1429,17 +1427,60 @@ func TestExportContainerNoId(t *testing.T) {
}
}
func TestUploadToContainer(t *testing.T) {
content := "File content"
in := stdinMock{bytes.NewBufferString(content)}
fakeRT := &FakeRoundTripper{status: http.StatusOK}
client := newTestClient(fakeRT)
opts := UploadToContainerOptions{
Path: "abc",
InputStream: in,
}
err := client.UploadToContainer("a123456", opts)
if err != nil {
t.Errorf("UploadToContainer: caught error %#v while uploading archive to container, expected nil", err)
}
req := fakeRT.requests[0]
if req.Method != "PUT" {
t.Errorf("UploadToContainer{Path:abc}: Wrong HTTP method. Want PUT. Got %s", req.Method)
}
if pathParam := req.URL.Query().Get("path"); pathParam != "abc" {
t.Errorf("ListImages({Path:abc}): Wrong parameter. Want path=abc. Got path=%s", pathParam)
}
}
func TestDownloadFromContainer(t *testing.T) {
filecontent := "File content"
client := newTestClient(&FakeRoundTripper{message: filecontent, status: http.StatusOK})
var out bytes.Buffer
opts := DownloadFromContainerOptions{
OutputStream: &out,
}
err := client.DownloadFromContainer("a123456", opts)
if err != nil {
t.Errorf("DownloadFromContainer: caught error %#v while downloading from container, expected nil", err.Error())
}
if out.String() != filecontent {
t.Errorf("DownloadFromContainer: wrong stdout. Want %#v. Got %#v.", filecontent, out.String())
}
}
func TestCopyFromContainer(t *testing.T) {
content := "File content"
out := stdoutMock{bytes.NewBufferString(content)}
client := newTestClient(&FakeRoundTripper{status: http.StatusOK})
opts := CopyFromContainerOptions{
Container: "a123456",
OutputStream: out,
OutputStream: &out,
}
err := client.CopyFromContainer(opts)
if err != nil {
t.Errorf("CopyFromContainer: caugh error %#v while copying from container, expected nil", err.Error())
t.Errorf("CopyFromContainer: caught error %#v while copying from container, expected nil", err.Error())
}
if out.String() != content {
t.Errorf("CopyFromContainer: wrong stdout. Want %#v. Got %#v.", content, out.String())
@ -1587,7 +1628,6 @@ func TestTopContainerWithPsArgs(t *testing.T) {
}
func TestStatsTimeout(t *testing.T) {
l, err := net.Listen("unix", "/tmp/docker_test.sock")
if err != nil {
t.Fatal(err)
@ -1597,7 +1637,7 @@ func TestStatsTimeout(t *testing.T) {
go func() {
l.Accept()
received = true
time.Sleep(time.Millisecond * 250)
time.Sleep(time.Second)
}()
client, _ := NewClient("unix:///tmp/docker_test.sock")
client.SkipServerVersionCheck = true

5
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/event.go generated vendored
View File

@ -5,7 +5,6 @@
package docker
import (
"crypto/tls"
"encoding/json"
"errors"
"fmt"
@ -260,9 +259,9 @@ func (c *Client) eventHijack(startTime int64, eventChan chan *APIEvents, errChan
var dial net.Conn
var err error
if c.TLSConfig == nil {
dial, err = net.Dial(protocol, address)
dial, err = c.Dialer.Dial(protocol, address)
} else {
dial, err = tls.Dial(protocol, address, c.TLSConfig)
dial, err = tlsDialWithDialer(c.Dialer, protocol, address, c.TLSConfig)
}
if err != nil {
return err

151
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/exec.go generated vendored
View File

@ -2,8 +2,6 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Docs can currently be found at https://github.com/docker/docker/blob/master/docs/sources/reference/api/docker_remote_api_v1.15.md#exec-create
package docker
import (
@ -15,9 +13,15 @@ import (
"strconv"
)
// Exec is the type representing a `docker exec` instance and containing the
// instance ID
type Exec struct {
ID string `json:"Id,omitempty" yaml:"Id,omitempty"`
}
// CreateExecOptions specify parameters to the CreateExecContainer function.
//
// See http://goo.gl/8izrzI for more details
// See https://goo.gl/1KSIb7 for more details
type CreateExecOptions struct {
AttachStdin bool `json:"AttachStdin,omitempty" yaml:"AttachStdin,omitempty"`
AttachStdout bool `json:"AttachStdout,omitempty" yaml:"AttachStdout,omitempty"`
@ -28,9 +32,31 @@ type CreateExecOptions struct {
User string `json:"User,omitempty" yaml:"User,omitempty"`
}
// CreateExec sets up an exec instance in a running container `id`, returning the exec
// instance, or an error in case of failure.
//
// See https://goo.gl/1KSIb7 for more details
func (c *Client) CreateExec(opts CreateExecOptions) (*Exec, error) {
path := fmt.Sprintf("/containers/%s/exec", opts.Container)
resp, err := c.do("POST", path, doOptions{data: opts})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: opts.Container}
}
return nil, err
}
defer resp.Body.Close()
var exec Exec
if err := json.NewDecoder(resp.Body).Decode(&exec); err != nil {
return nil, err
}
return &exec, nil
}
// StartExecOptions specify parameters to the StartExecContainer function.
//
// See http://goo.gl/JW8Lxl for more details
// See https://goo.gl/iQCnto for more details
type StartExecOptions struct {
Detach bool `json:"Detach,omitempty" yaml:"Detach,omitempty"`
@ -51,67 +77,11 @@ type StartExecOptions struct {
Success chan struct{} `json:"-"`
}
// Exec is the type representing a `docker exec` instance and containing the
// instance ID
type Exec struct {
ID string `json:"Id,omitempty" yaml:"Id,omitempty"`
}
// ExecProcessConfig is a type describing the command associated to a Exec
// instance. It's used in the ExecInspect type.
//
// See http://goo.gl/ypQULN for more details
type ExecProcessConfig struct {
Privileged bool `json:"privileged,omitempty" yaml:"privileged,omitempty"`
User string `json:"user,omitempty" yaml:"user,omitempty"`
Tty bool `json:"tty,omitempty" yaml:"tty,omitempty"`
EntryPoint string `json:"entrypoint,omitempty" yaml:"entrypoint,omitempty"`
Arguments []string `json:"arguments,omitempty" yaml:"arguments,omitempty"`
}
// ExecInspect is a type with details about a exec instance, including the
// exit code if the command has finished running. It's returned by a api
// call to /exec/(id)/json
//
// See http://goo.gl/ypQULN for more details
type ExecInspect struct {
ID string `json:"ID,omitempty" yaml:"ID,omitempty"`
Running bool `json:"Running,omitempty" yaml:"Running,omitempty"`
ExitCode int `json:"ExitCode,omitempty" yaml:"ExitCode,omitempty"`
OpenStdin bool `json:"OpenStdin,omitempty" yaml:"OpenStdin,omitempty"`
OpenStderr bool `json:"OpenStderr,omitempty" yaml:"OpenStderr,omitempty"`
OpenStdout bool `json:"OpenStdout,omitempty" yaml:"OpenStdout,omitempty"`
ProcessConfig ExecProcessConfig `json:"ProcessConfig,omitempty" yaml:"ProcessConfig,omitempty"`
Container Container `json:"Container,omitempty" yaml:"Container,omitempty"`
}
// CreateExec sets up an exec instance in a running container `id`, returning the exec
// instance, or an error in case of failure.
//
// See http://goo.gl/8izrzI for more details
func (c *Client) CreateExec(opts CreateExecOptions) (*Exec, error) {
path := fmt.Sprintf("/containers/%s/exec", opts.Container)
body, status, err := c.do("POST", path, doOptions{data: opts})
if status == http.StatusNotFound {
return nil, &NoSuchContainer{ID: opts.Container}
}
if err != nil {
return nil, err
}
var exec Exec
err = json.Unmarshal(body, &exec)
if err != nil {
return nil, err
}
return &exec, nil
}
// StartExec starts a previously set up exec instance id. If opts.Detach is
// true, it returns after starting the exec command. Otherwise, it sets up an
// interactive session with the exec command.
//
// See http://goo.gl/JW8Lxl for more details
// See https://goo.gl/iQCnto for more details
func (c *Client) StartExec(id string, opts StartExecOptions) error {
if id == "" {
return &NoSuchExec{ID: id}
@ -120,13 +90,14 @@ func (c *Client) StartExec(id string, opts StartExecOptions) error {
path := fmt.Sprintf("/exec/%s/start", id)
if opts.Detach {
_, status, err := c.do("POST", path, doOptions{data: opts})
if status == http.StatusNotFound {
return &NoSuchExec{ID: id}
}
resp, err := c.do("POST", path, doOptions{data: opts})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return &NoSuchExec{ID: id}
}
return err
}
defer resp.Body.Close()
return nil
}
@ -144,32 +115,62 @@ func (c *Client) StartExec(id string, opts StartExecOptions) error {
// is valid only if Tty was specified as part of creating and starting the exec
// command.
//
// See http://goo.gl/YDSx1f for more details
// See https://goo.gl/e1JpsA for more details
func (c *Client) ResizeExecTTY(id string, height, width int) error {
params := make(url.Values)
params.Set("h", strconv.Itoa(height))
params.Set("w", strconv.Itoa(width))
path := fmt.Sprintf("/exec/%s/resize?%s", id, params.Encode())
_, _, err := c.do("POST", path, doOptions{})
return err
resp, err := c.do("POST", path, doOptions{})
if err != nil {
return err
}
resp.Body.Close()
return nil
}
// ExecProcessConfig is a type describing the command associated to a Exec
// instance. It's used in the ExecInspect type.
type ExecProcessConfig struct {
Privileged bool `json:"privileged,omitempty" yaml:"privileged,omitempty"`
User string `json:"user,omitempty" yaml:"user,omitempty"`
Tty bool `json:"tty,omitempty" yaml:"tty,omitempty"`
EntryPoint string `json:"entrypoint,omitempty" yaml:"entrypoint,omitempty"`
Arguments []string `json:"arguments,omitempty" yaml:"arguments,omitempty"`
}
// ExecInspect is a type with details about a exec instance, including the
// exit code if the command has finished running. It's returned by a api
// call to /exec/(id)/json
//
// See https://goo.gl/gPtX9R for more details
type ExecInspect struct {
ID string `json:"ID,omitempty" yaml:"ID,omitempty"`
Running bool `json:"Running,omitempty" yaml:"Running,omitempty"`
ExitCode int `json:"ExitCode,omitempty" yaml:"ExitCode,omitempty"`
OpenStdin bool `json:"OpenStdin,omitempty" yaml:"OpenStdin,omitempty"`
OpenStderr bool `json:"OpenStderr,omitempty" yaml:"OpenStderr,omitempty"`
OpenStdout bool `json:"OpenStdout,omitempty" yaml:"OpenStdout,omitempty"`
ProcessConfig ExecProcessConfig `json:"ProcessConfig,omitempty" yaml:"ProcessConfig,omitempty"`
Container Container `json:"Container,omitempty" yaml:"Container,omitempty"`
}
// InspectExec returns low-level information about the exec command id.
//
// See http://goo.gl/ypQULN for more details
// See https://goo.gl/gPtX9R for more details
func (c *Client) InspectExec(id string) (*ExecInspect, error) {
path := fmt.Sprintf("/exec/%s/json", id)
body, status, err := c.do("GET", path, doOptions{})
if status == http.StatusNotFound {
return nil, &NoSuchExec{ID: id}
}
resp, err := c.do("GET", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, &NoSuchExec{ID: id}
}
return nil, err
}
defer resp.Body.Close()
var exec ExecInspect
err = json.Unmarshal(body, &exec)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&exec); err != nil {
return nil, err
}
return &exec, nil

179
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/image.go generated vendored
View File

@ -11,7 +11,6 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"os"
@ -46,16 +45,6 @@ type Image struct {
VirtualSize int64 `json:"VirtualSize,omitempty" yaml:"VirtualSize,omitempty"`
}
// ImageHistory represent a layer in an image's history returned by the
// ImageHistory call.
type ImageHistory struct {
ID string `json:"Id" yaml:"Id"`
Tags []string `json:"Tags,omitempty" yaml:"Tags,omitempty"`
Created int64 `json:"Created,omitempty" yaml:"Created,omitempty"`
CreatedBy string `json:"CreatedBy,omitempty" yaml:"CreatedBy,omitempty"`
Size int64 `json:"Size,omitempty" yaml:"Size,omitempty"`
}
// ImagePre012 serves the same purpose as the Image type except that it is for
// earlier versions of the Docker API (pre-012 to be specific)
type ImagePre012 struct {
@ -72,15 +61,6 @@ type ImagePre012 struct {
Size int64 `json:"size,omitempty"`
}
// ListImagesOptions specify parameters to the ListImages function.
//
// See http://goo.gl/HRVN1Z for more details.
type ListImagesOptions struct {
All bool
Filters map[string][]string
Digests bool
}
var (
// ErrNoSuchImage is the error returned when the image does not exist.
ErrNoSuchImage = errors.New("no such image")
@ -102,37 +82,56 @@ var (
ErrMustSpecifyNames = errors.New("must specify at least one name to export")
)
// ListImagesOptions specify parameters to the ListImages function.
//
// See https://goo.gl/xBe1u3 for more details.
type ListImagesOptions struct {
All bool
Filters map[string][]string
Digests bool
}
// ListImages returns the list of available images in the server.
//
// See http://goo.gl/HRVN1Z for more details.
// See https://goo.gl/xBe1u3 for more details.
func (c *Client) ListImages(opts ListImagesOptions) ([]APIImages, error) {
path := "/images/json?" + queryString(opts)
body, _, err := c.do("GET", path, doOptions{})
resp, err := c.do("GET", path, doOptions{})
if err != nil {
return nil, err
}
defer resp.Body.Close()
var images []APIImages
err = json.Unmarshal(body, &images)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&images); err != nil {
return nil, err
}
return images, nil
}
// ImageHistory represent a layer in an image's history returned by the
// ImageHistory call.
type ImageHistory struct {
ID string `json:"Id" yaml:"Id"`
Tags []string `json:"Tags,omitempty" yaml:"Tags,omitempty"`
Created int64 `json:"Created,omitempty" yaml:"Created,omitempty"`
CreatedBy string `json:"CreatedBy,omitempty" yaml:"CreatedBy,omitempty"`
Size int64 `json:"Size,omitempty" yaml:"Size,omitempty"`
}
// ImageHistory returns the history of the image by its name or ID.
//
// See http://goo.gl/2oJmNs for more details.
// See https://goo.gl/8bnTId for more details.
func (c *Client) ImageHistory(name string) ([]ImageHistory, error) {
body, status, err := c.do("GET", "/images/"+name+"/history", doOptions{})
if status == http.StatusNotFound {
return nil, ErrNoSuchImage
}
resp, err := c.do("GET", "/images/"+name+"/history", doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, ErrNoSuchImage
}
return nil, err
}
defer resp.Body.Close()
var history []ImageHistory
err = json.Unmarshal(body, &history)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&history); err != nil {
return nil, err
}
return history, nil
@ -140,19 +139,23 @@ func (c *Client) ImageHistory(name string) ([]ImageHistory, error) {
// RemoveImage removes an image by its name or ID.
//
// See http://goo.gl/znj0wM for more details.
// See https://goo.gl/V3ZWnK for more details.
func (c *Client) RemoveImage(name string) error {
_, status, err := c.do("DELETE", "/images/"+name, doOptions{})
if status == http.StatusNotFound {
return ErrNoSuchImage
resp, err := c.do("DELETE", "/images/"+name, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return ErrNoSuchImage
}
return err
}
return err
resp.Body.Close()
return nil
}
// RemoveImageOptions present the set of options available for removing an image
// from a registry.
//
// See http://goo.gl/6V48bF for more details.
// See https://goo.gl/V3ZWnK for more details.
type RemoveImageOptions struct {
Force bool `qs:"force"`
NoPrune bool `qs:"noprune"`
@ -161,40 +164,43 @@ type RemoveImageOptions struct {
// RemoveImageExtended removes an image by its name or ID.
// Extra params can be passed, see RemoveImageOptions
//
// See http://goo.gl/znj0wM for more details.
// See https://goo.gl/V3ZWnK for more details.
func (c *Client) RemoveImageExtended(name string, opts RemoveImageOptions) error {
uri := fmt.Sprintf("/images/%s?%s", name, queryString(&opts))
_, status, err := c.do("DELETE", uri, doOptions{})
if status == http.StatusNotFound {
return ErrNoSuchImage
resp, err := c.do("DELETE", uri, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return ErrNoSuchImage
}
return err
}
return err
resp.Body.Close()
return nil
}
// InspectImage returns an image by its name or ID.
//
// See http://goo.gl/Q112NY for more details.
// See https://goo.gl/jHPcg6 for more details.
func (c *Client) InspectImage(name string) (*Image, error) {
body, status, err := c.do("GET", "/images/"+name+"/json", doOptions{})
if status == http.StatusNotFound {
return nil, ErrNoSuchImage
}
resp, err := c.do("GET", "/images/"+name+"/json", doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, ErrNoSuchImage
}
return nil, err
}
defer resp.Body.Close()
var image Image
// if the caller elected to skip checking the server's version, assume it's the latest
if c.SkipServerVersionCheck || c.expectedAPIVersion.GreaterThanOrEqualTo(apiVersion112) {
err = json.Unmarshal(body, &image)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&image); err != nil {
return nil, err
}
} else {
var imagePre012 ImagePre012
err = json.Unmarshal(body, &imagePre012)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&imagePre012); err != nil {
return nil, err
}
@ -216,7 +222,7 @@ func (c *Client) InspectImage(name string) (*Image, error) {
// PushImageOptions represents options to use in the PushImage method.
//
// See http://goo.gl/pN8A3P for more details.
// See https://goo.gl/zPtZaT for more details.
type PushImageOptions struct {
// Name of the image
Name string
@ -236,7 +242,7 @@ type PushImageOptions struct {
// An empty instance of AuthConfiguration may be used for unauthenticated
// pushes.
//
// See http://goo.gl/pN8A3P for more details.
// See https://goo.gl/zPtZaT for more details.
func (c *Client) PushImage(opts PushImageOptions, auth AuthConfiguration) error {
if opts.Name == "" {
return ErrNoSuchImage
@ -259,7 +265,7 @@ func (c *Client) PushImage(opts PushImageOptions, auth AuthConfiguration) error
// PullImageOptions present the set of options available for pulling an image
// from a registry.
//
// See http://goo.gl/ACyYNS for more details.
// See https://goo.gl/iJkZjD for more details.
type PullImageOptions struct {
Repository string `qs:"fromImage"`
Registry string
@ -268,9 +274,10 @@ type PullImageOptions struct {
RawJSONStream bool `qs:"-"`
}
// PullImage pulls an image from a remote registry, logging progress to opts.OutputStream.
// PullImage pulls an image from a remote registry, logging progress to
// opts.OutputStream.
//
// See http://goo.gl/ACyYNS for more details.
// See https://goo.gl/iJkZjD for more details.
func (c *Client) PullImage(opts PullImageOptions, auth AuthConfiguration) error {
if opts.Repository == "" {
return ErrNoSuchImage
@ -296,14 +303,14 @@ func (c *Client) createImage(qs string, headers map[string]string, in io.Reader,
// LoadImageOptions represents the options for LoadImage Docker API Call
//
// See http://goo.gl/Y8NNCq for more details.
// See https://goo.gl/JyClMX for more details.
type LoadImageOptions struct {
InputStream io.Reader
}
// LoadImage imports a tarball docker image
//
// See http://goo.gl/Y8NNCq for more details.
// See https://goo.gl/JyClMX for more details.
func (c *Client) LoadImage(opts LoadImageOptions) error {
return c.stream("POST", "/images/load", streamOptions{
setRawTerminal: true,
@ -311,17 +318,17 @@ func (c *Client) LoadImage(opts LoadImageOptions) error {
})
}
// ExportImageOptions represent the options for ExportImage Docker API call
// ExportImageOptions represent the options for ExportImage Docker API call.
//
// See http://goo.gl/mi6kvk for more details.
// See https://goo.gl/le7vK8 for more details.
type ExportImageOptions struct {
Name string
OutputStream io.Writer
}
// ExportImage exports an image (as a tar file) into the stream
// ExportImage exports an image (as a tar file) into the stream.
//
// See http://goo.gl/mi6kvk for more details.
// See https://goo.gl/le7vK8 for more details.
func (c *Client) ExportImage(opts ExportImageOptions) error {
return c.stream("GET", fmt.Sprintf("/images/%s/get", opts.Name), streamOptions{
setRawTerminal: true,
@ -331,7 +338,7 @@ func (c *Client) ExportImage(opts ExportImageOptions) error {
// ExportImagesOptions represent the options for ExportImages Docker API call
//
// See http://goo.gl/YeZzQK for more details.
// See https://goo.gl/huC7HA for more details.
type ExportImagesOptions struct {
Names []string
OutputStream io.Writer `qs:"-"`
@ -339,7 +346,7 @@ type ExportImagesOptions struct {
// ExportImages exports one or more images (as a tar file) into the stream
//
// See http://goo.gl/YeZzQK for more details.
// See https://goo.gl/huC7HA for more details.
func (c *Client) ExportImages(opts ExportImagesOptions) error {
if opts.Names == nil || len(opts.Names) == 0 {
return ErrMustSpecifyNames
@ -353,7 +360,7 @@ func (c *Client) ExportImages(opts ExportImagesOptions) error {
// ImportImageOptions present the set of informations available for importing
// an image from a source file or the stdin.
//
// See http://goo.gl/PhBKnS for more details.
// See https://goo.gl/iJkZjD for more details.
type ImportImageOptions struct {
Repository string `qs:"repo"`
Source string `qs:"fromSrc"`
@ -366,7 +373,7 @@ type ImportImageOptions struct {
// ImportImage imports an image from a url, a file or stdin
//
// See http://goo.gl/PhBKnS for more details.
// See https://goo.gl/iJkZjD for more details.
func (c *Client) ImportImage(opts ImportImageOptions) error {
if opts.Repository == "" {
return ErrNoSuchImage
@ -379,8 +386,7 @@ func (c *Client) ImportImage(opts ImportImageOptions) error {
if err != nil {
return err
}
b, err := ioutil.ReadAll(f)
opts.InputStream = bytes.NewBuffer(b)
opts.InputStream = f
opts.Source = "-"
}
return c.createImage(queryString(&opts), nil, opts.InputStream, opts.OutputStream, opts.RawJSONStream)
@ -415,12 +421,12 @@ type BuildImageOptions struct {
// BuildImage builds an image from a tarball's url or a Dockerfile in the input
// stream.
//
// See http://goo.gl/7nuGXa for more details.
// See https://goo.gl/xySxCe for more details.
func (c *Client) BuildImage(opts BuildImageOptions) error {
if opts.OutputStream == nil {
return ErrMissingOutputStream
}
headers, err := headersWithAuth(opts.Auth, opts.AuthConfigs)
headers, err := headersWithAuth(opts.Auth, c.versionedAuthConfigs(opts.AuthConfigs))
if err != nil {
return err
}
@ -452,9 +458,19 @@ func (c *Client) BuildImage(opts BuildImageOptions) error {
})
}
func (c *Client) versionedAuthConfigs(authConfigs AuthConfigurations) interface{} {
if c.serverAPIVersion == nil {
c.checkAPIVersion()
}
if c.serverAPIVersion != nil && c.serverAPIVersion.GreaterThanOrEqualTo(apiVersion119) {
return AuthConfigurations119(authConfigs.Configs)
}
return authConfigs
}
// TagImageOptions present the set of options to tag an image.
//
// See http://goo.gl/5g6qFy for more details.
// See https://goo.gl/98ZzkU for more details.
type TagImageOptions struct {
Repo string
Tag string
@ -463,15 +479,16 @@ type TagImageOptions struct {
// TagImage adds a tag to the image identified by the given name.
//
// See http://goo.gl/5g6qFy for more details.
// See https://goo.gl/98ZzkU for more details.
func (c *Client) TagImage(name string, opts TagImageOptions) error {
if name == "" {
return ErrNoSuchImage
}
_, status, err := c.do("POST", fmt.Sprintf("/images/"+name+"/tag?%s",
resp, err := c.do("POST", fmt.Sprintf("/images/"+name+"/tag?%s",
queryString(&opts)), doOptions{})
defer resp.Body.Close()
if status == http.StatusNotFound {
if resp.StatusCode == http.StatusNotFound {
return ErrNoSuchImage
}
@ -497,7 +514,7 @@ func headersWithAuth(auths ...interface{}) (map[string]string, error) {
return nil, err
}
headers["X-Registry-Auth"] = base64.URLEncoding.EncodeToString(buf.Bytes())
case AuthConfigurations:
case AuthConfigurations, AuthConfigurations119:
var buf bytes.Buffer
if err := json.NewEncoder(&buf).Encode(auth); err != nil {
return nil, err
@ -509,9 +526,9 @@ func headersWithAuth(auths ...interface{}) (map[string]string, error) {
return headers, nil
}
// APIImageSearch reflect the result of a search on the dockerHub
// APIImageSearch reflect the result of a search on the Docker Hub.
//
// See http://goo.gl/xI5lLZ for more details.
// See https://goo.gl/AYjyrF for more details.
type APIImageSearch struct {
Description string `json:"description,omitempty" yaml:"description,omitempty"`
IsOfficial bool `json:"is_official,omitempty" yaml:"is_official,omitempty"`
@ -522,15 +539,15 @@ type APIImageSearch struct {
// SearchImages search the docker hub with a specific given term.
//
// See http://goo.gl/xI5lLZ for more details.
// See https://goo.gl/AYjyrF for more details.
func (c *Client) SearchImages(term string) ([]APIImageSearch, error) {
body, _, err := c.do("GET", "/images/search?term="+term, doOptions{})
resp, err := c.do("GET", "/images/search?term="+term, doOptions{})
defer resp.Body.Close()
if err != nil {
return nil, err
}
var searchResult []APIImageSearch
err = json.Unmarshal(body, &searchResult)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&searchResult); err != nil {
return nil, err
}
return searchResult, nil

4
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/image_test.go generated vendored
View File

@ -9,6 +9,7 @@ import (
"encoding/base64"
"encoding/json"
"io/ioutil"
"net"
"net/http"
"net/url"
"os"
@ -21,11 +22,14 @@ import (
func newTestClient(rt *FakeRoundTripper) Client {
endpoint := "http://localhost:4243"
u, _ := parseEndpoint("http://localhost:4243", false)
testAPIVersion, _ := NewAPIVersion("1.17")
client := Client{
HTTPClient: &http.Client{Transport: rt},
Dialer: &net.Dialer{},
endpoint: endpoint,
endpointURL: u,
SkipServerVersionCheck: true,
serverAPIVersion: testAPIVersion,
}
return client
}

20
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/misc.go generated vendored
View File

@ -4,21 +4,19 @@
package docker
import (
"bytes"
"strings"
)
import "strings"
// Version returns version information about the docker server.
//
// See http://goo.gl/BOZrF5 for more details.
// See https://goo.gl/ND9R8L for more details.
func (c *Client) Version() (*Env, error) {
body, _, err := c.do("GET", "/version", doOptions{})
resp, err := c.do("GET", "/version", doOptions{})
if err != nil {
return nil, err
}
defer resp.Body.Close()
var env Env
if err := env.Decode(bytes.NewReader(body)); err != nil {
if err := env.Decode(resp.Body); err != nil {
return nil, err
}
return &env, nil
@ -26,15 +24,15 @@ func (c *Client) Version() (*Env, error) {
// Info returns system-wide information about the Docker server.
//
// See http://goo.gl/wmqZsW for more details.
// See https://goo.gl/ElTHi2 for more details.
func (c *Client) Info() (*Env, error) {
body, _, err := c.do("GET", "/info", doOptions{})
resp, err := c.do("GET", "/info", doOptions{})
if err != nil {
return nil, err
}
defer resp.Body.Close()
var info Env
err = info.Decode(bytes.NewReader(body))
if err != nil {
if err := info.Decode(resp.Body); err != nil {
return nil, err
}
return &info, nil

33
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/network.go generated vendored
View File

@ -38,12 +38,13 @@ type Endpoint struct {
//
// See https://goo.gl/4hCNtZ for more details.
func (c *Client) ListNetworks() ([]Network, error) {
body, _, err := c.do("GET", "/networks", doOptions{})
resp, err := c.do("GET", "/networks", doOptions{})
if err != nil {
return nil, err
}
defer resp.Body.Close()
var networks []Network
if err := json.Unmarshal(body, &networks); err != nil {
if err := json.NewDecoder(resp.Body).Decode(&networks); err != nil {
return nil, err
}
return networks, nil
@ -54,15 +55,16 @@ func (c *Client) ListNetworks() ([]Network, error) {
// See https://goo.gl/4hCNtZ for more details.
func (c *Client) NetworkInfo(id string) (*Network, error) {
path := "/networks/" + id
body, status, err := c.do("GET", path, doOptions{})
if status == http.StatusNotFound {
return nil, &NoSuchNetwork{ID: id}
}
resp, err := c.do("GET", path, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, &NoSuchNetwork{ID: id}
}
return nil, err
}
defer resp.Body.Close()
var network Network
if err := json.Unmarshal(body, &network); err != nil {
if err := json.NewDecoder(resp.Body).Decode(&network); err != nil {
return nil, err
}
return &network, nil
@ -83,35 +85,34 @@ type CreateNetworkOptions struct {
//
// See http://goo.gl/mErxNp for more details.
func (c *Client) CreateNetwork(opts CreateNetworkOptions) (*Network, error) {
body, status, err := c.do(
resp, err := c.do(
"POST",
"/networks",
doOptions{
data: opts,
},
)
if status == http.StatusConflict {
return nil, ErrNetworkAlreadyExists
}
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusConflict {
return nil, ErrNetworkAlreadyExists
}
return nil, err
}
defer resp.Body.Close()
type createNetworkResponse struct {
ID string
}
var (
network Network
resp createNetworkResponse
cnr createNetworkResponse
)
err = json.Unmarshal(body, &resp)
if err != nil {
if err := json.NewDecoder(resp.Body).Decode(&cnr); err != nil {
return nil, err
}
network.Name = opts.Name
network.ID = resp.ID
network.ID = cnr.ID
network.Type = opts.NetworkType
return &network, nil

1
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/data/symlink generated vendored
View File

@ -1 +0,0 @@
doesnotexist

39
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/server.go generated vendored
View File

@ -12,6 +12,7 @@ import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
mathrand "math/rand"
"net"
"net/http"
@ -532,7 +533,7 @@ func (s *DockerServer) startContainer(w http.ResponseWriter, r *http.Request) {
}
container.HostConfig = &hostConfig
if container.State.Running {
http.Error(w, "Container already running", http.StatusBadRequest)
http.Error(w, "", http.StatusNotModified)
return
}
container.State.Running = true
@ -610,14 +611,34 @@ func (s *DockerServer) attachContainer(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
wg := sync.WaitGroup{}
if r.URL.Query().Get("stdin") == "1" {
wg.Add(1)
go func() {
ioutil.ReadAll(conn)
wg.Done()
}()
}
outStream := stdcopy.NewStdWriter(conn, stdcopy.Stdout)
if container.State.Running {
fmt.Fprintf(outStream, "Container %q is running\n", container.ID)
fmt.Fprintf(outStream, "Container is running\n")
} else {
fmt.Fprintf(outStream, "Container %q is not running\n", container.ID)
fmt.Fprintf(outStream, "Container is not running\n")
}
fmt.Fprintln(outStream, "What happened?")
fmt.Fprintln(outStream, "Something happened")
wg.Wait()
if r.URL.Query().Get("stream") == "1" {
for {
time.Sleep(1e6)
s.cMut.RLock()
if !container.State.Running {
s.cMut.RUnlock()
break
}
s.cMut.RUnlock()
}
}
conn.Close()
}
@ -936,7 +957,7 @@ func (s *DockerServer) createExecContainer(w http.ResponseWriter, r *http.Reques
func (s *DockerServer) startExecContainer(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
if exec, err := s.getExec(id); err == nil {
if exec, err := s.getExec(id, false); err == nil {
s.execMut.Lock()
exec.Running = true
s.execMut.Unlock()
@ -958,7 +979,7 @@ func (s *DockerServer) startExecContainer(w http.ResponseWriter, r *http.Request
func (s *DockerServer) resizeExecContainer(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
if _, err := s.getExec(id); err == nil {
if _, err := s.getExec(id, false); err == nil {
w.WriteHeader(http.StatusOK)
return
}
@ -967,7 +988,7 @@ func (s *DockerServer) resizeExecContainer(w http.ResponseWriter, r *http.Reques
func (s *DockerServer) inspectExecContainer(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
if exec, err := s.getExec(id); err == nil {
if exec, err := s.getExec(id, true); err == nil {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(exec)
@ -976,11 +997,15 @@ func (s *DockerServer) inspectExecContainer(w http.ResponseWriter, r *http.Reque
w.WriteHeader(http.StatusNotFound)
}
func (s *DockerServer) getExec(id string) (*docker.ExecInspect, error) {
func (s *DockerServer) getExec(id string, copy bool) (*docker.ExecInspect, error) {
s.execMut.RLock()
defer s.execMut.RUnlock()
for _, exec := range s.execs {
if exec.ID == id {
if copy {
cp := *exec
exec = &cp
}
return exec, nil
}
}

77
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/server_test.go generated vendored
View File

@ -5,9 +5,11 @@
package testing
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"math/rand"
"net"
"net/http"
@ -624,8 +626,8 @@ func TestStartContainerAlreadyRunning(t *testing.T) {
path := fmt.Sprintf("/containers/%s/start", server.containers[0].ID)
request, _ := http.NewRequest("POST", path, bytes.NewBuffer([]byte("null")))
server.ServeHTTP(recorder, request)
if recorder.Code != http.StatusBadRequest {
t.Errorf("StartContainer: wrong status code. Want %d. Got %d.", http.StatusBadRequest, recorder.Code)
if recorder.Code != http.StatusNotModified {
t.Errorf("StartContainer: wrong status code. Want %d. Got %d.", http.StatusNotModified, recorder.Code)
}
}
@ -845,22 +847,41 @@ func TestWaitContainerNotFound(t *testing.T) {
}
}
type HijackableResponseRecorder struct {
httptest.ResponseRecorder
readCh chan []byte
}
func (r *HijackableResponseRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) {
myConn, otherConn := net.Pipe()
r.readCh = make(chan []byte)
go func() {
data, _ := ioutil.ReadAll(myConn)
r.readCh <- data
}()
return otherConn, nil, nil
}
func (r *HijackableResponseRecorder) HijackBuffer() string {
return string(<-r.readCh)
}
func TestAttachContainer(t *testing.T) {
server := DockerServer{}
addContainers(&server, 1)
server.containers[0].State.Running = true
server.buildMuxer()
recorder := httptest.NewRecorder()
recorder := &HijackableResponseRecorder{}
path := fmt.Sprintf("/containers/%s/attach?logs=1", server.containers[0].ID)
request, _ := http.NewRequest("POST", path, nil)
server.ServeHTTP(recorder, request)
lines := []string{
fmt.Sprintf("\x01\x00\x00\x00\x03\x00\x00\x00Container %q is running", server.containers[0].ID),
"What happened?",
"Something happened",
"\x01\x00\x00\x00\x00\x00\x00\x15Container is running",
"\x01\x00\x00\x00\x00\x00\x00\x0fWhat happened?",
"\x01\x00\x00\x00\x00\x00\x00\x13Something happened",
}
expected := strings.Join(lines, "\n") + "\n"
if body := recorder.Body.String(); body == expected {
if body := recorder.HijackBuffer(); body != expected {
t.Errorf("AttachContainer: wrong body. Want %q. Got %q.", expected, body)
}
}
@ -868,7 +889,7 @@ func TestAttachContainer(t *testing.T) {
func TestAttachContainerNotFound(t *testing.T) {
server := DockerServer{}
server.buildMuxer()
recorder := httptest.NewRecorder()
recorder := &HijackableResponseRecorder{}
path := "/containers/abc123/attach?logs=1"
request, _ := http.NewRequest("POST", path, nil)
server.ServeHTTP(recorder, request)
@ -877,6 +898,44 @@ func TestAttachContainerNotFound(t *testing.T) {
}
}
func TestAttachContainerWithStreamBlocks(t *testing.T) {
server := DockerServer{}
addContainers(&server, 1)
server.containers[0].State.Running = true
server.buildMuxer()
path := fmt.Sprintf("/containers/%s/attach?logs=1&stdout=1&stream=1", server.containers[0].ID)
request, _ := http.NewRequest("POST", path, nil)
done := make(chan string)
go func() {
recorder := &HijackableResponseRecorder{}
server.ServeHTTP(recorder, request)
done <- recorder.HijackBuffer()
}()
select {
case <-done:
t.Fatalf("attach stream returned before container is stopped")
case <-time.After(500 * time.Millisecond):
}
server.cMut.Lock()
server.containers[0].State.Running = false
server.cMut.Unlock()
var body string
select {
case body = <-done:
case <-time.After(5 * time.Second):
t.Fatalf("timed out waiting for attach to finish")
}
lines := []string{
"\x01\x00\x00\x00\x00\x00\x00\x15Container is running",
"\x01\x00\x00\x00\x00\x00\x00\x0fWhat happened?",
"\x01\x00\x00\x00\x00\x00\x00\x13Something happened",
}
expected := strings.Join(lines, "\n") + "\n"
if body != expected {
t.Errorf("AttachContainer: wrong body. Want %q. Got %q.", expected, body)
}
}
func TestRemoveContainer(t *testing.T) {
server := DockerServer{}
addContainers(&server, 1)
@ -1690,7 +1749,7 @@ func addNetworks(server *DockerServer, n int) {
ID: fmt.Sprintf("%x", rand.Int()%10000),
Type: "bridge",
Endpoints: []*docker.Endpoint{
&docker.Endpoint{
{
Name: "blah",
ID: fmt.Sprintf("%x", rand.Int()%10000),
Network: netid,

4
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/tls.go generated vendored
View File

@ -94,7 +94,3 @@ func tlsDialWithDialer(dialer *net.Dialer, network, addr string, config *tls.Con
// wrapper which holds both the TLS and raw connections.
return &tlsClientCon{conn, rawConn}, nil
}
func tlsDial(network, addr string, config *tls.Config) (net.Conn, error) {
return tlsDialWithDialer(new(net.Dialer), network, addr, config)
}

127
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/volume.go generated vendored Normal file
View File

@ -0,0 +1,127 @@
// Copyright 2015 go-dockerclient authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package docker
import (
"encoding/json"
"errors"
"net/http"
)
var (
// ErrNoSuchVolume is the error returned when the volume does not exist.
ErrNoSuchVolume = errors.New("no such volume")
// ErrVolumeInUse is the error returned when the volume requested to be removed is still in use.
ErrVolumeInUse = errors.New("volume in use and cannot be removed")
)
// Volume represents a volume.
//
// See https://goo.gl/FZA4BK for more details.
type Volume struct {
Name string `json:"Name" yaml:"Name"`
Driver string `json:"Driver,omitempty" yaml:"Driver,omitempty"`
Mountpoint string `json:"Mountpoint,omitempty" yaml:"Mountpoint,omitempty"`
}
// ListVolumesOptions specify parameters to the ListVolumes function.
//
// See https://goo.gl/FZA4BK for more details.
type ListVolumesOptions struct {
Filters map[string][]string
}
// ListVolumes returns a list of available volumes in the server.
//
// See https://goo.gl/FZA4BK for more details.
func (c *Client) ListVolumes(opts ListVolumesOptions) ([]Volume, error) {
resp, err := c.do("GET", "/volumes?"+queryString(opts), doOptions{})
if err != nil {
return nil, err
}
defer resp.Body.Close()
m := make(map[string]interface{})
if err := json.NewDecoder(resp.Body).Decode(&m); err != nil {
return nil, err
}
var volumes []Volume
volumesJSON, ok := m["Volumes"]
if !ok {
return volumes, nil
}
data, err := json.Marshal(volumesJSON)
if err != nil {
return nil, err
}
if err := json.Unmarshal(data, &volumes); err != nil {
return nil, err
}
return volumes, nil
}
// CreateVolumeOptions specify parameters to the CreateVolume function.
//
// See https://goo.gl/pBUbZ9 for more details.
type CreateVolumeOptions struct {
Name string
Driver string
DriverOpts map[string]string
}
// CreateVolume creates a volume on the server.
//
// See https://goo.gl/pBUbZ9 for more details.
func (c *Client) CreateVolume(opts CreateVolumeOptions) (*Volume, error) {
resp, err := c.do("POST", "/volumes", doOptions{data: opts})
if err != nil {
return nil, err
}
defer resp.Body.Close()
var volume Volume
if err := json.NewDecoder(resp.Body).Decode(&volume); err != nil {
return nil, err
}
return &volume, nil
}
// InspectVolume returns a volume by its name.
//
// See https://goo.gl/0g9A6i for more details.
func (c *Client) InspectVolume(name string) (*Volume, error) {
resp, err := c.do("GET", "/volumes/"+name, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok && e.Status == http.StatusNotFound {
return nil, ErrNoSuchVolume
}
return nil, err
}
defer resp.Body.Close()
var volume Volume
if err := json.NewDecoder(resp.Body).Decode(&volume); err != nil {
return nil, err
}
return &volume, nil
}
// RemoveVolume removes a volume by its name.
//
// See https://goo.gl/79GNQz for more details.
func (c *Client) RemoveVolume(name string) error {
resp, err := c.do("DELETE", "/volumes/"+name, doOptions{})
if err != nil {
if e, ok := err.(*Error); ok {
if e.Status == http.StatusNotFound {
return ErrNoSuchVolume
}
if e.Status == http.StatusConflict {
return ErrVolumeInUse
}
}
return nil
}
defer resp.Body.Close()
return nil
}

142
Godeps/_workspace/src/github.com/fsouza/go-dockerclient/volume_test.go generated vendored Normal file
View File

@ -0,0 +1,142 @@
// Copyright 2015 go-dockerclient authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package docker
import (
"encoding/json"
"net/http"
"net/url"
"reflect"
"testing"
)
func TestListVolumes(t *testing.T) {
volumesData := `[
{
"Name": "tardis",
"Driver": "local",
"Mountpoint": "/var/lib/docker/volumes/tardis"
},
{
"Name": "foo",
"Driver": "bar",
"Mountpoint": "/var/lib/docker/volumes/bar"
}
]`
body := `{ "Volumes": ` + volumesData + ` }`
var expected []Volume
if err := json.Unmarshal([]byte(volumesData), &expected); err != nil {
t.Fatal(err)
}
client := newTestClient(&FakeRoundTripper{message: body, status: http.StatusOK})
volumes, err := client.ListVolumes(ListVolumesOptions{})
if err != nil {
t.Error(err)
}
if !reflect.DeepEqual(volumes, expected) {
t.Errorf("ListVolumes: Wrong return value. Want %#v. Got %#v.", expected, volumes)
}
}
func TestCreateVolume(t *testing.T) {
body := `{
"Name": "tardis",
"Driver": "local",
"Mountpoint": "/var/lib/docker/volumes/tardis"
}`
var expected Volume
if err := json.Unmarshal([]byte(body), &expected); err != nil {
t.Fatal(err)
}
fakeRT := &FakeRoundTripper{message: body, status: http.StatusOK}
client := newTestClient(fakeRT)
volume, err := client.CreateVolume(
CreateVolumeOptions{
Name: "tardis",
Driver: "local",
DriverOpts: map[string]string{
"foo": "bar",
},
},
)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(volume, &expected) {
t.Errorf("CreateVolume: Wrong return value. Want %#v. Got %#v.", expected, volume)
}
req := fakeRT.requests[0]
expectedMethod := "POST"
if req.Method != expectedMethod {
t.Errorf("CreateVolume(): Wrong HTTP method. Want %s. Got %s.", expectedMethod, req.Method)
}
u, _ := url.Parse(client.getURL("/volumes"))
if req.URL.Path != u.Path {
t.Errorf("CreateVolume(): Wrong request path. Want %q. Got %q.", u.Path, req.URL.Path)
}
}
func TestInspectVolume(t *testing.T) {
body := `{
"Name": "tardis",
"Driver": "local",
"Mountpoint": "/var/lib/docker/volumes/tardis"
}`
var expected Volume
if err := json.Unmarshal([]byte(body), &expected); err != nil {
t.Fatal(err)
}
fakeRT := &FakeRoundTripper{message: body, status: http.StatusOK}
client := newTestClient(fakeRT)
name := "tardis"
volume, err := client.InspectVolume(name)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(volume, &expected) {
t.Errorf("InspectVolume: Wrong return value. Want %#v. Got %#v.", expected, volume)
}
req := fakeRT.requests[0]
expectedMethod := "GET"
if req.Method != expectedMethod {
t.Errorf("InspectVolume(%q): Wrong HTTP method. Want %s. Got %s.", name, expectedMethod, req.Method)
}
u, _ := url.Parse(client.getURL("/volumes/" + name))
if req.URL.Path != u.Path {
t.Errorf("CreateVolume(%q): Wrong request path. Want %q. Got %q.", name, u.Path, req.URL.Path)
}
}
func TestRemoveVolume(t *testing.T) {
name := "test"
fakeRT := &FakeRoundTripper{message: "", status: http.StatusNoContent}
client := newTestClient(fakeRT)
if err := client.RemoveVolume(name); err != nil {
t.Fatal(err)
}
req := fakeRT.requests[0]
expectedMethod := "DELETE"
if req.Method != expectedMethod {
t.Errorf("RemoveVolume(%q): Wrong HTTP method. Want %s. Got %s.", name, expectedMethod, req.Method)
}
u, _ := url.Parse(client.getURL("/volumes/" + name))
if req.URL.Path != u.Path {
t.Errorf("RemoveVolume(%q): Wrong request path. Want %q. Got %q.", name, u.Path, req.URL.Path)
}
}
func TestRemoveVolumeNotFound(t *testing.T) {
client := newTestClient(&FakeRoundTripper{message: "no such volume", status: http.StatusNotFound})
if err := client.RemoveVolume("test:"); err != ErrNoSuchVolume {
t.Errorf("RemoveVolume: wrong error. Want %#v. Got %#v.", ErrNoSuchVolume, err)
}
}
func TestRemoveVolumeInUse(t *testing.T) {
client := newTestClient(&FakeRoundTripper{message: "volume in use and cannot be removed", status: http.StatusConflict})
if err := client.RemoveVolume("test:"); err != ErrVolumeInUse {
t.Errorf("RemoveVolume: wrong error. Want %#v. Got %#v.", ErrVolumeInUse, err)
}
}

67
Godeps/_workspace/src/github.com/influxdb/influxdb/.gitignore generated vendored Normal file
View File

@ -0,0 +1,67 @@
*~
src/
config.json
/bin/
/pkg/
TAGS
# vim temp files
*.swp
*.test
/query/a.out*
.DS_Store
# ignore generated files.
cmd/influxd/version.go
# executables
influxd
**/influxd
!**/influxd/
influx
**/influx
!**/influx/
influxdb
**/influxdb
!**/influxdb/
/benchmark-tool
/main
/benchmark-storage
godef
gosym
gocode
inspect-raft
# dependencies
out_rpm/
packages/
# autconf
autom4te.cache/
config.log
config.status
Makefile
# log file
influxdb.log
benchmark.log
# config file
config.toml
# test data files
integration/migration_data/
# goide project files
.idea
# goconvey config files
*.goconvey

1607
Godeps/_workspace/src/github.com/influxdb/influxdb/CHANGELOG.md generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

231
Godeps/_workspace/src/github.com/influxdb/influxdb/CONTRIBUTING.md generated vendored Normal file
View File

@ -0,0 +1,231 @@
Contributing to InfluxDB
========================
Bug reports
---------------
Before you file an issue, please search existing issues in case it has already been filed, or perhaps even fixed. If you file an issue, please include the following.
* Full details of your operating system (or distribution) e.g. 64-bit Ubuntu 14.04.
* The version of InfluxDB you are running
* Whether you installed it using a pre-built package, or built it from source.
* A small test case, if applicable, that demonstrates the issues.
Remember the golden rule of bug reports: **The easier you make it for us to reproduce the problem, the faster it will get fixed.**
If you have never written a bug report before, or if you want to brush up on your bug reporting skills, we recommend reading [Simon Tatham's essay "How to Report Bugs Effectively."](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
Test cases should be in the form of `curl` commands. For example:
```
# create database
curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"
# create retention policy
curl -G http://localhost:8086/query --data-urlencode "q=CREATE RETENTION POLICY myrp ON mydb DURATION 365d REPLICATION 1 DEFAULT"
# write data
curl -X POST http://localhost:8086/write --data-urlencode "db=mydb" --data-binary "cpu,region=useast,host=server_1,service=redis value=61"
# Delete a Measurement
curl -G http://localhost:8086/query --data-urlencode 'db=mydb' --data-urlencode 'q=DROP MEASUREMENT cpu'
# Query the Measurement
# Bug: expected it to return no data, but data comes back.
curl -G http://localhost:8086/query --data-urlencode 'db=mydb' --data-urlencode 'q=SELECT * from cpu'
```
**If you don't include a clear test case like this, your issue may not be investigated, and may even be closed**. If writing the data is too difficult, please zip up your data directory and include a link to it in your bug report.
Please note that issues are *not the place to file general questions* such as "how do I use collectd with InfluxDB?" Questions of this nature should be sent to the [Google Group](https://groups.google.com/forum/#!forum/influxdb), not filed as issues. Issues like this will be closed.
Feature requests
---------------
We really like to receive feature requests, as it helps us prioritize our work. Please be clear about your requirements, as incomplete feature requests may simply be closed if we don't understand what you would like to see added to InfluxDB.
Contributing to the source code
---------------
InfluxDB follows standard Go project structure. This means that all
your go development are done in `$GOPATH/src`. GOPATH can be any
directory under which InfluxDB and all its dependencies will be
cloned. For more details on recommended go project's structure, see
[How to Write Go Code](http://golang.org/doc/code.html) and
[Go: Best Practices for Production Environments](http://peter.bourgon.org/go-in-production/), or you can just follow
the steps below.
Submitting a pull request
------------
To submit a pull request you should fork the InfluxDB repository, and make your change on a feature branch of your fork. Then generate a pull request from your branch against *master* of the InfluxDB repository. Include in your pull request details of your change -- the why *and* the how -- as well as the testing your performed. Also, be sure to run the test suite with your change in place. Changes that cause tests to fail cannot be merged.
There will usually be some back and forth as we finalize the change, but once that completes it may be merged.
To assist in review for the PR, please add the following to your pull request comment:
```md
- [ ] CHANGELOG.md updated
- [ ] Rebased/mergable
- [ ] Tests pass
- [ ] Sign [CLA](http://influxdb.com/community/cla.html) (if not already signed)
```
Use of third-party packages
------------
A third-party package is defined as one that is not part of the standard Go distribution. Generally speaking we prefer to minimize our use of third-party packages, and avoid them unless absolutely necessarly. We'll often write a little bit of code rather than pull in a third-party package. Of course, we do use some third-party packages -- most importantly we use [BoltDB](https://github.com/boltdb/bolt) as the storage engine. So to maximise the chance your change will be accepted by us, use only the standard libaries, or the third-party packages we have decided to use.
For rationale, check out the post [The Case Against Third Party Libraries](http://blog.gopheracademy.com/advent-2014/case-against-3pl/).
Signing the CLA
---------------
If you are going to be contributing back to InfluxDB please take a
second to sign our CLA, which can be found
[on our website](http://influxdb.com/community/cla.html).
Installing Go
-------------
InfluxDB requires Go 1.4 or greater.
At InfluxDB we find gvm, a Go version manager, useful for installing Go. For instructions
on how to install it see [the gvm page on github](https://github.com/moovweb/gvm).
After installing gvm you can install and set the default go version by
running the following:
gvm install go1.4
gvm use go1.4 --default
Revision Control Systems
------
Go has the ability to import remote packages via revision control systems with the `go get` command. To ensure that you can retrieve any remote package, be sure to install the following rcs software to your system.
Currently the project only depends on `git` and `mercurial`.
* [Install Git](http://git-scm.com/book/en/Getting-Started-Installing-Git)
* [Install Mercurial](http://mercurial.selenic.com/wiki/Download)
Project structure
-----------------
First you need to setup the project structure:
export GOPATH=$HOME/gocodez
mkdir -p $GOPATH/src/github.com/influxdb
cd $GOPATH/src/github.com/influxdb
git clone git@github.com:influxdb/influxdb
You can add the line `export GOPATH=$HOME/gocodez` to your bash/zsh
file to be set for every shell instead of having to manually run it
everytime.
We have a pre commit hook to make sure code is formatted properly
and vetted before you commit any changes. We strongly recommend using the pre
commit hook to guard against accidentally committing unformatted
code. To use the pre-commit hook, run the following:
cd $GOPATH/src/github.com/influxdb/influxdb
cp .hooks/pre-commit .git/hooks/
In case the commit is rejected because it's not formatted you can run
the following to format the code:
```
go fmt ./...
go vet ./...
```
To install go vet, run the following command:
```
go get golang.org/x/tools/cmd/vet
```
NOTE: If you have not installed mercurial, the above command will fail. See [Revision Control Systems](#revision-control-systems) above.
For more information on `go vet`, [read the GoDoc](https://godoc.org/golang.org/x/tools/cmd/vet).
Build and Test
-----
Make sure you have Go installed and the project structure as shown above. To then build the project, execute the following commands:
```bash
cd $GOPATH/src/github.com/influxdb
go get -u -f -t ./...
go build ./...
```
To then install the binaries, run the following command. They can be found in `$GOPATH/bin`. Please note that the InfluxDB binary is named `influxd`, not `influxdb`.
```bash
go install ./...
```
To set the version and commit flags during the build pass the following to the build command:
```bash
-ldflags="-X main.version $VERSION -X main.branch $BRANCH -X main.commit $COMMIT"
```
where `$VERSION` is the version, `$BRANCH` is the branch, and `$COMMIT` is the git commit hash.
To run the tests, execute the following command:
```bash
cd $GOPATH/src/github.com/influxdb/influxdb
go test -v ./...
# run tests that match some pattern
go test -run=TestDatabase . -v
# run tests and show coverage
go test -coverprofile /tmp/cover . && go tool cover -html /tmp/cover
```
To install go cover, run the following command:
```
go get golang.org/x/tools/cmd/cover
```
Generated Google Protobuf code
-----------------
Most changes to the source do not require that the generated protocol buffer code be changed. But if you need to modify the protocol buffer code, you'll first need to install the protocol buffers toolchain.
First install the [protocol buffer compiler](https://developers.google.com/protocol-buffers/
) 2.6.1 or later for your OS:
Then install the go plugins:
```bash
go get github.com/gogo/protobuf/proto
go get github.com/gogo/protobuf/protoc-gen-gogo
go get github.com/gogo/protobuf/gogoproto
```
Finally run, `go generate` after updating any `*.proto` file:
```bash
go generate ./...
```
**Trouleshooting**
If generating the protobuf code is failing for you, check each of the following:
* Ensure the protobuf library can be found. Make sure that `LD_LIBRRARY_PATH` includes the directory in which the library `libprotoc.so` has been installed.
* Ensure the command `protoc-gen-gogo`, found in `GOPATH/bin`, is on your path. This can be done by adding `GOPATH/bin` to `PATH`.
Profiling
-----
When troubleshooting problems with CPU or memory the Go toolchain can be helpful. You can start InfluxDB with CPU or memory profiling turned on. For example:
```sh
# start influx with profiling
./influxd -cpuprofile influxd.prof
# run queries, writes, whatever you're testing
# open up pprof
go tool pprof influxd influxd.prof
# once inside run "web", opens up browser with the CPU graph
# can also run "web <function name>" to zoom in. Or "list <function name>" to see specific lines
```
Continuous Integration testing
-----
InfluxDB uses CirceCI for continuous integration testing. To see how the code is built and tested, check out [this file](https://github.com/influxdb/influxdb/blob/master/circle-test.sh). It closely follows the build and test process outlined above. You can see the exact version of Go InfluxDB uses for testing by consulting that file.
Useful links
------------
- [Useful techniques in Go](http://arslan.io/ten-useful-techniques-in-go)
- [Go in production](http://peter.bourgon.org/go-in-production/)
- [Principles of designing Go APIs with channels](https://inconshreveable.com/07-08-2014/principles-of-designing-go-apis-with-channels/)
- [Common mistakes in Golang](http://soryy.com/blog/2014/common-mistakes-with-go-lang/). Especially this section `Loops, Closures, and Local Variables`

44
Godeps/_workspace/src/github.com/influxdb/influxdb/DOCKER.md generated vendored Normal file
View File

@ -0,0 +1,44 @@
# Docker Setup
========================
This document describes how to build and run a minimal InfluxDB container under Docker. Currently, it has only been tested for local development and assumes that you have a working docker environment.
## Building Image
To build a docker image for InfluxDB from your current checkout, run the following:
```
$ ./build-docker.sh
```
This script uses the `golang:1.5` image to build a fully static binary of `influxd` and then adds it to a minimal `scratch` image.
To build the image using a different version of go:
```
$ GO_VER=1.4.2 ./build-docker.sh
```
Available version can be found [here](https://hub.docker.com/_/golang/).
## Single Node Container
This will start an interactive, single-node, that publishes the containers port `8086` and `8088` to the hosts ports `8086` and `8088` respectively. This is identical to starting `influxd` manually.
```
$ docker run -it -p 8086:8086 -p 8088:8088 influxdb
```
## Multi-Node Cluster
This will create a simple 3-node cluster. The data is stored within the container and will be lost when the container is removed. This is only useful for test clusters.
The `HOST_IP` env variable should be your host IP if running under linux or the virtualbox VM IP if running under OSX. On OSX, this would be something like: `$(docker-machine ip dev)` or `$(boot2docker ip)` depending on which docker tool you are using.
```
$ export HOST_IP=<your host/VM IP>
$ docker run -it -p 8086:8088 -p 8088:8088 influxdb -hostname $HOST_IP:8088
$ docker run -it -p 8186:8088 -p 8188:8088 influxdb -hostname $HOST_IP:8188 -join $HOST_IP:8088
$ docker run -it -p 8286:8088 -p 8288:8088 influxdb -hostname $HOST_IP:8288 -join $HOST_IP:8088
```

24
Godeps/_workspace/src/github.com/influxdb/influxdb/Dockerfile generated vendored Normal file
View File

@ -0,0 +1,24 @@
FROM busybox:ubuntu-14.04
MAINTAINER Jason Wilder "<jason@influxdb.com>"
# admin, http, udp, cluster, graphite, opentsdb, collectd
EXPOSE 8083 8086 8086/udp 8088 2003 4242 25826
WORKDIR /app
# copy binary into image
COPY influxd /app/
# Add influxd to the PATH
ENV PATH=/app:$PATH
# Generate a default config
RUN influxd config > /etc/influxdb.toml
# Use /data for all disk storage
RUN sed -i 's/dir = "\/.*influxdb/dir = "\/data/' /etc/influxdb.toml
VOLUME ["/data"]
ENTRYPOINT ["influxd", "--config", "/etc/influxdb.toml"]

20
Godeps/_workspace/src/github.com/influxdb/influxdb/LICENSE generated vendored Normal file
View File

@ -0,0 +1,20 @@
The MIT License (MIT)
Copyright (c) 2013-2015 Errplane Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

180
Godeps/_workspace/src/github.com/influxdb/influxdb/QUERIES.md generated vendored Normal file
View File

@ -0,0 +1,180 @@
The top level name is called a measurement. These names can contain any characters. Then there are field names, field values, tag keys and tag values, which can also contain any characters. However, if the measurement, field, or tag contains any character other than [A-Z,a-z,0-9,_], or if it starts with a digit, it must be double-quoted. Therefore anywhere a measurement name, field name, field value, tag name, or tag value appears it should be wrapped in double quotes.
# Databases & retention policies
```sql
-- create a database
CREATE DATABASE <name>
-- create a retention policy
CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
-- alter retention policy
ALTER RETENTION POLICY <rp-name> ON <db-name> (DURATION <duration> | REPLICATION <n> | DEFAULT)+
-- drop a database
DROP DATABASE <name>
-- drop a retention policy
DROP RETENTION POLICY <rp-name> ON <db-name>
```
where `<duration>` is either `INF` for infinite retention, or an integer followed by the desired unit of time: u,ms,s,m,h,d,w for microseconds, milliseconds, seconds, minutes, hours, days, or weeks, respectively. `<replication>` must be an integer.
If present, `DEFAULT` sets the retention policy as the default retention policy for writes and reads.
# Users and permissions
```sql
-- create user
CREATE USER <name> WITH PASSWORD '<password>'
-- grant privilege on a database
GRANT <privilege> ON <db> TO <user>
-- grant cluster admin privileges
GRANT ALL [PRIVILEGES] TO <user>
-- revoke privilege
REVOKE <privilege> ON <db> FROM <user>
-- revoke all privileges for a DB
REVOKE ALL [PRIVILEGES] ON <db> FROM <user>
-- revoke all privileges including cluster admin
REVOKE ALL [PRIVILEGES] FROM <user>
-- combine db creation with privilege assignment (user must already exist)
CREATE DATABASE <name> GRANT <privilege> TO <user>
CREATE DATABASE <name> REVOKE <privilege> FROM <user>
-- delete a user
DROP USER <name>
```
where `<privilege> := READ | WRITE | All `.
Authentication must be enabled in the influxdb.conf file for user permissions to be in effect.
By default, newly created users have no privileges to any databases.
Cluster administration privileges automatically grant full read and write permissions to all databases, regardless of subsequent database-specific privilege revocation statements.
# Select
```sql
SELECT mean(value) from cpu WHERE host = 'serverA' AND time > now() - 4h GROUP BY time(5m)
SELECT mean(value) from cpu WHERE time > now() - 4h GROUP BY time(5m), region
```
## Group By
# Delete
# Series
## Destroy
```sql
DROP MEASUREMENT <name>
DROP MEASUREMENT cpu WHERE region = 'uswest'
```
## Show
Show series queries are for pulling out individual series from measurement names and tag data. They're useful for discovery.
```sql
-- show all databases
SHOW DATABASES
-- show measurement names
SHOW MEASUREMENTS
SHOW MEASUREMENTS LIMIT 15
SHOW MEASUREMENTS LIMIT 10 OFFSET 40
SHOW MEASUREMENTS WHERE service = 'redis'
-- LIMIT and OFFSET can be applied to any of the SHOW type queries
-- show all series across all measurements/tagsets
SHOW SERIES
-- get a show of all series for any measurements where tag key region = tak value 'uswest'
SHOW SERIES WHERE region = 'uswest'
SHOW SERIES FROM cpu_load WHERE region = 'uswest' LIMIT 10
-- returns the 100 - 109 rows in the result. In the case of SHOW SERIES, which returns
-- series split into measurements. Each series counts as a row. So you could see only a
-- single measurement returned, but 10 series within it.
SHOW SERIES FROM cpu_load WHERE region = 'uswest' LIMIT 10 OFFSET 100
-- show all retention policies on a database
SHOW RETENTION POLICIES ON mydb
-- get a show of all tag keys across all measurements
SHOW TAG KEYS
-- show all the tag keys for a given measurement
SHOW TAG KEYS FROM cpu
SHOW TAG KEYS FROM temperature, wind_speed
-- show all the tag values. note that a single WHERE TAG KEY = '...' clause is required
SHOW TAG VALUES WITH TAG KEY = 'region'
SHOW TAG VALUES FROM cpu WHERE region = 'uswest' WITH TAG KEY = 'host'
-- and you can do stuff against fields
SHOW FIELD KEYS FROM cpu
-- but you can't do this
SHOW FIELD VALUES
-- we don't index field values, so this query should be invalid.
-- show all users
SHOW USERS
```
Note that `FROM` and `WHERE` are optional clauses in most of the show series queries.
And the show series output looks like this:
```json
[
{
"name": "cpu",
"columns": ["id", "region", "host"],
"values": [
1, "uswest", "servera",
2, "uswest", "serverb"
]
},
{
"name": "reponse_time",
"columns": ["id", "application", "host"],
"values": [
3, "myRailsApp", "servera"
]
}
]
```
# Continuous Queries
Continous queries are going to be inspired by MySQL `TRIGGER` syntax:
http://dev.mysql.com/doc/refman/5.0/en/trigger-syntax.html
Instead of having automatically-assigned ids, named continuous queries allows for some level of duplication prevention,
particularly in the case where creation is scripted.
## Create
CREATE CONTINUOUS QUERY <name> AS SELECT ... FROM ...
## Destroy
DROP CONTINUOUS QUERY <name>
## List
SHOW CONTINUOUS QUERIES

71
Godeps/_workspace/src/github.com/influxdb/influxdb/README.md generated vendored Normal file
View File

@ -0,0 +1,71 @@
# InfluxDB [![Circle CI](https://circleci.com/gh/influxdb/influxdb/tree/master.svg?style=svg)](https://circleci.com/gh/influxdb/influxdb/tree/master)
## An Open-Source, Distributed, Time Series Database
> InfluxDB v0.9.0 is now out. Going forward, the 0.9.x series of releases will not make breaking API changes or breaking changes to the underlying data storage. However, 0.9.0 clustering should be considered an alpha release.
InfluxDB is an open source **distributed time series database** with
**no external dependencies**. It's useful for recording metrics,
events, and performing analytics.
## Features
* Built-in [HTTP API](http://influxdb.com/docs/v0.9/concepts/reading_and_writing_data.html) so you don't have to write any server side code to get up and running.
* Data can be tagged, allowing very flexible querying.
* SQL-like query language.
* Clustering is supported out of the box, so that you can scale horizontally to handle your data.
* Simple to install and manage, and fast to get data in and out.
* It aims to answer queries in real-time. That means every data point is
indexed as it comes in and is immediately available in queries that
should return in < 100ms.
## Getting Started
*The following directions apply only to the 0.9.0 release or building from the source on master.*
### Building
You don't need to build the project to use it - you can use any of our
[pre-built packages](http://influxdb.com/download/index.html) to install InfluxDB. That's
the recommended way to get it running. However, if you want to contribute to the core of InfluxDB, you'll need to build.
For those adventurous enough, you can
[follow along on our docs](http://github.com/influxdb/influxdb/blob/master/CONTRIBUTING.md).
### Starting InfluxDB
* `service influxdb start` if you have installed InfluxDB using an official Debian or RPM package.
* `$GOPATH/bin/influxd` if you have built InfluxDB from source.
### Creating your first database
```
curl -G 'http://localhost:8086/query' --data-urlencode "q=CREATE DATABASE mydb"
```
### Insert some data
```
curl -XPOST 'http://localhost:8086/write?db=mydb' \
-d 'cpu,host=server01,region=uswest load=42 1434055562000000000'
curl -XPOST 'http://localhost:8086/write?db=mydb' \
-d 'cpu,host=server02,region=uswest load=78 1434055562000000000'
curl -XPOST 'http://localhost:8086/write?db=mydb' \
-d 'cpu,host=server03,region=useast load=15.4 1434055562000000000'
```
### Query for the data
```JSON
curl -G http://localhost:8086/query?pretty=true --data-urlencode "db=mydb" \
--data-urlencode "q=SELECT * FROM cpu WHERE host='server01' AND time < now - 1d"
```
### Analyze the data
```JSON
curl -G http://localhost:8086/query?pretty=true --data-urlencode "db=mydb" \
--data-urlencode "q=SELECT mean(load) FROM cpu WHERE region='uswest'"
```
## Helpful Links
* Understand the [design goals and motivations of the project](http://influxdb.com/docs/v0.9/introduction/overview.html).
* Follow the [getting started guide](http://influxdb.com/docs/v0.9/introduction/getting_started.html) to find out how to install InfluxDB, start writing more data, and issue more queries - in just a few minutes.
* See the [HTTP API documentation to start writing a library for your favorite language](http://influxdb.com/docs/v0.9/concepts/reading_and_writing_data.html).

78
Godeps/_workspace/src/github.com/influxdb/influxdb/balancer.go generated vendored Normal file
View File

@ -0,0 +1,78 @@
package influxdb
import (
"math/rand"
"github.com/influxdb/influxdb/meta"
)
// Balancer represents a load-balancing algorithm for a set of nodes
type Balancer interface {
// Next returns the next Node according to the balancing method
// or nil if there are no nodes available
Next() *meta.NodeInfo
}
type nodeBalancer struct {
nodes []meta.NodeInfo // data nodes to balance between
p int // current node index
}
// NewNodeBalancer create a shuffled, round-robin balancer so that
// multiple instances will return nodes in randomized order and each
// each returned node will be repeated in a cycle
func NewNodeBalancer(nodes []meta.NodeInfo) Balancer {
// make a copy of the node slice so we can randomize it
// without affecting the original instance as well as ensure
// that each Balancer returns nodes in a different order
b := &nodeBalancer{}
b.nodes = make([]meta.NodeInfo, len(nodes))
copy(b.nodes, nodes)
b.shuffle()
return b
}
// shuffle randomizes the ordering the balancers available nodes
func (b *nodeBalancer) shuffle() {
for i := range b.nodes {
j := rand.Intn(i + 1)
b.nodes[i], b.nodes[j] = b.nodes[j], b.nodes[i]
}
}
// online returns a slice of the nodes that are online
func (b *nodeBalancer) online() []meta.NodeInfo {
return b.nodes
// now := time.Now().UTC()
// up := []meta.NodeInfo{}
// for _, n := range b.nodes {
// if n.OfflineUntil.After(now) {
// continue
// }
// up = append(up, n)
// }
// return up
}
// Next returns the next available nodes
func (b *nodeBalancer) Next() *meta.NodeInfo {
// only use online nodes
up := b.online()
// no nodes online
if len(up) == 0 {
return nil
}
// rollover back to the beginning
if b.p >= len(up) {
b.p = 0
}
d := &up[b.p]
b.p += 1
return d
}

115
Godeps/_workspace/src/github.com/influxdb/influxdb/balancer_test.go generated vendored Normal file
View File

@ -0,0 +1,115 @@
package influxdb_test
import (
"fmt"
"testing"
"github.com/influxdb/influxdb"
"github.com/influxdb/influxdb/meta"
)
func NewNodes() []meta.NodeInfo {
var nodes []meta.NodeInfo
for i := 1; i <= 2; i++ {
nodes = append(nodes, meta.NodeInfo{
ID: uint64(i),
Host: fmt.Sprintf("localhost:999%d", i),
})
}
return nodes
}
func TestBalancerEmptyNodes(t *testing.T) {
b := influxdb.NewNodeBalancer([]meta.NodeInfo{})
got := b.Next()
if got != nil {
t.Errorf("expected nil, got %v", got)
}
}
func TestBalancerUp(t *testing.T) {
nodes := NewNodes()
b := influxdb.NewNodeBalancer(nodes)
// First node in randomized round-robin order
first := b.Next()
if first == nil {
t.Errorf("expected datanode, got %v", first)
}
// Second node in randomized round-robin order
second := b.Next()
if second == nil {
t.Errorf("expected datanode, got %v", second)
}
// Should never get the same node in order twice
if first.ID == second.ID {
t.Errorf("expected first != second. got %v = %v", first.ID, second.ID)
}
}
/*
func TestBalancerDown(t *testing.T) {
nodes := NewNodes()
b := influxdb.NewNodeBalancer(nodes)
nodes[0].Down()
// First node in randomized round-robin order
first := b.Next()
if first == nil {
t.Errorf("expected datanode, got %v", first)
}
// Second node should rollover to the first up node
second := b.Next()
if second == nil {
t.Errorf("expected datanode, got %v", second)
}
// Health node should be returned each time
if first.ID != 2 && first.ID != second.ID {
t.Errorf("expected first != second. got %v = %v", first.ID, second.ID)
}
}
*/
/*
func TestBalancerBackUp(t *testing.T) {
nodes := newDataNodes()
b := influxdb.NewNodeBalancer(nodes)
nodes[0].Down()
for i := 0; i < 3; i++ {
got := b.Next()
if got == nil {
t.Errorf("expected datanode, got %v", got)
}
if exp := uint64(2); got.ID != exp {
t.Errorf("wrong node id: exp %v, got %v", exp, got.ID)
}
}
nodes[0].Up()
// First node in randomized round-robin order
first := b.Next()
if first == nil {
t.Errorf("expected datanode, got %v", first)
}
// Second node should rollover to the first up node
second := b.Next()
if second == nil {
t.Errorf("expected datanode, got %v", second)
}
// Should get both nodes returned
if first.ID == second.ID {
t.Errorf("expected first != second. got %v = %v", first.ID, second.ID)
}
}
*/

7
Godeps/_workspace/src/github.com/influxdb/influxdb/build-docker.sh generated vendored Normal file
View File

@ -0,0 +1,7 @@
#!/bin/sh -x -e
GO_VER=${GO_VER:-1.5}
docker run -it -v "$GOPATH":/gopath -v "$(pwd)":/app -e "GOPATH=/gopath" -w /app golang:$GO_VER sh -c 'CGO_ENABLED=0 go build -a --installsuffix cgo --ldflags="-s" -o influxd ./cmd/influxd'
docker build -t influxdb .

63
Godeps/_workspace/src/github.com/influxdb/influxdb/circle-test.sh generated vendored Normal file
View File

@ -0,0 +1,63 @@
#!/bin/bash
#
# This is the InfluxDB CircleCI test script. Using this script allows total control
# the environment in which the build and test is run, and matches the official
# build process for InfluxDB.
BUILD_DIR=$HOME/influxdb-build
GO_VERSION=go1.4.2
PARALLELISM="-parallel 256"
TIMEOUT="-timeout 480s"
# Executes the given statement, and exits if the command returns a non-zero code.
function exit_if_fail {
command=$@
echo "Executing '$command'"
$command
rc=$?
if [ $rc -ne 0 ]; then
echo "'$command' returned $rc."
exit $rc
fi
}
source $HOME/.gvm/scripts/gvm
exit_if_fail gvm use $GO_VERSION
# Set up the build directory, and then GOPATH.
exit_if_fail mkdir $BUILD_DIR
export GOPATH=$BUILD_DIR
exit_if_fail mkdir -p $GOPATH/src/github.com/influxdb
# Dump some test config to the log.
echo "Test configuration"
echo "========================================"
echo "\$HOME: $HOME"
echo "\$GOPATH: $GOPATH"
echo "\$CIRCLE_BRANCH: $CIRCLE_BRANCH"
# Move the checked-out source to a better location.
exit_if_fail mv $HOME/influxdb $GOPATH/src/github.com/influxdb
exit_if_fail cd $GOPATH/src/github.com/influxdb/influxdb
exit_if_fail git branch --set-upstream-to=origin/$CIRCLE_BRANCH $CIRCLE_BRANCH
# Install the code.
exit_if_fail cd $GOPATH/src/github.com/influxdb/influxdb
exit_if_fail go get -t -d -v ./...
exit_if_fail git checkout $CIRCLE_BRANCH # 'go get' switches to master. Who knew? Switch back.
exit_if_fail go build -v ./...
# Run the tests.
exit_if_fail go tool vet --composites=false .
case $CIRCLE_NODE_INDEX in
0)
go test $PARALLELISM $TIMEOUT -v ./... 2>&1 | tee $CIRCLE_ARTIFACTS/test_logs.txt
rc=${PIPESTATUS[0]}
;;
1)
GORACE="halt_on_error=1" go test $PARALLELISM $TIMEOUT -v -race ./... 2>&1 | tee $CIRCLE_ARTIFACTS/test_logs_race.txt
rc=${PIPESTATUS[0]}
;;
esac
exit $rc

12
Godeps/_workspace/src/github.com/influxdb/influxdb/circle.yml generated vendored Normal file
View File

@ -0,0 +1,12 @@
machine:
pre:
- bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
- source $HOME/.gvm/scripts/gvm; gvm install go1.4.2 --binary
dependencies:
override:
- echo "Dummy override, so no Circle dependencies execute"
test:
override:
- bash circle-test.sh:
parallel: true

57
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/client_pool.go generated vendored Normal file
View File

@ -0,0 +1,57 @@
package cluster
import (
"net"
"sync"
"gopkg.in/fatih/pool.v2"
)
type clientPool struct {
mu sync.RWMutex
pool map[uint64]pool.Pool
}
func newClientPool() *clientPool {
return &clientPool{
pool: make(map[uint64]pool.Pool),
}
}
func (c *clientPool) setPool(nodeID uint64, p pool.Pool) {
c.mu.Lock()
c.pool[nodeID] = p
c.mu.Unlock()
}
func (c *clientPool) getPool(nodeID uint64) (pool.Pool, bool) {
c.mu.RLock()
p, ok := c.pool[nodeID]
c.mu.RUnlock()
return p, ok
}
func (c *clientPool) size() int {
c.mu.RLock()
var size int
for _, p := range c.pool {
size += p.Len()
}
c.mu.RUnlock()
return size
}
func (c *clientPool) conn(nodeID uint64) (net.Conn, error) {
c.mu.RLock()
conn, err := c.pool[nodeID].Get()
c.mu.RUnlock()
return conn, err
}
func (c *clientPool) close() {
c.mu.Lock()
for _, p := range c.pool {
p.Close()
}
c.mu.Unlock()
}

35
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/config.go generated vendored Normal file
View File

@ -0,0 +1,35 @@
package cluster
import (
"time"
"github.com/influxdb/influxdb/toml"
)
const (
// DefaultWriteTimeout is the default timeout for a complete write to succeed.
DefaultWriteTimeout = 5 * time.Second
// DefaultShardWriterTimeout is the default timeout set on shard writers.
DefaultShardWriterTimeout = 5 * time.Second
// DefaultShardMapperTimeout is the default timeout set on shard mappers.
DefaultShardMapperTimeout = 5 * time.Second
)
// Config represents the configuration for the clustering service.
type Config struct {
ForceRemoteShardMapping bool `toml:"force-remote-mapping"`
WriteTimeout toml.Duration `toml:"write-timeout"`
ShardWriterTimeout toml.Duration `toml:"shard-writer-timeout"`
ShardMapperTimeout toml.Duration `toml:"shard-mapper-timeout"`
}
// NewConfig returns an instance of Config with defaults.
func NewConfig() Config {
return Config{
WriteTimeout: toml.Duration(DefaultWriteTimeout),
ShardWriterTimeout: toml.Duration(DefaultShardWriterTimeout),
ShardMapperTimeout: toml.Duration(DefaultShardMapperTimeout),
}
}

27
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/config_test.go generated vendored Normal file
View File

@ -0,0 +1,27 @@
package cluster_test
import (
"testing"
"time"
"github.com/BurntSushi/toml"
"github.com/influxdb/influxdb/cluster"
)
func TestConfig_Parse(t *testing.T) {
// Parse configuration.
var c cluster.Config
if _, err := toml.Decode(`
shard-writer-timeout = "10s"
write-timeout = "20s"
`, &c); err != nil {
t.Fatal(err)
}
// Validate configuration.
if time.Duration(c.ShardWriterTimeout) != 10*time.Second {
t.Fatalf("unexpected shard-writer timeout: %s", c.ShardWriterTimeout)
} else if time.Duration(c.WriteTimeout) != 20*time.Second {
t.Fatalf("unexpected write timeout s: %s", c.WriteTimeout)
}
}

286
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/internal/data.pb.go generated vendored Normal file
View File

@ -0,0 +1,286 @@
// Code generated by protoc-gen-gogo.
// source: internal/data.proto
// DO NOT EDIT!
/*
Package internal is a generated protocol buffer package.
It is generated from these files:
internal/data.proto
It has these top-level messages:
WriteShardRequest
Field
Tag
Point
WriteShardResponse
MapShardRequest
MapShardResponse
*/
package internal
import proto "github.com/gogo/protobuf/proto"
import math "math"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf
type WriteShardRequest struct {
ShardID *uint64 `protobuf:"varint,1,req" json:"ShardID,omitempty"`
Points []*Point `protobuf:"bytes,2,rep" json:"Points,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *WriteShardRequest) Reset() { *m = WriteShardRequest{} }
func (m *WriteShardRequest) String() string { return proto.CompactTextString(m) }
func (*WriteShardRequest) ProtoMessage() {}
func (m *WriteShardRequest) GetShardID() uint64 {
if m != nil && m.ShardID != nil {
return *m.ShardID
}
return 0
}
func (m *WriteShardRequest) GetPoints() []*Point {
if m != nil {
return m.Points
}
return nil
}
type Field struct {
Name *string `protobuf:"bytes,1,req" json:"Name,omitempty"`
Int32 *int32 `protobuf:"varint,2,opt" json:"Int32,omitempty"`
Int64 *int64 `protobuf:"varint,3,opt" json:"Int64,omitempty"`
Float64 *float64 `protobuf:"fixed64,4,opt" json:"Float64,omitempty"`
Bool *bool `protobuf:"varint,5,opt" json:"Bool,omitempty"`
String_ *string `protobuf:"bytes,6,opt" json:"String,omitempty"`
Bytes []byte `protobuf:"bytes,7,opt" json:"Bytes,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Field) Reset() { *m = Field{} }
func (m *Field) String() string { return proto.CompactTextString(m) }
func (*Field) ProtoMessage() {}
func (m *Field) GetName() string {
if m != nil && m.Name != nil {
return *m.Name
}
return ""
}
func (m *Field) GetInt32() int32 {
if m != nil && m.Int32 != nil {
return *m.Int32
}
return 0
}
func (m *Field) GetInt64() int64 {
if m != nil && m.Int64 != nil {
return *m.Int64
}
return 0
}
func (m *Field) GetFloat64() float64 {
if m != nil && m.Float64 != nil {
return *m.Float64
}
return 0
}
func (m *Field) GetBool() bool {
if m != nil && m.Bool != nil {
return *m.Bool
}
return false
}
func (m *Field) GetString_() string {
if m != nil && m.String_ != nil {
return *m.String_
}
return ""
}
func (m *Field) GetBytes() []byte {
if m != nil {
return m.Bytes
}
return nil
}
type Tag struct {
Key *string `protobuf:"bytes,1,req" json:"Key,omitempty"`
Value *string `protobuf:"bytes,2,req" json:"Value,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Tag) Reset() { *m = Tag{} }
func (m *Tag) String() string { return proto.CompactTextString(m) }
func (*Tag) ProtoMessage() {}
func (m *Tag) GetKey() string {
if m != nil && m.Key != nil {
return *m.Key
}
return ""
}
func (m *Tag) GetValue() string {
if m != nil && m.Value != nil {
return *m.Value
}
return ""
}
type Point struct {
Name *string `protobuf:"bytes,1,req" json:"Name,omitempty"`
Time *int64 `protobuf:"varint,2,req" json:"Time,omitempty"`
Fields []*Field `protobuf:"bytes,3,rep" json:"Fields,omitempty"`
Tags []*Tag `protobuf:"bytes,4,rep" json:"Tags,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Point) Reset() { *m = Point{} }
func (m *Point) String() string { return proto.CompactTextString(m) }
func (*Point) ProtoMessage() {}
func (m *Point) GetName() string {
if m != nil && m.Name != nil {
return *m.Name
}
return ""
}
func (m *Point) GetTime() int64 {
if m != nil && m.Time != nil {
return *m.Time
}
return 0
}
func (m *Point) GetFields() []*Field {
if m != nil {
return m.Fields
}
return nil
}
func (m *Point) GetTags() []*Tag {
if m != nil {
return m.Tags
}
return nil
}
type WriteShardResponse struct {
Code *int32 `protobuf:"varint,1,req" json:"Code,omitempty"`
Message *string `protobuf:"bytes,2,opt" json:"Message,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *WriteShardResponse) Reset() { *m = WriteShardResponse{} }
func (m *WriteShardResponse) String() string { return proto.CompactTextString(m) }
func (*WriteShardResponse) ProtoMessage() {}
func (m *WriteShardResponse) GetCode() int32 {
if m != nil && m.Code != nil {
return *m.Code
}
return 0
}
func (m *WriteShardResponse) GetMessage() string {
if m != nil && m.Message != nil {
return *m.Message
}
return ""
}
type MapShardRequest struct {
ShardID *uint64 `protobuf:"varint,1,req" json:"ShardID,omitempty"`
Query *string `protobuf:"bytes,2,req" json:"Query,omitempty"`
ChunkSize *int32 `protobuf:"varint,3,req" json:"ChunkSize,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *MapShardRequest) Reset() { *m = MapShardRequest{} }
func (m *MapShardRequest) String() string { return proto.CompactTextString(m) }
func (*MapShardRequest) ProtoMessage() {}
func (m *MapShardRequest) GetShardID() uint64 {
if m != nil && m.ShardID != nil {
return *m.ShardID
}
return 0
}
func (m *MapShardRequest) GetQuery() string {
if m != nil && m.Query != nil {
return *m.Query
}
return ""
}
func (m *MapShardRequest) GetChunkSize() int32 {
if m != nil && m.ChunkSize != nil {
return *m.ChunkSize
}
return 0
}
type MapShardResponse struct {
Code *int32 `protobuf:"varint,1,req" json:"Code,omitempty"`
Message *string `protobuf:"bytes,2,opt" json:"Message,omitempty"`
Data []byte `protobuf:"bytes,3,opt" json:"Data,omitempty"`
TagSets []string `protobuf:"bytes,4,rep" json:"TagSets,omitempty"`
Fields []string `protobuf:"bytes,5,rep" json:"Fields,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *MapShardResponse) Reset() { *m = MapShardResponse{} }
func (m *MapShardResponse) String() string { return proto.CompactTextString(m) }
func (*MapShardResponse) ProtoMessage() {}
func (m *MapShardResponse) GetCode() int32 {
if m != nil && m.Code != nil {
return *m.Code
}
return 0
}
func (m *MapShardResponse) GetMessage() string {
if m != nil && m.Message != nil {
return *m.Message
}
return ""
}
func (m *MapShardResponse) GetData() []byte {
if m != nil {
return m.Data
}
return nil
}
func (m *MapShardResponse) GetTagSets() []string {
if m != nil {
return m.TagSets
}
return nil
}
func (m *MapShardResponse) GetFields() []string {
if m != nil {
return m.Fields
}
return nil
}
func init() {
}

49
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/internal/data.proto generated vendored Normal file
View File

@ -0,0 +1,49 @@
package internal;
message WriteShardRequest {
required uint64 ShardID = 1;
repeated Point Points = 2;
}
message Field {
required string Name = 1;
oneof Value {
int32 Int32 = 2;
int64 Int64 = 3;
double Float64 = 4;
bool Bool = 5;
string String = 6;
bytes Bytes = 7;
}
}
message Tag {
required string Key = 1;
required string Value = 2;
}
message Point {
required string Name = 1;
required int64 Time = 2;
repeated Field Fields = 3;
repeated Tag Tags = 4;
}
message WriteShardResponse {
required int32 Code = 1;
optional string Message = 2;
}
message MapShardRequest {
required uint64 ShardID = 1;
required string Query = 2;
required int32 ChunkSize = 3;
}
message MapShardResponse {
required int32 Code = 1;
optional string Message = 2;
optional bytes Data = 3;
repeated string TagSets = 4;
repeated string Fields = 5;
}

314
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/points_writer.go generated vendored Normal file
View File

@ -0,0 +1,314 @@
package cluster
import (
"errors"
"fmt"
"log"
"os"
"strings"
"sync"
"time"
"github.com/influxdb/influxdb"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/tsdb"
)
// ConsistencyLevel represent a required replication criteria before a write can
// be returned as successful
type ConsistencyLevel int
const (
// ConsistencyLevelAny allows for hinted hand off, potentially no write happened yet
ConsistencyLevelAny ConsistencyLevel = iota
// ConsistencyLevelOne requires at least one data node acknowledged a write
ConsistencyLevelOne
// ConsistencyLevelOne requires a quorum of data nodes to acknowledge a write
ConsistencyLevelQuorum
// ConsistencyLevelAll requires all data nodes to acknowledge a write
ConsistencyLevelAll
)
var (
// ErrTimeout is returned when a write times out.
ErrTimeout = errors.New("timeout")
// ErrPartialWrite is returned when a write partially succeeds but does
// not meet the requested consistency level.
ErrPartialWrite = errors.New("partial write")
// ErrWriteFailed is returned when no writes succeeded.
ErrWriteFailed = errors.New("write failed")
// ErrInvalidConsistencyLevel is returned when parsing the string version
// of a consistency level.
ErrInvalidConsistencyLevel = errors.New("invalid consistency level")
)
func ParseConsistencyLevel(level string) (ConsistencyLevel, error) {
switch strings.ToLower(level) {
case "any":
return ConsistencyLevelAny, nil
case "one":
return ConsistencyLevelOne, nil
case "quorum":
return ConsistencyLevelQuorum, nil
case "all":
return ConsistencyLevelAll, nil
default:
return 0, ErrInvalidConsistencyLevel
}
}
// PointsWriter handles writes across multiple local and remote data nodes.
type PointsWriter struct {
mu sync.RWMutex
closing chan struct{}
WriteTimeout time.Duration
Logger *log.Logger
MetaStore interface {
NodeID() uint64
Database(name string) (di *meta.DatabaseInfo, err error)
RetentionPolicy(database, policy string) (*meta.RetentionPolicyInfo, error)
CreateShardGroupIfNotExists(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error)
ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo)
}
TSDBStore interface {
CreateShard(database, retentionPolicy string, shardID uint64) error
WriteToShard(shardID uint64, points []tsdb.Point) error
}
ShardWriter interface {
WriteShard(shardID, ownerID uint64, points []tsdb.Point) error
}
HintedHandoff interface {
WriteShard(shardID, ownerID uint64, points []tsdb.Point) error
}
}
// NewPointsWriter returns a new instance of PointsWriter for a node.
func NewPointsWriter() *PointsWriter {
return &PointsWriter{
closing: make(chan struct{}),
WriteTimeout: DefaultWriteTimeout,
Logger: log.New(os.Stderr, "[write] ", log.LstdFlags),
}
}
// ShardMapping contains a mapping of a shards to a points.
type ShardMapping struct {
Points map[uint64][]tsdb.Point // The points associated with a shard ID
Shards map[uint64]*meta.ShardInfo // The shards that have been mapped, keyed by shard ID
}
// NewShardMapping creates an empty ShardMapping
func NewShardMapping() *ShardMapping {
return &ShardMapping{
Points: map[uint64][]tsdb.Point{},
Shards: map[uint64]*meta.ShardInfo{},
}
}
// MapPoint maps a point to shard
func (s *ShardMapping) MapPoint(shardInfo *meta.ShardInfo, p tsdb.Point) {
points, ok := s.Points[shardInfo.ID]
if !ok {
s.Points[shardInfo.ID] = []tsdb.Point{p}
} else {
s.Points[shardInfo.ID] = append(points, p)
}
s.Shards[shardInfo.ID] = shardInfo
}
func (w *PointsWriter) Open() error {
w.mu.Lock()
defer w.mu.Unlock()
if w.closing == nil {
w.closing = make(chan struct{})
}
return nil
}
func (w *PointsWriter) Close() error {
w.mu.Lock()
defer w.mu.Unlock()
if w.closing != nil {
close(w.closing)
w.closing = nil
}
return nil
}
// MapShards maps the points contained in wp to a ShardMapping. If a point
// maps to a shard group or shard that does not currently exist, it will be
// created before returning the mapping.
func (w *PointsWriter) MapShards(wp *WritePointsRequest) (*ShardMapping, error) {
// holds the start time ranges for required shard groups
timeRanges := map[time.Time]*meta.ShardGroupInfo{}
rp, err := w.MetaStore.RetentionPolicy(wp.Database, wp.RetentionPolicy)
if err != nil {
return nil, err
}
for _, p := range wp.Points {
timeRanges[p.Time().Truncate(rp.ShardGroupDuration)] = nil
}
// holds all the shard groups and shards that are required for writes
for t := range timeRanges {
sg, err := w.MetaStore.CreateShardGroupIfNotExists(wp.Database, wp.RetentionPolicy, t)
if err != nil {
return nil, err
}
timeRanges[t] = sg
}
mapping := NewShardMapping()
for _, p := range wp.Points {
sg := timeRanges[p.Time().Truncate(rp.ShardGroupDuration)]
sh := sg.ShardFor(p.HashID())
mapping.MapPoint(&sh, p)
}
return mapping, nil
}
// WritePoints writes across multiple local and remote data nodes according the consistency level.
func (w *PointsWriter) WritePoints(p *WritePointsRequest) error {
if p.RetentionPolicy == "" {
db, err := w.MetaStore.Database(p.Database)
if err != nil {
return err
} else if db == nil {
return influxdb.ErrDatabaseNotFound(p.Database)
}
p.RetentionPolicy = db.DefaultRetentionPolicy
}
shardMappings, err := w.MapShards(p)
if err != nil {
return err
}
// Write each shard in it's own goroutine and return as soon
// as one fails.
ch := make(chan error, len(shardMappings.Points))
for shardID, points := range shardMappings.Points {
go func(shard *meta.ShardInfo, database, retentionPolicy string, points []tsdb.Point) {
ch <- w.writeToShard(shard, p.Database, p.RetentionPolicy, p.ConsistencyLevel, points)
}(shardMappings.Shards[shardID], p.Database, p.RetentionPolicy, points)
}
for range shardMappings.Points {
select {
case <-w.closing:
return ErrWriteFailed
case err := <-ch:
if err != nil {
return err
}
}
}
return nil
}
// writeToShards writes points to a shard and ensures a write consistency level has been met. If the write
// partially succeeds, ErrPartialWrite is returned.
func (w *PointsWriter) writeToShard(shard *meta.ShardInfo, database, retentionPolicy string,
consistency ConsistencyLevel, points []tsdb.Point) error {
// The required number of writes to achieve the requested consistency level
required := len(shard.OwnerIDs)
switch consistency {
case ConsistencyLevelAny, ConsistencyLevelOne:
required = 1
case ConsistencyLevelQuorum:
required = required/2 + 1
}
// response channel for each shard writer go routine
ch := make(chan error, len(shard.OwnerIDs))
for _, nodeID := range shard.OwnerIDs {
go func(shardID, nodeID uint64, points []tsdb.Point) {
if w.MetaStore.NodeID() == nodeID {
err := w.TSDBStore.WriteToShard(shardID, points)
// If we've written to shard that should exist on the current node, but the store has
// not actually created this shard, tell it to create it and retry the write
if err == tsdb.ErrShardNotFound {
err = w.TSDBStore.CreateShard(database, retentionPolicy, shardID)
if err != nil {
ch <- err
return
}
err = w.TSDBStore.WriteToShard(shardID, points)
}
ch <- err
return
}
err := w.ShardWriter.WriteShard(shardID, nodeID, points)
if err != nil && tsdb.IsRetryable(err) {
// The remote write failed so queue it via hinted handoff
hherr := w.HintedHandoff.WriteShard(shardID, nodeID, points)
// If the write consistency level is ANY, then a successful hinted handoff can
// be considered a successful write so send nil to the response channel
// otherwise, let the original error propogate to the response channel
if hherr == nil && consistency == ConsistencyLevelAny {
ch <- nil
return
}
}
ch <- err
}(shard.ID, nodeID, points)
}
var wrote int
timeout := time.After(w.WriteTimeout)
var writeError error
for _, nodeID := range shard.OwnerIDs {
select {
case <-w.closing:
return ErrWriteFailed
case <-timeout:
// return timeout error to caller
return ErrTimeout
case err := <-ch:
// If the write returned an error, continue to the next response
if err != nil {
w.Logger.Printf("write failed for shard %d on node %d: %v", shard.ID, nodeID, err)
// Keep track of the first error we see to return back to the client
if writeError == nil {
writeError = err
}
continue
}
wrote += 1
}
}
// We wrote the required consistency level
if wrote >= required {
return nil
}
if wrote > 0 {
return ErrPartialWrite
}
if writeError != nil {
return fmt.Errorf("write failed: %v", writeError)
}
return ErrWriteFailed
}

436
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/points_writer_test.go generated vendored Normal file
View File

@ -0,0 +1,436 @@
package cluster_test
import (
"fmt"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/influxdb/influxdb/cluster"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/tsdb"
)
// Ensures the points writer maps a single point to a single shard.
func TestPointsWriter_MapShards_One(t *testing.T) {
ms := MetaStore{}
rp := NewRetentionPolicy("myp", time.Hour, 3)
ms.NodeIDFn = func() uint64 { return 1 }
ms.RetentionPolicyFn = func(db, retentionPolicy string) (*meta.RetentionPolicyInfo, error) {
return rp, nil
}
ms.CreateShardGroupIfNotExistsFn = func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
return &rp.ShardGroups[0], nil
}
c := cluster.PointsWriter{MetaStore: ms}
pr := &cluster.WritePointsRequest{
Database: "mydb",
RetentionPolicy: "myrp",
ConsistencyLevel: cluster.ConsistencyLevelOne,
}
pr.AddPoint("cpu", 1.0, time.Now(), nil)
var (
shardMappings *cluster.ShardMapping
err error
)
if shardMappings, err = c.MapShards(pr); err != nil {
t.Fatalf("unexpected an error: %v", err)
}
if exp := 1; len(shardMappings.Points) != exp {
t.Errorf("MapShards() len mismatch. got %v, exp %v", len(shardMappings.Points), exp)
}
}
// Ensures the points writer maps a multiple points across shard group boundaries.
func TestPointsWriter_MapShards_Multiple(t *testing.T) {
ms := MetaStore{}
rp := NewRetentionPolicy("myp", time.Hour, 3)
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
ms.NodeIDFn = func() uint64 { return 1 }
ms.RetentionPolicyFn = func(db, retentionPolicy string) (*meta.RetentionPolicyInfo, error) {
return rp, nil
}
ms.CreateShardGroupIfNotExistsFn = func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
for i, sg := range rp.ShardGroups {
if timestamp.Equal(sg.StartTime) || timestamp.After(sg.StartTime) && timestamp.Before(sg.EndTime) {
return &rp.ShardGroups[i], nil
}
}
panic("should not get here")
}
c := cluster.PointsWriter{MetaStore: ms}
pr := &cluster.WritePointsRequest{
Database: "mydb",
RetentionPolicy: "myrp",
ConsistencyLevel: cluster.ConsistencyLevelOne,
}
// Three points that range over the shardGroup duration (1h) and should map to two
// distinct shards
pr.AddPoint("cpu", 1.0, time.Unix(0, 0), nil)
pr.AddPoint("cpu", 2.0, time.Unix(0, 0).Add(time.Hour), nil)
pr.AddPoint("cpu", 3.0, time.Unix(0, 0).Add(time.Hour+time.Second), nil)
var (
shardMappings *cluster.ShardMapping
err error
)
if shardMappings, err = c.MapShards(pr); err != nil {
t.Fatalf("unexpected an error: %v", err)
}
if exp := 2; len(shardMappings.Points) != exp {
t.Errorf("MapShards() len mismatch. got %v, exp %v", len(shardMappings.Points), exp)
}
for _, points := range shardMappings.Points {
// First shard shoud have 1 point w/ first point added
if len(points) == 1 && points[0].Time() != pr.Points[0].Time() {
t.Fatalf("MapShards() value mismatch. got %v, exp %v", points[0].Time(), pr.Points[0].Time())
}
// Second shard shoud have the last two points added
if len(points) == 2 && points[0].Time() != pr.Points[1].Time() {
t.Fatalf("MapShards() value mismatch. got %v, exp %v", points[0].Time(), pr.Points[1].Time())
}
if len(points) == 2 && points[1].Time() != pr.Points[2].Time() {
t.Fatalf("MapShards() value mismatch. got %v, exp %v", points[1].Time(), pr.Points[2].Time())
}
}
}
func TestPointsWriter_WritePoints(t *testing.T) {
tests := []struct {
name string
database string
retentionPolicy string
consistency cluster.ConsistencyLevel
// the responses returned by each shard write call. node ID 1 = pos 0
err []error
expErr error
}{
// Consistency one
{
name: "write one success",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelOne,
err: []error{nil, nil, nil},
expErr: nil,
},
{
name: "write one error",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelOne,
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), fmt.Errorf("a failure")},
expErr: fmt.Errorf("write failed: a failure"),
},
// Consistency any
{
name: "write any success",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelAny,
err: []error{fmt.Errorf("a failure"), nil, fmt.Errorf("a failure")},
expErr: nil,
},
// Consistency all
{
name: "write all success",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelAll,
err: []error{nil, nil, nil},
expErr: nil,
},
{
name: "write all, 2/3, partial write",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelAll,
err: []error{nil, fmt.Errorf("a failure"), nil},
expErr: cluster.ErrPartialWrite,
},
{
name: "write all, 1/3 (failure)",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelAll,
err: []error{nil, fmt.Errorf("a failure"), fmt.Errorf("a failure")},
expErr: cluster.ErrPartialWrite,
},
// Consistency quorum
{
name: "write quorum, 1/3 failure",
consistency: cluster.ConsistencyLevelQuorum,
database: "mydb",
retentionPolicy: "myrp",
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), nil},
expErr: cluster.ErrPartialWrite,
},
{
name: "write quorum, 2/3 success",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelQuorum,
err: []error{nil, nil, fmt.Errorf("a failure")},
expErr: nil,
},
{
name: "write quorum, 3/3 success",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelQuorum,
err: []error{nil, nil, nil},
expErr: nil,
},
// Error write error
{
name: "no writes succeed",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelOne,
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), fmt.Errorf("a failure")},
expErr: fmt.Errorf("write failed: a failure"),
},
// Hinted handoff w/ ANY
{
name: "hinted handoff write succeed",
database: "mydb",
retentionPolicy: "myrp",
consistency: cluster.ConsistencyLevelAny,
err: []error{fmt.Errorf("a failure"), fmt.Errorf("a failure"), fmt.Errorf("a failure")},
expErr: nil,
},
// Write to non-existant database
{
name: "write to non-existant database",
database: "doesnt_exist",
retentionPolicy: "",
consistency: cluster.ConsistencyLevelAny,
err: []error{nil, nil, nil},
expErr: fmt.Errorf("database not found: doesnt_exist"),
},
}
for _, test := range tests {
pr := &cluster.WritePointsRequest{
Database: test.database,
RetentionPolicy: test.retentionPolicy,
ConsistencyLevel: test.consistency,
}
// Three points that range over the shardGroup duration (1h) and should map to two
// distinct shards
pr.AddPoint("cpu", 1.0, time.Unix(0, 0), nil)
pr.AddPoint("cpu", 2.0, time.Unix(0, 0).Add(time.Hour), nil)
pr.AddPoint("cpu", 3.0, time.Unix(0, 0).Add(time.Hour+time.Second), nil)
// copy to prevent data race
theTest := test
sm := cluster.NewShardMapping()
sm.MapPoint(
&meta.ShardInfo{ID: uint64(1), OwnerIDs: []uint64{uint64(1), uint64(2), uint64(3)}},
pr.Points[0])
sm.MapPoint(
&meta.ShardInfo{ID: uint64(2), OwnerIDs: []uint64{uint64(1), uint64(2), uint64(3)}},
pr.Points[1])
sm.MapPoint(
&meta.ShardInfo{ID: uint64(2), OwnerIDs: []uint64{uint64(1), uint64(2), uint64(3)}},
pr.Points[2])
// Local cluster.Node ShardWriter
// lock on the write increment since these functions get called in parallel
var mu sync.Mutex
sw := &fakeShardWriter{
ShardWriteFn: func(shardID, nodeID uint64, points []tsdb.Point) error {
mu.Lock()
defer mu.Unlock()
return theTest.err[int(nodeID)-1]
},
}
store := &fakeStore{
WriteFn: func(shardID uint64, points []tsdb.Point) error {
mu.Lock()
defer mu.Unlock()
return theTest.err[0]
},
}
hh := &fakeShardWriter{
ShardWriteFn: func(shardID, nodeID uint64, points []tsdb.Point) error {
return nil
},
}
ms := NewMetaStore()
ms.DatabaseFn = func(database string) (*meta.DatabaseInfo, error) {
return nil, nil
}
ms.NodeIDFn = func() uint64 { return 1 }
c := cluster.NewPointsWriter()
c.MetaStore = ms
c.ShardWriter = sw
c.TSDBStore = store
c.HintedHandoff = hh
err := c.WritePoints(pr)
if err == nil && test.expErr != nil {
t.Errorf("PointsWriter.WritePoints(): '%s' error: got %v, exp %v", test.name, err, test.expErr)
}
if err != nil && test.expErr == nil {
t.Errorf("PointsWriter.WritePoints(): '%s' error: got %v, exp %v", test.name, err, test.expErr)
}
if err != nil && test.expErr != nil && err.Error() != test.expErr.Error() {
t.Errorf("PointsWriter.WritePoints(): '%s' error: got %v, exp %v", test.name, err, test.expErr)
}
}
}
var shardID uint64
type fakeShardWriter struct {
ShardWriteFn func(shardID, nodeID uint64, points []tsdb.Point) error
}
func (f *fakeShardWriter) WriteShard(shardID, nodeID uint64, points []tsdb.Point) error {
return f.ShardWriteFn(shardID, nodeID, points)
}
type fakeStore struct {
WriteFn func(shardID uint64, points []tsdb.Point) error
CreateShardfn func(database, retentionPolicy string, shardID uint64) error
}
func (f *fakeStore) WriteToShard(shardID uint64, points []tsdb.Point) error {
return f.WriteFn(shardID, points)
}
func (f *fakeStore) CreateShard(database, retentionPolicy string, shardID uint64) error {
return f.CreateShardfn(database, retentionPolicy, shardID)
}
func NewMetaStore() *MetaStore {
ms := &MetaStore{}
rp := NewRetentionPolicy("myp", time.Hour, 3)
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
AttachShardGroupInfo(rp, []uint64{1, 2, 3})
ms.RetentionPolicyFn = func(db, retentionPolicy string) (*meta.RetentionPolicyInfo, error) {
return rp, nil
}
ms.CreateShardGroupIfNotExistsFn = func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
for i, sg := range rp.ShardGroups {
if timestamp.Equal(sg.StartTime) || timestamp.After(sg.StartTime) && timestamp.Before(sg.EndTime) {
return &rp.ShardGroups[i], nil
}
}
panic("should not get here")
}
return ms
}
type MetaStore struct {
NodeIDFn func() uint64
RetentionPolicyFn func(database, name string) (*meta.RetentionPolicyInfo, error)
CreateShardGroupIfNotExistsFn func(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error)
DatabaseFn func(database string) (*meta.DatabaseInfo, error)
ShardOwnerFn func(shardID uint64) (string, string, *meta.ShardGroupInfo)
}
func (m MetaStore) NodeID() uint64 { return m.NodeIDFn() }
func (m MetaStore) RetentionPolicy(database, name string) (*meta.RetentionPolicyInfo, error) {
return m.RetentionPolicyFn(database, name)
}
func (m MetaStore) CreateShardGroupIfNotExists(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error) {
return m.CreateShardGroupIfNotExistsFn(database, policy, timestamp)
}
func (m MetaStore) Database(database string) (*meta.DatabaseInfo, error) {
return m.DatabaseFn(database)
}
func (m MetaStore) ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo) {
return m.ShardOwnerFn(shardID)
}
func NewRetentionPolicy(name string, duration time.Duration, nodeCount int) *meta.RetentionPolicyInfo {
shards := []meta.ShardInfo{}
ownerIDs := []uint64{}
for i := 1; i <= nodeCount; i++ {
ownerIDs = append(ownerIDs, uint64(i))
}
// each node is fully replicated with each other
shards = append(shards, meta.ShardInfo{
ID: nextShardID(),
OwnerIDs: ownerIDs,
})
rp := &meta.RetentionPolicyInfo{
Name: "myrp",
ReplicaN: nodeCount,
Duration: duration,
ShardGroupDuration: duration,
ShardGroups: []meta.ShardGroupInfo{
meta.ShardGroupInfo{
ID: nextShardID(),
StartTime: time.Unix(0, 0),
EndTime: time.Unix(0, 0).Add(duration).Add(-1),
Shards: shards,
},
},
}
return rp
}
func AttachShardGroupInfo(rp *meta.RetentionPolicyInfo, ownerIDs []uint64) {
var startTime, endTime time.Time
if len(rp.ShardGroups) == 0 {
startTime = time.Unix(0, 0)
} else {
startTime = rp.ShardGroups[len(rp.ShardGroups)-1].StartTime.Add(rp.ShardGroupDuration)
}
endTime = startTime.Add(rp.ShardGroupDuration).Add(-1)
sh := meta.ShardGroupInfo{
ID: uint64(len(rp.ShardGroups) + 1),
StartTime: startTime,
EndTime: endTime,
Shards: []meta.ShardInfo{
meta.ShardInfo{
ID: nextShardID(),
OwnerIDs: ownerIDs,
},
},
}
rp.ShardGroups = append(rp.ShardGroups, sh)
}
func nextShardID() uint64 {
return atomic.AddUint64(&shardID, 1)
}

229
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/rpc.go generated vendored Normal file
View File

@ -0,0 +1,229 @@
package cluster
import (
"time"
"github.com/gogo/protobuf/proto"
"github.com/influxdb/influxdb/cluster/internal"
"github.com/influxdb/influxdb/tsdb"
)
//go:generate protoc --gogo_out=. internal/data.proto
// MapShardRequest represents the request to map a remote shard for a query.
type MapShardRequest struct {
pb internal.MapShardRequest
}
func (m *MapShardRequest) ShardID() uint64 { return m.pb.GetShardID() }
func (m *MapShardRequest) Query() string { return m.pb.GetQuery() }
func (m *MapShardRequest) ChunkSize() int32 { return m.pb.GetChunkSize() }
func (m *MapShardRequest) SetShardID(id uint64) { m.pb.ShardID = &id }
func (m *MapShardRequest) SetQuery(query string) { m.pb.Query = &query }
func (m *MapShardRequest) SetChunkSize(chunkSize int32) { m.pb.ChunkSize = &chunkSize }
// MarshalBinary encodes the object to a binary format.
func (m *MapShardRequest) MarshalBinary() ([]byte, error) {
return proto.Marshal(&m.pb)
}
// UnmarshalBinary populates MapShardRequest from a binary format.
func (m *MapShardRequest) UnmarshalBinary(buf []byte) error {
if err := proto.Unmarshal(buf, &m.pb); err != nil {
return err
}
return nil
}
// MapShardResponse represents the response returned from a remote MapShardRequest call
type MapShardResponse struct {
pb internal.MapShardResponse
}
func NewMapShardResponse(code int, message string) *MapShardResponse {
m := &MapShardResponse{}
m.SetCode(code)
m.SetMessage(message)
return m
}
func (r *MapShardResponse) Code() int { return int(r.pb.GetCode()) }
func (r *MapShardResponse) Message() string { return r.pb.GetMessage() }
func (r *MapShardResponse) TagSets() []string { return r.pb.GetTagSets() }
func (r *MapShardResponse) Fields() []string { return r.pb.GetFields() }
func (r *MapShardResponse) Data() []byte { return r.pb.GetData() }
func (r *MapShardResponse) SetCode(code int) { r.pb.Code = proto.Int32(int32(code)) }
func (r *MapShardResponse) SetMessage(message string) { r.pb.Message = &message }
func (r *MapShardResponse) SetTagSets(tagsets []string) { r.pb.TagSets = tagsets }
func (r *MapShardResponse) SetFields(fields []string) { r.pb.Fields = fields }
func (r *MapShardResponse) SetData(data []byte) { r.pb.Data = data }
// MarshalBinary encodes the object to a binary format.
func (r *MapShardResponse) MarshalBinary() ([]byte, error) {
return proto.Marshal(&r.pb)
}
// UnmarshalBinary populates WritePointRequest from a binary format.
func (r *MapShardResponse) UnmarshalBinary(buf []byte) error {
if err := proto.Unmarshal(buf, &r.pb); err != nil {
return err
}
return nil
}
// WritePointsRequest represents a request to write point data to the cluster
type WritePointsRequest struct {
Database string
RetentionPolicy string
ConsistencyLevel ConsistencyLevel
Points []tsdb.Point
}
// AddPoint adds a point to the WritePointRequest with field name 'value'
func (w *WritePointsRequest) AddPoint(name string, value interface{}, timestamp time.Time, tags map[string]string) {
w.Points = append(w.Points, tsdb.NewPoint(
name, tags, map[string]interface{}{"value": value}, timestamp,
))
}
// WriteShardRequest represents the a request to write a slice of points to a shard
type WriteShardRequest struct {
pb internal.WriteShardRequest
}
// WriteShardResponse represents the response returned from a remote WriteShardRequest call
type WriteShardResponse struct {
pb internal.WriteShardResponse
}
func (w *WriteShardRequest) SetShardID(id uint64) { w.pb.ShardID = &id }
func (w *WriteShardRequest) ShardID() uint64 { return w.pb.GetShardID() }
func (w *WriteShardRequest) Points() []tsdb.Point { return w.unmarshalPoints() }
func (w *WriteShardRequest) AddPoint(name string, value interface{}, timestamp time.Time, tags map[string]string) {
w.AddPoints([]tsdb.Point{tsdb.NewPoint(
name, tags, map[string]interface{}{"value": value}, timestamp,
)})
}
func (w *WriteShardRequest) AddPoints(points []tsdb.Point) {
w.pb.Points = append(w.pb.Points, w.marshalPoints(points)...)
}
// MarshalBinary encodes the object to a binary format.
func (w *WriteShardRequest) MarshalBinary() ([]byte, error) {
return proto.Marshal(&w.pb)
}
func (w *WriteShardRequest) marshalPoints(points []tsdb.Point) []*internal.Point {
pts := make([]*internal.Point, len(points))
for i, p := range points {
fields := []*internal.Field{}
for k, v := range p.Fields() {
name := k
f := &internal.Field{
Name: &name,
}
switch t := v.(type) {
case int:
f.Int64 = proto.Int64(int64(t))
case int32:
f.Int32 = proto.Int32(t)
case int64:
f.Int64 = proto.Int64(t)
case float64:
f.Float64 = proto.Float64(t)
case bool:
f.Bool = proto.Bool(t)
case string:
f.String_ = proto.String(t)
case []byte:
f.Bytes = t
}
fields = append(fields, f)
}
tags := []*internal.Tag{}
for k, v := range p.Tags() {
key := k
value := v
tags = append(tags, &internal.Tag{
Key: &key,
Value: &value,
})
}
name := p.Name()
pts[i] = &internal.Point{
Name: &name,
Time: proto.Int64(p.Time().UnixNano()),
Fields: fields,
Tags: tags,
}
}
return pts
}
// UnmarshalBinary populates WritePointRequest from a binary format.
func (w *WriteShardRequest) UnmarshalBinary(buf []byte) error {
if err := proto.Unmarshal(buf, &w.pb); err != nil {
return err
}
return nil
}
func (w *WriteShardRequest) unmarshalPoints() []tsdb.Point {
points := make([]tsdb.Point, len(w.pb.GetPoints()))
for i, p := range w.pb.GetPoints() {
pt := tsdb.NewPoint(
p.GetName(), map[string]string{},
map[string]interface{}{}, time.Unix(0, p.GetTime()))
for _, f := range p.GetFields() {
n := f.GetName()
if f.Int32 != nil {
pt.AddField(n, f.GetInt32())
} else if f.Int64 != nil {
pt.AddField(n, f.GetInt64())
} else if f.Float64 != nil {
pt.AddField(n, f.GetFloat64())
} else if f.Bool != nil {
pt.AddField(n, f.GetBool())
} else if f.String_ != nil {
pt.AddField(n, f.GetString_())
} else {
pt.AddField(n, f.GetBytes())
}
}
tags := tsdb.Tags{}
for _, t := range p.GetTags() {
tags[t.GetKey()] = t.GetValue()
}
pt.SetTags(tags)
points[i] = pt
}
return points
}
func (w *WriteShardResponse) SetCode(code int) { w.pb.Code = proto.Int32(int32(code)) }
func (w *WriteShardResponse) SetMessage(message string) { w.pb.Message = &message }
func (w *WriteShardResponse) Code() int { return int(w.pb.GetCode()) }
func (w *WriteShardResponse) Message() string { return w.pb.GetMessage() }
// MarshalBinary encodes the object to a binary format.
func (w *WriteShardResponse) MarshalBinary() ([]byte, error) {
return proto.Marshal(&w.pb)
}
// UnmarshalBinary populates WritePointRequest from a binary format.
func (w *WriteShardResponse) UnmarshalBinary(buf []byte) error {
if err := proto.Unmarshal(buf, &w.pb); err != nil {
return err
}
return nil
}

110
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/rpc_test.go generated vendored Normal file
View File

@ -0,0 +1,110 @@
package cluster
import (
"testing"
"time"
)
func TestWriteShardRequestBinary(t *testing.T) {
sr := &WriteShardRequest{}
sr.SetShardID(uint64(1))
if exp := uint64(1); sr.ShardID() != exp {
t.Fatalf("ShardID mismatch: got %v, exp %v", sr.ShardID(), exp)
}
sr.AddPoint("cpu", 1.0, time.Unix(0, 0), map[string]string{"host": "serverA"})
sr.AddPoint("cpu", 2.0, time.Unix(0, 0).Add(time.Hour), nil)
sr.AddPoint("cpu_load", 3.0, time.Unix(0, 0).Add(time.Hour+time.Second), nil)
b, err := sr.MarshalBinary()
if err != nil {
t.Fatalf("WritePointsRequest.MarshalBinary() failed: %v", err)
}
if len(b) == 0 {
t.Fatalf("WritePointsRequest.MarshalBinary() returned 0 bytes")
}
got := &WriteShardRequest{}
if err := got.UnmarshalBinary(b); err != nil {
t.Fatalf("WritePointsRequest.UnmarshalMarshalBinary() failed: %v", err)
}
if got.ShardID() != sr.ShardID() {
t.Errorf("ShardID mismatch: got %v, exp %v", got.ShardID(), sr.ShardID())
}
if len(got.Points()) != len(sr.Points()) {
t.Errorf("Points count mismatch: got %v, exp %v", len(got.Points()), len(sr.Points()))
}
srPoints := sr.Points()
gotPoints := got.Points()
for i, p := range srPoints {
g := gotPoints[i]
if g.Name() != p.Name() {
t.Errorf("Point %d name mismatch: got %v, exp %v", i, g.Name(), p.Name())
}
if !g.Time().Equal(p.Time()) {
t.Errorf("Point %d time mismatch: got %v, exp %v", i, g.Time(), p.Time())
}
if g.HashID() != p.HashID() {
t.Errorf("Point #%d HashID() mismatch: got %v, exp %v", i, g.HashID(), p.HashID())
}
for k, v := range p.Tags() {
if g.Tags()[k] != v {
t.Errorf("Point #%d tag mismatch: got %v, exp %v", i, k, v)
}
}
if len(p.Fields()) != len(g.Fields()) {
t.Errorf("Point %d field count mismatch: got %v, exp %v", i, len(g.Fields()), len(p.Fields()))
}
for j, f := range p.Fields() {
if g.Fields()[j] != f {
t.Errorf("Point %d field mismatch: got %v, exp %v", i, g.Fields()[j], f)
}
}
}
}
func TestWriteShardResponseBinary(t *testing.T) {
sr := &WriteShardResponse{}
sr.SetCode(10)
sr.SetMessage("foo")
b, err := sr.MarshalBinary()
if exp := 10; sr.Code() != exp {
t.Fatalf("Code mismatch: got %v, exp %v", sr.Code(), exp)
}
if exp := "foo"; sr.Message() != exp {
t.Fatalf("Message mismatch: got %v, exp %v", sr.Message(), exp)
}
if err != nil {
t.Fatalf("WritePointsResponse.MarshalBinary() failed: %v", err)
}
if len(b) == 0 {
t.Fatalf("WritePointsResponse.MarshalBinary() returned 0 bytes")
}
got := &WriteShardResponse{}
if err := got.UnmarshalBinary(b); err != nil {
t.Fatalf("WritePointsResponse.UnmarshalMarshalBinary() failed: %v", err)
}
if got.Code() != sr.Code() {
t.Errorf("Code mismatch: got %v, exp %v", got.Code(), sr.Code())
}
if got.Message() != sr.Message() {
t.Errorf("Message mismatch: got %v, exp %v", got.Message(), sr.Message())
}
}

338
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/service.go generated vendored Normal file
View File

@ -0,0 +1,338 @@
package cluster
import (
"encoding/binary"
"encoding/json"
"fmt"
"io"
"log"
"net"
"os"
"strings"
"sync"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/tsdb"
)
// MaxMessageSize defines how large a message can be before we reject it
const MaxMessageSize = 1024 * 1024 * 1024 // 1GB
// MuxHeader is the header byte used in the TCP mux.
const MuxHeader = 2
// Service processes data received over raw TCP connections.
type Service struct {
mu sync.RWMutex
wg sync.WaitGroup
closing chan struct{}
Listener net.Listener
MetaStore interface {
ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo)
}
TSDBStore interface {
CreateShard(database, policy string, shardID uint64) error
WriteToShard(shardID uint64, points []tsdb.Point) error
CreateMapper(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error)
}
Logger *log.Logger
}
// NewService returns a new instance of Service.
func NewService(c Config) *Service {
return &Service{
closing: make(chan struct{}),
Logger: log.New(os.Stderr, "[tcp] ", log.LstdFlags),
}
}
// Open opens the network listener and begins serving requests.
func (s *Service) Open() error {
s.Logger.Println("Starting cluster service")
// Begin serving conections.
s.wg.Add(1)
go s.serve()
return nil
}
// SetLogger sets the internal logger to the logger passed in.
func (s *Service) SetLogger(l *log.Logger) {
s.Logger = l
}
// serve accepts connections from the listener and handles them.
func (s *Service) serve() {
defer s.wg.Done()
for {
// Check if the service is shutting down.
select {
case <-s.closing:
return
default:
}
// Accept the next connection.
conn, err := s.Listener.Accept()
if err != nil {
if strings.Contains(err.Error(), "connection closed") {
s.Logger.Printf("cluster service accept error: %s", err)
return
}
s.Logger.Printf("accept error: %s", err)
continue
}
// Delegate connection handling to a separate goroutine.
s.wg.Add(1)
go func() {
defer s.wg.Done()
s.handleConn(conn)
}()
}
}
// Close shuts down the listener and waits for all connections to finish.
func (s *Service) Close() error {
if s.Listener != nil {
s.Listener.Close()
}
// Shut down all handlers.
close(s.closing)
// s.wg.Wait() // FIXME(benbjohnson)
return nil
}
// handleConn services an individual TCP connection.
func (s *Service) handleConn(conn net.Conn) {
// Ensure connection is closed when service is closed.
closing := make(chan struct{})
defer close(closing)
go func() {
select {
case <-closing:
case <-s.closing:
}
conn.Close()
}()
s.Logger.Printf("accept remote write connection from %v\n", conn.RemoteAddr())
defer func() {
s.Logger.Printf("close remote write connection from %v\n", conn.RemoteAddr())
}()
for {
// Read type-length-value.
typ, buf, err := ReadTLV(conn)
if err != nil {
if strings.HasSuffix(err.Error(), "EOF") {
return
}
s.Logger.Printf("unable to read type-length-value %s", err)
return
}
// Delegate message processing by type.
switch typ {
case writeShardRequestMessage:
err := s.processWriteShardRequest(buf)
if err != nil {
s.Logger.Printf("process write shard error: %s", err)
}
s.writeShardResponse(conn, err)
case mapShardRequestMessage:
err := s.processMapShardRequest(conn, buf)
if err != nil {
s.Logger.Printf("process map shard error: %s", err)
if err := writeMapShardResponseMessage(conn, NewMapShardResponse(1, err.Error())); err != nil {
s.Logger.Printf("process map shard error writing response: %s", err.Error())
}
}
default:
s.Logger.Printf("cluster service message type not found: %d", typ)
}
}
}
func (s *Service) processWriteShardRequest(buf []byte) error {
// Build request
var req WriteShardRequest
if err := req.UnmarshalBinary(buf); err != nil {
return err
}
err := s.TSDBStore.WriteToShard(req.ShardID(), req.Points())
// We may have received a write for a shard that we don't have locally because the
// sending node may have just created the shard (via the metastore) and the write
// arrived before the local store could create the shard. In this case, we need
// to check the metastore to determine what database and retention policy this
// shard should reside within.
if err == tsdb.ErrShardNotFound {
// Query the metastore for the owner of this shard
database, retentionPolicy, sgi := s.MetaStore.ShardOwner(req.ShardID())
if sgi == nil {
// If we can't find it, then we need to drop this request
// as it is no longer valid. This could happen if writes were queued via
// hinted handoff and delivered after a shard group was deleted.
s.Logger.Printf("drop write request: shard=%d", req.ShardID())
return nil
}
err = s.TSDBStore.CreateShard(database, retentionPolicy, req.ShardID())
if err != nil {
return err
}
return s.TSDBStore.WriteToShard(req.ShardID(), req.Points())
}
if err != nil {
return fmt.Errorf("write shard %d: %s", req.ShardID(), err)
}
return nil
}
func (s *Service) writeShardResponse(w io.Writer, e error) {
// Build response.
var resp WriteShardResponse
if e != nil {
resp.SetCode(1)
resp.SetMessage(e.Error())
} else {
resp.SetCode(0)
}
// Marshal response to binary.
buf, err := resp.MarshalBinary()
if err != nil {
s.Logger.Printf("error marshalling shard response: %s", err)
return
}
// Write to connection.
if err := WriteTLV(w, writeShardResponseMessage, buf); err != nil {
s.Logger.Printf("write shard response error: %s", err)
}
}
func (s *Service) processMapShardRequest(w io.Writer, buf []byte) error {
// Decode request
var req MapShardRequest
if err := req.UnmarshalBinary(buf); err != nil {
return err
}
m, err := s.TSDBStore.CreateMapper(req.ShardID(), req.Query(), int(req.ChunkSize()))
if err != nil {
return fmt.Errorf("create mapper: %s", err)
}
if m == nil {
return writeMapShardResponseMessage(w, NewMapShardResponse(0, ""))
}
if err := m.Open(); err != nil {
return fmt.Errorf("mapper open: %s", err)
}
defer m.Close()
var metaSent bool
for {
var resp MapShardResponse
if !metaSent {
resp.SetTagSets(m.TagSets())
resp.SetFields(m.Fields())
metaSent = true
}
chunk, err := m.NextChunk()
if err != nil {
return fmt.Errorf("next chunk: %s", err)
}
if chunk != nil {
b, err := json.Marshal(chunk)
if err != nil {
return fmt.Errorf("encoding: %s", err)
}
resp.SetData(b)
}
// Write to connection.
resp.SetCode(0)
if err := writeMapShardResponseMessage(w, &resp); err != nil {
return err
}
if chunk == nil {
// All mapper data sent.
return nil
}
}
}
func writeMapShardResponseMessage(w io.Writer, msg *MapShardResponse) error {
buf, err := msg.MarshalBinary()
if err != nil {
return err
}
return WriteTLV(w, mapShardResponseMessage, buf)
}
// ReadTLV reads a type-length-value record from r.
func ReadTLV(r io.Reader) (byte, []byte, error) {
var typ [1]byte
if _, err := io.ReadFull(r, typ[:]); err != nil {
return 0, nil, fmt.Errorf("read message type: %s", err)
}
// Read the size of the message.
var sz int64
if err := binary.Read(r, binary.BigEndian, &sz); err != nil {
return 0, nil, fmt.Errorf("read message size: %s", err)
}
if sz == 0 {
return 0, nil, fmt.Errorf("invalid message size: %d", sz)
}
if sz >= MaxMessageSize {
return 0, nil, fmt.Errorf("max message size of %d exceeded: %d", MaxMessageSize, sz)
}
// Read the value.
buf := make([]byte, sz)
if _, err := io.ReadFull(r, buf); err != nil {
return 0, nil, fmt.Errorf("read message value: %s", err)
}
return typ[0], buf, nil
}
// WriteTLV writes a type-length-value record to w.
func WriteTLV(w io.Writer, typ byte, buf []byte) error {
if _, err := w.Write([]byte{typ}); err != nil {
return fmt.Errorf("write message type: %s", err)
}
// Write the size of the message.
if err := binary.Write(w, binary.BigEndian, int64(len(buf))); err != nil {
return fmt.Errorf("write message size: %s", err)
}
// Write the value.
if _, err := w.Write(buf); err != nil {
return fmt.Errorf("write message value: %s", err)
}
return nil
}

103
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/service_test.go generated vendored Normal file
View File

@ -0,0 +1,103 @@
package cluster_test
import (
"fmt"
"net"
"time"
"github.com/influxdb/influxdb/cluster"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/tcp"
"github.com/influxdb/influxdb/tsdb"
)
type metaStore struct {
host string
}
func (m *metaStore) Node(nodeID uint64) (*meta.NodeInfo, error) {
return &meta.NodeInfo{
ID: nodeID,
Host: m.host,
}, nil
}
type testService struct {
nodeID uint64
ln net.Listener
muxln net.Listener
writeShardFunc func(shardID uint64, points []tsdb.Point) error
createShardFunc func(database, policy string, shardID uint64) error
createMapperFunc func(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error)
}
func newTestWriteService(f func(shardID uint64, points []tsdb.Point) error) testService {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
mux := tcp.NewMux()
muxln := mux.Listen(cluster.MuxHeader)
go mux.Serve(ln)
return testService{
writeShardFunc: f,
ln: ln,
muxln: muxln,
}
}
func (ts *testService) Close() {
if ts.ln != nil {
ts.ln.Close()
}
}
type serviceResponses []serviceResponse
type serviceResponse struct {
shardID uint64
ownerID uint64
points []tsdb.Point
}
func (t testService) WriteToShard(shardID uint64, points []tsdb.Point) error {
return t.writeShardFunc(shardID, points)
}
func (t testService) CreateShard(database, policy string, shardID uint64) error {
return t.createShardFunc(database, policy, shardID)
}
func (t testService) CreateMapper(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error) {
return t.createMapperFunc(shardID, query, chunkSize)
}
func writeShardSuccess(shardID uint64, points []tsdb.Point) error {
responses <- &serviceResponse{
shardID: shardID,
points: points,
}
return nil
}
func writeShardFail(shardID uint64, points []tsdb.Point) error {
return fmt.Errorf("failed to write")
}
var responses = make(chan *serviceResponse, 1024)
func (testService) ResponseN(n int) ([]*serviceResponse, error) {
var a []*serviceResponse
for {
select {
case r := <-responses:
a = append(a, r)
if len(a) == n {
return a, nil
}
case <-time.After(time.Second):
return a, fmt.Errorf("unexpected response count: expected: %d, actual: %d", n, len(a))
}
}
}

207
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_mapper.go generated vendored Normal file
View File

@ -0,0 +1,207 @@
package cluster
import (
"encoding/json"
"fmt"
"io"
"math/rand"
"net"
"time"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/tsdb"
"gopkg.in/fatih/pool.v2"
)
// ShardMapper is responsible for providing mappers for requested shards. It is
// responsible for creating those mappers from the local store, or reaching
// out to another node on the cluster.
type ShardMapper struct {
ForceRemoteMapping bool // All shards treated as remote. Useful for testing.
MetaStore interface {
NodeID() uint64
Node(id uint64) (ni *meta.NodeInfo, err error)
}
TSDBStore interface {
CreateMapper(shardID uint64, query string, chunkSize int) (tsdb.Mapper, error)
}
timeout time.Duration
pool *clientPool
}
// NewShardMapper returns a mapper of local and remote shards.
func NewShardMapper(timeout time.Duration) *ShardMapper {
return &ShardMapper{
pool: newClientPool(),
timeout: timeout,
}
}
// CreateMapper returns a Mapper for the given shard ID.
func (s *ShardMapper) CreateMapper(sh meta.ShardInfo, stmt string, chunkSize int) (tsdb.Mapper, error) {
var err error
var m tsdb.Mapper
if sh.OwnedBy(s.MetaStore.NodeID()) && !s.ForceRemoteMapping {
m, err = s.TSDBStore.CreateMapper(sh.ID, stmt, chunkSize)
if err != nil {
return nil, err
}
} else {
// Pick a node in a pseudo-random manner.
conn, err := s.dial(sh.OwnerIDs[rand.Intn(len(sh.OwnerIDs))])
if err != nil {
return nil, err
}
conn.SetDeadline(time.Now().Add(s.timeout))
rm := NewRemoteMapper(conn.(*pool.PoolConn), sh.ID, stmt, chunkSize)
m = rm
}
return m, nil
}
func (s *ShardMapper) dial(nodeID uint64) (net.Conn, error) {
// If we don't have a connection pool for that addr yet, create one
_, ok := s.pool.getPool(nodeID)
if !ok {
factory := &connFactory{nodeID: nodeID, clientPool: s.pool, timeout: s.timeout}
factory.metaStore = s.MetaStore
p, err := pool.NewChannelPool(1, 3, factory.dial)
if err != nil {
return nil, err
}
s.pool.setPool(nodeID, p)
}
return s.pool.conn(nodeID)
}
type remoteShardConn interface {
io.ReadWriter
Close() error
MarkUnusable()
}
// RemoteMapper implements the tsdb.Mapper interface. It connects to a remote node,
// sends a query, and interprets the stream of data that comes back.
type RemoteMapper struct {
shardID uint64
stmt string
chunkSize int
tagsets []string
fields []string
conn remoteShardConn
bufferedResponse *MapShardResponse
}
// NewRemoteMapper returns a new remote mapper using the given connection.
func NewRemoteMapper(c remoteShardConn, shardID uint64, stmt string, chunkSize int) *RemoteMapper {
return &RemoteMapper{
conn: c,
shardID: shardID,
stmt: stmt,
chunkSize: chunkSize,
}
}
// Open connects to the remote node and starts receiving data.
func (r *RemoteMapper) Open() (err error) {
defer func() {
if err != nil {
r.conn.Close()
}
}()
// Build Map request.
var request MapShardRequest
request.SetShardID(r.shardID)
request.SetQuery(r.stmt)
request.SetChunkSize(int32(r.chunkSize))
// Marshal into protocol buffers.
buf, err := request.MarshalBinary()
if err != nil {
return err
}
// Write request.
if err := WriteTLV(r.conn, mapShardRequestMessage, buf); err != nil {
r.conn.MarkUnusable()
return err
}
// Read the response.
_, buf, err = ReadTLV(r.conn)
if err != nil {
r.conn.MarkUnusable()
return err
}
// Unmarshal response.
r.bufferedResponse = &MapShardResponse{}
if err := r.bufferedResponse.UnmarshalBinary(buf); err != nil {
return err
}
if r.bufferedResponse.Code() != 0 {
return fmt.Errorf("error code %d: %s", r.bufferedResponse.Code(), r.bufferedResponse.Message())
}
// Decode the first response to get the TagSets.
r.tagsets = r.bufferedResponse.TagSets()
return nil
}
func (r *RemoteMapper) TagSets() []string {
return r.tagsets
}
func (r *RemoteMapper) Fields() []string {
return r.fields
}
// NextChunk returns the next chunk read from the remote node to the client.
func (r *RemoteMapper) NextChunk() (chunk interface{}, err error) {
output := &tsdb.MapperOutput{}
var response *MapShardResponse
if r.bufferedResponse != nil {
response = r.bufferedResponse
r.bufferedResponse = nil
} else {
response = &MapShardResponse{}
// Read the response.
_, buf, err := ReadTLV(r.conn)
if err != nil {
r.conn.MarkUnusable()
return nil, err
}
// Unmarshal response.
if err := response.UnmarshalBinary(buf); err != nil {
return nil, err
}
if response.Code() != 0 {
return nil, fmt.Errorf("error code %d: %s", response.Code(), response.Message())
}
}
if response.Data() == nil {
return nil, nil
}
err = json.Unmarshal(response.Data(), output)
return output, err
}
// Close the Mapper
func (r *RemoteMapper) Close() {
r.conn.Close()
}

96
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_mapper_test.go generated vendored Normal file
View File

@ -0,0 +1,96 @@
package cluster
import (
"bytes"
"encoding/json"
"io"
"testing"
"github.com/influxdb/influxdb/tsdb"
)
// remoteShardResponder implements the remoteShardConn interface.
type remoteShardResponder struct {
t *testing.T
rxBytes []byte
buffer *bytes.Buffer
}
func newRemoteShardResponder(outputs []*tsdb.MapperOutput, tagsets []string) *remoteShardResponder {
r := &remoteShardResponder{}
a := make([]byte, 0, 1024)
r.buffer = bytes.NewBuffer(a)
// Pump the outputs in the buffer for later reading.
for _, o := range outputs {
resp := &MapShardResponse{}
resp.SetCode(0)
if o != nil {
d, _ := json.Marshal(o)
resp.SetData(d)
resp.SetTagSets(tagsets)
}
g, _ := resp.MarshalBinary()
WriteTLV(r.buffer, mapShardResponseMessage, g)
}
return r
}
func (r remoteShardResponder) MarkUnusable() { return }
func (r remoteShardResponder) Close() error { return nil }
func (r remoteShardResponder) Read(p []byte) (n int, err error) {
return io.ReadFull(r.buffer, p)
}
func (r remoteShardResponder) Write(p []byte) (n int, err error) {
if r.rxBytes == nil {
r.rxBytes = make([]byte, 0)
}
r.rxBytes = append(r.rxBytes, p...)
return len(p), nil
}
// Ensure a RemoteMapper can process valid responses from a remote shard.
func TestShardWriter_RemoteMapper_Success(t *testing.T) {
expTagSets := []string{"tagsetA"}
expOutput := &tsdb.MapperOutput{
Name: "cpu",
Tags: map[string]string{"host": "serverA"},
}
c := newRemoteShardResponder([]*tsdb.MapperOutput{expOutput, nil}, expTagSets)
r := NewRemoteMapper(c, 1234, "SELECT * FROM CPU", 10)
if err := r.Open(); err != nil {
t.Fatalf("failed to open remote mapper: %s", err.Error())
}
if r.TagSets()[0] != expTagSets[0] {
t.Fatalf("incorrect tagsets received, exp %v, got %v", expTagSets, r.TagSets())
}
// Get first chunk from mapper.
chunk, err := r.NextChunk()
if err != nil {
t.Fatalf("failed to get next chunk from mapper: %s", err.Error())
}
output, ok := chunk.(*tsdb.MapperOutput)
if !ok {
t.Fatal("chunk is not of expected type")
}
if output.Name != "cpu" {
t.Fatalf("received output incorrect, exp: %v, got %v", expOutput, output)
}
// Next chunk should be nil, indicating no more data.
chunk, err = r.NextChunk()
if err != nil {
t.Fatalf("failed to get next chunk from mapper: %s", err.Error())
}
if chunk != nil {
t.Fatal("received more chunks when none expected")
}
}

163
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_writer.go generated vendored Normal file
View File

@ -0,0 +1,163 @@
package cluster
import (
"fmt"
"net"
"time"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/tsdb"
"gopkg.in/fatih/pool.v2"
)
const (
writeShardRequestMessage byte = iota + 1
writeShardResponseMessage
mapShardRequestMessage
mapShardResponseMessage
)
// ShardWriter writes a set of points to a shard.
type ShardWriter struct {
pool *clientPool
timeout time.Duration
MetaStore interface {
Node(id uint64) (ni *meta.NodeInfo, err error)
}
}
// NewShardWriter returns a new instance of ShardWriter.
func NewShardWriter(timeout time.Duration) *ShardWriter {
return &ShardWriter{
pool: newClientPool(),
timeout: timeout,
}
}
func (w *ShardWriter) WriteShard(shardID, ownerID uint64, points []tsdb.Point) error {
c, err := w.dial(ownerID)
if err != nil {
return err
}
conn, ok := c.(*pool.PoolConn)
if !ok {
panic("wrong connection type")
}
defer func(conn net.Conn) {
conn.Close() // return to pool
}(conn)
// Build write request.
var request WriteShardRequest
request.SetShardID(shardID)
request.AddPoints(points)
// Marshal into protocol buffers.
buf, err := request.MarshalBinary()
if err != nil {
return err
}
// Write request.
conn.SetWriteDeadline(time.Now().Add(w.timeout))
if err := WriteTLV(conn, writeShardRequestMessage, buf); err != nil {
conn.MarkUnusable()
return err
}
// Read the response.
conn.SetReadDeadline(time.Now().Add(w.timeout))
_, buf, err = ReadTLV(conn)
if err != nil {
conn.MarkUnusable()
return err
}
// Unmarshal response.
var response WriteShardResponse
if err := response.UnmarshalBinary(buf); err != nil {
return err
}
if response.Code() != 0 {
return fmt.Errorf("error code %d: %s", response.Code(), response.Message())
}
return nil
}
func (c *ShardWriter) dial(nodeID uint64) (net.Conn, error) {
// If we don't have a connection pool for that addr yet, create one
_, ok := c.pool.getPool(nodeID)
if !ok {
factory := &connFactory{nodeID: nodeID, clientPool: c.pool, timeout: c.timeout}
factory.metaStore = c.MetaStore
p, err := pool.NewChannelPool(1, 3, factory.dial)
if err != nil {
return nil, err
}
c.pool.setPool(nodeID, p)
}
return c.pool.conn(nodeID)
}
func (w *ShardWriter) Close() error {
if w.pool == nil {
return fmt.Errorf("client already closed")
}
w.pool.close()
w.pool = nil
return nil
}
const (
maxConnections = 500
maxRetries = 3
)
var errMaxConnectionsExceeded = fmt.Errorf("can not exceed max connections of %d", maxConnections)
type connFactory struct {
nodeID uint64
timeout time.Duration
clientPool interface {
size() int
}
metaStore interface {
Node(id uint64) (ni *meta.NodeInfo, err error)
}
}
func (c *connFactory) dial() (net.Conn, error) {
if c.clientPool.size() > maxConnections {
return nil, errMaxConnectionsExceeded
}
ni, err := c.metaStore.Node(c.nodeID)
if err != nil {
return nil, err
}
if ni == nil {
return nil, fmt.Errorf("node %d does not exist", c.nodeID)
}
conn, err := net.DialTimeout("tcp", ni.Host, c.timeout)
if err != nil {
return nil, err
}
// Write a marker byte for cluster messages.
_, err = conn.Write([]byte{MuxHeader})
if err != nil {
conn.Close()
return nil, err
}
return conn, nil
}

186
Godeps/_workspace/src/github.com/influxdb/influxdb/cluster/shard_writer_test.go generated vendored Normal file
View File

@ -0,0 +1,186 @@
package cluster_test
import (
"net"
"strings"
"testing"
"time"
"github.com/influxdb/influxdb/cluster"
"github.com/influxdb/influxdb/tsdb"
)
// Ensure the shard writer can successful write a single request.
func TestShardWriter_WriteShard_Success(t *testing.T) {
ts := newTestWriteService(writeShardSuccess)
s := cluster.NewService(cluster.Config{})
s.Listener = ts.muxln
s.TSDBStore = ts
if err := s.Open(); err != nil {
t.Fatal(err)
}
defer s.Close()
defer ts.Close()
w := cluster.NewShardWriter(time.Minute)
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
// Build a single point.
now := time.Now()
var points []tsdb.Point
points = append(points, tsdb.NewPoint("cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now))
// Write to shard and close.
if err := w.WriteShard(1, 2, points); err != nil {
t.Fatal(err)
} else if err := w.Close(); err != nil {
t.Fatal(err)
}
// Validate response.
responses, err := ts.ResponseN(1)
if err != nil {
t.Fatal(err)
} else if responses[0].shardID != 1 {
t.Fatalf("unexpected shard id: %d", responses[0].shardID)
}
// Validate point.
if p := responses[0].points[0]; p.Name() != "cpu" {
t.Fatalf("unexpected name: %s", p.Name())
} else if p.Fields()["value"] != int64(100) {
t.Fatalf("unexpected 'value' field: %d", p.Fields()["value"])
} else if p.Tags()["host"] != "server01" {
t.Fatalf("unexpected 'host' tag: %s", p.Tags()["host"])
} else if p.Time().UnixNano() != now.UnixNano() {
t.Fatalf("unexpected time: %s", p.Time())
}
}
// Ensure the shard writer can successful write a multiple requests.
func TestShardWriter_WriteShard_Multiple(t *testing.T) {
ts := newTestWriteService(writeShardSuccess)
s := cluster.NewService(cluster.Config{})
s.Listener = ts.muxln
s.TSDBStore = ts
if err := s.Open(); err != nil {
t.Fatal(err)
}
defer s.Close()
defer ts.Close()
w := cluster.NewShardWriter(time.Minute)
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
// Build a single point.
now := time.Now()
var points []tsdb.Point
points = append(points, tsdb.NewPoint("cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now))
// Write to shard twice and close.
if err := w.WriteShard(1, 2, points); err != nil {
t.Fatal(err)
} else if err := w.WriteShard(1, 2, points); err != nil {
t.Fatal(err)
} else if err := w.Close(); err != nil {
t.Fatal(err)
}
// Validate response.
responses, err := ts.ResponseN(1)
if err != nil {
t.Fatal(err)
} else if responses[0].shardID != 1 {
t.Fatalf("unexpected shard id: %d", responses[0].shardID)
}
// Validate point.
if p := responses[0].points[0]; p.Name() != "cpu" {
t.Fatalf("unexpected name: %s", p.Name())
} else if p.Fields()["value"] != int64(100) {
t.Fatalf("unexpected 'value' field: %d", p.Fields()["value"])
} else if p.Tags()["host"] != "server01" {
t.Fatalf("unexpected 'host' tag: %s", p.Tags()["host"])
} else if p.Time().UnixNano() != now.UnixNano() {
t.Fatalf("unexpected time: %s", p.Time())
}
}
// Ensure the shard writer returns an error when the server fails to accept the write.
func TestShardWriter_WriteShard_Error(t *testing.T) {
ts := newTestWriteService(writeShardFail)
s := cluster.NewService(cluster.Config{})
s.Listener = ts.muxln
s.TSDBStore = ts
if err := s.Open(); err != nil {
t.Fatal(err)
}
defer s.Close()
defer ts.Close()
w := cluster.NewShardWriter(time.Minute)
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
now := time.Now()
shardID := uint64(1)
ownerID := uint64(2)
var points []tsdb.Point
points = append(points, tsdb.NewPoint(
"cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now,
))
if err := w.WriteShard(shardID, ownerID, points); err == nil || err.Error() != "error code 1: write shard 1: failed to write" {
t.Fatalf("unexpected error: %v", err)
}
}
// Ensure the shard writer returns an error when dialing times out.
func TestShardWriter_Write_ErrDialTimeout(t *testing.T) {
ts := newTestWriteService(writeShardSuccess)
s := cluster.NewService(cluster.Config{})
s.Listener = ts.muxln
s.TSDBStore = ts
if err := s.Open(); err != nil {
t.Fatal(err)
}
defer s.Close()
defer ts.Close()
w := cluster.NewShardWriter(time.Nanosecond)
w.MetaStore = &metaStore{host: ts.ln.Addr().String()}
now := time.Now()
shardID := uint64(1)
ownerID := uint64(2)
var points []tsdb.Point
points = append(points, tsdb.NewPoint(
"cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now,
))
if err, exp := w.WriteShard(shardID, ownerID, points), "i/o timeout"; err == nil || !strings.Contains(err.Error(), exp) {
t.Fatalf("expected error %v, to contain %s", err, exp)
}
}
// Ensure the shard writer returns an error when reading times out.
func TestShardWriter_Write_ErrReadTimeout(t *testing.T) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
w := cluster.NewShardWriter(time.Millisecond)
w.MetaStore = &metaStore{host: ln.Addr().String()}
now := time.Now()
shardID := uint64(1)
ownerID := uint64(2)
var points []tsdb.Point
points = append(points, tsdb.NewPoint(
"cpu", tsdb.Tags{"host": "server01"}, map[string]interface{}{"value": int64(100)}, now,
))
if err := w.WriteShard(shardID, ownerID, points); err == nil || !strings.Contains(err.Error(), "i/o timeout") {
t.Fatalf("unexpected error: %s", err)
}
}

724
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx/main.go generated vendored Normal file
View File

@ -0,0 +1,724 @@
package main
import (
"encoding/csv"
"encoding/json"
"flag"
"fmt"
"io"
"net"
"net/url"
"os"
"os/user"
"path/filepath"
"sort"
"strconv"
"strings"
"text/tabwriter"
"github.com/influxdb/influxdb/client"
"github.com/influxdb/influxdb/importer/v8"
"github.com/peterh/liner"
)
// These variables are populated via the Go linker.
var (
version string = "0.9"
)
const (
// defaultFormat is the default format of the results when issuing queries
defaultFormat = "column"
// defaultPPS is the default points per second that the import will throttle at
// by default it's 0, which means it will not throttle
defaultPPS = 0
)
type CommandLine struct {
Client *client.Client
Line *liner.State
Host string
Port int
Username string
Password string
Database string
Ssl bool
RetentionPolicy string
Version string
Pretty bool // controls pretty print for json
Format string // controls the output format. Valid values are json, csv, or column
Execute string
ShowVersion bool
Import bool
PPS int // Controls how many points per second the import will allow via throttling
Path string
Compressed bool
}
func main() {
c := CommandLine{}
fs := flag.NewFlagSet("InfluxDB shell version "+version, flag.ExitOnError)
fs.StringVar(&c.Host, "host", client.DefaultHost, "Influxdb host to connect to.")
fs.IntVar(&c.Port, "port", client.DefaultPort, "Influxdb port to connect to.")
fs.StringVar(&c.Username, "username", c.Username, "Username to connect to the server.")
fs.StringVar(&c.Password, "password", c.Password, `Password to connect to the server. Leaving blank will prompt for password (--password="").`)
fs.StringVar(&c.Database, "database", c.Database, "Database to connect to the server.")
fs.BoolVar(&c.Ssl, "ssl", false, "Use https for connecting to cluster.")
fs.StringVar(&c.Format, "format", defaultFormat, "Format specifies the format of the server responses: json, csv, or column.")
fs.BoolVar(&c.Pretty, "pretty", false, "Turns on pretty print for the json format.")
fs.StringVar(&c.Execute, "execute", c.Execute, "Execute command and quit.")
fs.BoolVar(&c.ShowVersion, "version", false, "Displays the InfluxDB version.")
fs.BoolVar(&c.Import, "import", false, "Import a previous database.")
fs.IntVar(&c.PPS, "pps", defaultPPS, "How many points per second the import will allow. By default it is zero and will not throttle importing.")
fs.StringVar(&c.Path, "path", "", "path to the file to import")
fs.BoolVar(&c.Compressed, "compressed", false, "set to true if the import file is compressed")
// Define our own custom usage to print
fs.Usage = func() {
fmt.Println(`Usage of influx:
-version
Display the version and exit.
-host 'host name'
Host to connect to.
-port 'port #'
Port to connect to.
-database 'database name'
Database to connect to the server.
-password 'password'
Password to connect to the server. Leaving blank will prompt for password (--password '').
-username 'username'
Username to connect to the server.
-ssl
Use https for requests.
-execute 'command'
Execute command and quit.
-format 'json|csv|column'
Format specifies the format of the server responses: json, csv, or column.
-pretty
Turns on pretty print for the json format.
-import
Import a previous database export from file
-pps
How many points per second the import will allow. By default it is zero and will not throttle importing.
-path
Path to file to import
-compressed
Set to true if the import file is compressed
Examples:
# Use influx in a non-interactive mode to query the database "metrics" and pretty print json:
$ influx -database 'metrics' -execute 'select * from cpu' -format 'json' -pretty
# Connect to a specific database on startup and set database context:
$ influx -database 'metrics' -host 'localhost' -port '8086'
`)
}
fs.Parse(os.Args[1:])
if c.ShowVersion {
showVersion()
os.Exit(0)
}
var promptForPassword bool
// determine if they set the password flag but provided no value
for _, v := range os.Args {
v = strings.ToLower(v)
if (strings.HasPrefix(v, "-password") || strings.HasPrefix(v, "--password")) && c.Password == "" {
promptForPassword = true
break
}
}
c.Line = liner.NewLiner()
defer c.Line.Close()
if promptForPassword {
p, e := c.Line.PasswordPrompt("password: ")
if e != nil {
fmt.Println("Unable to parse password.")
} else {
c.Password = p
}
}
if err := c.connect(""); err != nil {
}
if c.Execute == "" && !c.Import {
fmt.Printf("Connected to %s version %s\n", c.Client.Addr(), c.Version)
}
if c.Execute != "" {
if err := c.ExecuteQuery(c.Execute); err != nil {
c.Line.Close()
os.Exit(1)
}
c.Line.Close()
os.Exit(0)
}
if c.Import {
path := net.JoinHostPort(c.Host, strconv.Itoa(c.Port))
u, e := client.ParseConnectionString(path, c.Ssl)
if e != nil {
fmt.Println(e)
return
}
config := v8.NewConfig()
config.Username = c.Username
config.Password = c.Password
config.Precision = "ns"
config.WriteConsistency = "any"
config.Path = c.Path
config.Version = version
config.URL = u
config.Compressed = c.Compressed
config.PPS = c.PPS
i := v8.NewImporter(config)
if err := i.Import(); err != nil {
fmt.Printf("ERROR: %s\n", err)
c.Line.Close()
os.Exit(1)
}
c.Line.Close()
os.Exit(0)
}
showVersion()
var historyFile string
usr, err := user.Current()
// Only load history if we can get the user
if err == nil {
historyFile = filepath.Join(usr.HomeDir, ".influx_history")
if f, err := os.Open(historyFile); err == nil {
c.Line.ReadHistory(f)
f.Close()
}
}
for {
l, e := c.Line.Prompt("> ")
if e != nil {
break
}
if c.ParseCommand(l) {
// write out the history
if len(historyFile) > 0 {
c.Line.AppendHistory(l)
if f, err := os.Create(historyFile); err == nil {
c.Line.WriteHistory(f)
f.Close()
}
}
} else {
break // exit main loop
}
}
}
func showVersion() {
fmt.Println("InfluxDB shell " + version)
}
func (c *CommandLine) ParseCommand(cmd string) bool {
lcmd := strings.TrimSpace(strings.ToLower(cmd))
switch {
case strings.HasPrefix(lcmd, "exit"):
// signal the program to exit
return false
case strings.HasPrefix(lcmd, "gopher"):
c.gopher()
case strings.HasPrefix(lcmd, "connect"):
c.connect(cmd)
case strings.HasPrefix(lcmd, "auth"):
c.SetAuth(cmd)
case strings.HasPrefix(lcmd, "help"):
c.help()
case strings.HasPrefix(lcmd, "format"):
c.SetFormat(cmd)
case strings.HasPrefix(lcmd, "settings"):
c.Settings()
case strings.HasPrefix(lcmd, "pretty"):
c.Pretty = !c.Pretty
if c.Pretty {
fmt.Println("Pretty print enabled")
} else {
fmt.Println("Pretty print disabled")
}
case strings.HasPrefix(lcmd, "use"):
c.use(cmd)
case strings.HasPrefix(lcmd, "insert"):
c.Insert(cmd)
case lcmd == "":
break
default:
c.ExecuteQuery(cmd)
}
return true
}
func (c *CommandLine) connect(cmd string) error {
var cl *client.Client
var u url.URL
// Remove the "connect" keyword if it exists
path := strings.TrimSpace(strings.Replace(cmd, "connect", "", -1))
// If they didn't provide a connection string, use the current settings
if path == "" {
path = net.JoinHostPort(c.Host, strconv.Itoa(c.Port))
}
var e error
u, e = client.ParseConnectionString(path, c.Ssl)
if e != nil {
return e
}
config := client.NewConfig()
config.URL = u
config.Username = c.Username
config.Password = c.Password
config.UserAgent = "InfluxDBShell/" + version
cl, err := client.NewClient(config)
if err != nil {
return fmt.Errorf("Could not create client %s", err)
}
c.Client = cl
if _, v, e := c.Client.Ping(); e != nil {
return fmt.Errorf("Failed to connect to %s\n", c.Client.Addr())
} else {
c.Version = v
}
return nil
}
func (c *CommandLine) SetAuth(cmd string) {
// If they pass in the entire command, we should parse it
// auth <username> <password>
args := strings.Fields(cmd)
if len(args) == 3 {
args = args[1:]
} else {
args = []string{}
}
if len(args) == 2 {
c.Username = args[0]
c.Password = args[1]
} else {
u, e := c.Line.Prompt("username: ")
if e != nil {
fmt.Printf("Unable to process input: %s", e)
return
}
c.Username = strings.TrimSpace(u)
p, e := c.Line.PasswordPrompt("password: ")
if e != nil {
fmt.Printf("Unable to process input: %s", e)
return
}
c.Password = p
}
// Update the client as well
c.Client.SetAuth(c.Username, c.Password)
}
func (c *CommandLine) use(cmd string) {
args := strings.Split(strings.TrimSuffix(strings.TrimSpace(cmd), ";"), " ")
if len(args) != 2 {
fmt.Printf("Could not parse database name from %q.\n", cmd)
return
}
d := args[1]
c.Database = d
fmt.Printf("Using database %s\n", d)
}
func (c *CommandLine) SetFormat(cmd string) {
// Remove the "format" keyword if it exists
cmd = strings.TrimSpace(strings.Replace(cmd, "format", "", -1))
// normalize cmd
cmd = strings.ToLower(cmd)
switch cmd {
case "json", "csv", "column":
c.Format = cmd
default:
fmt.Printf("Unknown format %q. Please use json, csv, or column.\n", cmd)
}
}
// isWhitespace returns true if the rune is a space, tab, or newline.
func isWhitespace(ch rune) bool { return ch == ' ' || ch == '\t' || ch == '\n' }
// isLetter returns true if the rune is a letter.
func isLetter(ch rune) bool { return (ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') }
// isDigit returns true if the rune is a digit.
func isDigit(ch rune) bool { return (ch >= '0' && ch <= '9') }
// isIdentFirstChar returns true if the rune can be used as the first char in an unquoted identifer.
func isIdentFirstChar(ch rune) bool { return isLetter(ch) || ch == '_' }
// isIdentChar returns true if the rune can be used in an unquoted identifier.
func isNotIdentChar(ch rune) bool { return !(isLetter(ch) || isDigit(ch) || ch == '_') }
func parseUnquotedIdentifier(stmt string) (string, string) {
if fields := strings.FieldsFunc(stmt, isNotIdentChar); len(fields) > 0 {
return fields[0], strings.TrimPrefix(stmt, fields[0])
}
return "", stmt
}
func parseDoubleQuotedIdentifier(stmt string) (string, string) {
escapeNext := false
fields := strings.FieldsFunc(stmt, func(ch rune) bool {
if ch == '\\' {
escapeNext = true
} else if ch == '"' {
if !escapeNext {
return true
}
escapeNext = false
}
return false
})
if len(fields) > 0 {
return fields[0], strings.TrimPrefix(stmt, "\""+fields[0]+"\"")
}
return "", stmt
}
func parseNextIdentifier(stmt string) (ident, remainder string) {
if len(stmt) > 0 {
switch {
case isWhitespace(rune(stmt[0])):
return parseNextIdentifier(stmt[1:])
case isIdentFirstChar(rune(stmt[0])):
return parseUnquotedIdentifier(stmt)
case stmt[0] == '"':
return parseDoubleQuotedIdentifier(stmt)
}
}
return "", stmt
}
func (c *CommandLine) parseInto(stmt string) string {
ident, stmt := parseNextIdentifier(stmt)
if strings.HasPrefix(stmt, ".") {
c.Database = ident
fmt.Printf("Using database %s\n", c.Database)
ident, stmt = parseNextIdentifier(stmt[1:])
}
if strings.HasPrefix(stmt, " ") {
c.RetentionPolicy = ident
fmt.Printf("Using retention policy %s\n", c.RetentionPolicy)
return stmt[1:]
}
return stmt
}
func (c *CommandLine) Insert(stmt string) error {
i, point := parseNextIdentifier(stmt)
if !strings.EqualFold(i, "insert") {
fmt.Printf("ERR: found %s, expected INSERT\n", i)
return nil
}
if i, r := parseNextIdentifier(point); strings.EqualFold(i, "into") {
point = c.parseInto(r)
}
_, err := c.Client.Write(client.BatchPoints{
Points: []client.Point{
client.Point{Raw: point},
},
Database: c.Database,
RetentionPolicy: c.RetentionPolicy,
Precision: "n",
WriteConsistency: client.ConsistencyAny,
})
if err != nil {
fmt.Printf("ERR: %s\n", err)
if c.Database == "" {
fmt.Println("Note: error may be due to not setting a database or retention policy.")
fmt.Println(`Please set a database with the command "use <database>" or`)
fmt.Println("INSERT INTO <database>.<retention-policy> <point>")
}
return err
}
return nil
}
func (c *CommandLine) ExecuteQuery(query string) error {
response, err := c.Client.Query(client.Query{Command: query, Database: c.Database})
if err != nil {
fmt.Printf("ERR: %s\n", err)
return err
}
c.FormatResponse(response, os.Stdout)
if err := response.Error(); err != nil {
fmt.Printf("ERR: %s\n", response.Error())
if c.Database == "" {
fmt.Println("Warning: It is possible this error is due to not setting a database.")
fmt.Println(`Please set a database with the command "use <database>".`)
}
return err
}
return nil
}
func (c *CommandLine) FormatResponse(response *client.Response, w io.Writer) {
switch c.Format {
case "json":
c.writeJSON(response, w)
case "csv":
c.writeCSV(response, w)
case "column":
c.writeColumns(response, w)
default:
fmt.Fprintf(w, "Unknown output format %q.\n", c.Format)
}
}
func (c *CommandLine) writeJSON(response *client.Response, w io.Writer) {
var data []byte
var err error
if c.Pretty {
data, err = json.MarshalIndent(response, "", " ")
} else {
data, err = json.Marshal(response)
}
if err != nil {
fmt.Fprintf(w, "Unable to parse json: %s\n", err)
return
}
fmt.Fprintln(w, string(data))
}
func (c *CommandLine) writeCSV(response *client.Response, w io.Writer) {
csvw := csv.NewWriter(w)
for _, result := range response.Results {
// Create a tabbed writer for each result as they won't always line up
rows := c.formatResults(result, "\t")
for _, r := range rows {
csvw.Write(strings.Split(r, "\t"))
}
csvw.Flush()
}
}
func (c *CommandLine) writeColumns(response *client.Response, w io.Writer) {
for _, result := range response.Results {
// Create a tabbed writer for each result a they won't always line up
w := new(tabwriter.Writer)
w.Init(os.Stdout, 0, 8, 1, '\t', 0)
csv := c.formatResults(result, "\t")
for _, r := range csv {
fmt.Fprintln(w, r)
}
w.Flush()
}
}
// formatResults will behave differently if you are formatting for columns or csv
func (c *CommandLine) formatResults(result client.Result, separator string) []string {
rows := []string{}
// Create a tabbed writer for each result a they won't always line up
for i, row := range result.Series {
// gather tags
tags := []string{}
for k, v := range row.Tags {
tags = append(tags, fmt.Sprintf("%s=%s", k, v))
sort.Strings(tags)
}
columnNames := []string{}
// Only put name/tags in a column if format is csv
if c.Format == "csv" {
if len(tags) > 0 {
columnNames = append([]string{"tags"}, columnNames...)
}
if row.Name != "" {
columnNames = append([]string{"name"}, columnNames...)
}
}
for _, column := range row.Columns {
columnNames = append(columnNames, column)
}
// Output a line separator if we have more than one set or results and format is column
if i > 0 && c.Format == "column" {
rows = append(rows, "")
}
// If we are column format, we break out the name/tag to seperate lines
if c.Format == "column" {
if row.Name != "" {
n := fmt.Sprintf("name: %s", row.Name)
rows = append(rows, n)
if len(tags) == 0 {
l := strings.Repeat("-", len(n))
rows = append(rows, l)
}
}
if len(tags) > 0 {
t := fmt.Sprintf("tags: %s", (strings.Join(tags, ", ")))
rows = append(rows, t)
}
}
rows = append(rows, strings.Join(columnNames, separator))
// if format is column, break tags to their own line/format
if c.Format == "column" && len(tags) > 0 {
lines := []string{}
for _, columnName := range columnNames {
lines = append(lines, strings.Repeat("-", len(columnName)))
}
rows = append(rows, strings.Join(lines, separator))
}
for _, v := range row.Values {
var values []string
if c.Format == "csv" {
if row.Name != "" {
values = append(values, row.Name)
}
if len(tags) > 0 {
values = append(values, strings.Join(tags, ","))
}
}
for _, vv := range v {
values = append(values, interfaceToString(vv))
}
rows = append(rows, strings.Join(values, separator))
}
// Outout a line separator if in column format
if c.Format == "column" {
rows = append(rows, "")
}
}
return rows
}
func interfaceToString(v interface{}) string {
switch t := v.(type) {
case nil:
return ""
case bool:
return fmt.Sprintf("%v", v)
case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, uintptr:
return fmt.Sprintf("%d", t)
case float32, float64:
return fmt.Sprintf("%v", t)
default:
return fmt.Sprintf("%v", t)
}
}
func (c *CommandLine) Settings() {
w := new(tabwriter.Writer)
w.Init(os.Stdout, 0, 8, 1, '\t', 0)
if c.Port > 0 {
fmt.Fprintf(w, "Host\t%s:%d\n", c.Host, c.Port)
} else {
fmt.Fprintf(w, "Host\t%s\n", c.Host)
}
fmt.Fprintf(w, "Username\t%s\n", c.Username)
fmt.Fprintf(w, "Database\t%s\n", c.Database)
fmt.Fprintf(w, "Pretty\t%v\n", c.Pretty)
fmt.Fprintf(w, "Format\t%s\n", c.Format)
fmt.Fprintln(w)
w.Flush()
}
func (c *CommandLine) help() {
fmt.Println(`Usage:
connect <host:port> connect to another node
auth prompt for username and password
pretty toggle pretty print
use <db_name> set current databases
format <format> set the output format: json, csv, or column
settings output the current settings for the shell
exit quit the influx shell
show databases show database names
show series show series information
show measurements show measurement information
show tag keys show tag key information
show tag values show tag value information
a full list of influxql commands can be found at:
https://influxdb.com/docs/v0.9/query_language/spec.html
`)
}
func (c *CommandLine) gopher() {
fmt.Println(`
.-::-::://:-::- .:/++/'
'://:-''/oo+//++o+/.://o- ./+:
.:-. '++- .o/ '+yydhy' o-
.:/. .h: :osoys .smMN- :/
-/:.' s- /MMMymh. '/y/ s'
-+s:'''' d -mMMms// '-/o:
-/++/++/////:. o: '... s- :s.
:+-+s-' ':/' 's- /+ 'o:
'+-'o: /ydhsh. '//. '-o- o-
.y. o: .MMMdm+y ':+++:::/+:.' s:
.-h/ y- 'sdmds'h -+ydds:::-.' 'h.
.//-.d' o: '.' 'dsNMMMNh:.:++' :y
+y. 'd 's. .s:mddds: ++ o/
'N- odd 'o/. './o-s-' .---+++' o-
'N' yNd .://:/:::::. -s -+/s/./s' 'o/'
so' .h '''' ////s: '+. .s +y'
os/-.y' 's' 'y::+ +d'
'.:o/ -+:-:.' so.---.'
o' 'd-.''/s'
.s' :y.''.y
-s mo:::'
:: yh
// '''' /M'
o+ .s///:/. 'N:
:+ /: -s' ho
's- -/s/:+/.+h' +h
ys' ':' '-. -d
oh .h
/o .s
s. .h
-y .d
m/ -h
+d /o
'N- y:
h: m.
s- -d
o- s+
+- 'm'
s/ oo--.
y- /s ':+'
s' 'od--' .d:
-+ ':o: ':+-/+
y- .:+- '
//o- '.:+/.
.-:+/' ''-/+/.
./:' ''.:o+/-'
.+o:/:/+-' ''.-+ooo/-'
o: -h///++////-.
/: .o/
//+ 'y
./sooy.
`)
}

194
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx/main_test.go generated vendored Normal file
View File

@ -0,0 +1,194 @@
package main_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/influxdb/influxdb/client"
main "github.com/influxdb/influxdb/cmd/influx"
)
func TestParseCommand_CommandsExist(t *testing.T) {
t.Parallel()
c := main.CommandLine{}
tests := []struct {
cmd string
}{
{cmd: "gopher"},
{cmd: "connect"},
{cmd: "help"},
{cmd: "pretty"},
{cmd: "use"},
{cmd: ""}, // test that a blank command just returns
}
for _, test := range tests {
if !c.ParseCommand(test.cmd) {
t.Fatalf(`Command failed for %q.`, test.cmd)
}
}
}
func TestParseCommand_TogglePretty(t *testing.T) {
t.Parallel()
c := main.CommandLine{}
if c.Pretty {
t.Fatalf(`Pretty should be false.`)
}
c.ParseCommand("pretty")
if !c.Pretty {
t.Fatalf(`Pretty should be true.`)
}
c.ParseCommand("pretty")
if c.Pretty {
t.Fatalf(`Pretty should be false.`)
}
}
func TestParseCommand_Exit(t *testing.T) {
t.Parallel()
c := main.CommandLine{}
tests := []struct {
cmd string
}{
{cmd: "exit"},
{cmd: " exit"},
{cmd: "exit "},
{cmd: "Exit "},
}
for _, test := range tests {
if c.ParseCommand(test.cmd) {
t.Fatalf(`Command "exit" failed for %q.`, test.cmd)
}
}
}
func TestParseCommand_Use(t *testing.T) {
t.Parallel()
c := main.CommandLine{}
tests := []struct {
cmd string
}{
{cmd: "use db"},
{cmd: " use db"},
{cmd: "use db "},
{cmd: "use db;"},
{cmd: "use db; "},
{cmd: "Use db"},
}
for _, test := range tests {
if !c.ParseCommand(test.cmd) {
t.Fatalf(`Command "use" failed for %q.`, test.cmd)
}
if c.Database != "db" {
t.Fatalf(`Command "use" changed database to %q. Expected db`, c.Database)
}
}
}
func TestParseCommand_Insert(t *testing.T) {
t.Parallel()
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data client.Response
w.WriteHeader(http.StatusNoContent)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
u, _ := url.Parse(ts.URL)
config := client.Config{URL: *u}
c, err := client.NewClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
m := main.CommandLine{Client: c}
tests := []struct {
cmd string
}{
{cmd: "INSERT cpu,host=serverA,region=us-west value=1.0"},
{cmd: " INSERT cpu,host=serverA,region=us-west value=1.0"},
{cmd: "INSERT cpu,host=serverA,region=us-west value=1.0"},
{cmd: "insert cpu,host=serverA,region=us-west value=1.0 "},
{cmd: "insert"},
{cmd: "Insert "},
{cmd: "insert c"},
{cmd: "insert int"},
}
for _, test := range tests {
if !m.ParseCommand(test.cmd) {
t.Fatalf(`Command "insert" failed for %q.`, test.cmd)
}
}
}
func TestParseCommand_InsertInto(t *testing.T) {
t.Parallel()
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data client.Response
w.WriteHeader(http.StatusNoContent)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
u, _ := url.Parse(ts.URL)
config := client.Config{URL: *u}
c, err := client.NewClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
m := main.CommandLine{Client: c}
tests := []struct {
cmd, db, rp string
}{
{
cmd: `INSERT INTO test cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test",
},
{
cmd: ` INSERT INTO .test cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test",
},
{
cmd: `INSERT INTO "test test" cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test test",
},
{
cmd: `Insert iNTO test.test cpu,host=serverA,region=us-west value=1.0`,
db: "test",
rp: "test",
},
{
cmd: `insert into "test test" cpu,host=serverA,region=us-west value=1.0`,
db: "test",
rp: "test test",
},
{
cmd: `insert into "d b"."test test" cpu,host=serverA,region=us-west value=1.0`,
db: "d b",
rp: "test test",
},
}
for _, test := range tests {
if !m.ParseCommand(test.cmd) {
t.Fatalf(`Command "insert into" failed for %q.`, test.cmd)
}
if m.Database != test.db {
t.Fatalf(`Command "insert into" db parsing failed, expected: %q, actual: %q`, test.db, m.Database)
}
if m.RetentionPolicy != test.rp {
t.Fatalf(`Command "insert into" rp parsing failed, expected: %q, actual: %q`, test.rp, m.RetentionPolicy)
}
}
}

154
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influx_stress/influx_stress.go generated vendored Normal file
View File

@ -0,0 +1,154 @@
package main
import (
"flag"
"fmt"
"math/rand"
"net/url"
"runtime"
"sort"
"sync"
"time"
"github.com/influxdb/influxdb/client"
)
var (
batchSize = flag.Int("batchsize", 5000, "number of points per batch")
seriesCount = flag.Int("series", 100000, "number of unique series to create")
pointCount = flag.Int("points", 100, "number of points per series to create")
concurrency = flag.Int("concurrency", 10, "number of simultaneous writes to run")
batchInterval = flag.Duration("batchinterval", 0*time.Second, "duration between batches")
database = flag.String("database", "stress", "name of database")
address = flag.String("addr", "localhost:8086", "IP address and port of database (e.g., localhost:8086)")
)
func main() {
flag.Parse()
runtime.GOMAXPROCS(runtime.NumCPU())
startTime := time.Now()
counter := NewConcurrencyLimiter(*concurrency)
u, _ := url.Parse(fmt.Sprintf("http://%s", *address))
c, err := client.NewClient(client.Config{URL: *u})
if err != nil {
panic(err)
}
var mu sync.Mutex
var wg sync.WaitGroup
responseTimes := make([]int, 0)
totalPoints := 0
batch := &client.BatchPoints{
Database: *database,
WriteConsistency: "any",
Time: time.Now(),
Precision: "n",
}
for i := 1; i <= *pointCount; i++ {
for j := 1; j <= *seriesCount; j++ {
p := client.Point{
Measurement: "cpu",
Tags: map[string]string{"region": "uswest", "host": fmt.Sprintf("host-%d", j)},
Fields: map[string]interface{}{"value": rand.Float64()},
}
batch.Points = append(batch.Points, p)
if len(batch.Points) >= *batchSize {
wg.Add(1)
counter.Increment()
totalPoints += len(batch.Points)
go func(b *client.BatchPoints, total int) {
st := time.Now()
if _, err := c.Write(*b); err != nil {
fmt.Println("ERROR: ", err.Error())
} else {
mu.Lock()
responseTimes = append(responseTimes, int(time.Since(st).Nanoseconds()))
mu.Unlock()
}
wg.Done()
counter.Decrement()
if total%500000 == 0 {
fmt.Printf("%d total points. %d in %s\n", total, *batchSize, time.Since(st))
}
}(batch, totalPoints)
batch = &client.BatchPoints{
Database: *database,
WriteConsistency: "any",
Precision: "n",
Time: time.Now(),
}
}
}
}
wg.Wait()
sort.Sort(sort.Reverse(sort.IntSlice(responseTimes)))
total := int64(0)
for _, t := range responseTimes {
total += int64(t)
}
mean := total / int64(len(responseTimes))
fmt.Printf("Wrote %d points at average rate of %.0f\n", totalPoints, float64(totalPoints)/time.Since(startTime).Seconds())
fmt.Println("Average response time: ", time.Duration(mean))
fmt.Println("Slowest response times:")
for _, r := range responseTimes[:100] {
fmt.Println(time.Duration(r))
}
}
// ConcurrencyLimiter is a go routine safe struct that can be used to
// ensure that no more than a specifid max number of goroutines are
// executing.
type ConcurrencyLimiter struct {
inc chan chan struct{}
dec chan struct{}
max int
count int
}
// NewConcurrencyLimiter returns a configured limiter that will
// ensure that calls to Increment will block if the max is hit.
func NewConcurrencyLimiter(max int) *ConcurrencyLimiter {
c := &ConcurrencyLimiter{
inc: make(chan chan struct{}),
dec: make(chan struct{}, max),
max: max,
}
go c.handleLimits()
return c
}
// Increment will increase the count of running goroutines by 1.
// if the number is currently at the max, the call to Increment
// will block until another goroutine decrements.
func (c *ConcurrencyLimiter) Increment() {
r := make(chan struct{})
c.inc <- r
<-r
}
// Decrement will reduce the count of running goroutines by 1
func (c *ConcurrencyLimiter) Decrement() {
c.dec <- struct{}{}
}
// handleLimits runs in a goroutine to manage the count of
// running goroutines.
func (c *ConcurrencyLimiter) handleLimits() {
for {
r := <-c.inc
if c.count >= c.max {
<-c.dec
c.count--
}
c.count++
r <- struct{}{}
}
}

170
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/backup/backup.go generated vendored Normal file
View File

@ -0,0 +1,170 @@
package backup
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"log"
"net"
"os"
"github.com/influxdb/influxdb/services/snapshotter"
"github.com/influxdb/influxdb/snapshot"
)
// Suffix is a suffix added to the backup while it's in-process.
const Suffix = ".pending"
// Command represents the program execution for "influxd backup".
type Command struct {
// The logger passed to the ticker during execution.
Logger *log.Logger
// Standard input/output, overridden for testing.
Stderr io.Writer
}
// NewCommand returns a new instance of Command with default settings.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
}
}
// Run executes the program.
func (cmd *Command) Run(args ...string) error {
// Set up logger.
cmd.Logger = log.New(cmd.Stderr, "", log.LstdFlags)
cmd.Logger.Printf("influxdb backup")
// Parse command line arguments.
host, path, err := cmd.parseFlags(args)
if err != nil {
return err
}
// Retrieve snapshot from local file.
m, err := snapshot.ReadFileManifest(path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read file snapshot: %s", err)
}
// Determine temporary path to download to.
tmppath := path + Suffix
// Calculate path of next backup file.
// This uses the path if it doesn't exist.
// Otherwise it appends an autoincrementing number.
path, err = cmd.nextPath(path)
if err != nil {
return fmt.Errorf("next path: %s", err)
}
// Retrieve snapshot.
if err := cmd.download(host, m, tmppath); err != nil {
return fmt.Errorf("download: %s", err)
}
// Rename temporary file to final path.
if err := os.Rename(tmppath, path); err != nil {
return fmt.Errorf("rename: %s", err)
}
// TODO: Check file integrity.
// Notify user of completion.
cmd.Logger.Println("backup complete")
return nil
}
// parseFlags parses and validates the command line arguments.
func (cmd *Command) parseFlags(args []string) (host string, path string, err error) {
fs := flag.NewFlagSet("", flag.ContinueOnError)
fs.StringVar(&host, "host", "localhost:8088", "")
fs.SetOutput(cmd.Stderr)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return "", "", err
}
// Ensure that only one arg is specified.
if fs.NArg() == 0 {
return "", "", errors.New("snapshot path required")
} else if fs.NArg() != 1 {
return "", "", errors.New("only one snapshot path allowed")
}
path = fs.Arg(0)
return host, path, nil
}
// nextPath returns the next file to write to.
func (cmd *Command) nextPath(path string) (string, error) {
// Use base path if it doesn't exist.
if _, err := os.Stat(path); os.IsNotExist(err) {
return path, nil
} else if err != nil {
return "", err
}
// Otherwise iterate through incremental files until one is available.
for i := 0; ; i++ {
s := fmt.Sprintf(path+".%d", i)
if _, err := os.Stat(s); os.IsNotExist(err) {
return s, nil
} else if err != nil {
return "", err
}
}
}
// download downloads a snapshot from a host to a given path.
func (cmd *Command) download(host string, m *snapshot.Manifest, path string) error {
// Create local file to write to.
f, err := os.Create(path)
if err != nil {
return fmt.Errorf("open temp file: %s", err)
}
defer f.Close()
// Connect to snapshotter service.
conn, err := net.Dial("tcp", host)
if err != nil {
return err
}
defer conn.Close()
// Send snapshotter marker byte.
if _, err := conn.Write([]byte{snapshotter.MuxHeader}); err != nil {
return fmt.Errorf("write snapshot header byte: %s", err)
}
// Write the manifest we currently have.
if err := json.NewEncoder(conn).Encode(m); err != nil {
return fmt.Errorf("encode snapshot manifest: %s", err)
}
// Read snapshot from the connection.
if _, err := io.Copy(f, conn); err != nil {
return fmt.Errorf("copy snapshot to file: %s", err)
}
// FIXME(benbjohnson): Verify integrity of snapshot.
return nil
}
// printUsage prints the usage message to STDERR.
func (cmd *Command) printUsage() {
fmt.Fprintf(cmd.Stderr, `usage: influxd backup [flags] PATH
backup downloads a snapshot of a data node and saves it to disk.
-host <host:port>
The host to connect to snapshot.
Defaults to 127.0.0.1:8088.
`)
}

125
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/backup/backup_test.go generated vendored Normal file
View File

@ -0,0 +1,125 @@
package backup_test
/*
import (
"bytes"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/influxdb/influxdb"
"github.com/influxdb/influxdb/cmd/influxd"
)
// Ensure the backup can download from the server and save to disk.
func TestBackupCommand(t *testing.T) {
// Mock the backup endpoint.
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/data/snapshot" {
t.Fatalf("unexpected url path: %s", r.URL.Path)
}
// Write a simple snapshot to the buffer.
sw := influxdb.NewSnapshotWriter()
sw.Snapshot = &influxdb.Snapshot{Files: []influxdb.SnapshotFile{
{Name: "meta", Size: 5, Index: 10},
}}
sw.FileWriters["meta"] = influxdb.NopWriteToCloser(bytes.NewBufferString("55555"))
if _, err := sw.WriteTo(w); err != nil {
t.Fatal(err)
}
}))
defer s.Close()
// Create a temp path and remove incremental backups at the end.
path := tempfile()
defer os.Remove(path)
defer os.Remove(path + ".0")
defer os.Remove(path + ".1")
// Execute the backup against the mock server.
for i := 0; i < 3; i++ {
if err := NewBackupCommand().Run("-host", s.URL, path); err != nil {
t.Fatal(err)
}
}
// Verify snapshot and two incremental snapshots were written.
if _, err := os.Stat(path); err != nil {
t.Fatalf("snapshot not found: %s", err)
} else if _, err = os.Stat(path + ".0"); err != nil {
t.Fatalf("incremental snapshot(0) not found: %s", err)
} else if _, err = os.Stat(path + ".1"); err != nil {
t.Fatalf("incremental snapshot(1) not found: %s", err)
}
}
// Ensure the backup command returns an error if flags cannot be parsed.
func TestBackupCommand_ErrFlagParse(t *testing.T) {
cmd := NewBackupCommand()
if err := cmd.Run("-bad-flag"); err == nil || err.Error() != `flag provided but not defined: -bad-flag` {
t.Fatal(err)
} else if !strings.Contains(cmd.Stderr.String(), "usage") {
t.Fatal("usage message not displayed")
}
}
// Ensure the backup command returns an error if the host cannot be parsed.
func TestBackupCommand_ErrInvalidHostURL(t *testing.T) {
if err := NewBackupCommand().Run("-host", "http://%f"); err == nil || err.Error() != `parse host url: parse http://%f: hexadecimal escape in host` {
t.Fatal(err)
}
}
// Ensure the backup command returns an error if the output path is not specified.
func TestBackupCommand_ErrPathRequired(t *testing.T) {
if err := NewBackupCommand().Run("-host", "//localhost"); err == nil || err.Error() != `snapshot path required` {
t.Fatal(err)
}
}
// Ensure the backup returns an error if it cannot connect to the server.
func TestBackupCommand_ErrConnectionRefused(t *testing.T) {
// Start and immediately stop a server so we have a dead port.
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
s.Close()
// Execute the backup command.
path := tempfile()
defer os.Remove(path)
if err := NewBackupCommand().Run("-host", s.URL, path); err == nil ||
!(strings.Contains(err.Error(), `connection refused`) || strings.Contains(err.Error(), `No connection could be made`)) {
t.Fatal(err)
}
}
// Ensure the backup returns any non-200 status codes.
func TestBackupCommand_ErrServerError(t *testing.T) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer s.Close()
// Execute the backup command.
path := tempfile()
defer os.Remove(path)
if err := NewBackupCommand().Run("-host", s.URL, path); err == nil || err.Error() != `download: snapshot error: status=500` {
t.Fatal(err)
}
}
// BackupCommand is a test wrapper for main.BackupCommand.
type BackupCommand struct {
*main.BackupCommand
Stderr bytes.Buffer
}
// NewBackupCommand returns a new instance of BackupCommand.
func NewBackupCommand() *BackupCommand {
cmd := &BackupCommand{BackupCommand: main.NewBackupCommand()}
cmd.BackupCommand.Stderr = &cmd.Stderr
return cmd
}
*/

46
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/help/help.go generated vendored Normal file
View File

@ -0,0 +1,46 @@
package help
import (
"fmt"
"io"
"os"
"strings"
)
// Command displays help for command-line sub-commands.
type Command struct {
Stdout io.Writer
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fmt.Fprintln(cmd.Stdout, strings.TrimSpace(usage))
return nil
}
const usage = `
Configure and start an InfluxDB server.
Usage:
influxd [[command] [arguments]]
The commands are:
backup downloads a snapshot of a data node and saves it to disk
config display the default configuration
restore uses a snapshot of a data node to rebuild a cluster
run run node with existing configuration
version displays the InfluxDB version
"run" is the default command.
Use "influxd help [command]" for more information about a command.
`

200
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/main.go generated vendored Normal file
View File

@ -0,0 +1,200 @@
package main
import (
"flag"
"fmt"
"io"
"log"
"math/rand"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/influxdb/influxdb/cmd/influxd/backup"
"github.com/influxdb/influxdb/cmd/influxd/help"
"github.com/influxdb/influxdb/cmd/influxd/restore"
"github.com/influxdb/influxdb/cmd/influxd/run"
)
// These variables are populated via the Go linker.
var (
version string = "0.9"
commit string
branch string
)
func init() {
// If commit or branch are not set, make that clear.
if commit == "" {
commit = "unknown"
}
if branch == "" {
branch = "unknown"
}
}
func main() {
rand.Seed(time.Now().UnixNano())
m := NewMain()
if err := m.Run(os.Args[1:]...); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
// Main represents the program execution.
type Main struct {
Logger *log.Logger
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewMain return a new instance of Main.
func NewMain() *Main {
return &Main{
Logger: log.New(os.Stderr, "[run] ", log.LstdFlags),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run determines and runs the command specified by the CLI args.
func (m *Main) Run(args ...string) error {
name, args := ParseCommandName(args)
// Extract name from args.
switch name {
case "", "run":
cmd := run.NewCommand()
// Tell the server the build details.
cmd.Version = version
cmd.Commit = commit
cmd.Branch = branch
if err := cmd.Run(args...); err != nil {
return fmt.Errorf("run: %s", err)
}
signalCh := make(chan os.Signal, 1)
signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM)
m.Logger.Println("Listening for signals")
// Block until one of the signals above is received
select {
case <-signalCh:
m.Logger.Println("Signal received, initializing clean shutdown...")
go func() {
cmd.Close()
}()
}
// Block again until another signal is received, a shutdown timeout elapses,
// or the Command is gracefully closed
m.Logger.Println("Waiting for clean shutdown...")
select {
case <-signalCh:
m.Logger.Println("second signal received, initializing hard shutdown")
case <-time.After(time.Second * 30):
m.Logger.Println("time limit reached, initializing hard shutdown")
case <-cmd.Closed:
m.Logger.Println("server shutdown completed")
}
// goodbye.
case "backup":
name := backup.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("backup: %s", err)
}
case "restore":
name := restore.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("restore: %s", err)
}
case "config":
if err := run.NewPrintConfigCommand().Run(args...); err != nil {
return fmt.Errorf("config: %s", err)
}
case "version":
if err := NewVersionCommand().Run(args...); err != nil {
return fmt.Errorf("version: %s", err)
}
case "help":
if err := help.NewCommand().Run(args...); err != nil {
return fmt.Errorf("help: %s", err)
}
default:
return fmt.Errorf(`unknown command "%s"`+"\n"+`Run 'influxd help' for usage`+"\n\n", name)
}
return nil
}
// ParseCommandName extracts the command name and args from the args list.
func ParseCommandName(args []string) (string, []string) {
// Retrieve command name as first argument.
var name string
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
name = args[0]
}
// Special case -h immediately following binary name
if len(args) > 0 && args[0] == "-h" {
name = "help"
}
// If command is "help" and has an argument then rewrite args to use "-h".
if name == "help" && len(args) > 1 {
args[0], args[1] = args[1], "-h"
name = args[0]
}
// If a named command is specified then return it with its arguments.
if name != "" {
return name, args[1:]
}
return "", args
}
// Command represents the command executed by "influxd version".
type VersionCommand struct {
Stdout io.Writer
Stderr io.Writer
}
// NewVersionCommand return a new instance of VersionCommand.
func NewVersionCommand() *VersionCommand {
return &VersionCommand{
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run prints the current version and commit info.
func (cmd *VersionCommand) Run(args ...string) error {
// Parse flags in case -h is specified.
fs := flag.NewFlagSet("", flag.ContinueOnError)
fs.Usage = func() { fmt.Fprintln(cmd.Stderr, strings.TrimSpace(versionUsage)) }
if err := fs.Parse(args); err != nil {
return err
}
// Print version info.
fmt.Fprintf(cmd.Stdout, "InfluxDB v%s (git: %s %s)\n", version, branch, commit)
return nil
}
var versionUsage = `
usage: version
version displays the InfluxDB version, build branch and git commit hash
`

250
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/restore/restore.go generated vendored Normal file
View File

@ -0,0 +1,250 @@
package restore
import (
"bytes"
"errors"
"flag"
"fmt"
"io"
"net"
"os"
"path/filepath"
"github.com/BurntSushi/toml"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/snapshot"
"github.com/influxdb/influxdb/tsdb"
)
// Command represents the program execution for "influxd restore".
type Command struct {
Stdout io.Writer
Stderr io.Writer
}
// NewCommand returns a new instance of Command with default settings.
func NewCommand() *Command {
return &Command{
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run executes the program.
func (cmd *Command) Run(args ...string) error {
config, path, err := cmd.parseFlags(args)
if err != nil {
return err
}
return cmd.Restore(config, path)
}
func (cmd *Command) Restore(config *Config, path string) error {
// Remove meta and data directories.
if err := os.RemoveAll(config.Meta.Dir); err != nil {
return fmt.Errorf("remove meta dir: %s", err)
} else if err := os.RemoveAll(config.Data.Dir); err != nil {
return fmt.Errorf("remove data dir: %s", err)
}
// Open snapshot file and all incremental backups.
mr, files, err := snapshot.OpenFileMultiReader(path)
if err != nil {
return fmt.Errorf("open multireader: %s", err)
}
defer closeAll(files)
// Unpack files from archive.
if err := cmd.unpack(mr, config); err != nil {
return fmt.Errorf("unpack: %s", err)
}
// Notify user of completion.
fmt.Fprintf(os.Stdout, "restore complete using %s", path)
return nil
}
// parseFlags parses and validates the command line arguments.
func (cmd *Command) parseFlags(args []string) (*Config, string, error) {
fs := flag.NewFlagSet("", flag.ContinueOnError)
configPath := fs.String("config", "", "")
fs.SetOutput(cmd.Stderr)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return nil, "", err
}
// Parse configuration file from disk.
if *configPath == "" {
return nil, "", fmt.Errorf("config required")
}
// Parse config.
config := Config{
Meta: meta.NewConfig(),
Data: tsdb.NewConfig(),
}
if _, err := toml.DecodeFile(*configPath, &config); err != nil {
return nil, "", err
}
// Require output path.
path := fs.Arg(0)
if path == "" {
return nil, "", fmt.Errorf("snapshot path required")
}
return &config, path, nil
}
func closeAll(a []io.Closer) {
for _, c := range a {
_ = c.Close()
}
}
// unpack expands the files in the snapshot archive into a directory.
func (cmd *Command) unpack(mr *snapshot.MultiReader, config *Config) error {
// Loop over files and extract.
for {
// Read entry header.
sf, err := mr.Next()
if err == io.EOF {
break
} else if err != nil {
return fmt.Errorf("next: entry=%s, err=%s", sf.Name, err)
}
// Log progress.
fmt.Fprintf(os.Stdout, "unpacking: %s (%d bytes)\n", sf.Name, sf.Size)
// Handle meta and tsdb files separately.
switch sf.Name {
case "meta":
if err := cmd.unpackMeta(mr, sf, config); err != nil {
return fmt.Errorf("meta: %s", err)
}
default:
if err := cmd.unpackData(mr, sf, config); err != nil {
return fmt.Errorf("data: %s", err)
}
}
}
return nil
}
// unpackMeta reads the metadata from the snapshot and initializes a raft
// cluster and replaces the root metadata.
func (cmd *Command) unpackMeta(mr *snapshot.MultiReader, sf snapshot.File, config *Config) error {
// Read meta into buffer.
var buf bytes.Buffer
if _, err := io.CopyN(&buf, mr, sf.Size); err != nil {
return fmt.Errorf("copy: %s", err)
}
// Unpack into metadata.
var data meta.Data
if err := data.UnmarshalBinary(buf.Bytes()); err != nil {
return fmt.Errorf("unmarshal: %s", err)
}
// Copy meta config and remove peers so it starts in single mode.
c := config.Meta
c.Peers = nil
// Initialize meta store.
store := meta.NewStore(config.Meta)
store.RaftListener = newNopListener()
store.ExecListener = newNopListener()
// Determine advertised address.
_, port, err := net.SplitHostPort(config.Meta.BindAddress)
if err != nil {
return fmt.Errorf("split bind address: %s", err)
}
hostport := net.JoinHostPort(config.Meta.Hostname, port)
// Resolve address.
addr, err := net.ResolveTCPAddr("tcp", hostport)
if err != nil {
return fmt.Errorf("resolve tcp: addr=%s, err=%s", hostport, err)
}
store.Addr = addr
// Open the meta store.
if err := store.Open(); err != nil {
return fmt.Errorf("open store: %s", err)
}
defer store.Close()
// Wait for the store to be ready or error.
select {
case <-store.Ready():
case err := <-store.Err():
return err
}
// Force set the full metadata.
if err := store.SetData(&data); err != nil {
return fmt.Errorf("set data: %s", err)
}
return nil
}
func (cmd *Command) unpackData(mr *snapshot.MultiReader, sf snapshot.File, config *Config) error {
path := filepath.Join(config.Data.Dir, sf.Name)
// Create parent directory for output file.
if err := os.MkdirAll(filepath.Dir(path), 0777); err != nil {
return fmt.Errorf("mkdir: entry=%s, err=%s", sf.Name, err)
}
// Create output file.
f, err := os.Create(path)
if err != nil {
return fmt.Errorf("create: entry=%s, err=%s", sf.Name, err)
}
defer f.Close()
// Copy contents from reader.
if _, err := io.CopyN(f, mr, sf.Size); err != nil {
return fmt.Errorf("copy: entry=%s, err=%s", sf.Name, err)
}
return nil
}
// printUsage prints the usage message to STDERR.
func (cmd *Command) printUsage() {
fmt.Fprintf(cmd.Stderr, `usage: influxd restore [flags] PATH
restore uses a snapshot of a data node to rebuild a cluster.
-config <path>
Set the path to the configuration file.
`)
}
// Config represents a partial config for rebuilding the server.
type Config struct {
Meta *meta.Config `toml:"meta"`
Data tsdb.Config `toml:"data"`
}
type nopListener struct {
closing chan struct{}
}
func newNopListener() *nopListener {
return &nopListener{make(chan struct{})}
}
func (ln *nopListener) Accept() (net.Conn, error) {
<-ln.closing
return nil, errors.New("listener closing")
}
func (ln *nopListener) Close() error { close(ln.closing); return nil }
func (ln *nopListener) Addr() net.Addr { return nil }

155
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/restore/restore_test.go generated vendored Normal file
View File

@ -0,0 +1,155 @@
package restore_test
/*
import (
"bytes"
"io/ioutil"
"os"
"path/filepath"
"reflect"
"testing"
"time"
main "github.com/influxdb/influxdb/cmd/influxd"
"github.com/influxdb/influxdb/tsdb"
)
func newConfig(path string, port int) main.Config {
config := main.NewConfig()
config.Port = port
config.Broker.Enabled = true
config.Broker.Dir = filepath.Join(path, "broker")
config.Data.Enabled = true
config.Data.Dir = filepath.Join(path, "data")
return *config
}
// Ensure the restore command can expand a snapshot and bootstrap a broker.
func TestRestoreCommand(t *testing.T) {
if testing.Short() {
t.Skip("skipping TestRestoreCommand")
}
now := time.Now()
// Create root path to server.
path := tempfile()
defer os.Remove(path)
// Parse configuration.
config := newConfig(path, 8900)
// Start server.
cmd := main.NewRunCommand()
node := cmd.Open(&config, "")
if node.Broker == nil {
t.Fatal("cannot run broker")
} else if node.DataNode == nil {
t.Fatal("cannot run server")
}
b := node.Broker
s := node.DataNode
// Create data.
if err := s.CreateDatabase("db"); err != nil {
t.Fatalf("cannot create database: %s", err)
}
if index, err := s.WriteSeries("db", "default", []tsdb.Point{tsdb.NewPoint("cpu", nil, map[string]interface{}{"value": float64(100)}, now)}); err != nil {
t.Fatalf("cannot write series: %s", err)
} else if err = s.Sync(1, index); err != nil {
t.Fatalf("shard sync: %s", err)
}
// Create snapshot writer.
sw, err := s.CreateSnapshotWriter()
if err != nil {
t.Fatalf("create snapshot writer: %s", err)
}
// Snapshot to file.
sspath := tempfile()
f, err := os.Create(sspath)
if err != nil {
t.Fatal(err)
}
sw.WriteTo(f)
f.Close()
// Stop server.
node.Close()
// Remove data & broker directories.
if err := os.RemoveAll(path); err != nil {
t.Fatalf("remove: %s", err)
}
// Execute the restore.
if err := NewRestoreCommand().Restore(&config, sspath); err != nil {
t.Fatal(err)
}
// Rewrite config to a new port and re-parse.
config = newConfig(path, 8910)
// Restart server.
cmd = main.NewRunCommand()
node = cmd.Open(&config, "")
if b == nil {
t.Fatal("cannot run broker")
} else if s == nil {
t.Fatal("cannot run server")
}
b = node.Broker
s = node.DataNode
// Write new data.
if err := s.CreateDatabase("newdb"); err != nil {
t.Fatalf("cannot create new database: %s", err)
}
if index, err := s.WriteSeries("newdb", "default", []tsdb.Point{tsdb.NewPoint("mem", nil, map[string]interface{}{"value": float64(1000)}, now)}); err != nil {
t.Fatalf("cannot write new series: %s", err)
} else if err = s.Sync(2, index); err != nil {
t.Fatalf("shard sync: %s", err)
}
// Read series data.
if v, err := s.ReadSeries("db", "default", "cpu", nil, now); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual(v, map[string]interface{}{"value": float64(100)}) {
t.Fatalf("read series(0) mismatch: %#v", v)
}
// Read new series data.
if v, err := s.ReadSeries("newdb", "default", "mem", nil, now); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual(v, map[string]interface{}{"value": float64(1000)}) {
t.Fatalf("read series(1) mismatch: %#v", v)
}
// Stop server.
node.Close()
}
// RestoreCommand is a test wrapper for main.RestoreCommand.
type RestoreCommand struct {
*main.RestoreCommand
Stderr bytes.Buffer
}
// NewRestoreCommand returns a new instance of RestoreCommand.
func NewRestoreCommand() *RestoreCommand {
cmd := &RestoreCommand{RestoreCommand: main.NewRestoreCommand()}
cmd.RestoreCommand.Stderr = &cmd.Stderr
return cmd
}
// MustReadFile reads data from a file. Panic on error.
func MustReadFile(filename string) []byte {
b, err := ioutil.ReadFile(filename)
if err != nil {
panic(err.Error())
}
return b
}
*/

235
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go generated vendored Normal file
View File

@ -0,0 +1,235 @@
package run
import (
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"path/filepath"
"runtime"
"strconv"
"strings"
"time"
"github.com/BurntSushi/toml"
)
const logo = `
8888888 .d888 888 8888888b. 888888b.
888 d88P" 888 888 "Y88b 888 "88b
888 888 888 888 888 888 .88P
888 88888b. 888888 888 888 888 888 888 888 888 8888888K.
888 888 "88b 888 888 888 888 Y8bd8P' 888 888 888 "Y88b
888 888 888 888 888 888 888 X88K 888 888 888 888
888 888 888 888 888 Y88b 888 .d8""8b. 888 .d88P 888 d88P
8888888 888 888 888 888 "Y88888 888 888 8888888P" 8888888P"
`
// Command represents the command executed by "influxd run".
type Command struct {
Version string
Branch string
Commit string
closing chan struct{}
Closed chan struct{}
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
Server *Server
}
// NewCommand return a new instance of Command.
func NewCommand() *Command {
return &Command{
closing: make(chan struct{}),
Closed: make(chan struct{}),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run parses the config from args and runs the server.
func (cmd *Command) Run(args ...string) error {
// Parse the command line flags.
options, err := cmd.ParseFlags(args...)
if err != nil {
return err
}
// Print sweet InfluxDB logo.
fmt.Print(logo)
// Write the PID file.
if err := cmd.writePIDFile(options.PIDFile); err != nil {
return fmt.Errorf("write pid file: %s", err)
}
// Set parallelism.
runtime.GOMAXPROCS(runtime.NumCPU())
// Turn on block profiling to debug stuck databases
runtime.SetBlockProfileRate(int(1 * time.Second))
// Parse config
config, err := cmd.ParseConfig(options.ConfigPath)
if err != nil {
return fmt.Errorf("parse config: %s", err)
}
// Apply any environment variables on top of the parsed config
if err := config.ApplyEnvOverrides(); err != nil {
return fmt.Errorf("apply env config: %v", err)
}
// Override config hostname if specified in the command line args.
if options.Hostname != "" {
config.Meta.Hostname = options.Hostname
}
if options.Join != "" {
config.Meta.Peers = strings.Split(options.Join, ",")
}
// Validate the configuration.
if err := config.Validate(); err != nil {
return fmt.Errorf("%s. To generate a valid configuration file run `influxd config > influxdb.generated.conf`.", err)
}
// Create server from config and start it.
s, err := NewServer(config, cmd.Version)
if err != nil {
return fmt.Errorf("create server: %s", err)
}
s.CPUProfile = options.CPUProfile
s.MemProfile = options.MemProfile
if err := s.Open(); err != nil {
return fmt.Errorf("open server: %s", err)
}
cmd.Server = s
// Mark start-up in log.
log.Printf("InfluxDB starting, version %s, branch %s, commit %s", cmd.Version, cmd.Branch, cmd.Commit)
log.Println("GOMAXPROCS set to", runtime.GOMAXPROCS(0))
// Begin monitoring the server's error channel.
go cmd.monitorServerErrors()
return nil
}
// Close shuts down the server.
func (cmd *Command) Close() error {
defer close(cmd.Closed)
close(cmd.closing)
if cmd.Server != nil {
return cmd.Server.Close()
}
return nil
}
func (cmd *Command) monitorServerErrors() {
logger := log.New(cmd.Stderr, "", log.LstdFlags)
for {
select {
case err := <-cmd.Server.Err():
logger.Println(err)
case <-cmd.closing:
return
}
}
}
// ParseFlags parses the command line flags from args and returns an options set.
func (cmd *Command) ParseFlags(args ...string) (Options, error) {
var options Options
fs := flag.NewFlagSet("", flag.ContinueOnError)
fs.StringVar(&options.ConfigPath, "config", "", "")
fs.StringVar(&options.PIDFile, "pidfile", "", "")
fs.StringVar(&options.Hostname, "hostname", "", "")
fs.StringVar(&options.Join, "join", "", "")
fs.StringVar(&options.CPUProfile, "cpuprofile", "", "")
fs.StringVar(&options.MemProfile, "memprofile", "", "")
fs.Usage = func() { fmt.Fprintln(cmd.Stderr, usage) }
if err := fs.Parse(args); err != nil {
return Options{}, err
}
return options, nil
}
// writePIDFile writes the process ID to path.
func (cmd *Command) writePIDFile(path string) error {
// Ignore if path is not set.
if path == "" {
return nil
}
// Ensure the required directory structure exists.
err := os.MkdirAll(filepath.Dir(path), 0777)
if err != nil {
return fmt.Errorf("mkdir: %s", err)
}
// Retrieve the PID and write it.
pid := strconv.Itoa(os.Getpid())
if err := ioutil.WriteFile(path, []byte(pid), 0666); err != nil {
return fmt.Errorf("write file: %s", err)
}
return nil
}
// ParseConfig parses the config at path.
// Returns a demo configuration if path is blank.
func (cmd *Command) ParseConfig(path string) (*Config, error) {
// Use demo configuration if no config path is specified.
if path == "" {
fmt.Fprintln(cmd.Stdout, "no configuration provided, using default settings")
return NewDemoConfig()
}
fmt.Fprintf(cmd.Stdout, "Using configuration at: %s\n", path)
config := NewConfig()
if _, err := toml.DecodeFile(path, &config); err != nil {
return nil, err
}
return config, nil
}
var usage = `usage: run [flags]
run starts the broker and data node server. If this is the first time running
the command then a new cluster will be initialized unless the -join argument
is used.
-config <path>
Set the path to the configuration file.
-hostname <name>
Override the hostname, the 'hostname' configuration
option will be overridden.
-join <url>
Joins the server to an existing cluster.
-pidfile <path>
Write process ID to a file.
`
// Options represents the command line options that can be parsed.
type Options struct {
ConfigPath string
PIDFile string
Hostname string
Join string
CPUProfile string
MemProfile string
}

227
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config.go generated vendored Normal file
View File

@ -0,0 +1,227 @@
package run
import (
"errors"
"fmt"
"os"
"os/user"
"path/filepath"
"reflect"
"strconv"
"strings"
"time"
"github.com/influxdb/influxdb/cluster"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/services/admin"
"github.com/influxdb/influxdb/services/collectd"
"github.com/influxdb/influxdb/services/continuous_querier"
"github.com/influxdb/influxdb/services/graphite"
"github.com/influxdb/influxdb/services/hh"
"github.com/influxdb/influxdb/services/httpd"
"github.com/influxdb/influxdb/services/monitor"
"github.com/influxdb/influxdb/services/opentsdb"
"github.com/influxdb/influxdb/services/precreator"
"github.com/influxdb/influxdb/services/retention"
"github.com/influxdb/influxdb/services/udp"
"github.com/influxdb/influxdb/tsdb"
)
// Config represents the configuration format for the influxd binary.
type Config struct {
Meta *meta.Config `toml:"meta"`
Data tsdb.Config `toml:"data"`
Cluster cluster.Config `toml:"cluster"`
Retention retention.Config `toml:"retention"`
Precreator precreator.Config `toml:"shard-precreation"`
Admin admin.Config `toml:"admin"`
HTTPD httpd.Config `toml:"http"`
Graphites []graphite.Config `toml:"graphite"`
Collectd collectd.Config `toml:"collectd"`
OpenTSDB opentsdb.Config `toml:"opentsdb"`
UDPs []udp.Config `toml:"udp"`
// Snapshot SnapshotConfig `toml:"snapshot"`
Monitoring monitor.Config `toml:"monitoring"`
ContinuousQuery continuous_querier.Config `toml:"continuous_queries"`
HintedHandoff hh.Config `toml:"hinted-handoff"`
// Server reporting
ReportingDisabled bool `toml:"reporting-disabled"`
}
// NewConfig returns an instance of Config with reasonable defaults.
func NewConfig() *Config {
c := &Config{}
c.Meta = meta.NewConfig()
c.Data = tsdb.NewConfig()
c.Cluster = cluster.NewConfig()
c.Precreator = precreator.NewConfig()
c.Admin = admin.NewConfig()
c.HTTPD = httpd.NewConfig()
c.Collectd = collectd.NewConfig()
c.OpenTSDB = opentsdb.NewConfig()
c.Graphites = append(c.Graphites, graphite.NewConfig())
c.Monitoring = monitor.NewConfig()
c.ContinuousQuery = continuous_querier.NewConfig()
c.Retention = retention.NewConfig()
c.HintedHandoff = hh.NewConfig()
return c
}
// NewDemoConfig returns the config that runs when no config is specified.
func NewDemoConfig() (*Config, error) {
c := NewConfig()
var homeDir string
// By default, store meta and data files in current users home directory
u, err := user.Current()
if err == nil {
homeDir = u.HomeDir
} else if os.Getenv("HOME") != "" {
homeDir = os.Getenv("HOME")
} else {
return nil, fmt.Errorf("failed to determine current user for storage")
}
c.Meta.Dir = filepath.Join(homeDir, ".influxdb/meta")
c.Data.Dir = filepath.Join(homeDir, ".influxdb/data")
c.HintedHandoff.Dir = filepath.Join(homeDir, ".influxdb/hh")
c.Data.WALDir = filepath.Join(homeDir, ".influxdb/wal")
c.Admin.Enabled = true
c.Monitoring.Enabled = false
return c, nil
}
// Validate returns an error if the config is invalid.
func (c *Config) Validate() error {
if c.Meta.Dir == "" {
return errors.New("Meta.Dir must be specified")
} else if c.Data.Dir == "" {
return errors.New("Data.Dir must be specified")
} else if c.HintedHandoff.Dir == "" {
return errors.New("HintedHandoff.Dir must be specified")
} else if c.Data.WALDir == "" {
return errors.New("Data.WALDir must be specified")
}
for _, g := range c.Graphites {
if err := g.Validate(); err != nil {
return fmt.Errorf("invalid graphite config: %v", err)
}
}
return nil
}
func (c *Config) ApplyEnvOverrides() error {
return c.applyEnvOverrides("INFLUXDB", reflect.ValueOf(c))
}
func (c *Config) applyEnvOverrides(prefix string, spec reflect.Value) error {
// If we have a pointer, dereference it
s := spec
if spec.Kind() == reflect.Ptr {
s = spec.Elem()
}
// Make sure we have struct
if s.Kind() != reflect.Struct {
return nil
}
typeOfSpec := s.Type()
for i := 0; i < s.NumField(); i++ {
f := s.Field(i)
// Get the toml tag to determine what env var name to use
configName := typeOfSpec.Field(i).Tag.Get("toml")
// Replace hyphens with underscores to avoid issues with shells
configName = strings.Replace(configName, "-", "_", -1)
fieldName := typeOfSpec.Field(i).Name
// Skip any fields that we cannot set
if f.CanSet() || f.Kind() == reflect.Slice {
// Use the upper-case prefix and toml name for the env var
key := strings.ToUpper(configName)
if prefix != "" {
key = strings.ToUpper(fmt.Sprintf("%s_%s", prefix, configName))
}
value := os.Getenv(key)
// If the type is s slice, apply to each using the index as a suffix
// e.g. GRAPHITE_0
if f.Kind() == reflect.Slice || f.Kind() == reflect.Array {
for i := 0; i < f.Len(); i++ {
if err := c.applyEnvOverrides(fmt.Sprintf("%s_%d", key, i), f.Index(i)); err != nil {
return err
}
}
continue
}
// If it's a sub-config, recursively apply
if f.Kind() == reflect.Struct || f.Kind() == reflect.Ptr {
if err := c.applyEnvOverrides(key, f); err != nil {
return err
}
continue
}
// Skip any fields we don't have a value to set
if value == "" {
continue
}
switch f.Kind() {
case reflect.String:
f.SetString(value)
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
var intValue int64
// Handle toml.Duration
if f.Type().Name() == "Duration" {
dur, err := time.ParseDuration(value)
if err != nil {
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
}
intValue = dur.Nanoseconds()
} else {
var err error
intValue, err = strconv.ParseInt(value, 0, f.Type().Bits())
if err != nil {
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
}
}
f.SetInt(intValue)
case reflect.Bool:
boolValue, err := strconv.ParseBool(value)
if err != nil {
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
}
f.SetBool(boolValue)
case reflect.Float32, reflect.Float64:
floatValue, err := strconv.ParseFloat(value, f.Type().Bits())
if err != nil {
return fmt.Errorf("failed to apply %v to %v using type %v and value '%v'", key, fieldName, f.Type().String(), value)
}
f.SetFloat(floatValue)
default:
if err := c.applyEnvOverrides(key, f); err != nil {
return err
}
}
}
}
return nil
}

73
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config_command.go generated vendored Normal file
View File

@ -0,0 +1,73 @@
package run
import (
"flag"
"fmt"
"io"
"os"
"github.com/BurntSushi/toml"
)
// PrintConfigCommand represents the command executed by "influxd config".
type PrintConfigCommand struct {
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewPrintConfigCommand return a new instance of PrintConfigCommand.
func NewPrintConfigCommand() *PrintConfigCommand {
return &PrintConfigCommand{
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run parses and prints the current config loaded.
func (cmd *PrintConfigCommand) Run(args ...string) error {
// Parse command flags.
fs := flag.NewFlagSet("", flag.ContinueOnError)
configPath := fs.String("config", "", "")
hostname := fs.String("hostname", "", "")
fs.Usage = func() { fmt.Fprintln(cmd.Stderr, printConfigUsage) }
if err := fs.Parse(args); err != nil {
return err
}
// Parse config from path.
config, err := cmd.parseConfig(*configPath)
if err != nil {
return fmt.Errorf("parse config: %s", err)
}
// Override config properties.
if *hostname != "" {
config.Meta.Hostname = *hostname
}
toml.NewEncoder(cmd.Stdout).Encode(config)
fmt.Fprint(cmd.Stdout, "\n")
return nil
}
// ParseConfig parses the config at path.
// Returns a demo configuration if path is blank.
func (cmd *PrintConfigCommand) parseConfig(path string) (*Config, error) {
if path == "" {
return NewDemoConfig()
}
config := NewConfig()
if _, err := toml.DecodeFile(path, &config); err != nil {
return nil, err
}
return config, nil
}
var printConfigUsage = `usage: config
config displays the default configuration
`

144
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/config_test.go generated vendored Normal file
View File

@ -0,0 +1,144 @@
package run_test
import (
"os"
"testing"
"github.com/BurntSushi/toml"
"github.com/influxdb/influxdb/cmd/influxd/run"
)
// Ensure the configuration can be parsed.
func TestConfig_Parse(t *testing.T) {
// Parse configuration.
var c run.Config
if _, err := toml.Decode(`
[meta]
dir = "/tmp/meta"
[data]
dir = "/tmp/data"
[cluster]
[admin]
bind-address = ":8083"
[http]
bind-address = ":8087"
[[graphite]]
protocol = "udp"
[[graphite]]
protocol = "tcp"
[collectd]
bind-address = ":1000"
[opentsdb]
bind-address = ":2000"
[[udp]]
bind-address = ":4444"
[monitoring]
enabled = true
[continuous_queries]
enabled = true
`, &c); err != nil {
t.Fatal(err)
}
// Validate configuration.
if c.Meta.Dir != "/tmp/meta" {
t.Fatalf("unexpected meta dir: %s", c.Meta.Dir)
} else if c.Data.Dir != "/tmp/data" {
t.Fatalf("unexpected data dir: %s", c.Data.Dir)
} else if c.Admin.BindAddress != ":8083" {
t.Fatalf("unexpected admin bind address: %s", c.Admin.BindAddress)
} else if c.HTTPD.BindAddress != ":8087" {
t.Fatalf("unexpected api bind address: %s", c.HTTPD.BindAddress)
} else if len(c.Graphites) != 2 {
t.Fatalf("unexpected graphites count: %d", len(c.Graphites))
} else if c.Graphites[0].Protocol != "udp" {
t.Fatalf("unexpected graphite protocol(0): %s", c.Graphites[0].Protocol)
} else if c.Graphites[1].Protocol != "tcp" {
t.Fatalf("unexpected graphite protocol(1): %s", c.Graphites[1].Protocol)
} else if c.Collectd.BindAddress != ":1000" {
t.Fatalf("unexpected collectd bind address: %s", c.Collectd.BindAddress)
} else if c.OpenTSDB.BindAddress != ":2000" {
t.Fatalf("unexpected opentsdb bind address: %s", c.OpenTSDB.BindAddress)
} else if c.UDPs[0].BindAddress != ":4444" {
t.Fatalf("unexpected udp bind address: %s", c.UDPs[0].BindAddress)
} else if c.Monitoring.Enabled != true {
t.Fatalf("unexpected monitoring enabled: %v", c.Monitoring.Enabled)
} else if c.ContinuousQuery.Enabled != true {
t.Fatalf("unexpected continuous query enabled: %v", c.ContinuousQuery.Enabled)
}
}
// Ensure the configuration can be parsed.
func TestConfig_Parse_EnvOverride(t *testing.T) {
// Parse configuration.
var c run.Config
if _, err := toml.Decode(`
[meta]
dir = "/tmp/meta"
[data]
dir = "/tmp/data"
[cluster]
[admin]
bind-address = ":8083"
[http]
bind-address = ":8087"
[[graphite]]
protocol = "udp"
[[graphite]]
protocol = "tcp"
[collectd]
bind-address = ":1000"
[opentsdb]
bind-address = ":2000"
[[udp]]
bind-address = ":4444"
[monitoring]
enabled = true
[continuous_queries]
enabled = true
`, &c); err != nil {
t.Fatal(err)
}
if err := os.Setenv("INFLUXDB_UDP_BIND_ADDRESS", ":1234"); err != nil {
t.Fatalf("failed to set env var: %v", err)
}
if err := os.Setenv("INFLUXDB_GRAPHITE_1_PROTOCOL", "udp"); err != nil {
t.Fatalf("failed to set env var: %v", err)
}
if err := c.ApplyEnvOverrides(); err != nil {
t.Fatalf("failed to apply env overrides: %v", err)
}
if c.UDPs[0].BindAddress != ":4444" {
t.Fatalf("unexpected udp bind address: %s", c.UDPs[0].BindAddress)
}
if c.Graphites[1].Protocol != "udp" {
t.Fatalf("unexpected graphite protocol(0): %s", c.Graphites[0].Protocol)
}
}

536
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go generated vendored Normal file
View File

@ -0,0 +1,536 @@
package run
import (
"bytes"
"fmt"
"log"
"net"
"net/http"
"os"
"runtime"
"runtime/pprof"
"strings"
"time"
"github.com/influxdb/influxdb/cluster"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/services/admin"
"github.com/influxdb/influxdb/services/collectd"
"github.com/influxdb/influxdb/services/continuous_querier"
"github.com/influxdb/influxdb/services/graphite"
"github.com/influxdb/influxdb/services/hh"
"github.com/influxdb/influxdb/services/httpd"
"github.com/influxdb/influxdb/services/opentsdb"
"github.com/influxdb/influxdb/services/precreator"
"github.com/influxdb/influxdb/services/retention"
"github.com/influxdb/influxdb/services/snapshotter"
"github.com/influxdb/influxdb/services/udp"
"github.com/influxdb/influxdb/tcp"
"github.com/influxdb/influxdb/tsdb"
_ "github.com/influxdb/influxdb/tsdb/engine"
)
// Server represents a container for the metadata and storage data and services.
// It is built using a Config and it manages the startup and shutdown of all
// services in the proper order.
type Server struct {
version string // Build version
err chan error
closing chan struct{}
Hostname string
BindAddress string
Listener net.Listener
MetaStore *meta.Store
TSDBStore *tsdb.Store
QueryExecutor *tsdb.QueryExecutor
PointsWriter *cluster.PointsWriter
ShardWriter *cluster.ShardWriter
ShardMapper *cluster.ShardMapper
HintedHandoff *hh.Service
Services []Service
// These references are required for the tcp muxer.
ClusterService *cluster.Service
SnapshotterService *snapshotter.Service
// Server reporting
reportingDisabled bool
// Profiling
CPUProfile string
MemProfile string
}
// NewServer returns a new instance of Server built from a config.
func NewServer(c *Config, version string) (*Server, error) {
// Construct base meta store and data store.
tsdbStore := tsdb.NewStore(c.Data.Dir)
tsdbStore.EngineOptions.Config = c.Data
s := &Server{
version: version,
err: make(chan error),
closing: make(chan struct{}),
Hostname: c.Meta.Hostname,
BindAddress: c.Meta.BindAddress,
MetaStore: meta.NewStore(c.Meta),
TSDBStore: tsdbStore,
reportingDisabled: c.ReportingDisabled,
}
// Copy TSDB configuration.
s.TSDBStore.EngineOptions.MaxWALSize = c.Data.MaxWALSize
s.TSDBStore.EngineOptions.WALFlushInterval = time.Duration(c.Data.WALFlushInterval)
s.TSDBStore.EngineOptions.WALPartitionFlushDelay = time.Duration(c.Data.WALPartitionFlushDelay)
// Set the shard mapper
s.ShardMapper = cluster.NewShardMapper(time.Duration(c.Cluster.ShardMapperTimeout))
s.ShardMapper.ForceRemoteMapping = c.Cluster.ForceRemoteShardMapping
s.ShardMapper.MetaStore = s.MetaStore
s.ShardMapper.TSDBStore = s.TSDBStore
// Initialize query executor.
s.QueryExecutor = tsdb.NewQueryExecutor(s.TSDBStore)
s.QueryExecutor.MetaStore = s.MetaStore
s.QueryExecutor.MetaStatementExecutor = &meta.StatementExecutor{Store: s.MetaStore}
s.QueryExecutor.ShardMapper = s.ShardMapper
// Set the shard writer
s.ShardWriter = cluster.NewShardWriter(time.Duration(c.Cluster.ShardWriterTimeout))
s.ShardWriter.MetaStore = s.MetaStore
// Create the hinted handoff service
s.HintedHandoff = hh.NewService(c.HintedHandoff, s.ShardWriter)
// Initialize points writer.
s.PointsWriter = cluster.NewPointsWriter()
s.PointsWriter.WriteTimeout = time.Duration(c.Cluster.WriteTimeout)
s.PointsWriter.MetaStore = s.MetaStore
s.PointsWriter.TSDBStore = s.TSDBStore
s.PointsWriter.ShardWriter = s.ShardWriter
s.PointsWriter.HintedHandoff = s.HintedHandoff
// Append services.
s.appendClusterService(c.Cluster)
s.appendPrecreatorService(c.Precreator)
s.appendSnapshotterService()
s.appendAdminService(c.Admin)
s.appendContinuousQueryService(c.ContinuousQuery)
s.appendHTTPDService(c.HTTPD)
s.appendCollectdService(c.Collectd)
if err := s.appendOpenTSDBService(c.OpenTSDB); err != nil {
return nil, err
}
for _, g := range c.UDPs {
s.appendUDPService(g)
}
s.appendRetentionPolicyService(c.Retention)
for _, g := range c.Graphites {
if err := s.appendGraphiteService(g); err != nil {
return nil, err
}
}
return s, nil
}
func (s *Server) appendClusterService(c cluster.Config) {
srv := cluster.NewService(c)
srv.TSDBStore = s.TSDBStore
srv.MetaStore = s.MetaStore
s.Services = append(s.Services, srv)
s.ClusterService = srv
}
func (s *Server) appendSnapshotterService() {
srv := snapshotter.NewService()
srv.TSDBStore = s.TSDBStore
srv.MetaStore = s.MetaStore
s.Services = append(s.Services, srv)
s.SnapshotterService = srv
}
func (s *Server) appendRetentionPolicyService(c retention.Config) {
if !c.Enabled {
return
}
srv := retention.NewService(c)
srv.MetaStore = s.MetaStore
srv.TSDBStore = s.TSDBStore
s.Services = append(s.Services, srv)
}
func (s *Server) appendAdminService(c admin.Config) {
if !c.Enabled {
return
}
srv := admin.NewService(c)
s.Services = append(s.Services, srv)
}
func (s *Server) appendHTTPDService(c httpd.Config) {
if !c.Enabled {
return
}
srv := httpd.NewService(c)
srv.Handler.MetaStore = s.MetaStore
srv.Handler.QueryExecutor = s.QueryExecutor
srv.Handler.PointsWriter = s.PointsWriter
srv.Handler.Version = s.version
// If a ContinuousQuerier service has been started, attach it.
for _, srvc := range s.Services {
if cqsrvc, ok := srvc.(continuous_querier.ContinuousQuerier); ok {
srv.Handler.ContinuousQuerier = cqsrvc
}
}
s.Services = append(s.Services, srv)
}
func (s *Server) appendCollectdService(c collectd.Config) {
if !c.Enabled {
return
}
srv := collectd.NewService(c)
srv.MetaStore = s.MetaStore
srv.PointsWriter = s.PointsWriter
s.Services = append(s.Services, srv)
}
func (s *Server) appendOpenTSDBService(c opentsdb.Config) error {
if !c.Enabled {
return nil
}
srv, err := opentsdb.NewService(c)
if err != nil {
return err
}
srv.PointsWriter = s.PointsWriter
srv.MetaStore = s.MetaStore
s.Services = append(s.Services, srv)
return nil
}
func (s *Server) appendGraphiteService(c graphite.Config) error {
if !c.Enabled {
return nil
}
srv, err := graphite.NewService(c)
if err != nil {
return err
}
srv.PointsWriter = s.PointsWriter
srv.MetaStore = s.MetaStore
s.Services = append(s.Services, srv)
return nil
}
func (s *Server) appendPrecreatorService(c precreator.Config) error {
if !c.Enabled {
return nil
}
srv, err := precreator.NewService(c)
if err != nil {
return err
}
srv.MetaStore = s.MetaStore
s.Services = append(s.Services, srv)
return nil
}
func (s *Server) appendUDPService(c udp.Config) {
if !c.Enabled {
return
}
srv := udp.NewService(c)
srv.PointsWriter = s.PointsWriter
s.Services = append(s.Services, srv)
}
func (s *Server) appendContinuousQueryService(c continuous_querier.Config) {
if !c.Enabled {
return
}
srv := continuous_querier.NewService(c)
srv.MetaStore = s.MetaStore
srv.QueryExecutor = s.QueryExecutor
srv.PointsWriter = s.PointsWriter
s.Services = append(s.Services, srv)
}
// Err returns an error channel that multiplexes all out of band errors received from all services.
func (s *Server) Err() <-chan error { return s.err }
// Open opens the meta and data store and all services.
func (s *Server) Open() error {
if err := func() error {
// Start profiling, if set.
startProfile(s.CPUProfile, s.MemProfile)
host, port, err := s.hostAddr()
if err != nil {
return err
}
hostport := net.JoinHostPort(host, port)
addr, err := net.ResolveTCPAddr("tcp", hostport)
if err != nil {
return fmt.Errorf("resolve tcp: addr=%s, err=%s", hostport, err)
}
s.MetaStore.Addr = addr
s.MetaStore.RemoteAddr = &tcpaddr{hostport}
// Open shared TCP connection.
ln, err := net.Listen("tcp", s.BindAddress)
if err != nil {
return fmt.Errorf("listen: %s", err)
}
s.Listener = ln
// The port 0 is used, we need to retrieve the port assigned by the kernel
if strings.HasSuffix(s.BindAddress, ":0") {
s.MetaStore.Addr = ln.Addr()
}
// Multiplex listener.
mux := tcp.NewMux()
s.MetaStore.RaftListener = mux.Listen(meta.MuxRaftHeader)
s.MetaStore.ExecListener = mux.Listen(meta.MuxExecHeader)
s.MetaStore.RPCListener = mux.Listen(meta.MuxRPCHeader)
s.ClusterService.Listener = mux.Listen(cluster.MuxHeader)
s.SnapshotterService.Listener = mux.Listen(snapshotter.MuxHeader)
go mux.Serve(ln)
// Open meta store.
if err := s.MetaStore.Open(); err != nil {
return fmt.Errorf("open meta store: %s", err)
}
go s.monitorErrorChan(s.MetaStore.Err())
// Wait for the store to initialize.
<-s.MetaStore.Ready()
// Open TSDB store.
if err := s.TSDBStore.Open(); err != nil {
return fmt.Errorf("open tsdb store: %s", err)
}
// Open the hinted handoff service
if err := s.HintedHandoff.Open(); err != nil {
return fmt.Errorf("open hinted handoff: %s", err)
}
for _, service := range s.Services {
if err := service.Open(); err != nil {
return fmt.Errorf("open service: %s", err)
}
}
// Start the reporting service, if not disabled.
if !s.reportingDisabled {
go s.startServerReporting()
}
return nil
}(); err != nil {
s.Close()
return err
}
return nil
}
// Close shuts down the meta and data stores and all services.
func (s *Server) Close() error {
stopProfile()
if s.Listener != nil {
s.Listener.Close()
}
if s.MetaStore != nil {
s.MetaStore.Close()
}
if s.TSDBStore != nil {
s.TSDBStore.Close()
}
if s.HintedHandoff != nil {
s.HintedHandoff.Close()
}
for _, service := range s.Services {
service.Close()
}
close(s.closing)
return nil
}
// startServerReporting starts periodic server reporting.
func (s *Server) startServerReporting() {
for {
select {
case <-s.closing:
return
default:
}
if err := s.MetaStore.WaitForLeader(30 * time.Second); err != nil {
log.Printf("no leader available for reporting: %s", err.Error())
time.Sleep(time.Second)
continue
}
s.reportServer()
<-time.After(24 * time.Hour)
}
}
// reportServer reports anonymous statistics about the system.
func (s *Server) reportServer() {
dis, err := s.MetaStore.Databases()
if err != nil {
log.Printf("failed to retrieve databases for reporting: %s", err.Error())
return
}
numDatabases := len(dis)
numMeasurements := 0
numSeries := 0
for _, di := range dis {
d := s.TSDBStore.DatabaseIndex(di.Name)
if d == nil {
// No data in this store for this database.
continue
}
m, s := d.MeasurementSeriesCounts()
numMeasurements += m
numSeries += s
}
clusterID, err := s.MetaStore.ClusterID()
if err != nil {
log.Printf("failed to retrieve cluster ID for reporting: %s", err.Error())
return
}
json := fmt.Sprintf(`[{
"name":"reports",
"columns":["os", "arch", "version", "server_id", "cluster_id", "num_series", "num_measurements", "num_databases"],
"points":[["%s", "%s", "%s", "%x", "%x", "%d", "%d", "%d"]]
}]`, runtime.GOOS, runtime.GOARCH, s.version, s.MetaStore.NodeID(), clusterID, numSeries, numMeasurements, numDatabases)
data := bytes.NewBufferString(json)
log.Printf("Sending anonymous usage statistics to m.influxdb.com")
client := http.Client{Timeout: time.Duration(5 * time.Second)}
go client.Post("http://m.influxdb.com:8086/db/reporting/series?u=reporter&p=influxdb", "application/json", data)
}
// monitorErrorChan reads an error channel and resends it through the server.
func (s *Server) monitorErrorChan(ch <-chan error) {
for {
select {
case err, ok := <-ch:
if !ok {
return
}
s.err <- err
case <-s.closing:
return
}
}
}
// hostAddr returns the host and port that remote nodes will use to reach this
// node.
func (s *Server) hostAddr() (string, string, error) {
// Resolve host to address.
_, port, err := net.SplitHostPort(s.BindAddress)
if err != nil {
return "", "", fmt.Errorf("split bind address: %s", err)
}
host := s.Hostname
// See if we might have a port that will override the BindAddress port
if host != "" && host[len(host)-1] >= '0' && host[len(host)-1] <= '9' && strings.Contains(host, ":") {
hostArg, portArg, err := net.SplitHostPort(s.Hostname)
if err != nil {
return "", "", err
}
if hostArg != "" {
host = hostArg
}
if portArg != "" {
port = portArg
}
}
return host, port, nil
}
// Service represents a service attached to the server.
type Service interface {
Open() error
Close() error
}
// prof stores the file locations of active profiles.
var prof struct {
cpu *os.File
mem *os.File
}
// StartProfile initializes the cpu and memory profile, if specified.
func startProfile(cpuprofile, memprofile string) {
if cpuprofile != "" {
f, err := os.Create(cpuprofile)
if err != nil {
log.Fatalf("cpuprofile: %v", err)
}
log.Printf("writing CPU profile to: %s\n", cpuprofile)
prof.cpu = f
pprof.StartCPUProfile(prof.cpu)
}
if memprofile != "" {
f, err := os.Create(memprofile)
if err != nil {
log.Fatalf("memprofile: %v", err)
}
log.Printf("writing mem profile to: %s\n", memprofile)
prof.mem = f
runtime.MemProfileRate = 4096
}
}
// StopProfile closes the cpu and memory profiles if they are running.
func stopProfile() {
if prof.cpu != nil {
pprof.StopCPUProfile()
prof.cpu.Close()
log.Println("CPU profile stopped")
}
if prof.mem != nil {
pprof.Lookup("heap").WriteTo(prof.mem, 0)
prof.mem.Close()
log.Println("mem profile stopped")
}
}
type tcpaddr struct{ host string }
func (a *tcpaddr) Network() string { return "tcp" }
func (a *tcpaddr) String() string { return a.host }

312
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_helpers_test.go generated vendored Normal file
View File

@ -0,0 +1,312 @@
// This package is a set of convenience helpers and structs to make integration testing easier
package run_test
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"math"
"net/http"
"net/url"
"os"
"regexp"
"strings"
"testing"
"time"
"github.com/influxdb/influxdb/cmd/influxd/run"
"github.com/influxdb/influxdb/meta"
"github.com/influxdb/influxdb/services/httpd"
"github.com/influxdb/influxdb/toml"
)
// Server represents a test wrapper for run.Server.
type Server struct {
*run.Server
Config *run.Config
}
// NewServer returns a new instance of Server.
func NewServer(c *run.Config) *Server {
srv, _ := run.NewServer(c, "testServer")
s := Server{
Server: srv,
Config: c,
}
s.TSDBStore.EngineOptions.Config = c.Data
configureLogging(&s)
return &s
}
// OpenServer opens a test server.
func OpenServer(c *run.Config, joinURLs string) *Server {
s := NewServer(c)
configureLogging(s)
if err := s.Open(); err != nil {
panic(err.Error())
}
return s
}
// OpenServerWithVersion opens a test server with a specific version.
func OpenServerWithVersion(c *run.Config, version string) *Server {
srv, _ := run.NewServer(c, version)
s := Server{
Server: srv,
Config: c,
}
configureLogging(&s)
if err := s.Open(); err != nil {
panic(err.Error())
}
return &s
}
// Close shuts down the server and removes all temporary paths.
func (s *Server) Close() {
os.RemoveAll(s.Config.Meta.Dir)
os.RemoveAll(s.Config.Data.Dir)
os.RemoveAll(s.Config.HintedHandoff.Dir)
s.Server.Close()
}
// URL returns the base URL for the httpd endpoint.
func (s *Server) URL() string {
for _, service := range s.Services {
if service, ok := service.(*httpd.Service); ok {
return "http://" + service.Addr().String()
}
}
panic("httpd server not found in services")
}
// CreateDatabaseAndRetentionPolicy will create the database and retention policy.
func (s *Server) CreateDatabaseAndRetentionPolicy(db string, rp *meta.RetentionPolicyInfo) error {
if _, err := s.MetaStore.CreateDatabase(db); err != nil {
return err
} else if _, err := s.MetaStore.CreateRetentionPolicy(db, rp); err != nil {
return err
}
return nil
}
// Query executes a query against the server and returns the results.
func (s *Server) Query(query string) (results string, err error) {
return s.QueryWithParams(query, nil)
}
// Query executes a query against the server and returns the results.
func (s *Server) QueryWithParams(query string, values url.Values) (results string, err error) {
if values == nil {
values = url.Values{}
}
values.Set("q", query)
resp, err := http.Get(s.URL() + "/query?" + values.Encode())
if err != nil {
return "", err
//} else if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusBadRequest {
}
body := string(MustReadAll(resp.Body))
switch resp.StatusCode {
case http.StatusBadRequest:
if !expectPattern(".*error parsing query*.", body) {
return "", fmt.Errorf("unexpected status code: code=%d, body=%s", resp.StatusCode, body)
}
return body, nil
case http.StatusOK:
return body, nil
default:
return "", fmt.Errorf("unexpected status code: code=%d, body=%s", resp.StatusCode, body)
}
}
// Write executes a write against the server and returns the results.
func (s *Server) Write(db, rp, body string, params url.Values) (results string, err error) {
if params == nil {
params = url.Values{}
}
if params.Get("db") == "" {
params.Set("db", db)
}
if params.Get("rp") == "" {
params.Set("rp", rp)
}
resp, err := http.Post(s.URL()+"/write?"+params.Encode(), "", strings.NewReader(body))
if err != nil {
return "", err
} else if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
return "", fmt.Errorf("invalid status code: code=%d, body=%s", resp.StatusCode, MustReadAll(resp.Body))
}
return string(MustReadAll(resp.Body)), nil
}
// NewConfig returns the default config with temporary paths.
func NewConfig() *run.Config {
c := run.NewConfig()
c.ReportingDisabled = true
c.Meta.Dir = MustTempFile()
c.Meta.BindAddress = "127.0.0.1:0"
c.Meta.HeartbeatTimeout = toml.Duration(50 * time.Millisecond)
c.Meta.ElectionTimeout = toml.Duration(50 * time.Millisecond)
c.Meta.LeaderLeaseTimeout = toml.Duration(50 * time.Millisecond)
c.Meta.CommitTimeout = toml.Duration(5 * time.Millisecond)
c.Data.Dir = MustTempFile()
c.Data.WALDir = MustTempFile()
c.HintedHandoff.Dir = MustTempFile()
c.HTTPD.Enabled = true
c.HTTPD.BindAddress = "127.0.0.1:0"
c.HTTPD.LogEnabled = testing.Verbose()
return c
}
func newRetentionPolicyInfo(name string, rf int, duration time.Duration) *meta.RetentionPolicyInfo {
return &meta.RetentionPolicyInfo{Name: name, ReplicaN: rf, Duration: duration}
}
func maxFloat64() string {
maxFloat64, _ := json.Marshal(math.MaxFloat64)
return string(maxFloat64)
}
func maxInt64() string {
maxInt64, _ := json.Marshal(^int64(0))
return string(maxInt64)
}
func now() time.Time {
return time.Now().UTC()
}
func yesterday() time.Time {
return now().Add(-1 * time.Hour * 24)
}
func mustParseTime(layout, value string) time.Time {
tm, err := time.Parse(layout, value)
if err != nil {
panic(err)
}
return tm
}
// MustReadAll reads r. Panic on error.
func MustReadAll(r io.Reader) []byte {
b, err := ioutil.ReadAll(r)
if err != nil {
panic(err)
}
return b
}
// MustTempFile returns a path to a temporary file.
func MustTempFile() string {
f, err := ioutil.TempFile("", "influxd-")
if err != nil {
panic(err)
}
f.Close()
os.Remove(f.Name())
return f.Name()
}
func expectPattern(exp, act string) bool {
re := regexp.MustCompile(exp)
if !re.MatchString(act) {
return false
}
return true
}
type Query struct {
name string
command string
params url.Values
exp, act string
pattern bool
skip bool
}
// Execute runs the command and returns an err if it fails
func (q *Query) Execute(s *Server) (err error) {
if q.params == nil {
q.act, err = s.Query(q.command)
return
}
q.act, err = s.QueryWithParams(q.command, q.params)
return
}
func (q *Query) success() bool {
if q.pattern {
return expectPattern(q.exp, q.act)
}
return q.exp == q.act
}
func (q *Query) Error(err error) string {
return fmt.Sprintf("%s: %v", q.name, err)
}
func (q *Query) failureMessage() string {
return fmt.Sprintf("%s: unexpected results\nquery: %s\nexp: %s\nactual: %s\n", q.name, q.command, q.exp, q.act)
}
type Test struct {
initialized bool
write string
params url.Values
db string
rp string
exp string
queries []*Query
}
func NewTest(db, rp string) Test {
return Test{
db: db,
rp: rp,
}
}
func (t *Test) addQueries(q ...*Query) {
t.queries = append(t.queries, q...)
}
func (t *Test) init(s *Server) error {
if t.write == "" || t.initialized {
return nil
}
t.initialized = true
if res, err := s.Write(t.db, t.rp, t.write, t.params); err != nil {
return err
} else if t.exp != res {
return fmt.Errorf("unexpected results\nexp: %s\ngot: %s\n", t.exp, res)
}
return nil
}
func configureLogging(s *Server) {
// Set the logger to discard unless verbose is on
if !testing.Verbose() {
type logSetter interface {
SetLogger(*log.Logger)
}
nullLogger := log.New(ioutil.Discard, "", 0)
s.MetaStore.Logger = nullLogger
s.TSDBStore.Logger = nullLogger
s.HintedHandoff.SetLogger(nullLogger)
for _, service := range s.Services {
if service, ok := service.(logSetter); ok {
service.SetLogger(nullLogger)
}
}
}
}

3719
Godeps/_workspace/src/github.com/influxdb/influxdb/cmd/influxd/run/server_test.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More