Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Wipe out workspace in preparation for v2 merge 

"Knock knock."

"Who's there?"

"InfluxDB Veet."

...
pull/11014/head
Mark Rushakoff 2019-01-11 10:38:50 -08:00
parent 26afe32611
commit f2898d1992
744 changed files with 0 additions and 236465 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
.circleci/config.yml
View File

@ -1,19 +0,0 @@
version: 2
jobs:
build:
machine:
enabled: true
docker_layer_caching: true
environment:
- PARALLELISM: 4 # Input to influxdb/build.py
parallelism: 5 # How many CircleCI test containers
steps:
- checkout
- run:
name: Ensure CircleCI parallelism matches "./test.sh count"
command: "[ `./test.sh count` -eq $CIRCLE_NODE_TOTAL ]"
- run:
name: Execute test
command: ./test.sh $CIRCLE_NODE_INDEX
no_output_timeout: 1500s

1
.dockerignore
View File

@ -1 +0,0 @@
build

38
.github/ISSUE_TEMPLATE/bug-report.md vendored
View File

@ -1,38 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
---
__System info:__ [Include InfluxDB version, operating system name, and other relevant details]
__Steps to reproduce:__
1. [First Step]
2. [Second Step]
3. [and so on...]
__Expected behavior:__ [What you expected to happen]
__Actual behavior:__ [What actually happened]
__Additional info:__ [Include gist of relevant config, logs, etc.]
If this is an issue of for performance, locking, etc the following commands are useful to create debug information for the team.
```
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
```
**Please note** It will take at least 30 seconds for the first cURL command above to return a response.
This is because it will run a CPU profile as part of its information gathering, which takes 30 seconds to collect.
Ideally you should run these commands when you're experiencing problems, so we can capture the state of the system at that time.
If you're concerned about running a CPU profile (which only has a small, temporary impact on performance), then you can set `?cpu=false` or omit `?cpu=true` altogether.
Please run those if possible and link them from a [gist](http://gist.github.com) or simply attach them as a comment to the issue.
*Please note, the quickest way to fix a bug is to open a Pull Request.*

15
.github/ISSUE_TEMPLATE/feature-request.md vendored
View File

@ -1,15 +0,0 @@
---
name: Feature request
about: Opening a feature request kicks off a discussion
---
__Proposal:__ [Description of the feature]
__Current behavior:__ [What currently happens]
__Desired behavior:__ [What you would like to happen]
__Use case:__ [Why is this important (helps with prioritizing requests)]
Requests may be closed if we're not actively planning to work on them.

10
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@ -1,10 +0,0 @@
###### Required for all non-trivial PRs
- [ ] Sign [CLA](https://influxdata.com/community/cla/) (if not already signed)
###### Required only if applicable
_You can erase any checkboxes below this note if they are not applicable to your Pull Request._
- [ ] [InfluxQL Spec](https://github.com/influxdata/influxdb/blob/master/influxql/README.md) updated
- [ ] Provide example syntax
- [ ] Update man page when modifying a command
- [ ] Config changes: update sample config (`etc/config.sample.toml`), server `NewDemoConfig` method, and `Diagnostics` methods reporting config settings, if necessary
- [ ] [InfluxData Documentation](https://github.com/influxdata/docs.influxdata.com): issue filed or pull request submitted \<link to issue or pull request\>

83
.gitignore vendored
View File

@ -1,83 +0,0 @@
# Keep editor-specific, non-project specific ignore rules in global .gitignore:
# https://help.github.com/articles/ignoring-files/#create-a-global-gitignore
*~
src/
config.json
/bin/
/query/a.out*
# ignore generated files.
cmd/influxd/version.go
# executables
*.test
influx_tsm
**/influx_tsm
!**/influx_tsm/
influx_stress
**/influx_stress
!**/influx_stress/
influxd
**/influxd
!**/influxd/
influx
**/influx
!**/influx/
influxdb
**/influxdb
!**/influxdb/
influx_inspect
**/influx_inspect
!**/influx_inspect/
/benchmark-tool
/main
/benchmark-storage
godef
gosym
gocode
inspect-raft
# dependencies
out_rpm/
packages/
# autconf
autom4te.cache/
config.log
config.status
# log file
influxdb.log
benchmark.log
# config file
config.toml
# test data files
integration/migration_data/
test-logs/
# man outputs
man/*.xml
man/*.1
man/*.1.gz
# test outputs
/test-results.xml
# profile data
/prof
# vendored files
/vendor

23
.hooks/pre-commit
View File

@ -1,23 +0,0 @@
#!/usr/bin/env bash
fmtcount=`git ls-files | grep '.go$' | xargs gofmt -l 2>&1 | wc -l`
if [ $fmtcount -gt 0 ]; then
echo "Some files aren't formatted, please run 'go fmt ./...' to format your source code before committing"
exit 1
fi
vetcount=`go tool vet ./ 2>&1 | wc -l`
if [ $vetcount -gt 0 ]; then
echo "Some files aren't passing vet heuristics, please run 'go vet ./...' to see the errors it flags and correct your source code before committing"
exit 1
fi
exit 0
# Ensure FIXME lines are removed before commit.
fixme_lines=$(git diff --cached | grep ^+ | grep -v pre-commit | grep FIXME | sed 's_^+\s*__g')
if [ "$fixme_lines" != "" ]; then
echo "Please remove the following lines:"
echo -e "$fixme_lines"
exit 1
fi

6
.mention-bot
View File

@ -1,6 +0,0 @@
{
"maxReviewers": 3,
"fileBlacklist": ["CHANGELOG.md"],
"userBlacklist": ["pauldix", "toddboom", "aviau", "mark-rushakoff"],
"requiredOrgs": ["influxdata"]
}

3221
CHANGELOG.md
View File

File diff suppressed because it is too large Load Diff

82
CODING_GUIDELINES.md
View File

@ -1,82 +0,0 @@
_This document is currently in draft form._
# Background
The goal of this guide is to capture some Do and Don'ts of Go code for the InfluxDB database. When it comes to Go, writing good code is often achieved with the help of tools like `go fmt` and `go vet`. However there are still some practices not enforceable by any tools. This guide lists some specific practices to follow when writing code for the database.
*Like everything, one needs to use good judgment.* There will always be times when it doesn't make sense to follow a guideline outlined in this document. If that case arises, be ready to justify your choices.
# The Guidelines
## Try not to use third-party libraries
A third-party package is defined as one that is not part of the standard Go distribution. Generally speaking we prefer to minimize our use of third-party packages, and avoid them unless absolutely necessarily. We'll often write a little bit of code rather than pull in a third-party package. Of course, we do use some third-party packages -- most importantly we use [BoltDB](https://github.com/boltdb/bolt) in some storage engines. So to maximise the chance your change will be accepted by us, use only the standard libraries, or the third-party packages we have decided to use.
For rationale, check out the post [The Case Against Third Party Libraries](http://blog.gopheracademy.com/advent-2014/case-against-3pl/).
## Always include a default case in a 'switch' statement
The lack of a `default` case in a `switch` statement can be a significant source of bugs. This is particularly true in the case of a type-assertions switch. So always include a `default` statement unless you have an explicit reason not to.
## When -- and when not -- set a channel to 'nil'
## Use defer with anonymous functions to handle complex locking
Consider a block of code like the following.
```
mu.Lock()
if foo == "quit" {
mu.Unlock()
return
} else if foo == "continue" {
if bar == "quit" {
mu.Unlock()
return
}
bar = "still going"
} else {
qux = "here at last"
mu.Unlock()
return
}
foo = "more to do"
bar = "still more to do"
mu.Unlock()
qux = "finished now"
return
```
While this is obviously contrived, complex lock control like this is sometimes required, and doesn't lend itself to `defer`. But as the code evolves, it's easy to introduce new cases, and forget to release locks. One way to address this is to use an anonymous function like so:
```
more := func() bool {
mu.Lock()
defer mu.Unlock()
if foo == "quit" {
return false
} else if foo == "continue" {
if bar == "quit" {
return false
}
bar = "still going"
} else {
qux = "here at last"
return false
}
foo = "more to do"
bar = "still more to do"
return true
}()
if more {
qux = "finished"
}
return
```
This allows us to use `defer` but ensures that if any new cases are added to the logic within the anonymous function, the lock will always be released. Another advantage of this approach is that `defer` will still run even in the event of a panic, ensuring the locks will be released even in that case.
## When to call 'panic()'
# Useful links
- [Useful techniques in Go](http://arslan.io/ten-useful-techniques-in-go)
- [Go in production](http://peter.bourgon.org/go-in-production/)
- [Principles of designing Go APIs with channels](https://inconshreveable.com/07-08-2014/principles-of-designing-go-apis-with-channels/)
- [Common mistakes in Golang](http://soryy.com/blog/2014/common-mistakes-with-go-lang/). Especially this section `Loops, Closures, and Local Variables`

282
CONTRIBUTING.md
View File

@ -1,282 +0,0 @@
Contributing to InfluxDB
========================
Bug reports
---------------
Before you file an issue, please search existing issues in case it has already been filed, or perhaps even fixed. If you file an issue, please include the following.
* Full details of your operating system (or distribution) e.g. 64-bit Ubuntu 14.04.
* The version of InfluxDB you are running
* Whether you installed it using a pre-built package, or built it from source.
* A small test case, if applicable, that demonstrates the issues.
Remember the golden rule of bug reports: **The easier you make it for us to reproduce the problem, the faster it will get fixed.**
If you have never written a bug report before, or if you want to brush up on your bug reporting skills, we recommend reading [Simon Tatham's essay "How to Report Bugs Effectively."](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
Test cases should be in the form of `curl` commands. For example:
```bash
# create database
curl -X POST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"
# create retention policy
curl -X POST http://localhost:8086/query --data-urlencode "q=CREATE RETENTION POLICY myrp ON mydb DURATION 365d REPLICATION 1 DEFAULT"
# write data
curl -X POST http://localhost:8086/write?db=mydb --data-binary "cpu,region=useast,host=server_1,service=redis value=61"
# Delete a Measurement
curl -X POST http://localhost:8086/query --data-urlencode 'db=mydb' --data-urlencode 'q=DROP MEASUREMENT cpu'
# Query the Measurement
# Bug: expected it to return no data, but data comes back.
curl -X POST http://localhost:8086/query --data-urlencode 'db=mydb' --data-urlencode 'q=SELECT * from cpu'
```
**If you don't include a clear test case like this, your issue may not be investigated, and may even be closed**. If writing the data is too difficult, please zip up your data directory and include a link to it in your bug report.
Please note that issues are *not the place to file general questions* such as "how do I use collectd with InfluxDB?" Questions of this nature should be sent to the [InfluxData Community](https://community.influxdata.com/), not filed as issues. Issues like this will be closed.
Feature requests
---------------
We really like to receive feature requests, as it helps us prioritize our work. Please be clear about your requirements, as incomplete feature requests may simply be closed if we don't understand what you would like to see added to InfluxDB.
Contributing to the source code
---------------
InfluxDB follows standard Go project structure. This means that all your Go development are done in `$GOPATH/src`. GOPATH can be any directory under which InfluxDB and all its dependencies will be cloned. For full details on the project structure, follow along below.
You should also read our [coding guide](https://github.com/influxdata/influxdb/blob/master/CODING_GUIDELINES.md), to understand better how to write code for InfluxDB.
Submitting a pull request
------------
To submit a pull request you should fork the InfluxDB repository, and make your change on a feature branch of your fork. Then generate a pull request from your branch against *master* of the InfluxDB repository. Include in your pull request details of your change -- the why *and* the how -- as well as the testing your performed. Also, be sure to run the test suite with your change in place. Changes that cause tests to fail cannot be merged.
There will usually be some back and forth as we finalize the change, but once that completes it may be merged.
To assist in review for the PR, please add the following to your pull request comment:
```md
- [ ] CHANGELOG.md updated
- [ ] Rebased/mergable
- [ ] Tests pass
- [ ] Sign [CLA](https://influxdata.com/community/cla/) (if not already signed)
```
Signing the CLA
---------------
If you are going to be contributing back to InfluxDB please take a
second to sign our CLA, which can be found
[on our website](https://influxdata.com/community/cla/).
Installing Go
-------------
InfluxDB requires Go 1.11.
At InfluxDB we find gvm, a Go version manager, useful for installing Go. For instructions
on how to install it see [the gvm page on github](https://github.com/moovweb/gvm).
After installing gvm you can install and set the default go version by
running the following:
gvm install go1.11
gvm use go1.11 --default
Installing Dep
-------------
InfluxDB uses [dep](https://github.com/golang/dep) to manage dependencies. Install it by running the following:
go get github.com/golang/dep/cmd/dep
Revision Control Systems
-------------
Go has the ability to import remote packages via revision control systems with the `go get` command. To ensure that you can retrieve any remote package, be sure to install the following rcs software to your system.
Currently the project only depends on `git` and `mercurial`.
* [Install Git](http://git-scm.com/book/en/Getting-Started-Installing-Git)
* [Install Mercurial](http://mercurial.selenic.com/wiki/Download)
Getting the source
------
Setup the project structure and fetch the repo like so:
```bash
mkdir $HOME/gocodez
export GOPATH=$HOME/gocodez
go get github.com/influxdata/influxdb
```
You can add the line `export GOPATH=$HOME/gocodez` to your bash/zsh file to be set for every shell instead of having to manually run it everytime.
Cloning a fork
-------------
If you wish to work with fork of InfluxDB, your own fork for example, you must still follow the directory structure above. But instead of cloning the main repo, instead clone your fork. Follow the steps below to work with a fork:
```bash
export GOPATH=$HOME/gocodez
mkdir -p $GOPATH/src/github.com/influxdata
cd $GOPATH/src/github.com/influxdata
git clone git@github.com:<username>/influxdb
```
Retaining the directory structure `$GOPATH/src/github.com/influxdata` is necessary so that Go imports work correctly.
Build and Test
-----
Make sure you have Go installed and the project structure as shown above. To then get the dependencies for the project, execute the following commands:
```bash
cd $GOPATH/src/github.com/influxdata/influxdb
dep ensure
```
To then build and install the binaries, run the following command.
```bash
go clean ./...
go install ./...
```
The binaries will be located in `$GOPATH/bin`. Please note that the InfluxDB binary is named `influxd`, not `influxdb`.
To set the version and commit flags during the build pass the following to the **install** command:
```bash
-ldflags="-X main.version=$VERSION -X main.branch=$BRANCH -X main.commit=$COMMIT"
```
where `$VERSION` is the version, `$BRANCH` is the branch, and `$COMMIT` is the git commit hash.
If you want to build packages, see `build.py` usage information:
```bash
python build.py --help
# Or to build a package for your current system
python build.py --package
```
To run the tests, execute the following command:
```bash
cd $GOPATH/src/github.com/influxdata/influxdb
go test -v ./...
# run tests that match some pattern
go test -run=TestDatabase . -v
# run tests and show coverage
go test -coverprofile /tmp/cover . && go tool cover -html /tmp/cover
```
To install go cover, run the following command:
```
go get golang.org/x/tools/cmd/cover
```
Generated Google Protobuf code
-----------------
Most changes to the source do not require that the generated protocol buffer code be changed. But if you need to modify the protocol buffer code, you'll first need to install the protocol buffers toolchain.
First install the [protocol buffer compiler](https://developers.google.com/protocol-buffers/
) 2.6.1 or later for your OS:
Then install the go plugins:
```bash
go get github.com/gogo/protobuf/proto
go get github.com/gogo/protobuf/protoc-gen-gogo
go get github.com/gogo/protobuf/gogoproto
```
Finally run, `go generate` after updating any `*.proto` file:
```bash
go generate ./...
```
**Troubleshooting**
If generating the protobuf code is failing for you, check each of the following:
* Ensure the protobuf library can be found. Make sure that `LD_LIBRARY_PATH` includes the directory in which the library `libprotoc.so` has been installed.
* Ensure the command `protoc-gen-gogo`, found in `GOPATH/bin`, is on your path. This can be done by adding `GOPATH/bin` to `PATH`.
Generated Go Templates
----------------------
The query engine requires optimized data structures for each data type so
instead of writing each implementation several times we use templates. _Do not
change code that ends in a `.gen.go` extension!_ Instead you must edit the
`.gen.go.tmpl` file that was used to generate it.
Once you've edited the template file, you'll need the [`tmpl`][tmpl] utility
to generate the code:
```sh
$ go get github.com/benbjohnson/tmpl
```
Then you can regenerate all templates in the project:
```sh
$ go generate ./...
```
[tmpl]: https://github.com/benbjohnson/tmpl
Pre-commit checks
-------------
We have a pre-commit hook to make sure code is formatted properly and vetted before you commit any changes. We strongly recommend using the pre-commit hook to guard against accidentally committing unformatted code. To use the pre-commit hook, run the following:
```bash
cd $GOPATH/src/github.com/influxdata/influxdb
cp .hooks/pre-commit .git/hooks/
```
In case the commit is rejected because it's not formatted you can run
the following to format the code:
```
go fmt ./...
go vet ./...
```
To install go vet, run the following command:
```
go get golang.org/x/tools/cmd/vet
```
NOTE: If you have not installed mercurial, the above command will fail. See [Revision Control Systems](#revision-control-systems) above.
For more information on `go vet`, [read the GoDoc](https://godoc.org/golang.org/x/tools/cmd/vet).
Profiling
-----
When troubleshooting problems with CPU or memory the Go toolchain can be helpful. You can start InfluxDB with CPU and memory profiling turned on. For example:
```sh
# start influx with profiling
./influxd -cpuprofile influxdcpu.prof -memprof influxdmem.prof
# run queries, writes, whatever you're testing
# Quit out of influxd and influxd.prof will then be written.
# open up pprof to examine the profiling data.
go tool pprof ./influxd influxd.prof
# once inside run "web", opens up browser with the CPU graph
# can also run "web <function name>" to zoom in. Or "list <function name>" to see specific lines
```
Note that when you pass the binary to `go tool pprof` *you must specify the path to the binary*.
If you are profiling benchmarks built with the `testing` package, you may wish
to use the [`github.com/pkg/profile`](github.com/pkg/profile) package to limit
the code being profiled:
```go
func BenchmarkSomething(b *testing.B) {
// do something intensive like fill database with data...
defer profile.Start(profile.ProfilePath("/tmp"), profile.MemProfile).Stop()
// do something that you want to profile...
}
```
Continuous Integration testing
-----
InfluxDB uses CircleCI for continuous integration testing. CircleCI executes [test.sh](https://github.com/influxdata/influxdb/blob/master/test.sh), so you may do the same on your local development environment before creating a pull request.
The `test.sh` script executes a test suite with 5 variants (standard 64 bit, 64 bit with race detection, 32 bit, TSI, go version 1.11), each executes with a different arg, 0 through 4. Unless you know differently, `./test.sh 0` is probably all you need.

61
DEPENDENCIES.md
View File

@ -1,61 +0,0 @@
- # List
- bootstrap 3.3.5 [MIT LICENSE](https://github.com/twbs/bootstrap/blob/master/LICENSE)
- collectd.org [ISC LICENSE](https://github.com/collectd/go-collectd/blob/master/LICENSE)
- github.com/BurntSushi/toml [MIT LICENSE](https://github.com/BurntSushi/toml/blob/master/COPYING)
- github.com/influxdata/roaring [APACHE LICENSE](https://github.com/RoaringBitmap/roaring/blob/master/LICENSE)
- github.com/beorn7/perks [MIT LICENSE](https://github.com/beorn7/perks/blob/master/LICENSE)
- github.com/bmizerany/pat [MIT LICENSE](https://github.com/bmizerany/pat#license)
- github.com/boltdb/bolt [MIT LICENSE](https://github.com/boltdb/bolt/blob/master/LICENSE)
- github.com/cespare/xxhash [MIT LICENSE](https://github.com/cespare/xxhash/blob/master/LICENSE.txt)
- github.com/clarkduvall/hyperloglog [MIT LICENSE](https://github.com/clarkduvall/hyperloglog/blob/master/LICENSE)
- github.com/davecgh/go-spew/spew [ISC LICENSE](https://github.com/davecgh/go-spew/blob/master/LICENSE)
- github.com/dgrijalva/jwt-go [MIT LICENSE](https://github.com/dgrijalva/jwt-go/blob/master/LICENSE)
- github.com/dgryski/go-bits [MIT LICENSE](https://github.com/dgryski/go-bits/blob/master/LICENSE)
- github.com/dgryski/go-bitstream [MIT LICENSE](https://github.com/dgryski/go-bitstream/blob/master/LICENSE)
- github.com/glycerine/go-unsnap-stream [MIT LICENSE](https://github.com/glycerine/go-unsnap-stream/blob/master/LICENSE)
- github.com/gogo/protobuf/proto [BSD LICENSE](https://github.com/gogo/protobuf/blob/master/LICENSE)
- github.com/golang/protobuf [BSD LICENSE](https://github.com/golang/protobuf/blob/master/LICENSE)
- github.com/golang/snappy [BSD LICENSE](https://github.com/golang/snappy/blob/master/LICENSE)
- github.com/google/go-cmp [BSD LICENSE](https://github.com/google/go-cmp/blob/master/LICENSE)
- github.com/influxdata/influxql [MIT LICENSE](https://github.com/influxdata/influxql/blob/master/LICENSE)
- github.com/influxdata/usage-client [MIT LICENSE](https://github.com/influxdata/usage-client/blob/master/LICENSE.txt)
- github.com/jsternberg/zap-logfmt [MIT LICENSE](https://github.com/jsternberg/zap-logfmt/blob/master/LICENSE)
- github.com/jwilder/encoding [MIT LICENSE](https://github.com/jwilder/encoding/blob/master/LICENSE)
- github.com/klauspost/pgzip [MIT LICENSE](https://github.com/klauspost/pgzip/blob/master/LICENSE)
- github.com/mattn/go-isatty [MIT LICENSE](https://github.com/mattn/go-isatty/blob/master/LICENSE)
- github.com/matttproud/golang_protobuf_extensions [APACHE LICENSE](https://github.com/matttproud/golang_protobuf_extensions/blob/master/LICENSE)
- github.com/opentracing/opentracing-go [APACHE LICENSE](https://github.com/opentracing/opentracing-go/blob/master/LICENSE)
- github.com/paulbellamy/ratecounter [MIT LICENSE](https://github.com/paulbellamy/ratecounter/blob/master/LICENSE)
- github.com/peterh/liner [MIT LICENSE](https://github.com/peterh/liner/blob/master/COPYING)
- github.com/philhofer/fwd [MIT LICENSE](https://github.com/philhofer/fwd/blob/master/LICENSE.md)
- github.com/prometheus/client_golang [MIT LICENSE](https://github.com/prometheus/client_golang/blob/master/LICENSE)
- github.com/prometheus/client_model [MIT LICENSE](https://github.com/prometheus/client_model/blob/master/LICENSE)
- github.com/prometheus/common [APACHE LICENSE](https://github.com/prometheus/common/blob/master/LICENSE)
- github.com/prometheus/procfs [APACHE LICENSE](https://github.com/prometheus/procfs/blob/master/LICENSE)
- github.com/rakyll/statik [APACHE LICENSE](https://github.com/rakyll/statik/blob/master/LICENSE)
- github.com/retailnext/hllpp [BSD LICENSE](https://github.com/retailnext/hllpp/blob/master/LICENSE)
- github.com/tinylib/msgp [MIT LICENSE](https://github.com/tinylib/msgp/blob/master/LICENSE)
- go.uber.org/atomic [MIT LICENSE](https://github.com/uber-go/atomic/blob/master/LICENSE.txt)
- go.uber.org/multierr [MIT LICENSE](https://github.com/uber-go/multierr/blob/master/LICENSE.txt)
- go.uber.org/zap [MIT LICENSE](https://github.com/uber-go/zap/blob/master/LICENSE.txt)
- golang.org/x/crypto [BSD LICENSE](https://github.com/golang/crypto/blob/master/LICENSE)
- golang.org/x/net [BSD LICENSE](https://github.com/golang/net/blob/master/LICENSE)
- golang.org/x/sys [BSD LICENSE](https://github.com/golang/sys/blob/master/LICENSE)
- golang.org/x/text [BSD LICENSE](https://github.com/golang/text/blob/master/LICENSE)
- golang.org/x/time [BSD LICENSE](https://github.com/golang/time/blob/master/LICENSE)
- jquery 2.1.4 [MIT LICENSE](https://github.com/jquery/jquery/blob/master/LICENSE.txt)
- github.com/xlab/treeprint [MIT LICENSE](https://github.com/xlab/treeprint/blob/master/LICENSE)

19
Dockerfile
View File

@ -1,19 +0,0 @@
FROM golang:1.11 as builder
RUN go get -u github.com/golang/dep/...
WORKDIR /go/src/github.com/influxdata/influxdb
COPY Gopkg.toml Gopkg.lock ./
RUN dep ensure -vendor-only
COPY . /go/src/github.com/influxdata/influxdb
RUN go install ./cmd/...
FROM debian:stretch
COPY --from=builder /go/bin/* /usr/bin/
COPY --from=builder /go/src/github.com/influxdata/influxdb/etc/config.sample.toml /etc/influxdb/influxdb.conf
EXPOSE 8086
VOLUME /var/lib/influxdb
COPY docker/entrypoint.sh /entrypoint.sh
COPY docker/init-influxdb.sh /init-influxdb.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["influxd"]

39
Dockerfile_build_ubuntu32
View File

@ -1,39 +0,0 @@
FROM ioft/i386-ubuntu:xenial
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
python-software-properties \
software-properties-common \
wget \
git \
mercurial \
make \
ruby \
ruby-dev \
rubygems \
autoconf \
libtool \
build-essential \
rpm \
zip \
python \
python-boto
RUN gem install fpm
# Install go
ENV GOPATH /root/go
ENV GO_VERSION 1.11
ENV GO_ARCH 386
RUN wget --no-verbose https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
tar -C /usr/local/ -xf /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
rm /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz
ENV PATH /usr/local/go/bin:$PATH
ENV PROJECT_DIR $GOPATH/src/github.com/influxdata/influxdb
ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
VOLUME $PROJECT_DIR
ENTRYPOINT [ "/root/go/src/github.com/influxdata/influxdb/build.py" ]

41
Dockerfile_build_ubuntu64
View File

@ -1,41 +0,0 @@
FROM ubuntu:xenial
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
python-software-properties \
software-properties-common \
wget \
git \
mercurial \
make \
ruby \
ruby-dev \
autoconf \
libtool \
build-essential \
rpm \
zip \
python \
python-boto \
asciidoc \
xmlto \
docbook-xsl
RUN gem install fpm
# Install go
ENV GOPATH /root/go
ENV GO_VERSION 1.11
ENV GO_ARCH amd64
RUN wget --no-verbose https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
tar -C /usr/local/ -xf /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
rm /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz
ENV PATH /usr/local/go/bin:$PATH
ENV PROJECT_DIR $GOPATH/src/github.com/influxdata/influxdb
ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
VOLUME $PROJECT_DIR
ENTRYPOINT [ "/root/go/src/github.com/influxdata/influxdb/build.py" ]

44
Dockerfile_build_ubuntu64_git
View File

@ -1,44 +0,0 @@
FROM ubuntu:xenial
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
python-software-properties \
software-properties-common \
wget \
git \
mercurial \
make \
ruby \
ruby-dev \
build-essential \
rpm \
zip \
python \
python-boto
RUN gem install fpm
# Setup env
ENV GOPATH /root/go
ENV PROJECT_DIR $GOPATH/src/github.com/influxdata/influxdb
ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p $PROJECT_DIR
VOLUME $PROJECT_DIR
# Install go
ENV GO_VERSION 1.11
ENV GO_ARCH amd64
RUN wget --no-verbose https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
tar -C /usr/local/ -xf /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
rm /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz
# Clone Go tip for compilation
ENV GOROOT_BOOTSTRAP /usr/local/go
RUN git clone https://go.googlesource.com/go
ENV PATH /go/bin:$PATH
# Add script for compiling go
ENV GO_CHECKOUT master
ADD ./gobuild.sh /gobuild.sh
ENTRYPOINT [ "/gobuild.sh" ]

44
Dockerfile_build_ubuntu64_go1.11
View File

@ -1,44 +0,0 @@
FROM ubuntu:xenial
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
python-software-properties \
software-properties-common \
wget \
git \
mercurial \
make \
ruby \
ruby-dev \
autoconf \
libtool \
build-essential \
rpm \
zip \
python \
python-boto \
asciidoc \
xmlto \
docbook-xsl
RUN gem install fpm
# Install go
ENV GOPATH /root/go
# TODO(edd) this needs to be updated to 1.11 when the branch is available.
ENV GO_VERSION 1.11rc1
ENV GO_ARCH amd64
RUN wget --no-verbose https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
tar -C /usr/local/ -xf /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
rm /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz
ENV PATH /usr/local/go/bin:$PATH
ENV PROJECT_DIR $GOPATH/src/github.com/influxdata/influxdb
ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
VOLUME $PROJECT_DIR
ENTRYPOINT [ "/root/go/src/github.com/influxdata/influxdb/build.py" ]

19
Dockerfile_jenkins_ubuntu32
View File

@ -1,19 +0,0 @@
FROM ioft/i386-ubuntu:xenial
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
wget \
mercurial \
gcc \
git && \
rm -rf /var/lib/apt/lists/*
# Install go
ENV GOPATH /go
ENV GO_VERSION 1.11
ENV GO_ARCH 386
RUN wget --no-verbose -q https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
tar -C /usr/local/ -xf /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz && \
mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH" && \
rm /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

12
Dockerfile_test_ubuntu32
View File

@ -1,12 +0,0 @@
FROM 32bit/ubuntu:14.04
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y python-software-properties software-properties-common git
RUN add-apt-repository ppa:evarlast/golang1.4
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-overwrite" golang-go
ENV GOPATH=/root/go
RUN mkdir -p /root/go/src/github.com/influxdata/influxdb
RUN mkdir -p /tmp/artifacts
VOLUME /root/go/src/github.com/influxdata/influxdb
VOLUME /tmp/artifacts

49
Godeps
View File

@ -1,49 +0,0 @@
collectd.org 2ce144541b8903101fb8f1483cc0497a68798122
github.com/BurntSushi/toml a368813c5e648fee92e5f6c30e3944ff9d5e8895
github.com/beorn7/perks 4c0e84591b9aa9e6dcfdf3e020114cd81f89d5f9
github.com/bmizerany/pat 6226ea591a40176dd3ff9cd8eff81ed6ca721a00
github.com/boltdb/bolt 2f1ce7a837dcb8da3ec595b1dac9d0632f0f99e8
github.com/cespare/xxhash 5c37fe3735342a2e0d01c87a907579987c8936cc
github.com/davecgh/go-spew 346938d642f2ec3594ed81d874461961cd0faa76
github.com/dgrijalva/jwt-go 06ea1031745cb8b3dab3f6a236daf2b0aa468b7e
github.com/dgryski/go-bitstream 9f22ccc24718d9643ac427c8c897ae1a01575783
github.com/glycerine/go-unsnap-stream 62a9a9eb44fd8932157b1a8ace2149eff5971af6
github.com/gogo/protobuf 1adfc126b41513cc696b209667c8656ea7aac67c
github.com/golang/protobuf b4deda0973fb4c70b50d226b1af49f3da59f5265
github.com/golang/snappy d9eb7a3d35ec988b8585d4a0068e462c27d28380
github.com/google/go-cmp 3af367b6b30c263d47e8895973edcca9a49cf029
github.com/influxdata/influxql 5e999e6a81820d4450f2a1f35c5597b569258f01
github.com/influxdata/roaring fc520f41fab6dcece280e8d4853d87a09a67f9e0
github.com/influxdata/usage-client 6d3895376368aa52a3a81d2a16e90f0f52371967
github.com/jsternberg/zap-logfmt ac4bd917e18a4548ce6e0e765b29a4e7f397b0b6
github.com/jwilder/encoding b4e1701a28efcc637d9afcca7d38e495fe909a09
github.com/klauspost/compress 6c8db69c4b49dd4df1fff66996cf556176d0b9bf
github.com/klauspost/cpuid ae7887de9fa5d2db4eaa8174a7eff2c1ac00f2da
github.com/klauspost/crc32 cb6bfca970f6908083f26f39a79009d608efd5cd
github.com/klauspost/pgzip 0bf5dcad4ada2814c3c00f996a982270bb81a506
github.com/mattn/go-isatty 6ca4dbf54d38eea1a992b3c722a76a5d1c4cb25c
github.com/matttproud/golang_protobuf_extensions 3247c84500bff8d9fb6d579d800f20b3e091582c
github.com/mschoch/smat 90eadee771aeab36e8bf796039b8c261bebebe4f
github.com/opentracing/opentracing-go 328fceb7548c744337cd010914152b74eaf4c4ab
github.com/paulbellamy/ratecounter 524851a93235ac051e3540563ed7909357fe24ab
github.com/peterh/liner 6106ee4fe3e8435f18cd10e34557e5e50f0e792a
github.com/philhofer/fwd bb6d471dc95d4fe11e432687f8b70ff496cf3136
github.com/prometheus/client_golang 661e31bf844dfca9aeba15f27ea8aa0d485ad212
github.com/prometheus/client_model 99fa1f4be8e564e8a6b613da7fa6f46c9edafc6c
github.com/prometheus/common e4aa40a9169a88835b849a6efb71e05dc04b88f0
github.com/prometheus/procfs 54d17b57dd7d4a3aa092476596b3f8a933bde349
github.com/retailnext/hllpp 101a6d2f8b52abfc409ac188958e7e7be0116331
github.com/tinylib/msgp b2b6a672cf1e5b90748f79b8b81fc8c5cf0571a1
github.com/willf/bitset d860f346b89450988a379d7d705e83c58d1ea227
github.com/xlab/treeprint f3a15cfd24bf976c724324cb6846a8b54b88b639
go.uber.org/atomic 8474b86a5a6f79c443ce4b2992817ff32cf208b8
go.uber.org/multierr 3c4937480c32f4c13a875a1829af76c98ca3d40a
go.uber.org/zap 35aad584952c3e7020db7b839f6b102de6271f89
golang.org/x/crypto c3a3ad6d03f7a915c0f7e194b7152974bb73d287
golang.org/x/net 92b859f39abd2d91a854c9f9c4621b2f5054a92d
golang.org/x/sync 1d60e4601c6fd243af51cc01ddf169918a5407ca
golang.org/x/sys d8e400bc7db4870d786864138af681469693d18c
golang.org/x/text f21a4dfb5e38f5895301dc265a8def02365cc3d0
golang.org/x/time 26559e0f760e39c24d730d3224364aef164ee23f
google.golang.org/genproto ff3583edef7de132f219f0efc00e097cabcc0ec0
google.golang.org/grpc 168a6198bcb0ef175f7dacec0b8691fc141dc9b8

1140
Gopkg.lock generated
View File

File diff suppressed because it is too large Load Diff

77
Gopkg.toml
View File

@ -1,77 +0,0 @@
required = ["github.com/influxdata/platform/pkg/data/gen"]
[[constraint]]
name = "collectd.org"
version = "0.3.0"
[[constraint]]
name = "github.com/BurntSushi/toml"
branch = "master"
[[constraint]]
name = "github.com/influxdata/roaring"
branch = "master"
[[constraint]]
name = "github.com/boltdb/bolt"
version = "1.3.1"
[[constraint]]
name = "github.com/cespare/xxhash"
version = "1.0.0"
[[constraint]]
name = "github.com/dgrijalva/jwt-go"
version = "3.2.0"
[[constraint]]
name = "github.com/gogo/protobuf"
version = "1.0.0"
[[constraint]]
name = "github.com/golang/snappy"
revision = "d9eb7a3d35ec988b8585d4a0068e462c27d28380"
[[constraint]]
name = "github.com/mattn/go-isatty"
branch = "master"
[[constraint]]
name = "github.com/opentracing/opentracing-go"
branch = "master"
[[constraint]]
name = "github.com/paulbellamy/ratecounter"
version = "0.2.0"
[[constraint]]
name = "github.com/retailnext/hllpp"
branch = "master"
[[constraint]]
name = "github.com/tinylib/msgp"
version = "1.0.2"
[[constraint]]
name = "go.uber.org/zap"
version = "1.7.1"
[[constraint]]
name = "github.com/jsternberg/zap-logfmt"
version = "1.0.0"
[[constraint]]
name = "github.com/prometheus/client_golang"
revision = "661e31bf844dfca9aeba15f27ea8aa0d485ad212"
[[constraint]]
name = "github.com/klauspost/pgzip"
version = "1.1.0"
[prune]
go-tests = true
unused-packages = true
[[constraint]]
name = "github.com/influxdata/flux"
version = "0.12.0"

95
Jenkinsfile vendored
View File

@ -1,95 +0,0 @@
readTrusted 'Dockerfile_jenkins_ubuntu32'
pipeline {
agent none
stages {
stage('Update changelog') {
agent any
when {
anyOf {
expression { BRANCH_NAME ==~ /^1(.\d+)*$/ }
}
}
steps {
sh "docker pull jsternberg/changelog"
withDockerContainer(image: "jsternberg/changelog") {
withCredentials(
[[$class: "UsernamePasswordMultiBinding",
credentialsId: "hercules-username-password",
usernameVariable: "GITHUB_USER",
passwordVariable: "GITHUB_TOKEN"]]) {
script {
if (env.GIT_PREVIOUS_SUCCESSFUL_COMMIT) {
sh "git changelog ${env.GIT_PREVIOUS_SUCCESSFUL_COMMIT}"
} else {
sh "git changelog"
}
}
}
}
sshagent(credentials: ['jenkins-hercules-ssh']) {
sh """
set -e
if ! git diff --quiet; then
git config remote.origin.pushurl git@github.com:influxdata/influxdb.git
git commit -am 'Update changelog'
git push origin HEAD:${BRANCH_NAME}
fi
"""
}
}
}
stage('64bit') {
agent {
docker {
image 'golang:1.11'
}
}
steps {
sh """
mkdir -p /go/src/github.com/influxdata
cp -a $WORKSPACE /go/src/github.com/influxdata/influxdb
cd /go/src/github.com/influxdata/influxdb
go get github.com/golang/dep/cmd/dep
dep ensure -vendor-only
"""
sh """
cd /go/src/github.com/influxdata/influxdb
go test -parallel=1 ./...
"""
}
}
stage('32bit') {
agent {
dockerfile {
filename 'Dockerfile_jenkins_ubuntu32'
}
}
steps {
sh """
mkdir -p /go/src/github.com/influxdata
cp -a $WORKSPACE /go/src/github.com/influxdata/influxdb
cd /go/src/github.com/influxdata/influxdb
go get github.com/golang/dep/cmd/dep
dep ensure -vendor-only
"""
sh """
cd /go/src/github.com/influxdata/influxdb
go test -parallel=1 ./...
"""
}
}
}
}

20
LICENSE
View File

@ -1,20 +0,0 @@
The MIT License (MIT)
Copyright (c) 2013-2018 InfluxData Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

190
QUERIES.md
View File

@ -1,190 +0,0 @@
The top level name is called a measurement. These names can contain any characters. Then there are field names, field values, tag keys and tag values, which can also contain any characters. However, if the measurement, field, or tag contains any character other than [A-Z,a-z,0-9,_], or if it starts with a digit, it must be double-quoted. Therefore anywhere a measurement name, field key, or tag key appears it should be wrapped in double quotes.
# Databases & retention policies
```sql
-- create a database
CREATE DATABASE <name>
-- create a retention policy
CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
-- alter retention policy
ALTER RETENTION POLICY <rp-name> ON <db-name> (DURATION <duration> | REPLICATION <n> | DEFAULT)+
-- drop a database
DROP DATABASE <name>
-- drop a retention policy
DROP RETENTION POLICY <rp-name> ON <db-name>
```
where `<duration>` is either `INF` for infinite retention, or an integer followed by the desired unit of time: u,ms,s,m,h,d,w for microseconds, milliseconds, seconds, minutes, hours, days, or weeks, respectively. `<replication>` must be an integer.
If present, `DEFAULT` sets the retention policy as the default retention policy for writes and reads.
# Users and permissions
```sql
-- create user
CREATE USER <name> WITH PASSWORD '<password>'
-- grant privilege on a database
GRANT <privilege> ON <db> TO <user>
-- grant cluster admin privileges
GRANT ALL [PRIVILEGES] TO <user>
-- revoke privilege
REVOKE <privilege> ON <db> FROM <user>
-- revoke all privileges for a DB
REVOKE ALL [PRIVILEGES] ON <db> FROM <user>
-- revoke all privileges including cluster admin
REVOKE ALL [PRIVILEGES] FROM <user>
-- combine db creation with privilege assignment (user must already exist)
CREATE DATABASE <name> GRANT <privilege> TO <user>
CREATE DATABASE <name> REVOKE <privilege> FROM <user>
-- delete a user
DROP USER <name>
```
where `<privilege> := READ | WRITE | All `.
Authentication must be enabled in the influxdb.conf file for user permissions to be in effect.
By default, newly created users have no privileges to any databases.
Cluster administration privileges automatically grant full read and write permissions to all databases, regardless of subsequent database-specific privilege revocation statements.
# Select
```sql
SELECT * FROM just_my_type
```
## Group By
```sql
SELECT mean(value) from cpu WHERE host = 'serverA' AND time > now() - 4h GROUP BY time(5m)
SELECT mean(value) from cpu WHERE time > now() - 4h GROUP BY time(5m), region
```
# Delete
```sql
DELETE FROM "cpu"
DELETE FROM "cpu" WHERE time < '2000-01-01T00:00:00Z'
DELETE WHERE time < '2000-01-01T00:00:00Z'
```
# Series
## Destroy
```sql
DROP MEASUREMENT <name>
DROP MEASUREMENT cpu WHERE region = 'uswest'
```
## Show
Show series queries are for pulling out individual series from measurement names and tag data. They're useful for discovery.
```sql
-- show all databases
SHOW DATABASES
-- show measurement names
SHOW MEASUREMENTS
SHOW MEASUREMENTS LIMIT 15
SHOW MEASUREMENTS LIMIT 10 OFFSET 40
SHOW MEASUREMENTS WHERE service = 'redis'
-- LIMIT and OFFSET can be applied to any of the SHOW type queries
-- show all series across all measurements/tagsets
SHOW SERIES
-- get a show of all series for any measurements where tag key region = tak value 'uswest'
SHOW SERIES WHERE region = 'uswest'
SHOW SERIES FROM cpu_load WHERE region = 'uswest' LIMIT 10
-- returns the 100 - 109 rows in the result. In the case of SHOW SERIES, which returns
-- series split into measurements. Each series counts as a row. So you could see only a
-- single measurement returned, but 10 series within it.
SHOW SERIES FROM cpu_load WHERE region = 'uswest' LIMIT 10 OFFSET 100
-- show all retention policies on a database
SHOW RETENTION POLICIES ON mydb
-- get a show of all tag keys across all measurements
SHOW TAG KEYS
-- show all the tag keys for a given measurement
SHOW TAG KEYS FROM cpu
SHOW TAG KEYS FROM temperature, wind_speed
-- show all the tag values. note that a single WHERE TAG KEY = '...' clause is required
SHOW TAG VALUES WITH TAG KEY = 'region'
SHOW TAG VALUES FROM cpu WHERE region = 'uswest' WITH TAG KEY = 'host'
-- and you can do stuff against fields
SHOW FIELD KEYS FROM cpu
-- but you can't do this
SHOW FIELD VALUES
-- we don't index field values, so this query should be invalid.
-- show all users
SHOW USERS
```
Note that `FROM` and `WHERE` are optional clauses in most of the show series queries.
And the show series output looks like this:
```json
[
{
"name": "cpu",
"columns": ["id", "region", "host"],
"values": [
1, "uswest", "servera",
2, "uswest", "serverb"
]
},
{
"name": "reponse_time",
"columns": ["id", "application", "host"],
"values": [
3, "myRailsApp", "servera"
]
}
]
```
# Continuous Queries
Continuous queries are going to be inspired by MySQL `TRIGGER` syntax:
http://dev.mysql.com/doc/refman/5.0/en/trigger-syntax.html
Instead of having automatically-assigned ids, named continuous queries allows for some level of duplication prevention,
particularly in the case where creation is scripted.
## Create
CREATE CONTINUOUS QUERY <name> AS SELECT ... FROM ...
## Destroy
DROP CONTINUOUS QUERY <name>
## List
SHOW CONTINUOUS QUERIES

79
README.md
View File

@ -1,79 +0,0 @@
# InfluxDB [![Circle CI](https://circleci.com/gh/influxdata/influxdb/tree/master.svg?style=svg)](https://circleci.com/gh/influxdata/influxdb/tree/master) [![Go Report Card](https://goreportcard.com/badge/github.com/influxdata/influxdb)](https://goreportcard.com/report/github.com/influxdata/influxdb) [![Docker pulls](https://img.shields.io/docker/pulls/library/influxdb.svg)](https://hub.docker.com/_/influxdb/)
# ATTENTION:
Around January 11th, 2019, `master` on this repository will become InfluxDB 2.0 code. The content of `infludata/platform` will be moved to this repository. If you rely on `master`, you should update your dependencies to track the `1.7` branch.
## An Open-Source Time Series Database
InfluxDB is an open source **time series database** with
**no external dependencies**. It's useful for recording metrics,
events, and performing analytics.
## Features
* Built-in [HTTP API](https://docs.influxdata.com/influxdb/latest/guides/writing_data/) so you don't have to write any server side code to get up and running.
* Data can be tagged, allowing very flexible querying.
* SQL-like query language.
* Simple to install and manage, and fast to get data in and out.
* It aims to answer queries in real-time. That means every data point is
indexed as it comes in and is immediately available in queries that
should return in < 100ms.
## Installation
We recommend installing InfluxDB using one of the [pre-built packages](https://influxdata.com/downloads/#influxdb). Then start InfluxDB using:
* `service influxdb start` if you have installed InfluxDB using an official Debian or RPM package.
* `systemctl start influxdb` if you have installed InfluxDB using an official Debian or RPM package, and are running a distro with `systemd`. For example, Ubuntu 15 or later.
* `$GOPATH/bin/influxd` if you have built InfluxDB from source.
## Getting Started
### Create your first database
```
curl -XPOST "http://localhost:8086/query" --data-urlencode "q=CREATE DATABASE mydb"
```
### Insert some data
```
curl -XPOST "http://localhost:8086/write?db=mydb" \
-d 'cpu,host=server01,region=uswest load=42 1434055562000000000'
curl -XPOST "http://localhost:8086/write?db=mydb" \
-d 'cpu,host=server02,region=uswest load=78 1434055562000000000'
curl -XPOST "http://localhost:8086/write?db=mydb" \
-d 'cpu,host=server03,region=useast load=15.4 1434055562000000000'
```
### Query for the data
```JSON
curl -G "http://localhost:8086/query?pretty=true" --data-urlencode "db=mydb" \
--data-urlencode "q=SELECT * FROM cpu WHERE host='server01' AND time < now() - 1d"
```
### Analyze the data
```JSON
curl -G "http://localhost:8086/query?pretty=true" --data-urlencode "db=mydb" \
--data-urlencode "q=SELECT mean(load) FROM cpu WHERE region='uswest'"
```
## Documentation
* Read more about the [design goals and motivations of the project](https://docs.influxdata.com/influxdb/latest/).
* Follow the [getting started guide](https://docs.influxdata.com/influxdb/latest/introduction/getting_started/) to learn the basics in just a few minutes.
* Learn more about [InfluxDB's key concepts](https://docs.influxdata.com/influxdb/latest/concepts/key_concepts/).
## Contributing
If you're feeling adventurous and want to contribute to InfluxDB, see our [contributing doc](https://github.com/influxdata/influxdb/blob/master/CONTRIBUTING.md) for info on how to make feature requests, build from source, and run tests.
## Licensing
See [LICENSE](./LICENSE) and [DEPENDENCIES](./DEPENDENCIES).
## Looking for Support?
InfluxDB offers a number of services to help your project succeed. We offer Developer Support for organizations in active development, Managed Hosting to make it easy to move into production, and Enterprise Support for companies requiring the best response times, SLAs, and technical fixes. Visit our [support page](https://influxdata.com/services/) or contact [sales@influxdb.com](mailto:sales@influxdb.com) to learn how we can best help you succeed.

9
TODO.md
View File

@ -1,9 +0,0 @@
# TODO
## v2
TODO list for v2. Here is a list of things we want to add to v1, but can't because they would be a breaking change.
- [#1834](https://github.com/influxdata/influxdb/issues/1834): Disallow using time as a tag key or field key.
- [#2124](https://github.com/influxdata/influxdb/issues/2124): Prohibit writes with precision, but without an explicit timestamp.
- [#4461](https://github.com/influxdata/influxdb/issues/4461): Change default time boundaries.

251
_tools/tmpl/main.go
View File

@ -1,251 +0,0 @@
package main
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"go/format"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"strings"
"text/template"
)
const Ext = ".tmpl"
type pathSpec struct {
in, out string
}
func (p *pathSpec) String() string { return p.in + " → " + p.out }
func (p *pathSpec) IsGoFile() bool { return filepath.Ext(p.out) == ".go" }
func parsePath(path string) (string, string) {
p := strings.IndexByte(path, '=')
if p == -1 {
if filepath.Ext(path) != Ext {
errExit("template file '%s' must have .tmpl extension", path)
}
return path, path[:len(path)-len(Ext)]
}
return path[:p], path[p+1:]
}
type data struct {
In interface{}
D listValue
}
func errExit(format string, a ...interface{}) {
fmt.Fprintf(os.Stderr, format, a...)
fmt.Fprintln(os.Stderr)
os.Exit(1)
}
type listValue map[string]string
func (l listValue) String() string {
res := make([]string, 0, len(l))
for k, v := range l {
res = append(res, fmt.Sprintf("%s=%s", k, v))
}
return strings.Join(res, ", ")
}
func (l listValue) Set(v string) error {
nv := strings.Split(v, "=")
if len(nv) != 2 {
return fmt.Errorf("expected NAME=VALUE, got %s", v)
}
l[nv[0]] = nv[1]
return nil
}
func main() {
var (
dataArg = flag.String("data", "", "input JSON data")
gi = flag.Bool("i", false, "run goimports")
in = &data{D: make(listValue)}
)
flag.Var(&in.D, "d", "-d NAME=VALUE")
flag.Parse()
if *dataArg == "" {
errExit("data option is required")
}
if *gi {
if _, err := exec.LookPath("goimports"); err != nil {
errExit("failed to find goimports: %s", err.Error())
}
formatter = formatSource
} else {
formatter = format.Source
}
paths := flag.Args()
if len(paths) == 0 {
errExit("no tmpl files specified")
}
specs := make([]pathSpec, len(paths))
for i, p := range paths {
in, out := parsePath(p)
specs[i] = pathSpec{in: in, out: out}
}
in.In = readData(*dataArg)
process(in, specs)
}
func mustReadAll(path string) []byte {
data, err := ioutil.ReadFile(path)
if err != nil {
errExit(err.Error())
}
return data
}
func readData(path string) interface{} {
data := mustReadAll(path)
var v interface{}
if err := json.Unmarshal(StripComments(data), &v); err != nil {
errExit("invalid JSON data: %s", err.Error())
}
return v
}
func fileMode(path string) os.FileMode {
stat, err := os.Stat(path)
if err != nil {
errExit(err.Error())
}
return stat.Mode()
}
var funcs = template.FuncMap{
"lower": strings.ToLower,
"upper": strings.ToUpper,
}
func process(data interface{}, specs []pathSpec) {
for _, spec := range specs {
var (
t *template.Template
err error
)
t, err = template.New("gen").Funcs(funcs).Parse(string(mustReadAll(spec.in)))
if err != nil {
errExit("error processing template '%s': %s", spec.in, err.Error())
}
var buf bytes.Buffer
if spec.IsGoFile() {
// preamble
fmt.Fprintf(&buf, "// Code generated by %s. DO NOT EDIT.\n", spec.in)
fmt.Fprintln(&buf)
}
err = t.Execute(&buf, data)
if err != nil {
errExit("error executing template '%s': %s", spec.in, err.Error())
}
generated := buf.Bytes()
if spec.IsGoFile() {
generated, err = formatter(generated)
if err != nil {
errExit("error formatting '%s': %s", spec.in, err.Error())
}
}
ioutil.WriteFile(spec.out, generated, fileMode(spec.in))
}
}
var (
formatter func([]byte) ([]byte, error)
)
func formatSource(in []byte) ([]byte, error) {
r := bytes.NewReader(in)
cmd := exec.Command("goimports")
cmd.Stdin = r
out, err := cmd.Output()
if err != nil {
if ee, ok := err.(*exec.ExitError); ok {
return nil, fmt.Errorf("error running goimports: %s", string(ee.Stderr))
}
return nil, fmt.Errorf("error running goimports: %s", string(out))
}
return out, nil
}
func StripComments(raw []byte) []byte {
var (
quoted, esc bool
comment bool
)
buf := bytes.Buffer{}
for i := 0; i < len(raw); i++ {
b := raw[i]
if comment {
switch b {
case '/':
comment = false
j := bytes.IndexByte(raw[i+1:], '\n')
if j == -1 {
i = len(raw)
} else {
i += j // keep new line
}
case '*':
j := bytes.Index(raw[i+1:], []byte("*/"))
if j == -1 {
i = len(raw)
} else {
i += j + 2
comment = false
}
}
continue
}
if esc {
esc = false
continue
}
if b == '\\' && quoted {
esc = true
continue
}
if b == '"' || b == '\'' {
quoted = !quoted
}
if b == '/' && !quoted {
comment = true
continue
}
buf.WriteByte(b)
}
if quoted || esc || comment {
// unexpected state, so return raw bytes
return raw
}
return buf.Bytes()
}

57
_tools/tmpl/main_test.go
View File

@ -1,57 +0,0 @@
package main
import (
"testing"
)
func TestStripComments(t *testing.T) {
tests := []struct {
name string
in string
exp string
}{
{name: "none", in: `[1,2,3]`, exp: `[1,2,3]`},
{name: "single-line, line comment at end", in: `[1,2,3] // foo bar`, exp: `[1,2,3] `},
{name: "single-line, block comment at end", in: `[1,2,3] /* foo bar */ `, exp: `[1,2,3] `},
{name: "single-line, block comment at end", in: `[1,2,3] /* /* // */`, exp: `[1,2,3] `},
{name: "single-line, block comment in middle", in: `[1,/* foo bar */2,3]`, exp: `[1,2,3]`},
{name: "single-line, block comment in string", in: `[1,"/* foo bar */"]`, exp: `[1,"/* foo bar */"]`},
{name: "single-line, malformed block comment", in: `[1,2,/*]`, exp: `[1,2,/*]`},
{name: "single-line, malformed JSON", in: `[1,2,/]`, exp: `[1,2,/]`},
{
name: "multi-line",
in: `[
1,
2,
3
]`,
exp: `[
1,
2,
3
]`,
},
{
name: "multi-line, multiple line comments",
in: `[ // foo
1, // bar
2,
3
] // fit`,
exp: `[
1,
2,
3
] `,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got := string(StripComments([]byte(test.in)))
if got != test.exp {
t.Errorf("got:\n%s\nexp:\n%s", got, test.exp)
}
})
}
}

37
appveyor.yml
View File

@ -1,37 +0,0 @@
version: 0.{build}
pull_requests:
do_not_increment_build_number: true
branches:
only:
- master
os: Windows Server 2012 R2
# Custom clone folder (variables are not expanded here).
clone_folder: c:\gopath\src\github.com\influxdata\influxdb
# Environment variables
environment:
GOROOT: C:\go110
GOPATH: C:\gopath
# Scripts that run after cloning repository
install:
- set PATH=%GOROOT%\bin;%GOPATH%\bin;%PATH%
- rmdir c:\go /s /q
- echo %PATH%
- echo %GOPATH%
- cd C:\gopath\src\github.com\influxdata\influxdb
- go version
- go env
- go get github.com/golang/dep/cmd/dep
- cd C:\gopath\src\github.com\influxdata\influxdb
- dep ensure -vendor-only
# To run your custom scripts instead of automatic MSBuild
build_script:
- go get -t -v ./...
- go test -timeout 15m -v ./...
# To disable deployment
deploy: off

991
build.py
View File

@ -1,991 +0,0 @@
#!/usr/bin/python2.7 -u
import sys
import os
import subprocess
import time
from datetime import datetime
import shutil
import tempfile
import hashlib
import re
import logging
import argparse
################
#### InfluxDB Variables
################
# Packaging variables
PACKAGE_NAME = "influxdb"
INSTALL_ROOT_DIR = "/usr/bin"
LOG_DIR = "/var/log/influxdb"
DATA_DIR = "/var/lib/influxdb"
SCRIPT_DIR = "/usr/lib/influxdb/scripts"
CONFIG_DIR = "/etc/influxdb"
LOGROTATE_DIR = "/etc/logrotate.d"
MAN_DIR = "/usr/share/man"
INIT_SCRIPT = "scripts/init.sh"
SYSTEMD_SCRIPT = "scripts/influxdb.service"
PREINST_SCRIPT = "scripts/pre-install.sh"
POSTINST_SCRIPT = "scripts/post-install.sh"
POSTUNINST_SCRIPT = "scripts/post-uninstall.sh"
LOGROTATE_SCRIPT = "scripts/logrotate"
DEFAULT_CONFIG = "etc/config.sample.toml"
# Default AWS S3 bucket for uploads
DEFAULT_BUCKET = "dl.influxdata.com/influxdb/artifacts"
CONFIGURATION_FILES = [
CONFIG_DIR + '/influxdb.conf',
LOGROTATE_DIR + '/influxdb',
]
PACKAGE_LICENSE = "MIT"
PACKAGE_URL = "https://github.com/influxdata/influxdb"
MAINTAINER = "support@influxdb.com"
VENDOR = "InfluxData"
DESCRIPTION = "Distributed time-series database."
prereqs = [ 'git', 'go' ]
go_vet_command = "go vet ./..."
optional_prereqs = [ 'fpm', 'rpmbuild', 'gpg' ]
fpm_common_args = "-f -s dir --log error \
--vendor {} \
--url {} \
--after-install {} \
--before-install {} \
--after-remove {} \
--license {} \
--maintainer {} \
--directories {} \
--directories {} \
--directories {} \
--description \"{}\"".format(
VENDOR,
PACKAGE_URL,
POSTINST_SCRIPT,
PREINST_SCRIPT,
POSTUNINST_SCRIPT,
PACKAGE_LICENSE,
MAINTAINER,
LOG_DIR,
DATA_DIR,
MAN_DIR,
DESCRIPTION)
for f in CONFIGURATION_FILES:
fpm_common_args += " --config-files {}".format(f)
targets = {
'influx' : './cmd/influx',
'influxd' : './cmd/influxd',
'influx_stress' : './cmd/influx_stress',
'influx_inspect' : './cmd/influx_inspect',
'influx_tsm' : './cmd/influx_tsm',
}
supported_builds = {
'darwin': [ "amd64" ],
'windows': [ "amd64" ],
'linux': [ "amd64", "i386", "armhf", "arm64", "armel", "static_i386", "static_amd64" ]
}
supported_packages = {
"darwin": [ "tar" ],
"linux": [ "deb", "rpm", "tar" ],
"windows": [ "zip" ],
}
################
#### InfluxDB Functions
################
def print_banner():
logging.info("""
___ __ _ ___ ___
|_ _|_ _ / _| |_ ___ _| \\| _ )
| || ' \\| _| | || \\ \\ / |) | _ \\
|___|_||_|_| |_|\\_,_/_\\_\\___/|___/
Build Script
""")
def create_package_fs(build_root):
"""Create a filesystem structure to mimic the package filesystem.
"""
logging.debug("Creating package filesystem at location: {}".format(build_root))
# Using [1:] for the path names due to them being absolute
# (will overwrite previous paths, per 'os.path.join' documentation)
dirs = [ INSTALL_ROOT_DIR[1:],
LOG_DIR[1:],
DATA_DIR[1:],
SCRIPT_DIR[1:],
CONFIG_DIR[1:],
LOGROTATE_DIR[1:],
MAN_DIR[1:] ]
for d in dirs:
os.makedirs(os.path.join(build_root, d))
os.chmod(os.path.join(build_root, d), 0o755)
def package_scripts(build_root, config_only=False, windows=False):
"""Copy the necessary scripts and configuration files to the package
filesystem.
"""
if config_only:
logging.debug("Copying configuration to build directory.")
shutil.copyfile(DEFAULT_CONFIG, os.path.join(build_root, "influxdb.conf"))
os.chmod(os.path.join(build_root, "influxdb.conf"), 0o644)
else:
logging.debug("Copying scripts and sample configuration to build directory.")
shutil.copyfile(INIT_SCRIPT, os.path.join(build_root, SCRIPT_DIR[1:], INIT_SCRIPT.split('/')[1]))
os.chmod(os.path.join(build_root, SCRIPT_DIR[1:], INIT_SCRIPT.split('/')[1]), 0o644)
shutil.copyfile(SYSTEMD_SCRIPT, os.path.join(build_root, SCRIPT_DIR[1:], SYSTEMD_SCRIPT.split('/')[1]))
os.chmod(os.path.join(build_root, SCRIPT_DIR[1:], SYSTEMD_SCRIPT.split('/')[1]), 0o644)
shutil.copyfile(LOGROTATE_SCRIPT, os.path.join(build_root, LOGROTATE_DIR[1:], "influxdb"))
os.chmod(os.path.join(build_root, LOGROTATE_DIR[1:], "influxdb"), 0o644)
shutil.copyfile(DEFAULT_CONFIG, os.path.join(build_root, CONFIG_DIR[1:], "influxdb.conf"))
os.chmod(os.path.join(build_root, CONFIG_DIR[1:], "influxdb.conf"), 0o644)
def package_man_files(build_root):
"""Copy and gzip man pages to the package filesystem."""
logging.debug("Installing man pages.")
run("make -C man/ clean install DESTDIR={}/usr".format(build_root))
for path, dir, files in os.walk(os.path.join(build_root, MAN_DIR[1:])):
for f in files:
run("gzip -9n {}".format(os.path.join(path, f)))
def go_get(branch, update=False, no_uncommitted=False):
"""Retrieve build dependencies or restore pinned dependencies.
"""
if local_changes() and no_uncommitted:
logging.error("There are uncommitted changes in the current directory.")
return False
if not check_path_for("dep"):
logging.info("Downloading `dep`...")
get_command = "go get github.com/golang/dep/cmd/dep"
run(get_command)
logging.info("Retrieving dependencies with `dep`...")
sys.stdout.flush()
run("{}/bin/dep ensure -v -vendor-only".format(os.environ.get("GOPATH")))
return True
def run_tests(race, parallel, timeout, no_vet, junit=False):
"""Run the Go test suite on binary output.
"""
logging.info("Starting tests...")
if race:
logging.info("Race is enabled.")
if parallel is not None:
logging.info("Using parallel: {}".format(parallel))
if timeout is not None:
logging.info("Using timeout: {}".format(timeout))
out = run("go fmt ./...")
if len(out) > 0:
logging.error("Code not formatted. Please use 'go fmt ./...' to fix formatting errors.")
logging.error("{}".format(out))
return False
if not no_vet:
logging.info("Running 'go vet'...")
out = run(go_vet_command)
if len(out) > 0:
logging.error("Go vet failed. Please run 'go vet ./...' and fix any errors.")
logging.error("{}".format(out))
return False
else:
logging.info("Skipping 'go vet' call...")
test_command = "go test -v"
if race:
test_command += " -race"
if parallel is not None:
test_command += " -parallel {}".format(parallel)
if timeout is not None:
test_command += " -timeout {}".format(timeout)
test_command += " ./..."
if junit:
logging.info("Retrieving go-junit-report...")
run("go get github.com/jstemmer/go-junit-report")
# Retrieve the output from this command.
logging.info("Running tests...")
logging.debug("{}".format(test_command))
proc = subprocess.Popen(test_command.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output, unused_err = proc.communicate()
output = output.decode('utf-8').strip()
# Process the output through go-junit-report.
with open('test-results.xml', 'w') as f:
logging.debug("{}".format("go-junit-report"))
junit_proc = subprocess.Popen(["go-junit-report"], stdin=subprocess.PIPE, stdout=f, stderr=subprocess.PIPE)
unused_output, err = junit_proc.communicate(output.encode('ascii', 'ignore'))
if junit_proc.returncode != 0:
logging.error("Command '{}' failed with error: {}".format("go-junit-report", err))
sys.exit(1)
if proc.returncode != 0:
logging.error("Command '{}' failed with error: {}".format(test_command, output.encode('ascii', 'ignore')))
sys.exit(1)
else:
logging.info("Running tests...")
output = run(test_command)
logging.debug("Test output:\n{}".format(out.encode('ascii', 'ignore')))
return True
################
#### All InfluxDB-specific content above this line
################
def run(command, allow_failure=False, shell=False):
"""Run shell command (convenience wrapper around subprocess).
"""
out = None
logging.debug("{}".format(command))
try:
if shell:
out = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=shell)
else:
out = subprocess.check_output(command.split(), stderr=subprocess.STDOUT)
out = out.decode('utf-8').strip()
# logging.debug("Command output: {}".format(out))
except subprocess.CalledProcessError as e:
if allow_failure:
logging.warn("Command '{}' failed with error: {}".format(command, e.output))
return None
else:
logging.error("Command '{}' failed with error: {}".format(command, e.output))
sys.exit(1)
except OSError as e:
if allow_failure:
logging.warn("Command '{}' failed with error: {}".format(command, e))
return out
else:
logging.error("Command '{}' failed with error: {}".format(command, e))
sys.exit(1)
else:
return out
def create_temp_dir(prefix = None):
""" Create temporary directory with optional prefix.
"""
if prefix is None:
return tempfile.mkdtemp(prefix="{}-build.".format(PACKAGE_NAME))
else:
return tempfile.mkdtemp(prefix=prefix)
def increment_minor_version(version):
"""Return the version with the minor version incremented and patch
version set to zero.
"""
ver_list = version.split('.')
if len(ver_list) != 3:
logging.warn("Could not determine how to increment version '{}', will just use provided version.".format(version))
return version
ver_list[1] = str(int(ver_list[1]) + 1)
ver_list[2] = str(0)
inc_version = '.'.join(ver_list)
logging.debug("Incremented version from '{}' to '{}'.".format(version, inc_version))
return inc_version
def get_current_version_tag():
"""Retrieve the raw git version tag.
"""
version = run("git describe --always --tags --abbrev=0")
return version
def get_current_version():
"""Parse version information from git tag output.
"""
version_tag = get_current_version_tag()
# Remove leading 'v'
if version_tag[0] == 'v':
version_tag = version_tag[1:]
# Replace any '-'/'_' with '~'
if '-' in version_tag:
version_tag = version_tag.replace("-","~")
if '_' in version_tag:
version_tag = version_tag.replace("_","~")
return version_tag
def get_current_commit(short=False):
"""Retrieve the current git commit.
"""
command = None
if short:
command = "git log --pretty=format:'%h' -n 1"
else:
command = "git rev-parse HEAD"
out = run(command)
return out.strip('\'\n\r ')
def get_current_branch():
"""Retrieve the current git branch.
"""
command = "git rev-parse --abbrev-ref HEAD"
out = run(command)
return out.strip()
def local_changes():
"""Return True if there are local un-committed changes.
"""
output = run("git diff-files --ignore-submodules --").strip()
if len(output) > 0:
return True
return False
def get_system_arch():
"""Retrieve current system architecture.
"""
arch = os.uname()[4]
if arch == "x86_64":
arch = "amd64"
elif arch == "386":
arch = "i386"
elif arch == "aarch64":
arch = "arm64"
elif 'arm' in arch:
# Prevent uname from reporting full ARM arch (eg 'armv7l')
arch = "arm"
return arch
def get_system_platform():
"""Retrieve current system platform.
"""
if sys.platform.startswith("linux"):
return "linux"
else:
return sys.platform
def get_go_version():
"""Retrieve version information for Go.
"""
out = run("go version")
matches = re.search('go version go(\S+)', out)
if matches is not None:
return matches.groups()[0].strip()
return None
def check_path_for(b):
"""Check the the user's path for the provided binary.
"""
def is_exe(fpath):
return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
for path in os.environ["PATH"].split(os.pathsep):
path = path.strip('"')
full_path = os.path.join(path, b)
if os.path.isfile(full_path) and os.access(full_path, os.X_OK):
return full_path
def check_environ(build_dir = None):
"""Check environment for common Go variables.
"""
logging.info("Checking environment...")
for v in [ "GOPATH", "GOBIN", "GOROOT" ]:
logging.debug("Using '{}' for {}".format(os.environ.get(v), v))
cwd = os.getcwd()
if build_dir is None and os.environ.get("GOPATH") and os.environ.get("GOPATH") not in cwd:
logging.warn("Your current directory is not under your GOPATH. This may lead to build failures.")
return True
def check_prereqs():
"""Check user path for required dependencies.
"""
logging.info("Checking for dependencies...")
for req in prereqs:
if not check_path_for(req):
logging.error("Could not find dependency: {}".format(req))
return False
return True
def upload_packages(packages, bucket_name=None, overwrite=False):
"""Upload provided package output to AWS S3.
"""
logging.debug("Uploading files to bucket '{}': {}".format(bucket_name, packages))
try:
import boto
from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat
logging.getLogger("boto").setLevel(logging.WARNING)
except ImportError:
logging.warn("Cannot upload packages without 'boto' Python library!")
return False
logging.info("Connecting to AWS S3...")
# Up the number of attempts to 10 from default of 1
boto.config.add_section("Boto")
boto.config.set("Boto", "metadata_service_num_attempts", "10")
c = boto.connect_s3(calling_format=OrdinaryCallingFormat())
if bucket_name is None:
bucket_name = DEFAULT_BUCKET
bucket = c.get_bucket(bucket_name.split('/')[0])
for p in packages:
if '/' in bucket_name:
# Allow for nested paths within the bucket name (ex:
# bucket/folder). Assuming forward-slashes as path
# delimiter.
name = os.path.join('/'.join(bucket_name.split('/')[1:]),
os.path.basename(p))
else:
name = os.path.basename(p)
logging.debug("Using key: {}".format(name))
if bucket.get_key(name) is None or overwrite:
logging.info("Uploading file {}".format(name))
k = Key(bucket)
k.key = name
if overwrite:
n = k.set_contents_from_filename(p, replace=True)
else:
n = k.set_contents_from_filename(p, replace=False)
k.make_public()
else:
logging.warn("Not uploading file {}, as it already exists in the target bucket.".format(name))
return True
def go_list(vendor=False, relative=False):
"""
Return a list of packages
If vendor is False vendor package are not included
If relative is True the package prefix defined by PACKAGE_URL is stripped
"""
p = subprocess.Popen(["go", "list", "./..."], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
packages = out.split('\n')
if packages[-1] == '':
packages = packages[:-1]
if not vendor:
non_vendor = []
for p in packages:
if '/vendor/' not in p:
non_vendor.append(p)
packages = non_vendor
if relative:
relative_pkgs = []
for p in packages:
r = p.replace(PACKAGE_URL, '.')
if r != '.':
relative_pkgs.append(r)
packages = relative_pkgs
return packages
def build(version=None,
platform=None,
arch=None,
nightly=False,
race=False,
clean=False,
outdir=".",
tags=[],
static=False):
"""Build each target for the specified architecture and platform.
"""
logging.info("Starting build for {}/{}...".format(platform, arch))
logging.info("Using Go version: {}".format(get_go_version()))
logging.info("Using git branch: {}".format(get_current_branch()))
logging.info("Using git commit: {}".format(get_current_commit()))
if static:
logging.info("Using statically-compiled output.")
if race:
logging.info("Race is enabled.")
if len(tags) > 0:
logging.info("Using build tags: {}".format(','.join(tags)))
logging.info("Sending build output to: {}".format(outdir))
if not os.path.exists(outdir):
os.makedirs(outdir)
elif clean and outdir != '/' and outdir != ".":
logging.info("Cleaning build directory '{}' before building.".format(outdir))
shutil.rmtree(outdir)
os.makedirs(outdir)
logging.info("Using version '{}' for build.".format(version))
for target, path in targets.items():
logging.info("Building target: {}".format(target))
build_command = ""
# Handle static binary output
if static is True or "static_" in arch:
if "static_" in arch:
static = True
arch = arch.replace("static_", "")
build_command += "CGO_ENABLED=0 "
# Handle variations in architecture output
if arch == "i386" or arch == "i686":
arch = "386"
elif "arm" in arch:
arch = "arm"
build_command += "GOOS={} GOARCH={} ".format(platform, arch)
if "arm" in arch:
if arch == "armel":
build_command += "GOARM=5 "
elif arch == "armhf" or arch == "arm":
build_command += "GOARM=6 "
elif arch == "arm64":
# TODO(rossmcdonald) - Verify this is the correct setting for arm64
build_command += "GOARM=7 "
else:
logging.error("Invalid ARM architecture specified: {}".format(arch))
logging.error("Please specify either 'armel', 'armhf', or 'arm64'.")
return False
if platform == 'windows':
target = target + '.exe'
build_command += "go build -o {} ".format(os.path.join(outdir, target))
if race:
build_command += "-race "
if len(tags) > 0:
build_command += "-tags {} ".format(','.join(tags))
if "1.4" in get_go_version():
if static:
build_command += "-ldflags=\"-s -X main.version {} -X main.branch {} -X main.commit {}\" ".format(version,
get_current_branch(),
get_current_commit())
else:
build_command += "-ldflags=\"-X main.version {} -X main.branch {} -X main.commit {}\" ".format(version,
get_current_branch(),
get_current_commit())
else:
# Starting with Go 1.5, the linker flag arguments changed to 'name=value' from 'name value'
if static:
build_command += "-ldflags=\"-s -X main.version={} -X main.branch={} -X main.commit={}\" ".format(version,
get_current_branch(),
get_current_commit())
else:
build_command += "-ldflags=\"-X main.version={} -X main.branch={} -X main.commit={}\" ".format(version,
get_current_branch(),
get_current_commit())
if static:
build_command += "-a -installsuffix cgo "
build_command += path
start_time = datetime.utcnow()
run(build_command, shell=True)
end_time = datetime.utcnow()
logging.info("Time taken: {}s".format((end_time - start_time).total_seconds()))
return True
def generate_md5_from_file(path):
"""Generate MD5 signature based on the contents of the file at path.
"""
m = hashlib.md5()
with open(path, 'rb') as f:
for chunk in iter(lambda: f.read(4096), b""):
m.update(chunk)
return m.hexdigest()
def generate_sig_from_file(path):
"""Generate a detached GPG signature from the file at path.
"""
logging.debug("Generating GPG signature for file: {}".format(path))
gpg_path = check_path_for('gpg')
if gpg_path is None:
logging.warn("gpg binary not found on path! Skipping signature creation.")
return False
if os.environ.get("GNUPG_HOME") is not None:
run('gpg --homedir {} --armor --yes --detach-sign {}'.format(os.environ.get("GNUPG_HOME"), path))
else:
run('gpg --armor --detach-sign --yes {}'.format(path))
return True
def package(build_output, pkg_name, version, nightly=False, iteration=1, static=False, release=False):
"""Package the output of the build process.
"""
outfiles = []
tmp_build_dir = create_temp_dir()
logging.debug("Packaging for build output: {}".format(build_output))
logging.info("Using temporary directory: {}".format(tmp_build_dir))
try:
for platform in build_output:
# Create top-level folder displaying which platform (linux, etc)
os.makedirs(os.path.join(tmp_build_dir, platform))
for arch in build_output[platform]:
logging.info("Creating packages for {}/{}".format(platform, arch))
# Create second-level directory displaying the architecture (amd64, etc)
current_location = build_output[platform][arch]
# Create directory tree to mimic file system of package
build_root = os.path.join(tmp_build_dir,
platform,
arch,
'{}-{}-{}'.format(PACKAGE_NAME, version, iteration))
os.makedirs(build_root)
# Copy packaging scripts to build directory
if platform == "windows":
# For windows and static builds, just copy
# binaries to root of package (no other scripts or
# directories)
package_scripts(build_root, config_only=True, windows=True)
elif static or "static_" in arch:
package_scripts(build_root, config_only=True)
else:
create_package_fs(build_root)
package_scripts(build_root)
if platform != "windows":
package_man_files(build_root)
for binary in targets:
# Copy newly-built binaries to packaging directory
if platform == 'windows':
binary = binary + '.exe'
if platform == 'windows' or static or "static_" in arch:
# Where the binary should go in the package filesystem
to = os.path.join(build_root, binary)
# Where the binary currently is located
fr = os.path.join(current_location, binary)
else:
# Where the binary currently is located
fr = os.path.join(current_location, binary)
# Where the binary should go in the package filesystem
to = os.path.join(build_root, INSTALL_ROOT_DIR[1:], binary)
shutil.copy(fr, to)
for package_type in supported_packages[platform]:
# Package the directory structure for each package type for the platform
logging.debug("Packaging directory '{}' as '{}'.".format(build_root, package_type))
name = pkg_name
# Reset version, iteration, and current location on each run
# since they may be modified below.
package_version = version
package_iteration = iteration
if "static_" in arch:
# Remove the "static_" from the displayed arch on the package
package_arch = arch.replace("static_", "")
else:
package_arch = arch
if not release and not nightly:
# For non-release builds, just use the commit hash as the version
package_version = "{}~{}".format(version,
get_current_commit(short=True))
package_iteration = "0"
package_build_root = build_root
current_location = build_output[platform][arch]
if package_type in ['zip', 'tar']:
# For tars and zips, start the packaging one folder above
# the build root (to include the package name)
package_build_root = os.path.join('/', '/'.join(build_root.split('/')[:-1]))
if nightly:
if static or "static_" in arch:
name = '{}-static-nightly_{}_{}'.format(name,
platform,
package_arch)
else:
name = '{}-nightly_{}_{}'.format(name,
platform,
package_arch)
else:
if static or "static_" in arch:
name = '{}-{}-static_{}_{}'.format(name,
package_version,
platform,
package_arch)
else:
name = '{}-{}_{}_{}'.format(name,
package_version,
platform,
package_arch)
current_location = os.path.join(os.getcwd(), current_location)
if package_type == 'tar':
tar_command = "cd {} && tar -cvzf {}.tar.gz --owner=root ./*".format(package_build_root, name)
run(tar_command, shell=True)
run("mv {}.tar.gz {}".format(os.path.join(package_build_root, name), current_location), shell=True)
outfile = os.path.join(current_location, name + ".tar.gz")
outfiles.append(outfile)
elif package_type == 'zip':
zip_command = "cd {} && zip -r {}.zip ./*".format(package_build_root, name)
run(zip_command, shell=True)
run("mv {}.zip {}".format(os.path.join(package_build_root, name), current_location), shell=True)
outfile = os.path.join(current_location, name + ".zip")
outfiles.append(outfile)
elif package_type not in ['zip', 'tar'] and static or "static_" in arch:
logging.info("Skipping package type '{}' for static builds.".format(package_type))
else:
fpm_command = "fpm {} --name {} -a {} -t {} --version {} --iteration {} -C {} -p {} ".format(
fpm_common_args,
name,
package_arch,
package_type,
package_version,
package_iteration,
package_build_root,
current_location)
if package_type == "rpm":
fpm_command += "--depends coreutils --depends shadow-utils --rpm-posttrans {}".format(POSTINST_SCRIPT)
out = run(fpm_command, shell=True)
matches = re.search(':path=>"(.*)"', out)
outfile = None
if matches is not None:
outfile = matches.groups()[0]
if outfile is None:
logging.warn("Could not determine output from packaging output!")
else:
if nightly:
# Strip nightly version from package name
new_outfile = outfile.replace("{}-{}".format(package_version, package_iteration), "nightly")
os.rename(outfile, new_outfile)
outfile = new_outfile
else:
if package_type == 'rpm':
# rpm's convert any dashes to underscores
package_version = package_version.replace("-", "_")
new_outfile = outfile.replace("{}-{}".format(package_version, package_iteration), package_version)
os.rename(outfile, new_outfile)
outfile = new_outfile
outfiles.append(os.path.join(os.getcwd(), outfile))
logging.debug("Produced package files: {}".format(outfiles))
return outfiles
finally:
# Cleanup
shutil.rmtree(tmp_build_dir)
def main(args):
global PACKAGE_NAME
if args.release and args.nightly:
logging.error("Cannot be both a nightly and a release.")
return 1
if args.nightly:
args.version = increment_minor_version(args.version)
args.version = "{}~n{}".format(args.version,
datetime.utcnow().strftime("%Y%m%d%H%M"))
args.iteration = 0
# Pre-build checks
check_environ()
if not check_prereqs():
return 1
if args.build_tags is None:
args.build_tags = []
else:
args.build_tags = args.build_tags.split(',')
orig_commit = get_current_commit(short=True)
orig_branch = get_current_branch()
if args.platform not in supported_builds and args.platform != 'all':
logging.error("Invalid build platform: {}".format(target_platform))
return 1
build_output = {}
if args.branch != orig_branch and args.commit != orig_commit:
logging.error("Can only specify one branch or commit to build from.")
return 1
elif args.branch != orig_branch:
logging.info("Moving to git branch: {}".format(args.branch))
run("git checkout {}".format(args.branch))
elif args.commit != orig_commit:
logging.info("Moving to git commit: {}".format(args.commit))
run("git checkout {}".format(args.commit))
if not args.no_get:
if not go_get(args.branch, update=args.update, no_uncommitted=args.no_uncommitted):
return 1
if args.test:
if not run_tests(args.race, args.parallel, args.timeout, args.no_vet, args.junit_report):
return 1
platforms = []
single_build = True
if args.platform == 'all':
platforms = supported_builds.keys()
single_build = False
else:
platforms = [args.platform]
for platform in platforms:
build_output.update( { platform : {} } )
archs = []
if args.arch == "all":
single_build = False
archs = supported_builds.get(platform)
else:
archs = [args.arch]
for arch in archs:
od = args.outdir
if not single_build:
od = os.path.join(args.outdir, platform, arch)
if not build(version=args.version,
platform=platform,
arch=arch,
nightly=args.nightly,
race=args.race,
clean=args.clean,
outdir=od,
tags=args.build_tags,
static=args.static):
return 1
build_output.get(platform).update( { arch : od } )
# Build packages
if args.package:
if not check_path_for("fpm"):
logging.error("FPM ruby gem required for packaging. Stopping.")
return 1
packages = package(build_output,
args.name,
args.version,
nightly=args.nightly,
iteration=args.iteration,
static=args.static,
release=args.release)
if args.sign:
logging.debug("Generating GPG signatures for packages: {}".format(packages))
sigs = [] # retain signatures so they can be uploaded with packages
for p in packages:
if generate_sig_from_file(p):
sigs.append(p + '.asc')
else:
logging.error("Creation of signature for package [{}] failed!".format(p))
return 1
packages += sigs
if args.upload:
logging.debug("Files staged for upload: {}".format(packages))
if args.nightly:
args.upload_overwrite = True
if not upload_packages(packages, bucket_name=args.bucket, overwrite=args.upload_overwrite):
return 1
logging.info("Packages created:")
for p in packages:
logging.info("{} (MD5={})".format(p.split('/')[-1:][0],
generate_md5_from_file(p)))
if orig_branch != get_current_branch():
logging.info("Moving back to original git branch: {}".format(orig_branch))
run("git checkout {}".format(orig_branch))
return 0
if __name__ == '__main__':
LOG_LEVEL = logging.INFO
if '--debug' in sys.argv[1:]:
LOG_LEVEL = logging.DEBUG
log_format = '[%(levelname)s] %(funcName)s: %(message)s'
logging.basicConfig(level=LOG_LEVEL,
format=log_format)
parser = argparse.ArgumentParser(description='InfluxDB build and packaging script.')
parser.add_argument('--verbose','-v','--debug',
action='store_true',
help='Use debug output')
parser.add_argument('--outdir', '-o',
metavar='<output directory>',
default='./build/',
type=os.path.abspath,
help='Output directory')
parser.add_argument('--name', '-n',
metavar='<name>',
default=PACKAGE_NAME,
type=str,
help='Name to use for package name (when package is specified)')
parser.add_argument('--arch',
metavar='<amd64|i386|armhf|arm64|armel|all>',
type=str,
default=get_system_arch(),
help='Target architecture for build output')
parser.add_argument('--platform',
metavar='<linux|darwin|windows|all>',
type=str,
default=get_system_platform(),
help='Target platform for build output')
parser.add_argument('--branch',
metavar='<branch>',
type=str,
default=get_current_branch(),
help='Build from a specific branch')
parser.add_argument('--commit',
metavar='<commit>',
type=str,
default=get_current_commit(short=True),
help='Build from a specific commit')
parser.add_argument('--version',
metavar='<version>',
type=str,
default=get_current_version(),
help='Version information to apply to build output (ex: 0.12.0)')
parser.add_argument('--iteration',
metavar='<package iteration>',
type=str,
default="1",
help='Package iteration to apply to build output (defaults to 1)')
parser.add_argument('--stats',
action='store_true',
help='Emit build metrics (requires InfluxDB Python client)')
parser.add_argument('--stats-server',
metavar='<hostname:port>',
type=str,
help='Send build stats to InfluxDB using provided hostname and port')
parser.add_argument('--stats-db',
metavar='<database name>',
type=str,
help='Send build stats to InfluxDB using provided database name')
parser.add_argument('--nightly',
action='store_true',
help='Mark build output as nightly build (will incremement the minor version)')
parser.add_argument('--update',
action='store_true',
help='Update build dependencies prior to building')
parser.add_argument('--package',
action='store_true',
help='Package binary output')
parser.add_argument('--release',
action='store_true',
help='Mark build output as release')
parser.add_argument('--clean',
action='store_true',
help='Clean output directory before building')
parser.add_argument('--no-get',
action='store_true',
help='Do not retrieve pinned dependencies when building')
parser.add_argument('--no-uncommitted',
action='store_true',
help='Fail if uncommitted changes exist in the working directory')
parser.add_argument('--upload',
action='store_true',
help='Upload output packages to AWS S3')
parser.add_argument('--upload-overwrite','-w',
action='store_true',
help='Upload output packages to AWS S3')
parser.add_argument('--bucket',
metavar='<S3 bucket name>',
type=str,
default=DEFAULT_BUCKET,
help='Destination bucket for uploads')
parser.add_argument('--build-tags',
metavar='<tags>',
help='Optional build tags to use for compilation')
parser.add_argument('--static',
action='store_true',
help='Create statically-compiled binary output')
parser.add_argument('--sign',
action='store_true',
help='Create GPG detached signatures for packages (when package is specified)')
parser.add_argument('--test',
action='store_true',
help='Run tests (does not produce build output)')
parser.add_argument('--junit-report',
action='store_true',
help='Output tests in the JUnit XML format')
parser.add_argument('--no-vet',
action='store_true',
help='Do not run "go vet" when running tests')
parser.add_argument('--race',
action='store_true',
help='Enable race flag for build output')
parser.add_argument('--parallel',
metavar='<num threads>',
type=int,
help='Number of tests to run simultaneously')
parser.add_argument('--timeout',
metavar='<timeout>',
type=str,
help='Timeout for tests before failing')
args = parser.parse_args()
print_banner()
sys.exit(main(args))

22
build.sh
View File

@ -1,22 +0,0 @@
#!/bin/bash
# Run the build utility via Docker
set -e
# Make sure our working dir is the dir of the script
DIR=$(cd $(dirname ${BASH_SOURCE[0]}) && pwd)
cd $DIR
# Build new docker image
docker build -f Dockerfile_build_ubuntu64 -t influxdb-builder $DIR
echo "Running build.py"
# Run docker
docker run --rm \
-e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
-e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
-v $HOME/.aws.conf:/root/.aws.conf \
-v $DIR:/root/go/src/github.com/influxdata/influxdb \
influxdb-builder \
"$@"

313
client/README.md
View File

@ -1,313 +0,0 @@
# InfluxDB Client
[![GoDoc](https://godoc.org/github.com/influxdata/influxdb?status.svg)](http://godoc.org/github.com/influxdata/influxdb/client/v2)
## Description
**NOTE:** The Go client library now has a "v2" version, with the old version
being deprecated. The new version can be imported at
`import "github.com/influxdata/influxdb/client/v2"`. It is not backwards-compatible.
A Go client library written and maintained by the **InfluxDB** team.
This package provides convenience functions to read and write time series data.
It uses the HTTP protocol to communicate with your **InfluxDB** cluster.
## Getting Started
### Connecting To Your Database
Connecting to an **InfluxDB** database is straightforward. You will need a host
name, a port and the cluster user credentials if applicable. The default port is
8086. You can customize these settings to your specific installation via the
**InfluxDB** configuration file.
Though not necessary for experimentation, you may want to create a new user
and authenticate the connection to your database.
For more information please check out the
[Admin Docs](https://docs.influxdata.com/influxdb/latest/administration/).
For the impatient, you can create a new admin user _bubba_ by firing off the
[InfluxDB CLI](https://github.com/influxdata/influxdb/blob/master/cmd/influx/main.go).
```shell
influx
> create user bubba with password 'bumblebeetuna'
> grant all privileges to bubba
```
And now for good measure set the credentials in you shell environment.
In the example below we will use $INFLUX_USER and $INFLUX_PWD
Now with the administrivia out of the way, let's connect to our database.
NOTE: If you've opted out of creating a user, you can omit Username and Password in
the configuration below.
```go
package main
import (
"log"
"time"
"github.com/influxdata/influxdb/client/v2"
)
const (
MyDB = "square_holes"
username = "bubba"
password = "bumblebeetuna"
)
func main() {
// Create a new HTTPClient
c, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
Username: username,
Password: password,
})
if err != nil {
log.Fatal(err)
}
defer c.Close()
// Create a new point batch
bp, err := client.NewBatchPoints(client.BatchPointsConfig{
Database: MyDB,
Precision: "s",
})
if err != nil {
log.Fatal(err)
}
// Create a point and add to batch
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err != nil {
log.Fatal(err)
}
bp.AddPoint(pt)
// Write the batch
if err := c.Write(bp); err != nil {
log.Fatal(err)
}
// Close client resources
if err := c.Close(); err != nil {
log.Fatal(err)
}
}
```
### Inserting Data
Time series data aka *points* are written to the database using batch inserts.
The mechanism is to create one or more points and then create a batch aka
*batch points* and write these to a given database and series. A series is a
combination of a measurement (time/values) and a set of tags.
In this sample we will create a batch of a 1,000 points. Each point has a time and
a single value as well as 2 tags indicating a shape and color. We write these points
to a database called _square_holes_ using a measurement named _shapes_.
NOTE: You can specify a RetentionPolicy as part of the batch points. If not
provided InfluxDB will use the database _default_ retention policy.
```go
func writePoints(clnt client.Client) {
sampleSize := 1000
bp, err := client.NewBatchPoints(client.BatchPointsConfig{
Database: "systemstats",
Precision: "us",
})
if err != nil {
log.Fatal(err)
}
rand.Seed(time.Now().UnixNano())
for i := 0; i < sampleSize; i++ {
regions := []string{"us-west1", "us-west2", "us-west3", "us-east1"}
tags := map[string]string{
"cpu": "cpu-total",
"host": fmt.Sprintf("host%d", rand.Intn(1000)),
"region": regions[rand.Intn(len(regions))],
}
idle := rand.Float64() * 100.0
fields := map[string]interface{}{
"idle": idle,
"busy": 100.0 - idle,
}
pt, err := client.NewPoint(
"cpu_usage",
tags,
fields,
time.Now(),
)
if err != nil {
log.Fatal(err)
}
bp.AddPoint(pt)
}
if err := clnt.Write(bp); err != nil {
log.Fatal(err)
}
}
```
#### Uint64 Support
The `uint64` data type is supported if your server is version `1.4.0` or
greater. To write a data point as an unsigned integer, you must insert
the point as `uint64`. You cannot use `uint` or any of the other
derivatives because previous versions of the client have supported
writing those types as an integer.
### Querying Data
One nice advantage of using **InfluxDB** the ability to query your data using familiar
SQL constructs. In this example we can create a convenience function to query the database
as follows:
```go
// queryDB convenience function to query the database
func queryDB(clnt client.Client, cmd string) (res []client.Result, err error) {
q := client.Query{
Command: cmd,
Database: MyDB,
}
if response, err := clnt.Query(q); err == nil {
if response.Error() != nil {
return res, response.Error()
}
res = response.Results
} else {
return res, err
}
return res, nil
}
```
#### Creating a Database
```go
_, err := queryDB(clnt, fmt.Sprintf("CREATE DATABASE %s", MyDB))
if err != nil {
log.Fatal(err)
}
```
#### Count Records
```go
q := fmt.Sprintf("SELECT count(%s) FROM %s", "value", MyMeasurement)
res, err := queryDB(clnt, q)
if err != nil {
log.Fatal(err)
}
count := res[0].Series[0].Values[0][1]
log.Printf("Found a total of %v records\n", count)
```
#### Find the last 10 _shapes_ records
```go
q := fmt.Sprintf("SELECT * FROM %s LIMIT %d", MyMeasurement, 10)
res, err = queryDB(clnt, q)
if err != nil {
log.Fatal(err)
}
for i, row := range res[0].Series[0].Values {
t, err := time.Parse(time.RFC3339, row[0].(string))
if err != nil {
log.Fatal(err)
}
val := row[1].(string)
log.Printf("[%2d] %s: %s\n", i, t.Format(time.Stamp), val)
}
```
### Using the UDP Client
The **InfluxDB** client also supports writing over UDP.
```go
func WriteUDP() {
// Make client
// PayloadSize is default value(512)
c, err := client.NewUDPClient(client.UDPConfig{Addr: "localhost:8089"})
if err != nil {
panic(err.Error())
}
// Create a new point batch
bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
Precision: "s",
})
// Create a point and add to batch
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err != nil {
panic(err.Error())
}
bp.AddPoint(pt)
// Write the batch
c.Write(bp)
}
```
### Point Splitting
The UDP client now supports splitting single points that exceed the configured
payload size. The logic for processing each point is listed here, starting with
an empty payload.
1. If adding the point to the current (non-empty) payload would exceed the
configured size, send the current payload. Otherwise, add it to the current
payload.
1. If the point is smaller than the configured size, add it to the payload.
1. If the point has no timestamp, just try to send the entire point as a single
UDP payload, and process the next point.
1. Since the point has a timestamp, re-use the existing measurement name,
tagset, and timestamp and create multiple new points by splitting up the
fields. The per-point length will be kept close to the configured size,
staying under it if possible. This does mean that one large field, maybe a
long string, could be sent as a larger-than-configured payload.
The above logic attempts to respect configured payload sizes, but not sacrifice
any data integrity. Points without a timestamp can't be split, as that may
cause fields to have differing timestamps when processed by the server.
## Go Docs
Please refer to
[http://godoc.org/github.com/influxdata/influxdb/client/v2](http://godoc.org/github.com/influxdata/influxdb/client/v2)
for documentation.
## See Also
You can also examine how the client library is used by the
[InfluxDB CLI](https://github.com/influxdata/influxdb/blob/master/cmd/influx/main.go).

113
client/example_test.go
View File

@ -1,113 +0,0 @@
package client_test
import (
"fmt"
"log"
"math/rand"
"net/url"
"os"
"strconv"
"time"
"github.com/influxdata/influxdb/client"
)
func ExampleNewClient() {
host, err := url.Parse(fmt.Sprintf("http://%s:%d", "localhost", 8086))
if err != nil {
log.Fatal(err)
}
// NOTE: this assumes you've setup a user and have setup shell env variables,
// namely INFLUX_USER/INFLUX_PWD. If not just omit Username/Password below.
conf := client.Config{
URL: *host,
Username: os.Getenv("INFLUX_USER"),
Password: os.Getenv("INFLUX_PWD"),
}
con, err := client.NewClient(conf)
if err != nil {
log.Fatal(err)
}
log.Println("Connection", con)
}
func ExampleClient_Ping() {
host, err := url.Parse(fmt.Sprintf("http://%s:%d", "localhost", 8086))
if err != nil {
log.Fatal(err)
}
con, err := client.NewClient(client.Config{URL: *host})
if err != nil {
log.Fatal(err)
}
dur, ver, err := con.Ping()
if err != nil {
log.Fatal(err)
}
log.Printf("Happy as a hippo! %v, %s", dur, ver)
}
func ExampleClient_Query() {
host, err := url.Parse(fmt.Sprintf("http://%s:%d", "localhost", 8086))
if err != nil {
log.Fatal(err)
}
con, err := client.NewClient(client.Config{URL: *host})
if err != nil {
log.Fatal(err)
}
q := client.Query{
Command: "select count(value) from shapes",
Database: "square_holes",
}
if response, err := con.Query(q); err == nil && response.Error() == nil {
log.Println(response.Results)
}
}
func ExampleClient_Write() {
host, err := url.Parse(fmt.Sprintf("http://%s:%d", "localhost", 8086))
if err != nil {
log.Fatal(err)
}
con, err := client.NewClient(client.Config{URL: *host})
if err != nil {
log.Fatal(err)
}
var (
shapes = []string{"circle", "rectangle", "square", "triangle"}
colors = []string{"red", "blue", "green"}
sampleSize = 1000
pts = make([]client.Point, sampleSize)
)
rand.Seed(42)
for i := 0; i < sampleSize; i++ {
pts[i] = client.Point{
Measurement: "shapes",
Tags: map[string]string{
"color": strconv.Itoa(rand.Intn(len(colors))),
"shape": strconv.Itoa(rand.Intn(len(shapes))),
},
Fields: map[string]interface{}{
"value": rand.Intn(sampleSize),
},
Time: time.Now(),
Precision: "s",
}
}
bps := client.BatchPoints{
Points: pts,
Database: "BumbeBeeTuna",
RetentionPolicy: "default",
}
_, err = con.Write(bps)
if err != nil {
log.Fatal(err)
}
}

870
client/influxdb.go
View File

@ -1,870 +0,0 @@
// Package client implements a now-deprecated client for InfluxDB;
// use github.com/influxdata/influxdb/client/v2 instead.
package client // import "github.com/influxdata/influxdb/client"
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"net"
"net/http"
"net/url"
"path"
"strconv"
"strings"
"time"
"github.com/influxdata/influxdb/models"
)
const (
// DefaultHost is the default host used to connect to an InfluxDB instance
DefaultHost = "localhost"
// DefaultPort is the default port used to connect to an InfluxDB instance
DefaultPort = 8086
// DefaultTimeout is the default connection timeout used to connect to an InfluxDB instance
DefaultTimeout = 0
)
// Query is used to send a command to the server. Both Command and Database are required.
type Query struct {
Command string
Database string
// RetentionPolicy tells the server which retention policy to use by default.
// This option is only effective when querying a server of version 1.6.0 or later.
RetentionPolicy string
// Chunked tells the server to send back chunked responses. This places
// less load on the server by sending back chunks of the response rather
// than waiting for the entire response all at once.
Chunked bool
// ChunkSize sets the maximum number of rows that will be returned per
// chunk. Chunks are either divided based on their series or if they hit
// the chunk size limit.
//
// Chunked must be set to true for this option to be used.
ChunkSize int
// NodeID sets the data node to use for the query results. This option only
// has any effect in the enterprise version of the software where there can be
// more than one data node and is primarily useful for analyzing differences in
// data. The default behavior is to automatically select the appropriate data
// nodes to retrieve all of the data. On a database where the number of data nodes
// is greater than the replication factor, it is expected that setting this option
// will only retrieve partial data.
NodeID int
}
// ParseConnectionString will parse a string to create a valid connection URL
func ParseConnectionString(path string, ssl bool) (url.URL, error) {
var host string
var port int
h, p, err := net.SplitHostPort(path)
if err != nil {
if path == "" {
host = DefaultHost
} else {
host = path
}
// If they didn't specify a port, always use the default port
port = DefaultPort
} else {
host = h
port, err = strconv.Atoi(p)
if err != nil {
return url.URL{}, fmt.Errorf("invalid port number %q: %s\n", path, err)
}
}
u := url.URL{
Scheme: "http",
Host: host,
}
if ssl {
u.Scheme = "https"
if port != 443 {
u.Host = net.JoinHostPort(host, strconv.Itoa(port))
}
} else if port != 80 {
u.Host = net.JoinHostPort(host, strconv.Itoa(port))
}
return u, nil
}
// Config is used to specify what server to connect to.
// URL: The URL of the server connecting to.
// Username/Password are optional. They will be passed via basic auth if provided.
// UserAgent: If not provided, will default "InfluxDBClient",
// Timeout: If not provided, will default to 0 (no timeout)
type Config struct {
URL url.URL
UnixSocket string
Username string
Password string
UserAgent string
Timeout time.Duration
Precision string
WriteConsistency string
UnsafeSsl bool
Proxy func(req *http.Request) (*url.URL, error)
TLS *tls.Config
}
// NewConfig will create a config to be used in connecting to the client
func NewConfig() Config {
return Config{
Timeout: DefaultTimeout,
}
}
// Client is used to make calls to the server.
type Client struct {
url url.URL
unixSocket string
username string
password string
httpClient *http.Client
userAgent string
precision string
}
const (
// ConsistencyOne requires at least one data node acknowledged a write.
ConsistencyOne = "one"
// ConsistencyAll requires all data nodes to acknowledge a write.
ConsistencyAll = "all"
// ConsistencyQuorum requires a quorum of data nodes to acknowledge a write.
ConsistencyQuorum = "quorum"
// ConsistencyAny allows for hinted hand off, potentially no write happened yet.
ConsistencyAny = "any"
)
// NewClient will instantiate and return a connected client to issue commands to the server.
func NewClient(c Config) (*Client, error) {
tlsConfig := new(tls.Config)
if c.TLS != nil {
tlsConfig = c.TLS.Clone()
}
tlsConfig.InsecureSkipVerify = c.UnsafeSsl
tr := &http.Transport{
Proxy: c.Proxy,
TLSClientConfig: tlsConfig,
}
if c.UnixSocket != "" {
// No need for compression in local communications.
tr.DisableCompression = true
tr.DialContext = func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial("unix", c.UnixSocket)
}
}
client := Client{
url: c.URL,
unixSocket: c.UnixSocket,
username: c.Username,
password: c.Password,
httpClient: &http.Client{Timeout: c.Timeout, Transport: tr},
userAgent: c.UserAgent,
precision: c.Precision,
}
if client.userAgent == "" {
client.userAgent = "InfluxDBClient"
}
return &client, nil
}
// SetAuth will update the username and passwords
func (c *Client) SetAuth(u, p string) {
c.username = u
c.password = p
}
// SetPrecision will update the precision
func (c *Client) SetPrecision(precision string) {
c.precision = precision
}
// Query sends a command to the server and returns the Response
func (c *Client) Query(q Query) (*Response, error) {
return c.QueryContext(context.Background(), q)
}
// QueryContext sends a command to the server and returns the Response
// It uses a context that can be cancelled by the command line client
func (c *Client) QueryContext(ctx context.Context, q Query) (*Response, error) {
u := c.url
u.Path = path.Join(u.Path, "query")
values := u.Query()
values.Set("q", q.Command)
values.Set("db", q.Database)
if q.RetentionPolicy != "" {
values.Set("rp", q.RetentionPolicy)
}
if q.Chunked {
values.Set("chunked", "true")
if q.ChunkSize > 0 {
values.Set("chunk_size", strconv.Itoa(q.ChunkSize))
}
}
if q.NodeID > 0 {
values.Set("node_id", strconv.Itoa(q.NodeID))
}
if c.precision != "" {
values.Set("epoch", c.precision)
}
u.RawQuery = values.Encode()
req, err := http.NewRequest("POST", u.String(), nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", c.userAgent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
req = req.WithContext(ctx)
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
var response Response
if q.Chunked {
cr := NewChunkedResponse(resp.Body)
for {
r, err := cr.NextResponse()
if err != nil {
// If we got an error while decoding the response, send that back.
return nil, err
}
if r == nil {
break
}
response.Results = append(response.Results, r.Results...)
if r.Err != nil {
response.Err = r.Err
break
}
}
} else {
dec := json.NewDecoder(resp.Body)
dec.UseNumber()
if err := dec.Decode(&response); err != nil {
// Ignore EOF errors if we got an invalid status code.
if !(err == io.EOF && resp.StatusCode != http.StatusOK) {
return nil, err
}
}
}
// If we don't have an error in our json response, and didn't get StatusOK,
// then send back an error.
if resp.StatusCode != http.StatusOK && response.Error() == nil {
return &response, fmt.Errorf("received status code %d from server", resp.StatusCode)
}
return &response, nil
}
// Write takes BatchPoints and allows for writing of multiple points with defaults
// If successful, error is nil and Response is nil
// If an error occurs, Response may contain additional information if populated.
func (c *Client) Write(bp BatchPoints) (*Response, error) {
u := c.url
u.Path = path.Join(u.Path, "write")
var b bytes.Buffer
for _, p := range bp.Points {
err := checkPointTypes(p)
if err != nil {
return nil, err
}
if p.Raw != "" {
if _, err := b.WriteString(p.Raw); err != nil {
return nil, err
}
} else {
for k, v := range bp.Tags {
if p.Tags == nil {
p.Tags = make(map[string]string, len(bp.Tags))
}
p.Tags[k] = v
}
if _, err := b.WriteString(p.MarshalString()); err != nil {
return nil, err
}
}
if err := b.WriteByte('\n'); err != nil {
return nil, err
}
}
req, err := http.NewRequest("POST", u.String(), &b)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "")
req.Header.Set("User-Agent", c.userAgent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
precision := bp.Precision
if precision == "" {
precision = c.precision
}
params := req.URL.Query()
params.Set("db", bp.Database)
params.Set("rp", bp.RetentionPolicy)
params.Set("precision", precision)
params.Set("consistency", bp.WriteConsistency)
req.URL.RawQuery = params.Encode()
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
var response Response
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
var err = fmt.Errorf(string(body))
response.Err = err
return &response, err
}
return nil, nil
}
// WriteLineProtocol takes a string with line returns to delimit each write
// If successful, error is nil and Response is nil
// If an error occurs, Response may contain additional information if populated.
func (c *Client) WriteLineProtocol(data, database, retentionPolicy, precision, writeConsistency string) (*Response, error) {
u := c.url
u.Path = path.Join(u.Path, "write")
r := strings.NewReader(data)
req, err := http.NewRequest("POST", u.String(), r)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "")
req.Header.Set("User-Agent", c.userAgent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
params := req.URL.Query()
params.Set("db", database)
params.Set("rp", retentionPolicy)
params.Set("precision", precision)
params.Set("consistency", writeConsistency)
req.URL.RawQuery = params.Encode()
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
var response Response
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
err := fmt.Errorf(string(body))
response.Err = err
return &response, err
}
return nil, nil
}
// Ping will check to see if the server is up
// Ping returns how long the request took, the version of the server it connected to, and an error if one occurred.
func (c *Client) Ping() (time.Duration, string, error) {
now := time.Now()
u := c.url
u.Path = path.Join(u.Path, "ping")
req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
return 0, "", err
}
req.Header.Set("User-Agent", c.userAgent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return 0, "", err
}
defer resp.Body.Close()
version := resp.Header.Get("X-Influxdb-Version")
return time.Since(now), version, nil
}
// Structs
// Message represents a user message.
type Message struct {
Level string `json:"level,omitempty"`
Text string `json:"text,omitempty"`
}
// Result represents a resultset returned from a single statement.
type Result struct {
Series []models.Row
Messages []*Message
Err error
}
// MarshalJSON encodes the result into JSON.
func (r *Result) MarshalJSON() ([]byte, error) {
// Define a struct that outputs "error" as a string.
var o struct {
Series []models.Row `json:"series,omitempty"`
Messages []*Message `json:"messages,omitempty"`
Err string `json:"error,omitempty"`
}
// Copy fields to output struct.
o.Series = r.Series
o.Messages = r.Messages
if r.Err != nil {
o.Err = r.Err.Error()
}
return json.Marshal(&o)
}
// UnmarshalJSON decodes the data into the Result struct
func (r *Result) UnmarshalJSON(b []byte) error {
var o struct {
Series []models.Row `json:"series,omitempty"`
Messages []*Message `json:"messages,omitempty"`
Err string `json:"error,omitempty"`
}
dec := json.NewDecoder(bytes.NewBuffer(b))
dec.UseNumber()
err := dec.Decode(&o)
if err != nil {
return err
}
r.Series = o.Series
r.Messages = o.Messages
if o.Err != "" {
r.Err = errors.New(o.Err)
}
return nil
}
// Response represents a list of statement results.
type Response struct {
Results []Result
Err error
}
// MarshalJSON encodes the response into JSON.
func (r *Response) MarshalJSON() ([]byte, error) {
// Define a struct that outputs "error" as a string.
var o struct {
Results []Result `json:"results,omitempty"`
Err string `json:"error,omitempty"`
}
// Copy fields to output struct.
o.Results = r.Results
if r.Err != nil {
o.Err = r.Err.Error()
}
return json.Marshal(&o)
}
// UnmarshalJSON decodes the data into the Response struct
func (r *Response) UnmarshalJSON(b []byte) error {
var o struct {
Results []Result `json:"results,omitempty"`
Err string `json:"error,omitempty"`
}
dec := json.NewDecoder(bytes.NewBuffer(b))
dec.UseNumber()
err := dec.Decode(&o)
if err != nil {
return err
}
r.Results = o.Results
if o.Err != "" {
r.Err = errors.New(o.Err)
}
return nil
}
// Error returns the first error from any statement.
// Returns nil if no errors occurred on any statements.
func (r *Response) Error() error {
if r.Err != nil {
return r.Err
}
for _, result := range r.Results {
if result.Err != nil {
return result.Err
}
}
return nil
}
// duplexReader reads responses and writes it to another writer while
// satisfying the reader interface.
type duplexReader struct {
r io.Reader
w io.Writer
}
func (r *duplexReader) Read(p []byte) (n int, err error) {
n, err = r.r.Read(p)
if err == nil {
r.w.Write(p[:n])
}
return n, err
}
// ChunkedResponse represents a response from the server that
// uses chunking to stream the output.
type ChunkedResponse struct {
dec *json.Decoder
duplex *duplexReader
buf bytes.Buffer
}
// NewChunkedResponse reads a stream and produces responses from the stream.
func NewChunkedResponse(r io.Reader) *ChunkedResponse {
resp := &ChunkedResponse{}
resp.duplex = &duplexReader{r: r, w: &resp.buf}
resp.dec = json.NewDecoder(resp.duplex)
resp.dec.UseNumber()
return resp
}
// NextResponse reads the next line of the stream and returns a response.
func (r *ChunkedResponse) NextResponse() (*Response, error) {
var response Response
if err := r.dec.Decode(&response); err != nil {
if err == io.EOF {
return nil, nil
}
// A decoding error happened. This probably means the server crashed
// and sent a last-ditch error message to us. Ensure we have read the
// entirety of the connection to get any remaining error text.
io.Copy(ioutil.Discard, r.duplex)
return nil, errors.New(strings.TrimSpace(r.buf.String()))
}
r.buf.Reset()
return &response, nil
}
// Point defines the fields that will be written to the database
// Measurement, Time, and Fields are required
// Precision can be specified if the time is in epoch format (integer).
// Valid values for Precision are n, u, ms, s, m, and h
type Point struct {
Measurement string
Tags map[string]string
Time time.Time
Fields map[string]interface{}
Precision string
Raw string
}
// MarshalJSON will format the time in RFC3339Nano
// Precision is also ignored as it is only used for writing, not reading
// Or another way to say it is we always send back in nanosecond precision
func (p *Point) MarshalJSON() ([]byte, error) {
point := struct {
Measurement string `json:"measurement,omitempty"`
Tags map[string]string `json:"tags,omitempty"`
Time string `json:"time,omitempty"`
Fields map[string]interface{} `json:"fields,omitempty"`
Precision string `json:"precision,omitempty"`
}{
Measurement: p.Measurement,
Tags: p.Tags,
Fields: p.Fields,
Precision: p.Precision,
}
// Let it omit empty if it's really zero
if !p.Time.IsZero() {
point.Time = p.Time.UTC().Format(time.RFC3339Nano)
}
return json.Marshal(&point)
}
// MarshalString renders string representation of a Point with specified
// precision. The default precision is nanoseconds.
func (p *Point) MarshalString() string {
pt, err := models.NewPoint(p.Measurement, models.NewTags(p.Tags), p.Fields, p.Time)
if err != nil {
return "# ERROR: " + err.Error() + " " + p.Measurement
}
if p.Precision == "" || p.Precision == "ns" || p.Precision == "n" {
return pt.String()
}
return pt.PrecisionString(p.Precision)
}
// UnmarshalJSON decodes the data into the Point struct
func (p *Point) UnmarshalJSON(b []byte) error {
var normal struct {
Measurement string `json:"measurement"`
Tags map[string]string `json:"tags"`
Time time.Time `json:"time"`
Precision string `json:"precision"`
Fields map[string]interface{} `json:"fields"`
}
var epoch struct {
Measurement string `json:"measurement"`
Tags map[string]string `json:"tags"`
Time *int64 `json:"time"`
Precision string `json:"precision"`
Fields map[string]interface{} `json:"fields"`
}
if err := func() error {
var err error
dec := json.NewDecoder(bytes.NewBuffer(b))
dec.UseNumber()
if err = dec.Decode(&epoch); err != nil {
return err
}
// Convert from epoch to time.Time, but only if Time
// was actually set.
var ts time.Time
if epoch.Time != nil {
ts, err = EpochToTime(*epoch.Time, epoch.Precision)
if err != nil {
return err
}
}
p.Measurement = epoch.Measurement
p.Tags = epoch.Tags
p.Time = ts
p.Precision = epoch.Precision
p.Fields = normalizeFields(epoch.Fields)
return nil
}(); err == nil {
return nil
}
dec := json.NewDecoder(bytes.NewBuffer(b))
dec.UseNumber()
if err := dec.Decode(&normal); err != nil {
return err
}
normal.Time = SetPrecision(normal.Time, normal.Precision)
p.Measurement = normal.Measurement
p.Tags = normal.Tags
p.Time = normal.Time
p.Precision = normal.Precision
p.Fields = normalizeFields(normal.Fields)
return nil
}
// Remove any notion of json.Number
func normalizeFields(fields map[string]interface{}) map[string]interface{} {
newFields := map[string]interface{}{}
for k, v := range fields {
switch v := v.(type) {
case json.Number:
jv, e := v.Float64()
if e != nil {
panic(fmt.Sprintf("unable to convert json.Number to float64: %s", e))
}
newFields[k] = jv
default:
newFields[k] = v
}
}
return newFields
}
// BatchPoints is used to send batched data in a single write.
// Database and Points are required
// If no retention policy is specified, it will use the databases default retention policy.
// If tags are specified, they will be "merged" with all points. If a point already has that tag, it will be ignored.
// If time is specified, it will be applied to any point with an empty time.
// Precision can be specified if the time is in epoch format (integer).
// Valid values for Precision are n, u, ms, s, m, and h
type BatchPoints struct {
Points []Point `json:"points,omitempty"`
Database string `json:"database,omitempty"`
RetentionPolicy string `json:"retentionPolicy,omitempty"`
Tags map[string]string `json:"tags,omitempty"`
Time time.Time `json:"time,omitempty"`
Precision string `json:"precision,omitempty"`
WriteConsistency string `json:"-"`
}
// UnmarshalJSON decodes the data into the BatchPoints struct
func (bp *BatchPoints) UnmarshalJSON(b []byte) error {
var normal struct {
Points []Point `json:"points"`
Database string `json:"database"`
RetentionPolicy string `json:"retentionPolicy"`
Tags map[string]string `json:"tags"`
Time time.Time `json:"time"`
Precision string `json:"precision"`
}
var epoch struct {
Points []Point `json:"points"`
Database string `json:"database"`
RetentionPolicy string `json:"retentionPolicy"`
Tags map[string]string `json:"tags"`
Time *int64 `json:"time"`
Precision string `json:"precision"`
}
if err := func() error {
var err error
if err = json.Unmarshal(b, &epoch); err != nil {
return err
}
// Convert from epoch to time.Time
var ts time.Time
if epoch.Time != nil {
ts, err = EpochToTime(*epoch.Time, epoch.Precision)
if err != nil {
return err
}
}
bp.Points = epoch.Points
bp.Database = epoch.Database
bp.RetentionPolicy = epoch.RetentionPolicy
bp.Tags = epoch.Tags
bp.Time = ts
bp.Precision = epoch.Precision
return nil
}(); err == nil {
return nil
}
if err := json.Unmarshal(b, &normal); err != nil {
return err
}
normal.Time = SetPrecision(normal.Time, normal.Precision)
bp.Points = normal.Points
bp.Database = normal.Database
bp.RetentionPolicy = normal.RetentionPolicy
bp.Tags = normal.Tags
bp.Time = normal.Time
bp.Precision = normal.Precision
return nil
}
// utility functions
// Addr provides the current url as a string of the server the client is connected to.
func (c *Client) Addr() string {
if c.unixSocket != "" {
return c.unixSocket
}
return c.url.String()
}
// checkPointTypes ensures no unsupported types are submitted to influxdb, returning error if they are found.
func checkPointTypes(p Point) error {
for _, v := range p.Fields {
switch v.(type) {
case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64, bool, string, nil:
return nil
default:
return fmt.Errorf("unsupported point type: %T", v)
}
}
return nil
}
// helper functions
// EpochToTime takes a unix epoch time and uses precision to return back a time.Time
func EpochToTime(epoch int64, precision string) (time.Time, error) {
if precision == "" {
precision = "s"
}
var t time.Time
switch precision {
case "h":
t = time.Unix(0, epoch*int64(time.Hour))
case "m":
t = time.Unix(0, epoch*int64(time.Minute))
case "s":
t = time.Unix(0, epoch*int64(time.Second))
case "ms":
t = time.Unix(0, epoch*int64(time.Millisecond))
case "u":
t = time.Unix(0, epoch*int64(time.Microsecond))
case "n":
t = time.Unix(0, epoch)
default:
return time.Time{}, fmt.Errorf("Unknown precision %q", precision)
}
return t, nil
}
// SetPrecision will round a time to the specified precision
func SetPrecision(t time.Time, precision string) time.Time {
switch precision {
case "n":
case "u":
return t.Round(time.Microsecond)
case "ms":
return t.Round(time.Millisecond)
case "s":
return t.Round(time.Second)
case "m":
return t.Round(time.Minute)
case "h":
return t.Round(time.Hour)
}
return t
}

1022
client/influxdb_test.go
View File

File diff suppressed because it is too large Load Diff

729
client/v2/client.go
View File

@ -1,729 +0,0 @@
// Package client (v2) is the current official Go client for InfluxDB.
package client // import "github.com/influxdata/influxdb/client/v2"
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"mime"
"net"
"net/http"
"net/url"
"path"
"strconv"
"strings"
"time"
"github.com/influxdata/influxdb/models"
)
// HTTPConfig is the config data needed to create an HTTP Client.
type HTTPConfig struct {
// Addr should be of the form "http://host:port"
// or "http://[ipv6-host%zone]:port".
Addr string
// Username is the influxdb username, optional.
Username string
// Password is the influxdb password, optional.
Password string
// UserAgent is the http User Agent, defaults to "InfluxDBClient".
UserAgent string
// Timeout for influxdb writes, defaults to no timeout.
Timeout time.Duration
// InsecureSkipVerify gets passed to the http client, if true, it will
// skip https certificate verification. Defaults to false.
InsecureSkipVerify bool
// TLSConfig allows the user to set their own TLS config for the HTTP
// Client. If set, this option overrides InsecureSkipVerify.
TLSConfig *tls.Config
// Proxy configures the Proxy function on the HTTP client.
Proxy func(req *http.Request) (*url.URL, error)
// DialContext specifies the dial function for creating unencrypted TCP connections.
// If DialContext is nil then the transport dials using package net.
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
}
// BatchPointsConfig is the config data needed to create an instance of the BatchPoints struct.
type BatchPointsConfig struct {
// Precision is the write precision of the points, defaults to "ns".
Precision string
// Database is the database to write points to.
Database string
// RetentionPolicy is the retention policy of the points.
RetentionPolicy string
// Write consistency is the number of servers required to confirm write.
WriteConsistency string
}
// Client is a client interface for writing & querying the database.
type Client interface {
// Ping checks that status of cluster, and will always return 0 time and no
// error for UDP clients.
Ping(timeout time.Duration) (time.Duration, string, error)
// Write takes a BatchPoints object and writes all Points to InfluxDB.
Write(bp BatchPoints) error
// Query makes an InfluxDB Query on the database. This will fail if using
// the UDP client.
Query(q Query) (*Response, error)
// QueryAsChunk makes an InfluxDB Query on the database. This will fail if using
// the UDP client.
QueryAsChunk(q Query) (*ChunkedResponse, error)
// Close releases any resources a Client may be using.
Close() error
}
// NewHTTPClient returns a new Client from the provided config.
// Client is safe for concurrent use by multiple goroutines.
func NewHTTPClient(conf HTTPConfig) (Client, error) {
if conf.UserAgent == "" {
conf.UserAgent = "InfluxDBClient"
}
u, err := url.Parse(conf.Addr)
if err != nil {
return nil, err
} else if u.Scheme != "http" && u.Scheme != "https" {
m := fmt.Sprintf("Unsupported protocol scheme: %s, your address"+
" must start with http:// or https://", u.Scheme)
return nil, errors.New(m)
}
tr := &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: conf.InsecureSkipVerify,
},
Proxy: conf.Proxy,
DialContext: conf.DialContext,
}
if conf.TLSConfig != nil {
tr.TLSClientConfig = conf.TLSConfig
}
return &client{
url: *u,
username: conf.Username,
password: conf.Password,
useragent: conf.UserAgent,
httpClient: &http.Client{
Timeout: conf.Timeout,
Transport: tr,
},
transport: tr,
}, nil
}
// Ping will check to see if the server is up with an optional timeout on waiting for leader.
// Ping returns how long the request took, the version of the server it connected to, and an error if one occurred.
func (c *client) Ping(timeout time.Duration) (time.Duration, string, error) {
now := time.Now()
u := c.url
u.Path = path.Join(u.Path, "ping")
req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
return 0, "", err
}
req.Header.Set("User-Agent", c.useragent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
if timeout > 0 {
params := req.URL.Query()
params.Set("wait_for_leader", fmt.Sprintf("%.0fs", timeout.Seconds()))
req.URL.RawQuery = params.Encode()
}
resp, err := c.httpClient.Do(req)
if err != nil {
return 0, "", err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return 0, "", err
}
if resp.StatusCode != http.StatusNoContent {
var err = errors.New(string(body))
return 0, "", err
}
version := resp.Header.Get("X-Influxdb-Version")
return time.Since(now), version, nil
}
// Close releases the client's resources.
func (c *client) Close() error {
c.transport.CloseIdleConnections()
return nil
}
// client is safe for concurrent use as the fields are all read-only
// once the client is instantiated.
type client struct {
// N.B - if url.UserInfo is accessed in future modifications to the
// methods on client, you will need to synchronize access to url.
url url.URL
username string
password string
useragent string
httpClient *http.Client
transport *http.Transport
}
// BatchPoints is an interface into a batched grouping of points to write into
// InfluxDB together. BatchPoints is NOT thread-safe, you must create a separate
// batch for each goroutine.
type BatchPoints interface {
// AddPoint adds the given point to the Batch of points.
AddPoint(p *Point)
// AddPoints adds the given points to the Batch of points.
AddPoints(ps []*Point)
// Points lists the points in the Batch.
Points() []*Point
// Precision returns the currently set precision of this Batch.
Precision() string
// SetPrecision sets the precision of this batch.
SetPrecision(s string) error
// Database returns the currently set database of this Batch.
Database() string
// SetDatabase sets the database of this Batch.
SetDatabase(s string)
// WriteConsistency returns the currently set write consistency of this Batch.
WriteConsistency() string
// SetWriteConsistency sets the write consistency of this Batch.
SetWriteConsistency(s string)
// RetentionPolicy returns the currently set retention policy of this Batch.
RetentionPolicy() string
// SetRetentionPolicy sets the retention policy of this Batch.
SetRetentionPolicy(s string)
}
// NewBatchPoints returns a BatchPoints interface based on the given config.
func NewBatchPoints(conf BatchPointsConfig) (BatchPoints, error) {
if conf.Precision == "" {
conf.Precision = "ns"
}
if _, err := time.ParseDuration("1" + conf.Precision); err != nil {
return nil, err
}
bp := &batchpoints{
database: conf.Database,
precision: conf.Precision,
retentionPolicy: conf.RetentionPolicy,
writeConsistency: conf.WriteConsistency,
}
return bp, nil
}
type batchpoints struct {
points []*Point
database string
precision string
retentionPolicy string
writeConsistency string
}
func (bp *batchpoints) AddPoint(p *Point) {
bp.points = append(bp.points, p)
}
func (bp *batchpoints) AddPoints(ps []*Point) {
bp.points = append(bp.points, ps...)
}
func (bp *batchpoints) Points() []*Point {
return bp.points
}
func (bp *batchpoints) Precision() string {
return bp.precision
}
func (bp *batchpoints) Database() string {
return bp.database
}
func (bp *batchpoints) WriteConsistency() string {
return bp.writeConsistency
}
func (bp *batchpoints) RetentionPolicy() string {
return bp.retentionPolicy
}
func (bp *batchpoints) SetPrecision(p string) error {
if _, err := time.ParseDuration("1" + p); err != nil {
return err
}
bp.precision = p
return nil
}
func (bp *batchpoints) SetDatabase(db string) {
bp.database = db
}
func (bp *batchpoints) SetWriteConsistency(wc string) {
bp.writeConsistency = wc
}
func (bp *batchpoints) SetRetentionPolicy(rp string) {
bp.retentionPolicy = rp
}
// Point represents a single data point.
type Point struct {
pt models.Point
}
// NewPoint returns a point with the given timestamp. If a timestamp is not
// given, then data is sent to the database without a timestamp, in which case
// the server will assign local time upon reception. NOTE: it is recommended to
// send data with a timestamp.
func NewPoint(
name string,
tags map[string]string,
fields map[string]interface{},
t ...time.Time,
) (*Point, error) {
var T time.Time
if len(t) > 0 {
T = t[0]
}
pt, err := models.NewPoint(name, models.NewTags(tags), fields, T)
if err != nil {
return nil, err
}
return &Point{
pt: pt,
}, nil
}
// String returns a line-protocol string of the Point.
func (p *Point) String() string {
return p.pt.String()
}
// PrecisionString returns a line-protocol string of the Point,
// with the timestamp formatted for the given precision.
func (p *Point) PrecisionString(precision string) string {
return p.pt.PrecisionString(precision)
}
// Name returns the measurement name of the point.
func (p *Point) Name() string {
return string(p.pt.Name())
}
// Tags returns the tags associated with the point.
func (p *Point) Tags() map[string]string {
return p.pt.Tags().Map()
}
// Time return the timestamp for the point.
func (p *Point) Time() time.Time {
return p.pt.Time()
}
// UnixNano returns timestamp of the point in nanoseconds since Unix epoch.
func (p *Point) UnixNano() int64 {
return p.pt.UnixNano()
}
// Fields returns the fields for the point.
func (p *Point) Fields() (map[string]interface{}, error) {
return p.pt.Fields()
}
// NewPointFrom returns a point from the provided models.Point.
func NewPointFrom(pt models.Point) *Point {
return &Point{pt: pt}
}
func (c *client) Write(bp BatchPoints) error {
var b bytes.Buffer
for _, p := range bp.Points() {
if p == nil {
continue
}
if _, err := b.WriteString(p.pt.PrecisionString(bp.Precision())); err != nil {
return err
}
if err := b.WriteByte('\n'); err != nil {
return err
}
}
u := c.url
u.Path = path.Join(u.Path, "write")
req, err := http.NewRequest("POST", u.String(), &b)
if err != nil {
return err
}
req.Header.Set("Content-Type", "")
req.Header.Set("User-Agent", c.useragent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
params := req.URL.Query()
params.Set("db", bp.Database())
params.Set("rp", bp.RetentionPolicy())
params.Set("precision", bp.Precision())
params.Set("consistency", bp.WriteConsistency())
req.URL.RawQuery = params.Encode()
resp, err := c.httpClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
var err = errors.New(string(body))
return err
}
return nil
}
// Query defines a query to send to the server.
type Query struct {
Command string
Database string
RetentionPolicy string
Precision string
Chunked bool
ChunkSize int
Parameters map[string]interface{}
}
// NewQuery returns a query object.
// The database and precision arguments can be empty strings if they are not needed for the query.
func NewQuery(command, database, precision string) Query {
return Query{
Command: command,
Database: database,
Precision: precision,
Parameters: make(map[string]interface{}),
}
}
// NewQueryWithRP returns a query object.
// The database, retention policy, and precision arguments can be empty strings if they are not needed
// for the query. Setting the retention policy only works on InfluxDB versions 1.6 or greater.
func NewQueryWithRP(command, database, retentionPolicy, precision string) Query {
return Query{
Command: command,
Database: database,
RetentionPolicy: retentionPolicy,
Precision: precision,
Parameters: make(map[string]interface{}),
}
}
// NewQueryWithParameters returns a query object.
// The database and precision arguments can be empty strings if they are not needed for the query.
// parameters is a map of the parameter names used in the command to their values.
func NewQueryWithParameters(command, database, precision string, parameters map[string]interface{}) Query {
return Query{
Command: command,
Database: database,
Precision: precision,
Parameters: parameters,
}
}
// Response represents a list of statement results.
type Response struct {
Results []Result
Err string `json:"error,omitempty"`
}
// Error returns the first error from any statement.
// It returns nil if no errors occurred on any statements.
func (r *Response) Error() error {
if r.Err != "" {
return errors.New(r.Err)
}
for _, result := range r.Results {
if result.Err != "" {
return errors.New(result.Err)
}
}
return nil
}
// Message represents a user message.
type Message struct {
Level string
Text string
}
// Result represents a resultset returned from a single statement.
type Result struct {
Series []models.Row
Messages []*Message
Err string `json:"error,omitempty"`
}
// Query sends a command to the server and returns the Response.
func (c *client) Query(q Query) (*Response, error) {
req, err := c.createDefaultRequest(q)
if err != nil {
return nil, err
}
params := req.URL.Query()
if q.Chunked {
params.Set("chunked", "true")
if q.ChunkSize > 0 {
params.Set("chunk_size", strconv.Itoa(q.ChunkSize))
}
req.URL.RawQuery = params.Encode()
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if err := checkResponse(resp); err != nil {
return nil, err
}
var response Response
if q.Chunked {
cr := NewChunkedResponse(resp.Body)
for {
r, err := cr.NextResponse()
if err != nil {
if err == io.EOF {
break
}
// If we got an error while decoding the response, send that back.
return nil, err
}
if r == nil {
break
}
response.Results = append(response.Results, r.Results...)
if r.Err != "" {
response.Err = r.Err
break
}
}
} else {
dec := json.NewDecoder(resp.Body)
dec.UseNumber()
decErr := dec.Decode(&response)
// ignore this error if we got an invalid status code
if decErr != nil && decErr.Error() == "EOF" && resp.StatusCode != http.StatusOK {
decErr = nil
}
// If we got a valid decode error, send that back
if decErr != nil {
return nil, fmt.Errorf("unable to decode json: received status code %d err: %s", resp.StatusCode, decErr)
}
}
// If we don't have an error in our json response, and didn't get statusOK
// then send back an error
if resp.StatusCode != http.StatusOK && response.Error() == nil {
return &response, fmt.Errorf("received status code %d from server", resp.StatusCode)
}
return &response, nil
}
// QueryAsChunk sends a command to the server and returns the Response.
func (c *client) QueryAsChunk(q Query) (*ChunkedResponse, error) {
req, err := c.createDefaultRequest(q)
if err != nil {
return nil, err
}
params := req.URL.Query()
params.Set("chunked", "true")
if q.ChunkSize > 0 {
params.Set("chunk_size", strconv.Itoa(q.ChunkSize))
}
req.URL.RawQuery = params.Encode()
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
if err := checkResponse(resp); err != nil {
return nil, err
}
return NewChunkedResponse(resp.Body), nil
}
func checkResponse(resp *http.Response) error {
// If we lack a X-Influxdb-Version header, then we didn't get a response from influxdb
// but instead some other service. If the error code is also a 500+ code, then some
// downstream loadbalancer/proxy/etc had an issue and we should report that.
if resp.Header.Get("X-Influxdb-Version") == "" && resp.StatusCode >= http.StatusInternalServerError {
body, err := ioutil.ReadAll(resp.Body)
if err != nil || len(body) == 0 {
return fmt.Errorf("received status code %d from downstream server", resp.StatusCode)
}
return fmt.Errorf("received status code %d from downstream server, with response body: %q", resp.StatusCode, body)
}
// If we get an unexpected content type, then it is also not from influx direct and therefore
// we want to know what we received and what status code was returned for debugging purposes.
if cType, _, _ := mime.ParseMediaType(resp.Header.Get("Content-Type")); cType != "application/json" {
// Read up to 1kb of the body to help identify downstream errors and limit the impact of things
// like downstream serving a large file
body, err := ioutil.ReadAll(io.LimitReader(resp.Body, 1024))
if err != nil || len(body) == 0 {
return fmt.Errorf("expected json response, got empty body, with status: %v", resp.StatusCode)
}
return fmt.Errorf("expected json response, got %q, with status: %v and response body: %q", cType, resp.StatusCode, body)
}
return nil
}
func (c *client) createDefaultRequest(q Query) (*http.Request, error) {
u := c.url
u.Path = path.Join(u.Path, "query")
jsonParameters, err := json.Marshal(q.Parameters)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", u.String(), nil)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "")
req.Header.Set("User-Agent", c.useragent)
if c.username != "" {
req.SetBasicAuth(c.username, c.password)
}
params := req.URL.Query()
params.Set("q", q.Command)
params.Set("db", q.Database)
if q.RetentionPolicy != "" {
params.Set("rp", q.RetentionPolicy)
}
params.Set("params", string(jsonParameters))
if q.Precision != "" {
params.Set("epoch", q.Precision)
}
req.URL.RawQuery = params.Encode()
return req, nil
}
// duplexReader reads responses and writes it to another writer while
// satisfying the reader interface.
type duplexReader struct {
r io.ReadCloser
w io.Writer
}
func (r *duplexReader) Read(p []byte) (n int, err error) {
n, err = r.r.Read(p)
if err == nil {
r.w.Write(p[:n])
}
return n, err
}
// Close closes the response.
func (r *duplexReader) Close() error {
return r.r.Close()
}
// ChunkedResponse represents a response from the server that
// uses chunking to stream the output.
type ChunkedResponse struct {
dec *json.Decoder
duplex *duplexReader
buf bytes.Buffer
}
// NewChunkedResponse reads a stream and produces responses from the stream.
func NewChunkedResponse(r io.Reader) *ChunkedResponse {
rc, ok := r.(io.ReadCloser)
if !ok {
rc = ioutil.NopCloser(r)
}
resp := &ChunkedResponse{}
resp.duplex = &duplexReader{r: rc, w: &resp.buf}
resp.dec = json.NewDecoder(resp.duplex)
resp.dec.UseNumber()
return resp
}
// NextResponse reads the next line of the stream and returns a response.
func (r *ChunkedResponse) NextResponse() (*Response, error) {
var response Response
if err := r.dec.Decode(&response); err != nil {
if err == io.EOF {
return nil, err
}
// A decoding error happened. This probably means the server crashed
// and sent a last-ditch error message to us. Ensure we have read the
// entirety of the connection to get any remaining error text.
io.Copy(ioutil.Discard, r.duplex)
return nil, errors.New(strings.TrimSpace(r.buf.String()))
}
r.buf.Reset()
return &response, nil
}
// Close closes the response.
func (r *ChunkedResponse) Close() error {
return r.duplex.Close()
}

939
client/v2/client_test.go
View File

@ -1,939 +0,0 @@
package client
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"net/url"
"path"
"reflect"
"strings"
"sync"
"testing"
"time"
)
func TestUDPClient_Query(t *testing.T) {
config := UDPConfig{Addr: "localhost:8089"}
c, err := NewUDPClient(config)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
defer c.Close()
query := Query{}
_, err = c.Query(query)
if err == nil {
t.Error("Querying UDP client should fail")
}
}
func TestUDPClient_Ping(t *testing.T) {
config := UDPConfig{Addr: "localhost:8089"}
c, err := NewUDPClient(config)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
defer c.Close()
rtt, version, err := c.Ping(0)
if rtt != 0 || version != "" || err != nil {
t.Errorf("unexpected error. expected (%v, '%v', %v), actual (%v, '%v', %v)", 0, "", nil, rtt, version, err)
}
}
func TestUDPClient_Write(t *testing.T) {
config := UDPConfig{Addr: "localhost:8089"}
c, err := NewUDPClient(config)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
defer c.Close()
bp, err := NewBatchPoints(BatchPointsConfig{})
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
fields := make(map[string]interface{})
fields["value"] = 1.0
pt, _ := NewPoint("cpu", make(map[string]string), fields)
bp.AddPoint(pt)
err = c.Write(bp)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestUDPClient_BadAddr(t *testing.T) {
config := UDPConfig{Addr: "foobar@wahoo"}
c, err := NewUDPClient(config)
if err == nil {
defer c.Close()
t.Error("Expected resolve error")
}
}
func TestUDPClient_Batches(t *testing.T) {
var logger writeLogger
var cl udpclient
cl.conn = &logger
cl.payloadSize = 20 // should allow for two points per batch
// expected point should look like this: "cpu a=1i"
fields := map[string]interface{}{"a": 1}
p, _ := NewPoint("cpu", nil, fields, time.Time{})
bp, _ := NewBatchPoints(BatchPointsConfig{})
for i := 0; i < 9; i++ {
bp.AddPoint(p)
}
if err := cl.Write(bp); err != nil {
t.Fatalf("Unexpected error during Write: %v", err)
}
if len(logger.writes) != 5 {
t.Errorf("Mismatched write count: got %v, exp %v", len(logger.writes), 5)
}
}
func TestUDPClient_Split(t *testing.T) {
var logger writeLogger
var cl udpclient
cl.conn = &logger
cl.payloadSize = 1 // force one field per point
fields := map[string]interface{}{"a": 1, "b": 2, "c": 3, "d": 4}
p, _ := NewPoint("cpu", nil, fields, time.Unix(1, 0))
bp, _ := NewBatchPoints(BatchPointsConfig{})
bp.AddPoint(p)
if err := cl.Write(bp); err != nil {
t.Fatalf("Unexpected error during Write: %v", err)
}
if len(logger.writes) != len(fields) {
t.Errorf("Mismatched write count: got %v, exp %v", len(logger.writes), len(fields))
}
}
type writeLogger struct {
writes [][]byte
}
func (w *writeLogger) Write(b []byte) (int, error) {
w.writes = append(w.writes, append([]byte(nil), b...))
return len(b), nil
}
func (w *writeLogger) Close() error { return nil }
func TestClient_Query(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data Response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
_, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClient_QueryWithRP(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
params := r.URL.Query()
if got, exp := params.Get("db"), "db0"; got != exp {
t.Errorf("unexpected db query parameter: %s != %s", exp, got)
}
if got, exp := params.Get("rp"), "rp0"; got != exp {
t.Errorf("unexpected rp query parameter: %s != %s", exp, got)
}
var data Response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := NewQueryWithRP("SELECT * FROM m0", "db0", "rp0", "")
_, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClientDownstream500WithBody_Query(t *testing.T) {
const err500page = `<html>
<head>
<title>500 Internal Server Error</title>
</head>
<body>Internal Server Error</body>
</html>`
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(err500page))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
_, err := c.Query(query)
expected := fmt.Sprintf("received status code 500 from downstream server, with response body: %q", err500page)
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClientDownstream500_Query(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
_, err := c.Query(query)
expected := "received status code 500 from downstream server"
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClientDownstream400WithBody_Query(t *testing.T) {
const err403page = `<html>
<head>
<title>403 Forbidden</title>
</head>
<body>Forbidden</body>
</html>`
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(err403page))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
_, err := c.Query(query)
expected := fmt.Sprintf(`expected json response, got "text/html", with status: %v and response body: %q`, http.StatusForbidden, err403page)
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClientDownstream400_Query(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusForbidden)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
_, err := c.Query(query)
expected := fmt.Sprintf(`expected json response, got empty body, with status: %v`, http.StatusForbidden)
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClient500_Query(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Influxdb-Version", "1.3.1")
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(`{"error":"test"}`))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
resp, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected nothing, actual %v", err)
}
if resp.Err != "test" {
t.Errorf(`unexpected response error. expected "test", actual %v`, resp.Err)
}
}
func TestClient_ChunkedQuery(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data Response
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Influxdb-Version", "1.3.1")
w.WriteHeader(http.StatusOK)
enc := json.NewEncoder(w)
_ = enc.Encode(data)
_ = enc.Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, err := NewHTTPClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
query := Query{Chunked: true}
_, err = c.Query(query)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClientDownstream500WithBody_ChunkedQuery(t *testing.T) {
const err500page = `<html>
<head>
<title>500 Internal Server Error</title>
</head>
<body>Internal Server Error</body>
</html>`
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(err500page))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, err := NewHTTPClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
query := Query{Chunked: true}
_, err = c.Query(query)
expected := fmt.Sprintf("received status code 500 from downstream server, with response body: %q", err500page)
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClientDownstream500_ChunkedQuery(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{Chunked: true}
_, err := c.Query(query)
expected := "received status code 500 from downstream server"
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClient500_ChunkedQuery(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Influxdb-Version", "1.3.1")
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(`{"error":"test"}`))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{Chunked: true}
resp, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected nothing, actual %v", err)
}
if resp.Err != "test" {
t.Errorf(`unexpected response error. expected "test", actual %v`, resp.Err)
}
}
func TestClientDownstream400WithBody_ChunkedQuery(t *testing.T) {
const err403page = `<html>
<head>
<title>403 Forbidden</title>
</head>
<body>Forbidden</body>
</html>`
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(err403page))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{Chunked: true}
_, err := c.Query(query)
expected := fmt.Sprintf(`expected json response, got "text/html", with status: %v and response body: %q`, http.StatusForbidden, err403page)
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClientDownstream400_ChunkedQuery(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusForbidden)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{Chunked: true}
_, err := c.Query(query)
expected := fmt.Sprintf(`expected json response, got empty body, with status: %v`, http.StatusForbidden)
if err.Error() != expected {
t.Errorf("unexpected error. expected %v, actual %v", expected, err)
}
}
func TestClient_BoundParameters(t *testing.T) {
var parameterString string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data Response
r.ParseForm()
parameterString = r.FormValue("params")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
expectedParameters := map[string]interface{}{
"testStringParameter": "testStringValue",
"testNumberParameter": 12.3,
}
query := Query{
Parameters: expectedParameters,
}
_, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
var actualParameters map[string]interface{}
err = json.Unmarshal([]byte(parameterString), &actualParameters)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
if !reflect.DeepEqual(expectedParameters, actualParameters) {
t.Errorf("unexpected parameters. expected %v, actual %v", expectedParameters, actualParameters)
}
}
func TestClient_BasicAuth(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
u, p, ok := r.BasicAuth()
if !ok {
t.Errorf("basic auth error")
}
if u != "username" {
t.Errorf("unexpected username, expected %q, actual %q", "username", u)
}
if p != "password" {
t.Errorf("unexpected password, expected %q, actual %q", "password", p)
}
var data Response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL, Username: "username", Password: "password"}
c, _ := NewHTTPClient(config)
defer c.Close()
query := Query{}
_, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClient_Ping(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data Response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusNoContent)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
_, _, err := c.Ping(0)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClient_Concurrent_Use(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{}`))
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
var wg sync.WaitGroup
wg.Add(3)
n := 1000
errC := make(chan error)
go func() {
defer wg.Done()
bp, err := NewBatchPoints(BatchPointsConfig{})
if err != nil {
errC <- fmt.Errorf("got error %v", err)
return
}
for i := 0; i < n; i++ {
if err = c.Write(bp); err != nil {
errC <- fmt.Errorf("got error %v", err)
return
}
}
}()
go func() {
defer wg.Done()
var q Query
for i := 0; i < n; i++ {
if _, err := c.Query(q); err != nil {
errC <- fmt.Errorf("got error %v", err)
return
}
}
}()
go func() {
defer wg.Done()
for i := 0; i < n; i++ {
c.Ping(time.Second)
}
}()
go func() {
wg.Wait()
close(errC)
}()
for err := range errC {
if err != nil {
t.Error(err)
}
}
}
func TestClient_Write(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
in, err := ioutil.ReadAll(r.Body)
if err != nil {
t.Fatalf("unexpected error: %s", err)
} else if have, want := strings.TrimSpace(string(in)), `m0,host=server01 v1=2,v2=2i,v3=2u,v4="foobar",v5=true 0`; have != want {
t.Errorf("unexpected write protocol: %s != %s", have, want)
}
var data Response
w.WriteHeader(http.StatusNoContent)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, _ := NewHTTPClient(config)
defer c.Close()
bp, err := NewBatchPoints(BatchPointsConfig{})
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
pt, err := NewPoint(
"m0",
map[string]string{
"host": "server01",
},
map[string]interface{}{
"v1": float64(2),
"v2": int64(2),
"v3": uint64(2),
"v4": "foobar",
"v5": true,
},
time.Unix(0, 0).UTC(),
)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
bp.AddPoint(pt)
err = c.Write(bp)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClient_UserAgent(t *testing.T) {
receivedUserAgent := ""
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
receivedUserAgent = r.UserAgent()
var data Response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
_, err := http.Get(ts.URL)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
tests := []struct {
name string
userAgent string
expected string
}{
{
name: "Empty user agent",
userAgent: "",
expected: "InfluxDBClient",
},
{
name: "Custom user agent",
userAgent: "Test Influx Client",
expected: "Test Influx Client",
},
}
for _, test := range tests {
config := HTTPConfig{Addr: ts.URL, UserAgent: test.userAgent}
c, _ := NewHTTPClient(config)
defer c.Close()
receivedUserAgent = ""
query := Query{}
_, err = c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
if !strings.HasPrefix(receivedUserAgent, test.expected) {
t.Errorf("Unexpected user agent. expected %v, actual %v", test.expected, receivedUserAgent)
}
receivedUserAgent = ""
bp, _ := NewBatchPoints(BatchPointsConfig{})
err = c.Write(bp)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
if !strings.HasPrefix(receivedUserAgent, test.expected) {
t.Errorf("Unexpected user agent. expected %v, actual %v", test.expected, receivedUserAgent)
}
receivedUserAgent = ""
_, err := c.Query(query)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
if receivedUserAgent != test.expected {
t.Errorf("Unexpected user agent. expected %v, actual %v", test.expected, receivedUserAgent)
}
}
}
func TestClient_PointString(t *testing.T) {
const shortForm = "2006-Jan-02"
time1, _ := time.Parse(shortForm, "2013-Feb-03")
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{"idle": 10.1, "system": 50.9, "user": 39.0}
p, _ := NewPoint("cpu_usage", tags, fields, time1)
s := "cpu_usage,cpu=cpu-total idle=10.1,system=50.9,user=39 1359849600000000000"
if p.String() != s {
t.Errorf("Point String Error, got %s, expected %s", p.String(), s)
}
s = "cpu_usage,cpu=cpu-total idle=10.1,system=50.9,user=39 1359849600000"
if p.PrecisionString("ms") != s {
t.Errorf("Point String Error, got %s, expected %s",
p.PrecisionString("ms"), s)
}
}
func TestClient_PointWithoutTimeString(t *testing.T) {
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{"idle": 10.1, "system": 50.9, "user": 39.0}
p, _ := NewPoint("cpu_usage", tags, fields)
s := "cpu_usage,cpu=cpu-total idle=10.1,system=50.9,user=39"
if p.String() != s {
t.Errorf("Point String Error, got %s, expected %s", p.String(), s)
}
if p.PrecisionString("ms") != s {
t.Errorf("Point String Error, got %s, expected %s",
p.PrecisionString("ms"), s)
}
}
func TestClient_PointName(t *testing.T) {
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{"idle": 10.1, "system": 50.9, "user": 39.0}
p, _ := NewPoint("cpu_usage", tags, fields)
exp := "cpu_usage"
if p.Name() != exp {
t.Errorf("Error, got %s, expected %s",
p.Name(), exp)
}
}
func TestClient_PointTags(t *testing.T) {
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{"idle": 10.1, "system": 50.9, "user": 39.0}
p, _ := NewPoint("cpu_usage", tags, fields)
if !reflect.DeepEqual(tags, p.Tags()) {
t.Errorf("Error, got %v, expected %v",
p.Tags(), tags)
}
}
func TestClient_PointUnixNano(t *testing.T) {
const shortForm = "2006-Jan-02"
time1, _ := time.Parse(shortForm, "2013-Feb-03")
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{"idle": 10.1, "system": 50.9, "user": 39.0}
p, _ := NewPoint("cpu_usage", tags, fields, time1)
exp := int64(1359849600000000000)
if p.UnixNano() != exp {
t.Errorf("Error, got %d, expected %d",
p.UnixNano(), exp)
}
}
func TestClient_PointFields(t *testing.T) {
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{"idle": 10.1, "system": 50.9, "user": 39.0}
p, _ := NewPoint("cpu_usage", tags, fields)
pfields, err := p.Fields()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(fields, pfields) {
t.Errorf("Error, got %v, expected %v",
pfields, fields)
}
}
func TestBatchPoints_PrecisionError(t *testing.T) {
_, err := NewBatchPoints(BatchPointsConfig{Precision: "foobar"})
if err == nil {
t.Errorf("Precision: foobar should have errored")
}
bp, _ := NewBatchPoints(BatchPointsConfig{Precision: "ns"})
err = bp.SetPrecision("foobar")
if err == nil {
t.Errorf("Precision: foobar should have errored")
}
}
func TestBatchPoints_SettersGetters(t *testing.T) {
bp, _ := NewBatchPoints(BatchPointsConfig{
Precision: "ns",
Database: "db",
RetentionPolicy: "rp",
WriteConsistency: "wc",
})
if bp.Precision() != "ns" {
t.Errorf("Expected: %s, got %s", bp.Precision(), "ns")
}
if bp.Database() != "db" {
t.Errorf("Expected: %s, got %s", bp.Database(), "db")
}
if bp.RetentionPolicy() != "rp" {
t.Errorf("Expected: %s, got %s", bp.RetentionPolicy(), "rp")
}
if bp.WriteConsistency() != "wc" {
t.Errorf("Expected: %s, got %s", bp.WriteConsistency(), "wc")
}
bp.SetDatabase("db2")
bp.SetRetentionPolicy("rp2")
bp.SetWriteConsistency("wc2")
err := bp.SetPrecision("s")
if err != nil {
t.Errorf("Did not expect error: %s", err.Error())
}
if bp.Precision() != "s" {
t.Errorf("Expected: %s, got %s", bp.Precision(), "s")
}
if bp.Database() != "db2" {
t.Errorf("Expected: %s, got %s", bp.Database(), "db2")
}
if bp.RetentionPolicy() != "rp2" {
t.Errorf("Expected: %s, got %s", bp.RetentionPolicy(), "rp2")
}
if bp.WriteConsistency() != "wc2" {
t.Errorf("Expected: %s, got %s", bp.WriteConsistency(), "wc2")
}
}
func TestClientConcatURLPath(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.URL.String(), "/influxdbproxy/ping") || strings.Contains(r.URL.String(), "/ping/ping") {
t.Errorf("unexpected error. expected %v contains in %v", "/influxdbproxy/ping", r.URL)
}
var data Response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusNoContent)
_ = json.NewEncoder(w).Encode(data)
}))
defer ts.Close()
url, _ := url.Parse(ts.URL)
url.Path = path.Join(url.Path, "influxdbproxy")
fmt.Println("TestClientConcatURLPath: concat with path 'influxdbproxy' result ", url.String())
c, _ := NewHTTPClient(HTTPConfig{Addr: url.String()})
defer c.Close()
_, _, err := c.Ping(0)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
_, _, err = c.Ping(0)
if err != nil {
t.Errorf("unexpected error. expected %v, actual %v", nil, err)
}
}
func TestClientProxy(t *testing.T) {
pinged := false
ts := httptest.NewServer(http.HandlerFunc(func(resp http.ResponseWriter, req *http.Request) {
if got, want := req.URL.String(), "http://example.com:8086/ping"; got != want {
t.Errorf("invalid url in request: got=%s want=%s", got, want)
}
resp.WriteHeader(http.StatusNoContent)
pinged = true
}))
defer ts.Close()
proxyURL, _ := url.Parse(ts.URL)
c, _ := NewHTTPClient(HTTPConfig{
Addr: "http://example.com:8086",
Proxy: http.ProxyURL(proxyURL),
})
if _, _, err := c.Ping(0); err != nil {
t.Fatalf("could not ping server: %s", err)
}
if !pinged {
t.Fatalf("no http request was received")
}
}
func TestClient_QueryAsChunk(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data Response
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Influxdb-Version", "1.3.1")
w.WriteHeader(http.StatusOK)
enc := json.NewEncoder(w)
_ = enc.Encode(data)
_ = enc.Encode(data)
}))
defer ts.Close()
config := HTTPConfig{Addr: ts.URL}
c, err := NewHTTPClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
query := Query{Chunked: true}
resp, err := c.QueryAsChunk(query)
defer resp.Close()
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
}

265
client/v2/example_test.go
View File

@ -1,265 +0,0 @@
package client_test
import (
"fmt"
"math/rand"
"os"
"time"
"github.com/influxdata/influxdb/client/v2"
)
// Create a new client
func ExampleClient() {
// NOTE: this assumes you've setup a user and have setup shell env variables,
// namely INFLUX_USER/INFLUX_PWD. If not just omit Username/Password below.
_, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
Username: os.Getenv("INFLUX_USER"),
Password: os.Getenv("INFLUX_PWD"),
})
if err != nil {
fmt.Println("Error creating InfluxDB Client: ", err.Error())
}
}
// Write a point using the UDP client
func ExampleClient_uDP() {
// Make client
config := client.UDPConfig{Addr: "localhost:8089"}
c, err := client.NewUDPClient(config)
if err != nil {
fmt.Println("Error: ", err.Error())
}
defer c.Close()
// Create a new point batch
bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
Precision: "s",
})
// Create a point and add to batch
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err != nil {
fmt.Println("Error: ", err.Error())
}
bp.AddPoint(pt)
// Write the batch
c.Write(bp)
}
// Ping the cluster using the HTTP client
func ExampleClient_Ping() {
// Make client
c, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
})
if err != nil {
fmt.Println("Error creating InfluxDB Client: ", err.Error())
}
defer c.Close()
_, _, err = c.Ping(0)
if err != nil {
fmt.Println("Error pinging InfluxDB Cluster: ", err.Error())
}
}
// Write a point using the HTTP client
func ExampleClient_write() {
// Make client
c, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
})
if err != nil {
fmt.Println("Error creating InfluxDB Client: ", err.Error())
}
defer c.Close()
// Create a new point batch
bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
Database: "BumbleBeeTuna",
Precision: "s",
})
// Create a point and add to batch
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err != nil {
fmt.Println("Error: ", err.Error())
}
bp.AddPoint(pt)
// Write the batch
c.Write(bp)
}
// Create a batch and add a point
func ExampleBatchPoints() {
// Create a new point batch
bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
Database: "BumbleBeeTuna",
Precision: "s",
})
// Create a point and add to batch
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err != nil {
fmt.Println("Error: ", err.Error())
}
bp.AddPoint(pt)
}
// Using the BatchPoints setter functions
func ExampleBatchPoints_setters() {
// Create a new point batch
bp, _ := client.NewBatchPoints(client.BatchPointsConfig{})
bp.SetDatabase("BumbleBeeTuna")
bp.SetPrecision("ms")
// Create a point and add to batch
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err != nil {
fmt.Println("Error: ", err.Error())
}
bp.AddPoint(pt)
}
// Create a new point with a timestamp
func ExamplePoint() {
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields, time.Now())
if err == nil {
fmt.Println("We created a point: ", pt.String())
}
}
// Create a new point without a timestamp
func ExamplePoint_withoutTime() {
tags := map[string]string{"cpu": "cpu-total"}
fields := map[string]interface{}{
"idle": 10.1,
"system": 53.3,
"user": 46.6,
}
pt, err := client.NewPoint("cpu_usage", tags, fields)
if err == nil {
fmt.Println("We created a point w/o time: ", pt.String())
}
}
// Write 1000 points
func ExampleClient_write1000() {
sampleSize := 1000
// Make client
c, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
})
if err != nil {
fmt.Println("Error creating InfluxDB Client: ", err.Error())
}
defer c.Close()
rand.Seed(42)
bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
Database: "systemstats",
Precision: "us",
})
for i := 0; i < sampleSize; i++ {
regions := []string{"us-west1", "us-west2", "us-west3", "us-east1"}
tags := map[string]string{
"cpu": "cpu-total",
"host": fmt.Sprintf("host%d", rand.Intn(1000)),
"region": regions[rand.Intn(len(regions))],
}
idle := rand.Float64() * 100.0
fields := map[string]interface{}{
"idle": idle,
"busy": 100.0 - idle,
}
pt, err := client.NewPoint(
"cpu_usage",
tags,
fields,
time.Now(),
)
if err != nil {
println("Error:", err.Error())
continue
}
bp.AddPoint(pt)
}
err = c.Write(bp)
if err != nil {
fmt.Println("Error: ", err.Error())
}
}
// Make a Query
func ExampleClient_query() {
// Make client
c, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
})
if err != nil {
fmt.Println("Error creating InfluxDB Client: ", err.Error())
}
defer c.Close()
q := client.NewQuery("SELECT count(value) FROM shapes", "square_holes", "ns")
if response, err := c.Query(q); err == nil && response.Error() == nil {
fmt.Println(response.Results)
}
}
// Create a Database with a query
func ExampleClient_createDatabase() {
// Make client
c, err := client.NewHTTPClient(client.HTTPConfig{
Addr: "http://localhost:8086",
})
if err != nil {
fmt.Println("Error creating InfluxDB Client: ", err.Error())
}
defer c.Close()
q := client.NewQuery("CREATE DATABASE telegraf", "", "")
if response, err := c.Query(q); err == nil && response.Error() == nil {
fmt.Println(response.Results)
}
}

116
client/v2/udp.go
View File

@ -1,116 +0,0 @@
package client
import (
"fmt"
"io"
"net"
"time"
)
const (
// UDPPayloadSize is a reasonable default payload size for UDP packets that
// could be travelling over the internet.
UDPPayloadSize = 512
)
// UDPConfig is the config data needed to create a UDP Client.
type UDPConfig struct {
// Addr should be of the form "host:port"
// or "[ipv6-host%zone]:port".
Addr string
// PayloadSize is the maximum size of a UDP client message, optional
// Tune this based on your network. Defaults to UDPPayloadSize.
PayloadSize int
}
// NewUDPClient returns a client interface for writing to an InfluxDB UDP
// service from the given config.
func NewUDPClient(conf UDPConfig) (Client, error) {
var udpAddr *net.UDPAddr
udpAddr, err := net.ResolveUDPAddr("udp", conf.Addr)
if err != nil {
return nil, err
}
conn, err := net.DialUDP("udp", nil, udpAddr)
if err != nil {
return nil, err
}
payloadSize := conf.PayloadSize
if payloadSize == 0 {
payloadSize = UDPPayloadSize
}
return &udpclient{
conn: conn,
payloadSize: payloadSize,
}, nil
}
// Close releases the udpclient's resources.
func (uc *udpclient) Close() error {
return uc.conn.Close()
}
type udpclient struct {
conn io.WriteCloser
payloadSize int
}
func (uc *udpclient) Write(bp BatchPoints) error {
var b = make([]byte, 0, uc.payloadSize) // initial buffer size, it will grow as needed
var d, _ = time.ParseDuration("1" + bp.Precision())
var delayedError error
var checkBuffer = func(n int) {
if len(b) > 0 && len(b)+n > uc.payloadSize {
if _, err := uc.conn.Write(b); err != nil {
delayedError = err
}
b = b[:0]
}
}
for _, p := range bp.Points() {
p.pt.Round(d)
pointSize := p.pt.StringSize() + 1 // include newline in size
//point := p.pt.RoundedString(d) + "\n"
checkBuffer(pointSize)
if p.Time().IsZero() || pointSize <= uc.payloadSize {
b = p.pt.AppendString(b)
b = append(b, '\n')
continue
}
points := p.pt.Split(uc.payloadSize - 1) // account for newline character
for _, sp := range points {
checkBuffer(sp.StringSize() + 1)
b = sp.AppendString(b)
b = append(b, '\n')
}
}
if len(b) > 0 {
if _, err := uc.conn.Write(b); err != nil {
return err
}
}
return delayedError
}
func (uc *udpclient) Query(q Query) (*Response, error) {
return nil, fmt.Errorf("Querying via UDP is not supported")
}
func (uc *udpclient) QueryAsChunk(q Query) (*ChunkedResponse, error) {
return nil, fmt.Errorf("Querying via UDP is not supported")
}
func (uc *udpclient) Ping(timeout time.Duration) (time.Duration, string, error) {
return 0, "", nil
}

1232
cmd/influx/cli/cli.go
View File

File diff suppressed because it is too large Load Diff

58
cmd/influx/cli/cli_internal_test.go
View File

@ -1,58 +0,0 @@
package cli
import "testing"
func TestParseCommand_InsertInto(t *testing.T) {
t.Parallel()
c := CommandLine{}
tests := []struct {
cmd, db, rp string
}{
{
cmd: `INSERT INTO test cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test",
},
{
cmd: ` INSERT INTO .test cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test",
},
{
cmd: `INSERT INTO "test test" cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test test",
},
{
cmd: `Insert iNTO test.test cpu,host=serverA,region=us-west value=1.0`,
db: "test",
rp: "test",
},
{
cmd: `insert into "test test" cpu,host=serverA,region=us-west value=1.0`,
db: "",
rp: "test test",
},
{
cmd: `insert into "d b"."test test" cpu,host=serverA,region=us-west value=1.0`,
db: "d b",
rp: "test test",
},
}
for _, test := range tests {
t.Logf("command: %s", test.cmd)
bp, err := c.parseInsert(test.cmd)
if err != nil {
t.Fatal(err)
}
if bp.Database != test.db {
t.Fatalf(`Command "insert into" db parsing failed, expected: %q, actual: %q`, test.db, bp.Database)
}
if bp.RetentionPolicy != test.rp {
t.Fatalf(`Command "insert into" rp parsing failed, expected: %q, actual: %q`, test.rp, bp.RetentionPolicy)
}
}
}

634
cmd/influx/cli/cli_test.go
View File

@ -1,634 +0,0 @@
package cli_test
import (
"bufio"
"bytes"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"strings"
"testing"
"github.com/influxdata/influxdb/client"
"github.com/influxdata/influxdb/cmd/influx/cli"
"github.com/influxdata/influxql"
"github.com/peterh/liner"
)
const (
CLIENT_VERSION = "y.y"
SERVER_VERSION = "x.x"
)
func TestNewCLI(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
if c == nil {
t.Fatal("CommandLine shouldn't be nil.")
}
if c.ClientVersion != CLIENT_VERSION {
t.Fatalf("CommandLine version is %s but should be %s", c.ClientVersion, CLIENT_VERSION)
}
}
func TestRunCLI_ExecuteInsert(t *testing.T) {
t.Parallel()
ts := emptyTestServer()
defer ts.Close()
u, _ := url.Parse(ts.URL)
h, p, _ := net.SplitHostPort(u.Host)
c := cli.New(CLIENT_VERSION)
c.Host = h
c.Port, _ = strconv.Atoi(p)
c.ClientConfig.Precision = "ms"
c.Execute = "INSERT sensor,floor=1 value=2"
c.IgnoreSignals = true
c.ForceTTY = true
if err := c.Run(); err != nil {
t.Fatalf("Run failed with error: %s", err)
}
}
func TestRunCLI_ExecuteInsert_WithSignals(t *testing.T) {
t.Parallel()
ts := emptyTestServer()
defer ts.Close()
u, _ := url.Parse(ts.URL)
h, p, _ := net.SplitHostPort(u.Host)
c := cli.New(CLIENT_VERSION)
c.Host = h
c.Port, _ = strconv.Atoi(p)
c.ClientConfig.Precision = "ms"
c.Execute = "INSERT sensor,floor=1 value=2"
c.IgnoreSignals = false
c.ForceTTY = true
if err := c.Run(); err != nil {
t.Fatalf("Run failed with error: %s", err)
}
}
func TestSetAuth(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
config := client.NewConfig()
client, _ := client.NewClient(config)
c.Client = client
u := "userx"
p := "pwdy"
c.SetAuth("auth " + u + " " + p)
// validate CLI configuration
if c.ClientConfig.Username != u {
t.Fatalf("Username is %s but should be %s", c.ClientConfig.Username, u)
}
if c.ClientConfig.Password != p {
t.Fatalf("Password is %s but should be %s", c.ClientConfig.Password, p)
}
}
func TestSetPrecision(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
config := client.NewConfig()
client, _ := client.NewClient(config)
c.Client = client
// validate set non-default precision
p := "ns"
c.SetPrecision("precision " + p)
if c.ClientConfig.Precision != p {
t.Fatalf("Precision is %s but should be %s", c.ClientConfig.Precision, p)
}
up := "NS"
c.SetPrecision("PRECISION " + up)
if c.ClientConfig.Precision != p {
t.Fatalf("Precision is %s but should be %s", c.ClientConfig.Precision, p)
}
mixed := "ns"
c.SetPrecision("PRECISION " + mixed)
if c.ClientConfig.Precision != p {
t.Fatalf("Precision is %s but should be %s", c.ClientConfig.Precision, p)
}
// validate set default precision which equals empty string
p = "rfc3339"
c.SetPrecision("precision " + p)
if c.ClientConfig.Precision != "" {
t.Fatalf("Precision is %s but should be empty", c.ClientConfig.Precision)
}
p = "RFC3339"
c.SetPrecision("precision " + p)
if c.ClientConfig.Precision != "" {
t.Fatalf("Precision is %s but should be empty", c.ClientConfig.Precision)
}
}
func TestSetFormat(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
config := client.NewConfig()
client, _ := client.NewClient(config)
c.Client = client
// validate set non-default format
f := "json"
c.SetFormat("format " + f)
if c.Format != f {
t.Fatalf("Format is %s but should be %s", c.Format, f)
}
uf := "JSON"
c.SetFormat("format " + uf)
if c.Format != f {
t.Fatalf("Format is %s but should be %s", c.Format, f)
}
mixed := "json"
c.SetFormat("FORMAT " + mixed)
if c.Format != f {
t.Fatalf("Format is %s but should be %s", c.Format, f)
}
}
func Test_SetChunked(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
config := client.NewConfig()
client, _ := client.NewClient(config)
c.Client = client
// make sure chunked is on by default
if got, exp := c.Chunked, true; got != exp {
t.Fatalf("chunked should be on by default. got %v, exp %v", got, exp)
}
// turn chunked off
if err := c.ParseCommand("Chunked"); err != nil {
t.Fatalf("setting chunked failed: err: %s", err)
}
if got, exp := c.Chunked, false; got != exp {
t.Fatalf("setting chunked failed. got %v, exp %v", got, exp)
}
// turn chunked back on
if err := c.ParseCommand("Chunked"); err != nil {
t.Fatalf("setting chunked failed: err: %s", err)
}
if got, exp := c.Chunked, true; got != exp {
t.Fatalf("setting chunked failed. got %v, exp %v", got, exp)
}
}
func Test_SetChunkSize(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
config := client.NewConfig()
client, _ := client.NewClient(config)
c.Client = client
// check default chunk size
if got, exp := c.ChunkSize, 0; got != exp {
t.Fatalf("unexpected chunk size. got %d, exp %d", got, exp)
}
tests := []struct {
command string
exp int
}{
{"chunk size 20", 20},
{" CHunk siZE 55 ", 55},
{"chunk 10", 10},
{" chuNK 15", 15},
{"chunk size -60", 0},
{"chunk size 10", 10},
{"chunk size 0", 0},
{"chunk size 10", 10},
{"chunk size junk", 10},
}
for _, test := range tests {
if err := c.ParseCommand(test.command); err != nil {
t.Logf("command: %q", test.command)
t.Fatalf("setting chunked failed: err: %s", err)
}
if got, exp := c.ChunkSize, test.exp; got != exp {
t.Logf("command: %q", test.command)
t.Fatalf("unexpected chunk size. got %d, exp %d", got, exp)
}
}
}
func TestSetWriteConsistency(t *testing.T) {
t.Parallel()
c := cli.New(CLIENT_VERSION)
config := client.NewConfig()
client, _ := client.NewClient(config)
c.Client = client
// set valid write consistency
consistency := "all"
c.SetWriteConsistency("consistency " + consistency)
if c.ClientConfig.WriteConsistency != consistency {
t.Fatalf("WriteConsistency is %s but should be %s", c.ClientConfig.WriteConsistency, consistency)
}
// set different valid write consistency and validate change
consistency = "quorum"
c.SetWriteConsistency("consistency " + consistency)
if c.ClientConfig.WriteConsistency != consistency {
t.Fatalf("WriteConsistency is %s but should be %s", c.ClientConfig.WriteConsistency, consistency)
}
consistency = "QUORUM"
c.SetWriteConsistency("consistency " + consistency)
if c.ClientConfig.WriteConsistency != "quorum" {
t.Fatalf("WriteConsistency is %s but should be %s", c.ClientConfig.WriteConsistency, "quorum")
}
consistency = "quorum"
c.SetWriteConsistency("CONSISTENCY " + consistency)
if c.ClientConfig.WriteConsistency != consistency {
t.Fatalf("WriteConsistency is %s but should be %s", c.ClientConfig.WriteConsistency, consistency)
}
// set invalid write consistency and verify there was no change
invalidConsistency := "invalid_consistency"
c.SetWriteConsistency("consistency " + invalidConsistency)
if c.ClientConfig.WriteConsistency == invalidConsistency {
t.Fatalf("WriteConsistency is %s but should be %s", c.ClientConfig.WriteConsistency, consistency)
}
}
func TestParseCommand_CommandsExist(t *testing.T) {
t.Parallel()
c, err := client.NewClient(client.Config{})
if err != nil {
t.Fatalf("unexpected error %v", err)
}
m := cli.CommandLine{Client: c, Line: liner.NewLiner()}
tests := []struct {
cmd string
}{
{cmd: "gopher"},
{cmd: "auth"},
{cmd: "help"},
{cmd: "format"},
{cmd: "precision"},
{cmd: "settings"},
}
for _, test := range tests {
if err := m.ParseCommand(test.cmd); err != nil {
t.Fatalf(`Got error %v for command %q, expected nil`, err, test.cmd)
}
}
}
func TestParseCommand_Connect(t *testing.T) {
t.Parallel()
ts := emptyTestServer()
defer ts.Close()
u, _ := url.Parse(ts.URL)
cmd := "connect " + u.Host
c := cli.CommandLine{}
// assert connection is established
if err := c.ParseCommand(cmd); err != nil {
t.Fatalf("There was an error while connecting to %v: %v", u.Path, err)
}
// assert server version is populated
if c.ServerVersion != SERVER_VERSION {
t.Fatalf("Server version is %s but should be %s.", c.ServerVersion, SERVER_VERSION)
}
}
func TestParseCommand_TogglePretty(t *testing.T) {
t.Parallel()
c := cli.CommandLine{}
if c.Pretty {
t.Fatalf(`Pretty should be false.`)
}
c.ParseCommand("pretty")
if !c.Pretty {
t.Fatalf(`Pretty should be true.`)
}
c.ParseCommand("pretty")
if c.Pretty {
t.Fatalf(`Pretty should be false.`)
}
}
func TestParseCommand_Exit(t *testing.T) {
t.Parallel()
tests := []struct {
cmd string
}{
{cmd: "exit"},
{cmd: " exit"},
{cmd: "exit "},
{cmd: "Exit "},
}
for _, test := range tests {
c := cli.CommandLine{Quit: make(chan struct{}, 1)}
c.ParseCommand(test.cmd)
// channel should be closed
if _, ok := <-c.Quit; ok {
t.Fatalf(`Command "exit" failed for %q.`, test.cmd)
}
}
}
func TestParseCommand_Quit(t *testing.T) {
t.Parallel()
tests := []struct {
cmd string
}{
{cmd: "quit"},
{cmd: " quit"},
{cmd: "quit "},
{cmd: "Quit "},
}
for _, test := range tests {
c := cli.CommandLine{Quit: make(chan struct{}, 1)}
c.ParseCommand(test.cmd)
// channel should be closed
if _, ok := <-c.Quit; ok {
t.Fatalf(`Command "quit" failed for %q.`, test.cmd)
}
}
}
func TestParseCommand_Use(t *testing.T) {
t.Parallel()
ts := emptyTestServer()
defer ts.Close()
u, _ := url.Parse(ts.URL)
config := client.Config{URL: *u}
c, err := client.NewClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
tests := []struct {
cmd string
db string
}{
{cmd: "use db", db: "db"},
{cmd: " use db", db: "db"},
{cmd: "use db ", db: "db"},
{cmd: "use db;", db: "db"},
{cmd: "use db; ", db: "db"},
{cmd: "Use db", db: "db"},
{cmd: `Use "db"`, db: "db"},
{cmd: `Use "db db"`, db: "db db"},
}
for _, test := range tests {
m := cli.CommandLine{Client: c}
if err := m.ParseCommand(test.cmd); err != nil {
t.Fatalf(`Got error %v for command %q, expected nil.`, err, test.cmd)
}
if m.Database != test.db {
t.Fatalf(`Command "%s" changed database to %q. Expected %s`, test.cmd, m.Database, test.db)
}
}
}
func TestParseCommand_UseAuth(t *testing.T) {
t.Parallel()
ts := emptyTestServer()
defer ts.Close()
u, _ := url.Parse(ts.URL)
tests := []struct {
cmd string
user string
database string
}{
{
cmd: "use db",
user: "admin",
database: "db",
},
{
cmd: "use blank",
user: "admin",
database: "",
},
{
cmd: "use db",
user: "anonymous",
database: "db",
},
{
cmd: "use blank",
user: "anonymous",
database: "blank",
},
}
for i, tt := range tests {
config := client.Config{URL: *u, Username: tt.user}
fmt.Println("using auth:", tt.user)
c, err := client.NewClient(config)
if err != nil {
t.Errorf("%d. unexpected error. expected %v, actual %v", i, nil, err)
continue
}
m := cli.CommandLine{Client: c}
m.ClientConfig.Username = tt.user
if err := m.ParseCommand(tt.cmd); err != nil {
t.Fatalf(`%d. Got error %v for command %q, expected nil.`, i, err, tt.cmd)
}
if m.Database != tt.database {
t.Fatalf(`%d. Command "use" changed database to %q. Expected %q`, i, m.Database, tt.database)
}
}
}
func TestParseCommand_Consistency(t *testing.T) {
t.Parallel()
c := cli.CommandLine{}
tests := []struct {
cmd string
}{
{cmd: "consistency one"},
{cmd: " consistency one"},
{cmd: "consistency one "},
{cmd: "consistency one;"},
{cmd: "consistency one; "},
{cmd: "Consistency one"},
}
for _, test := range tests {
if err := c.ParseCommand(test.cmd); err != nil {
t.Fatalf(`Got error %v for command %q, expected nil.`, err, test.cmd)
}
if c.ClientConfig.WriteConsistency != "one" {
t.Fatalf(`Command "consistency" changed consistency to %q. Expected one`, c.ClientConfig.WriteConsistency)
}
}
}
func TestParseCommand_Insert(t *testing.T) {
t.Parallel()
ts := emptyTestServer()
defer ts.Close()
u, _ := url.Parse(ts.URL)
config := client.Config{URL: *u}
c, err := client.NewClient(config)
if err != nil {
t.Fatalf("unexpected error. expected %v, actual %v", nil, err)
}
m := cli.CommandLine{Client: c}
tests := []struct {
cmd string
}{
{cmd: "INSERT cpu,host=serverA,region=us-west value=1.0"},
{cmd: " INSERT cpu,host=serverA,region=us-west value=1.0"},
{cmd: "INSERT cpu,host=serverA,region=us-west value=1.0"},
{cmd: "insert cpu,host=serverA,region=us-west value=1.0 "},
{cmd: "insert"},
{cmd: "Insert "},
{cmd: "insert c"},
{cmd: "insert int"},
}
for _, test := range tests {
if err := m.ParseCommand(test.cmd); err != nil {
t.Fatalf(`Got error %v for command %q, expected nil.`, err, test.cmd)
}
}
}
func TestParseCommand_History(t *testing.T) {
t.Parallel()
c := cli.CommandLine{Line: liner.NewLiner()}
defer c.Line.Close()
// append one entry to history
c.Line.AppendHistory("abc")
tests := []struct {
cmd string
}{
{cmd: "history"},
{cmd: " history"},
{cmd: "history "},
{cmd: "History "},
}
for _, test := range tests {
if err := c.ParseCommand(test.cmd); err != nil {
t.Fatalf(`Got error %v for command %q, expected nil.`, err, test.cmd)
}
}
// buf size should be at least 1
var buf bytes.Buffer
c.Line.WriteHistory(&buf)
if buf.Len() < 1 {
t.Fatal("History is borked")
}
}
func TestParseCommand_HistoryWithBlankCommand(t *testing.T) {
t.Parallel()
c := cli.CommandLine{Line: liner.NewLiner()}
defer c.Line.Close()
// append one entry to history
c.Line.AppendHistory("x")
tests := []struct {
cmd string
err error
}{
{cmd: "history"},
{cmd: " history"},
{cmd: "history "},
{cmd: "", err: cli.ErrBlankCommand}, // shouldn't be persisted in history
{cmd: " ", err: cli.ErrBlankCommand}, // shouldn't be persisted in history
{cmd: " ", err: cli.ErrBlankCommand}, // shouldn't be persisted in history
}
// a blank command will return cli.ErrBlankCommand.
for _, test := range tests {
if err := c.ParseCommand(test.cmd); err != test.err {
t.Errorf(`Got error %v for command %q, expected %v`, err, test.cmd, test.err)
}
}
// buf shall not contain empty commands
var buf bytes.Buffer
c.Line.WriteHistory(&buf)
scanner := bufio.NewScanner(&buf)
for scanner.Scan() {
if strings.TrimSpace(scanner.Text()) == "" {
t.Fatal("Empty commands should not be persisted in history.")
}
}
}
// helper methods
func emptyTestServer() *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Influxdb-Version", SERVER_VERSION)
// Fake authorization entirely based on the username.
authorized := false
user, _, _ := r.BasicAuth()
switch user {
case "", "admin":
authorized = true
}
switch r.URL.Path {
case "/query":
values := r.URL.Query()
parser := influxql.NewParser(bytes.NewBufferString(values.Get("q")))
q, err := parser.ParseQuery()
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
stmt := q.Statements[0]
switch stmt.(type) {
case *influxql.ShowDatabasesStatement:
if authorized {
io.WriteString(w, `{"results":[{"series":[{"name":"databases","columns":["name"],"values":[["db", "db db"]]}]}]}`)
} else {
w.WriteHeader(http.StatusUnauthorized)
io.WriteString(w, fmt.Sprintf(`{"error":"error authorizing query: %s not authorized to execute statement 'SHOW DATABASES', requires admin privilege"}`, user))
}
case *influxql.ShowDiagnosticsStatement:
io.WriteString(w, `{"results":[{}]}`)
}
case "/write":
w.WriteHeader(http.StatusOK)
}
}))
}

39
cmd/influx/cli/flux.go
View File

@ -1,39 +0,0 @@
package cli
import (
"context"
"github.com/influxdata/flux"
"github.com/influxdata/flux/csv"
"github.com/influxdata/flux/repl"
_ "github.com/influxdata/influxdb/flux/builtin"
"github.com/influxdata/influxdb/flux/client"
)
// QueryService represents a type capable of performing queries.
type fluxClient interface {
// Query submits a query for execution returning a results iterator.
// Cancel must be called on any returned results to free resources.
Query(ctx context.Context, req *client.ProxyRequest) (flux.ResultIterator, error)
}
// replQuerier implements the repl.Querier interface while consuming a fluxClient
type replQuerier struct {
client fluxClient
}
func (q *replQuerier) Query(ctx context.Context, compiler flux.Compiler) (flux.ResultIterator, error) {
req := &client.ProxyRequest{
Compiler: compiler,
Dialect: csv.DefaultDialect(),
}
return q.client.Query(ctx, req)
}
func getFluxREPL(host string, port int, ssl bool) (*repl.REPL, error) {
c, err := client.NewHTTP(host, port, ssl)
if err != nil {
return nil, err
}
return repl.New(&replQuerier{client: c}), nil
}

34
cmd/influx/cli/parser.go
View File

@ -1,34 +0,0 @@
package cli
import (
"bytes"
"fmt"
)
func parseDatabaseAndRetentionPolicy(stmt []byte) (string, string, error) {
var db, rp []byte
var quoted bool
var seperatorCount int
stmt = bytes.TrimSpace(stmt)
for _, b := range stmt {
if b == '"' {
quoted = !quoted
continue
}
if b == '.' && !quoted {
seperatorCount++
if seperatorCount > 1 {
return "", "", fmt.Errorf("unable to parse database and retention policy from %s", string(stmt))
}
continue
}
if seperatorCount == 1 {
rp = append(rp, b)
continue
}
db = append(db, b)
}
return string(db), string(rp), nil
}

90
cmd/influx/cli/parser_internal_test.go
View File

@ -1,90 +0,0 @@
package cli
import (
"errors"
"testing"
)
func Test_parseDatabaseAndretentionPolicy(t *testing.T) {
tests := []struct {
stmt string
db string
rp string
err error
}{
{
stmt: `foo`,
db: "foo",
},
{
stmt: `"foo.bar"`,
db: "foo.bar",
},
{
stmt: `"foo.bar".`,
db: "foo.bar",
},
{
stmt: `."foo.bar"`,
rp: "foo.bar",
},
{
stmt: `foo.bar`,
db: "foo",
rp: "bar",
},
{
stmt: `"foo".bar`,
db: "foo",
rp: "bar",
},
{
stmt: `"foo"."bar"`,
db: "foo",
rp: "bar",
},
{
stmt: `"foo.bin"."bar"`,
db: "foo.bin",
rp: "bar",
},
{
stmt: `"foo.bin"."bar.baz...."`,
db: "foo.bin",
rp: "bar.baz....",
},
{
stmt: ` "foo.bin"."bar.baz...." `,
db: "foo.bin",
rp: "bar.baz....",
},
{
stmt: `"foo.bin"."bar".boom`,
err: errors.New("foo"),
},
{
stmt: "foo.bar.",
err: errors.New("foo"),
},
}
for _, test := range tests {
db, rp, err := parseDatabaseAndRetentionPolicy([]byte(test.stmt))
if err != nil && test.err == nil {
t.Errorf("unexpected error: got %s", err)
continue
}
if test.err != nil && err == nil {
t.Errorf("expected err: got: nil, exp: %s", test.err)
continue
}
if db != test.db {
t.Errorf("unexpected database: got: %s, exp: %s", db, test.db)
}
if rp != test.rp {
t.Errorf("unexpected retention policy: got: %s, exp: %s", rp, test.rp)
}
}
}

131
cmd/influx/main.go
View File

@ -1,131 +0,0 @@
// The influx command is a CLI client to InfluxDB.
package main
import (
"flag"
"fmt"
"os"
"strings"
"github.com/influxdata/influxdb/client"
"github.com/influxdata/influxdb/cmd/influx/cli"
)
// These variables are populated via the Go linker.
var (
version string
)
const (
// defaultFormat is the default format of the results when issuing queries
defaultFormat = "column"
// defaultPrecision is the default timestamp format of the results when issuing queries
defaultPrecision = "ns"
// defaultPPS is the default points per second that the import will throttle at
// by default it's 0, which means it will not throttle
defaultPPS = 0
)
func init() {
// If version is not set, make that clear.
if version == "" {
version = "unknown"
}
}
func main() {
c := cli.New(version)
fs := flag.NewFlagSet("InfluxDB shell version "+version, flag.ExitOnError)
fs.StringVar(&c.Host, "host", client.DefaultHost, "Influxdb host to connect to.")
fs.IntVar(&c.Port, "port", client.DefaultPort, "Influxdb port to connect to.")
fs.StringVar(&c.ClientConfig.UnixSocket, "socket", "", "Influxdb unix socket to connect to.")
fs.StringVar(&c.ClientConfig.Username, "username", "", "Username to connect to the server.")
fs.StringVar(&c.ClientConfig.Password, "password", "", `Password to connect to the server. Leaving blank will prompt for password (--password="").`)
fs.StringVar(&c.Database, "database", c.Database, "Database to connect to the server.")
fs.Var(&c.Type, "type", "query language for executing commands or invoking the REPL: influxql, flux")
fs.BoolVar(&c.Ssl, "ssl", false, "Use https for connecting to cluster.")
fs.BoolVar(&c.ClientConfig.UnsafeSsl, "unsafeSsl", false, "Set this when connecting to the cluster using https and not use SSL verification.")
fs.StringVar(&c.Format, "format", defaultFormat, "Format specifies the format of the server responses: json, csv, or column.")
fs.StringVar(&c.ClientConfig.Precision, "precision", defaultPrecision, "Precision specifies the format of the timestamp: rfc3339,h,m,s,ms,u or ns.")
fs.StringVar(&c.ClientConfig.WriteConsistency, "consistency", "all", "Set write consistency level: any, one, quorum, or all.")
fs.BoolVar(&c.Pretty, "pretty", false, "Turns on pretty print for the json format.")
fs.IntVar(&c.NodeID, "node", 0, "Specify the node that data should be retrieved from (enterprise only).")
fs.StringVar(&c.Execute, "execute", c.Execute, "Execute command and quit.")
fs.BoolVar(&c.ShowVersion, "version", false, "Displays the InfluxDB version.")
fs.BoolVar(&c.Import, "import", false, "Import a previous database.")
fs.IntVar(&c.ImporterConfig.PPS, "pps", defaultPPS, "How many points per second the import will allow. By default it is zero and will not throttle importing.")
fs.StringVar(&c.ImporterConfig.Path, "path", "", "path to the file to import")
fs.BoolVar(&c.ImporterConfig.Compressed, "compressed", false, "set to true if the import file is compressed")
// Define our own custom usage to print
fs.Usage = func() {
fmt.Println(`Usage of influx:
-version
Display the version and exit.
-host 'host name'
Host to connect to.
-port 'port #'
Port to connect to.
-socket 'unix domain socket'
Unix socket to connect to.
-database 'database name'
Database to connect to the server.
-password 'password'
Password to connect to the server. Leaving blank will prompt for password (--password '').
-username 'username'
Username to connect to the server.
-ssl
Use https for requests.
-unsafeSsl
Set this when connecting to the cluster using https and not use SSL verification.
-execute 'command'
Execute command and quit.
-type 'influxql|flux'
Type specifies the query language for executing commands or when invoking the REPL.
-format 'json|csv|column'
Format specifies the format of the server responses: json, csv, or column.
-precision 'rfc3339|h|m|s|ms|u|ns'
Precision specifies the format of the timestamp: rfc3339, h, m, s, ms, u or ns.
-consistency 'any|one|quorum|all'
Set write consistency level: any, one, quorum, or all
-pretty
Turns on pretty print for the json format.
-import
Import a previous database export from file
-pps
How many points per second the import will allow. By default it is zero and will not throttle importing.
-path
Path to file to import
-compressed
Set to true if the import file is compressed
Examples:
# Use influx in a non-interactive mode to query the database "metrics" and pretty print json:
$ influx -database 'metrics' -execute 'select * from cpu' -format 'json' -pretty
# Connect to a specific database on startup and set database context:
$ influx -database 'metrics' -host 'localhost' -port '8086'`)
}
fs.Parse(os.Args[1:])
argsNotParsed := fs.Args()
if len(argsNotParsed) > 0 {
fmt.Fprintf(os.Stderr, "unknown arguments: %s\n", strings.Join(argsNotParsed, " "))
fs.Usage()
os.Exit(1)
}
if c.ShowVersion {
c.Version()
os.Exit(0)
}
if err := c.Run(); err != nil {
fmt.Fprintf(os.Stderr, "%s\n", err)
os.Exit(1)
}
}

107
cmd/influx_inspect/README.md
View File

@ -1,107 +0,0 @@
# `influx_inspect`
## Ways to run
### `influx_inspect`
Will print usage for the tool.
### `influx_inspect report`
Displays series meta-data for all shards. Default location [$HOME/.influxdb]
### `influx_inspect dumptsm`
Dumps low-level details about tsm1 files
#### Flags
##### `-index` bool
Dump raw index data.
`default` = false
#### `-blocks` bool
Dump raw block data.
`default` = false
#### `-all`
Dump all data. Caution: This may print a lot of information.
`default` = false
#### `-filter-key`
Only display index and block data match this key substring.
`default` = ""
### `influx_inspect export`
Exports all tsm files to line protocol. This output file can be imported via the [influx](https://github.com/influxdata/influxdb/tree/master/importer#running-the-import-command) command.
#### `-datadir` string
Data storage path.
`default` = "$HOME/.influxdb/data"
#### `-waldir` string
WAL storage path.
`default` = "$HOME/.influxdb/wal"
#### `-out` string
Destination file to export to
`default` = "$HOME/.influxdb/export"
#### `-database` string (optional)
Database to export.
`default` = ""
#### `-retention` string (optional)
Retention policy to export.
`default` = ""
#### `-start` string (optional)
Optional. The time range to start at.
#### `-end` string (optional)
Optional. The time range to end at.
#### `-compress` bool (optional)
Compress the output.
`default` = false
#### Sample Commands
Export entire database and compress output:
```
influx_inspect export --compress
```
Export specific retention policy:
```
influx_inspect export --database mydb --retention autogen
```
##### Sample Data
This is a sample of what the output will look like.
```
# DDL
CREATE DATABASE MY_DB_NAME
CREATE RETENTION POLICY autogen ON MY_DB_NAME DURATION inf REPLICATION 1
# DML
# CONTEXT-DATABASE:MY_DB_NAME
# CONTEXT-RETENTION-POLICY:autogen
randset value=97.9296104805 1439856000000000000
randset value=25.3849066842 1439856100000000000
```
# Caveats
The system does not have access to the meta store when exporting TSM shards. As such, it always creates the retention policy with infinite duration and replication factor of 1.
End users may want to change this prior to re-importing if they are importing to a cluster or want a different duration for retention.

420
cmd/influx_inspect/buildtsi/buildtsi.go
View File

@ -1,420 +0,0 @@
// Package buildtsi reads an in-memory index and exports it as a TSI index.
package buildtsi
import (
"flag"
"fmt"
"io"
"io/ioutil"
"os"
"os/user"
"path/filepath"
"runtime"
"strconv"
"strings"
"sync/atomic"
"github.com/influxdata/influxdb/logger"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"github.com/influxdata/influxdb/tsdb/index/tsi1"
"go.uber.org/zap"
)
const defaultBatchSize = 10000
// Command represents the program execution for "influx_inspect buildtsi".
type Command struct {
Stderr io.Writer
Stdout io.Writer
Verbose bool
Logger *zap.Logger
concurrency int // Number of goroutines to dedicate to shard index building.
databaseFilter string
retentionFilter string
shardFilter string
maxLogFileSize int64
maxCacheSize uint64
batchSize int
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
Logger: zap.NewNop(),
batchSize: defaultBatchSize,
concurrency: runtime.GOMAXPROCS(0),
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fs := flag.NewFlagSet("buildtsi", flag.ExitOnError)
dataDir := fs.String("datadir", "", "data directory")
walDir := fs.String("waldir", "", "WAL directory")
fs.IntVar(&cmd.concurrency, "concurrency", runtime.GOMAXPROCS(0), "Number of workers to dedicate to shard index building. Defaults to GOMAXPROCS")
fs.StringVar(&cmd.databaseFilter, "database", "", "optional: database name")
fs.StringVar(&cmd.retentionFilter, "retention", "", "optional: retention policy")
fs.StringVar(&cmd.shardFilter, "shard", "", "optional: shard id")
fs.Int64Var(&cmd.maxLogFileSize, "max-log-file-size", tsdb.DefaultMaxIndexLogFileSize, "optional: maximum log file size")
fs.Uint64Var(&cmd.maxCacheSize, "max-cache-size", tsdb.DefaultCacheMaxMemorySize, "optional: maximum cache size")
fs.IntVar(&cmd.batchSize, "batch-size", defaultBatchSize, "optional: set the size of the batches we write to the index. Setting this can have adverse affects on performance and heap requirements")
fs.BoolVar(&cmd.Verbose, "v", false, "verbose")
fs.SetOutput(cmd.Stdout)
if err := fs.Parse(args); err != nil {
return err
} else if fs.NArg() > 0 || *dataDir == "" || *walDir == "" {
fs.Usage()
return nil
}
cmd.Logger = logger.New(cmd.Stderr)
return cmd.run(*dataDir, *walDir)
}
func (cmd *Command) run(dataDir, walDir string) error {
// Verify the user actually wants to run as root.
if isRoot() {
fmt.Println("You are currently running as root. This will build your")
fmt.Println("index files with root ownership and will be inaccessible")
fmt.Println("if you run influxd as a non-root user. You should run")
fmt.Println("buildtsi as the same user you are running influxd.")
fmt.Print("Are you sure you want to continue? (y/N): ")
var answer string
if fmt.Scanln(&answer); !strings.HasPrefix(strings.TrimSpace(strings.ToLower(answer)), "y") {
return fmt.Errorf("operation aborted")
}
}
fis, err := ioutil.ReadDir(dataDir)
if err != nil {
return err
}
for _, fi := range fis {
name := fi.Name()
if !fi.IsDir() {
continue
} else if cmd.databaseFilter != "" && name != cmd.databaseFilter {
continue
}
if err := cmd.processDatabase(name, filepath.Join(dataDir, name), filepath.Join(walDir, name)); err != nil {
return err
}
}
return nil
}
func (cmd *Command) processDatabase(dbName, dataDir, walDir string) error {
cmd.Logger.Info("Rebuilding database", zap.String("name", dbName))
sfile := tsdb.NewSeriesFile(filepath.Join(dataDir, tsdb.SeriesFileDirectory))
sfile.Logger = cmd.Logger
if err := sfile.Open(); err != nil {
return err
}
defer sfile.Close()
fis, err := ioutil.ReadDir(dataDir)
if err != nil {
return err
}
for _, fi := range fis {
rpName := fi.Name()
if !fi.IsDir() {
continue
} else if rpName == tsdb.SeriesFileDirectory {
continue
} else if cmd.retentionFilter != "" && rpName != cmd.retentionFilter {
continue
}
if err := cmd.processRetentionPolicy(sfile, dbName, rpName, filepath.Join(dataDir, rpName), filepath.Join(walDir, rpName)); err != nil {
return err
}
}
return nil
}
func (cmd *Command) processRetentionPolicy(sfile *tsdb.SeriesFile, dbName, rpName, dataDir, walDir string) error {
cmd.Logger.Info("Rebuilding retention policy", logger.Database(dbName), logger.RetentionPolicy(rpName))
fis, err := ioutil.ReadDir(dataDir)
if err != nil {
return err
}
type shard struct {
ID uint64
Path string
}
var shards []shard
for _, fi := range fis {
if !fi.IsDir() {
continue
} else if cmd.shardFilter != "" && fi.Name() != cmd.shardFilter {
continue
}
shardID, err := strconv.ParseUint(fi.Name(), 10, 64)
if err != nil {
continue
}
shards = append(shards, shard{shardID, fi.Name()})
}
errC := make(chan error, len(shards))
var maxi uint32 // index of maximum shard being worked on.
for k := 0; k < cmd.concurrency; k++ {
go func() {
for {
i := int(atomic.AddUint32(&maxi, 1) - 1) // Get next partition to work on.
if i >= len(shards) {
return // No more work.
}
id, name := shards[i].ID, shards[i].Path
log := cmd.Logger.With(logger.Database(dbName), logger.RetentionPolicy(rpName), logger.Shard(id))
errC <- IndexShard(sfile, filepath.Join(dataDir, name), filepath.Join(walDir, name), cmd.maxLogFileSize, cmd.maxCacheSize, cmd.batchSize, log, cmd.Verbose)
}
}()
}
// Check for error
for i := 0; i < cap(errC); i++ {
if err := <-errC; err != nil {
return err
}
}
return nil
}
func IndexShard(sfile *tsdb.SeriesFile, dataDir, walDir string, maxLogFileSize int64, maxCacheSize uint64, batchSize int, log *zap.Logger, verboseLogging bool) error {
log.Info("Rebuilding shard")
// Check if shard already has a TSI index.
indexPath := filepath.Join(dataDir, "index")
log.Info("Checking index path", zap.String("path", indexPath))
if _, err := os.Stat(indexPath); !os.IsNotExist(err) {
log.Info("tsi1 index already exists, skipping", zap.String("path", indexPath))
return nil
}
log.Info("Opening shard")
// Remove temporary index files if this is being re-run.
tmpPath := filepath.Join(dataDir, ".index")
log.Info("Cleaning up partial index from previous run, if any")
if err := os.RemoveAll(tmpPath); err != nil {
return err
}
// Open TSI index in temporary path.
tsiIndex := tsi1.NewIndex(sfile, "",
tsi1.WithPath(tmpPath),
tsi1.WithMaximumLogFileSize(maxLogFileSize),
tsi1.DisableFsync(),
// Each new series entry in a log file is ~12 bytes so this should
// roughly equate to one flush to the file for every batch.
tsi1.WithLogFileBufferSize(12*batchSize),
)
tsiIndex.WithLogger(log)
log.Info("Opening tsi index in temporary location", zap.String("path", tmpPath))
if err := tsiIndex.Open(); err != nil {
return err
}
defer tsiIndex.Close()
// Write out tsm1 files.
// Find shard files.
tsmPaths, err := collectTSMFiles(dataDir)
if err != nil {
return err
}
log.Info("Iterating over tsm files")
for _, path := range tsmPaths {
log.Info("Processing tsm file", zap.String("path", path))
if err := IndexTSMFile(tsiIndex, path, batchSize, log, verboseLogging); err != nil {
return err
}
}
// Write out wal files.
walPaths, err := collectWALFiles(walDir)
if err != nil {
if !os.IsNotExist(err) {
return err
}
} else {
log.Info("Building cache from wal files")
cache := tsm1.NewCache(maxCacheSize)
loader := tsm1.NewCacheLoader(walPaths)
loader.WithLogger(log)
if err := loader.Load(cache); err != nil {
return err
}
log.Info("Iterating over cache")
keysBatch := make([][]byte, 0, batchSize)
namesBatch := make([][]byte, 0, batchSize)
tagsBatch := make([]models.Tags, 0, batchSize)
for _, key := range cache.Keys() {
seriesKey, _ := tsm1.SeriesAndFieldFromCompositeKey(key)
name, tags := models.ParseKeyBytes(seriesKey)
if verboseLogging {
log.Info("Series", zap.String("name", string(name)), zap.String("tags", tags.String()))
}
keysBatch = append(keysBatch, seriesKey)
namesBatch = append(namesBatch, name)
tagsBatch = append(tagsBatch, tags)
// Flush batch?
if len(keysBatch) == batchSize {
if err := tsiIndex.CreateSeriesListIfNotExists(keysBatch, namesBatch, tagsBatch); err != nil {
return fmt.Errorf("problem creating series: (%s)", err)
}
keysBatch = keysBatch[:0]
namesBatch = namesBatch[:0]
tagsBatch = tagsBatch[:0]
}
}
// Flush any remaining series in the batches
if len(keysBatch) > 0 {
if err := tsiIndex.CreateSeriesListIfNotExists(keysBatch, namesBatch, tagsBatch); err != nil {
return fmt.Errorf("problem creating series: (%s)", err)
}
keysBatch = nil
namesBatch = nil
tagsBatch = nil
}
}
// Attempt to compact the index & wait for all compactions to complete.
log.Info("compacting index")
tsiIndex.Compact()
tsiIndex.Wait()
// Close TSI index.
log.Info("Closing tsi index")
if err := tsiIndex.Close(); err != nil {
return err
}
// Rename TSI to standard path.
log.Info("Moving tsi to permanent location")
return os.Rename(tmpPath, indexPath)
}
func IndexTSMFile(index *tsi1.Index, path string, batchSize int, log *zap.Logger, verboseLogging bool) error {
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
r, err := tsm1.NewTSMReader(f)
if err != nil {
log.Warn("Unable to read, skipping", zap.String("path", path), zap.Error(err))
return nil
}
defer r.Close()
keysBatch := make([][]byte, 0, batchSize)
namesBatch := make([][]byte, 0, batchSize)
tagsBatch := make([]models.Tags, batchSize)
var ti int
for i := 0; i < r.KeyCount(); i++ {
key, _ := r.KeyAt(i)
seriesKey, _ := tsm1.SeriesAndFieldFromCompositeKey(key)
var name []byte
name, tagsBatch[ti] = models.ParseKeyBytesWithTags(seriesKey, tagsBatch[ti])
if verboseLogging {
log.Info("Series", zap.String("name", string(name)), zap.String("tags", tagsBatch[ti].String()))
}
keysBatch = append(keysBatch, seriesKey)
namesBatch = append(namesBatch, name)
ti++
// Flush batch?
if len(keysBatch) == batchSize {
if err := index.CreateSeriesListIfNotExists(keysBatch, namesBatch, tagsBatch[:ti]); err != nil {
return fmt.Errorf("problem creating series: (%s)", err)
}
keysBatch = keysBatch[:0]
namesBatch = namesBatch[:0]
ti = 0 // Reset tags.
}
}
// Flush any remaining series in the batches
if len(keysBatch) > 0 {
if err := index.CreateSeriesListIfNotExists(keysBatch, namesBatch, tagsBatch[:ti]); err != nil {
return fmt.Errorf("problem creating series: (%s)", err)
}
}
return nil
}
func collectTSMFiles(path string) ([]string, error) {
fis, err := ioutil.ReadDir(path)
if err != nil {
return nil, err
}
var paths []string
for _, fi := range fis {
if filepath.Ext(fi.Name()) != "."+tsm1.TSMFileExtension {
continue
}
paths = append(paths, filepath.Join(path, fi.Name()))
}
return paths, nil
}
func collectWALFiles(path string) ([]string, error) {
if path == "" {
return nil, os.ErrNotExist
}
if _, err := os.Stat(path); os.IsNotExist(err) {
return nil, err
}
fis, err := ioutil.ReadDir(path)
if err != nil {
return nil, err
}
var paths []string
for _, fi := range fis {
if filepath.Ext(fi.Name()) != "."+tsm1.WALFileExtension {
continue
}
paths = append(paths, filepath.Join(path, fi.Name()))
}
return paths, nil
}
func isRoot() bool {
user, _ := user.Current()
return user != nil && user.Username == "root"
}

156
cmd/influx_inspect/deletetsm/deletetsm.go
View File

@ -1,156 +0,0 @@
// Package deletetsm bulk deletes a measurement from a raw tsm file.
package deletetsm
import (
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"time"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
)
// Command represents the program execution for "influxd deletetsm".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
measurement string // measurement to delete
sanitize bool // remove all keys with non-printable unicode
verbose bool // verbose logging
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) (err error) {
fs := flag.NewFlagSet("deletetsm", flag.ExitOnError)
fs.StringVar(&cmd.measurement, "measurement", "", "")
fs.BoolVar(&cmd.sanitize, "sanitize", false, "")
fs.BoolVar(&cmd.verbose, "v", false, "")
fs.SetOutput(cmd.Stdout)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
} else if fs.NArg() == 0 {
fmt.Printf("path required\n\n")
fs.Usage()
return nil
}
if !cmd.verbose {
log.SetOutput(ioutil.Discard)
}
// Validate measurement or sanitize flag.
if cmd.measurement == "" && !cmd.sanitize {
return fmt.Errorf("-measurement or -sanitize flag required")
}
// Process each TSM file.
for _, path := range fs.Args() {
log.Printf("processing: %s", path)
if err := cmd.process(path); err != nil {
return err
}
}
return nil
}
func (cmd *Command) process(path string) error {
// Open TSM reader.
input, err := os.Open(path)
if err != nil {
return err
}
defer input.Close()
r, err := tsm1.NewTSMReader(input)
if err != nil {
return fmt.Errorf("unable to read %s: %s", path, err)
}
defer r.Close()
// Remove previous temporary files.
outputPath := path + ".rewriting.tmp"
if err := os.RemoveAll(outputPath); err != nil {
return err
} else if err := os.RemoveAll(outputPath + ".idx.tmp"); err != nil {
return err
}
// Create TSMWriter to temporary location.
output, err := os.Create(outputPath)
if err != nil {
return err
}
defer output.Close()
w, err := tsm1.NewTSMWriter(output)
if err != nil {
return err
}
defer w.Close()
// Iterate over the input blocks.
itr := r.BlockIterator()
for itr.Next() {
// Read key & time range.
key, minTime, maxTime, _, _, block, err := itr.Read()
if err != nil {
return err
}
// Skip block if this is the measurement and time range we are deleting.
series, _ := tsm1.SeriesAndFieldFromCompositeKey(key)
measurement, tags := models.ParseKey(series)
if string(measurement) == cmd.measurement || (cmd.sanitize && !models.ValidKeyTokens(measurement, tags)) {
log.Printf("deleting block: %s (%s-%s) sz=%d",
key,
time.Unix(0, minTime).UTC().Format(time.RFC3339Nano),
time.Unix(0, maxTime).UTC().Format(time.RFC3339Nano),
len(block),
)
continue
}
if err := w.WriteBlock(key, minTime, maxTime, block); err != nil {
return err
}
}
// Write index & close.
if err := w.WriteIndex(); err != nil {
return err
} else if err := w.Close(); err != nil {
return err
}
// Replace original file with new file.
return os.Rename(outputPath, path)
}
func (cmd *Command) printUsage() {
fmt.Print(`Deletes a measurement from a raw tsm file.
Usage: influx_inspect deletetsm [flags] path...
-measurement NAME
The name of the measurement to remove.
-sanitize
Remove all keys with non-printable unicode characters.
-v
Enable verbose logging.`)
}

528
cmd/influx_inspect/dumptsi/dumptsi.go
View File

@ -1,528 +0,0 @@
// Package dumptsi inspects low-level details about tsi1 files.
package dumptsi
import (
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"text/tabwriter"
"github.com/influxdata/influxdb/logger"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/index/tsi1"
)
// Command represents the program execution for "influxd dumptsi".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
seriesFilePath string
paths []string
showSeries bool
showMeasurements bool
showTagKeys bool
showTagValues bool
showTagValueSeries bool
measurementFilter *regexp.Regexp
tagKeyFilter *regexp.Regexp
tagValueFilter *regexp.Regexp
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
var measurementFilter, tagKeyFilter, tagValueFilter string
fs := flag.NewFlagSet("dumptsi", flag.ExitOnError)
fs.StringVar(&cmd.seriesFilePath, "series-file", "", "Path to series file")
fs.BoolVar(&cmd.showSeries, "series", false, "Show raw series data")
fs.BoolVar(&cmd.showMeasurements, "measurements", false, "Show raw measurement data")
fs.BoolVar(&cmd.showTagKeys, "tag-keys", false, "Show raw tag key data")
fs.BoolVar(&cmd.showTagValues, "tag-values", false, "Show raw tag value data")
fs.BoolVar(&cmd.showTagValueSeries, "tag-value-series", false, "Show raw series data for each value")
fs.StringVar(&measurementFilter, "measurement-filter", "", "Regex measurement filter")
fs.StringVar(&tagKeyFilter, "tag-key-filter", "", "Regex tag key filter")
fs.StringVar(&tagValueFilter, "tag-value-filter", "", "Regex tag value filter")
fs.SetOutput(cmd.Stdout)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
}
// Parse filters.
if measurementFilter != "" {
re, err := regexp.Compile(measurementFilter)
if err != nil {
return err
}
cmd.measurementFilter = re
}
if tagKeyFilter != "" {
re, err := regexp.Compile(tagKeyFilter)
if err != nil {
return err
}
cmd.tagKeyFilter = re
}
if tagValueFilter != "" {
re, err := regexp.Compile(tagValueFilter)
if err != nil {
return err
}
cmd.tagValueFilter = re
}
// Validate series file path.
if cmd.seriesFilePath == "" {
return errors.New("series file path required")
}
cmd.paths = fs.Args()
if len(cmd.paths) == 0 {
fmt.Printf("at least one path required\n\n")
fs.Usage()
return nil
}
// Some flags imply other flags.
if cmd.showTagValueSeries {
cmd.showTagValues = true
}
if cmd.showTagValues {
cmd.showTagKeys = true
}
if cmd.showTagKeys {
cmd.showMeasurements = true
}
return cmd.run()
}
func (cmd *Command) run() error {
sfile := tsdb.NewSeriesFile(cmd.seriesFilePath)
sfile.Logger = logger.New(os.Stderr)
if err := sfile.Open(); err != nil {
return err
}
defer sfile.Close()
// Build a file set from the paths on the command line.
idx, fs, err := cmd.readFileSet(sfile)
if err != nil {
return err
}
if cmd.showSeries {
if err := cmd.printSeries(sfile); err != nil {
return err
}
}
// If this is an ad-hoc fileset then process it and close afterward.
if fs != nil {
defer fs.Release()
defer fs.Close()
if cmd.showSeries || cmd.showMeasurements {
return cmd.printMeasurements(sfile, fs)
}
return cmd.printFileSummaries(fs)
}
// Otherwise iterate over each partition in the index.
defer idx.Close()
for i := 0; i < int(idx.PartitionN); i++ {
if err := func() error {
fs, err := idx.PartitionAt(i).RetainFileSet()
if err != nil {
return err
}
defer fs.Release()
if cmd.showSeries || cmd.showMeasurements {
return cmd.printMeasurements(sfile, fs)
}
return cmd.printFileSummaries(fs)
}(); err != nil {
return err
}
}
return nil
}
func (cmd *Command) readFileSet(sfile *tsdb.SeriesFile) (*tsi1.Index, *tsi1.FileSet, error) {
// If only one path exists and it's a directory then open as an index.
if len(cmd.paths) == 1 {
fi, err := os.Stat(cmd.paths[0])
if err != nil {
return nil, nil, err
} else if fi.IsDir() {
// Verify directory is an index before opening it.
if ok, err := tsi1.IsIndexDir(cmd.paths[0]); err != nil {
return nil, nil, err
} else if !ok {
return nil, nil, fmt.Errorf("Not an index directory: %q", cmd.paths[0])
}
idx := tsi1.NewIndex(sfile,
"",
tsi1.WithPath(cmd.paths[0]),
tsi1.DisableCompactions(),
)
if err := idx.Open(); err != nil {
return nil, nil, err
}
return idx, nil, nil
}
}
// Open each file and group into a fileset.
var files []tsi1.File
for _, path := range cmd.paths {
switch ext := filepath.Ext(path); ext {
case tsi1.LogFileExt:
f := tsi1.NewLogFile(sfile, path)
if err := f.Open(); err != nil {
return nil, nil, err
}
files = append(files, f)
case tsi1.IndexFileExt:
f := tsi1.NewIndexFile(sfile)
f.SetPath(path)
if err := f.Open(); err != nil {
return nil, nil, err
}
files = append(files, f)
default:
return nil, nil, fmt.Errorf("unexpected file extension: %s", ext)
}
}
fs, err := tsi1.NewFileSet(nil, sfile, files)
if err != nil {
return nil, nil, err
}
fs.Retain()
return nil, fs, nil
}
func (cmd *Command) printSeries(sfile *tsdb.SeriesFile) error {
if !cmd.showSeries {
return nil
}
// Print header.
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
fmt.Fprintln(tw, "Series\t")
// Iterate over each series.
itr := sfile.SeriesIDIterator()
for {
e, err := itr.Next()
if err != nil {
return err
} else if e.SeriesID == 0 {
break
}
name, tags := tsdb.ParseSeriesKey(sfile.SeriesKey(e.SeriesID))
if !cmd.matchSeries(name, tags) {
continue
}
deleted := sfile.IsDeleted(e.SeriesID)
fmt.Fprintf(tw, "%s%s\t%v\n", name, tags.HashKey(), deletedString(deleted))
}
// Flush & write footer spacing.
if err := tw.Flush(); err != nil {
return err
}
fmt.Fprint(cmd.Stdout, "\n\n")
return nil
}
func (cmd *Command) printMeasurements(sfile *tsdb.SeriesFile, fs *tsi1.FileSet) error {
if !cmd.showMeasurements {
return nil
}
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
fmt.Fprintln(tw, "Measurement\t")
// Iterate over each series.
if itr := fs.MeasurementIterator(); itr != nil {
for e := itr.Next(); e != nil; e = itr.Next() {
if cmd.measurementFilter != nil && !cmd.measurementFilter.Match(e.Name()) {
continue
}
fmt.Fprintf(tw, "%s\t%v\n", e.Name(), deletedString(e.Deleted()))
if err := tw.Flush(); err != nil {
return err
}
if err := cmd.printTagKeys(sfile, fs, e.Name()); err != nil {
return err
}
}
}
fmt.Fprint(cmd.Stdout, "\n\n")
return nil
}
func (cmd *Command) printTagKeys(sfile *tsdb.SeriesFile, fs *tsi1.FileSet, name []byte) error {
if !cmd.showTagKeys {
return nil
}
// Iterate over each key.
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
itr := fs.TagKeyIterator(name)
for e := itr.Next(); e != nil; e = itr.Next() {
if cmd.tagKeyFilter != nil && !cmd.tagKeyFilter.Match(e.Key()) {
continue
}
fmt.Fprintf(tw, " %s\t%v\n", e.Key(), deletedString(e.Deleted()))
if err := tw.Flush(); err != nil {
return err
}
if err := cmd.printTagValues(sfile, fs, name, e.Key()); err != nil {
return err
}
}
fmt.Fprint(cmd.Stdout, "\n")
return nil
}
func (cmd *Command) printTagValues(sfile *tsdb.SeriesFile, fs *tsi1.FileSet, name, key []byte) error {
if !cmd.showTagValues {
return nil
}
// Iterate over each value.
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
itr := fs.TagValueIterator(name, key)
for e := itr.Next(); e != nil; e = itr.Next() {
if cmd.tagValueFilter != nil && !cmd.tagValueFilter.Match(e.Value()) {
continue
}
fmt.Fprintf(tw, " %s\t%v\n", e.Value(), deletedString(e.Deleted()))
if err := tw.Flush(); err != nil {
return err
}
if err := cmd.printTagValueSeries(sfile, fs, name, key, e.Value()); err != nil {
return err
}
}
fmt.Fprint(cmd.Stdout, "\n")
return nil
}
func (cmd *Command) printTagValueSeries(sfile *tsdb.SeriesFile, fs *tsi1.FileSet, name, key, value []byte) error {
if !cmd.showTagValueSeries {
return nil
}
// Iterate over each series.
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
itr, err := fs.TagValueSeriesIDIterator(name, key, value)
if err != nil {
return err
}
for {
e, err := itr.Next()
if err != nil {
return err
} else if e.SeriesID == 0 {
break
}
name, tags := tsdb.ParseSeriesKey(sfile.SeriesKey(e.SeriesID))
if !cmd.matchSeries(name, tags) {
continue
}
fmt.Fprintf(tw, " %s%s\n", name, tags.HashKey())
if err := tw.Flush(); err != nil {
return err
}
}
fmt.Fprint(cmd.Stdout, "\n")
return nil
}
func (cmd *Command) printFileSummaries(fs *tsi1.FileSet) error {
for _, f := range fs.Files() {
switch f := f.(type) {
case *tsi1.LogFile:
if err := cmd.printLogFileSummary(f); err != nil {
return err
}
case *tsi1.IndexFile:
if err := cmd.printIndexFileSummary(f); err != nil {
return err
}
default:
panic("unreachable")
}
fmt.Fprintln(cmd.Stdout, "")
}
return nil
}
func (cmd *Command) printLogFileSummary(f *tsi1.LogFile) error {
fmt.Fprintf(cmd.Stdout, "[LOG FILE] %s\n", filepath.Base(f.Path()))
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
fmt.Fprintf(tw, "Series:\t%d\n", f.SeriesN())
fmt.Fprintf(tw, "Measurements:\t%d\n", f.MeasurementN())
fmt.Fprintf(tw, "Tag Keys:\t%d\n", f.TagKeyN())
fmt.Fprintf(tw, "Tag Values:\t%d\n", f.TagValueN())
return tw.Flush()
}
func (cmd *Command) printIndexFileSummary(f *tsi1.IndexFile) error {
fmt.Fprintf(cmd.Stdout, "[INDEX FILE] %s\n", filepath.Base(f.Path()))
// Calculate summary stats.
var measurementN, measurementSeriesN, measurementSeriesSize uint64
var keyN uint64
var valueN, valueSeriesN, valueSeriesSize uint64
if mitr := f.MeasurementIterator(); mitr != nil {
for me, _ := mitr.Next().(*tsi1.MeasurementBlockElem); me != nil; me, _ = mitr.Next().(*tsi1.MeasurementBlockElem) {
kitr := f.TagKeyIterator(me.Name())
for ke, _ := kitr.Next().(*tsi1.TagBlockKeyElem); ke != nil; ke, _ = kitr.Next().(*tsi1.TagBlockKeyElem) {
vitr := f.TagValueIterator(me.Name(), ke.Key())
for ve, _ := vitr.Next().(*tsi1.TagBlockValueElem); ve != nil; ve, _ = vitr.Next().(*tsi1.TagBlockValueElem) {
valueN++
valueSeriesN += uint64(ve.SeriesN())
valueSeriesSize += uint64(len(ve.SeriesData()))
}
keyN++
}
measurementN++
measurementSeriesN += uint64(me.SeriesN())
measurementSeriesSize += uint64(len(me.SeriesData()))
}
}
// Write stats.
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
fmt.Fprintf(tw, "Measurements:\t%d\n", measurementN)
fmt.Fprintf(tw, " Series data size:\t%d (%s)\n", measurementSeriesSize, formatSize(measurementSeriesSize))
fmt.Fprintf(tw, " Bytes per series:\t%.01fb\n", float64(measurementSeriesSize)/float64(measurementSeriesN))
fmt.Fprintf(tw, "Tag Keys:\t%d\n", keyN)
fmt.Fprintf(tw, "Tag Values:\t%d\n", valueN)
fmt.Fprintf(tw, " Series:\t%d\n", valueSeriesN)
fmt.Fprintf(tw, " Series data size:\t%d (%s)\n", valueSeriesSize, formatSize(valueSeriesSize))
fmt.Fprintf(tw, " Bytes per series:\t%.01fb\n", float64(valueSeriesSize)/float64(valueSeriesN))
return tw.Flush()
}
// matchSeries returns true if the command filters matches the series.
func (cmd *Command) matchSeries(name []byte, tags models.Tags) bool {
// Filter by measurement.
if cmd.measurementFilter != nil && !cmd.measurementFilter.Match(name) {
return false
}
// Filter by tag key/value.
if cmd.tagKeyFilter != nil || cmd.tagValueFilter != nil {
var matched bool
for _, tag := range tags {
if (cmd.tagKeyFilter == nil || cmd.tagKeyFilter.Match(tag.Key)) && (cmd.tagValueFilter == nil || cmd.tagValueFilter.Match(tag.Value)) {
matched = true
break
}
}
if !matched {
return false
}
}
return true
}
// printUsage prints the usage message to STDERR.
func (cmd *Command) printUsage() {
usage := `Dumps low-level details about tsi1 files.
Usage: influx_inspect dumptsi [flags] path...
-series
Dump raw series data
-measurements
Dump raw measurement data
-tag-keys
Dump raw tag keys
-tag-values
Dump raw tag values
-tag-value-series
Dump raw series for each tag value
-measurement-filter REGEXP
Filters data by measurement regular expression
-series-file PATH
Path to the "_series" directory under the database data directory.
Required.
-tag-key-filter REGEXP
Filters data by tag key regular expression
-tag-value-filter REGEXP
Filters data by tag value regular expression
One or more paths are required. Path must specify either a TSI index directory
or it should specify one or more .tsi/.tsl files. If no flags are specified
then summary stats are provided for each file.
`
fmt.Fprintf(cmd.Stdout, usage)
}
// deletedString returns "(deleted)" if v is true.
func deletedString(v bool) string {
if v {
return "(deleted)"
}
return ""
}
func formatSize(v uint64) string {
denom := uint64(1)
var uom string
for _, uom = range []string{"b", "kb", "mb", "gb", "tb"} {
if denom*1024 > v {
break
}
denom *= 1024
}
return fmt.Sprintf("%0.01f%s", float64(v)/float64(denom), uom)
}

335
cmd/influx_inspect/dumptsm/dumptsm.go
View File

@ -1,335 +0,0 @@
// Package dumptsm inspects low-level details about tsm1 files.
package dumptsm
import (
"encoding/binary"
"flag"
"fmt"
"io"
"os"
"strconv"
"strings"
"text/tabwriter"
"time"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
)
// Command represents the program execution for "influxd dumptsm".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
dumpIndex bool
dumpBlocks bool
dumpAll bool
filterKey string
path string
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fs := flag.NewFlagSet("file", flag.ExitOnError)
fs.BoolVar(&cmd.dumpIndex, "index", false, "Dump raw index data")
fs.BoolVar(&cmd.dumpBlocks, "blocks", false, "Dump raw block data")
fs.BoolVar(&cmd.dumpAll, "all", false, "Dump all data. Caution: This may print a lot of information")
fs.StringVar(&cmd.filterKey, "filter-key", "", "Only display index and block data match this key substring")
fs.SetOutput(cmd.Stdout)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
}
if fs.Arg(0) == "" {
fmt.Printf("TSM file not specified\n\n")
fs.Usage()
return nil
}
cmd.path = fs.Args()[0]
cmd.dumpBlocks = cmd.dumpBlocks || cmd.dumpAll || cmd.filterKey != ""
cmd.dumpIndex = cmd.dumpIndex || cmd.dumpAll || cmd.filterKey != ""
return cmd.dump()
}
func (cmd *Command) dump() error {
var errors []error
f, err := os.Open(cmd.path)
if err != nil {
return err
}
// Get the file size
stat, err := f.Stat()
if err != nil {
return err
}
b := make([]byte, 8)
r, err := tsm1.NewTSMReader(f)
if err != nil {
return fmt.Errorf("Error opening TSM files: %s", err.Error())
}
defer r.Close()
minTime, maxTime := r.TimeRange()
keyCount := r.KeyCount()
blockStats := &blockStats{}
println("Summary:")
fmt.Printf(" File: %s\n", cmd.path)
fmt.Printf(" Time Range: %s - %s\n",
time.Unix(0, minTime).UTC().Format(time.RFC3339Nano),
time.Unix(0, maxTime).UTC().Format(time.RFC3339Nano),
)
fmt.Printf(" Duration: %s ", time.Unix(0, maxTime).Sub(time.Unix(0, minTime)))
fmt.Printf(" Series: %d ", keyCount)
fmt.Printf(" File Size: %d\n", stat.Size())
println()
tw := tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
if cmd.dumpIndex {
println("Index:")
tw.Flush()
println()
fmt.Fprintln(tw, " "+strings.Join([]string{"Pos", "Min Time", "Max Time", "Ofs", "Size", "Key", "Field"}, "\t"))
var pos int
for i := 0; i < keyCount; i++ {
key, _ := r.KeyAt(i)
for _, e := range r.Entries(key) {
pos++
split := strings.Split(string(key), "#!~#")
// Possible corruption? Try to read as much as we can and point to the problem.
measurement := split[0]
field := split[1]
if cmd.filterKey != "" && !strings.Contains(string(key), cmd.filterKey) {
continue
}
fmt.Fprintln(tw, " "+strings.Join([]string{
strconv.FormatInt(int64(pos), 10),
time.Unix(0, e.MinTime).UTC().Format(time.RFC3339Nano),
time.Unix(0, e.MaxTime).UTC().Format(time.RFC3339Nano),
strconv.FormatInt(int64(e.Offset), 10),
strconv.FormatInt(int64(e.Size), 10),
measurement,
field,
}, "\t"))
tw.Flush()
}
}
}
tw = tabwriter.NewWriter(cmd.Stdout, 8, 8, 1, '\t', 0)
fmt.Fprintln(tw, " "+strings.Join([]string{"Blk", "Chk", "Ofs", "Len", "Type", "Min Time", "Points", "Enc [T/V]", "Len [T/V]"}, "\t"))
// Starting at 5 because the magic number is 4 bytes + 1 byte version
i := int64(5)
var blockCount, pointCount, blockSize int64
indexSize := r.IndexSize()
// Start at the beginning and read every block
for j := 0; j < keyCount; j++ {
key, _ := r.KeyAt(j)
for _, e := range r.Entries(key) {
f.Seek(int64(e.Offset), 0)
f.Read(b[:4])
chksum := binary.BigEndian.Uint32(b[:4])
buf := make([]byte, e.Size-4)
f.Read(buf)
blockSize += int64(e.Size)
if cmd.filterKey != "" && !strings.Contains(string(key), cmd.filterKey) {
i += blockSize
blockCount++
continue
}
blockType := buf[0]
encoded := buf[1:]
var v []tsm1.Value
v, err := tsm1.DecodeBlock(buf, v)
if err != nil {
return err
}
startTime := time.Unix(0, v[0].UnixNano())
pointCount += int64(len(v))
// Length of the timestamp block
tsLen, j := binary.Uvarint(encoded)
// Unpack the timestamp bytes
ts := encoded[int(j) : int(j)+int(tsLen)]
// Unpack the value bytes
values := encoded[int(j)+int(tsLen):]
tsEncoding := timeEnc[int(ts[0]>>4)]
vEncoding := encDescs[int(blockType+1)][values[0]>>4]
typeDesc := blockTypes[blockType]
blockStats.inc(0, ts[0]>>4)
blockStats.inc(int(blockType+1), values[0]>>4)
blockStats.size(len(buf))
if cmd.dumpBlocks {
fmt.Fprintln(tw, " "+strings.Join([]string{
strconv.FormatInt(blockCount, 10),
strconv.FormatUint(uint64(chksum), 10),
strconv.FormatInt(i, 10),
strconv.FormatInt(int64(len(buf)), 10),
typeDesc,
startTime.UTC().Format(time.RFC3339Nano),
strconv.FormatInt(int64(len(v)), 10),
fmt.Sprintf("%s/%s", tsEncoding, vEncoding),
fmt.Sprintf("%d/%d", len(ts), len(values)),
}, "\t"))
}
i += blockSize
blockCount++
}
}
if cmd.dumpBlocks {
println("Blocks:")
tw.Flush()
println()
}
var blockSizeAvg int64
if blockCount > 0 {
blockSizeAvg = blockSize / blockCount
}
fmt.Printf("Statistics\n")
fmt.Printf(" Blocks:\n")
fmt.Printf(" Total: %d Size: %d Min: %d Max: %d Avg: %d\n",
blockCount, blockSize, blockStats.min, blockStats.max, blockSizeAvg)
fmt.Printf(" Index:\n")
fmt.Printf(" Total: %d Size: %d\n", blockCount, indexSize)
fmt.Printf(" Points:\n")
fmt.Printf(" Total: %d", pointCount)
println()
println(" Encoding:")
for i, counts := range blockStats.counts {
if len(counts) == 0 {
continue
}
fmt.Printf(" %s: ", strings.Title(fieldType[i]))
for j, v := range counts {
fmt.Printf("\t%s: %d (%d%%) ", encDescs[i][j], v, int(float64(v)/float64(blockCount)*100))
}
println()
}
fmt.Printf(" Compression:\n")
fmt.Printf(" Per block: %0.2f bytes/point\n", float64(blockSize)/float64(pointCount))
fmt.Printf(" Total: %0.2f bytes/point\n", float64(stat.Size())/float64(pointCount))
if len(errors) > 0 {
println()
fmt.Printf("Errors (%d):\n", len(errors))
for _, err := range errors {
fmt.Printf(" * %v\n", err)
}
println()
return fmt.Errorf("error count %d", len(errors))
}
return nil
}
// printUsage prints the usage message to STDERR.
func (cmd *Command) printUsage() {
usage := `Dumps low-level details about tsm1 files.
Usage: influx_inspect dumptsm [flags] <path
-index
Dump raw index data
-blocks
Dump raw block data
-all
Dump all data. Caution: This may print a lot of information
-filter-key <name>
Only display index and block data match this key substring
`
fmt.Fprintf(cmd.Stdout, usage)
}
var (
fieldType = []string{
"timestamp", "float", "int", "bool", "string", "unsigned",
}
blockTypes = []string{
"float64", "int64", "bool", "string", "unsigned",
}
timeEnc = []string{
"none", "s8b", "rle",
}
floatEnc = []string{
"none", "gor",
}
intEnc = []string{
"none", "s8b", "rle",
}
boolEnc = []string{
"none", "bp",
}
stringEnc = []string{
"none", "snpy",
}
unsignedEnc = []string{
"none", "s8b", "rle",
}
encDescs = [][]string{
timeEnc, floatEnc, intEnc, boolEnc, stringEnc, unsignedEnc,
}
)
type blockStats struct {
min, max int
counts [][]int
}
func (b *blockStats) inc(typ int, enc byte) {
for len(b.counts) <= typ {
b.counts = append(b.counts, []int{})
}
for len(b.counts[typ]) <= int(enc) {
b.counts[typ] = append(b.counts[typ], 0)
}
b.counts[typ][enc]++
}
func (b *blockStats) size(sz int) {
if b.min == 0 || sz < b.min {
b.min = sz
}
if b.min == 0 || sz > b.max {
b.max = sz
}
}

3
cmd/influx_inspect/dumptsm/dumptsm_test.go
View File

@ -1,3 +0,0 @@
package dumptsm_test
// TODO: write some tests

162
cmd/influx_inspect/dumptsmwal/dumptsmwal.go
View File

@ -1,162 +0,0 @@
// Package dumptsmwal dumps all data from a WAL file.
package dumptsmwal
import (
"flag"
"fmt"
"io"
"log"
"os"
"path/filepath"
"sort"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
)
// Command represents the program execution for "influxd dumptsmwal".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
showDuplicates bool
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) (err error) {
fs := flag.NewFlagSet("dumptsmwal", flag.ExitOnError)
fs.SetOutput(cmd.Stdout)
fs.BoolVar(&cmd.showDuplicates, "show-duplicates", false, "prints keys with out-of-order or duplicate values")
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
} else if fs.NArg() == 0 {
fmt.Printf("path required\n\n")
fs.Usage()
return nil
}
// Process each TSM WAL file.
for _, path := range fs.Args() {
if err := cmd.process(path); err != nil {
return err
}
}
return nil
}
func (cmd *Command) process(path string) error {
if filepath.Ext(path) != "."+tsm1.WALFileExtension {
log.Printf("invalid wal filename, skipping %s", path)
return nil
}
// Track the earliest timestamp for each key and a set of keys with out-of-order points.
minTimestampByKey := make(map[string]int64)
duplicateKeys := make(map[string]struct{})
// Open WAL reader.
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
r := tsm1.NewWALSegmentReader(f)
// Iterate over the WAL entries.
for r.Next() {
entry, err := r.Read()
if err != nil {
return fmt.Errorf("cannot read entry: %s", err)
}
switch entry := entry.(type) {
case *tsm1.WriteWALEntry:
if !cmd.showDuplicates {
fmt.Printf("[write] sz=%d\n", entry.MarshalSize())
}
keys := make([]string, 0, len(entry.Values))
for k := range entry.Values {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
for _, v := range entry.Values[k] {
t := v.UnixNano()
// Check for duplicate/out of order keys.
if min, ok := minTimestampByKey[k]; ok && t <= min {
duplicateKeys[k] = struct{}{}
}
minTimestampByKey[k] = t
// Skip printing if we are only showing duplicate keys.
if cmd.showDuplicates {
continue
}
switch v := v.(type) {
case tsm1.IntegerValue:
fmt.Printf("%s %vi %d\n", k, v.Value(), t)
case tsm1.UnsignedValue:
fmt.Printf("%s %vu %d\n", k, v.Value(), t)
case tsm1.FloatValue:
fmt.Printf("%s %v %d\n", k, v.Value(), t)
case tsm1.BooleanValue:
fmt.Printf("%s %v %d\n", k, v.Value(), t)
case tsm1.StringValue:
fmt.Printf("%s %q %d\n", k, v.Value(), t)
default:
fmt.Printf("%s EMPTY\n", k)
}
}
}
case *tsm1.DeleteWALEntry:
fmt.Printf("[delete] sz=%d\n", entry.MarshalSize())
for _, k := range entry.Keys {
fmt.Printf("%s\n", string(k))
}
case *tsm1.DeleteRangeWALEntry:
fmt.Printf("[delete-range] min=%d max=%d sz=%d\n", entry.Min, entry.Max, entry.MarshalSize())
for _, k := range entry.Keys {
fmt.Printf("%s\n", string(k))
}
default:
return fmt.Errorf("invalid wal entry: %#v", entry)
}
}
// Print keys with duplicate or out-of-order points, if requested.
if cmd.showDuplicates {
keys := make([]string, 0, len(duplicateKeys))
for k := range duplicateKeys {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
fmt.Println(k)
}
}
return nil
}
func (cmd *Command) printUsage() {
fmt.Print(`Dumps all entries from one or more TSM WAL files.
Usage: influx_inspect dumptsmwal path...`)
}

419
cmd/influx_inspect/export/export.go
View File

@ -1,419 +0,0 @@
// Package export exports TSM files into InfluxDB line protocol format.
package export
import (
"bufio"
"compress/gzip"
"flag"
"fmt"
"io"
"math"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/pkg/escape"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"github.com/influxdata/influxql"
)
// Command represents the program execution for "influx_inspect export".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
dataDir string
walDir string
out string
database string
retentionPolicy string
startTime int64
endTime int64
compress bool
manifest map[string]struct{}
tsmFiles map[string][]string
walFiles map[string][]string
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
manifest: make(map[string]struct{}),
tsmFiles: make(map[string][]string),
walFiles: make(map[string][]string),
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
var start, end string
fs := flag.NewFlagSet("export", flag.ExitOnError)
fs.StringVar(&cmd.dataDir, "datadir", os.Getenv("HOME")+"/.influxdb/data", "Data storage path")
fs.StringVar(&cmd.walDir, "waldir", os.Getenv("HOME")+"/.influxdb/wal", "WAL storage path")
fs.StringVar(&cmd.out, "out", os.Getenv("HOME")+"/.influxdb/export", "Destination file to export to")
fs.StringVar(&cmd.database, "database", "", "Optional: the database to export")
fs.StringVar(&cmd.retentionPolicy, "retention", "", "Optional: the retention policy to export (requires -database)")
fs.StringVar(&start, "start", "", "Optional: the start time to export (RFC3339 format)")
fs.StringVar(&end, "end", "", "Optional: the end time to export (RFC3339 format)")
fs.BoolVar(&cmd.compress, "compress", false, "Compress the output")
fs.SetOutput(cmd.Stdout)
fs.Usage = func() {
fmt.Fprintf(cmd.Stdout, "Exports TSM files into InfluxDB line protocol format.\n\n")
fmt.Fprintf(cmd.Stdout, "Usage: %s export [flags]\n\n", filepath.Base(os.Args[0]))
fs.PrintDefaults()
}
if err := fs.Parse(args); err != nil {
return err
}
// set defaults
if start != "" {
s, err := time.Parse(time.RFC3339, start)
if err != nil {
return err
}
cmd.startTime = s.UnixNano()
} else {
cmd.startTime = math.MinInt64
}
if end != "" {
e, err := time.Parse(time.RFC3339, end)
if err != nil {
return err
}
cmd.endTime = e.UnixNano()
} else {
// set end time to max if it is not set.
cmd.endTime = math.MaxInt64
}
if err := cmd.validate(); err != nil {
return err
}
return cmd.export()
}
func (cmd *Command) validate() error {
if cmd.retentionPolicy != "" && cmd.database == "" {
return fmt.Errorf("must specify a db")
}
if cmd.startTime != 0 && cmd.endTime != 0 && cmd.endTime < cmd.startTime {
return fmt.Errorf("end time before start time")
}
return nil
}
func (cmd *Command) export() error {
if err := cmd.walkTSMFiles(); err != nil {
return err
}
if err := cmd.walkWALFiles(); err != nil {
return err
}
return cmd.write()
}
func (cmd *Command) walkTSMFiles() error {
return filepath.Walk(cmd.dataDir, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
// check to see if this is a tsm file
if filepath.Ext(path) != "."+tsm1.TSMFileExtension {
return nil
}
relPath, err := filepath.Rel(cmd.dataDir, path)
if err != nil {
return err
}
dirs := strings.Split(relPath, string(byte(os.PathSeparator)))
if len(dirs) < 2 {
return fmt.Errorf("invalid directory structure for %s", path)
}
if dirs[0] == cmd.database || cmd.database == "" {
if dirs[1] == cmd.retentionPolicy || cmd.retentionPolicy == "" {
key := filepath.Join(dirs[0], dirs[1])
cmd.manifest[key] = struct{}{}
cmd.tsmFiles[key] = append(cmd.tsmFiles[key], path)
}
}
return nil
})
}
func (cmd *Command) walkWALFiles() error {
return filepath.Walk(cmd.walDir, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
// check to see if this is a wal file
fileName := filepath.Base(path)
if filepath.Ext(path) != "."+tsm1.WALFileExtension || !strings.HasPrefix(fileName, tsm1.WALFilePrefix) {
return nil
}
relPath, err := filepath.Rel(cmd.walDir, path)
if err != nil {
return err
}
dirs := strings.Split(relPath, string(byte(os.PathSeparator)))
if len(dirs) < 2 {
return fmt.Errorf("invalid directory structure for %s", path)
}
if dirs[0] == cmd.database || cmd.database == "" {
if dirs[1] == cmd.retentionPolicy || cmd.retentionPolicy == "" {
key := filepath.Join(dirs[0], dirs[1])
cmd.manifest[key] = struct{}{}
cmd.walFiles[key] = append(cmd.walFiles[key], path)
}
}
return nil
})
}
func (cmd *Command) write() error {
// open our output file and create an output buffer
f, err := os.Create(cmd.out)
if err != nil {
return err
}
defer f.Close()
// Because calling (*os.File).Write is relatively expensive,
// and we don't *need* to sync to disk on every written line of export,
// use a sized buffered writer so that we only sync the file every megabyte.
bw := bufio.NewWriterSize(f, 1024*1024)
defer bw.Flush()
var w io.Writer = bw
if cmd.compress {
gzw := gzip.NewWriter(w)
defer gzw.Close()
w = gzw
}
s, e := time.Unix(0, cmd.startTime).Format(time.RFC3339), time.Unix(0, cmd.endTime).Format(time.RFC3339)
fmt.Fprintf(w, "# INFLUXDB EXPORT: %s - %s\n", s, e)
// Write out all the DDL
fmt.Fprintln(w, "# DDL")
for key := range cmd.manifest {
keys := strings.Split(key, string(os.PathSeparator))
db, rp := influxql.QuoteIdent(keys[0]), influxql.QuoteIdent(keys[1])
fmt.Fprintf(w, "CREATE DATABASE %s WITH NAME %s\n", db, rp)
}
fmt.Fprintln(w, "# DML")
for key := range cmd.manifest {
keys := strings.Split(key, string(os.PathSeparator))
fmt.Fprintf(w, "# CONTEXT-DATABASE:%s\n", keys[0])
fmt.Fprintf(w, "# CONTEXT-RETENTION-POLICY:%s\n", keys[1])
if files, ok := cmd.tsmFiles[key]; ok {
fmt.Fprintf(cmd.Stdout, "writing out tsm file data for %s...", key)
if err := cmd.writeTsmFiles(w, files); err != nil {
return err
}
fmt.Fprintln(cmd.Stdout, "complete.")
}
if _, ok := cmd.walFiles[key]; ok {
fmt.Fprintf(cmd.Stdout, "writing out wal file data for %s...", key)
if err := cmd.writeWALFiles(w, cmd.walFiles[key], key); err != nil {
return err
}
fmt.Fprintln(cmd.Stdout, "complete.")
}
}
return nil
}
func (cmd *Command) writeTsmFiles(w io.Writer, files []string) error {
fmt.Fprintln(w, "# writing tsm data")
// we need to make sure we write the same order that the files were written
sort.Strings(files)
for _, f := range files {
if err := cmd.exportTSMFile(f, w); err != nil {
return err
}
}
return nil
}
func (cmd *Command) exportTSMFile(tsmFilePath string, w io.Writer) error {
f, err := os.Open(tsmFilePath)
if err != nil {
if os.IsNotExist(err) {
fmt.Fprintf(w, "skipped missing file: %s", tsmFilePath)
return nil
}
return err
}
defer f.Close()
r, err := tsm1.NewTSMReader(f)
if err != nil {
fmt.Fprintf(cmd.Stderr, "unable to read %s, skipping: %s\n", tsmFilePath, err.Error())
return nil
}
defer r.Close()
if sgStart, sgEnd := r.TimeRange(); sgStart > cmd.endTime || sgEnd < cmd.startTime {
return nil
}
for i := 0; i < r.KeyCount(); i++ {
key, _ := r.KeyAt(i)
values, err := r.ReadAll(key)
if err != nil {
fmt.Fprintf(cmd.Stderr, "unable to read key %q in %s, skipping: %s\n", string(key), tsmFilePath, err.Error())
continue
}
measurement, field := tsm1.SeriesAndFieldFromCompositeKey(key)
field = escape.Bytes(field)
if err := cmd.writeValues(w, measurement, string(field), values); err != nil {
// An error from writeValues indicates an IO error, which should be returned.
return err
}
}
return nil
}
func (cmd *Command) writeWALFiles(w io.Writer, files []string, key string) error {
fmt.Fprintln(w, "# writing wal data")
// we need to make sure we write the same order that the wal received the data
sort.Strings(files)
var once sync.Once
warnDelete := func() {
once.Do(func() {
msg := fmt.Sprintf(`WARNING: detected deletes in wal file.
Some series for %q may be brought back by replaying this data.
To resolve, you can either let the shard snapshot prior to exporting the data
or manually editing the exported file.
`, key)
fmt.Fprintln(cmd.Stderr, msg)
})
}
for _, f := range files {
if err := cmd.exportWALFile(f, w, warnDelete); err != nil {
return err
}
}
return nil
}
// exportWAL reads every WAL entry from r and exports it to w.
func (cmd *Command) exportWALFile(walFilePath string, w io.Writer, warnDelete func()) error {
f, err := os.Open(walFilePath)
if err != nil {
if os.IsNotExist(err) {
fmt.Fprintf(w, "skipped missing file: %s", walFilePath)
return nil
}
return err
}
defer f.Close()
r := tsm1.NewWALSegmentReader(f)
defer r.Close()
for r.Next() {
entry, err := r.Read()
if err != nil {
n := r.Count()
fmt.Fprintf(cmd.Stderr, "file %s corrupt at position %d: %v", walFilePath, n, err)
break
}
switch t := entry.(type) {
case *tsm1.DeleteWALEntry, *tsm1.DeleteRangeWALEntry:
warnDelete()
continue
case *tsm1.WriteWALEntry:
for key, values := range t.Values {
measurement, field := tsm1.SeriesAndFieldFromCompositeKey([]byte(key))
// measurements are stored escaped, field names are not
field = escape.Bytes(field)
if err := cmd.writeValues(w, measurement, string(field), values); err != nil {
// An error from writeValues indicates an IO error, which should be returned.
return err
}
}
}
}
return nil
}
// writeValues writes every value in values to w, using the given series key and field name.
// If any call to w.Write fails, that error is returned.
func (cmd *Command) writeValues(w io.Writer, seriesKey []byte, field string, values []tsm1.Value) error {
buf := []byte(string(seriesKey) + " " + field + "=")
prefixLen := len(buf)
for _, value := range values {
ts := value.UnixNano()
if (ts < cmd.startTime) || (ts > cmd.endTime) {
continue
}
// Re-slice buf to be "<series_key> <field>=".
buf = buf[:prefixLen]
// Append the correct representation of the value.
switch v := value.Value().(type) {
case float64:
buf = strconv.AppendFloat(buf, v, 'g', -1, 64)
case int64:
buf = strconv.AppendInt(buf, v, 10)
buf = append(buf, 'i')
case uint64:
buf = strconv.AppendUint(buf, v, 10)
buf = append(buf, 'u')
case bool:
buf = strconv.AppendBool(buf, v)
case string:
buf = append(buf, '"')
buf = append(buf, models.EscapeStringField(v)...)
buf = append(buf, '"')
default:
// This shouldn't be possible, but we'll format it anyway.
buf = append(buf, fmt.Sprintf("%v", v)...)
}
// Now buf has "<series_key> <field>=<value>".
// Append the timestamp and a newline, then write it.
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, ts, 10)
buf = append(buf, '\n')
if _, err := w.Write(buf); err != nil {
// Underlying IO error needs to be returned.
return err
}
}
return nil
}

358
cmd/influx_inspect/export/export_test.go
View File

@ -1,358 +0,0 @@
package export
import (
"bytes"
"fmt"
"io/ioutil"
"math"
"math/rand"
"os"
"sort"
"strconv"
"strings"
"testing"
"github.com/golang/snappy"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
)
type corpus map[string][]tsm1.Value
var (
basicCorpus = corpus{
tsm1.SeriesFieldKey("floats,k=f", "f"): []tsm1.Value{
tsm1.NewValue(1, float64(1.5)),
tsm1.NewValue(2, float64(3)),
},
tsm1.SeriesFieldKey("ints,k=i", "i"): []tsm1.Value{
tsm1.NewValue(10, int64(15)),
tsm1.NewValue(20, int64(30)),
},
tsm1.SeriesFieldKey("bools,k=b", "b"): []tsm1.Value{
tsm1.NewValue(100, true),
tsm1.NewValue(200, false),
},
tsm1.SeriesFieldKey("strings,k=s", "s"): []tsm1.Value{
tsm1.NewValue(1000, "1k"),
tsm1.NewValue(2000, "2k"),
},
tsm1.SeriesFieldKey("uints,k=u", "u"): []tsm1.Value{
tsm1.NewValue(3000, uint64(45)),
tsm1.NewValue(4000, uint64(60)),
},
}
basicCorpusExpLines = []string{
"floats,k=f f=1.5 1",
"floats,k=f f=3 2",
"ints,k=i i=15i 10",
"ints,k=i i=30i 20",
"bools,k=b b=true 100",
"bools,k=b b=false 200",
`strings,k=s s="1k" 1000`,
`strings,k=s s="2k" 2000`,
`uints,k=u u=45u 3000`,
`uints,k=u u=60u 4000`,
}
escapeStringCorpus = corpus{
tsm1.SeriesFieldKey("t", "s"): []tsm1.Value{
tsm1.NewValue(1, `1. "quotes"`),
tsm1.NewValue(2, `2. back\slash`),
tsm1.NewValue(3, `3. bs\q"`),
},
}
escCorpusExpLines = []string{
`t s="1. \"quotes\"" 1`,
`t s="2. back\\slash" 2`,
`t s="3. bs\\q\"" 3`,
}
)
func Test_exportWALFile(t *testing.T) {
for _, c := range []struct {
corpus corpus
lines []string
}{
{corpus: basicCorpus, lines: basicCorpusExpLines},
{corpus: escapeStringCorpus, lines: escCorpusExpLines},
} {
walFile := writeCorpusToWALFile(c.corpus)
defer os.Remove(walFile.Name())
var out bytes.Buffer
if err := newCommand().exportWALFile(walFile.Name(), &out, func() {}); err != nil {
t.Fatal(err)
}
lines := strings.Split(out.String(), "\n")
for _, exp := range c.lines {
found := false
for _, l := range lines {
if exp == l {
found = true
break
}
}
if !found {
t.Fatalf("expected line %q to be in exported output:\n%s", exp, out.String())
}
}
}
// Missing .wal file should not cause a failure.
var out bytes.Buffer
if err := newCommand().exportWALFile("file-that-does-not-exist.wal", &out, func() {}); err != nil {
t.Fatal(err)
}
}
func Test_exportTSMFile(t *testing.T) {
for _, c := range []struct {
corpus corpus
lines []string
}{
{corpus: basicCorpus, lines: basicCorpusExpLines},
{corpus: escapeStringCorpus, lines: escCorpusExpLines},
} {
tsmFile := writeCorpusToTSMFile(c.corpus)
defer os.Remove(tsmFile.Name())
var out bytes.Buffer
if err := newCommand().exportTSMFile(tsmFile.Name(), &out); err != nil {
t.Fatal(err)
}
lines := strings.Split(out.String(), "\n")
for _, exp := range c.lines {
found := false
for _, l := range lines {
if exp == l {
found = true
break
}
}
if !found {
t.Fatalf("expected line %q to be in exported output:\n%s", exp, out.String())
}
}
}
// Missing .tsm file should not cause a failure.
var out bytes.Buffer
if err := newCommand().exportTSMFile("file-that-does-not-exist.tsm", &out); err != nil {
t.Fatal(err)
}
}
var sink interface{}
func benchmarkExportTSM(c corpus, b *testing.B) {
// Garbage collection is relatively likely to happen during export, so track allocations.
b.ReportAllocs()
f := writeCorpusToTSMFile(c)
defer os.Remove(f.Name())
cmd := newCommand()
var out bytes.Buffer
b.ResetTimer()
b.StartTimer()
for i := 0; i < b.N; i++ {
if err := cmd.exportTSMFile(f.Name(), &out); err != nil {
b.Fatal(err)
}
sink = out.Bytes()
out.Reset()
}
}
func BenchmarkExportTSMFloats_100s_250vps(b *testing.B) {
benchmarkExportTSM(makeFloatsCorpus(100, 250), b)
}
func BenchmarkExportTSMInts_100s_250vps(b *testing.B) {
benchmarkExportTSM(makeIntsCorpus(100, 250), b)
}
func BenchmarkExportTSMBools_100s_250vps(b *testing.B) {
benchmarkExportTSM(makeBoolsCorpus(100, 250), b)
}
func BenchmarkExportTSMStrings_100s_250vps(b *testing.B) {
benchmarkExportTSM(makeStringsCorpus(100, 250), b)
}
func benchmarkExportWAL(c corpus, b *testing.B) {
// Garbage collection is relatively likely to happen during export, so track allocations.
b.ReportAllocs()
f := writeCorpusToWALFile(c)
defer os.Remove(f.Name())
cmd := newCommand()
var out bytes.Buffer
b.ResetTimer()
b.StartTimer()
for i := 0; i < b.N; i++ {
if err := cmd.exportWALFile(f.Name(), &out, func() {}); err != nil {
b.Fatal(err)
}
sink = out.Bytes()
out.Reset()
}
}
func BenchmarkExportWALFloats_100s_250vps(b *testing.B) {
benchmarkExportWAL(makeFloatsCorpus(100, 250), b)
}
func BenchmarkExportWALInts_100s_250vps(b *testing.B) {
benchmarkExportWAL(makeIntsCorpus(100, 250), b)
}
func BenchmarkExportWALBools_100s_250vps(b *testing.B) {
benchmarkExportWAL(makeBoolsCorpus(100, 250), b)
}
func BenchmarkExportWALStrings_100s_250vps(b *testing.B) {
benchmarkExportWAL(makeStringsCorpus(100, 250), b)
}
// newCommand returns a command that discards its output and that accepts all timestamps.
func newCommand() *Command {
return &Command{
Stderr: ioutil.Discard,
Stdout: ioutil.Discard,
startTime: math.MinInt64,
endTime: math.MaxInt64,
}
}
// makeCorpus returns a new corpus filled with values generated by fn.
// The RNG passed to fn is seeded with numSeries * numValuesPerSeries, for predictable output.
func makeCorpus(numSeries, numValuesPerSeries int, fn func(*rand.Rand) interface{}) corpus {
rng := rand.New(rand.NewSource(int64(numSeries) * int64(numValuesPerSeries)))
var unixNano int64
corpus := make(corpus, numSeries)
for i := 0; i < numSeries; i++ {
vals := make([]tsm1.Value, numValuesPerSeries)
for j := 0; j < numValuesPerSeries; j++ {
vals[j] = tsm1.NewValue(unixNano, fn(rng))
unixNano++
}
k := fmt.Sprintf("m,t=%d", i)
corpus[tsm1.SeriesFieldKey(k, "x")] = vals
}
return corpus
}
func makeFloatsCorpus(numSeries, numFloatsPerSeries int) corpus {
return makeCorpus(numSeries, numFloatsPerSeries, func(rng *rand.Rand) interface{} {
return rng.Float64()
})
}
func makeIntsCorpus(numSeries, numIntsPerSeries int) corpus {
return makeCorpus(numSeries, numIntsPerSeries, func(rng *rand.Rand) interface{} {
// This will only return positive integers. That's probably okay.
return rng.Int63()
})
}
func makeBoolsCorpus(numSeries, numBoolsPerSeries int) corpus {
return makeCorpus(numSeries, numBoolsPerSeries, func(rng *rand.Rand) interface{} {
return rand.Int63n(2) == 1
})
}
func makeStringsCorpus(numSeries, numStringsPerSeries int) corpus {
return makeCorpus(numSeries, numStringsPerSeries, func(rng *rand.Rand) interface{} {
// The string will randomly have 2-6 parts
parts := make([]string, rand.Intn(4)+2)
for i := range parts {
// Each part is a random base36-encoded number
parts[i] = strconv.FormatInt(rand.Int63(), 36)
}
// Join the individual parts with underscores.
return strings.Join(parts, "_")
})
}
// writeCorpusToWALFile writes the given corpus as a WAL file, and returns a handle to that file.
// It is the caller's responsibility to remove the returned temp file.
// writeCorpusToWALFile will panic on any error that occurs.
func writeCorpusToWALFile(c corpus) *os.File {
walFile, err := ioutil.TempFile("", "export_test_corpus_wal")
if err != nil {
panic(err)
}
e := &tsm1.WriteWALEntry{Values: c}
b, err := e.Encode(nil)
if err != nil {
panic(err)
}
w := tsm1.NewWALSegmentWriter(walFile)
if err := w.Write(e.Type(), snappy.Encode(nil, b)); err != nil {
panic(err)
}
if err := w.Flush(); err != nil {
panic(err)
}
// (*tsm1.WALSegmentWriter).sync isn't exported, but it only Syncs the file anyway.
if err := walFile.Sync(); err != nil {
panic(err)
}
return walFile
}
// writeCorpusToTSMFile writes the given corpus as a TSM file, and returns a handle to that file.
// It is the caller's responsibility to remove the returned temp file.
// writeCorpusToTSMFile will panic on any error that occurs.
func writeCorpusToTSMFile(c corpus) *os.File {
tsmFile, err := ioutil.TempFile("", "export_test_corpus_tsm")
if err != nil {
panic(err)
}
w, err := tsm1.NewTSMWriter(tsmFile)
if err != nil {
panic(err)
}
// Write the series in alphabetical order so that each test run is comparable,
// given an identical corpus.
keys := make([]string, 0, len(c))
for k := range c {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if err := w.Write([]byte(k), c[k]); err != nil {
panic(err)
}
}
if err := w.WriteIndex(); err != nil {
panic(err)
}
if err := w.Close(); err != nil {
panic(err)
}
return tsmFile
}

47
cmd/influx_inspect/help/help.go
View File

@ -1,47 +0,0 @@
// Package help contains the help for the influx_inspect command.
package help
import (
"fmt"
"io"
"os"
"strings"
)
// Command displays help for command-line sub-commands.
type Command struct {
Stdout io.Writer
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fmt.Fprintln(cmd.Stdout, strings.TrimSpace(usage))
return nil
}
const usage = `
Usage: influx_inspect [[command] [arguments]]
The commands are:
deletetsm bulk measurement deletion of raw tsm file
dumptsi dumps low-level details about tsi1 files
dumptsm dumps low-level details about tsm1 files
export exports raw data from a shard to line protocol
buildtsi generates tsi1 indexes from tsm1 data
help display this help message
report displays a shard level report
verify verifies integrity of TSM files
verify-seriesfile verifies integrity of the Series file
"help" is the default command.
Use "influx_inspect [command] -help" for more information about a command.
`

3
cmd/influx_inspect/help/help_test.go
View File

@ -1,3 +0,0 @@
package help_test
// TODO: write some tests

120
cmd/influx_inspect/main.go
View File

@ -1,120 +0,0 @@
// The influx_inspect command displays detailed information about InfluxDB data files.
package main
import (
"fmt"
"io"
"log"
"os"
"github.com/influxdata/influxdb/cmd"
"github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi"
"github.com/influxdata/influxdb/cmd/influx_inspect/deletetsm"
"github.com/influxdata/influxdb/cmd/influx_inspect/dumptsi"
"github.com/influxdata/influxdb/cmd/influx_inspect/dumptsm"
"github.com/influxdata/influxdb/cmd/influx_inspect/dumptsmwal"
"github.com/influxdata/influxdb/cmd/influx_inspect/export"
"github.com/influxdata/influxdb/cmd/influx_inspect/help"
"github.com/influxdata/influxdb/cmd/influx_inspect/report"
"github.com/influxdata/influxdb/cmd/influx_inspect/reporttsi"
"github.com/influxdata/influxdb/cmd/influx_inspect/verify/seriesfile"
"github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm"
_ "github.com/influxdata/influxdb/tsdb/engine"
)
func main() {
m := NewMain()
if err := m.Run(os.Args[1:]...); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
// Main represents the program execution.
type Main struct {
Logger *log.Logger
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewMain returns a new instance of Main.
func NewMain() *Main {
return &Main{
Logger: log.New(os.Stderr, "[influx_inspect] ", log.LstdFlags),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run determines and runs the command specified by the CLI args.
func (m *Main) Run(args ...string) error {
name, args := cmd.ParseCommandName(args)
// Extract name from args.
switch name {
case "", "help":
if err := help.NewCommand().Run(args...); err != nil {
return fmt.Errorf("help: %s", err)
}
case "deletetsm":
name := deletetsm.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("deletetsm: %s", err)
}
case "dumptsi":
name := dumptsi.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("dumptsi: %s", err)
}
case "dumptsmdev":
fmt.Fprintf(m.Stderr, "warning: dumptsmdev is deprecated, use dumptsm instead.\n")
fallthrough
case "dumptsm":
name := dumptsm.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("dumptsm: %s", err)
}
case "dumptsmwal":
name := dumptsmwal.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("dumptsmwal: %s", err)
}
case "export":
name := export.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("export: %s", err)
}
case "buildtsi":
name := buildtsi.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("buildtsi: %s", err)
}
case "report":
name := report.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("report: %s", err)
}
case "reporttsi":
name := reporttsi.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("reporttsi: %s", err)
}
case "verify":
name := tsm.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("verify: %s", err)
}
case "verify-seriesfile":
name := seriesfile.NewCommand()
if err := name.Run(args...); err != nil {
return fmt.Errorf("verify-seriesfile: %s", err)
}
default:
return fmt.Errorf(`unknown command "%s"`+"\n"+`Run 'influx_inspect help' for usage`+"\n\n", name)
}
return nil
}

326
cmd/influx_inspect/report/report.go
View File

@ -1,326 +0,0 @@
// Package report reports statistics about TSM files.
package report
import (
"flag"
"fmt"
"io"
"math"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"text/tabwriter"
"time"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"github.com/retailnext/hllpp"
)
// Command represents the program execution for "influxd report".
type Command struct {
Stderr io.Writer
Stdout io.Writer
dir string
pattern string
detailed, exact bool
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fs := flag.NewFlagSet("report", flag.ExitOnError)
fs.StringVar(&cmd.pattern, "pattern", "", "Include only files matching a pattern")
fs.BoolVar(&cmd.detailed, "detailed", false, "Report detailed cardinality estimates")
fs.BoolVar(&cmd.exact, "exact", false, "Report exact counts")
fs.SetOutput(cmd.Stdout)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
}
newCounterFn := newHLLCounter
estTitle := " (est)"
if cmd.exact {
estTitle = ""
newCounterFn = newExactCounter
}
cmd.dir = fs.Arg(0)
err := cmd.isShardDir(cmd.dir)
if cmd.detailed && err != nil {
return fmt.Errorf("-detailed only supported for shard dirs")
}
totalSeries := newCounterFn()
tagCardinalities := map[string]counter{}
measCardinalities := map[string]counter{}
fieldCardinalities := map[string]counter{}
dbCardinalities := map[string]counter{}
start := time.Now()
tw := tabwriter.NewWriter(cmd.Stdout, 8, 2, 1, ' ', 0)
fmt.Fprintln(tw, strings.Join([]string{"DB", "RP", "Shard", "File", "Series", "New" + estTitle, "Min Time", "Max Time", "Load Time"}, "\t"))
minTime, maxTime := int64(math.MaxInt64), int64(math.MinInt64)
var fileCount int
if err := cmd.walkShardDirs(cmd.dir, func(db, rp, id, path string) error {
if cmd.pattern != "" && strings.Contains(path, cmd.pattern) {
return nil
}
file, err := os.OpenFile(path, os.O_RDONLY, 0600)
if err != nil {
fmt.Fprintf(cmd.Stderr, "error: %s: %v. Skipping.\n", path, err)
return nil
}
loadStart := time.Now()
reader, err := tsm1.NewTSMReader(file)
if err != nil {
fmt.Fprintf(cmd.Stderr, "error: %s: %v. Skipping.\n", file.Name(), err)
return nil
}
loadTime := time.Since(loadStart)
fileCount++
dbCount := dbCardinalities[db]
if dbCount == nil {
dbCount = newCounterFn()
dbCardinalities[db] = dbCount
}
oldCount := dbCount.Count()
seriesCount := reader.KeyCount()
for i := 0; i < seriesCount; i++ {
key, _ := reader.KeyAt(i)
totalSeries.Add([]byte(key))
dbCount.Add([]byte(key))
if cmd.detailed {
sep := strings.Index(string(key), "#!~#")
seriesKey, field := key[:sep], key[sep+4:]
measurement, tags := models.ParseKey(seriesKey)
measCount := measCardinalities[measurement]
if measCount == nil {
measCount = newCounterFn()
measCardinalities[measurement] = measCount
}
measCount.Add([]byte(key))
fieldCount := fieldCardinalities[measurement]
if fieldCount == nil {
fieldCount = newCounterFn()
fieldCardinalities[measurement] = fieldCount
}
fieldCount.Add([]byte(field))
for _, t := range tags {
tagCount := tagCardinalities[string(t.Key)]
if tagCount == nil {
tagCount = newCounterFn()
tagCardinalities[string(t.Key)] = tagCount
}
tagCount.Add(t.Value)
}
}
}
minT, maxT := reader.TimeRange()
if minT < minTime {
minTime = minT
}
if maxT > maxTime {
maxTime = maxT
}
reader.Close()
fmt.Fprintln(tw, strings.Join([]string{
db, rp, id,
filepath.Base(file.Name()),
strconv.FormatInt(int64(seriesCount), 10),
strconv.FormatInt(int64(dbCount.Count()-oldCount), 10),
time.Unix(0, minT).UTC().Format(time.RFC3339Nano),
time.Unix(0, maxT).UTC().Format(time.RFC3339Nano),
loadTime.String(),
}, "\t"))
if cmd.detailed {
tw.Flush()
}
return nil
}); err != nil {
return err
}
tw.Flush()
println()
println("Summary:")
fmt.Printf(" Files: %d\n", fileCount)
fmt.Printf(" Time Range: %s - %s\n",
time.Unix(0, minTime).UTC().Format(time.RFC3339Nano),
time.Unix(0, maxTime).UTC().Format(time.RFC3339Nano),
)
fmt.Printf(" Duration: %s \n", time.Unix(0, maxTime).Sub(time.Unix(0, minTime)))
println()
fmt.Printf("Statistics\n")
fmt.Printf(" Series:\n")
for db, counts := range dbCardinalities {
fmt.Printf(" - %s%s: %d (%d%%)\n", db, estTitle, counts.Count(), int(float64(counts.Count())/float64(totalSeries.Count())*100))
}
fmt.Printf(" Total%s: %d\n", estTitle, totalSeries.Count())
if cmd.detailed {
fmt.Printf("\n Measurements (est):\n")
for _, t := range sortKeys(measCardinalities) {
fmt.Printf(" - %v: %d (%d%%)\n", t, measCardinalities[t].Count(), int((float64(measCardinalities[t].Count())/float64(totalSeries.Count()))*100))
}
fmt.Printf("\n Fields (est):\n")
for _, t := range sortKeys(fieldCardinalities) {
fmt.Printf(" - %v: %d\n", t, fieldCardinalities[t].Count())
}
fmt.Printf("\n Tags (est):\n")
for _, t := range sortKeys(tagCardinalities) {
fmt.Printf(" - %v: %d\n", t, tagCardinalities[t].Count())
}
}
fmt.Printf("Completed in %s\n", time.Since(start))
return nil
}
// sortKeys is a quick helper to return the sorted set of a map's keys
func sortKeys(vals map[string]counter) (keys []string) {
for k := range vals {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
func (cmd *Command) isShardDir(dir string) error {
name := filepath.Base(dir)
if id, err := strconv.Atoi(name); err != nil || id < 1 {
return fmt.Errorf("not a valid shard dir: %v", dir)
}
return nil
}
func (cmd *Command) walkShardDirs(root string, fn func(db, rp, id, path string) error) error {
type location struct {
db, rp, id, path string
}
var dirs []location
if err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if filepath.Ext(info.Name()) == "."+tsm1.TSMFileExtension {
shardDir := filepath.Dir(path)
if err := cmd.isShardDir(shardDir); err != nil {
return err
}
absPath, err := filepath.Abs(path)
if err != nil {
return err
}
parts := strings.Split(absPath, string(filepath.Separator))
db, rp, id := parts[len(parts)-4], parts[len(parts)-3], parts[len(parts)-2]
dirs = append(dirs, location{db: db, rp: rp, id: id, path: path})
return nil
}
return nil
}); err != nil {
return err
}
sort.Slice(dirs, func(i, j int) bool {
a, _ := strconv.Atoi(dirs[i].id)
b, _ := strconv.Atoi(dirs[j].id)
return a < b
})
for _, shard := range dirs {
if err := fn(shard.db, shard.rp, shard.id, shard.path); err != nil {
return err
}
}
return nil
}
// printUsage prints the usage message to STDERR.
func (cmd *Command) printUsage() {
usage := `Displays shard level report.
Usage: influx_inspect report [flags]
-pattern <pattern>
Include only files matching a pattern.
-exact
Report exact cardinality counts instead of estimates. Note: this can use a lot of memory.
Defaults to "false".
-detailed
Report detailed cardinality estimates.
Defaults to "false".
`
fmt.Fprintf(cmd.Stdout, usage)
}
// counter abstracts a a method of counting keys.
type counter interface {
Add(key []byte)
Count() uint64
}
// newHLLCounter returns an approximate counter using HyperLogLogs for cardinality estimation.
func newHLLCounter() counter {
return hllpp.New()
}
// exactCounter returns an exact count for keys using counting all distinct items in a set.
type exactCounter struct {
m map[string]struct{}
}
func (c *exactCounter) Add(key []byte) {
c.m[string(key)] = struct{}{}
}
func (c *exactCounter) Count() uint64 {
return uint64(len(c.m))
}
func newExactCounter() counter {
return &exactCounter{
m: make(map[string]struct{}),
}
}

3
cmd/influx_inspect/report/report_test.go
View File

@ -1,3 +0,0 @@
package report_test
// TODO: write some tests

487
cmd/influx_inspect/reporttsi/report.go
View File

@ -1,487 +0,0 @@
// Package reporttsi provides a report about the series cardinality in one or more TSI indexes.
package reporttsi
import (
"errors"
"flag"
"fmt"
"io"
"math"
"os"
"path"
"path/filepath"
"runtime"
"sort"
"strconv"
"sync/atomic"
"text/tabwriter"
"github.com/influxdata/influxdb/logger"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/index/tsi1"
)
const (
// Number of series IDs to stored in slice before we convert to a roaring
// bitmap. Roaring bitmaps have a non-trivial initial cost to construct.
useBitmapN = 25
)
// Command represents the program execution for "influxd reporttsi".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
dbPath string
shardPaths map[uint64]string
shardIdxs map[uint64]*tsi1.Index
cardinalities map[uint64]map[string]*cardinality
seriesFilePath string // optional. Defaults to dbPath/_series
sfile *tsdb.SeriesFile
topN int
byMeasurement bool
byTagKey bool
// How many goroutines to dedicate to calculating cardinality.
concurrency int
}
// NewCommand returns a new instance of Command with default setting applied.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
shardPaths: map[uint64]string{},
shardIdxs: map[uint64]*tsi1.Index{},
cardinalities: map[uint64]map[string]*cardinality{},
topN: 0,
byMeasurement: true,
byTagKey: false,
concurrency: runtime.GOMAXPROCS(0),
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fs := flag.NewFlagSet("reporttsi", flag.ExitOnError)
fs.StringVar(&cmd.dbPath, "db-path", "", "Path to database. Required.")
fs.StringVar(&cmd.seriesFilePath, "series-file", "", "Optional path to series file. Defaults /path/to/db-path/_series")
fs.BoolVar(&cmd.byMeasurement, "measurements", true, "Segment cardinality by measurements")
// TODO(edd): Not yet implemented.
// fs.BoolVar(&cmd.byTagKey, "tag-key", false, "Segment cardinality by tag keys (overrides `measurements`")
fs.IntVar(&cmd.topN, "top", 0, "Limit results to top n")
fs.IntVar(&cmd.concurrency, "c", runtime.GOMAXPROCS(0), "Set worker concurrency. Defaults to GOMAXPROCS setting.")
fs.SetOutput(cmd.Stdout)
if err := fs.Parse(args); err != nil {
return err
}
if cmd.byTagKey {
return errors.New("Segmenting cardinality by tag key is not yet implemented")
}
if cmd.dbPath == "" {
return errors.New("path to database must be provided")
}
if cmd.seriesFilePath == "" {
cmd.seriesFilePath = path.Join(cmd.dbPath, tsdb.SeriesFileDirectory)
}
// Walk database directory to get shards.
if err := filepath.Walk(cmd.dbPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if !info.IsDir() {
return nil
}
// TODO(edd): this would be a problem if the retention policy was named
// "index".
if info.Name() == tsdb.SeriesFileDirectory || info.Name() == "index" {
return filepath.SkipDir
}
id, err := strconv.Atoi(info.Name())
if err != nil {
return nil
}
cmd.shardPaths[uint64(id)] = path
return nil
}); err != nil {
return err
}
if len(cmd.shardPaths) == 0 {
fmt.Fprintf(cmd.Stderr, "No shards under %s\n", cmd.dbPath)
return nil
}
return cmd.run()
}
func (cmd *Command) run() error {
cmd.sfile = tsdb.NewSeriesFile(cmd.seriesFilePath)
cmd.sfile.Logger = logger.New(os.Stderr)
if err := cmd.sfile.Open(); err != nil {
return err
}
defer cmd.sfile.Close()
// Open all the indexes.
for id, pth := range cmd.shardPaths {
pth = path.Join(pth, "index")
// Verify directory is an index before opening it.
if ok, err := tsi1.IsIndexDir(pth); err != nil {
return err
} else if !ok {
return fmt.Errorf("not a TSI index directory: %q", pth)
}
cmd.shardIdxs[id] = tsi1.NewIndex(cmd.sfile,
"",
tsi1.WithPath(pth),
tsi1.DisableCompactions(),
)
if err := cmd.shardIdxs[id].Open(); err != nil {
return err
}
defer cmd.shardIdxs[id].Close()
// Initialise cardinality set to store cardinalities for this shard.
cmd.cardinalities[id] = map[string]*cardinality{}
}
// Calculate cardinalities of shards.
fn := cmd.cardinalityByMeasurement
// if cmd.byTagKey {
// TODO(edd)
// }
// Blocks until all work done.
cmd.calculateCardinalities(fn)
// Print summary.
if err := cmd.printSummaryByMeasurement(); err != nil {
return err
}
allIDs := make([]uint64, 0, len(cmd.shardIdxs))
for id := range cmd.shardIdxs {
allIDs = append(allIDs, id)
}
sort.Slice(allIDs, func(i int, j int) bool { return allIDs[i] < allIDs[j] })
for _, id := range allIDs {
if err := cmd.printShardByMeasurement(id); err != nil {
return err
}
}
return nil
}
// calculateCardinalities calculates the cardinalities of the set of shard being
// worked on concurrently. The provided function determines how cardinality is
// calculated and broken down.
func (cmd *Command) calculateCardinalities(fn func(id uint64) error) error {
// Get list of shards to work on.
shardIDs := make([]uint64, 0, len(cmd.shardIdxs))
for id := range cmd.shardIdxs {
shardIDs = append(shardIDs, id)
}
errC := make(chan error, len(shardIDs))
var maxi uint32 // index of maximumm shard being worked on.
for k := 0; k < cmd.concurrency; k++ {
go func() {
for {
i := int(atomic.AddUint32(&maxi, 1) - 1) // Get next partition to work on.
if i >= len(shardIDs) {
return // No more work.
}
errC <- fn(shardIDs[i])
}
}()
}
// Check for error
for i := 0; i < cap(errC); i++ {
if err := <-errC; err != nil {
return err
}
}
return nil
}
type cardinality struct {
name []byte
short []uint32
set *tsdb.SeriesIDSet
}
func (c *cardinality) add(x uint64) {
if c.set != nil {
c.set.AddNoLock(x)
return
}
c.short = append(c.short, uint32(x)) // Series IDs never get beyond 2^32
// Cheaper to store in bitmap.
if len(c.short) > useBitmapN {
c.set = tsdb.NewSeriesIDSet()
for i := 0; i < len(c.short); i++ {
c.set.AddNoLock(uint64(c.short[i]))
}
c.short = nil
return
}
}
func (c *cardinality) cardinality() int64 {
if c == nil || (c.short == nil && c.set == nil) {
return 0
}
if c.short != nil {
return int64(len(c.short))
}
return int64(c.set.Cardinality())
}
type cardinalities []*cardinality
func (a cardinalities) Len() int { return len(a) }
func (a cardinalities) Less(i, j int) bool { return a[i].cardinality() < a[j].cardinality() }
func (a cardinalities) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (cmd *Command) cardinalityByMeasurement(shardID uint64) error {
idx := cmd.shardIdxs[shardID]
itr, err := idx.MeasurementIterator()
if err != nil {
return err
} else if itr == nil {
return nil
}
defer itr.Close()
OUTER:
for {
name, err := itr.Next()
if err != nil {
return err
} else if name == nil {
break OUTER
}
// Get series ID set to track cardinality under measurement.
c, ok := cmd.cardinalities[shardID][string(name)]
if !ok {
c = &cardinality{name: name}
cmd.cardinalities[shardID][string(name)] = c
}
sitr, err := idx.MeasurementSeriesIDIterator(name)
if err != nil {
return err
} else if sitr == nil {
continue
}
var e tsdb.SeriesIDElem
for e, err = sitr.Next(); err == nil && e.SeriesID != 0; e, err = sitr.Next() {
if e.SeriesID > math.MaxUint32 {
panic(fmt.Sprintf("series ID is too large: %d (max %d). Corrupted series file?", e.SeriesID, uint32(math.MaxUint32)))
}
c.add(e.SeriesID)
}
sitr.Close()
if err != nil {
return err
}
}
return nil
}
type result struct {
name []byte
count int64
// For low cardinality measurements just track series using map
lowCardinality map[uint32]struct{}
// For higher cardinality measurements track using bitmap.
set *tsdb.SeriesIDSet
}
func (r *result) addShort(ids []uint32) {
// There is already a bitset of this result.
if r.set != nil {
for _, id := range ids {
r.set.AddNoLock(uint64(id))
}
return
}
// Still tracking low cardinality sets
if r.lowCardinality == nil {
r.lowCardinality = map[uint32]struct{}{}
}
for _, id := range ids {
r.lowCardinality[id] = struct{}{}
}
// Cardinality is large enough that we will benefit from using a bitmap
if len(r.lowCardinality) > useBitmapN {
r.set = tsdb.NewSeriesIDSet()
for id := range r.lowCardinality {
r.set.AddNoLock(uint64(id))
}
r.lowCardinality = nil
}
}
func (r *result) merge(other *tsdb.SeriesIDSet) {
if r.set == nil {
r.set = tsdb.NewSeriesIDSet()
for id := range r.lowCardinality {
r.set.AddNoLock(uint64(id))
}
r.lowCardinality = nil
}
r.set.Merge(other)
}
type results []*result
func (a results) Len() int { return len(a) }
func (a results) Less(i, j int) bool { return a[i].count < a[j].count }
func (a results) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (cmd *Command) printSummaryByMeasurement() error {
// Get global set of measurement names across shards.
idxs := &tsdb.IndexSet{SeriesFile: cmd.sfile}
for _, idx := range cmd.shardIdxs {
idxs.Indexes = append(idxs.Indexes, idx)
}
mitr, err := idxs.MeasurementIterator()
if err != nil {
return err
} else if mitr == nil {
return errors.New("got nil measurement iterator for index set")
}
defer mitr.Close()
var name []byte
var totalCardinality int64
measurements := results{}
for name, err = mitr.Next(); err == nil && name != nil; name, err = mitr.Next() {
res := &result{name: name}
for _, shardCards := range cmd.cardinalities {
other, ok := shardCards[string(name)]
if !ok {
continue // this shard doesn't have anything for this measurement.
}
if other.short != nil && other.set != nil {
panic("cardinality stored incorrectly")
}
if other.short != nil { // low cardinality case
res.addShort(other.short)
} else if other.set != nil { // High cardinality case
res.merge(other.set)
}
// Shard does not have any series for this measurement.
}
// Determine final cardinality and allow intermediate structures to be
// GCd.
if res.lowCardinality != nil {
res.count = int64(len(res.lowCardinality))
} else {
res.count = int64(res.set.Cardinality())
}
totalCardinality += res.count
res.set = nil
res.lowCardinality = nil
measurements = append(measurements, res)
}
if err != nil {
return err
}
// sort measurements by cardinality.
sort.Sort(sort.Reverse(measurements))
if cmd.topN > 0 {
// There may not be "topN" measurement cardinality to sub-slice.
n := int(math.Min(float64(cmd.topN), float64(len(measurements))))
measurements = measurements[:n]
}
tw := tabwriter.NewWriter(cmd.Stdout, 4, 4, 1, '\t', 0)
fmt.Fprintf(tw, "Summary\nDatabase Path: %s\nCardinality (exact): %d\n\n", cmd.dbPath, totalCardinality)
fmt.Fprint(tw, "Measurement\tCardinality (exact)\n\n")
for _, res := range measurements {
fmt.Fprintf(tw, "%q\t\t%d\n", res.name, res.count)
}
if err := tw.Flush(); err != nil {
return err
}
fmt.Fprint(cmd.Stdout, "\n\n")
return nil
}
func (cmd *Command) printShardByMeasurement(id uint64) error {
allMap, ok := cmd.cardinalities[id]
if !ok {
return nil
}
var totalCardinality int64
all := make(cardinalities, 0, len(allMap))
for _, card := range allMap {
n := card.cardinality()
if n == 0 {
continue
}
totalCardinality += n
all = append(all, card)
}
sort.Sort(sort.Reverse(all))
// Trim to top-n
if cmd.topN > 0 {
// There may not be "topN" measurement cardinality to sub-slice.
n := int(math.Min(float64(cmd.topN), float64(len(all))))
all = all[:n]
}
tw := tabwriter.NewWriter(cmd.Stdout, 4, 4, 1, '\t', 0)
fmt.Fprintf(tw, "===============\nShard ID: %d\nPath: %s\nCardinality (exact): %d\n\n", id, cmd.shardPaths[id], totalCardinality)
fmt.Fprint(tw, "Measurement\tCardinality (exact)\n\n")
for _, card := range all {
fmt.Fprintf(tw, "%q\t\t%d\n", card.name, card.cardinality())
}
fmt.Fprint(tw, "===============\n\n")
if err := tw.Flush(); err != nil {
return err
}
fmt.Fprint(cmd.Stdout, "\n\n")
return nil
}

120
cmd/influx_inspect/verify/seriesfile/command.go
View File

@ -1,120 +0,0 @@
// Package seriesfile verifies integrity of series files.
package seriesfile
import (
"flag"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"runtime"
"github.com/influxdata/influxdb/logger"
"go.uber.org/zap/zapcore"
)
// Command represents the program execution for "influx_inspect verify-seriesfile".
type Command struct {
Stdout io.Writer
Stderr io.Writer
dir string
db string
seriesFile string
verbose bool
concurrent int
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fs := flag.NewFlagSet("verify-seriesfile", flag.ExitOnError)
fs.StringVar(&cmd.dir, "dir", filepath.Join(os.Getenv("HOME"), ".influxdb", "data"),
"Data directory.")
fs.StringVar(&cmd.db, "db", "",
"Only use this database inside of the data directory.")
fs.StringVar(&cmd.seriesFile, "series-file", "",
"Path to a series file. This overrides -db and -dir.")
fs.BoolVar(&cmd.verbose, "v", false,
"Verbose output.")
fs.IntVar(&cmd.concurrent, "c", runtime.GOMAXPROCS(0),
"How many concurrent workers to run.")
fs.SetOutput(cmd.Stdout)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
}
config := logger.NewConfig()
config.Level = zapcore.WarnLevel
if cmd.verbose {
config.Level = zapcore.InfoLevel
}
logger, err := config.New(cmd.Stderr)
if err != nil {
return err
}
v := NewVerify()
v.Logger = logger
v.Concurrent = cmd.concurrent
if cmd.seriesFile != "" {
_, err := v.VerifySeriesFile(cmd.seriesFile)
return err
}
if cmd.db != "" {
_, err := v.VerifySeriesFile(filepath.Join(cmd.dir, cmd.db, "_series"))
return err
}
dbs, err := ioutil.ReadDir(cmd.dir)
if err != nil {
return err
}
for _, db := range dbs {
if !db.IsDir() {
continue
}
_, err := v.VerifySeriesFile(filepath.Join(cmd.dir, db.Name(), "_series"))
if err != nil {
return err
}
}
return nil
}
func (cmd *Command) printUsage() {
usage := `Verifies the integrity of Series files.
Usage: influx_inspect verify-seriesfile [flags]
-dir <path>
Root data path.
Defaults to "%[1]s/.influxdb/data".
-db <name>
Only verify this database inside of the data directory.
-series-file <path>
Path to a series file. This overrides -db and -dir.
-v
Enable verbose logging.
-c
How many concurrent workers to run.
Defaults to "%[2]d" on this machine.
`
fmt.Printf(usage, os.Getenv("HOME"), runtime.GOMAXPROCS(0))
}

406
cmd/influx_inspect/verify/seriesfile/verify.go
View File

@ -1,406 +0,0 @@
package seriesfile
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"runtime"
"sort"
"sync"
"github.com/influxdata/influxdb/tsdb"
"go.uber.org/zap"
)
// verifyResult contains the result of a Verify... call
type verifyResult struct {
valid bool
err error
}
// Verify contains configuration for running verification of series files.
type Verify struct {
Concurrent int
Logger *zap.Logger
done chan struct{}
}
// NewVerify constructs a Verify with good defaults.
func NewVerify() Verify {
return Verify{
Concurrent: runtime.GOMAXPROCS(0),
Logger: zap.NewNop(),
}
}
// VerifySeriesFile performs verifications on a series file. The error is only returned
// if there was some fatal problem with operating, not if there was a problem with the series file.
func (v Verify) VerifySeriesFile(filePath string) (valid bool, err error) {
v.Logger = v.Logger.With(zap.String("path", filePath))
v.Logger.Info("Verifying series file")
defer func() {
if rec := recover(); rec != nil {
v.Logger.Error("Panic verifying file", zap.String("recovered", fmt.Sprint(rec)))
valid = false
}
}()
partitionInfos, err := ioutil.ReadDir(filePath)
if os.IsNotExist(err) {
v.Logger.Error("Series file does not exist")
return false, nil
}
if err != nil {
return false, err
}
// Check every partition in concurrently.
concurrent := v.Concurrent
if concurrent <= 0 {
concurrent = 1
}
in := make(chan string, len(partitionInfos))
out := make(chan verifyResult, len(partitionInfos))
// Make sure all the workers are cleaned up when we return.
var wg sync.WaitGroup
defer wg.Wait()
// Set up cancellation. Any return will cause the workers to be cancelled.
v.done = make(chan struct{})
defer close(v.done)
for i := 0; i < concurrent; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for partitionPath := range in {
valid, err := v.VerifyPartition(partitionPath)
select {
case out <- verifyResult{valid: valid, err: err}:
case <-v.done:
return
}
}
}()
}
// send off the work and read the results.
for _, partitionInfo := range partitionInfos {
in <- filepath.Join(filePath, partitionInfo.Name())
}
close(in)
for range partitionInfos {
result := <-out
if result.err != nil {
return false, err
} else if !result.valid {
return false, nil
}
}
return true, nil
}
// VerifyPartition performs verifications on a partition of a series file. The error is only returned
// if there was some fatal problem with operating, not if there was a problem with the partition.
func (v Verify) VerifyPartition(partitionPath string) (valid bool, err error) {
v.Logger = v.Logger.With(zap.String("partition", filepath.Base(partitionPath)))
v.Logger.Info("Verifying partition")
defer func() {
if rec := recover(); rec != nil {
v.Logger.Error("Panic verifying partition", zap.String("recovered", fmt.Sprint(rec)))
valid = false
}
}()
segmentInfos, err := ioutil.ReadDir(partitionPath)
if err != nil {
return false, err
}
segments := make([]*tsdb.SeriesSegment, 0, len(segmentInfos))
ids := make(map[uint64]IDData)
// check every segment
for _, segmentInfo := range segmentInfos {
select {
default:
case <-v.done:
return false, nil
}
segmentPath := filepath.Join(partitionPath, segmentInfo.Name())
segmentID, err := tsdb.ParseSeriesSegmentFilename(segmentInfo.Name())
if err != nil {
continue
}
if valid, err := v.VerifySegment(segmentPath, ids); err != nil {
return false, err
} else if !valid {
return false, nil
}
// open the segment for verifying the index. we want it to be open outside
// the for loop as well, so the defer is ok.
segment := tsdb.NewSeriesSegment(segmentID, segmentPath)
if err := segment.Open(); err != nil {
return false, err
}
defer segment.Close()
segments = append(segments, segment)
}
// check the index
indexPath := filepath.Join(partitionPath, "index")
if valid, err := v.VerifyIndex(indexPath, segments, ids); err != nil {
return false, err
} else if !valid {
return false, nil
}
return true, nil
}
// IDData keeps track of data about a series ID.
type IDData struct {
Offset int64
Key []byte
Deleted bool
}
// VerifySegment performs verifications on a segment of a series file. The error is only returned
// if there was some fatal problem with operating, not if there was a problem with the partition.
// The ids map is populated with information about the ids stored in the segment.
func (v Verify) VerifySegment(segmentPath string, ids map[uint64]IDData) (valid bool, err error) {
segmentName := filepath.Base(segmentPath)
v.Logger = v.Logger.With(zap.String("segment", segmentName))
v.Logger.Info("Verifying segment")
// Open up the segment and grab it's data.
segmentID, err := tsdb.ParseSeriesSegmentFilename(segmentName)
if err != nil {
return false, err
}
segment := tsdb.NewSeriesSegment(segmentID, segmentPath)
if err := segment.Open(); err != nil {
v.Logger.Error("Error opening segment", zap.Error(err))
return false, nil
}
defer segment.Close()
buf := newBuffer(segment.Data())
defer func() {
if rec := recover(); rec != nil {
v.Logger.Error("Panic verifying segment", zap.String("recovered", fmt.Sprint(rec)),
zap.Int64("offset", buf.offset))
valid = false
}
}()
// Skip the header: it has already been verified by the Open call.
if err := buf.advance(tsdb.SeriesSegmentHeaderSize); err != nil {
v.Logger.Error("Unable to advance buffer",
zap.Int64("offset", buf.offset),
zap.Error(err))
return false, nil
}
prevID, firstID := uint64(0), true
entries:
for len(buf.data) > 0 {
select {
default:
case <-v.done:
return false, nil
}
flag, id, key, sz := tsdb.ReadSeriesEntry(buf.data)
// Check the flag is valid and for id monotonicity.
hasKey := true
switch flag {
case tsdb.SeriesEntryInsertFlag:
if !firstID && prevID > id {
v.Logger.Error("ID is not monotonically increasing",
zap.Uint64("prev_id", prevID),
zap.Uint64("id", id),
zap.Int64("offset", buf.offset))
return false, nil
}
firstID = false
prevID = id
if ids != nil {
keyCopy := make([]byte, len(key))
copy(keyCopy, key)
ids[id] = IDData{
Offset: tsdb.JoinSeriesOffset(segment.ID(), uint32(buf.offset)),
Key: keyCopy,
}
}
case tsdb.SeriesEntryTombstoneFlag:
hasKey = false
if ids != nil {
data := ids[id]
data.Deleted = true
ids[id] = data
}
case 0: // if zero, there are no more entries
if err := buf.advance(sz); err != nil {
v.Logger.Error("Unable to advance buffer",
zap.Int64("offset", buf.offset),
zap.Error(err))
return false, nil
}
break entries
default:
v.Logger.Error("Invalid flag",
zap.Uint8("flag", flag),
zap.Int64("offset", buf.offset))
return false, nil
}
// Ensure the key parses. This may panic, but our defer handler should
// make the error message more usable by providing the key.
if hasKey {
parsed := false
func() {
defer func() {
if rec := recover(); rec != nil {
v.Logger.Error("Panic parsing key",
zap.String("key", fmt.Sprintf("%x", key)),
zap.Int64("offset", buf.offset),
zap.String("recovered", fmt.Sprint(rec)))
}
}()
tsdb.ParseSeriesKey(key)
parsed = true
}()
if !parsed {
return false, nil
}
}
// Advance past the entry.
if err := buf.advance(sz); err != nil {
v.Logger.Error("Unable to advance buffer",
zap.Int64("offset", buf.offset),
zap.Error(err))
return false, nil
}
}
return true, nil
}
// VerifyIndex performs verification on an index in a series file. The error is only returned
// if there was some fatal problem with operating, not if there was a problem with the partition.
// The ids map must be built from verifying the passed in segments.
func (v Verify) VerifyIndex(indexPath string, segments []*tsdb.SeriesSegment,
ids map[uint64]IDData) (valid bool, err error) {
v.Logger.Info("Verifying index")
defer func() {
if rec := recover(); rec != nil {
v.Logger.Error("Panic verifying index", zap.String("recovered", fmt.Sprint(rec)))
valid = false
}
}()
index := tsdb.NewSeriesIndex(indexPath)
if err := index.Open(); err != nil {
v.Logger.Error("Error opening index", zap.Error(err))
return false, nil
}
defer index.Close()
if err := index.Recover(segments); err != nil {
v.Logger.Error("Error recovering index", zap.Error(err))
return false, nil
}
// we check all the ids in a consistent order to get the same errors if
// there is a problem
idsList := make([]uint64, 0, len(ids))
for id := range ids {
idsList = append(idsList, id)
}
sort.Slice(idsList, func(i, j int) bool {
return idsList[i] < idsList[j]
})
for _, id := range idsList {
select {
default:
case <-v.done:
return false, nil
}
IDData := ids[id]
expectedOffset, expectedID := IDData.Offset, id
if IDData.Deleted {
expectedOffset, expectedID = 0, 0
}
// check both that the offset is right and that we get the right
// id for the key
if gotOffset := index.FindOffsetByID(id); gotOffset != expectedOffset {
v.Logger.Error("Index inconsistency",
zap.Uint64("id", id),
zap.Int64("got_offset", gotOffset),
zap.Int64("expected_offset", expectedOffset))
return false, nil
}
if gotID := index.FindIDBySeriesKey(segments, IDData.Key); gotID != expectedID {
v.Logger.Error("Index inconsistency",
zap.Uint64("id", id),
zap.Uint64("got_id", gotID),
zap.Uint64("expected_id", expectedID))
return false, nil
}
}
return true, nil
}
// buffer allows one to safely advance a byte slice and keep track of how many bytes were advanced.
type buffer struct {
offset int64
data []byte
}
// newBuffer constructs a buffer with the provided data.
func newBuffer(data []byte) *buffer {
return &buffer{
offset: 0,
data: data,
}
}
// advance will consume n bytes from the data slice and return an error if there is not enough
// data to do so.
func (b *buffer) advance(n int64) error {
if int64(len(b.data)) < n {
return fmt.Errorf("unable to advance %d bytes: %d remaining", n, len(b.data))
}
b.data = b.data[n:]
b.offset += n
return nil
}

168
cmd/influx_inspect/verify/seriesfile/verify_test.go
View File

@ -1,168 +0,0 @@
package seriesfile_test
import (
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"testing"
"time"
"github.com/influxdata/influxdb/cmd/influx_inspect/verify/seriesfile"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"go.uber.org/zap"
)
func TestVerifies_Valid(t *testing.T) {
test := NewTest(t)
defer test.Close()
verify := seriesfile.NewVerify()
if testing.Verbose() {
verify.Logger, _ = zap.NewDevelopment()
}
passed, err := verify.VerifySeriesFile(test.Path)
test.AssertNoError(err)
test.Assert(passed)
}
func TestVerifies_Invalid(t *testing.T) {
test := NewTest(t)
defer test.Close()
test.AssertNoError(filepath.Walk(test.Path, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
test.Backup(path)
defer test.Restore(path)
fh, err := os.OpenFile(path, os.O_RDWR, 0)
test.AssertNoError(err)
defer fh.Close()
_, err = fh.WriteAt([]byte("BOGUS"), 0)
test.AssertNoError(err)
test.AssertNoError(fh.Close())
passed, err := seriesfile.NewVerify().VerifySeriesFile(test.Path)
test.AssertNoError(err)
test.Assert(!passed)
return nil
}))
}
//
// helpers
//
type Test struct {
*testing.T
Path string
}
func NewTest(t *testing.T) *Test {
t.Helper()
dir, err := ioutil.TempDir("", "verify-seriesfile-")
if err != nil {
t.Fatal(err)
}
// create a series file in the directory
err = func() error {
seriesFile := tsdb.NewSeriesFile(dir)
if err := seriesFile.Open(); err != nil {
return err
}
defer seriesFile.Close()
seriesFile.EnableCompactions()
const (
compactionThreshold = 100
numSeries = 2 * tsdb.SeriesFilePartitionN * compactionThreshold
)
for _, partition := range seriesFile.Partitions() {
partition.CompactThreshold = compactionThreshold
}
var names [][]byte
var tagsSlice []models.Tags
for i := 0; i < numSeries; i++ {
names = append(names, []byte(fmt.Sprintf("series%d", i)))
tagsSlice = append(tagsSlice, nil)
}
_, err := seriesFile.CreateSeriesListIfNotExists(names, tagsSlice)
if err != nil {
return err
}
// wait for compaction to make sure we detect issues with the index
partitions := seriesFile.Partitions()
wait:
for _, partition := range partitions {
if partition.Compacting() {
time.Sleep(100 * time.Millisecond)
goto wait
}
}
return seriesFile.Close()
}()
if err != nil {
os.RemoveAll(dir)
t.Fatal(err)
}
return &Test{
T: t,
Path: dir,
}
}
func (t *Test) Close() {
os.RemoveAll(t.Path)
}
func (t *Test) AssertNoError(err error) {
t.Helper()
if err != nil {
t.Fatal("unexpected error:", err)
}
}
func (t *Test) Assert(x bool) {
t.Helper()
if !x {
t.Fatal("unexpected condition")
}
}
// Backup makes a copy of the path for a later Restore.
func (t *Test) Backup(path string) {
in, err := os.Open(path)
t.AssertNoError(err)
defer in.Close()
out, err := os.Create(path + ".backup")
t.AssertNoError(err)
defer out.Close()
_, err = io.Copy(out, in)
t.AssertNoError(err)
}
// Restore restores the file at the path to the time when Backup was called last.
func (t *Test) Restore(path string) {
t.AssertNoError(os.Rename(path+".backup", path))
}

224
cmd/influx_inspect/verify/tsm/verify.go
View File

@ -1,224 +0,0 @@
// Package tsm verifies integrity of TSM files.
package tsm
import (
"flag"
"fmt"
"hash/crc32"
"io"
"os"
"path/filepath"
"text/tabwriter"
"time"
"unicode/utf8"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"github.com/pkg/errors"
)
// Command represents the program execution for "influx_inspect verify".
type Command struct {
Stderr io.Writer
Stdout io.Writer
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
var path string
fs := flag.NewFlagSet("verify", flag.ExitOnError)
fs.StringVar(&path, "dir", os.Getenv("HOME")+"/.influxdb", "Root storage path. [$HOME/.influxdb]")
var checkUTF8 bool
fs.BoolVar(&checkUTF8, "check-utf8", false, "Verify series keys are valid UTF-8")
fs.SetOutput(cmd.Stdout)
fs.Usage = cmd.printUsage
if err := fs.Parse(args); err != nil {
return err
}
dataPath := filepath.Join(path, "data")
tw := tabwriter.NewWriter(cmd.Stdout, 16, 8, 0, '\t', 0)
var runner verifier
if checkUTF8 {
runner = &verifyUTF8{}
} else {
runner = &verifyChecksums{}
}
err := runner.Run(tw, dataPath)
tw.Flush()
return err
}
// printUsage prints the usage message to STDERR.
func (cmd *Command) printUsage() {
usage := fmt.Sprintf(`Verifies the integrity of TSM files.
Usage: influx_inspect verify [flags]
-dir <path>
Root storage path
Defaults to "%[1]s/.influxdb".
-check-utf8
Verify series keys are valid UTF-8.
This check skips verification of block checksums.
`, os.Getenv("HOME"))
fmt.Fprintf(cmd.Stdout, usage)
}
type verifyTSM struct {
files []string
f string
start time.Time
err error
}
func (v *verifyTSM) loadFiles(dataPath string) {
err := filepath.Walk(dataPath, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
if filepath.Ext(path) == "."+tsm1.TSMFileExtension {
v.files = append(v.files, path)
}
return nil
})
if err != nil {
panic(err)
}
}
func (v *verifyTSM) Next() bool {
if len(v.files) == 0 {
return false
}
v.f, v.files = v.files[0], v.files[1:]
return true
}
func (v *verifyTSM) TSMReader() (string, *tsm1.TSMReader) {
file, err := os.OpenFile(v.f, os.O_RDONLY, 0600)
if err != nil {
v.err = err
return "", nil
}
reader, err := tsm1.NewTSMReader(file)
if err != nil {
file.Close()
v.err = err
return "", nil
}
return v.f, reader
}
func (v *verifyTSM) Start() {
v.start = time.Now()
}
func (v *verifyTSM) Elapsed() time.Duration {
return time.Since(v.start)
}
type verifyChecksums struct {
verifyTSM
totalErrors int
total int
}
func (v *verifyChecksums) Run(w io.Writer, dataPath string) error {
v.loadFiles(dataPath)
v.Start()
for v.Next() {
f, reader := v.TSMReader()
if reader == nil {
break
}
blockItr := reader.BlockIterator()
fileErrors := 0
count := 0
for blockItr.Next() {
v.total++
key, _, _, _, checksum, buf, err := blockItr.Read()
if err != nil {
v.totalErrors++
fileErrors++
fmt.Fprintf(w, "%s: could not get checksum for key %v block %d due to error: %q\n", f, key, count, err)
} else if expected := crc32.ChecksumIEEE(buf); checksum != expected {
v.totalErrors++
fileErrors++
fmt.Fprintf(w, "%s: got %d but expected %d for key %v, block %d\n", f, checksum, expected, key, count)
}
count++
}
if fileErrors == 0 {
fmt.Fprintf(w, "%s: healthy\n", f)
}
reader.Close()
}
fmt.Fprintf(w, "Broken Blocks: %d / %d, in %vs\n", v.totalErrors, v.total, v.Elapsed().Seconds())
return v.err
}
type verifyUTF8 struct {
verifyTSM
totalErrors int
total int
}
func (v *verifyUTF8) Run(w io.Writer, dataPath string) error {
v.loadFiles(dataPath)
v.Start()
for v.Next() {
f, reader := v.TSMReader()
if reader == nil {
break
}
n := reader.KeyCount()
fileErrors := 0
v.total += n
for i := 0; i < n; i++ {
key, _ := reader.KeyAt(i)
if !utf8.Valid(key) {
v.totalErrors++
fileErrors++
fmt.Fprintf(w, "%s: key #%d is not valid UTF-8\n", f, i)
}
}
if fileErrors == 0 {
fmt.Fprintf(w, "%s: healthy\n", f)
}
}
fmt.Fprintf(w, "Invalid Keys: %d / %d, in %vs\n", v.totalErrors, v.total, v.Elapsed().Seconds())
if v.totalErrors > 0 && v.err == nil {
v.err = errors.New("check-utf8: failed")
}
return v.err
}
type verifier interface {
Run(w io.Writer, dataPath string) error
}

3
cmd/influx_inspect/verify/tsm/verify_test.go
View File

@ -1,3 +0,0 @@
package tsm_test
// TODO: write some tests

43
cmd/influx_stress/README.md
View File

@ -1,43 +0,0 @@
# `influx_stress`
If you run into any issues with this tool please mention @jackzampolin when you create an issue.
## Ways to run
### `influx_stress`
This runs a basic stress test with the [default config](https://github.com/influxdata/influxdb/blob/master/stress/stress.toml) For more information on the configuration file please see the default.
### `influx_stress -config someConfig.toml`
This runs the stress test with a valid configuration file located at `someConfig.tom`
### `influx_stress -v2 -config someConfig.iql`
This runs the stress test with a valid `v2` configuration file. For more information about the `v2` stress test see the [v2 stress README](https://github.com/influxdata/influxdb/blob/master/stress/v2/README.md).
## Flags
If flags are defined they overwrite the config from any file passed in.
### `-addr` string
IP address and port of database where response times will persist (e.g., localhost:8086)
`default` = "http://localhost:8086"
### `-config` string
The relative path to the stress test configuration file.
`default` = [config](https://github.com/influxdata/influxdb/blob/master/stress/stress.toml)
### `-cpuprofile` filename
Writes the result of Go's cpu profile to filename
`default` = no profiling
### `-database` string
Name of database on `-addr` that `influx_stress` will persist write and query response times
`default` = "stress"
### `-tags` value
A comma separated list of tags to add to write and query response times.
`default` = ""

92
cmd/influx_stress/examples/template.toml
View File

@ -1,92 +0,0 @@
# This section can be removed
[provision]
# The basic provisioner simply deletes and creates database.
# If `reset_database` is false, it will not attempt to delete the database
[provision.basic]
# If enabled the provisioner will actually run
enabled = true
# Address of the instance that is to be provisioned
address = "localhost:8086"
# Database that will be created/deleted
database = "stress"
# Attempt to delete database
reset_database = true
# This section cannot be commented out
# To prevent writes set `enabled=false`
# in [write.influx_client.basic]
[write]
[write.point_generator]
# The basic point generator will generate points of the form
# `cpu,host=server-%v,location=us-west value=234 123456`
[write.point_generator.basic]
# number of points that will be written for each of the series
point_count = 100
# number of series
series_count = 100000
# How much time between each timestamp
tick = "10s"
# Randomize timestamp a bit (not functional)
jitter = true
# Precision of points that are being written
precision = "s"
# name of the measurement that will be written
measurement = "cpu"
# The date for the first point that is written into influx
start_date = "2006-Jan-02"
# Defines a tag for a series
[[write.point_generator.basic.tag]]
key = "host"
value = "server"
[[write.point_generator.basic.tag]]
key = "location"
value = "us-west"
# Defines a field for a series
[[write.point_generator.basic.field]]
key = "value"
value = "float64" # supported types: float64, int, bool
[write.influx_client]
[write.influx_client.basic]
# If enabled the writer will actually write
enabled = true
# Addresses is an array of the Influxdb instances
addresses = ["localhost:8086"] # stress_test_server runs on port 1234
# Database that is being written to
database = "stress"
# Precision of points that are being written
precision = "s"
# Size of batches that are sent to db
batch_size = 10000
# Interval between each batch
batch_interval = "0s"
# How many concurrent writers to the db
concurrency = 10
# ssl enabled?
ssl = false
# format of points that are written to influxdb
format = "line_http" # line_udp (not supported yet), graphite_tcp (not supported yet), graphite_udp (not supported yet)
# This section can be removed
[read]
[read.query_generator]
[read.query_generator.basic]
# Template of the query that will be ran against the instance
template = "SELECT count(value) FROM cpu where host='server-%v'"
# How many times the templated query will be ran
query_count = 250
[read.query_client]
[read.query_client.basic]
# if enabled the reader will actually read
enabled = true
# Address of the instance that will be queried
addresses = ["localhost:8086"]
# Database that will be queried
database = "stress"
# Interval bewteen queries
query_interval = "100ms"
# Number of concurrent queriers
concurrency = 1

71
cmd/influx_stress/influx_stress.go
View File

@ -1,71 +0,0 @@
// Command influx_stress is deprecated; use github.com/influxdata/influx-stress instead.
package main
import (
"flag"
"fmt"
"log"
"os"
"runtime/pprof"
"github.com/influxdata/influxdb/stress"
v2 "github.com/influxdata/influxdb/stress/v2"
)
var (
useV2 = flag.Bool("v2", false, "Use version 2 of stress tool")
config = flag.String("config", "", "The stress test file")
cpuprofile = flag.String("cpuprofile", "", "Write the cpu profile to `filename`")
db = flag.String("db", "", "target database within test system for write and query load")
)
func main() {
o := stress.NewOutputConfig()
flag.Parse()
if *cpuprofile != "" {
f, err := os.Create(*cpuprofile)
if err != nil {
fmt.Println(err)
return
}
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
}
if *useV2 {
if *config != "" {
v2.RunStress(*config)
} else {
v2.RunStress("stress/v2/iql/file.iql")
}
} else {
c, err := stress.NewConfig(*config)
if err != nil {
log.Fatal(err)
return
}
if *db != "" {
c.Provision.Basic.Database = *db
c.Write.InfluxClients.Basic.Database = *db
c.Read.QueryClients.Basic.Database = *db
}
w := stress.NewWriter(c.Write.PointGenerators.Basic, &c.Write.InfluxClients.Basic)
r := stress.NewQuerier(&c.Read.QueryGenerators.Basic, &c.Read.QueryClients.Basic)
s := stress.NewStressTest(&c.Provision.Basic, w, r)
bw := stress.NewBroadcastChannel()
bw.Register(c.Write.InfluxClients.Basic.BasicWriteHandler)
bw.Register(o.HTTPHandler("write"))
br := stress.NewBroadcastChannel()
br.Register(c.Read.QueryClients.Basic.BasicReadHandler)
br.Register(o.HTTPHandler("read"))
s.Start(bw.Handle, br.Handle)
}
}

310
cmd/influx_tools/compact/command.go
View File

@ -1,310 +0,0 @@
package compact
import (
"bufio"
"bytes"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"sort"
"strings"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/errlist"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/binary"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/line"
"github.com/influxdata/influxdb/logger"
"github.com/influxdata/influxdb/pkg/limiter"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"go.uber.org/zap"
)
var (
_ line.Writer
_ binary.Writer
)
// Command represents the program execution for "influx-tools compact".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
Logger *zap.Logger
path string
force bool
verbose bool
}
// NewCommand returns a new instance of the export Command.
func NewCommand() *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
}
}
// Run executes the export command using the specified args.
func (cmd *Command) Run(args []string) (err error) {
err = cmd.parseFlags(args)
if err != nil {
return err
}
var log = zap.NewNop()
if cmd.verbose {
cfg := logger.Config{Format: "logfmt"}
log, err = cfg.New(os.Stdout)
if err != nil {
return err
}
}
fmt.Fprintf(cmd.Stdout, "opening shard at path %q\n\n", cmd.path)
sc, err := newShardCompactor(cmd.path, log)
if err != nil {
return err
}
fmt.Fprintln(cmd.Stdout)
fmt.Fprintln(cmd.Stdout, "The following files will be compacted:")
fmt.Fprintln(cmd.Stdout)
fmt.Fprintln(cmd.Stdout, sc.String())
if !cmd.force {
fmt.Fprint(cmd.Stdout, "Proceed? [N] ")
scan := bufio.NewScanner(os.Stdin)
scan.Scan()
if scan.Err() != nil {
return fmt.Errorf("error reading STDIN: %v", scan.Err())
}
if strings.ToLower(scan.Text()) != "y" {
return nil
}
}
fmt.Fprintln(cmd.Stdout, "Compacting shard.")
err = sc.CompactShard()
if err != nil {
return fmt.Errorf("compaction failed: %v", err)
}
fmt.Fprintln(cmd.Stdout, "Compaction succeeded. New files:")
for _, f := range sc.newTSM {
fmt.Fprintf(cmd.Stdout, " %s\n", f)
}
return nil
}
func (cmd *Command) parseFlags(args []string) error {
fs := flag.NewFlagSet("compact-shard", flag.ContinueOnError)
fs.StringVar(&cmd.path, "path", "", "path of shard to be compacted")
fs.BoolVar(&cmd.force, "force", false, "Force compaction without prompting")
fs.BoolVar(&cmd.verbose, "verbose", false, "Enable verbose logging")
if err := fs.Parse(args); err != nil {
return err
}
if cmd.path == "" {
return errors.New("shard-path is required")
}
return nil
}
type shardCompactor struct {
logger *zap.Logger
path string
tsm []string
tombstone []string
readers []*tsm1.TSMReader
files map[string]*tsm1.TSMReader
newTSM []string
}
func newShardCompactor(path string, logger *zap.Logger) (sc *shardCompactor, err error) {
sc = &shardCompactor{
logger: logger,
path: path,
files: make(map[string]*tsm1.TSMReader),
}
sc.tsm, err = filepath.Glob(filepath.Join(path, fmt.Sprintf("*.%s", tsm1.TSMFileExtension)))
if err != nil {
return nil, fmt.Errorf("newFileStore: error reading tsm files at path %q: %v", path, err)
}
if len(sc.tsm) == 0 {
return nil, fmt.Errorf("newFileStore: no tsm files at path %q", path)
}
sort.Strings(sc.tsm)
sc.tombstone, err = filepath.Glob(filepath.Join(path, fmt.Sprintf("*.%s", "tombstone")))
if err != nil {
return nil, fmt.Errorf("error reading tombstone files: %v", err)
}
if err := sc.openFiles(); err != nil {
return nil, err
}
return sc, nil
}
func (sc *shardCompactor) openFiles() error {
sc.readers = make([]*tsm1.TSMReader, 0, len(sc.tsm))
// struct to hold the result of opening each reader in a goroutine
type res struct {
r *tsm1.TSMReader
err error
}
lim := limiter.NewFixed(runtime.GOMAXPROCS(0))
readerC := make(chan *res)
for i, fn := range sc.tsm {
file, err := os.OpenFile(fn, os.O_RDONLY, 0666)
if err != nil {
return fmt.Errorf("newFileStore: failed to open file %q: %v", fn, err)
}
go func(idx int, file *os.File) {
// Ensure a limited number of TSM files are loaded at once.
// Systems which have very large datasets (1TB+) can have thousands
// of TSM files which can cause extremely long load times.
lim.Take()
defer lim.Release()
start := time.Now()
df, err := tsm1.NewTSMReader(file)
sc.logger.Info("Opened file",
zap.String("path", file.Name()),
zap.Int("id", idx),
zap.Duration("duration", time.Since(start)))
// If we are unable to read a TSM file then log the error, rename
// the file, and continue loading the shard without it.
if err != nil {
sc.logger.Error("Cannot read corrupt tsm file, renaming", zap.String("path", file.Name()), zap.Int("id", idx), zap.Error(err))
if e := os.Rename(file.Name(), file.Name()+"."+tsm1.BadTSMFileExtension); e != nil {
sc.logger.Error("Cannot rename corrupt tsm file", zap.String("path", file.Name()), zap.Int("id", idx), zap.Error(e))
readerC <- &res{r: df, err: fmt.Errorf("cannot rename corrupt file %s: %v", file.Name(), e)}
return
}
}
readerC <- &res{r: df}
}(i, file)
}
for range sc.tsm {
res := <-readerC
if res.err != nil {
return res.err
} else if res.r == nil {
continue
}
sc.readers = append(sc.readers, res.r)
sc.files[res.r.Path()] = res.r
}
close(readerC)
sort.Sort(tsmReaders(sc.readers))
return nil
}
func (sc *shardCompactor) CompactShard() (err error) {
c := tsm1.NewCompactor()
c.Dir = sc.path
c.Size = tsm1.DefaultSegmentSize
c.FileStore = sc
c.Open()
tsmFiles, err := c.CompactFull(sc.tsm)
if err == nil {
sc.newTSM, err = sc.replace(tsmFiles)
}
return err
}
// replace replaces the existing shard files with temporary tsmFiles
func (sc *shardCompactor) replace(tsmFiles []string) ([]string, error) {
// rename .tsm.tmp → .tsm
var newNames []string
for _, file := range tsmFiles {
var newName = file[:len(file)-4] // remove extension
if err := os.Rename(file, newName); err != nil {
return nil, err
}
newNames = append(newNames, newName)
}
var errs errlist.ErrorList
// close all readers
for _, r := range sc.readers {
r.Close()
}
sc.readers = nil
sc.files = nil
// remove existing .tsm and .tombstone
for _, file := range sc.tsm {
errs.Add(os.Remove(file))
}
for _, file := range sc.tombstone {
errs.Add(os.Remove(file))
}
return newNames, errs.Err()
}
func (sc *shardCompactor) NextGeneration() int {
panic("not implemented")
}
func (sc *shardCompactor) TSMReader(path string) *tsm1.TSMReader {
r := sc.files[path]
if r != nil {
r.Ref()
}
return r
}
func (sc *shardCompactor) String() string {
var sb bytes.Buffer
sb.WriteString("TSM:\n")
for _, f := range sc.tsm {
sb.WriteString(" ")
sb.WriteString(f)
sb.WriteByte('\n')
}
if len(sc.tombstone) > 0 {
sb.WriteString("\nTombstone:\n")
for _, f := range sc.tombstone {
sb.WriteString(" ")
sb.WriteString(f)
sb.WriteByte('\n')
}
}
return sb.String()
}
type tsmReaders []*tsm1.TSMReader
func (a tsmReaders) Len() int { return len(a) }
func (a tsmReaders) Less(i, j int) bool { return a[i].Path() < a[j].Path() }
func (a tsmReaders) Swap(i, j int) { a[i], a[j] = a[j], a[i] }

76
cmd/influx_tools/export/README.md
View File

@ -1,76 +0,0 @@
`influx-tools export`
=====================
Used with `influx-tools import`, the export tool transforms existing shards to a new shard duration in order to
consolidate into fewer shards. It is also possible to separate into a greater number of shards.
Field type conflicts
--------------------
A field type for a given measurement can be different per shard. This creates the potential for field type conflicts
when exporting new shard durations. If this happens, the field type will be determined by the first shard containing
data for that field in order to fulfil the target shard duration. All conflicting data will be written as line protocol
and gzipped to the path specified by `-conflict-path`, unless the `-no-conflict-path` option is specified.
Range
-----
The optional `-range <range>` option specifies which target shards should be exported, based on their sequence number.
A use case for the `range` option is to parallelize the reshaping of a large data set. A machine with 4 cores could run two
export / import jobs concurrently over a subset of the total target shards.
The sequence number is included in the plan output. For example:
```sh
$ influx-tools export -config config.toml -database foo -rp autogen -duration 24h -format=binary -no-conflict-path -print-only
Source data from: 2018-02-19 00:00:00 +0000 UTC -> 2018-04-09 00:00:00 +0000 UTC
Converting source from 4 shard group(s) to 28 shard groups
Seq # ID Start End
0 608 2018-02-19 00:00:00 +0000 UTC 2018-02-26 00:00:00 +0000 UTC
1 609 2018-03-19 00:00:00 +0000 UTC 2018-03-26 00:00:00 +0000 UTC
2 610 2018-03-26 00:00:00 +0000 UTC 2018-04-02 00:00:00 +0000 UTC
3 612 2018-04-02 00:00:00 +0000 UTC 2018-04-09 00:00:00 +0000 UTC
Seq # ID Start End
0 0 2018-02-19 00:00:00 +0000 UTC 2018-02-20 00:00:00 +0000 UTC
1 1 2018-02-20 00:00:00 +0000 UTC 2018-02-21 00:00:00 +0000 UTC
2 2 2018-02-21 00:00:00 +0000 UTC 2018-02-22 00:00:00 +0000 UTC
3 3 2018-02-22 00:00:00 +0000 UTC 2018-02-23 00:00:00 +0000 UTC
4 4 2018-02-23 00:00:00 +0000 UTC 2018-02-24 00:00:00 +0000 UTC
5 5 2018-02-24 00:00:00 +0000 UTC 2018-02-25 00:00:00 +0000 UTC
6 6 2018-02-25 00:00:00 +0000 UTC 2018-02-26 00:00:00 +0000 UTC
7 28 2018-03-19 00:00:00 +0000 UTC 2018-03-20 00:00:00 +0000 UTC
8 29 2018-03-20 00:00:00 +0000 UTC 2018-03-21 00:00:00 +0000 UTC
9 30 2018-03-21 00:00:00 +0000 UTC 2018-03-22 00:00:00 +0000 UTC
...
26 47 2018-04-07 00:00:00 +0000 UTC 2018-04-08 00:00:00 +0000 UTC
27 48 2018-04-08 00:00:00 +0000 UTC 2018-04-09 00:00:00 +0000 UTC
```
Adding `-range 2-4` would return the following plan:
```sh
$ influx-tools export -config config.toml -database foo -rp autogen -duration 24h -format=binary -no-conflict-path -print-only -range=2-4
Source data from: 2018-02-19 00:00:00 +0000 UTC -> 2018-04-09 00:00:00 +0000 UTC
Converting source from 4 shard group(s) to 3 shard groups
Seq # ID Start End
0 608 2018-02-19 00:00:00 +0000 UTC 2018-02-26 00:00:00 +0000 UTC
1 609 2018-03-19 00:00:00 +0000 UTC 2018-03-26 00:00:00 +0000 UTC
2 610 2018-03-26 00:00:00 +0000 UTC 2018-04-02 00:00:00 +0000 UTC
3 612 2018-04-02 00:00:00 +0000 UTC 2018-04-09 00:00:00 +0000 UTC
Seq # ID Start End
2 2 2018-02-21 00:00:00 +0000 UTC 2018-02-22 00:00:00 +0000 UTC
3 3 2018-02-22 00:00:00 +0000 UTC 2018-02-23 00:00:00 +0000 UTC
4 4 2018-02-23 00:00:00 +0000 UTC 2018-02-24 00:00:00 +0000 UTC
```
A range can either be a single sequence number or an interval as shown previously.
**Hint**: Include the `-print-only` option to display the plan and exit without exporting any data.

48
cmd/influx_tools/export/bucket.go
View File

@ -1,48 +0,0 @@
package export
import (
"time"
"github.com/influxdata/influxdb/services/meta"
)
func makeShardGroupsForDuration(min, max time.Time, d time.Duration) meta.ShardGroupInfos {
start := min.Truncate(d).UTC()
end := max.Truncate(d).Add(d).UTC()
groups := make(meta.ShardGroupInfos, end.Sub(start)/d)
var i uint64
for start.Before(end) {
groups[i] = meta.ShardGroupInfo{
ID: i,
StartTime: start,
EndTime: start.Add(d),
}
i++
start = start.Add(d)
}
return groups[:i]
}
// PlanShardGroups creates a new ShardGroup set using a shard group duration of d, for the time spanning min to max.
func planShardGroups(sourceShards []meta.ShardGroupInfo, min, max time.Time, d time.Duration) meta.ShardGroupInfos {
groups := makeShardGroupsForDuration(min, max, d)
var target []meta.ShardGroupInfo
for i := 0; i < len(groups); i++ {
g := groups[i]
// NOTE: EndTime.Add(-1) matches the Contains interval of [start, end)
if hasShardsGroupForTimeRange(sourceShards, g.StartTime, g.EndTime.Add(-1)) {
target = append(target, g)
}
}
return target
}
func hasShardsGroupForTimeRange(groups []meta.ShardGroupInfo, min, max time.Time) bool {
for _, g := range groups {
if g.Overlaps(min, max) {
return true
}
}
return false
}

116
cmd/influx_tools/export/bucket_test.go
View File

@ -1,116 +0,0 @@
package export
import (
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/influxdata/influxdb/services/meta"
)
func ts(s int64) time.Time { return time.Unix(s, 0).UTC() }
func ms(times ...int64) meta.ShardGroupInfos {
sgis := make(meta.ShardGroupInfos, len(times)-1)
for i := range sgis {
sgis[i] = meta.ShardGroupInfo{ID: uint64(i), StartTime: ts(times[i]), EndTime: ts(times[i+1])}
}
return sgis
}
func TestMakeShardGroupsForDuration(t *testing.T) {
tests := []struct {
name string
min time.Time
max time.Time
d time.Duration
exp meta.ShardGroupInfos
}{
{
min: ts(15),
max: ts(25),
d: 10 * time.Second,
exp: ms(10, 20, 30),
},
{
min: ts(15),
max: ts(20),
d: 10 * time.Second,
exp: ms(10, 20, 30),
},
{
min: ts(15),
max: ts(17),
d: 10 * time.Second,
exp: ms(10, 20),
},
{
min: ts(10),
max: ts(20),
d: 10 * time.Second,
exp: ms(10, 20, 30),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := makeShardGroupsForDuration(tt.min, tt.max, tt.d); !cmp.Equal(got, tt.exp) {
t.Errorf("unexpected value -got/+exp\n%s", cmp.Diff(got, tt.exp))
}
})
}
}
func ms2(times ...int64) meta.ShardGroupInfos {
sgis := make(meta.ShardGroupInfos, len(times)/2)
for i := 0; i < len(times); i += 2 {
sgis[i/2] = meta.ShardGroupInfo{ID: uint64(i / 2), StartTime: ts(times[i]), EndTime: ts(times[i+1])}
}
return sgis
}
func shardGroupEqual(x, y meta.ShardGroupInfo) bool {
return x.StartTime == y.StartTime && x.EndTime == y.EndTime
}
func TestPlanShardGroups(t *testing.T) {
tests := []struct {
name string
g meta.ShardGroupInfos
d time.Duration
exp meta.ShardGroupInfos
}{
{
name: "20s->10s.nogap",
g: ms2(20, 40, 40, 60, 60, 80),
d: 10 * time.Second,
exp: ms2(20, 30, 30, 40, 40, 50, 50, 60, 60, 70, 70, 80),
},
{
name: "20s->10s.gap",
g: ms2(20, 40, 60, 80, 80, 100),
d: 10 * time.Second,
exp: ms2(20, 30, 30, 40, 60, 70, 70, 80, 80, 90, 90, 100),
},
{
name: "05s->10s.nogap",
g: ms2(15, 20, 20, 25, 25, 30),
d: 10 * time.Second,
exp: ms2(10, 20, 20, 30),
},
{
name: "05s->10s.gap",
g: ms2(15, 20, 20, 25, 50, 55, 55, 60),
d: 10 * time.Second,
exp: ms2(10, 20, 20, 30, 50, 60),
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
min, max := tc.g[0].StartTime, tc.g[len(tc.g)-1].EndTime
got := planShardGroups(tc.g, min, max, tc.d)
if !cmp.Equal(got, tc.exp, cmp.Comparer(shardGroupEqual)) {
t.Errorf("unexpected value -got/+exp\n%s", cmp.Diff(got, tc.exp))
}
})
}
}

217
cmd/influx_tools/export/command.go
View File

@ -1,217 +0,0 @@
package export
import (
"compress/gzip"
"errors"
"flag"
"fmt"
"io"
"math"
"os"
"strconv"
"strings"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/binary"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/line"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/text"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
"go.uber.org/zap"
)
var (
_ line.Writer
_ binary.Writer
)
// Command represents the program execution for "store query".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdout io.Writer
Logger *zap.Logger
server server.Interface
conflicts io.WriteCloser
configPath string
database string
rp string
shardDuration time.Duration
format string
r rangeValue
conflictPath string
ignore bool
print bool
}
// NewCommand returns a new instance of the export Command.
func NewCommand(server server.Interface) *Command {
return &Command{
Stderr: os.Stderr,
Stdout: os.Stdout,
server: server,
}
}
// Run executes the export command using the specified args.
func (cmd *Command) Run(args []string) (err error) {
err = cmd.parseFlags(args)
if err != nil {
return err
}
err = cmd.server.Open(cmd.configPath)
if err != nil {
return err
}
defer cmd.server.Close()
e, err := cmd.openExporter()
if err != nil {
return err
}
defer e.Close()
e.PrintPlan(cmd.Stderr)
if cmd.print {
return nil
}
if !cmd.ignore {
if f, err := os.Create(cmd.conflictPath); err != nil {
return err
} else {
cmd.conflicts = gzip.NewWriter(f)
defer func() {
cmd.conflicts.Close()
f.Close()
}()
}
}
var wr format.Writer
switch cmd.format {
case "line":
wr = line.NewWriter(cmd.Stdout)
case "binary":
wr = binary.NewWriter(cmd.Stdout, cmd.database, cmd.rp, cmd.shardDuration)
case "series":
wr = text.NewWriter(cmd.Stdout, text.Series)
case "values":
wr = text.NewWriter(cmd.Stdout, text.Values)
case "discard":
wr = format.Discard
}
defer func() {
err = wr.Close()
}()
if cmd.conflicts != nil {
wr = format.NewConflictWriter(wr, line.NewWriter(cmd.conflicts))
} else {
wr = format.NewConflictWriter(wr, format.DevNull)
}
return e.WriteTo(wr)
}
func (cmd *Command) openExporter() (*exporter, error) {
cfg := &exporterConfig{Database: cmd.database, RP: cmd.rp, ShardDuration: cmd.shardDuration, Min: cmd.r.Min(), Max: cmd.r.Max()}
e, err := newExporter(cmd.server, cfg)
if err != nil {
return nil, err
}
return e, e.Open()
}
func (cmd *Command) parseFlags(args []string) error {
fs := flag.NewFlagSet("export", flag.ContinueOnError)
fs.StringVar(&cmd.configPath, "config", "", "Config file")
fs.StringVar(&cmd.database, "database", "", "Database name")
fs.StringVar(&cmd.rp, "rp", "", "Retention policy name")
fs.StringVar(&cmd.format, "format", "line", "Output format (line, binary)")
fs.StringVar(&cmd.conflictPath, "conflict-path", "", "File name for writing field conflicts using line protocol and gzipped")
fs.BoolVar(&cmd.ignore, "no-conflict-path", false, "Disable writing field conflicts to a file")
fs.Var(&cmd.r, "range", "Range of target shards to export (default: all)")
fs.BoolVar(&cmd.print, "print-only", false, "Print plan to stderr and exit")
fs.DurationVar(&cmd.shardDuration, "duration", time.Hour*24*7, "Target shard duration")
if err := fs.Parse(args); err != nil {
return err
}
if cmd.database == "" {
return errors.New("database is required")
}
switch cmd.format {
case "line", "binary", "series", "values", "discard":
default:
return fmt.Errorf("invalid format '%s'", cmd.format)
}
if cmd.conflictPath == "" && !cmd.ignore {
return errors.New("missing conflict-path")
}
return nil
}
type rangeValue struct {
min, max uint64
set bool
}
func (rv *rangeValue) Min() uint64 { return rv.min }
func (rv *rangeValue) Max() uint64 {
if !rv.set {
return math.MaxUint64
}
return rv.max
}
func (rv *rangeValue) String() string {
if rv.Min() == rv.Max() {
return fmt.Sprint(rv.min)
}
return fmt.Sprintf("[%d,%d]", rv.Min(), rv.Max())
}
func (rv *rangeValue) Set(v string) (err error) {
p := strings.Split(v, "-")
switch {
case len(p) == 1:
rv.min, err = strconv.ParseUint(p[0], 10, 64)
if err != nil {
return fmt.Errorf("range error: invalid number %s", v)
}
rv.max = rv.min
case len(p) == 2:
rv.min, err = strconv.ParseUint(p[0], 10, 64)
if err != nil {
return fmt.Errorf("range error: min value %q is not a positive number", p[0])
}
rv.max = math.MaxUint64
if len(p[1]) > 0 {
rv.max, err = strconv.ParseUint(p[1], 10, 64)
if err != nil {
return fmt.Errorf("range error: max value %q is not empty or a positive number", p[1])
}
}
default:
return fmt.Errorf("range error: %q is not a valid range", v)
}
if rv.min > rv.max {
return errors.New("range error: min > max")
}
rv.set = true
return nil
}

236
cmd/influx_tools/export/exporter.go
View File

@ -1,236 +0,0 @@
package export
import (
"context"
"fmt"
"io"
"sort"
"text/tabwriter"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/storage"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/services/meta"
"github.com/influxdata/influxdb/tsdb"
)
type exporterConfig struct {
Database string
RP string
ShardDuration time.Duration
Min, Max uint64
}
type exporter struct {
metaClient server.MetaClient
tsdbStore *tsdb.Store
store *storage.Store
min, max uint64
db, rp string
d time.Duration
sourceGroups []meta.ShardGroupInfo
targetGroups []meta.ShardGroupInfo
// source data time range
startDate time.Time
endDate time.Time
}
func newExporter(server server.Interface, cfg *exporterConfig) (*exporter, error) {
client := server.MetaClient()
dbi := client.Database(cfg.Database)
if dbi == nil {
return nil, fmt.Errorf("database '%s' does not exist", cfg.Database)
}
if cfg.RP == "" {
// select default RP
cfg.RP = dbi.DefaultRetentionPolicy
}
rpi, err := client.RetentionPolicy(cfg.Database, cfg.RP)
if rpi == nil || err != nil {
return nil, fmt.Errorf("retention policy '%s' does not exist", cfg.RP)
}
store := tsdb.NewStore(server.TSDBConfig().Dir)
if server.Logger() != nil {
store.WithLogger(server.Logger())
}
store.EngineOptions.MonitorDisabled = true
store.EngineOptions.CompactionDisabled = true
store.EngineOptions.Config = server.TSDBConfig()
store.EngineOptions.EngineVersion = server.TSDBConfig().Engine
store.EngineOptions.IndexVersion = server.TSDBConfig().Index
store.EngineOptions.DatabaseFilter = func(database string) bool {
return database == cfg.Database
}
store.EngineOptions.RetentionPolicyFilter = func(_, rp string) bool {
return rp == cfg.RP
}
store.EngineOptions.ShardFilter = func(_, _ string, _ uint64) bool {
return false
}
return &exporter{
metaClient: client,
tsdbStore: store,
store: &storage.Store{TSDBStore: store},
min: cfg.Min,
max: cfg.Max,
db: cfg.Database,
rp: cfg.RP,
d: cfg.ShardDuration,
}, nil
}
func (e *exporter) Open() (err error) {
err = e.tsdbStore.Open()
if err != nil {
return err
}
err = e.loadShardGroups()
if err != nil {
return err
}
e.targetGroups = planShardGroups(e.sourceGroups, e.startDate, e.endDate, e.d)
if e.max >= uint64(len(e.targetGroups)) {
e.max = uint64(len(e.targetGroups) - 1)
}
if e.min > e.max {
return fmt.Errorf("invalid shard group range %d to %d", e.min, e.max)
}
e.targetGroups = e.targetGroups[e.min : e.max+1]
return nil
}
func (e *exporter) PrintPlan(w io.Writer) {
fmt.Fprintf(w, "Source data from: %s -> %s\n\n", e.startDate, e.endDate)
fmt.Fprintf(w, "Converting source from %d shard group(s) to %d shard groups:\n\n", len(e.sourceGroups), len(e.targetGroups))
e.printShardGroups(w, 0, e.sourceGroups)
fmt.Fprintln(w)
e.printShardGroups(w, int(e.min), e.targetGroups)
}
func (e *exporter) printShardGroups(w io.Writer, base int, target []meta.ShardGroupInfo) {
tw := tabwriter.NewWriter(w, 10, 8, 1, '\t', 0)
fmt.Fprintln(tw, "Seq #\tID\tStart\tEnd")
for i := 0; i < len(target); i++ {
g := target[i]
fmt.Fprintf(tw, "%d\t%d\t%s\t%s\n", i+base, g.ID, g.StartTime, g.EndTime)
}
tw.Flush()
}
func (e *exporter) SourceTimeRange() (time.Time, time.Time) { return e.startDate, e.endDate }
func (e *exporter) SourceShardGroups() []meta.ShardGroupInfo { return e.sourceGroups }
func (e *exporter) TargetShardGroups() []meta.ShardGroupInfo { return e.targetGroups }
func (e *exporter) loadShardGroups() error {
min := time.Unix(0, models.MinNanoTime)
max := time.Unix(0, models.MaxNanoTime)
groups, err := e.metaClient.NodeShardGroupsByTimeRange(e.db, e.rp, min, max)
if err != nil {
return err
}
if len(groups) == 0 {
return nil
}
sort.Sort(meta.ShardGroupInfos(groups))
e.sourceGroups = groups
e.startDate = groups[0].StartTime
e.endDate = groups[len(groups)-1].EndTime
return nil
}
func (e *exporter) shardsGroupsByTimeRange(min, max time.Time) []meta.ShardGroupInfo {
groups := make([]meta.ShardGroupInfo, 0, len(e.sourceGroups))
for _, g := range e.sourceGroups {
if !g.Overlaps(min, max) {
continue
}
groups = append(groups, g)
}
return groups
}
func (e *exporter) WriteTo(w format.Writer) error {
for _, g := range e.targetGroups {
min, max := g.StartTime, g.EndTime
rs, err := e.read(min, max.Add(-1))
if err != nil || rs == nil {
return err
}
format.WriteBucket(w, min.UnixNano(), max.UnixNano(), rs)
rs.Close()
}
return nil
}
// Read creates a ResultSet that reads all points with a timestamp ts, such that start ≤ ts < end.
func (e *exporter) read(min, max time.Time) (*storage.ResultSet, error) {
shards, err := e.getShards(min, max)
if err != nil {
return nil, err
}
req := storage.ReadRequest{
Database: e.db,
RP: e.rp,
Shards: shards,
Start: min.UnixNano(),
End: max.UnixNano(),
}
return e.store.Read(context.Background(), &req)
}
func (e *exporter) Close() error {
return e.tsdbStore.Close()
}
func (e *exporter) getShards(min, max time.Time) ([]*tsdb.Shard, error) {
groups := e.shardsGroupsByTimeRange(min, max)
var ids []uint64
for _, g := range groups {
for _, s := range g.Shards {
ids = append(ids, s.ID)
}
}
shards := e.tsdbStore.Shards(ids)
if len(shards) == len(ids) {
return shards, nil
}
return e.openStoreWithShardsIDs(ids)
}
func (e *exporter) openStoreWithShardsIDs(ids []uint64) ([]*tsdb.Shard, error) {
e.tsdbStore.Close()
e.tsdbStore.EngineOptions.ShardFilter = func(_, _ string, id uint64) bool {
for i := range ids {
if id == ids[i] {
return true
}
}
return false
}
if err := e.tsdbStore.Open(); err != nil {
return nil, err
}
return e.tsdbStore.Shards(ids), nil
}

163
cmd/influx_tools/generate/exec/command.go
View File

@ -1,163 +0,0 @@
package exec
import (
"context"
"errors"
"flag"
"fmt"
"io"
"math"
"os"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/generate"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/profile"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
"github.com/influxdata/influxdb/services/meta"
"github.com/influxdata/platform/pkg/data/gen"
)
// Command represents the program execution for "store query".
type Command struct {
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
deps Dependencies
server server.Interface
filter SeriesGeneratorFilter
configPath string
printOnly bool
noTSI bool
concurrency int
spec generate.Spec
profile profile.Config
}
type SeriesGeneratorFilter func(sgi meta.ShardGroupInfo, g SeriesGenerator) SeriesGenerator
type Dependencies struct {
Server server.Interface
// SeriesGeneratorFilter wraps g with a SeriesGenerator that
// returns a subset of keys from g
SeriesGeneratorFilter SeriesGeneratorFilter
}
// NewCommand returns a new instance of Command.
func NewCommand(deps Dependencies) *Command {
return &Command{
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
server: deps.Server,
filter: deps.SeriesGeneratorFilter,
}
}
func (cmd *Command) Run(args []string) (err error) {
err = cmd.parseFlags(args)
if err != nil {
return err
}
err = cmd.server.Open(cmd.configPath)
if err != nil {
return err
}
plan, err := cmd.spec.Plan(cmd.server)
if err != nil {
return err
}
plan.PrintPlan(cmd.Stdout)
if cmd.printOnly {
return nil
}
if err = plan.InitFileSystem(cmd.server.MetaClient()); err != nil {
return err
}
return cmd.exec(plan)
}
func (cmd *Command) parseFlags(args []string) error {
fs := flag.NewFlagSet("gen-init", flag.ContinueOnError)
fs.StringVar(&cmd.configPath, "config", "", "Config file")
fs.BoolVar(&cmd.printOnly, "print", false, "Print data spec only")
fs.BoolVar(&cmd.noTSI, "no-tsi", false, "Skip building TSI index")
fs.IntVar(&cmd.concurrency, "c", 1, "Number of shards to generate concurrently")
fs.StringVar(&cmd.profile.CPU, "cpuprofile", "", "Collect a CPU profile")
fs.StringVar(&cmd.profile.Memory, "memprofile", "", "Collect a memory profile")
cmd.spec.AddFlags(fs)
if err := fs.Parse(args); err != nil {
return err
}
if cmd.spec.Database == "" {
return errors.New("database is required")
}
if cmd.spec.Retention == "" {
return errors.New("retention policy is required")
}
return nil
}
func (cmd *Command) exec(p *generate.Plan) error {
groups := p.ShardGroups()
gens := make([]SeriesGenerator, len(groups))
for i := range gens {
var (
name []byte
keys []string
tv []gen.CountableSequence
)
name = []byte("m0")
tv = make([]gen.CountableSequence, len(p.Tags))
setTagVals(p.Tags, tv)
keys = make([]string, len(p.Tags))
setTagKeys("tag", keys)
sgi := groups[i]
vg := gen.NewIntegerConstantValuesSequence(p.PointsPerSeriesPerShard, sgi.StartTime, p.ShardDuration/time.Duration(p.PointsPerSeriesPerShard), 1)
gens[i] = NewSeriesGenerator(name, []byte("v0"), vg, gen.NewTagsValuesSequenceKeysValues(keys, tv))
if cmd.filter != nil {
gens[i] = cmd.filter(sgi, gens[i])
}
}
stop := cmd.profile.Start()
defer stop()
start := time.Now().UTC()
defer func() {
elapsed := time.Since(start)
fmt.Println()
fmt.Printf("Total time: %0.1f seconds\n", elapsed.Seconds())
}()
g := Generator{Concurrency: cmd.concurrency, BuildTSI: !cmd.noTSI}
return g.Run(context.Background(), p.Database, p.ShardPath(), p.NodeShardGroups(), gens)
}
func setTagVals(tags []int, tv []gen.CountableSequence) {
for j := range tags {
tv[j] = gen.NewCounterByteSequenceCount(tags[j])
}
}
func setTagKeys(prefix string, keys []string) {
tw := int(math.Ceil(math.Log10(float64(len(keys)))))
tf := fmt.Sprintf("%s%%0%dd", prefix, tw)
for i := range keys {
keys[i] = fmt.Sprintf(tf, i)
}
}

187
cmd/influx_tools/generate/exec/generator.go
View File

@ -1,187 +0,0 @@
package exec
import (
"context"
"fmt"
"path"
"path/filepath"
"strconv"
"sync"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/errlist"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/shard"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/services/meta"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"github.com/influxdata/influxdb/tsdb/index/tsi1"
)
type Generator struct {
Concurrency int
BuildTSI bool
sfile *tsdb.SeriesFile
}
func (g *Generator) Run(ctx context.Context, database, shardPath string, groups []meta.ShardGroupInfo, gens []SeriesGenerator) (err error) {
limit := make(chan struct{}, g.Concurrency)
for i := 0; i < g.Concurrency; i++ {
limit <- struct{}{}
}
var (
wg sync.WaitGroup
errs errlist.ErrorList
ch = make(chan error, len(groups))
)
dbPath := path.Dir(shardPath)
g.sfile = tsdb.NewSeriesFile(filepath.Join(dbPath, tsdb.SeriesFileDirectory))
if err := g.sfile.Open(); err != nil {
return err
}
defer g.sfile.Close()
g.sfile.DisableCompactions()
wg.Add(len(groups))
for i := 0; i < len(groups); i++ {
go func(n int) {
<-limit
defer func() {
wg.Done()
limit <- struct{}{}
}()
sgi := &groups[n]
if len(sgi.Shards) > 1 {
ch <- fmt.Errorf("multiple shards for the same owner %v", sgi.Shards[0].Owners)
return
}
id := sgi.Shards[0].ID
var (
idx seriesIndex
ti *tsi1.Index
)
if g.BuildTSI {
ti = tsi1.NewIndex(g.sfile, database, tsi1.WithPath(filepath.Join(shardPath, strconv.Itoa(int(id)), "index")))
if err := ti.Open(); err != nil {
ch <- fmt.Errorf("error opening TSI1 index %d: %s", id, err.Error())
return
}
idx = ti
} else {
idx = &seriesFileAdapter{sf: g.sfile, buf: make([]byte, 0, 2048)}
}
if err := g.writeShard(idx, gens[n], id, shardPath); err != nil {
ch <- fmt.Errorf("error writing shard %d: %s", id, err.Error())
}
if ti != nil {
ti.Compact()
ti.Wait()
if err := ti.Close(); err != nil {
ch <- fmt.Errorf("error compacting TSI1 index %d: %s", id, err.Error())
}
}
}(i)
}
wg.Wait()
close(ch)
for e := range ch {
errs.Add(e)
}
parts := g.sfile.Partitions()
wg.Add(len(parts))
ch = make(chan error, len(parts))
for i := range parts {
go func(n int) {
<-limit
defer func() {
wg.Done()
limit <- struct{}{}
}()
p := parts[n]
c := tsdb.NewSeriesPartitionCompactor()
if err := c.Compact(p); err != nil {
ch <- fmt.Errorf("error compacting series partition %d: %s", n, err.Error())
}
}(i)
}
wg.Wait()
close(ch)
for e := range ch {
errs.Add(e)
}
return errs.Err()
}
// seriesBatchSize specifies the number of series keys passed to the index.
const seriesBatchSize = 1000
func (g *Generator) writeShard(idx seriesIndex, sg SeriesGenerator, id uint64, path string) error {
sw := shard.NewWriter(id, path)
defer sw.Close()
var (
keys [][]byte
names [][]byte
tags []models.Tags
)
for sg.Next() {
seriesKey := sg.Key()
keys = append(keys, seriesKey)
names = append(names, sg.Name())
tags = append(tags, sg.Tags())
if len(keys) == seriesBatchSize {
if err := idx.CreateSeriesListIfNotExists(keys, names, tags); err != nil {
return err
}
keys = keys[:0]
names = names[:0]
tags = tags[:0]
}
vg := sg.ValuesGenerator()
key := tsm1.SeriesFieldKeyBytes(string(seriesKey), string(sg.Field()))
for vg.Next() {
sw.WriteV(key, vg.Values())
}
if err := sw.Err(); err != nil {
return err
}
}
if len(keys) > seriesBatchSize {
if err := idx.CreateSeriesListIfNotExists(keys, names, tags); err != nil {
return err
}
}
return nil
}
type seriesIndex interface {
CreateSeriesListIfNotExists(keys [][]byte, names [][]byte, tagsSlice []models.Tags) error
}
type seriesFileAdapter struct {
sf *tsdb.SeriesFile
buf []byte
}
func (s *seriesFileAdapter) CreateSeriesListIfNotExists(keys [][]byte, names [][]byte, tagsSlice []models.Tags) (err error) {
_, err = s.sf.CreateSeriesListIfNotExists(names, tagsSlice)
return
}

89
cmd/influx_tools/generate/exec/series_generator.go
View File

@ -1,89 +0,0 @@
package exec
import (
"github.com/influxdata/influxdb/models"
"github.com/influxdata/platform/pkg/data/gen"
)
type SeriesGenerator interface {
// Next advances the series generator to the next series key.
Next() bool
// Key returns the series key.
// The returned value may be cached.
Key() []byte
// Name returns the name of the measurement.
// The returned value may be cached.
Name() []byte
// Tags returns the tag set.
// The returned value may be cached.
Tags() models.Tags
// Field returns the name of the field.
// The returned value may be cached.
Field() []byte
// ValuesGenerator returns a values sequence for the current series.
ValuesGenerator() gen.ValuesSequence
}
type cache struct {
key []byte
tags models.Tags
}
type seriesGenerator struct {
name []byte
tags gen.TagsSequence
field []byte
vg gen.ValuesSequence
c cache
}
func NewSeriesGenerator(name []byte, field []byte, vg gen.ValuesSequence, tags gen.TagsSequence) SeriesGenerator {
return &seriesGenerator{
name: name,
field: field,
vg: vg,
tags: tags,
}
}
func (g *seriesGenerator) Next() bool {
if g.tags.Next() {
g.c = cache{}
g.vg.Reset()
return true
}
return false
}
func (g *seriesGenerator) Key() []byte {
if len(g.c.key) == 0 {
g.c.key = models.MakeKey(g.name, g.tags.Value())
}
return g.c.key
}
func (g *seriesGenerator) Name() []byte {
return g.name
}
func (g *seriesGenerator) Tags() models.Tags {
if len(g.c.tags) == 0 {
g.c.tags = g.tags.Value().Clone()
}
return g.c.tags
}
func (g *seriesGenerator) Field() []byte {
return g.field
}
func (g *seriesGenerator) ValuesGenerator() gen.ValuesSequence {
return g.vg
}

79
cmd/influx_tools/generate/init/command.go
View File

@ -1,79 +0,0 @@
package init
import (
"errors"
"flag"
"io"
"os"
"github.com/influxdata/influxdb/cmd/influx_tools/generate"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
)
// Command represents the program execution for "store query".
type Command struct {
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
server server.Interface
configPath string
printOnly bool
spec generate.Spec
}
// NewCommand returns a new instance of Command.
func NewCommand(server server.Interface) *Command {
return &Command{
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
server: server,
}
}
func (cmd *Command) Run(args []string) (err error) {
err = cmd.parseFlags(args)
if err != nil {
return err
}
err = cmd.server.Open(cmd.configPath)
if err != nil {
return err
}
plan, err := cmd.spec.Plan(cmd.server)
if err != nil {
return err
}
plan.PrintPlan(cmd.Stdout)
if !cmd.printOnly {
return plan.InitMetadata(cmd.server.MetaClient())
}
return nil
}
func (cmd *Command) parseFlags(args []string) error {
fs := flag.NewFlagSet("gen-init", flag.ContinueOnError)
fs.StringVar(&cmd.configPath, "config", "", "Config file")
fs.BoolVar(&cmd.printOnly, "print", false, "Print data spec only")
cmd.spec.AddFlags(fs)
if err := fs.Parse(args); err != nil {
return err
}
if cmd.spec.Database == "" {
return errors.New("database is required")
}
if cmd.spec.Retention == "" {
return errors.New("retention policy is required")
}
return nil
}

230
cmd/influx_tools/generate/plan.go
View File

@ -1,230 +0,0 @@
package generate
import (
"fmt"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"text/tabwriter"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/errlist"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
"github.com/influxdata/influxdb/services/meta"
"github.com/pkg/errors"
)
type Plan struct {
Database string
Retention string
ReplicaN int
StartTime time.Time
ShardCount int
ShardDuration time.Duration
Tags TagCardinalities
PointsPerSeriesPerShard int
DatabasePath string
info *meta.DatabaseInfo
groups []meta.ShardGroupInfo
}
func (p *Plan) String() string {
sb := new(strings.Builder)
p.PrintPlan(sb)
return sb.String()
}
func (p *Plan) PrintPlan(w io.Writer) {
tw := tabwriter.NewWriter(w, 25, 4, 2, ' ', 0)
fmt.Fprintf(tw, "Data Path\t%s\n", p.ShardPath())
fmt.Fprintf(tw, "Tag cardinalities\t%s\n", p.Tags)
fmt.Fprintf(tw, "Points per series per shard\t%d\n", p.PointsPerSeriesPerShard)
fmt.Fprintf(tw, "Total points per shard\t%d\n", p.Tags.Cardinality()*p.PointsPerSeriesPerShard)
fmt.Fprintf(tw, "Total series\t%d\n", p.Tags.Cardinality())
fmt.Fprintf(tw, "Total points\t%d\n", p.Tags.Cardinality()*p.ShardCount*p.PointsPerSeriesPerShard)
fmt.Fprintf(tw, "Shard Count\t%d\n", p.ShardCount)
fmt.Fprintf(tw, "Database\t%s/%s (Shard duration: %s)\n", p.Database, p.Retention, p.ShardDuration)
fmt.Fprintf(tw, "Start time\t%s\n", p.StartTime)
fmt.Fprintf(tw, "End time\t%s\n", p.EndTime())
tw.Flush()
}
func (p *Plan) ShardPath() string {
return filepath.Join(p.DatabasePath, p.Retention)
}
// TimeSpan returns the total duration for which the data set.
func (p *Plan) TimeSpan() time.Duration {
return p.ShardDuration * time.Duration(p.ShardCount)
}
func (p *Plan) EndTime() time.Time {
return p.StartTime.Add(p.TimeSpan())
}
func (p *Plan) InitMetadata(client server.MetaClient) (err error) {
if err = client.DropDatabase(p.Database); err != nil {
return err
}
rp := meta.RetentionPolicySpec{
Name: p.Retention,
ShardGroupDuration: p.ShardDuration,
ReplicaN: &p.ReplicaN,
}
info, err := client.CreateDatabaseWithRetentionPolicy(p.Database, &rp)
if err != nil {
return err
}
return p.createShardGroupMetadata(client, info.DefaultRetentionPolicy)
}
// InitFileSystem initializes the file system structure, cleaning up
// existing files and re-creating the appropriate shard directories.
func (p *Plan) InitFileSystem(client server.MetaClient) error {
var err error
if err = os.RemoveAll(p.DatabasePath); err != nil {
return err
}
minT, maxT := p.TimeRange()
groups, err := client.NodeShardGroupsByTimeRange(p.Database, p.Retention, minT, maxT)
if err != nil {
return err
}
p.groups = groups
for i := 0; i < len(groups); i++ {
sgi := &groups[i]
if len(sgi.Shards) > 1 {
return fmt.Errorf("multiple shards for the same owner %v", sgi.Shards[0].Owners)
}
if err = os.MkdirAll(filepath.Join(p.ShardPath(), strconv.Itoa(int(sgi.Shards[0].ID))), 0777); err != nil {
return err
}
}
p.info = client.Database(p.Database)
return nil
}
// NodeShardGroups returns ShardGroupInfo with Shards limited to the current node
func (p *Plan) NodeShardGroups() []meta.ShardGroupInfo {
return p.groups
}
func (p *Plan) ShardGroups() []meta.ShardGroupInfo {
return p.info.RetentionPolicy(p.info.DefaultRetentionPolicy).ShardGroups
}
func (p *Plan) createShardGroupMetadata(client server.MetaClient, rp string) error {
ts := p.StartTime.Truncate(p.ShardDuration).UTC()
var err error
groups := make([]*meta.ShardGroupInfo, p.ShardCount)
for i := 0; i < p.ShardCount; i++ {
groups[i], err = client.CreateShardGroup(p.Database, rp, ts)
if err != nil {
return err
}
ts = ts.Add(p.ShardDuration)
}
return nil
}
func (p *Plan) TimeRange() (start, end time.Time) {
start = p.StartTime.Truncate(p.ShardDuration).UTC()
end = start.Add(time.Duration(p.ShardDuration.Nanoseconds() * int64(p.ShardCount)))
return start, end
}
func (p *Plan) Validate() error {
// build default values
def := &planDefaults{}
WalkPlan(def, p)
// validate
val := &planValidator{}
WalkPlan(val, p)
return val.Err()
}
type Visitor interface {
Visit(node Node) Visitor
}
type Node interface{ node() }
func (*Plan) node() {}
func WalkPlan(v Visitor, node Node) {
if v = v.Visit(node); v == nil {
return
}
switch n := node.(type) {
case *Plan:
default:
panic(fmt.Sprintf("WalkConfig: unexpected node type %T", n))
}
}
type planValidator struct {
errs errlist.ErrorList
}
func (v *planValidator) Visit(node Node) Visitor {
switch n := node.(type) {
case *Plan:
if n.DatabasePath == "" {
v.errs.Add(errors.New("missing DataPath"))
}
if n.StartTime.Add(n.TimeSpan()).After(time.Now()) {
v.errs.Add(fmt.Errorf("start time must be ≤ %s", time.Now().Truncate(n.ShardDuration).UTC().Add(-n.TimeSpan())))
}
}
return v
}
func (v *planValidator) Err() error {
return v.errs.Err()
}
type planDefaults struct{}
func (v *planDefaults) Visit(node Node) Visitor {
switch n := node.(type) {
case *Plan:
if n.DatabasePath == "" {
n.DatabasePath = "${HOME}/.influxdb/data"
}
if n.Database == "" {
n.Database = "db"
}
if n.Retention == "" {
n.Retention = "autogen"
}
if n.ShardDuration == 0 {
n.ShardDuration = 24 * time.Hour
}
if n.ShardCount == 0 {
n.ShardCount = 1
}
if n.StartTime.IsZero() {
n.StartTime = time.Now().Truncate(n.ShardDuration).Add(-n.TimeSpan())
}
}
return v
}

92
cmd/influx_tools/generate/spec.go
View File

@ -1,92 +0,0 @@
package generate
import (
"flag"
"fmt"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
)
type TagCardinalities []int
func (t TagCardinalities) String() string {
s := make([]string, 0, len(t))
for i := 0; i < len(t); i++ {
s = append(s, strconv.Itoa(t[i]))
}
return fmt.Sprintf("[%s]", strings.Join(s, ","))
}
func (t TagCardinalities) Cardinality() int {
n := 1
for i := range t {
n *= t[i]
}
return n
}
func (t *TagCardinalities) Set(tags string) error {
*t = (*t)[:0]
for _, s := range strings.Split(tags, ",") {
v, err := strconv.Atoi(s)
if err != nil {
return fmt.Errorf("cannot parse tag cardinality: %s", s)
}
*t = append(*t, v)
}
return nil
}
type Spec struct {
StartTime string
Database string
Retention string
ReplicaN int
ShardCount int
ShardDuration time.Duration
Tags TagCardinalities
PointsPerSeriesPerShard int
}
func (a *Spec) AddFlags(fs *flag.FlagSet) {
fs.StringVar(&a.StartTime, "start-time", "", "Start time")
fs.StringVar(&a.Database, "db", "db", "Name of database to create")
fs.StringVar(&a.Retention, "rp", "rp", "Name of retention policy")
fs.IntVar(&a.ReplicaN, "rf", 1, "Replication factor")
fs.IntVar(&a.ShardCount, "shards", 1, "Number of shards to create")
fs.DurationVar(&a.ShardDuration, "shard-duration", 24*time.Hour, "Shard duration (default 24h)")
a.Tags = []int{10, 10, 10}
fs.Var(&a.Tags, "t", "Tag cardinality")
fs.IntVar(&a.PointsPerSeriesPerShard, "p", 100, "Points per series per shard")
}
func (a *Spec) Plan(server server.Interface) (*Plan, error) {
plan := &Plan{
Database: a.Database,
Retention: a.Retention,
ReplicaN: a.ReplicaN,
ShardCount: a.ShardCount,
ShardDuration: a.ShardDuration,
Tags: a.Tags,
PointsPerSeriesPerShard: a.PointsPerSeriesPerShard,
DatabasePath: filepath.Join(server.TSDBConfig().Dir, a.Database),
}
if a.StartTime != "" {
if t, err := time.Parse(time.RFC3339, a.StartTime); err != nil {
return nil, err
} else {
plan.StartTime = t.UTC()
}
}
if err := plan.Validate(); err != nil {
return nil, err
}
return plan, nil
}

43
cmd/influx_tools/help/help.go
View File

@ -1,43 +0,0 @@
// Package help is the help subcommand of the influxd command.
package help
import (
"fmt"
"io"
"os"
"strings"
)
// Command displays help for command-line sub-commands.
type Command struct {
Stdout io.Writer
}
// NewCommand returns a new instance of Command.
func NewCommand() *Command {
return &Command{
Stdout: os.Stdout,
}
}
// Run executes the command.
func (cmd *Command) Run(args ...string) error {
fmt.Fprintln(cmd.Stdout, strings.TrimSpace(usage))
return nil
}
const usage = `
Tools for managing and querying InfluxDB data.
Usage: influx-tools command [arguments]
The commands are:
export reshapes existing shards to a new shard duration
compact-shard fully compacts the specified shard
gen-init creates database and retention policy metadata
gen-exec generates data
help display this help message
Use "influx-tools command -help" for more information about a command.
`

12
cmd/influx_tools/importer/README.md
View File

@ -1,12 +0,0 @@
`influx-tools import`
=====================
The import tool consumes binary data produced by `influx-tools export -format
binary` to write data directly to disk possibly under a new retention policy.
This tool handles the binary format only - exports of line protocol data should
be handled using the existing endpoints. Influx should be offline while this
tool is run.
If the target retention policy already exists, the tool will error out if you
attempt to change the retention policy settings. However, it is possible to
replace on disk shards with the `-replace` option.

144
cmd/influx_tools/importer/command.go
View File

@ -1,144 +0,0 @@
package importer
import (
"errors"
"flag"
"io"
"os"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/errlist"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/binary"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
"github.com/influxdata/influxdb/services/meta"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"go.uber.org/zap"
)
// Command represents the program execution for "store query".
type Command struct {
// Standard input/output, overridden for testing.
Stderr io.Writer
Stdin io.Reader
Logger *zap.Logger
server server.Interface
configPath string
database string
retentionPolicy string
replication int
duration time.Duration
shardDuration time.Duration
buildTSI bool
replace bool
}
// NewCommand returns a new instance of Command.
func NewCommand(server server.Interface) *Command {
return &Command{
Stderr: os.Stderr,
Stdin: os.Stdin,
server: server,
}
}
// Run executes the import command using the specified args.
func (cmd *Command) Run(args []string) (err error) {
err = cmd.parseFlags(args)
if err != nil {
return err
}
err = cmd.server.Open(cmd.configPath)
if err != nil {
return err
}
i := newImporter(cmd.server, cmd.database, cmd.retentionPolicy, cmd.replace, cmd.buildTSI, cmd.Logger)
reader := binary.NewReader(cmd.Stdin)
_, err = reader.ReadHeader()
if err != nil {
return err
}
rp := &meta.RetentionPolicySpec{Name: cmd.retentionPolicy, ShardGroupDuration: cmd.shardDuration}
if cmd.duration >= time.Hour {
rp.Duration = &cmd.duration
}
if cmd.replication > 0 {
rp.ReplicaN = &cmd.replication
}
err = i.CreateDatabase(rp)
if err != nil {
return err
}
var bh *binary.BucketHeader
for bh, err = reader.NextBucket(); (bh != nil) && (err == nil); bh, err = reader.NextBucket() {
err = importShard(reader, i, bh.Start, bh.End)
if err != nil {
return err
}
}
return err
}
func importShard(reader *binary.Reader, i *importer, start int64, end int64) error {
err := i.StartShardGroup(start, end)
if err != nil {
return err
}
el := errlist.NewErrorList()
var sh *binary.SeriesHeader
var next bool
for sh, err = reader.NextSeries(); (sh != nil) && (err == nil); sh, err = reader.NextSeries() {
i.AddSeries(sh.SeriesKey)
pr := reader.Points()
seriesFieldKey := tsm1.SeriesFieldKeyBytes(string(sh.SeriesKey), string(sh.Field))
for next, err = pr.Next(); next && (err == nil); next, err = pr.Next() {
err = i.Write(seriesFieldKey, pr.Values())
if err != nil {
break
}
}
if err != nil {
break
}
}
el.Add(err)
el.Add(i.CloseShardGroup())
return el.Err()
}
func (cmd *Command) parseFlags(args []string) error {
fs := flag.NewFlagSet("import", flag.ContinueOnError)
fs.StringVar(&cmd.configPath, "config", "", "Config file")
fs.StringVar(&cmd.database, "database", "", "Database name")
fs.StringVar(&cmd.retentionPolicy, "rp", "", "Retention policy")
fs.IntVar(&cmd.replication, "replication", 0, "Retention policy replication")
fs.DurationVar(&cmd.duration, "duration", time.Hour*0, "Retention policy duration")
fs.DurationVar(&cmd.shardDuration, "shard-duration", time.Hour*24*7, "Retention policy shard duration")
fs.BoolVar(&cmd.buildTSI, "build-tsi", false, "Build the on disk TSI")
fs.BoolVar(&cmd.replace, "replace", false, "Enables replacing an existing retention policy")
if err := fs.Parse(args); err != nil {
return err
}
if cmd.database == "" {
return errors.New("database is required")
}
if cmd.retentionPolicy == "" {
return errors.New("retention policy is required")
}
return nil
}

233
cmd/influx_tools/importer/importer.go
View File

@ -1,233 +0,0 @@
package importer
import (
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/errlist"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/shard"
"github.com/influxdata/influxdb/cmd/influx_tools/server"
"github.com/influxdata/influxdb/services/meta"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"go.uber.org/zap"
)
type importer struct {
MetaClient server.MetaClient
db string
dataDir string
replace bool
rpi *meta.RetentionPolicyInfo
log *zap.Logger
skipShard bool
currentShard uint64
sh *shard.Writer
sfile *tsdb.SeriesFile
sw *seriesWriter
buildTsi bool
seriesBuf []byte
}
const seriesBatchSize = 1000
func newImporter(server server.Interface, db string, rp string, replace bool, buildTsi bool, log *zap.Logger) *importer {
i := &importer{MetaClient: server.MetaClient(), db: db, dataDir: server.TSDBConfig().Dir, replace: replace, buildTsi: buildTsi, log: log, skipShard: false}
if !buildTsi {
i.seriesBuf = make([]byte, 0, 2048)
}
return i
}
func (i *importer) Close() error {
el := errlist.NewErrorList()
if i.sh != nil {
el.Add(i.CloseShardGroup())
}
return el.Err()
}
func (i *importer) CreateDatabase(rp *meta.RetentionPolicySpec) error {
var rpi *meta.RetentionPolicyInfo
dbInfo := i.MetaClient.Database(i.db)
if dbInfo == nil {
return i.createDatabaseWithRetentionPolicy(rp)
}
rpi, err := i.MetaClient.RetentionPolicy(i.db, rp.Name)
if err != nil {
return err
}
nonmatchingRp := (rpi != nil) && ((rp.Duration != nil && rpi.Duration != *rp.Duration) ||
(rp.ReplicaN != nil && rpi.ReplicaN != *rp.ReplicaN) ||
(rpi.ShardGroupDuration != rp.ShardGroupDuration))
if nonmatchingRp {
return fmt.Errorf("retention policy %v already exists with different parameters", rp.Name)
} else {
if _, err := i.MetaClient.CreateRetentionPolicy(i.db, rp, false); err != nil {
return err
}
}
i.rpi, err = i.MetaClient.RetentionPolicy(i.db, rp.Name)
return err
}
func (i *importer) createDatabaseWithRetentionPolicy(rp *meta.RetentionPolicySpec) error {
var err error
var dbInfo *meta.DatabaseInfo
if len(rp.Name) == 0 {
dbInfo, err = i.MetaClient.CreateDatabase(i.db)
} else {
dbInfo, err = i.MetaClient.CreateDatabaseWithRetentionPolicy(i.db, rp)
}
if err != nil {
return err
}
i.rpi = dbInfo.RetentionPolicy(rp.Name)
return nil
}
func (i *importer) StartShardGroup(start int64, end int64) error {
existingSg, err := i.MetaClient.NodeShardGroupsByTimeRange(i.db, i.rpi.Name, time.Unix(0, start), time.Unix(0, end))
if err != nil {
return err
}
var sgi *meta.ShardGroupInfo
var shardID uint64
shardsPath := i.shardPath(i.rpi.Name)
var shardPath string
if len(existingSg) > 0 {
sgi = &existingSg[0]
if len(sgi.Shards) > 1 {
return fmt.Errorf("multiple shards for the same owner %v and time range %v to %v", sgi.Shards[0].Owners, start, end)
}
shardID = sgi.Shards[0].ID
shardPath = filepath.Join(shardsPath, strconv.Itoa(int(shardID)))
_, err = os.Stat(shardPath)
if err != nil {
if !os.IsNotExist(err) {
return err
}
} else {
if i.replace {
if err := os.RemoveAll(shardPath); err != nil {
return err
}
} else {
if i.log != nil {
i.log.Error(fmt.Sprintf("shard %d already exists, skipping over new shard data", sgi.ID))
}
i.skipShard = true
return nil
}
}
} else {
sgi, err = i.MetaClient.CreateShardGroup(i.db, i.rpi.Name, time.Unix(0, start))
if err != nil {
return err
}
shardID = sgi.Shards[0].ID
}
shardPath = filepath.Join(shardsPath, strconv.Itoa(int(shardID)))
if err = os.MkdirAll(shardPath, 0777); err != nil {
return err
}
i.skipShard = false
i.sh = shard.NewWriter(shardID, shardsPath)
i.currentShard = shardID
i.startSeriesFile()
return nil
}
func (i *importer) shardPath(rp string) string {
return filepath.Join(i.dataDir, i.db, rp)
}
func (i *importer) removeShardGroup(rp string, shardID uint64) error {
shardPath := i.shardPath(rp)
err := os.RemoveAll(filepath.Join(shardPath, strconv.Itoa(int(shardID))))
return err
}
func (i *importer) Write(key []byte, values tsm1.Values) error {
if i.skipShard {
return nil
}
if i.sh == nil {
return errors.New("importer not currently writing a shard")
}
i.sh.Write(key, values)
if i.sh.Err() != nil {
el := errlist.NewErrorList()
el.Add(i.sh.Err())
el.Add(i.CloseShardGroup())
el.Add(i.removeShardGroup(i.rpi.Name, i.currentShard))
i.sh = nil
i.currentShard = 0
return el.Err()
}
return nil
}
func (i *importer) CloseShardGroup() error {
if i.skipShard {
i.skipShard = false
return nil
}
el := errlist.NewErrorList()
el.Add(i.closeSeriesFile())
i.sh.Close()
if i.sh.Err() != nil {
el.Add(i.sh.Err())
}
i.sh = nil
return el.Err()
}
func (i *importer) startSeriesFile() error {
dataPath := filepath.Join(i.dataDir, i.db)
shardPath := filepath.Join(i.dataDir, i.db, i.rpi.Name)
i.sfile = tsdb.NewSeriesFile(filepath.Join(dataPath, tsdb.SeriesFileDirectory))
if err := i.sfile.Open(); err != nil {
return err
}
var err error
if i.buildTsi {
i.sw, err = newTSI1SeriesWriter(i.sfile, i.db, dataPath, shardPath, int(i.sh.ShardID()))
} else {
i.sw, err = newInMemSeriesWriter(i.sfile, i.db, dataPath, shardPath, int(i.sh.ShardID()), i.seriesBuf)
}
if err != nil {
return err
}
return nil
}
func (i *importer) AddSeries(seriesKey []byte) error {
if i.skipShard {
return nil
}
return i.sw.AddSeries(seriesKey)
}
func (i *importer) closeSeriesFile() error {
return i.sw.Close()
}

114
cmd/influx_tools/importer/series_writer.go
View File

@ -1,114 +0,0 @@
package importer
import (
"fmt"
"path/filepath"
"strconv"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/errlist"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
"github.com/influxdata/influxdb/tsdb/index/tsi1"
)
type seriesWriter struct {
keys [][]byte
names [][]byte
tags []models.Tags
seriesBatchSize int
sfile *tsdb.SeriesFile
idx seriesIndex
}
func newInMemSeriesWriter(sfile *tsdb.SeriesFile, db string, dataPath string, shardPath string, shardID int, buf []byte) (*seriesWriter, error) {
return &seriesWriter{seriesBatchSize: seriesBatchSize, sfile: sfile, idx: &seriesFileAdapter{sf: sfile, buf: buf}}, nil
}
func newTSI1SeriesWriter(sfile *tsdb.SeriesFile, db string, dataPath string, shardPath string, shardID int) (*seriesWriter, error) {
ti := tsi1.NewIndex(sfile, db, tsi1.WithPath(filepath.Join(shardPath, strconv.Itoa(shardID), "index")))
if err := ti.Open(); err != nil {
return nil, fmt.Errorf("error opening TSI1 index %d: %s", shardID, err.Error())
}
return &seriesWriter{seriesBatchSize: seriesBatchSize, sfile: sfile, idx: &tsi1Adapter{ti: ti}}, nil
}
func (sw *seriesWriter) AddSeries(key []byte) error {
seriesKey, _ := tsm1.SeriesAndFieldFromCompositeKey(key)
sw.keys = append(sw.keys, seriesKey)
name, tag := models.ParseKeyBytes(seriesKey)
sw.names = append(sw.names, name)
sw.tags = append(sw.tags, tag)
if len(sw.keys) == sw.seriesBatchSize {
if err := sw.idx.CreateSeriesListIfNotExists(sw.keys, sw.names, sw.tags); err != nil {
return err
}
sw.keys = sw.keys[:0]
sw.names = sw.names[:0]
sw.tags = sw.tags[:0]
}
return nil
}
func (sw *seriesWriter) Close() error {
el := errlist.NewErrorList()
el.Add(sw.idx.CreateSeriesListIfNotExists(sw.keys, sw.names, sw.tags))
el.Add(sw.idx.Compact())
el.Add(sw.idx.Close())
el.Add(sw.sfile.Close())
return el.Err()
}
type seriesIndex interface {
CreateSeriesListIfNotExists(keys [][]byte, names [][]byte, tagsSlice []models.Tags) (err error)
Compact() error
Close() error
}
type seriesFileAdapter struct {
sf *tsdb.SeriesFile
buf []byte
}
func (s *seriesFileAdapter) CreateSeriesListIfNotExists(keys [][]byte, names [][]byte, tagsSlice []models.Tags) (err error) {
_, err = s.sf.CreateSeriesListIfNotExists(names, tagsSlice)
return err
}
func (s *seriesFileAdapter) Compact() error {
parts := s.sf.Partitions()
for i, p := range parts {
c := tsdb.NewSeriesPartitionCompactor()
if err := c.Compact(p); err != nil {
return fmt.Errorf("error compacting series partition %d: %s", i, err.Error())
}
}
return nil
}
func (s *seriesFileAdapter) Close() error {
return nil
}
type tsi1Adapter struct {
ti *tsi1.Index
}
func (t *tsi1Adapter) CreateSeriesListIfNotExists(keys [][]byte, names [][]byte, tagsSlice []models.Tags) (err error) {
return t.ti.CreateSeriesListIfNotExists(keys, names, tagsSlice)
}
func (t *tsi1Adapter) Compact() error {
t.ti.Compact()
t.ti.Wait()
return nil
}
func (t *tsi1Adapter) Close() error {
return t.ti.Close()
}

37
cmd/influx_tools/internal/errlist/errlist.go
View File

@ -1,37 +0,0 @@
package errlist
import (
"bytes"
)
// ErrorList is a simple error aggregator to return multiple errors as one.
type ErrorList struct {
errs []error
}
func NewErrorList() *ErrorList {
return &ErrorList{errs: make([]error, 0)}
}
func (el *ErrorList) Add(err error) {
if err == nil {
return
}
el.errs = append(el.errs, err)
}
func (el *ErrorList) Err() error {
if len(el.errs) == 0 {
return nil
}
return el
}
func (el *ErrorList) Error() string {
var buf bytes.Buffer
for _, err := range el.errs {
buf.WriteString(err.Error())
buf.WriteByte('\n')
}
return buf.String()
}

2231
cmd/influx_tools/internal/format/binary/binary.pb.go
View File

File diff suppressed because it is too large Load Diff

72
cmd/influx_tools/internal/format/binary/binary.proto
View File

@ -1,72 +0,0 @@
syntax = "proto3";
package binary;
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
option (gogoproto.goproto_getters_all) = false;
message Header {
enum Version {
option (gogoproto.goproto_enum_prefix) = false;
VERSION_0 = 0 [(gogoproto.enumvalue_customname) = "Version0"];
}
Version version = 1;
string database = 2;
string retention_policy = 3;
int64 shard_duration = 4 [(gogoproto.stdduration) = true];
}
message BucketHeader {
sfixed64 start = 1;
sfixed64 end = 2;
}
message BucketFooter {
}
message FloatPoints {
repeated sfixed64 timestamps = 1;
repeated double values = 2;
}
message IntegerPoints {
repeated sfixed64 timestamps = 1;
repeated int64 values = 2;
}
message UnsignedPoints {
repeated sfixed64 timestamps = 1;
repeated uint64 values = 2;
}
message BooleanPoints {
repeated sfixed64 timestamps = 1;
repeated bool values = 2;
}
message StringPoints {
repeated sfixed64 timestamps = 1;
repeated string values = 2;
}
enum FieldType {
option (gogoproto.goproto_enum_prefix) = false;
FLOAT = 0 [(gogoproto.enumvalue_customname) = "FloatFieldType"];
INTEGER = 1 [(gogoproto.enumvalue_customname) = "IntegerFieldType"];
UNSIGNED = 2 [(gogoproto.enumvalue_customname) = "UnsignedFieldType"];
BOOLEAN = 3 [(gogoproto.enumvalue_customname) = "BooleanFieldType"];
STRING = 4 [(gogoproto.enumvalue_customname) = "StringFieldType"];
}
message SeriesHeader {
FieldType field_type = 1;
bytes series_key = 2;
bytes field = 3;
}
message SeriesFooter {
}

60
cmd/influx_tools/internal/format/binary/common.go
View File

@ -1,60 +0,0 @@
package binary
//go:generate protoc -I$GOPATH/src/github.com/influxdata/influxdb/vendor -I. --gogofaster_out=Mgoogle/protobuf/empty.proto=github.com/gogo/protobuf/types:. binary.proto
//go:generate stringer -type=MessageType
import "errors"
var (
ErrWriteAfterClose = errors.New("format/binary: write after close")
ErrWriteBucketAfterClose = errors.New("format/binary: write to closed bucket")
)
var (
Magic = [...]byte{0x49, 0x46, 0x4c, 0x58, 0x44, 0x55, 0x4d, 0x50} // IFLXDUMP
)
type MessageType byte
const (
HeaderType MessageType = iota + 1
BucketHeaderType
BucketFooterType
SeriesHeaderType
FloatPointsType
IntegerPointsType
UnsignedPointsType
BooleanPointsType
StringPointsType
SeriesFooterType
)
type message interface {
Size() int
MarshalTo(dAtA []byte) (int, error)
}
/*
Stream format
FILE:
┌─────────────────┬────────────────────┬─────────────────┐
│ │ │ │
│ IFLXDUMP (8) │ Header │ BUCKET 0..n │
│ │ │ │
└─────────────────┴────────────────────┴─────────────────┘
BUCKET:
┌─────────────────┬────────────────────┬─────────────────┐
│ │ │ │
│ Bucket Header │ SERIES DATA 0..n │ Bucket Footer │
│ │ │ │
└─────────────────┴────────────────────┴─────────────────┘
SERIES DATA:
┌─────────────────┬────────────────────┬─────────────────┐
│ │ │ │
│ Series Header │ POINTS 0..n │ Series Footer │
│ │ │ │
└─────────────────┴────────────────────┴─────────────────┘
*/

17
cmd/influx_tools/internal/format/binary/messagetype_string.go
View File

@ -1,17 +0,0 @@
// Code generated by "stringer -type=MessageType"; DO NOT EDIT.
package binary
import "fmt"
const _MessageType_name = "HeaderTypeBucketHeaderTypeBucketFooterTypeSeriesHeaderTypeFloatPointsTypeIntegerPointsTypeUnsignedPointsTypeBooleanPointsTypeStringPointsTypeSeriesFooterType"
var _MessageType_index = [...]uint8{0, 10, 26, 42, 58, 73, 90, 108, 125, 141, 157}
func (i MessageType) String() string {
i -= 1
if i >= MessageType(len(_MessageType_index)-1) {
return fmt.Sprintf("MessageType(%d)", i+1)
}
return _MessageType_name[_MessageType_index[i]:_MessageType_index[i+1]]
}

281
cmd/influx_tools/internal/format/binary/reader.go
View File

@ -1,281 +0,0 @@
package binary
import (
"bytes"
"errors"
"fmt"
"io"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/tlv"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxdb/tsdb/engine/tsm1"
)
type Reader struct {
r io.Reader
pr *PointsReader
state *readerState
stats *readerStats
}
type readerStats struct {
series int
counts [8]struct {
series, values int
}
}
type readerState byte
const (
readHeader readerState = iota + 1
readBucket
readSeries
readPoints
done
)
func NewReader(reader io.Reader) *Reader {
state := readHeader
var stats readerStats
r := &Reader{r: reader, state: &state, stats: &stats,
pr: &PointsReader{r: reader, values: make(tsm1.Values, tsdb.DefaultMaxPointsPerBlock), state: &state, stats: &stats}}
return r
}
func (r *Reader) ReadHeader() (*Header, error) {
if *r.state != readHeader {
return nil, fmt.Errorf("expected reader in state %v, was in state %v\n", readHeader, *r.state)
}
var magic [len(Magic)]byte
n, err := r.r.Read(magic[:])
if err != nil {
return nil, err
}
if n < len(Magic) || !bytes.Equal(magic[:], Magic[:]) {
return nil, errors.New("IFLXDUMP header not present")
}
t, lv, err := tlv.ReadTLV(r.r)
if err != nil {
return nil, err
}
if t != byte(HeaderType) {
return nil, fmt.Errorf("expected header type, got %v", t)
}
h := &Header{}
err = h.Unmarshal(lv)
*r.state = readBucket
return h, err
}
func (r *Reader) Close() error {
return nil
}
func (r *Reader) NextBucket() (*BucketHeader, error) {
if *r.state != readBucket {
return nil, fmt.Errorf("expected reader in state %v, was in state %v", readBucket, *r.state)
}
t, lv, err := tlv.ReadTLV(r.r)
if err != nil {
if err == io.EOF {
*r.state = done
return nil, nil
}
return nil, err
}
if t != byte(BucketHeaderType) {
return nil, fmt.Errorf("expected bucket header type, got %v", t)
}
bh := &BucketHeader{}
err = bh.Unmarshal(lv)
if err != nil {
return nil, err
}
*r.state = readSeries
return bh, nil
}
func (r *Reader) NextSeries() (*SeriesHeader, error) {
if *r.state != readSeries {
return nil, fmt.Errorf("expected reader in state %v, was in state %v", readSeries, *r.state)
}
t, lv, err := tlv.ReadTLV(r.r)
if err != nil {
return nil, err
}
if t == byte(BucketFooterType) {
*r.state = readBucket
return nil, nil
}
if t != byte(SeriesHeaderType) {
return nil, fmt.Errorf("expected series header type, got %v", t)
}
sh := &SeriesHeader{}
err = sh.Unmarshal(lv)
if err != nil {
return nil, err
}
r.stats.series++
r.stats.counts[sh.FieldType&7].series++
var pointsType MessageType
switch sh.FieldType {
case FloatFieldType:
pointsType = FloatPointsType
case IntegerFieldType:
pointsType = IntegerPointsType
case UnsignedFieldType:
pointsType = UnsignedPointsType
case BooleanFieldType:
pointsType = BooleanPointsType
case StringFieldType:
pointsType = StringPointsType
default:
return nil, fmt.Errorf("unsupported series field type %v", sh.FieldType)
}
*r.state = readPoints
r.pr.Reset(pointsType)
return sh, nil
}
func (r *Reader) Points() *PointsReader {
return r.pr
}
type PointsReader struct {
pointsType MessageType
r io.Reader
values tsm1.Values
n int
state *readerState
stats *readerStats
}
func (pr *PointsReader) Reset(pointsType MessageType) {
pr.pointsType = pointsType
pr.n = 0
}
func (pr *PointsReader) Next() (bool, error) {
if *pr.state != readPoints {
return false, fmt.Errorf("expected reader in state %v, was in state %v", readPoints, *pr.state)
}
t, lv, err := tlv.ReadTLV(pr.r)
if err != nil {
return false, err
}
if t == byte(SeriesFooterType) {
*pr.state = readSeries
return false, nil
}
if t != byte(pr.pointsType) {
return false, fmt.Errorf("expected message type %v, got %v", pr.pointsType, t)
}
err = pr.marshalValues(lv)
if err != nil {
return false, err
}
return true, nil
}
func (pr *PointsReader) Values() tsm1.Values {
return pr.values[:pr.n]
}
func (pr *PointsReader) marshalValues(lv []byte) error {
switch pr.pointsType {
case FloatPointsType:
return pr.marshalFloats(lv)
case IntegerPointsType:
return pr.marshalIntegers(lv)
case UnsignedPointsType:
return pr.marshalUnsigned(lv)
case BooleanPointsType:
return pr.marshalBooleans(lv)
case StringPointsType:
return pr.marshalStrings(lv)
default:
return fmt.Errorf("unsupported points type %v", pr.pointsType)
}
}
func (pr *PointsReader) marshalFloats(lv []byte) error {
fp := &FloatPoints{}
err := fp.Unmarshal(lv)
if err != nil {
return err
}
for i, t := range fp.Timestamps {
pr.values[i] = tsm1.NewFloatValue(t, fp.Values[i])
}
pr.stats.counts[0].values += len(fp.Timestamps)
pr.n = len(fp.Timestamps)
return nil
}
func (pr *PointsReader) marshalIntegers(lv []byte) error {
ip := &IntegerPoints{}
err := ip.Unmarshal(lv)
if err != nil {
return err
}
for i, t := range ip.Timestamps {
pr.values[i] = tsm1.NewIntegerValue(t, ip.Values[i])
}
pr.stats.counts[1].values += len(ip.Timestamps)
pr.n = len(ip.Timestamps)
return nil
}
func (pr *PointsReader) marshalUnsigned(lv []byte) error {
up := &UnsignedPoints{}
err := up.Unmarshal(lv)
if err != nil {
return err
}
for i, t := range up.Timestamps {
pr.values[i] = tsm1.NewUnsignedValue(t, up.Values[i])
}
pr.stats.counts[2].values += len(up.Timestamps)
pr.n = len(up.Timestamps)
return nil
}
func (pr *PointsReader) marshalBooleans(lv []byte) error {
bp := &BooleanPoints{}
err := bp.Unmarshal(lv)
if err != nil {
return err
}
for i, t := range bp.Timestamps {
pr.values[i] = tsm1.NewBooleanValue(t, bp.Values[i])
}
pr.stats.counts[3].values += len(bp.Timestamps)
pr.n = len(bp.Timestamps)
return nil
}
func (pr *PointsReader) marshalStrings(lv []byte) error {
sp := &StringPoints{}
err := sp.Unmarshal(lv)
if err != nil {
return err
}
for i, t := range sp.Timestamps {
pr.values[i] = tsm1.NewStringValue(t, sp.Values[i])
}
pr.stats.counts[4].values += len(sp.Timestamps)
pr.n = len(sp.Timestamps)
return nil
}

466
cmd/influx_tools/internal/format/binary/reader_test.go
View File

@ -1,466 +0,0 @@
package binary_test
import (
"bytes"
"fmt"
"math"
"testing"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/binary"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
func TestReader_OneBucketOneIntegerSeries(t *testing.T) {
var buf bytes.Buffer
ts := []int64{0, 1, 2}
ints := []int64{10, 11, 12}
vs := make([]interface{}, len(ints))
for i, v := range ints {
vs[i] = v
}
s := &oneSeriesData{
db: "database",
rp: "default",
sd: time.Hour * 24,
start: int64(0),
end: int64(time.Hour * 24),
seriesName: []byte("series"),
seriesField: []byte("field"),
seriesTags: models.NewTags(map[string]string{"k": "v"}),
fieldType: binary.IntegerFieldType,
ts: ts,
vs: vs,
}
w := binary.NewWriter(&buf, s.db, s.rp, s.sd)
bw, _ := w.NewBucket(s.start, s.end)
bw.BeginSeries(s.seriesName, s.seriesField, influxql.Integer, s.seriesTags)
bw.WriteIntegerCursor(&intCursor{1, s.ts, ints})
bw.EndSeries()
bw.Close()
w.Close()
verifySingleSeries(t, buf, s)
}
func TestReader_OneBucketOneFloatSeries(t *testing.T) {
var buf bytes.Buffer
ts := []int64{0, 1, 2}
floats := []float64{0.1, 11.1, 1200.0}
vs := make([]interface{}, len(floats))
for i, v := range floats {
vs[i] = v
}
s := &oneSeriesData{
db: "database",
rp: "default",
sd: time.Hour * 24,
start: int64(0),
end: int64(time.Hour * 24),
seriesName: []byte("series"),
seriesField: []byte("field"),
seriesTags: models.NewTags(map[string]string{"k": "v"}),
fieldType: binary.FloatFieldType,
ts: ts,
vs: vs,
}
w := binary.NewWriter(&buf, s.db, s.rp, s.sd)
bw, _ := w.NewBucket(s.start, s.end)
bw.BeginSeries(s.seriesName, s.seriesField, influxql.Float, s.seriesTags)
bw.WriteFloatCursor(&floatCursor{1, s.ts, floats})
bw.EndSeries()
bw.Close()
w.Close()
verifySingleSeries(t, buf, s)
}
func TestReader_OneBucketOneUnsignedSeries(t *testing.T) {
var buf bytes.Buffer
ts := []int64{0, 1, 2}
uints := []uint64{0, 1, math.MaxUint64}
vs := make([]interface{}, len(uints))
for i, v := range uints {
vs[i] = v
}
s := &oneSeriesData{
db: "database",
rp: "default",
sd: time.Hour * 24,
start: int64(0),
end: int64(time.Hour * 24),
seriesName: []byte("series"),
seriesField: []byte("field"),
seriesTags: models.NewTags(map[string]string{"k": "v"}),
fieldType: binary.UnsignedFieldType,
ts: ts,
vs: vs,
}
w := binary.NewWriter(&buf, s.db, s.rp, s.sd)
bw, _ := w.NewBucket(s.start, s.end)
bw.BeginSeries(s.seriesName, s.seriesField, influxql.Unsigned, s.seriesTags)
bw.WriteUnsignedCursor(&unsignedCursor{1, s.ts, uints})
bw.EndSeries()
bw.Close()
w.Close()
verifySingleSeries(t, buf, s)
}
func TestReader_OneBucketOneBooleanSeries(t *testing.T) {
var buf bytes.Buffer
ts := []int64{0, 1, 2}
bools := []bool{true, true, false}
vs := make([]interface{}, len(bools))
for i, v := range bools {
vs[i] = v
}
s := &oneSeriesData{
db: "database",
rp: "default",
sd: time.Hour * 24,
start: int64(0),
end: int64(time.Hour * 24),
seriesName: []byte("series"),
seriesField: []byte("field"),
seriesTags: models.NewTags(map[string]string{"k": "v"}),
fieldType: binary.BooleanFieldType,
ts: ts,
vs: vs,
}
w := binary.NewWriter(&buf, s.db, s.rp, s.sd)
bw, _ := w.NewBucket(s.start, s.end)
bw.BeginSeries(s.seriesName, s.seriesField, influxql.Boolean, s.seriesTags)
bw.WriteBooleanCursor(&booleanCursor{1, s.ts, bools})
bw.EndSeries()
bw.Close()
w.Close()
verifySingleSeries(t, buf, s)
}
func TestReader_OneBucketOneStringSeries(t *testing.T) {
var buf bytes.Buffer
ts := []int64{0, 1, 2}
strings := []string{"", "a", "a《 》"}
vs := make([]interface{}, len(strings))
for i, v := range strings {
vs[i] = v
}
s := &oneSeriesData{
db: "database",
rp: "default",
sd: time.Hour * 24,
start: int64(0),
end: int64(time.Hour * 24),
seriesName: []byte("series"),
seriesField: []byte("field"),
seriesTags: models.NewTags(map[string]string{"k": "v"}),
fieldType: binary.StringFieldType,
ts: ts,
vs: vs,
}
w := binary.NewWriter(&buf, s.db, s.rp, s.sd)
bw, _ := w.NewBucket(s.start, s.end)
bw.BeginSeries(s.seriesName, s.seriesField, influxql.String, s.seriesTags)
bw.WriteStringCursor(&stringCursor{1, s.ts, strings})
bw.EndSeries()
bw.Close()
w.Close()
verifySingleSeries(t, buf, s)
}
type oneSeriesData struct {
db string
rp string
sd time.Duration
start int64
end int64
seriesName []byte
seriesField []byte
seriesTags models.Tags
fieldType binary.FieldType
ts []int64
vs []interface{}
}
func verifySingleSeries(t *testing.T, buf bytes.Buffer, s *oneSeriesData) {
t.Helper()
r := binary.NewReader(&buf)
h, err := r.ReadHeader()
assertNoError(t, err)
assertEqual(t, h, &binary.Header{Database: s.db, RetentionPolicy: s.rp, ShardDuration: s.sd})
bh, err := r.NextBucket()
assertNoError(t, err)
assertEqual(t, bh, &binary.BucketHeader{Start: s.start, End: s.end})
sh, err := r.NextSeries()
assertNoError(t, err)
seriesKey := make([]byte, 0)
seriesKey = models.AppendMakeKey(seriesKey[:0], s.seriesName, s.seriesTags)
assertEqual(t, sh, &binary.SeriesHeader{FieldType: s.fieldType, SeriesKey: seriesKey, Field: s.seriesField})
for i := 0; i < len(s.ts); i++ {
next, err := r.Points().Next()
assertNoError(t, err)
assertEqual(t, next, true)
values := r.Points().Values()
assertEqual(t, len(values), 1)
assertEqual(t, values[0].UnixNano(), s.ts[i])
assertEqual(t, values[0].Value(), s.vs[i])
}
next, err := r.Points().Next()
assertNoError(t, err)
assertEqual(t, next, false)
sh, err = r.NextSeries()
assertNoError(t, err)
assertNil(t, sh)
bh, err = r.NextBucket()
assertNoError(t, err)
assertNil(t, bh)
}
func TestReader_OneBucketMixedSeries(t *testing.T) {
var buf bytes.Buffer
db := "db"
rp := "rp"
start := int64(0)
end := int64(time.Hour * 24)
seriesName := []byte("cpu")
seriesField := []byte("idle")
seriesTags1 := models.NewTags(map[string]string{"host": "host1", "region": "us-west-1"})
seriesTags2 := models.NewTags(map[string]string{"host": "host2", "region": "us-west-1"})
w := binary.NewWriter(&buf, db, rp, time.Hour*24)
bw, _ := w.NewBucket(start, end)
bw.BeginSeries(seriesName, seriesField, influxql.Integer, seriesTags1)
t1s := []int64{0, 1, 2}
v1s := []int64{10, 11, 12}
bw.WriteIntegerCursor(&intCursor{1, t1s, v1s})
bw.EndSeries()
bw.BeginSeries(seriesName, seriesField, influxql.Integer, seriesTags2)
t2s := []int64{1, 2, 3}
v2s := []float64{7, 8, 9}
bw.WriteFloatCursor(&floatCursor{1, t2s, v2s})
bw.EndSeries()
bw.Close()
w.Close()
r := binary.NewReader(&buf)
h, err := r.ReadHeader()
assertNoError(t, err)
assertEqual(t, h, &binary.Header{Database: db, RetentionPolicy: rp, ShardDuration: time.Hour * 24})
bh, err := r.NextBucket()
assertNoError(t, err)
assertEqual(t, bh, &binary.BucketHeader{Start: start, End: end})
sh, err := r.NextSeries()
assertNoError(t, err)
seriesKey := make([]byte, 0)
seriesKey = models.AppendMakeKey(seriesKey[:0], seriesName, seriesTags1)
assertEqual(t, sh, &binary.SeriesHeader{FieldType: binary.IntegerFieldType, SeriesKey: seriesKey, Field: seriesField})
for i := 0; i < len(t1s); i++ {
next, err := r.Points().Next()
assertNoError(t, err)
assertEqual(t, next, true)
values := r.Points().Values()
assertEqual(t, len(values), 1)
assertEqual(t, values[0].UnixNano(), t1s[i])
assertEqual(t, values[0].Value(), v1s[i])
}
next, err := r.Points().Next()
assertNoError(t, err)
assertEqual(t, next, false)
sh, err = r.NextSeries()
assertNoError(t, err)
seriesKey = models.AppendMakeKey(seriesKey[:0], seriesName, seriesTags2)
assertEqual(t, sh, &binary.SeriesHeader{FieldType: binary.FloatFieldType, SeriesKey: seriesKey, Field: seriesField})
for i := 0; i < len(t2s); i++ {
next, err := r.Points().Next()
assertNoError(t, err)
assertEqual(t, next, true)
values := r.Points().Values()
assertEqual(t, len(values), 1)
assertEqual(t, values[0].UnixNano(), t2s[i])
assertEqual(t, values[0].Value(), v2s[i])
}
next, err = r.Points().Next()
assertNoError(t, err)
assertEqual(t, next, false)
sh, err = r.NextSeries()
assertNoError(t, err)
assertNil(t, sh)
bh, err = r.NextBucket()
assertNoError(t, err)
assertNil(t, bh)
}
func TestReader_EmptyBucket(t *testing.T) {
var buf bytes.Buffer
db := "db"
rp := "default"
start := int64(0)
end := int64(time.Hour * 24)
w := binary.NewWriter(&buf, db, rp, time.Hour*24)
bw, _ := w.NewBucket(start, end)
bw.Close()
w.Close()
r := binary.NewReader(&buf)
h, err := r.ReadHeader()
assertNoError(t, err)
assertEqual(t, h, &binary.Header{Database: db, RetentionPolicy: rp, ShardDuration: time.Hour * 24})
bh, err := r.NextBucket()
assertNoError(t, err)
assertEqual(t, bh, &binary.BucketHeader{Start: start, End: end})
sh, err := r.NextSeries()
assertNoError(t, err)
assertNil(t, sh)
bh, err = r.NextBucket()
assertNoError(t, err)
assertNil(t, bh)
}
func TestReader_States(t *testing.T) {
var buf bytes.Buffer
r := binary.NewReader(&buf)
next, err := r.Points().Next()
assertError(t, err, fmt.Errorf("expected reader in state %v, was in state %v", 4, 1))
assertEqual(t, next, false)
sh, err := r.NextSeries()
assertError(t, err, fmt.Errorf("expected reader in state %v, was in state %v", 3, 1))
assertNil(t, sh)
bh, err := r.NextBucket()
assertError(t, err, fmt.Errorf("expected reader in state %v, was in state %v", 2, 1))
assertNil(t, bh)
}
type floatCursor struct {
c int // number of values to return per call to Next
keys []int64
vals []float64
}
func (c *floatCursor) Close() {}
func (c *floatCursor) Err() error { return nil }
func (c *floatCursor) Stats() tsdb.CursorStats { return tsdb.CursorStats{} }
func (c *floatCursor) Next() *tsdb.FloatArray {
if c.c > len(c.keys) {
c.c = len(c.keys)
}
var a tsdb.FloatArray
a.Timestamps, a.Values = c.keys[:c.c], c.vals[:c.c]
c.keys, c.vals = c.keys[c.c:], c.vals[c.c:]
return &a
}
type unsignedCursor struct {
c int // number of values to return per call to Next
keys []int64
vals []uint64
}
func (c *unsignedCursor) Close() {}
func (c *unsignedCursor) Err() error { return nil }
func (c *unsignedCursor) Stats() tsdb.CursorStats { return tsdb.CursorStats{} }
func (c *unsignedCursor) Next() *tsdb.UnsignedArray {
if c.c > len(c.keys) {
c.c = len(c.keys)
}
var a tsdb.UnsignedArray
a.Timestamps, a.Values = c.keys[:c.c], c.vals[:c.c]
c.keys, c.vals = c.keys[c.c:], c.vals[c.c:]
return &a
}
type booleanCursor struct {
c int // number of values to return per call to Next
keys []int64
vals []bool
}
func (c *booleanCursor) Close() {}
func (c *booleanCursor) Err() error { return nil }
func (c *booleanCursor) Stats() tsdb.CursorStats { return tsdb.CursorStats{} }
func (c *booleanCursor) Next() *tsdb.BooleanArray {
if c.c > len(c.keys) {
c.c = len(c.keys)
}
var a tsdb.BooleanArray
a.Timestamps, a.Values = c.keys[:c.c], c.vals[:c.c]
c.keys, c.vals = c.keys[c.c:], c.vals[c.c:]
return &a
}
type stringCursor struct {
c int // number of values to return per call to Next
keys []int64
vals []string
}
func (c *stringCursor) Close() {}
func (c *stringCursor) Err() error { return nil }
func (c *stringCursor) Stats() tsdb.CursorStats { return tsdb.CursorStats{} }
func (c *stringCursor) Next() *tsdb.StringArray {
if c.c > len(c.keys) {
c.c = len(c.keys)
}
var a tsdb.StringArray
a.Timestamps, a.Values = c.keys[:c.c], c.vals[:c.c]
c.keys, c.vals = c.keys[c.c:], c.vals[c.c:]
return &a
}
func assertNil(t *testing.T, got interface{}) {
t.Helper()
if got == nil {
t.Fatalf("not nil: got:\n%s", got)
}
}
func assertError(t *testing.T, got error, exp error) {
t.Helper()
if got == nil {
t.Fatalf("did not receive expected error: %s", exp)
} else {
assertEqual(t, got.Error(), exp.Error())
}
}

375
cmd/influx_tools/internal/format/binary/writer.go
View File

@ -1,375 +0,0 @@
package binary
import (
"bufio"
"fmt"
"io"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/tlv"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
type Writer struct {
w *bufio.Writer
buf []byte
db, rp string
duration time.Duration
err error
bw *bucketWriter
state writeState
wroteHeader bool
msg struct {
bucketHeader BucketHeader
bucketFooter BucketFooter
seriesHeader SeriesHeader
seriesFooter SeriesFooter
}
stats struct {
series int
counts [8]struct {
series, values int
}
}
}
type writeState int
const (
writeHeader writeState = iota
writeBucket
writeSeries
writeSeriesHeader
writePoints
)
func NewWriter(w io.Writer, database, rp string, duration time.Duration) *Writer {
var wr *bufio.Writer
if wr, _ = w.(*bufio.Writer); wr == nil {
wr = bufio.NewWriter(w)
}
return &Writer{w: wr, db: database, rp: rp, duration: duration}
}
func (w *Writer) WriteStats(o io.Writer) {
fmt.Fprintf(o, "total series: %d\n", w.stats.series)
for i := 0; i < 5; i++ {
ft := FieldType(i)
fmt.Fprintf(o, "%s unique series: %d\n", ft, w.stats.counts[i].series)
fmt.Fprintf(o, "%s total values : %d\n", ft, w.stats.counts[i].values)
}
}
func (w *Writer) NewBucket(start, end int64) (format.BucketWriter, error) {
if w.state == writeHeader {
w.writeHeader()
}
if w.err != nil {
return nil, w.err
}
if w.state != writeBucket {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", w.state, writeBucket))
}
w.bw = &bucketWriter{w: w, start: start, end: end}
w.writeBucketHeader(start, end)
return w.bw, w.err
}
func (w *Writer) Close() error {
if w.err == ErrWriteAfterClose {
return nil
}
if w.err != nil {
return w.err
}
w.err = ErrWriteAfterClose
return nil
}
func (w *Writer) writeHeader() {
w.state = writeBucket
w.wroteHeader = true
w.write(Magic[:])
h := Header{
Version: Version0,
Database: w.db,
RetentionPolicy: w.rp,
ShardDuration: w.duration,
}
w.writeTypeMessage(HeaderType, &h)
}
func (w *Writer) writeBucketHeader(start, end int64) {
w.state = writeSeries
w.msg.bucketHeader.Start = start
w.msg.bucketHeader.End = end
w.writeTypeMessage(BucketHeaderType, &w.msg.bucketHeader)
}
func (w *Writer) writeBucketFooter() {
w.state = writeBucket
w.writeTypeMessage(BucketFooterType, &w.msg.bucketFooter)
}
func (w *Writer) writeSeriesHeader(key, field []byte, ft FieldType) {
w.state = writePoints
w.stats.series++
w.stats.counts[ft&7].series++
w.msg.seriesHeader.SeriesKey = key
w.msg.seriesHeader.Field = field
w.msg.seriesHeader.FieldType = ft
w.writeTypeMessage(SeriesHeaderType, &w.msg.seriesHeader)
}
func (w *Writer) writeSeriesFooter(ft FieldType, count int) {
w.stats.counts[ft&7].values += count
w.writeTypeMessage(SeriesFooterType, &w.msg.seriesFooter)
}
func (w *Writer) write(p []byte) {
if w.err != nil {
return
}
_, w.err = w.w.Write(p)
}
func (w *Writer) writeTypeMessage(typ MessageType, msg message) {
if w.err != nil {
return
}
// ensure size
n := msg.Size()
if n > cap(w.buf) {
w.buf = make([]byte, n)
} else {
w.buf = w.buf[:n]
}
_, w.err = msg.MarshalTo(w.buf)
w.writeTypeBytes(typ, w.buf)
}
func (w *Writer) writeTypeBytes(typ MessageType, b []byte) {
if w.err != nil {
return
}
w.err = tlv.WriteTLV(w.w, byte(typ), w.buf)
}
type bucketWriter struct {
w *Writer
err error
start, end int64
key []byte
field []byte
n int
closed bool
}
func (bw *bucketWriter) Err() error {
if bw.w.err != nil {
return bw.w.err
}
return bw.err
}
func (bw *bucketWriter) hasErr() bool {
return bw.w.err != nil || bw.err != nil
}
func (bw *bucketWriter) BeginSeries(name, field []byte, typ influxql.DataType, tags models.Tags) {
if bw.hasErr() {
return
}
if bw.w.state != writeSeries {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", bw.w.state, writeSeries))
}
bw.w.state = writeSeriesHeader
bw.key = models.AppendMakeKey(bw.key[:0], name, tags)
bw.field = field
}
func (bw *bucketWriter) EndSeries() {
if bw.hasErr() {
return
}
if bw.w.state != writePoints && bw.w.state != writeSeriesHeader {
panic(fmt.Sprintf("writer state: got=%v, exp=%v,%v", bw.w.state, writeSeriesHeader, writePoints))
}
if bw.w.state == writePoints {
bw.w.writeSeriesFooter(IntegerFieldType, bw.n)
}
bw.w.state = writeSeries
}
func (bw *bucketWriter) WriteIntegerCursor(cur tsdb.IntegerArrayCursor) {
if bw.hasErr() {
return
}
if bw.w.state == writeSeriesHeader {
bw.w.writeSeriesHeader(bw.key, bw.field, IntegerFieldType)
}
if bw.w.state != writePoints {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", bw.w.state, writePoints))
}
var msg IntegerPoints
for {
a := cur.Next()
if a.Len() == 0 {
break
}
bw.n += a.Len()
msg.Timestamps = a.Timestamps
msg.Values = a.Values
bw.w.writeTypeMessage(IntegerPointsType, &msg)
}
}
func (bw *bucketWriter) WriteFloatCursor(cur tsdb.FloatArrayCursor) {
if bw.hasErr() {
return
}
if bw.w.state == writeSeriesHeader {
bw.w.writeSeriesHeader(bw.key, bw.field, FloatFieldType)
}
if bw.w.state != writePoints {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", bw.w.state, writePoints))
}
var msg FloatPoints
for {
a := cur.Next()
if a.Len() == 0 {
break
}
bw.n += a.Len()
msg.Timestamps = a.Timestamps
msg.Values = a.Values
bw.w.writeTypeMessage(FloatPointsType, &msg)
}
}
func (bw *bucketWriter) WriteUnsignedCursor(cur tsdb.UnsignedArrayCursor) {
if bw.hasErr() {
return
}
if bw.w.state == writeSeriesHeader {
bw.w.writeSeriesHeader(bw.key, bw.field, UnsignedFieldType)
}
if bw.w.state != writePoints {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", bw.w.state, writePoints))
}
var msg UnsignedPoints
for {
a := cur.Next()
if a.Len() == 0 {
break
}
bw.n += a.Len()
msg.Timestamps = a.Timestamps
msg.Values = a.Values
bw.w.writeTypeMessage(UnsignedPointsType, &msg)
}
}
func (bw *bucketWriter) WriteBooleanCursor(cur tsdb.BooleanArrayCursor) {
if bw.hasErr() {
return
}
if bw.w.state == writeSeriesHeader {
bw.w.writeSeriesHeader(bw.key, bw.field, BooleanFieldType)
}
if bw.w.state != writePoints {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", bw.w.state, writePoints))
}
var msg BooleanPoints
for {
a := cur.Next()
if a.Len() == 0 {
break
}
bw.n += a.Len()
msg.Timestamps = a.Timestamps
msg.Values = a.Values
bw.w.writeTypeMessage(BooleanPointsType, &msg)
}
}
func (bw *bucketWriter) WriteStringCursor(cur tsdb.StringArrayCursor) {
if bw.hasErr() {
return
}
if bw.w.state == writeSeriesHeader {
bw.w.writeSeriesHeader(bw.key, bw.field, StringFieldType)
}
if bw.w.state != writePoints {
panic(fmt.Sprintf("writer state: got=%v, exp=%v", bw.w.state, writePoints))
}
var msg StringPoints
for {
a := cur.Next()
if a.Len() == 0 {
break
}
bw.n += a.Len()
msg.Timestamps = a.Timestamps
msg.Values = a.Values
bw.w.writeTypeMessage(StringPointsType, &msg)
}
}
func (bw *bucketWriter) Close() error {
if bw.closed {
return nil
}
bw.closed = true
if bw.hasErr() {
return bw.Err()
}
bw.w.bw = nil
bw.w.writeBucketFooter()
bw.err = ErrWriteBucketAfterClose
return bw.w.w.Flush()
}

117
cmd/influx_tools/internal/format/binary/writer_test.go
View File

@ -1,117 +0,0 @@
package binary_test
import (
"bytes"
"io"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format/binary"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/tlv"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
func TestWriter_WriteOneBucketOneSeries(t *testing.T) {
var buf bytes.Buffer
w := binary.NewWriter(&buf, "db", "rp", time.Second)
bw, _ := w.NewBucket(0, int64(time.Second))
bw.BeginSeries([]byte("cpu"), []byte("idle"), influxql.Integer, models.NewTags(map[string]string{"host": "host1", "region": "us-west-1"}))
ts := []int64{0, 1, 2}
vs := []int64{10, 11, 12}
bw.WriteIntegerCursor(&intCursor{1, ts, vs})
bw.EndSeries()
bw.Close()
w.Close()
// magic
var in [8]byte
buf.Read(in[:])
assertEqual(t, in[:], binary.Magic[:])
// header
var hdr binary.Header
assertTypeValue(t, &buf, binary.HeaderType, &hdr)
assertEqual(t, hdr, binary.Header{Version: binary.Version0, Database: "db", RetentionPolicy: "rp", ShardDuration: time.Second})
// bucket header
var bh binary.BucketHeader
assertTypeValue(t, &buf, binary.BucketHeaderType, &bh)
assertEqual(t, bh, binary.BucketHeader{Start: 0, End: int64(time.Second)})
// series
var sh binary.SeriesHeader
assertTypeValue(t, &buf, binary.SeriesHeaderType, &sh)
assertEqual(t, sh, binary.SeriesHeader{
FieldType: binary.IntegerFieldType,
SeriesKey: []byte("cpu,host=host1,region=us-west-1"),
Field: []byte("idle"),
})
// values
for i := 0; i < len(ts); i++ {
var ip binary.IntegerPoints
assertTypeValue(t, &buf, binary.IntegerPointsType, &ip)
assertEqual(t, ip, binary.IntegerPoints{Timestamps: ts[i : i+1], Values: vs[i : i+1]})
}
// series footer
var sf binary.SeriesFooter
assertTypeValue(t, &buf, binary.SeriesFooterType, &sf)
// bucket footer
var bf binary.BucketFooter
assertTypeValue(t, &buf, binary.BucketFooterType, &bf)
}
type intCursor struct {
c int // number of values to return per call to Next
keys []int64
vals []int64
}
func (c *intCursor) Close() {}
func (c *intCursor) Err() error { return nil }
func (c *intCursor) Stats() tsdb.CursorStats { return tsdb.CursorStats{} }
func (c *intCursor) Next() *tsdb.IntegerArray {
if c.c > len(c.keys) {
c.c = len(c.keys)
}
var a tsdb.IntegerArray
a.Timestamps, a.Values = c.keys[:c.c], c.vals[:c.c]
c.keys, c.vals = c.keys[c.c:], c.vals[c.c:]
return &a
}
func assertEqual(t *testing.T, got, exp interface{}) {
t.Helper()
if !cmp.Equal(got, exp) {
t.Fatalf("not equal: -got/+exp\n%s", cmp.Diff(got, exp))
}
}
func assertNoError(t *testing.T, err error) {
t.Helper()
if err == nil {
return
}
t.Fatalf("unexpected error: %v", err)
}
type message interface {
Unmarshal([]byte) error
}
func assertTypeValue(t *testing.T, r io.Reader, expType binary.MessageType, m message) {
t.Helper()
typ, d, err := tlv.ReadTLV(r)
assertNoError(t, err)
assertEqual(t, typ, byte(expType))
err = m.Unmarshal(d)
assertNoError(t, err)
}

176
cmd/influx_tools/internal/format/conflictwriter.go
View File

@ -1,176 +0,0 @@
package format
import (
"bytes"
"fmt"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
// ConflictWriter is a Writer that redirects conflicting data to an alternate output.
type ConflictWriter struct {
w Writer
c Writer
bw aggregateBucketWriter
}
// NewConflictWriter returns a Writer that redirects invalid point data to the conflict Writer.
func NewConflictWriter(w, conflict Writer) *ConflictWriter {
return &ConflictWriter{w: w, c: conflict}
}
func (cw *ConflictWriter) NewBucket(start, end int64) (bw BucketWriter, err error) {
cw.bw.w, err = cw.w.NewBucket(start, end)
if err != nil {
return nil, err
}
cw.bw.c, err = cw.c.NewBucket(start, end)
if err != nil {
cw.bw.w.Close()
return nil, err
}
return &cw.bw, nil
}
func (cw *ConflictWriter) Close() error {
// we care if either error and prioritize the conflict writer lower.
cerr := cw.c.Close()
if err := cw.w.Close(); err != nil {
return err
}
return cerr
}
type bucketState int
const (
beginSeriesBucketState bucketState = iota
writeBucketState
writeConflictsBucketState
)
type aggregateBucketWriter struct {
w BucketWriter
c BucketWriter
state bucketState
// current series
name []byte
field []byte
typ influxql.DataType
tags models.Tags
mf map[string]influxql.DataType
}
func (bw *aggregateBucketWriter) Err() error {
switch {
case bw.w.Err() != nil:
return bw.w.Err()
case bw.c.Err() != nil:
return bw.c.Err()
default:
return nil
}
}
func (bw *aggregateBucketWriter) BeginSeries(name, field []byte, typ influxql.DataType, tags models.Tags) {
bw.w.BeginSeries(name, field, typ, tags)
if !bytes.Equal(bw.name, name) {
// new measurement
bw.name = append(bw.name[:0], name...)
bw.mf = make(map[string]influxql.DataType)
}
bw.field = append(bw.field[:0], field...)
bw.tags = tags
var ok bool
bw.typ, ok = bw.mf[string(field)]
if !ok {
bw.mf[string(field)] = typ
bw.typ = typ
}
bw.state = writeBucketState
}
func (bw *aggregateBucketWriter) EndSeries() {
switch {
case bw.state == writeBucketState:
bw.w.EndSeries()
case bw.state == writeConflictsBucketState:
bw.w.EndSeries()
bw.c.EndSeries()
default:
panic(fmt.Sprintf("ConflictWriter state: got=%v, exp=%v,%v", bw.state, writeBucketState, writeConflictsBucketState))
}
bw.state = beginSeriesBucketState
}
func (bw *aggregateBucketWriter) conflictState(other influxql.DataType) {
if bw.state == writeBucketState {
bw.c.BeginSeries(bw.name, bw.field, bw.typ, bw.tags)
bw.state = writeConflictsBucketState
}
}
func (bw *aggregateBucketWriter) WriteIntegerCursor(cur tsdb.IntegerArrayCursor) {
if bw.typ == influxql.Integer {
bw.w.WriteIntegerCursor(cur)
} else {
bw.conflictState(influxql.Integer)
bw.c.WriteIntegerCursor(cur)
}
}
func (bw *aggregateBucketWriter) WriteFloatCursor(cur tsdb.FloatArrayCursor) {
if bw.typ == influxql.Float {
bw.w.WriteFloatCursor(cur)
} else {
bw.conflictState(influxql.Float)
bw.c.WriteFloatCursor(cur)
}
}
func (bw *aggregateBucketWriter) WriteUnsignedCursor(cur tsdb.UnsignedArrayCursor) {
if bw.typ == influxql.Unsigned {
bw.w.WriteUnsignedCursor(cur)
} else {
bw.conflictState(influxql.Unsigned)
bw.c.WriteUnsignedCursor(cur)
}
}
func (bw *aggregateBucketWriter) WriteBooleanCursor(cur tsdb.BooleanArrayCursor) {
if bw.typ == influxql.Boolean {
bw.w.WriteBooleanCursor(cur)
} else {
bw.conflictState(influxql.Boolean)
bw.c.WriteBooleanCursor(cur)
}
}
func (bw *aggregateBucketWriter) WriteStringCursor(cur tsdb.StringArrayCursor) {
if bw.typ == influxql.String {
bw.w.WriteStringCursor(cur)
} else {
bw.conflictState(influxql.String)
bw.c.WriteStringCursor(cur)
}
}
func (bw *aggregateBucketWriter) Close() error {
// we care if either error and prioritize the conflict writer lower.
cerr := bw.c.Close()
if err := bw.w.Close(); err != nil {
return err
}
return cerr
}

181
cmd/influx_tools/internal/format/line/writer.go
View File

@ -1,181 +0,0 @@
package line
import (
"bufio"
"fmt"
"io"
"strconv"
"time"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/pkg/escape"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
type Writer struct {
w *bufio.Writer
key []byte
err error
}
func NewWriter(w io.Writer) *Writer {
var wr *bufio.Writer
if wr, _ = w.(*bufio.Writer); wr == nil {
wr = bufio.NewWriter(w)
}
return &Writer{
w: wr,
}
}
func (w *Writer) NewBucket(start, end int64) (format.BucketWriter, error) {
fmt.Fprintf(w.w, "# new shard group start: %s -> end: %s\n", time.Unix(0, start).UTC(), time.Unix(0, end).UTC())
return w, nil
}
func (w *Writer) Close() error { return w.w.Flush() }
func (w *Writer) Err() error { return w.err }
func (w *Writer) BeginSeries(name, field []byte, typ influxql.DataType, tags models.Tags) {
if w.err != nil {
return
}
w.key = models.AppendMakeKey(w.key[:0], name, tags)
w.key = append(w.key, ' ')
w.key = append(w.key, escape.Bytes(field)...)
w.key = append(w.key, '=')
}
func (w *Writer) EndSeries() {}
func (w *Writer) WriteIntegerCursor(cur tsdb.IntegerArrayCursor) {
if w.err != nil {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:len(w.key)] // Re-slice buf to be "<series_key> <field>=".
buf = strconv.AppendInt(buf, a.Values[i], 10)
buf = append(buf, 'i')
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteFloatCursor(cur tsdb.FloatArrayCursor) {
if w.err != nil {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:len(w.key)] // Re-slice buf to be "<series_key> <field>=".
buf = strconv.AppendFloat(buf, a.Values[i], 'g', -1, 64)
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteUnsignedCursor(cur tsdb.UnsignedArrayCursor) {
if w.err != nil {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:len(w.key)] // Re-slice buf to be "<series_key> <field>=".
buf = strconv.AppendUint(buf, a.Values[i], 10)
buf = append(buf, 'u')
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteBooleanCursor(cur tsdb.BooleanArrayCursor) {
if w.err != nil {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:len(w.key)] // Re-slice buf to be "<series_key> <field>=".
buf = strconv.AppendBool(buf, a.Values[i])
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteStringCursor(cur tsdb.StringArrayCursor) {
if w.err != nil {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:len(w.key)] // Re-slice buf to be "<series_key> <field>=".
buf = append(buf, '"')
buf = append(buf, models.EscapeStringField(a.Values[i])...)
buf = append(buf, '"')
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}

191
cmd/influx_tools/internal/format/text/writer.go
View File

@ -1,191 +0,0 @@
package text
import (
"bufio"
"io"
"strconv"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/format"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/pkg/escape"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
type Writer struct {
w *bufio.Writer
key []byte
err error
m Mode
}
type Mode bool
const (
Series Mode = false
Values Mode = true
)
func NewWriter(w io.Writer, mode Mode) *Writer {
var wr *bufio.Writer
if wr, _ = w.(*bufio.Writer); wr == nil {
wr = bufio.NewWriter(w)
}
return &Writer{
w: wr,
key: make([]byte, 1024),
m: mode,
}
}
func (w *Writer) NewBucket(start, end int64) (format.BucketWriter, error) {
return w, nil
}
func (w *Writer) Close() error { return w.w.Flush() }
func (w *Writer) Err() error { return w.err }
func (w *Writer) BeginSeries(name, field []byte, typ influxql.DataType, tags models.Tags) {
if w.err != nil {
return
}
if w.m == Series {
w.key = models.AppendMakeKey(w.key[:0], name, tags)
w.key = append(w.key, ' ')
w.key = append(w.key, escape.Bytes(field)...)
w.w.Write(w.key)
w.w.WriteByte('\n')
}
}
func (w *Writer) EndSeries() {}
func (w *Writer) WriteIntegerCursor(cur tsdb.IntegerArrayCursor) {
if w.err != nil || w.m == Series {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:0]
buf = strconv.AppendInt(buf, a.Values[i], 10)
buf = append(buf, 'i')
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteFloatCursor(cur tsdb.FloatArrayCursor) {
if w.err != nil || w.m == Series {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:0]
buf = strconv.AppendFloat(buf, a.Values[i], 'g', -1, 64)
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteUnsignedCursor(cur tsdb.UnsignedArrayCursor) {
if w.err != nil || w.m == Series {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:0]
buf = strconv.AppendUint(buf, a.Values[i], 10)
buf = append(buf, 'u')
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteBooleanCursor(cur tsdb.BooleanArrayCursor) {
if w.err != nil || w.m == Series {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:0]
buf = strconv.AppendBool(buf, a.Values[i])
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}
func (w *Writer) WriteStringCursor(cur tsdb.StringArrayCursor) {
if w.err != nil || w.m == Series {
return
}
buf := w.key
for {
a := cur.Next()
if a.Len() == 0 {
break
}
for i := range a.Timestamps {
buf = buf[:0]
buf = append(buf, '"')
buf = append(buf, models.EscapeStringField(a.Values[i])...)
buf = append(buf, '"')
buf = append(buf, ' ')
buf = strconv.AppendInt(buf, a.Timestamps[i], 10)
buf = append(buf, '\n')
if _, w.err = w.w.Write(buf); w.err != nil {
return
}
}
}
}

156
cmd/influx_tools/internal/format/writer.go
View File

@ -1,156 +0,0 @@
package format
import (
"fmt"
"github.com/influxdata/influxdb/cmd/influx_tools/internal/storage"
"github.com/influxdata/influxdb/models"
"github.com/influxdata/influxdb/tsdb"
"github.com/influxdata/influxql"
)
var (
// Discard is a Writer where all write calls succeed. The source data is also read completely, which can be useful
// for testing performance.
Discard Writer = &devNull{true}
// DevNull is a Writer where all write calls succeed, however, no source data is read.
DevNull Writer = &devNull{}
)
type Writer interface {
NewBucket(start, end int64) (BucketWriter, error)
Close() error
}
type BucketWriter interface {
Err() error
BeginSeries(name, field []byte, typ influxql.DataType, tags models.Tags)
EndSeries()
WriteIntegerCursor(cur tsdb.IntegerArrayCursor)
WriteFloatCursor(cur tsdb.FloatArrayCursor)
WriteUnsignedCursor(cur tsdb.UnsignedArrayCursor)
WriteBooleanCursor(cur tsdb.BooleanArrayCursor)
WriteStringCursor(cur tsdb.StringArrayCursor)
Close() error
}
// WriteBucket reads data from rs covering the time range [start, end) and streams to w.
// The ResultSet must guarantee series+field keys are produced in ascending lexicographical order and values in
// ascending time order.
func WriteBucket(w Writer, start, end int64, rs *storage.ResultSet) error {
bw, err := w.NewBucket(start, end)
if err != nil {
return err
}
defer bw.Close()
for rs.Next() {
bw.BeginSeries(rs.Name(), rs.Field(), rs.FieldType(), rs.Tags())
ci := rs.CursorIterator()
for ci.Next() {
cur := ci.Cursor()
switch c := cur.(type) {
case tsdb.IntegerArrayCursor:
bw.WriteIntegerCursor(c)
case tsdb.FloatArrayCursor:
bw.WriteFloatCursor(c)
case tsdb.UnsignedArrayCursor:
bw.WriteUnsignedCursor(c)
case tsdb.BooleanArrayCursor:
bw.WriteBooleanCursor(c)
case tsdb.StringArrayCursor:
bw.WriteStringCursor(c)
case nil:
// no data for series key + field combination in this shard
continue
default:
panic(fmt.Sprintf("unreachable: %T", c))
}
cur.Close()
}
bw.EndSeries()
if bw.Err() != nil {
return bw.Err()
}
}
return nil
}
type devNull struct {
r bool
}
func (w *devNull) NewBucket(start, end int64) (BucketWriter, error) {
return w, nil
}
func (w *devNull) BeginSeries(name, field []byte, typ influxql.DataType, tags models.Tags) {}
func (w *devNull) EndSeries() {}
func (w *devNull) Err() error { return nil }
func (w *devNull) Close() error { return nil }
func (w *devNull) WriteIntegerCursor(cur tsdb.IntegerArrayCursor) {
if !w.r {
return
}
for {
a := cur.Next()
if a.Len() == 0 {
break
}
}
}
func (w *devNull) WriteFloatCursor(cur tsdb.FloatArrayCursor) {
if !w.r {
return
}
for {
a := cur.Next()
if a.Len() == 0 {
break
}
}
}
func (w *devNull) WriteUnsignedCursor(cur tsdb.UnsignedArrayCursor) {
if !w.r {
return
}
for {
a := cur.Next()
if a.Len() == 0 {
break
}
}
}
func (w *devNull) WriteBooleanCursor(cur tsdb.BooleanArrayCursor) {
if !w.r {
return
}
for {
a := cur.Next()
if a.Len() == 0 {
break
}
}
}
func (w *devNull) WriteStringCursor(cur tsdb.StringArrayCursor) {
if !w.r {
return
}
for {
a := cur.Next()
if a.Len() == 0 {
break
}
}
}

Some files were not shown because too many files have changed in this diff Show More