Commit Graph

1978 Commits (db/6263/compaction-debug-logging)

Author SHA1 Message Date
Ben Johnson b42245acbe
Refactoring 'buildtsi' command. 2018-01-30 11:01:35 -07:00
Edd Robinson 13a1505ba2 Cleanup cmd package 2018-01-21 12:08:25 -08:00
Edd Robinson 6a66b5faf0 Cleanup services package 2018-01-21 10:52:37 -08:00
Adam 28f3912491
Renamed references for "enterprise" backup/restore mode to "portable" (#9346) 2018-01-19 16:53:43 -05:00
Adam 938db68198
Update restore functionality to run in online mode, consume Enterprise backup files. (#9207)
* Live Restore + Enterprise data format compatability

* Extended ImportData to import all DB's if no db name given

* Added a new enterprise data test, and backup command now prints the backup file paths at conclusion

* Added whole-system backup test

* Update to use protobuf in all enterprise data cases

* Update to test to do cross-testing with enterprise version

* incremental enterprise backup format support
2018-01-10 13:59:18 -05:00
Ben Johnson 98486a284a
Merge pull request #9265 from benbjohnson/series-file-compaction
Sequential series file id & series file segmentation
2018-01-03 10:05:59 -07:00
Edd Robinson 5efeed4ccc Add import 2018-01-03 16:50:44 +00:00
Ben Johnson 52630e69d7
Integrate SeriesFileCompactor 2018-01-02 12:20:03 -07:00
Jonathan A. Sternberg af23897940 Allow setting the node id in the influx cli program
The string `node <n>` can be used to specify which data node the data
should be retrieved from. This uses the `node_id=X` query parameter that
is supported, but wasn't exposed anywhere in the client library.

We use this feature enough internally when attempting to find
inconsistencies or network errors that it is easier if this is just
supported. Otherwise, I continue having to recompile the CLI program
every time I need to do this.

To clear a previously set node, you can use `node 0` or `node clear`.
2018-01-02 11:15:19 -06:00
Ben Johnson d8b1d208c0
rebase 2017-12-20 15:13:34 -07:00
Edd Robinson 289d1f8d44 Allow iterators to return if shard is closing 2017-12-15 00:46:43 +00:00
Ben Johnson 288c5217e8
Fix tsi1 tools. 2017-12-08 16:12:33 -07:00
Edd Robinson 7d13bf3262 merge master 2017-12-08 17:21:58 +00:00
Edd Robinson f6835632e7 Merge master into branch 2017-12-08 17:11:07 +00:00
Adam a0b2195d6b
Pulled in backup-relevant code for review (#9193)
for issue #8879
2017-12-07 11:35:20 -05:00
Jonathan A. Sternberg 95e1e3b332
Merge pull request #8015 from influxdata/js-code-coverage
Expand code coverage for undercovered packages
2017-11-29 19:30:47 -06:00
Andrew Hare 28ec02a7c1
Merge branch 'master' into amh-8789 2017-11-28 17:05:42 -07:00
Jonathan A. Sternberg b775ad3d5d Expand unit test code coverage in services that were undercovered
This expands code coverage for the following packages:
* monitor (3.5% -> 86.9%)
* services/precreator (31.6% -> 83.8%)
* services/retention (83.0% -> 84.9%)
* services/snapshotter (0.0% -> 82.1%)
* tcp (48.7% -> 60.0%)
2017-11-28 15:44:35 -06:00
Jonathan A. Sternberg 00ab411a6c Group the "strings" import with the other stdlib imports 2017-11-27 17:10:28 -06:00
Ben Johnson fc966a1b67
Add series file backup/restore. 2017-11-22 08:55:54 -07:00
Andrew Hare 9ac37ca459 A non-regex solution 2017-11-20 16:30:27 -07:00
Andrew Hare 89589adea5 Fix CLI to allow quoted database names
Change the CLI to support quoted database names in `use` statements.
This also allows for all database names to be specified, including names
that contain spaces.
2017-11-20 16:30:27 -07:00
wyc c0d184ee93 Influx CLI: More Connection Warnings
If we don't detect a server version, then there's a good chance that
we're not speaking to an InfluxDB server. We should warn the user about
this to make it easier for them to debug.
2017-11-16 11:59:29 -05:00
Ben Johnson ba4c9e0317
Merge remote-tracking branch 'upstream/master' into er-tsi-index-part 2017-11-14 16:14:13 -07:00
Jason Wilder 48e21e6fc8
Merge pull request #9084 from influxdata/jw-delete-time
Handle high cardinality deletes in TSM engine
2017-11-13 14:39:54 -07:00
Stuart Carnie 40ffa49a62 rename config section to ifql
Renaming for discoverability for the initial release of ifql
2017-11-13 11:05:02 -08:00
e-wave 75e142c41a use of flag.Args instead of flag.NArg 2017-11-13 18:07:00 +02:00
Jason Wilder ca4998f29e Fix pid test to ignore local data
If you have lots of data stored locally, this test takes a while to
complete since it loads it all up from the users home dir.
2017-11-13 08:50:07 -07:00
e-wave 1c256e1441 shell shows error and usage when given extra parameters 2017-11-13 17:06:58 +02:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Ben Johnson 919f99f34d
Fixing tests. 2017-11-09 09:30:19 -07:00
Edd Robinson bf132004a3
Implement basic partition layout 2017-11-09 09:28:37 -07:00
Edd Robinson 7aa9de508d
Initial refactor of tsi1.Index
This commit carries out the initial refactor of the tsi1.Index into
tsi1.Partition. We then create a new tsi1.Index that will be an
abstraction over a collection of Partitions.
2017-11-09 09:27:56 -07:00
Ben Johnson f223153078
Initial working version of series file. 2017-11-09 09:18:33 -07:00
Ben Johnson e05d4fdeeb
intermediate 2017-11-09 09:18:33 -07:00
Jonathan A. Sternberg e13ad18160 Use the isolated temp dir for the test pid server 2017-11-07 10:40:46 -06:00
Jonathan A. Sternberg 0b5178ca73
Merge pull request #9074 from influxdata/js-fix-cmd-integration-test
Use the appropriate environment variable for the wal directory
2017-11-07 10:30:02 -06:00
Jonathan A. Sternberg 0a73dcec96 Use the appropriate environment variable for the wal directory
The integration test was intended to use the temporary directory for the
files that were created, but `INFLUXDB_WAL_DIR` is supposed to be
`INFLUXDB_DATA_WAL_DIR`.
2017-11-07 10:07:33 -06:00
Edd Robinson 3e0bf925f8 Use random port in PID file test 2017-11-06 19:35:17 +00:00
Stuart Carnie cf2227def1 add expected data type to series frame 2017-11-06 11:12:27 -07:00
Jonathan A. Sternberg fade2ba9a8
Merge pull request #9043 from influxdata/js-8965-handle-utf16-bom
Handle utf16 files when reading the configuration file
2017-10-31 14:16:09 -05:00
Jonathan A. Sternberg 9dc60962c7 Handle utf16 files when reading the configuration file
Windows computers may produce a utf16 file from the command line that
contains a byte-order-mark. Along with handling the utf8
byte-order-mark, this also handles the utf16 for better Windows
compatibility.
2017-10-30 22:48:44 -05:00
Stuart Carnie f3d45ba301 influxdata/influxdb/influxql -> influxdata/influxql 2017-10-30 14:40:26 -07:00
Stuart Carnie 9e836aefb2 add additional frame types to store tool; count total points processed 2017-10-30 09:24:15 -07:00
Curt Bushko 37c38326a4 fix linting on cmd 2017-10-26 21:15:27 -04:00
Ben Johnson 49c1fca036
Handle nil MeasurementIterator. 2017-10-26 11:25:46 -06:00
Stuart Carnie dc04eaa8f3 Amendments based on feedback
* Fprint* functions
* No nakedness
* clarify panic messages
* spacing between case statements
* remove break in favor of return
* remove goto in favor of for { continue }
2017-10-25 13:38:07 -07:00
Stuart Carnie b8a6ee7c12 store command to query storage RPC APIs 2017-10-25 13:38:07 -07:00
Stuart Carnie 415ed14c53 storage service
* storage service is disabled by default
* default port 8082
* RPC interface generated using yarpc via service.proto
2017-10-25 13:38:07 -07:00
Ben Johnson 1f4e070b3a
Switch to use tsm1.CacheLoader. 2017-10-20 15:14:17 -06:00
Ben Johnson ea43660a55
Streaming inmem2tsi conversion.
Changes the `influx_inspect inmem2tsi` tool to stream each TSM/WAL
file and convert to a TSI index instead of loading the entire shard's
in-memory index first.
2017-10-20 15:14:17 -06:00
Jonathan A. Sternberg 75b8d9cd4f Code cleanup for removing the pid file 2017-10-20 09:17:16 -05:00
Jonathan A. Sternberg 2005a8c8e5 Remove the pidfile after the server has exited 2017-10-19 16:33:59 -05:00
Mark Rushakoff 4ed2e6f21e Minor cleanup 2017-10-13 17:28:24 -07:00
Mark Rushakoff f3f1cc1064 Initial integration tests for config settings 2017-10-11 17:16:42 -07:00
Jason Wilder c1b17114f5 Add time range summary 2017-10-09 14:35:30 -06:00
Jason Wilder 742c871302 Add exact counting
This adds a new flag -exact that will return exact counts instead of
estimates.  The default is to return estimates since exact counting
on a problem shard could consume a lot of memory.
2017-10-09 14:01:57 -06:00
Jason Wilder 2abd7bd1d8 Report shards in sorted shard ID order 2017-10-09 13:50:37 -06:00
Jason Wilder c69ad0978c Update influx_inspect to report total cardinalities
This updates the report command to be able to report cardinalities across
all dbs and shards
2017-10-06 16:00:20 -06:00
Jonathan A. Sternberg abf2649d3a Update influx inspect to support unsigned types 2017-10-05 10:26:14 -05:00
Joe LeGasse 1525069213 Merge pull request #8892 from influxdata/jl-tag-values
auth: add series auth to 'show tag values'
2017-10-03 08:47:39 -04:00
Mark Rushakoff 8c70cfd74b Merge pull request #8808 from liketic/feature/issues/8590
Influx CLI: Make cli case insensitive
2017-09-28 09:13:19 -07:00
Joe LeGasse 1443b22379 auth: add series auth to 'show tag values' 2017-09-27 20:01:18 -04:00
nwante e995bba24e simple typo fix
just a simple wording fix when I was reading the docs.
2017-09-25 08:16:57 -04:00
liketic 1dfdc30377 Influx CLI: Make cli case insensitive 2017-09-08 23:20:54 +08:00
emluque fa681edcb9 6563 Support Ctrl+C to cancel a running query in the Influx CLI
Solved the issue by using context on the http request on the client.
2017-09-01 08:02:27 -05:00
Joe LeGasse 732a0c2eaa Merge pull request #8769 from influxdata/jl-map-cleanup
cleanup: remove poor usage of ',ok' with maps
2017-08-31 09:18:42 -04:00
David Norton 2aa446bf24 fix #8638: inspect shouldn't err on missing file
influx_inspect walks the data and wal directories building a list of
files to export. It then opens, reads, and exports each. If the file was
deleted between the time it was added to the list and the time the
inspect tool attempts to read it, the file is now skipped without
emitting an error.
2017-08-30 09:59:03 -04:00
Joe LeGasse a95647b720 cleanup: remove poor usage of ',ok' with maps
There are several places in the code where comma-ok map retrieval was
being used poorly. Some were benign, like checking existence before
issuing an unconditional delete with no cleanup. Others were potentially
far more serious: assuming that if 'ok' was true, then the resulting
pointer retrieved from the map would be non-nil. `nil` is a perfectly
valid value to store in a map of pointers, and the comma-ok syntax is
meant for when membership is distinct from having a non-zero value.
There was only one or two cases that I saw that being used correctly for
maps of pointers.
2017-08-30 09:49:31 -04:00
Jonathan A. Sternberg 9a2357c2c0 Separate the query engine into a separate package
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
Ben Johnson 06bc3b6fbf
TSI Index Migration 2017-08-15 11:40:24 -06:00
Edd Robinson 442581d299 Merge pull request #8592 from influxdata/er-mutex-profile
Adds mutex profile
2017-08-14 22:09:19 +01:00
Edd Robinson 81c06b58fa Merge pull request #8462 from xginn8/backup_stdout
backup output should go to stdout not stderr
2017-08-14 22:08:25 +01:00
Edd Robinson 8934aacebb Adds mutex profile
Mutex profiles are now available via the:

  - /debug/pprof/mutex
  - /debug/pprof/all

endpoints.
2017-08-04 14:36:39 +01:00
Jason Wilder 778000435a Conver all keys from string to []byte in TSM engine
This switches all the interfaces that take string series key to
take a []byte.  This eliminates many small allocations where we
convert between to two repeatedly.  Eventually, this change should
propogate futher up the stack.
2017-07-28 11:00:50 -06:00
xginn8 faec9625bc Merge branch 'master' into backup_stdout 2017-07-13 18:10:54 -04:00
Edd Robinson d2ce7060c5 Merge branch 'master' into backup_stdout 2017-07-07 16:27:39 +01:00
Adam 2259ada8c3 adds a new header key/value X-Influxdb-Build that has value OSS if called from open-source build, and ENT if called from enterprise. This commit sets the value for the OSS case, and also creates the proper flag 2017-07-07 10:59:26 -04:00
Jason Wilder 5512c97dac Merge pull request #8515 from influxdata/jw-subscriber
Append subscribers service after it's open
2017-06-21 12:32:52 -06:00
Jason Wilder ba8649a6d5 Merge pull request #8503 from eheydrick/backup-docs
Add a timezone component to the example backup command to make it RFC3339 compliant
2017-06-21 09:40:05 -06:00
Jason Wilder 72eeacaed4 Append subscribers service after it's open
The Points channel is nil until after the subscriber service is opened.
If it is append before it's opened, the PointsWriter holds onto the
old reference.
2017-06-21 09:28:59 -06:00
Ben Johnson 431b947149
Add dumptsi command to help command. 2017-06-19 09:49:59 -06:00
Eric Heydrick 810c183963 Add a timezone component to the example backup command to make it
RFC3339 compliant
2017-06-16 15:43:33 -07:00
Matthew McGinn 4159fd693e break out backup logging into 2 loggers, one for Stdout and one for Stderr 2017-06-14 15:20:58 -04:00
Stuart Carnie 932edd90b2 Merge branch 'master' into sgc-8188 2017-06-14 10:55:06 +10:00
Jason Wilder 206fd857c3 Merge pull request #8481 from influxdata/jw-subscriber
Decouple Subscriber from PointsWriter
2017-06-13 15:55:43 -06:00
Jason Wilder 71579fcbe9 Remove direct Subscriber dependency from PointsWriter 2017-06-13 12:51:02 -06:00
lrita 20e6b820bb remove duplicated assignment in cmd/influxd/run/server 2017-06-12 15:58:52 +08:00
Stuart Carnie 2de52834f0 CQ statistics written to monitor database, addresses #8188
* off by default, enabled by `query-stats-enabled`
* writes to cq_query measurement of configured monitor database
* see CHANGELOG for schema of individual points
2017-06-10 09:20:38 +08:00
Matthew McGinn d9f984a886 backup output should go to stdout not stderr 2017-06-07 13:11:58 -04:00
Joe LeGasse 815f740f4c initial fga work
wip

wip

fix tests / build
2017-05-26 13:16:27 -07:00
Ben Johnson c744e2f562
TSI pull request fixes. 2017-05-23 09:01:05 -06:00
Mark Rushakoff c4f11afc90 Default RPC bind to localhost
Prior to this change, the default configuration would listen on all
interfaces, potentially exposing the RPC to the public internet.
2017-05-12 17:02:51 -07:00
Jason Wilder 2cac46ebbc Convert usage of strings to []byte
Measurement name and field were converted between []byte and string
repetively causing lots of garbage.  This switches the code to use
[]byte in the write path.
2017-05-12 14:05:19 -06:00
Jason Wilder b150a6293c Merge pull request #8380 from influxdata/jw-wal-buffer
Use buffer writer for wal segments
2017-05-11 08:34:44 -06:00
Jason Wilder e102fcca9c Use buffer writer for wal segments 2017-05-10 11:42:32 -06:00
Sebastian Borza 6bb85f809a
sort influx_inspect detailed report results 2017-05-08 23:30:40 -05:00
Ben Johnson 489c89bea4
Add tsi support tooling. 2017-05-08 11:00:15 -06:00
Jonathan A. Sternberg 1300f4cc6c Remove the admin UI 2017-04-25 16:58:24 -05:00
Ben Johnson 9c97cd8601
Merge remote-tracking branch 'upstream/master' into tsi 2017-04-04 12:46:09 -06:00
Jason Wilder 8da84e6144 Merge branch 'master' into tsi 2017-04-03 11:21:02 -06:00
Cory LaNou bc0759d0fc redact passwords before saving history in cli 2017-04-03 11:20:14 -05:00
Edd Robinson fddaff2cc8 Merge master in 2017-03-29 18:00:28 +01:00
Cory LaNou 9f674ccec4
suppress headers in output for influx when they are the same 2017-03-28 12:50:23 -05:00
Cory LaNou 80948223f4 add chunked/chunksize as setting/options 2017-03-28 12:48:21 -05:00
Cory LaNou e07d84525d
add chunked/chunksize as setting/options 2017-03-17 18:25:52 -05:00
Ben Johnson 358b1e0b05
Merge remote-tracking branch 'upstream/master' into tsi 2017-03-15 10:13:32 -06:00
Mark Rushakoff 535cf597f1 Report subset of config values in SHOW DIAGNOSTICS
This includes hand-selected config settings that are safe to expose and
not expected to include any kind of secrets.

Fixes #7821
2017-03-14 11:34:19 -07:00
Jason Wilder 78b7815c49 Add block type for BlockIterator 2017-03-09 09:16:59 -07:00
Mark Rushakoff 601cbcd084 Merge branch '1.2' into mr-merge-12 2017-02-17 16:14:22 -08:00
Jonathan A. Sternberg 2fe48d6781 Rename zap import back to github.com/uber-go/zap
They rebased a revision we were previously relying upon that allowed us
to use the vanity name so we are reverting back to an older version with
the old import path.
2017-02-17 17:17:22 -06:00
Jason Wilder 524169b814 Move integration tests to tests dir 2017-02-15 10:37:29 -07:00
Jason Wilder 517df0f939 Make integration tests able to run against remote server
URL=http://localhost:8086 go test -parallel 1 ./cmd/influxd/run

will run the tests over HTTP against localhost:8086.  They currently
need to be run serially since they all write to the same DB.
2017-02-15 10:07:40 -07:00
Mark Rushakoff c762ab49ee Merge pull request #7974 from influxdata/mr-4785-show-databases
Allow non-admin users to execute SHOW DATABASES
2017-02-13 15:04:00 -08:00
Jason Wilder c3de210ded Merge branch '1.2' into jw-merge-12 2017-02-13 11:45:27 -07:00
Mark Rushakoff 53699aa24f Allow non-admin users to execute SHOW DATABASES
This commit introduces a new interface type, influxql.Authorizer, that
is passed as part of a statement's execution context and determines
whether the context is permitted to access a given database. In the
future, the Authorizer interface may be expanded to other resources
besides databases. In this commit, the Authorizer interface is
specifically used to determine which databases are returned when
executing SHOW DATABASES.

When HTTP authentication is enabled, the existing meta.UserInfo struct
implements Authorizer, meaning admin users can SHOW every database, and
non-admin users can SHOW only databases for which they have read and/or
write permission.

When HTTP authentication is disabled, all databases are visible through
SHOW DATABASES.

This addresses a long-standing issue where Chronograf or Grafana would
be unable to list databases if the logged-in user did not have admin
privileges.

Fixes #4785.
2017-02-13 08:59:16 -08:00
Jonathan A. Sternberg 2ad1668c2a Prevent a panic when aggregates are used in an inner query with a raw query
The following types of queries will panic:

    SELECT mean, host FROM (SELECT mean(value) FROM cpu GROUP BY host)
    SELECT top(sum, host, 3) FROM (SELECT sum(value) FROM cpu GROUP BY host)

These queries _should_ work, but due to a current limitation with
aggregate functions, the aggregate functions won't return any auxiliary
fields. So even if a tag is not an auxiliary field, it is treated that
way by the query engine and this query will fail.

Fixing this properly will take a longer period of time. This fix just
prevents the panic from killing the server while we fix this for real.
2017-02-08 11:44:56 -06:00
Jason Wilder 1bc0f68490 Merge branch '1.2' into jw-merge-12 2017-02-07 12:48:36 -07:00
Jonathan A. Sternberg e1fa48d0dd Fix ORDER BY time DESC with ordering series keys
The order of series keys is in ascending alphabetical order, not
descending alphabetical order, when it is ordered by descending time.
This fixes the ordering so points are returned in descending order. The
emitter also had the conditions for choosing which iterator to use in
the wrong direction (which only affects aggregates with `FILL(none)`).
2017-02-06 15:49:12 -06:00
Jason Wilder 2e95b4043c Merge branch '1.2' into jw-merge-12 2017-02-02 16:40:36 -07:00
Jonathan A. Sternberg e49ba016fa Fix incorrect math when aggregates that emit different times are used
When using `non_negative_derivative()` and `last()` in a math aggregate
with each other, the math would not be matched with each other because
one of those aggregates would emit one fewer point than the others. The
math iterators have been modified so they now track the name and tags of
a point and match based on those.

This isn't necessarily ideal and may come to bite us in the future. We
don't necessarily have a defined structure for all iterators so it can
be difficult to know which of two points is supposed to come first in
the ordering. This uses the common ordering that usually makes sense,
but the query engine is getting complicated enough where I am not 100%
certain that this is correct in all circumstances.
2017-02-02 14:40:41 -06:00
Ben Johnson c246f3d9b0
Use inmem index on existing shards. 2017-02-02 10:04:25 -07:00
Joe LeGasse dd9278a098 regex: don't use exact match for case insensitive expression
Fixes #7906

In an attempt to reduce the overhead of using regex for exact matches,
the query parser will replace `=~ /^thing$/` with `== 'thing'`, but the
conditions being checked would ignore if any flags were set on the
expression, so `=~ /(?i)^THING$/` was replaced with `== 'THING'`, which
will fail unless the case was already exact. This change ensures that no
flags have been changed from those defaulted by the parser.
2017-02-02 10:49:12 -05:00
Joe LeGasse 93d18d42a6 regex: don't use exact match for case insensitive expression
Fixes #7906

In an attempt to reduce the overhead of using regex for exact matches,
the query parser will replace `=~ /^thing$/` with `== 'thing'`, but the
conditions being checked would ignore if any flags were set on the
expression, so `=~ /(?i)^THING$/` was replaced with `== 'THING'`, which
will fail unless the case was already exact. This change ensures that no
flags have been changed from those defaulted by the parser.
2017-02-02 10:25:08 -05:00
Cory LaNou e3e319a176 fix panic in query execution 2017-02-01 10:51:26 -06:00
Paul Dix a801c9dea6 Merge pull request #7889 from influxdata/js-subquery-fixes
Cherry-pick 1.2 fixes for subqueries into master
2017-01-26 10:49:37 -05:00
Edd Robinson 91ee34b111 Merge pull request #7837 from influxdata/er-tidy
General tidy up and subtle bug fixes
2017-01-26 13:43:07 +00:00
Jonathan A. Sternberg 2980f5b2b4 Fix mapping of types when the measurement uses a regex
With the new shard mapper implementation, regexes were just ignored so
it attempted to look up the field type inside of a measurement with no
name (which cannot possibly exist) so it would think the field didn't
exist and map it as the unknown type.
2017-01-25 16:32:57 -06:00
Jonathan A. Sternberg 552408c949 Fix mapping of types when the measurement uses a regex
With the new shard mapper implementation, regexes were just ignored so
it attempted to look up the field type inside of a measurement with no
name (which cannot possibly exist) so it would think the field didn't
exist and map it as the unknown type.
2017-01-25 09:49:51 -06:00
Ben Johnson 047c21f4d9
Merge remote-tracking branch 'upstream/master' into tsi 2017-01-24 09:28:58 -07:00
Jonathan A. Sternberg 83c6d53294 Support the WHERE clause in outer queries with subqueries 2017-01-23 15:01:32 -06:00
Jonathan A. Sternberg 3d4d9062a0 Update subqueries so groupings are propagated to inner queries
Previously, only time expressions got propagated inwards. The reason for
this was simple. If the outer query was going to filter to a specific
time range, then it would be unnecessary for the inner query to output
points within that time frame. It started as an optimization, but became
a feature because there was no reason to have the user repeat the same
time clause for the inner query as the outer query. So we allowed an
aggregate query with an interval to pass validation in the subquery if
the outer query had a time range. But `GROUP BY` clauses were not
propagated because that same logic didn't apply to them. It's not an
optimization there. So while grouping by a tag in the outer query
without grouping by it in the inner query was useless, there wasn't any
particular reason to care.

Then a bug was found where wildcards would propagate the dimensions
correctly, but the outer query containing a group by with the inner
query omitting it wouldn't correctly filter out the outer group by. We
could fix that filtering, but on further review, I had been seeing
people make that same mistake a lot. People seem to just believe that
the grouping should be propagated inwards. Instead of trying to fight
what the user wanted and explicitly erase groupings that weren't
propagated manually, we might as well just propagate them for the user
to make their lives easier. There is no useful situation where you would
want to group into buckets that can't physically exist so we might as
well do _something_ useful.

This will also now propagate time intervals to inner queries since the
same applies there. But, while the interval propagates, the following
query will not pass validation since it is still not possible to use a
grouping interval with a raw query (even if the inner query is an
aggregate):

    SELECT * FROM (SELECT mean(value) FROM cpu) WHERE time > now() - 5m GROUP BY time(1m)

This also means wildcards will behave a bit differently. They will
retrieve dimensions from the sources in the inner query rather than just
using the dimensions in the group by.

Fixing top() and bottom() to return the correct auxiliary fields.
Unfortunately, we were not copying the buffer with the auxiliary fields
so those values would be overwritten by a later point.
2017-01-23 15:01:19 -06:00
Jonathan A. Sternberg 6cd5b690d1 Support the WHERE clause in outer queries with subqueries 2017-01-23 14:49:04 -06:00
Jonathan A. Sternberg f199c50d25 Merge pull request #7854 from influxdata/js-7846-subquery-tag-propagation
Update subqueries so groupings are propagated to inner queries
2017-01-23 14:47:18 -06:00
Edd Robinson a67b5457f5 Merge pull request #7869 from influxdata/er-rp-validate-1.2
[Backport 1.2] #7866
2017-01-23 19:36:12 +00:00
Edd Robinson d30819b978 Ensure rp names validated in CREATE DATABASE WITH 2017-01-23 19:18:07 +00:00
Edd Robinson 0804cdb7b5 Ensure rp names validated in CREATE DATABASE WITH 2017-01-23 19:00:19 +00:00
Cory LaNou d54a955068 allow partial writes on field conflicts 2017-01-23 11:54:46 -07:00
Jonathan A. Sternberg f628b4a198 Update subqueries so groupings are propagated to inner queries
Previously, only time expressions got propagated inwards. The reason for
this was simple. If the outer query was going to filter to a specific
time range, then it would be unnecessary for the inner query to output
points within that time frame. It started as an optimization, but became
a feature because there was no reason to have the user repeat the same
time clause for the inner query as the outer query. So we allowed an
aggregate query with an interval to pass validation in the subquery if
the outer query had a time range. But `GROUP BY` clauses were not
propagated because that same logic didn't apply to them. It's not an
optimization there. So while grouping by a tag in the outer query
without grouping by it in the inner query was useless, there wasn't any
particular reason to care.

Then a bug was found where wildcards would propagate the dimensions
correctly, but the outer query containing a group by with the inner
query omitting it wouldn't correctly filter out the outer group by. We
could fix that filtering, but on further review, I had been seeing
people make that same mistake a lot. People seem to just believe that
the grouping should be propagated inwards. Instead of trying to fight
what the user wanted and explicitly erase groupings that weren't
propagated manually, we might as well just propagate them for the user
to make their lives easier. There is no useful situation where you would
want to group into buckets that can't physically exist so we might as
well do _something_ useful.

This will also now propagate time intervals to inner queries since the
same applies there. But, while the interval propagates, the following
query will not pass validation since it is still not possible to use a
grouping interval with a raw query (even if the inner query is an
aggregate):

    SELECT * FROM (SELECT mean(value) FROM cpu) WHERE time > now() - 5m GROUP BY time(1m)

This also means wildcards will behave a bit differently. They will
retrieve dimensions from the sources in the inner query rather than just
using the dimensions in the group by.

Fixing top() and bottom() to return the correct auxiliary fields.
Unfortunately, we were not copying the buffer with the auxiliary fields
so those values would be overwritten by a later point.
2017-01-23 12:38:10 -06:00
Cory LaNou 0103e44896
allow partial writes on field conflicts 2017-01-23 12:25:35 -06:00
Edd Robinson fb7388cdfc Remove dead code from various pkgs 2017-01-17 09:47:34 -08:00
gunnaraasen c8e15da54d Remove token message; Fixes #7823 2017-01-11 13:43:45 -08:00
Mark Rushakoff bbb43faad2 Add more config validation 2017-01-10 10:28:49 -08:00
Jonathan A. Sternberg e7b7984c27 Merge pull request #7817 from influxdata/js-7326-verbose-output-for-ssl-connection-errors
Verbose output for SSL connection errors
2017-01-10 12:10:55 -06:00
Jonathan A. Sternberg 73b76d1227 Verbose output for SSL connection errors
When an error that appears to be an SSL error happens without SSL
enabled, the client will attempt to reconnect with SSL just to see if
that works. If it works, it exits with an error message telling the user
to add `-ssl`. It will also do the same if the SSL connection is unsafe
although it will warn that this is insecure.
2017-01-10 11:53:17 -06:00
Jonathan A. Sternberg b58d1778e2 Remove improper newlines from logging statements 2017-01-10 11:20:09 -06:00
Mark Rushakoff a135906b43 Merge pull request #7747 from influxdata/mr-lint-cleanup
Miscellaneous lint cleanup
2017-01-10 08:22:00 -08:00
Mark Rushakoff 8c2cfd14af Merge pull request #7808 from influxdata/mr-fix-benchmarks
Fix broken server benchmarks
2017-01-09 15:13:01 -08:00
Jonathan A. Sternberg 4a559c4620 Merge pull request #7646 from influxdata/js-4619-subqueries
Support subquery execution in the query language
2017-01-09 14:14:01 -06:00
Mark Rushakoff c126dc5f19 Fix broken server benchmarks
These seem to have been broken in #7368.
2017-01-09 11:09:25 -08:00
Jason Wilder eb4d311c0a Add retry/backup when backing up a shard fails
The backup command can fail if a snapshot is running which silently
closes the connection.  This causes the backup shard command to continue
on as if nothing failed.
2017-01-09 11:28:48 -07:00
Ben Johnson 64c7715243
Rebase fixes. 2017-01-09 10:10:12 -07:00
Jonathan A. Sternberg d7c8c7ca4f Support subquery execution in the query language
This adds query syntax support for subqueries and adds support to the
query engine to execute queries on subqueries.

Subqueries act as a source for another query. It is the equivalent of
writing the results of a query to a temporary database, executing
a query on that temporary database, and then deleting the database
(except this is all performed in-memory).

The syntax is like this:

    SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *)

This will execute derivative and then sum the result of those derivatives.
Another example:

    SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host)

This would let you find the maximum minimum value of each host.

There is complete freedom to mix subqueries with auxiliary fields. The only
caveat is that the following two queries:

    SELECT mean(value) FROM cpu
    SELECT mean(value) FROM (SELECT value FROM cpu)

Have different performance characteristics. The first will calculate
`mean(value)` at the shard level and will be faster, especially when it comes to
clustered setups. The second will process the mean at the top level and will not
include that optimization.
2017-01-07 13:00:48 -06:00
Jason Wilder bbd9d97d73 Re-enabled TestServer_BackupAndRestore
It was failing intermittently, but seems to fail consistently one
re-enabled.  A slice pointer was incremented too early causing a
panic.

Fixes #6590
2017-01-06 16:55:12 -07:00
Ben Johnson 2b3cd415e2
Fixing rebase. 2017-01-06 09:52:16 -07:00
Edd Robinson 0f9b2bfe6a
Fix tests 2017-01-05 10:16:15 -07:00
Ben Johnson 4aa78383d1
Fix tsi1 series deletion. 2017-01-05 10:14:48 -07:00
Ben Johnson 409b0165f5
shared in-memory index 2017-01-05 10:09:57 -07:00
Edd Robinson 05bc4dec00
Refactor 2017-01-05 09:50:23 -07:00
Edd Robinson 2171d9471b
Initialise index in shards 2017-01-05 09:42:48 -07:00
Mark Rushakoff 390a16925d Merge pull request #7781 from influxdata/mr-godoc
Godoc cleanup
2017-01-04 14:11:51 -08:00
Mark Rushakoff 07b87f2630 Miscellaneous lint cleanup 2017-01-03 09:47:32 -08:00
Michael Nikitochkin 5ebd4244b1 Merge branch 'master' into env-array-config 2017-01-02 16:35:55 +01:00
Mark Rushakoff 6768c6ed3b Update godoc for the cmd package and subpackages 2016-12-30 11:58:43 -08:00
Gustav Westling 69c5354d98
Use length instead of removing it 2016-12-30 12:23:40 +01:00
Gustav Westling 56d98325da
Removed ineffective assignments, and added checks for errors that previsouly was not checked 2016-12-29 20:26:15 +01:00
Michael Nikitochkin 65b08e56f7 [#7323]: Allow add items to array config via ENV
Allow to create a new templates or tags configs, if there are no records
in the default config.

Fixes: #6943
2016-12-23 09:20:46 +01:00
Cory LaNou bc5736f59d Merge pull request #7672 from influxdata/cjl-7563-rp-duration-inf
Enforce minimum shard duration when creating retention policies
2016-12-20 12:16:07 -06:00
Cory LaNou 880c7cdcc8 Merge branch 'master' into cjl-3188-cli-rp-context 2016-12-20 09:55:40 -06:00
Cory LaNou 0cbdea531a add the ability for retention policy context in cli with use command 2016-12-20 09:15:38 -06:00
Cory LaNou fbc9e3cfcc add clear command to cli 2016-12-20 09:14:20 -06:00
Cory LaNou 572da8985c enforce minimum shard duration when creating retention policies 2016-12-20 09:11:43 -06:00
Mark Rushakoff 295a29d4ea Fix quoting on exported string fields
The previous implementation was wrong and double-escaped quotes and
backslashes.
2016-12-18 00:37:48 -08:00
Mark Rushakoff ff78c84b0f Improve export performance
Benchmark improvements with this change:

benchmark                                   old ns/op     new ns/op     delta
BenchmarkExportTSMFloats_100s_250vps-4      23206480      10279106      -55.71%
BenchmarkExportTSMInts_100s_250vps-4        17995000      5762310       -67.98%
BenchmarkExportTSMBools_100s_250vps-4       17067605      4235467       -75.18%
BenchmarkExportTSMStrings_100s_250vps-4     54846997      34682568      -36.76%
BenchmarkExportWALFloats_100s_250vps-4      23459937      10436297      -55.51%
BenchmarkExportWALInts_100s_250vps-4        18747150      6236062       -66.74%
BenchmarkExportWALBools_100s_250vps-4       17988273      4814358       -73.24%
BenchmarkExportWALStrings_100s_250vps-4     59700802      35815739      -40.01%

benchmark                                   old allocs     new allocs     delta
BenchmarkExportTSMFloats_100s_250vps-4      201442         51738          -74.32%
BenchmarkExportTSMInts_100s_250vps-4        201442         51728          -74.32%
BenchmarkExportTSMBools_100s_250vps-4       201441         51638          -74.37%
BenchmarkExportTSMStrings_100s_250vps-4     404092         201584         -50.11%
BenchmarkExportWALFloats_100s_250vps-4      250322         75627          -69.79%
BenchmarkExportWALInts_100s_250vps-4        250323         75617          -69.79%
BenchmarkExportWALBools_100s_250vps-4       250321         75527          -69.83%
BenchmarkExportWALStrings_100s_250vps-4     452868         225291         -50.25%

benchmark                                   old bytes     new bytes     delta
BenchmarkExportTSMFloats_100s_250vps-4      5170539       2351789       -54.52%
BenchmarkExportTSMInts_100s_250vps-4        5143189       2331276       -54.67%
BenchmarkExportTSMBools_100s_250vps-4       3724951       2143780       -42.45%
BenchmarkExportTSMStrings_100s_250vps-4     17131400      10796281      -36.98%
BenchmarkExportWALFloats_100s_250vps-4      4487868       1468109       -67.29%
BenchmarkExportWALInts_100s_250vps-4        4458395       1452359       -67.42%
BenchmarkExportWALBools_100s_250vps-4       2838719       1258755       -55.66%
BenchmarkExportWALStrings_100s_250vps-4     16787201      10010700      -40.37%

Also, after improving those benchmarks, I did a time-filtered export on
a 450MB TSM file to a 21GB plain text output, with and without the
bufio.BufferedWriter.

Without buffering, it took about 263s, and with buffering, it took about
60s, for a delta of about -77%.
2016-12-17 20:15:39 -08:00
Mark Rushakoff da45aab52c Clean up export code, add tests and benchmarks
The export code was moved around a bit, particularly to ease testing
export of a single TSM or WAL file. The functionality should not have
changed.
2016-12-17 18:17:18 -08:00
Jonathan A. Sternberg ec57108520 Use proper uber-go/zap import path
It looks like the real import path to the project is go.uber.org/zap
instead of github.com/uber-go/zap since the example in the project
references that path.
2016-12-15 08:54:14 -06:00
Jonathan A. Sternberg 74d474594f Remove the override of GOMAXPROCS
This was needed when we were on go 1.4 but hasn't been needed since go
1.5. It was kept because we weren't sure if we were going to have to
rollback to an older version of Go at that time and we kept it so we
wouldn't forget to readd it.

Now that we are on go 1.7 with go 1.4 deprecated, there is no going back
so we might as well remove this so people can set GOMAXPROCS to a custom
value using environment variables.
2016-12-14 12:00:40 -06:00
Jonathan A. Sternberg 21502a39e8 Switch logging to use structured logging everywhere
The logging library has been switched to use uber-go/zap. While the
logging has been changed to use structured logging, this commit does not
change any of the logging statements to take advantage of the new
structured log or new log levels. Those changes will come in future
commits.
2016-12-14 10:45:15 -06:00
Cory LaNou 6f027ac60e Merge pull request #7702 from influxdata/cjl-6527-cli-precision-case-sensitive
fix precision being case sensitive in cli
2016-12-07 12:15:34 -06:00
Cory LaNou dddc3c6690
fix precision being case sensitive in cli 2016-12-07 12:04:19 -06:00
Cory LaNou 4442615f15 use spaces instead of tabs for CLI column output format 2016-12-07 11:21:04 -06:00
Cory LaNou a90a1f4b7f
cli was caching db/rp for insert into statements 2016-12-07 09:22:48 -06:00
Edd Robinson a98ad483ee Refactor how CLI manages configuration options 2016-12-02 18:26:47 +00:00
Edd Robinson 840fb6af38 Ensure unsafessl option passed to importer 2016-12-02 18:26:47 +00:00
Jonathan A. Sternberg bffc759cf9 Return the time from a percentile call on an integer
`percentile()` is supposed to be a selector and return the time of the
point, but that only got changed when the input was a float. Updating
the integer processor to also return the time of the point rather than
the beginning of the interval.
2016-12-01 12:34:48 -06:00
Jonathan A. Sternberg 3dc5165b3c Merge pull request #7505 from oiooj/pr-influx-unixsocket
support unix socket connection for influx CLI
2016-11-30 09:02:32 -06:00
oiooj 8ddb11cee2 support unix socket connect for influx CLI 2016-11-30 22:37:56 +08:00
Allen Petersen 31129ab0e9 Use slash separator for filenames in tar archives
NO-OP on platforms with unix path separator.
On Windows paths get converted to slashes before adding to archive and back to backslashes during restore.
2016-11-29 09:44:08 -08:00
Jonathan A. Sternberg b4db76cee2 Introduce syntax for marking a partial response with chunking
The `partial` tag has been added to the JSON response of a series and
the result so that a client knows when more of the series or result will
be sent in a future JSON chunk.

This helps interactive clients who don't want to wait for all of the
data to know if it is done processing the current series or the current
result. Previously, the client had to guess if the next chunk would
refer to the same result or a new result and it had to match the name
and tags of the two series to know if they were the same series. Now,
the client just needs to check the `partial` field included with the
response to know if it should expect more.

Fixed `max-row-limit` so it counts rows instead of results and it
truncates the response when the `max-row-limit` is reached.
2016-11-22 11:16:22 -06:00
Jonathan A. Sternberg e885fe5117 Expand string and boolean fields when using a wildcard with sample() 2016-11-15 15:56:47 -06:00
Marc 7d277d0def Add support for collectd security policy
There are 2 new keys in the configuration file.
- security-level: "none", "sign", or "encrypt".
- auth-file: The location of the user/password file.

Please see the collectd network doc for more details.
2016-11-08 08:42:09 +08:00
Mark Rushakoff d52eb01c17 Merge pull request #7492 from influxdata/mr-influx_inspect-help-verify
Mention verify subcommand in influx_inspect help
2016-11-07 13:23:39 -08:00
Jason Wilder 3c0aaae8ff Merge pull request #7556 from influxdata/rk-readme-fix
Update example in influx_inspect README.md
2016-11-07 14:16:43 -07:00
Jonathan A. Sternberg be49f3ee2c Merge pull request #7586 from influxdata/js-7575-fix-execute-flag-with-no-tty
Fix the `-execute` and `-import` when there is no TTY
2016-11-07 15:07:21 -06:00
Jason Wilder b6863b04d1 Disable/deprecate admin interface 2016-11-07 12:04:51 -07:00
Jonathan A. Sternberg 8a57f27eb9 Fix the `-execute` and `-import` when there is no TTY 2016-11-04 16:35:38 -05:00
Cory LaNou 6e290040bb
remove SetDefaultRetentionPolicy method from meta.Client 2016-11-03 09:39:41 -05:00
Cory LaNou cd272ce6c3 fix retention policy creation inconsistencies 2016-11-03 09:09:43 -05:00
Jonathan A. Sternberg 1b2fa645ee Fix incorrect grouping when multiple aggregates are used with sparse data
When a query would use a grouping with two different aggregates, it was
possible for one of the aggregates to return a value from a different
series key than the second aggregate. When these series keys didn't
match, the returned grouping would be screwed up because it sorted by
time before checking for name and tags.

This did not happen when the aggregates returned values for the same
series keys because then the iterators were aligned with each other.
2016-11-02 13:35:22 -05:00
Regan Kuchan 3ebaba4714 Update example in influx_inspect README.md 2016-11-01 10:16:23 -07:00
Jason Wilder 0b6f5441b9 Add config option to messages when limits exceeded
When a limit is exceeded, we return errors and sometimes log (if appropriate)
that a limit was exceeded.  The messages don't always provide an indication
as to where or how they are configured.

Instead, return the config option (easily searchable for) as well as the limit
currently set and the value that exceeded it when possible.
2016-10-28 14:54:45 -06:00
Jonathan A. Sternberg 3e29d3d9ca Truncate the version string when linking to the documentation
The admin console would dynamically discover the version from the
InfluxDB server, but for patch releases, it included the patch in the
link to the documentation and that wasn't a valid link.

Truncate the version so the documentation url is correct since we only
do documentation for `major.minor`.
2016-10-27 16:40:08 -05:00
joelegasse 05c252696e Merge pull request #7499 from influxdata/jl-export-doc
Update `influx_inspect export` doc
2016-10-26 14:17:27 -04:00
Jonathan A. Sternberg f1fbb55909 Change default time boundaries for raw queries
Changes the default time boundaries for raw queries so raw queries will
range until the end of time. Aggregate queries continue to have their
default end time be `now()`.
2016-10-25 15:08:51 -05:00
Cory LaNou cc8e34886c escape fields when exporting tsm/wal files 2016-10-25 13:36:45 -05:00
Joe LeGasse ad62d04ce3 influx_inspect: update flag descriptions 2016-10-25 11:05:06 -04:00
Edd Robinson b12b0d12fb Add regex benchmarks and fix existing approach 2016-10-25 11:10:03 +01:00
Jason Wilder 686d1a7ba4 Remove unused config options 2016-10-24 15:32:38 -06:00
Mark Rushakoff ab7bfe2473 Mention verify subcommand in influx_inspect help 2016-10-24 08:46:48 -07:00
Edd Robinson 0ae775f230 Prevent influx from panicking on connect 2016-10-24 16:09:06 +01:00
Joe LeGasse aa9f832166 influx_inspect: make 'go vet' happy 2016-10-21 10:52:13 -04:00
Joe LeGasse 5be37c014d influx_inspect: Update usage doc for 'export' 2016-10-21 10:35:54 -04:00
Cory LaNou 5b72b874d8
remove ProcessContinousQueries from httpd endpoint 2016-10-20 11:22:36 -05:00
Jason Wilder 6fd74a6a2a Merge pull request #7423 from mnuessler/usage-help
Fix help command usage string
2016-10-19 16:22:21 -06:00
Matthias Nüßler d2b987eac8 Fix help command usage string
The information in the usage string for the `help` command about
how to get more information about a specific command was incorrect:

"influxd help [command]" does not work, but "influxd [command] --help"
does.
2016-10-19 22:30:36 +02:00
Jason Wilder 1aafea2e07 Merge pull request #7428 from ch33hau/prevent-trim-auth
auth command with leading spaces will be written to history
2016-10-18 15:12:01 -06:00
Jonathan A. Sternberg 19a61dbb44 Align binary math expression streams by time
Also fills in missing values using the fill expression for any binary
aggregation.
2016-10-18 13:31:13 -05:00
Jonathan A. Sternberg 2f5f995782 Correctly read in input from a non-interactive stream for the CLI
If you pipe in a file to the `influx` CLI, it will not try to open the
interactive line reader, but instead just send the contents of the
entire file to the server.
2016-10-17 12:58:44 -05:00
Jonathan A. Sternberg e457cefb24 Support INFLUX_USERNAME and INFLUX_PASSWORD for setting username/password in the CLI 2016-10-17 12:26:39 -05:00
Jonathan A. Sternberg e955d16f82 Merge pull request #7388 from influxdata/js-cumulative-sum-over-time
Implement cumulative_sum() function
2016-10-07 15:43:23 -05:00
ch33hau b4e98e992d Fix tiny typo 2016-10-08 02:48:27 +08:00
ch33hau 933fd8e8fe Prevent auth command with leading spaces writing to history 2016-10-08 02:45:22 +08:00
Jonathan A. Sternberg 6afc2a77a5 Implement cumulative_sum() function
The `cumulative_sum()` function can be used to sum each new point and
output the current total. For the following points:

    cpu value=2 0
    cpu value=4 10
    cpu value=6 20

This would output the following points:

    > SELECT cumulative_sum(value) FROM cpu
    time    value
    ----    -----
    0       2
    10      6
    20      12

As can be seen, each new point adds to the sum of the previous point and
outputs the value with the same timestamp.

The function can also be used with an aggregate like `derivative()`.

    > SELECT cumulative_sum(mean(value) FROM cpu WHERE time >= now() - 10m GROUP BY time(1m)
2016-10-07 10:11:53 -05:00
Michael Desa 1030306d3e Prevent writing auth command to influx_history 2016-10-06 09:30:04 -07:00
Edd Robinson 537316eb7b Ensure pprof-enabled config option is respected 2016-09-30 18:41:38 +01:00
Jason Wilder b42a2316b1 Merge pull request #7324 from miry/go1.7.1
Use go 1.7.1 for testing
2016-09-29 09:40:55 -06:00
Cory LaNou 4f952ad483 export wal files when exporting shard data 2016-09-23 15:09:17 -05:00
Michael Nikitochkin c70efc8261
Use go 1.7.1 for testing
Updated failed test that specific for duration format in string.
2016-09-19 12:28:58 +02:00
Jason Wilder 14b0f6e496 Fix panic in subscriber.Statistics
The tags passed into Statistics() calls are not supposed to be modified.
The balanceWriter in subscribers tried to modify them triggering a panic
because they can be nil.
2016-09-16 11:49:15 -06:00
Jonathan A. Sternberg 477d6231db Update source files to pass vet checks for go 1.7
The vet checks for some files did not pass for go 1.7. As part of a
preliminary start to making go 1.7 work with this software, go vet
should pass.

Also updated the gogo/protobuf dependency which fixed the code generator
to work with go 1.7 too. Ran `go generate` on the entire repository to
ensure every file was up to date.
2016-09-14 15:01:22 -05:00
Jonathan A. Sternberg 35b7460ad3 Use consistent column output from the CLI for column formatted responses
There were three different outputs that could be output with columns
that were rather strange depending on if there was a name and if there
were tags with the response.

Normalized output now has the dashes always under the column names and
no dashes anywhere else for consistency.
2016-09-13 14:20:49 -05:00
sdtsui 6471986eb1 Lint influx_tsm; Relates to #4098
- Single commit, PR follows conventions laid out by @Gouthamve in #5822
* main.go: struct field CpuFile should be CPUFile
* influx_inspect: loop equivalent to `for key := range...`
* adds comments to exported fields and consts
* fixes typo in `CHANGELOG.md`: text for #4702 now matches number
2016-09-12 18:35:14 -07:00
Gunnar df7e609c3e Merge pull request #7189 from influxdata/ga-inspect
Add time range options to influx_inspect export
2016-08-26 17:03:54 -07:00
Jonathan A. Sternberg 8b234546a8 Merge pull request #7204 from influxdata/1.0
Merge 1.0 branch to master
2016-08-25 15:20:30 -05:00
gunnaraasen 5a7d99cc93 Improve db/rp creation in influx_inspect 2016-08-25 11:53:07 -07:00
gunnaraasen d28a0a9b4e Add time range option to influx_inspect 2016-08-25 11:39:54 -07:00
Ashish Gaurav 4e17f9bb13 add mode() function & tests 2016-08-23 19:31:41 -05:00
Ben Johnson 65536676a4 Merge pull request #7138 from benbjohnson/optimize-shard-open
Reduce memory allocations in index
2016-08-17 15:27:33 -06:00
Jack Zampolin 3a179982a6 Fixes issue #7141 2016-08-17 11:10:06 -04:00
Ben Johnson 8aa224b22d
reduce memory allocations in index
This commit changes the index to point to index data in the shards
instead of keeping it in-memory on the heap.
2016-08-16 14:09:00 -06:00
Jonathan A. Sternberg 87f7c66b8a Merge pull request #7119 from influxdata/js-create-database-use-defaults
Use defaults from `meta` package for `CREATE DATABASE`
2016-08-11 10:34:22 -05:00
Jonathan A. Sternberg ab049d7f0a Support mixed duration units
It is now possible to use a mixed duration unit like `1h30m`. The
duration units can be in whatever order as long as they are connected to
each other.

There is a change to the scanner. A token such as `10x` will be scanned
as a duration literal, but will then fail to parse as an invalid
duration. This should not be a breaking change as there is no situation
where `10m10` was a valid order of tokens for the parser.

Fixes #3634.
2016-08-10 13:34:19 -05:00
Jonathan A. Sternberg 530b00bd76 Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-09 12:00:06 -05:00
Ben Johnson 55b3e63ced
concurrent series limit
This commit fixes the `MaxSelectSeriesN` limit which was broken by
the implementation of lazy iterators. The setting previously limited
the total number of series but the new implementation limits the
concurrent number of series being processed.
2016-08-09 08:58:01 -06:00
Mark Rushakoff 7787703ca8 Ensure gzip writer closed in influx_inspect export
Previously, the gzip writer may have had unflushed data when the
underlying writer was closed, and the output stream would have been
truncated early.
2016-08-08 08:52:22 -07:00
Jonathan A. Sternberg 4a89ab4482 Merge pull request #6815 from influxdata/js-content-type-response-writer
Setup an interface for writing http responses based on the Content-Type header
2016-07-29 13:28:37 -05:00
Jonathan A. Sternberg d85072ada9 Setup an interface for writing http responses based on the Content-Type header
Only `application/json` is supported right now, but this opens up the
easier possibility of additional content types to be returned from the
server.
2016-07-29 13:11:51 -05:00
Cory LaNou 1117526873 remove IF EXISTS/IF NOT EXISTS from influxql language 2016-07-29 12:58:05 -05:00
Cory LaNou ae4a1a4d00 Merge pull request #7049 from influxdata/cjl-influx-inspect-doc-update
Update influx inspect readme with DDL information
2016-07-28 18:19:12 -05:00
Jonathan A. Sternberg 23ef9484a4 Support wildcards in aggregate functions 2016-07-28 17:56:32 -05:00
Jason Wilder 6bee354530 Add Load Time column 2016-07-26 10:23:45 -06:00
Jason Wilder 78cfe441aa Skip TSM files that fail to load
This can be used a quick way to determine if any TSM files are
corrupted.
2016-07-26 10:23:44 -06:00
Jason Wilder 5bdc01a3b2 Replace info w/ report command
info was not safe to run and not really useful.
2016-07-26 10:23:44 -06:00
Jason Wilder fadc227c93 Flush tab writer after each row
tab writer was flushed after all the rows were processed which consumed
a lot of memory and was very slow.
2016-07-26 10:17:58 -06:00
Cory LaNou 7ec80e05ce update influx inspect readme with DDL information 2016-07-25 14:09:53 -05:00
Cory LaNou 968d322d6d finish tsm file exporter 2016-07-21 17:20:51 -05:00
Cory LaNou 8871f5bf8c update connection settings when changing hosts in cli 2016-07-08 11:35:42 -05:00
Jonathan A. Sternberg 036bf7d2d1 Allow any variant of the help option to trigger the help 2016-07-07 16:54:26 -05:00
Jonathan A. Sternberg bd24dc7135 Merge pull request #6970 from influxdata/js-6968-load-default-configuration-file
Always use the demo config when outputting a new config
2016-07-07 16:51:08 -05:00
Jonathan A. Sternberg ad97903a20 Always use the demo config when outputting a new config
Updating the help formatting for all of the commands for consistency.
Removed tabs from the output in favor of using spaces so it is more
clear how the output is intended to look.
2016-07-07 16:29:24 -05:00
Jonathan A. Sternberg 837a9804cf Refactoring the monitor service to avoid expvar
Truncate the time interval output of the monitor service to be on even
time intervals rather than on every minute based on the start time. This
normalizes the output from the monitor service.
2016-07-07 11:13:58 -05:00
Jonathan A. Sternberg b12f2c829b The data only build tag is no longer used
Removing the data only build tag because it is no longer used. It was
previously used for building a data only node from the open source build
that could be used in clustering, but all of the RPC code was removed
since the separation was too much of a hindrance on development whenever
we found a bug.

Since the original purpose of this is now useless, there's no point in
having a confusing build tag around.
2016-07-02 17:44:34 -05:00
Jonathan A. Sternberg 3bd51d3537 Fix fill(previous) when used with math operators 2016-06-29 09:54:12 -05:00
Cameron Sparr 98360c50d5 Trim BOM from config file for windows support
Default windows text editor (Notepad) adds a BOM to the beginning of the
file. This needs to be trimmed otherwise we will get an "invalid toml"
error.

see https://github.com/influxdata/telegraf/issues/1378
and http://utf8everywhere.org/#faq.boms
2016-06-24 07:50:13 +01:00
Jonathan A. Sternberg b8e52ce39a Merge pull request #6889 from influxdata/js-update-config-options
Update help and remove unused config options from the configuration file
2016-06-23 11:49:10 -05:00
Jack Zampolin e39dcaf8f0 Clean up code, fix struct and file names to fit with golang convention 2016-06-22 13:13:36 -07:00
Jonathan A. Sternberg 1d20498273 Allow a non-admin to call "use" for the influx cli
Previously, a non-admin could not call "use" in the influx cli since the
`SHOW DATABASES` command requires admin permissions to run. The correct
solution to this is likely to allow non-admins to call `SHOW DATABASES`,
but only see the databases they should be capable of seeing.

Since we don't have this kind of fine-grained authorization yet and
plans for it are still in the works, we do need someway to not
arbitrarily cripple non-admins attempting to use the cli program. This
is a temporary solution that will ignore any authorization errors from
`SHOW DATABASES` if authorization has been set. A warning message will
be printed and the database will be switched. This should be enough to
ensure that there is some warning that you may not have switched to a
valid database while not crippling non-admin users.

A temporary solution for #6397.
2016-06-22 08:15:18 -05:00
Jonathan A. Sternberg 22173acb70 Update help and remove unused config options from the configuration file
Normalize the output for the various help options so they all follow the
same format and display all relevant options.

Removing some of the unused config options from the configuration file
and updating the help documentation. Removing some remaining references
to clustering within the open source version.
2016-06-21 14:06:05 -05:00
Jonathan A. Sternberg 2a1e6533b7 Rename dumptsmdev to dumptsm in influx_inspect
Removes the old implementation of `dumptsm`. It was for an older version
of the tsm1 files that is no longer used and now just panics when used
on a tsm1 file. dumptsmdev has been renamed to `dumptsm`, but the old
`dumptsmdev` command still works for compatibility.
2016-06-21 10:31:05 -05:00
Jonathan A. Sternberg 1d03151631 Remove FieldCodec from tsdb package
Updated `influx_inspect` to use the `FieldDimensions` method instead
(more reliable anyway). The `influx_tsm` program used its own vendored
copy of `FieldCodec` so it is not affected by this change. `FieldCodec`
was only used for the `b1` and `bz1` engines which were removed in 0.12,
but the code that created the field codec was never removed. This
limited the maximum number of fields to 255 even though that restriction
was removed with the `tsm1` engine.

Fixes #6869.
2016-06-19 21:38:43 -05:00
Paul Dix 04d1393425 Update usage url 2016-06-17 14:27:11 -04:00
Paul Dix 79d1f31d0b Update usage info to remove anonymous 2016-06-17 14:19:10 -04:00
Jonathan A. Sternberg 9837de793c Support regex and other operations for selecting the key in SHOW TAG VALUES
This adds support for using regex expressions in SHOW TAG VALUES when
selecting the key. Also supporting the `!=` operation for the
comparison. Now you can do any of the following:

    SHOW TAG VALUES WITH KEY != "region"
    SHOW TAG VALUES WITH KEY =~ /region/
    SHOW TAG VALUES WITH KEY !~ /region/

It also adds a new SetLiteral AST node that will potentially be used in
the future to allow set operations for other comparisons in the future.

Fixes #4532.
2016-06-13 10:03:14 -05:00
Jonathan A. Sternberg 55973d2815 Separate the task manager from the query executor
The task manager now acts as its own statement executor so that a custom
statement executor can perform custom actions for KillQueryStatement and
ShowQueriesStatement.
2016-06-10 12:30:51 -05:00
Jonathan A. Sternberg b8e22d9d79 Merge pull request #6586 from influxdata/js-3733-rename-default-retention-policy
Modify the default retention policy name and make it configurable
2016-06-06 15:05:29 -05:00
Jonathan A. Sternberg 5b26780ead Perform a proper exit for the reportServer goroutine
Previously, the goroutine would block for 24 hours and would only check
every 24 hours if the server was closed. Since this never happened, the
goroutine would just be abruptly killed when the main process ended
(since nobody waited for this goroutine to end).

Updating the code to exit when the server is closed immediately and
switching the timer to a ticker instance that can be stopped.
2016-06-05 10:45:19 -05:00
Jason Wilder 619c3c4229 Merge pull request #6735 from influxdata/jw-backup
Switch backup to use shard.Snapshot
2016-05-27 11:19:15 -06:00
Jason Wilder 0fb82d8787 Use UTC for expected times in TestServer_Query_LargeTimestamp 2016-05-27 09:46:07 -06:00
Jason Wilder 11959005f4 Switch backup to use shard.Snapshot
This switch the backup shard call to use the shard Snapshot that
internally creates a snapshot by hardlinking all of the TSM and
tombstone files instead.  This reduces the time that the FileStore
is locked and will allow for larger shards to be backup more easily.
2016-05-27 09:30:25 -06:00
Edd Robinson f4fc905fa9 Reject timestamps too far in future 2016-05-27 11:07:48 +01:00
Edd Robinson bc6e1e84af Change max time for SHOW meta queries
If all points in a series are timestamped in the future, the SHOW
queries will not return anything from these series.

This commit changes the to time used when querying shards for the SHOW
queries to the maximum time, in order to ensure future points are
considered in the results for these queries.

Fixes #6599.
2016-05-26 23:22:23 +01:00
Jonathan A. Sternberg 907c88d4b5 Truncate the shard group end time if it exceeds MaxNanoTime
Related to #6599.
2016-05-25 21:25:10 -04:00
Jonathan A. Sternberg cebe256773 Merge pull request #6706 from influxdata/js-remove-deprecated-features
Remove features that were deprecated for 1.0
2016-05-25 11:56:18 -04:00
Jonathan A. Sternberg baaa782c95 Modify the default retention policy name and make it configurable
The default retention policy name is changed to "autogen" instead of
"default" since it ends up being ambiguous when we tell a user to check
the default retention policy, it is uncertain if we are referring to the
default retention policy (which can be changed) or the retention policy
with the name "default".

Now the automatically generated retention policy name is "autogen".

The default retention policy is now also configurable through the
configuration file so an administrator can customize what they think
should be the default.

Fixes #3733.
2016-05-24 09:51:23 -04:00
Jonathan A. Sternberg 1518e5eb2c Update IF NOT EXISTS and IF EXISTS deprecated message
Removed the version that it would be removed in as we will not be
removing it for 1.0.
2016-05-24 09:50:23 -04:00
Jonathan A. Sternberg 6a3f7f5ded Remove support for fixing [[collectd]] and [[opentsdb]] sections 2016-05-24 09:50:15 -04:00
Jonathan A. Sternberg 5e7e0bd19b Filter out sources that do not match the shard database/retention policy
If you use a statement like this:

    SELECT value FROM one..cpu, two..cpu

It will access both the `one` and `two` databases as if you had selected
the `cpu` measurement twice for both of them. Updated the `tsdb.Shard`
create iterator function to filter out any sources that do not apply to
that shard so this duplication doesn't happen.

Fixes #6701.
2016-05-23 17:05:33 -04:00
Jack Zampolin 1f62ecd778 Merge pull request #6443 from influxdata/js-new-stress
Add v2 stress test
2016-05-23 13:47:33 -07:00
David Norton b32208a850 fix #2048: check that RPs exist before creating CQ 2016-05-20 12:40:36 -04:00
Edd Robinson afb56c33c1 Merge pull request #6678 from influxdata/er-client-fix
Ensure client sends correct precision
2016-05-20 11:13:03 +01:00
Edd Robinson 40732a35d0 Merge pull request #6660 from influxdata/er-vet
Fix vet issues
2016-05-20 11:12:25 +01:00
Edd Robinson effcccdcaf Set default consistency to all 2016-05-19 18:54:07 +01:00
Nathaniel Cook 904e8cc825 add config for http timeout 2016-05-19 10:53:29 -06:00
Edd Robinson 1b5c6b75c2 Ensure client sends correct precision 2016-05-19 17:27:32 +01:00
Edd Robinson f680ab0f0d Fix vet issues 2016-05-18 13:34:11 +01:00
Jonathan A. Sternberg 23f6a706bb Support cast syntax for selecting a specific type
Casting syntax is done with the PostgreSQL syntax `field1::float` to
specify which type should be used when selecting a field. You can also
do `field1::field` or `tag1::tag` to specify that a field or tag should
be selected.

This makes it possible to select a tag when a field key and a tag key
conflict with each other in a measurement. It also means it's possible
to choose a field with a specific type if multiple shards disagree. If
no types are given, the same ordering for how a type is chosen is used
to determine which type to return.

The FieldDimensions method has been updated to return the data type for
the fields that get returned. The SeriesKeys function has also been
removed since it is no longer needed. SeriesKeys was originally used for
the fill iterator, but then expanded to be used by auxiliary iterators
for determining the channel iterator types. The fill iterator doesn't
need it anymore and the auxiliary types are better served by
FieldDimensions implementing that functionality, so SeriesKeys is no
longer needed.

Fixes #6519.
2016-05-16 12:08:29 -04:00
Jonathan A. Sternberg a17f3d960a SHOW TAG VALUES accepts != and !~ in WHERE clause
Fixes #6607.
2016-05-16 08:51:09 -04:00
Jonathan A. Sternberg ecacdb7d21 query-log-enabled not ignored in config anymore
Fixes #6629.
2016-05-15 22:36:18 -04:00
Jason Wilder db7d58c867 Remove old cluster config options 2016-05-11 12:59:30 -06:00
Jason Wilder 75e89e35d8 Convert deprecated [cluster] config to [coordinator] 2016-05-11 12:35:19 -06:00