Commit Graph

1553 Commits (a015996c3e6830dbca2167112f67440a824ecf6f)

Author SHA1 Message Date
Jason Wilder b42a2316b1 Merge pull request #7324 from miry/go1.7.1
Use go 1.7.1 for testing
2016-09-29 09:40:55 -06:00
Cory LaNou 4f952ad483 export wal files when exporting shard data 2016-09-23 15:09:17 -05:00
Michael Nikitochkin c70efc8261
Use go 1.7.1 for testing
Updated failed test that specific for duration format in string.
2016-09-19 12:28:58 +02:00
Jason Wilder 14b0f6e496 Fix panic in subscriber.Statistics
The tags passed into Statistics() calls are not supposed to be modified.
The balanceWriter in subscribers tried to modify them triggering a panic
because they can be nil.
2016-09-16 11:49:15 -06:00
Jonathan A. Sternberg 477d6231db Update source files to pass vet checks for go 1.7
The vet checks for some files did not pass for go 1.7. As part of a
preliminary start to making go 1.7 work with this software, go vet
should pass.

Also updated the gogo/protobuf dependency which fixed the code generator
to work with go 1.7 too. Ran `go generate` on the entire repository to
ensure every file was up to date.
2016-09-14 15:01:22 -05:00
Jonathan A. Sternberg 35b7460ad3 Use consistent column output from the CLI for column formatted responses
There were three different outputs that could be output with columns
that were rather strange depending on if there was a name and if there
were tags with the response.

Normalized output now has the dashes always under the column names and
no dashes anywhere else for consistency.
2016-09-13 14:20:49 -05:00
sdtsui 6471986eb1 Lint influx_tsm; Relates to #4098
- Single commit, PR follows conventions laid out by @Gouthamve in #5822
* main.go: struct field CpuFile should be CPUFile
* influx_inspect: loop equivalent to `for key := range...`
* adds comments to exported fields and consts
* fixes typo in `CHANGELOG.md`: text for #4702 now matches number
2016-09-12 18:35:14 -07:00
Gunnar df7e609c3e Merge pull request #7189 from influxdata/ga-inspect
Add time range options to influx_inspect export
2016-08-26 17:03:54 -07:00
Jonathan A. Sternberg 8b234546a8 Merge pull request #7204 from influxdata/1.0
Merge 1.0 branch to master
2016-08-25 15:20:30 -05:00
gunnaraasen 5a7d99cc93 Improve db/rp creation in influx_inspect 2016-08-25 11:53:07 -07:00
gunnaraasen d28a0a9b4e Add time range option to influx_inspect 2016-08-25 11:39:54 -07:00
Ashish Gaurav 4e17f9bb13 add mode() function & tests 2016-08-23 19:31:41 -05:00
Ben Johnson 65536676a4 Merge pull request #7138 from benbjohnson/optimize-shard-open
Reduce memory allocations in index
2016-08-17 15:27:33 -06:00
Jack Zampolin 3a179982a6 Fixes issue #7141 2016-08-17 11:10:06 -04:00
Ben Johnson 8aa224b22d
reduce memory allocations in index
This commit changes the index to point to index data in the shards
instead of keeping it in-memory on the heap.
2016-08-16 14:09:00 -06:00
Jonathan A. Sternberg 87f7c66b8a Merge pull request #7119 from influxdata/js-create-database-use-defaults
Use defaults from `meta` package for `CREATE DATABASE`
2016-08-11 10:34:22 -05:00
Jonathan A. Sternberg ab049d7f0a Support mixed duration units
It is now possible to use a mixed duration unit like `1h30m`. The
duration units can be in whatever order as long as they are connected to
each other.

There is a change to the scanner. A token such as `10x` will be scanned
as a duration literal, but will then fail to parse as an invalid
duration. This should not be a breaking change as there is no situation
where `10m10` was a valid order of tokens for the parser.

Fixes #3634.
2016-08-10 13:34:19 -05:00
Jonathan A. Sternberg 530b00bd76 Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-09 12:00:06 -05:00
Ben Johnson 55b3e63ced
concurrent series limit
This commit fixes the `MaxSelectSeriesN` limit which was broken by
the implementation of lazy iterators. The setting previously limited
the total number of series but the new implementation limits the
concurrent number of series being processed.
2016-08-09 08:58:01 -06:00
Mark Rushakoff 7787703ca8 Ensure gzip writer closed in influx_inspect export
Previously, the gzip writer may have had unflushed data when the
underlying writer was closed, and the output stream would have been
truncated early.
2016-08-08 08:52:22 -07:00
Jonathan A. Sternberg 4a89ab4482 Merge pull request #6815 from influxdata/js-content-type-response-writer
Setup an interface for writing http responses based on the Content-Type header
2016-07-29 13:28:37 -05:00
Jonathan A. Sternberg d85072ada9 Setup an interface for writing http responses based on the Content-Type header
Only `application/json` is supported right now, but this opens up the
easier possibility of additional content types to be returned from the
server.
2016-07-29 13:11:51 -05:00
Cory LaNou 1117526873 remove IF EXISTS/IF NOT EXISTS from influxql language 2016-07-29 12:58:05 -05:00
Cory LaNou ae4a1a4d00 Merge pull request #7049 from influxdata/cjl-influx-inspect-doc-update
Update influx inspect readme with DDL information
2016-07-28 18:19:12 -05:00
Jonathan A. Sternberg 23ef9484a4 Support wildcards in aggregate functions 2016-07-28 17:56:32 -05:00
Jason Wilder 6bee354530 Add Load Time column 2016-07-26 10:23:45 -06:00
Jason Wilder 78cfe441aa Skip TSM files that fail to load
This can be used a quick way to determine if any TSM files are
corrupted.
2016-07-26 10:23:44 -06:00
Jason Wilder 5bdc01a3b2 Replace info w/ report command
info was not safe to run and not really useful.
2016-07-26 10:23:44 -06:00
Jason Wilder fadc227c93 Flush tab writer after each row
tab writer was flushed after all the rows were processed which consumed
a lot of memory and was very slow.
2016-07-26 10:17:58 -06:00
Cory LaNou 7ec80e05ce update influx inspect readme with DDL information 2016-07-25 14:09:53 -05:00
Cory LaNou 968d322d6d finish tsm file exporter 2016-07-21 17:20:51 -05:00
Cory LaNou 8871f5bf8c update connection settings when changing hosts in cli 2016-07-08 11:35:42 -05:00
Jonathan A. Sternberg 036bf7d2d1 Allow any variant of the help option to trigger the help 2016-07-07 16:54:26 -05:00
Jonathan A. Sternberg bd24dc7135 Merge pull request #6970 from influxdata/js-6968-load-default-configuration-file
Always use the demo config when outputting a new config
2016-07-07 16:51:08 -05:00
Jonathan A. Sternberg ad97903a20 Always use the demo config when outputting a new config
Updating the help formatting for all of the commands for consistency.
Removed tabs from the output in favor of using spaces so it is more
clear how the output is intended to look.
2016-07-07 16:29:24 -05:00
Jonathan A. Sternberg 837a9804cf Refactoring the monitor service to avoid expvar
Truncate the time interval output of the monitor service to be on even
time intervals rather than on every minute based on the start time. This
normalizes the output from the monitor service.
2016-07-07 11:13:58 -05:00
Jonathan A. Sternberg b12f2c829b The data only build tag is no longer used
Removing the data only build tag because it is no longer used. It was
previously used for building a data only node from the open source build
that could be used in clustering, but all of the RPC code was removed
since the separation was too much of a hindrance on development whenever
we found a bug.

Since the original purpose of this is now useless, there's no point in
having a confusing build tag around.
2016-07-02 17:44:34 -05:00
Jonathan A. Sternberg 3bd51d3537 Fix fill(previous) when used with math operators 2016-06-29 09:54:12 -05:00
Cameron Sparr 98360c50d5 Trim BOM from config file for windows support
Default windows text editor (Notepad) adds a BOM to the beginning of the
file. This needs to be trimmed otherwise we will get an "invalid toml"
error.

see https://github.com/influxdata/telegraf/issues/1378
and http://utf8everywhere.org/#faq.boms
2016-06-24 07:50:13 +01:00
Jonathan A. Sternberg b8e52ce39a Merge pull request #6889 from influxdata/js-update-config-options
Update help and remove unused config options from the configuration file
2016-06-23 11:49:10 -05:00
Jack Zampolin e39dcaf8f0 Clean up code, fix struct and file names to fit with golang convention 2016-06-22 13:13:36 -07:00
Jonathan A. Sternberg 1d20498273 Allow a non-admin to call "use" for the influx cli
Previously, a non-admin could not call "use" in the influx cli since the
`SHOW DATABASES` command requires admin permissions to run. The correct
solution to this is likely to allow non-admins to call `SHOW DATABASES`,
but only see the databases they should be capable of seeing.

Since we don't have this kind of fine-grained authorization yet and
plans for it are still in the works, we do need someway to not
arbitrarily cripple non-admins attempting to use the cli program. This
is a temporary solution that will ignore any authorization errors from
`SHOW DATABASES` if authorization has been set. A warning message will
be printed and the database will be switched. This should be enough to
ensure that there is some warning that you may not have switched to a
valid database while not crippling non-admin users.

A temporary solution for #6397.
2016-06-22 08:15:18 -05:00
Jonathan A. Sternberg 22173acb70 Update help and remove unused config options from the configuration file
Normalize the output for the various help options so they all follow the
same format and display all relevant options.

Removing some of the unused config options from the configuration file
and updating the help documentation. Removing some remaining references
to clustering within the open source version.
2016-06-21 14:06:05 -05:00
Jonathan A. Sternberg 2a1e6533b7 Rename dumptsmdev to dumptsm in influx_inspect
Removes the old implementation of `dumptsm`. It was for an older version
of the tsm1 files that is no longer used and now just panics when used
on a tsm1 file. dumptsmdev has been renamed to `dumptsm`, but the old
`dumptsmdev` command still works for compatibility.
2016-06-21 10:31:05 -05:00
Jonathan A. Sternberg 1d03151631 Remove FieldCodec from tsdb package
Updated `influx_inspect` to use the `FieldDimensions` method instead
(more reliable anyway). The `influx_tsm` program used its own vendored
copy of `FieldCodec` so it is not affected by this change. `FieldCodec`
was only used for the `b1` and `bz1` engines which were removed in 0.12,
but the code that created the field codec was never removed. This
limited the maximum number of fields to 255 even though that restriction
was removed with the `tsm1` engine.

Fixes #6869.
2016-06-19 21:38:43 -05:00
Paul Dix 04d1393425 Update usage url 2016-06-17 14:27:11 -04:00
Paul Dix 79d1f31d0b Update usage info to remove anonymous 2016-06-17 14:19:10 -04:00
Jonathan A. Sternberg 9837de793c Support regex and other operations for selecting the key in SHOW TAG VALUES
This adds support for using regex expressions in SHOW TAG VALUES when
selecting the key. Also supporting the `!=` operation for the
comparison. Now you can do any of the following:

    SHOW TAG VALUES WITH KEY != "region"
    SHOW TAG VALUES WITH KEY =~ /region/
    SHOW TAG VALUES WITH KEY !~ /region/

It also adds a new SetLiteral AST node that will potentially be used in
the future to allow set operations for other comparisons in the future.

Fixes #4532.
2016-06-13 10:03:14 -05:00
Jonathan A. Sternberg 55973d2815 Separate the task manager from the query executor
The task manager now acts as its own statement executor so that a custom
statement executor can perform custom actions for KillQueryStatement and
ShowQueriesStatement.
2016-06-10 12:30:51 -05:00
Jonathan A. Sternberg b8e22d9d79 Merge pull request #6586 from influxdata/js-3733-rename-default-retention-policy
Modify the default retention policy name and make it configurable
2016-06-06 15:05:29 -05:00
Jonathan A. Sternberg 5b26780ead Perform a proper exit for the reportServer goroutine
Previously, the goroutine would block for 24 hours and would only check
every 24 hours if the server was closed. Since this never happened, the
goroutine would just be abruptly killed when the main process ended
(since nobody waited for this goroutine to end).

Updating the code to exit when the server is closed immediately and
switching the timer to a ticker instance that can be stopped.
2016-06-05 10:45:19 -05:00
Jason Wilder 619c3c4229 Merge pull request #6735 from influxdata/jw-backup
Switch backup to use shard.Snapshot
2016-05-27 11:19:15 -06:00
Jason Wilder 0fb82d8787 Use UTC for expected times in TestServer_Query_LargeTimestamp 2016-05-27 09:46:07 -06:00
Jason Wilder 11959005f4 Switch backup to use shard.Snapshot
This switch the backup shard call to use the shard Snapshot that
internally creates a snapshot by hardlinking all of the TSM and
tombstone files instead.  This reduces the time that the FileStore
is locked and will allow for larger shards to be backup more easily.
2016-05-27 09:30:25 -06:00
Edd Robinson f4fc905fa9 Reject timestamps too far in future 2016-05-27 11:07:48 +01:00
Edd Robinson bc6e1e84af Change max time for SHOW meta queries
If all points in a series are timestamped in the future, the SHOW
queries will not return anything from these series.

This commit changes the to time used when querying shards for the SHOW
queries to the maximum time, in order to ensure future points are
considered in the results for these queries.

Fixes #6599.
2016-05-26 23:22:23 +01:00
Jonathan A. Sternberg 907c88d4b5 Truncate the shard group end time if it exceeds MaxNanoTime
Related to #6599.
2016-05-25 21:25:10 -04:00
Jonathan A. Sternberg cebe256773 Merge pull request #6706 from influxdata/js-remove-deprecated-features
Remove features that were deprecated for 1.0
2016-05-25 11:56:18 -04:00
Jonathan A. Sternberg baaa782c95 Modify the default retention policy name and make it configurable
The default retention policy name is changed to "autogen" instead of
"default" since it ends up being ambiguous when we tell a user to check
the default retention policy, it is uncertain if we are referring to the
default retention policy (which can be changed) or the retention policy
with the name "default".

Now the automatically generated retention policy name is "autogen".

The default retention policy is now also configurable through the
configuration file so an administrator can customize what they think
should be the default.

Fixes #3733.
2016-05-24 09:51:23 -04:00
Jonathan A. Sternberg 1518e5eb2c Update IF NOT EXISTS and IF EXISTS deprecated message
Removed the version that it would be removed in as we will not be
removing it for 1.0.
2016-05-24 09:50:23 -04:00
Jonathan A. Sternberg 6a3f7f5ded Remove support for fixing [[collectd]] and [[opentsdb]] sections 2016-05-24 09:50:15 -04:00
Jonathan A. Sternberg 5e7e0bd19b Filter out sources that do not match the shard database/retention policy
If you use a statement like this:

    SELECT value FROM one..cpu, two..cpu

It will access both the `one` and `two` databases as if you had selected
the `cpu` measurement twice for both of them. Updated the `tsdb.Shard`
create iterator function to filter out any sources that do not apply to
that shard so this duplication doesn't happen.

Fixes #6701.
2016-05-23 17:05:33 -04:00
Jack Zampolin 1f62ecd778 Merge pull request #6443 from influxdata/js-new-stress
Add v2 stress test
2016-05-23 13:47:33 -07:00
David Norton b32208a850 fix #2048: check that RPs exist before creating CQ 2016-05-20 12:40:36 -04:00
Edd Robinson afb56c33c1 Merge pull request #6678 from influxdata/er-client-fix
Ensure client sends correct precision
2016-05-20 11:13:03 +01:00
Edd Robinson 40732a35d0 Merge pull request #6660 from influxdata/er-vet
Fix vet issues
2016-05-20 11:12:25 +01:00
Edd Robinson effcccdcaf Set default consistency to all 2016-05-19 18:54:07 +01:00
Nathaniel Cook 904e8cc825 add config for http timeout 2016-05-19 10:53:29 -06:00
Edd Robinson 1b5c6b75c2 Ensure client sends correct precision 2016-05-19 17:27:32 +01:00
Edd Robinson f680ab0f0d Fix vet issues 2016-05-18 13:34:11 +01:00
Jonathan A. Sternberg 23f6a706bb Support cast syntax for selecting a specific type
Casting syntax is done with the PostgreSQL syntax `field1::float` to
specify which type should be used when selecting a field. You can also
do `field1::field` or `tag1::tag` to specify that a field or tag should
be selected.

This makes it possible to select a tag when a field key and a tag key
conflict with each other in a measurement. It also means it's possible
to choose a field with a specific type if multiple shards disagree. If
no types are given, the same ordering for how a type is chosen is used
to determine which type to return.

The FieldDimensions method has been updated to return the data type for
the fields that get returned. The SeriesKeys function has also been
removed since it is no longer needed. SeriesKeys was originally used for
the fill iterator, but then expanded to be used by auxiliary iterators
for determining the channel iterator types. The fill iterator doesn't
need it anymore and the auxiliary types are better served by
FieldDimensions implementing that functionality, so SeriesKeys is no
longer needed.

Fixes #6519.
2016-05-16 12:08:29 -04:00
Jonathan A. Sternberg a17f3d960a SHOW TAG VALUES accepts != and !~ in WHERE clause
Fixes #6607.
2016-05-16 08:51:09 -04:00
Jonathan A. Sternberg ecacdb7d21 query-log-enabled not ignored in config anymore
Fixes #6629.
2016-05-15 22:36:18 -04:00
Jason Wilder db7d58c867 Remove old cluster config options 2016-05-11 12:59:30 -06:00
Jason Wilder 75e89e35d8 Convert deprecated [cluster] config to [coordinator] 2016-05-11 12:35:19 -06:00
Jason Wilder 6cc1a34704 Rename cluster package to coordinator 2016-05-11 11:41:05 -06:00
Jason Wilder de04d0972d Remove old cluster code 2016-05-11 11:39:23 -06:00
Jonathan A. Sternberg 89346bb618 Merge pull request #6600 from influxdata/0.13
Merge 0.13 release candidate back to master
2016-05-11 13:04:26 -04:00
Jonathan A. Sternberg 733a17d9e9 Update SHOW FIELD KEYS to return the field type with the field key
Fixes #3451.
2016-05-10 13:16:57 -04:00
Cory LaNou 10db0aafeb Merge pull request #6591 from influxdata/cjl-issue-6590
intermittent test failing
2016-05-10 08:57:16 -05:00
Cory LaNou f39e9e910d intermittent test failing 2016-05-10 08:30:19 -05:00
Cory LaNou 4484182e86 remove old shard copier service 2016-05-10 08:14:51 -05:00
Jonathan A. Sternberg 5a6a34144e Fix config file loading to read from a file
Accidentally used `FromToml()` when trying to read from a file path
rather than `FromTomlFile()` which will read the file and then call
`FromToml()`.

Fixes #6578.
2016-05-08 09:00:29 -04:00
Jonathan A. Sternberg 3c3f7be355 Support the deprecated [collectd] and [opentsdb] sections temporarily
Performs an automatic search and replace on these sections of the
configuration file so they work for the 0.13 release. Will be removed
for 1.0.
2016-05-05 14:09:50 -04:00
Edd Robinson 0b338dddd8 Add remote monitor writer 2016-05-05 11:08:35 +01:00
Edd Robinson 2ae6a43c36 Disable Monitor when running dataonly mode 2016-05-04 18:22:01 +01:00
Jason Wilder 57cb3fdbc0 Merge pull request #6522 from influxdata/tp-tsm-dump
Dump TSM files to line protocol
2016-05-03 10:44:33 -06:00
Ben Johnson 49eb3b8d04
optimize show series iterator
This commit changes the `SeriesIterator` to process one measurement
at a time and uses a `floatFastDedupeIterator` to avoid point
encoding during deduplication.
2016-05-03 08:52:44 -06:00
Todd Persen efdc54ac5d Display value according to type. 2016-05-02 22:08:15 -07:00
Todd Persen 318a966b94 Change `dump` to `export` 2016-05-02 21:42:42 -07:00
Jonathan A. Sternberg 0cc99a3e27 Merge pull request #6504 from influxdata/js-2074-group-by-offsets
Support offset argument in the GROUP BY time(...) call
2016-05-02 15:00:26 -04:00
Jonathan A. Sternberg 64556e4f8e Support offset argument in the GROUP BY time(...) call
An offset of `time(1m, now())` will anchor the offset to the current
time of the query. The default offset is `0s` which is the current
default anyway.

This fixes #2074 by making time zone offset support unnecessary. Time
comparisons can use timezones inside of the time clause and the offset
needed for non-hour timezone differences can be used as part of the
offset argument.
2016-05-02 14:02:35 -04:00
Seif Lotfy ced79acac5 Remove redundant error return from (Client) Database(name string) and (Client) Databases() 2016-04-30 17:04:38 -05:00
Todd Persen 1378b8a3c8 Dump TSM files to line protocol. 2016-04-29 23:01:18 -07:00
Jonathan A. Sternberg 043bd6031e Return the time with a selector when there is no group by interval
The time of the point will be returned with a selector when there is no
group by interval and when there is only one selector. Any other
conditions will return the start time of the interval.

Fixes #5890.
2016-04-29 11:27:26 -04:00
Jonathan A. Sternberg 6f61c0ea4a Add POST /query endpoint and warning messages for using GET with write operations
In order to follow REST a bit more carefully, all write operations
should go through a POST in the future. We still allow read operations
through either GET or POST (similar to the Graphite /render endpoint),
but write operations will trigger a returned warning as part of the JSON
response and will eventually return an error.

Also updates the Golang client libraries to always use POST instead of
GET.

Fixes #6290.
2016-04-29 09:00:23 -04:00
Jonathan A. Sternberg 2ccbe26447 Fix the CLI not to enter an infinite loop when the liner has an error
This also removes the dependency on `os/user` and uses the `HOME`
environment variable which is more common on Linux and Mac OS X for
customizing the history file location.

Removing this import also lets the `influx` binary be cross-compiled as
`os/user` relies on cgo.
2016-04-28 08:35:54 -04:00
Ben Johnson f7af787aef
add DELETE query support
This commit adds query language support for deleting series with a
`DELETE` query.
2016-04-27 15:16:23 -06:00
Jason Wilder c8bd41c2d8 Remove TSM reader Keys func
It's very inneficient and should never be used.
2016-04-27 13:09:52 -06:00
Jason Wilder a789e819a3 Remove NewTSMReaderWithOptions
There are two TSMIndex implementations, the directIndex and the
indirectIndex.  Originally, we only had the directIndex and later
added the indirectIndex and NewTSMReaderWithOptions in order to
allow both indexes to be used in tests and code.  This has created
a problem since we really only use the directIndex for writing and
always use the indirectIndex for reading.

This changes removes the NewTSMReaderWithOptions func so that it is
no longer possible to create a TSMReader with a directIndex.  This
will allow a lot of the block reading code used by the directIndex
to be removed and simplify maintainence.  It also gives better test
coverage of the code that is actually used by the TSM engine now.
2016-04-27 13:09:52 -06:00