Commit Graph

2213 Commits (cbef811c63bf76ae0ce844fc1d5e95e39d53a871)

Author SHA1 Message Date
Edd Robinson 550fd4a8c2 Merge pull request #9017 from influxdata/er-8819-retention-index
Ensure retention service removes shards locally
2017-10-26 17:15:43 +01:00
Edd Robinson 50d73497c5 CHANGELOG 2017-10-26 16:22:48 +01:00
Edd Robinson 2ea2abb001 Remove possibility of race when dropping shards
Fixes #8819.

Previously, the process of dropping expired shards according to the
retention policy duration, was managed by two independent goroutines in
the retention policy service. This behaviour was introduced in #2776,
at a time when there were both data and meta nodes in the OSS codebase.
The idea was that only the leader meta node would run the meta data
deletions in the first goroutine, and all other nodes would run the
local deletions in the second goroutine.

InfluxDB no longer operates in that way and so we ended up with two
independent goroutines that were carrying out an action that was really
dependent on each other.

If the second goroutine runs before the first then it may not see the
meta data changes indicating shards should be deleted and it won't
delete any shards locally. Shortly after this the first goroutine will
run and remove the meta data for the shard groups.

This results in a situation where it looks like the shards have gone,
but in fact they remain on disk (and importantly, their series within
the index) until the next time the second goroutine runs. By default
that's 30 minutes.

In the case where the shards to be removed would have removed the last
occurences of some series, then it's possible that if the database was already at its
maximum series limit (or tag limit for that matter), no further new series
can be inserted.
2017-10-26 16:15:13 +01:00
Edd Robinson c8a679421f Update CHANGELOG.md 2017-10-26 15:41:30 +01:00
Andrew Montgomery-Hurrell 2ed0d2d1c9
Make client errors more helpful on downstream errs
When a downstream server such as a proxy or loadbalancer between
influxdb and the client produces an error, the client currently does
not make this very obvious.

This change introduces checks on both the content type and the
influx version header to identify whether a request was served by
influxdb itself and returns a more appropriate error in the cases
where it can be determined a downstream issue is at play.
2017-10-26 11:56:29 +01:00
Edd Robinson 0f4c251148 Update CHANGELOG 2017-10-25 13:34:44 +01:00
Stuart Carnie 63d7310572 update CHANGELOG 2017-10-24 14:17:02 -07:00
Ben Johnson 5a77238f30
Sort & validate TSI key value insertion. 2017-10-23 10:46:01 -06:00
Ben Johnson 5af2d73867
CHANGELOG 2017-10-20 15:14:39 -06:00
Stuart Carnie d7b73f9b02 Update CHANGELOG 2017-10-20 08:58:45 -07:00
Andrew Hare 13c3808aff Add test 2017-10-19 15:57:16 -06:00
Jonathan A. Sternberg 2005a8c8e5 Remove the pidfile after the server has exited 2017-10-19 16:33:59 -05:00
Jonathan A. Sternberg 83ecab6299 Prevent deadlock during collectd, graphite, opentsdb, and udp shutdown
All of these services start up goroutines and then wait for the
goroutines to finish. Each of them has a `tsdb.PointBatcher` that may
return a point during the shutdown sequence. During the shutdown
sequence, a lock was held. This lock may get accessed when attempting to
write the point that came back from the `tsdb.PointBatcher`. This caused
the read lock attempt to wait forever for the write lock to be unlocked
during `Close()`.

This modifies these methods so that the write lock is released while
waiting for goroutines to finish in these three services.
2017-10-19 15:57:05 -05:00
Jason Wilder 05131f4453 Fix indirectIndex not removing fully deleted series
If multiple tombstones exists for a series that ended up causing the
full data to be deleted, the blocks were not removed from the offsets
in the index.  This causes the TSMReader to report that a key exist
but does not have any data.

During a compaction, every key should have at least one value.  Since
this invariant was broken, the compaction aborted early and ends up
dropping all series keys that are lexigraphically greater than where
the breakage occured.  This would cause data to be dropped during the
compaction.
2017-10-18 18:16:41 -06:00
Jason Wilder 4d171f3f40 Fix data deleted outside of time range 2017-10-18 13:39:47 -06:00
Ben Johnson 264971fc5a
CHANGELOG 2017-10-18 07:22:49 -06:00
Andrew Hare e6aa5023eb Create a command to truncated shard groups 2017-10-16 20:34:26 -06:00
Jason Wilder fb7135ddc8 Fix corrupted wal segment panic on 32 bit systems 2017-10-16 09:41:20 -06:00
Jonathan A. Sternberg f20cab6e99 Implicitly decide on the lower limit for fill queries when none is present
This allows the query:

    SELECT mean(value) FROM cpu GROUP BY time(1d)

To function in some way that makes sense. The upper limit is implicitly
the `now()` starting time and the lower limit will be whichever interval
the lowest point falls into.

When no lower bound is specified and `max-select-buckets` is specified,
the query will only consider points that would satisfy
`max-select-buckets`. So if you have one point written in 1970, have
another point within the last minute, and then do the above query with
`max-select-buckets` being equal to 10, the older point from 1970 will
not be considered.
2017-10-05 15:56:44 -05:00
Jason Wilder 7e2b22f8d4 Merge pull request #8886 from influxdata/jw-planner
Improved compaction scheduling
2017-10-04 09:03:59 -06:00
Mark Rushakoff bb6cb80d45 Update changelog 2017-10-03 16:43:26 -07:00
Mark Rushakoff 0aaede8421 Update CHANGELOG 2017-10-03 11:28:16 -07:00
Edd Robinson c80c3636dd Merge pull request #8904 from influxdata/er-changelog
Update 1.3.3 release date
2017-10-03 18:43:56 +01:00
Mark Rushakoff 7525d331d1 Merge branch 'master' into mr-systemd-detection 2017-10-03 09:54:45 -07:00
Jason Wilder a43e652985 Update changelog 2017-10-03 10:49:46 -06:00
Edd Robinson defb81413e Merge branch 'master' into er-changelog 2017-10-03 12:20:49 +01:00
Jonathan A. Sternberg 06994a6585 Revert "Convert duration to string without dot and zeros" 2017-10-02 16:31:03 -05:00
Mark Rushakoff 4d32c157a9 Update changelog 2017-10-02 11:49:16 -07:00
Lyon Hill 38dc837910 Fix a minor memory leak when batching points for some services.
fixes #8895
2017-10-02 11:26:25 -06:00
Jonathan A. Sternberg 1a920c5c42 Merge pull request #8897 from influxdata/js-msgpack-formatter
Add message pack formatter
2017-10-02 11:31:35 -05:00
Jonathan A. Sternberg a0df2fe504 Add message pack formatter 2017-09-29 14:58:38 -05:00
Andrew Hare 4d6672fa66 Merge with master 2017-09-29 11:07:40 -06:00
Jonathan A. Sternberg 9db4439984 Merge pull request #8840 from liketic/bugfix/issues/8813
Convert duration to string without dot and zeros
2017-09-29 11:24:40 -05:00
liketic 986a0e18da Convert duration to string without dot and zeros 2017-09-28 11:56:25 -05:00
Jonathan A. Sternberg 7d1c32fb96 Merge pull request #8763 from influxdata/js-8762-document-subquery-ordering-change
Document that query ordering must be consistent in subqueries
2017-09-28 11:15:22 -05:00
Mark Rushakoff 8c70cfd74b Merge pull request #8808 from liketic/feature/issues/8590
Influx CLI: Make cli case insensitive
2017-09-28 09:13:19 -07:00
Jonathan A. Sternberg 311f1fa619 Merge pull request #8864 from influxdata/js-8848-subquery-math-on-greater-than-two-fields
Prevent deadlock when doing math on the result of a subquery
2017-09-25 11:58:02 -05:00
Jonathan A. Sternberg bcf2e8fca5 Prevent deadlock when doing math on the result of a subquery
The `fill(none)` attribute got set on subqueries, but that can cause an
issue with certain subqueries just like it caused a deadlock on outer
queries.
2017-09-22 14:45:53 -05:00
Edd Robinson 5b7fc517fa Improve performance of TSI bloom filter
This commit replaces the previous hashing algorithm used by the pkg.Filter with
one based on xxhash. Further, taking from the hashing literature, we can
represent k hashes with only two hash function, where previously Filter was using
four.

Further, unlike `murmur3`, `xxhash` is allocation-free, so allocations have
dramatically reduced when inserting and checking for hashes.
2017-09-22 17:59:39 +01:00
Edd Robinson ccb6f7451f Ensure importer emits errors to stderr (#8833) 2017-09-22 17:17:20 +01:00
Edd Robinson 44691847e9 Merge branch 'master' into er-8678-tsi1-where 2017-09-22 16:54:49 +01:00
Stuart Carnie 976d214a5e Merge remote-tracking branch 'origin/master' into sgc-readentries-perf
# Conflicts:
#	CHANGELOG.md
2017-09-19 13:28:23 -07:00
Jonathan A. Sternberg 107feebb81 Report the task status for a query
This more accurately shows whether or not a query has been killed.
Instead of automatically removing it from the query table when it's
killed even though goroutines and iterators may still be open, it now
marks the process as killed. This should allow us to more accurately
determine if a query has been stalled and is still using resources on
the server.

This is related to #8848, but not directly connected.
2017-09-19 14:48:56 -05:00
Stuart Carnie f8688c5588 update CHANGELOG 2017-09-19 11:57:15 -07:00
Stuart Carnie ec852dbafe update CHANGELOG 2017-09-19 10:00:11 -07:00
Joe LeGasse da743321d8 Update CHANGELOG 2017-09-14 14:24:49 -04:00
Jonathan A. Sternberg 6e60edc4bd Merge pull request #8771 from influxdata/js-restore-old-timerange-behavior
This restores the old time range behavior
2017-09-14 12:00:50 -05:00
Jonathan A. Sternberg c8440b4778 This restores the old time range behavior
All time ranges are combined with AND regardless of context and
regardless of whether it makes any logical sense.

That was the previous behavior and, unfortunately, a lot of people rely
on it.
2017-09-14 09:10:37 -05:00
liketic 152aec6fb2 Add CHANGELOG 2017-09-09 11:31:09 +08:00
Jason Wilder 94e229ff59 Merge branch 'master' into jw-drop-series 2017-09-08 15:34:32 -06:00
Jason Wilder 44e1d3f185 Merge pull request #8804 from influxdata/jw-wal-oom
Fix increased memory usage in cache and wal
2017-09-08 15:10:53 -06:00
Jason Wilder 7d05601ba5 Update changelog 2017-09-07 16:52:00 -06:00
Paul Dix 2e3c0a8171 Update CHANGELOG with Prometheus feature 2017-09-07 13:55:12 -04:00
Jason Wilder a3ab827e2e Update changelog 2017-09-07 11:46:32 -06:00
Jonathan A. Sternberg e18425757d Merge pull request #8791 from influxdata/js-explain-cached-values
Include the number of scanned cached values in the iterator cost
2017-09-06 16:00:30 -05:00
Jonathan A. Sternberg 590be193e5 Include the number of scanned cached values in the iterator cost 2017-09-06 15:41:07 -05:00
Stuart Carnie aa6ef36051 update CHANGELOG 2017-09-05 11:22:18 -07:00
Jonathan A. Sternberg 50d404e690 Initial implementation of explain plan
It prints the statistics of each iterator that will access the storage
engine. For each access of the storage engine, it will print the number
of shards that will potentially be accessed, the number of files that
may be accessed, the number of series that will be created, the number
of blocks, and the size of those blocks.
2017-09-01 09:01:10 -05:00
emluque fa681edcb9 6563 Support Ctrl+C to cancel a running query in the Influx CLI
Solved the issue by using context on the http request on the client.
2017-09-01 08:02:27 -05:00
Jonathan A. Sternberg 466fc9026e Reduce how long it takes to walk the varrefs in an expression
This is used quite a bit to determine which fields are needed in a
condition. When the condition gets large, the memory usage begins to
slow it down considerably and it doesn't take care of duplicates.
2017-08-31 09:33:45 -05:00
Ben Johnson 1dbe0662d8
Use system cursors for measurement, series, and tag key meta queries. 2017-08-30 08:35:20 -06:00
David Norton 2aa446bf24 fix #8638: inspect shouldn't err on missing file
influx_inspect walks the data and wal directories building a list of
files to export. It then opens, reads, and exports each. If the file was
deleted between the time it was added to the list and the time the
inspect tool attempts to read it, the file is now skipped without
emitting an error.
2017-08-30 09:59:03 -04:00
Stuart Carnie 9f7f225c10 update CHANGELOG 2017-08-29 16:16:10 -07:00
Jonathan A. Sternberg c41360abae Document that query ordering must be consistent in subqueries 2017-08-29 14:48:37 -05:00
Stuart Carnie 4129c2a606 Merge branch 'master' into sgc-inmem-race 2017-08-28 15:49:10 -07:00
Jonathan A. Sternberg 1c7bafcd3e Force subqueries to match the parent queries ordering
Previously, subqueries would honor their own ordering. We never really
supported that and I have no idea if it would work since most parts in
the query engine assume that points are being delivered in only one
ordering.

Subqueries have now been modified so if a person tries to do different
ordering, they get an error when running the query. If they specify an
ordering in the top most query, that ordering gets propagated to all
subqueries.

Fixes #8699.
2017-08-28 15:57:40 -05:00
Stuart Carnie 7a1a3ebe04 update CHANGELOG 2017-08-28 13:39:18 -07:00
Ben Johnson bd2963dc05 Merge pull request #8735 from influxdata/er-8695-iterators
Ensure that sorted heaps are merged correctly
2017-08-24 08:45:16 -06:00
Edd Robinson 8c4686fb1b Ensure that sorted heaps are merged correctly
When merging streams of system iterators we don't use tags or time.
Instead we add series keys (in the case of, for example, `SHOW SERIES`)
to the `Aux` field of the iterators' elements. This is because we only
emit merged and sorted sets of series key to the client.

We currently use `SortedMergeHeap`s to merge together multiple
iterators, and the comparitor function did not consider `Aux` fields
when determining which heap to pop the next item off during a merge. As
such, `SHOW SERIES` and `SHOW TAG KEYS` (any meta query that gets
converted into a special type of `SELECT`) were returning results in
arbitrary order.

This issue was never noticed on the `inmem` index because the streams
are always duplicates of each other, and of course it doesn't matter if
you arbitrarily merge together two idential, sorted streams...

The issue first manifested itself on the `tsi1` index, but this fix will
apply to both indexes.
2017-08-23 17:21:24 +01:00
Edd Robinson 6e3b8d4684 Merge branch 'master' into er-8668-toml 2017-08-23 17:16:31 +01:00
Edd Robinson 64933fea17 Support pre-existing request ID headers
Other applications or services sometimes expose a header containing a
unique ID, which can then be included in logging or response information
to allow an operator to link inter-service requests. The most common
header name used by services in the wild appears to be `X-Request-ID`,
but `Request-Id` is also used.

This commit adds support for specifying either `X-Request-ID` or
`Request-Id` headers, which will then be used by InfluxDB when logging
request information, and also in the `X-Request-ID` and `Request-Id`
response headers.

We populate both `X-Request-ID` and `Request-Id` to maintain backwards
compatibility with previous version, and to support the more common
`X-Request-ID` header name.

If both `X-Request-ID` and `Request-Id` are specified, then
`X-Request-ID` is used.

If neither header is specified, then in line with previous behaviour, we
generate a v1 UUID.
2017-08-23 13:42:33 +01:00
Edd Robinson 45a6e53d7b Bump toml parser to relicensed MIT version 2017-08-23 11:00:31 +01:00
Edd Robinson 74297a38d6 CHANGELOG 2017-08-23 10:47:01 +01:00
Edd Robinson a472af6418 Update 1.3.3 release date 2017-08-22 17:37:31 +01:00
Jason Wilder e265d150be Fix leaking tmp file when large compaction aborted
If a large compaction was running and was aborted. It could would leave
some tmp files around for files that it had fully written.  The current
active file was cleaned up, but already completed ones would not.  This
would occur when a TSM file needed to rollover due to size.
2017-08-21 17:04:57 -06:00
Stuart Carnie 4233763321 update CHANGELOG 2017-08-17 11:12:48 -07:00
Jason Wilder 9135b21efd Merge pull request #8714 from influxdata/jw-delete-deadlock
Fix deadlock when deleting measurement and writing to it
2017-08-16 17:30:40 -06:00
Jason Wilder a393458f4d Update changelog 2017-08-16 16:45:02 -06:00
Edd Robinson 6b749f459e Merge pull request #8572 from influxdata/er-log-response
Log server error response messages
2017-08-16 23:17:57 +01:00
Jonathan A. Sternberg 8bd04ebe39 Remove TimeRange function and replace with a more accurate ConditionExpr function
The ConditionExpr function is more accurate because it parses the
condition and ensures that time conditions are actually used correctly.
That means that attempting to combine conditions with OR will not result
in the query silently pretending it's an AND and nested conditions work
correctly so there is only one way to read the query.

It also extracts the non-time conditions into a separate condition so we
can stop attempting to parse around the time conditions in lower layers
of the storage engine. This change does not remove those hacks, but a
following commit should be able to sanitize the condition and remove
them.
2017-08-16 16:45:35 -05:00
Edd Robinson a1d5913329 Log server error response messages
This commit provides more insight into server errors by both setting
the error on a response header, and, in the case of server errors (5xx),
logging those error messages to the HTTPD log, if [http] log_enabled =
true.
2017-08-16 21:32:58 +01:00
Jason Wilder ce90eca4a6 Update changelog 2017-08-16 13:13:40 -06:00
Stuart Carnie 3caeee8a24 fix: cursor leak when cur == nil and aux or conds is not empty 2017-08-16 09:17:20 -07:00
Ben Johnson 60ab1282ea
Refactor system iterators.
Previously pseudo iterators could be created for meta data such
as series, measurement, and tag data. These iterators were created
at a higher level and lacked a lot of the power of the query engine.

This commit moves system iterators down to the series level and
supports the following:

	- _name
	- _seriesKey
	- _tagKey
	- _tagValue
	- _fieldKey

These can be used as normal fields such as:

	SELECT _seriesKey FROM cpu

This will return all the series keys for `cpu`.
2017-08-16 09:27:29 -06:00
Ben Johnson c4e2ba25c3 Merge pull request #8669 from benbjohnson/1392-tsi-index-migration
TSI Index Migration Tool
2017-08-16 09:16:03 -06:00
David Norton 1d8d739418 fix #8677: check for snapshot size == 0 2017-08-16 09:43:56 -04:00
Jason Wilder 186e44d227 Merge pull request #8702 from influxdata/jw-monitor-cpu
Reduce CPU usage when checking series cardinality
2017-08-15 16:02:17 -06:00
Jason Wilder b7b11811ce Update changelog 2017-08-15 12:21:24 -06:00
Jason Wilder 21f6ed2c2e Update changelog 2017-08-15 12:03:01 -06:00
Ben Johnson 06bc3b6fbf
TSI Index Migration 2017-08-15 11:40:24 -06:00
Edd Robinson 442581d299 Merge pull request #8592 from influxdata/er-mutex-profile
Adds mutex profile
2017-08-14 22:09:19 +01:00
Edd Robinson 81c06b58fa Merge pull request #8462 from xginn8/backup_stdout
backup output should go to stdout not stderr
2017-08-14 22:08:25 +01:00
Edd Robinson 45969ef3c6 Allow tag filtering when using DELETE with tsi1 2017-08-14 19:09:36 +01:00
Stuart Carnie bf30282e53 update CHANGELOG 2017-08-09 14:53:14 -07:00
Edd Robinson 8934aacebb Adds mutex profile
Mutex profiles are now available via the:

  - /debug/pprof/mutex
  - /debug/pprof/all

endpoints.
2017-08-04 14:36:39 +01:00
Edd Robinson 7c67718d60 Update CHANGELOG 2017-08-02 18:19:04 +01:00
Stuart Carnie 6311a19ad4 update CHANGELOG 2017-08-01 10:20:05 -07:00
Jonathan A. Sternberg fcc5184a81 Merge pull request #8642 from influxdata/js-8639-parse-time-literals-with-location
Parse time literals using the time zone in the select statement
2017-07-28 12:35:50 -05:00
Ryan Betts df88fa3736 Update CHANGELOG with 1.3 release dates. 2017-07-27 16:23:22 -07:00
Jason Wilder 85642e2831 Merge pull request #8630 from influxdata/jw-drop-oom
Prevent excessive memory usage when dropping series
2017-07-27 16:24:52 -06:00
Jason Wilder 6e6cc991ee Merge pull request #8629 from influxdata/jw-compaction-abort
Interrupt in progress TSM compactions
2017-07-27 16:12:40 -06:00
Jason Wilder c75ac3076f Limit delete to run one shard at a time
There was a change to speed up deleting and dropping measurements
that executed the deletes in parallel for all shards at once. #7015

When TSI was merged in #7618, the series keys passed into Shard.DeleteMeasurement
were removed and were expanded lower down.  This causes memory to blow up
when a delete across many shards occurs as we now expand the set of series
keys N times instead of just once as before.

While running the deletes in parallel would be ideal, there have been a number
of optimizations in the delete path that make running deletes serially pretty
good.  This change just limits the concurrency of the deletes which keeps memory
more stable.
2017-07-27 16:01:47 -06:00
Jason Wilder 18a02d50d7 Interrupt in progress TSM compactions
When snapshots and compactions are disabled, the check to see if
the compaction should be aborted occurs in between writing to the
next TSM file.  If a large compaction is running, it might take
a while for the file to be finished writing causing long delays.

This now interrupts compactions while iterating over the blocks to
write which allows them to abort immediately.
2017-07-27 15:58:56 -06:00
Jonathan A. Sternberg 950753d036 Parse time literals using the time zone in the select statement 2017-07-27 13:05:51 -05:00
Jonathan A. Sternberg 343cd2e120 Fix time zone shifts when the shift happens on a time zone boundary
When a time zone shift happened at the edge of a time bucket, the logic
was incorrect. When we added an hour to the day with a time grouping of
2 hours, we took the hour away from the previous bucket instead of the
next bucket so the first bucket of the day would be from midnight to 1
AM and the second bucket would include 1 AM to 4 AM (2 AM to 3 AM does
not exist when shifting forwards). The correct buckets should have been
12 AM to 2 AM for the first bucket of the day and 3 AM to 4 AM for the
second (since 2 AM to 3 AM does not exist).

This also modifies the tests to use table tests and sub tests.
2017-07-25 23:36:03 -05:00
Connor Peet 4459e1e8d8 Return errors in CSV responses 2017-07-25 09:44:02 -05:00
Jonathan A. Sternberg 8bb80321d9 Merge pull request #8561 from influxdata/js-8525-query-http-pipelining
Support http pipelining for /query endpoint
2017-07-21 11:15:28 -05:00
Jonathan A. Sternberg 506c06241b Fixed time boundaries for continuous queries with time zones
This calculates the start and end time along with any time zones shifts
so that continuous queries are run at the correct time when a time zone
is included in the query.
2017-07-20 14:30:52 -05:00
Jonathan A. Sternberg 39591e5375 Support http pipelining for /query endpoint
Removing the forced `Connection: close` header from the `/query`
endpoint. This was originally added because of golang/go#13165, but it
seems like it's possible to use pipelining with go 1.8 and http 1.1,
just not recommended.

After some testing, it appears that the channel returned by
`ResponseWriter.CloseNotify()` will not send a value if the connection
was not interrupted. We already account for this in /query by exiting
from the goroutine if the request has finished by signaling another
channel.

Since the handler already accounts for the possibility that the channel
will not signal and since `CloseNotify()` doesn't interfere with a
pipelined request, we can remove the forced `Connection: close` that was
added to force clients to establish a new connection.
2017-07-20 09:38:01 -05:00
Jonathan A. Sternberg f7d07910aa Fix the cq start and end times to use unix timestamps
The Go timestamp leads Truncate to start a week on Monday, but the query
engine uses unix timestamps which has the week start on a Thursday.

Updating the service so it uses a custom truncate method that uses the
unix timestamp instead of `time.Time`.

Fixes #8569.
2017-07-17 14:59:45 -05:00
xginn8 faec9625bc Merge branch 'master' into backup_stdout 2017-07-13 18:10:54 -04:00
Jason Wilder 4244d0e053 Merge pull request #8568 from influxdata/jw-tombstone-compress
Compress tombstone files
2017-07-10 11:28:09 -06:00
Jason Wilder c25f7b8b3f Fix duplicate points returned after delete
The sortedSeriesIds slice was not getting reset to 0 which caused
the same series ids to exist in the slice more than once.  Since
the size of the slice never matched the size of the seriesID map,
it kept appendending to the slice and sorting it which cause multiple
cursor to get created for the same series.

Fixes #8531
2017-07-10 10:37:01 -06:00
Stuart Carnie 649beba8b3 update CHANGELOG 2017-07-08 08:46:27 -07:00
Adam 7ac4f5b8c2 Merge pull request #8574 from influxdata/plutonium#1240_master
Add X-Influxdb-Build  value to HTTP response header
2017-07-07 12:04:58 -04:00
Edd Robinson d2ce7060c5 Merge branch 'master' into backup_stdout 2017-07-07 16:27:39 +01:00
Edd Robinson 0d7059af04 Update CHANGELOG 2017-07-07 16:23:38 +01:00
Edd Robinson a43238618e Merge pull request #8512 from axiomhq/loglogbeta
Switch to LogLog-Beta Cardinality estimation
2017-07-07 16:14:16 +01:00
Adam 460b30bd08 removed blank line from changelog 2017-07-07 11:06:01 -04:00
Adam 2259ada8c3 adds a new header key/value X-Influxdb-Build that has value OSS if called from open-source build, and ENT if called from enterprise. This commit sets the value for the OSS case, and also creates the proper flag 2017-07-07 10:59:26 -04:00
Jason Wilder 3e7dfad7c4 Compress tombstone files
This adds a v3 format that is a gzip compressed version of the v2
format.  It reduces the size of tombstone files substantially without
having to support a more feature rich file format for tombstones.
2017-07-06 10:10:31 -06:00
Edd Robinson 7374e4e8a4 Merge pull request #8550 from influxdata/er-8548-panic
Allow panic recovery mechanism to be disabled
2017-07-05 22:09:09 +01:00
Edd Robinson 101af89987 Update CHANGELOG 2017-07-05 16:35:41 +01:00
Edd Robinson 12248b7233 Allow panic recovery to be disabled
This commit adds a new environment variable INFLUXDB_PANIC_CRASH, which
when set to a truthy value, e.g., true, TRUE, 1, will prevent the server
from recovering from a panic.

Recover currently occurs in two places: the HTTP handler and the
QueryExecutor. INFLUXDB_PANIC_CRASH will control both.

Further, this commit adds _internal stats that will monitor the
occurrence of panics all the time (regardless of if INFLUXDB_PANIC_CRASH
has been set to true or not).

The recovered panic frequency can be inspected with the following
queries:

SELECT "recoveredPanics" FROM "_internal"."monitor"."httpd";
SELECT "recoveredPanics" FROM "_internal"."monitor"."queryExecutor";
2017-06-29 19:44:25 +01:00
Ben Johnson 9e64813db8
Defer unlock all write locks in inmem index.
Currently two write locks in `inmem` are obtained and then
manually unlocked at function exit points. However, we have
reports that the `inmem` index is hanging on a write lock and
cannot track the issue down to anything else besides a lock
that could have been left unlocked because of a panic.

This commit changes the two locks to always defer their unlocks
to prevent these hangs.
2017-06-29 10:23:13 -06:00
Edd Robinson e1a5ee4ede Ensure privileges can't be set on non-existent DB 2017-06-26 14:15:52 +01:00
Adam b12ee4e6ee 8426: updated code based on pull-request feedback 2017-06-23 14:59:01 -04:00
Adam 6ac3e0ab98 8426 Updated changelog to more clearly describe the new feature 2017-06-23 09:22:14 -04:00
Adam 0fb2de16ee 8426 updated changelog 2017-06-22 18:56:38 -04:00
Jason Wilder 12c7063566 Update changelog 2017-06-21 09:20:39 -06:00
Seif Lotfy 643b2eb30c Switch to LogLog-Beta Cardinality estimation
The new algorithm uses only one formula and needs no additional bias corrections for the entire range of cardinalities,
therefore, it is more efficient and simpler to implement. Our simulations show that the accuracy provided by the new
algorithm is as good as or better than the accuracy provided by either of HyperLogLog or HyperLogLog++. The sparse
representation was kept in to provide better low cardinality accuracy. However the linear counting and range estimations
are replaced.
2017-06-20 15:25:01 +02:00
Stuart Carnie 932edd90b2 Merge branch 'master' into sgc-8188 2017-06-14 10:55:06 +10:00
Stuart Carnie 3657dbc256 update CHANGELOG with key names 2017-06-14 10:37:04 +10:00
Jonathan A. Sternberg e2c1da05e4 Merge pull request #8480 from influxdata/js-stats-interval
Change the default stats interval to 1 second instead of 10 seconds
2017-06-13 12:08:23 -05:00
Ben Johnson b51f604030
Fix TSI non-contiguous compaction panic.
This fixes the case where log files are compacted out of order
and cause non-contiguous sets of index files to be compacted.

Previously, the compaction planner would fetch a list of index files
for each level and compact them in order starting with the oldest
ones. This can be a problem for level 1 because level 0 (log files)
are compacted individually and in some cases a log file can finish
compacting before older log files are finished compacting. This
causes there to be a gap in the list of level 1 files that is
ignored when fetching a list of index files.

Now, the planner reads the list of index files starting from the
oldest but stops once it hits a log file. This prevents that gap
from being ignored.
2017-06-13 10:53:26 -06:00
Jonathan A. Sternberg f7382982fd Change the default stats interval to 1 second instead of 10 seconds 2017-06-12 13:15:25 -05:00
marchtea 16dfe2a0ae update CHANGELOG.md 2017-06-12 11:00:27 +08:00
Stuart Carnie 2de52834f0 CQ statistics written to monitor database, addresses #8188
* off by default, enabled by `query-stats-enabled`
* writes to cq_query measurement of configured monitor database
* see CHANGELOG for schema of individual points
2017-06-10 09:20:38 +08:00
Ben Johnson bcc6ef769b
Check file count before attempting a TSI level compaction.
This check was previously in a different section of code which
was lost during a refactor to the new compaction strategy. The
compaction planning now makes a check to ensure at least two
files are available for compaction in a level.
2017-06-06 11:08:59 -06:00
Stuart Carnie 98f2050bcb Update config sample and CHANGELOG 2017-06-05 22:05:00 +08:00
Ben Johnson 3128c6a42e
Fix SHOW TAG VALUES deduplication. 2017-06-01 15:38:35 -06:00
Jonathan A. Sternberg 6a78f1cf4a URL query parameter credentials take priority over Authentication header 2017-05-30 09:26:24 -05:00
Jason Wilder f1181cc402 Update changelog 2017-05-24 14:47:01 -06:00
Jonathan A. Sternberg 9edf236cc8 Maintain the tags of points selected by top() or bottom() when writing the results
When a `SELECT ... INTO ...` is used with `top()` or `bottom()` used
with tags, the points will be written with the tags still intact instead
of converted to fields.
2017-05-23 15:00:21 -05:00
Ryan Betts b18a7e8deb Merge pull request #8396 from influxdata/changelog-merges
Add 1.2.4 and 1.1.5 CHANGELOG updates.
2017-05-23 14:31:09 -04:00
Jason Wilder 31d2309177 Update changelog 2017-05-22 14:53:06 -06:00
Jonathan A. Sternberg 4bdce21a9a Merge pull request #8394 from influxdata/js-top-bottom-performance
Optimize top() and bottom() using an incremental aggregator
2017-05-19 14:32:55 -05:00
Jason Wilder 55f2f83e34 Merge pull request #8407 from influxdata/jw-8392
Return partial write error when points outside of retention policy ar…
2017-05-19 11:25:08 -06:00
Jonathan A. Sternberg 7b9b55bfc0 Optimize top() and bottom() using an incremental aggregator
The previous version of `top()` and `bottom()` would gather all of the
points to use in a slice, filter them (if necessary), then use a
slightly modified heap sort to retrieve the top or bottom values.

This performed horrendously from the standpoint of memory. Since it
consumed so much memory and spent so much time in allocations (along
with sorting a potentially very large slice), this affected speed too.

These calls have now been modified so they keep the top or bottom points
in a min or max heap. For `top()`, a new point will read the minimum
value from the heap. If the new point is greater than the minimum point,
it will replace the minimum point and fix the heap with the new value.
If the new point is smaller, it discards that point. For `bottom()`, the
process is the opposite.

It will then sort the final result to ensure the correct ordering of the
selected points.

When `top()` or `bottom()` contain a tag to select, they have now been
modified so this query:

    SELECT top(value, host, 2) FROM cpu

Essentially becomes this query:

    SELECT top(value, 2), host FROM (
        SELECT max(value) FROM cpu GROUP BY host
    )

This should drastically increase the performance of all `top()` and
`bottom()` queries.
2017-05-19 11:56:46 -05:00
Jason Wilder afb1027bed Return partial write error when points outside of retention policy are dropped
Writing points outside of a retention policy range were silently dropped. They
are dropped to prevent creating a shard that will be immediately deleted.  These
dropped points were silent and did not return an error respone to the caller.

Fixes #8392
2017-05-19 10:50:03 -06:00
Jonathan A. Sternberg 7d043dbc61 Add nanosecond duration literal support 2017-05-19 10:44:11 -05:00
Edd Robinson a5fed3d296 Merge pull request #7862 from influxdata/er-debug-all
Adds handler for returning a profile archive
2017-05-17 17:09:39 +01:00
Ryan Betts a43856adc6 Add 1.2.4 and 1.1.5 CHANGELOG updates. 2017-05-16 16:51:29 -04:00
Edd Robinson 1cbbaa9317 Add support for shards, stats and diagnostics 2017-05-15 14:12:00 +01:00
Edd Robinson 8f8ff0ec61 Adds handler for returning a profile archive
Currently, when debugging issues with InfluxDB we often ask for the
following profiles:

  curl -o block.txt "http://localhost:8086/debug/pprof/block?debug=1"
  curl -o goroutine.txt
"http://localhost:8086/debug/pprof/goroutine?debug=1"
  curl -o heap.txt "http://localhost:8086/debug/pprof/heap?debug=1"
  curl -o cpu.txt "http://localhost:8086/debug/pprof/profile

This can be bothersome for users, or even difficult if they're
unfamiliar with cURL (or it's not on their system).

This commit adds a new endpoint: /debug/pprof/all which will return a
single compressed archive of all of the above profiles. The CPU profile
is optional, and not returned by default. To include a CPU profile the
URL to request should be: /debug/pprof/all?cpu=true. It's also possible
to vary the length of the CPU profile by adding a `seconds=x` parameter,
where x defaults to 30, if absent.

The new command for gathering profiles from users should now be:

  curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all"

Or, if we need to see a CPU profile:

  curl -o profiles.tar.gz
"http://localhost:8086/debug/pprof/all?cpu=true"

It's important to remember that a CPU profile is a blocking operation
and by default it will take 30 seconds for the response to be returned
to the user.

Finally, if the user is unfamiliar with cURL, they will now be able to
visit http://localhost:8086/debug/pprof/all in a web browser, and the
archive will be downloaded to their machine.
2017-05-15 14:11:38 +01:00
Mark Rushakoff 6f438ea467 Update CHANGELOG 2017-05-12 17:09:09 -07:00
Jason Wilder 0b7c0b680c Update changelog 2017-05-12 14:05:24 -06:00
Jonathan A. Sternberg dea02009e0 Small edits to the etc/config.sample.toml file 2017-05-10 10:56:34 -05:00
Jonathan A. Sternberg 2780630a5f Track HTTP client requests for /write and /query with /debug/requests
After using `/debug/requests`, the client will wait for 30 seconds
(configurable by specifying `seconds=` in the query parameters) and the
HTTP handler will track every incoming query and write to the system.
After that time period has passed, it will output a JSON blob that looks
very similar to `/debug/vars` that shows every IP address and user
account (if authentication is used) that connected to the host during
that time.

In the future, we can add more metrics to track. This is an initial
start to aid with debugging machines that connect too often by looking
at a sample of time (like `/debug/pprof`).
2017-05-09 10:18:33 -05:00
Jason Wilder 29c2b1958e Fix deletes triggering unnecessary compactions
Tombstone files would be written to all TSM files even if the deleted
keys or timerange did not exist in the TSM file.  This had the side
effect of causing shards to get recompacted back to the same state. If
any shards or large numbers of TSM files existed, disk usage and CPU
utilization would spike causing issues.

This prevents tombstones being written for TSM files that could not
possiby contain the series keys being deleted or if the delted time
range is outside the range of the file.
2017-05-08 14:52:28 -06:00
Ben Johnson 489c89bea4
Add tsi support tooling. 2017-05-08 11:00:15 -06:00
Jonathan A. Sternberg 260bdef3d4 Set the CSV output to an empty string for null values 2017-05-04 20:51:58 -05:00
Jason Wilder 684f5d884a Update changelog 2017-05-03 16:31:57 -06:00
Jonathan A. Sternberg df30a4d9c9 Refactor the subquery code and fix outer condition queries
This change refactors the subquery code into a separate builder class to
help allow for more reuse and make the functions smaller and easier to
read.

The previous function that handled most of the code was too big and
impossible to reason through.

This also goes and replaces the complicated logic of aggregates that had
a subquery source with the simpler IteratorMapper. I think the overhead
from the IteratorMapper will be more, but I also believe that the actual
code is simpler and more robust to produce more accurate answers. It
might be a future project to optimize that section of code, but I don't
have any actual numbers for the efficiency of one method and I believe
accuracy and code clarity may be more important at the moment since I am
otherwise incapable of reading my own code.
2017-04-28 17:12:32 -05:00
Jonathan A. Sternberg addc12561f Fix LIMIT and OFFSET for certain aggregate queries
When LIMIT and OFFSET were used with any functions that were not handled
directly by the query engine (anything other than count, max, min, mean,
first, or last), the input to the function would be limited instead of
receiving the full stream of values it was supposed to receive.

This also fixes a bug that caused the server to hang when LIMIT and
OFFSET were used with a selector. When using a selector, the limit and
offset should be handled before the points go to the auxiliary iterator
to be split into different iterators. Limiting happened afterwards which
caused the auxiliary iterator to hang forever.
2017-04-28 15:55:06 -05:00
Ben Johnson 3a46e5dd9e
Remove default upper time bound for DELETE queries. 2017-04-28 12:26:26 -06:00
Jason Wilder a736f186f0 Merge pull request #8327 from influxdata/jw-go181
Update to go 1.8.1
2017-04-27 08:42:30 -06:00
Jonathan A. Sternberg be3bce5212 top() and bottom() now returns the time for every point
`top()` and `bottom()` will now organize the points by time and also
keep the points original time even when a time grouping is used. At the
same time, `top()` and `bottom()` will no longer honor any fill options
that are present since they don't really make sense for these specific
functions.

This also fixes the aggregate and selectors to honor the ordered
iterator option so iterator remain ordered and to also respect the
buckets that are created by the final dimensions of the query so that
two buckets don't overlap each other within the same reducer. A test has
been added for this situation. This should clarify and encourage the use
of the ordered attribute within the query engine.
2017-04-26 15:07:10 -05:00
Jonathan A. Sternberg 4776b216a4 Merge pull request #8253 from influxdata/js-8065-restrict-top-bottom-query
Restrict top() and bottom() selectors to be used with no other functions
2017-04-26 15:06:30 -05:00
Jason Wilder 4db3b69b9d Update to go1.8.1 2017-04-26 11:32:42 -06:00
Jonathan A. Sternberg 1300f4cc6c Remove the admin UI 2017-04-25 16:58:24 -05:00
Jason Wilder 71825d20c8 Update changelog 2017-04-20 12:31:06 -06:00
Jason Wilder 5c51ae7319 Merge branch '1.2' into jw-merge-123 2017-04-14 14:36:54 -06:00
Cory LaNou 8c0f5a7dbe
redact passwords before saving history in cli 2017-04-14 13:13:56 -05:00
Jonathan A. Sternberg 57a2abbc87 Restrict top() and bottom() selectors to be used with no other functions 2017-04-14 10:23:07 -05:00
Cory LaNou 775c5d243d Add changelog for 8187 2017-04-13 13:33:25 -05:00
Cory LaNou f96b59ed20 Add changelog for 8187 2017-04-13 10:31:31 -05:00
Jonathan A. Sternberg a550d323c4 Restrict fill(none) and fill(linear) to be usable only with aggregate queries 2017-04-10 15:58:05 -05:00
Jonathan A. Sternberg 0a5e4bd92b Implicitly cast null to false in binary expressions with a boolean
Also more consistently treat a binary expression with strings so it
produces the same value no matter the direction of the expression.
2017-04-06 12:26:04 -05:00
Jonathan A. Sternberg 45895862b7 Merge pull request #8058 from karlding/service-golinting
Make services/{admin, httpd, subscriber, udp} golintable
2017-04-05 12:30:11 -05:00
Jason Wilder 91bfc5772a Update changelog 2017-04-04 16:39:53 -06:00
Jason Wilder 5fa8073fc2 Merge branch '1.2' into jw-merge-123 2017-04-04 11:12:06 -06:00
Jonathan A. Sternberg b1caafe82f Ensure the input for certain functions in the query engine are ordered
The following functions require ordered input but were not guaranteed to
received ordered input:

* `distinct()`
* `sample()`
* `holt_winters()`
* `holt_winters_with_fit()`
* `derivative()`
* `non_negative_derivative()`
* `difference()`
* `moving_average()`
* `elapsed()`
* `cumulative_sum()`
* `top()`
* `bottom()`

These function calls have now been modified to request that their input
be ordered by the query engine. This will prevent the improper output
that could have been caused by multiple series being merged together or
multiple shards being merged together potentially incorrectly when no
time grouping was specified.

Two additional functions were already correct to begin with (so there
are no bugs with these two, but I'm including their names for
completeness).

* `median()`
* `percentile()`
2017-04-04 09:20:43 -05:00
Jonathan A. Sternberg 3c0d1c1bb5 Fix a regression when math was used with selectors
If there were multiple selectors and math, the query engine would
mistakenly think it was the only selector in the query and would not
match their timestamps.

Fixed the query engine to pass whether the selector should be treated as
a selector so queries like `max(value) * 1, min(value) * 1` will match
the timestamps of the result.
2017-04-04 09:20:43 -05:00
Ryan Betts 4b1977673c Merge branch 'master' into timhallinflux-patch-2-1 2017-04-04 09:57:19 -04:00
Cory LaNou bc0759d0fc redact passwords before saving history in cli 2017-04-03 11:20:14 -05:00
Cory LaNou 7a6243eb58 Merge pull request #8122 from influxdata/cjl-influx-suppress-headers
suppress headers in output for influx when they are the same
2017-04-03 10:07:48 -05:00
Edd Robinson 57b8993e7b Reduce cost of admin user check
This commits adds a caching mechanism to the Data object, such that
when large numbers of users exist in the system, the cost of determining
if there is at least one admin user will be low.

To ensure that previously marshalled Data objects contain the correct
cached admin user value, we exhaustively determine if there is an admin
user present whenever we unmarshal a Data object.
2017-04-03 12:06:44 +01:00
Karl b22783f127 Update CHANGELOG 2017-04-02 18:33:09 -04:00
Tom Young d2fd3f50aa Add bitwise AND, OR and XOR operators to InfluxQL. 2017-03-31 21:02:02 +01:00
Jonathan A. Sternberg 211e7ea65d Merge pull request #8234 from influxdata/js-8230-fix-window-computation-overflow
Prevent overflowing or underflowing during window computation
2017-03-31 11:09:30 -05:00
zhexuany 232fdae6dd introduce a new function non_negative_difference 2017-03-31 23:08:36 +08:00
Jonathan A. Sternberg 64fb1db5f5 Prevent overflowing or underflowing during window computation
The Window function will now check before it adjusts the offset whether
it is going to overflow or underflow. If it is going to do either, it
sets the start or end time to MinTime or MaxTime.
2017-03-30 16:35:22 -05:00
timhallinflux e50ad6615d Update CHANGELOG.md 2017-03-30 12:46:53 -07:00
timhallinflux 9f88e63ccf Changelog addition for #8231 2017-03-30 12:44:59 -07:00
Jonathan A. Sternberg 2ea805c928 Interpolate between different intervals to find the whole area under the curve 2017-03-30 12:51:52 -05:00
Edd Robinson 7644ab1fc4 Fix race in test helpers. Fixes #8177 2017-03-29 12:31:04 +01:00
Edd Robinson 45f843fc91 Don't unassign shards when system shutting down 2017-03-29 11:57:38 +01:00
Jonathan A. Sternberg 7e0ed1f5e5 Ensure the input for certain functions in the query engine are ordered
The following functions require ordered input but were not guaranteed to
received ordered input:

* `distinct()`
* `sample()`
* `holt_winters()`
* `holt_winters_with_fit()`
* `derivative()`
* `non_negative_derivative()`
* `difference()`
* `moving_average()`
* `elapsed()`
* `cumulative_sum()`
* `top()`
* `bottom()`

These function calls have now been modified to request that their input
be ordered by the query engine. This will prevent the improper output
that could have been caused by multiple series being merged together or
multiple shards being merged together potentially incorrectly when no
time grouping was specified.

Two additional functions were already correct to begin with (so there
are no bugs with these two, but I'm including their names for
completeness).

* `median()`
* `percentile()`
2017-03-28 13:55:37 -05:00
Jonathan A. Sternberg 24109468c3 Merge pull request #8168 from influxdata/js-8167-math-with-multiple-selectors
Fix a regression when math was used with selectors
2017-03-28 13:31:57 -05:00
Cory LaNou 9f674ccec4
suppress headers in output for influx when they are the same 2017-03-28 12:50:23 -05:00
Jonathan A. Sternberg 3e52ec7ca2 Merge pull request #7762 from influxdata/js-6541-timezone-support
Support timezone offsets for queries
2017-03-28 10:39:07 -05:00
Cory LaNou b4d61d4ef4 Merge pull request #8119 from influxdata/cjl-influx-chunked
add chunked/chunk size as setting/options in cli
2017-03-28 10:38:56 -05:00
Jonathan A. Sternberg b14c292cba Fix a regression when math was used with selectors
If there were multiple selectors and math, the query engine would
mistakenly think it was the only selector in the query and would not
match their timestamps.

Fixed the query engine to pass whether the selector should be treated as
a selector so queries like `max(value) * 1, min(value) * 1` will match
the timestamps of the result.
2017-03-27 14:12:15 -05:00
Jonathan A. Sternberg ccf0cb8371 Fix query parser when using addition and subtraction without spaces
Additionally, support unary addition and subtraction for variables,
calls, and parenthesis expressions. Doing `-value` will be the
equivalent of doing `-1 * value` now.
2017-03-24 12:52:19 -05:00
Jason Wilder 2972a3f223 Remove MMAP derefencing code
This code was added to address some slow startup issues.  It is believed
to be the cause of some segfault panic's that occur at query time when
the underlying MMAP array has been unmapped.  The current structure of
code makes this change unnecessary now.
2017-03-23 12:46:23 -06:00
Jonathan A. Sternberg 347b01814e Support timezone offsets for queries
The timezone for a query can now be added to the end with something like
`TZ("America/Los_Angeles")` and it will localize the results of the
query to be in that timezone. The offset will automatically be set to
the offset for that timezone and offsets will automatically adjust for
daylight savings time so grouping by a day will result in a 25 hour day
once a year and a 23 hour day another day of the year.

The automatic adjustment of intervals for timezone offsets changing will
only happen if the group by period is greater than the timezone offset
would be. That means grouping by an hour or less will not be affected by
daylight savings time, but a 2 hour or 1 day interval will be.

The default timezone is UTC and existing queries are unaffected by this
change.

When times are returned as strings (when `epoch=1` is not used), the
results will be returned using the requested timezone format in RFC3339
format.
2017-03-22 15:09:41 -05:00
Jonathan A. Sternberg 33981277bc Fix the time range when an exact timestamp is selected
There is a lot of confusion in the code if the range is [start, end) or
[start, end]. This is not made easier because it is acts one way in some
areas and in another way in some other areas, but it is usually [start,
end]. The `time = ?` syntax assumed that it was [start, end) and added
an extra nanosecond to the end time to accomodate for that, but the
range was actually [start, end] and that caused it to include one extra
nanosecond when it shouldn't have.

This change fixes it so exactly one timestamp is selected when `time = ?`
is used.
2017-03-21 14:55:31 -05:00
Jonathan A. Sternberg a6c09e58a0 Return an error when an invalid duration literal is parsed 2017-03-21 12:10:41 -05:00
Edd Robinson f89de550ed Significantly speed up DROP DATABASE 2017-03-21 11:35:31 +00:00
Edd Robinson 255992f5ec Reduce cost of admin user check
This commits adds a caching mechanism to the Data object, such that
when large numbers of users exist in the system, the cost of determining
if there is at least one admin user will be low.

To ensure that previously marshalled Data objects contain the correct
cached admin user value, we exhaustively determine if there is an admin
user present whenever we unmarshal a Data object.
2017-03-20 12:04:03 +00:00
Cory LaNou e07d84525d
add chunked/chunksize as setting/options 2017-03-17 18:25:52 -05:00
Jason Wilder 00306336ee Update changelog 2017-03-17 16:13:36 -06:00
Jonathan A. Sternberg 5fba1bdcd3 Update liner dependency to handle docker exec
The liner dependency now handles the scenario where the terminal width
is reported as zero. Previously, liner would panic when it tried to
divide by the width (which was zero). Now it falls back onto a dumb
prompt rather than attempting to use a smart prompt and panicking.
2017-03-17 08:42:53 -05:00
Jonathan A. Sternberg 41c8370bbc Fix fill(linear) when multiple series exist and there are null values
When there were multiple series and anything other than the last series
had any null values, the series would start using the first point from
the next series to interpolate points.

Interpolation should not cross between series. Now, the linear fill
checks to make sure the next point is within the same series before
using it to perform interpolation.
2017-03-16 15:54:20 -05:00
Jonathan A. Sternberg 5072db40c2 Forbid wildcards in binary expressions
When rewriting fields, wildcards within binary expressions were skipped.
This now throws an error whenever it finds a wildcard within a binary
expression in order to prevent the panic that occurs.
2017-03-16 14:26:10 -05:00
Jonathan A. Sternberg 9cdfdd04e9 Do not increment the continuous query statistic if no query is run
Instead of incrementing the `queryOk` statistic with or without the
continuous query running, it will only increment when the query is
actually executed.
2017-03-16 10:36:00 -05:00
Jason Wilder e9eb925170 Coalesce multiple WAL fsyncs
Fsyncs to the WAL can cause higher IO with lots of small writes or
slower disks.  This reworks the previous wal fsyncing to remove the
extra goroutine and remove the hard-coded 100ms delay.  Writes to
the wal still maintain the invariant that they do not return to the
caller until the write is fsync'd.

This also adds a new config options wal-fsync-delay (default 0s)
which can be increased if a delay is desired.  This is somewhat useful
for system with slower disks, but the current default works well as
is.
2017-03-15 16:31:03 -06:00
Jonathan A. Sternberg 208d8507f1 Implement both single and multiline comments in influxql
A single line comment will read until the end of a line and is started
with `--` (just like SQL). A multiline comment is with `/* */`. You
cannot nest multiline comments.
2017-03-15 14:24:09 -05:00
Jason Wilder 713a1d2fab Merge pull request #8137 from influxdata/jw-merge-12
Merge 1.2.2 to master
2017-03-14 17:51:34 -06:00
Jason Wilder a16d86ebaa Merge pull request #8136 from influxdata/sort-changelog
Place CHANGELOG.md in descending sort order.
2017-03-14 15:27:46 -06:00
Jason Wilder e62c72d1f9 Merge branch '1.2' into jw-merge-12 2017-03-14 15:15:50 -06:00
Ryan Betts 46fa0c33a0 Place CHANGELOG.md in descending sort order. 2017-03-14 16:38:06 -04:00
Jason Wilder c9740f753b Disable max-row-limit by default
max-row-limit was set at 10000 since 1.0, but due to a bug it was
effectively 0 (disabled).  1.2 fixed this bug via #7368, but this
caused a breaking change w/ Grafana and any users upgrading from <1.2
who had not disabled the config manually.
2017-03-14 12:47:32 -06:00
Mark Rushakoff 32a961005d Update CHANGELOG 2017-03-14 11:34:49 -07:00
Edd Robinson bdc10a0e51 Failed imports will now alter exit code 2017-03-14 15:57:21 +00:00
Jason Wilder b9e5375043 Merge branch '1.2' into jw-merge-12 2017-03-08 13:16:50 -07:00
Jason Wilder 3ec60fe264 Update v1.2.1 release date 2017-03-08 12:26:07 -07:00
Jonathan A. Sternberg 83cf8893e1 Include IsRawQuery in the rewritten statement for meta queries 2017-03-06 14:46:33 -06:00
Jason Wilder 675d7c9d65 Merge branch '1.2' into jw-merge12 2017-03-06 11:09:05 -07:00
Jason Wilder eab012ef61 Fix points missing after compaction
If blocks containing overlapping ranges of time where partially
recombined, it was possible for the some points to get dropped
during compactions.  This occurred because the window of time of
the points we need to merge did not account for the partial blocks
created from a prior merge.

Fixes #8084
2017-03-06 10:17:11 -07:00
Jason Wilder 29f8d8de76 Fix race in WALEntry.Encode and Value.Deduplicate
Under high query load, a race exists in the cache and the WAL.  Since
writes currently hit the cache first, they are availble for query before
they hit the WAL.  If the WAL is writing and accessign the Value slice
at the same time that a query is run that needs to dedup the same slice,
a race occurs.

To fix this, the cache now just copies the values instead of storing the
slice passed in.  Another way to fix this might be to have the writes go
to the wal before the cache.  I think the latter would be better, but it
introduces some larger write path issues that we'd need to also address.
e.g. if the cache was full, writes to the WAL would need to be rejected
to avoid filling the disk.

Copying the slice in the cache is simpler for now and does not appear to
dramatically affect performance.
2017-03-06 09:38:22 -07:00
Ben Johnson 4c202eea09
Re-check field type under write lock. 2017-03-03 09:47:43 -07:00
Ben Johnson dffd12319c
Add point.UnmarshalBinary() bounds checking. 2017-03-01 12:01:25 -07:00
Jonathan A. Sternberg c5970b59b4 Map types correctly when selecting a field with multiple measurements where one of the measurements is empty 2017-03-01 11:47:26 -06:00
Jonathan A. Sternberg b942f3a373 Merge pull request #8069 from influxdata/js-8044-measurement-with-underscore-prefix
Treat non-reserved measurement names with underscores as normal measurements
2017-02-28 10:21:26 -06:00
Jason Wilder 414fe1349e Merge 1.1.4 changelog 2017-02-27 16:51:07 -07:00
Jonathan A. Sternberg 1081785cb4 Treat non-reserved measurement names with underscores as normal measurements
A measurement name that begins with an underscore and does not conflict
with one of the reserved measurement names will now be passed untouched
to the underlying shards rather than being intercepted as an empty
measurement.

A user still shouldn't rely on measurements that begin with underscores
to always be accessible, but this will prevent the most common use case
from causing unexpected behavior since we will very rarely, if ever, add
additional system sources.
2017-02-27 16:49:02 -06:00
Jonathan A. Sternberg 1fb34e3eef Dividing aggregate functions with different outputs doesn't panic 2017-02-23 18:38:29 -06:00
David Norton c7fa58473f fix #8028: call api.NewTypesDB() instead of new
The code was calling new(api.TypesDB) which didn't initialize an
unexported map inside of the type. Call api.NewTypesDB() instead.
2017-02-23 18:04:21 -05:00
Jonathan A. Sternberg 72e4dd01b9 Properly select a tag within a subquery
Previously, subqueries would only be able to select tag fields within a
subquery if the subquery used a selector. But, it didn't filter out
aggregates like `mean()` so it would panic instead.

Now it is possible to select the tag directly instead of rewriting the
query in an invalid way.

Some queries in this form will still not work though. For example, the
following still does not function (but also doesn't panic):

    SELECT count(host) FROM (SELECT mean, host FROM (SELECT mean(value) FROM cpu GROUP BY host))
2017-02-23 11:16:22 -06:00
Jonathan A. Sternberg 5a2b458180 Reduce the expression in a subquery to avoid a panic
The builder used for subqueries does not handle parenthesis, but a set
of parenthesis wrapping a field would cause it to panic. This code now
reduces the expression so the parenthesis are removed before being
processed.
2017-02-23 10:14:05 -06:00
Jason Wilder a024003f2c Merge branch '1.2' into jw-merge-12 2017-02-22 12:13:29 -07:00
Ben Johnson 78a9bb2527 Remove Tags.shouldCopy, replace with forceCopy on series creation.
Previously, tags had a `shouldCopy` flag to indicate if those tags
referenced an underlying buffer and should be copied to allow GC.
Unfortunately, this prevented tags from being copied that were
created and referenced the mmap which caused segfaults.

This change removes the `shouldCopy` flag and replaces it with a
`forceCopy` argument in `CreateSeriesIfNotExists()`. This allows
the write path to indicate that tags must be cloned on insert.
2017-02-21 11:13:35 -07:00
Mark Rushakoff 601cbcd084 Merge branch '1.2' into mr-merge-12 2017-02-17 16:14:22 -08:00
Ben Johnson 8e79ca5d75
Fix tag dereferencing panic.
Clones series tags under lock during var ref iterator creation.
2017-02-15 17:56:47 -07:00
Jonathan A. Sternberg 71f62d33e6 Map types correctly when using a regex and one of the measurements is empty 2017-02-13 18:14:29 -06:00
Mark Rushakoff c762ab49ee Merge pull request #7974 from influxdata/mr-4785-show-databases
Allow non-admin users to execute SHOW DATABASES
2017-02-13 15:04:00 -08:00
Jason Wilder c3de210ded Merge branch '1.2' into jw-merge-12 2017-02-13 11:45:27 -07:00
Mark Rushakoff 53699aa24f Allow non-admin users to execute SHOW DATABASES
This commit introduces a new interface type, influxql.Authorizer, that
is passed as part of a statement's execution context and determines
whether the context is permitted to access a given database. In the
future, the Authorizer interface may be expanded to other resources
besides databases. In this commit, the Authorizer interface is
specifically used to determine which databases are returned when
executing SHOW DATABASES.

When HTTP authentication is enabled, the existing meta.UserInfo struct
implements Authorizer, meaning admin users can SHOW every database, and
non-admin users can SHOW only databases for which they have read and/or
write permission.

When HTTP authentication is disabled, all databases are visible through
SHOW DATABASES.

This addresses a long-standing issue where Chronograf or Grafana would
be unable to list databases if the logged-in user did not have admin
privileges.

Fixes #4785.
2017-02-13 08:59:16 -08:00