Previously, only time expressions got propagated inwards. The reason for
this was simple. If the outer query was going to filter to a specific
time range, then it would be unnecessary for the inner query to output
points within that time frame. It started as an optimization, but became
a feature because there was no reason to have the user repeat the same
time clause for the inner query as the outer query. So we allowed an
aggregate query with an interval to pass validation in the subquery if
the outer query had a time range. But `GROUP BY` clauses were not
propagated because that same logic didn't apply to them. It's not an
optimization there. So while grouping by a tag in the outer query
without grouping by it in the inner query was useless, there wasn't any
particular reason to care.
Then a bug was found where wildcards would propagate the dimensions
correctly, but the outer query containing a group by with the inner
query omitting it wouldn't correctly filter out the outer group by. We
could fix that filtering, but on further review, I had been seeing
people make that same mistake a lot. People seem to just believe that
the grouping should be propagated inwards. Instead of trying to fight
what the user wanted and explicitly erase groupings that weren't
propagated manually, we might as well just propagate them for the user
to make their lives easier. There is no useful situation where you would
want to group into buckets that can't physically exist so we might as
well do _something_ useful.
This will also now propagate time intervals to inner queries since the
same applies there. But, while the interval propagates, the following
query will not pass validation since it is still not possible to use a
grouping interval with a raw query (even if the inner query is an
aggregate):
SELECT * FROM (SELECT mean(value) FROM cpu) WHERE time > now() - 5m GROUP BY time(1m)
This also means wildcards will behave a bit differently. They will
retrieve dimensions from the sources in the inner query rather than just
using the dimensions in the group by.
Fixing top() and bottom() to return the correct auxiliary fields.
Unfortunately, we were not copying the buffer with the auxiliary fields
so those values would be overwritten by a later point.
When an error that appears to be an SSL error happens without SSL
enabled, the client will attempt to reconnect with SSL just to see if
that works. If it works, it exits with an error message telling the user
to add `-ssl`. It will also do the same if the SSL connection is unsafe
although it will warn that this is insecure.
The backup command can fail if a snapshot is running which silently
closes the connection. This causes the backup shard command to continue
on as if nothing failed.
This adds query syntax support for subqueries and adds support to the
query engine to execute queries on subqueries.
Subqueries act as a source for another query. It is the equivalent of
writing the results of a query to a temporary database, executing
a query on that temporary database, and then deleting the database
(except this is all performed in-memory).
The syntax is like this:
SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *)
This will execute derivative and then sum the result of those derivatives.
Another example:
SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host)
This would let you find the maximum minimum value of each host.
There is complete freedom to mix subqueries with auxiliary fields. The only
caveat is that the following two queries:
SELECT mean(value) FROM cpu
SELECT mean(value) FROM (SELECT value FROM cpu)
Have different performance characteristics. The first will calculate
`mean(value)` at the shard level and will be faster, especially when it comes to
clustered setups. The second will process the mean at the top level and will not
include that optimization.
Benchmark improvements with this change:
benchmark old ns/op new ns/op delta
BenchmarkExportTSMFloats_100s_250vps-4 23206480 10279106 -55.71%
BenchmarkExportTSMInts_100s_250vps-4 17995000 5762310 -67.98%
BenchmarkExportTSMBools_100s_250vps-4 17067605 4235467 -75.18%
BenchmarkExportTSMStrings_100s_250vps-4 54846997 34682568 -36.76%
BenchmarkExportWALFloats_100s_250vps-4 23459937 10436297 -55.51%
BenchmarkExportWALInts_100s_250vps-4 18747150 6236062 -66.74%
BenchmarkExportWALBools_100s_250vps-4 17988273 4814358 -73.24%
BenchmarkExportWALStrings_100s_250vps-4 59700802 35815739 -40.01%
benchmark old allocs new allocs delta
BenchmarkExportTSMFloats_100s_250vps-4 201442 51738 -74.32%
BenchmarkExportTSMInts_100s_250vps-4 201442 51728 -74.32%
BenchmarkExportTSMBools_100s_250vps-4 201441 51638 -74.37%
BenchmarkExportTSMStrings_100s_250vps-4 404092 201584 -50.11%
BenchmarkExportWALFloats_100s_250vps-4 250322 75627 -69.79%
BenchmarkExportWALInts_100s_250vps-4 250323 75617 -69.79%
BenchmarkExportWALBools_100s_250vps-4 250321 75527 -69.83%
BenchmarkExportWALStrings_100s_250vps-4 452868 225291 -50.25%
benchmark old bytes new bytes delta
BenchmarkExportTSMFloats_100s_250vps-4 5170539 2351789 -54.52%
BenchmarkExportTSMInts_100s_250vps-4 5143189 2331276 -54.67%
BenchmarkExportTSMBools_100s_250vps-4 3724951 2143780 -42.45%
BenchmarkExportTSMStrings_100s_250vps-4 17131400 10796281 -36.98%
BenchmarkExportWALFloats_100s_250vps-4 4487868 1468109 -67.29%
BenchmarkExportWALInts_100s_250vps-4 4458395 1452359 -67.42%
BenchmarkExportWALBools_100s_250vps-4 2838719 1258755 -55.66%
BenchmarkExportWALStrings_100s_250vps-4 16787201 10010700 -40.37%
Also, after improving those benchmarks, I did a time-filtered export on
a 450MB TSM file to a 21GB plain text output, with and without the
bufio.BufferedWriter.
Without buffering, it took about 263s, and with buffering, it took about
60s, for a delta of about -77%.
It looks like the real import path to the project is go.uber.org/zap
instead of github.com/uber-go/zap since the example in the project
references that path.
This was needed when we were on go 1.4 but hasn't been needed since go
1.5. It was kept because we weren't sure if we were going to have to
rollback to an older version of Go at that time and we kept it so we
wouldn't forget to readd it.
Now that we are on go 1.7 with go 1.4 deprecated, there is no going back
so we might as well remove this so people can set GOMAXPROCS to a custom
value using environment variables.
The logging library has been switched to use uber-go/zap. While the
logging has been changed to use structured logging, this commit does not
change any of the logging statements to take advantage of the new
structured log or new log levels. Those changes will come in future
commits.
`percentile()` is supposed to be a selector and return the time of the
point, but that only got changed when the input was a float. Updating
the integer processor to also return the time of the point rather than
the beginning of the interval.
NO-OP on platforms with unix path separator.
On Windows paths get converted to slashes before adding to archive and back to backslashes during restore.
The `partial` tag has been added to the JSON response of a series and
the result so that a client knows when more of the series or result will
be sent in a future JSON chunk.
This helps interactive clients who don't want to wait for all of the
data to know if it is done processing the current series or the current
result. Previously, the client had to guess if the next chunk would
refer to the same result or a new result and it had to match the name
and tags of the two series to know if they were the same series. Now,
the client just needs to check the `partial` field included with the
response to know if it should expect more.
Fixed `max-row-limit` so it counts rows instead of results and it
truncates the response when the `max-row-limit` is reached.
There are 2 new keys in the configuration file.
- security-level: "none", "sign", or "encrypt".
- auth-file: The location of the user/password file.
Please see the collectd network doc for more details.
Jonathan A. Sternberg
Edd Robinson
Cory LaNou
gunnaraasen
Mark Rushakoff
Mark Rushakoff
Jason Wilder
Michael Nikitochkin
Gustav Westling
oiooj
Allen Petersen
Marc
Jason Wilder