This commit refactors the tsdb query engine to use separate aggregate
and raw execution paths, encapsulates cursor functionality, and removes
the TagSetCursor from the aggregate path. By removing the TagSetCursor,
we can pass sets of unordered values to the map functions and bypass
the `container/heap` entirely.
This commit changes `tsdb.mapFunc` to use `tsdb.MapInput` instead
of an iterator. This will make it easier and faster to pass blocks
of values from the new storage engine into the engine.
This commit fixes `SelectMapper.Open()` so that it properly rolls back
transactions. Previously, this caused transactions to stay open
indefinitely which caused mmap resizes to hang indefinitely.
The TSDBStore was returning a nil mapper if the shard did not exist. The caller always
assumed the mapper would not be nil causing a panic. Instead, have the mapper skip the mapping
phase if it's shard reference is nil. This fixes queries against data-only nodes and against
shards that are not fully replicated in the cluster.
Fixes#3574
If we've seeked a cursor, then we can be sure that there will be no
data between it and the point that was seeked to. Take advantage of
this fact to only seek when it would yield us a value that would be
different from the last.
In addition, only init the pointsheap when doing a raw query. For
aggregate queries, it is reinitialized on every time bucket, so no
need to seek through all the cursors
For a synthetic database where there was only entries for a tiny
slice of time, it cut queries from 112 seconds to 30 seconds doing
`select mean(value) from cpu where time > now - 2h group by time(1h)`
Since we already got a pointsHeapItem, let's just reuse it instead
of allocating a new one. This cuts allocated memory of a 1 million
points aggregate query from 4881.97MB to 4139.86MB
CPU profiling shows that computing the tagset-based key of each point is significant CPU cost. These keys actually don't change per cursor, so precompute once at mapper-open time, and then use those values as points are drained from the cursor.
Before this change the cursor tag was getting computed on every point, which involved marshalling tags.
liang@qiniu.com
Ben Johnson
Cory LaNou
Philip O'Toole
Daniel Morsing
Daniel Morsing
Jason Wilder
dgnorton
Paul Dix
Philip O'Toole