* TSI index should compact old or too-large log files
* Old tsl files should be compacted without new writes
* Add extra logging when disk size test fails
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
When using queries like 'select count(_seriesKey) from bigmeasurement`, we
should iterate over the tsi structures to serve the query instead of loading
all the series into memory up front.
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
When the compaction planner runs, if it cannot acquire
a lock on the files it plans to compact, it returns a
nil list of compaction groups. This, in turn, sets the
engine statistics for compactions queues to zero,
which is incorrect. Instead, use the length of pending
files which would have been returned.
closes https://github.com/influxdata/influxdb/issues/22138
(cherry picked from commit 7d3efe1e9e)
closes https://github.com/influxdata/influxdb/issues/22141
This fix ensures that memory-mapped files are not released
before pointers into them are copied into heap memory.
MeasurementNamesByExpr() and MeasurementNamesByPredicate() can
cause panics by copying memory from mmapped files that have been
released. The functions they call use iterators to files which
are closed (releasing the mmapped files) before the memory is
safely copied to the heap.
closes https://github.com/influxdata/influxdb/issues/22000
(cherry picked from commit a989f8f8b6)
closes https://github.com/influxdata/influxdb/issues/22002
tsdb.Engine.IsIdle and tsdb.Engine.Digest now return a reason string for why the engine & shard are not idle.
Callers can then use this string for logging, if desired. The returned reason does not allocate memory, so the
caller may want to add the shard ID and path for more information in the log. This is intended to be used in
calls from the anti-entropy service in Enterprise.
(cherry picked from commit bf45841359)
fixes https://github.com/influxdata/influxdb/issues/21448
(cherry picked from commit c8da9bafbf)
closes https://github.com/influxdata/influxdb/issues/21894
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.
Closes#21577
(cherry picked from commit f64be286be)
Closes https://github.com/influxdata/influxdb/issues/21598
The cursors were previously [start, stop) to be consistent with how flux
requests data, but the underlying storage file store was [start, stop]
because that's how influxql read data. This reverts back the cursor
behavior so that it is now [start, stop] everywhere and the conversion
from [start, stop) to [start, stop] is performed when doing the cursor
request to get the next cursor.
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written
in memory and when the new file is being renamed.
Test verification of correct behavior by checking the fields.idx
file matches the in-memory copy after heavily parallel measurement addition.
Co-authored-by: davidby-influx <72418212+davidby-influx@users.noreply.github.com>
This use-after-free bug may lead to segfault. The iterators that have
reference to the underlying index files were closed too early while
the bitmaps were still used afterwards. If a compaction occurs
concurrently and removes the index files, it would result in accessing
unmap'd memory address.
davidby-influx
Daniel Moran
Jonathan A. Sternberg
Sam Arnold
Dane Strandboge
Yun Zhao
LLThomas
Tristan Su
Ben Johnson
Stuart Carnie