There exists a possibility for an in-flight read on a TSMReader to read
a stale reference to an mmapped TSM file index, which has become
unmapped.
This commit resolves that issue by simply renaming the file, leaving the
original file handler open and the data mapped. The path is updated so
that if any callers need to refer to the name of the TSM file after it's
renamed, the new name will be reflected.
The orphaned file handler will be closed when the TSM file is closed.
StringArrayEncodeAll will panic if the total length of strings
contained in the src slice is > 0xffffffff. This change adds a unit
test to replicate the issue and an associated fix to return an error.
This also raises an issue that compactions will be unable to make
progress under the following condition:
* multiple string blocks are to be merged to a single block and
* the total length of all strings exceeds the maximum block size that
snappy will encode (0xffffffff)
The observable effect of this is errors in the logs indicating a
compaction failure.
Fixes#13687
This commit teaches the storage schema APIs how to track statistics
and make them available via the returned `cursors.StringIterator`.
Statistics are only tracked when decoding TSM blocks or when scanning
the in-memory cache.
Closes#13541
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagKeys expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement. This is tracked by issue #13497.
When a tsi1 partition closes, it waits on the wait group for compactions
and then acquires the lock. Unfortunately, a compaction may start in the
mean time, holding on to some resources. Then, close will attempt to
close those resources while holding the lock. That will block until
the compaction has finished, but it also needs to acquire the lock
in order to finish, leading to deadlock.
One cannot just move the wait group wait into the lock because, once
again, the compaction must acquire the lock before finishing. Compaction
can't finish before acquiring the lock because then it might be operating
on an invalid resource.
This change splits the locks into two: one to protect just against
concurrent Open and Close calls, and one to protect all of the other
state. We then just close the partition, acquire the lock, then free
the resources. Starting a compaction requires acquiring a resource
to the partition itself, so that it can't start one after it has
started closing.
This change also introduces a cancellation channel into a reference
to a resource that is closed when the resource is being closed, allowing
processes that have acquired a reference to clean up quicker if someone
is trying to close the resource.
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagValues expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement.
The TimeRangeIterator permits linear or random index scans and
can answer whether the current key has data for the specified time
interval, considering any tombstones.
When there are no tombstones there are some opportunities for
optimization to skip decoding blocks. Specifically, if the
queried time interval overlaps any boundaries of the TSM index entries.
Add a Contains API which is a peer to the TimestampArray.Contains
function. This is used by the schema APIs to determine if data exists
in the cache for a given key and time interval.
Permits random access of the iterator, correctly maintaining state,
so that Next may be called to iterator from a given key.
This API will be used by the schema APIs when a predicate is specified,
typically requiring random access.
TimestampArray.Contains(min,max) API performs a binary search to
determine if timestamps exist for the given time interval.
It also implements Exclude to drop timestamps that have been tombstoned.
DecodeTimestampArrayBlock decodes only the timestamps of the provided
block.
Removes the `STATS` file generated during TSI compaction as it had
potential for becoming inconsistent with the index data. Instead,
stats are recalculated on start up and on each compaction on a
per-partition basis.
Computing stats for 10M series across 10K measurements takes
approximately 0.171s.
The storer interface isn't necessary if the init/Free logic is
removed, which is unnecessary in a world with only one shard.
Additionally, there were some cases where an init/Free call could
race and cause data loss in the cache. Not doing it at all fixes
all of those races.
This change fixes#10511 that manifests when a shard is considered cold
faster than its cache is snapshotted. Previously the code only looked at
the last modification of compacted tsm1 files. Instead the (restored)
Engine.lastModified() also takes the cache into account.
Ports #10522 to master where engine.go has moved and Engine.LastModified()
was deleted because it was unused.
This commit adds a reason label to the total compaction metric. For
snapshots, the reason will indicate why the cache was snapshotted. For
other compactions, the reason label will be blank.
This commit adds a new Cache option, via the
`tsm1.CacheConfig.SnapshotAgeDuration` field, which controls the maximum
age the cache can reach before it is snapshotted to a TSM file.
The default value for this option is `0`, which means that the cache
will never be snapshotted based only on age. Setting this value to, for
example, 10 seconds, would result in the cache snapshotting every 10
seconds.
Snapshotting the cache more frequently can provide better durability
guarantees in some circumstances, though more, smaller TSM files will
lead to more work needed to compact them down to larger, more dense
files.
When using InfluxDB with a WAL there isn't really a strong reason to
alter `tsm1.CacheConfig.SnapshotAgeDuration` from `0`.
Alirie Gray
Nathaniel Cook
Christopher Wolff
Jacob Marble
Edd Robinson
Jeff Wendling
Stuart Carnie
Jeff Wendling
Ben Johnson
Tanya Gordeeva
Todd Persen
Jacob Marble
Jonas Hahnfeld