Avoid that the querier accesses files that were flagged for deletion a
long time ago. This would happen if the following conditions hold:
- we have very long-running querier pods (e.g. over holidays)
- the table doesn't receive any writes (or the partition if we ever
change the cache granularity), hence the querier is never informed
that its state is out-of-date
- a compactor runs a cold compaction, and by doing so flags a file for
deletion
- the GC finally wants to delete it
This is mostly a safety measure to prevent weird internal server errors
that should nearly never happen. On the other hand I do not want to hunt
Heisenbugs.
This commit adds an `inspect` command to read through the sequenced
operations in a WAL file and debug pretty print their contents to
stdout, optionally filtering by a sequence number range.
Historically the authz crate didn't distinguish between an invalid
token and a valid token without the required permissions. Recently
errors were added to distinguish these cases. This means that an
invalid token now returns an "Internal" error if supplied an invalid
token. Detect this case and return a "Permission Denied" error,
which is the error type that was previously returned in this case.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* docs: Remove TODO comment that's TODONE
* docs: Oops, turns out the TODO comment was this enum's documentation
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: Allow passing service protection limits in create db gRPC call
* fix: Move the impl into the catalog namespace trait
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Nothing gets the partition ID out of the metadata. The parts of the code
interacting with object storage that need the ID to create the object
store path were using the partition ID from the metadata out of
convenience, but I changed those places to pass in the partition ID in a
separate argument instead.
This will make the transition to deterministic partition IDs a bit
smoother.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Props to proptesting for this one - the prop_arbitrary_strftime_format()
randomly generated the formatting sequence "%#z" which turns out to be
an undocumented way of causing a panic in chrono:
088b69372e/src/format/mod.rs (L673)
In fact, the docs actually list is as a usable sequence!
This adds a little extra layer of type safety and should be optimised
by the compiler. This commit also makes sure the ingester's WAL sink
tests assert the behaviour for partitioned sequence numbering on an
operation that hits multiple tables & thus partitions.
This commit changes the partitioner to skip generating partition keys
for successive rows that would generate identical partition keys.
Often successive rows in a batch will map to the same partition key -
for example, if multiple measurements are taken at the same time, then
the strftime formatter will output the same partition key part for each
row.
This commit changes the partitioner to only generate the first key
string in such a batch of identical keys. This is cheap to pre-compute,
as we only allow tag & time columns to be partitioned, both of which are
64-bit integers (dictionary key & timestamp respectively), making it
cheaper to check equality than to allocate & generate the partition key
string and check that.
Combined with the default YYYY-MM-DD precision reduction optimisation in
a prior commit, this optimisation is particularly effective for writes
with timestamps that span a single day (the typical case).
This change doubles the rows/s throughput for a modest 1,000 line batch,
with improvements across the board. I'd expect the performance benefit
to increase as the batch size increases, and/or as more partition
template parts are added.
Allows the StftimeFormatter to perform an equality match against a
timestamp and the last rendered timestamp, potentially after applying
the precision reduction optimisation if appropriate.
* fix(garbage-collector): just test parquet file existence
The GC, when checking files in object store against the catalog, only
cares if the parquet file for the given object store id exists in the
catalog. It doesn't need the full parquet file. Let's not transmit it
over the wire.
This PR uses a SELECT 1 and boolean to test for parquet file existing.
* helps #7784
* chore: use struct for from_row
* chore: satisfy clippy
* chore: fmt
Joe-Blount
kodiakhq[bot]
Fraser Savage
Marco Neumann
dependabot[bot]
Joe-Blount
Martin Hilton
Carol (Nichols || Goulding)
Andrew Lamb
Dom
Marko Mikulicic
CircleCI[bot]
Phil Bracikowski
Carol (Nichols || Goulding)