Abstract
========
We need to be careful w/ tombstone that fall exactly in sequence number range of a parquet file.
Current Bug
===========
Imagine the following order of events:
1. Router creates write at sequence number 1:
- `table,selector=1 payload=1 1`
- `table,selector=2 payload=2 2`
2. Ingester pulls write, waits a bit and persists it to parquet file 1:
- `table,selector=1 payload=1 1`
- `table,selector=2 payload=2 2`
4. Router creates write at sequence number 2:
- `table,selector=1 payload=3 3`
- `table,selector=2 payload=4 4`
5. Ingester pulls write
6. Router create delete at sequencer number 3: full time range, `selector=1`
7. Ingeser pulls delete and creates tombstone 1
8. Router creates write at sequence number 4:
- `table,selector=1 payload=5 5`
- `table,selector=2 payload=6 6`
9. Ingester pulls write
10. Ingester persists parquet file 2:
- `table,selector=2 payload=4 4`
- `table,selector=1 payload=5 5`
- `table,selector=2 payload=6 6`
When reading parquet file 2, the tombstone MUST NOT be applied. Otherwise `table,selector=1 payload=5 5` will be
deleted.
Notes
=====
Technically this issue also applies to files created by the compactor, however the compactor marks tombstones as
processed that fall into the sequence number range. It even does that in a single transaction:
fc4635a334/compactor/src/compact.rs (L821-L861)
Alternative
===========
An alternative solution would be if the ingester would mark tombstones that it materialized during persistence as
"processed" (tombstone 1 for parquet file 2 in the example above). However "processed" markers are currently a mere
optimization and don't affect correctness, which is nice for caching on the querier side as well as reasoning.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
5bebc73e3f