influxdb/iox_query/src
Marco Neumann 4638b89d93
refactor: migrate retention to proper predicates (#8092)
Do not (ab)use per-chunk delete predicates for the retention policy.
Instead use a per-table predicate.

This makes the code way cleaner, since the scoping is correct (i.e.
delete predicates are a table-wide attribute, not a chunk-based one) and
it is consistent time predicates that the user providers (e.g. via
`WHERE time > x`).

It also allows us to remove delete predicates (in their current,
non-scalable form) from the query path. A potential future version would
likely not use per chunk predicates (and "is processed" markers) but use
the timestamp / chunk order to determine to which data the predicate
should be applied.

Note that the lowering of the retention policy changed slightly from

```text
(time > (now() - retention)) AND (time < MAX)
```

to

```text
time > (now() - retention)
```

Since the `MAX` cut is just an artifact of the lowering and was unnecessary.

Closes #7409.
Closes #7410.
2023-06-29 08:36:37 +00:00
..
exec Revert "chore: Update datafusion + arrow/arrow-flight/parquet to version `42.0.0` (#8036)" (#8049) 2023-06-22 11:03:25 +00:00
frontend refactor: remove output sorting from scan provider (#7798) 2023-05-17 05:59:04 +00:00
logical_optimizer chore: Update DataFusion and other dependencies (#8014) 2023-06-16 10:39:36 +00:00
physical_optimizer chore: Update DataFusion and other dependencies (#8014) 2023-06-16 10:39:36 +00:00
plan refactor: Move InfluxQL behaviour from iox_query to new crate (#7156) 2023-03-08 22:29:20 +00:00
provider Revert "chore: Update datafusion + arrow/arrow-flight/parquet to version `42.0.0` (#8036)" (#8049) 2023-06-22 11:03:25 +00:00
config.rs fix: remove de-dup even if we have many partitions (#8004) 2023-06-15 15:25:25 +00:00
exec.rs feat(parquet): reduce and limit buffering when writing parquet files (#7880) 2023-05-31 13:27:32 +00:00
frontend.rs refactor: remove output sorting from scan provider (#7798) 2023-05-17 05:59:04 +00:00
lib.rs refactor: migrate retention to proper predicates (#8092) 2023-06-29 08:36:37 +00:00
plan.rs refactor: Move InfluxQL behaviour from iox_query to new crate (#7156) 2023-03-08 22:29:20 +00:00
provider.rs refactor: migrate retention to proper predicates (#8092) 2023-06-29 08:36:37 +00:00
pruning.rs chore: Update DataFusion pin (#7916) 2023-06-05 18:38:59 +00:00
statistics.rs refactor: migrate `iox_query` to use DataFusion statistics (#7908) 2023-06-02 09:18:59 +00:00
test.rs refactor: migrate retention to proper predicates (#8092) 2023-06-29 08:36:37 +00:00
util.rs chore: clippy 🧠 2023-06-04 07:23:11 +10:00