influxdb/iox_catalog/src
Marco Neumann 743a59aa64
feat: use single per-migration txn when possible (#8373)
* test: improve `test_step_sql_statement_no_transaction`

* feat: also print number of steps in "applying migration step"

* feat: use single per-migration txn when possible

If all steps can (and want) to run in a transaction block, then wrap the
migration bookkeeping and the migration script into a single
transaction. This way we avoid the dirty state altogether because its
now an "all or nothing" migration.

Note that we still guarantee that there is only a single migration
running at the same time due to the locking mechanism. Otherwise we
would potentially run into nasty transaction failures during schema
modifications.

This is related to #7897 but only fixes / self-heals the "dirty" state
for transaction that can run in transactions. For concurrent index
migrations (which we need in prod) we need to be a bit smarter and this
will be done in a follow-up. However I feel that not leaving half-done
migrations for the cases where it's technically possible (e.g. adding
columns) is already a huge step forward.

* test: make `test_migrator_uses_single_transaction_when_possible` harder

* test: explain test

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-01 08:18:39 +00:00
..
interface.rs feat: Make parquet_file.partition_id optional in the catalog (#8339) 2023-07-31 12:40:56 +00:00
kafkaless_transition.rs refactor: add `parquet_file` PG index for querier (#7842) 2023-05-31 10:56:32 +00:00
lib.rs feat: Make parquet_file.partition_id optional in the catalog (#8339) 2023-07-31 12:40:56 +00:00
mem.rs feat: Make parquet_file.partition_id optional in the catalog (#8339) 2023-07-31 12:40:56 +00:00
metrics.rs feat: add batch partition getters (#8268) 2023-07-19 15:05:41 +00:00
migrate.rs feat: use single per-migration txn when possible (#8373) 2023-08-01 08:18:39 +00:00
postgres.rs feat: Make parquet_file.partition_id optional in the catalog (#8339) 2023-07-31 12:40:56 +00:00
sqlite.rs feat: Make parquet_file.partition_id optional in the catalog (#8339) 2023-07-31 12:40:56 +00:00