Commit Graph

12 Commits (07505c8f721993c4b38a111458eea2023cfa974d)

Author SHA1 Message Date
Nga Tran 9fe7a50129
feat: call soft delete partitions from garbage collector (#6090) 2022-11-09 14:48:26 +00:00
Luke Bond dfb820615c
feat: deletion flagging in GC based on retention policy (#6073)
* feat: deletion flagging in GC based on retention policy

* chore: typo in comment

* fix: only soft delete parquet files that aren't yet soft deleted

* fix: guard against flakiness in catalog test

* chore: some better tests for parquet file delete flagging

Co-authored-by: Nga Tran <nga-tran@live.com>
2022-11-08 20:22:35 +00:00
Carol (Nichols || Goulding) c94c7eabbc fix: Replace deprecated parse(try_from_str(..)) with value_parser
See <https://github.com/clap-rs/clap/pull/3742>
2022-09-30 16:59:03 -04:00
Dom Dwyer cd4087e00d style: add no todo!() or dbg!() lints
Some crates had theme, some not - lets be consistent and have the
compiler spot dbg!() and todo!() macro calls - they should never be in
prod code!
2022-09-29 13:10:07 +02:00
Marco Neumann 84e8a4ac41
fix: GC `parquet_file` table in batches (#5691)
* fix: GC `parquet_file` table in batches

Otherwise this transaction will never finish in prod.

* fix: GC shutdown

* refactor: use constant
2022-09-20 11:14:39 +00:00
Carol (Nichols || Goulding) 414b0f02ca
fix: Use time helper methods in more places 2022-09-19 13:24:08 -04:00
Luke Bond b52865e018
feat: garbage collector now cleans up old parquet files (#5588)
* feat: garbage collector now cleans up old parquet files

* chore: clarifying comment in GC

* chore: typos in GC

* chore: typos in GC

* fix: cmdline arg in GC test needs updating after refactor

* fix: use select! on shutdown rx in GC

* fix: recalc cutoff in GD each loop

* chore: add delete_old that returns IDs only, for GC

* chore: use duration in GC args instead of usize days

* chore: GC lister runs forever w/ sleep; tests updated accordingly

* docs: fix link in GC comments to automatic link

* chore: test for delete_old_ids_only; refactor mem impl thereof

* chore: make GC test less flakey

* chore: make GC test less flakey

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 14:09:28 +00:00
Carol (Nichols || Goulding) 58f0b63cdc
refactor: Rename KafkaTopic to Topic or TopicMetadata or topic name as appropriate 2022-08-29 14:27:02 -04:00
Carol (Nichols || Goulding) 74c9529062
fix: Rename KafkaPartition to ShardIndex 2022-08-29 14:07:18 -04:00
Jake Goulding 4abf21c724
refactor: Rename Sequencer (and its entourage) to Shard 2022-08-29 14:06:43 -04:00
Marco Neumann 4da124d862
feat: concurrent garbage collector deletes (#5364)
This should speed up the prod process a bit.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 09:14:46 +00:00
Andrew Lamb e4dc8c2067
refactor: rename garbage collector crates for consistency (#5196)
* refactor: rename garbage collector crates for consistency

* fix: cargo fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00