217094029f
* feat: remove gen1 snapshot files with api (#2396) * feat: add core gen1 cleanup handler * feat: wired the Gen1 cleanup handler into the enterprise buffer so it's accessible via the WriteBuffer trait * feat: added http paths * feat: e2e tests added * feat: tests + tidy ups closes: https://github.com/influxdata/influxdb_pro/issues/2314 * refactor: address copilot feedback * fix: ingest only mode wiring * refactor: address PR feedback - update ordering comments - use 24h as default + tests to use min_age=0 * refactor: simplify gen1 cleanup handler - remove incorrect or outdated comments - use a single list and reuse obj store metadata - don't try to delete files that are already deleted, by not letting table index cache purging to make obj store delete calls again - simplify loop but still lookup the next compaction marker in case compaction has moved along * refactor: tidy ups - switch to rstest instead of test per case - remove unnecessary functions by reusing existing functions for OS delete - tidy up cleanup loop to accumulate results as it proceeds and avoid multiple allocations * refactor: concurrent deletions * chore: use scopeguard to allow resetting atomic bool * chore: doc comments fixes * refactor: hoist the imports and remove full path imports * refactor: tidy ups * refactor: update names of methods/constants * feat: additional tests for gen1 cleanup + minor tidy ups * refactor: small tidy ups * feat: add query param to configure deletion concurrency * refactor: test fixes * refactor: tidy ups - remove `unwrap()` on object store delete - tidy up errors / http layer result match - return None for a missing snapshot * refactor: return Option and rename method * chore(cherry-pick): add scopeguard dependency * chore(cherry-pick): move persister tests into separate file * chore(cherry-pick): add rustc-hash dependency * chore(cherry-pick): remove problematic merge conflict code I attempted to resolve the implicit merge conflict for the code being removed in this PR by cherry-picking another commit from enterprise (25ea412fbf30c37195be1242c25a2fda7bf73106). However, this led to a series of cascading merge conflicts that I don't think it's worth resolving for the 3.8.1 release. * chore(cherry-pick): move table_index_cache tests into separate file * feat: allow partial deletion of gen1 files in snapshot files (#2432) * feat: allow partial deletion of gen1 files in snapshot files Currently the snapshot files with references to leftover files are completely ignored. In this commit gen1 files that aren't leftovers are deleted with snapshot files having partial references to gen1 files already deleted. Following changes have been made to achieve that, - snapshot eligibility is now only relying on whether there are files in the snapshot that aren't processed by compactor. Note: compactor moves the next file id if it still has leftover files from a snapshot. - gen1 files are checked to see if it's leftover and if it is, then it just skips the deletion of that file alone and moves to next file. - separate counter to track not_found is added. Now that snapshot files can have partial references then the same files are picked up in subsequent cycle to delete so this ends up reporting that files are deleted however those deletes happened the first time these files were encountered. So this counter allows to differentiate between attempted to delete vs actual deletes. - object store now exposes a raw delete function that returns the "NotFound" error which is then used at a higher level to update the not_found counter. Existing functions that does the translation of not found error uses the same raw delete function under the hood closes: https://github.com/influxdata/influxdb_pro/issues/2426 * chore: remove unused/dead code and tests --------- Co-authored-by: praveen-influx <pkumar@influxdata.com> |
||
|---|---|---|
| .. | ||
| retryable_object_store | ||
| lib.rs | ||
| retryable_object_store.rs | ||
| test_object_store.rs | ||