Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/garbage_collector/Cargo.toml

37 lines
1.1 KiB
TOML
Raw Normal View History

feat: Walking skeleton of the object store GC tool
2022-06-17 19:55:51 +00:00
[package]
refactor: rename garbage collector crates for consistency (#5196) * refactor: rename garbage collector crates for consistency * fix: cargo fmt Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00
name = "garbage_collector"
feat: Use workspace package metadata to reduce differences and repetition
2022-09-26 14:43:00 +00:00
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
feat: Walking skeleton of the object store GC tool
2022-06-17 19:55:51 +00:00
[dependencies]
refactor: remove unneeded dependency features
2022-07-01 18:36:05 +00:00
chrono = { version = "0.4", default-features = false }
chore(deps): Bump clap from 3.2.22 to 4.0.7 Bumps [clap](https://github.com/clap-rs/clap) from 3.2.22 to 4.0.7. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](https://github.com/clap-rs/clap/compare/v3.2.22...v4.0.7)
2022-09-29 11:14:38 +00:00
clap = { version = "4", features = ["derive", "env"] }
chore: Move garbage collection configuration into clap_blocks (#6678) Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-25 11:31:48 +00:00
clap_blocks = { path = "../clap_blocks" }
feat: garbage collector now cleans up old parquet files (#5588) * feat: garbage collector now cleans up old parquet files * chore: clarifying comment in GC * chore: typos in GC * chore: typos in GC * fix: cmdline arg in GC test needs updating after refactor * fix: use select! on shutdown rx in GC * fix: recalc cutoff in GD each loop * chore: add delete_old that returns IDs only, for GC * chore: use duration in GC args instead of usize days * chore: GC lister runs forever w/ sleep; tests updated accordingly * docs: fix link in GC comments to automatic link * chore: test for delete_old_ids_only; refactor mem impl thereof * chore: make GC test less flakey * chore: make GC test less flakey Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 14:09:28 +00:00
data_types = { path = "../data_types" }
feat: Walking skeleton of the object store GC tool
2022-06-17 19:55:51 +00:00
futures = "0.3"
feat: garbage collector now cleans up old parquet files (#5588) * feat: garbage collector now cleans up old parquet files * chore: clarifying comment in GC * chore: typos in GC * chore: typos in GC * fix: cmdline arg in GC test needs updating after refactor * fix: use select! on shutdown rx in GC * fix: recalc cutoff in GD each loop * chore: add delete_old that returns IDs only, for GC * chore: use duration in GC args instead of usize days * chore: GC lister runs forever w/ sleep; tests updated accordingly * docs: fix link in GC comments to automatic link * chore: test for delete_old_ids_only; refactor mem impl thereof * chore: make GC test less flakey * chore: make GC test less flakey Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 14:09:28 +00:00
humantime = "2.1.0"
feat: make object store garbage collector into a long-running service (#5135) * refactor: remove unused logging config * chore: remove the object store garbage collector CLI tool * refactor: accept an object store and catalog * refactor: make Result type alias public like the error * refactor: remove public modifier from modules * refactor: allow shutting down the object store garbage collector * feat: Introduce the object-store garbage collection server Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 21:27:38 +00:00
iox_catalog = { path = "../iox_catalog" }
chore(garbage collector): backoff first connect to objectstore (#7652) * chore(garbage collector): backoff first connect to objectstore This pr replaces the initial sleep added in #7562 with expential backoff retry of connecting to objectstore. This avoids the issue of waiting the configured sleep which can be quite long in a world where the service is getting redeployed often. * for #7562 * chore: rewrite lister::perform based on pr feedback This commit redoes perform as a do..while loop, putting the call to list from the object store at the top of the loop so the infinite backoff retry and be used for each loop iteration - it might fail on more than the first time! There are 3 selects as there are 3 wait stages and each needs to check for shutdown: os list, processing the list, and sleeping on the poll interval. * chore: hoist cancellation check higher; limit listing to 1000 files Responding to PR feedback. * chore: add error info message * chore: make build. :| * chore: linter
2023-04-27 17:51:52 +00:00
backoff = { path = "../backoff" }
chore: use workspace dependencies for `object_store`
2023-05-26 11:03:42 +00:00
object_store = { workspace = true }
feat: Add trogging logging config to object store garbage collector
2022-06-30 19:31:27 +00:00
observability_deps = { path = "../observability_deps" }
feat: Walking skeleton of the object store GC tool
2022-06-17 19:55:51 +00:00
snafu = "0.7"
tokio = { version = "1", features = ["macros", "rt", "sync"] }
feat: concurrent garbage collector deletes (#5364) This should speed up the prod process a bit. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 09:14:46 +00:00
tokio-stream = "0.1"
chore(deps): Bump tokio-util from 0.7.7 to 0.7.8 (#7665) Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.7 to 0.7.8. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.7...tokio-util-0.7.8) --- updated-dependencies: - dependency-name: tokio-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 09:24:39 +00:00
tokio-util = { version = "0.7.8" }
feat: Walking skeleton of the object store GC tool
2022-06-17 19:55:51 +00:00
uuid = { version = "1", features = ["v4"] }
fix: garbage_collector no longer uses chrono-english so it can use the workspace hack crate
2023-05-01 15:08:34 +00:00
workspace-hack = { version = "0.1", path = "../workspace-hack" }
fix: Test logic for whether to delete object store files And fix the bugs the tests found
2022-06-20 21:12:05 +00:00
[dev-dependencies]
feat(garbage-collector): batch parquet existence checks to catalog (#7964) * feat(garbage-collector): batch parquet existence checks to catalog The core feature of this PR is batching the existence checks of parquet files in object store against the catalog. Before, there was 1 catalog query per each parquet file in object store. This can be a lot of requests. This PR can perform one query of at most 100 parquet file uuids against the catalog in one query. A hundred seems like a decent starting place. The batch may not reach 100 because there is also a timeout on receiving object store meta objects from the object store lister thread. That timeout is set to 100 milliseconds. If more than 100 are received, they are batched into 100 for the catalog. Additionally, this PR includes surrounding code changes to make it more idiomatic (but not perfect). It follows up some suggested work from #7652 for watching for shutdown on the threads. * fixes #7784 * use hashset instead of vec to test for contains * chore: add test for db failure path * remove ParquetFileExistsByOSID and other single field structs that are just for sql deserialization; map to uuid explicitly * fix the sqlite query by using a blob literal X'<hex>' for uuids * comment clarifications * adjust loggings to warn from debug for expected rare events Many thanks to Carol for help implementing this!
2023-06-14 14:59:00 +00:00
async-trait = "0.1"
chore(deps): Bump bytes from 1.4.0 to 1.5.0 (#8692) Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/compare/v1.4.0...v1.5.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-08 12:17:12 +00:00
bytes = "1.5"
fix: Test logic for whether to delete object store files And fix the bugs the tests found
2022-06-20 21:12:05 +00:00
data_types = { path = "../data_types" }
test: Add basic end-to-end tests of the garbage collector
2022-07-01 17:56:57 +00:00
filetime = "0.2"
feat: make object store garbage collector into a long-running service (#5135) * refactor: remove unused logging config * chore: remove the object store garbage collector CLI tool * refactor: accept an object store and catalog * refactor: make Result type alias public like the error * refactor: remove public modifier from modules * refactor: allow shutting down the object store garbage collector * feat: Introduce the object-store garbage collection server Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 21:27:38 +00:00
metric = { path = "../metric" }
chore(deps): Bump once_cell from 1.17.2 to 1.18.0 Bumps [once_cell](https://github.com/matklad/once_cell) from 1.17.2 to 1.18.0. - [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md) - [Commits](https://github.com/matklad/once_cell/compare/v1.17.2...v1.18.0) --- updated-dependencies: - dependency-name: once_cell dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-06-05 02:03:15 +00:00
once_cell = { version = "1.18", features = ["parking_lot"] }
fix: Test logic for whether to delete object store files And fix the bugs the tests found
2022-06-20 21:12:05 +00:00
parquet_file = { path = "../parquet_file" }
test: Add basic end-to-end tests of the garbage collector
2022-07-01 17:56:57 +00:00
tempfile = "3"
chore: upgrade to sqlx 0.7.1 (#8266) There are a bunch of dependencies in `Cargo.lock` that are related to mysql. These are NOT compiled at all, and are also not part of `cargo tree`. The reason for the inclusion is a bug in cargo: https://github.com/rust-lang/cargo/issues/10801 Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-19 12:18:57 +00:00
sqlx = { version = "0.7.1", features = [ "runtime-tokio-rustls" ] }
feat(garbage-collector): batch parquet existence checks to catalog (#7964) * feat(garbage-collector): batch parquet existence checks to catalog The core feature of this PR is batching the existence checks of parquet files in object store against the catalog. Before, there was 1 catalog query per each parquet file in object store. This can be a lot of requests. This PR can perform one query of at most 100 parquet file uuids against the catalog in one query. A hundred seems like a decent starting place. The batch may not reach 100 because there is also a timeout on receiving object store meta objects from the object store lister thread. That timeout is set to 100 milliseconds. If more than 100 are received, they are batched into 100 for the catalog. Additionally, this PR includes surrounding code changes to make it more idiomatic (but not perfect). It follows up some suggested work from #7652 for watching for shutdown on the threads. * fixes #7784 * use hashset instead of vec to test for contains * chore: add test for db failure path * remove ParquetFileExistsByOSID and other single field structs that are just for sql deserialization; map to uuid explicitly * fix the sqlite query by using a blob literal X'<hex>' for uuids * comment clarifications * adjust loggings to warn from debug for expected rare events Many thanks to Carol for help implementing this!
2023-06-14 14:59:00 +00:00