Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/compactor2/Cargo.toml

40 lines
1.2 KiB
TOML
Raw Normal View History

feat: very basic compactor2 skeleton (#6614) Sets up crate and wires up the main binary. No tests yet, no algorithm framework, just the bare minimum. Also I decided to not offer a gRPC server in `compactor2` at the moment and hence did not implement any handle/delegate infrastructure. We add this later if we need it. This also means compactor2 does NOT provide a catalog service for now. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
[package]
name = "compactor2"
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 (#7374) * chore(deps): Bump async-trait from 0.1.66 to 0.1.68 Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.66 to 0.1.68. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68) --- updated-dependencies: - dependency-name: async-trait dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
async-trait = "0.1.68"
feat: invokes catalog functions for compactor2 (#6619) Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 10:33:57 +00:00
backoff = { path = "../backoff" }
chore(deps): Bump bytes from 1.3.0 to 1.4.0 Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/compare/v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-02-01 00:30:56 +00:00
bytes = "1.4"
refactor: reorganize compact_files core function (#6636) * refactor: reorganize compact_files core function * chore: smore more in-progress structure * refactor: further reorganization for compact_files Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 13:41:23 +00:00
datafusion = { workspace = true }
feat: basic compactor2 algo layout (#6616)
2023-01-18 18:51:59 +00:00
data_types = { path = "../data_types" }
feat: very basic compactor2 skeleton (#6614) Sets up crate and wires up the main binary. No tests yet, no algorithm framework, just the bare minimum. Also I decided to not offer a gRPC server in `compactor2` at the moment and hence did not implement any handle/delegate infrastructure. We add this later if we need it. This also means compactor2 does NOT provide a catalog service for now. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
futures = "0.3"
iox_catalog = { path = "../iox_catalog" }
iox_query = { path = "../iox_query" }
iox_time = { path = "../iox_time" }
feat: add compaction level to commit metrics (#6985) * feat: add compaction level to commit metrics * test: more realism
2023-02-15 09:28:19 +00:00
itertools = "0.10.5"
feat: very basic compactor2 skeleton (#6614) Sets up crate and wires up the main binary. No tests yet, no algorithm framework, just the bare minimum. Also I decided to not offer a gRPC server in `compactor2` at the moment and hence did not implement any handle/delegate infrastructure. We add this later if we need it. This also means compactor2 does NOT provide a catalog service for now. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
metric = { path = "../metric" }
chore(deps): Bump object_store from 0.5.5 to 0.5.6 (#7433) Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.5 to 0.5.6. - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md) - [Commits](https://github.com/apache/arrow-rs/commits) --- updated-dependencies: - dependency-name: object_store dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dom <dom@itsallbroken.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 08:43:34 +00:00
object_store = "0.5.6"
feat: very basic compactor2 skeleton (#6614) Sets up crate and wires up the main binary. No tests yet, no algorithm framework, just the bare minimum. Also I decided to not offer a gRPC server in `compactor2` at the moment and hence did not implement any handle/delegate infrastructure. We add this later if we need it. This also means compactor2 does NOT provide a catalog service for now. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
observability_deps = { path = "../observability_deps" }
parquet_file = { path = "../parquet_file" }
refactor: reorganize compact_files core function (#6636) * refactor: reorganize compact_files core function * chore: smore more in-progress structure * refactor: further reorganization for compact_files Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 13:41:23 +00:00
predicate = { path = "../predicate" }
feat: basic compactor2 algo layout (#6616)
2023-01-18 18:51:59 +00:00
rand = "0.8.3"
refactor: reorganize compact_files core function (#6636) * refactor: reorganize compact_files core function * chore: smore more in-progress structure * refactor: further reorganization for compact_files Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 13:41:23 +00:00
schema = { path = "../schema" }
feat: compactor sharding (#6729) I'm not saying we have to use this, but this is a demonstration how easy it would be to add sharding to the compaction tier and also acts as a "backup / insurance" if we ever need it. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-31 14:37:06 +00:00
sharder = { path = "../sharder" }
feat: very basic compactor2 skeleton (#6614) Sets up crate and wires up the main binary. No tests yet, no algorithm framework, just the bare minimum. Also I decided to not offer a gRPC server in `compactor2` at the moment and hence did not implement any handle/delegate infrastructure. We add this later if we need it. This also means compactor2 does NOT provide a catalog service for now. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
tokio = { version = "1", features = ["macros", "rt", "sync"] }
chore(deps): Bump tokio-util from 0.7.5 to 0.7.7 (#6964) --- updated-dependencies: - dependency-name: tokio-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-13 10:10:53 +00:00
tokio-util = { version = "0.7.7" }
feat: job semaphore (#6696) * refactor: avoid too-many-arguments * refactor: extract `fetch_partition_info` * feat: job semaphore
2023-01-25 10:35:07 +00:00
tracker = { path = "../tracker" }
refactor: extract "fetch files" component (#6632)
2023-01-19 13:11:44 +00:00
uuid = { version = "1", features = ["v4"] }
chore: Rerun cargo hakari manage-deps
2023-02-24 18:02:23 +00:00
workspace-hack = { version = "0.1", path = "../workspace-hack" }
feat: very basic compactor2 skeleton (#6614) Sets up crate and wires up the main binary. No tests yet, no algorithm framework, just the bare minimum. Also I decided to not offer a gRPC server in `compactor2` at the moment and hence did not implement any handle/delegate infrastructure. We add this later if we need it. This also means compactor2 does NOT provide a catalog service for now. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
[dev-dependencies]
refactor: execute compaction plan (#6654) * chore: address review comment of previous PR * refactor: execute compact plan * refactor: we will now compact all L0 and L1 files of a partition and split them as needed * chore: comnents Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 22:34:50 +00:00
arrow_util = { path = "../arrow_util" }
feat: Don't add a partition to skipped_compactions if it makes progress If a partition takes longer than `partition_timeout` to compact, but it did make _some_ progress, let the compactor try that partition again at a later time so that compaction for the partition will eventually complete. If a partition times out and _no_ progress has been made, then still add it to the skipped_compactions table because it's either too big to ever compact or is otherwise stuck. Closes influxdata/idpe#17234.
2023-03-10 22:14:47 +00:00
assert_matches = "1"
refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test (#6960) * refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test * fix: Update compactor2/src/components/mod.rs Co-authored-by: Marco Neumann <marco@crepererum.net> --------- Co-authored-by: Marco Neumann <marco@crepererum.net> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:06:42 +00:00
compactor2_test_utils = { path = "../compactor2_test_utils" }
refactor: execute compaction plan (#6654) * chore: address review comment of previous PR * refactor: execute compact plan * refactor: we will now compact all L0 and L1 files of a partition and split them as needed * chore: comnents Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 22:34:50 +00:00
iox_tests = { path = "../iox_tests" }
refactor: more components for compactor2 (#6630) * refactor: rename "rules" to "components" * refactor: replace `Trait::name` w/ `impl Display` * refactor: remodel partitions source * refactor: add TODO * refactor: fully components-based partitions source * test: add tests for partitions sources * docs: improve Co-authored-by: Dom <dom@itsallbroken.com> Co-authored-by: Dom <dom@itsallbroken.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 12:29:38 +00:00
test_helpers = { path = "../test_helpers"}
chore(deps): Bump insta from 1.28.0 to 1.29.0 (#7322) Bumps [insta](https://github.com/mitsuhiko/insta) from 1.28.0 to 1.29.0. - [Release notes](https://github.com/mitsuhiko/insta/releases) - [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md) - [Commits](https://github.com/mitsuhiko/insta/compare/1.28.0...1.29.0) --- updated-dependencies: - dependency-name: insta dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:25:01 +00:00
insta = { version = "1.29.0", features = ["yaml"] }