Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/ingester/Cargo.toml

83 lines
3.0 KiB
TOML
Raw Normal View History

feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
[package]
fix: Rename ingester2 to ingester
2023-05-05 16:03:50 +00:00
name = "ingester"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
arrow = { workspace = true, features = ["prettyprint"] }
arrow_util = { version = "0.1.0", path = "../arrow_util" }
test: add ingester2_test_ctx Moves the ingester2 TestContext & builder to its own crate for reuse between integration tests & benchmarks (and more?!) This allows us to DRY & reuse the test code with it's nice API for benchmarks too, without forcing it all to be in the "prod" build of ingester2 (and the dependencies, and their dependencies, etc). This also means this code can be built and cached instead of being rebuilt all the time during normal development.
2023-04-12 10:35:26 +00:00
arrow-flight = { workspace = true }
feat(ingester2): optimal persist parallelism This commit changes the behaviour of the persist system to enable optimal parallelism of persist operations, and improve the accuracy of the outstanding job bound / back-pressure. Previously all persist operations for a given partition were consistently hashed to a single worker task. This serialised persistence per partition, ensuring all updates to the partition sort key were serialised. However, this also unnecessarily serialises persist operations that do not need to update the sort key, reducing the potential throughput of the system; in the worst case of a single partition receiving all the writes, only one worker would be persisting, and the other N-1 workers would be idle. After this change, the sort key is inspected when enqueuing the persist operation and if it can be determined that no sort key update is necessary (the typical case), then the persist task is placed into a global work queue from which all workers consume. This allows for maximal parallelisation of these jobs, and the removes the per-worker head-of-line blocking. In the case that the sort key does need updating, these jobs continue to be consistently hashed to a single worker, ensuring serialised sort key updates only where necessary. To support these changes, the back-pressure system has been changed to account for all outstanding persist jobs in the system, regardless of type or assigned worker - a logical, bounded queue is composed together of a semaphore limiting the number of persist tasks overall, and a series of physical, unbounded queues - one to each worker & the global queue. The overall system remains bounded by the INFLUXDB_IOX_PERSIST_QUEUE_DEPTH value, and is now simpler to reason about (it is independent of the number of workers, etc).
2022-12-15 14:14:23 +00:00
async-channel = "1.8.0"
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 (#7374) * chore(deps): Bump async-trait from 0.1.66 to 0.1.68 Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.66 to 0.1.68. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68) --- updated-dependencies: - dependency-name: async-trait dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
async-trait = "0.1.68"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
backoff = { version = "0.1.0", path = "../backoff" }
chore(deps): Bump bytes from 1.3.0 to 1.4.0 Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/compare/v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-02-01 00:30:56 +00:00
bytes = "1.4.0"
chore(deps): Bump crossbeam-utils from 0.8.14 to 0.8.15 (#7089) Bumps [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam) from 0.8.14 to 0.8.15. - [Release notes](https://github.com/crossbeam-rs/crossbeam/releases) - [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md) - [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-utils-0.8.14...crossbeam-utils-0.8.15) --- updated-dependencies: - dependency-name: crossbeam-utils dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-01 10:16:09 +00:00
crossbeam-utils = "0.8.15"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
data_types = { version = "0.1.0", path = "../data_types" }
test: add ingester2_test_ctx Moves the ingester2 TestContext & builder to its own crate for reuse between integration tests & benchmarks (and more?!) This allows us to DRY & reuse the test code with it's nice API for benchmarks too, without forcing it all to be in the "prod" build of ingester2 (and the dependencies, and their dependencies, etc). This also means this code can be built and cached instead of being rebuilt all the time during normal development.
2023-04-12 10:35:26 +00:00
datafusion.workspace = true
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
dml = { version = "0.1.0", path = "../dml" }
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` (#6756) * chore: update DataFusion * fix: test * chore: format * chore: clippy * chore: update arrow * chore: arrow upgrade fallout * chore: Run cargo hakari tasks * chore: remove failing warm compaction test * fix: flight error propagation * chore: update parquet size * fix: Update error message * chore: Update parquet metadata test --------- Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: Andrew Lamb <alamb@influxdata.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:35:39 +00:00
flatbuffers = "23.1.21"
chore(deps): Bump futures from 0.3.27 to 0.3.28 (#7397) Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.27 to 0.3.28. - [Release notes](https://github.com/rust-lang/futures-rs/releases) - [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.27...0.3.28) --- updated-dependencies: - dependency-name: futures dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-31 10:44:23 +00:00
futures = "0.3.28"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
generated_types = { version = "0.1.0", path = "../generated_types" }
hashbrown.workspace = true
fix: Move ingester-querier gRPC communication to its own crate
2023-05-12 15:35:29 +00:00
ingester_query_grpc = { path = "../ingester_query_grpc" }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
iox_catalog = { version = "0.1.0", path = "../iox_catalog" }
iox_query = { version = "0.1.0", path = "../iox_query" }
iox_time = { path = "../iox_time" }
metric = { version = "0.1.0", path = "../metric" }
mutable_batch = { version = "0.1.0", path = "../mutable_batch" }
mutable_batch_pb = { version = "0.1.0", path = "../mutable_batch_pb" }
observability_deps = { version = "0.1.0", path = "../observability_deps" }
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 (#6473) * chore(deps): Bump once_cell from 1.16.0 to 1.17.0 Bumps [once_cell](https://github.com/matklad/once_cell) from 1.16.0 to 1.17.0. - [Release notes](https://github.com/matklad/once_cell/releases) - [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md) - [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0) --- updated-dependencies: - dependency-name: once_cell dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Change once_cell version specifier to major.minor for less churn Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
once_cell = "1.17"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
parking_lot = "0.12.1"
feat(ingester2): parallel partition persistence Implements actor-based, parallel persistence in ingester2 with controllable fan-out parallelism and queue depths. This implementation encapsulates the complexity of persistence, queuing and parallelism - the caller simply uses the handle to persist a partition, while the actor handles fan-out to a set of persistence workers, compaction in a separate thread-pool, and optional completion notifications. By consistently hashing persist jobs onto workers, parallelism is achieved across partitions, but serialisation of partition persists is enforced so that the sort key update is correctly serialised.
2022-12-01 09:37:25 +00:00
parquet_file = { version = "0.1.0", path = "../parquet_file" }
chore(deps): Bump pin-project from 1.0.12 to 1.1.0 Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.0.12 to 1.1.0. - [Release notes](https://github.com/taiki-e/pin-project/releases) - [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md) - [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.12...v1.1.0) --- updated-dependencies: - dependency-name: pin-project dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-05-15 02:02:32 +00:00
pin-project = "1.1.0"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
predicate = { version = "0.1.0", path = "../predicate" }
chore(deps): Bump prost from 0.11.8 to 0.11.9 (#7537) * chore(deps): Bump prost from 0.11.8 to 0.11.9 Bumps [prost](https://github.com/tokio-rs/prost) from 0.11.8 to 0.11.9. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Commits](https://github.com/tokio-rs/prost/compare/v0.11.8...v0.11.9) --- updated-dependencies: - dependency-name: prost dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): Bump prost-build from 0.11.8 to 0.11.9 Bumps [prost-build](https://github.com/tokio-rs/prost) from 0.11.8 to 0.11.9. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Commits](https://github.com/tokio-rs/prost/compare/v0.11.8...v0.11.9) --- updated-dependencies: - dependency-name: prost-build dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): Bump prost-types from 0.11.8 to 0.11.9 Bumps [prost-types](https://github.com/tokio-rs/prost) from 0.11.8 to 0.11.9. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Commits](https://github.com/tokio-rs/prost/compare/v0.11.8...v0.11.9) --- updated-dependencies: - dependency-name: prost-types dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 10:27:15 +00:00
prost = { version = "0.11.9", default-features = false, features = ["std"] }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
rand = "0.8.5"
schema = { version = "0.1.0", path = "../schema" }
service_grpc_catalog = { version = "0.1.0", path = "../service_grpc_catalog" }
feat(ingester2): parallel partition persistence Implements actor-based, parallel persistence in ingester2 with controllable fan-out parallelism and queue depths. This implementation encapsulates the complexity of persistence, queuing and parallelism - the caller simply uses the handle to persist a partition, while the actor handles fan-out to a set of persistence workers, compaction in a separate thread-pool, and optional completion notifications. By consistently hashing persist jobs onto workers, parallelism is achieved across partitions, but serialisation of partition persists is enforced so that the sort key update is correctly serialised.
2022-12-01 09:37:25 +00:00
sharder = { version = "0.1.0", path = "../sharder" }
fix: drop WAL segments after replay Changes the WAL replay logic to: * Replay a segment file * Persist all replayed data * Drop segment file * ...repeat... This ensures old WAL segments are removed once their contents have been made durable, fixing #6461.
2022-12-22 15:53:48 +00:00
test_helpers = { path = "../test_helpers", features = ["future_timeout"], optional = true }
test: add ingester2_test_ctx Moves the ingester2 TestContext & builder to its own crate for reuse between integration tests & benchmarks (and more?!) This allows us to DRY & reuse the test code with it's nice API for benchmarks too, without forcing it all to be in the "prod" build of ingester2 (and the dependencies, and their dependencies, etc). This also means this code can be built and cached instead of being rebuilt all the time during normal development.
2023-04-12 10:35:26 +00:00
thiserror = "1.0.40"
chore(deps): Bump tokio from 1.27.0 to 1.28.0 (#7667) * chore(deps): Bump tokio from 1.27.0 to 1.28.0 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.27.0 to 1.28.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
tokio = { version = "1.28", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
chore(deps): Bump tokio-util from 0.7.7 to 0.7.8 (#7665) Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.7 to 0.7.8. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.7...tokio-util-0.7.8) --- updated-dependencies: - dependency-name: tokio-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 09:24:39 +00:00
tokio-util = "0.7.8"
refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc (#7515) * refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc * chore: Run cargo hakari tasks --------- Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:07:19 +00:00
tonic = { workspace = true }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
trace = { version = "0.1.0", path = "../trace" }
chore(deps): Bump uuid from 1.3.2 to 1.3.3 Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.3.2 to 1.3.3. - [Release notes](https://github.com/uuid-rs/uuid/releases) - [Commits](https://github.com/uuid-rs/uuid/compare/1.3.2...1.3.3) --- updated-dependencies: - dependency-name: uuid dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2023-05-16 02:00:24 +00:00
uuid = "1.3.3"
feat(ingester2): commit writes to write-ahead log Adds WalSink, an implementation of the DmlSink trait that commits DML operations to the write-ahead log before passing the DML op into the decorated inner DmlSink chain. By structuring the WAL logic as a decorator, the chain of inner DmlSink handlers within it are only ever invoked after the WAL commit has successfully completed, which keeps all the WAL commit code in a singly-responsible component. This also lets us layer on the WAL commit logic to the DML sink chain after replaying any existing WAL files, avoiding a circular WAL mess. The application-logic level WAL code abstracts over the underlying WAL implementation & codec through the WalAppender trait. This decouples the business logic from the WAL implementation both for testing purposes, and for trying different WAL implementations in the future.
2022-11-28 22:09:03 +00:00
wal = { version = "0.1.0", path = "../wal" }
chore: Rerun cargo hakari manage-deps
2023-02-24 18:02:23 +00:00
workspace-hack = { version = "0.1", path = "../workspace-hack" }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
[dev-dependencies]
assert_matches = "1.5.0"
test(ingester2): WAL replay benchmark This adds a simple WAL replay benchmark to ingester2 that executes a replay of a single line of LP. Unfortunately each file in the benches directory is compiled as it's own binary/crate, and as such is restricted to importing only "pub" types. This sucks, as it requires you to either benchmark at a high level (macro, not microbenchmarks - i.e. benchmarking the ingester startup, not just the WAL replay) or you are forced to mark the reliant types & functions as "pub", as well as all the other types/traits they reference in their signatures. Because the performance sensitive code is usually towards the lower end of the call stack, this can quickly lead to an explosion of "pub" types causing a large amount of internal code to be exported. Instead this commit uses a middle-ground; benchmarked types & fns are conditionally marked as "pub" iff the "benches" feature is enabled. This prevents them from being visible by default, but allows the benchmark function to call them. The benchmark itself is also restricted to only run when this feature is enabled.
2022-12-12 13:24:11 +00:00
criterion = { version = "0.4", default-features = false, features = ["async_tokio"]}
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
datafusion_util = { path = "../datafusion_util" }
test: add ingester2_test_ctx Moves the ingester2 TestContext & builder to its own crate for reuse between integration tests & benchmarks (and more?!) This allows us to DRY & reuse the test code with it's nice API for benchmarks too, without forcing it all to be in the "prod" build of ingester2 (and the dependencies, and their dependencies, etc). This also means this code can be built and cached instead of being rebuilt all the time during normal development.
2023-04-12 10:35:26 +00:00
influxdb_iox_client = { path = "../influxdb_iox_client" }
fix: Rename ingester2_test_ctx to ingester_test_ctx
2023-05-09 00:23:02 +00:00
ingester_test_ctx = { path = "../ingester_test_ctx" }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
lazy_static = "1.4.0"
mutable_batch_lp = { path = "../mutable_batch_lp" }
build: remove unused dependencies from crates This commit fixes loads of crates (47!) had unused dependencies, or mis-configured dependencies (test deps as normal deps). I added the "unused_crate_dependencies" to all crates to help prevent this mess from growing again! https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html This has the minor downside of false-positives when specifying dev-dependencies for test/bench binaries - these are files in /test or /benches (not normal tests). This commit includes a workaround, importing them in lib.rs (gated by a feature flag). I think the trade-off of better dependency management is worth it!
2023-05-22 13:08:20 +00:00
object_store = "0.5.6"
chore(deps): Bump paste from 1.0.11 to 1.0.12 (#7130) Bumps [paste](https://github.com/dtolnay/paste) from 1.0.11 to 1.0.12. - [Release notes](https://github.com/dtolnay/paste/releases) - [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12) --- updated-dependencies: - dependency-name: paste dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:40:41 +00:00
paste = "1.0.12"
chore(deps): Bump tempfile from 3.4.0 to 3.5.0 (#7376) * chore(deps): Bump tempfile from 3.4.0 to 3.5.0 Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.4.0 to 3.5.0. - [Release notes](https://github.com/Stebalien/tempfile/releases) - [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS) - [Commits](https://github.com/Stebalien/tempfile/commits) --- updated-dependencies: - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:15:54 +00:00
tempfile = "3.5.0"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
test_helpers = { path = "../test_helpers", features = ["future_timeout"] }
chore(deps): Bump tokio from 1.27.0 to 1.28.0 (#7667) * chore(deps): Bump tokio from 1.27.0 to 1.28.0 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.27.0 to 1.28.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
tokio = { version = "1.28", features = ["macros", "time", "test-util"] }
test(ingester2): WAL replay benchmark This adds a simple WAL replay benchmark to ingester2 that executes a replay of a single line of LP. Unfortunately each file in the benches directory is compiled as it's own binary/crate, and as such is restricted to importing only "pub" types. This sucks, as it requires you to either benchmark at a high level (macro, not microbenchmarks - i.e. benchmarking the ingester startup, not just the WAL replay) or you are forced to mark the reliant types & functions as "pub", as well as all the other types/traits they reference in their signatures. Because the performance sensitive code is usually towards the lower end of the call stack, this can quickly lead to an explosion of "pub" types causing a large amount of internal code to be exported. Instead this commit uses a middle-ground; benchmarked types & fns are conditionally marked as "pub" iff the "benches" feature is enabled. This prevents them from being visible by default, but allows the benchmark function to call them. The benchmark itself is also restricted to only run when this feature is enabled.
2022-12-12 13:24:11 +00:00
[features]
test: add ingester2_test_ctx Moves the ingester2 TestContext & builder to its own crate for reuse between integration tests & benchmarks (and more?!) This allows us to DRY & reuse the test code with it's nice API for benchmarks too, without forcing it all to be in the "prod" build of ingester2 (and the dependencies, and their dependencies, etc). This also means this code can be built and cached instead of being rebuilt all the time during normal development.
2023-04-12 10:35:26 +00:00
# Export some internal types for benchmark purposes only.
benches = ["test_helpers"]
test(ingester2): WAL replay benchmark This adds a simple WAL replay benchmark to ingester2 that executes a replay of a single line of LP. Unfortunately each file in the benches directory is compiled as it's own binary/crate, and as such is restricted to importing only "pub" types. This sucks, as it requires you to either benchmark at a high level (macro, not microbenchmarks - i.e. benchmarking the ingester startup, not just the WAL replay) or you are forced to mark the reliant types & functions as "pub", as well as all the other types/traits they reference in their signatures. Because the performance sensitive code is usually towards the lower end of the call stack, this can quickly lead to an explosion of "pub" types causing a large amount of internal code to be exported. Instead this commit uses a middle-ground; benchmarked types & fns are conditionally marked as "pub" iff the "benches" feature is enabled. This prevents them from being visible by default, but allows the benchmark function to call them. The benchmark itself is also restricted to only run when this feature is enabled.
2022-12-12 13:24:11 +00:00
[lib]
bench = false
[[bench]]
name = "wal"
harness = false
# Require some internal types be made visible for benchmark code.
required-features = ["benches"]
test(bench): ingester2 write benchmarks Adds a single-line & batched write request benchmarks to ingester2. When using the WAL, a floor of 10ms is added to all writes (the WAL linger time, configurable). With the WAL disabled in code, benchmarked writes complete in less than a millisecond: single row/write time: [1.9138 µs 1.9168 µs 1.9197 µs] thrpt: [520.92 Kelem/s 521.70 Kelem/s 522.51 Kelem/s] batched/write/1000 time: [129.15 µs 129.97 µs 131.28 µs] thrpt: [7.6173 Melem/s 7.6941 Melem/s 7.7429 Melem/s] Note these benchmarks exclude network I/O, and measure single-threaded, synchronous write client performance.
2023-04-12 10:38:05 +00:00
[[bench]]
name = "write"
harness = false
# Require some internal types be made visible for benchmark code.