Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/write_buffer/Cargo.toml

20 lines
452 B
TOML
Raw Normal View History

chore: move write buffer into its own crate
2021-07-14 09:39:49 +00:00
[package]
name = "write_buffer"
version = "0.1.0"
edition = "2018"
[dependencies]
async-trait = "0.1"
feat: add ingest timestamp to `Sequence` This allows us to track wall-clock ingest time for entries that we receive via write buffer (e.g. Kafka).
2021-07-26 14:37:18 +00:00
chrono = "0.4"
chore: move write buffer into its own crate
2021-07-14 09:39:49 +00:00
data_types = { path = "../data_types" }
entry = { path = "../entry" }
futures = "0.3"
refactor: use one stream per sequencer/partition Advantages are: - for large DBs w/ many partitions we can ingest data in-parallel - on top of this change we can implement per-sequencer seeking, which is required for replay
2021-07-19 08:34:58 +00:00
observability_deps = { path = "../observability_deps" }
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2. - [Release notes](https://github.com/Amanieu/parking_lot/releases) - [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md) - [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2) --- updated-dependencies: - dependency-name: parking_lot dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
parking_lot = "0.11.2"
chore: move write buffer into its own crate
2021-07-14 09:39:49 +00:00
rdkafka = "0.26.0"
refactor: use one stream per sequencer/partition Advantages are: - for large DBs w/ many partitions we can ingest data in-parallel - on top of this change we can implement per-sequencer seeking, which is required for replay
2021-07-19 08:34:58 +00:00
tokio = { version = "1.0", features = ["macros", "fs"] }
refactor: write buffer testing + better mocking This refactors the write buffer a bit for: - **Testing:** Add generic tests for the Kafka and the mocking implementation. The same interface can be used easily add new implementations (e.g. via Redis, filesystem, ...). - **Partition on Write:** The caller of the writer operation must now specify the partition/sequencer ID. The implicit partitioning of the Kafka writer would have lead to broken data since we must never spill entries w/ the same primary key over multiple partitions. At the moment we will only use partition 0 but we can easily implement better logic in the future. - **Improved Mocking:** The mocked implementation now simulates a system that feels more real. Especially the handling around multiple streams and "write while read" has been improved. This will be helpful for testing and for new features like seeking (during replay). A solid realistic mock also helps us to ensure that the tests using the mock do not rely on unrealistic behavior too much.
2021-07-15 12:02:49 +00:00
[dev-dependencies]
dotenv = "0.15.0"
uuid = { version = "0.8", features = ["serde", "v4"] }