Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/ingester2/Cargo.toml

54 lines
2.0 KiB
TOML
Raw Normal View History

feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
[package]
name = "ingester2"
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
arrow = { workspace = true, features = ["prettyprint"] }
chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` (#6279) * chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` * chore: Update thrift to 0.17 * fix: use workspace arrow-flight in ingester2 * chore: Update for API changes * fix: test * chore: Update hakari * chore: Update hakari again * chore: Update trace_exporters to latest thrift * fix: update test Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 14:12:30 +00:00
arrow-flight = { workspace = true }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
arrow_util = { version = "0.1.0", path = "../arrow_util" }
async-trait = "0.1.58"
backoff = { version = "0.1.0", path = "../backoff" }
feat(ingester2): gRPC methods & type-erased init This commit implements the gRPC direct-write RPC interface (largely copied from the ingester crate), and adds a much improved RPC query handler. Compared to the ingester crate, the query API is now split into two defined halves - the API handler side, and types necessary to support it (server/grpc/query.rs) and the Ingester query execution side (a stub in query/exec.rs). These two halves maintain a separation of concerns, and are interfaced by an abstract QueryExec trait (in query/trait.rs). I also added the catalog RPC interface as it is currently exposed on the ingester, though I am unsure if it is used by anything. This commit also introduces the "init" module, and the IngesterRpcInterface trait within it. This trait forms the public ingester2 crate API, defining the complete set of methods external crates can expect to utilise in a stable, unchanging and decoupled way. The IngesterRpcInterface trait also serves as a method of type-erasure on the underlying handler implementations, avoiding the need to expose/pub the types, abstractions, and internal implementation details of the ingester to external crates.
2022-11-25 10:58:32 +00:00
bytes = "1.3.0"
feat: operation timestamp sequencer Adds a TimestampOracle to provide an ingester-internal ordering to incoming DmlOperations using a logical clock.
2022-11-28 12:23:16 +00:00
crossbeam-utils = "0.8.14"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
data_types = { version = "0.1.0", path = "../data_types" }
datafusion.workspace = true
perf(ingester2): stream BufferTree partition data This commit implements the QueryExec trait for the BufferTree, allow it to be queried for the partition data it contains. With this change, the BufferTree now provides "read your writes" functionality. Notably the implementation streams the contents of individual partitions to the caller on demand (pull-based execution), deferring acquiring the partition lock until actually necessary and minimising the duration of time a strong reference to a specific RecordBatch is held in order to minimise the memory overhead. During query execution a client sees a consistent snapshot of partitions: once a client begins streaming the query response, incoming writes that create new partitions do not become visible. However incoming writes to an existing partition that forms part of the snapshot set become visible iff they are ordered before the acquisition of the partition lock when streaming that partition data to the client.
2022-11-25 14:35:39 +00:00
datafusion_util = { path = "../datafusion_util" }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
dml = { version = "0.1.0", path = "../dml" }
feat(ingester2): gRPC methods & type-erased init This commit implements the gRPC direct-write RPC interface (largely copied from the ingester crate), and adds a much improved RPC query handler. Compared to the ingester crate, the query API is now split into two defined halves - the API handler side, and types necessary to support it (server/grpc/query.rs) and the Ingester query execution side (a stub in query/exec.rs). These two halves maintain a separation of concerns, and are interfaced by an abstract QueryExec trait (in query/trait.rs). I also added the catalog RPC interface as it is currently exposed on the ingester, though I am unsure if it is used by anything. This commit also introduces the "init" module, and the IngesterRpcInterface trait within it. This trait forms the public ingester2 crate API, defining the complete set of methods external crates can expect to utilise in a stable, unchanging and decoupled way. The IngesterRpcInterface trait also serves as a method of type-erasure on the underlying handler implementations, avoiding the need to expose/pub the types, abstractions, and internal implementation details of the ingester to external crates.
2022-11-25 10:58:32 +00:00
flatbuffers = "22"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
futures = "0.3.25"
generated_types = { version = "0.1.0", path = "../generated_types" }
hashbrown.workspace = true
iox_catalog = { version = "0.1.0", path = "../iox_catalog" }
iox_query = { version = "0.1.0", path = "../iox_query" }
iox_time = { path = "../iox_time" }
metric = { version = "0.1.0", path = "../metric" }
mutable_batch = { version = "0.1.0", path = "../mutable_batch" }
mutable_batch_pb = { version = "0.1.0", path = "../mutable_batch_pb" }
object_store = "0.5.1"
observability_deps = { version = "0.1.0", path = "../observability_deps" }
once_cell = "1.16.0"
parking_lot = "0.12.1"
feat(ingester2): gRPC methods & type-erased init This commit implements the gRPC direct-write RPC interface (largely copied from the ingester crate), and adds a much improved RPC query handler. Compared to the ingester crate, the query API is now split into two defined halves - the API handler side, and types necessary to support it (server/grpc/query.rs) and the Ingester query execution side (a stub in query/exec.rs). These two halves maintain a separation of concerns, and are interfaced by an abstract QueryExec trait (in query/trait.rs). I also added the catalog RPC interface as it is currently exposed on the ingester, though I am unsure if it is used by anything. This commit also introduces the "init" module, and the IngesterRpcInterface trait within it. This trait forms the public ingester2 crate API, defining the complete set of methods external crates can expect to utilise in a stable, unchanging and decoupled way. The IngesterRpcInterface trait also serves as a method of type-erasure on the underlying handler implementations, avoiding the need to expose/pub the types, abstractions, and internal implementation details of the ingester to external crates.
2022-11-25 10:58:32 +00:00
pin-project = "1.0.12"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
predicate = { version = "0.1.0", path = "../predicate" }
feat(ingester2): gRPC methods & type-erased init This commit implements the gRPC direct-write RPC interface (largely copied from the ingester crate), and adds a much improved RPC query handler. Compared to the ingester crate, the query API is now split into two defined halves - the API handler side, and types necessary to support it (server/grpc/query.rs) and the Ingester query execution side (a stub in query/exec.rs). These two halves maintain a separation of concerns, and are interfaced by an abstract QueryExec trait (in query/trait.rs). I also added the catalog RPC interface as it is currently exposed on the ingester, though I am unsure if it is used by anything. This commit also introduces the "init" module, and the IngesterRpcInterface trait within it. This trait forms the public ingester2 crate API, defining the complete set of methods external crates can expect to utilise in a stable, unchanging and decoupled way. The IngesterRpcInterface trait also serves as a method of type-erasure on the underlying handler implementations, avoiding the need to expose/pub the types, abstractions, and internal implementation details of the ingester to external crates.
2022-11-25 10:58:32 +00:00
prost = { version = "0.11.2", default-features = false, features = ["std"] }
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
rand = "0.8.5"
schema = { version = "0.1.0", path = "../schema" }
service_grpc_catalog = { version = "0.1.0", path = "../service_grpc_catalog" }
thiserror = "1.0.37"
tokio = { version = "1.22", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
chore(deps): Bump tonic from 0.8.2 to 0.8.3 (#6249) Bumps [tonic](https://github.com/hyperium/tonic) from 0.8.2 to 0.8.3. - [Release notes](https://github.com/hyperium/tonic/releases) - [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/tonic/compare/v0.8.2...v0.8.3) --- updated-dependencies: - dependency-name: tonic dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 09:53:30 +00:00
tonic = "0.8.3"
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
trace = { version = "0.1.0", path = "../trace" }
uuid = "1.2.2"
chore: Run cargo hakari tasks
2022-11-24 14:51:21 +00:00
workspace-hack = { path = "../workspace-hack"}
feat(ingester): ingester2 init Adds an ingester2 crate to hold the MVP of the Kafkaless project. This was necessary due to the tight coupling of the ingester internals with tests in external crates, and eases the parallel development of two version of the ingester. This commit contains various changes from the "ingester" crate, mostly removing the concept/references to a "shard" or "ShardId" where possible. This commit does not copy over all of the "ingester" crate - only those components that are definitely needed. I will drag across more as functionality is implemented.
2022-11-24 14:26:34 +00:00
[dev-dependencies]
assert_matches = "1.5.0"
datafusion_util = { path = "../datafusion_util" }
lazy_static = "1.4.0"
mutable_batch_lp = { path = "../mutable_batch_lp" }
paste = "1.0.9"
test_helpers = { path = "../test_helpers", features = ["future_timeout"] }