Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/querier/Cargo.toml

48 lines
1.4 KiB
TOML
Raw Normal View History

feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
[package]
name = "querier"
version = "0.1.0"
edition = "2021"
[dependencies]
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 (#4119) * chore: update datafusion * chore(deps): Bump arrow from 10.0.0 to 11.0.0 Bumps [arrow](https://github.com/apache/arrow-rs) from 10.0.0 to 11.0.0. - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0) --- updated-dependencies: - dependency-name: arrow dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0 Bumps [arrow-flight](https://github.com/apache/arrow-rs) from 10.0.0 to 11.0.0. - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0) --- updated-dependencies: - dependency-name: arrow-flight dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update parquet to 11.0.0 * fix: error on create schema, test for same * fix: upgrade zstd * chore: Run cargo hakari tasks * fix: fix logical merge conflict * fix: hakari * fix: hakari * fix: update newly introduced dep Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
arrow = "11"
arrow-flight = "11"
chore(deps): Bump async-trait from 0.1.52 to 0.1.53 (#4141) Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.52 to 0.1.53. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.52...0.1.53) --- updated-dependencies: - dependency-name: async-trait dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:55:24 +00:00
async-trait = "0.1.53"
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
backoff = { path = "../backoff" }
feat: Querier able to send Flight queries to Ingester Fixes #3773.
2022-02-17 15:42:32 +00:00
bytes = "1.0"
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
client_util = { path = "../client_util" }
refactor: Extract a crate for shared data types in the new design
2022-02-25 15:10:23 +00:00
data_types2 = { path = "../data_types2" }
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
datafusion = { path = "../datafusion" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
futures = "0.3"
generated_types = { path = "../generated_types" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
hyper = "0.14"
iox_catalog = { path = "../iox_catalog" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
iox_object_store = { path = "../iox_object_store" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
metric = { path = "../metric" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
object_store = { path = "../object_store" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
observability_deps = { path = "../observability_deps" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
parking_lot = "0.12"
parquet_file = { path = "../parquet_file" }
test: querier shutdown and background task handling (#3881) This is similar to what we already have for the ingester.
2022-03-02 08:59:50 +00:00
pin-project = "1.0"
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
predicate = { path = "../predicate" }
feat: Querier able to send Flight queries to Ingester Fixes #3773.
2022-02-17 15:42:32 +00:00
prost = "0.9"
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
query = { path = "../query" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
rand = "0.8.3"
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd (#4106) * refactor: extract grpc service implementations out of influxdb_ioxd * chore: Run cargo hakari tasks * refactor: rename server_common to service_common * refactor: rename server_grpc_influxrpc to service_grpc_influxrpc * refactor: rename server_grpc_flight to service_grpc_flight * refactor: rename server_grpc_testing to service_grpc_testing * fix: Cargo.toml Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-23 20:14:45 +00:00
service_common = { path = "../service_common" }
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
schema = { path = "../schema" }
refactor: `querier` w/o `db` (#4063) * feat: `TombstoneRepo::list_by_table` * feat: `ParquetFileRepo::list_by_table_not_to_delete` * refactor: `querier` w/o `db` Get the `querier` to work w/o relying on `db`. A few notes: - Testing is kinda shallow, we really need to get `query_tests` working w/ `querier` (see #3934). - We still run a sync loop for namespaces, tables and schemas. This will be a replaced by "update namespace incl. tables and schemas on demand". Note however that we cannot fetch single tables and schemas on demand at the moment, because DataFusion doesn't implement async schema inspection (only `scan` / "give me all the chunks" is async). I think that's OK for now and we can address this later. - There is NO cache for parquet files and tombstones at the moment. For correctness, they need to be fetched in a single transaction (or we need a kinda tricky sequence number / logical clock tracking) and I am not sure yet how this makes sense when we have the ingester data wired up and predicates pushed down to the catalog (see next point). So let's measure first and then decide on a caching strategy for this. - Predicates are currently NOT pushed down to the catalog. I'll need to figure out how to extract time range from generic DataFusion expressions to make that work (it's easier for InfluxRPC queries, but they are not tested at the moment, see first point). Sorry that this commit is kinda huge. I initially planned to only migrate the chunks away from `db` and leave the tables and schemas for a follow-up PR, but the DataFusion trait structure (chunks are bound to their tables) makes this kinda pointless. Closes #3974. * docs: explain what we're doing Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * docs: mention tracking issues * docs: explain what we're doing Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-03-21 16:58:00 +00:00
snafu = "0.7"
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
thiserror = "1.0"
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
time = { path = "../time" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
tokio = { version = "1.17", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 (#4154) * chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.0 to 0.7.1. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1) --- updated-dependencies: - dependency-name: tokio-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
tokio-util = { version = "0.7.1" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
tonic = { version = "0.6" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
trace = { path = "../trace" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
uuid = { version = "0.8", features = ["v4"] }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
workspace-hack = { path = "../workspace-hack"}
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
[dev-dependencies]
feat: create CatalogChunk in querier (#3862) * feat: `NamespaceRepo::get_by_id` * feat: create `CatalogChunk` in querier
2022-02-28 17:20:38 +00:00
arrow_util = { path = "../arrow_util" }
bytes = "1.0"
refactor: move querier's test utils to a new crate to get reused by tests in other crates (#4013) * refactor: move querier's test utils to a new crate to be able resued by tests in other crates * chore: remove unused import
2022-03-10 18:17:58 +00:00
iox_tests = { path = "../iox_tests" }
feat: sync chunks in querier (#3911) * feat: `ParquetFileRepo::list_by_namespace_not_to_delete` * feat: `ChunkAddr: Clone` * test: ensure that querier keeps same partition objects * test: improve `create_parquet_file` flexibility * feat: sync chunks in querier * test: improve `test_parquet_file`
2022-03-04 08:53:39 +00:00
mutable_batch_lp = { path = "../mutable_batch_lp" }
test: add proptest for `AddressableHeap` (#4025) * test: add proptest for `AddressableHeap` For #3985. * refactor: simplify code Co-authored-by: Edd Robinson <me@edd.io> Co-authored-by: Edd Robinson <me@edd.io>
2022-03-14 12:38:30 +00:00
proptest = { version = "1", default_features = false, features = ["std"] }