Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/querier/Cargo.toml

53 lines
1.7 KiB
TOML
Raw Normal View History

feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
[package]
name = "querier"
feat: Use workspace package metadata to reduce differences and repetition
2022-09-26 14:43:00 +00:00
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
[dependencies]
chore: Update datafusion and arrow / parquet / arrow-flight 25.0.0 (#5900) * chore: Update datafusion and `arrow` / `parquet` / `arrow-flight` 25.0.0 * chore: Update for structure changes * chore: Update for new projection pushdown * chore: Run cargo hakari tasks * fix: fmt Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 20:58:47 +00:00
arrow = "25.0.0"
chore(deps): Bump async-trait from 0.1.57 to 0.1.58 (#5904) Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.57 to 0.1.58. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.57...0.1.58) --- updated-dependencies: - dependency-name: async-trait dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-19 09:40:26 +00:00
async-trait = "0.1.58"
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
backoff = { path = "../backoff" }
feat: querier object store cache (#5527) * feat: querier object store cache * docs: improve Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-09-02 09:48:53 +00:00
bytes = "1.2"
refactor: move `querier::cache_system` into its own crate (#4592)
2022-05-13 13:12:07 +00:00
cache_system = { path = "../cache_system" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
client_util = { path = "../client_util" }
fix: Rename data_types2 to data_types
2022-05-05 19:29:24 +00:00
data_types = { path = "../data_types" }
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
datafusion = { path = "../datafusion" }
feat: MVP include un-persisted results from the ingester in query results (#4255) * feat: Return not-yet-persisted data in query results * fix: comments from code review * fix: update for logical merge conflict Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-12 11:03:19 +00:00
datafusion_util = { path = "../datafusion_util" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
futures = "0.3"
generated_types = { path = "../generated_types" }
refactor: use a single flight client implementation (#4273) "end-user -> querier" and "querier -> ingester" should use a single Flight client implementation. The difference is just the request and response metadata. This changes our default Flight client to use protobuf instead of JSON for the ticket format.
2022-04-12 09:08:25 +00:00
influxdb_iox_client = { path = "../influxdb_iox_client" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
iox_catalog = { path = "../iox_catalog" }
metric = { path = "../metric" }
chore(deps): Bump object_store from 0.5.0 to 0.5.1 Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.0 to 0.5.1. - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md) - [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.0...object_store_0.5.1) --- updated-dependencies: - dependency-name: object_store dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2022-10-11 01:19:10 +00:00
object_store = "0.5.1"
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
observability_deps = { path = "../observability_deps" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
parking_lot = "0.12"
parquet_file = { path = "../parquet_file" }
test: querier shutdown and background task handling (#3881) This is similar to what we already have for the ingester.
2022-03-02 08:59:50 +00:00
pin-project = "1.0"
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
predicate = { path = "../predicate" }
ci: fix cargo deny (#4629) * ci: fix cargo deny * chore: downgrade `socket2`, version 0.4.5 was yanked * chore: rename `query` to `iox_query` `query` is already taken on crates.io and yanked and I am getting tired of working around that.
2022-05-18 09:38:35 +00:00
iox_query = { path = "../iox_query" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
rand = "0.8.3"
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd (#4106) * refactor: extract grpc service implementations out of influxdb_ioxd * chore: Run cargo hakari tasks * refactor: rename server_common to service_common * refactor: rename server_grpc_influxrpc to service_grpc_influxrpc * refactor: rename server_grpc_flight to service_grpc_flight * refactor: rename server_grpc_testing to service_grpc_testing * fix: Cargo.toml Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-23 20:14:45 +00:00
service_common = { path = "../service_common" }
feat: Interrogate schema from querier (as well as router) (#4557) * refactor: move SchemaService into `service_grpc_schema` * feat: implement schema gRPC for querier * chore: Run cargo hakari tasks Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-10 20:55:58 +00:00
service_grpc_schema = { path = "../service_grpc_schema" }
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
schema = { path = "../schema" }
feat: Make specifying the write buffer, and thus getting a sharder, optional in querier
2022-06-10 16:03:26 +00:00
sharder = { path = "../sharder" }
refactor: `querier` w/o `db` (#4063) * feat: `TombstoneRepo::list_by_table` * feat: `ParquetFileRepo::list_by_table_not_to_delete` * refactor: `querier` w/o `db` Get the `querier` to work w/o relying on `db`. A few notes: - Testing is kinda shallow, we really need to get `query_tests` working w/ `querier` (see #3934). - We still run a sync loop for namespaces, tables and schemas. This will be a replaced by "update namespace incl. tables and schemas on demand". Note however that we cannot fetch single tables and schemas on demand at the moment, because DataFusion doesn't implement async schema inspection (only `scan` / "give me all the chunks" is async). I think that's OK for now and we can address this later. - There is NO cache for parquet files and tombstones at the moment. For correctness, they need to be fetched in a single transaction (or we need a kinda tricky sequence number / logical clock tracking) and I am not sure yet how this makes sense when we have the ingester data wired up and predicates pushed down to the catalog (see next point). So let's measure first and then decide on a caching strategy for this. - Predicates are currently NOT pushed down to the catalog. I'll need to figure out how to extract time range from generic DataFusion expressions to make that work (it's easier for InfluxRPC queries, but they are not tested at the moment, see first point). Sorry that this commit is kinda huge. I initially planned to only migrate the chunks away from `db` and leave the tables and schemas for a follow-up PR, but the DataFusion trait structure (chunks are bound to their tables) makes this kinda pointless. Closes #3974. * docs: explain what we're doing Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * docs: mention tracking issues * docs: explain what we're doing Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-03-21 16:58:00 +00:00
snafu = "0.7"
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
thiserror = "1.0"
refactor: Rename time to iox_time (#4416) Signed-off-by: hi-rustin <rustin.liu@gmail.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
iox_time = { path = "../iox_time" }
ci: fix (#5569) * ci: use same feature set in `build_dev` and `build_release` * ci: also enable unstable tokio for `build_dev` * chore: update tokio to 1.21 (to fix console-subscriber 0.1.8 * fix: "must use"
2022-09-06 14:13:28 +00:00
tokio = { version = "1.21", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
chore(deps): Bump tokio-util from 0.7.3 to 0.7.4 (#5596) Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.3 to 0.7.4. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.3...tokio-util-0.7.4) --- updated-dependencies: - dependency-name: tokio-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 07:40:16 +00:00
tokio-util = { version = "0.7.4" }
chore: Update datafusion / arrow/parquet/arrow-flight and prost/tonic ecosystem (#5360) * chore: Update datafusion and arrow * chore: Update Cargo.lock * chore: update to Decimal128 * chore: Update tonic/prost/pbjson/etc * chore: Run cargo hakari tasks * fix: doctest in generated types Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 17:30:44 +00:00
tonic = { version = "0.8" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
trace = { path = "../trace" }
feat: expose query semaphore metrics (#4836) The groundwork for that was already done, just needed a bit of wiring. This might help us to judge timeouts.
2022-06-13 09:36:50 +00:00
tracker = { path = "../tracker" }
chore(deps): Bump sqlx to 0.6.0 and uuid to 1 (#4894) Closes #4889. Closes #4890. Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-17 10:28:28 +00:00
uuid = { version = "1", features = ["v4"] }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
workspace-hack = { path = "../workspace-hack"}
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
[dev-dependencies]
feat: create CatalogChunk in querier (#3862) * feat: `NamespaceRepo::get_by_id` * feat: create `CatalogChunk` in querier
2022-02-28 17:20:38 +00:00
arrow_util = { path = "../arrow_util" }
test: mock gRPC ingester response for querier Add infrastructure to test how the querier processes ingester gRPC responses w/o performing a full query or end2end test.
2022-04-14 08:33:32 +00:00
assert_matches = "1.5"
refactor: move querier's test utils to a new crate to get reused by tests in other crates (#4013) * refactor: move querier's test utils to a new crate to be able resued by tests in other crates * chore: remove unused import
2022-03-10 18:17:58 +00:00
iox_tests = { path = "../iox_tests" }
feat: sync chunks in querier (#3911) * feat: `ParquetFileRepo::list_by_namespace_not_to_delete` * feat: `ChunkAddr: Clone` * test: ensure that querier keeps same partition objects * test: improve `create_parquet_file` flexibility * feat: sync chunks in querier * test: improve `test_parquet_file`
2022-03-04 08:53:39 +00:00
mutable_batch_lp = { path = "../mutable_batch_lp" }
feat: querier object store cache (#5527) * feat: querier object store cache * docs: improve Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-09-02 09:48:53 +00:00
object_store_metrics = { path = "../object_store_metrics" }
test: Added explain tests for querier. Found and fixed #4468 (#4469) * test: Added explain tests for querier. Found and fixed #4468 * chore: cleanup * chore: Apply suggestions from code review Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-29 14:15:30 +00:00
test_helpers = { path = "../test_helpers" }