Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/querier/Cargo.toml

56 lines
1.9 KiB
TOML
Raw Normal View History

feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
[package]
name = "querier"
feat: Use workspace package metadata to reduce differences and repetition
2022-09-26 14:43:00 +00:00
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
[dependencies]
feat: Use workspace dep inheritance for the arrow crate
2022-10-24 17:58:38 +00:00
arrow = { workspace = true }
refactor: remove iox_arrow_flight use in `influxdb_iox_client ` and `querier` (#6624) Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 18:48:23 +00:00
arrow-flight = { workspace = true }
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 (#7374) * chore(deps): Bump async-trait from 0.1.66 to 0.1.68 Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.66 to 0.1.68. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68) --- updated-dependencies: - dependency-name: async-trait dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
async-trait = "0.1.68"
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
backoff = { path = "../backoff" }
chore(deps): Bump bytes from 1.3.0 to 1.4.0 Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/compare/v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-02-01 00:30:56 +00:00
bytes = "1.4"
refactor: move `querier::cache_system` into its own crate (#4592)
2022-05-13 13:12:07 +00:00
cache_system = { path = "../cache_system" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
client_util = { path = "../client_util" }
fix: Rename data_types2 to data_types
2022-05-05 19:29:24 +00:00
data_types = { path = "../data_types" }
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-24 17:47:45 +00:00
datafusion = { workspace = true }
feat: MVP include un-persisted results from the ingester in query results (#4255) * feat: Return not-yet-persisted data in query results * fix: comments from code review * fix: update for logical merge conflict Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-12 11:03:19 +00:00
datafusion_util = { path = "../datafusion_util" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
futures = "0.3"
refactor: use a single flight client implementation (#4273) "end-user -> querier" and "querier -> ingester" should use a single Flight client implementation. The difference is just the request and response metadata. This changes our default Flight client to use protobuf instead of JSON for the ticket format.
2022-04-12 09:08:25 +00:00
influxdb_iox_client = { path = "../influxdb_iox_client" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
iox_catalog = { path = "../iox_catalog" }
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
iox_query = { path = "../iox_query" }
iox_time = { path = "../iox_time" }
fix: Move ingester-querier gRPC communication to its own crate
2023-05-12 15:35:29 +00:00
ingester_query_grpc = { path = "../ingester_query_grpc" }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
metric = { path = "../metric" }
chore: use workspace dependencies for `object_store`
2023-05-26 11:03:42 +00:00
object_store = { workspace = true }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
observability_deps = { path = "../observability_deps" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
parking_lot = "0.12"
parquet_file = { path = "../parquet_file" }
chore(deps): Bump pin-project from 1.0.12 to 1.1.0 Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.0.12 to 1.1.0. - [Release notes](https://github.com/taiki-e/pin-project/releases) - [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md) - [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.12...v1.1.0) --- updated-dependencies: - dependency-name: pin-project dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-05-15 02:02:32 +00:00
pin-project = "1.1"
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
predicate = { path = "../predicate" }
refactor: remove iox_arrow_flight use in `influxdb_iox_client ` and `querier` (#6624) Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 18:48:23 +00:00
prost = { version = "0.11" }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
rand = "0.8.3"
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd (#4106) * refactor: extract grpc service implementations out of influxdb_ioxd * chore: Run cargo hakari tasks * refactor: rename server_common to service_common * refactor: rename server_grpc_influxrpc to service_grpc_influxrpc * refactor: rename server_grpc_flight to service_grpc_flight * refactor: rename server_grpc_testing to service_grpc_testing * fix: Cargo.toml Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-23 20:14:45 +00:00
service_common = { path = "../service_common" }
feat: wire up flight into the querier (#3889)
2022-03-02 09:20:26 +00:00
schema = { path = "../schema" }
refactor: `querier` w/o `db` (#4063) * feat: `TombstoneRepo::list_by_table` * feat: `ParquetFileRepo::list_by_table_not_to_delete` * refactor: `querier` w/o `db` Get the `querier` to work w/o relying on `db`. A few notes: - Testing is kinda shallow, we really need to get `query_tests` working w/ `querier` (see #3934). - We still run a sync loop for namespaces, tables and schemas. This will be a replaced by "update namespace incl. tables and schemas on demand". Note however that we cannot fetch single tables and schemas on demand at the moment, because DataFusion doesn't implement async schema inspection (only `scan` / "give me all the chunks" is async). I think that's OK for now and we can address this later. - There is NO cache for parquet files and tombstones at the moment. For correctness, they need to be fetched in a single transaction (or we need a kinda tricky sequence number / logical clock tracking) and I am not sure yet how this makes sense when we have the ingester data wired up and predicates pushed down to the catalog (see next point). So let's measure first and then decide on a caching strategy for this. - Predicates are currently NOT pushed down to the catalog. I'll need to figure out how to extract time range from generic DataFusion expressions to make that work (it's easier for InfluxRPC queries, but they are not tested at the moment, see first point). Sorry that this commit is kinda huge. I initially planned to only migrate the chunks away from `db` and leave the tables and schemas for a follow-up PR, but the DataFusion trait structure (chunks are bound to their tables) makes this kinda pointless. Closes #3974. * docs: explain what we're doing Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * docs: mention tracking issues * docs: explain what we're doing Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-03-21 16:58:00 +00:00
snafu = "0.7"
chore(deps): Bump tokio from 1.28.2 to 1.29.0 (#8100) Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.28.2 to 1.29.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 13:18:08 +00:00
tokio = { version = "1.29", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
chore(deps): Bump tokio-util from 0.7.7 to 0.7.8 (#7665) Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.7 to 0.7.8. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.7...tokio-util-0.7.8) --- updated-dependencies: - dependency-name: tokio-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 09:24:39 +00:00
tokio-util = { version = "0.7.8" }
chore: Update DataFusion + arrow ecosystem to 37 (#7544) * chore: Update datafusion and arrow/parquet to 37, tonic to 0.9.1 * refactor: Update for FieldRef and other API changes * fix: Update field size calculation * fix: Use `NullBuffer` directly * fix: remove outdated comment * chore: Update test for tonic * chore: Run cargo hakari tasks * chore: cargo update --------- Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-14 12:43:01 +00:00
tonic = { workspace = true }
feat: skeleton of querier CLI (#3843) * feat: skeleton of querier CLI * chore: wrap metrics in opt&arc in querier to satisfy new api * chore: derive debug in querier handler * chore: add join handles and their shutdown to nascent querier server * chore: querier server http unimpl -> 404 * fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
trace = { path = "../trace" }
refactor: Extract common ArrowFlight client into iox_arrow_flight (#6427) * refactor: Extract common ArrowFlight client into iox_arrow_flight * chore: Run cargo hakari tasks * fix: clarify intent of iox_arrow_flight crate * refactor: Apply suggestions from code review Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> * fix: loop --> while let * fix: REmove make_tonic_error in favor of From impl Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
trace_exporters = { path = "../trace_exporters" }
trace_http = { path = "../trace_http" }
feat: expose query semaphore metrics (#4836) The groundwork for that was already done, just needed a bit of wiring. This might help us to judge timeouts.
2022-06-13 09:36:50 +00:00
tracker = { path = "../tracker" }
chore(deps): Bump sqlx to 0.6.0 and uuid to 1 (#4894) Closes #4889. Closes #4890. Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-17 10:28:28 +00:00
uuid = { version = "1", features = ["v4"] }
chore: Rerun cargo hakari manage-deps
2023-02-24 18:02:23 +00:00
workspace-hack = { version = "0.1", path = "../workspace-hack" }
feat: create `ParquetChunk` in querier (#3857) Adds a small adapter that is able to produce `ParquetChunk`s for NG. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
[dev-dependencies]
feat: create CatalogChunk in querier (#3862) * feat: `NamespaceRepo::get_by_id` * feat: create `CatalogChunk` in querier
2022-02-28 17:20:38 +00:00
arrow_util = { path = "../arrow_util" }
test: mock gRPC ingester response for querier Add infrastructure to test how the querier processes ingester gRPC responses w/o performing a full query or end2end test.
2022-04-14 08:33:32 +00:00
assert_matches = "1.5"
feat: cache partition template in querier (#7987) * feat: impl `Eq` for `TablePartitionTemplateOverride` * feat: `TablePartitionTemplateOverride::size` * feat: cache partition template in querier Required for #7974.
2023-06-15 10:30:56 +00:00
generated_types = { path = "../generated_types" }
chore(deps): Bump insta from 1.29.0 to 1.30.0 (#8059) Bumps [insta](https://github.com/mitsuhiko/insta) from 1.29.0 to 1.30.0. - [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md) - [Commits](https://github.com/mitsuhiko/insta/compare/1.29.0...1.30.0) --- updated-dependencies: - dependency-name: insta dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-23 07:45:41 +00:00
insta = { version = "1.30.0", features = ["yaml"] }
refactor: move querier's test utils to a new crate to get reused by tests in other crates (#4013) * refactor: move querier's test utils to a new crate to be able resued by tests in other crates * chore: remove unused import
2022-03-10 18:17:58 +00:00
iox_tests = { path = "../iox_tests" }
feat: sync chunks in querier (#3911) * feat: `ParquetFileRepo::list_by_namespace_not_to_delete` * feat: `ChunkAddr: Clone` * test: ensure that querier keeps same partition objects * test: improve `create_parquet_file` flexibility * feat: sync chunks in querier * test: improve `test_parquet_file`
2022-03-04 08:53:39 +00:00
mutable_batch_lp = { path = "../mutable_batch_lp" }
feat: querier object store cache (#5527) * feat: querier object store cache * docs: improve Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-09-02 09:48:53 +00:00
object_store_metrics = { path = "../object_store_metrics" }
test: Added explain tests for querier. Found and fixed #4468 (#4469) * test: Added explain tests for querier. Found and fixed #4468 * chore: cleanup * chore: Apply suggestions from code review Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-29 14:15:30 +00:00
test_helpers = { path = "../test_helpers" }