3ebd07358b
chore: Update DataFusion pin, upgrade `date_bin` and `InfluxQL` to use `Interval(MonthDayNano)` ( #7516 )
...
* chore: Update datafusion
* chore: Update for change in PhysicalSortExpr
* refactor: Update date_bin_gapfill to take IntervalMonthDayNano, fix FlightSQL
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 10:43:32 +00:00
53ddca45d8
Merge branch 'main' into cn/remove-write-summary
2023-04-12 16:07:35 +00:00
20e9c91866
refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc ( #7515 )
...
* refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:07:19 +00:00
6387a9576a
fix: Remove the write_summary crate and write info service
2023-04-12 11:31:23 -04:00
65c90b0281
fix: Remove the write buffer crate
2023-04-12 11:01:26 -04:00
02c7ec0727
test: add ingester2_test_ctx
...
Moves the ingester2 TestContext & builder to its own crate for reuse
between integration tests & benchmarks (and more?!)
This allows us to DRY & reuse the test code with it's nice API for
benchmarks too, without forcing it all to be in the "prod" build of
ingester2 (and the dependencies, and their dependencies, etc).
This also means this code can be built and cached instead of being
rebuilt all the time during normal development.
2023-04-12 12:51:47 +02:00
6c2ce01f1e
fix: Remove old ingester and ioxd_ingester
2023-04-07 11:06:37 -04:00
a5769a461b
fix: Remove old compactor and ioxd_compactor
2023-04-06 16:49:52 -04:00
e8b7d69b0f
chore: Update datafusion again ( #7442 )
...
* chore: Update datafusion
* chore: Fix up plans for datafusion API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-05 18:21:53 +00:00
badc8865ef
chore: Update datafusion again ( #7440 )
...
* chore: Update DataFusion
* chore: Update for new API
* chore: Run cargo hakari tasks
* fix: cargo doc
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-04 15:45:46 +00:00
d8b0139ea9
chore: Update datafusion + arrow/parquet/arrow-flight to 36 ( #7354 )
...
* chore: Update datafusion + arrow/parquet/arrow-flight to 36
* refactor: update optimize for new API
* refactor: update parquet for new API
* chore: Update more dependencies
* chore: Update to use the new buffer creation APIs
* chore: Run cargo hakari tasks
* fix: bad len
* fix: update for API change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 13:41:59 +00:00
43e236e040
chore: Update datafusion again ( #7353 )
...
* chore: Update DataFusion
* refactor: Update predicate crate for new transform API
* refactor: Update iox_query crate for new APIs
* refactor: Update influxql for new API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 16:21:49 +00:00
1817104236
chore: Update DataFusion again ( #7329 )
...
* chore: update datafusion
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:45:53 +00:00
5dd71998a1
chore: Update datafusion ( #7318 )
...
* chore: Update datafusion
* chore: Update for API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 15:07:23 +00:00
6660b86f6c
fix: Put authorization header in request extension
2023-03-22 04:12:07 +01:00
8ab553180e
chore: Update datafusion again ( #7276 )
...
* chore: Update DataFusion pin
* chore: Update for new Expr type
* fix: deprecated
* chore: Run cargo hakari tasks
* docs: Update predicate/src/lib.rs
* fix: fmt
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-21 18:25:04 +00:00
96c2094302
refactor(iox_query): extract influxrpc planner to its own crate ( #7241 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-17 18:48:55 +00:00
13657d5bcc
feat(authz): authorization service client and write integration ( #7216 )
...
* feat(authz): add authorization client.
Add a new authz crate to provide the interface for making authorization
checks from within IOx. This includes the default client that uses
the influxdata.iox.authz.v1 gRPC protocol. This feature is not used
by any IOx component yet.
* feat: optional authorization on write path
Support optionally enabling authorization checks on the /api/v2/write
handler. If an authrorizer is configured then the handler will
attempt to retrieve a token from the request's Authorization header.
If no such token exists then a response with a 401 error code is
returned. If the token is not valid, or does not have write permission
for the requested namespace then a response with a 403 error is
returned.
* chore: add unit test for authz in write handler
Add unit tests that test the correct functioning of the /api/v2/write
handler when an Authorizer is configured.
* chore(authz): use lazy connection
Change the initialization of the authz client to use a lazy connection.
This allows the client to be initialised synchronously.
* chore: Run cargo hakari tasks
* fix(authz): protolint complaints
* fix: authz tests
* fix: benches and lint
* chore: Update clap_blocks/src/authz.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: Update authz/src/lib.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: Update clap_blocks/src/authz.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: review suggestions
* chore: review suggestions
Apply a number of suggestions from review comments. The main
behavioural change is that if the authz service is configured
applictions will perform a probe request to ensure it can communicate
before continuing startup.
* chore: Update router/src/server/http.rs
Co-authored-by: Dom <dom@itsallbroken.com>
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-17 15:20:14 +00:00
7dfaa05e8a
chore: Update datafusion again ( #7208 )
...
* chore: update datafusion again
* fix: update test
* fix: use table_reference
* fix: clean up import
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 14:34:40 +00:00
35bd5ccd13
chore: Update datafusion pin ( #7207 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-15 11:35:15 +00:00
0eb858c70d
chore: Update datafusion ( #7167 )
...
* chore: Update datafusion
* chore: Update datafusion
* refactor: use UserDefinedLogicalNodeCore
* fix: remove stray comment
* fix: clippy
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 16:41:32 +00:00
fe48a685ec
refactor: Move InfluxQL behaviour from iox_query to new crate ( #7156 )
...
* refactor: Break unnecessary dependencies from `iox_query` crate
In the process, the test code has been simplified.
* refactor: Move InfluxQL plan module to iox_query_influxql crate
* refactor: Move remaining behaviour from iox_query to iox_query_influxql
* chore: rustfmt 🙄
I was under the impression `clippy` would catch formatting
2023-03-08 22:29:20 +00:00
3828d2a50e
chore: update DataFusion to `deeaa5632ed99a58b91767261570756db736d158` ( #7158 )
...
* chore: update DataFusion to `deeaa5632ed99a58b91767261570756db736d158`
I want to get pull:
- https://github.com/apache/arrow-datafusion/pull/5495
Changes in the IOx code base due to:
- https://github.com/apache/arrow-datafusion/pull/5423
- https://github.com/apache/arrow-datafusion/pull/5421
- https://github.com/apache/arrow-datafusion/pull/5450
* refactor: simplify expression simplifcation
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: remove upstreamed code
* test: update snapshots
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-03-08 13:05:31 +00:00
ed0704ac8d
chore: Update datafusion ( #7100 )
...
* chore: Update datafusion
* chore: iox_query to compile for API changes + update tests
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-06 17:59:24 +00:00
e19ce98407
chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0 ( #7084 )
...
* chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0
* chore: Run cargo hakari tasks
* chore: Update plans
* chore: Update querier expected output
* chore: Update querier tests to use insta
* fix: sort output too
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-01 11:25:01 +00:00
f93baf7693
chore: Update DataFusion and `arrow` / `arrow-flight` / `parquet` to `33.0.0` ( #7045 )
...
* chore: Update DataFusion and arrow/arrow-flight/parquet to 33.0.0
* fix: Update test output
* fix: update more test output
* fix: Update querier test output
* chore: Run cargo hakari tasks
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting for selector tests
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: Christopher Wolff <chris.wolff@influxdata.com>
2023-02-22 21:24:20 +00:00
27890b313f
chore: Update datafusion ( #6997 )
...
* chore: Update datafusion
* chore: update the plans
* fix: update some plans
* chore: Update plans and port some explain plans to use insta snapshots
* fix: another plan
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 17:03:25 +00:00
4d7aa1e48b
refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test ( #6960 )
...
* refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test
* fix: Update compactor2/src/components/mod.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:06:42 +00:00
143cb26e87
chore: Update datafusion again ( #6958 )
...
* chore: Update datafusion pin
* chore: Update plans
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 10:28:44 +00:00
501032c2fb
chore: Update datafusion again ( #6929 )
...
* chore: Update DataFusion
* fix: disable repartitioned sorting
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 11:51:47 +00:00
2f4d901fbe
chore: Update datafusion ( #6893 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: update for api change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 18:27:18 +00:00
ae10b33f0c
chore: Update datafusion ( #6868 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update error message
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-06 14:22:45 +00:00
d3601a59f8
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` ( #6756 )
...
* chore: update DataFusion
* fix: test
* chore: format
* chore: clippy
* chore: update arrow
* chore: arrow upgrade fallout
* chore: Run cargo hakari tasks
* chore: remove failing warm compaction test
* fix: flight error propagation
* chore: update parquet size
* fix: Update error message
* chore: Update parquet metadata test
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:35:39 +00:00
b55e4ffa21
test: Port table_schema tests to end-to-end and get rid of query_tests ( #6828 )
...
* test: Port table_schema query_tests to end-to-end
* fix: Remove now outdated query_tests crate
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 17:41:08 +00:00
5b14caa780
chore: Update DataFusion ( #6753 )
...
* chore: Update datafusion
* fix: Update for changes
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-30 14:48:52 +00:00
0d32662eea
chore: Update datafusion again ( #6722 )
...
* chore: Update datafusion
* fix: Update for API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-27 18:59:27 +00:00
c3bc61f10e
refactor: Move `flightsql` code into its own module, add docs and tests ( #6640 )
...
* refactor: Move `flightsql` code into its own module
* fix: get schema from LogicalPlan
* refactor: use arrow_flight::sql::Any instead of prost_types::any
* fix: cleanup docs and avoid as_ref
* fix: Use Bytes
* fix: use Any::pack
* fix: doclink
2023-01-24 18:24:32 +00:00
9a61f36a53
chore: Update datafusion again ( #6656 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-23 12:13:07 +00:00
b09691dc6b
chore: Upgrade datafusion (again, I know) ( #6639 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-23 11:24:22 +00:00
24ca1e6f8c
Merge branch 'main' into cn/test-old-with-new
2023-01-20 15:20:40 +00:00
d808c57cdc
chore: Remove `iox_arrow_flight` ( #6621 )
...
* chore: Remove iox_arrow_flight
* fix: hack around tonic status errors
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 11:16:19 +00:00
6afd782b3f
fix: Move query_tests2 into influxdb_iox/tests so that the code rebuilds
2023-01-19 16:44:31 -05:00
f639bf3e23
chore: refactor ingester to use upstream arrow-flight ( #6622 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 15:16:13 +00:00
6a4fa11dba
chore: Update to latest datafusion / arrow `31.0.0` ( #6618 )
...
* chore: Update datafusion and arrow/parquet/arrow-flight `31.0.0`
* chore: Update for new API
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 20:21:10 +00:00
33168b97f0
Merge branch 'main' into cn/query-tests-grpc
2023-01-18 19:03:51 +00:00
e72173d58d
feat: very basic compactor2 skeleton ( #6614 )
...
Sets up crate and wires up the main binary. No tests yet, no algorithm
framework, just the bare minimum.
Also I decided to not offer a gRPC server in `compactor2` at the moment
and hence did not implement any handle/delegate infrastructure. We add
this later if we need it. This also means compactor2 does NOT provide a
catalog service for now.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
8410998408
chore: Update datafusion to Jan 17, 2023 (2 / 2) and arrow/parquet `30.0.1` ( #6604 )
...
* chore: Update datafusion to Jan 9, 2023 (2 / 2) and arrow/parquet `30.0.1`
* chore: Update for changes in arrow ipc
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-18 15:51:24 +00:00
f3b5dcaab7
feat: Reimagining query_tests
2023-01-18 10:24:17 -05:00
57f08dbccd
chore: Update datafusion to Jan 9, 2023 (1 / 2) ( #6603 )
...
* refactor: Update DataFusion pin to early Jan 2023
* fix: Update tests now that planning is async
* fix: Updates for API changes
* chore: Run cargo hakari tasks
* fix: Update comment
* refactor: nicer config setup
* fix: gapfill async
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-18 12:19:32 +00:00
cc6d8709c9
chore(deps): Bump hashbrown from 0.13.1 to 0.13.2 ( #6574 )
...
Bumps [hashbrown](https://github.com/rust-lang/hashbrown ) from 0.13.1 to 0.13.2.
- [Release notes](https://github.com/rust-lang/hashbrown/releases )
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.1...v0.13.2 )
---
updated-dependencies:
- dependency-name: hashbrown
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-12 09:47:15 +00:00
dbe52f1ca1
chore: Upgrade datafusion ( #6467 )
...
* chore: Update datafusion
* fix: Update for new apis
* chore: Update expected plan
* fix: Update for new config construction
* chore: update clippy
* fix: Fix error codes
* fix: update another test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
d0d5906476
chore: Update datafusion pin ( #6442 )
...
* chore: Update datafusion pin
* refactor: Update iox_query for new apis
* chore: Update some more apis
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-19 20:02:42 +00:00
94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight ( #6427 )
...
* refactor: Extract common ArrowFlight client into iox_arrow_flight
* chore: Run cargo hakari tasks
* fix: clarify intent of iox_arrow_flight crate
* refactor: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: loop --> while let
* fix: REmove make_tonic_error in favor of From impl
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
be45889be1
chore: Upgrade datafusion ( #6407 )
...
* chore: Update datafusion
* chore: Update for API change
2022-12-15 06:51:35 +00:00
8729977851
chore: Upgrade datafusion / arrow to 29.0.0 to get flightsql client ( #6396 )
...
* chore: Update datafusion pin
* chore: Update for API change
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-13 20:16:09 +00:00
336ca761a3
chore: Update datafusion pin (to get sqlparser update) ( #6378 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-12 14:39:42 +00:00
e0ecacf6cc
chore: Update DataFusion (get median fix and automatic string to timestamp coercion) ( #6363 )
...
* chore: Update DataFusion pin to get median fix
* chore: Update for new Expr node
* test: add test for median
* test: add test for coercion of strings to timestamps
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 12:14:00 +00:00
061d582a9b
chore: patch object_store to get apache#3274 ( #6362 )
...
* chore: patch object_store to get apache#3274
* chore: Run cargo hakari tasks
* fix: add issue breadcrumb
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 14:52:25 +00:00
9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers ( #6349 )
...
* test: add tests for periods in measurement names
* chore: Update Datafusion
* chore: Update for changed APIs
* chore: Update expected plan output
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 11:27:13 +00:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
d0f1f6a4fd
chore: Upgrade datafusion to get memory limits ( #6297 )
...
* chore: Update datafusion
* fix: use correctly qualified column names
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 16:40:26 +00:00
99698d0f46
feat: Create an ioxd_ingester2 crate
2022-12-01 11:37:21 -05:00
14a9bc92e9
Revert "Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )" ( #6296 )
...
This reverts commit b7e52c0d8d
2022-12-01 14:20:43 +00:00
b7e52c0d8d
Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )
...
This reverts commit 039a45ddd1
2022-12-01 11:38:42 +00:00
039a45ddd1
chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )
...
* chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0`
* chore: Update thrift to 0.17
* fix: use workspace arrow-flight in ingester2
* chore: Update for API changes
* fix: test
* chore: Update hakari
* chore: Update hakari again
* chore: Update trace_exporters to latest thrift
* fix: update test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 14:12:30 +00:00
f22b1e1a09
chore: Update datafusion (to get memory limiting code) ( #6246 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update to newer api
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 12:05:17 +00:00
fc5697b8e7
chore: Update datafusion again (N of N) ( #6218 )
...
* chore: Update datafusion again (4 of N)
* fix: Update plans
* fix: Update for renamed API
* fix: Update more plans
* chore: Update to datafusion @ d355f69aae2cc951cfd021e5c0b690861ba0c4ac
* fix: update explain plan tests
* fix: update test after schema error
* chore: Update datafusion again
* fix: Add size() calculation to selectors
* chore: Run cargo hakari tasks
* fix: Update newly added test
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-28 17:09:40 +00:00
a66fc0b645
feat(ingester): ingester2 init
...
Adds an ingester2 crate to hold the MVP of the Kafkaless project.
This was necessary due to the tight coupling of the ingester internals
with tests in external crates, and eases the parallel development of two
version of the ingester.
This commit contains various changes from the "ingester" crate, mostly
removing the concept/references to a "shard" or "ShardId" where
possible.
This commit does not copy over all of the "ingester" crate - only those
components that are definitely needed. I will drag across more as
functionality is implemented.
2022-11-24 15:34:02 +01:00
db63dc55a2
feat: Initial WAL implementation
2022-11-23 17:07:43 -05:00
71b29ae442
chore: Update DataFusion (3 of N) and arrow/parquet to 27.0.0 ( #6208 )
...
* chore: Update DataFusion and arrow/parquet to 27.0.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-23 14:21:47 +00:00
9fb1de0428
chore: Update datafusion (2 of N) right before arrow 27 upgrade ( #6207 )
...
* chore: Update datafusion (2 of N) right before arrow 27 upgrade
* fix: Update tests for better unsigned pushdown
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-23 11:04:14 +00:00
1a1ea74cb7
chore: Upgrade datafusion again ( #6160 )
...
* Revert "Revert "chore: Update datafusion again (#6108 )""
This reverts commit 766b3bbeb440618cfe332f6ee7d4f8a8217acc48.
* fix: Respect the partition sort key
* chore: update plans
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 19:28:26 +00:00
67712b595c
Revert "chore: Update datafusion again ( #6108 )" ( #6159 )
...
This reverts commit fbe9f27f10
2022-11-16 21:14:55 +00:00
9365d933f1
chore: router namespace api ( #6151 )
...
* chore: move ns api from querier to router
* chore: add explanatory comment in querier about moved namespace API
* fix: add namespace service to router
* fix: querier returns unimplemented error for ns retention, not panic
* chore: reuse namespace -> proto in router ns api
* chore: grpc namespace - consume ns to avoid clone
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 15:25:49 +00:00
fbe9f27f10
chore: Update datafusion again ( #6108 )
...
* chore: Update datafusion pin + api code
* chore: Run cargo hakari tasks
* refactor: combine_sort_key is more idomatic and add rationale comments
* refactor: satisfy borrow checker and updated comments
* fix: Add test case for combine_sort_key
* fix: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* fix: Add back test for deeply nested expression
* fix: Update output ordering
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 14:41:52 +00:00
cc17e5a54b
refactor: use a workspace dependency for hashbrown
2022-11-11 13:25:39 -05:00
034d9b371d
chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0` ( #6061 )
...
* chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0`
* fix: Update query_functions
* fix: update for TimestampNanosecondArray API changes
* fix: update for TimestampNanosecondArray API changes
* chore: Update flatbuffers and remove rustsec warning
* chore: Update text
* fix: update more test
* fix: Lock ahash to exactly 0.8.0
* fix: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-07 11:01:58 +00:00
8c8e607dca
chore: Update datafusion pin ( #6054 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
* chore: Update expected error
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-03 19:41:31 +00:00
e7e9945207
chore: Update datafusion pin ( #6017 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 11:12:21 +00:00
00953460fb
chore: Update datafusion pin ( #6010 )
2022-10-31 17:14:56 +00:00
ace3c11f12
chore: Update datafusion ( #6004 )
...
* chore: Update datafusion
* chore: change path
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-31 16:16:28 +00:00
b27e2bd7d1
feat: Use workspace dep inheritance for the parquet crate
2022-10-26 10:37:51 -04:00
88c3a1f5e7
feat: Use workspace dep inheritance for the arrow-flight crate
2022-10-26 10:34:54 -04:00
3145e2c05b
feat: Use workspace dep inheritance for the arrow crate
2022-10-26 10:34:29 -04:00
44936f661a
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-26 10:33:56 -04:00
99f36f8889
chore: unpatch clap V3 ( #5971 )
...
Upstream issue was solved by a new release.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-25 09:16:16 +00:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
4ca869fcd9
fix: textwrap 0.15.1 was yanked ( #5953 )
...
Need to pull an unreleased clap v3 so we no longer depend on a yanked
version of textwrap. Also see https://github.com/clap-rs/clap/issues/4418
2022-10-24 07:19:51 +00:00
76d3fbbe53
refactor: remove old `packers` crate ( #5893 )
...
* refactor: remove old packers code
* refactor: Remove from workspace
2022-10-18 19:00:28 +00:00
9b7a7b5e40
refactor: remove read buffer ( #5891 )
2022-10-18 12:46:17 +00:00
81722dc19b
feat: AST traversal using Visitor pattern ( #5796 )
...
* feat: Partition implementation of Visitable for InfluxQL AST
* feat: Added consistent structures for each clause to simplify visitor
Continued to expand `accept` and `pre` / `post` visit implementations.
* feat: Added insta and tests using snapshots (thanks @crepererum)
The insta crate simplifies the process of validating the combination of
visitor and accept implementations are called and in the correct order.
* chore: Run cargo hakari tasks
* feat: Added remaining snapshot tests
Some tests are failing as some minor type changes must be added along
with the addition of related visitor functions.
* feat: Add types to represent each clause in numerous statements
These clauses permit distinct visit functions on the `Visitor` type.
* chore: Reformat `SELECT`
* chore: Explicitly specify access to export selected types only
This required completing all the missing documentation for the exported
types.
* chore: Update Cargo.lock
* chore: macro to implement common traits and hide 0th tuple element
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-10-13 22:37:49 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol ( #5734 )
...
* feat: add `influxdb_iox debug parquet_to_lp` command
* chore: Run cargo hakari tasks
* fix: update command description
* fix: remove unecessary Result import
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-26 14:17:27 +00:00
f86d3e31da
chore: Update datafusion + object_store ( #5619 )
...
* chore: Update datafusion pin
* chore: update object_store to 0.5.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-13 12:34:54 +00:00
15369d4c52
feat: Implement gRPC binary logger ( #5473 )
...
* feat: Implement gRPC binary logger
* chore: Run cargo hakari tasks
* fix: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-01 10:57:02 +00:00
711ba77341
chore: update object_store to test IMDSv1 fallback ( #5509 )
...
* chore: update object_store to test IMDSv1 fallback
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-30 12:31:49 +00:00
66f0b59dbb
revert: remove Azure SDK / bump object_store
...
This reverts commit c2f8efa03a
2022-08-30 10:41:29 +02:00
c2f8efa03a
build: remove Azure SDK / bump object_store
...
Bumps the object_store pin to master to pick up:
https://github.com/apache/arrow-rs/pull/2509
This removes the Azure SDK.
2022-08-29 14:06:51 +02:00
b4e5895d7a
feat: Add influxdb_influxql_parser crate ( #5415 )
...
* feat: Add crate; parse quoted identifiers
* chore: Run cargo hakari tasks
* chore: satisfy linter
* chore: Use `test_helpers::Result`
* feat: Add all InfluxQL keywords
* chore: Update influxdb_influxql_parser/src/lib.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* chore: PR feedback
* chore: PR Feedback, remove Result<()>
* chore: Update Cargo.lock
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-08-18 23:09:45 +00:00
Andrew Lamb
kodiakhq[bot]
Carol (Nichols || Goulding)
Dom Dwyer
Marko Mikulicic
Martin Hilton
Stuart Carnie
Marco Neumann
Raphael Taylor-Davies
Carol (Nichols || Goulding)
dependabot[bot]
Paul Dix
Luke Bond
Jake Goulding
Andrew Lamb