dd9c5d1b13
fix: Rename compactor2 to compactor
2023-05-09 10:58:55 +02:00
0849ce6f2b
fix: Rename ingester2_test_ctx to ingester_test_ctx
2023-05-08 20:23:02 -04:00
19a56bdf4b
fix: Rename ioxd_ingester2 to ioxd_ingester
2023-05-08 20:20:03 -04:00
56916cf942
fix: Rename ingester2 to ingester
2023-05-08 12:03:05 -04:00
2860d87fe1
chore: Update DataFusion ( #7756 )
...
* chore: Update DataFusion pin
* chore: Update explain plans
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-05 18:58:18 +00:00
abe5d26d2f
chore: update DataFusion to `2787e7a36a6be83d91201df20827d3695f933300` ( #7732 )
...
Required to get:
- https://github.com/apache/arrow-datafusion/pull/6199
2023-05-03 16:40:00 +00:00
f6dea224e8
feat(wal): Add wal_inspect crate & a write op entry decoder
...
This adds a new crate with a type capable of converting
decoded WAL Write Op entries to line protocol and writing
the result to a namespaced destination. The wal crate
now exports a type which reads the sequenced wal ops and
decodes them as namespaced table batch writes.
2023-05-03 11:50:59 +01:00
65ca52ce3b
chore: Update DataFusion again ( #7722 )
...
* chore: Update DF again
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-02 18:34:45 +00:00
2b1f8b56e2
chore: Update DataFusion ( #7719 )
...
* chore: Update DataFusion
* chore: update for API change
* chore: update some tests
* fix: Update plans in optimizer
* chore: Update plans
* chore: Update error messages
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-02 17:55:04 +00:00
0c4a86cbaa
fix: Use humantime rather than chrono-english
2023-05-01 11:31:42 -04:00
bc324c4ac2
chore: Update DataFusion and arrow/parquet/arrow-flight to `38.0.0` ( #7662 )
...
* chore: Update DataFusion and arrow/parquet/arrow-flight to `38.0.0`
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-25 14:29:23 -04:00
f411f5d6d5
chore: update DF to `181e5ccf2816ccaa05d8aaef0b375d4b7bbceece` ( #7630 )
...
I need the following two PRs:
- https://github.com/apache/arrow-datafusion/pull/6045
- https://github.com/apache/arrow-datafusion/pull/6085
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-24 09:46:54 +00:00
26575f8bb9
Merge branch 'main' into dom/tidy-authz
2023-04-18 10:47:18 +00:00
a409fd45f7
chore(deps): Bump tonic-reflection from 0.9.1 to 0.9.2 ( #7581 )
...
Bumps [tonic-reflection](https://github.com/hyperium/tonic ) from 0.9.1 to 0.9.2.
- [Release notes](https://github.com/hyperium/tonic/releases )
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/tonic/compare/v0.9.1...v0.9.2 )
---
updated-dependencies:
- dependency-name: tonic-reflection
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-18 10:40:49 +00:00
2b9a809de4
refactor: move HTTP authz helpers into authz
...
The "server_util" crate exists only to support HTTP authz operations, so
this commit moves it under the authz crate. This helper is gated by a
feature flag allowing callers to opt into this extra HTTP dependency
(disabled by default).
2023-04-18 12:30:56 +02:00
36947616fd
chore(deps): Bump tonic-health from 0.9.1 to 0.9.2 ( #7580 )
...
Bumps [tonic-health](https://github.com/hyperium/tonic ) from 0.9.1 to 0.9.2.
- [Release notes](https://github.com/hyperium/tonic/releases )
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/tonic/compare/v0.9.1...v0.9.2 )
---
updated-dependencies:
- dependency-name: tonic-health
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-18 10:04:53 +00:00
2e818fb370
chore(deps): Bump tonic from 0.9.1 to 0.9.2
...
Bumps [tonic](https://github.com/hyperium/tonic ) from 0.9.1 to 0.9.2.
- [Release notes](https://github.com/hyperium/tonic/releases )
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/tonic/compare/v0.9.1...v0.9.2 )
---
updated-dependencies:
- dependency-name: tonic
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-04-18 07:54:43 +00:00
91d1441031
chore(deps): Bump tonic-build from 0.9.1 to 0.9.2 ( #7579 )
...
Bumps [tonic-build](https://github.com/hyperium/tonic ) from 0.9.1 to 0.9.2.
- [Release notes](https://github.com/hyperium/tonic/releases )
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/tonic/compare/v0.9.1...v0.9.2 )
---
updated-dependencies:
- dependency-name: tonic-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-18 07:53:01 +00:00
808a13cf40
chore: remove `time` 0.1 & fix RUSTSEC-2020-0071 ( #7568 )
...
`time` 0.1 suffers from [RUSTSEC-2020-0071] and many upstream crates
have tried to remove it for years. The last dependency is
1. `chrono-english`
2. `chrono` (default features)
3. `chrono` (oldtime)
4. `time` 0.1
`chrono-english` doesn't seem to be super well maintained, but I
couldn't find a nice replacement for it. Luckily the master branch of
`chrono-english` is already fixed, so let's just directly use that.
[RUSTSEC-2020-0071]: https://rustsec.org/advisories/RUSTSEC-2020-0071
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-17 12:36:10 +00:00
c26981d51b
chore: Update datafusion again ( #7557 )
...
* chore: Update datafusion again
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-14 17:37:00 +00:00
f46d06d56f
chore: Update DataFusion + arrow ecosystem to 37 ( #7544 )
...
* chore: Update datafusion and arrow/parquet to 37, tonic to 0.9.1
* refactor: Update for FieldRef and other API changes
* fix: Update field size calculation
* fix: Use `NullBuffer` directly
* fix: remove outdated comment
* chore: Update test for tonic
* chore: Run cargo hakari tasks
* chore: cargo update
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-14 12:43:01 +00:00
134ff2ef83
chore: update DataFusion pin (right before arrow 37 update) ( #7540 )
...
* chore: update DataFusion pin
* refactor: Update for deprecated API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 17:25:24 +00:00
3ebd07358b
chore: Update DataFusion pin, upgrade `date_bin` and `InfluxQL` to use `Interval(MonthDayNano)` ( #7516 )
...
* chore: Update datafusion
* chore: Update for change in PhysicalSortExpr
* refactor: Update date_bin_gapfill to take IntervalMonthDayNano, fix FlightSQL
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 10:43:32 +00:00
53ddca45d8
Merge branch 'main' into cn/remove-write-summary
2023-04-12 16:07:35 +00:00
20e9c91866
refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc ( #7515 )
...
* refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:07:19 +00:00
6387a9576a
fix: Remove the write_summary crate and write info service
2023-04-12 11:31:23 -04:00
65c90b0281
fix: Remove the write buffer crate
2023-04-12 11:01:26 -04:00
02c7ec0727
test: add ingester2_test_ctx
...
Moves the ingester2 TestContext & builder to its own crate for reuse
between integration tests & benchmarks (and more?!)
This allows us to DRY & reuse the test code with it's nice API for
benchmarks too, without forcing it all to be in the "prod" build of
ingester2 (and the dependencies, and their dependencies, etc).
This also means this code can be built and cached instead of being
rebuilt all the time during normal development.
2023-04-12 12:51:47 +02:00
6c2ce01f1e
fix: Remove old ingester and ioxd_ingester
2023-04-07 11:06:37 -04:00
a5769a461b
fix: Remove old compactor and ioxd_compactor
2023-04-06 16:49:52 -04:00
e8b7d69b0f
chore: Update datafusion again ( #7442 )
...
* chore: Update datafusion
* chore: Fix up plans for datafusion API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-05 18:21:53 +00:00
badc8865ef
chore: Update datafusion again ( #7440 )
...
* chore: Update DataFusion
* chore: Update for new API
* chore: Run cargo hakari tasks
* fix: cargo doc
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-04 15:45:46 +00:00
d8b0139ea9
chore: Update datafusion + arrow/parquet/arrow-flight to 36 ( #7354 )
...
* chore: Update datafusion + arrow/parquet/arrow-flight to 36
* refactor: update optimize for new API
* refactor: update parquet for new API
* chore: Update more dependencies
* chore: Update to use the new buffer creation APIs
* chore: Run cargo hakari tasks
* fix: bad len
* fix: update for API change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 13:41:59 +00:00
43e236e040
chore: Update datafusion again ( #7353 )
...
* chore: Update DataFusion
* refactor: Update predicate crate for new transform API
* refactor: Update iox_query crate for new APIs
* refactor: Update influxql for new API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 16:21:49 +00:00
1817104236
chore: Update DataFusion again ( #7329 )
...
* chore: update datafusion
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:45:53 +00:00
5dd71998a1
chore: Update datafusion ( #7318 )
...
* chore: Update datafusion
* chore: Update for API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 15:07:23 +00:00
6660b86f6c
fix: Put authorization header in request extension
2023-03-22 04:12:07 +01:00
8ab553180e
chore: Update datafusion again ( #7276 )
...
* chore: Update DataFusion pin
* chore: Update for new Expr type
* fix: deprecated
* chore: Run cargo hakari tasks
* docs: Update predicate/src/lib.rs
* fix: fmt
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-21 18:25:04 +00:00
96c2094302
refactor(iox_query): extract influxrpc planner to its own crate ( #7241 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-17 18:48:55 +00:00
13657d5bcc
feat(authz): authorization service client and write integration ( #7216 )
...
* feat(authz): add authorization client.
Add a new authz crate to provide the interface for making authorization
checks from within IOx. This includes the default client that uses
the influxdata.iox.authz.v1 gRPC protocol. This feature is not used
by any IOx component yet.
* feat: optional authorization on write path
Support optionally enabling authorization checks on the /api/v2/write
handler. If an authrorizer is configured then the handler will
attempt to retrieve a token from the request's Authorization header.
If no such token exists then a response with a 401 error code is
returned. If the token is not valid, or does not have write permission
for the requested namespace then a response with a 403 error is
returned.
* chore: add unit test for authz in write handler
Add unit tests that test the correct functioning of the /api/v2/write
handler when an Authorizer is configured.
* chore(authz): use lazy connection
Change the initialization of the authz client to use a lazy connection.
This allows the client to be initialised synchronously.
* chore: Run cargo hakari tasks
* fix(authz): protolint complaints
* fix: authz tests
* fix: benches and lint
* chore: Update clap_blocks/src/authz.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: Update authz/src/lib.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: Update clap_blocks/src/authz.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: review suggestions
* chore: review suggestions
Apply a number of suggestions from review comments. The main
behavioural change is that if the authz service is configured
applictions will perform a probe request to ensure it can communicate
before continuing startup.
* chore: Update router/src/server/http.rs
Co-authored-by: Dom <dom@itsallbroken.com>
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-17 15:20:14 +00:00
7dfaa05e8a
chore: Update datafusion again ( #7208 )
...
* chore: update datafusion again
* fix: update test
* fix: use table_reference
* fix: clean up import
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 14:34:40 +00:00
35bd5ccd13
chore: Update datafusion pin ( #7207 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-15 11:35:15 +00:00
0eb858c70d
chore: Update datafusion ( #7167 )
...
* chore: Update datafusion
* chore: Update datafusion
* refactor: use UserDefinedLogicalNodeCore
* fix: remove stray comment
* fix: clippy
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 16:41:32 +00:00
fe48a685ec
refactor: Move InfluxQL behaviour from iox_query to new crate ( #7156 )
...
* refactor: Break unnecessary dependencies from `iox_query` crate
In the process, the test code has been simplified.
* refactor: Move InfluxQL plan module to iox_query_influxql crate
* refactor: Move remaining behaviour from iox_query to iox_query_influxql
* chore: rustfmt 🙄
I was under the impression `clippy` would catch formatting
2023-03-08 22:29:20 +00:00
3828d2a50e
chore: update DataFusion to `deeaa5632ed99a58b91767261570756db736d158` ( #7158 )
...
* chore: update DataFusion to `deeaa5632ed99a58b91767261570756db736d158`
I want to get pull:
- https://github.com/apache/arrow-datafusion/pull/5495
Changes in the IOx code base due to:
- https://github.com/apache/arrow-datafusion/pull/5423
- https://github.com/apache/arrow-datafusion/pull/5421
- https://github.com/apache/arrow-datafusion/pull/5450
* refactor: simplify expression simplifcation
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: remove upstreamed code
* test: update snapshots
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-03-08 13:05:31 +00:00
ed0704ac8d
chore: Update datafusion ( #7100 )
...
* chore: Update datafusion
* chore: iox_query to compile for API changes + update tests
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-06 17:59:24 +00:00
e19ce98407
chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0 ( #7084 )
...
* chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0
* chore: Run cargo hakari tasks
* chore: Update plans
* chore: Update querier expected output
* chore: Update querier tests to use insta
* fix: sort output too
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-01 11:25:01 +00:00
f93baf7693
chore: Update DataFusion and `arrow` / `arrow-flight` / `parquet` to `33.0.0` ( #7045 )
...
* chore: Update DataFusion and arrow/arrow-flight/parquet to 33.0.0
* fix: Update test output
* fix: update more test output
* fix: Update querier test output
* chore: Run cargo hakari tasks
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting for selector tests
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: Christopher Wolff <chris.wolff@influxdata.com>
2023-02-22 21:24:20 +00:00
27890b313f
chore: Update datafusion ( #6997 )
...
* chore: Update datafusion
* chore: update the plans
* fix: update some plans
* chore: Update plans and port some explain plans to use insta snapshots
* fix: another plan
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 17:03:25 +00:00
4d7aa1e48b
refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test ( #6960 )
...
* refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test
* fix: Update compactor2/src/components/mod.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:06:42 +00:00
143cb26e87
chore: Update datafusion again ( #6958 )
...
* chore: Update datafusion pin
* chore: Update plans
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 10:28:44 +00:00
501032c2fb
chore: Update datafusion again ( #6929 )
...
* chore: Update DataFusion
* fix: disable repartitioned sorting
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 11:51:47 +00:00
2f4d901fbe
chore: Update datafusion ( #6893 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: update for api change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 18:27:18 +00:00
ae10b33f0c
chore: Update datafusion ( #6868 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update error message
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-06 14:22:45 +00:00
d3601a59f8
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` ( #6756 )
...
* chore: update DataFusion
* fix: test
* chore: format
* chore: clippy
* chore: update arrow
* chore: arrow upgrade fallout
* chore: Run cargo hakari tasks
* chore: remove failing warm compaction test
* fix: flight error propagation
* chore: update parquet size
* fix: Update error message
* chore: Update parquet metadata test
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:35:39 +00:00
b55e4ffa21
test: Port table_schema tests to end-to-end and get rid of query_tests ( #6828 )
...
* test: Port table_schema query_tests to end-to-end
* fix: Remove now outdated query_tests crate
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 17:41:08 +00:00
5b14caa780
chore: Update DataFusion ( #6753 )
...
* chore: Update datafusion
* fix: Update for changes
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-30 14:48:52 +00:00
0d32662eea
chore: Update datafusion again ( #6722 )
...
* chore: Update datafusion
* fix: Update for API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-27 18:59:27 +00:00
c3bc61f10e
refactor: Move `flightsql` code into its own module, add docs and tests ( #6640 )
...
* refactor: Move `flightsql` code into its own module
* fix: get schema from LogicalPlan
* refactor: use arrow_flight::sql::Any instead of prost_types::any
* fix: cleanup docs and avoid as_ref
* fix: Use Bytes
* fix: use Any::pack
* fix: doclink
2023-01-24 18:24:32 +00:00
9a61f36a53
chore: Update datafusion again ( #6656 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-23 12:13:07 +00:00
b09691dc6b
chore: Upgrade datafusion (again, I know) ( #6639 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-23 11:24:22 +00:00
24ca1e6f8c
Merge branch 'main' into cn/test-old-with-new
2023-01-20 15:20:40 +00:00
d808c57cdc
chore: Remove `iox_arrow_flight` ( #6621 )
...
* chore: Remove iox_arrow_flight
* fix: hack around tonic status errors
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-20 11:16:19 +00:00
6afd782b3f
fix: Move query_tests2 into influxdb_iox/tests so that the code rebuilds
2023-01-19 16:44:31 -05:00
f639bf3e23
chore: refactor ingester to use upstream arrow-flight ( #6622 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 15:16:13 +00:00
6a4fa11dba
chore: Update to latest datafusion / arrow `31.0.0` ( #6618 )
...
* chore: Update datafusion and arrow/parquet/arrow-flight `31.0.0`
* chore: Update for new API
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 20:21:10 +00:00
33168b97f0
Merge branch 'main' into cn/query-tests-grpc
2023-01-18 19:03:51 +00:00
e72173d58d
feat: very basic compactor2 skeleton ( #6614 )
...
Sets up crate and wires up the main binary. No tests yet, no algorithm
framework, just the bare minimum.
Also I decided to not offer a gRPC server in `compactor2` at the moment
and hence did not implement any handle/delegate infrastructure. We add
this later if we need it. This also means compactor2 does NOT provide a
catalog service for now.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-18 16:36:40 +00:00
8410998408
chore: Update datafusion to Jan 17, 2023 (2 / 2) and arrow/parquet `30.0.1` ( #6604 )
...
* chore: Update datafusion to Jan 9, 2023 (2 / 2) and arrow/parquet `30.0.1`
* chore: Update for changes in arrow ipc
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-18 15:51:24 +00:00
f3b5dcaab7
feat: Reimagining query_tests
2023-01-18 10:24:17 -05:00
57f08dbccd
chore: Update datafusion to Jan 9, 2023 (1 / 2) ( #6603 )
...
* refactor: Update DataFusion pin to early Jan 2023
* fix: Update tests now that planning is async
* fix: Updates for API changes
* chore: Run cargo hakari tasks
* fix: Update comment
* refactor: nicer config setup
* fix: gapfill async
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-18 12:19:32 +00:00
cc6d8709c9
chore(deps): Bump hashbrown from 0.13.1 to 0.13.2 ( #6574 )
...
Bumps [hashbrown](https://github.com/rust-lang/hashbrown ) from 0.13.1 to 0.13.2.
- [Release notes](https://github.com/rust-lang/hashbrown/releases )
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.1...v0.13.2 )
---
updated-dependencies:
- dependency-name: hashbrown
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-12 09:47:15 +00:00
dbe52f1ca1
chore: Upgrade datafusion ( #6467 )
...
* chore: Update datafusion
* fix: Update for new apis
* chore: Update expected plan
* fix: Update for new config construction
* chore: update clippy
* fix: Fix error codes
* fix: update another test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
d0d5906476
chore: Update datafusion pin ( #6442 )
...
* chore: Update datafusion pin
* refactor: Update iox_query for new apis
* chore: Update some more apis
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-19 20:02:42 +00:00
94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight ( #6427 )
...
* refactor: Extract common ArrowFlight client into iox_arrow_flight
* chore: Run cargo hakari tasks
* fix: clarify intent of iox_arrow_flight crate
* refactor: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: loop --> while let
* fix: REmove make_tonic_error in favor of From impl
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
be45889be1
chore: Upgrade datafusion ( #6407 )
...
* chore: Update datafusion
* chore: Update for API change
2022-12-15 06:51:35 +00:00
8729977851
chore: Upgrade datafusion / arrow to 29.0.0 to get flightsql client ( #6396 )
...
* chore: Update datafusion pin
* chore: Update for API change
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-13 20:16:09 +00:00
336ca761a3
chore: Update datafusion pin (to get sqlparser update) ( #6378 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-12 14:39:42 +00:00
e0ecacf6cc
chore: Update DataFusion (get median fix and automatic string to timestamp coercion) ( #6363 )
...
* chore: Update DataFusion pin to get median fix
* chore: Update for new Expr node
* test: add test for median
* test: add test for coercion of strings to timestamps
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 12:14:00 +00:00
061d582a9b
chore: patch object_store to get apache#3274 ( #6362 )
...
* chore: patch object_store to get apache#3274
* chore: Run cargo hakari tasks
* fix: add issue breadcrumb
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 14:52:25 +00:00
9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers ( #6349 )
...
* test: add tests for periods in measurement names
* chore: Update Datafusion
* chore: Update for changed APIs
* chore: Update expected plan output
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 11:27:13 +00:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
d0f1f6a4fd
chore: Upgrade datafusion to get memory limits ( #6297 )
...
* chore: Update datafusion
* fix: use correctly qualified column names
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 16:40:26 +00:00
99698d0f46
feat: Create an ioxd_ingester2 crate
2022-12-01 11:37:21 -05:00
14a9bc92e9
Revert "Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )" ( #6296 )
...
This reverts commit b7e52c0d8d
2022-12-01 14:20:43 +00:00
b7e52c0d8d
Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )
...
This reverts commit 039a45ddd1
2022-12-01 11:38:42 +00:00
039a45ddd1
chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )
...
* chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0`
* chore: Update thrift to 0.17
* fix: use workspace arrow-flight in ingester2
* chore: Update for API changes
* fix: test
* chore: Update hakari
* chore: Update hakari again
* chore: Update trace_exporters to latest thrift
* fix: update test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 14:12:30 +00:00
f22b1e1a09
chore: Update datafusion (to get memory limiting code) ( #6246 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update to newer api
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 12:05:17 +00:00
fc5697b8e7
chore: Update datafusion again (N of N) ( #6218 )
...
* chore: Update datafusion again (4 of N)
* fix: Update plans
* fix: Update for renamed API
* fix: Update more plans
* chore: Update to datafusion @ d355f69aae2cc951cfd021e5c0b690861ba0c4ac
* fix: update explain plan tests
* fix: update test after schema error
* chore: Update datafusion again
* fix: Add size() calculation to selectors
* chore: Run cargo hakari tasks
* fix: Update newly added test
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-28 17:09:40 +00:00
a66fc0b645
feat(ingester): ingester2 init
...
Adds an ingester2 crate to hold the MVP of the Kafkaless project.
This was necessary due to the tight coupling of the ingester internals
with tests in external crates, and eases the parallel development of two
version of the ingester.
This commit contains various changes from the "ingester" crate, mostly
removing the concept/references to a "shard" or "ShardId" where
possible.
This commit does not copy over all of the "ingester" crate - only those
components that are definitely needed. I will drag across more as
functionality is implemented.
2022-11-24 15:34:02 +01:00
db63dc55a2
feat: Initial WAL implementation
2022-11-23 17:07:43 -05:00
71b29ae442
chore: Update DataFusion (3 of N) and arrow/parquet to 27.0.0 ( #6208 )
...
* chore: Update DataFusion and arrow/parquet to 27.0.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-23 14:21:47 +00:00
9fb1de0428
chore: Update datafusion (2 of N) right before arrow 27 upgrade ( #6207 )
...
* chore: Update datafusion (2 of N) right before arrow 27 upgrade
* fix: Update tests for better unsigned pushdown
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-23 11:04:14 +00:00
1a1ea74cb7
chore: Upgrade datafusion again ( #6160 )
...
* Revert "Revert "chore: Update datafusion again (#6108 )""
This reverts commit 766b3bbeb440618cfe332f6ee7d4f8a8217acc48.
* fix: Respect the partition sort key
* chore: update plans
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 19:28:26 +00:00
67712b595c
Revert "chore: Update datafusion again ( #6108 )" ( #6159 )
...
This reverts commit fbe9f27f10
2022-11-16 21:14:55 +00:00
9365d933f1
chore: router namespace api ( #6151 )
...
* chore: move ns api from querier to router
* chore: add explanatory comment in querier about moved namespace API
* fix: add namespace service to router
* fix: querier returns unimplemented error for ns retention, not panic
* chore: reuse namespace -> proto in router ns api
* chore: grpc namespace - consume ns to avoid clone
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 15:25:49 +00:00
fbe9f27f10
chore: Update datafusion again ( #6108 )
...
* chore: Update datafusion pin + api code
* chore: Run cargo hakari tasks
* refactor: combine_sort_key is more idomatic and add rationale comments
* refactor: satisfy borrow checker and updated comments
* fix: Add test case for combine_sort_key
* fix: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* fix: Add back test for deeply nested expression
* fix: Update output ordering
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 14:41:52 +00:00
cc17e5a54b
refactor: use a workspace dependency for hashbrown
2022-11-11 13:25:39 -05:00
034d9b371d
chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0` ( #6061 )
...
* chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0`
* fix: Update query_functions
* fix: update for TimestampNanosecondArray API changes
* fix: update for TimestampNanosecondArray API changes
* chore: Update flatbuffers and remove rustsec warning
* chore: Update text
* fix: update more test
* fix: Lock ahash to exactly 0.8.0
* fix: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-07 11:01:58 +00:00
8c8e607dca
chore: Update datafusion pin ( #6054 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
* chore: Update expected error
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-03 19:41:31 +00:00
Carol (Nichols || Goulding)
Andrew Lamb
Marco Neumann
Fraser Savage
kodiakhq[bot]
dependabot[bot]
Dom Dwyer
Marko Mikulicic
Martin Hilton
Stuart Carnie
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Paul Dix
Luke Bond
Jake Goulding