08ceb4ee48
test: Check catalog for new Parquet files to know when data is persisted
2023-01-04 10:06:57 -05:00
e49bee0c26
test: Make test ingester2 instances either persist very quickly or not at all
2023-01-04 10:06:57 -05:00
96029654ab
test: Add a shared MiniCluster for version 2 services
2023-01-04 10:06:56 -05:00
dbe52f1ca1
chore: Upgrade datafusion ( #6467 )
...
* chore: Update datafusion
* fix: Update for new apis
* chore: Update expected plan
* fix: Update for new config construction
* chore: update clippy
* fix: Fix error codes
* fix: update another test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
0aacef3c59
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 ( #6473 )
...
* chore(deps): Bump once_cell from 1.16.0 to 1.17.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Change once_cell version specifier to major.minor for less churn
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
d27e137c39
chore: add debug info for the investigation ( #6472 )
2022-12-29 23:49:29 +00:00
7c6ccdb6d7
fix: Use keys and values functions. Thanks clippy!
2022-12-21 14:32:35 -05:00
ed17069087
chore(deps): Bump num_cpus from 1.14.0 to 1.15.0 ( #6451 )
...
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus ) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases )
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.14.0...v1.15.0 )
---
updated-dependencies:
- dependency-name: num_cpus
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-21 15:17:55 +00:00
e1059a9009
feat: FlightSQL Milestone 2 basic FlightSQL client and FlightSQL server implementation and plumbing ( #6398 )
...
* feat: Add basic Flight and FlightSQL client into IOx codebase
Basic flight end to end test
* fix: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-20 17:34:00 +00:00
9b22ede3f0
refactor: Make arrow flight client return `futures::Streams` ( #6438 )
...
* refactor: Make arrow flight client use futures::Streams
* refactor: concision
2022-12-19 17:09:26 +00:00
c0f2ba09ee
Merge branch 'main' into cn/compactor2
2022-12-19 14:22:56 +00:00
94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight ( #6427 )
...
* refactor: Extract common ArrowFlight client into iox_arrow_flight
* chore: Run cargo hakari tasks
* fix: clarify intent of iox_arrow_flight crate
* refactor: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: loop --> while let
* fix: REmove make_tonic_error in favor of From impl
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
299f0e99f9
chore(deps): Bump thiserror from 1.0.37 to 1.0.38
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.37 to 1.0.38.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.37...1.0.38 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 10:33:32 +00:00
7f2aa8b10c
chore(deps): Bump serde_json from 1.0.89 to 1.0.91
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.89 to 1.0.91.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.89...v1.0.91 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 01:44:18 +00:00
d7e75d43ea
fix: Make shard ID optional for compactor queries in RPC write mode
2022-12-16 17:28:53 -05:00
2406cdb24b
feat: Create a compactor2 cli
2022-12-16 17:22:06 -05:00
f419e2c378
feat: warm compaction ( #6192 )
...
* feat: warm compaction
chore: add missing warm compaction config
chore: tests for warm compaction
chore: modify count usage in warm compaction sql
chore: catalog test for warm compaction; sql fixes
feat: settable target level for compact w/ budget
chore: tests for warm compaction
chore: clarifying comments in warm compaction test
chore: fixed erroneous comment in catalog test
chore: improve warm compactor test by checking file exists
chore: tests for warm compaction
chore: warm compactor test tidy-ups
* chore: improve test for warm compaction
* chore: fix erroneous comment in warm compaction code
2022-12-16 15:59:45 +00:00
2a1e540ee3
fix: Rename INFLUXDB_IOX_MODE to INFLUXDB_IOX_RPC_MODE
2022-12-15 14:13:01 -05:00
7d216ba1fd
feat: Error if you run the wrong command with the wrong env var set
...
Connects to #6402 .
2022-12-15 14:06:59 -05:00
aec98015d7
fix: Remove the rpc_write feature flag and use INFLUXDB_IOX_MODE env var instead
...
And standardize on ingester2 and router2 for consistency.
Connects to #6402 .
2022-12-15 14:06:59 -05:00
8e1bf9cdf7
feat: Increase default all-in-one compactor memory to 300MB and expose it as a setting ( #6405 )
2022-12-14 20:35:48 +00:00
66c610f7b1
Merge branch 'main' into cn/ingester-persisted-file-count
2022-12-14 14:58:31 +00:00
47cd6821e1
feat: Document IOx Flight API and add convenience methods ( #6392 )
...
* feat: Document IOx Flight API and add convenience methods
* fix: InfluxQL handling
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-13 17:32:37 +00:00
fdbf9e112e
fix: Actually switch into rpc_write mode in querier
...
Only when the feature flag is set *and* --ingester-addresses is set. I
had documented that intention, but didn't actually implement it
correctly.
2022-12-12 16:37:11 -05:00
1c7f322a4e
feat: Keep track of and report number of Parquet files persisted
...
Per partition and starting over each time the ingester restarts.
Fixes #6334 .
2022-12-12 11:45:00 -05:00
727efcbdee
Merge branch 'main' into cn/ingester2-uuid
2022-12-12 16:21:15 +00:00
e5322b24b9
feat: Add --token CLI argument, improve update docs about writing ( #6356 )
...
* feat: Add --token CLI argument, improve update docs about writing
* fix: support environment tokens too
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 13:43:15 +00:00
c3a7575d46
feat: Enable rpc_write on the inner command if it's enabled for tests
...
And only run rpc_write specific tests if the feature is enabled when
running the tests.
2022-12-08 17:45:30 -05:00
5141cba1db
fix: Only switch into querier RPC write path if ingester addresses specified
...
This enables testing of the querier using the old path with the
rpc_write feature turned on.
2022-12-08 17:40:04 -05:00
2fd2d05ef6
feat: Identify each run of an ingester with a Uuid
...
And send that UUID in the Flight response for queries to that ingester
run.
Fixes #6333 .
2022-12-08 17:22:52 -05:00
619a2d0856
fix: Remove conflicting arguments from the RouterRpcWriteConfig ( #6355 )
...
These were added in
https://github.com/influxdata/influxdb_iox/pull/6346 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 20:21:37 +00:00
4ded68de62
test: "not found" end2end tests for querier ( #6352 )
...
I couldn't find any end2end tests for these cases and I was kinda
worried that our error codes were wrong. Turns out they are correct, but
let's have some nice tests for this behavior.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 18:17:53 +00:00
6f7cb5ccf0
Merge branch 'main' into cn/ingester2-querier
2022-12-08 14:00:49 +00:00
e13e668d26
refactor: Share more code in the querier in the RPC write path mode
2022-12-07 13:54:08 -05:00
551bb0ef6a
feat: allow enabling/disabling ns autocreation in router ( #6346 )
...
* feat: allow enabling/disabling ns autocreation in router
* fix: missed an import for something behind router2 compile flag
2022-12-07 16:12:00 +00:00
9166ace796
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-07 09:56:50 -05:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
f62b270852
fix: gRPC errors regarding group cols ( #6314 )
...
* fix: gRPC errors regarding group cols
- missing group col prev. produced an "internal error" but should be
"invalid argument"
- duplicate group cols produced a panic but should also be "invalid
argument"
* docs: clarify
2022-12-06 07:36:32 +00:00
cd6a8a1a82
refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics ( #6313 )
...
* refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics
Closes #6310 .
* refactor: rename and tune default exec mem limits
* fix: ingester2 bits after rebase
2022-12-05 12:38:28 +00:00
fc9431f6ae
chore(deps): Bump assert_cmd from 2.0.6 to 2.0.7 ( #6330 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.6...v2.0.7 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:05:56 +00:00
66aab55534
feat(ingester2): run persistence task
...
Configures the initialisation of an ingester2 instance to spawn a
persistence task (currently unused) and plumbs in various configuration
parameters.
2022-12-02 17:18:39 +01:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
77cbc880f6
feat: Add cap limit on number of partitions to be compacted in parallel ( #6305 )
...
* feat: Add cap limit on number of partitions to be comapcted in parallel
* chore: cleanup
* chore: clearer comments
2022-12-01 21:23:44 +00:00
b9e424582f
refactor: Extract a clap block for the ingester2 RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
fef3bc02cd
refactor: Extract a clap block for the router RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
0133479122
feat: Create an ingester2 server
2022-12-01 11:39:29 -05:00
99698d0f46
feat: Create an ioxd_ingester2 crate
2022-12-01 11:37:21 -05:00
fc694e7273
feat: Add an ingester2 command behind the rpc_write feature flag
2022-12-01 11:37:21 -05:00
c008219692
feat: Add a feature flag to switch to the router RPC write path ( #6247 )
...
* feat: Add a feature flag to switch to the router RPC write path
Fixes #6242 .
* refactor: Remove a weird arc clone/rename that's not needed
I'm sure this was needed at some point, but it doesn't make much sense.
I wasn't going to change this, but I'm now trying to minimize the
differences between this function and the write path init function, so
make this one better too.
* fix: Add the namespace autocreation to the RPC write path too
The topic/query pool don't really apply to this case, but use them
anyway to be able to use the existing catalog methods.
Also add a bunch of comments pointing out where the RPC write path
initializer and the old router's initializer are the same and where
they're different, so that perhaps it'll be easier to keep them in sync
while they both exist.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 11:05:39 +00:00
d07658282c
feat: add router config parameter for retention ( #6278 )
...
* chore: remove unused/moved ns_autocreation dml handler
* feat(router): expose new ns retention as config
* fix: forgot to set default value for router retention arg
* chore: make new namespace retention param an option
2022-11-30 13:14:39 +00:00
Carol (Nichols || Goulding)
Andrew Lamb
dependabot[bot]
Nga Tran
kodiakhq[bot]
Luke Bond
Carol (Nichols || Goulding)
Marco Neumann
Dom Dwyer