66047f4372
feat: InfluxQL learns how to plan some InfluxQL queries ( #6520 )
...
* feat: InfluxQL learns how to plan some queries
Also added a means to test the planner and execution
* chore: Update module docs
* chore: Document the planner functions
* chore: Update end_to_end_cases crate
* chore: Clarify why `SLIMIT` and `SOFFSET` return `NotImplemented`
* chore: Address lint issues
* chore: Fix rustdoc link issue
* chore: Remove InfluxQL tests from query_tests crate
Will follow conventions established by @carols10cents when
new query_tests crate is merged.
* chore: `now` field
`now` is a DataFusion built-in scalar function
* chore: remove unused code
* chore: Add additional arithmetic expression tests
* chore: Establish pattern for identifying and tracking InfluxQL issues
* chore: Add tests for case sensitivity issues
* chore: group tests into modules and functions
This avoids mass rewriting of insta snapshots as new
tests are added to each function. When tests are added in the middle,
existing snapshots are renamed (-N+1, -N+2, etc) resulting in
having to review numerous additional snapshots.
2023-01-11 02:50:49 +00:00
62c0f3dbdd
feat: have cold compaction work with Compactor2 ( #6542 )
...
* feat: cold
* chore: debug info
* feat: only compact qualified cold partition candidates
* fix: catalog test
* chore: cleanup
* chore: add new config flag for cold partition candidates
* chore: implement display for CompactionType and add tests for max num partitions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 16:42:57 +00:00
b49cc2e35e
chore(deps): Bump tokio from 1.24.0 to 1.24.1 ( #6545 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.0 to 1.24.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.24.0...tokio-1.24.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:48:44 +00:00
09746c3aea
chore(deps): Bump assert_cmd from 2.0.7 to 2.0.8
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.7 to 2.0.8.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.7...v2.0.8 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-10 01:06:02 +00:00
828992c9c5
feat: Ingest replica skeleton ( #6529 )
...
* feat: Update replication.proto
* Remove the PartitionId in the replicate request as a single replicate request can have the data for many partitions.
* Add namespace_id and table_id to persist complete request to make data easier to lookup in buffer.
* feat: Initial ingest_replica skeleton
A bunch of copy pasta here from ingester2, but this takes out a ton of stuff that isn't used in replicas.
Also lays the groundwork for the simpler buffer structure to keep the data and a basic cache for catalog information that will be required.
* feat: update replication.proto GetPartitionBufferResponse
* chore: PR cleanup
* chore: PR cleanup
2023-01-09 16:53:49 +00:00
afaaedc758
test: Reproduce the test for querier caching
2023-01-04 10:15:35 -05:00
9f7dede433
refactor: Reorganize tests to make deleting half the tests easier
...
When we switch to RPC write mode, the with_kafka module can be deleted
and the kafkaless_rpc_write module can be unwrapped.
2023-01-04 10:15:35 -05:00
c464487bb2
test: Ability to set up a querier2 that doesn't connect to any ingesters
2023-01-04 10:06:57 -05:00
08ceb4ee48
test: Check catalog for new Parquet files to know when data is persisted
2023-01-04 10:06:57 -05:00
e49bee0c26
test: Make test ingester2 instances either persist very quickly or not at all
2023-01-04 10:06:57 -05:00
96029654ab
test: Add a shared MiniCluster for version 2 services
2023-01-04 10:06:56 -05:00
dbe52f1ca1
chore: Upgrade datafusion ( #6467 )
...
* chore: Update datafusion
* fix: Update for new apis
* chore: Update expected plan
* fix: Update for new config construction
* chore: update clippy
* fix: Fix error codes
* fix: update another test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
0aacef3c59
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 ( #6473 )
...
* chore(deps): Bump once_cell from 1.16.0 to 1.17.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Change once_cell version specifier to major.minor for less churn
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
d27e137c39
chore: add debug info for the investigation ( #6472 )
2022-12-29 23:49:29 +00:00
7c6ccdb6d7
fix: Use keys and values functions. Thanks clippy!
2022-12-21 14:32:35 -05:00
ed17069087
chore(deps): Bump num_cpus from 1.14.0 to 1.15.0 ( #6451 )
...
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus ) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases )
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.14.0...v1.15.0 )
---
updated-dependencies:
- dependency-name: num_cpus
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-21 15:17:55 +00:00
e1059a9009
feat: FlightSQL Milestone 2 basic FlightSQL client and FlightSQL server implementation and plumbing ( #6398 )
...
* feat: Add basic Flight and FlightSQL client into IOx codebase
Basic flight end to end test
* fix: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-20 17:34:00 +00:00
9b22ede3f0
refactor: Make arrow flight client return `futures::Streams` ( #6438 )
...
* refactor: Make arrow flight client use futures::Streams
* refactor: concision
2022-12-19 17:09:26 +00:00
c0f2ba09ee
Merge branch 'main' into cn/compactor2
2022-12-19 14:22:56 +00:00
94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight ( #6427 )
...
* refactor: Extract common ArrowFlight client into iox_arrow_flight
* chore: Run cargo hakari tasks
* fix: clarify intent of iox_arrow_flight crate
* refactor: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: loop --> while let
* fix: REmove make_tonic_error in favor of From impl
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
299f0e99f9
chore(deps): Bump thiserror from 1.0.37 to 1.0.38
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.37 to 1.0.38.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.37...1.0.38 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 10:33:32 +00:00
7f2aa8b10c
chore(deps): Bump serde_json from 1.0.89 to 1.0.91
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.89 to 1.0.91.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.89...v1.0.91 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 01:44:18 +00:00
d7e75d43ea
fix: Make shard ID optional for compactor queries in RPC write mode
2022-12-16 17:28:53 -05:00
2406cdb24b
feat: Create a compactor2 cli
2022-12-16 17:22:06 -05:00
f419e2c378
feat: warm compaction ( #6192 )
...
* feat: warm compaction
chore: add missing warm compaction config
chore: tests for warm compaction
chore: modify count usage in warm compaction sql
chore: catalog test for warm compaction; sql fixes
feat: settable target level for compact w/ budget
chore: tests for warm compaction
chore: clarifying comments in warm compaction test
chore: fixed erroneous comment in catalog test
chore: improve warm compactor test by checking file exists
chore: tests for warm compaction
chore: warm compactor test tidy-ups
* chore: improve test for warm compaction
* chore: fix erroneous comment in warm compaction code
2022-12-16 15:59:45 +00:00
2a1e540ee3
fix: Rename INFLUXDB_IOX_MODE to INFLUXDB_IOX_RPC_MODE
2022-12-15 14:13:01 -05:00
7d216ba1fd
feat: Error if you run the wrong command with the wrong env var set
...
Connects to #6402 .
2022-12-15 14:06:59 -05:00
aec98015d7
fix: Remove the rpc_write feature flag and use INFLUXDB_IOX_MODE env var instead
...
And standardize on ingester2 and router2 for consistency.
Connects to #6402 .
2022-12-15 14:06:59 -05:00
8e1bf9cdf7
feat: Increase default all-in-one compactor memory to 300MB and expose it as a setting ( #6405 )
2022-12-14 20:35:48 +00:00
66c610f7b1
Merge branch 'main' into cn/ingester-persisted-file-count
2022-12-14 14:58:31 +00:00
47cd6821e1
feat: Document IOx Flight API and add convenience methods ( #6392 )
...
* feat: Document IOx Flight API and add convenience methods
* fix: InfluxQL handling
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-13 17:32:37 +00:00
fdbf9e112e
fix: Actually switch into rpc_write mode in querier
...
Only when the feature flag is set *and* --ingester-addresses is set. I
had documented that intention, but didn't actually implement it
correctly.
2022-12-12 16:37:11 -05:00
1c7f322a4e
feat: Keep track of and report number of Parquet files persisted
...
Per partition and starting over each time the ingester restarts.
Fixes #6334 .
2022-12-12 11:45:00 -05:00
727efcbdee
Merge branch 'main' into cn/ingester2-uuid
2022-12-12 16:21:15 +00:00
e5322b24b9
feat: Add --token CLI argument, improve update docs about writing ( #6356 )
...
* feat: Add --token CLI argument, improve update docs about writing
* fix: support environment tokens too
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 13:43:15 +00:00
c3a7575d46
feat: Enable rpc_write on the inner command if it's enabled for tests
...
And only run rpc_write specific tests if the feature is enabled when
running the tests.
2022-12-08 17:45:30 -05:00
5141cba1db
fix: Only switch into querier RPC write path if ingester addresses specified
...
This enables testing of the querier using the old path with the
rpc_write feature turned on.
2022-12-08 17:40:04 -05:00
2fd2d05ef6
feat: Identify each run of an ingester with a Uuid
...
And send that UUID in the Flight response for queries to that ingester
run.
Fixes #6333 .
2022-12-08 17:22:52 -05:00
619a2d0856
fix: Remove conflicting arguments from the RouterRpcWriteConfig ( #6355 )
...
These were added in
https://github.com/influxdata/influxdb_iox/pull/6346 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 20:21:37 +00:00
4ded68de62
test: "not found" end2end tests for querier ( #6352 )
...
I couldn't find any end2end tests for these cases and I was kinda
worried that our error codes were wrong. Turns out they are correct, but
let's have some nice tests for this behavior.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 18:17:53 +00:00
6f7cb5ccf0
Merge branch 'main' into cn/ingester2-querier
2022-12-08 14:00:49 +00:00
e13e668d26
refactor: Share more code in the querier in the RPC write path mode
2022-12-07 13:54:08 -05:00
551bb0ef6a
feat: allow enabling/disabling ns autocreation in router ( #6346 )
...
* feat: allow enabling/disabling ns autocreation in router
* fix: missed an import for something behind router2 compile flag
2022-12-07 16:12:00 +00:00
9166ace796
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-07 09:56:50 -05:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
f62b270852
fix: gRPC errors regarding group cols ( #6314 )
...
* fix: gRPC errors regarding group cols
- missing group col prev. produced an "internal error" but should be
"invalid argument"
- duplicate group cols produced a panic but should also be "invalid
argument"
* docs: clarify
2022-12-06 07:36:32 +00:00
cd6a8a1a82
refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics ( #6313 )
...
* refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics
Closes #6310 .
* refactor: rename and tune default exec mem limits
* fix: ingester2 bits after rebase
2022-12-05 12:38:28 +00:00
fc9431f6ae
chore(deps): Bump assert_cmd from 2.0.6 to 2.0.7 ( #6330 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.6...v2.0.7 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:05:56 +00:00
66aab55534
feat(ingester2): run persistence task
...
Configures the initialisation of an ingester2 instance to spawn a
persistence task (currently unused) and plumbs in various configuration
parameters.
2022-12-02 17:18:39 +01:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
Stuart Carnie
Nga Tran
dependabot[bot]
Paul Dix
Carol (Nichols || Goulding)
Andrew Lamb
kodiakhq[bot]
Luke Bond
Carol (Nichols || Goulding)
Marco Neumann
Dom Dwyer