2a658ec0ab
Merge pull request #8289 from influxdata/cn/cleanups
...
fix: Clean up some comments and unused imports that were allowed
2023-07-21 13:27:37 +00:00
76ecfcc815
Merge branch 'main' into cn/cleanups
2023-07-21 13:22:50 +00:00
3ac0e30ac9
fix: Remove namespace ID from a partition identifier type ( #8288 )
...
I'm going to make a change in the future that removes the access to the
namespace ID from this code, and it's not needed anyway as partitions
are uniquely identifiable by only table ID and partition key.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-21 13:22:33 +00:00
a14afafcd9
Merge branch 'main' into savage/await-enqueue-rotation-returned-receiver-during-shutdown
2023-07-21 12:45:55 +01:00
6347d833e7
fix(ingester): Graceful shutdown must wait for file rotation receiver
...
This adds a second level of wait to the WAL draing & rotate before
allowing shutdown to proceed. Only once the returned receiver has
notified the called that the notification has been handled may it await
the empty waker set.
2023-07-21 12:31:04 +01:00
3eb48ef210
chore: Update datafusion again ( #8247 )
...
* chore: Update datafusion to get new grouping
* chore: Update for new API
* chore: update tests
* fix: new API
* fix: state type
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-21 11:20:36 +00:00
ac9d1946e9
fix: add retry loop to avoid CI flake in build-catalog test ( #8271 )
...
* fix: add retry loop to avoid CI flake in build-catalog test
* fix: Update influxdb_iox/tests/end_to_end_cases/debug.rs
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-21 10:14:06 +00:00
0d0f07b34e
chore(deps): Bump tempfile from 3.6.0 to 3.7.0 ( #8297 )
...
* chore(deps): Bump tempfile from 3.6.0 to 3.7.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.6.0 to 3.7.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.6.0...v3.7.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-21 09:06:34 +00:00
867937157c
chore(deps): Bump num-traits from 0.2.15 to 0.2.16 ( #8296 )
...
Bumps [num-traits](https://github.com/rust-num/num-traits ) from 0.2.15 to 0.2.16.
- [Changelog](https://github.com/rust-num/num-traits/blob/master/RELEASES.md )
- [Commits](https://github.com/rust-num/num-traits/compare/num-traits-0.2.15...num-traits-0.2.16 )
---
updated-dependencies:
- dependency-name: num-traits
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-21 09:00:43 +00:00
732ae81f0b
Merge pull request #8290 from influxdata/jrb_67_update_new_file_at_for_l2
...
fix: update new_file_at following all compactions
2023-07-20 14:13:04 -05:00
629f9d20db
fix: update new_file_at following all compactions
2023-07-20 13:27:54 -05:00
3dd29384ed
fix: Remove unneeded allow unused_imports and remove unused imports
2023-07-20 14:20:55 -04:00
fce4f3f346
test: Remove outdated test comments
...
This is not at all what this test is doing; probably copypasta
2023-07-20 14:13:15 -04:00
8575f3c022
chore(docs): add `influxdb_iox debug build-catalog` to debug.md ( #8273 )
...
* chore(docs): add `influxdb_iox debug build-catalog` to debug.md
chore(docs): wording
chore(docs): rephrase
* chore: remove personal shell prompt and make cmds cleaner
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 15:42:19 +00:00
1bed99567c
chore: add DF metrics to compaction spans ( #8270 )
...
* chore: add DF metrics to compaction spans
* chore: update string for test verification
* chore: update comment
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 15:00:22 +00:00
668a1c3d8e
fix: aggregate fns called on tags should return null ( #8274 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 14:55:16 +00:00
58181bab0a
Merge pull request #8285 from influxdata/savage/allow-kafka-columns-to-be-nullable
...
refactor(catalog): Allow kafka columns to be nullable
2023-07-20 14:25:23 +00:00
bdc884e6e8
Merge branch 'main' into savage/allow-kafka-columns-to-be-nullable
2023-07-20 14:20:42 +00:00
a49a0aaa21
chore(deps): Bump zstd from 0.12.3+zstd.1.5.2 to 0.12.4 ( #8280 )
...
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.12.3+zstd.1.5.2 to 0.12.4.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-20 14:14:08 +00:00
0173c50ba1
fix: use correct error code when querier is shutting down ( #8282 )
...
When a long running query is in process and the querier is shutting
down, it might happen that the executor (= thread pool and tokio
executor responsible for the CPU-bound DataFusion execution) is shut
down while the query is running. From a "systems interaction" PoV I
think this is totally fine and I would like to avoid some weird
ref-counting. Or in other words: if the system is shutting down, shut it
down.
However the error was treated as "internal" which is not useful. The
client should rather be informed that its server was gone and that it is
OK (and desired) to retry. So as per
<https://grpc.github.io/grpc/core/md_doc_statuscodes.html > I think this
should signal "unavailable".
This change wires the error code in such a way that the gRPC service
layer can properly inspect it and then changes the error mapping.
Ref https://github.com/influxdata/idpe/issues/17917 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 12:08:22 +00:00
9263b42c35
chore(deps): Bump clap from 4.3.16 to 4.3.17 ( #8277 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.16 to 4.3.17.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.16...v4.3.17 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 12:03:06 +00:00
6ae9143742
fix: `end_to_end_cases::cli::query_ingester` flakyness ( #8281 )
...
While I cannot reproduce the CI flakyness locally (probably because the
local system is fast enough), looking at the test convinced me that the
ingester should not persist.
Closes #8245 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 11:48:59 +00:00
e894ea73f7
refactor(catalog): Allow kafka columns to be nullable
2023-07-20 11:18:02 +01:00
d3432198b6
revert: batch partition catalog requests in querier ( #8269 ) ( #8283 )
...
Panics in prod.
This reverts commit 0c347e8e64
2023-07-20 09:42:40 +00:00
0994f54af4
chore(deps): Bump sqlparser from 0.35.0 to 0.36.0 ( #8279 )
...
* chore(deps): Bump sqlparser from 0.35.0 to 0.36.0
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.35.0 to 0.36.0.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.35.0...v0.36.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 09:30:14 +00:00
0f2cbcaef5
chore(deps): Bump zstd-safe from 6.0.5+zstd.1.5.4 to 6.0.6 ( #8275 )
...
Bumps [zstd-safe](https://github.com/gyscos/zstd-rs ) from 6.0.5+zstd.1.5.4 to 6.0.6.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd-safe
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-20 09:05:54 +00:00
4eff0fb359
chore(deps): Bump tower-http from 0.4.1 to 0.4.2 ( #8278 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.4.1...tower-http-0.4.2 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 08:36:49 +00:00
0c347e8e64
feat: batch partition catalog requests in querier ( #8269 )
...
This is mostly wiring that builds on top of the other PRs linked to #8089 .
I think we eventually could make the batching code nicer by adding
better wrappers / helpers, but lets do that if we have other batched
caches and this patterns proofs to be useful.
Closes #8089 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 08:31:49 +00:00
59c532ee56
chore(deps): Bump serde from 1.0.171 to 1.0.173 ( #8276 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.171 to 1.0.173.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.171...v1.0.173 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 08:05:38 +00:00
2f2fcb6f05
refactor(influxql): make DERIVATIVE a user-defined window function ( #8265 )
...
Now that user-defined window functions are available, change the
DERIVATIVE and NON_NEGATIVE_DERIVATIVE function implementations to
use user-defined windows functions. This should improve performance
by allowing the entire window to be processed in one go, rather
than processing one row at a time.
The implementation is also moved out of the planner module alongside
the other user-defined window functions.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-20 06:19:02 +00:00
b3fca844d5
chore: update doc to `remote store get-table` ( #8261 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-19 20:23:41 +00:00
2519c9c6d2
Merge pull request #8262 from influxdata/jrb_65_move_progress_reporting
...
fix(compactor timeout): chunk up high plan counts in the compactor
2023-07-19 14:16:48 -05:00
ccb950445c
Merge remote-tracking branch 'origin/main' into jrb_65_move_progress_reporting
2023-07-19 14:05:48 -05:00
4c5c2473e6
chore: move scratchpad cleanup earlier
2023-07-19 10:16:28 -05:00
4e88571142
feat: add batch partition getters ( #8268 )
2023-07-19 15:05:41 +00:00
004b401a05
chore: upgrade to sqlx 0.7.1 ( #8266 )
...
There are a bunch of dependencies in `Cargo.lock` that are related to
mysql. These are NOT compiled at all, and are also not part of `cargo
tree`. The reason for the inclusion is a bug in cargo:
https://github.com/rust-lang/cargo/issues/10801
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-19 12:18:57 +00:00
d3b3805e5f
chore(deps): Bump clap from 4.3.15 to 4.3.16 ( #8263 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.15 to 4.3.16.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.15...v4.3.16 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-19 09:26:04 +00:00
fbbaf155ec
feat: ability to batch cache loader requests ( #8259 )
...
Core functionality for #8089 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-19 09:20:40 +00:00
69e4f998dd
fix: chunk up high plan counts in the compactor
2023-07-18 15:16:04 -05:00
b79009b128
refactor: refactor `Schemas` type in InfluxQL planner ( #8237 )
...
* refactor: refactor `Schemas` type in InfluxQL planner
* chore: update after merging main
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-18 17:14:39 +00:00
efae0f108a
feat(idpe-17887): enable `/` in db name for v1 write. ( #8235 )
...
* test case for proposed new behavior in v1 write endpoint.
* autogen and default are equivalent reserved words for rp
* have write endpoint match query endpoint, in that db and rp are always concated
2023-07-18 09:36:25 -07:00
1723aa46a6
docs: adjust querier cache docs ( #8260 )
...
`parquet_file` cache has a TTL as of #7975 .
2023-07-18 15:13:15 +00:00
67d1d41b11
chore(deps): Bump scopeguard from 1.1.0 to 1.2.0 ( #8255 )
...
Bumps [scopeguard](https://github.com/bluss/scopeguard ) from 1.1.0 to 1.2.0.
- [Commits](https://github.com/bluss/scopeguard/compare/v1.1.0...v1.2.0 )
---
updated-dependencies:
- dependency-name: scopeguard
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-18 12:49:34 +00:00
4c4c9f731c
chore(deps): Bump uuid from 1.4.0 to 1.4.1 ( #8256 )
...
Bumps [uuid](https://github.com/uuid-rs/uuid ) from 1.4.0 to 1.4.1.
- [Release notes](https://github.com/uuid-rs/uuid/releases )
- [Commits](https://github.com/uuid-rs/uuid/compare/1.4.0...1.4.1 )
---
updated-dependencies:
- dependency-name: uuid
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-18 12:30:42 +00:00
39450c54da
refactor: cache system code moves ( #8258 )
...
* refactor: fix `TestLoader` arg order
* refactor: extract test utils
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-18 12:20:27 +00:00
14f99f219b
chore(deps): Bump clap from 4.3.12 to 4.3.15 ( #8254 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.12 to 4.3.15.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.12...v4.3.15 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-18 09:18:58 +00:00
0bb9e059e5
refactor(influxql): make difference a user-defined window function ( #8257 )
...
Change the implementation of DIFFERENCE, and NON_NEGATIVE_DIFFERENCE,
to use a user-defined window function, rather than a user-defined
aggregator function. This slightly simplifies the implementation.
The window function calculates difference operation on the full
window in one operation rather than a row at a time. Where ther are
no NULL values in the input this function uses arrow compute kernels.
As part of this refactor the window function's implementation is
moved out of the planner module.
2023-07-18 09:03:58 +00:00
d1640bb926
feat(influxql): CUMULATIVE_SUM window function ( #8248 )
...
* feat(influxql): CUMULATIVE_SUM window function
Implement the InfluxQL CUMULATIVE_SUM window function. This is
implemented as described in
https://docs.influxdata.com/influxdb/v1.8/query_language/functions/#cumulative_sum .
* chore: Add a test demonstrating NULL handling of CUMULATIVE_SUM
---------
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-18 06:13:58 +00:00
33e41fc5cb
fix: improve error for malformed gap fill query ( #8252 )
...
* fix: improve error for malformed gap fill query
* fix: code review feedback
2023-07-17 21:20:34 +00:00
b916a89159
fix: recurse through SubqueryAlias when finding gap fill time range ( #8249 )
2023-07-17 19:39:30 +00:00
kodiakhq[bot]
Carol (Nichols || Goulding)
Fraser Savage
Andrew Lamb
dependabot[bot]
Joe-Blount
Joe-Blount
Carol (Nichols || Goulding)
Chunchun Ye
Christopher M. Wolff
Marco Neumann
Martin Hilton
wiedld