f2ae47f72e
Merge pull request #7392 from influxdata/cn/redo
...
fix: Only look at the current branch files' status when looking for concurrent modifications
2023-03-30 20:34:05 +00:00
a1389e5962
Merge branch 'main' into cn/redo
2023-03-30 20:27:25 +00:00
807082ae69
Merge pull request #7391 from influxdata/pjb-update-readme
...
chore: update readme after going through it.
2023-03-30 13:26:22 -07:00
144a0948a2
Merge branch 'main' into pjb-update-readme
2023-03-30 13:16:43 -07:00
4adb4975e1
chore: update readme after going through it.
2023-03-30 13:15:40 -07:00
8718aaa148
fix: Change test file ID to match intent
2023-03-30 16:04:52 -04:00
c37f908349
docs: Update comments based on the new criteria for changed files
2023-03-30 16:04:51 -04:00
bf026d1f74
fix: Only log that we've detected changed files we're about to compact
2023-03-30 15:10:35 -04:00
48b102f037
fix: Only check that existing files continue to exist at their current compaction level
2023-03-30 14:06:39 -04:00
0a51fd55a6
chore: move compaction progress notification up (to be more frequent) ( #7389 )
2023-03-30 17:49:09 +00:00
7e6518c7ca
Merge pull request #7390 from influxdata/cn/revert
...
revert: Merge pull request #7369
2023-03-30 16:44:44 +00:00
956d7bcee4
revert: "revert: Merge pull request #7369 from influxdata/cn/parquet-file-saved-status"
...
This reverts commit 0d7393f2c1
2023-03-30 12:39:03 -04:00
0d7393f2c1
revert: Merge pull request #7369 from influxdata/cn/parquet-file-saved-status
...
This reverts commit c320ed11d4555f2a67aa
2023-03-30 12:37:36 -04:00
c320ed11d4
Merge pull request #7369 from influxdata/cn/parquet-file-saved-status
...
feat: Save Parquet file state and check it remains the same before committing compaction
2023-03-30 15:12:43 +00:00
68abc42cda
test: Add more unit cases for SavedParquetFileState comparisons
2023-03-30 11:05:13 -04:00
a0890bf8d3
refactor: Extract a function for repeated code to get and save parquet file state
2023-03-30 11:05:13 -04:00
03e5b1ac6a
feat: Save Parquet file state and check it remains the same before committing compaction
2023-03-30 11:05:13 -04:00
8b360e013a
fix: Split long line in a macro that rustfmt doesn't handle
2023-03-30 11:05:13 -04:00
555f2a67aa
refactor(flightsql): Build `GetDBSchemas` directly from DataFusion catalog ( #7386 )
...
* refactor: Build `GetDBSchemas` directly from DataFusion catalog
* fix: Apply suggestions from code review
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 13:28:34 +00:00
5f372b022c
test: add new test for decoded table_schema ( #7385 )
...
Co-authored-by: Chunchun <14298407+appletreeisyellow@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 13:20:48 +00:00
26802c81e5
chore(deps): Bump clap from 4.1.11 to 4.2.1 ( #7378 )
...
* chore(deps): Bump clap from 4.1.11 to 4.2.1
Bumps [clap](https://github.com/clap-rs/clap ) from 4.1.11 to 4.2.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.1.11...v4.2.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 11:30:50 +00:00
86c55918ec
chore(deps): Bump tempfile from 3.4.0 to 3.5.0 ( #7376 )
...
* chore(deps): Bump tempfile from 3.4.0 to 3.5.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:15:54 +00:00
6091cf31bb
chore(deps): Bump prost-build from 0.11.7 to 0.11.8 ( #7379 )
...
Bumps [prost-build](https://github.com/tokio-rs/prost ) from 0.11.7 to 0.11.8.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/commits/v0.11.8 )
---
updated-dependencies:
- dependency-name: prost-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:04:05 +00:00
4a067a635b
test: improve query test filter normalization ( #7382 )
...
- Only redact the actual time value in `FilteExec`, not the entire
expression. This preserves important information about filter
pushdowns.
- Apply similar time filter to `ParquetExec` because with #6098 we will
push down more filters into `ParquetExec`, including retention
policies.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:50:31 +00:00
9ddca96336
chore(deps): Bump thiserror from 1.0.39 to 1.0.40 ( #7375 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.39 to 1.0.40.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.39...1.0.40 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:43:28 +00:00
d2f3f279f3
fix: projection pushdown w/ resorting ( #7381 )
...
We should resort properly when performing projection pushdown. Extended
test utils to actually catch this by checking the plan schemas.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:24:23 +00:00
4eedb7ea77
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 ( #7374 )
...
* chore(deps): Bump async-trait from 0.1.66 to 0.1.68
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.66 to 0.1.68.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 ( #7373 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
008270b346
chore(deps): Bump serde from 1.0.156 to 1.0.159 ( #7372 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.156 to 1.0.159.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.156...v1.0.159 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:23:03 +00:00
97dd61bfb1
chore(deps): Bump is-terminal from 0.4.5 to 0.4.6 ( #7371 )
...
* chore(deps): Bump is-terminal from 0.4.5 to 0.4.6
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal ) from 0.4.5 to 0.4.6.
- [Release notes](https://github.com/sunfishcode/is-terminal/releases )
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.5...v0.4.6 )
---
updated-dependencies:
- dependency-name: is-terminal
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 08:18:16 +00:00
066c3280eb
fix: phys. optimizers must respect sort partitioning ( #7362 )
...
* fix: sort pushdown must preserve partioning
* fix: projection pushdown must preserve sort partitioning
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 08:10:21 +00:00
19a0c7fe9c
feat: Teach InfluxQL how to process `FILL(null|previous|<value>)` ( #7359 )
...
* chore: Publicise gap-filling APIs
Helps #6916
* feat: IOx learns `FILL(null|previous|<value>)`
Helps #6916
* chore: More test cases
* chore: Revert change to TreeNodeVisitor
* chore: Update snapshot with expected gap-filling changes
2023-03-29 23:11:20 +00:00
f41c1a7945
feat: update gap fill planner rule to use LOCF ( #7358 )
...
* feat: update gap fill planner rule to use LOCF
* chore: cargo fmt
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 15:45:34 +00:00
d8b0139ea9
chore: Update datafusion + arrow/parquet/arrow-flight to 36 ( #7354 )
...
* chore: Update datafusion + arrow/parquet/arrow-flight to 36
* refactor: update optimize for new API
* refactor: update parquet for new API
* chore: Update more dependencies
* chore: Update to use the new buffer creation APIs
* chore: Run cargo hakari tasks
* fix: bad len
* fix: update for API change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 13:41:59 +00:00
208d83d982
feat(flight): Support influxql queries via JSON encoded tickets ( #7357 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 13:34:05 +00:00
581d402368
Merge pull request #7343 from influxdata/savage/add-tests-for-hot-partition-persistence
...
feat(ingester2): Add test coverage for hot partition persistence
2023-03-29 11:06:34 +00:00
2e0d7fa4a9
Merge branch 'main' into savage/add-tests-for-hot-partition-persistence
2023-03-29 11:59:48 +01:00
d2a561b177
chore(deps): Bump rustix from 0.36.11 to 0.37.4 ( #7360 )
...
* chore(deps): Bump rustix from 0.36.11 to 0.37.4
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.11 to 0.37.4.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.11...v0.37.4 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-29 10:48:35 +00:00
39856ad432
fix: projection pushdown should project `ParquetExec` ordering ( #7356 )
...
* fix: projection pushdown should project `ParquetExec` ordering
Bug found while working on the final steps for #6098 .
* fix: Update expected output
* test: make test even harder
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-03-29 09:05:19 +00:00
52e54e0f8d
feat: more aggressive `CombineChunks` ( #7355 )
...
Try to combine chunks even when not all Union-arms/inputs are
combinable. This will later help to transform
```yaml
---
union:
- parquet:
files: [f1]
- parquet:
files: [f2]
- dedup:
parquet:
files: [f3]
```
into
```yaml
---
union:
- parquet:
files: [f1, f2]
- dedup:
parquet:
files: [f3]
```
Helps #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 06:24:17 +00:00
75dba43ced
test: extend retention policy query test ( #7352 )
...
Add an ingester chunk to the parquet chunks.
Helpful for #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 05:32:22 +00:00
a26cb6032a
feat: InfluxQL learns LOG function (arity 2) ( #7333 )
2023-03-28 20:22:27 +00:00
43e236e040
chore: Update datafusion again ( #7353 )
...
* chore: Update DataFusion
* refactor: Update predicate crate for new transform API
* refactor: Update iox_query crate for new APIs
* refactor: Update influxql for new API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 16:21:49 +00:00
b277b620fa
Merge pull request #7326 from influxdata/jrb_21_backfill_handling
...
feat: vertical split files when L0s are highly overlapped
2023-03-28 10:57:53 -05:00
041f14ae50
Merge remote-tracking branch 'origin/main' into jrb_21_backfill_handling
2023-03-28 10:34:05 -05:00
f834df004b
chore: insta churn
2023-03-28 10:23:57 -05:00
a241ba1596
chore: sort files before selecting split candidate
2023-03-28 10:23:26 -05:00
dbf6493312
feat: add scalar function LOCF ( #7347 )
...
* feat: add scalar function LOCF
* chore: cargo update spin@0.9.6
Apparently this version was yanked
2023-03-28 14:35:27 +00:00
f16622fdc1
docs: add compactor invariants and rules ( #7346 )
...
* docs: add compactor invariants and rules
* chore: address review commnents
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 13:35:04 +00:00
95732fe221
docs: Update gRPC example in README ( #7345 )
...
Given that `ManagementService` no longer exists, the example is misleading and returns an error.
- Replace `ManagementService` example with `NamespaceService` examples.
- Revise instruction for providing `.proto` files and using the wrapper script.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 11:26:40 +00:00
kodiakhq[bot]
Phil Bracikowski
Phil Bracikowski
Carol (Nichols || Goulding)
Joe-Blount
Andrew Lamb
dependabot[bot]
Marco Neumann
Stuart Carnie
Christopher M. Wolff
Fraser Savage
Joe-Blount
Nga Tran
Jason Stirnaman