8b360e013a
fix: Split long line in a macro that rustfmt doesn't handle
2023-03-30 11:05:13 -04:00
555f2a67aa
refactor(flightsql): Build `GetDBSchemas` directly from DataFusion catalog ( #7386 )
...
* refactor: Build `GetDBSchemas` directly from DataFusion catalog
* fix: Apply suggestions from code review
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 13:28:34 +00:00
5f372b022c
test: add new test for decoded table_schema ( #7385 )
...
Co-authored-by: Chunchun <14298407+appletreeisyellow@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 13:20:48 +00:00
26802c81e5
chore(deps): Bump clap from 4.1.11 to 4.2.1 ( #7378 )
...
* chore(deps): Bump clap from 4.1.11 to 4.2.1
Bumps [clap](https://github.com/clap-rs/clap ) from 4.1.11 to 4.2.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.1.11...v4.2.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 11:30:50 +00:00
86c55918ec
chore(deps): Bump tempfile from 3.4.0 to 3.5.0 ( #7376 )
...
* chore(deps): Bump tempfile from 3.4.0 to 3.5.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:15:54 +00:00
6091cf31bb
chore(deps): Bump prost-build from 0.11.7 to 0.11.8 ( #7379 )
...
Bumps [prost-build](https://github.com/tokio-rs/prost ) from 0.11.7 to 0.11.8.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/commits/v0.11.8 )
---
updated-dependencies:
- dependency-name: prost-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:04:05 +00:00
4a067a635b
test: improve query test filter normalization ( #7382 )
...
- Only redact the actual time value in `FilteExec`, not the entire
expression. This preserves important information about filter
pushdowns.
- Apply similar time filter to `ParquetExec` because with #6098 we will
push down more filters into `ParquetExec`, including retention
policies.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:50:31 +00:00
9ddca96336
chore(deps): Bump thiserror from 1.0.39 to 1.0.40 ( #7375 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.39 to 1.0.40.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.39...1.0.40 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:43:28 +00:00
d2f3f279f3
fix: projection pushdown w/ resorting ( #7381 )
...
We should resort properly when performing projection pushdown. Extended
test utils to actually catch this by checking the plan schemas.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:24:23 +00:00
4eedb7ea77
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 ( #7374 )
...
* chore(deps): Bump async-trait from 0.1.66 to 0.1.68
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.66 to 0.1.68.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 ( #7373 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
008270b346
chore(deps): Bump serde from 1.0.156 to 1.0.159 ( #7372 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.156 to 1.0.159.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.156...v1.0.159 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:23:03 +00:00
97dd61bfb1
chore(deps): Bump is-terminal from 0.4.5 to 0.4.6 ( #7371 )
...
* chore(deps): Bump is-terminal from 0.4.5 to 0.4.6
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal ) from 0.4.5 to 0.4.6.
- [Release notes](https://github.com/sunfishcode/is-terminal/releases )
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.5...v0.4.6 )
---
updated-dependencies:
- dependency-name: is-terminal
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 08:18:16 +00:00
066c3280eb
fix: phys. optimizers must respect sort partitioning ( #7362 )
...
* fix: sort pushdown must preserve partioning
* fix: projection pushdown must preserve sort partitioning
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 08:10:21 +00:00
19a0c7fe9c
feat: Teach InfluxQL how to process `FILL(null|previous|<value>)` ( #7359 )
...
* chore: Publicise gap-filling APIs
Helps #6916
* feat: IOx learns `FILL(null|previous|<value>)`
Helps #6916
* chore: More test cases
* chore: Revert change to TreeNodeVisitor
* chore: Update snapshot with expected gap-filling changes
2023-03-29 23:11:20 +00:00
f41c1a7945
feat: update gap fill planner rule to use LOCF ( #7358 )
...
* feat: update gap fill planner rule to use LOCF
* chore: cargo fmt
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 15:45:34 +00:00
d8b0139ea9
chore: Update datafusion + arrow/parquet/arrow-flight to 36 ( #7354 )
...
* chore: Update datafusion + arrow/parquet/arrow-flight to 36
* refactor: update optimize for new API
* refactor: update parquet for new API
* chore: Update more dependencies
* chore: Update to use the new buffer creation APIs
* chore: Run cargo hakari tasks
* fix: bad len
* fix: update for API change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 13:41:59 +00:00
208d83d982
feat(flight): Support influxql queries via JSON encoded tickets ( #7357 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 13:34:05 +00:00
581d402368
Merge pull request #7343 from influxdata/savage/add-tests-for-hot-partition-persistence
...
feat(ingester2): Add test coverage for hot partition persistence
2023-03-29 11:06:34 +00:00
2e0d7fa4a9
Merge branch 'main' into savage/add-tests-for-hot-partition-persistence
2023-03-29 11:59:48 +01:00
d2a561b177
chore(deps): Bump rustix from 0.36.11 to 0.37.4 ( #7360 )
...
* chore(deps): Bump rustix from 0.36.11 to 0.37.4
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.11 to 0.37.4.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.11...v0.37.4 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-29 10:48:35 +00:00
39856ad432
fix: projection pushdown should project `ParquetExec` ordering ( #7356 )
...
* fix: projection pushdown should project `ParquetExec` ordering
Bug found while working on the final steps for #6098 .
* fix: Update expected output
* test: make test even harder
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-03-29 09:05:19 +00:00
52e54e0f8d
feat: more aggressive `CombineChunks` ( #7355 )
...
Try to combine chunks even when not all Union-arms/inputs are
combinable. This will later help to transform
```yaml
---
union:
- parquet:
files: [f1]
- parquet:
files: [f2]
- dedup:
parquet:
files: [f3]
```
into
```yaml
---
union:
- parquet:
files: [f1, f2]
- dedup:
parquet:
files: [f3]
```
Helps #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 06:24:17 +00:00
75dba43ced
test: extend retention policy query test ( #7352 )
...
Add an ingester chunk to the parquet chunks.
Helpful for #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 05:32:22 +00:00
a26cb6032a
feat: InfluxQL learns LOG function (arity 2) ( #7333 )
2023-03-28 20:22:27 +00:00
43e236e040
chore: Update datafusion again ( #7353 )
...
* chore: Update DataFusion
* refactor: Update predicate crate for new transform API
* refactor: Update iox_query crate for new APIs
* refactor: Update influxql for new API
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 16:21:49 +00:00
b277b620fa
Merge pull request #7326 from influxdata/jrb_21_backfill_handling
...
feat: vertical split files when L0s are highly overlapped
2023-03-28 10:57:53 -05:00
041f14ae50
Merge remote-tracking branch 'origin/main' into jrb_21_backfill_handling
2023-03-28 10:34:05 -05:00
f834df004b
chore: insta churn
2023-03-28 10:23:57 -05:00
a241ba1596
chore: sort files before selecting split candidate
2023-03-28 10:23:26 -05:00
dbf6493312
feat: add scalar function LOCF ( #7347 )
...
* feat: add scalar function LOCF
* chore: cargo update spin@0.9.6
Apparently this version was yanked
2023-03-28 14:35:27 +00:00
f16622fdc1
docs: add compactor invariants and rules ( #7346 )
...
* docs: add compactor invariants and rules
* chore: address review commnents
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 13:35:04 +00:00
95732fe221
docs: Update gRPC example in README ( #7345 )
...
Given that `ManagementService` no longer exists, the example is misleading and returns an error.
- Replace `ManagementService` example with `NamespaceService` examples.
- Revise instruction for providing `.proto` files and using the wrapper script.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 11:26:40 +00:00
4466e16018
chore(deps): Bump reqwest from 0.11.15 to 0.11.16 ( #7349 )
...
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.15 to 0.11.16.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.15...v0.11.16 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 11:19:51 +00:00
1ad0160d30
chore(deps): Bump serde_json from 1.0.94 to 1.0.95 ( #7348 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.94 to 1.0.95.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.94...v1.0.95 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 11:09:59 +00:00
5a965e2481
Merge branch 'main' into savage/add-tests-for-hot-partition-persistence
2023-03-28 11:37:20 +01:00
c7b3565990
refactor: Compare data inside the partition as persisted
...
Use tokio::task::yield_now to allow the persist task to
be enqueued instead of tokio::time::advance.
2023-03-28 11:28:53 +01:00
621672b38f
test: Add an e2e test for write replication and fix it because it didn't work 😜 ( #7344 )
...
* test: Add an e2e test for write replication
* fix: Pass through rpc_write_replicas configuration to RpcWrite handler
---------
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-28 09:31:16 +00:00
f588bd5e8d
chore: update `spin` to 0.9.7 ( #7350 )
...
0.9.6 was yanked.
2023-03-28 08:47:25 +00:00
0db347bc1a
Merge remote-tracking branch 'origin/main' into jrb_21_backfill_handling
...
# Conflicts:
# compactor2/tests/layouts/knobs.rs
# compactor2/tests/layouts/large_files.rs
# compactor2/tests/layouts/many_files.rs
2023-03-27 14:20:16 -05:00
638a8ea4a8
chore: move random backfill tests to their own file
2023-03-27 13:48:16 -05:00
ed282ed738
chore: add case to cause non-overlapped file to be processed by high_l0_overlap_split
2023-03-27 13:40:06 -05:00
6a60914f95
chore: insta updates
2023-03-27 11:15:24 -05:00
af71e3295c
Merge pull request #7342 from influxdata/dom/cleanup-mock-iter
...
refactor(test): generic return iterator
2023-03-27 15:01:25 +01:00
de043f93e5
feat(ingester2): Add test coverage for hot partition persistence
...
This commit adds a test for the hot partition persistence
post-write observer functionality.
2023-03-27 12:46:54 +01:00
874325d9ce
refactor(test): generic return iterator
...
This change allows the MockWriteClient to accept more input types and
internalises the box dyn bits; I got tired of writing "Box::new()"
everywhere.
2023-03-27 13:31:31 +02:00
71b88b22b9
fix: ensure we don't loose predicates in chunk roundtrips ( #7340 )
...
`extract_chunks` never runs after predicate pushdown. However IF this
should ever happen, we would potentially forget the predicates attached
to `ParquetExec`. So let's make sure we refuse chunk extraction in this
case. This is similar to the existing behavior, i.e. we don't support
chunk extraction after filter pushdown (i.e. if there is a filter around
an `RecordBatchesExec`).
For #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-27 11:18:56 +00:00
44551e7519
chore(deps): Bump regex from 1.7.2 to 1.7.3 ( #7338 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.2 to 1.7.3.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.2...1.7.3 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-27 09:20:35 +00:00
e067cfd3d7
Merge pull request #7316 from influxdata/dom/replicate
...
feat: write replication
2023-03-27 10:13:11 +01:00
48333f66a4
Merge branch 'main' into dom/replicate
2023-03-27 10:06:07 +01:00
Carol (Nichols || Goulding)
Andrew Lamb
dependabot[bot]
Marco Neumann
Stuart Carnie
Christopher M. Wolff
kodiakhq[bot]
Fraser Savage
Joe-Blount
Joe-Blount
Nga Tran
Jason Stirnaman
Carol (Nichols || Goulding)
Dom