Commit Graph

313 Commits (377e10815230c604f84020696cc05e8bf890af9a)

Author SHA1 Message Date
Marco Neumann bc18c6dc5f
refactor: re-land #7815. (#7852)
* refactor: consolidate pruning code

Let's have a single chunk pruning implementation in our code, not two.

Also removes a bit of crust from `QueryChunk` since it is technically no
longer responsible for pruning (this part has been pushed into the
querier for early pruning and bits for the `iox_query_influxrpc` for
some RPC shenanigans).

* test: regression test for incident

* fix: chunk pruning

* docs: add some test notes
2023-05-24 09:46:49 +00:00
Dom Dwyer 928a4d163e
build: remove unused dependencies from crates
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).

I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!

    https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html

This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
2023-05-23 14:55:43 +02:00
Marco Neumann 6c0f50a473
revert: refactor: consolidate pruning code (#7815) (#7847)
This reverts commit db9fe92981.

Likely causing an incident, see https://app.incident.io/incidents/267 .
2023-05-23 08:01:53 +00:00
Marco Neumann db9fe92981
refactor: consolidate pruning code (#7815)
Let's have a single chunk pruning implementation in our code, not two.

Also removes a bit of crust from `QueryChunk` since it is technically no
longer responsible for pruning (this part has been pushed into the
querier for early pruning and bits for the `iox_query_influxrpc` for
some RPC shenanigans).
2023-05-22 08:42:20 +00:00
Andrew Lamb 6344fe8c3f
chore: Add rationale for `clippy::future_not_send` (#7822)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:58:56 +00:00
Marco Neumann 7e64264eef
refactor: remove `RedudantSort` optimizer pass (#7809)
* test: add dedup test for multiple partitions and ranges

* refactor: remove `RedudantSort` optimizer pass

Similar to #7807 this is now covered by DataFusion, as demonstrated by
the fact that all query tests (incl. explain tests) still pass.

The good thing is: passes that are no longer required don't require any
upstreaming, so this also closes #7411.
2023-05-17 09:30:04 +00:00
Marco Neumann 931b4488bd
refactor: remove `SortPushdown` optimizer pass (#7807)
DataFusion is now smart enough to do that using the builtin passes. No
`EXPLAIN` tests regressed.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-17 06:26:31 +00:00
Nga Tran ca12f1c03d
fix: correctly recurse in `ParquetSortness` (#7778)
* test: reproducer for idpe_17556

* fix: `ParquetSortness` and partial opt

1. correctly handle cases where `ParquetSortness` would optimize one
   child branch but not the other
2. handle cases where `ParquetSortness` recusion should stop a bit
   clearer (using `TreeNodeRewriter`)
3. rename query tests to be a bit clearer
4. add test case with many (but not too many) duplicate files and an
   ingester (basically a prod use case where the compactor is slightly
   behind)

---------

Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-17 06:09:23 +00:00
Marco Neumann d3ff945117
refactor: remove output sorting from scan provider (#7798)
This is somewhat a left-over from the old phys. plan construction where
we tried to fold in the sorts at the right place. Now the optimizer
takes care of that, so we can just express this as a standard logical
node (the same as SQL and InfluxQL). This makes the plan construction a
bit cleaner since the actual scan provider only performs the minimal
work that is required by DataFusion and the users (SQL, InfluxQL, reorg)
request what they actually need.

The tests in `iox_query::frontend::reorg::tests` that assert the tests
still pass and proof that the actual physical plans are identical w/
this approach.

Closes #7785.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-17 05:59:04 +00:00
Chunchun Ye 2bb6445668
chore: update DataFusion and arrow / arrow-flight / parquet to `39.0.0` (#7793)
* chore: update DataFusion and arrow/parquet/arrow-flight to 39.0.0

* chore: update DataFusion and arrow/parquet/arrow-flight to 39.0.0 in workspace-hack/Cargo.toml

* chore: Run cargo hakari tasks

* chore: fix CI test and lint

* chore: update csv schema

* refactor: remove type-annotate for `Arc`

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-16 13:42:26 +00:00
Andrew Lamb 7735e7c95b
chore: Update DataFusion again (#7777)
* chore: Update datafusion again

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-15 12:38:45 +00:00
Andrew Lamb 2860d87fe1
chore: Update DataFusion (#7756)
* chore: Update DataFusion pin

* chore: Update explain plans

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-05 18:58:18 +00:00
Christopher M. Wolff 55b35367ac
test: add test for gap fill query missing time bounds (#7747)
* test: add test for gap fill query missing time bounds

* chore: update unit test

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 21:01:45 +00:00
Christopher M. Wolff 05688799c4
fix: handle aliases in gapfill aggregate columns (#7725)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-03 15:20:14 +00:00
Andrew Lamb 2b1f8b56e2
chore: Update DataFusion (#7719)
* chore: Update DataFusion

* chore: update for API change

* chore: update some tests

* fix: Update plans in optimizer

* chore: Update plans

* chore: Update error messages

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-02 17:55:04 +00:00
Andrew Lamb 530ee94558
fix: use correct sort key in projection_pushdown (#7718)
* fix: use correct sort key in projection_pushdown

* fix: tabs in docs

* refactor: Use Serde to format test results
2023-05-02 16:50:04 +00:00
Christopher M. Wolff 493b26831d
fix: make influx RPC interface break up series into multiple frames (#7691)
* fix: make influx RPC interface break up series into multiple frames

* refactor: code review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-01 20:18:05 +00:00
Marco Neumann 0556fdae53
refactor: remove `QueryChunk::partition_sort_key` (#7680)
As of #7250 / #7449 the partition sort key is no longer required for
query planning. Instead we use a combination of
`QueryChunk::partition_id` and `QueryChunk::sort_key` which is more
robust and easier to reason about.

Removing it simplifies the querier code a lot since we no longer need to
have a sort key for the ingester chunks and also don't need to "sync"
the sort key between chunks for consistency.
2023-04-27 10:54:41 +00:00
dependabot[bot] bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 (#7667)
* chore(deps): Bump tokio from 1.27.0 to 1.28.0

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
Christopher M. Wolff 7a6862ee3a
refactor: let date_bin_gapfill allow omitted origin (#7595)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 14:56:52 +00:00
Marco Neumann d7dc305972
feat: allow overwriting DataFusion's default config (#7586)
This is helpful to test changes in our defaults but also for testing.

Required for https://github.com/influxdata/idpe/issues/17474 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-18 11:28:45 +00:00
Andrew Lamb f46d06d56f
chore: Update DataFusion + arrow ecosystem to 37 (#7544)
* chore: Update datafusion and arrow/parquet to 37, tonic to 0.9.1

* refactor: Update for FieldRef and other API changes

* fix: Update field size calculation

* fix: Use `NullBuffer` directly

* fix: remove outdated comment

* chore: Update test for tonic

* chore: Run cargo hakari tasks

* chore: cargo update

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-14 12:43:01 +00:00
Andrew Lamb 134ff2ef83
chore: update DataFusion pin (right before arrow 37 update) (#7540)
* chore: update DataFusion pin

* refactor: Update for deprecated API

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 17:25:24 +00:00
Andrew Lamb 3ebd07358b
chore: Update DataFusion pin, upgrade `date_bin` and `InfluxQL` to use `Interval(MonthDayNano)` (#7516)
* chore: Update datafusion

* chore: Update for change in PhysicalSortExpr

* refactor: Update date_bin_gapfill to take IntervalMonthDayNano, fix FlightSQL

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 10:43:32 +00:00
Christopher M. Wolff cbd747db44
feat: update gap fill planner rule to use `interpolate` (#7494)
* feat: add INTERPOLATE fn and update planner gap-fill planner rule

* test: add an end-to-end test for interpolate()
2023-04-12 21:51:44 +00:00
Christopher M. Wolff 0937615dba
fix: make interpolate() fill null values in input (#7490)
* fix: make interpolate() fill null values in input

* chore: cargo doc
2023-04-12 21:41:11 +00:00
Christopher M. Wolff 3e60369eff
refactor: input buffering for gap filling interpolate null-as-missing (#7478)
* refactor: move logic for knowing how much to buffer into GapFiller

* chore: clippy

* chore: add some clarifying comments

* refactor: clean up relationships between gap filling types

* refactor: remove use of RefCell from BufferedInput

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 21:08:51 +00:00
Andrew Lamb 8c42fedf33
chore: Remove dead code (#7475)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-11 10:44:49 +00:00
Andrew Lamb 1a80b8073c
fix: Improve span names for query access (#7476)
* fix: Improve span names for query access

* fix: update test

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-11 10:34:09 +00:00
Marco Neumann 5f43f2a719
refactor: remove old query planning code (#7449)
Closes #7406.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-06 16:05:08 +00:00
Marco Neumann 30b1878171
test: `ChunkTableProvider::scan` + fix "not dedup" (#7448)
1. Add loads of tests for `ChunkTableProvider::scan` (= the naive phys.
   plan before running any phys. optimizers)
2. Fix interaction of "no de-dup" and predicate pushdown. This might
   be used by the ingester at some point and I would like to have this
   correct before someone silently introduces a bug by pushing field
   predicates into the ingester.

This is mostly prep-work for #7406 so I know that test coverage is
sufficient.
2023-04-06 08:39:53 +00:00
Andrew Lamb e8b7d69b0f
chore: Update datafusion again (#7442)
* chore: Update datafusion

* chore: Fix up plans for datafusion API change

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-05 18:21:53 +00:00
Andrew Lamb 94d390f31e
test: Add additional tests for reorg plans (#7444)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-05 11:15:23 +00:00
Christopher M. Wolff d57a4f8947
refactor: make null-as-missing default behavior for LOCF (#7443)
* refactor: make null-as-missing default behavior for LOCF

* test: update InfluxQL test

---------

Co-authored-by: Christopher Wolff <cwolff@athena.tail244ec.ts.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 18:03:09 +00:00
Andrew Lamb badc8865ef
chore: Update datafusion again (#7440)
* chore: Update DataFusion

* chore: Update for new API

* chore: Run cargo hakari tasks

* fix: cargo doc

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-04 15:45:46 +00:00
dependabot[bot] 66982f988b
chore(deps): Bump object_store from 0.5.5 to 0.5.6 (#7433)
Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.5 to 0.5.6.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md)
- [Commits](https://github.com/apache/arrow-rs/commits)

---
updated-dependencies:
- dependency-name: object_store
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 08:43:34 +00:00
Marco Neumann e9bdf96457
refactor: remove DF-clean-DF phys. optimizer pass hack (#7428)
As discussed in https://github.com/influxdata/influxdb_iox/pull/7250#discussion_r1155684471

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 08:09:35 +00:00
Marco Neumann f04962d630
feat: new query planning (#7250)
Closes #6098.
2023-04-03 10:31:03 +00:00
Marco Neumann e3b802cd25
feat: "parquet sortness" optimizer pass (#7383)
* feat: "parquet sortness" optimizer pass

Trade wider fan-out for the not having to fully sort parquet files.

For #6098.

* test: rename

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-31 08:01:33 +00:00
Marco Neumann 2d7bff91b5
feat: allow gap-fill logical opt. to handle inline filters (#7384)
With #6098 our `TableProvider` will declare `supports_filter_pushdown`
as "exact" since we handle the predicate pushdown ourselves. This has
two effects:

1. The phys. plan no longer contains an additional `FilterExec` node
   even if we already do all the correct filtering. This will improve
   performance.
2. The logical plan no longer contains a `Filter` node but instead the
   predicate is part of the `TableScan`. This simplifies the logical
   plan.

For (2) we need to adjust the gap fill logical optimizer to find the
time range again. Otherwise the optimizer pass will fail (which is
currently somewhat swallowed by DataFusion even though it is logged) and
the physical plan will contain our placeholder UDFs that are not
executable.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-31 06:09:51 +00:00
Marco Neumann d2f3f279f3
fix: projection pushdown w/ resorting (#7381)
We should resort properly when performing projection pushdown. Extended
test utils to actually catch this by checking the plan schemas.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:24:23 +00:00
dependabot[bot] 9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 (#7373)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
Marco Neumann 066c3280eb
fix: phys. optimizers must respect sort partitioning (#7362)
* fix: sort pushdown must preserve partioning

* fix: projection pushdown must preserve sort partitioning

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 08:10:21 +00:00
Stuart Carnie 19a0c7fe9c
feat: Teach InfluxQL how to process `FILL(null|previous|<value>)` (#7359)
* chore: Publicise gap-filling APIs

Helps #6916

* feat: IOx learns `FILL(null|previous|<value>)`

Helps #6916

* chore: More test cases

* chore: Revert change to TreeNodeVisitor

* chore: Update snapshot with expected gap-filling changes
2023-03-29 23:11:20 +00:00
Christopher M. Wolff f41c1a7945
feat: update gap fill planner rule to use LOCF (#7358)
* feat: update gap fill planner rule to use LOCF

* chore: cargo fmt

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 15:45:34 +00:00
Marco Neumann 39856ad432
fix: projection pushdown should project `ParquetExec` ordering (#7356)
* fix: projection pushdown should project `ParquetExec` ordering

Bug found while working on the final steps for #6098.

* fix: Update expected output

* test: make test even harder

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-03-29 09:05:19 +00:00
Marco Neumann 52e54e0f8d
feat: more aggressive `CombineChunks` (#7355)
Try to combine chunks even when not all Union-arms/inputs are
combinable. This will later help to transform

```yaml
---
union:
  - parquet:
      files: [f1]
  - parquet:
      files: [f2]
  - dedup:
      parquet:
        files: [f3]
```

into

```yaml
---
union:
  - parquet:
      files: [f1, f2]
  - dedup:
      parquet:
        files: [f3]

```

Helps #6098.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-29 06:24:17 +00:00
Andrew Lamb 43e236e040
chore: Update datafusion again (#7353)
* chore: Update DataFusion

* refactor: Update predicate crate for new transform API

* refactor: Update iox_query crate for new APIs

* refactor: Update influxql for new API

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 16:21:49 +00:00
Christopher M. Wolff dbf6493312
feat: add scalar function LOCF (#7347)
* feat: add scalar function LOCF

* chore: cargo update spin@0.9.6

Apparently this version was yanked
2023-03-28 14:35:27 +00:00
Marco Neumann 71b88b22b9
fix: ensure we don't loose predicates in chunk roundtrips (#7340)
`extract_chunks` never runs after predicate pushdown. However IF this
should ever happen, we would potentially forget the predicates attached
to `ParquetExec`. So let's make sure we refuse chunk extraction in this
case. This is similar to the existing behavior, i.e. we don't support
chunk extraction after filter pushdown (i.e. if there is a filter around
an `RecordBatchesExec`).

For #6098.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-27 11:18:56 +00:00