8670b28445
Merge branch 'main' into sgc/issue/7600_moving_average
2023-06-18 09:41:19 +10:00
5889c96501
chore: Update `datafusion` and other dependencies ( #7981 )
...
* chore: Update DatFaFusion pin
* chore: Update other dependencies
* chore: Update hakari
* fix: Update for API changes
* fix: Update explain plan
* fix: Update influxql plans
* fix: rustdoc links
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-16 09:48:55 +00:00
2407be8062
feat: trialed retractable UDAF
...
Unfortunately, this is not suitable when the source data has nulls,
as InfluxQL OG ignores these values.
2023-06-16 13:10:47 +10:00
22baf5096e
feat: define place-holder UDFs for `moving_average` and `difference`
2023-06-16 13:10:47 +10:00
bd4e7e563f
chore: rustfmt 🧹
2023-06-15 12:42:32 +10:00
5c9e8dba5b
chore: Fix tests and clippy warnings
2023-06-15 12:32:45 +10:00
e55f45fee2
chore: expand time range using `extra_intervals` and capture ordering
2023-06-15 12:23:23 +10:00
3dd6c9a6e2
feat: distinguish window from aggregate functions
2023-06-15 12:19:37 +10:00
32b668c848
chore: match InfluxQL OG behaviour for time ranges
2023-06-15 12:15:57 +10:00
7bd2a7bfdb
chore: Extract interval duration and offset from `TIME`
2023-06-15 12:15:56 +10:00
f114842711
feat: Push outer query time-range to subqueries
...
Added additional end-to-end tests to validate time-range behaviour
2023-06-06 16:33:01 +10:00
8c71f30145
chore: Don't wrap `TimeRange` in an `Option`
...
It is unnecessary, because both the lower and upper are already
optional values.
2023-06-06 15:07:47 +10:00
9e2550c933
Merge branch 'main' into sgc/issue/7829_time_bounds_3
...
# Conflicts:
# iox_query_influxql/src/plan/planner.rs
2023-06-06 12:55:43 +10:00
01a22f8a2c
chore: Simplify WHERE clause handling
...
We don't need `Context` in the expression rewriting, which eliminates
a redundant `Context::default()` call in all the `SHOW` (metadata)
queries. Also simplified the time-range handling of metadata queries,
as we now have API to extract the time range from a `WHERE` clause
predicate and add the time filter to the same `Filter` logical plan
node.
2023-06-06 12:54:05 +10:00
f571aeb445
chore: Update DataFusion pin ( #7916 )
...
* chore: Update DataFusion pin
* chore: Update cargo
* fix: update for API changes
* fix: Update plans
* chore: Update for new api
* fix: Update plans
* chore: Update for API changes more
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-05 18:38:59 +00:00
d8c2f2c679
refactor: Simplify `TimeRange` to match InfluxQL OG behaviour explicitly
2023-06-05 15:14:13 +10:00
8c51d855e4
chore: `super` is shorter
2023-06-02 13:56:58 +10:00
61e19c9036
refactor: Use InfluxQL implementation for simplifying time ranges
2023-06-02 13:40:02 +10:00
d5719f9be2
refactor: Moved simplification of time range expressions to parser
2023-06-02 09:50:01 +10:00
8c02f81456
chore: Add some docs to the execution_props API
2023-05-31 12:31:38 +10:00
d9d7419693
refactor: move time range logic to separate module
...
The `rewrite_expression` module was getting large, so made sense to
move time range logic to its own module.
2023-05-31 12:28:03 +10:00
0a7e162911
refactor: rename into planner submodule
2023-05-31 12:14:12 +10:00
8f2c235b15
refactor: these are APIs for transforming InfluxQL expressions
2023-05-31 12:09:32 +10:00
600ed6652c
refactor: rewrite time-range expressions to a single range
...
Fixes gap filling, which was confused by multiple lower or upper
time bounds.
2023-05-30 15:46:45 +10:00
2a07b53879
feat: add more tag predicate rewrite logic for InfluxQL ( #7869 )
...
* feat: add more tag predicate rewrite logic for InfluxQL
* chore: cargo fmt
* chore: fmt
* test: add more tests
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-26 21:53:52 +00:00
6c8cab1073
chore: Separate scope from context, as it is a separate concern
2023-05-26 12:12:11 +10:00
377e108152
chore: No need to be exported from current module
2023-05-26 12:12:10 +10:00
928a4d163e
build: remove unused dependencies from crates
...
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).
I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!
https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html
This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
2023-05-23 14:55:43 +02:00
46031b2031
chore: rustfmt 🧹
2023-05-23 10:10:41 +10:00
66623398a8
chore: add some additional tests to validate multiple data sources
2023-05-23 09:52:01 +10:00
963e9ea60e
chore: Use BTreeSet per PR feedback
...
See: https://github.com/influxdata/influxdb_iox/pull/7808#discussion_r1200974643
2023-05-23 09:43:05 +10:00
e9cbe4e589
chore: Fix broken reference
2023-05-22 16:55:41 +10:00
50289ebab7
chore: Update insta snapshots
2023-05-22 16:30:55 +10:00
c9b6695a44
chore: No need to use deref
2023-05-22 16:30:38 +10:00
af76865b2c
feat: complete InfluxQL subquery compatibility
...
Closes #7794
2023-05-22 16:22:54 +10:00
ccd73a0b32
chore: remove duplicate code
2023-05-22 16:20:26 +10:00
e88d8befdb
Merge branch 'main' into sgc/issue/7794_subquery_inconsistency
2023-05-19 18:45:58 +10:00
4d069800dd
feat: Generate plan for each table and then union final plans together
...
Closes #7794
2023-05-19 18:45:35 +10:00
8bb98ab21c
chore: Improve docs
2023-05-19 09:24:33 +10:00
6344fe8c3f
chore: Add rationale for `clippy::future_not_send` ( #7822 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:58:56 +00:00
04617a4362
Merge branch 'main' into sgc/issue/7794_subquery_inconsistency
2023-05-18 07:24:00 +10:00
62fed73bcd
refactor: upgrade DataFusion to `19b03240920ad63cac916b42951754c0337bdac8#19b03240920ad63cac916b42951754c0337bdac8` ( #7813 )
...
I need:
- https://github.com/apache/arrow-datafusion/pull/6226 .
Changes in code due to:
- https://github.com/apache/arrow-datafusion/pull/6332
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-17 13:57:12 +00:00
39cffcbc92
chore: Add API to find all VarRef instances of a `SELECT` statement
2023-05-17 16:43:16 +10:00
b299937d18
chore: Resolve tags in GROUP BY clause
...
Any `VarRef`s in the `Select` where the `data_type` field is `None`
indicates there is no field or tag found in the schema of any children
in the `FROM` clause
2023-05-17 14:55:10 +10:00
9cecb3504c
chore: Remove redundant call to `to_lowercase`
...
Function call names are normalised to lowercase by the parser.
2023-05-17 14:32:57 +10:00
c9f8151302
feat: Resolve data types for `VarRef` nodes in the `WHERE` condition
2023-05-17 12:31:37 +10:00
76aa1ac3d6
feat: Refactor node used to represent a tag in a `GROUP BY`
...
* Use `VarRef` for `Tag`, to ensure a consistent representation of a
column reference across `GROUP BY`, `SELECT` projection and `WHERE`
clause.
* Rename `tags` to `tag_names`
* `tags` now returns `VarRef`
2023-05-17 12:29:02 +10:00
90328e5d0f
chore: PR feedback
2023-05-17 09:24:55 +10:00
d9729ede7d
chore: Fix rustdoc links
2023-05-16 09:21:49 +10:00
2e30442a86
Merge branch 'main' into sgc/issue/6879_subquery_01
2023-05-16 09:15:03 +10:00
4068249349
chore: Fix rustdoc links
2023-05-16 09:13:08 +10:00
7735e7c95b
chore: Update DataFusion again ( #7777 )
...
* chore: Update datafusion again
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-15 12:38:45 +00:00
7ba619a32b
feat: outer GROUP BY pushed down to subqueries; more Cloud 2 examples
2023-05-15 15:31:20 +10:00
e75a95bca7
chore: clippy 🧹
2023-05-15 15:30:39 +10:00
62a4c02836
feat: Handle default FILL behaviour for subqueries
2023-05-15 11:22:26 +10:00
c77c4b3d23
feat: support nested aggregate subqueries
2023-05-15 09:31:06 +10:00
0dd1826e3c
feat: add the merged tag set for the `Select` query
2023-05-15 09:30:28 +10:00
f7a8850f62
chore: docs
2023-05-15 09:29:45 +10:00
914d2a9e02
chore: docs
2023-05-15 07:27:47 +10:00
553476c646
fix: time should be added to all subqueries of arbitrary depth
2023-05-15 07:22:24 +10:00
4e96f814db
chore: Improve docs
2023-05-15 07:21:36 +10:00
f4a19fc6c1
fix: Aggregate subqueries with push-down `GROUP BY tags`
2023-05-12 16:53:16 +10:00
87dca540a7
feat: More subqueries succeeding
2023-05-12 13:55:42 +10:00
c2335b9f26
chore: Add compatibility mode, to follow InfluxQL OG behaviour 😬
2023-05-12 13:53:59 +10:00
1a947d6bf1
chore: Planner can use Arrow type
2023-05-12 12:50:16 +10:00
248f596bda
chore: Additional subquery tests
2023-05-12 12:49:26 +10:00
9560e50fa2
chore: `let` syntax is clearer
2023-05-12 12:49:06 +10:00
92414f224f
feat: Planner can depend on `data_type` of `Field`
2023-05-12 12:48:12 +10:00
80545feee0
feat: `TypeEvaluator` needs to know about `DISTINCT`
2023-05-12 12:46:45 +10:00
e9e0f22d64
feat: Teach `TypeEvaluator` about scalar functions
2023-05-12 12:13:50 +10:00
687e2f1003
chore: Move TagSet and more cleanup
2023-05-11 12:42:43 +10:00
f234442534
chore: some initial tests for validating column schema inference
2023-05-11 09:46:38 +10:00
26c96b356f
chore: rename test function to avoid collisions with future tests
2023-05-11 09:46:15 +10:00
1d837b175e
feat: add schema support to DataSource::Subquery; fix incompatibilities
2023-05-10 14:33:34 +10:00
3c25272eeb
chore: More cleanup
2023-05-09 13:00:12 +10:00
707a700481
chore: Remove unnecessary Result type
2023-05-09 12:59:56 +10:00
54c063cc67
fix: field_and_dimensions doesn't error
2023-05-09 12:48:22 +10:00
01744865b2
fix: Don't skip explicit reference to Tag fields
2023-05-09 12:47:24 +10:00
bd760330f2
feat: Initial experimental subquery support
2023-05-09 09:42:49 +10:00
4fc5a90d60
chore: Update docs
2023-05-09 09:24:19 +10:00
74b5c32440
refactor: Cleanup rewriter
2023-05-09 09:24:19 +10:00
e4b6a8f77a
chore: Resolve aliases for all `SELECT` queries, not just the outer
2023-05-09 09:24:18 +10:00
9227bc032f
chore: Remove fragile `is_tag_field` API
2023-05-05 17:15:59 +10:00
3099733de6
chore: Introduce root `SelectQuery`
...
This is used to gather shared state for the query vs `Select`, which
represents state for each `SELECT` statement.
First example is tracking whether a statement projects multiple
unique measurements. This changed improved one of the plans, as it
no longer needs to sort the `iox::measurement` column.
2023-05-05 16:48:27 +10:00
f026d546b0
chore: Refactor `select_statement_to_plan`
...
This is an improvement over the previous version, and prepares the
planner for implementing subqueries and passing schema to the
`project_select` function.
2023-05-05 13:35:32 +10:00
e7d6819602
chore: Correct docs
2023-05-05 12:25:58 +10:00
a218d97dcf
chore: Use `Select` rather than `SelectStatement` in planner
2023-05-05 12:17:15 +10:00
03b4195976
chore: Rename TableReference to DataSource to avoid conflicts with DF
2023-05-05 11:03:56 +10:00
8deb1832e7
chore: Expose TypeEvaluator for more efficient type evaluation
2023-05-05 10:54:45 +10:00
7b19bf014b
chore: Reorganise `rewrite_statement` unit tests
2023-05-05 10:12:17 +10:00
8b60a95a77
chore: Unnecessary use of `borrow`
2023-05-05 10:12:02 +10:00
f8b1d37d5a
fix: Consistent alias names with InfluxQL OG
...
Fixes #7750
2023-05-05 10:11:37 +10:00
7e47d48d01
chore: SLIMIT and SOFFSET are not implemented
2023-05-05 07:48:20 +10:00
85f461fbda
Merge branch 'main' into sgc/issue/7739_select_ir_01
2023-05-05 06:29:21 +10:00
2cfb30d5df
refactor: remove non-struct selector functions ( #7743 )
...
Clean up for #7533 .
2023-05-04 10:25:52 +00:00
b47e0efc85
feat: Step 1 on N for an intermediate representation of `SELECT`
...
The end goal is that each `Select` node will contain a schema to be
referenced directly by the InfluxQL planner. Additionally, further
refinement of the field data types used by the `Select` node
are expected, to remove ambiguity from the planner.
2023-05-04 18:27:32 +10:00
43baecbb1a
chore: handle aliased time column in sort expression
2023-05-03 12:04:24 +10:00
19ea80390a
chore: ignore time column when processing SELECT
2023-05-03 11:04:54 +10:00
91f5464375
chore: add additional tests
2023-05-03 09:23:15 +10:00
24378bd460
chore: `time` is always present and at first position of projection
2023-05-03 07:47:14 +10:00
Stuart Carnie
Andrew Lamb
Christopher M. Wolff
Dom Dwyer
Marco Neumann