ace3c11f12
chore: Update datafusion ( #6004 )
...
* chore: Update datafusion
* chore: change path
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-31 16:16:28 +00:00
9b48437711
refactor: make influx column type mandatory ( #5978 )
...
We basically assume everywhere that a column falls into one of the three
known categories (time, tag, field), so lets encode this in our type
system instead of defining "unknown" as "undefined behavior, may or may
not crash".
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-26 11:20:29 +00:00
3e4db81bc6
refactor: make `SchemaBuilder::field` fallible
...
It would be nice if the IOx data type would not be optional and this is
a prep clean-up to achieve that.
2022-10-24 18:12:42 +02:00
d706f8221d
chore: Update datafusion and arrow / parquet / arrow-flight 25.0.0 ( #5900 )
...
* chore: Update datafusion and `arrow` / `parquet` / `arrow-flight` 25.0.0
* chore: Update for structure changes
* chore: Update for new projection pushdown
* chore: Run cargo hakari tasks
* fix: fmt
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 20:58:47 +00:00
ece0ce440d
chore: update datafusion ( #5869 )
...
* chore: update datafusion
* chore: Update split conjunction api
* chore: remove code that was upstreamed
* chore: simplification is better upstream
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-17 13:22:16 +00:00
9134ccd6c3
chore: Update datafusion again ( #5855 )
...
* chore: Update datafusion
* chore: Updates for changes in datafusion
* chore: more updates
* fix: update doc example
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-13 19:18:57 +00:00
227ecd4da1
refactor: Move http delete parsing code into router along side other http code ( #5841 )
2022-10-13 11:08:42 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
8013781ac2
feat: rewrite missing column references to NULL ( #5818 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-07 18:05:54 +00:00
cd4087e00d
style: add no todo!() or dbg!() lints
...
Some crates had theme, some not - lets be consistent and have the
compiler spot dbg!() and todo!() macro calls - they should never be in
prod code!
2022-09-29 13:10:07 +02:00
159250e776
refactor: concurrent table planning in InfluxRPC ( #5647 )
...
* refactor: concurrent table planning in InfluxRPC
Some InfluxRPC can scan multiple tables. Prior to this PR we were always
scanning the tables in sequence, adding up potential latencies (catalog,
ingester, object store). There is no reason we need to do this,
"ordinary" SQL queries would not serialize this way either.
So let's scan tables concurrently. This add concurrency to:
- read filter
- read group
- read window aggregate
There are other query types that could benefit from a similar treatment.
They will be changed in a follow-up.
* docs: improve
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* test: explain `Send` assertion
* refactor: change `CONCURRENT_TABLE_JOBS` to 10
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-09-15 13:55:22 +00:00
f86d3e31da
chore: Update datafusion + object_store ( #5619 )
...
* chore: Update datafusion pin
* chore: update object_store to 0.5.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-13 12:34:54 +00:00
1fd31ee3bf
chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0 ( #5591 )
...
* chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0
* fix: enable dynamic comparison flag
* chore: derive Eq for clippy
* chore: update explain plans
* chore: Update sizes for ReadBuffer encoding
* chore: update more tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-12 17:45:03 +00:00
52ae60bf2e
chore: fix comment typo ( #5551 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2022-09-07 08:49:29 +00:00
bd4b708055
chore: Update datafusion pin + other dependencies ( #5418 )
...
* chore: Update datafusion pin
* chore: Update other depdecies
* fix: Update for changes in API
2022-08-17 10:42:37 +00:00
b982bdaf2f
fix: Derive Eq when we derive PartialEq and members can derive Eq
...
Allow this in generated code that we don't control, though.
Recommended by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq
2022-08-11 15:04:06 -04:00
7066c4e679
fix: make it clear rpc_predicates are only ever specialized when a schema is known ( #5315 )
...
* fix: make it clear rpc_predicates are only ever specialized when a schema is known
* fix: handle case of no schema
* fix: Update predicate/src/rpc_predicate.rs
2022-08-06 10:56:53 +00:00
e0ea335b70
fix: Support RegExMatch and RegExNotMatch predicates on `_field` ( #5301 )
...
* test: add tests for regex_match_on_field
* feat: more general `_field` predicate handling
* fix: remove old comment
* fix: update tests
* fix: improve test a little more
* fix: fmt
* fix: Update predicate/src/rpc_predicate/field_rewrite.rs
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: Handle predicates that can not be evaluated
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 19:42:16 +00:00
3989ac1386
refactor: remove `split_members` and use `split_conjunction` from upstream DataFusion ( #5308 )
...
* refactor: remove split_members and use split_conjunction from datafusion
* fix: clippy
2022-08-04 13:58:59 +00:00
76171c5cea
refactor: simplify creating AND expressions ( #5298 )
...
* refactor: simplify creating AND expressions
* fix: fmt + clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 11:25:43 +00:00
d0f88c664c
docs: improve some docstrings in `iox_query` ( #5291 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:20:12 +00:00
3fbe860bb9
fix: interpret [MIN_NANO_TIME, MAX_NANO_TIME) range as all time for optimization ( #5231 )
...
InfluxQL queries can send (technically incorrect) ranges like this, meaning all time
but excluding the max nanosecond time.
Since this is an important case, we should handle it specially and use the optimized
'all time' handling for meta queries even though this is technically wrong in that
it does not filter out column names / measurement names at MAX_NANO_TIME exactly.
Closes: https://github.com/influxdata/conductor/issues/1072
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 12:24:26 +00:00
96c3a05481
feat: add debug log to `with_clear_timestamp_if_max_range` ( #5199 )
2022-07-25 11:43:54 +00:00
449b4cf1d6
chore: Update datafusion pin ( #5148 )
...
* chore: Update datafusion pin
* fix: Update for changes upstream
2022-07-19 13:13:37 +00:00
c46e1c6347
chore: Update datafusion + arrow/parquet/arrow-flight to `17.0.0` ( #5021 )
...
* fix: correct nullability declaration of system tables
* chore: Update datafusion and arrow/parquet/arrow-flight
* chore: Run cargo hakari tasks
* fix: Update tests
* fix: Update tests
* fix: predicate pruning
* fix: add some tests
* fix: query_functions
* fix: fix read_buffer test
* fix: fix clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-07 19:22:15 +00:00
e193913ed3
fix: optimize field columns for all-time predicates ( #5046 )
...
* fix: optimize field columns for all-time predicates
Also fix timestamp range to allow selecting points at MAX_NANO_TIME
* fix: clamp end to MIN_NANO_TIME for safety
* refactor: add contains_all method to TimestampRange
2022-07-06 12:01:28 +00:00
03f456d8fd
fix: optimize tag_keys to go only to schema when predicate is empty ( #4985 )
...
* docs: fix comment
* test: add test for delete behaviour
* fix: tag_keys optimization for empty predicate
Also need to eliminate 'true' predicates from simplified predicate so
is_empty works correctly.
* refactor: use lit instead of spelling out literal true
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-05 12:45:25 +00:00
6f445ccd94
feat: Prune chunks using table summary (stats) ( #5017 )
...
* feat: easy tests of table summary against predicate
Helps with #4976 .
Alternative to #4995 .
* refactor: address review comments
* refactor: address review comments
* refactor: address review comments
2022-07-04 09:01:34 +00:00
1993448abf
refactor: remove `Predicat::partition_key` ( #5016 )
...
There is no way a user can filter for partition keys (neither via
InfluxRPC nor via SQL) and the query engine doesn't use this field at
all. So let's remove it.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-01 17:17:29 +00:00
2ec7764fdd
refactor: rename builder like predicate methods to be `with_` ( #4808 )
...
* refactor: rename builder like predicate methods to be `with_`
* fix: merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-09 11:26:03 +00:00
5e4fcfaa4d
refactor: reduce mut usage in Predicate ( #4807 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-09 10:46:01 +00:00
46de8d6cb3
refactor: remove redundant code in predicate ( #4805 )
2022-06-08 15:03:26 +00:00
afc1c12062
refactor: consolidate `PredicateBuilder` into `Predicate` ( #4799 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-08 12:21:24 +00:00
8e96a2721d
chore: Update datafusion (again) ( #4788 )
...
* chore: Update datafusion
* chore: Update imports
* refactor: update API usage
* refactor: clean up some uses of binary_expr
* fix: remove unused export
* fix: update explain output
* chore: update more explain tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-07 08:17:56 +00:00
a37c553545
refactor: Split up rpc_predicate module a bit ( #4763 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-01 19:56:11 +00:00
257aaa7e7b
fix: Support `_field != <name>` predicates ( #4721 )
...
* fix: Support `_field != <name>` predicates
* fix: update test
* fix: add negative test
* fix: improve comments
* refactor: make `add_include` and `add_exclude` infallible
* chore: add type annotations
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-01 16:04:53 +00:00
95e6a8ed46
chore: Update datafusion (again) ( #4679 )
...
* chore: Update datafusion deps
* fix: fix for changes in ScalarValue
* fix: fix for using TableSource rather than TableProvider
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-24 15:54:39 +00:00
3a33e806c7
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `14.0.0` ( #4619 )
...
* chore: Update datafusion deps
* chore: update arrow/parquet/arrow flight deps
* chore: Run cargo hakari tasks
* chore: Update location of utils
* chore: Update some more APIs
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-17 14:13:03 +00:00
068096e7e1
fix: Rename data_types2 to data_types
2022-05-06 14:45:39 -04:00
f39b093357
fix: Only use data_types2 in predicate
2022-05-06 14:45:37 -04:00
eb31b347b0
refactor: Move tombstones_to_delete_predicates to the predicate crate
2022-05-06 14:45:37 -04:00
e13d3433ae
feat: Use datafusion serialization code rather than our own copy of it ( #4421 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 13:03:34 +00:00
115f007317
refactor: Use DataFusion `Expr` instead of our own custom wrapper for `ValueExpr` ( #4440 )
...
* refactor: Use DataFusion `Expr` instead of custom wrapper for BinaryExprs
* fix: apply code review suggestions
* fix: more code review suggestions
2022-04-27 19:20:15 +00:00
f0ceca985b
feat: Implement IOx Function Registry for regex_match/regex_not_match ( #4431 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 10:33:19 +00:00
9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) ( #4428 )
...
* refactor: Move all UDF implementations to query_function crate
* refactor: Move regex udf to query_functions
* refactor: Move functions out of query
* fix: lints, imports
* chore: Run cargo hakari tasks
* fix: clipy + benches
* fix: reduce borrowing and fix clippy
* fix: moar clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 17:30:27 +00:00
e3d83fe757
chore: update datafusion ( #4342 )
...
* chore: update datafusion
* fix: Update imports for change in datafusion organization
2022-04-19 13:38:12 +00:00
e97cec3f4d
chore: use Nga's DF branch with debug information for compactor ( #4314 )
2022-04-13 22:20:16 +00:00
22b24bdab3
chore: Update datafusion again ( #4148 )
...
* chore: update datafusoon
* refactor: Update for DataFusion API changes
* chore: TEMP TEMP change df to local copy
* chore: Update to datafusion again
* fix: Update Cargo.lock
* fix: logical conflict
2022-03-30 16:51:48 +00:00
b83b000590
chore: Update datafusion ( #4071 )
...
* chore: update to datafusion 5936edc2a94d5fb20702a41eab2b80695961b9dc
* chore: Update apis to match datafusion changes
2022-03-22 13:17:41 +00:00
c9908b260c
refactor: dyn-dispatch database in query subsystem ( #4083 )
...
* refactor: dyn-dispatch database in query subsystem
This is similar to #4080 but concerns the database itself.
For #3934 .
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-22 09:15:52 +00:00
Andrew Lamb
Marco Neumann
Dom Dwyer
Andrew Lamb
YIXIAO SHI
Carol (Nichols || Goulding)
Sam Arnold
Sam Arnold
Nga Tran