c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
d2c2143277
fix: wrong results in read_filter with "complex" OR predicate and IS_NULL ( #2977 )
...
* fix: Allow evaluating general purpose predicates in gRPC by rewriting missing columns to null
* fix: update commands and add some additional tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 14:30:28 +00:00
6760212e86
Merge branch 'main' into ntran/group
2021-10-26 21:25:57 +00:00
5c2a5e1030
fix: fmt after appying review comments
2021-10-26 17:17:35 -04:00
6950e7a9b6
chore: Apply suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-26 17:08:07 -04:00
8c43b24745
chore: cleanup
2021-10-26 16:46:50 -04:00
46cb4df14d
fix: now no rows will be return for read_group if the table has no data or all its data is soft deleted or the table does not exist
2021-10-26 16:40:36 -04:00
39b3b8d33f
fix: Logical merge conflicts
2021-10-25 13:47:10 -04:00
7cd56cbc56
Merge remote-tracking branch 'origin/main' into er/fix/flux/2691
2021-10-25 13:41:08 -04:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate ( #2947 )
...
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate
* fix: make apply_predicate_to_metadata precise
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
28962038c1
fix: properly handle null values in table_name plan ( #2946 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:35:02 +00:00
eb71102d82
Merge branch 'main' into er/fix/flux/2691
2021-10-22 20:21:22 +01:00
e8ddc4e0db
feat: Add custom `NonNullChecker` operator that checks for non-null values in columns ( #2944 )
...
* feat: Add custom `NonNullChecker` operator that checks for non-null values in columns
* fix: remove println
* fix: typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-22 18:53:20 +00:00
fe155e15fb
fix: fix flaky test ( #2939 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-22 17:08:27 +00:00
4464114330
Merge branch 'main' into er/fix/flux/2691
2021-10-22 09:43:35 +01:00
654d33684a
fix: Move code shared between if/else blocks out
2021-10-21 20:47:35 -04:00
8898631034
refactor: update query/src/frontend/influxrpc.rs
2021-10-21 22:38:20 +01:00
fc7ce3535d
refactor: fix _value filtering for non-selector aggregates
2021-10-21 22:28:41 +01:00
f8489fc774
refactor: ensure _value filtering works read_filter
2021-10-21 22:28:41 +01:00
8a70bd898b
feat: add _value support to selector aggregates
2021-10-21 22:28:38 +01:00
0cb9751b0a
refactor: extract _value expr from predicate
2021-10-21 22:26:32 +01:00
f7ca80e29f
test: ensure query cancellation (somewhat) works ( #2931 )
...
* feat: enable reconfiguration of in-use throttled store
This is handy for tests for which a part should run "normal" and another
one should be throttled/blocked.
* feat: keep track of the number of tasks within a `DedicatedExecutor`
* test: ensure query cancellation (somewhat) works
We cannot really test that query cancellation finishes all subtasks
because _tokio_ doesn't provide sufficient stats / inspection, at least
as long we don't want to rely heavily on _tokio_ tracing. So let's at
least check that tasks from the dedicated executors are pruned properly.
For all other regressions we need to add unit tests to the affected
components. See for example:
- https://github.com/apache/arrow-datafusion/issues/1103
- https://github.com/apache/arrow-datafusion/pull/1105
- https://github.com/apache/arrow-datafusion/pull/1112
- https://github.com/apache/arrow-datafusion/pull/1121
Closes #2027 .
2021-10-21 19:10:58 +00:00
aecbdc0468
chore: merge main to branch
2021-10-20 17:09:02 -04:00
995ffd43f6
chore: run fmt
2021-10-20 16:56:21 -04:00
d9b034cfaa
chore: cleanup
2021-10-20 16:08:14 -04:00
97d5760347
feat: support delelete for table_names
2021-10-20 15:58:25 -04:00
ee2ca8fc32
fix(read_group): Support grouping on `_start` and `_stop` ( #2918 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 15:16:47 +00:00
438cd1767a
refactor: update query/src/frontend/influxrpc.rs
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-20 15:00:07 +01:00
a96284e120
test: use rewriter properly
2021-10-20 14:59:27 +01:00
a0ba75576e
fix: add support for rewriting field column
2021-10-20 14:59:27 +01:00
7d86d1144e
test: add coverage for field value rewriter
2021-10-20 14:59:27 +01:00
3fe8f64aa4
feat: add support for filtering on _value
2021-10-20 14:59:27 +01:00
f85a9f15b1
refactor: plumb schema to TableNormalizedPredicate
2021-10-20 14:59:27 +01:00
9974a5364c
chore(security): Replace prettytable with comfy-table ( #2905 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 10:44:36 +00:00
8c058f791b
chore: cleanup
2021-10-19 15:43:06 -04:00
b55ca06fe3
fix: read_window_aggregate overflow ( #2908 )
2021-10-19 19:20:39 +00:00
e664799593
feat: support delete for tag_values
2021-10-19 14:18:01 -04:00
254a842eed
Merge branch 'main' into ntran/table_names
2021-10-19 17:53:25 +00:00
d5cffb5f54
fix(read_window_aggregates): return aggregates as integer rather than unsigned ( #2906 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:47:15 +00:00
c755678aa1
refactor: address review comments
2021-10-19 13:41:41 -04:00
cabb007956
chore: Merge branch 'main' into ntran/table_names
2021-10-19 13:22:28 -04:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
ea85d6478e
chore: remove code changes for table_names abecause normal plan needs to be implemented first
2021-10-19 10:36:10 -04:00
afa6e50c9c
feat: make tag_keys work with delete
2021-10-18 15:36:19 -04:00
f5a84122e3
feat: Support grouping by _field and _measurement ( #2874 )
...
* feat: Support grouping by _field and _measurement
* fix: clippy
* fix: doclink
* refactor: rename SeriesOrGroup --> Either
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 15:32:24 +00:00
51276119df
docs: Add better SeriesSet explanation ( #2857 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 21:53:51 +00:00
beaf77cecf
refactor: move Series translation logic into query crate, update gRPC tests ( #2852 )
...
* refactor: move Series translation logic into query crate
* refactor: update grpc_tests to use new display
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 11:06:40 +00:00
08e6a01e69
refactor: Move series set converter into its own module ( #2847 )
...
* refactor: Move series set converter into its own module
* fix: add file
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 14:16:32 +00:00
8342e138b0
refactor: PR feedback
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-14 09:23:29 +01:00
fcf31eebb2
fix: use correct timestamps for selectors
2021-10-14 09:23:29 +01:00
e0929f20ae
refactor: Pull out read_group order creation ( #2832 )
2021-10-13 19:05:15 +00:00
d2cf6fa9f7
docs: improve docstring for InfluxRPCPlanner::read_group ( #2827 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:48:32 +00:00
04468f3208
chore: more cleanup
2021-10-12 17:27:04 -04:00
459dd46ae9
refactor: move delete tests to .sql
2021-10-12 15:49:23 -04:00
0b4ae95ca4
refactor: exhaust scenarios for one-chunk test
2021-10-11 17:47:41 -04:00
6b42f1cdbb
Merge branch 'main' into ntran/count_star
2021-10-11 15:01:00 +00:00
1718b55283
chore: Merge branch 'ntran/count_star' of https://github.com/influxdata/influxdb_iox into ntran/count_star
2021-10-11 10:47:38 -04:00
fbf5539336
chore: merge main to branch
2021-10-11 10:47:10 -04:00
24ae269b3a
refactor: cancel executor jobs on drop
...
Our executor is not meant as a fire-and-forget system. Instead the
submitter should always poll the result. Dropping the receiver side (aka
the job handle) should cancel the job.
2021-10-11 16:13:04 +02:00
d13e61c201
fix: Apply suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-11 09:33:23 -04:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
d0a17ca79d
refactor: address Edd's review comments
2021-10-08 16:47:37 -04:00
f7475322a6
chore: merge main to branch, resolve conflicts, and discover an inconsitent bug
2021-10-08 15:50:46 -04:00
bea310db76
chore: remove comments
2021-10-08 14:58:06 -04:00
2556639bb5
chore: more cleanup
2021-10-08 14:57:22 -04:00
f2cdb9531f
chore: cleanup
2021-10-08 14:52:15 -04:00
adbcd85c26
fix: fully fix 2745
2021-10-08 14:37:34 -04:00
2072b4066e
feat: Implement support for `_measurement` predicate in gRPC plans ( #2772 )
...
* feat: Implement filtering for _measurement in general purpose gRPC plans
* docs: fixup docstrings
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-08 17:25:33 +00:00
22d6f11bea
fix: add cols of delete predicates into the schema of scanning columns
2021-10-07 17:37:34 -04:00
e590ac4da2
fix: remove outdated "supported predicate" check in gRPC planner ( #2763 )
2021-10-07 20:34:05 +00:00
c7727f1b5b
chore: Update datafusion + other deps ( #2760 )
...
* chore: Update datafusion and other deps
* fix: fmt
* fix: cleanup workaround
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-07 16:48:11 +00:00
d4bc33b921
refactor: use RWLock instead of TaskTracker for query exec shutdown ( #2761 )
2021-10-07 14:18:40 +00:00
0b4fd01d04
fix: Cast count aggregates correctly ( #2756 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-07 10:49:15 +00:00
63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
de148337e8
fix: half way fix the bug to inlcude schema of column in delete predicate into the schema of IOx scan to avoid missing reading columns
2021-10-06 17:43:48 -04:00
efa2316626
fix: do not sort the output of read_group with no group keys ( #2755 )
2021-10-06 18:59:58 +00:00
65a02f7085
refactor: Apply suggestions from code review
...
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-06 10:04:28 -04:00
ea26a77217
Merge branch 'main' into ntran/no_use_stats
2021-10-06 09:30:18 +01:00
aa64daca86
feat: dDisable using statistics to query data if there are soft deleted rows
2021-10-05 17:52:32 -04:00
785a62c114
fix: include all group tags, not just group_keys in GroupFrame response ( #2741 )
...
* fix: include all group tags, not just group_keys in GroupFrame response
* docs: fix test comments, add doc strings for group_description_to_frames
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-05 19:08:04 +00:00
97881079e8
refactor: make `ChunkOrder` non-zero
...
This will make it easier to handle missing values.
Helps with #2633 .
2021-10-04 17:49:12 +02:00
5a5a929b9e
refactor: introduce `DeletePredicate`
...
`DeletePredicate` is a simpler version of `Predicate` that is based on
IOx `DeleteExpr` instead of the full-blown DataFusion `Expr`. This will
allow us to do a couple of things (in follow-up changes):
- Order and de-duplicate delete predicates
- Normalize predicates
- Infallible serialization
- Smaller memory footprint
Note that this change only affects delete expressions. Query expressions
that are supported via the API are not changed. The query subsystem also
still uses the full-featured expressions/predicates (delete
expressions/predicates are converted to the more powerful DataFusion
version on-the-fly).
2021-10-04 16:36:20 +02:00
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
134fb96b26
feat: add UInt64 support for gRPC query results ( #2701 )
2021-10-01 17:18:32 +00:00
2db56a0332
chore: Make query logging a bit less verbose ( #2655 )
...
* chore: Make query logging a bit less verbose
* fix: remove unused use
2021-09-28 20:58:37 +00:00
a55a21c644
chore: Update datafusion ( #2635 )
...
* chore: Update datafusion and sqlparser
* fix: remove STACK_SIZE workaround
* chore: update datafusion_util
* chore: update predicate
* chore: update query_tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-27 14:13:19 +00:00
d38648952c
chore: Update datafusion ( #2602 )
...
* chore: Update datafusion + other deps
* refactor: update query crate for new async interfaces
* refactor: update server crate for new async interface
* refactor: update query_tests crate for new async interfaces
* refactor: update ioxd and server to use new async interface
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-22 10:33:25 +00:00
0fc2567161
refactor: address review comment
2021-09-21 16:50:29 -04:00
a06afb3932
feat: optimize scan for chunks without delete preidcates and without the need to sort output
2021-09-21 15:56:19 -04:00
61ed67a5d9
refactor: cleanup
2021-09-21 12:28:18 -04:00
93551bdd1e
fix: all chunks now are applied delete predicates during scan
2021-09-21 12:17:59 -04:00
77d84ca5ab
Merge branch 'main' into crepererum/chunk_id
2021-09-20 13:39:05 +00:00
cef5aeee52
refactor: introduce `ChunkId` type
2021-09-20 13:10:41 +02:00
364d245eae
feat: apply negated delete predicates during scan
2021-09-17 16:20:42 -04:00
ec943081c7
refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates
...
The motivations are:
1. The API uses a SINGLE predicate and adds that to many chunks. With
`Arc<Vec<...>>` you gain nothing, with `Vec<Arc<...>>` the predicate
is only stored once (in many vectors)
2. While we currently add predicates blindly to all chunks, we can be way
smarter in the future and prune out tables, partitions or even single
chunks (based on statistics). With that, it will be rare that many
chunks share the exact same set of predicates.
3. It would be nice if we could de-duplicate predicates when writing them
to the preserved catalog without needing to repeat the pruning
discussed in point 2. This is way easier to implement whan chunks
exists in `Arc`s.
4. As a side-note: the `Arc<Vec<...>>` wasn't really cloned around but
instead was created many time. So the new version should be more
memory efficient out of the box.
2021-09-16 17:16:09 +02:00
ce224bd37f
fix: Capture query execution traces for storage gRPC queries as well ( #2553 )
...
* fix: Capture query execution traces for storage gRPC queries as well
* refactor: remove debugging droppings
* refactor: do not Box::pin within TracedStream
* refactor: Use Futures::TryStreamExt rather than custom collect function
* fix: remove wild println
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 14:45:20 +00:00
33cd1cffad
Merge branch 'main' into ntran/delete_read
2021-09-16 13:22:50 +00:00
a478138756
refactor: Add SpanContext:new() to make a new span ( #2551 )
...
* refactor: Add SpanContext::new() and remove make_span
* fix: generate random trace_id and span_ids
* docs: Update trace/src/ctx.rs
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 11:02:28 +00:00
dependabot[bot]
Andrew Lamb
kodiakhq[bot]
Nga Tran
Andrew Lamb
Marco Neumann
Edd Robinson
Carol (Nichols || Goulding)
Raphael Taylor-Davies