8c058f791b
chore: cleanup
2021-10-19 15:43:06 -04:00
b55ca06fe3
fix: read_window_aggregate overflow ( #2908 )
2021-10-19 19:20:39 +00:00
e664799593
feat: support delete for tag_values
2021-10-19 14:18:01 -04:00
254a842eed
Merge branch 'main' into ntran/table_names
2021-10-19 17:53:25 +00:00
d5cffb5f54
fix(read_window_aggregates): return aggregates as integer rather than unsigned ( #2906 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:47:15 +00:00
c755678aa1
refactor: address review comments
2021-10-19 13:41:41 -04:00
cabb007956
chore: Merge branch 'main' into ntran/table_names
2021-10-19 13:22:28 -04:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
ea85d6478e
chore: remove code changes for table_names abecause normal plan needs to be implemented first
2021-10-19 10:36:10 -04:00
afa6e50c9c
feat: make tag_keys work with delete
2021-10-18 15:36:19 -04:00
f5a84122e3
feat: Support grouping by _field and _measurement ( #2874 )
...
* feat: Support grouping by _field and _measurement
* fix: clippy
* fix: doclink
* refactor: rename SeriesOrGroup --> Either
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 15:32:24 +00:00
51276119df
docs: Add better SeriesSet explanation ( #2857 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 21:53:51 +00:00
beaf77cecf
refactor: move Series translation logic into query crate, update gRPC tests ( #2852 )
...
* refactor: move Series translation logic into query crate
* refactor: update grpc_tests to use new display
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 11:06:40 +00:00
08e6a01e69
refactor: Move series set converter into its own module ( #2847 )
...
* refactor: Move series set converter into its own module
* fix: add file
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 14:16:32 +00:00
8342e138b0
refactor: PR feedback
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-14 09:23:29 +01:00
fcf31eebb2
fix: use correct timestamps for selectors
2021-10-14 09:23:29 +01:00
e0929f20ae
refactor: Pull out read_group order creation ( #2832 )
2021-10-13 19:05:15 +00:00
d2cf6fa9f7
docs: improve docstring for InfluxRPCPlanner::read_group ( #2827 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:48:32 +00:00
04468f3208
chore: more cleanup
2021-10-12 17:27:04 -04:00
459dd46ae9
refactor: move delete tests to .sql
2021-10-12 15:49:23 -04:00
0b4ae95ca4
refactor: exhaust scenarios for one-chunk test
2021-10-11 17:47:41 -04:00
6b42f1cdbb
Merge branch 'main' into ntran/count_star
2021-10-11 15:01:00 +00:00
1718b55283
chore: Merge branch 'ntran/count_star' of https://github.com/influxdata/influxdb_iox into ntran/count_star
2021-10-11 10:47:38 -04:00
fbf5539336
chore: merge main to branch
2021-10-11 10:47:10 -04:00
24ae269b3a
refactor: cancel executor jobs on drop
...
Our executor is not meant as a fire-and-forget system. Instead the
submitter should always poll the result. Dropping the receiver side (aka
the job handle) should cancel the job.
2021-10-11 16:13:04 +02:00
d13e61c201
fix: Apply suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-11 09:33:23 -04:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
d0a17ca79d
refactor: address Edd's review comments
2021-10-08 16:47:37 -04:00
f7475322a6
chore: merge main to branch, resolve conflicts, and discover an inconsitent bug
2021-10-08 15:50:46 -04:00
bea310db76
chore: remove comments
2021-10-08 14:58:06 -04:00
2556639bb5
chore: more cleanup
2021-10-08 14:57:22 -04:00
f2cdb9531f
chore: cleanup
2021-10-08 14:52:15 -04:00
adbcd85c26
fix: fully fix 2745
2021-10-08 14:37:34 -04:00
2072b4066e
feat: Implement support for `_measurement` predicate in gRPC plans ( #2772 )
...
* feat: Implement filtering for _measurement in general purpose gRPC plans
* docs: fixup docstrings
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-08 17:25:33 +00:00
22d6f11bea
fix: add cols of delete predicates into the schema of scanning columns
2021-10-07 17:37:34 -04:00
e590ac4da2
fix: remove outdated "supported predicate" check in gRPC planner ( #2763 )
2021-10-07 20:34:05 +00:00
c7727f1b5b
chore: Update datafusion + other deps ( #2760 )
...
* chore: Update datafusion and other deps
* fix: fmt
* fix: cleanup workaround
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-07 16:48:11 +00:00
d4bc33b921
refactor: use RWLock instead of TaskTracker for query exec shutdown ( #2761 )
2021-10-07 14:18:40 +00:00
0b4fd01d04
fix: Cast count aggregates correctly ( #2756 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-07 10:49:15 +00:00
63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
de148337e8
fix: half way fix the bug to inlcude schema of column in delete predicate into the schema of IOx scan to avoid missing reading columns
2021-10-06 17:43:48 -04:00
efa2316626
fix: do not sort the output of read_group with no group keys ( #2755 )
2021-10-06 18:59:58 +00:00
65a02f7085
refactor: Apply suggestions from code review
...
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-06 10:04:28 -04:00
ea26a77217
Merge branch 'main' into ntran/no_use_stats
2021-10-06 09:30:18 +01:00
aa64daca86
feat: dDisable using statistics to query data if there are soft deleted rows
2021-10-05 17:52:32 -04:00
785a62c114
fix: include all group tags, not just group_keys in GroupFrame response ( #2741 )
...
* fix: include all group tags, not just group_keys in GroupFrame response
* docs: fix test comments, add doc strings for group_description_to_frames
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-05 19:08:04 +00:00
97881079e8
refactor: make `ChunkOrder` non-zero
...
This will make it easier to handle missing values.
Helps with #2633 .
2021-10-04 17:49:12 +02:00
5a5a929b9e
refactor: introduce `DeletePredicate`
...
`DeletePredicate` is a simpler version of `Predicate` that is based on
IOx `DeleteExpr` instead of the full-blown DataFusion `Expr`. This will
allow us to do a couple of things (in follow-up changes):
- Order and de-duplicate delete predicates
- Normalize predicates
- Infallible serialization
- Smaller memory footprint
Note that this change only affects delete expressions. Query expressions
that are supported via the API are not changed. The query subsystem also
still uses the full-featured expressions/predicates (delete
expressions/predicates are converted to the more powerful DataFusion
version on-the-fly).
2021-10-04 16:36:20 +02:00
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
134fb96b26
feat: add UInt64 support for gRPC query results ( #2701 )
2021-10-01 17:18:32 +00:00
2db56a0332
chore: Make query logging a bit less verbose ( #2655 )
...
* chore: Make query logging a bit less verbose
* fix: remove unused use
2021-09-28 20:58:37 +00:00
a55a21c644
chore: Update datafusion ( #2635 )
...
* chore: Update datafusion and sqlparser
* fix: remove STACK_SIZE workaround
* chore: update datafusion_util
* chore: update predicate
* chore: update query_tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-27 14:13:19 +00:00
d38648952c
chore: Update datafusion ( #2602 )
...
* chore: Update datafusion + other deps
* refactor: update query crate for new async interfaces
* refactor: update server crate for new async interface
* refactor: update query_tests crate for new async interfaces
* refactor: update ioxd and server to use new async interface
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-22 10:33:25 +00:00
0fc2567161
refactor: address review comment
2021-09-21 16:50:29 -04:00
a06afb3932
feat: optimize scan for chunks without delete preidcates and without the need to sort output
2021-09-21 15:56:19 -04:00
61ed67a5d9
refactor: cleanup
2021-09-21 12:28:18 -04:00
93551bdd1e
fix: all chunks now are applied delete predicates during scan
2021-09-21 12:17:59 -04:00
77d84ca5ab
Merge branch 'main' into crepererum/chunk_id
2021-09-20 13:39:05 +00:00
cef5aeee52
refactor: introduce `ChunkId` type
2021-09-20 13:10:41 +02:00
364d245eae
feat: apply negated delete predicates during scan
2021-09-17 16:20:42 -04:00
ec943081c7
refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates
...
The motivations are:
1. The API uses a SINGLE predicate and adds that to many chunks. With
`Arc<Vec<...>>` you gain nothing, with `Vec<Arc<...>>` the predicate
is only stored once (in many vectors)
2. While we currently add predicates blindly to all chunks, we can be way
smarter in the future and prune out tables, partitions or even single
chunks (based on statistics). With that, it will be rare that many
chunks share the exact same set of predicates.
3. It would be nice if we could de-duplicate predicates when writing them
to the preserved catalog without needing to repeat the pruning
discussed in point 2. This is way easier to implement whan chunks
exists in `Arc`s.
4. As a side-note: the `Arc<Vec<...>>` wasn't really cloned around but
instead was created many time. So the new version should be more
memory efficient out of the box.
2021-09-16 17:16:09 +02:00
ce224bd37f
fix: Capture query execution traces for storage gRPC queries as well ( #2553 )
...
* fix: Capture query execution traces for storage gRPC queries as well
* refactor: remove debugging droppings
* refactor: do not Box::pin within TracedStream
* refactor: Use Futures::TryStreamExt rather than custom collect function
* fix: remove wild println
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 14:45:20 +00:00
33cd1cffad
Merge branch 'main' into ntran/delete_read
2021-09-16 13:22:50 +00:00
a478138756
refactor: Add SpanContext:new() to make a new span ( #2551 )
...
* refactor: Add SpanContext::new() and remove make_span
* fix: generate random trace_id and span_ids
* docs: Update trace/src/ctx.rs
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 11:02:28 +00:00
7175488133
chore: add some comments
2021-09-15 14:45:04 -04:00
3486cc8b38
fix: should not send an empty delete predicate predicate which means delete everything (no time range)
2021-09-15 14:14:26 -04:00
74d3c2e6d2
feat: Translate DataFusion execution metrics to IOx Spans ( #2529 )
...
* feat: Translate DataFusion execution metrics to IOx Spans
* fix: add end to end test to ensure plumbing is hookedup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 18:14:23 +00:00
de732b4273
Merge branch 'main' into crepererum/parquet_file_wo_query
2021-09-15 07:15:19 +00:00
63cc7b3fb0
test: more tests to discover what still need to be done
2021-09-14 17:57:30 -04:00
f4f140d3b7
chore: merge main to branch
2021-09-14 13:25:32 -04:00
509c07330d
refactor: decouple `parquet_file` from `query`
2021-09-14 18:26:16 +02:00
d60aa5940b
Merge branch 'main' into crepererum/chunk_order_type
2021-09-14 16:25:17 +00:00
bfaba78dc3
refactor: move `predicate` into its own crate
...
Two reasons:
1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
small follow-up PR).
2. `predicate` will have more and more features (like serialization)
which justifies a new home
2021-09-14 17:13:02 +02:00
becef1c75f
refactor: introduce `ChunkOrder` type
2021-09-14 17:10:23 +02:00
9ea61cd434
Merge branch 'main' into crepererum/issue1963
2021-09-14 11:38:59 +00:00
4795bd5c9c
refactor: stricter delete predicate TS parsing
...
As a a nice side effect, the parser no longer depends on the line
protocol parser.
2021-09-14 13:14:19 +02:00
1b788732da
fix: order chunks correctly during query processing
...
The query processing was implicitly relying on the order provided by the
catalog. This had two issues:
- this ordering was not defined in the API contract (neither via docs
nor via typing)
- the order was based on chunk IDs which is not adequate in some cases
(e.g. when chunks are created while a persistence operations is in
progress)
Now we explicitly sort chunks by `(order, ID)`.
Fixes #1963 .
2021-09-14 13:00:55 +02:00
042a78e5a7
feat: apply delete predicate during query to emilimate deleted data
2021-09-13 18:02:55 -04:00
5eef76c868
chore: Update dependencies (including datafusion) ( #2521 )
...
* chore: Update datafusion deps to pre-release
* refactor: Update IOx to use new datafusion Statistics
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-13 21:30:44 +00:00
20143e4f4e
feat: migrate chunk pruning metrics ( #2516 )
2021-09-13 13:13:47 +00:00
eb72799f0d
chore: Update datafusion (and arrow et al) dependencies ( #2509 )
...
* chore: update datafusion and other deps
* fix: Update InfluxRPC frontend with new op types
* fix: Update test output for new column names
* fix: typos and unintended changes
* fix: Update query_tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-10 17:57:25 +00:00
368f0369ee
chore: Rust 1.55
2021-09-10 12:36:49 +02:00
5d2b4a87c3
refactor: use snafu
2021-09-09 16:34:20 -04:00
1effb11ad9
refactor: move delete parseing work out of influxdb_line_protocol crate
2021-09-09 15:27:17 -04:00
b4f8fad400
refactor: address review comments and remove uno longer needed dependencies
2021-09-09 14:24:58 -04:00
00df7b064c
feat: finally have the delete predicate parsed
2021-09-08 17:30:10 -04:00
dbe4bcff22
chore: merge main to branch
2021-09-07 10:54:59 -04:00
9ee1bdeeb9
refactor: address review comments
2021-09-07 10:24:38 -04:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2
...
Bumps [parking_lot](https://github.com/Amanieu/parking_lot ) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2 )
---
updated-dependencies:
- dependency-name: parking_lot
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
d8b3208896
refactor: sstart building parser in the client side
2021-09-03 17:56:45 -04:00
a4183de411
feat: more progress on the delete flow from grpc API to catalog chunks
2021-08-31 17:42:07 -04:00
b42784d8a0
refactor: address review comments
2021-08-30 16:31:59 -04:00
7edb3fd270
fix: time column is alwyas last even if the column is sorted lexicographically
2021-08-30 14:48:18 -04:00
f42f0349ed
feat: Implement basic metrics for `DeduplicateExec`, `IOxReadFilterNode`, `SchemaPivotExec` and `StreamSplitExec` ( #2387 )
...
* feat: Add baseline metrics to DeduplicateExec
* feat: Add metrics to `IOxReadFilterNode`
* feat: Add metrics for SchemaPivotExec
* feat: Add metrics to StreamSplitExec
* fix: Update for new API, cleanups
* test: Add tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 20:31:28 +00:00
e3e801d29a
feat: propagate span context into storage RPC queries ( #2407 )
...
* feat: propagate span context into storage RPC queries
* refactor: create ExecutionContextProvider trait
* chore: cleanup imports
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-26 17:11:49 +00:00
ddf6c6362e
chore: update DataFusion again ( #2411 )
...
* chore: update datafusion ref
* chore: run cargo update
* refactor: Rename concurrency to target_partitions, avoid deprecation warning
2021-08-26 08:03:13 +00:00
5d97acb2f3
Merge branch 'main' into crepererum/issue2372
2021-08-25 07:08:15 +00:00
f7792aafe6
feat: query tracing ( #2273 ) ( #2391 )
...
* feat: query tracing (#2273 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-24 17:35:59 +00:00
745eaa9248
chore: Update DataFusion + other deps (to get new Metric API) ( #2385 )
...
* chore: Update deps
* refactor: Update IOx to use new DataFusion Metric API
* fix: update Modulus --> Modulo
2021-08-24 16:07:23 +00:00
Nga Tran
Andrew Lamb
kodiakhq[bot]
Edd Robinson
Marco Neumann
Raphael Taylor-Davies
dependabot[bot]