2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
77f6153f72
refactor: remove `QueryDatabase::chunk_summaries` ( #3977 )
...
- This is not used by the query engine at all.
- The query engine should not care about ALL chunks but only about the
chunks it gets via `QueryDatabase::chunks` (which includes a table
name and a predicate).
- All other users of that API are NOT really query-related.
2022-03-08 11:34:26 +00:00
5cc1c697fc
refactor: remove `QueryDatabase::partition_addr` ( #3976 )
...
- This was not actually used by the query engine.
- The query engine doesn't have a concept of a "partition", it only
cares about chunks.
- Unbound access to all partitions in the database is quite expensive
(esp. on NG).
2022-03-08 11:17:31 +00:00
80fb75d90b
feat: add a flag to enable per-partition tracing ( #3928 )
...
* feat: add a flag to enable per-partition tracing
* chore: rename constant
* feat: use BooleanFlag and cache result
2022-03-07 13:49:23 +00:00
7b28fb4366
feat: improve trace naming ( #3931 )
...
* feat: improve trace naming
* test: test span description
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-07 11:49:19 +00:00
9d8bceccbf
test: Add test to verify deduplicating is working ( #3937 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 20:05:17 +00:00
e09f39d6a0
chore: Update datafusion ( #3943 )
...
* chore: Update datafusion
* refactor: update for new datafusion
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-04 19:37:46 +00:00
e304613546
feat: include trace ID in query log ( #3912 ) ( #3923 )
...
* feat: include trace ID in query log (#3912 )
* chore: fmt
* chore: lint
2022-03-03 17:50:49 +00:00
de7c46c9bb
feat: add read_window_aggregate tracing
2022-03-03 14:30:27 +00:00
ea32bc366a
feat: add read_group tracing
2022-03-03 14:27:01 +00:00
32baaa1ee7
feat: add tracing to field_columns
2022-03-03 14:27:01 +00:00
787a848bf5
feat: add tracing for tag_values
2022-03-03 14:27:01 +00:00
6a6fbf73ae
feat: add tracing support tag_keys
2022-03-03 14:27:01 +00:00
998e205c2c
feat: trace table_names
2022-03-03 14:27:01 +00:00
301ae886ce
feat: add tracing down to the chunk level ( #3804 )
...
* refactor: wire exectution context to Deduplicator
* feat: example trace to chunk read_filter
* refactor: make execution context required
* refactor: expose metadata API
* refactor: more span context for chunk read_filter
* refactor: fix build
* refactor: push context into result stream
* refactor: make executor optional
2022-03-03 14:27:00 +00:00
f2126c76d3
refactor: address PR feedback ( #3906 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 11:04:05 +00:00
3d047073b9
feat: add tracing down to the chunk level ( #3804 )
...
* refactor: wire exectution context to Deduplicator
* feat: example trace to chunk read_filter
* refactor: make execution context required
* refactor: expose metadata API
* refactor: more span context for chunk read_filter
* refactor: fix build
* refactor: push context into result stream
* refactor: make executor optional
2022-03-02 19:08:22 +00:00
286d5f7b2b
feat: add `success` column to system.queries ( #3891 )
...
* feat: add `success` column to system.queries
* refactor: Remove lifetime from QueryCompletedToken and thread through flight
* test: update test to make incomplete query clearer
* refactor: use better patter to set complete
* fix: logical merge conflict
2022-03-02 15:05:06 +00:00
ace4af1b66
feat: `DedicatedExecutor` async `join` and job `detach`. ( #3835 )
...
* feat: detach dedicated exec jobs
* feat: async `DedicatedExecutor::join`
Now `DedicatedExecutor` follows the system we use for other server
components:
- `shutdown`: a quick sync call that signals the shutdown but doesn't
drop
- `join`: async awaits until the executor has finished shutdown
- `drop`: warn but still try to shut down
* test: irmpove `detach_receiver` test
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-03-01 15:25:31 +00:00
33851be3a5
chore: upgrade Rust to 1.59 ( #3875 )
...
Mostly a few new clippy crates around `flat_map`, `and_then`, and
"underscore locks" (!!!):
https://rust-lang.github.io/rust-clippy/master/index.html#let_underscore_lock
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 15:14:19 +00:00
2a842fbb1a
feat: correctly sort data and store in catalog metadata ( #3864 )
...
* feat: respect sort order in ChunkTableProvider (#3214 )
feat: persist sort order in catalog (#3845 )
refactor: owned SortKey (#3845 )
* fix: size tests
* refactor: immutable SortKey
* test: test sort order restart (#3845 )
* chore: explicit None for sort key
* chore: test cleanup
* fix: handling of sort keys containing fields
* chore: remove unused selected_sort_key
* chore: more docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 17:56:27 +00:00
8edc462c37
fix: while executing deduplication, do not return empty record batches as a result of deduplication ( #3861 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 15:00:13 +00:00
a32b952104
fix: chunk overlap missing columns ( #3844 )
...
* fix: chunk overlap missing columns
* chore: more tests
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 09:46:03 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` ( #3819 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:10:43 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
9588b43a90
fix: Make errors in rewriting return `Error` rather than a `panic` ( #3767 )
...
* test: add test for predicate errors
* fix: Return errors properly rather than panic
* fix: handle errors in influxrpc planner
* fix: appease clippy
* fix: tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-17 15:39:14 +00:00
0b3f76462d
feat: build Query Plan that queries QueryableBatch with filters ( #3742 )
...
* feat: initial implementaion the Query Plan that query QueryableBatch with filters
* fix: read_filter of QueryableBatch should provide the shema of the columns/projection it needs
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: address review comment
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-15 16:06:26 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
d1c71ba5d8
feat: predicate pushdown for Ingester's QueryableBatch ( #3728 )
...
* feat: predicate pushdown for Ingester's QueryableBatch
* chore: comment cleanup
* chore: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: address review comments
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:28:52 +00:00
d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively ( #3633 )
...
* chore: update datafusion
* fix: Update to use new datafusion api
* chore: update expected plans
* fix: support zero output partitions
* fix: update test
* fix: Update for new DataFusion API
* fix: newly added system table
* fix: update cargo lock
2022-02-09 19:53:41 +00:00
c18ad4ac97
feat: special case max timestamp range for table_names and field_columns ( #3642 )
2022-02-08 16:09:36 +00:00
be662ec731
feat: lazy query log! ( #3654 )
...
* feat: lazy query log
* chore: fmt
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:07:28 +00:00
e6ec8ef5f3
test: tests to show predicate simplification on chunks ( #3649 )
...
* test: tests to show predicate simplification on chunks
* fix: clippy
* refactor: less Box
* refactor: make typealias + add comments, hopefully to improve clarity
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 15:04:20 +00:00
2e30483f1f
refactor: Remove predicate module from predicate crate ( #3648 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 14:54:07 +00:00
77b80e7618
fix(InfluxQL): treat null tags as `''` rather than `null` in storagerpc queries ( #3557 )
...
* fix(InfluxQL): treat null tags as `''` rather than `null` in storage rpc queries
* test: add one more case
* fix: Update comment
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 12:14:43 +00:00
a2ed6a1b75
Merge branch 'main' into combine-non-overlapping-chunks
2022-02-02 20:47:51 +00:00
429d59f1b6
feat: Simplify predicates in the `InfluxRpcFrontend` before using them ( #3588 )
...
* feat: normalize + simplify RPC predicates before using them
* docs: Update predicate/src/rpc_predicate.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 19:46:57 +00:00
8a8de19fb5
feat: combine non-overlapping chunks without deletes
2022-02-02 16:40:30 +00:00
030a2cb4c1
chore: Update datafusion ( #3613 )
...
* chore: Update datafusion
* fix: update for latest DF API
* fix: another API change
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 16:27:11 +00:00
b03c900174
fix: handle IS NULL / IS NOT NULL during rpc rewrite
2022-02-01 13:48:58 -05:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
7b96a37165
chore: Update datafusion ( #3586 )
...
* chore: update DataFusion to f849968057ddddccc9aa19915ef3ea56bf14d80d
* fix: reduce overhead of creating physical expressions
* chore: use MemTrackingMetrics
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-31 18:15:28 +00:00
32d7c4cbfe
refactor: remove InfluxColumnType::IOx ( #3565 )
...
* refactor: remove InfluxColumnType::IOx
Remove unused column variant - see #3554 for context.
* refactor: reserve SEMANTIC_TYPE_IOX name in proto
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 21:15:36 +00:00
d1d45fe818
feat: columnar predicate pruning across `Chunks` ( #3553 )
...
* feat: columnar predicate pruning
* fix: doc
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 17:02:46 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
8dd96127d6
fix: Reuse the same DataFusion DiskManager and MemoryManager (Do not recreate temp files) ( #3515 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:04:06 +00:00
21c1824a7a
refactor: remove table_names from Predicate ( #3545 )
...
* refactor: remove table_names from Predicate
* chore: fix benchmarks
* chore: review feedback
Co-authored-by: Edd Robinson <me@edd.io>
* chore: review feedback
* chore: replace Default::default with InfluxRpcPredicate::default()
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:44:49 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
Andrew Lamb
Marco Neumann
Raphael Taylor-Davies
Edd Robinson
Nga Tran
dependabot[bot]
Carol (Nichols || Goulding)
Andrew Lamb
kodiakhq[bot]
Raphael Taylor-Davies
Dom