3f0073807f
chore(flightsql): `insta`ize flightsql end to end tests ( #7236 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 14:41:49 +00:00
7dfaa05e8a
chore: Update datafusion again ( #7208 )
...
* chore: update datafusion again
* fix: update test
* fix: use table_reference
* fix: clean up import
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 14:34:40 +00:00
d2874f5c02
test: fix chunk boundaries for retention query tests ( #7235 )
...
See code comment, found while working on #6098 .
2023-03-16 13:14:08 +00:00
c819d4f865
feat(flightsql): Implement `GetDbSchemas` metadata API ( #7213 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 10:13:51 +00:00
6bfae7bb52
chore(deps): Bump predicates from 2.1.5 to 3.0.1 ( #7229 )
...
* chore(deps): Bump predicates from 2.1.5 to 3.0.1
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 2.1.5 to 3.0.1.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.1.5...v3.0.1 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-16 09:23:02 +00:00
2dde0658c6
chore(deps): Bump assert_cmd from 2.0.8 to 2.0.9 ( #7228 )
...
* chore(deps): Bump assert_cmd from 2.0.8 to 2.0.9
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.8 to 2.0.9.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.8...v2.0.9 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 08:52:27 +00:00
6d6fd8f663
feat(flightsql): implement basic `CommandGetCatalogs` support ( #7212 )
...
* refactor: reduce redundancy in test
* chore: implement basic get_catalog support
* fix: clippy
2023-03-15 21:52:59 +00:00
493f331e4b
fix: Remove the max_compact_size knob and hardcode a multiple ( #7197 )
...
* fix: Remove the max_compact_size knob and hardcode a multiple
Rather than panic if the user hasn't set this knob in a particular way,
set the max_compact_size to the minimum value we need by multiplying
max_desired_file_size_bytes by MIN_COMPACT_SIZE_MULTIPLE.
Fixes influxdata/idpe#17259 .
* refactor: Move computation of max_compact_size_bytes into compactor config
* test: change test setups to reflect the purposes of the tests
---------
Co-authored-by: NGA-TRAN <nga-tran@live.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-15 11:21:28 +00:00
9ef32b682c
chore(deps): Bump serde from 1.0.155 to 1.0.156 ( #7214 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.155 to 1.0.156.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.155...v1.0.156 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-15 07:14:27 -04:00
5fa88746d3
chore(deps): Bump nu-ansi-term from 0.46.0 to 0.47.0 ( #7201 )
...
Bumps [nu-ansi-term](https://github.com/nushell/nu-ansi-term ) from 0.46.0 to 0.47.0.
- [Release notes](https://github.com/nushell/nu-ansi-term/releases )
- [Changelog](https://github.com/nushell/nu-ansi-term/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nushell/nu-ansi-term/compare/v0.46.0...v0.47.0 )
---
updated-dependencies:
- dependency-name: nu-ansi-term
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-14 10:52:00 +00:00
1ad394edbe
chore(deps): Bump serde from 1.0.154 to 1.0.155 ( #7187 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.154 to 1.0.155.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.154...v1.0.155 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-13 09:49:04 +00:00
4eecf144bc
chore(deps): Bump serde from 1.0.153 to 1.0.154 ( #7162 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.153 to 1.0.154.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.153...v1.0.154 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-09 09:12:01 +00:00
c87113ccbf
chore(iox/compactor): rename max_input_parquet_bytes_per_partition ( #7160 )
2023-03-08 17:08:08 +00:00
f05f274304
chore(deps): Bump serde from 1.0.152 to 1.0.153 ( #7155 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.152 to 1.0.153.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.152...v1.0.153 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-08 09:58:07 +00:00
04202bffb6
feat: Output formatting for InfluxQL ( #7133 )
...
* feat: Output formatting for InfluxQL
* chore: Use constant for metadata key based on PR feedback
* chore: Add lock changes
2023-03-08 00:42:05 +00:00
2b74f07fe5
feat: Support `GROUP BY` with tags in raw `SELECT` queries ( #7109 )
...
* chore: Normalise name of Call expression to lowercase
Simplifies matching functions in planner, as they are guaranteed to be
lowercase.
This also ensures compatibility with InfluxQL when generating column
alias names, which are reflected in updated tests.
* chore: Ensure aggregate functions fail gracefully.
* feat: GROUP BY tag support
* feat: Ensure schema-level metadata is propagated
Requires: https://github.com/apache/arrow-rs/issues/3779
* chore: Add some tests to validate GROUP BY output
* chore: Add clarifying comment
* chore: Declare message in flight.proto
The metadata is public API, so best practice is to encode this in a way
that is most compatible for clients in other languages, and will also
document the history of schema changes.
Added tests to validate the metadata is encoded correctly.
* chore: Placate linters
* chore: Use correct column in test cases
* chore: Add `is_projected` to the TagKeyColumn message
`is_projected` is necessary to inform a client whether it should include
the tag key is used exclusively for the group key (false) or also
projected in the `SELECT` column list.
* refactor: Move constants to `schema` crate per PR feedback
* chore: rustfmt 🙄
* chore: Update docs for InfluxQlMetadata
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-03-07 22:40:23 +00:00
ed0704ac8d
chore: Update datafusion ( #7100 )
...
* chore: Update datafusion
* chore: iox_query to compile for API changes + update tests
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-06 17:59:24 +00:00
535f3d92a2
chore(deps): Bump serde_json from 1.0.93 to 1.0.94 ( #7132 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.93 to 1.0.94.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.93...v1.0.94 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 12:12:23 +00:00
8771dcb645
chore(deps): Bump thiserror from 1.0.38 to 1.0.39 ( #7131 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.38 to 1.0.39.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.38...1.0.39 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-06 10:55:26 +00:00
3256fcc72e
chore(deps): Bump object_store from 0.5.4 to 0.5.5
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.4 to 0.5.5.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.4...object_store_0.5.5 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-03 02:00:51 +00:00
c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 ( #7107 )
...
* chore(deps): Bump tokio from 1.25.0 to 1.26.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
e19ce98407
chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0 ( #7084 )
...
* chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0
* chore: Run cargo hakari tasks
* chore: Update plans
* chore: Update querier expected output
* chore: Update querier tests to use insta
* fix: sort output too
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-01 11:25:01 +00:00
e8b6e66f75
feat: misc logging/tracing improvements ( #7081 )
...
* feat: print query language/variant in info log
* feat: allow overwriting trace header name in CLI
2023-02-28 11:14:42 +00:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
4cf066680e
chore(deps): Bump tempfile from 3.3.0 to 3.4.0 ( #7069 )
...
* chore(deps): Bump tempfile from 3.3.0 to 3.4.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-27 10:02:21 +00:00
2ed5758ddb
feat: InfluxQL planner learns how to project multiple measurements ( #7063 )
...
* feat: Planner learns how to project multiple measurements
Closes #6896
* chore: Update test
* chore: PR feedback
2023-02-27 00:50:06 +00:00
08578cded5
refactor: n_threads and n_target_partitions are non-zero ( #7047 )
...
* refactor: n_threads and n_target_partitions are non-zero
Zero values will just panic. Prevent that earlier.
* fix: typo
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
---------
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-02-23 16:57:00 +00:00
0282eb4750
feat: streaming implementation of gap filling ( #7037 )
...
* feat: streaming implementation of gap filling
* chore: cargo fmt
* refactor: when gapfilling, concatenate input batches less frequently
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-23 15:42:41 +00:00
f93baf7693
chore: Update DataFusion and `arrow` / `arrow-flight` / `parquet` to `33.0.0` ( #7045 )
...
* chore: Update DataFusion and arrow/arrow-flight/parquet to 33.0.0
* fix: Update test output
* fix: update more test output
* fix: Update querier test output
* chore: Run cargo hakari tasks
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting for selector tests
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: Christopher Wolff <chris.wolff@influxdata.com>
2023-02-22 21:24:20 +00:00
48e6cce746
Merge branch 'main' into cn/less-shard-id
2023-02-22 15:05:37 +00:00
80ba49b611
Merge branch 'main' into alamb/remove_old_algorithm
2023-02-22 08:30:30 -06:00
18815b951d
chore: update TODO comment ( #7044 )
2023-02-22 14:17:30 +00:00
6274252a17
chore(deps): Bump rustyline from 10.1.1 to 11.0.0 ( #7030 )
...
* chore(deps): Bump rustyline from 10.1.1 to 11.0.0
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 10.1.1 to 11.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v10.1.1...v11.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* fix: rustyline 11 breakage
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-02-22 10:05:44 +00:00
929ac9081e
feat: Rewrite logical expression to match InfluxQL behaviour ( #7031 )
...
* chore: Move to inline snapshots
* chore: Container for the DataFusion and IOx schema
* chore: Simplify using logical expression helper functions
* feat: Rewrite conditional expressions using InfluxQL rules
* feat: Add tests to validation conditional expression rewriting
* feat: Rewrite column expressions
* chore: Rewrite expression to use false when possible
This allows the planner to optimise away the entire logical plan to an
empty plan in many cases.
* feat: Complete cast postfix operator support
Added `unsigned` postfix operator, as the feature was mostly complete.
Closes #6895
* chore: Remove redundant attribute
2023-02-21 20:01:31 +00:00
88d2882350
Merge branch 'main' into alamb/remove_old_algorithm
2023-02-21 09:02:35 -06:00
19be6df6cd
Merge branch 'main' into jrb_6_compactor_query_thread_default
2023-02-21 08:17:42 -06:00
716c469324
feat: Implement all-in-one local persistence testing mode ( #7027 )
...
* feat: Implement all-in-one local persistence testing mode
* fix: Apply suggestions from code review
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
---------
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-21 12:45:25 +00:00
77ee4d512a
chore(deps): Bump http from 0.2.8 to 0.2.9
...
Bumps [http](https://github.com/hyperium/http ) from 0.2.8 to 0.2.9.
- [Release notes](https://github.com/hyperium/http/releases )
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/http/compare/v0.2.8...v0.2.9 )
---
updated-dependencies:
- dependency-name: http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 02:54:56 +00:00
b1b7865d35
fix: Refactor storage to properly handle binary and text meta tags ( #7012 )
...
* fix: Refactor storage to properly handle binary and text meta tags
* fix: placate linter
2023-02-19 21:12:41 +00:00
b3927c89a7
chore: clippy cleanup
2023-02-17 15:30:13 -06:00
3997de6a52
chore: cleanup formating
2023-02-17 15:29:44 -06:00
7b97cdd69c
chore: change default for compactor_config.query_exec_thread_count to be 1 less than the CPU count
2023-02-17 15:29:44 -06:00
65ba208f88
fix: Remove shard_id from the Parquet File protobuf in the catalog service
2023-02-17 13:53:03 -05:00
21a3c8c40d
refactor: delete all at once algorithm
2023-02-17 06:24:26 -05:00
342ef56b03
feat: streaming parquet --> line protocol conversion ( #6989 )
...
* feat: streaming parquet file conversion
* fix: use async io
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-17 11:14:39 +00:00
f69c8adc7c
feat: Compact partition with many L0 files ( #7007 )
...
* feat: initial implementation of the split
* feat: split many L0 files in groups and compact them into new and fewer L0 files
* test: remove iappropriate AllAtOnce test
* refactor: move file classification for initial target to its own function
* fix: pop the branch from start to end
* chore: address review comments
* feat: support splitting to many L1 files
* feat: only add extra round to compact level-n files to same level-n files if their files plus overlapped level-n-plus-1 over limit
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: final cleanup and address comments
* chore: run fmt
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 21:17:25 +00:00
27890b313f
chore: Update datafusion ( #6997 )
...
* chore: Update datafusion
* chore: update the plans
* fix: update some plans
* chore: Update plans and port some explain plans to use insta snapshots
* fix: another plan
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 17:03:25 +00:00
b11228c72e
test: Write an e2e test of namespace soft-deletion ( #7002 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-16 09:59:14 +00:00
4db9df7197
feat: Add logging of commands sent via the `influxdb storage` command ( #6987 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-14 22:41:54 +00:00
5c506058da
feat: skip partitions of wide tables ( #6978 )
...
* feat: skip partitions of wide tables
* test: one more test
* refactor: address review comments
2023-02-14 16:42:13 +00:00
Andrew Lamb
Marco Neumann
dependabot[bot]
Carol (Nichols || Goulding)
Joe-Blount
Stuart Carnie
Carol (Nichols || Goulding)
Christopher M. Wolff
kodiakhq[bot]
Joe-Blount
Andrew Lamb
Nga Tran