ae1db613eb
Merge branch 'main' into dom/test-helpers
2023-02-22 10:28:59 +00:00
1c6020d5f6
chore: update heappy to `1d6ac77a4026fffce8680a7b31a9f6e9859b5e73` ( #7039 )
...
Removes A LOT of duplicate dependencies.
2023-02-22 10:20:45 +00:00
4c4137a1b8
Merge branch 'main' into dom/test-helpers
2023-02-22 10:15:42 +00:00
6274252a17
chore(deps): Bump rustyline from 10.1.1 to 11.0.0 ( #7030 )
...
* chore(deps): Bump rustyline from 10.1.1 to 11.0.0
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 10.1.1 to 11.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v10.1.1...v11.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* fix: rustyline 11 breakage
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-02-22 10:05:44 +00:00
e9ec213b72
refactor: remove `TaskConfig` param from `chunks_to_physical_nodes` ( #7019 )
...
This makes it easier to use it from optimizer passes.
Ref #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-22 09:19:59 +00:00
929ac9081e
feat: Rewrite logical expression to match InfluxQL behaviour ( #7031 )
...
* chore: Move to inline snapshots
* chore: Container for the DataFusion and IOx schema
* chore: Simplify using logical expression helper functions
* feat: Rewrite conditional expressions using InfluxQL rules
* feat: Add tests to validation conditional expression rewriting
* feat: Rewrite column expressions
* chore: Rewrite expression to use false when possible
This allows the planner to optimise away the entire logical plan to an
empty plan in many cases.
* feat: Complete cast postfix operator support
Added `unsigned` postfix operator, as the feature was mostly complete.
Closes #6895
* chore: Remove redundant attribute
2023-02-21 20:01:31 +00:00
c9417bb7f1
chore(deps): Bump croaring from 0.8.0 to 0.8.1 ( #7034 )
...
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.8.0 to 0.8.1.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.8.0...0.8.1 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-21 16:39:05 +00:00
aa7d458a81
chore(deps): Bump tokio-stream from 0.1.11 to 0.1.12 ( #7035 )
...
Bumps [tokio-stream](https://github.com/tokio-rs/tokio ) from 0.1.11 to 0.1.12.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.11...tokio-stream-0.1.12 )
---
updated-dependencies:
- dependency-name: tokio-stream
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-21 16:31:55 +00:00
bfa476c08f
test: PartitionStream construction helper
...
Mocking out query responses requires constructing a PartitionResponse
containing the set of PartitionStream, itself a stream of RecordBatch.
This nested stream of structures is required to enable a pull-based /
streaming query response, but makes testing difficult because the types
are hard to initialise.
This commit adds a helper macro make_partition_stream! which when
combined with make_batch! to initialise the inner RecordBatch instances,
reduces the developer burden when writing test code that interacts with
query responses:
let stream = make_partition_stream!(
PartitionId::new(1) => [
make_batch!(
Int64Array("a" => vec![1, 2, 3, 4, 5]),
Float32Array("b" => vec![4.1, 4.2, 4.3, 4.4, 5.0]),
),
make_batch!(
Int64Array("c" => vec![1, 2, 3, 4, 5]),
),
],
PartitionId::new(2) => [
make_batch!(
Float32Array("d" => vec![1.1, 2.2, 3.3, 4.4, 5.5]),
),
],
);
The above yields a PartitionStream containing two partitions, with their
respective RecordBatch instances.
2023-02-21 16:09:54 +01:00
88d2882350
Merge branch 'main' into alamb/remove_old_algorithm
2023-02-21 09:02:35 -06:00
9720096f78
test: RecordBatch constructor helper
...
I always find it tedious initialising a RecordBatch (including its
schema) with a given set of rows/columns - this macro simplifies it to:
let (batch, schema) = make_batch!(
Int64Array("a" => vec![1, 2, 3, 4]),
Float32Array("b" => vec![4.1, 4.2, 4.3, 4.4]),
);
Resulting in a 4 row, 2 column ("a" and "b") RecordBatch & Schema.
2023-02-21 15:47:32 +01:00
49b9794869
Merge pull request #7025 from influxdata/jrb_6_compactor_query_thread_default
...
chore: change default for compactor_config.query_exec_thread_count
2023-02-21 08:26:23 -06:00
19be6df6cd
Merge branch 'main' into jrb_6_compactor_query_thread_default
2023-02-21 08:17:42 -06:00
716c469324
feat: Implement all-in-one local persistence testing mode ( #7027 )
...
* feat: Implement all-in-one local persistence testing mode
* fix: Apply suggestions from code review
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
---------
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-21 12:45:25 +00:00
a6111044aa
Merge pull request #7029 from influxdata/dependabot/cargo/http-0.2.9
...
chore(deps): Bump http from 0.2.8 to 0.2.9
2023-02-20 10:51:29 +00:00
77ee4d512a
chore(deps): Bump http from 0.2.8 to 0.2.9
...
Bumps [http](https://github.com/hyperium/http ) from 0.2.8 to 0.2.9.
- [Release notes](https://github.com/hyperium/http/releases )
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/http/compare/v0.2.8...v0.2.9 )
---
updated-dependencies:
- dependency-name: http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 02:54:56 +00:00
b1b7865d35
fix: Refactor storage to properly handle binary and text meta tags ( #7012 )
...
* fix: Refactor storage to properly handle binary and text meta tags
* fix: placate linter
2023-02-19 21:12:41 +00:00
b3927c89a7
chore: clippy cleanup
2023-02-17 15:30:13 -06:00
3997de6a52
chore: cleanup formating
2023-02-17 15:29:44 -06:00
7b97cdd69c
chore: change default for compactor_config.query_exec_thread_count to be 1 less than the CPU count
2023-02-17 15:29:44 -06:00
d82d00b847
docs(compactor2): Update compactor2 config parameter documentation ( #7022 )
...
* chore: Update compactor2 config parameter documentaton
* fix: clarify ording
2023-02-17 21:09:17 +00:00
65ba208f88
fix: Remove shard_id from the Parquet File protobuf in the catalog service
2023-02-17 13:53:03 -05:00
20250d883e
fix: Remove shard_id from the catalog service Partition
2023-02-17 12:56:51 -05:00
fb5aa25c5b
fix: Separate most_recent_n into filtering by shard and not
2023-02-17 12:56:51 -05:00
9351dc6c17
fix: Remove unused method list_by_shard
2023-02-17 12:56:51 -05:00
b785f751b3
feat(compactor): add simulator output ( #7021 )
2023-02-17 15:04:26 +00:00
7ff3b8b19d
Merge remote-tracking branch 'origin/main' into alamb/remove_old_algorithm
2023-02-17 09:45:20 -05:00
d90443d9e6
refactor: remove files_filter too
2023-02-17 09:45:00 -05:00
bda2310ca1
feat: extract chunks from phys. plan ( #7018 )
...
* feat: extract chunks from phys. plan
For #6098 .
* test: ensure that `extract_chunks` does NOT scan through other nodes
2023-02-17 11:41:39 +00:00
21a3c8c40d
refactor: delete all at once algorithm
2023-02-17 06:24:26 -05:00
342ef56b03
feat: streaming parquet --> line protocol conversion ( #6989 )
...
* feat: streaming parquet file conversion
* fix: use async io
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-17 11:14:39 +00:00
ae58831467
test: add a test that have over 2 times ax limit files per plan ( #7017 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-17 10:42:31 +00:00
49dbb04559
chore(deps): Bump mockito from 0.31.1 to 0.32.3 ( #6966 )
...
* chore(deps): Bump mockito from 0.31.1 to 0.32.0
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.31.1 to 0.32.0.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.31.1...0.32.0 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: use async methods
* chore: update mockito to 0.32.3
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-17 09:46:02 +00:00
a8feed120c
test: `chunks_to_physical_nodes` ( #7013 )
...
No new actual code but sets up some test infra that I need for #6098 .
2023-02-17 09:37:43 +00:00
f69c8adc7c
feat: Compact partition with many L0 files ( #7007 )
...
* feat: initial implementation of the split
* feat: split many L0 files in groups and compact them into new and fewer L0 files
* test: remove iappropriate AllAtOnce test
* refactor: move file classification for initial target to its own function
* fix: pop the branch from start to end
* chore: address review comments
* feat: support splitting to many L1 files
* feat: only add extra round to compact level-n files to same level-n files if their files plus overlapped level-n-plus-1 over limit
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: final cleanup and address comments
* chore: run fmt
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 21:17:25 +00:00
a06f64b198
chore(deps): Bump insta from 1.26.0 to 1.28.0 ( #7016 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.26.0 to 1.28.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.26.0...1.28.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-16 18:14:25 +00:00
d093f2b3ca
Merge pull request #7009 from influxdata/cn/more-ingester-tests
...
test: Ingester integration tests that can have a little a internal state
2023-02-16 17:32:30 +00:00
0904cc987a
Merge branch 'main' into cn/more-ingester-tests
2023-02-16 17:25:45 +00:00
27890b313f
chore: Update datafusion ( #6997 )
...
* chore: Update datafusion
* chore: update the plans
* fix: update some plans
* chore: Update plans and port some explain plans to use insta snapshots
* fix: another plan
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 17:03:25 +00:00
fea5245148
refactor: move GapFillParams to its own module ( #7014 )
...
* refactor: move params to own module
* chore: cargo fmt
2023-02-16 16:52:52 +00:00
1d4f8d2c8d
test: Ingester integration tests that can have a little a internal state
...
As a treat.
2023-02-16 11:06:44 -05:00
2fe9d9647f
refactor: Change the types returned from the IngesterRpcInterface
2023-02-16 10:02:17 -05:00
b840ed0ad9
fix: Use `as_expr` vs `col` to avoid splitting identifiers with periods ( #7011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 11:03:06 +00:00
b11228c72e
test: Write an e2e test of namespace soft-deletion ( #7002 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-16 09:59:14 +00:00
95736ddd1a
chore(deps): Bump clap from 4.1.4 to 4.1.6 ( #7010 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.1.4 to 4.1.6.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.1.4...v4.1.6 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 08:26:22 +00:00
822063b7f2
feat: remember `QueryChunk` for every parquet file ( #7000 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 08:02:13 +00:00
e41cf080b4
feat: `RecordBatchesExec` remembers chunks ( #6999 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 07:55:35 +00:00
67794bccdb
refactor: `group_potential_duplicates` cannot fail ( #6998 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-15 19:28:02 +00:00
9e7153daee
Merge pull request #7006 from influxdata/alamb/target_level_choice
...
chore(compactor): Move target level choice into round info
2023-02-15 14:19:27 -05:00
63877ab314
chore: Move target level choice into round info
2023-02-15 14:00:35 -05:00
Dom
Marco Neumann
dependabot[bot]
Stuart Carnie
Joe-Blount
Andrew Lamb
Joe-Blount
Carol (Nichols || Goulding)
Andrew Lamb
Nga Tran
kodiakhq[bot]
Christopher M. Wolff
Carol (Nichols || Goulding)