1e129c1793
Merge pull request #7219 from influxdata/dom/metric-assert-scope
...
refactor(metric): assert macro import scopes
2023-03-15 14:33:36 +00:00
afb3f768cf
Merge branch 'main' into dom/metric-assert-scope
2023-03-15 14:26:11 +00:00
98daccc463
Merge branch 'main' into dom/wal-refs
2023-03-15 14:24:30 +00:00
84a5f291ee
docs: use correct var name
...
Co-authored-by: Fraser Savage <fraser@savage.engineer>
2023-03-15 14:24:19 +00:00
1832f682e4
test: compaction output of common use cases ( #7211 )
...
* test: common comapctor use cases
* test: add 3 L0 files during last comapction
* chore: clearer comments
* test: add intermediate test results per review request
* chore: comment only change to trigger circle CI
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* test: add test with less ingested data and fix output after main merge
* chore: run format after pulling suggestions
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-03-15 14:22:07 +00:00
45c8519a0b
refactor(metric): assert macro import scopes
...
The assert_counter! and assert_histogram! macros use items in the metric
crate, but the macros can be called from other crates/modules that may
not have those items in scope.
2023-03-15 15:10:37 +01:00
35bd5ccd13
chore: Update datafusion pin ( #7207 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-15 11:35:15 +00:00
f752e7993d
chore(deps): Bump bitflags from 1.3.2 to 2.0.0 ( #7215 )
...
* chore(deps): Bump bitflags from 1.3.2 to 2.0.0
Bumps [bitflags](https://github.com/bitflags/bitflags ) from 1.3.2 to 2.0.0.
- [Release notes](https://github.com/bitflags/bitflags/releases )
- [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md )
- [Commits](https://github.com/bitflags/bitflags/compare/1.3.2...2.0.0 )
---
updated-dependencies:
- dependency-name: bitflags
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-15 11:28:05 +00:00
493f331e4b
fix: Remove the max_compact_size knob and hardcode a multiple ( #7197 )
...
* fix: Remove the max_compact_size knob and hardcode a multiple
Rather than panic if the user hasn't set this knob in a particular way,
set the max_compact_size to the minimum value we need by multiplying
max_desired_file_size_bytes by MIN_COMPACT_SIZE_MULTIPLE.
Fixes influxdata/idpe#17259 .
* refactor: Move computation of max_compact_size_bytes into compactor config
* test: change test setups to reflect the purposes of the tests
---------
Co-authored-by: NGA-TRAN <nga-tran@live.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-15 11:21:28 +00:00
9ef32b682c
chore(deps): Bump serde from 1.0.155 to 1.0.156 ( #7214 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.155 to 1.0.156.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.155...v1.0.156 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-15 07:14:27 -04:00
ad9e3bd1ef
feat: further split the smallest possible set to compact but too large to do so ( #7180 )
...
* test: add a panic limit_files_to_compact function to it is used in the right way
* test: provide correct output to the tests
* chore: remove no-longer valid comments
* feat: have the function limit_files_to_compact to also return files_to_further_split if the minimum set to compact is too large to do so
* refactor: rename files_to_split to start_level_files_to_split
* refactor: rename identify_files_to_split to identify_start_level_files_to_split before adding new split function
* feat: split 2 files of minimum set of compacting files if they are over max compact size
* test: since now we may split files in different levels, let us remove the missleading at level from the simulation tests
* chore: clearer comments
* test: add tests for tiny time ranges
* chore: address review comments
2023-03-14 20:08:58 +00:00
bef29a6837
feat(wal): segement file reference counting
...
Reference count rotated WAL segment files, tracking the number of
unpersisted operations in each and deleting the file once all the data
within them has been persisted to object storage.
This commit contains the implementation of the reference counting logic,
and is currently unused. A follow-up PR will wire this into the various
places needed to feed it the necessary information.
Part of https://github.com/influxdata/influxdb_iox/issues/6566 .
2023-03-14 17:29:32 +01:00
f5c50094ca
refactor: owned persisted SequenceNumberSet copy
...
Allow an owned SequenceNumberSet to be destructured from an owned
CompletedPersist notification.
Additionally allow an owned SequenceNumberSet to be obtained from a
(potentially) shared Arc<CompletedPersist> in a memory efficient / no
cloning way. A CompletedPersist notification is typically wrapped in an
Arc, and at time of writing, typically referenced only once.
2023-03-14 17:29:16 +01:00
d9ca8f948a
refactor(wal): const SegmentId constructor
...
Allow a SegmentId to be constructed in a const context.
2023-03-14 17:29:15 +01:00
570c61f9a7
refactor: formalize abstraction for building gap filled columns ( #7179 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-14 15:14:02 +00:00
84a0b9acbe
refactor: Make the filter name reflect what excatly it does ( #7199 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-14 13:43:34 +00:00
bdfab5fb64
chore: opt-in for sparse registry protocol ( #7206 )
...
Faster registry updates for our developers and the CI/CD pipeline.
See <https://blog.rust-lang.org/2023/03/09/Rust-1.68.0.html#cargos-sparse-protocol >.
2023-03-14 13:09:13 +00:00
ca40177695
chore(deps): Bump mockito from 0.32.5 to 1.0.0 ( #7200 )
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.32.5 to 1.0.0.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.32.5...1.0.0 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-14 10:58:43 +00:00
5fa88746d3
chore(deps): Bump nu-ansi-term from 0.46.0 to 0.47.0 ( #7201 )
...
Bumps [nu-ansi-term](https://github.com/nushell/nu-ansi-term ) from 0.46.0 to 0.47.0.
- [Release notes](https://github.com/nushell/nu-ansi-term/releases )
- [Changelog](https://github.com/nushell/nu-ansi-term/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nushell/nu-ansi-term/compare/v0.46.0...v0.47.0 )
---
updated-dependencies:
- dependency-name: nu-ansi-term
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-14 10:52:00 +00:00
86ddf893b1
chore(deps): Bump bitflags from 1.3.2 to 2.0.0 ( #7202 )
...
* chore(deps): Bump bitflags from 1.3.2 to 2.0.0
Bumps [bitflags](https://github.com/bitflags/bitflags ) from 1.3.2 to 2.0.0.
- [Release notes](https://github.com/bitflags/bitflags/releases )
- [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md )
- [Commits](https://github.com/bitflags/bitflags/compare/1.3.2...2.0.0 )
---
updated-dependencies:
- dependency-name: bitflags
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-14 10:51:45 +00:00
f775b6bbf5
Merge pull request #7203 from influxdata/dependabot/cargo/toml-0.7.3
...
chore(deps): Bump toml from 0.7.2 to 0.7.3
2023-03-14 10:43:49 +00:00
6cc5961075
chore(deps): Bump toml from 0.7.2 to 0.7.3
...
Bumps [toml](https://github.com/toml-rs/toml ) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/toml-rs/toml/releases )
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.2...toml-v0.7.3 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-14 02:06:04 +00:00
0eb858c70d
chore: Update datafusion ( #7167 )
...
* chore: Update datafusion
* chore: Update datafusion
* refactor: use UserDefinedLogicalNodeCore
* fix: remove stray comment
* fix: clippy
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 16:41:32 +00:00
ffab683ead
refactor: move trailing_gaps bit into cursor ( #7178 )
...
* refactor: push trailing_gap bit into cursor
* chore: clippy
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 15:40:55 +00:00
a415a5b53a
chore: Update dependencies (to fix cargo_audit failures) ( #7196 )
...
* chore: Update dependencies
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-13 15:28:59 +00:00
78926490f9
docs: Update dml_handlers module docs ( #7194 )
...
Update the module diagram to reflect kafkaless
changes.
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-13 13:28:29 +00:00
737ea15d07
feat: projection pushdown phys. optimizer ( #7161 )
...
* feat: projection pushdown phys. optimizer
The is by far the largest pass (at least test-wise), because projections
are added last in the naive plan and you have to push them through
everything else. The actual code however isn't that complicated mostly
because we can reuse some DataFusion functionality and the different
variants for the different "child nodes" are very similar.
For #6098 .
* feat: projection pushdown for `RecordBatchesExec`
* test: `test_ignore_when_partial_impure_projection_rename`
* test: more dedup projection tests
* test: integration
2023-03-13 12:59:45 +00:00
3bb5347d4c
chore: Remove unused dependencies found with cargo-machete @alamb ( #7183 )
...
* chore: Remove unused dependencies found with cargo-machete
* chore: Remove unused dependencies
* fix: fixup
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 11:06:53 +00:00
3a9ca8879b
chore(deps): Bump futures from 0.3.26 to 0.3.27 ( #7193 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 10:53:59 +00:00
16e36a672d
chore(deps): Bump hyper from 0.14.24 to 0.14.25 ( #7190 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.24 to 0.14.25.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/v0.14.25/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.24...v0.14.25 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 10:47:09 +00:00
b35208be96
chore(deps): Bump futures-util from 0.3.26 to 0.3.27 ( #7189 )
...
Bumps [futures-util](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-13 10:40:29 +00:00
3a4e7c6ddd
chore(deps): Bump futures-channel from 0.3.26 to 0.3.27 ( #7188 )
...
Bumps [futures-channel](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures-channel
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 10:31:10 +00:00
41802b7b5b
feat: `SchemaAdapterStream` may create virtual columns ( #7173 )
...
* feat: `SchemaAdapterStream` may create virtual columns
For chunk order handling in #6098 .
* fix: improve `SchemaAdapterStream` docs and error handling
2023-03-13 10:02:13 +00:00
1ad394edbe
chore(deps): Bump serde from 1.0.154 to 1.0.155 ( #7187 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.154 to 1.0.155.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.154...v1.0.155 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-13 09:49:04 +00:00
70c3a261a2
chore(deps): Bump chrono from 0.4.23 to 0.4.24 ( #7186 )
...
* chore(deps): Bump chrono from 0.4.23 to 0.4.24
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.23 to 0.4.24.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.23...v0.4.24 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 09:35:17 +00:00
95ce2199ea
chore(deps): Bump futures-core from 0.3.26 to 0.3.27 ( #7185 )
...
Bumps [futures-core](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-13 09:26:20 +00:00
9150b9feb2
chore(deps): Bump futures-sink from 0.3.26 to 0.3.27 ( #7184 )
...
Bumps [futures-sink](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures-sink
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-13 09:14:32 +00:00
cc7c44f76a
chore: Upgrade to Rust 1.68 ( #7175 )
...
* chore: Upgrade to Rust 1.68
* fix: Remove unnecessary into_iter, thanks Clippy!
* fix: Use the size of the type, not a reference to the type... oops.
Thanks clippy!
* fix: Return block directly instead of creating a variable
Thanks clippy!
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-12 13:22:20 +00:00
5c00819727
test: provide correct input to the limit_files_to_compact function ( #7176 )
...
* test: add a panic limit_files_to_compact function to it is used in the right way
* test: provide correct output to the tests
* chore: Apply suggestions from code review
Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
---------
Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
2023-03-10 19:18:00 +00:00
870d2830da
feat: upgrade upgradable files when there are nothing else to compact or split ( #7174 )
...
* feat: upgrade upgradable files when there are nothing else to compact or split
* chore: fix a typo in a comment
2023-03-10 16:08:42 +00:00
cb1ef88a5d
chore(deps): Bump libc from 0.2.139 to 0.2.140 ( #7170 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.139 to 0.2.140.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.139...0.2.140 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-10 15:35:53 +00:00
8d16ba29f3
Merge pull request #7169 from influxdata/jrb_16_test_improvements
...
feat(iox/compactor/simulator): include max_l0_created_at in parquet file info printed
2023-03-10 09:06:46 -06:00
5f0059ed00
Merge remote-tracking branch 'origin/main' into jrb_16_test_improvements
...
# Conflicts:
# compactor2/tests/layouts/core.rs
# compactor2/tests/layouts/knobs.rs
# compactor2/tests/layouts/large_files.rs
# compactor2/tests/layouts/large_overlaps.rs
# compactor2/tests/layouts/many_files.rs
2023-03-10 08:55:15 -06:00
a2928db0a5
chore(iox/compactor): doc updates for max_l0_created_at addition
2023-03-10 08:45:54 -06:00
7b3db5c2b2
chore(iox/compactor): insta test updates for simulator width change
2023-03-10 08:39:43 -06:00
c5a8cd2ac8
chore(iox/compactor): update width for simulator file printing
2023-03-10 08:37:45 -06:00
6e2f5bae22
chore(iox/compactor): 3 more insta test updates
2023-03-10 08:31:05 -06:00
6a85e8644b
chore(compactor2): Assign all rows and sizes in simlator ( #7124 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-10 12:55:10 +00:00
5b359c3e30
feat: Enable TLS feature so that CLI can speak to https ( #7172 )
...
* feat: Enable TLS feature so that CLI can speak to https
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-10 08:15:12 +00:00
0868c8634e
chore(iox/compactor): update insta tests now that max_l0_created_at is displayed
2023-03-09 17:00:35 -06:00
kodiakhq[bot]
Dom
Nga Tran
Andrew Lamb
dependabot[bot]
Carol (Nichols || Goulding)
Christopher M. Wolff
Marco Neumann
Fraser Savage
Joe-Blount
Joe-Blount
Marko Mikulicic