041f14ae50
Merge remote-tracking branch 'origin/main' into jrb_21_backfill_handling
2023-03-28 10:34:05 -05:00
f834df004b
chore: insta churn
2023-03-28 10:23:57 -05:00
a241ba1596
chore: sort files before selecting split candidate
2023-03-28 10:23:26 -05:00
f16622fdc1
docs: add compactor invariants and rules ( #7346 )
...
* docs: add compactor invariants and rules
* chore: address review commnents
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 13:35:04 +00:00
0db347bc1a
Merge remote-tracking branch 'origin/main' into jrb_21_backfill_handling
...
# Conflicts:
# compactor2/tests/layouts/knobs.rs
# compactor2/tests/layouts/large_files.rs
# compactor2/tests/layouts/many_files.rs
2023-03-27 14:20:16 -05:00
638a8ea4a8
chore: move random backfill tests to their own file
2023-03-27 13:48:16 -05:00
ed282ed738
chore: add case to cause non-overlapped file to be processed by high_l0_overlap_split
2023-03-27 13:40:06 -05:00
6a60914f95
chore: insta updates
2023-03-27 11:15:24 -05:00
976dadb4d4
chore: split out the time range selection for splitting into separate function
2023-03-24 16:59:02 -05:00
4b888c7255
chore(deps): Bump insta from 1.28.0 to 1.29.0 ( #7322 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.28.0 to 1.29.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.28.0...1.29.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:25:01 +00:00
07d5011d7a
refactor: Rename TriedToCompactTooMuch to CompactAndSplitOutput
2023-03-24 12:05:15 -04:00
5dd08a233d
refactor: Rename FoundFilesToSplit to ReduceLargeFileSize
2023-03-24 11:59:52 -04:00
fc1bea8502
docs: Add comments when the reason chosen is irrelevant to the test
2023-03-24 11:56:40 -04:00
3eb5c916d9
chore: insta updates
2023-03-24 10:01:37 -05:00
5a49ae0efd
feat: vertical split files when L0s are highly overlapped
2023-03-24 10:01:07 -05:00
3fe8bda51a
feat: Record why compact, split, or nothing plans were picked
...
And display the reason in the simulator tests to ensure decisions are
consistent.
Fixes influxdata/idpe#17306 .
2023-03-23 16:58:17 -04:00
acb491aa98
Merge branch 'main' into cn/refactor
2023-03-23 15:27:46 +00:00
2379f00929
test: set max_l0_created_at appropriately for tests of many_files ( #7304 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 15:27:28 +00:00
98ec67c564
docs: Add suggested clarification comment
2023-03-23 11:20:06 -04:00
eb232cadc8
test: set max_l0_created_at approriately for tests in single_timestamp ( #7305 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 15:19:31 +00:00
efa9732ff1
docs: Copy edit doc comments
2023-03-23 11:15:35 -04:00
28bab6899a
test: set max_l0_created_at to more appropriately value for large_overlaps tests ( #7303 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 13:47:16 +00:00
77948e3341
chore(iox/compactor): add test cases tracking max_l0_created_at ( #7181 )
...
* chore(iox/compactor): add test cases tracking max_l0_created_at
* chore(iox/compactor): add invariant check to simulator for misordered max_l0_created_at
* chore(iox/compactor): test updates for max_l0_created_at
This commit updates the newly added tests to comply with the new invariant check.
Several test cases are deleted because the changes to comply with the invariant make the permutations pointless.
* chore(iox/simulator): Add test cases hitting new invariant check for illegal max_l0_created_at
* chore: update previous tests to comply with new invariant
* chore: address comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-22 22:14:06 +00:00
2b8db8d645
refactor: Move decision about what plan to create into IRPlanner
2023-03-22 17:03:04 -04:00
4c551a3ce9
refactor: Move uuid merging further into plan building
2023-03-22 16:53:28 -04:00
626dd236f9
refactor: Have run_plans take ownership of split_or_compact
...
Making this clone more explicit, higher up. Deleting the old files from
the catalog might only need the IDs, and the compaction levels for the
metrics, but that can be later.
2023-03-22 16:45:04 -04:00
a58ef38bb0
refactor: Move target_level into PlanIR
...
Because this info needs to stay with the rest of the plan info
2023-03-22 16:15:16 -04:00
a20c5dcd8b
refactor: Move the logic for getting the input paths into FilesToSplitOrCompact
2023-03-22 15:51:43 -04:00
e2dcf0b5e9
refactor: Share execute_plan calls between compact and split paths
2023-03-22 15:17:40 -04:00
3235be572d
refactor: Inline run_split_plan in anticipation of combining more code
2023-03-22 15:08:16 -04:00
91ac9627fa
refactor: Inline run_compaction_plan in anticipation of combining code with run_split_plan
2023-03-22 14:44:01 -04:00
7100be4103
refactor: Inline run_split_plans in anticipation of combining with run_compaction_plan
...
Also set a capacity on the Vec to avoid some reallocations.
2023-03-22 14:18:44 -04:00
45a32f80f9
refactor: Move the decision for making progress into the filter
...
And define the conditions for that decision on a type responsible for
the data that goes into that condition.
Also start moving away from methods that only work on one variant of
FilesToSplitOrCompact, and add a state for when FilesToSplitOrCompact
doesn't actually contain any files.
2023-03-22 13:24:09 -04:00
c0b99011c8
refactor: Make some names more consistent and less redundant
...
I can't ever remember whether it's "compact or split" or "split or
compact", so now I think it's always "split or compact".
Also remove "FilesTo" from the enum variants because "FilesTo" is in the
overall enum name.
2023-03-22 13:06:11 -04:00
a3fa986f6e
refactor: Give post-classification partition filter a separate trait
...
I want to change the types, but for now, this is mostly a copy of the
partition filter trait and setting up the associated
logging/metrics/mock.
2023-03-22 13:06:11 -04:00
63bb4db6e9
refactor: Rename unable_to_compact to possible_progress
2023-03-22 13:06:11 -04:00
b78b3d3ee2
refactor: Rename UnableToCompactPartitionFilter to PossibleProgressFilter
...
I'm going to move all the decisions about whether there's progress to be
made or not into here.
2023-03-22 13:06:11 -04:00
353eda377e
refactor: Reduce complexity where there's only one filter remaining
...
The vec and the AndFilter aren't needed because there's only one filter
happening in this spot now.
2023-03-22 13:06:11 -04:00
50d9d40322
refactor: Rename a filter slot based on when it's applied
...
The filter in this spot could potentially do whatever, but the important
part of what should go in this field is that it will be called on the
files after they're classified by the file classifier.
2023-03-22 13:06:11 -04:00
3681003f44
refactor: Destructure FileClassification so the parts can be moved different places
2023-03-22 13:06:10 -04:00
222d765323
chore(compactor/simulator): add data backfill test cases ( #7293 )
...
* chore(compactor/simulator): add data backfill test cases
* chore: remove extra comment (left from copy & paste)
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-22 13:51:55 +00:00
7ee40acb22
test: set max_l0_created_at appropriately in the core tests ( #7292 )
...
* test: make max_l0_created_at more sense in the core tests
* chore: fix typo
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-21 20:40:09 +00:00
f780aba353
test: set max_l0_created_at to reasonable values for the tests and al… ( #7286 )
...
* test: set max_l0_created_at to reasonable values for the tests and also verify it using both test layout and catalog function
* fix: typo
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-21 18:57:10 +00:00
efae5246cf
docs: add comments for split/compact ( #7269 )
...
* docs: add comments for split/compact
* chore: Apply suggestions from code review
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-20 19:58:39 +00:00
46b94c1362
refactor: Rename module and file to just timeout
2023-03-16 13:40:28 -04:00
85687ecd4c
refactor: Move timeout with progress to its own module
2023-03-16 11:50:23 -04:00
c9bfcb4501
test: Actually make this test a different case, oops
2023-03-15 17:10:10 -04:00
6fd910a091
fix: Make really really sure this test hits the timeout
2023-03-15 17:06:58 -04:00
413635d25a
feat: Don't add a partition to skipped_compactions if it makes progress
...
If a partition takes longer than `partition_timeout` to compact, but it
did make _some_ progress, let the compactor try that partition again at
a later time so that compaction for the partition will eventually
complete.
If a partition times out and _no_ progress has been made, then still add
it to the skipped_compactions table because it's either too big to ever
compact or is otherwise stuck.
Closes influxdata/idpe#17234 .
2023-03-15 14:04:08 -04:00
afb3f768cf
Merge branch 'main' into dom/metric-assert-scope
2023-03-15 14:26:11 +00:00
Joe-Blount
Nga Tran
dependabot[bot]
Carol (Nichols || Goulding)
kodiakhq[bot]
Joe-Blount