753f8c8410
enhance: [2.5] Reduce the allocation of ID ( #41005 )
...
This PR is a supplement to PR
https://github.com/milvus-io/milvus/pull/40960 .
issue: https://github.com/milvus-io/milvus/issues/40207
pr: https://github.com/milvus-io/milvus/pull/40960
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 17:58:21 +08:00
1caeac937b
fix: [2.5] Fix delete data loss due to duplicate binlogID ( #40985 )
...
This PR is a supplement to PR
[#40960 ](https://github.com/milvus-io/milvus/pull/40960 ).
issue: https://github.com/milvus-io/milvus/issues/40207
pr: https://github.com/milvus-io/milvus/pull/40960
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 10:38:31 +08:00
27ea5d14dc
fix: [2.5] Fix delete data loss due to duplicate binlogID ( #40976 )
...
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816 ), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.
issue: https://github.com/milvus-io/milvus/issues/40207
pr: https://github.com/milvus-io/milvus/pull/40960
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-28 14:34:21 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
bd5c56a77b
fix: [cp25]Negative -1 executing compaction tasks ( #39955 )
...
See also: #39675
pr: #39954
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-19 18:42:53 +08:00
e46c8ba7fb
fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats ( #39490 )
...
issue: #39333
master pr: #39489
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-01-22 10:27:05 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
0693634f62
enhance: add db name in replica description ( #38673 )
...
issue: #36621
pr: #38672
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:43:04 +08:00
306e5e6898
enhance: clean compaction task in compactionHandler ( #38170 )
...
issue: #35711
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-12-19 12:36:47 +08:00
60dd55f292
enhance: add end time stats for compaction and index build tasks ( #38048 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 16:28:40 +08:00
1dbb6cd7cb
enhance: refine the datacoord meta related interfaces ( #37957 )
...
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
1a49da2cc0
fix: refuse schedule compaction tasks if there is no slot ( #37589 )
...
See #37621
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-11-13 15:12:30 +08:00
5ab3c562da
enhance: tidy compaction logs ( #37595 )
...
Remove some annoying logs and lower a log level from warn to info
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-12 21:18:29 +08:00
4926021c02
fix: Skip mark compaction timeout for mix and l0 compaction ( #37118 )
...
Timeout is a bad design for long running tasks, especially using a
static timeout config. We should monitor execution progress and fail the
task if the progress has been stale for a long time.
This pr is a small patch to stop DC from marking compaction tasks
timeout, while still waiting for DN to finish. The design is
self-conflicted. After this pr, mix and L0 compaction are no longer
controlled by DC timeout, but clustering is still under timeout control.
The compaction queue capacity grows larger for priority calc, hence
timeout compactions appears more often, and when timeout, the queuing
tasks will be timeout too, no compaction will success after.
See also: #37108 , #37015
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-28 14:33:29 +08:00
49b562207c
fix: Refine compactionTask to avoid data race ( #36936 )
...
issue: #36897
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-24 09:55:28 +08:00
277c50fd31
fix:add log in mixCompactionTask and set fail/timeout task to clean ( #35970 )
...
issue: #35966
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-17 18:05:23 +08:00
d230b91bd1
enhance: Add PreallocatedSegmentIDs for the compaction task ( #36734 )
...
Add `PreallocatedSegmentIDs` field to the compaction task, allowing the
`ResultSegments` in the compaction task to represent the final segments
produced by the compaction.
issue: https://github.com/milvus-io/milvus/issues/36733
also related: https://github.com/milvus-io/milvus/issues/36686
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-13 17:59:21 +08:00
c7a81d5de3
enhance:use proto.clone in compaction tasks to prevent omissions ( #36624 )
...
issue: #36623
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-08 11:23:23 +08:00
7ff41697f9
enhance: make slotUsage a field of CompactionTask ( #36510 )
...
#36509
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-26 20:19:14 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
fd7221f4be
fix: Fix accidentlly exit MixCompaction task loop ( #34688 )
...
See also: #33431 , #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 15:57:42 +08:00
1495b7a2c9
fix: Fix no L0 compaction tasks are executed ( #34537 )
...
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.
issue: https://github.com/milvus-io/milvus/issues/34460
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-15 17:01:37 +08:00
d7a3697fb5
enhance: Add back compactionTaskNum metrics ( #34583 )
...
Fix L0 compaction task recover unable to set segment not isCompacting
See also: #34460
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
d18c49013b
enhance: Refine compaction ( #33982 )
...
issue : https://github.com/milvus-io/milvus/issues/32939
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-25 10:08:03 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
feeb869ff9
enhance: Remove compaction plans on the datanode ( #33548 )
...
issue: #33546
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yihao.dai
congqixia
XuanYang-cn
cai.zhang
Zhen Ye
jaime
tinswzy
Ted Xu
wayblink
zhenshan.cao