753f8c8410
enhance: [2.5] Reduce the allocation of ID ( #41005 )
...
This PR is a supplement to PR
https://github.com/milvus-io/milvus/pull/40960 .
issue: https://github.com/milvus-io/milvus/issues/40207
pr: https://github.com/milvus-io/milvus/pull/40960
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 17:58:21 +08:00
27ea5d14dc
fix: [2.5] Fix delete data loss due to duplicate binlogID ( #40976 )
...
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816 ), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.
issue: https://github.com/milvus-io/milvus/issues/40207
pr: https://github.com/milvus-io/milvus/pull/40960
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-28 14:34:21 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
bd5c56a77b
fix: [cp25]Negative -1 executing compaction tasks ( #39955 )
...
See also: #39675
pr: #39954
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-19 18:42:53 +08:00
e46c8ba7fb
fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats ( #39490 )
...
issue: #39333
master pr: #39489
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-01-22 10:27:05 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
0693634f62
enhance: add db name in replica description ( #38673 )
...
issue: #36621
pr: #38672
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:43:04 +08:00
306e5e6898
enhance: clean compaction task in compactionHandler ( #38170 )
...
issue: #35711
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-12-19 12:36:47 +08:00
60dd55f292
enhance: add end time stats for compaction and index build tasks ( #38048 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 16:28:40 +08:00
1dbb6cd7cb
enhance: refine the datacoord meta related interfaces ( #37957 )
...
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
26028f4137
fix: Exlude L0 compaction when clustering is executing ( #37141 )
...
Also remove conflit check when executing L0. The exclusive is already
guarenteed in scheduler
See also: #37140
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-29 06:28:24 +08:00
4926021c02
fix: Skip mark compaction timeout for mix and l0 compaction ( #37118 )
...
Timeout is a bad design for long running tasks, especially using a
static timeout config. We should monitor execution progress and fail the
task if the progress has been stale for a long time.
This pr is a small patch to stop DC from marking compaction tasks
timeout, while still waiting for DN to finish. The design is
self-conflicted. After this pr, mix and L0 compaction are no longer
controlled by DC timeout, but clustering is still under timeout control.
The compaction queue capacity grows larger for priority calc, hence
timeout compactions appears more often, and when timeout, the queuing
tasks will be timeout too, no compaction will success after.
See also: #37108 , #37015
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-28 14:33:29 +08:00
49b562207c
fix: Refine compactionTask to avoid data race ( #36936 )
...
issue: #36897
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-24 09:55:28 +08:00
277c50fd31
fix:add log in mixCompactionTask and set fail/timeout task to clean ( #35970 )
...
issue: #35966
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-17 18:05:23 +08:00
c7a81d5de3
enhance:use proto.clone in compaction tasks to prevent omissions ( #36624 )
...
issue: #36623
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-08 11:23:23 +08:00
7ff41697f9
enhance: make slotUsage a field of CompactionTask ( #36510 )
...
#36509
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-26 20:19:14 +08:00
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
06f9ba294e
enhance: Add more ut for l0CompactionTask ( #35100 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 14:24:18 +08:00
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
9a07790215
enhance: Add ut for l0CompactionTask processExecuting ( #34800 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-29 10:53:48 +08:00
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
90e765db87
fix: fix bug if datacoord restart when l0 compaction saveSegmentMeta ( #34679 )
...
#34681
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 13:03:41 +08:00
fd7221f4be
fix: Fix accidentlly exit MixCompaction task loop ( #34688 )
...
See also: #33431 , #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 15:57:42 +08:00
1495b7a2c9
fix: Fix no L0 compaction tasks are executed ( #34537 )
...
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.
issue: https://github.com/milvus-io/milvus/issues/34460
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-15 17:01:37 +08:00
d7966f46ad
fix: Compaction task l0 state transfer wrong ( #34597 )
...
This bug caused failed L0 compaction tasks never end.
See also: #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 21:39:36 +08:00
d7a3697fb5
enhance: Add back compactionTaskNum metrics ( #34583 )
...
Fix L0 compaction task recover unable to set segment not isCompacting
See also: #34460
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
d18c49013b
enhance: Refine compaction ( #33982 )
...
issue : https://github.com/milvus-io/milvus/issues/32939
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-25 10:08:03 +08:00
622be36ad7
fix: fix l0 compaction may miss some sealed segments ( #33980 )
...
issue: https://github.com/milvus-io/milvus/issues/33955
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-21 16:52:15 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
feeb869ff9
enhance: Remove compaction plans on the datanode ( #33548 )
...
issue: #33546
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yihao.dai
congqixia
XuanYang-cn
cai.zhang
Zhen Ye
jaime
tinswzy
wayblink
aoiasd
zhenshan.cao
Bingyi Sun