817b616eb4
fix: [2.5]Restore the compacting state for stats task during recovery ( #39460 )
...
issue: #39333
master pr: #39459
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-21 01:03:05 +08:00
15d60c60c2
enhance: [2.5] remain token after authorize ( #39455 )
...
relate: https://github.com/milvus-io/milvus/issues/39285
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-20 20:27:04 +08:00
e4c6ef9d57
enhance: [2.5] serializes the resp body and attach some details ( #39401 )
...
issue: #37611
pr: #39140
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-20 18:17:10 +08:00
ff53d4f742
enhance: [2.5]add some apis in Restful ( #39425 )
...
pr: #39105
issue: #38709
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-20 18:15:22 +08:00
8e2e828a78
enhance: [2.5] Skip loading pk index for sorted segment in loader ( #39438 )
...
Cherry pick from master
pr: #39437
Related to #39339
Previous PR #39389 only skips append index into segment
Also related to #39428
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-20 18:07:04 +08:00
964000f645
fix: deleted the sealed segment data accidentally ( #39422 )
...
issue:https://github.com/milvus-io/milvus/issues/39333
pr: https://github.com/milvus-io/milvus/pull/39421
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-01-20 17:49:03 +08:00
6342237ca8
enhance: add workaround for concurrent tantivy reload and directory removal ( #39441 )
...
related pr: https://github.com/milvus-io/milvus/pull/39440
issue: https://github.com/milvus-io/milvus/issues/39428
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 17:33:05 +08:00
6f9f840080
fix: Fix panic caused by removing directory ( #38622 ) ( #39434 )
...
related pr: https://github.com/milvus-io/milvus/pull/38622
issue: https://github.com/milvus-io/milvus/issues/39428
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 16:13:03 +08:00
8c35064d01
enhance: [2.5] clean up logs of empty op key ( #39394 )
...
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39335
related: https://github.com/milvus-io/milvus/issues/39334
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-20 12:15:10 +08:00
4464966462
enhance: [2.5] Remove frequent observe log ( #39414 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/39413
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-20 11:01:10 +08:00
6d5f174142
enhance: [2.5] add recalls response for restful api ( #39409 )
...
issue: #37899
pr: #39408
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 10:59:10 +08:00
efe76905c5
fix: correct remote centroids path in clustering compaction ( #39399 )
...
issue: https://github.com/milvus-io/milvus/issues/39353
pr: #39398
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 10:57:21 +08:00
c9b0859b16
fix: [cp25]Record active collections for l0Policy ( #39217 ) ( #39383 )
...
By recording the active collection lists, The l0 compaction trigger of
view change and idle won't influence each other.
Also this pr replaces the L0View cache with real L0 segments' change.
Save some memory and make L0 compaction triggers more accurate.
See also: #39187
pr: #39217
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-20 10:47:03 +08:00
57bcb1bfc0
fix: sparse: add inverted_index_algo and dim_max_score_ratio config ( #39362 )
...
issue: #39332
pr: #39358
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
2025-01-18 12:17:03 +08:00
0c00f94965
fix: [2.5] Skip load extra indexes for sorted segment pk field ( #39390 )
...
Cherry-pick from master
pr: #39389
Related to #39339
Extra indexes can be ignored for most cases since sorted pk column
already provided indexing features
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-17 14:53:03 +08:00
89a183c7c2
enhance: [2.5] enable task delta cache ( #39349 )
...
When there are many segment tasks in the querycoord scheduler, the
traversal in GetSegmentTaskDelta checks becomes time-consuming. This PR
adds caching for segment deltas.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/39307
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
2025-01-17 12:01:03 +08:00
5330fc981c
fix: panic when streaming release if using msgstream ( #39375 )
...
issue: #39367
pr: #39374
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-17 11:55:03 +08:00
56cb1683eb
fix: Fix performance issue and use after free bug ( #39343 )
...
cherry pick some fixes in https://github.com/milvus-io/milvus/pull/39249
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-17 11:51:03 +08:00
47e410feb1
fix: [2.5] fix assignment operator in AssertInfo to comparison operator ( #39351 )
...
cp https://github.com/milvus-io/milvus/pull/39347
issue: #39346
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-17 11:27:02 +08:00
0391b8c7d3
fix: update rg failed when alter database ( #39356 )
...
issue: #36933
pr: #39355
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-17 10:37:03 +08:00
b69994272f
enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails ( #39359 )
...
1. Limit the maximum number of restored segments to 1024.
2. Fail the import job if saving binlog fails.
3. Fail the import job if saving the import task fails to prevent
repeatedly generating dirty importing segments.
issue: https://github.com/milvus-io/milvus/issues/39331
pr: https://github.com/milvus-io/milvus/pull/39344
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-17 10:27:04 +08:00
6773fb10a8
enhance: [2.5] Read metadata concurrently to accelerate recovery ( #38900 )
...
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38403
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:53:01 +08:00
9d2a0e775c
fix: [2.5] Fix slow dist handle and slow observe ( #38905 )
...
1. Provide partition&channel level indexing in the collection target.
2. Make SegmentAction not wait for distribution.
3. Remove scheduler and target manager mutex
4. Optimize logging to reduce CPU overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38566
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:07:02 +08:00
448eb3b63a
fix: [2.5]bulkimport binlogs skip diskquota check ( #39319 )
...
bulkimport binlogs skip diskquota check
issue:https://github.com/milvus-io/milvus/issues/38845
pr: https://github.com/milvus-io/milvus/pull/39315
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-16 15:33:05 +08:00
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers ( #38915 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38455
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 15:19:03 +08:00
4cb47b3f67
feat: [2.5] root privileges can be customized ( #39324 )
...
- issue: #39184
- pr: #39191
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-16 14:27:11 +08:00
c945efa853
enhance: [2.5] Remove mutex from SegmentManger in querynode ( #39051 ) ( #39282 )
...
Remove mutex from SegmentManger in querynode to prevent mutex
contention.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/39051
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 13:27:02 +08:00
c741b8be2b
fix: [2.5] Remove frequently updating metric to avoid mutex contention ( #38778 )
...
issue: https://github.com/milvus-io/milvus/issues/37630
Reduce the frequency of `updateIndexTasksMetrics` to avoid holding the
mutex for long periods.
pr: https://github.com/milvus-io/milvus/pull/38775
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 11:51:02 +08:00
1602390734
enhance: support null expr( #38772 ) ( #38931 )
...
pr: #38772
issue: #31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-16 10:55:07 +08:00
76ed552b00
enhance: Add logs for check health failed ( #39208 ) ( #39302 )
...
pr: #39208
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-16 10:31:04 +08:00
21df11b37c
enhance: [2.5] Add try-catch and return CStatus for NewCollection ( #39279 ) ( #39303 )
...
Cherry pick from master
pr: #39279
Related to #28795
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 10:13:06 +08:00
1d9788e4bc
fix: [2.5] enable to build index with single segment ( #39272 )
...
issue: #39232
cherry-pick https://github.com/milvus-io/milvus/pull/39233
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 09:59:02 +08:00
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf ( #39206 ) ( #39255 )
...
Cherry-pick from master
pr: #39206 #39308
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 00:33:03 +08:00
9b916f2cf8
enhance: [2.5] Skip creating partition rate limiters when not enable ( #38911 )
...
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38816
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 22:19:02 +08:00
dd44a58381
enhance: [2.5] pass partition key scalar info if enable for vector mem index ( #39245 )
...
issue: #34332
pr: #39123
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-15 21:45:01 +08:00
2fe245f918
fix: [2.5] Add index param duplication check ( #39289 ) ( #39304 )
...
Cherry-pick from master
pr: #39289
Related to #39288
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-15 21:04:06 +08:00
8bf0f0b921
fix: [2.5] Fix rootcoord meta mutex contention ( #38902 )
...
RootCoord meta uses copy-on-write, allowing the removal of unnecessary
copies.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38799
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:11:08 +08:00
2439278f93
enhance: add some apis in Restful( #38733 ) ( #38875 )
...
issue: #38709
pr: #38733 #38979
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-15 15:19:00 +08:00
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode ( #38913 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38129
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 13:59:00 +08:00
74246d8167
fix: wrong streaming mockery package name ( #39261 )
...
issue: #39095
pr: #39260
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-15 11:44:59 +08:00
4c91e05a5d
enhance: fix inconsistenty of alias and db for query iterator( #39045 ) ( #39248 )
...
related: #39045
pr: https://github.com/milvus-io/milvus/pull/39216
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-01-15 10:36:59 +08:00
6816ee4cf5
fix: [2.5] Record a map to avoid repeatedly traversing the CompactionFrom ( #38926 )
...
issue: #38811
master pr: #38925
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-15 10:31:00 +08:00
2e4a1052aa
enhance: [2.5] Reduce mutex contention in datacoord meta ( #38904 )
...
1. Using secondary index to avoid retrieving all segments at
GetSegmentsChanPart.
2. Perform batch SetAllocations to reduce the number of times the meta
lock is acquired.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38219
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 00:55:00 +08:00
4270174899
fix: [2.5] Add scalar index engine version for compatibility ( #39236 )
...
issue: #39203
master pr: #39204
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-14 21:01:01 +08:00
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex ( #38907 )
...
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38599
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-14 20:43:06 +08:00
51994158d9
fix: channel unbalance during stopping balance progress ( #38971 ) ( #39200 )
...
issue: #38970
pr: #38971
cause the stopping balance channel still use the row_count_based policy,
which may causes channel unbalance in multi-collection case.
This PR impl a score based stopping balance channel policy.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-14 18:25:00 +08:00
4fd56e4773
fix: Prevent leader checker from generating excessive duplicate leader tasks ( #39000 ) ( #39160 )
...
issue: #39001
pr: #39000
Background:
Segment Load Version: Each segment load request assigns a timestamp as
its version. When multiple copies of a segment are loaded on different
QueryNodes, the leader checker uses this version to identify the latest
copy and updates the routing table in the leader view to point to it.
Delegator Router Version: When a delegator builds a route to a QueryNode
that has loaded a segment, it also records the segment's version.
Router Table Update Logic: If the leader checker detects that the
version of a segment in the routing table does not match the version in
the worker, it updates the routing table to point to the QueryNode with
the latest version. Additionally, it updates the segment's load version
in the QueryNode during this process.
Issue:
When a channel is undergoing load balancing, the leader checker may sync
the routing table to a new delegator. This sync operation modifies the
segment's load version, which invalidates the routing in the old
delegator. Subsequently, the leader checker updates the routing table in
the old delegator, breaking the routing in the new delegator. This cycle
continues, causing repeated updates and inconsistencies.
Fix:
This PR introduces two changes to address the issue:
1. Use NodeID to verify whether the delegator's routing table needs an
update, avoiding unnecessary modifications.
2. Ensure compatibility by using the latest segment's load version as
the version recorded in the routing table.
These changes resolve the cyclic updates and prevent the leader checker
from generating excessive duplicate tasks, ensuring routing stability
across delegators during load balancing.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-14 18:11:06 +08:00
b4ddf746c1
enhance: [2.5][bitset] extend op_find() to be able to search both 0 and 1 ( #39237 )
...
Cherry-pick from master
pr: #39176 #39229
issue: #39124
`bitset::find_first()` and `bitset::find_next()` now accept one more
parameter, which allows to search for `0` bit instead of `1` bit
---------
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Alexander Guzhva <alexanderguzhva@gmail.com>
2025-01-14 17:59:00 +08:00
e6ac2fe063
enhance: [Cherry-pick] report auth log to hook when grpc authenticate failed ( #39192 )
...
relate: https://github.com/milvus-io/milvus/pull/38943
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-14 17:47:04 +08:00
4a47f760b3
fix: [2.5] rbac custom group privilege level check ( #39164 ) ( #39224 )
...
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-14 16:50:59 +08:00
cai.zhang
aoiasd
smellthemoon
congqixia
zhenshan.cao
Bingyi Sun
sthuang
yihao.dai
Gao
XuanYang-cn
sparknack
Zhen Ye
Spade A
Xianhui Lin
SimFG
wei liu
Chun Han