a41de38f21
enhance: remain token after authorize ( #39453 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-20 20:27:23 +08:00
140c5a0a75
enhance: add unit test for string pk ( #39329 )
...
https://github.com/milvus-io/milvus/issues/39107
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 19:03:04 +08:00
fb5761af8d
enhance: Skip loading pk index for sorted segment in loader ( #39437 )
...
Related to #39339
Previous PR #39389 only skips append index into segment
Also related to #39428
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-20 18:17:10 +08:00
513b489c85
enhance: add some apis in Restful ( #39105 )
...
- drop/alter database properties
- simplify the structure of search_params
- flush
- compact
- get_compact_status
issue: #38709
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-20 18:15:22 +08:00
45d49df89b
fix: Skip load extra indexes for sorted segment pk field ( #39389 )
...
Related to #39339
Extra indexes can be ignored for most cases since sorted pk column
already provided indexing features
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-20 18:13:15 +08:00
38b96090a8
fix: deleted the sealed segment data accidentally ( #39421 )
...
issue: https://github.com/milvus-io/milvus/issues/39333
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-01-20 15:29:02 +08:00
cb959cd1f9
enhance: upgrade rust version to 1.83 ( #39295 )
...
#39254
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 11:15:03 +08:00
e55d6506e3
enhance: Remove frequent observe log ( #39413 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-20 11:01:10 +08:00
1a680c29e2
fix: correct remote centroids path in clustering compaction ( #39398 )
...
issue: https://github.com/milvus-io/milvus/issues/39353
The path was modified unintentionally, change it back.
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 10:59:10 +08:00
9c4a81c80c
enhance: add recalls response for restful api ( #39408 )
...
issue: #37899
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 10:55:04 +08:00
fdb968d0ea
[automated] Update Knowhere Commit ( #39420 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-01-20 01:17:02 +08:00
f54e8e9554
fix: Folders are also considered for WalkWithObjects to adhere bulk insert failure on using GCS buckets ( #39352 )
...
Related task: [#39134
](https://github.com/milvus-io/milvus/issues/39134 )
Previous PR: [#39150 ](https://github.com/milvus-io/milvus/pull/39150 )
2025-01-19 02:49:03 +08:00
5b35fc700d
enhance: [skip-e2e] Use template to remove duplicate unittest ( #39396 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-18 10:33:01 +08:00
7cac87caca
fix: Skip erase field if index build on PK field ( #39370 )
...
Related to #39339
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-17 20:31:02 +08:00
c22e457c59
feat: root privileges can be customized ( #39191 )
...
- issue: #39184
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-17 16:25:04 +08:00
64feeb0e2b
enhance: Rename API GenDataset to GenFieldData in unittest ( #39386 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-17 15:55:03 +08:00
3ca6c0127d
fix: update rg failed when alter database ( #39355 )
...
issue: #36933
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-17 14:27:03 +08:00
92bde5b4f6
fix: panic when streaming release if using msgstream ( #39374 )
...
issue: #39367
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-17 11:47:02 +08:00
c38bca8d80
enhance: serializes the resp body and attach some details ( #39140 )
...
#37611
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-17 11:21:02 +08:00
9209a70bb6
fix: clang format broken under osx ( #38427 )
...
See: #38434
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-17 10:43:03 +08:00
3c9f26942d
enhance: clean up logs of empty op key ( #39335 )
...
related: https://github.com/milvus-io/milvus/issues/39334
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-17 10:29:03 +08:00
4c76f0403f
fix: bulkimport binlogs skip diskquota check ( #39315 )
...
bulkimport binlogs skip diskquota check
issue:https://github.com/milvus-io/milvus/issues/38845
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-17 10:03:03 +08:00
0461ddf776
fix: phrase match does not support offset input ( #39338 )
...
fix : #39337
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 22:05:01 +08:00
74b4369c5b
fix: Record active collections for l0Policy ( #39217 )
...
By recording the active collection lists, The l0 compaction trigger
of view change and idle won't influence each other.
Also this pr replaces the L0View cache with real L0 segments' change.
Save some memory and make L0 compaction triggers more accurate.
See also: #39187
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-16 20:19:02 +08:00
75d7978a18
enhance: pass partition key scalar info if enable for vector mem index ( #39123 )
...
issue: #34332
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-16 14:33:03 +08:00
1f6fd54146
fix: [skip e2e] Make test case assigner result deterministic ( #39317 )
...
Related to #39296
The case initialized with {100:8 ,101: 16}. After first assignment, the
slots become {100:8, 101:8} and the following result is not stable.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 14:25:02 +08:00
8c4ba70a4c
fix: enable to build index with single segment ( #39233 )
...
fix https://github.com/milvus-io/milvus/issues/39232
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 11:01:06 +08:00
bca2a62b78
enhance: Handle PutOrRef collection schema failure error ( #39310 )
...
Related to previous pr #39279
When NewCollection returns nil, the error shall be returned and handled
by caller
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 10:13:06 +08:00
57e5652f1a
enhance: Log error instead of panicking if load lock wait timeout ( #39308 )
...
Related to #39205
Previous PR #39206
This PR change wait timeout behavior to log error and return to avoid
making other collection read failure in only some collections have
deadlock
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 02:31:02 +08:00
a5a83a0904
fix: Fix consume blocked due to too many consumers ( #38455 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 21:37:01 +08:00
657550cf06
fix: Fix slow dist handle and slow observe ( #38566 )
...
1. Provide partition&channel level indexing in the collection target.
2. Make `SegmentAction` not wait for distribution.
3. Remove scheduler and target manager mutex.
4. Optimize logging to reduce CPU overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:17:00 +08:00
38881bf591
enhance: prevent multiple query nodes from causing excessive occupancy of a single node, leading to GPU memory overflow ( #39276 ) ( #38617 )
...
issue: #39276
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2025-01-15 20:15:01 +08:00
0df2c75b77
fix: Fix rootcoord meta mutex contention ( #38799 )
...
RootCoord meta uses copy-on-write, allowing the removal of unnecessary
copies.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:11:08 +08:00
82bdf9a6a8
fix: Add index param duplication check ( #39289 )
...
Related to #39288
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-15 19:23:01 +08:00
eb63334312
enhance: Add try-catch and return CStatus for NewCollection ( #39279 )
...
Related to #28795
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-15 19:17:01 +08:00
d2834a1812
enhance: Add logs for check health failed ( #39208 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-15 17:31:00 +08:00
7addeea70c
fix: wrong streaming mockery package name ( #39260 )
...
issue: #39095
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-15 15:51:00 +08:00
ce41778fe6
enhance: Optimize GetLocalDiskSize and segment loader mutex ( #38599 )
...
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 15:45:01 +08:00
e501025bba
enhance: simplify compaction tasks to reduce their memory overhead ( #39121 )
...
See #39080
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-15 14:51:00 +08:00
f5234c3c11
enhance: Remove mutex from SegmentManger in querynode ( #39051 )
...
Remove mutex from SegmentManger in querynode to prevent mutex
contention.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 12:45:00 +08:00
e8f76cd2d9
fix: unstable ut in leader_vew_manager.go file ( #39161 )
...
issue: #38672
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-15 12:26:59 +08:00
6d45dd5666
fix: Add scalar index engine version for compatibility ( #39204 )
...
issue: #39203
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-15 12:25:00 +08:00
3a6408b237
fix: Record a map to avoid repeatedly traversing the CompactionFrom ( #38925 )
...
issue: #38811
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-15 10:02:58 +08:00
ed31a5a4bf
enhance: fix inconsistenty of alias and db for query iterator( #39045 ) ( #39216 )
...
related: #39045
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-01-15 09:48:59 +08:00
55dcac375c
[automated] Update Knowhere Commit ( #39263 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-01-15 02:52:59 +08:00
ec2e77b5d7
enhance: Reduce memory usage of BF in DataNode and QueryNode ( #38129 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 01:59:01 +08:00
272d95ad79
enhance: Reduce mutex contention in datacoord meta ( #38219 )
...
1. Using secondary index to avoid retrieving all segments at
`GetSegmentsChanPart`.
2. Perform batch SetAllocations to reduce the number of times the meta
lock is acquired.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 01:15:02 +08:00
5bf1b2b929
feat: Support Int8Vector in go ( #38990 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-14 20:43:06 +08:00
d89768f9e0
enhance: Unify LoadStateLock RLock & PinIf ( #39206 )
...
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-14 18:38:59 +08:00
da07993082
enhance: report auth log to hook when grpc authenticate failed ( #38943 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-14 17:47:04 +08:00
aoiasd
Bingyi Sun
congqixia
smellthemoon
zhenshan.cao
yihao.dai
Gao
sre-ci-robot
Gifi Siby
Cai Yudong
SimFG
Zhen Ye
Ted Xu
sthuang
Xianhui Lin
Spade A
XuanYang-cn
presburger
wei liu
jaime
cai.zhang
Chun Han