15d60c60c2
enhance: [2.5] remain token after authorize ( #39455 )
...
relate: https://github.com/milvus-io/milvus/issues/39285
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-20 20:27:04 +08:00
f62eca09a0
enhance: set materializedView.enabled default to true ( #39450 )
...
pr: #39449
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 19:55:04 +08:00
3893f24323
fix: apply custom producer config for kafkaHealthCheck ( #39364 )
...
issue: https://github.com/milvus-io/milvus/issues/39287
pr: https://github.com/milvus-io/milvus/pull/39283
Signed-off-by: DLT1412 <tuduc93@gmail.com>
2025-01-20 17:27:04 +08:00
bb32bda423
enhance: [2.5] Sync multipleChunkEnabled default value & milvus yaml ( #39373 )
...
Cherry-pick from master
pr: #39372
The default value and yaml have different values which may cause
confusion when upgrading from older version.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-17 12:07:02 +08:00
e7520599df
fix: [2.5] remove param num_build_thread ( #39341 )
...
issue: #39090
pr: #39340
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-01-16 18:05:01 +08:00
6773fb10a8
enhance: [2.5] Read metadata concurrently to accelerate recovery ( #38900 )
...
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38403
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:53:01 +08:00
9d2a0e775c
fix: [2.5] Fix slow dist handle and slow observe ( #38905 )
...
1. Provide partition&channel level indexing in the collection target.
2. Make SegmentAction not wait for distribution.
3. Remove scheduler and target manager mutex
4. Optimize logging to reduce CPU overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38566
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:07:02 +08:00
e68d161e60
enhance: Bump milvus & proto version to v2.5.4 ( #39330 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 15:39:03 +08:00
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers ( #38915 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38455
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 15:19:03 +08:00
4cb47b3f67
feat: [2.5] root privileges can be customized ( #39324 )
...
- issue: #39184
- pr: #39191
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-16 14:27:11 +08:00
1602390734
enhance: support null expr( #38772 ) ( #38931 )
...
pr: #38772
issue: #31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-16 10:55:07 +08:00
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf ( #39206 ) ( #39255 )
...
Cherry-pick from master
pr: #39206 #39308
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 00:33:03 +08:00
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode ( #38913 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38129
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 13:59:00 +08:00
74246d8167
fix: wrong streaming mockery package name ( #39261 )
...
issue: #39095
pr: #39260
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-15 11:44:59 +08:00
4270174899
fix: [2.5] Add scalar index engine version for compatibility ( #39236 )
...
issue: #39203
master pr: #39204
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-14 21:01:01 +08:00
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex ( #38907 )
...
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38599
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-14 20:43:06 +08:00
4a47f760b3
fix: [2.5] rbac custom group privilege level check ( #39164 ) ( #39224 )
...
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-14 16:50:59 +08:00
99a8274326
enhance: Add primary field name in SearchResult and QueryResults ( #39222 )
...
pr: https://github.com/milvus-io/milvus/pull/39220
issue: https://github.com/milvus-io/milvus/issues/39219
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-01-14 16:10:58 +08:00
54036bcafd
enhance: add broadcast operation for msgstream ( #39119 )
...
issue: #38399
pr: #39040
- make broadcast service available for msgstream by reusing the
architecture streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:11:00 +08:00
7c8db6f61f
fix: [2.5] use the object heap to keep the min ddl ts order ( #39195 )
...
- issue: #39002
- pr: #39118
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-14 14:23:05 +08:00
adfc3f945e
enhance: record memory size (uncompressed) item for index ( #38844 )
...
issue: #38715
pr: #38770
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
ac730be5aa
enhance: Bump milvus & proto version to v2.5.3 ( #39198 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-13 12:56:57 +08:00
39fa00fc1a
enhance: add intent ctx to rootcoord init ( #38948 )
...
issue: #38439
cherry pick "enhance: add intent ctx to rootcoord init"
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-01-11 23:54:57 +08:00
6b127d4f5f
fix: [2.5] Return io error other than `NotExist` refreshing config ( #38924 ) ( #39072 )
...
Cherry pick from master
pr: #38924
Related to #38923
This PR:
- Check whether `os.Stat` config file error is io.ErrNotExist
- Panic when get config return error during Milvus initialization
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-10 14:09:04 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
0693634f62
enhance: add db name in replica description ( #38673 )
...
issue: #36621
pr: #38672
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:43:04 +08:00
f2e581e2d7
enhance: add broadcast for streaming service ( #39047 )
...
issue: #38399
pr: #39020
- Add new rpc for transfer broadcast to streaming coord
- Add broadcast service at streaming coord to make broadcast message
sent automicly
also cherry pick the pr #38400
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-09 16:46:56 +08:00
6f1febe881
enhance: move streaming coord from datacoord to rootcoord ( #39009 )
...
issue: #38399
pr: #39007
We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-07 17:56:56 +08:00
b457c2f415
enhance: [2.5]Add missing delete metrics ( #38634 ) ( #38747 )
...
Add 2 counter metrics:
- Total delete entries from deltalog:
milvus_datanode_compaction_delete_count
- Total missing deletes: milvus_datanode_compaction_missing_delete_count
See also: #34665
pr: #38634
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-07 11:20:56 +08:00
6433f76c61
enhance: [2.5] Bump golang/x/net to v0.33 fixing security alarm ( #38850 ) ( #38932 )
...
Cherry-pick from master
pr: #38850
Related to https://github.com/milvus-io/milvus/security/dependabot/106
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-03 14:50:54 +08:00
5fb8313d6f
enhance: excessive disk usage logs generated if the path does not exist ( #38822 )
...
issue: #38820
pr: #38821
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-02 14:28:53 +08:00
1d55ad6a1d
enhance: [2.5] add param for tuning max VARCHAR length and restore limit to 65535 ( #38883 )
...
issue: #38882
pr: https://github.com/milvus-io/milvus/pull/38884
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit a64a7377ea8258efe36c74bd1cf9db556726ba2d)
2025-01-02 13:56:54 +08:00
a2c4cd59ce
fix: drop partition can not be successful if load failed[2.5] ( #38874 )
...
fix https://github.com/milvus-io/milvus/issues/38649
pr: #38793
when partition load failed, the partition drop will also fail due to the
wrong error message
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-01-02 09:56:53 +08:00
84f8737ee9
enhance: Bump milvus version to v2.5.2 ( #38880 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-31 15:15:59 +08:00
fa7028177c
enhance: [2.5] update expr version to support method parameter type conversion ( #38782 )
...
/kind improvement
issue: #38780
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-30 15:59:06 +08:00
a67b57275c
enhance: Bump milvus & proto version to v2.5.1 ( #38774 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-26 16:36:50 +08:00
a4901ef7ec
fix: [2.5] fix privilege group list and list collections ( #38738 )
...
cherry-pick from: https://github.com/milvus-io/milvus/pull/38684
related: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:06:50 +08:00
52de43dbeb
enhance: add multiply factor when loading index ( #38722 )
...
issue: #38715
pr: #38716
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:50:58 +08:00
11bedf5e76
fix: Revert "Expose metrics of stanby coordinators ( #27698 )" ( #38621 )
...
issue: #38608
pr: #38620
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-20 18:04:47 +08:00
b132438c03
enhance: Bump milvus & proto verison to v2.5.0 ( #38610 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-20 14:30:46 +08:00
306e5e6898
enhance: clean compaction task in compactionHandler ( #38170 )
...
issue: #35711
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-12-19 12:36:47 +08:00
78438ef41e
fix: revert optimize CPU usage for CheckHealth requests ( #35589 ) ( #38555 )
...
issue: #35563
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-19 00:38:45 +08:00
1639779b5d
fix: [skip e2e]metrics josn tag unmarshall failed ( #38554 )
...
issue: #38551
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-18 18:02:46 +08:00
b18a3cfce2
fix: kafka use shared channel to receive produce result ( #38532 )
...
issue: #38531
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 23:10:45 +08:00
4d5572d7ed
enhance: [skip e2e] refine display format ( #38537 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 22:30:44 +08:00
29e620fa6d
fix: sync task still running after DataNode has stopped ( #38377 )
...
issue: #38319
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 18:06:44 +08:00
b9d8c5c1d7
fix: privilege group privileges granted from ClusterAdmin permission denied ( #38514 )
...
related: https://github.com/milvus-io/milvus/issues/38460
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-17 17:34:44 +08:00
afac153c26
enhance: move the lifetime implementation out of server level lifetime ( #38442 )
...
issue: #38399
- move the lifetime implementation of common code out of the server
level lifetime implementation
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 11:42:44 +08:00
28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests ( #35589 )
...
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 11:02:45 +08:00
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
aoiasd
Gao
Duc Le Tu
congqixia
foxspy
yihao.dai
SimFG
smellthemoon
Zhen Ye
cai.zhang
sthuang
zhenshan.cao
tinswzy
jaime
XuanYang-cn
Patrick Weizhi Xu
Xiaofan