fd84ed817c
enhance: add broadcast operation for msgstream ( #39040 )
...
issue: #38399
- make broadcast service available for msgstream by reusing the
architecture streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:14:59 +08:00
3e788f0fbd
enhance: record memory size (uncompressed) item for index ( #38770 )
...
issue: #38715
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field `MemSize` (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
b3daae8e40
fix: Make KeyLock obj pool size unlimited ( #39182 )
...
Related to #39181
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-13 18:04:58 +08:00
5c5948cb70
fix: rbac custom group privilege level check ( #39164 )
...
related: https://github.com/milvus-io/milvus/issues/39086
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-13 11:24:59 +08:00
032292a432
feat: support phrase match query ( #38869 )
...
The relevant issue: https://github.com/milvus-io/milvus/issues/38930
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-12 20:24:58 +08:00
86d665a50f
enhance: keylock object background recycling ( #38805 )
...
issue: #38587
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-10 18:20:59 +08:00
357eaf0d71
fix: use the object heap to keep the min ddl ts order ( #39118 )
...
issue: #39002
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-10 18:16:58 +08:00
4355b485e5
enhance: remove compaction parallelism control ( #39081 )
...
See #39080
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-10 13:23:00 +08:00
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
22b8f6a58e
enhance: excessive disk usage logs generated if the path does not exist ( #38821 )
...
issue: #38820
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:43:04 +08:00
f03a85725a
enhance: add db name in replica ( #38672 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:40:59 +08:00
3bcdd92915
enhance: add broadcast for streaming service ( #39020 )
...
issue: #38399
- Add new rpc for transfer broadcast to streaming coord
- Add broadcast service at streaming coord to make broadcast message
sent automicly
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-09 16:24:55 +08:00
a208170c1f
fix: Make runtime param accessor concurrent safe ( #39050 )
...
Related to #39049
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-08 17:08:57 +08:00
f076898761
fix: Return io error other than `NotExist` refreshing config ( #38924 )
...
Related to #38923
This PR:
- Check whether `os.Stat` config file error is io.ErrNotExist
- Panic when get config return error during Milvus initialization
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-08 12:00:56 +08:00
c5a7000a92
enhance: move streaming coord from datacoord to rootcoord ( #39007 )
...
issue: #38399
We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-07 17:42:57 +08:00
4245c5bed1
fix: text match panics when enable_match is set be false ( #38950 )
...
fix: https://github.com/milvus-io/milvus/issues/38949
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-03 14:20:55 +08:00
d3a5282eaa
enhance: add param for tuning max VARCHAR length and restore limit to 65535 ( #38884 )
...
issue: #38882
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 8e740e004151dd2c11918aad2857c1c8d1bd98f5)
2025-01-02 14:42:53 +08:00
19a5c31907
enhance: Bump golang/x/net to v0.33 fixing security alarm ( #38850 )
...
Related to https://github.com/milvus-io/milvus/security/dependabot/106
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-02 10:50:55 +08:00
11f8f4a378
enhance: add intent ctx to rootcoord init ( #38439 )
...
issue: #35917
add intent ctx to rootcoord initialization process
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
cb6eca8e91
fix: drop partition can not be successful if load failed ( #38793 )
...
fix #38649
when partition load failed, the partition drop will also fail due to the
wrong error message
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-12-30 19:42:52 +08:00
56c5b66619
fix: the close operation of rmq consumer is not sync ( #38734 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 16:04:51 +08:00
ecc820e030
enhance: update expr version to support method parameter type conversion ( #38780 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-30 15:59:06 +08:00
6bc799061e
fix: fix privilege group list and list collections ( #38684 )
...
related: #37031
* built-in privilege group privileges in listPrivilegeGroups() should be
the same as in milvus.yaml
* collections granted by collection level built-in privilege group
should be list in showCollections()
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:00:51 +08:00
5395ec19ad
enhance: add mem size for index file and fallback to multiply with serialized size ( #38716 )
...
issue: #38715
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 14:46:49 +08:00
c731357538
enhance: Add missing delete metrics ( #38634 )
...
Add 2 counter metrics:
- Total delete entries from deltalog:
milvus_datanode_compaction_delete_count
- Total missing deletes: milvus_datanode_compaction_missing_delete_count
See also: #34665
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-25 10:24:50 +08:00
285289d5b0
enhance: implement kafka for wal ( #38598 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00
118678bd91
enhance: make runtime config into a global environment table ( #38671 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-24 21:00:49 +08:00
69a9fd6ead
enhance: enable rmq for streaming ( #38669 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-24 20:24:48 +08:00
5afd0c0a2b
fix: Revert "Expose metrics of stanby coordinators ( #27698 )" ( #38620 )
...
issue: #38608
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-23 11:46:57 +08:00
984a605d47
enhance: use lazy initializing client for streaming node ( #38400 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-23 11:36:47 +08:00
306e5e6898
enhance: clean compaction task in compactionHandler ( #38170 )
...
issue: #35711
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-12-19 12:36:47 +08:00
78438ef41e
fix: revert optimize CPU usage for CheckHealth requests ( #35589 ) ( #38555 )
...
issue: #35563
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-19 00:38:45 +08:00
1639779b5d
fix: [skip e2e]metrics josn tag unmarshall failed ( #38554 )
...
issue: #38551
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-18 18:02:46 +08:00
b18a3cfce2
fix: kafka use shared channel to receive produce result ( #38532 )
...
issue: #38531
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 23:10:45 +08:00
4d5572d7ed
enhance: [skip e2e] refine display format ( #38537 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 22:30:44 +08:00
29e620fa6d
fix: sync task still running after DataNode has stopped ( #38377 )
...
issue: #38319
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 18:06:44 +08:00
b9d8c5c1d7
fix: privilege group privileges granted from ClusterAdmin permission denied ( #38514 )
...
related: https://github.com/milvus-io/milvus/issues/38460
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-17 17:34:44 +08:00
afac153c26
enhance: move the lifetime implementation out of server level lifetime ( #38442 )
...
issue: #38399
- move the lifetime implementation of common code out of the server
level lifetime implementation
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 11:42:44 +08:00
28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests ( #35589 )
...
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 11:02:45 +08:00
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically ( #38178 )
...
- issue: #38177
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
fe79babdb3
enhance: Bump golang/x/crypto for all packages ( #38445 )
...
Related to #38446
See also https://github.com/milvus-io/milvus/security/dependabot/103
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-13 19:12:42 +08:00
833c74aa66
enhance: add detail, replica count for resource group ( #38314 )
...
issue: #30647
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-13 14:14:50 +08:00
c2855a5c74
enhance: add privilege group privilege into built-in privilege group ( #38393 )
...
related issue: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-12 17:20:42 +08:00
d0a8110a7a
enhance: alterdatabase support delete property ( #38380 )
...
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 17:04:43 +08:00
75e64b993f
enhance: add metrics for counting number of nun-zeros/tokens of sparse/FTS search ( #38329 )
...
sparse vectors may have arbitrary number of non zeros and it is hard to
optimize without knowing the actual distribution of nnz. this PR adds a
metric for analyzing that.
issue: https://github.com/milvus-io/milvus/issues/35853
comparing with https://github.com/milvus-io/milvus/pull/38328 , this
includes also metric for FTS in query node delegator
also fixed a bug of sparse when searching by pk
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-12-12 16:22:43 +08:00
a514f839b2
enhance: Speed up meta recovery ( #38285 )
...
Increase the batchSize in WalkWithPrefix operations to 10000.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-12 10:58:43 +08:00
8977454311
enhance: support recall estimation ( #38017 )
...
issue: #37899
Only `search` api will be supported
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 20:40:48 +08:00
dc85d8e968
enhance: improve mix compaction performance by removing max segment limitations ( #38344 )
...
See #37234
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-11 20:38:42 +08:00
Zhen Ye
congqixia
sthuang
Spade A
SimFG
Ted Xu
jaime
Patrick Weizhi Xu
tinswzy
Xiaofan
XuanYang-cn
cai.zhang
Xianhui Lin
Buqian Zheng
yihao.dai
Gao