cae8f75e47
enhance: Bump milvus version to 2.5.5 ( #40017 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-19 18:48:54 +08:00
3c2d8c1419
enhance: [2.5] Add management api to check querycoord balance status ( #37784 ) ( #39909 )
...
issue: #37783
pr: #37784
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-02-19 10:56:49 +08:00
04175d8549
enhance: [2.5]compress jsonstatslog path only save filename ( #39979 )
...
enhance: compress jsonstatslog path only save filename
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-02-19 10:20:52 +08:00
7a92121c92
fix: [2.5] Fix failed to seek to earliest position ( #39966 )
...
If it is the earliest message ID, skip the seek to prevent failure.
issue: https://github.com/milvus-io/milvus/issues/39964
pr: https://github.com/milvus-io/milvus/pull/39965
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-18 20:36:53 +08:00
de8abe7970
fix:reset default auto index for varchar ( #39908 )
...
pr: #39819
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-17 20:55:26 +08:00
52434ccc78
enhance: [2.5] Limit the speed of the generating stats task ( #39645 )
...
master pr: #39644
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-17 16:06:17 +08:00
8067113133
enhance: [cp25]Enable to observe write amplification ( #39661 ) ( #39743 )
...
pr: #39661
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-17 16:00:17 +08:00
56c1a8d462
fix: ignore growing option is lost at hibridsearch ( #39900 )
...
issue: #39892
pr: #39799
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-17 15:30:15 +08:00
ee25af4c9b
enhance: Add configs for compaction schedule ( #39010 ) ( #39511 )
...
pr: #39010
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-17 11:44:15 +08:00
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization ( #39876 )
...
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
16ea4fcb90
Revert "fix:reset default auto index type for scalar ( #37086 )" ( #39820 )
...
This reverts commit eeb67a3845#39819
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-13 17:36:14 +08:00
a50249e0bb
enhance: [2.5] dump pprof info if component stop progress timeout ( #39760 )
...
issue: #39735
pr: #39726
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-02-13 15:30:47 +08:00
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) ( #39180 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data:
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40 ">
<head>
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
.font5
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;}
tr
{mso-height-source:auto;
mso-ruby-visibility:none;}
col
{mso-width-source:auto;
mso-ruby-visibility:none;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-number-format:General;
text-align:general;
vertical-align:middle;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
ruby
{ruby-align:left;}
rt
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-char-type:none;
display:none;}
-->
</head>
<body link="#0563C1" vlink="#954F72">
sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
| |
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no
</body>
</html>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-02-13 09:56:47 +08:00
667c84740c
enhance: [2.5] Add metrics for parse expression ( #39716 )
...
master pr: #39654
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-12 10:30:46 +08:00
c2986d4ab9
fix: [2.5] Fix concurrent map ( #39776 )
...
issue: https://github.com/milvus-io/milvus/issues/39778
pr: https://github.com/milvus-io/milvus/pull/39775
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-11 22:12:46 +08:00
f542c866b3
fix: [2.5] root check the role list if `rootShouldBindRole` is true ( #39714 )
...
- issue: #39712
- pr: #39713
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-02-11 10:10:51 +08:00
4a8e6fc59c
fix: [2.5] Fix flowgraph leak ( #39686 )
...
Consider lag targets when determining whether to remove the dispatcher
manager.
issue: https://github.com/milvus-io/milvus/issues/39642
pr: https://github.com/milvus-io/milvus/pull/39656
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-08 15:40:45 +08:00
858dc10ef9
enhance: broadcast with event-based notification ( #39550 )
...
issue: #38399
pr: #39522
- broadcast message can carry multi resource key now.
- implement event-based notification for broadcast messages
- broadcast message use broadcast id as a unique identifier in message
- broadcasted message on vchannels keep the broadcasted vchannel now.
- broadcasted message and broadcast message have a common broadcast
header now.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-07 11:50:50 +08:00
ddc5b299ad
enhance: expose more metrics data ( #39466 )
...
issue: #36621 #39417
pr: #39456
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-02-07 11:48:45 +08:00
c32ec451b0
fix: [2.5] Use param item formatter to avoid SetConfig to overlay ( #39597 ) ( #39636 )
...
Cherry-pick from master
pr: #39597
Related to #39596
When updating the build param configuration, the `Formatter` could be
used to do so and completed avoid touching the `overlay` config items
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-07 10:02:44 +08:00
30411d6d3a
fix: [2.5] deny to set the mmap param for the alter index api ( #39520 )
...
- issue: #39517
- pr: #39518
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-22 23:55:06 +08:00
20c6de156e
feat: update milvusproto version to 2.5.5 ( #39478 )
...
feat: update milvusproto version to 2.5
issue: https://github.com/milvus-io/milvus/issues/39093
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-21 15:11:05 +08:00
3b3959c625
enhance: [2.5]Add schema update time verification for insert and upsert to use cache ( #39382 )
...
enhance: Add schema update time verification for insert and upsert to
use cache
issue: https://github.com/milvus-io/milvus/issues/39093
Related to pr: https://github.com/milvus-io/milvus/pull/39096
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-21 10:19:11 +08:00
15d60c60c2
enhance: [2.5] remain token after authorize ( #39455 )
...
relate: https://github.com/milvus-io/milvus/issues/39285
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-20 20:27:04 +08:00
f62eca09a0
enhance: set materializedView.enabled default to true ( #39450 )
...
pr: #39449
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 19:55:04 +08:00
3893f24323
fix: apply custom producer config for kafkaHealthCheck ( #39364 )
...
issue: https://github.com/milvus-io/milvus/issues/39287
pr: https://github.com/milvus-io/milvus/pull/39283
Signed-off-by: DLT1412 <tuduc93@gmail.com>
2025-01-20 17:27:04 +08:00
bb32bda423
enhance: [2.5] Sync multipleChunkEnabled default value & milvus yaml ( #39373 )
...
Cherry-pick from master
pr: #39372
The default value and yaml have different values which may cause
confusion when upgrading from older version.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-17 12:07:02 +08:00
e7520599df
fix: [2.5] remove param num_build_thread ( #39341 )
...
issue: #39090
pr: #39340
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-01-16 18:05:01 +08:00
6773fb10a8
enhance: [2.5] Read metadata concurrently to accelerate recovery ( #38900 )
...
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38403
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:53:01 +08:00
9d2a0e775c
fix: [2.5] Fix slow dist handle and slow observe ( #38905 )
...
1. Provide partition&channel level indexing in the collection target.
2. Make SegmentAction not wait for distribution.
3. Remove scheduler and target manager mutex
4. Optimize logging to reduce CPU overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38566
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:07:02 +08:00
e68d161e60
enhance: Bump milvus & proto version to v2.5.4 ( #39330 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 15:39:03 +08:00
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers ( #38915 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38455
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 15:19:03 +08:00
4cb47b3f67
feat: [2.5] root privileges can be customized ( #39324 )
...
- issue: #39184
- pr: #39191
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-16 14:27:11 +08:00
1602390734
enhance: support null expr( #38772 ) ( #38931 )
...
pr: #38772
issue: #31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-16 10:55:07 +08:00
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf ( #39206 ) ( #39255 )
...
Cherry-pick from master
pr: #39206 #39308
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 00:33:03 +08:00
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode ( #38913 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38129
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 13:59:00 +08:00
74246d8167
fix: wrong streaming mockery package name ( #39261 )
...
issue: #39095
pr: #39260
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-15 11:44:59 +08:00
4270174899
fix: [2.5] Add scalar index engine version for compatibility ( #39236 )
...
issue: #39203
master pr: #39204
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-14 21:01:01 +08:00
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex ( #38907 )
...
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38599
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-14 20:43:06 +08:00
4a47f760b3
fix: [2.5] rbac custom group privilege level check ( #39164 ) ( #39224 )
...
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-14 16:50:59 +08:00
99a8274326
enhance: Add primary field name in SearchResult and QueryResults ( #39222 )
...
pr: https://github.com/milvus-io/milvus/pull/39220
issue: https://github.com/milvus-io/milvus/issues/39219
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-01-14 16:10:58 +08:00
54036bcafd
enhance: add broadcast operation for msgstream ( #39119 )
...
issue: #38399
pr: #39040
- make broadcast service available for msgstream by reusing the
architecture streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:11:00 +08:00
7c8db6f61f
fix: [2.5] use the object heap to keep the min ddl ts order ( #39195 )
...
- issue: #39002
- pr: #39118
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-14 14:23:05 +08:00
adfc3f945e
enhance: record memory size (uncompressed) item for index ( #38844 )
...
issue: #38715
pr: #38770
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
ac730be5aa
enhance: Bump milvus & proto version to v2.5.3 ( #39198 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-13 12:56:57 +08:00
39fa00fc1a
enhance: add intent ctx to rootcoord init ( #38948 )
...
issue: #38439
cherry pick "enhance: add intent ctx to rootcoord init"
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-01-11 23:54:57 +08:00
6b127d4f5f
fix: [2.5] Return io error other than `NotExist` refreshing config ( #38924 ) ( #39072 )
...
Cherry pick from master
pr: #38924
Related to #38923
This PR:
- Check whether `os.Stat` config file error is io.ErrNotExist
- Panic when get config return error during Milvus initialization
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-10 14:09:04 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
0693634f62
enhance: add db name in replica description ( #38673 )
...
issue: #36621
pr: #38672
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:43:04 +08:00
f2e581e2d7
enhance: add broadcast for streaming service ( #39047 )
...
issue: #38399
pr: #39020
- Add new rpc for transfer broadcast to streaming coord
- Add broadcast service at streaming coord to make broadcast message
sent automicly
also cherry pick the pr #38400
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-09 16:46:56 +08:00
congqixia
wei liu
Xianhui Lin
yihao.dai
zhagnlu
cai.zhang
XuanYang-cn
Zhen Ye
cqy123456
SimFG
jaime
aoiasd
Gao
Duc Le Tu
foxspy
smellthemoon
sthuang
zhenshan.cao
tinswzy