0db5e0c4f6
enhance: [2.5]Deprecate disk params about indexing ( #41078 )
...
issue: #40863
master pr: #41045
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-04-07 11:36:34 +08:00
37a533fe6d
fix: [2.5] Address manual balance and balance check issues ( #41038 )
...
issue: #37651
pr: #41037
- Fix context propagation for manual balance segment task creation from
PR #38080 .
- Optimize stopping balance by preventing redundant checks per round,
addressing performance regression from PR #40297 .
- Decrease default `checkBalanceInterval` from 3000ms to 300ms.
- Correct minor log messages in `BalanceChecker`.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-04-03 01:26:23 +08:00
b64bb63e77
enhance: [2.5] Add trigger interval config for auto balance ( #39154 ) ( #39918 )
...
issue: #39156
pr: #39154
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-03-27 16:40:23 +08:00
ec0c25aaec
fix: [2.5] As the segment size increases the number of slots should also increase ( #40862 )
...
issue: #40858
master pr: #39084
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-25 14:42:24 +08:00
b534c9d804
enhance: [2.5] Introduce batch subscription in msgdispatcher ( #40596 )
...
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.
issue: https://github.com/milvus-io/milvus/issues/39862
pr: https://github.com/milvus-io/milvus/pull/39863
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-24 10:18:17 +08:00
0cfe835cac
enhance: [2.5] Support slot for index task and stats task ( #40288 )
...
issue: #39101
master pr: #39084
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-19 16:50:11 +08:00
e49294cd16
enhance: [2.5] Add buffer forwarder for stream delta loading ( #40559 ) ( #40699 )
...
Cherry-pick from master
pr: #40559
See also #40558
Related to #35303 & #38066 as well
This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-19 10:28:14 +08:00
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats ( #40661 )
...
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-18 17:32:16 +08:00
b8abd91962
enhance[2.5]: add more config for interimIndex to support more refine mode ( #40429 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-03-17 22:04:12 +08:00
a3c60a03b5
feat: [2.5] add DDLDB rate type and related quota configurations ( #40653 )
...
- issue: #40650
- pr: #40651
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-14 15:36:17 +08:00
6b9e141ada
enhance: reorder sub expr for conjunct expr ( #40186 )
...
pr:#39872
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-14 15:16:08 +08:00
d8a2c1a907
enhance: [2.5] Add channel seal policy based on blocking l0 ( #40505 ) ( #40535 )
...
Cherry-pick from 2.5
pr: #40505
Related to #40502
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-14 10:02:12 +08:00
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer ( #40039 )
...
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-13 20:18:10 +08:00
c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats ( #40108 )
...
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-02-22 14:35:54 +08:00
04175d8549
enhance: [2.5]compress jsonstatslog path only save filename ( #39979 )
...
enhance: compress jsonstatslog path only save filename
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-02-19 10:20:52 +08:00
ee25af4c9b
enhance: Add configs for compaction schedule ( #39010 ) ( #39511 )
...
pr: #39010
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-17 11:44:15 +08:00
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization ( #39876 )
...
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
a50249e0bb
enhance: [2.5] dump pprof info if component stop progress timeout ( #39760 )
...
issue: #39735
pr: #39726
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-02-13 15:30:47 +08:00
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) ( #39180 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data:
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40 ">
<head>
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
.font5
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;}
tr
{mso-height-source:auto;
mso-ruby-visibility:none;}
col
{mso-width-source:auto;
mso-ruby-visibility:none;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-number-format:General;
text-align:general;
vertical-align:middle;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
ruby
{ruby-align:left;}
rt
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-char-type:none;
display:none;}
-->
</head>
<body link="#0563C1" vlink="#954F72">
sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
| |
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no
</body>
</html>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-02-13 09:56:47 +08:00
f542c866b3
fix: [2.5] root check the role list if `rootShouldBindRole` is true ( #39714 )
...
- issue: #39712
- pr: #39713
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-02-11 10:10:51 +08:00
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers ( #38915 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38455
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 15:19:03 +08:00
4cb47b3f67
feat: [2.5] root privileges can be customized ( #39324 )
...
- issue: #39184
- pr: #39191
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-16 14:27:11 +08:00
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf ( #39206 ) ( #39255 )
...
Cherry-pick from master
pr: #39206 #39308
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 00:33:03 +08:00
54036bcafd
enhance: add broadcast operation for msgstream ( #39119 )
...
issue: #38399
pr: #39040
- make broadcast service available for msgstream by reusing the
architecture streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:11:00 +08:00
a4901ef7ec
fix: [2.5] fix privilege group list and list collections ( #38738 )
...
cherry-pick from: https://github.com/milvus-io/milvus/pull/38684
related: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:06:50 +08:00
52de43dbeb
enhance: add multiply factor when loading index ( #38722 )
...
issue: #38715
pr: #38716
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:50:58 +08:00
78438ef41e
fix: revert optimize CPU usage for CheckHealth requests ( #35589 ) ( #38555 )
...
issue: #35563
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-19 00:38:45 +08:00
29e620fa6d
fix: sync task still running after DataNode has stopped ( #38377 )
...
issue: #38319
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 18:06:44 +08:00
28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests ( #35589 )
...
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 11:02:45 +08:00
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
c2855a5c74
enhance: add privilege group privilege into built-in privilege group ( #38393 )
...
related issue: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-12 17:20:42 +08:00
dc85d8e968
enhance: improve mix compaction performance by removing max segment limitations ( #38344 )
...
See #37234
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-11 20:38:42 +08:00
e279ccf109
enhance: Enable score based balance channel policy ( #38143 )
...
issue: #38142
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.
This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-11 17:20:43 +08:00
41b19c6b1d
enhance: Determine the number of buffers based on the resource limits of the DataNode ( #38209 )
...
issue: #28410
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-08 18:02:40 +08:00
1f66b9ebfb
feat: add config field to set internal tls sni ( #38124 )
...
/cc @xiaofan-luan @jaime0815 @nish112022
part of https://github.com/milvus-io/milvus/issues/36864
Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2024-12-04 14:56:47 +08:00
23dc313c44
fix: fix grant/revoke v2 meta and unclear error messages ( #38110 )
...
related issue: https://github.com/milvus-io/milvus/issues/37031
fixed issues:
#37974 : better error messages for grant v2 interface
#37903 : fix meta built-in privilege group object name
#37843 : better error messages for custom privilege group interface
#38002 : fix built-in privilege group meta to pass proxy interceptor
check
#38008 : fix revoke v2 to support revoking v1 granted privileges
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 11:36:39 +08:00
2208b7c2ef
fix: the too long default root password does not take effect ( #37983 )
...
- issue: #36987
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-26 17:24:35 +08:00
2b4f211d84
enhance: add switch for local rpc enabled ( #37985 )
...
issue: #33285
- Add switch for local rpc
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-26 17:00:54 +08:00
83df725146
enhance: Revert default l0 forward policy to FilterByBF ( #37867 )
...
Related to #37767
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 09:00:33 +08:00
d7dcc752f1
enhance: Increase task capacity and clean illegal task ( #37896 )
...
1. taskQueueCapacity 256 is too small for production when we want to
re-write the entire collection
2. tasks should be cleaned when unable to recover, or the meta will
remain in etcd forever later.
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 18:48:32 +08:00
0a440e0d38
fix: Prevent simultaneous balance of segments and channels ( #37850 )
...
issue: #33550
balance segment and balance segment execute at same time, which will
cause bounch of corner case.
This PR disable simultaneous balance of segments and channels
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:56:55 +08:00
484c6b5c44
feat: Added code for Internal-tls ( #36865 )
...
issue : https://github.com/milvus-io/milvus/issues/36864
I have a few questions regarding my approach.I will consolidate them
here for feedback and review.Thanks
---------
Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
Signed-off-by: Nischay <Nischay.Yadav@ibm.com>
2024-11-20 06:00:32 +08:00
dae4160466
enhance: Whether to enable mergeSort mode when performing mixCompaction ( #37664 )
...
issue: #37579
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-19 11:28:31 +08:00
2d72ad33f2
enhance: RBAC built in privilege groups ( #37720 )
...
issue: #37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-18 20:38:39 +08:00
0fc0d1a888
fix: Limit the concurrency of channel tasks ( #37740 )
...
Limit the maximum concurrency of channel tasks for each DataNode to
prevent excessive subscriptions from causing DataNode OOM.
issue: https://github.com/milvus-io/milvus/issues/37665
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-18 16:26:30 +08:00
81fa7dd52c
fix: add ddl and dcl concurrency to avoid competition ( #37672 )
...
issue: #37166
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-15 15:04:31 +08:00
66bf254437
enhance: Enable `RemoteLoad` l0 forward policy by default ( #37678 )
...
Related to #35303
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-15 10:22:30 +08:00
a9538f6e96
fix: config file validator missed the file tail validation ( #37608 )
...
See: #32168
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-11-13 15:14:30 +08:00
a45a288a25
fix: Separate L0 and Mix trigger interval ( #37190 )
...
See also: #37108
- Add MixCompactionTriggerInterval, default 60s
- Add L0CompactionTriggerInterval, default 10s
- Export Single related compaction configs
- Raise SingleCompactionDeltaLogMaxSize from 2MB to 16MB
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-12 10:56:37 +08:00
c1eccce2fa
enhance: enable multiple chunked segment by default ( #37570 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 09:20:28 +08:00
cai.zhang
wei liu
yihao.dai
congqixia
Xianhui Lin
cqy123456
SimFG
zhagnlu
XuanYang-cn
Zhen Ye
sthuang
jaime
Ted Xu
shaoyue
nish112022
Bingyi Sun