05c01070c6
fix: fix arm64 compile issue ( #41058 )
...
issue: #41059
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-04-02 17:44:22 +08:00
a906466d8f
fix: [2.5]Add bypass thread pool mode to avoid growing indexes blocking insert/load ( #41013 )
...
issue: #40825
related: #41012
pr: #41012
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-04-02 16:58:25 +08:00
0d3bd3131c
fix:fix format error for json ( #41031 )
...
#41026
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-04-02 14:32:32 +08:00
7945ff8784
enhance: increate tantivy index worker and memory budget for json key stats for buliding index ( #40912 )
...
issue: #40897
This PR, combined with https://github.com/milvus-io/milvus/pull/40898 ,
makes tantivy total duration decrease roughly
from 6.3s to 1.68s for the case in the issue.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-04-02 08:23:08 +08:00
28c1ab8a16
enhance: add documents in batch for json key stats ( #40898 )
...
issue: #40897
After this, the document add operations scheduling duration is
**decreased** roughly from **6s to 0.9s** for the case in the issue.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-04-02 08:22:24 +08:00
c7ca352de5
enhance: Initialize field id to avoid negative number ( #38789 ) ( #41017 )
...
pr: #38789
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-04-01 10:08:21 +08:00
9108cee958
feat: add json null/exists expression ( #41002 )
...
issue: #35528
pr: https://github.com/milvus-io/milvus/pull/41004
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-31 21:52:23 +08:00
48eabe3d2e
fix:[2.5]growing mmap data type crashed by nullable input ( #40980 )
...
issue: https://github.com/milvus-io/milvus/issues/40981
master pr: https://github.com/milvus-io/milvus/pull/40994
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-03-31 14:42:20 +08:00
d420f100c0
enhance: refine array view to optimize memory usage( #38736 ) ( #40206 )
...
related: #38736
pr: https://github.com/milvus-io/milvus/pull/38808
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-03-28 11:08:18 +08:00
f0346a149a
fix: Fix using json index when iterative_filter is specified ( #40946 )
...
issue: #40934
pr: #40945
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-27 23:02:20 +08:00
deed5b5df4
enhance:change multi or expr to in expr ( #40751 )
...
pr: #40757
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-27 11:08:20 +08:00
7c716d61dc
fix:[2.5]wrong p.InterimIndexRefineQuantType default value and reduce ut run time ( #40830 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-03-21 19:10:13 +08:00
d09483809c
fix: Fix tasks will panic if one of them throw an exception. ( #40760 )
...
issue: #40690
pr: https://github.com/milvus-io/milvus/pull/40691
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-21 17:50:18 +08:00
0a1272fc6f
fix: Revert "enhance: update tantivy version ( #39253 )" ( #40769 )
...
issue: #35528
pr: #40313
This reverts commit d3cec7f277
2025-03-21 11:24:17 +08:00
14259a3f8a
fix:fix exception when do arith expr when using index ( #40799 )
...
pr: #40783
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-20 19:34:49 +08:00
2710eb3fc9
fix: [2.5] Json Stats filter the data is double but the filter expr is int ( #40771 )
...
Json Stats filter the data is double but the filter is int
issue: https://github.com/milvus-io/milvus/issues/40707
master-pr:https://github.com/milvus-io/milvus/pull/38039
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-19 20:22:13 +08:00
78f3d69a2f
fix: jsonstats Negative numbers cannot be filtered out ( #40734 )
...
fix: jsonstats Negative numbers cannot be filtered out
issue: https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-18 22:35:53 +08:00
d4e2f581f9
fix: [2.5] Do not delete the centroids file when sampling fails instead wait GC ( #40702 )
...
issue: #40700
pr: #40701
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-18 22:00:20 +08:00
2fbfd86146
fix:[2.5]fix growing index ut ( #40724 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-03-18 19:04:15 +08:00
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats ( #40661 )
...
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-18 17:32:16 +08:00
e47682aa55
fix:add clear bitmap input for every batch loop ( #40722 )
...
#40685
cherry-pick part from pr: #39872
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-18 17:27:02 +08:00
0dc4b73c81
fix: [2.5] avoid segmentation faults caused by retrieving empty vector datasets ( #40546 )
...
issue: #40544
pr: #40545
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-03-18 14:58:13 +08:00
b0078ceae3
fix: Fix json test ( #40710 )
...
issue: #35528
pr: https://github.com/milvus-io/milvus/pull/40671
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-18 12:52:13 +08:00
b8abd91962
enhance[2.5]: add more config for interimIndex to support more refine mode ( #40429 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-03-17 22:04:12 +08:00
9145885f40
enhance: Record simdjson error ( #40671 )
...
issue: #35528
pr: #40647
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-15 12:48:08 +08:00
ae8a9cebb8
fix: fix json index not-equal filter ( #40648 )
...
issue: #35528
pr: https://github.com/milvus-io/milvus/pull/40647
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-14 21:36:10 +08:00
0a3435f53a
fix: Fix null offset loading in inverted index ( #40524 )
...
issue: #40516
pr: https://github.com/milvus-io/milvus/pull/40523
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-14 18:32:09 +08:00
6b9e141ada
enhance: reorder sub expr for conjunct expr ( #40186 )
...
pr:#39872
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-14 15:16:08 +08:00
33e9db1539
fix: cherry pick json prs ( #40621 )
...
issue: #35528
pr: #40604
this pr includes some prs:
1. #40604
2. #40611
3. #40619
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-13 20:36:10 +08:00
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer ( #40039 )
...
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-13 20:18:10 +08:00
d8f51ca852
fix: Catch invalid json pointer error ( #40626 )
...
issue: #35528
pr: https://github.com/milvus-io/milvus/pull/40625
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-13 16:10:09 +08:00
ec6f8bac6d
fix: [2.5] avoid panic when field not exists in schema in query node ( #40542 )
...
ref https://github.com/milvus-io/milvus/issues/40473
cherry-pick: https://github.com/milvus-io/milvus/pull/40541
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-03-12 22:14:09 +08:00
bd4170106d
fix: cherry pick json index related prs ( #40568 )
...
issue: #35528
pr: #40407
this pr includes some json index related pr:
1. #40407
2. #40538
3. #40419
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-11 21:00:09 +08:00
d0763cb84d
enhance: [2.5] Fix Array.h clang format issue ( #40536 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-11 15:58:09 +08:00
683b26ffb7
feat: cherry pick json path index ( #40313 )
...
issue: #35528
pr: #36750
this pr includes json path index pr and some related prs:
1. update tantivy version #39253
2. json path index #36750
3. fall back to brute force #40076
4. term filter #40140
5. bug fix #40336
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-10 22:14:05 +08:00
ed71932c0a
fix:array corrupt for ignoring validity( #40384 ) ( #40433 )
...
related: #40384
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-03-07 18:15:48 +08:00
32b6f721eb
fix: Null expr does not work for json field ( #40457 )
...
issue: #40455
pr: #40456
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-07 18:00:03 +08:00
08a8e01f72
fix: [2.5]store wrong offset when build tantivy in nullable field ( #40453 )
...
pr: #40452
issue: #40454
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-03-07 17:54:06 +08:00
8d76d58d20
enhance: [2.5]Only check element type with not null array ( #40447 )
...
master pr: #40446
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-07 12:49:08 +08:00
345ca01634
fix: [2.5]fix concurrent issues in null offset #40363 ( #40365 )
...
issue: https://github.com/milvus-io/milvus/issues/40308
cherry-pick: https://github.com/milvus-io/milvus/pull/40363
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-03-05 16:54:09 +08:00
34f2bc0a68
enhance: add more metrics for async cgo component ( #40232 )
...
issue: #40014
pr: #40136
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-05 09:16:00 +08:00
825316670d
enhance: upgrade rust version to 1.83 ( #39295 ) ( #40317 )
...
issue: #39254
pr: https://github.com/milvus-io/milvus/pull/39295
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-04 17:42:04 +08:00
c45771af6a
enhance: add monitor metric for retrieve raw data ( #40155 )
...
pr: #40141
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-25 14:41:56 +08:00
29579a8ec9
fix: Fix search failure of null expression ( #40128 )
...
issue: https://github.com/milvus-io/milvus/issues/40095
pr: #40129
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-02-24 18:35:55 +08:00
d66d3f0d47
enhance:decrease dump snapshot limit from 10w to 1w ( #40102 )
...
pr: #40101
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-22 11:29:53 +08:00
b8a758b6c4
enhance: [2.5] Add get vector latency metric and refine request limit error message ( #40085 )
...
issue: https://github.com/milvus-io/milvus/issues/40078
pr: https://github.com/milvus-io/milvus/pull/40083
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-21 20:19:55 +08:00
ce6d8ba836
enhance: optimize expr performace for some points ( #39695 ) ( #39938 )
...
1. skip get expr arguments which deserialize proto for every batch
execute.
2. replace unordered_set with sort array that has better performace for
small set.
pr: #39695
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-19 19:52:53 +08:00
6cd00880e2
fix: Skip create primary key index when loading pk index ( #39922 )
...
issue: https://github.com/milvus-io/milvus/issues/39907
pr: https://github.com/milvus-io/milvus/pull/39921
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-02-19 15:54:52 +08:00
04175d8549
enhance: [2.5]compress jsonstatslog path only save filename ( #39979 )
...
enhance: compress jsonstatslog path only save filename
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-02-19 10:20:52 +08:00
63576738b6
fix: [2.5]exists expr on nested dictionaries fail in JSONkeyIndex ( #39972 )
...
fix: exists expr on nested dictionaries fail in JSONkeyIndex
issue: https://github.com/milvus-io/milvus/issues/39963
pr: https://github.com/milvus-io/milvus/pull/38039
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-02-19 10:18:51 +08:00
2eaa1a4869
fix: [2.5] Use `text_log` prefix for TextMatchIndex null offset file ( #39936 )
...
Cherry-pick from master
pr: #39935
Related to #39933
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-18 00:43:07 +08:00
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization ( #39876 )
...
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
e372b1733c
enhance: [2.5]update raft to cuvs ( #39221 )
...
pr: #39873
issue: #39883
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2025-02-14 16:52:14 +08:00
401794f405
fix: [2.5] Fix module for long type ( #39802 )
...
issue: #39640
master pr: #39722
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-14 15:18:21 +08:00
df28d2200d
fix: [2.5] Remove load field & schema column size check #39834 ( #39835 )
...
Cherry-pick from master
pr: #39833
Related to #39788
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-13 12:42:53 +08:00
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) ( #39180 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data:
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40 ">
<head>
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
.font5
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;}
tr
{mso-height-source:auto;
mso-ruby-visibility:none;}
col
{mso-width-source:auto;
mso-ruby-visibility:none;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-number-format:General;
text-align:general;
vertical-align:middle;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
ruby
{ruby-align:left;}
rt
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-char-type:none;
display:none;}
-->
</head>
<body link="#0563C1" vlink="#954F72">
sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
| |
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no
</body>
</html>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-02-13 09:56:47 +08:00
0f4d3ef8b6
enhance[2.5]: move GetVector_EmptySparseVector ut to growing test ( #39286 )
...
issue: https://github.com/milvus-io/milvus/issues/35853
pr: https://github.com/milvus-io/milvus/pull/39234
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-02-12 16:00:49 +08:00
b02d6423a3
enhance: [2.5]update knowhere version ( #39673 )
...
/kind branch-feature
release note :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.3
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-02-07 12:46:45 +08:00
aee9368c6e
enhance: [2.5] update knowhere version and IsAdditionalScalarSupported interface ( #39574 )
...
pr: #39573
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-02-05 19:53:11 +08:00
6cdabd9002
fix:fix GetValueFromConfig for bool type ( #39527 )
...
pr: #39526
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-23 00:03:07 +08:00
6342237ca8
enhance: add workaround for concurrent tantivy reload and directory removal ( #39441 )
...
related pr: https://github.com/milvus-io/milvus/pull/39440
issue: https://github.com/milvus-io/milvus/issues/39428
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 17:33:05 +08:00
6f9f840080
fix: Fix panic caused by removing directory ( #38622 ) ( #39434 )
...
related pr: https://github.com/milvus-io/milvus/pull/38622
issue: https://github.com/milvus-io/milvus/issues/39428
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 16:13:03 +08:00
efe76905c5
fix: correct remote centroids path in clustering compaction ( #39399 )
...
issue: https://github.com/milvus-io/milvus/issues/39353
pr: #39398
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 10:57:21 +08:00
57bcb1bfc0
fix: sparse: add inverted_index_algo and dim_max_score_ratio config ( #39362 )
...
issue: #39332
pr: #39358
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
2025-01-18 12:17:03 +08:00
0c00f94965
fix: [2.5] Skip load extra indexes for sorted segment pk field ( #39390 )
...
Cherry-pick from master
pr: #39389
Related to #39339
Extra indexes can be ignored for most cases since sorted pk column
already provided indexing features
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-17 14:53:03 +08:00
56cb1683eb
fix: Fix performance issue and use after free bug ( #39343 )
...
cherry pick some fixes in https://github.com/milvus-io/milvus/pull/39249
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-17 11:51:03 +08:00
47e410feb1
fix: [2.5] fix assignment operator in AssertInfo to comparison operator ( #39351 )
...
cp https://github.com/milvus-io/milvus/pull/39347
issue: #39346
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-17 11:27:02 +08:00
1602390734
enhance: support null expr( #38772 ) ( #38931 )
...
pr: #38772
issue: #31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-16 10:55:07 +08:00
21df11b37c
enhance: [2.5] Add try-catch and return CStatus for NewCollection ( #39279 ) ( #39303 )
...
Cherry pick from master
pr: #39279
Related to #28795
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 10:13:06 +08:00
1d9788e4bc
fix: [2.5] enable to build index with single segment ( #39272 )
...
issue: #39232
cherry-pick https://github.com/milvus-io/milvus/pull/39233
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 09:59:02 +08:00
dd44a58381
enhance: [2.5] pass partition key scalar info if enable for vector mem index ( #39245 )
...
issue: #34332
pr: #39123
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-15 21:45:01 +08:00
b4ddf746c1
enhance: [2.5][bitset] extend op_find() to be able to search both 0 and 1 ( #39237 )
...
Cherry-pick from master
pr: #39176 #39229
issue: #39124
`bitset::find_first()` and `bitset::find_next()` now accept one more
parameter, which allows to search for `0` bit instead of `1` bit
---------
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Alexander Guzhva <alexanderguzhva@gmail.com>
2025-01-14 17:59:00 +08:00
9a63383878
fix:[2.5] fail to get empty index num rows( #39155 ) ( #39210 )
...
pr: #39155
issue: #39125
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-14 13:24:59 +08:00
adfc3f945e
enhance: record memory size (uncompressed) item for index ( #38844 )
...
issue: #38715
pr: #38770
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
5b113064fe
enhance: update knowhere version ( #39218 )
...
/kind branch-feature
release-note :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.2
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-01-14 10:21:06 +08:00
0aa592b081
fix: data race when accessing field_ when retrieving ( #39152 )
...
issue: #39148
pr: #39151
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-13 11:23:04 +08:00
a43720f779
fix: [2.5] fix chunk cache madvise when sparse raw data is mmaped ( #39146 )
...
instead of marking as not supported,
ChunkedSparseFloatColumn::DataByteSize can simply use the impl of super
class.
pr: https://github.com/milvus-io/milvus/pull/39145
issue: #39158
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-01-10 16:24:58 +08:00
f70262c980
fix: build text index when loading field data ( #39070 ) ( #39113 )
...
fix: https://github.com/milvus-io/milvus/issues/39053 may fix
https://github.com/milvus-io/milvus/issues/38644 which could be caused
by https://github.com/milvus-io/milvus/issues/39053
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-09 19:30:58 +08:00
dad0d3b6e4
fix: Fix in filter search result is empty if pk type is varchar ( #39108 )
...
Cherry pick from master
pr: #39016
https://github.com/milvus-io/milvus/issues/39107
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-09 17:04:56 +08:00
aecf04e369
fix: fix query incorrect in case of concurrent delete ( #38991 ) ( #39054 )
...
pr: #38991
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-08 17:48:56 +08:00
b2b7cca910
fix: [2.5] set iterative filter hint to false when no expr specified ( #39034 )
...
issue: https://github.com/milvus-io/milvus/issues/39013
pr: #39033
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-08 12:58:56 +08:00
0b62c1e692
enhance: [2.5] Optimize retrieve performance utilizing bitset SIMD methods ( #39041 )
...
Cherry pick from master
pr: #39004 #39037
Related to #39003
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-07 17:45:03 +08:00
015e50e4de
fix: [2.5] Only generate the index_null_offset file when the field support null value ( #38834 )
...
issue: #38832
master pr: #38833
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-12-30 17:02:52 +08:00
ef400227ad
enhance: [2.5][cp] speed up search iterator stage 1 ( #38678 )
...
pr: https://github.com/milvus-io/milvus/pull/37947
issue: https://github.com/milvus-io/milvus/issues/37548
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 9016c4adcd765c0766b01e7e5d465c915e176a6f)
2024-12-27 18:48:52 +08:00
3d723676ac
enhance: [2.5]update knowhere version ( #38785 )
...
/kind branch-feature
release note: https://github.com/zilliztech/knowhere/releases/tag/v2.5.1
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-12-27 10:24:50 +08:00
930f9e9ef8
fix:[Cherr-pick] sealed segment get empty index params when brute force search for bm25 ( #38752 )
...
relate: https://github.com/milvus-io/milvus/issues/38236
pr: https://github.com/milvus-io/milvus/pull/38707
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-26 10:26:51 +08:00
d47372eb9c
fix: report error when hints not supported ( #38730 )
...
issue: #38705
pr: #38717
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-25 19:02:56 +08:00
8fcb33c21d
fix:fix delete record assert failed ( #38580 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-19 18:22:47 +08:00
b537a72309
fix: interted index out of range ( #38577 )
...
issue: #38546 , #38486
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-19 15:20:47 +08:00
06d410b70f
enhance: update knowhere version ( #38544 )
...
related: #37730
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-12-18 16:52:45 +08:00
87056be748
fix: fix snapshot or size when query ( #38549 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-18 16:42:45 +08:00
ffd3c5d2f5
[automated] Update Knowhere Commit ( #38542 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-18 01:24:47 +08:00
decdfdae10
fix: growing-groupby-crush( #38533 ) ( #38538 )
...
related: #38533
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-12-17 21:05:12 +08:00
f0096ec292
fix: Fix IsMmapSupported for scalar index ( #38135 )
...
https://github.com/milvus-io/milvus/issues/38134
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 20:30:44 +08:00
9afcc5bc5c
fix:fix incorrect dir operations when create or load inverted index ( #38359 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 20:06:45 +08:00
d0a7e98a27
fix:remove incorrect assert for delete query ( #38509 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 17:48:44 +08:00
dd4f33ae19
fix: Fix chunked segment can not warmup using mmap ( #38492 )
...
issue: #38410
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 13:42:45 +08:00
33aecb0655
fix: build break on target test_cpp under OSX ( #38479 )
...
See: #38434
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-17 13:38:45 +08:00
3e2a2f278b
enhance: Handle rust error in c++ ( #38113 )
...
https://github.com/milvus-io/milvus/issues/37930
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-16 19:40:45 +08:00
4919ccf543
enhance: eliminate compile warnings ( #38420 )
...
See: #38435
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-16 09:58:43 +08:00
01de0afc4e
enhance: refactor delete mvcc function ( #38066 )
...
#37413
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-15 18:02:43 +08:00
6ea15265e1
enhance: add file info log when mmap failed. ( #38386 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-15 17:36:43 +08:00
1e274384cd
[automated] Update Knowhere Commit ( #38458 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-14 00:54:43 +08:00
c1f9158996
fix: search-group-by failed to get data from multi-chunked-segment(##… ( #38383 )
...
related: #38343
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-12-13 16:54:43 +08:00
3038383e36
fix: UT compile broken under osx ( #38432 )
...
See: #38434
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-13 16:24:43 +08:00
efbfa1cc3e
fix:fix ut failed for debug ( #38384 )
...
#38382
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-13 14:38:43 +08:00
e404123e3e
[automated] Update Knowhere Commit ( #38422 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-13 02:40:44 +08:00
b14a0c4bf5
fix:GrowingDataGetter get the wrong string data ( #38015 )
...
issue: https://github.com/milvus-io/milvus/issues/37994
2.4 pr: https://github.com/milvus-io/milvus/pull/37995
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-12-12 14:50:42 +08:00
994fc544e7
enhance: support iterative filter execution ( #37363 )
...
issue: #37360
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 11:32:44 +08:00
9ef76971ce
fix:add more info to local chunk manager log ( #38357 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-11 10:24:49 +08:00
32f575be0f
enhance: change bitmap index mmap mode to view mode ( #38179 )
...
#38138
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-08 17:26:41 +08:00
6d0a4fdb31
fix: Fix bug for Search fails with filter expression contains underscore ( #38085 )
...
Enhance the matching for elements within the UnaryRangeArray
https://github.com/milvus-io/milvus/issues/38068
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-05 10:18:40 +08:00
262f6db3d8
enhance: Add mmap file usage metric ( #38193 )
...
issue: #38156 Add mmap file usage metric
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 16:12:47 +08:00
87aa9a0f2d
fix: empty analyzer params not use standard tokenizer ( #38148 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-04 14:58:39 +08:00
3445b8bd44
[automated] Update Knowhere Commit ( #38192 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-04 02:30:39 +08:00
767b7e6218
enhance: Use fdopen, fwrite to reduce direct syscall ( #38157 )
...
`File.Write` and `File.WriteInt` use `write`, which may be just direct
syscall in some systems. When mappding field data and write line by
line, this could cost lost of CPU time when the row number is large.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-03 15:24:39 +08:00
90064cd47b
fix: Fix variable redeclaration in term filter ( #38045 )
...
https://github.com/milvus-io/milvus/issues/38046
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-02 15:10:38 +08:00
c6dcef7b84
enhance: move segcore codes of segment into one package ( #37722 )
...
issue: #33285
- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-29 10:22:36 +08:00
0894ed0016
[automated] Update Knowhere Commit ( #38082 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-29 01:08:36 +08:00
e6af806a0d
enhance: optimize self defined rust error ( #37975 )
...
Prepare for issue: https://github.com/milvus-io/milvus/issues/37930
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-28 20:30:36 +08:00
cb6542339e
enhance: Mark cgo thread with tag name ( #38000 )
...
Related to #37999
This PR add `SetThreadName` API for marking cgo thread and utilize it
when initializing cgo worker.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-26 11:22:35 +08:00
fbb68ca370
enhance: make all index operation async scheduled by tokio ( #37946 )
...
issue: #37851
related pr: https://github.com/milvus-io/tantivy/pull/3
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-25 10:12:34 +08:00
ed73dfca3f
[automated] Update Knowhere Commit ( #37965 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-25 01:28:32 +08:00
62af24c1a1
fix: change search latency metric from us unit to ms unit ( #37806 )
...
#37805
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-24 17:26:33 +08:00
700a448a54
fix: Escape prefix before search in inverted index ( #37925 )
...
issue: https://github.com/milvus-io/milvus/issues/37912
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 14:10:33 +08:00
06d73cf2e2
enhance: Remove raw tokenizer register. ( #37886 )
...
tantivy already register raw tokenizer by default
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 12:02:32 +08:00
1dc1a97e65
fix: use different thread pool for scheduler and merger ( #37911 )
...
issue: #37895
related pr: https://github.com/milvus-io/tantivy/pull/2
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-21 21:34:33 +08:00
f3a36f8a29
fix: use global pool but not dedicated pool for every index ( #37852 )
...
issue: #37851
- make a global thread pool at tantivy temporally.
- set 1 but not 4 threads for inverted text index.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-20 20:44:32 +08:00
8216345b07
enhance: reduce copy of bitset and id conversion of brurtforce search ( #37675 )
...
issue: https://github.com/milvus-io/milvus/issues/37798
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-11-19 15:48:40 +08:00
6b82320953
fix: Fix using wrong upperbound when searching by pk ( #37769 )
...
issue: https://github.com/milvus-io/milvus/issues/37649
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-19 10:16:31 +08:00
3d28d99411
fix: to use the correct offset in span ( #37780 )
...
#37734
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-18 21:56:30 +08:00
16e206167c
enhance: analyzer length filter max should be close interval instead open interval ( #37770 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 19:30:31 +08:00
e9391acf80
fix: bm25 brute force search need index params k1 and b ( #37721 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 15:44:31 +08:00
3f1614e9d9
enhance: add trace_id into segcore logs ( #37656 )
...
issue: #37655
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-18 10:20:30 +08:00
3b5a0df159
enhance: Optimize chinese analyzer and support CnAlphaNumFilter ( #37727 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-16 10:28:30 +08:00
0ba868ae64
enhance: update knowhere version ( #37730 )
...
release note draft :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.0
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-16 10:08:30 +08:00
7999367c0c
fix: use not retried err when get wrong parameter ( #37707 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-15 19:14:30 +08:00
e4b6773d0a
fix: fix create text index dir conflict bug ( #37693 )
...
#37623
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-15 18:26:30 +08:00
65d3c6622a
enhance: Optimize GetChunkIDByOffset and add ut ( #37704 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 14:16:31 +08:00
d1596297d9
fix: Fix query failure with inverted index ( #37686 )
...
https://github.com/milvus-io/milvus/issues/37649
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 10:28:31 +08:00
5ae347aba0
enhance: update knowhere version ( #37688 )
...
issue: #37665 #37631 #37620 #37587 #36906
knowhere has add default nlist value, so some invalid param test ut with
no nlist param will be valid.
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-15 10:10:31 +08:00
1b4f7e3ac1
enhance: Add more expr ut for chunked segment ( #37600 )
...
related pr: #37570
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-14 18:40:32 +08:00
993051bb49
fix: brute force bm25 search lack avgdl param ( #37650 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 14:58:31 +08:00
0565300b7f
fix: Sparse to use CC index as growing/temp index ( #37591 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-14 10:54:31 +08:00
1c5b5e1e3d
feat: Add chinese and english analyzer with refactor jieba tokenizer ( #37494 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 10:34:31 +08:00
cf883b114e
enhance: update knowhere version ( #37510 )
...
issue: #36925
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-13 16:36:27 +08:00
3389a6b500
enhance: support null in text match index ( #37517 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-13 11:08:29 +08:00
3c225e5c94
fix: data race when using fields_ ( #37612 )
...
issue: #37609
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-13 04:06:30 +08:00
2d29dcd30c
enhance:refine group_strict_size parameter( #37482 ) ( #37483 )
...
related: #37482
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-11-12 09:56:28 +08:00
c1eccce2fa
enhance: enable multiple chunked segment by default ( #37570 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 09:20:28 +08:00
Spade A
foxspy
zhagnlu
Bingyi Sun
cqy123456
Chun Han
Xianhui Lin
cai.zhang
congqixia
smellthemoon
Zhen Ye
yihao.dai
presburger
Buqian Zheng
Gao
sparknack
Patrick Weizhi Xu
aoiasd
sre-ci-robot
Ted Xu
tinswzy