2eaa1a4869
fix: [2.5] Use `text_log` prefix for TextMatchIndex null offset file ( #39936 )
...
Cherry-pick from master
pr: #39935
Related to #39933
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-18 00:43:07 +08:00
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization ( #39876 )
...
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
e372b1733c
enhance: [2.5]update raft to cuvs ( #39221 )
...
pr: #39873
issue: #39883
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2025-02-14 16:52:14 +08:00
401794f405
fix: [2.5] Fix module for long type ( #39802 )
...
issue: #39640
master pr: #39722
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-14 15:18:21 +08:00
df28d2200d
fix: [2.5] Remove load field & schema column size check #39834 ( #39835 )
...
Cherry-pick from master
pr: #39833
Related to #39788
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-13 12:42:53 +08:00
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) ( #39180 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data:
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40 ">
<head>
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
.font5
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;}
tr
{mso-height-source:auto;
mso-ruby-visibility:none;}
col
{mso-width-source:auto;
mso-ruby-visibility:none;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-number-format:General;
text-align:general;
vertical-align:middle;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
ruby
{ruby-align:left;}
rt
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-char-type:none;
display:none;}
-->
</head>
<body link="#0563C1" vlink="#954F72">
sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
| |
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no
</body>
</html>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-02-13 09:56:47 +08:00
0f4d3ef8b6
enhance[2.5]: move GetVector_EmptySparseVector ut to growing test ( #39286 )
...
issue: https://github.com/milvus-io/milvus/issues/35853
pr: https://github.com/milvus-io/milvus/pull/39234
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-02-12 16:00:49 +08:00
b02d6423a3
enhance: [2.5]update knowhere version ( #39673 )
...
/kind branch-feature
release note :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.3
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-02-07 12:46:45 +08:00
aee9368c6e
enhance: [2.5] update knowhere version and IsAdditionalScalarSupported interface ( #39574 )
...
pr: #39573
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-02-05 19:53:11 +08:00
6cdabd9002
fix:fix GetValueFromConfig for bool type ( #39527 )
...
pr: #39526
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-23 00:03:07 +08:00
6342237ca8
enhance: add workaround for concurrent tantivy reload and directory removal ( #39441 )
...
related pr: https://github.com/milvus-io/milvus/pull/39440
issue: https://github.com/milvus-io/milvus/issues/39428
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 17:33:05 +08:00
6f9f840080
fix: Fix panic caused by removing directory ( #38622 ) ( #39434 )
...
related pr: https://github.com/milvus-io/milvus/pull/38622
issue: https://github.com/milvus-io/milvus/issues/39428
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 16:13:03 +08:00
efe76905c5
fix: correct remote centroids path in clustering compaction ( #39399 )
...
issue: https://github.com/milvus-io/milvus/issues/39353
pr: #39398
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-20 10:57:21 +08:00
57bcb1bfc0
fix: sparse: add inverted_index_algo and dim_max_score_ratio config ( #39362 )
...
issue: #39332
pr: #39358
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
2025-01-18 12:17:03 +08:00
0c00f94965
fix: [2.5] Skip load extra indexes for sorted segment pk field ( #39390 )
...
Cherry-pick from master
pr: #39389
Related to #39339
Extra indexes can be ignored for most cases since sorted pk column
already provided indexing features
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-17 14:53:03 +08:00
56cb1683eb
fix: Fix performance issue and use after free bug ( #39343 )
...
cherry pick some fixes in https://github.com/milvus-io/milvus/pull/39249
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-17 11:51:03 +08:00
47e410feb1
fix: [2.5] fix assignment operator in AssertInfo to comparison operator ( #39351 )
...
cp https://github.com/milvus-io/milvus/pull/39347
issue: #39346
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-17 11:27:02 +08:00
1602390734
enhance: support null expr( #38772 ) ( #38931 )
...
pr: #38772
issue: #31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-16 10:55:07 +08:00
21df11b37c
enhance: [2.5] Add try-catch and return CStatus for NewCollection ( #39279 ) ( #39303 )
...
Cherry pick from master
pr: #39279
Related to #28795
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 10:13:06 +08:00
1d9788e4bc
fix: [2.5] enable to build index with single segment ( #39272 )
...
issue: #39232
cherry-pick https://github.com/milvus-io/milvus/pull/39233
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 09:59:02 +08:00
dd44a58381
enhance: [2.5] pass partition key scalar info if enable for vector mem index ( #39245 )
...
issue: #34332
pr: #39123
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-15 21:45:01 +08:00
b4ddf746c1
enhance: [2.5][bitset] extend op_find() to be able to search both 0 and 1 ( #39237 )
...
Cherry-pick from master
pr: #39176 #39229
issue: #39124
`bitset::find_first()` and `bitset::find_next()` now accept one more
parameter, which allows to search for `0` bit instead of `1` bit
---------
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Alexander Guzhva <alexanderguzhva@gmail.com>
2025-01-14 17:59:00 +08:00
9a63383878
fix:[2.5] fail to get empty index num rows( #39155 ) ( #39210 )
...
pr: #39155
issue: #39125
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-14 13:24:59 +08:00
adfc3f945e
enhance: record memory size (uncompressed) item for index ( #38844 )
...
issue: #38715
pr: #38770
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
5b113064fe
enhance: update knowhere version ( #39218 )
...
/kind branch-feature
release-note :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.2
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-01-14 10:21:06 +08:00
0aa592b081
fix: data race when accessing field_ when retrieving ( #39152 )
...
issue: #39148
pr: #39151
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-13 11:23:04 +08:00
a43720f779
fix: [2.5] fix chunk cache madvise when sparse raw data is mmaped ( #39146 )
...
instead of marking as not supported,
ChunkedSparseFloatColumn::DataByteSize can simply use the impl of super
class.
pr: https://github.com/milvus-io/milvus/pull/39145
issue: #39158
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-01-10 16:24:58 +08:00
f70262c980
fix: build text index when loading field data ( #39070 ) ( #39113 )
...
fix: https://github.com/milvus-io/milvus/issues/39053 may fix
https://github.com/milvus-io/milvus/issues/38644 which could be caused
by https://github.com/milvus-io/milvus/issues/39053
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-09 19:30:58 +08:00
dad0d3b6e4
fix: Fix in filter search result is empty if pk type is varchar ( #39108 )
...
Cherry pick from master
pr: #39016
https://github.com/milvus-io/milvus/issues/39107
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-09 17:04:56 +08:00
aecf04e369
fix: fix query incorrect in case of concurrent delete ( #38991 ) ( #39054 )
...
pr: #38991
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-08 17:48:56 +08:00
b2b7cca910
fix: [2.5] set iterative filter hint to false when no expr specified ( #39034 )
...
issue: https://github.com/milvus-io/milvus/issues/39013
pr: #39033
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-08 12:58:56 +08:00
0b62c1e692
enhance: [2.5] Optimize retrieve performance utilizing bitset SIMD methods ( #39041 )
...
Cherry pick from master
pr: #39004 #39037
Related to #39003
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-07 17:45:03 +08:00
015e50e4de
fix: [2.5] Only generate the index_null_offset file when the field support null value ( #38834 )
...
issue: #38832
master pr: #38833
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-12-30 17:02:52 +08:00
ef400227ad
enhance: [2.5][cp] speed up search iterator stage 1 ( #38678 )
...
pr: https://github.com/milvus-io/milvus/pull/37947
issue: https://github.com/milvus-io/milvus/issues/37548
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 9016c4adcd765c0766b01e7e5d465c915e176a6f)
2024-12-27 18:48:52 +08:00
3d723676ac
enhance: [2.5]update knowhere version ( #38785 )
...
/kind branch-feature
release note: https://github.com/zilliztech/knowhere/releases/tag/v2.5.1
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-12-27 10:24:50 +08:00
930f9e9ef8
fix:[Cherr-pick] sealed segment get empty index params when brute force search for bm25 ( #38752 )
...
relate: https://github.com/milvus-io/milvus/issues/38236
pr: https://github.com/milvus-io/milvus/pull/38707
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-26 10:26:51 +08:00
d47372eb9c
fix: report error when hints not supported ( #38730 )
...
issue: #38705
pr: #38717
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-25 19:02:56 +08:00
8fcb33c21d
fix:fix delete record assert failed ( #38580 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-19 18:22:47 +08:00
b537a72309
fix: interted index out of range ( #38577 )
...
issue: #38546 , #38486
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-19 15:20:47 +08:00
06d410b70f
enhance: update knowhere version ( #38544 )
...
related: #37730
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-12-18 16:52:45 +08:00
87056be748
fix: fix snapshot or size when query ( #38549 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-18 16:42:45 +08:00
ffd3c5d2f5
[automated] Update Knowhere Commit ( #38542 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-18 01:24:47 +08:00
decdfdae10
fix: growing-groupby-crush( #38533 ) ( #38538 )
...
related: #38533
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-12-17 21:05:12 +08:00
f0096ec292
fix: Fix IsMmapSupported for scalar index ( #38135 )
...
https://github.com/milvus-io/milvus/issues/38134
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 20:30:44 +08:00
9afcc5bc5c
fix:fix incorrect dir operations when create or load inverted index ( #38359 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 20:06:45 +08:00
d0a7e98a27
fix:remove incorrect assert for delete query ( #38509 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 17:48:44 +08:00
dd4f33ae19
fix: Fix chunked segment can not warmup using mmap ( #38492 )
...
issue: #38410
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 13:42:45 +08:00
33aecb0655
fix: build break on target test_cpp under OSX ( #38479 )
...
See: #38434
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-17 13:38:45 +08:00
3e2a2f278b
enhance: Handle rust error in c++ ( #38113 )
...
https://github.com/milvus-io/milvus/issues/37930
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-16 19:40:45 +08:00
4919ccf543
enhance: eliminate compile warnings ( #38420 )
...
See: #38435
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-16 09:58:43 +08:00
01de0afc4e
enhance: refactor delete mvcc function ( #38066 )
...
#37413
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-15 18:02:43 +08:00
6ea15265e1
enhance: add file info log when mmap failed. ( #38386 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-15 17:36:43 +08:00
1e274384cd
[automated] Update Knowhere Commit ( #38458 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-14 00:54:43 +08:00
c1f9158996
fix: search-group-by failed to get data from multi-chunked-segment(##… ( #38383 )
...
related: #38343
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-12-13 16:54:43 +08:00
3038383e36
fix: UT compile broken under osx ( #38432 )
...
See: #38434
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-13 16:24:43 +08:00
efbfa1cc3e
fix:fix ut failed for debug ( #38384 )
...
#38382
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-13 14:38:43 +08:00
e404123e3e
[automated] Update Knowhere Commit ( #38422 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-13 02:40:44 +08:00
b14a0c4bf5
fix:GrowingDataGetter get the wrong string data ( #38015 )
...
issue: https://github.com/milvus-io/milvus/issues/37994
2.4 pr: https://github.com/milvus-io/milvus/pull/37995
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-12-12 14:50:42 +08:00
994fc544e7
enhance: support iterative filter execution ( #37363 )
...
issue: #37360
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 11:32:44 +08:00
9ef76971ce
fix:add more info to local chunk manager log ( #38357 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-11 10:24:49 +08:00
32f575be0f
enhance: change bitmap index mmap mode to view mode ( #38179 )
...
#38138
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-08 17:26:41 +08:00
6d0a4fdb31
fix: Fix bug for Search fails with filter expression contains underscore ( #38085 )
...
Enhance the matching for elements within the UnaryRangeArray
https://github.com/milvus-io/milvus/issues/38068
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-05 10:18:40 +08:00
262f6db3d8
enhance: Add mmap file usage metric ( #38193 )
...
issue: #38156 Add mmap file usage metric
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 16:12:47 +08:00
87aa9a0f2d
fix: empty analyzer params not use standard tokenizer ( #38148 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-04 14:58:39 +08:00
3445b8bd44
[automated] Update Knowhere Commit ( #38192 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-04 02:30:39 +08:00
767b7e6218
enhance: Use fdopen, fwrite to reduce direct syscall ( #38157 )
...
`File.Write` and `File.WriteInt` use `write`, which may be just direct
syscall in some systems. When mappding field data and write line by
line, this could cost lost of CPU time when the row number is large.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-03 15:24:39 +08:00
90064cd47b
fix: Fix variable redeclaration in term filter ( #38045 )
...
https://github.com/milvus-io/milvus/issues/38046
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-02 15:10:38 +08:00
c6dcef7b84
enhance: move segcore codes of segment into one package ( #37722 )
...
issue: #33285
- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-29 10:22:36 +08:00
0894ed0016
[automated] Update Knowhere Commit ( #38082 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-29 01:08:36 +08:00
e6af806a0d
enhance: optimize self defined rust error ( #37975 )
...
Prepare for issue: https://github.com/milvus-io/milvus/issues/37930
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-28 20:30:36 +08:00
cb6542339e
enhance: Mark cgo thread with tag name ( #38000 )
...
Related to #37999
This PR add `SetThreadName` API for marking cgo thread and utilize it
when initializing cgo worker.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-26 11:22:35 +08:00
fbb68ca370
enhance: make all index operation async scheduled by tokio ( #37946 )
...
issue: #37851
related pr: https://github.com/milvus-io/tantivy/pull/3
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-25 10:12:34 +08:00
ed73dfca3f
[automated] Update Knowhere Commit ( #37965 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-25 01:28:32 +08:00
62af24c1a1
fix: change search latency metric from us unit to ms unit ( #37806 )
...
#37805
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-24 17:26:33 +08:00
700a448a54
fix: Escape prefix before search in inverted index ( #37925 )
...
issue: https://github.com/milvus-io/milvus/issues/37912
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 14:10:33 +08:00
06d73cf2e2
enhance: Remove raw tokenizer register. ( #37886 )
...
tantivy already register raw tokenizer by default
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 12:02:32 +08:00
1dc1a97e65
fix: use different thread pool for scheduler and merger ( #37911 )
...
issue: #37895
related pr: https://github.com/milvus-io/tantivy/pull/2
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-21 21:34:33 +08:00
f3a36f8a29
fix: use global pool but not dedicated pool for every index ( #37852 )
...
issue: #37851
- make a global thread pool at tantivy temporally.
- set 1 but not 4 threads for inverted text index.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-20 20:44:32 +08:00
8216345b07
enhance: reduce copy of bitset and id conversion of brurtforce search ( #37675 )
...
issue: https://github.com/milvus-io/milvus/issues/37798
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-11-19 15:48:40 +08:00
6b82320953
fix: Fix using wrong upperbound when searching by pk ( #37769 )
...
issue: https://github.com/milvus-io/milvus/issues/37649
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-19 10:16:31 +08:00
3d28d99411
fix: to use the correct offset in span ( #37780 )
...
#37734
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-18 21:56:30 +08:00
16e206167c
enhance: analyzer length filter max should be close interval instead open interval ( #37770 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 19:30:31 +08:00
e9391acf80
fix: bm25 brute force search need index params k1 and b ( #37721 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 15:44:31 +08:00
3f1614e9d9
enhance: add trace_id into segcore logs ( #37656 )
...
issue: #37655
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-18 10:20:30 +08:00
3b5a0df159
enhance: Optimize chinese analyzer and support CnAlphaNumFilter ( #37727 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-16 10:28:30 +08:00
0ba868ae64
enhance: update knowhere version ( #37730 )
...
release note draft :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.0
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-16 10:08:30 +08:00
7999367c0c
fix: use not retried err when get wrong parameter ( #37707 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-15 19:14:30 +08:00
e4b6773d0a
fix: fix create text index dir conflict bug ( #37693 )
...
#37623
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-15 18:26:30 +08:00
65d3c6622a
enhance: Optimize GetChunkIDByOffset and add ut ( #37704 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 14:16:31 +08:00
d1596297d9
fix: Fix query failure with inverted index ( #37686 )
...
https://github.com/milvus-io/milvus/issues/37649
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 10:28:31 +08:00
5ae347aba0
enhance: update knowhere version ( #37688 )
...
issue: #37665 #37631 #37620 #37587 #36906
knowhere has add default nlist value, so some invalid param test ut with
no nlist param will be valid.
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-15 10:10:31 +08:00
1b4f7e3ac1
enhance: Add more expr ut for chunked segment ( #37600 )
...
related pr: #37570
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-14 18:40:32 +08:00
993051bb49
fix: brute force bm25 search lack avgdl param ( #37650 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 14:58:31 +08:00
0565300b7f
fix: Sparse to use CC index as growing/temp index ( #37591 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-14 10:54:31 +08:00
1c5b5e1e3d
feat: Add chinese and english analyzer with refactor jieba tokenizer ( #37494 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 10:34:31 +08:00
cf883b114e
enhance: update knowhere version ( #37510 )
...
issue: #36925
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-13 16:36:27 +08:00
3389a6b500
enhance: support null in text match index ( #37517 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-13 11:08:29 +08:00
3c225e5c94
fix: data race when using fields_ ( #37612 )
...
issue: #37609
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-13 04:06:30 +08:00
2d29dcd30c
enhance:refine group_strict_size parameter( #37482 ) ( #37483 )
...
related: #37482
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-11-12 09:56:28 +08:00
c1eccce2fa
enhance: enable multiple chunked segment by default ( #37570 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 09:20:28 +08:00
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params ( #37478 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
40ba5a3414
fix: fix chunked segment term filter expression and add ut ( #37392 )
...
issue: https://github.com/milvus-io/milvus/issues/37143
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-07 11:04:19 -08:00
5310d3469f
fix: Escape brace of dumped JSON for index err message ( #37504 )
...
Related to #37503
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-07 18:58:25 +08:00
9b6dd23f8e
fix: wrong path spelling when use rootpath in segcore ( #37453 )
...
#36532
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-07 11:22:25 +08:00
d67853fa89
feat: Tokenizer support build with params and clone for concurrency ( #37048 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
625b6176cd
fix: Search for pk using raw data to reduce the overhead caused by views ( #37202 )
...
issue: #37152
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-05 20:36:24 +08:00
bd04cac4b3
fix: fix group by on chunked segment ( #37292 )
...
https://github.com/milvus-io/milvus/issues/37244
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-05 17:12:23 +08:00
9a0e1c82bc
fix: repeated error code in milvus and segcore ( #37359 )
...
issue: #37357
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-05 16:28:23 +08:00
0c4321cf57
fix: crash when startup if the milvus volume is on-operation concurrently ( #37312 )
...
issue: #37311
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-04 14:50:23 +08:00
c27f477b6c
enhance: Update Knowhere version ( #37333 )
...
issue: #37269
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-04 11:56:31 +08:00
51cb2fbf97
fix: parse fail in empty json ( #37294 )
...
#37200
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-03 19:00:21 +08:00
5aad000a93
enhance: [skip ci] remove tools/core_gen after #35251 ( #36306 )
...
`tools/core_gen` python scripts are useless after
https://github.com/milvus-io/milvus/pull/35251
fixes : #36305
Signed-off-by: Yinzuo Jiang <yinzuo.jiang@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-11-01 13:08:21 +08:00
cd2655c861
fix: fix wrong method is called to fetch variable valid data ( #37304 )
...
issue: https://github.com/milvus-io/milvus/issues/37147
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-01 01:52:20 +08:00
63843dce33
fix: Fix conan gdal building problem ( #37338 )
...
issue:https://github.com/milvus-io/milvus/issues/27576
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server ( #35990 )
...
issue:https://github.com/milvus-io/milvus/issues/27576
# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.
# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.
---------
Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
4bac2eb13e
enhance: Update Knowhere version ( #37315 )
...
Signed-off-by: Li Liu <li.liu@zilliz.com>
2024-10-31 17:24:20 +08:00
b8492498ac
fix: mask with valid data when preCheckOverflow ( #37221 )
...
#37175
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-31 10:44:26 +08:00
2092dc0ba1
enhance: reserve vector space to reduce reallocate cost in Views() and StringViews() ( #37182 )
...
issue: #37152
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-31 10:02:21 +08:00
7961568223
fix: Rectify `OffsetOrderedArray` contain logic ( #37305 )
...
Related to #36887
Remove non-hit pk delete record logic does not work since
`insert_record_.contain` does not work due to logic problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-30 21:26:19 +08:00
90948e9444
fix: add SearchOnSealed unit test and fix a bug ( #37241 )
...
issue: https://github.com/milvus-io/milvus/issues/37244
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-30 10:26:19 +08:00
86687bd8ed
enhance: Refine code for get_deleted_bitmap ( #36819 )
...
issue: #33744
Check whether the PK is truly sorted in the debug model.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-28 15:19:30 +08:00
b81f162f6a
fix: fix several bugs and refactor some codes related with chunked segment ( #37168 )
...
issue: https://github.com/milvus-io/milvus/issues/37147
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-28 14:17:30 +08:00
d7b2ffe5aa
enhance: add an unify vector index config checker ( #36844 )
...
issue: #34298
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-28 10:11:37 +08:00
a2f0092e39
fix: check sparse float before calling get_dim ( #37145 )
...
https://github.com/milvus-io/milvus/issues/37146
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-26 16:25:29 +08:00
44ddcb5a63
fix: not check has_value before get value in JSON ( #37128 )
...
https://github.com/milvus-io/milvus/issues/36236
also: https://github.com/milvus-io/milvus/issues/37113
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-25 17:19:28 +08:00
ff0b7ea0ef
enhance: build interim index for mmapped vector in ChunkedSealedSegment ( #36993 )
...
issue:https://github.com/milvus-io/milvus/issues/36392
related pr: https://github.com/milvus-io/milvus/pull/36391
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-10-25 15:55:28 +08:00
3628593d20
feat: Implement custom function module in milvus expr ( #36560 )
...
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org
Solutions:
- parser (planparserv2)
- add CallExpr in planparserv2/Plan.g4
- update parser_visitor and show_visitor
- grpc protobuf
- add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
- create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
- empty(string)
- starts_with(string, string)
- add cpp/go unittests and E2E tests
closes : #36559
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-10-25 15:25:30 +08:00
6ef014d931
fix: get correct size when sealed segment chunked ( #37062 )
...
#37019
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-25 12:01:31 +08:00
ad2df904c6
fix: correctly set ExecTermArrayVariableInField bitset result ( #37111 )
...
issue: https://github.com/milvus-io/milvus/issues/37110
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-24 18:52:02 -07:00
bf956a3ec2
fix: fix string field has invalid utf-8 ( #37104 )
...
issue: https://github.com/milvus-io/milvus/issues/37083
We use vector of string_view to save data temporally but real string
data will be released after record batch is deconstructed.
Change it to vector of string to avoid memory corruption.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-24 18:33:47 -07:00
2b3f5bec07
fix: panic when create index on all none data ( #37046 )
...
#37045
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-24 17:09:28 +08:00
8902e2220e
enhance: enable asan for cpp unittest ( #37041 )
...
https://github.com/milvus-io/milvus/issues/35854
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2024-10-23 17:21:27 +08:00
90b3907a92
fix: fix missing return value in chunked column ( #37064 )
...
issue: https://github.com/milvus-io/milvus/issues/36834
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-22 10:29:19 -07:00
5a1f752272
enhance: [bitset] multiple 'and' and 'or' in a single op ( #33345 )
...
issue #34117
* Refactoring
* Added a capability to perform multiple bitwise `and` and `or`
operations in a single op
* AVX2, AVX512, ARM NEON, ARM SVE backed bitwise `and`, `op`, `xor` and
`sub` ops
* more unit tests for bitset
* fixed a bug in `or_with_count` for certain bitset sizes
* fixed a bug for certain offset values for inplace operations that take
two bitsets
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-10-22 16:25:33 +08:00
6bedc7e8c8
fix: not set valid_data in bitmap index when mmap ( #37023 )
...
#37013
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-22 12:03:26 +08:00
346510ed23
enhance: Update Knowhere version ( #37000 )
...
Signed-off-by: foxspy <xian_hust@foxmail.com>
2024-10-21 11:39:26 +08:00
304098cd40
fix:Chunk Id out of range in vector BF search, after the growing index removes the vec chunks. ( #36939 )
...
issue: https://github.com/milvus-io/milvus/issues/36871
related pr: https://github.com/milvus-io/milvus/pull/36938
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-10-18 12:21:25 +08:00
903c18ba26
enhance: consider the mmap chunck cache config when resource usage estimate ( #36814 )
...
- issue: #36530
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-18 10:17:23 +08:00
3de57ec4fa
enhance: add vector index mgr to remove vector index type dependency ( #36843 )
...
issue: #34298
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-17 22:15:25 +08:00
eb3e4583ec
enhance: all op(Null) is false in expr ( #35527 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-17 21:14:30 +08:00
b474374ea5
enhance: use growingMmapEnabled to control the behavior of interim index, not vectorField ( #36500 )
...
issue:https://github.com/milvus-io/milvus/issues/36392
related pr: https://github.com/milvus-io/milvus/pull/36391
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-10-17 20:25:24 +08:00
b2037c95a8
fix: use chunk_row_nums to iterate ( #36882 )
...
Fix segmentation fault error and remove useless codes.
https://github.com/milvus-io/milvus/issues/36834
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-16 11:15:25 +08:00
9997c5de34
fix: remove excessive logging ( #36859 )
...
issue: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-16 10:47:22 +08:00
aa904be6ec
enhance: support sparse vector mmap in growing segment type ( #36566 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/36565
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-10-15 10:59:23 +08:00
f46c3acea9
fix: heap buffer overflow when unittest at index wrapper ( #36838 )
...
issue: #35852
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-14 18:13:22 +08:00
3a09b438c2
fix: fix macos code checker ( #36817 )
...
https://github.com/milvus-io/milvus/issues/36829
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-14 11:11:51 +08:00
e170991a10
[automated] Update Knowhere Commit ( #36823 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-13 01:21:20 +08:00
ef0c649bda
enhance: update knowhere version to support diskann iterator ( #36813 )
...
issue: #36812
Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-10-12 18:05:22 +08:00
a75bb85f3a
feat: support chunked column for sealed segment ( #35764 )
...
This PR splits sealed segment to chunked data to avoid unnecessary
memory copy and save memory usage when loading segments so that loading
can be accelerated.
To support rollback to previous version, we add an option
`multipleChunkedEnable` which is false by default.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-12 15:04:52 +08:00
db34572c56
feat: support load and query with bm25 metric ( #36071 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
b1e678dcba
fix: fix json in [] expr bug ( #36721 )
...
#36718
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-10-11 01:11:20 +08:00
f7b811450d
feat: add enable_tokenizer params to VarChar field ( #36480 )
...
issue: #35922
add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-10 20:33:21 +08:00
130a923dec
enhance: the estimate method when loading the collection ( #36307 )
...
- issue: #36530
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-09 17:35:19 +08:00
c3d910756b
enhance: Update knowhere commit to fix mac compilation ( #36706 )
...
Related to zilliztech/knowhere#879
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-09 16:05:20 +08:00
3936d12661
[automated] Update Knowhere Commit ( #36634 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-01 01:05:15 +08:00
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) ( #36214 )
...
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.
Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:
1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.
To address these limitations, This enhancement is needed.
Related Issue: #36212
2024-09-30 13:23:32 +08:00
94005b7198
fix: Sparse float vector incorrectly ExpandData at mmap mode ( #36603 )
...
issue: #36561
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-30 10:39:16 +08:00
8ed34dce84
enhance: Reopen chunk cache cpp ut ( #33622 )
...
issue: https://github.com/milvus-io/milvus/issues/33210
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 18:19:15 +08:00
9e3efa06be
fix:fix empty search result bug ( #36582 )
...
#36450
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-28 17:45:16 +08:00
0799d927c6
fix:fix term expr overflow bug ( #36525 )
...
#36520
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-26 15:01:14 +08:00
447e326629
[automated] Update Knowhere Commit ( #36527 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-26 01:15:13 +08:00
8495bc6bbc
fix: fix broken Sparse Float Vector raw data mmap ( #36183 )
...
issue: https://github.com/milvus-io/milvus/issues/36182
* improved `Column.h` to make the code much more readable and
maintainable, and added detailed comments.
* fixed an issue where `ArrayColumn::NumRows()` always returns 0 when
the mmap backing storage is a file.
* removed unused `ColumnBase` constructors and unnecessary members so we
don't get confused.
* Updated `test_chunk_cache.cpp` to make the tests parameterized: to
test both mmap enabled and disabled. Added sparse field in the test to
add coverage.
* re-enabled test `Sealed::GetSparseVectorFromChunkCache`.
* But 2 other disabled tests `Sealed::WarmupChunkCache` and
`Sealed::GetVectorFromChunkCache` remain disabled, there seems to be
errors. @bigsheeper PTAL.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-25 18:59:13 +08:00
8cda48a96a
enhance: Use mmap.scalarIndex config for text index ( #36400 )
...
issue: https://github.com/milvus-io/milvus/issues/35273
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-24 12:21:13 +08:00
167e4fb10d
[automated] Update Knowhere Commit ( #36352 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-19 01:01:10 +08:00
23b95aeba3
fix: remove element type check ( #35828 )
...
https://github.com/milvus-io/milvus/issues/36275
Array's element type is not same with schema's. It is INT32 for INT16
and INT8
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-09-18 11:37:10 +08:00
2ff3765058
enhance: catch std::stoi exception and improve error msg ( #36267 )
...
issue: #36255
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-14 16:17:08 +08:00
489087d18b
enhance: refactor executor framework V2 ( #35251 )
...
#32636
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-13 20:57:09 +08:00
58d3200986
enhance: Filter out non-hit delete records during load delta ( #36207 )
...
Related to #35303
This PR utilizes pk index in segment to exclude non-hit delete record
during load delete records. This ability is crucial when l0/delete
forward policy only replies on segment itself(without BF filtering).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-13 19:05:08 +08:00
f0f2fb4cf0
enhance: span tracing of c++ part ( #36205 )
...
fix: https://github.com/milvus-io/milvus/issues/36204
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-13 11:19:09 +08:00
5e5e87cc2f
enhance: rename some params and reduce default bitmapCardinalityLimit… ( #36138 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-12 12:09:08 +08:00
89bf226f0b
feat: support keyword text match ( #35923 )
...
fix : #35922
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-10 15:11:08 +08:00
53a8a24554
fix: fix empty indices of sparse float ( #35403 )
...
https://github.com/milvus-io/milvus/issues/35401
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-09-10 14:23:07 +08:00
851f3b9883
fix: Make legacy non-lexicographic branch break swtich ( #36125 )
...
Related to #35941
Previous PR: #36034
This patch makes the switch branching logic correct and make the unit
test work for cases which does not select the whole dataset.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 10:15:07 +08:00
3123093dd7
enhance: Use `MARISA_LABEL_ORDER` when building trie index ( #36034 )
...
Related to #35941
Previous PR: #35943
This PR make `Trie` index using `MARISA_LABEL_ORDER`, which make
predictive search iterating in lexicographic order.
When trie index is build in label order, lexicographc could be utilized
accelerating `Range` operations.
However according to the official document, using `MARISA_LABEL_ORDER`
will make "exact match lookup, common prefix search, and predictive
search" slower.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 14:29:05 +08:00
a103dd5eb3
enhance: Fix SearchOnSealed clang-format lint ( #36056 )
...
Related to #36008
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-06 16:47:04 +08:00
21b135c7c2
fix: not append valid data when transfer to insert record ( #36027 )
...
fix not append valid data when transfer to insert record and add a tiny
check when in groupBy field.
#35924
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-06 14:53:04 +08:00
5247631289
fix: fill the metric type field in the LoadMetaInfo object ( #35962 )
...
- issue: #35960
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-05 20:50:23 -07:00
11325d9ed5
fix: binary arith expression on inverted index ( #35945 )
...
issue: https://github.com/milvus-io/milvus/issues/35946
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-05 20:01:05 +08:00
560e8e70b0
enhance: reduce mmap_rss after chunkcache warmup ( #35974 )
...
related pr: https://github.com/milvus-io/milvus/pull/35965
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-09-05 18:07:05 +08:00
c61eea737b
enhance: Fix trace.cpp lint format issue ( #36004 )
...
Introduced by #35928
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:33:04 +08:00
7b21032d19
fix: Check all values for `trie.predictive_search` ( #35943 )
...
Related to #35941
For marisa trie `predictive_search` default behavior, it value iterated
is not in lexicographic order.
This PR is a brute force fix to make range operator returns correct
values.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 15:01:04 +08:00
9e96ed4873
fix: Fix tracing config update logic ( #35928 )
...
Related to #35927
There are serveral issue this PR addresses:
- Use `ResetTraceConfig` method instead init one in update event handler
- Implement dynamic stats.Handler to receive tracing config update event
- Update `enable_trace` flag when `ResetTraceConfig` is invoked
- Change `enable_trace` to `std::atomic<bool>` in case of data race
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 14:27:04 +08:00
cc02dc0a55
fix: Handle Input/Output Errors in vsnprintf and snprintf ( #35898 )
...
Fix for Issue: #35897
2024-09-04 08:15:04 +08:00
9da86529a7
enhance: Add disk filemananger parallel load control to reduce the memory consumption ( #35281 )
...
issue: #35280
add parallel control to limit the memory consumption during index file
loading
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-09-03 18:01:03 +08:00
f68df9a11e
fix: SkipIndex cause segment fault ( #35907 )
...
issue: #35882
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-03 17:15:03 +08:00
74048ce34f
fix:rename mmap file path to avoid directory conflict ( #35810 )
...
#35784
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-03 16:05:03 +08:00
4641fd9195
enhance: make search groupby stop when reaching topk groups ( #35814 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-02 18:25:03 +08:00
b2eb9fe2a7
fix: memory leak in unittest and open the USE_ASAN option when build unittest ( #35855 )
...
issue: #35854
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-02 15:59:04 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
576ac2bbed
fix: Fix the reference to a variable after it has been moved ( #35875 )
...
#35607
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 10:05:02 +08:00
5ea2454fdf
feat: tantivy tokenizer binding ( #35801 )
...
fix : #35800
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-01 17:13:03 +08:00
671112d17b
enhance: add more info to hybrid index log ( #35808 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-29 21:07:04 +08:00
a3f2f044d6
fix: not set nullable when stream writer write headers ( #35799 )
...
#35802
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 20:59:00 +08:00
b3089b5bdc
feat: support range search pagination retains order ( #35738 )
...
issue: #35464
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-29 14:09:00 +08:00
b51b4a2838
fix: try get not exist file after upgrade ( #35740 )
...
https://github.com/milvus-io/milvus/issues/35741
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 11:09:01 +08:00
9b96841ae9
fix: wrong construction in evalctx ( #35772 )
...
issue: #35771
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-28 19:21:00 +08:00
a52ba3d09d
enhance: allow many segments for inverted index ( #35616 )
...
fix: https://github.com/milvus-io/milvus/issues/35615
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-28 11:30:59 +08:00
98866205fa
fix: munmap deallocate too much memory ( #35725 )
...
issue: #35693
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-27 17:18:59 +08:00
4d2f96c760
enhance: support bitmap mmap ( #35399 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-27 16:34:59 +08:00
6ddfd02f01
[automated] Update Knowhere Commit ( #35688 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-26 01:04:57 +08:00
congqixia
Xianhui Lin
presburger
cai.zhang
cqy123456
Buqian Zheng
foxspy
Gao
zhagnlu
Bingyi Sun
sparknack
Spade A
smellthemoon
Zhen Ye
Patrick Weizhi Xu
aoiasd
sre-ci-robot
Chun Han
Ted Xu
tinswzy
Yinzuo Jiang
zhenshan.cao
Hao Tan
liliu-z
Yinzuo Jiang
yellow-shine
Alexander Guzhva
foxspy
SimFG
Min Tian
Rijin-N
yihao.dai
jaime
Jiquan Long
Abdullah Ahmed