80dbe87759
enhance: support null value in index ( #35238 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-16 15:30:54 +08:00
f4a91e135b
enhance: Allow empty sparse row ( #34700 )
...
issue: #29419
* If a sparse vector with 0 non-zero value is inserted, no ANN search on
this sparse vector field will return it as a result. User may retrieve
this row via scalar query or ANN search on another vector field though.
* If the user uses an empty sparse vector as the query vector for a ANN
search, no neighbor will be returned.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-08-16 14:14:54 +08:00
b896143965
enhance: Improve bitset performance for AVX512 ( #35479 )
...
see #35478
optimized functions take 20%+ less time to run
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-08-16 07:44:53 +08:00
d20d6ea551
fix: Functional-notation casting vulnerability fix ( #35252 )
...
Fix for issue: https://github.com/milvus-io/milvus/issues/35200
2024-08-15 16:20:53 +08:00
b1ea645390
[automated] Update Knowhere Commit ( #35474 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-15 01:14:53 +08:00
e1d725b938
fix: rollback grpc to 1.50.1 ( #35385 )
...
issue: #35298
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-09 14:08:17 +08:00
626b1b2f5e
fix:redefine hybrid internal index type ( #35314 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-08 10:32:16 +08:00
7420115b5e
enhance: pinpoint revision of 3thirdparty package ( #35325 )
...
issue: https://github.com/milvus-io/milvus/issues/35340
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-08 09:46:22 +08:00
1d5bf34e38
fix: init telemetry fails when set otlp as trace exporter ( #35301 )
...
issue: #35298
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-07 14:20:21 +08:00
c19fe95154
fix: support string match for hybrid and bitmap index ( #35294 )
...
#34841
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-07 09:54:22 +08:00
e75efa7359
[automated] Update Knowhere Commit ( #35328 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-07 01:00:16 +08:00
3c9a47c8db
feat: Encode traceID and spanID as hex string ( #34807 )
...
Issue: https://github.com/zilliztech/knowhere/pull/714
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-08-06 15:20:16 +08:00
91df03afe8
feat: put inverted index into ram ( #35222 )
...
fix: https://github.com/milvus-io/milvus/issues/35224
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-06 11:54:16 +08:00
4b553b0333
enhance: revert remove duplicated pk function ( #35103 )
...
issue: #34778
Revert "fix: fix query count(*) concurrently"
Revert "enhance: mark duplicated pk as deleted "
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-05 10:48:17 +08:00
16dd53e7cf
enhance: remove timestamp_filter after retrieve ( #35207 )
...
#35226
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-02 19:32:46 +08:00
475c333fa2
enhance: add valid_data in span ( #35030 )
...
#31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-02 15:40:14 +08:00
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue ( #35166 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 16:52:13 +08:00
f8c1b138a8
fix:fix get array error for int type ( #35154 )
...
#35055
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-01 14:30:12 +08:00
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
f229f244d2
enhance: add chunk basic impl ( #34634 )
...
https://github.com/milvus-io/milvus/issues/35112
This pr would not affect milvus functionality by now.
It implments a Chunk memory layout that looks like
```
VariableColumn
|offset|offset|offset|
|data|data|data|
```
We maybe move offsets to the beginning and add null bitmaps later but
not in this PR.
And mmap test will also be added in another PR.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-01 10:29:51 +08:00
c15d074b21
[automated] Update Knowhere Commit ( #35157 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-01 01:17:49 +08:00
48b14c56f6
fix: add prometheus dependency for milvus_monitor ( #35078 )
...
add prometheus dependency for monitor module. Or else Some compilers may
report a compilation failure.
issue: #35077
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-31 10:15:48 +08:00
dd0c26cf58
enhance: redefine variable column block size ( #35040 )
...
#35013
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-30 19:23:50 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
a8a4779749
fix: fix parse plan proto failed for search type ( #34944 )
...
#25848
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 21:19:49 +08:00
86322e0468
fix: fix query count(*) concurrently ( #35007 )
...
#34778
#34849
fix two problems:
1. count(*) incorrect, if growing insert duplicated (pk, timestamp)
pairs that pk and timestamp all same, need to keep just one pair.
2. count(*) may core dump, if get_real_count interface get snapshot and
do mvcc at not consistency status, mainly happens under concurrency.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 19:53:50 +08:00
972752258a
enhance: Support otlp http exporter ( #35053 )
...
See also #35052
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 17:43:49 +08:00
f77f5364b2
fix: disable use_index when some array expr ( #34894 )
...
#34797
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 00:17:46 +08:00
9463eeef2b
fix: Avoided dereferencing NULL pointer ( #34836 )
...
issue : https://github.com/milvus-io/milvus/issues/34835
Signed-off-by: Ald392 <imagesai32@gmail.com>
2024-07-27 17:27:52 +08:00
804ec24c02
fix:fix retrieve raw data from bitmap array index ( #34848 )
...
#34795
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-27 01:53:47 +08:00
e93687cdd3
[automated] Update Knowhere Commit ( #35039 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-27 00:59:46 +08:00
7ec9d856f3
fix: access address was not malloc ( #34971 )
...
issue: #34972
fix string type data use memcpy to fill cause segv for not malloc enough
memory in advance.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-25 02:31:44 +08:00
5616b7e8d2
enhance: support null in c data_datacodec and load null value ( #32183 )
...
1. support read and write null in segcore
will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3. retrieve valid_data.
parse valid_data in search/query.
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-23 16:07:51 +08:00
a4aed9b0b5
enhance: new knowhere param for range_search ( #34686 )
...
issue: #34685
knowhere needs a new json param `range_search_k` for RangeSearch to
early terminate the iterator.
Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-07-23 11:45:43 +08:00
12b49859c0
[automated] Update Knowhere Commit ( #34909 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-23 00:53:43 +08:00
6c19f9baf8
enhance: optimize search reduce perf( #32507 ) ( #34607 )
...
related: #32507
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-22 20:57:43 +08:00
ed057e6fce
fix: non-init seg_offset for growing raw-data when doing groupby ( #34748 )
...
related: #34713
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-19 17:01:40 +08:00
8e64bf929c
enhance: add scalar filtering and vector search latency metrics ( #34785 )
...
add scalar filtering and vector search latency metrics to distinguish
the cost of scalar filtering.
To add metrics in query chain, add a monitor module and move the metric
files from original storage module.
issue: #34780
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-19 14:01:39 +08:00
c61592dcea
[automated] Update Knowhere Commit ( #34819 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-19 02:09:42 +08:00
f1b2f7b640
enhance: refactor bitmap index and internal hybrid index ( #34450 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-18 10:39:42 +08:00
aa5418a5a9
fix: correctly set search params when using knowhere iterator ( #34731 )
...
issue: #34730
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-17 15:13:41 +08:00
804dd5409a
enhance: mark duplicated pk as deleted ( #34586 )
...
fix #34247
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-16 14:25:39 +08:00
bd9727a1f7
fix: fix bug that set incorrect info to columnbase ( #34428 )
...
#34427
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-14 22:27:46 +08:00
18c83c6466
fix: fix auto merge error ( #34661 )
...
#33704
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-13 13:07:37 +08:00
f00c529aea
feat: support group_size for search_group_by( #33544 ) ( #33720 )
...
related: #33544
mainly changes in three aspects:
1. enable setting group_size for group by function
2. separate normal reduce and group by reduce
3. eleminate uncessary padding in search result for reducing
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-12 10:17:36 +08:00
104d0966b7
feat: support partition key isolation ( #34336 )
...
issue: #34332
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
4850336ca3
fix: Write padding at end of mmap file not chunk ( #34529 )
...
Related to #34508
The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 11:12:14 +08:00
0b3f317b14
[automated] Update Knowhere Commit ( #34549 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-10 01:26:14 +08:00
6b4d977a10
fix: Write padding into mmap file in case of SIGBUS ( #34443 )
...
See also #34442
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-05 17:44:09 +08:00
d65b6895d7
[automated] Update Knowhere Commit ( #34432 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-05 01:18:09 +08:00
ec5db7a57d
enhance: update knowhere version ( #34361 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-02 19:18:24 +08:00
734415b8a2
fix: Reduce duplicate PKs in segcore ( #34267 )
...
issue: https://github.com/milvus-io/milvus/issues/34247
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-01 17:42:06 +08:00
ef3ced8138
fix: descriptor event in previous version not has nullable to parse error ( #34235 )
...
#34176
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-01 16:38:06 +08:00
14e827dc6c
fix: Implement singleflight for segcore ChunkCache ( #34250 )
...
See also #34249
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 11:46:06 +08:00
3030e4625e
enhance: refactor variable column to reduce memory cost ( #33875 )
...
#33874
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:16:06 +08:00
cc1bc07bfd
enhance: add log to bitmap index ( #34197 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:02:06 +08:00
ad90360162
enhance: Update knowhere commit ( #34223 )
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-27 18:20:06 +08:00
a933f6731b
fix: centroids file not removed when data skew in major compaction ( #34050 )
...
issue: https://github.com/milvus-io/milvus/issues/30633
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-26 10:48:04 +08:00
cc54ab2e47
[automated] Update Knowhere Commit ( #34172 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-26 02:20:03 +08:00
254b555d53
[automated] Update Knowhere Commit ( #34119 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-25 01:22:02 +08:00
aa36f9feed
fix: [ut] regex query under unsupported index ( #34087 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/29988
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-24 14:08:03 +08:00
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
03a3f50892
enhance: add skip using array index when some situation ( #33947 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:26:02 +08:00
0d7ea8ec42
enhance: Enhance and correct exception module ( #33705 )
...
#33704
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:22:01 +08:00
259a682673
enhance: async search and retrieve in cgo ( #33228 )
...
issue: #30926 , #33132
related pr: #33133
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-22 09:38:02 +08:00
d9f986439d
[automated] Update Knowhere Commit ( #34040 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-21 01:50:01 +08:00
dac20d41c0
fix: fix milvus gpu compile error ( #33995 )
...
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: https://github.com/milvus-io/milvus/issues/33991
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-20 20:30:01 +08:00
dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk ( #33993 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-20 17:42:02 +08:00
2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
7b9462c0d3
enhance: fix copying hits of inverted index twice ( #33968 )
...
issue: https://github.com/milvus-io/milvus/issues/29793
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-19 12:40:01 +08:00
298e50b834
enhance: check index with data type ( #33880 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
0d20303e54
fix: fix binary vector data size ( #33750 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
- fix byte size wrong for binary vectors
- fix the expect/actual error msg
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-18 21:39:59 +08:00
b460862537
fix: can't find Chunk struct after growing support mmap ( #33951 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 18:37:58 +08:00
3fdaae8792
fix: Return record with largest timestamp for entires with same PK ( #33936 )
...
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
e83ecd5074
enhance: fix not find arrow::parquet lib ( #33928 )
...
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-18 00:22:01 +08:00
d43ec4db0b
enhance: support array bitmap index ( #33527 )
...
#32900
---------
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:51:58 +08:00
e422168f09
fix: readd timestamp index because segment timestamp not ordered ( #33856 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:48:01 +08:00
7d74edd6dd
fix: update clang-tidy and clang-format from 10 to 12 ( #33141 )
...
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.
issue: #33142
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-06-13 15:27:58 +08:00
47b04ea167
enhance: support sparse cardinal hnsw index ( #33656 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 16:57:55 +08:00
8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector ( #33209 )
...
issue: #29419
* sparse float vector to support raw data mmap
For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 10:09:55 +08:00
ecf2bcee42
enhance: speed up array-equal operator via inverted index ( #33633 )
...
fix : #33632
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-11 14:13:54 +08:00
f53ab54c5d
enhance: async cgo utility ( #33133 )
...
issue: #30926 , #33132
- implement future-based cgo utility.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-09 22:55:53 +08:00
6ce9df913f
fix: clean vector memory ( #33692 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-07 10:37:54 +08:00
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
b69740c8f3
enhance: Remove unnecessary log info during load segment ( #33663 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-06 14:13:50 +08:00
fd191dd7db
[automated] Update Knowhere Commit ( #33655 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-06 01:53:50 +08:00
703fc73f71
enhance: disk index support binary vector ( #33631 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-05 19:37:57 +08:00
8ad26093ba
fix: fix load failure ( #33599 )
...
issue: #33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-05 19:19:51 +08:00
545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector ( #33377 )
...
related #22837
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-05 14:31:57 +08:00
597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit ( #33609 )
...
See also milvus-io/milvus-sdk-go#756
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 11:51:51 +08:00
c6f8a73bb2
enhance: optimize some cache to reduce memory usage ( #33534 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-04 14:09:47 +08:00
d25c755480
[automated] Update Knowhere Commit ( #33573 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-04 01:55:46 +08:00
2b285e5573
fix: Wrap init segcore tracing with golang timeout ( #33494 )
...
See also #33483
Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-03 19:25:51 +08:00
4171414222
enhance: update knowhere version ( #33490 )
...
issue: https://github.com/milvus-io/milvus/issues/33489
update knowhere version to latest. remove usage of `seed_ef` as it be
replaced by existing `ef`.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-31 10:17:50 +08:00
0c5d8660aa
feat: support inverted index for array ( #33452 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-31 09:47:47 +08:00
416a2cf507
fix: query iterator lack results( #33137 ) ( #33422 )
...
related: #33137
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
589d4dfd82
enhance: optimize bitmap index ( #33358 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
9f9dff33f1
fix: make milvus compilable using gcc-13 ( #30149 )
...
add a missing header
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-27 14:52:53 +08:00
5cf4161394
fix: Fix exception info is missing ( #33393 )
...
Replace based std::exception to prevent "object slicing"
issue: https://github.com/milvus-io/milvus/issues/33392
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-27 14:33:41 +08:00
b391781a2e
[automated] Update Knowhere Commit ( #33340 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-24 01:43:40 +08:00
3bec2c4928
enhance: Update Knowhere version ( #33309 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-23 16:57:41 +08:00
c5918ffbdb
enhance: mark sparse inverted index as mmap-able ( #33281 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-23 14:11:42 +08:00
32d3e22d7d
fix: Throw an exception after all the threads in thread pool finished ( #32810 )
...
issue: #32487
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 11:47:40 +08:00
be77ceba84
enhance: Use proto for passing info in cgo ( #33184 )
...
issue: #33183
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 10:31:40 +08:00
3c4df81261
enhance: Assert insert data length not overflow int ( #33248 )
...
When InsertData is too large for cpp proto unmarshalling, the error
message is confusing since the length is overflowed
This PR adds assertion for insert data length.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-22 15:11:39 +08:00
648d5661ca
enhance: Upgrade bitset for ARM SVE ( #32718 )
...
issue: #32826
improve ARM SVE performance for `internal/core/src/bitset`
Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83
Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-22 11:37:40 +08:00
c6e2dd05fc
[automated] Update Knowhere Commit ( #33147 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-20 01:51:37 +08:00
d669fbcf46
enhance: support bitmap index for scalar type ( #32902 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-19 21:49:38 +08:00
7c60d725cc
fix: validate sparse vector in search request ( #32856 )
...
issue: #32368
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-15 15:39:33 +08:00
3777eb2f68
enhance: update knowhere new version ( #33043 )
...
for milvus new release, knowhere update to latest release version
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-05-14 20:45:32 +08:00
09cca1fb51
[automated] Update Knowhere Commit ( #33030 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-14 01:47:32 +08:00
f6777267e3
enhance: add score compute consistency config for knowhere ( #32997 )
...
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-13 14:21:31 +08:00
f7d29118a5
enhance: update knowhere version ( #32931 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-05-10 16:13:31 +08:00
bb7765cbd6
fix: fix Indexing.Iterator ut: build index with all data at once ( #32844 )
...
issue: #32843
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-10 11:31:30 +08:00
54a51b1236
enhance: Support dynamic config for opentelemetry trace ( #32169 )
...
relate: https://github.com/milvus-io/milvus/issues/31940
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-09 17:43:30 +08:00
3d78b90fe7
[automated] Update Knowhere Commit ( #32872 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-09 01:57:30 +08:00
035a508722
fix: make sure inverted index has only one segment ( #32858 )
...
issue: #32717
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-08 21:25:30 +08:00
01c2684355
enhance: [skip e2e] disable unstable ut temporarily ( #32836 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-08 12:17:29 +08:00
9837ad6a8d
enhance: remove deprecated api ( #32808 )
...
issue: #32728
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 20:03:30 +08:00
1f58cda957
enhance: add more trace for search & query ( #32734 )
...
issue: https://github.com/milvus-io/milvus/issues/32728
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 13:03:29 +08:00
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
a43baa0323
enhance: Upgrade knowhere to 2.3.1 ( #32699 )
...
/kind improvement
Signed-off-by: Li Liu <li.liu@zilliz.com>
2024-04-29 17:21:26 +08:00
6289f3a9eb
fix: build milvus in rockylinux8 ( #32619 )
...
issue: #32299
1. xz utils recovers
2. forget to install ninja
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-29 14:53:26 +08:00
858599d831
enhance: sparse float vector to support brute force iterator and range search ( #32635 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-29 14:35:26 +08:00
cb1dbf20c7
enhance:remove openblas from conan ( #32002 )
...
issue: #30683
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-28 20:13:25 +08:00
46d7298407
fix: file not find test_parquet.cpp ( #32644 )
...
#32646
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-04-26 16:57:27 +08:00
4fb8044a27
enhance: delete some no lint code ( #32182 )
...
#31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-04-26 14:15:26 +08:00
ccce1e928a
fix: regex query can't handle text with newline ( #32569 )
...
issue: https://github.com/milvus-io/milvus/issues/32482
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-26 12:01:26 +08:00
c002745902
enhance: retrieve output fields after local reduce ( #32346 )
...
issue: #31822
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-25 09:49:26 +08:00
b703d6cdbe
[automated] Update Knowhere Commit ( #32586 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-25 02:27:24 +08:00
29922829da
fix: [restful v2] parameters of create collection ( #32485 )
...
issue: #31176
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-24 09:43:29 +08:00
15558112c5
[automated] Update Knowhere Commit ( #32516 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-23 01:33:23 +08:00
0fab265eed
enhance: update knowhere and some header changes ( #32468 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-04-22 15:47:26 +08:00
93f0c262c1
fix: add bitset sve support macro protect ( #32463 )
...
#32137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-19 18:05:21 +08:00
e2541f04eb
fix: crash on arm64 ( #31951 ) ( #31980 )
...
issue: #31953
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-17 18:03:20 +08:00
9248a6a149
fix: remove sve flags ( #32270 )
...
#32129
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-16 15:21:19 +08:00
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData ( #32012 )
...
issue: #30361
- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.
- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`
- Remove repeat load delta log operation in lru.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-16 11:17:20 +08:00
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
337cc0756d
fix: lack good results for insufficient ef( #29883 ) ( #32080 )
...
related: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-13 22:13:23 +08:00
4fb85be525
fix: put inverted index into local storage ( #32209 )
...
issue: https://github.com/milvus-io/milvus/issues/32154
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-13 21:57:19 +08:00
454984aa4e
[automated] Update Knowhere Commit ( #32181 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-12 01:25:19 +08:00
b5455d176e
fix: dynamically resolve whether SVE is available for bitset ( #32137 )
...
Issue: #32129
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-11 19:27:18 +08:00
52ae47c850
enhance: gather materialized view search info once per request ( #31996 )
...
issue: #29892
This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-11 15:21:19 +08:00
f3f2a5a7e9
fix: evicted segments in the serverlss mode( #31959 ) ( #31961 )
...
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-10 15:15:19 +08:00
a0a4ec8b67
enhance: make range search param check message more meaningful ( #32006 )
...
Issue: #31970
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-09 16:17:26 +08:00
1b767669a4
enhance: Throw error instead of crash when index cannot be built ( #31844 )
...
issue: #27589
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-04-09 11:51:18 +08:00
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug ( #31977 )
...
issue: #30361
- Add configurable disk capacity limit
- fix bitset reset logic
- make insert record reinsert after clear
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. ( #31771 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-07 14:13:16 +08:00
cae5722229
enhance: performance improvements for the bitset ( #31753 )
...
Issue: #31752
This PR improves the performance for bitset utilities (introduced in PR
#30454 ), including varchar filtering
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-06 05:19:22 +08:00
b2669e26dc
fix:reduce thread pool test time ( #31893 )
...
#31877
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-05 10:05:12 +08:00
d6d3b01a04
fix:remove thread pool timeout test because of high load cpu ( #31879 )
...
#31877
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-03 15:55:38 +08:00
03e0db109e
fix: udpate Cargo.lock ( #31859 )
...
issue: #31681
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-03 14:18:23 +08:00
246586be27
enhance: Unify data type check APIs under internal/core ( #31800 )
...
Issue: #22837
Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-02 19:15:14 +08:00
d299fa502e
fix: use milvus-io/vcpkg ( #31770 )
...
issue: #31769
GitHub Disables The XZ Repository because of CVE-2024-3094
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-01 15:01:13 +08:00
5655ec4fc0
enhance: add mmap usage metrics ( #31708 )
...
issue: #31707
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 11:35:12 +08:00
3ffe126dc7
enhance: Refine error message when search vector type not matched ( #31725 )
...
Previously the error message only reports the case happened without
field name and vector type.
This PR add field name and vector type information in the error
messages.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-01 10:11:12 +08:00
675a5dc822
fix: Save traceID and spanID as std::vector into search config ( #31278 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-29 14:29:11 +08:00
9750e78f1d
enhance: lock tantivy dependencies ( #31688 )
...
issue: https://github.com/milvus-io/milvus/issues/31681
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-29 10:15:17 +08:00
b99c46246c
enhance: ban groupby on binary vector( #31134 ) ( #31659 )
...
related: #31134
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-28 15:19:10 +08:00
e33dba8afe
fix: [skip-e2e] use zstd-sys 2.0.9 ( #31682 )
...
fix : #31681
/kind improvement
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-28 15:14:10 +08:00
b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
4eb4df1e81
fix: predict inverted index resource usage more reasonably ( #31615 )
...
/kind improvement
issue: #31617
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-27 19:33:09 +08:00
655097f171
fix: Verify PlaceHolderValue type before search ( #31626 )
...
See also #31625
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-27 17:49:10 +08:00
5be395354c
fix: minio ssl compatible issue ( #31607 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-03-27 14:41:20 +08:00
678cb187e8
[automated] Update Knowhere Commit ( #31630 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-27 01:15:10 +08:00
659ad81ab7
fix: remove deprecated ut test ( #31499 )
...
#31498
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-26 14:01:07 +08:00
c4b37fb285
enhance: Custom bitset and bitsetview prototypes ( #30454 )
...
Issue: #31285
Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.
This PR is for internal use only.
Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[ OK ] CApiTest.AssembeChunkPerfTest (183 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[ FAILED ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[ OK ] Expr.TestExprs (9 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 124.8us
start test type:3
cost: 163.1us
start test type:4
cost: 275.9us
start test type:5
cost: 590.9us
start test type:10
cost: 62.7us
start test type:11
cost: 65.9us
[ OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 151.4us
start test type:3
cost: 198.4us
start test type:4
cost: 361.9us
start test type:5
cost: 753.9us
start test type:10
cost: 64.6us
start test type:11
cost: 62.2us
[ OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 121.14us
start test type:3
cost: 156.84us
start test type:4
cost: 249.76us
start test type:5
cost: 534.44us
start test type:10
cost: 82.2us
start test type:11
cost: 83.52us
[ OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 80.64us
start test type:3
cost: 78.22us
start test type:4
cost: 255.76us
start test type:5
cost: 532.04us
start test type:10
cost: 89.26us
start test type:11
cost: 90us
[ OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 401.7us
start test type:3
cost: 420.96us
start test type:4
cost: 418.04us
start test type:5
cost: 470.54us
start test type:10
cost: 250.32us
start test type:11
cost: 850.08us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 162us
start test type:3
cost: 142us
start test type:4
cost: 374us
start test type:5
cost: 674us
start test type:10
cost: 366us
start test type:11
cost: 645us
[ OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[ OK ] Expr.TestRefactorExprs (1142 ms)
```
Candidate:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[ OK ] CApiTest.AssembeChunkPerfTest (153 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[ OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[ OK ] Expr.TestExprs (8 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 55.7us
start test type:3
cost: 79.8us
start test type:4
cost: 177.6us
start test type:5
cost: 337.2us
start test type:10
cost: 16.9us
start test type:11
cost: 15.7us
[ OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 57.1us
start test type:3
cost: 87us
start test type:4
cost: 177.5us
start test type:5
cost: 342.7us
start test type:10
cost: 17.9us
start test type:11
cost: 16.7us
[ OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 34.58us
start test type:3
cost: 68.86us
start test type:4
cost: 151.38us
start test type:5
cost: 286.8us
start test type:10
cost: 16.54us
start test type:11
cost: 16.7us
[ OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 20us
start test type:3
cost: 17.1us
start test type:4
cost: 154.12us
start test type:5
cost: 286.1us
start test type:10
cost: 19.6us
start test type:11
cost: 19.24us
[ OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 125.7us
start test type:3
cost: 111.34us
start test type:4
cost: 148.02us
start test type:5
cost: 306.7us
start test type:10
cost: 149.3us
start test type:11
cost: 282.94us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 89us
start test type:3
cost: 79us
start test type:4
cost: 323us
start test type:5
cost: 629us
start test type:10
cost: 313us
start test type:11
cost: 591us
[ OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[ OK ] Expr.TestRefactorExprs (1149 ms)
```
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-03-24 21:49:07 +08:00
982dd2834b
enhance: add materialized view search info ( #30888 )
...
issue: #29892
This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.
## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.
## Detailed MV Info
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-03-21 11:19:07 +08:00
c81909bfab
enhance: Support MinIO TLS connection ( #31311 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
cf5109ec17
fix: fix mmap failed when string field all value is empty ( #31406 )
...
#31162
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-21 10:41:07 +08:00
66d679ecbb
fix: clear binlog files in CleanData ( #31039 )
...
issue: https://github.com/milvus-io/milvus/issues/31042
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-20 11:11:07 +08:00
6edd06083f
chore: remove repetitive words ( #31153 )
...
Signed-off-by: gcmutator <329964069@qq.com>
2024-03-20 10:17:07 +08:00
b35ecebcc3
enhance: Update Knowhere version ( #31392 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-19 15:11:07 +08:00
d7727dd087
enhance: fetch simdjson directly in the format of targz ( #31369 )
...
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-03-18 18:55:11 +08:00
1c930e560c
enhance: Update Knowhere version ( #31312 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-18 11:29:04 +08:00
038c570ef3
enhance: upgrade folly to run on arm ( #31284 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-03-15 15:39:03 +08:00
6939ad15f2
fix:possible out-of-bound due to groupby when reduing( #30711 ) ( #31200 )
...
related: #30711
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-14 13:07:03 +08:00
7fc3094a42
fix: fix growing index data race and properly handle build error ( #31170 )
...
issue: https://github.com/milvus-io/milvus/issues/31169
also properly handling index build error by re-create a new index so
that nothing will be left in the previous failed index build attempt.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 20:19:04 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
c8b54f321a
fix:restrict pk in [...] optimization situations ( #31184 )
...
#31154
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-12 14:49:03 +08:00
6a83f16871
feat: Support for multiple forms of JSON ( #31052 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-11 19:55:02 +08:00
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building ( #30357 )
...
This commit adds sparse float vector support to segcore with the
following:
1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built
This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
a99143dd52
fix: Save traceID and spanID as hex string into search config ( #31071 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-11 14:21:01 +08:00
53af6d8c59
[automated] Update Knowhere Commit ( #31151 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-09 01:55:02 +08:00
122981aeb9
fix: Disable knowhere trace as a quick fix ( #31055 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-08 15:27:01 +08:00
3574bdf858
enhance: ban range-search iteration for search-group-by ( #30824 )
...
related: #30033
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-08 14:17:00 +08:00
19c64067af
enhance: jemalloc aarch64 platform use 64k pagesize. ( #29522 )
...
enhance: jemalloc aarch64 platform use 64k pagesize.
issue: #28843
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2024-03-07 21:01:01 +08:00
2d9de233fc
[automated] Update Knowhere Commit ( #31089 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-07 12:05:02 +08:00
c047f09110
[automated] Update Knowhere Commit ( #31015 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-05 16:31:00 +08:00
4bda6c33ad
fix: binary vector should not limit dimension to 32768 ( #30676 )
...
all the vector dimension check should happen on collection creation but
not index build
fix #30285
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-05 14:21:00 +08:00
3dc5e38240
[automated] Update Knowhere Commit ( #30989 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-04 16:34:59 +08:00
29f44f840a
enhance: refine groupBy error msg( #29968 ) ( #30920 )
...
related: #29968
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-01 18:53:03 +08:00
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
e2f35954d4
enhance: support pattern matching on json field ( #30779 )
...
issue: https://github.com/milvus-io/milvus/issues/30714
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
16b785e149
enhance: optimize the memory usage and speed up loading variable length data ( #30787 )
...
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
4459078e0b
fix: wrong num_entities used when mmap variable length data ( #30848 )
...
https://github.com/milvus-io/milvus/issues/30728
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
a115b731ed
enhance: fix old pr cpp format issue ( #30894 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
f658dd5faa
enhance: update knowhere version to 60a5c9c ( #30788 )
...
/kind improvement
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-02-28 14:18:55 +08:00
8a219e0102
feat: Support knowhere trace using OpenTelemetry ( #30750 )
...
Issue: #21508
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
6e9f3ea531
[automated] Update Knowhere Commit ( #30744 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-28 10:50:57 +08:00
57397b1307
enhance: add new LRU cache impl ( #30360 )
...
- remove the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache
related #30361
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
3e82d21ca1
enhance: reduce 1x memory copy when loading json ( #30753 )
...
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-27 10:18:55 +08:00
e2330f02f8
fix: pattern match use incorrect raw data ( #30764 )
...
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
77eb6defb1
feat: support groupby on growing and non-indexed sealed egment( #30307 ) ( #30644 )
...
related: #30308
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-21 14:02:53 +08:00
18aac076de
fix: move test from NEON to X86 ( #30324 )
...
#26137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 11:58:53 +08:00
0118bef2a2
fix: replace sse2 simd interface with older version ( #30668 )
...
#30667
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 10:04:54 +08:00
976b6fc0e4
enhance: change opendal as compile configurable ( #30384 )
...
#30373
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-20 19:16:52 +08:00
b74673c147
enhance: calculate the accuracy memory usage while loading segment ( #30473 )
...
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.
Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-20 14:52:51 +08:00
43e8cd531d
enhance: Update Knowhere version ( #30675 )
...
issue: #30669
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-02-19 22:04:51 +08:00
18c351efa6
fix: Prevent ChunkCache use absolute path in All-in-one mode ( #30666 )
...
See also #30651
Append operator of `std::filesystem::path` will replace whole path when
the param of "/" operation is an absolute path.
In "All-in-one" mode, this shall cause ChunkCache removing the original
vector data file when building chunk cache during/after load procedure.
This PR changes the ChunkCache path generation logic to a separate
function in which will check whether the file path is absolute or not.
If the file path is absolute, it removes the root path prefix and return
concatenated file path.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-19 20:58:51 +08:00
5bb28a9ea4
enhance: Print out range_filter and radius when range search param check fail ( #30623 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-18 15:40:48 +08:00
a297baae9d
enhance: remove unused code ( #30601 )
...
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-02-13 10:26:47 +08:00
e8a6f1ea2b
fix: erase pk empty check when pk index replace raw data ( #30432 )
...
#30350
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-07 14:56:47 +08:00
92d1d744ae
fix: groupby results lack good results( #29883 ) ( #30428 )
...
related: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-06 17:08:34 +08:00
5449e862d5
fix: safety access unordered_map and remove some useless code excute ( #30504 )
...
issue: https://github.com/milvus-io/milvus/issues/30358 and
https://github.com/milvus-io/milvus/issues/30491
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-02-05 22:03:09 +08:00
ebbe32df9a
[automated] Update Knowhere Commit ( #30515 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-05 01:32:44 +08:00
a587450e56
enhance: [skip-e2e] disable asan ( #30498 )
...
fix : #30511
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-04 21:25:05 +08:00
20c9cfc587
[automated] Update Knowhere Commit ( #30487 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-04 01:23:04 +08:00
e549148a19
enhance: full-support for wildcard pattern matching ( #30288 )
...
issue: #29988
This pr adds full-support for wildcard pattern matching from end to end.
Before this pr, the users can only use prefix match in their expression,
for example, "like 'prefix%'". With this pr, more flexible syntax can be
combined.
To do so, this pr makes these changes:
- 1. support regex query both on index and raw data;
- 2. translate the pattern matching to regex query, so that it can be
handled by the regex query logic;
- 3. loose the limit of the expression parsing, which allows general
pattern matching syntax;
With the support of regex query in segcore backend, we can also add
mysql-like `REGEXP` syntax later easily.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-01 12:37:04 +08:00
5cf9bb236e
enhance: restful support import jobs ( #30343 )
...
issue: #28521 #29732
include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job
fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-31 17:57:04 +08:00
878c4c9463
enhance: limit the max pool size to 16 ( #30371 )
...
according to our benchmark, concurrency level 16 is enough to fully
utilize the object storage network bandwidth
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-31 14:13:06 +08:00
74cfba0249
enhance:limit binlog index rows num ( #30173 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
also relate issue: https://github.com/milvus-io/milvus/issues/30065
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-29 19:49:02 +08:00
0542a0e7dc
[automated] Update Knowhere Commit ( #30332 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-29 01:05:01 +08:00
aeb1e36f00
enhance: change plan desc log from info to debug ( #30304 )
...
#30172
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-28 16:04:38 +08:00
e9fdd2475d
fix: fix searchPlan metricType modified concurrently ( #30227 )
...
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-26 14:03:09 +08:00
116d0f20b8
fix: groupby bug for ut ( #30272 )
...
related: #29965
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-25 20:57:00 +08:00
c02fb64ad6
enhance: Allows proactive warming up of chunk cache ( #30182 )
...
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-25 19:55:39 +08:00
a27c0e86fd
enhance: reduce many I/O operations while loading disk index ( #30189 )
...
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file
this optimized this to open only once and continiously write all data.
This also makes it concurrent to load the files from object storage
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-25 15:23:02 +08:00
8c58d9af67
enhance: optimize marisa trie range search for performance ( #30079 )
...
#30078
#29986
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-25 10:07:00 +08:00
0907d76253
enhance: pass partition key scalar info if enabled when build vector index ( #29931 )
...
issue: #29892
Pass optional scalar IVF offsets to Cardinal
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
42bb4e37e5
fix:diskann search crash when search list = 9999999999 ( #30185 )
...
issue: https://github.com/milvus-io/milvus/issues/29020
Json can't not pass a max_int32 value to int32_t, so let knowhere check
value range by itself.
After fix this, pymilvus will report:
pymilvus.exceptions.MilvusException: <MilvusException: (code=65535,
message=fail to search on QueryNode 6: worker(6) query failed: => failed
to search: arithmetic overflow: param search_list_size should be at most
2147483647)>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-23 14:46:55 +08:00
6cf2f09b60
feat: Support tencent cloud object storage for milvus ( #30163 )
...
issue: #30162
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 11:28:56 +08:00
a77693aa19
enhance: convert the `GetObject` util to async ( #30166 )
...
This makes it much easier to use
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 19:20:57 +08:00
e967949cc5
[automated] Update Knowhere Commit ( #30120 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-22 18:40:54 +08:00
4436effdc3
enhance: support groupby based on scalar-index( #29965 ) ( #30091 )
...
related: #29965
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 10:50:54 +08:00
aee19dcd6b
enhance: Opt vector dimension mismatch error message ( #29928 )
...
issue: https://github.com/milvus-io/milvus/issues/29791
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-19 17:52:54 +08:00
f542bdbf3c
enhance: calc the accurate mem size of segment ( #30093 )
...
this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve #30095
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-19 12:32:53 +08:00
fa7cf587b0
enhance: Opt metric type does not match error message ( #29927 )
...
issue: #29791
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-17 20:25:03 +08:00
1185e4dcd5
fix: written file size is over the int32 range and raises error ( #30057 )
...
we sum the total data size in int32, which could lead to an overflow
error
related #30056
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 16:42:54 +08:00
8030b90891
fix: correct file name when loading index ( #29985 )
...
issue: #29973
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-16 10:24:52 +08:00
c31e68446e
enhance: refine groupby-performance ( #29933 )
...
related: #29844
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-15 14:12:52 +08:00
def717af55
fix: SealedIndexingEntry in SealedIndexingRecord may leak without smart pointer protect. ( #29932 )
...
may related issue: #29828
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:28:51 +08:00
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
f2e36db488
enhance: optimize the loading index performance ( #29894 )
...
this utilizes concurrent loading
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 17:44:51 +08:00
6c477ce3a7
enhance: optimize the loading strategy ( #29910 )
...
as we have the pool size limit so we don't need to limit the concurrency
manually
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:26:50 +08:00
aba2656e68
fix: missing field data after appending scalar index to loaded segment ( #29912 )
...
related #29843
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:04:54 +08:00
4d11525f55
[automated] Update Knowhere Commit ( #29904 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-12 14:00:50 +08:00
e429965f32
Add float16 approve for multi-type part ( #28427 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Add bfloat16 vector, add the index part of float16 vector.
Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
67ab5be15a
enhance: optimize search performance of inverted index ( #29794 )
...
issue: #29793
Use `DocSetCollector` instead of `TopDocsCollector`, which will avoid
scoring and sorting.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-11 11:12:49 +08:00
smellthemoon
Buqian Zheng
Alexander Guzhva
Abdullah Ahmed
sre-ci-robot
Patrick Weizhi Xu
zhagnlu
yellow-shine
Cai Yudong
Jiquan Long
congqixia
zhenshan.cao
Bingyi Sun
foxspy
Aldrin
Min Tian
Chun Han
Gao
yihao.dai
chyezh
presburger
cqy123456
Yinzuo Jiang
cai.zhang
wei liu
aoiasd
liliu-z
PowderLi
SimFG
groot
gcmutator
Xiaofan
yah01
xige-16
chyezh
Xu Tong