2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
7b9462c0d3
enhance: fix copying hits of inverted index twice ( #33968 )
...
issue: https://github.com/milvus-io/milvus/issues/29793
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-19 12:40:01 +08:00
298e50b834
enhance: check index with data type ( #33880 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
0d20303e54
fix: fix binary vector data size ( #33750 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
- fix byte size wrong for binary vectors
- fix the expect/actual error msg
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-18 21:39:59 +08:00
b460862537
fix: can't find Chunk struct after growing support mmap ( #33951 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 18:37:58 +08:00
3fdaae8792
fix: Return record with largest timestamp for entires with same PK ( #33936 )
...
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
e83ecd5074
enhance: fix not find arrow::parquet lib ( #33928 )
...
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-18 00:22:01 +08:00
d43ec4db0b
enhance: support array bitmap index ( #33527 )
...
#32900
---------
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:51:58 +08:00
e422168f09
fix: readd timestamp index because segment timestamp not ordered ( #33856 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:48:01 +08:00
7d74edd6dd
fix: update clang-tidy and clang-format from 10 to 12 ( #33141 )
...
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.
issue: #33142
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-06-13 15:27:58 +08:00
47b04ea167
enhance: support sparse cardinal hnsw index ( #33656 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 16:57:55 +08:00
8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector ( #33209 )
...
issue: #29419
* sparse float vector to support raw data mmap
For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 10:09:55 +08:00
ecf2bcee42
enhance: speed up array-equal operator via inverted index ( #33633 )
...
fix : #33632
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-11 14:13:54 +08:00
f53ab54c5d
enhance: async cgo utility ( #33133 )
...
issue: #30926 , #33132
- implement future-based cgo utility.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-09 22:55:53 +08:00
6ce9df913f
fix: clean vector memory ( #33692 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-07 10:37:54 +08:00
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
b69740c8f3
enhance: Remove unnecessary log info during load segment ( #33663 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-06 14:13:50 +08:00
fd191dd7db
[automated] Update Knowhere Commit ( #33655 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-06 01:53:50 +08:00
703fc73f71
enhance: disk index support binary vector ( #33631 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-05 19:37:57 +08:00
8ad26093ba
fix: fix load failure ( #33599 )
...
issue: #33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-05 19:19:51 +08:00
545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector ( #33377 )
...
related #22837
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-05 14:31:57 +08:00
597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit ( #33609 )
...
See also milvus-io/milvus-sdk-go#756
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 11:51:51 +08:00
c6f8a73bb2
enhance: optimize some cache to reduce memory usage ( #33534 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-04 14:09:47 +08:00
d25c755480
[automated] Update Knowhere Commit ( #33573 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-04 01:55:46 +08:00
2b285e5573
fix: Wrap init segcore tracing with golang timeout ( #33494 )
...
See also #33483
Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-03 19:25:51 +08:00
4171414222
enhance: update knowhere version ( #33490 )
...
issue: https://github.com/milvus-io/milvus/issues/33489
update knowhere version to latest. remove usage of `seed_ef` as it be
replaced by existing `ef`.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-31 10:17:50 +08:00
0c5d8660aa
feat: support inverted index for array ( #33452 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-31 09:47:47 +08:00
416a2cf507
fix: query iterator lack results( #33137 ) ( #33422 )
...
related: #33137
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
589d4dfd82
enhance: optimize bitmap index ( #33358 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
9f9dff33f1
fix: make milvus compilable using gcc-13 ( #30149 )
...
add a missing header
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-27 14:52:53 +08:00
5cf4161394
fix: Fix exception info is missing ( #33393 )
...
Replace based std::exception to prevent "object slicing"
issue: https://github.com/milvus-io/milvus/issues/33392
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-27 14:33:41 +08:00
b391781a2e
[automated] Update Knowhere Commit ( #33340 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-24 01:43:40 +08:00
3bec2c4928
enhance: Update Knowhere version ( #33309 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-23 16:57:41 +08:00
c5918ffbdb
enhance: mark sparse inverted index as mmap-able ( #33281 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-23 14:11:42 +08:00
32d3e22d7d
fix: Throw an exception after all the threads in thread pool finished ( #32810 )
...
issue: #32487
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 11:47:40 +08:00
be77ceba84
enhance: Use proto for passing info in cgo ( #33184 )
...
issue: #33183
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 10:31:40 +08:00
3c4df81261
enhance: Assert insert data length not overflow int ( #33248 )
...
When InsertData is too large for cpp proto unmarshalling, the error
message is confusing since the length is overflowed
This PR adds assertion for insert data length.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-22 15:11:39 +08:00
648d5661ca
enhance: Upgrade bitset for ARM SVE ( #32718 )
...
issue: #32826
improve ARM SVE performance for `internal/core/src/bitset`
Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83
Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-22 11:37:40 +08:00
c6e2dd05fc
[automated] Update Knowhere Commit ( #33147 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-20 01:51:37 +08:00
d669fbcf46
enhance: support bitmap index for scalar type ( #32902 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-19 21:49:38 +08:00
7c60d725cc
fix: validate sparse vector in search request ( #32856 )
...
issue: #32368
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-15 15:39:33 +08:00
3777eb2f68
enhance: update knowhere new version ( #33043 )
...
for milvus new release, knowhere update to latest release version
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-05-14 20:45:32 +08:00
09cca1fb51
[automated] Update Knowhere Commit ( #33030 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-14 01:47:32 +08:00
f6777267e3
enhance: add score compute consistency config for knowhere ( #32997 )
...
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-13 14:21:31 +08:00
f7d29118a5
enhance: update knowhere version ( #32931 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-05-10 16:13:31 +08:00
bb7765cbd6
fix: fix Indexing.Iterator ut: build index with all data at once ( #32844 )
...
issue: #32843
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-10 11:31:30 +08:00
54a51b1236
enhance: Support dynamic config for opentelemetry trace ( #32169 )
...
relate: https://github.com/milvus-io/milvus/issues/31940
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-09 17:43:30 +08:00
3d78b90fe7
[automated] Update Knowhere Commit ( #32872 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-09 01:57:30 +08:00
035a508722
fix: make sure inverted index has only one segment ( #32858 )
...
issue: #32717
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-08 21:25:30 +08:00
01c2684355
enhance: [skip e2e] disable unstable ut temporarily ( #32836 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-08 12:17:29 +08:00
9837ad6a8d
enhance: remove deprecated api ( #32808 )
...
issue: #32728
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 20:03:30 +08:00
1f58cda957
enhance: add more trace for search & query ( #32734 )
...
issue: https://github.com/milvus-io/milvus/issues/32728
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 13:03:29 +08:00
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
a43baa0323
enhance: Upgrade knowhere to 2.3.1 ( #32699 )
...
/kind improvement
Signed-off-by: Li Liu <li.liu@zilliz.com>
2024-04-29 17:21:26 +08:00
6289f3a9eb
fix: build milvus in rockylinux8 ( #32619 )
...
issue: #32299
1. xz utils recovers
2. forget to install ninja
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-29 14:53:26 +08:00
858599d831
enhance: sparse float vector to support brute force iterator and range search ( #32635 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-29 14:35:26 +08:00
cb1dbf20c7
enhance:remove openblas from conan ( #32002 )
...
issue: #30683
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-28 20:13:25 +08:00
46d7298407
fix: file not find test_parquet.cpp ( #32644 )
...
#32646
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-04-26 16:57:27 +08:00
4fb8044a27
enhance: delete some no lint code ( #32182 )
...
#31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-04-26 14:15:26 +08:00
ccce1e928a
fix: regex query can't handle text with newline ( #32569 )
...
issue: https://github.com/milvus-io/milvus/issues/32482
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-26 12:01:26 +08:00
c002745902
enhance: retrieve output fields after local reduce ( #32346 )
...
issue: #31822
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-25 09:49:26 +08:00
b703d6cdbe
[automated] Update Knowhere Commit ( #32586 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-25 02:27:24 +08:00
29922829da
fix: [restful v2] parameters of create collection ( #32485 )
...
issue: #31176
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-24 09:43:29 +08:00
15558112c5
[automated] Update Knowhere Commit ( #32516 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-23 01:33:23 +08:00
0fab265eed
enhance: update knowhere and some header changes ( #32468 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-04-22 15:47:26 +08:00
93f0c262c1
fix: add bitset sve support macro protect ( #32463 )
...
#32137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-19 18:05:21 +08:00
e2541f04eb
fix: crash on arm64 ( #31951 ) ( #31980 )
...
issue: #31953
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-17 18:03:20 +08:00
9248a6a149
fix: remove sve flags ( #32270 )
...
#32129
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-16 15:21:19 +08:00
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData ( #32012 )
...
issue: #30361
- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.
- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`
- Remove repeat load delta log operation in lru.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-16 11:17:20 +08:00
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
337cc0756d
fix: lack good results for insufficient ef( #29883 ) ( #32080 )
...
related: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-13 22:13:23 +08:00
4fb85be525
fix: put inverted index into local storage ( #32209 )
...
issue: https://github.com/milvus-io/milvus/issues/32154
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-13 21:57:19 +08:00
454984aa4e
[automated] Update Knowhere Commit ( #32181 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-12 01:25:19 +08:00
b5455d176e
fix: dynamically resolve whether SVE is available for bitset ( #32137 )
...
Issue: #32129
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-11 19:27:18 +08:00
52ae47c850
enhance: gather materialized view search info once per request ( #31996 )
...
issue: #29892
This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-11 15:21:19 +08:00
f3f2a5a7e9
fix: evicted segments in the serverlss mode( #31959 ) ( #31961 )
...
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-10 15:15:19 +08:00
a0a4ec8b67
enhance: make range search param check message more meaningful ( #32006 )
...
Issue: #31970
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-09 16:17:26 +08:00
1b767669a4
enhance: Throw error instead of crash when index cannot be built ( #31844 )
...
issue: #27589
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-04-09 11:51:18 +08:00
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug ( #31977 )
...
issue: #30361
- Add configurable disk capacity limit
- fix bitset reset logic
- make insert record reinsert after clear
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. ( #31771 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-07 14:13:16 +08:00
cae5722229
enhance: performance improvements for the bitset ( #31753 )
...
Issue: #31752
This PR improves the performance for bitset utilities (introduced in PR
#30454 ), including varchar filtering
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-06 05:19:22 +08:00
b2669e26dc
fix:reduce thread pool test time ( #31893 )
...
#31877
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-05 10:05:12 +08:00
d6d3b01a04
fix:remove thread pool timeout test because of high load cpu ( #31879 )
...
#31877
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-03 15:55:38 +08:00
03e0db109e
fix: udpate Cargo.lock ( #31859 )
...
issue: #31681
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-03 14:18:23 +08:00
246586be27
enhance: Unify data type check APIs under internal/core ( #31800 )
...
Issue: #22837
Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-02 19:15:14 +08:00
d299fa502e
fix: use milvus-io/vcpkg ( #31770 )
...
issue: #31769
GitHub Disables The XZ Repository because of CVE-2024-3094
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-01 15:01:13 +08:00
5655ec4fc0
enhance: add mmap usage metrics ( #31708 )
...
issue: #31707
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 11:35:12 +08:00
3ffe126dc7
enhance: Refine error message when search vector type not matched ( #31725 )
...
Previously the error message only reports the case happened without
field name and vector type.
This PR add field name and vector type information in the error
messages.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-01 10:11:12 +08:00
675a5dc822
fix: Save traceID and spanID as std::vector into search config ( #31278 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-29 14:29:11 +08:00
9750e78f1d
enhance: lock tantivy dependencies ( #31688 )
...
issue: https://github.com/milvus-io/milvus/issues/31681
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-29 10:15:17 +08:00
b99c46246c
enhance: ban groupby on binary vector( #31134 ) ( #31659 )
...
related: #31134
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-28 15:19:10 +08:00
e33dba8afe
fix: [skip-e2e] use zstd-sys 2.0.9 ( #31682 )
...
fix : #31681
/kind improvement
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-28 15:14:10 +08:00
b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
4eb4df1e81
fix: predict inverted index resource usage more reasonably ( #31615 )
...
/kind improvement
issue: #31617
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-27 19:33:09 +08:00
655097f171
fix: Verify PlaceHolderValue type before search ( #31626 )
...
See also #31625
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-27 17:49:10 +08:00
5be395354c
fix: minio ssl compatible issue ( #31607 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-03-27 14:41:20 +08:00
678cb187e8
[automated] Update Knowhere Commit ( #31630 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-27 01:15:10 +08:00
659ad81ab7
fix: remove deprecated ut test ( #31499 )
...
#31498
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-26 14:01:07 +08:00
c4b37fb285
enhance: Custom bitset and bitsetview prototypes ( #30454 )
...
Issue: #31285
Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.
This PR is for internal use only.
Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[ OK ] CApiTest.AssembeChunkPerfTest (183 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[ FAILED ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[ OK ] Expr.TestExprs (9 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 124.8us
start test type:3
cost: 163.1us
start test type:4
cost: 275.9us
start test type:5
cost: 590.9us
start test type:10
cost: 62.7us
start test type:11
cost: 65.9us
[ OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 151.4us
start test type:3
cost: 198.4us
start test type:4
cost: 361.9us
start test type:5
cost: 753.9us
start test type:10
cost: 64.6us
start test type:11
cost: 62.2us
[ OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 121.14us
start test type:3
cost: 156.84us
start test type:4
cost: 249.76us
start test type:5
cost: 534.44us
start test type:10
cost: 82.2us
start test type:11
cost: 83.52us
[ OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 80.64us
start test type:3
cost: 78.22us
start test type:4
cost: 255.76us
start test type:5
cost: 532.04us
start test type:10
cost: 89.26us
start test type:11
cost: 90us
[ OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 401.7us
start test type:3
cost: 420.96us
start test type:4
cost: 418.04us
start test type:5
cost: 470.54us
start test type:10
cost: 250.32us
start test type:11
cost: 850.08us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 162us
start test type:3
cost: 142us
start test type:4
cost: 374us
start test type:5
cost: 674us
start test type:10
cost: 366us
start test type:11
cost: 645us
[ OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[ OK ] Expr.TestRefactorExprs (1142 ms)
```
Candidate:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[ OK ] CApiTest.AssembeChunkPerfTest (153 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[ OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[ OK ] Expr.TestExprs (8 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 55.7us
start test type:3
cost: 79.8us
start test type:4
cost: 177.6us
start test type:5
cost: 337.2us
start test type:10
cost: 16.9us
start test type:11
cost: 15.7us
[ OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 57.1us
start test type:3
cost: 87us
start test type:4
cost: 177.5us
start test type:5
cost: 342.7us
start test type:10
cost: 17.9us
start test type:11
cost: 16.7us
[ OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 34.58us
start test type:3
cost: 68.86us
start test type:4
cost: 151.38us
start test type:5
cost: 286.8us
start test type:10
cost: 16.54us
start test type:11
cost: 16.7us
[ OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 20us
start test type:3
cost: 17.1us
start test type:4
cost: 154.12us
start test type:5
cost: 286.1us
start test type:10
cost: 19.6us
start test type:11
cost: 19.24us
[ OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 125.7us
start test type:3
cost: 111.34us
start test type:4
cost: 148.02us
start test type:5
cost: 306.7us
start test type:10
cost: 149.3us
start test type:11
cost: 282.94us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 89us
start test type:3
cost: 79us
start test type:4
cost: 323us
start test type:5
cost: 629us
start test type:10
cost: 313us
start test type:11
cost: 591us
[ OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[ OK ] Expr.TestRefactorExprs (1149 ms)
```
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-03-24 21:49:07 +08:00
982dd2834b
enhance: add materialized view search info ( #30888 )
...
issue: #29892
This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.
## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.
## Detailed MV Info
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-03-21 11:19:07 +08:00
c81909bfab
enhance: Support MinIO TLS connection ( #31311 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
cf5109ec17
fix: fix mmap failed when string field all value is empty ( #31406 )
...
#31162
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-21 10:41:07 +08:00
66d679ecbb
fix: clear binlog files in CleanData ( #31039 )
...
issue: https://github.com/milvus-io/milvus/issues/31042
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-20 11:11:07 +08:00
6edd06083f
chore: remove repetitive words ( #31153 )
...
Signed-off-by: gcmutator <329964069@qq.com>
2024-03-20 10:17:07 +08:00
b35ecebcc3
enhance: Update Knowhere version ( #31392 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-19 15:11:07 +08:00
d7727dd087
enhance: fetch simdjson directly in the format of targz ( #31369 )
...
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-03-18 18:55:11 +08:00
1c930e560c
enhance: Update Knowhere version ( #31312 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-18 11:29:04 +08:00
038c570ef3
enhance: upgrade folly to run on arm ( #31284 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-03-15 15:39:03 +08:00
6939ad15f2
fix:possible out-of-bound due to groupby when reduing( #30711 ) ( #31200 )
...
related: #30711
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-14 13:07:03 +08:00
7fc3094a42
fix: fix growing index data race and properly handle build error ( #31170 )
...
issue: https://github.com/milvus-io/milvus/issues/31169
also properly handling index build error by re-create a new index so
that nothing will be left in the previous failed index build attempt.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 20:19:04 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
c8b54f321a
fix:restrict pk in [...] optimization situations ( #31184 )
...
#31154
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-12 14:49:03 +08:00
6a83f16871
feat: Support for multiple forms of JSON ( #31052 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-11 19:55:02 +08:00
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building ( #30357 )
...
This commit adds sparse float vector support to segcore with the
following:
1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built
This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
a99143dd52
fix: Save traceID and spanID as hex string into search config ( #31071 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-11 14:21:01 +08:00
53af6d8c59
[automated] Update Knowhere Commit ( #31151 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-09 01:55:02 +08:00
122981aeb9
fix: Disable knowhere trace as a quick fix ( #31055 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-08 15:27:01 +08:00
3574bdf858
enhance: ban range-search iteration for search-group-by ( #30824 )
...
related: #30033
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-08 14:17:00 +08:00
19c64067af
enhance: jemalloc aarch64 platform use 64k pagesize. ( #29522 )
...
enhance: jemalloc aarch64 platform use 64k pagesize.
issue: #28843
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2024-03-07 21:01:01 +08:00
2d9de233fc
[automated] Update Knowhere Commit ( #31089 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-07 12:05:02 +08:00
c047f09110
[automated] Update Knowhere Commit ( #31015 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-05 16:31:00 +08:00
4bda6c33ad
fix: binary vector should not limit dimension to 32768 ( #30676 )
...
all the vector dimension check should happen on collection creation but
not index build
fix #30285
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-05 14:21:00 +08:00
3dc5e38240
[automated] Update Knowhere Commit ( #30989 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-04 16:34:59 +08:00
29f44f840a
enhance: refine groupBy error msg( #29968 ) ( #30920 )
...
related: #29968
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-01 18:53:03 +08:00
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
e2f35954d4
enhance: support pattern matching on json field ( #30779 )
...
issue: https://github.com/milvus-io/milvus/issues/30714
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
16b785e149
enhance: optimize the memory usage and speed up loading variable length data ( #30787 )
...
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
4459078e0b
fix: wrong num_entities used when mmap variable length data ( #30848 )
...
https://github.com/milvus-io/milvus/issues/30728
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
a115b731ed
enhance: fix old pr cpp format issue ( #30894 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
f658dd5faa
enhance: update knowhere version to 60a5c9c ( #30788 )
...
/kind improvement
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-02-28 14:18:55 +08:00
8a219e0102
feat: Support knowhere trace using OpenTelemetry ( #30750 )
...
Issue: #21508
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
6e9f3ea531
[automated] Update Knowhere Commit ( #30744 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-28 10:50:57 +08:00
57397b1307
enhance: add new LRU cache impl ( #30360 )
...
- remove the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache
related #30361
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
3e82d21ca1
enhance: reduce 1x memory copy when loading json ( #30753 )
...
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-27 10:18:55 +08:00
e2330f02f8
fix: pattern match use incorrect raw data ( #30764 )
...
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
77eb6defb1
feat: support groupby on growing and non-indexed sealed egment( #30307 ) ( #30644 )
...
related: #30308
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-21 14:02:53 +08:00
18aac076de
fix: move test from NEON to X86 ( #30324 )
...
#26137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 11:58:53 +08:00
0118bef2a2
fix: replace sse2 simd interface with older version ( #30668 )
...
#30667
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 10:04:54 +08:00
976b6fc0e4
enhance: change opendal as compile configurable ( #30384 )
...
#30373
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-20 19:16:52 +08:00
b74673c147
enhance: calculate the accuracy memory usage while loading segment ( #30473 )
...
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.
Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-20 14:52:51 +08:00
43e8cd531d
enhance: Update Knowhere version ( #30675 )
...
issue: #30669
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-02-19 22:04:51 +08:00
18c351efa6
fix: Prevent ChunkCache use absolute path in All-in-one mode ( #30666 )
...
See also #30651
Append operator of `std::filesystem::path` will replace whole path when
the param of "/" operation is an absolute path.
In "All-in-one" mode, this shall cause ChunkCache removing the original
vector data file when building chunk cache during/after load procedure.
This PR changes the ChunkCache path generation logic to a separate
function in which will check whether the file path is absolute or not.
If the file path is absolute, it removes the root path prefix and return
concatenated file path.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-19 20:58:51 +08:00
5bb28a9ea4
enhance: Print out range_filter and radius when range search param check fail ( #30623 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-18 15:40:48 +08:00
a297baae9d
enhance: remove unused code ( #30601 )
...
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-02-13 10:26:47 +08:00
e8a6f1ea2b
fix: erase pk empty check when pk index replace raw data ( #30432 )
...
#30350
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-07 14:56:47 +08:00
92d1d744ae
fix: groupby results lack good results( #29883 ) ( #30428 )
...
related: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-06 17:08:34 +08:00
5449e862d5
fix: safety access unordered_map and remove some useless code excute ( #30504 )
...
issue: https://github.com/milvus-io/milvus/issues/30358 and
https://github.com/milvus-io/milvus/issues/30491
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-02-05 22:03:09 +08:00
ebbe32df9a
[automated] Update Knowhere Commit ( #30515 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-05 01:32:44 +08:00
a587450e56
enhance: [skip-e2e] disable asan ( #30498 )
...
fix : #30511
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-04 21:25:05 +08:00
20c9cfc587
[automated] Update Knowhere Commit ( #30487 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-04 01:23:04 +08:00
e549148a19
enhance: full-support for wildcard pattern matching ( #30288 )
...
issue: #29988
This pr adds full-support for wildcard pattern matching from end to end.
Before this pr, the users can only use prefix match in their expression,
for example, "like 'prefix%'". With this pr, more flexible syntax can be
combined.
To do so, this pr makes these changes:
- 1. support regex query both on index and raw data;
- 2. translate the pattern matching to regex query, so that it can be
handled by the regex query logic;
- 3. loose the limit of the expression parsing, which allows general
pattern matching syntax;
With the support of regex query in segcore backend, we can also add
mysql-like `REGEXP` syntax later easily.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-01 12:37:04 +08:00
5cf9bb236e
enhance: restful support import jobs ( #30343 )
...
issue: #28521 #29732
include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job
fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-31 17:57:04 +08:00
878c4c9463
enhance: limit the max pool size to 16 ( #30371 )
...
according to our benchmark, concurrency level 16 is enough to fully
utilize the object storage network bandwidth
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-31 14:13:06 +08:00
74cfba0249
enhance:limit binlog index rows num ( #30173 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
also relate issue: https://github.com/milvus-io/milvus/issues/30065
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-29 19:49:02 +08:00
0542a0e7dc
[automated] Update Knowhere Commit ( #30332 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-29 01:05:01 +08:00
aeb1e36f00
enhance: change plan desc log from info to debug ( #30304 )
...
#30172
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-28 16:04:38 +08:00
e9fdd2475d
fix: fix searchPlan metricType modified concurrently ( #30227 )
...
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-26 14:03:09 +08:00
116d0f20b8
fix: groupby bug for ut ( #30272 )
...
related: #29965
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-25 20:57:00 +08:00
c02fb64ad6
enhance: Allows proactive warming up of chunk cache ( #30182 )
...
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-25 19:55:39 +08:00
a27c0e86fd
enhance: reduce many I/O operations while loading disk index ( #30189 )
...
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file
this optimized this to open only once and continiously write all data.
This also makes it concurrent to load the files from object storage
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-25 15:23:02 +08:00
8c58d9af67
enhance: optimize marisa trie range search for performance ( #30079 )
...
#30078
#29986
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-25 10:07:00 +08:00
0907d76253
enhance: pass partition key scalar info if enabled when build vector index ( #29931 )
...
issue: #29892
Pass optional scalar IVF offsets to Cardinal
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
42bb4e37e5
fix:diskann search crash when search list = 9999999999 ( #30185 )
...
issue: https://github.com/milvus-io/milvus/issues/29020
Json can't not pass a max_int32 value to int32_t, so let knowhere check
value range by itself.
After fix this, pymilvus will report:
pymilvus.exceptions.MilvusException: <MilvusException: (code=65535,
message=fail to search on QueryNode 6: worker(6) query failed: => failed
to search: arithmetic overflow: param search_list_size should be at most
2147483647)>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-23 14:46:55 +08:00
6cf2f09b60
feat: Support tencent cloud object storage for milvus ( #30163 )
...
issue: #30162
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 11:28:56 +08:00
a77693aa19
enhance: convert the `GetObject` util to async ( #30166 )
...
This makes it much easier to use
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 19:20:57 +08:00
e967949cc5
[automated] Update Knowhere Commit ( #30120 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-22 18:40:54 +08:00
4436effdc3
enhance: support groupby based on scalar-index( #29965 ) ( #30091 )
...
related: #29965
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 10:50:54 +08:00
aee19dcd6b
enhance: Opt vector dimension mismatch error message ( #29928 )
...
issue: https://github.com/milvus-io/milvus/issues/29791
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-19 17:52:54 +08:00
f542bdbf3c
enhance: calc the accurate mem size of segment ( #30093 )
...
this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve #30095
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-19 12:32:53 +08:00
fa7cf587b0
enhance: Opt metric type does not match error message ( #29927 )
...
issue: #29791
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-17 20:25:03 +08:00
1185e4dcd5
fix: written file size is over the int32 range and raises error ( #30057 )
...
we sum the total data size in int32, which could lead to an overflow
error
related #30056
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 16:42:54 +08:00
8030b90891
fix: correct file name when loading index ( #29985 )
...
issue: #29973
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-16 10:24:52 +08:00
c31e68446e
enhance: refine groupby-performance ( #29933 )
...
related: #29844
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-15 14:12:52 +08:00
def717af55
fix: SealedIndexingEntry in SealedIndexingRecord may leak without smart pointer protect. ( #29932 )
...
may related issue: #29828
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:28:51 +08:00
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
f2e36db488
enhance: optimize the loading index performance ( #29894 )
...
this utilizes concurrent loading
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 17:44:51 +08:00
6c477ce3a7
enhance: optimize the loading strategy ( #29910 )
...
as we have the pool size limit so we don't need to limit the concurrency
manually
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:26:50 +08:00
aba2656e68
fix: missing field data after appending scalar index to loaded segment ( #29912 )
...
related #29843
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:04:54 +08:00
4d11525f55
[automated] Update Knowhere Commit ( #29904 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-12 14:00:50 +08:00
e429965f32
Add float16 approve for multi-type part ( #28427 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Add bfloat16 vector, add the index part of float16 vector.
Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
67ab5be15a
enhance: optimize search performance of inverted index ( #29794 )
...
issue: #29793
Use `DocSetCollector` instead of `TopDocsCollector`, which will avoid
scoring and sorting.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-11 11:12:49 +08:00
5164d30287
fix: increase expr recursion depth to avoid parse failed ( #29860 )
...
#29759
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-11 10:26:50 +08:00
031243fee7
feat: support mmap for marisa trie ( #29613 )
...
this supports mmap for marisa trie index
related https://github.com/milvus-io/milvus/issues/21866
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 10:22:50 +08:00
d6429933a7
enhance: make Load process traceable in querynode & segcore ( #29858 )
...
See also #29803
This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 21:58:51 +08:00
cb9d9ec0f0
enhance: Correct sampleFraction's type to float ( #29810 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-01-10 13:18:50 +08:00
600f6eff06
enhance: Upgrade gtest to 1.13.0 ( #29805 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-01-10 13:16:57 +08:00
601a8b801b
fix: add move cursor function to physical expr ( #29603 )
...
#29570
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-09 17:08:48 +08:00
60e88fb833
fix: Restore the MVCC functionality. ( #29749 )
...
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:
1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.
issue: #29656
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
9702cef2b5
feat: Support multiple vector search ( #29433 )
...
issue #25639
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
e9f3df3626
fix: inverted index file not found ( #29695 )
...
issue: https://github.com/milvus-io/milvus/issues/29654
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-07 20:26:49 +08:00
d07197ab1a
enhance: add compare simd function ( #29432 )
...
#26137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-07 20:20:57 +08:00
271edc6669
fix: throw exception when upload file failed for DiskIndex ( #29627 )
...
related to : #29417
cardinal indexes upload index files in `Serialize` interface, and throw
exception when the `Serialize` failed.
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-01-07 20:03:13 +08:00
5dc300c4a9
fix: Fix bug for pk index doesn't have raw data ( #29711 )
...
issue: #29697
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-07 19:36:48 +08:00
9e2e7157e9
feat: support search_group_by for milvus( #25324 ) ( #28983 )
...
related: #25324
Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:
1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-05 15:50:47 +08:00
22bb84fa9d
feat:add new gpu index:GPU_BRUTE_FORCE and limit gpu index metric type ( #29590 )
...
issue: https://github.com/milvus-io/milvus/issues/29230
this pr do these things:
1. add gpu brute force;
2. limit gpu index only support l2 / ip;
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-05 15:24:48 +08:00
c8db36a63a
enhance: get a blob to check object storage config ( #29703 )
...
issue: #29672
the storage account need privileges of actions
`Microsoft.Storage/storageAccounts/blobServices/containers/blobs/*` at
least
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-05 14:50:46 +08:00
0ae90443ba
enhance: fill missed info for segcore error ( #29610 )
...
- fill missed error info
- format the error message directly
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-04 17:54:46 +08:00
99e0f1e65a
enhance: unable to compile C++ tests ( #29616 )
...
The tests need to call a private method, Milvus uses `#define` to
replace private with public, the hack trick works but would be broken if
the including order changed.
This uses friend to make all things work well
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-04 13:20:46 +08:00
5f00bad4b8
fix: link with install path's libblob-chunk-manager ( #29496 )
...
issue: #29494
1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-31 20:02:48 +08:00
3f46c6d459
feat: support inverted index ( #28783 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
Add inverted index for some data types in Milvus. This index type can
save a lot of memory compared to loading all data into RAM and speed up
the term query and range query.
Supported: `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BOOL`
and `VARCHAR`.
Not supported: `ARRAY` and `JSON`.
Note:
- The inverted index for `VARCHAR` is not designed to serve full-text
search now. We will treat every row as a whole keyword instead of
tokenizing it into multiple terms.
- The inverted index don't support retrieval well, so if you create
inverted index for field, those operations which depend on the raw data
will fallback to use chunk storage, which will bring some performance
loss. For example, comparisons between two columns and retrieval of
output fields.
The inverted index is very easy to be used.
Taking below collection as an example:
```python
fields = [
FieldSchema(name="pk", dtype=DataType.VARCHAR, is_primary=True, auto_id=False, max_length=100),
FieldSchema(name="int8", dtype=DataType.INT8),
FieldSchema(name="int16", dtype=DataType.INT16),
FieldSchema(name="int32", dtype=DataType.INT32),
FieldSchema(name="int64", dtype=DataType.INT64),
FieldSchema(name="float", dtype=DataType.FLOAT),
FieldSchema(name="double", dtype=DataType.DOUBLE),
FieldSchema(name="bool", dtype=DataType.BOOL),
FieldSchema(name="varchar", dtype=DataType.VARCHAR, max_length=1000),
FieldSchema(name="random", dtype=DataType.DOUBLE),
FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
collection = Collection("demo", schema)
```
Then we can simply create inverted index for field via:
```python
index_type = "INVERTED"
collection.create_index("int8", {"index_type": index_type})
collection.create_index("int16", {"index_type": index_type})
collection.create_index("int32", {"index_type": index_type})
collection.create_index("int64", {"index_type": index_type})
collection.create_index("float", {"index_type": index_type})
collection.create_index("double", {"index_type": index_type})
collection.create_index("bool", {"index_type": index_type})
collection.create_index("varchar", {"index_type": index_type})
```
Then, term query and range query on the field can be speed up
automatically by the inverted index:
```python
result = collection.query(expr='int64 in [1, 2, 3]', output_fields=["pk"])
result = collection.query(expr='int64 < 5', output_fields=["pk"])
result = collection.query(expr='int64 > 2997', output_fields=["pk"])
result = collection.query(expr='1 < int64 < 5', output_fields=["pk"])
```
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-31 19:50:47 +08:00
79c417b14e
fix: pass active count to query context instead of timestamp ( #29541 )
...
#29319
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-31 16:08:48 +08:00
c2345daf3a
[automated] Update Knowhere Commit ( #29578 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-29 18:56:46 +08:00
6f4791da0b
fix: panic in concurrent insert/query scenario ( #29408 )
...
issue: https://github.com/milvus-io/milvus/issues/29405
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-26 15:10:48 +08:00
b8318fcd7d
enhance: improve the handling for segcore error ( #29471 )
...
- fix lost exception details in segcore
- improve the logs of handling errors from segcore
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-26 14:06:46 +08:00
4c979538a4
enhance: update cagra index params in config and add params check ( #29045 )
...
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
a.milvus yaml config support gpu memory settings
b.add cagra-params check
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2023-12-26 11:04:47 +08:00
fce1a8dafb
[automated] Update Knowhere Commit ( #29412 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-25 17:58:46 +08:00
aef483806d
enhance: improve the segcore logs ( #29372 )
...
- remove the streaming logging
- refine existing logs
fix #29366
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-23 21:52:43 +08:00
1b7f1d7067
enhance: mmap data corrupted after seal the column ( #29422 )
...
this bug was introduced in recent changes
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-23 15:20:43 +08:00
1cbe3cd5fc
fix: fix memory leak when cancel segcore task ( #29431 )
...
#29430
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-22 20:28:43 +08:00
a6eb7e5f9a
enhance: skip segment when using pk in (..) expr ( #29394 )
...
#29293
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-21 20:06:42 +08:00
7a2374e698
enhance: reduce the memory usage of variable length data ( #29387 )
...
add all loading data into a buffer and then copy them into the a
fit-in-size memory
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:02:42 +08:00
be87c18b44
fix: fixup data race at generate binlog index ( #29370 )
...
issue: #29339
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-12-21 14:58:49 +08:00
04b2518ae7
enhance: fix the incorrect init parameter ( #29357 )
...
as the `driver_` field is not used so this doesn't matter for now
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 20:50:43 +08:00
9b52cb6417
enhance: improve reducing results when many segments are filtered ( #29073 )
...
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-20 12:56:42 +08:00
8f89e9cf75
enhance: remove all unnecessary string formatting ( #29323 )
...
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
related: #28811
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 10:04:43 +08:00
89b208d27a
enhance: Fix format message ( #29159 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-20 09:30:44 +08:00
bfca0a7926
fix: refine skipIndex to resolve cyclic dependcy( #29132 ) ( #29189 )
...
related: #29132
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-19 10:26:40 +08:00
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
26409d801e
enhance: Remove omp from segcore ( #29207 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-12-15 14:00:39 +08:00
3e66e78508
[automated] Update Knowhere Commit ( #29178 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-14 17:16:39 +08:00
49b8657f95
enhance: Support implicit type conversion for parquet ( #29046 )
...
issue: #29019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:44 +08:00
0e65e90338
enhance: Support otlp with insecure ( #29115 )
...
issue: https://github.com/milvus-io/milvus/issues/28914
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 11:14:37 +08:00
9d54d6f590
fix: change Abseil to shared library to solve macos compilation issue ( #28986 )
...
fix the compilation error on macos 14.0 with x86 arch processor
related to #28985
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2023-12-11 21:54:37 +08:00
464bc9e8f4
fix: fix reduce precision for search( #27325 ) ( #29031 )
...
related: #27325
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 10:04:37 +08:00
dcb662d9ed
enhance: Refine C.NewSegment response and handle exception ( #28952 )
...
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-07 13:34:35 +08:00
fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data ( #28888 )
...
issue: #28886
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-06 20:36:36 +08:00
36f69ea031
feat: integrate storagev2 in building index of segcore ( #28768 )
...
issue: https://github.com/milvus-io/milvus/issues/28655
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 16:48:54 +08:00
f01e507b15
[automated] Update Knowhere Commit ( #28965 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-05 15:56:35 +08:00
9b6cbe956a
[automated] Update Knowhere Commit ( #28917 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-04 15:42:34 +08:00
c8b1a4618a
enhance: Resolve libunwind requirement conflict using 1.7.2 ( #28929 )
...
Try to resolve libunwind dependency requirement conflict between glog &
folly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 14:06:40 +08:00
20fc90c591
enhance: find collection schema from cache ( #28782 )
...
issue: #28781 #28329
1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
342635ed61
enhance: enable assert method to format arguments ( #28812 )
...
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.
make it happy
related #28811
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-01 18:04:33 +08:00
f5856812a2
fix: Fix get binary vector from chunk cache ( #28866 )
...
The way of getting binary vector size is wrong. This PR will fix it.
issue: https://github.com/milvus-io/milvus/issues/28865
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-01 14:40:32 +08:00
7206795e91
fix: update folly to resolve simd issue ( #28878 )
...
related #27552 , after this, milvus could run successfully on sse4.2
only machine
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-01 13:50:32 +08:00
8036ee13fa
feat: avoid dereferencing nullptr ( #28862 )
...
issue: #28793
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-01 10:20:32 +08:00
ecc3ca374c
[automated] Update Knowhere Commit ( #28882 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-01 02:28:31 +08:00
cac802ef7f
enhance: use already installed vcpkg ( #28703 )
...
issue #28686
1. Update Builder gpu image changes, see changes #28505
2. update azure-identity-cpp from beta to release
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-30 15:58:32 +08:00
d69440524b
fix: bypass growing index if no index meta ( #28791 )
...
we shouldn't panic if no index meta, just skip building it
fix #28022
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-30 14:10:27 +08:00
1dc086496f
fix: schema->size() check logic with system field ( #28802 )
...
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 22:40:28 +08:00
3b1b14dd78
fix: update binlog index memory uasge before loading segments ( #28528 )
...
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
86ccb8e146
[automated] Update Knowhere Commit ( #28704 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-24 16:56:24 +08:00
6f7a9264d5
enhance: Handle knowhere error for creare diskann index ( #28690 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-24 11:58:23 +08:00
0d9d098186
enhance: Add precheck when chunk manager init ( #28330 )
...
#28329
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:56:32 +08:00
e9e9b30e8e
enhance: Disable stdout buffer ( #28391 )
...
#28390
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:26:25 +08:00
e5ce385ffd
enhance: remove -inl.h files ( #28674 )
...
issue: https://github.com/milvus-io/milvus/issues/28673
Move template implementations from -inl.h to .cpp file and make explicit
instantiation
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:20:25 +08:00
c96d07682e
enhance: enhance the accuracy of memory usage ( #28554 )
...
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.
we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage
resolve #28553
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-23 15:12:23 +08:00
e8c5e75786
enhance: ignore proto generated files ( #28537 )
...
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-22 10:20:22 +08:00
bfccfcd0ca
enhance: refine error messages ( #28424 )
...
- Split the simple reason and full detail
- Refine existing error messages
related: #28422
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
02c5a649cf
enhance: store system fields in segcore ( #28524 )
...
we need the system fields info for some usacase
fix : #28523
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 09:28:22 +08:00
smellthemoon
Jiquan Long
cqy123456
Gao
congqixia
presburger
zhagnlu
Yinzuo Jiang
Buqian Zheng
chyezh
cai.zhang
wei liu
sre-ci-robot
Chun Han
Alexander Guzhva
yihao.dai
foxspy
aoiasd
Bingyi Sun
liliu-z
PowderLi
Patrick Weizhi Xu
Cai Yudong
SimFG
groot
gcmutator
sammy.huang
Xiaofan
yah01
xige-16
chyezh
Xu Tong
zhenshan.cao
yah01
Enwei Jiao