32d5c5b3c3
fix: add bitset sve support macro protect ( #32464 )
...
pr: #32463
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-29 14:01:32 +08:00
d05926f7ac
fix: remove sve flags ( #32275 )
...
pr: #32270
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-29 14:01:32 +08:00
5f7edc7304
fix: lack good results for insufficient ef( #29883 ) ( #32151 )
...
related: #29883
pr: https://github.com/milvus-io/milvus/pull/32080
see also: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-13 22:13:23 +08:00
3a1e0c3b18
fix: put inverted index into local storage ( #32209 ) ( #32210 )
...
issue: https://github.com/milvus-io/milvus/issues/32154
pr: #32209
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-13 21:53:18 +08:00
a72a9cd9cb
fix: [2.4] [Cherry-pick] dynamically resolve whether SVE is available for bitset ( #32139 )
...
Issue: https://github.com/milvus-io/milvus/issues/32129
pr: #32137
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-12 10:01:20 +08:00
1cfc681461
enhance:remove openblas from conan ( #31413 )
...
issue: #30683
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-08 19:17:17 +08:00
312d570855
fix: crash on arm64 ( #31951 )
...
issue: #31953
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-08 02:15:15 +08:00
7580fa8e75
fix: [cherry-pick]fix some fp16/bf16 code missing in segcore ( #31841 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/31771
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-07 14:27:22 +08:00
841f9e4f80
enhance: [Cherry-pick] performance improvements for the bitset ( #31754 )
...
Issue: #31752
see also: #30454
pr: #31753
This PR improves the performance for bitset utilities (introduced in PR
https://github.com/milvus-io/milvus/pull/30454 ), including varchar
filtering
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-06 05:19:21 +08:00
1add739de1
fix:reduce thread pool test time ( #31894 )
...
pr: #31893
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-05 10:05:04 +08:00
3904719287
enhance: Throw error instead of crash when index cannot be built ( #31845 )
...
issue: #27589
pr: #31844
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-04-03 16:27:16 +08:00
d4f59a6059
fix:remove thread pool timeout test because of high load cpu ( #31880 )
...
pr: #31879
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-03 15:55:12 +08:00
76483b2bec
enhance: Unify data type check APIs under internal/core ( #31857 )
...
Issue: #22837 #31800
Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-03 15:01:14 +08:00
17a847a3b9
fix: [cherry-pick] use milvus-io/vcpkg ( #31799 )
...
issue: #31769
master pr: #31770
GitHub Disables The XZ Repository because of CVE-2024-3094
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-01 22:11:13 +08:00
7523107c80
enhance: [cherry-pick] Refine error message when search vector type not matched ( #31725 ) ( #31766 )
...
Cherry-pick from master
pr: #31725
Previously the error message only reports the case happened without
field name and vector type.
This PR add field name and vector type information in the error
messages.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-01 16:53:21 +08:00
f3216bfe18
enhance: ban groupby on binary vector( #31134 ) ( #31735 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/31659
See also: https://github.com/milvus-io/milvus/issues/31134
Currently, don't support brute force search iterator for binary_vector
so group_by in such cases will fail and to avoid inconsistent behavior,
we ban groupby on binary vector for the time being.
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-01 14:13:12 +08:00
2f5ddfab22
enhance: Update Knowhere version ( #31313 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-29 17:35:12 +08:00
ef267b5c34
enhance: lock tantivy dependencies ( #31688 ) ( #31690 )
...
issue: https://github.com/milvus-io/milvus/issues/31681
pr: #31688
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-29 10:15:17 +08:00
bd6f123c59
fix: [skip-e2e] use zstd-sys 2.0.9 ( #31682 ) ( #31683 )
...
issue: #31681
pr: #31682
/kind improvement
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-28 15:11:57 +08:00
97c28edb04
fix: [Cherry-pick] Verify PlaceHolderValue type before search ( #31626 ) ( #31653 )
...
Cherry-pick from master
pr: #31626
See also #31625
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-28 10:21:11 +08:00
d37e1fdd9f
fix: predict inverted index resource usage more reasonably ( #31615 ) ( #31641 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/31617
pr: #31615
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-27 21:07:11 +08:00
b0cbddae8d
fix: minio ssl compatible issue ( #31618 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: https://github.com/milvus-io/milvus/pull/31607
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-03-27 14:41:20 +08:00
ded1fcc555
fix: remove deprecated ut test ( #31499 ) ( #31636 )
...
pr: #31499
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-27 12:49:10 +08:00
3addc68c66
enhance: [Cherry-pick] Custom bitset and bitset_view implementations ( #31592 )
...
Issue: https://github.com/milvus-io/milvus/issues/31285
pr: https://github.com/milvus-io/milvus/pull/30454
Basically, I've replaced FixedVector<bool> and boost::dynamic_bitset
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.
(cherry picked from commit 5dcecc882d
2024-03-26 10:05:09 +08:00
a0535edb67
enhance: Support MinIO TLS connection ( #31396 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: https://github.com/milvus-io/milvus/pull/31292
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
6856ba1e69
fix: fix mmap failed when string field all value is empty ( #31418 )
...
pr: #31406
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-20 14:45:10 +08:00
b773581fde
enhance: fetch simdjson directly in the format of targz ( #31370 )
...
pr: #31369
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-03-18 18:55:10 +08:00
fdb3231151
enhance: Upgrade Knowhere ( #31308 )
...
/kind improvement
Signed-off-by: Li Liu <li.liu@zilliz.com>
2024-03-18 14:21:04 +08:00
038c570ef3
enhance: upgrade folly to run on arm ( #31284 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-03-15 15:39:03 +08:00
6939ad15f2
fix:possible out-of-bound due to groupby when reduing( #30711 ) ( #31200 )
...
related: #30711
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-14 13:07:03 +08:00
7fc3094a42
fix: fix growing index data race and properly handle build error ( #31170 )
...
issue: https://github.com/milvus-io/milvus/issues/31169
also properly handling index build error by re-create a new index so
that nothing will be left in the previous failed index build attempt.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 20:19:04 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
c8b54f321a
fix:restrict pk in [...] optimization situations ( #31184 )
...
#31154
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-12 14:49:03 +08:00
6a83f16871
feat: Support for multiple forms of JSON ( #31052 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-11 19:55:02 +08:00
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building ( #30357 )
...
This commit adds sparse float vector support to segcore with the
following:
1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built
This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
a99143dd52
fix: Save traceID and spanID as hex string into search config ( #31071 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-11 14:21:01 +08:00
53af6d8c59
[automated] Update Knowhere Commit ( #31151 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-09 01:55:02 +08:00
122981aeb9
fix: Disable knowhere trace as a quick fix ( #31055 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-08 15:27:01 +08:00
3574bdf858
enhance: ban range-search iteration for search-group-by ( #30824 )
...
related: #30033
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-08 14:17:00 +08:00
19c64067af
enhance: jemalloc aarch64 platform use 64k pagesize. ( #29522 )
...
enhance: jemalloc aarch64 platform use 64k pagesize.
issue: #28843
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2024-03-07 21:01:01 +08:00
2d9de233fc
[automated] Update Knowhere Commit ( #31089 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-07 12:05:02 +08:00
c047f09110
[automated] Update Knowhere Commit ( #31015 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-05 16:31:00 +08:00
4bda6c33ad
fix: binary vector should not limit dimension to 32768 ( #30676 )
...
all the vector dimension check should happen on collection creation but
not index build
fix #30285
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-05 14:21:00 +08:00
3dc5e38240
[automated] Update Knowhere Commit ( #30989 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-04 16:34:59 +08:00
29f44f840a
enhance: refine groupBy error msg( #29968 ) ( #30920 )
...
related: #29968
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-01 18:53:03 +08:00
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
e2f35954d4
enhance: support pattern matching on json field ( #30779 )
...
issue: https://github.com/milvus-io/milvus/issues/30714
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
16b785e149
enhance: optimize the memory usage and speed up loading variable length data ( #30787 )
...
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
4459078e0b
fix: wrong num_entities used when mmap variable length data ( #30848 )
...
https://github.com/milvus-io/milvus/issues/30728
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
a115b731ed
enhance: fix old pr cpp format issue ( #30894 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
zhagnlu
Chun Han
Jiquan Long
Alexander Guzhva
presburger
cqy123456
cai.zhang
Cai Yudong
PowderLi
congqixia
foxspy
groot
sammy.huang
liliu-z
Gao
Buqian Zheng
sre-ci-robot
Xiaofan