0d20303e54
fix: fix binary vector data size ( #33750 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
- fix byte size wrong for binary vectors
- fix the expect/actual error msg
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-18 21:39:59 +08:00
c5918ffbdb
enhance: mark sparse inverted index as mmap-able ( #33281 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-23 14:11:42 +08:00
bb7765cbd6
fix: fix Indexing.Iterator ut: build index with all data at once ( #32844 )
...
issue: #32843
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-10 11:31:30 +08:00
01c2684355
enhance: [skip e2e] disable unstable ut temporarily ( #32836 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-08 12:17:29 +08:00
337cc0756d
fix: lack good results for insufficient ef( #29883 ) ( #32080 )
...
related: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-13 22:13:23 +08:00
aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. ( #31771 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-07 14:13:16 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building ( #30357 )
...
This commit adds sparse float vector support to segcore with the
following:
1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built
This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
8a219e0102
feat: Support knowhere trace using OpenTelemetry ( #30750 )
...
Issue: #21508
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
e2330f02f8
fix: pattern match use incorrect raw data ( #30764 )
...
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
77eb6defb1
feat: support groupby on growing and non-indexed sealed egment( #30307 ) ( #30644 )
...
related: #30308
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-21 14:02:53 +08:00
a587450e56
enhance: [skip-e2e] disable asan ( #30498 )
...
fix : #30511
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-04 21:25:05 +08:00
e429965f32
Add float16 approve for multi-type part ( #28427 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Add bfloat16 vector, add the index part of float16 vector.
Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
d6429933a7
enhance: make Load process traceable in querynode & segcore ( #29858 )
...
See also #29803
This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 21:58:51 +08:00
36f69ea031
feat: integrate storagev2 in building index of segcore ( #28768 )
...
issue: https://github.com/milvus-io/milvus/issues/28655
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 16:48:54 +08:00
267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment ( #28323 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-10 15:44:22 +08:00
7a65b6fb85
Limit faiss ivf index build thread num and fix ut ( #27567 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-10-11 10:33:33 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
3d05ddf505
Reduce cpp test time ( #27043 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-13 15:41:18 +08:00
c3f15c6b95
Refactor duplicate error class into one place ( #26985 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-11 20:43:17 +08:00
5b8d716cbc
Add ut for growing segment load binlog ( #26268 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-13 20:41:31 +08:00
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager ( #26194 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
300fef446b
Enable mmap for vector index ( #25877 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-10 13:59:15 +08:00
9618bd9b42
Set channel capacity before consuming it ( #25895 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-26 17:35:01 +08:00
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE ( #25708 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
a519213316
Update knowhere version, update diskann api and generate cache nodes in build process ( #24898 )
...
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-06-16 14:20:39 +08:00
092d743917
Add support for getting vectors by ids ( #23450 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-23 09:00:32 +08:00
ef63e64ded
Remove ANNOY index type ( #23189 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-04-04 16:30:27 +08:00
9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData ( #22726 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-03-20 10:41:56 +08:00
a36fefb009
Fix cpplint ( #22657 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-10 09:47:54 +08:00
9e0ec15436
Support range search ( #21652 )
...
Signed-off-by: smellthemoon <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
2023-02-21 09:48:32 +08:00
8ec5ba13dc
Fix search failed on disk index when search_list equal to limit ( #21113 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-02-16 11:02:34 +08:00
9950cacd10
support knowhere 2.0 ( #21857 )
...
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-02-10 14:24:32 +08:00
d7156812c1
Try using ASAN in ci ut ( #21089 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-12-29 15:29:31 +08:00
158787811e
Move assemble/disassemble func to core ( #19420 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-16 21:17:25 +08:00
a1db9038fb
Move disk index params to config file ( #19714 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
8c9c1672ae
Assign different storage config for indexes ( #19517 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
428840178c
Support diskann index for vector field ( #19093 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
a287a2b3fd
Return empty result in advance if all data filtered out ( #18329 ) ( #18438 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-28 12:36:30 +08:00
638f6c36e9
Support to get real row count of segment ( #18115 )
...
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
015a2f0866
Let growing segment call knowhere brute search API ( #18227 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-07-12 11:58:25 +08:00
a001412e12
Replace faiss::MetricType with knowhere::MetricType ( #17891 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-29 14:20:19 +08:00
7385770014
Upgrade to knowhere-v1.1.12 ( #17692 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-24 10:34:18 +08:00
08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks ( #17022 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-23 16:41:58 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
58ea38142f
Use boost dynamic_bitset in segcore ( #16476 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-14 22:37:34 +08:00
a37479d728
Upgrade to knowhere-v1.1.2 to support all index types for mac ( #16416 )
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2022-04-08 15:29:31 +08:00
f4ebd3a9ce
Upgrade to knowhere v1.1.0 ( #16186 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-25 13:49:25 +08:00
Gao
Buqian Zheng
Chun Han
cqy123456
Cai Yudong
Jiquan Long
Xu Tong
congqixia
Bingyi Sun
yah01
foxspy
Enwei Jiao
xige-16
zhagnlu
yah01
yihao.dai
smellthemoon
presburger
zhenshan.cao
Jeng.Gwan