be77ceba84
enhance: Use proto for passing info in cgo ( #33184 )
...
issue: #33183
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 10:31:40 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building ( #30357 )
...
This commit adds sparse float vector support to segcore with the
following:
1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built
This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment ( #28323 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-10 15:44:22 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
5b8d716cbc
Add ut for growing segment load binlog ( #26268 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-13 20:41:31 +08:00
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager ( #26194 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE ( #25708 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
ef63e64ded
Remove ANNOY index type ( #23189 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-04-04 16:30:27 +08:00
a36fefb009
Fix cpplint ( #22657 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-10 09:47:54 +08:00
9950cacd10
support knowhere 2.0 ( #21857 )
...
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-02-10 14:24:32 +08:00
d7156812c1
Try using ASAN in ci ut ( #21089 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-12-29 15:29:31 +08:00
158787811e
Move assemble/disassemble func to core ( #19420 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-16 21:17:25 +08:00
8c9c1672ae
Assign different storage config for indexes ( #19517 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
428840178c
Support diskann index for vector field ( #19093 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
4de1bfe5bc
Add cpp data codec ( #18538 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: ( #18935 )
...
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-05 10:41:11 +08:00
7385770014
Upgrade to knowhere-v1.1.12 ( #17692 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-24 10:34:18 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
b03da87df5
Accelerate query speed in sealed segment ( #16287 )
...
Fix delete duplicate key
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-31 17:33:28 +08:00
f4ebd3a9ce
Upgrade to knowhere v1.1.0 ( #16186 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-25 13:49:25 +08:00
f8d9bc919d
Unify interface of vector index & scalar index. ( #15959 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-21 14:23:24 +08:00
142848fcc3
Abandon using protobuf to pass binaryset parameter ( #15626 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-02-18 18:39:50 +08:00
307a8ce535
Support compile and run on Mac ( #15491 )
...
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
2022-02-09 14:27:46 +08:00
d8044299a0
Add uids and seg_offsets check in segcore ( #10983 )
...
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-11-01 15:21:52 +08:00
57783827fb
Add cpp coverage ( #10801 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-10-28 18:54:35 +08:00
e790ba3e41
Reorder header files ( #9700 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-12 12:44:33 +08:00
8b81ceb5d7
Reduce cpp unittest time cost ( #8403 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-09-23 16:55:54 +08:00
76066b5659
Fix IVF_FLAT index's wrong meta_data ( #7167 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-08-19 14:26:11 +08:00
e43b43e7d8
Refactor `RangeExpr` & `CompareExpr` ( #6786 )
...
* Changed `RangeExpr` proto to `UnaryRangeExpr` & `BinaryRangeExpr`
Several unit test unpassed.
Signed-off-by: xaxys <tpnnghd@163.com>
* Fix bugs to pass unit test. Fix format.
Signed-off-by: xaxys <tpnnghd@163.com>
* Remove debug information.
Signed-off-by: xaxys <tpnnghd@163.com>
* Fix format.
Remove debug information.
Unify variable name.
Add error information.
Remove `CompareExpr` test in `test_c_api.cpp`.
Signed-off-by: xaxys <tpnnghd@163.com>
* Fix code format.
Signed-off-by: xaxys <tpnnghd@163.com>
* Update `Plan.cpp`.
Signed-off-by: xaxys <tpnnghd@163.com>
2021-07-25 10:23:19 +08:00
2821c08560
Add unittests for metric_type: tanimoto ( #6773 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-07-23 20:59:33 +08:00
8dd2051a9a
Fix 5045, decrease the nb and nlist in unittest ( #6282 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-07-02 18:20:14 +08:00
3f42a9ed12
reduce unittest time ( #5363 )
...
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-05-24 03:54:06 +00:00
9d062b54ee
Add benchmark for indexbuilder, refactor test utils
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-03-23 18:44:57 +08:00
68518fec38
Fix memory leak in indexnode
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-22 18:33:40 +08:00
1104f059ee
Update doc:module interfaces
...
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-13 11:08:03 +08:00
8e9d8e36e1
Update run_go_unittest.sh
...
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-01-13 10:40:46 +08:00
4cd42c553f
Rename field_name, make field_id strongly typed, skip multithread test
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-12 18:31:52 +08:00
3ae04794f6
Fix bug in describeIndexProgressTask
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-07 15:39:20 +08:00
34a00aacfa
Enable binary index in segcore
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-07 15:00:42 +08:00
ced56c84f8
Add support to build binary vector index
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-01-07 14:56:17 +08:00
8f1f118b26
Add query opentracing
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-01-07 14:24:35 +08:00
a1e2ff3c26
Add gcNode to write node
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-06 16:44:12 +08:00
4015d7245d
Merge operation
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-06 14:45:50 +08:00
e16a7478da
Add verifier
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-05 20:13:54 +08:00
bccf8d2098
Add query support for ivf-flat, nsg
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-01-05 19:20:55 +08:00
1ba47ac433
Check system field in segCore
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-04 10:13:01 +08:00
59878b1dbb
Add lazy connect logic for indexbuilder
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-12-31 15:44:06 +08:00
cai.zhang
Buqian Zheng
yah01
foxspy
xige-16
zhagnlu
Cai Yudong
Jiquan Long
presburger
zhenshan.cao
jaime
yukun
groot
xaxys
FluorineDog
GuoRentong
quicksilver
FluorineDog
sunby
godchen
bigsheeper
neza2017