milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
19,505 Commits
97 Branches
161 Tags
577 MiB
Commit Graph

481 Commits (2.4-hotfix)

Author SHA1 Message Date
Jiquan Long 8139106b51
Feat: count entities by expression (#22765) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-16 19:31:55 +08:00
Cai Yudong ab3cbdfc61
Partial change to prepare for GPU index type support (#22591) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-14 23:21:56 +08:00
Jiquan Long a36fefb009
Fix cpplint (#22657) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-10 09:47:54 +08:00
XuanYang-cn 955bc06165
Fix search binary pagination failure (#22477) 
See also: #22168

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-08 11:03:51 +08:00
smellthemoon 0dae20c938
Layer range search results (#22539) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-03 15:19:48 +08:00
yah01 7478e44911
Support using mmap to load data (#22052) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-01 18:07:49 +08:00
smellthemoon 8d05874ff9
Fix get index mode error (#22364) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-27 14:45:47 +08:00
smellthemoon 820d0bc060
Fix range search error (#22352) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-23 14:21:46 +08:00
smellthemoon 9e0ec15436
Support range search (#21652) 
Signed-off-by: smellthemoon <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
 2023-02-21 09:48:32 +08:00
xige-16 8ec5ba13dc
Fix search failed on disk index when search_list equal to limit (#21113) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-02-16 11:02:34 +08:00
presburger 9950cacd10
support knowhere 2.0 (#21857) 
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
 2023-02-10 14:24:32 +08:00
yah01 73ce87dfe5
Fix reduce decreasing recall (#21981) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-06 11:23:53 +08:00
yah01 836773f1a0
Optimize performance of reducing segments (#21722) 
- Improve the performance of reducing from O(knlogn) to O(nlogk)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-16 15:25:42 +08:00
smellthemoon bf3c02155a
Support update (#20875) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-04 17:21:36 +08:00
Jiquan Long d7156812c1
Try using ASAN in ci ut (#21089) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-29 15:29:31 +08:00
smellthemoon 1e5cd9dc6e
Modify comment (#21362) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-26 15:27:30 +08:00
xige-16 a20770c172
Delete logs that print sensitive information (#20889) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-12-01 10:35:16 +08:00
Enwei Jiao 958e94f6f0
Use Conan as c++ package manager (#19920) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-23 10:39:11 +08:00
xige-16 34c88cea32
Fix GetDiskUsedSize not work (#20474) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:47:17 +08:00
smellthemoon f5ab719f21
timestamp decided if the pks were the same (#20166) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-21 10:55:10 +08:00
xige-16 4a66965df4
Delete RAW_DATA copy when load IVF_FLAT index data (#20274) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-05 17:33:05 +08:00
zhagnlu cb2591d1fe
Parallel operations of disk-ann file manager (#20120) (#20122) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-11-03 14:39:40 +08:00
zhagnlu 02a8a07322
Fix BucketExists bug in minio chunk manager (#19548) (#19854) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-10-20 10:23:27 +08:00
xige-16 158787811e
Move assemble/disassemble func to core (#19420) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-16 21:17:25 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 17:51:24 +08:00
xige-16 8c9c1672ae
Assign different storage config for indexes (#19517) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 14:45:23 +08:00
congqixia 089912baa7
Fix segcore unit test name (#19783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-14 14:29:24 +08:00
Cai Yudong 87d78a4a85
Ignore cases when comparing metric type in segcore (#19437) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-26 17:58:52 +08:00
aoiasd 2b58bd5c0a
Optimize large memory usage of InsertRecord by using vector instead of unordered_map if InsertRecord used in sealed segment (#19245) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-23 17:08:51 +08:00
xige-16 428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00
aoiasd 58d3a49b62
Optimize large memory usage of Variant used in InsertRecord (#19197) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-16 10:52:47 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00
Cai Yudong 7b5eb85205
Remove redundant parameter for segcore Search() (#19025) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 12:05:11 +08:00
Cai Yudong 765907ab77
Optimize segcore Reduce (#18902) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 10:55:12 +08:00
Cai Yudong 686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935) 
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-05 10:41:11 +08:00
Cai Yudong da96659569
[skip e2e] Add more testcases with different parameter combinations in test_reduce (#18967) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 16:59:02 +08:00
Cai Yudong 4ded45394a
[skip e2e] Optimize test_reduce (#18957) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 11:09:00 +08:00
Cai Yudong 9dc3bbecbd
Change the type of slice_nqs and slice_topks from int32_t[] to int64_t[] (#18867) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-29 11:36:56 +08:00
Letian Jiang c6d9d0df5e
Fix segcore benchmark (#18675) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-08-17 20:24:49 +08:00
Cai Yudong d878cc2ec0
Rename real_topK_per_nq_ to topks_per_nq_prefix_sum_ in SearchResult struct (#18634) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-16 00:32:48 +08:00
zhenshan.cao a287a2b3fd
Return empty result in advance if all data filtered out (#18329) (#18438) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-28 12:36:30 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc (#18349) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-20 22:22:31 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment (#18115) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-18 09:58:28 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API (#18227) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-07-12 11:58:25 +08:00
bigsheeper 22508f36d3
Get search field id from search plan and log if loaded index when search segments (#18183) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-08 20:18:22 +08:00
zhagnlu 257da153ce
Fix core dump when nq has no topk result (#17923) (#18051) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-07-05 19:48:20 +08:00
xige-16 0702993106
Fix query not empty after delete string (#17953) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-07-04 16:54:20 +08:00
Jiquan Long 6954a5ba3e
Fix search successfully with invalid metric type (#17977) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-01 22:28:23 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully (#17959) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-01 10:44:18 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-29 14:20:19 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-24 10:34:18 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore (#17700) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow (#17706) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-22 16:56:13 +08:00
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search (#17565) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-17 17:46:10 +08:00
xige-16 36ad989590
Fix segOffset grater than insert barrier when mark delete (#17444) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-10 20:02:08 +08:00
bigsheeper f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 21:34:05 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS (#17316) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-02 10:48:03 +08:00
xige-16 56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
xige-16 7d810ac159
Fix query empty when insert same pk after deletion (#17222) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-25 22:06:00 +08:00
xige-16 a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-24 21:56:00 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
Jiquan Long 77b0f290a8
Fix prefix match on sealed segment (#17050) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-17 20:15:55 +08:00
xige-16 935b729a0c
Fix multiple deletes not taking effect (#17033) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-16 18:39:56 +08:00
Jiquan Long c5149c7e85
Replace text codec of pb with binary codec (#16955) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-13 09:53:53 +08:00
xige-16 08ad77c71b
Delete all repeated primary keys (#16863) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-12 14:09:53 +08:00
bigsheeper 64d2793fe9
Fix wrong result topks when reduce search result (#16815) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-06 21:17:51 +08:00
Cai Yudong 21a1311f66
Merge utils/Utils.h into common/Utils.h (#16762) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-05-03 12:05:50 +08:00
Cai Yudong 6a62ff18bf
Support easylogging config for segcore and knowhere (#16751) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-05-03 08:39:49 +08:00
Cai Yudong d5db4ae463
Merge utils/Types.h with common/Types.h (#16445) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-05-02 22:21:51 +08:00
xige-16 515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
Aivin V. Solatorio 04fffb08ef
Support arithmetic operations on numerical fields for scalar filtering (#16520) 
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
 2022-04-24 16:43:45 +08:00
yah01 3f5733b738
Return the local variable directly, make RVO enabled (#16201) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-22 17:43:42 +08:00
zhenshan.cao 58ea38142f
Use boost dynamic_bitset in segcore (#16476) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-14 22:37:34 +08:00
Cai Yudong 70bd6b6e6d
Remove dependency on knowhere::scalar (#16458) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-04-12 09:49:33 +08:00
xige-16 27b4cbc098
Cherry pick remove translateHits commit to mater (#16436) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-08 20:27:31 +08:00
Cai Yudong a37479d728
Upgrade to knowhere-v1.1.2 to support all index types for mac (#16416) 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-04-08 15:29:31 +08:00
Cai Yudong 6c4d497d73
Fix mac build error (#16415) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-04-07 17:29:31 +08:00
Cai Yudong 543ec4a209
Rename log to milvus_log (#16383) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-04-05 22:55:30 +08:00
Cai Yudong 7323180861
Move bench to directory unittest (#16386) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-04-05 12:43:30 +08:00
Cai Yudong 0c53f06a33
Fix build error on mac (#16380) 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-04-04 21:09:32 +08:00
Jiquan Long fd589baca7
Integrates marisa trie index (#16192) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-01 15:31:29 +08:00
zhenshan.cao b03da87df5
Accelerate query speed in sealed segment (#16287) 
Fix delete duplicate key

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-31 17:33:28 +08:00
Cai Yudong f4ebd3a9ce
Upgrade to knowhere v1.1.0 (#16186) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-25 13:49:25 +08:00
Jiquan Long 48706f416f
Migrate scalar index from knowhere (#16174) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-24 14:57:26 +08:00
Letian Jiang 321105cc01
Add column-based insert interface in segcore (#16100) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-23 19:05:25 +08:00
Jiquan Long f8d9bc919d
Unify interface of vector index & scalar index. (#15959) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-21 14:23:24 +08:00
Letian Jiang 8f52e5b6c7
Add unit tests in segcore (#15960) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-10 16:33:59 +08:00
Cai Yudong 9a6ee61731
Change all fields in Expr to const to avoid modification (#15663) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-22 14:45:51 +08:00
zhenshan.cao 142848fcc3
Abandon using protobuf to pass binaryset parameter (#15626) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-02-18 18:39:50 +08:00
jaime 307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
bigsheeper ebed1a68ff
Add log for segcore search (#15159) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-11 18:07:34 +08:00
zhenshan.cao f729853be7
Add unittest for delete and retrieve (#14373) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-28 22:18:47 +08:00
Cai Yudong 4884541896
Update unittest log (#13161) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-10 15:55:08 +08:00
Cai Yudong 15b932c63c
Add unittest IVFFlatNM for Indexing (#13044) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-09 14:21:28 +08:00
Cai Yudong 1ae249adb5
Update profiler CMakeLists.txt (#13001) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-08 23:23:06 +08:00
Cai Yudong 45bac3e4ec
Move profiler under core/thirdparty (#12949) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-08 13:01:04 +08:00
Cai Yudong 41ba52bb28
Support C unittest profiling (#12899) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-07 20:17:03 +08:00
Cai Yudong b053317545
Rename FaissMetricTypeToString to MetricTypeToString (#12767) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-07 10:13:53 +08:00
Cai Yudong cbb01051f0
Update Search return type (#12578) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-02 11:45:32 +08:00
dragondriver a44c00b0d0
Format test_c_api.cpp (#12515) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-01 15:28:00 +08:00
dragondriver 1db1cf2104
Fix similarity correlation (#12511) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-01 15:26:02 +08:00
Cai Yudong 365b5a5d01
Rename SubSearchResult fields for better readability (#12341) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-29 17:07:40 +08:00
Cai Yudong cb952d6036
Rename SearchResult fields for better readability (#12327) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-29 14:57:18 +08:00
Cai Yudong 45fac75889
Fix segcore unittest fail (#12326) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-29 12:27:17 +08:00
dragondriver 73f18c564f
Refine logs when search with unsupported metric type (#12185) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-23 10:13:14 +08:00
Cai Yudong 3aca73969f
Optimize segcore API arrangement (#12135) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-19 17:23:12 +08:00
dragondriver 8c951217ee
Fix wrong distances caused by metric type (#11901) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-16 19:11:10 +08:00
Cai Yudong b0054f1b48
Update C API NewSegment (#11809) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-15 16:57:09 +08:00
Cai Yudong 5fdc6626cb
Change API retrieve return type from CProtoResult to CProto (#11555) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-12 10:04:49 +08:00
dragondriver daaeb27ee1
Make AdapterMgr.GetAdapter thread-safe (#11674) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-11 20:42:44 +08:00
Cai Yudong db2a0a3bd3
Fix reduce panic (#11325) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-05 18:17:00 +08:00
zhenshan.cao cd79fd07c4
Fix timetravel bug on sealed segment (#11152) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-11-03 12:40:12 +08:00
Cai Yudong 4241326524
Update test API CheckSearchResultDuplicate (#11019) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-01 18:31:59 +08:00
yukun d8044299a0
Add uids and seg_offsets check in segcore (#10983) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-11-01 15:21:52 +08:00
groot 57783827fb
Add cpp coverage (#10801) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-28 18:54:35 +08:00
Cai Yudong 1227b9ebae
Rename utils/tools.h to utils/Utils.h (#10665) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-26 17:28:23 +08:00
yukun 7acb48919b
Add Delete func in Segcore SegmentSealed (#10501) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-10-24 14:19:10 +08:00
yukun bc083950a8
Add delete in segcore segment sealed (#10470) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-10-22 23:35:18 +08:00
yukun fe1927c05d
Support delete in SegmentSealed (#10181) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-10-22 13:11:11 +08:00
Cai Yudong 48648c818b
Remove duplicated search results in segcore reduce (#10117) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-20 15:40:49 +08:00
Cai Yudong eda9d550a8
Code clean for test_timestamp_index (#10148) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-19 11:10:34 +08:00
Cai Yudong d54f3426e8
Reorder header files for test_segcore (#10051) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-18 18:38:47 +08:00
Cai Yudong 027103b4f6
Reorder header files for test_span (#10053) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-18 18:10:57 +08:00
Cai Yudong 3e3f8b825a
Reorder header files for test_sealed (#10049) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-18 18:09:04 +08:00
Cai Yudong ffd67de9cb
Reorder header files for test_reduce (#9934) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-15 17:00:43 +08:00
Cai Yudong 356b7e36ba
Reorder header files for test_query (#9932) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-15 16:56:51 +08:00
Cai Yudong 5a8aca01fd
Reorder header files for test_plan_proto (#9871) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-14 16:52:33 +08:00
Cai Yudong 8dbc3db393
Remove dummy test_naive (#9850) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-14 12:52:33 +08:00
Cai Yudong 30661321f2
Remove dummy test_interface (#9848) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-14 12:50:37 +08:00
Cai Yudong 937e59f6c3
Remove dummy test_load (#9847) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-14 11:06:39 +08:00
yukun 6c88774624
Add get final filtered bitmap in SegmentGrowing (#9780) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-10-13 16:54:34 +08:00
Cai Yudong 92c7eb5a09
Code clean for test_interface (#9767) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-13 11:06:33 +08:00
Cai Yudong ca5e3e8644
Code clean for test_indexing (#9765) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-13 11:00:39 +08:00
Cai Yudong 01db5dc6dd
Code clean for test_init (#9766) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-13 10:58:33 +08:00
Cai Yudong 2d8e86692a
Format test_expr.cpp (#9698) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-12 12:48:34 +08:00
Cai Yudong e790ba3e41
Reorder header files (#9700) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-12 12:44:33 +08:00
Cai Yudong 6f7031111a
Update header file (#9696) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-12 12:42:33 +08:00
Cai Yudong 9229d56c96
Clear redundant code line for test_bitmap (#9632) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-11 17:08:29 +08:00
Cai Yudong b8519416c0
Clear redundant code line (#9631) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-11 17:04:29 +08:00
Cai Yudong 4908b1c461
Update header file (#9625) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-11 16:35:52 +08:00
elfisworking d4232f88a2
For search funtion adding a round_decimal paramter to precision control (#8574) 
Signed-off-by: elfisworking <zymustb@126.com>
 2021-10-08 17:39:55 +08:00
Cai Yudong 3e487bde0b
Clean test_concurrent_vector.cpp (#8532) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-25 12:13:56 +08:00
groot af1bdaf202
Reduce cpp unittest time cost (#8409) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-23 17:55:54 +08:00
groot 8b81ceb5d7
Reduce cpp unittest time cost (#8403) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-23 16:55:54 +08:00
Cai Yudong 83ffa18c37
[skip ci] Update test_concurrent_vector (#8273) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-22 17:21:53 +08:00
Cai Yudong 07aa453270
[skip ci] Update test_segcore (#8270) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-22 17:19:53 +08:00
yukun 82e48fb379
Add more error messages in segcore visitors (#8195) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-09-18 15:53:51 +08:00
Cai Yudong 2a946707b9
[skip ci] Update test_concurrent_vector.cpp (#8205) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 14:41:50 +08:00
zhenshan.cao 4c184921f0
Add comments and enchanme unittest for retrieve (#8114) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-16 23:43:49 +08:00
zhenshan.cao 5cd9ea5f04
Fixbug: init optional retrieve_ret_ when query result is empty (#8068) 
Add unittest

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-16 20:11:49 +08:00
Cai Yudong 14ef405a6b
Rename chunk_size to chunk_rows for better code readability (#7953) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-15 15:15:52 +08:00
Cai Yudong a0fd2707cb
Support config SIMD type (#7942) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-15 12:57:48 +08:00
Cai Yudong e6c384b4d4
Move segcore chunk_size configuration to querynode.yaml (#7913) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-15 10:35:52 +08:00
Cai Yudong b5923ff27d
Update segcore unittest format (#7860) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-14 11:22:57 +08:00
Cai Yudong 74ac528095
Improve segcore (#7851) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-14 10:53:04 +08:00
Cai Yudong 27dcf698d3
Support set segcore chunk_size via config file (#7635) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-11 14:40:01 +08:00
yukun 94272bba87
Support query by expression (#7386) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-09-03 17:12:55 +08:00
xige-16 76066b5659
Fix IVF_FLAT index's wrong meta_data (#7167) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-08-19 14:26:11 +08:00
Cai Yudong 6c75301c70
optimize search reduce logic (#7066) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-12 18:00:11 +08:00
xaxys e43b43e7d8
Refactor `RangeExpr` & `CompareExpr` (#6786) 
* Changed `RangeExpr` proto to `UnaryRangeExpr` & `BinaryRangeExpr`
Several unit test unpassed.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix bugs to pass unit test. Fix format.

Signed-off-by: xaxys <tpnnghd@163.com>

* Remove debug information.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix format.
Remove debug information.
Unify variable name.
Add error information.
Remove `CompareExpr` test in `test_c_api.cpp`.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix code format.

Signed-off-by: xaxys <tpnnghd@163.com>

* Update `Plan.cpp`.

Signed-off-by: xaxys <tpnnghd@163.com>
 2021-07-25 10:23:19 +08:00
dragondriver 2821c08560
Add unittests for metric_type: tanimoto (#6773) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-07-23 20:59:33 +08:00
FluorineDog aba21baf82
refactor expr executor to use single bitset (#6667) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-23 15:30:12 +08:00
Cai Yudong 744f2c7f51
optimize test_c_api.cpp (#6637) 
* optimize test_c_api.cpp

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* clean redundant file

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* change DIM to const

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize test_c_api.cpp

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update reduce testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-20 17:13:21 +08:00
FluorineDog 234954931f
remove deprecated (#6623) 
* remove deprecated

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* fix timeout

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-20 15:18:08 +08:00
dragondriver 99249a0224
Fix bug: override the compare function of SearchResultPair (#6628) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-07-20 14:38:11 +08:00
Cai Yudong 724f10b9a0
Unify the usage of query and search (#6467) 
Unify the usage of query and search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-13 22:20:33 +08:00
xaxys e5f8a77a2d
Extend grammar of boolean expression (#6460) 
* Extend grammar of boolean expression

Signed-off-by: xaxys <tpnnghd@163.com>
 2021-07-13 14:37:03 +08:00
FluorineDog 2ff072ea7b
Use larger dim for binary test to avoid false positive test failed (#6416) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-09 18:12:40 +08:00
FluorineDog 97b1426666
fix benchmark, add support for primary key in schemaHelper (#6281) 
* fix benchmark

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* fix tests

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-06 09:50:01 +08:00
dragondriver 8dd2051a9a
Fix 5045, decrease the nb and nlist in unittest (#6282) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-07-02 18:20:14 +08:00
FluorineDog bf8b2be4a7
Deprecate num_groups to simplify search API (#6230) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-01 10:32:15 +08:00
FluorineDog 255e3959af
support time travel (#5894) 
* support time travel

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-19 17:38:11 +08:00
FluorineDog 9a90313390
Support GetEntityByIDs in CGo, fix segcore bugs (#5563) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-04 10:38:34 +08:00
FluorineDog b1a9aea6a6
support get entity by ids in segcore (#5456) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-05-28 10:39:30 +08:00
FluorineDog 3f42a9ed12
reduce unittest time (#5363) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-05-24 03:54:06 +00:00
yukun ecaef24fea
Change should_not to must_not in dsl (#5131) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-07 11:52:24 +00:00
dragondriver b7977698a8
Change the codec style of expr plan from text to binary (#5129) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 11:27:17 +00:00
dragondriver e5d4963ba0
Distinguish the dsl and expr in proxy and query node (#5118) 
Pass a parameter `IsExpr` from proxy to query node, so query 
node can create search plan according to the variable `IsExpr`.
It makes `segcore` unnecessary to distinguish `dsl` or `expr`
using dynamic exception. When `IsExpr` is set to true, query
node will call `CreatePlanByExpr` according to the passed
information about expression. Otherwise query node will keep
still to use `CreatePlan` according to the `dsl` information. At
the same time, this pr adds some unittests to `CreatePlanByExpr`,
these unittests translate already exist case with `dsl` to case with
`expr`.

Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-07 07:20:47 +00:00
FluorineDog 396b3f33e9
Support TermExpr, NotExpr, LogicalExpr (#5096) 
1. Support Term, like `A in [1, 2, 3]`
2. Support Not, like `! A < 3`
3. Support logical combination, like `A < 3 && B > 5 or C == 0`

Type: Feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-04-30 07:19:52 +00:00
FluorineDog 88f5642603
Add plan proto and support basic boolean expr parser (#5088) 
**What type of PR is this?**
- [x] Feature

**What this PR does / why we need it:**
This PR supports boolean expression as DSL.
1. The goal of this PR is to support predicates
    like `A > 3 && not B < 5 or C in [1, 2, 3]`. 
2. Defines `plan.proto`, as Intermediate Representation (IR) 
    used between go and cpp. 
3. Support expr parser, convert predicate expr to IR
    in proxynode, while doing static check there
4. Support IR to AST in cpp, enable the execution
 2021-04-29 08:48:06 +00:00
FluorineDog 6059558698 Add license files 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-19 11:16:16 +08:00
FluorineDog ce7a5ea699 Support segcoreinit 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-04-16 14:02:49 +08:00
XuanYang-cn 46d27e37d7 Skip install gtest when build with test 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-08 10:01:18 +08:00
dragondriver 9d062b54ee Add benchmark for indexbuilder, refactor test utils 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-23 18:44:57 +08:00
zhenshan.cao c2734fa55f Fix bug and enchance system 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-22 16:36:10 +08:00
FluorineDog 2cec04ed90 Fix empty schema proto hack 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-17 11:35:28 +08:00
xige-16 8a5c039137 Fix search error when running single node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-12 19:23:06 +08:00
FluorineDog e33d0a797c Migrate knowhere to segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-12 15:54:59 +08:00
godchen f3649f0419 Refactor interface and proto 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-12 14:22:09 +08:00
FluorineDog ef98dab2a9 Support segcore config 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-04 17:09:48 +08:00
FluorineDog 66146223ca Support flat 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-02-27 12:46:37 +08:00
zhenshan.cao 6b392cbe58 Fix memory leak 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-23 14:35:00 +08:00
dragondriver 68518fec38 Fix memory leak in indexnode 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-22 18:33:40 +08:00
xige-16 f62078c027 Fix crash error when search 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-20 10:14:03 +08:00
xige-16 2ca53fa668 Fix msgstream deadlock when loadCollection 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-07 15:47:10 +08:00
bigsheeper 01e9dc8e3f Remove collection name 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 11:52:19 +08:00
sunby f3aad3a71c Change SegmentInfo 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 18:53:10 +08:00
bigsheeper 5e781b9370 Remove field name in query node and segCore 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 10:10:07 +08:00