milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,542 Commits
146 Branches
120 Tags
573 MiB
Commit Graph

426 Commits (038eebba4db629edd2498abf1b51009a4b91bf22)

Author SHA1 Message Date
zhagnlu 9489e14000
Optimize multi logical exprs performance when meet some situations (#26265) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 15:31:29 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
Enwei Jiao ca1349708b
Remove time travel ralted testcase (#26119) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-10 18:53:17 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
yah01 300fef446b
Enable mmap for vector index (#25877) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-10 13:59:15 +08:00
Jiquan Long 6578a27fe7
Support empty expression (#26153) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-10 08:37:15 +08:00
Jiquan Long bafb183a2b
Optimize bitset usage (#26096) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-03 15:25:09 +08:00
MrPresent-Han 5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
Jiquan Long 5c1f79dc54
Push down the limit operator to segcore (#25959) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-01 20:29:05 +08:00
yah01 9618bd9b42
Set channel capacity before consuming it (#25895) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-26 17:35:01 +08:00
foxspy 5437fcce8e
fix range search (#25880) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-07-25 10:07:01 +08:00
yah01 53c3bf053e
Fix unstable sealed segment bruteforce unittest (#25867) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-07-25 09:05:00 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
zhagnlu 65cb52d06b
Support dynamic simd framework and using term expr as example (#25260) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-13 16:22:30 +08:00
yah01 dd5f896dc8
Load batch by batch (#25212) 
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 13:58:27 +08:00
Jiquan Long c2a2ece551
Fix expression on integer overflow case (#25320) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-07-05 10:18:24 +08:00
foxspy 31173727b2
growing segment index memory opt & get vector bugfix (#25272) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-07-05 00:04:25 +08:00
Enwei Jiao 816158e4af
Remove outdated searchplan (#25282) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-04 18:30:25 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
PowderLi 3f4356df10
fix the spelling of `field` (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yah01 a413842e38
Fix deleted data is still visible (#24849) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-16 17:16:41 +08:00
zhagnlu c5b1533fdc
Change segcore search_id from travelling all bits to select true bits(#24659) (#24800) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-16 16:48:44 +08:00
cqy123456 a519213316
Update knowhere version, update diskann api and generate cache nodes in build process (#24898) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-06-16 14:20:39 +08:00
Jiquan Long 26a6e1b946
Fix BinaryRange expression on integer overflow case (#24920) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-16 09:56:39 +08:00
zhagnlu f60b839127
Support element in json array in segcore part(#24677) (#24829) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-14 14:38:37 +08:00
Jiquan Long c6a6998ba7
Fix term expression on interger overflow case (#24867) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-14 11:44:38 +08:00
Cai Yudong 1b3c4b26f1
Optimize range search result sort in segcore (#24837) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-06-13 19:22:38 +08:00
yah01 ceda0ed598
Optimize the performance of filter by JSON field (#24268) 
- Construct JSON pointer only once
- Avoid copying nested path for each row

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-22 00:47:25 +08:00
yah01 c75e7a5d05
Fix failed to compare int value with double value (#24229) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-05-19 12:57:23 +08:00
zhagnlu 113f9a0ebc
Support SIMD of several Expr (#23715) (#23717) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-05-12 14:11:20 +08:00
cai.zhang 9715a850fa
Support expr with json field (#23804) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-05-10 10:19:19 +08:00
yah01 62eea5286f
Support to filter with json expr (#23739) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-30 20:36:39 +08:00
yah01 60fdd7e4f4
Introduce simdjson (#23644) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-26 10:30:34 +08:00
foxspy 6f4ed517de
add growing segment index (#23615) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-04-26 10:14:41 +08:00
yihao.dai 092d743917
Add support for getting vectors by ids (#23450) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-23 09:00:32 +08:00
yah01 546080dcdd
Support to retrieve json (#23563) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-21 11:46:32 +08:00
Cai Yudong 5f4673fd16
Optimize unittest to save runtime (#23248) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-07 14:20:29 +08:00
Cai Yudong ef63e64ded
Remove ANNOY index type (#23189) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-04 16:30:27 +08:00
Enwei Jiao 50dab2d394
Add tracing for retrieve (#23033) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-29 18:10:02 +08:00
Enwei Jiao ef57ba3bac
Remove easylogging and replace by GLog (#22979) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-27 18:22:02 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Cai Yudong 0e9a4478e3
Remove useless index mode (#22934) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-23 21:39:59 +08:00
Enwei Jiao fd974df09f
OTEL integration in cpp (#22175) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-21 16:31:57 +08:00
zhagnlu 7c633e9b9d
Support aliyun OSS for chunk manager (#22663) (#22842) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-03-20 14:45:57 +08:00
xige-16 9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData (#22726) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-03-20 10:41:56 +08:00
Jiquan Long 8139106b51
Feat: count entities by expression (#22765) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-16 19:31:55 +08:00
Cai Yudong ab3cbdfc61
Partial change to prepare for GPU index type support (#22591) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-14 23:21:56 +08:00
Jiquan Long a36fefb009
Fix cpplint (#22657) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-10 09:47:54 +08:00
XuanYang-cn 955bc06165
Fix search binary pagination failure (#22477) 
See also: #22168

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-08 11:03:51 +08:00
smellthemoon 0dae20c938
Layer range search results (#22539) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-03 15:19:48 +08:00
yah01 7478e44911
Support using mmap to load data (#22052) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-01 18:07:49 +08:00
smellthemoon 8d05874ff9
Fix get index mode error (#22364) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-27 14:45:47 +08:00
smellthemoon 820d0bc060
Fix range search error (#22352) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-23 14:21:46 +08:00
smellthemoon 9e0ec15436
Support range search (#21652) 
Signed-off-by: smellthemoon <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
 2023-02-21 09:48:32 +08:00
xige-16 8ec5ba13dc
Fix search failed on disk index when search_list equal to limit (#21113) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-02-16 11:02:34 +08:00
presburger 9950cacd10
support knowhere 2.0 (#21857) 
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
 2023-02-10 14:24:32 +08:00
yah01 73ce87dfe5
Fix reduce decreasing recall (#21981) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-06 11:23:53 +08:00
yah01 836773f1a0
Optimize performance of reducing segments (#21722) 
- Improve the performance of reducing from O(knlogn) to O(nlogk)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-16 15:25:42 +08:00
smellthemoon bf3c02155a
Support update (#20875) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-04 17:21:36 +08:00
Jiquan Long d7156812c1
Try using ASAN in ci ut (#21089) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-29 15:29:31 +08:00
smellthemoon 1e5cd9dc6e
Modify comment (#21362) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-26 15:27:30 +08:00
xige-16 a20770c172
Delete logs that print sensitive information (#20889) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-12-01 10:35:16 +08:00
Enwei Jiao 958e94f6f0
Use Conan as c++ package manager (#19920) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-23 10:39:11 +08:00
xige-16 34c88cea32
Fix GetDiskUsedSize not work (#20474) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:47:17 +08:00
smellthemoon f5ab719f21
timestamp decided if the pks were the same (#20166) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-21 10:55:10 +08:00
xige-16 4a66965df4
Delete RAW_DATA copy when load IVF_FLAT index data (#20274) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-05 17:33:05 +08:00
zhagnlu cb2591d1fe
Parallel operations of disk-ann file manager (#20120) (#20122) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-11-03 14:39:40 +08:00
zhagnlu 02a8a07322
Fix BucketExists bug in minio chunk manager (#19548) (#19854) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-10-20 10:23:27 +08:00
xige-16 158787811e
Move assemble/disassemble func to core (#19420) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-16 21:17:25 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 17:51:24 +08:00
xige-16 8c9c1672ae
Assign different storage config for indexes (#19517) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 14:45:23 +08:00
congqixia 089912baa7
Fix segcore unit test name (#19783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-14 14:29:24 +08:00
Cai Yudong 87d78a4a85
Ignore cases when comparing metric type in segcore (#19437) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-26 17:58:52 +08:00
aoiasd 2b58bd5c0a
Optimize large memory usage of InsertRecord by using vector instead of unordered_map if InsertRecord used in sealed segment (#19245) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-23 17:08:51 +08:00
xige-16 428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00
aoiasd 58d3a49b62
Optimize large memory usage of Variant used in InsertRecord (#19197) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-16 10:52:47 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00
Cai Yudong 7b5eb85205
Remove redundant parameter for segcore Search() (#19025) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 12:05:11 +08:00
Cai Yudong 765907ab77
Optimize segcore Reduce (#18902) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 10:55:12 +08:00
Cai Yudong 686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935) 
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-05 10:41:11 +08:00
Cai Yudong da96659569
[skip e2e] Add more testcases with different parameter combinations in test_reduce (#18967) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 16:59:02 +08:00
Cai Yudong 4ded45394a
[skip e2e] Optimize test_reduce (#18957) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 11:09:00 +08:00
Cai Yudong 9dc3bbecbd
Change the type of slice_nqs and slice_topks from int32_t[] to int64_t[] (#18867) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-29 11:36:56 +08:00
Letian Jiang c6d9d0df5e
Fix segcore benchmark (#18675) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-08-17 20:24:49 +08:00
Cai Yudong d878cc2ec0
Rename real_topK_per_nq_ to topks_per_nq_prefix_sum_ in SearchResult struct (#18634) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-16 00:32:48 +08:00
zhenshan.cao a287a2b3fd
Return empty result in advance if all data filtered out (#18329) (#18438) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-28 12:36:30 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc (#18349) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-20 22:22:31 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment (#18115) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-18 09:58:28 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API (#18227) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-07-12 11:58:25 +08:00
bigsheeper 22508f36d3
Get search field id from search plan and log if loaded index when search segments (#18183) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-08 20:18:22 +08:00
zhagnlu 257da153ce
Fix core dump when nq has no topk result (#17923) (#18051) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-07-05 19:48:20 +08:00
xige-16 0702993106
Fix query not empty after delete string (#17953) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-07-04 16:54:20 +08:00
Jiquan Long 6954a5ba3e
Fix search successfully with invalid metric type (#17977) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-01 22:28:23 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully (#17959) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-01 10:44:18 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-29 14:20:19 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-24 10:34:18 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore (#17700) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow (#17706) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-22 16:56:13 +08:00