Commit Graph

1106 Commits (32827f538a8fa9c08cd8c71879e90bea2ad06653)

Author SHA1 Message Date
yah01 99f31dc37a
Fix data after travel timestamp is still visible (#22813)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-17 18:03:55 +08:00
Jiquan Long dff15c3488
Check dimension of inserted records (#22819)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-17 17:33:58 +08:00
presburger 5841a3707b
suppport for raft gpu (#22790)
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-03-17 11:59:54 +08:00
yah01 ab1ac69712
Fix inserting into growing segment copies data (#22792)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-16 20:11:55 +08:00
Jiquan Long 8139106b51
Feat: count entities by expression (#22765)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-16 19:31:55 +08:00
Cai Yudong ab3cbdfc61
Partial change to prepare for GPU index type support (#22591)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-03-14 23:21:56 +08:00
yah01 a4031da634
Refine string parameters, avoid coping or deref (#22708)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-13 17:53:53 +08:00
yah01 8f3d6e08df
Optimize the performance of matching prefix for string index (#22712)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-13 17:31:53 +08:00
yah01 7bc3309918
Replace NULL with nullptr (#22701)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-13 10:35:52 +08:00
Jiquan Long a36fefb009
Fix cpplint (#22657)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-10 09:47:54 +08:00
yah01 e4120d1671
Upgrade proto to add error code field of Status (#22656)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-09 18:49:51 +08:00
XuanYang-cn 955bc06165
Fix search binary pagination failure (#22477)
See also: #22168

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-08 11:03:51 +08:00
smellthemoon c196c55381
Improve the assert information (#22578)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-03-06 14:25:50 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
smellthemoon 0dae20c938
Layer range search results (#22539)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-03-03 15:19:48 +08:00
yah01 bdd6bc7695
Re-format cpp code (#22513)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-02 15:55:49 +08:00
yah01 4d4bf99098
Fix DiskANN can't recovery after querynode reboots (#22488)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-02 10:17:48 +08:00
yah01 1ee10f3224
Refine the information while assertion failed (#22434)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-02 09:07:48 +08:00
cai.zhang 5cae49e28d
Store index files in slices to reduce memory peaks (#22369) (#22455)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-03-01 19:43:47 +08:00
yah01 7478e44911
Support using mmap to load data (#22052)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-01 18:07:49 +08:00
smellthemoon 8d05874ff9
Fix get index mode error (#22364)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-02-27 14:45:47 +08:00
smellthemoon cf999b68dd
Fix range search can not get status from knowhere (#22384)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-02-24 17:59:45 +08:00
smellthemoon 820d0bc060
Fix range search error (#22352)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-02-23 14:21:46 +08:00
yah01 3e9d497157
Avoid allocating a zero-bytes vector each search time (#22219)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-23 10:37:46 +08:00
SimFG 7e19a924b7
Add the rbac for the GetLoadingProgress and GetLoadState api (#22320)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-02-22 20:37:45 +08:00
smellthemoon 7a4dfcc72b
Add status error output (#22325)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-02-22 15:51:44 +08:00
smellthemoon 9e0ec15436
Support range search (#21652)
Signed-off-by: smellthemoon <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
2023-02-21 09:48:32 +08:00
Bennu ac6e879811
[skip e2e] fix etcd image for healthcheck (#22294)
Signed-off-by: Bennu-Li <yunmei.li@zilliz.com>
2023-02-20 17:08:25 +08:00
Enwei Jiao b25b3ef431
Integreation with Velox (#22102)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-16 17:26:35 +08:00
yah01 187788059b
Fix double copy varchar field while loading (#22114)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-16 17:16:35 +08:00
xige-16 8ec5ba13dc
Fix search failed on disk index when search_list equal to limit (#21113)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-02-16 11:02:34 +08:00
Cai Yudong 68f146d5df
Show knowhere version when system up (#22172)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-02-14 19:04:34 +08:00
presburger 9950cacd10
support knowhere 2.0 (#21857)
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-02-10 14:24:32 +08:00
wei liu d078441522
support rg api rbac (#22097)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-10 10:54:33 +08:00
liliu-z 915503de0b
Upgrade knowhere to 1.3.8 (#22087)
Signed-off-by: Li Liu <li.liu@zilliz.com>
2023-02-09 18:58:32 +08:00
Cai Yudong fc1d2d18cf
Optimize knowhere CMakeLists.txt (#22000)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2023-02-07 16:12:04 +08:00
yah01 73ce87dfe5
Fix reduce decreasing recall (#21981)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-06 11:23:53 +08:00
zhenshan.cao 5316743a00
Reduce memory usage of scalar index (#21970)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-02-03 19:01:53 +08:00
Enwei Jiao 6d05f102c1
Upgrade boost to 1.81 (#21954)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-03 17:21:53 +08:00
yah01 bdcff5b921
Avoid counting all bits to improve query performance (#21909)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-01 18:49:51 +08:00
Cai Yudong e3d1624e97
Handle distance Inf correctly (#21828)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2023-02-01 14:59:51 +08:00
yah01 61c2f29ab3
Fix miss to throw exception when failed to create dir (#21886)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-31 13:49:50 +08:00
wei liu 73c44d4b29
resource group impl (#21609)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
congqixia 0480719e1a
Fix useasan format not correct (#21829)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-28 11:43:47 +08:00
jaime 1dcd9eeb79
Support rename collection (#21693)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-19 14:13:43 +08:00
presburger e64f55cd27
downgrade curl for compiled on aarch64 platform (#21739)
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-01-17 14:53:42 +08:00
jaime 81326ca0a1
[skip e2e]Fix cpp lint errors (#21750)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-17 09:43:42 +08:00
Xiaofan e7f947ed0a
Check ScalarIndex deserialization (#21724)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-16 17:29:43 +08:00
yah01 836773f1a0
Optimize performance of reducing segments (#21722)
- Improve the performance of reducing from O(knlogn) to O(nlogk)

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-16 15:25:42 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown (#21653)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-13 15:31:41 +08:00
Enwei Jiao 564173dcbc
Upgrade protobuf to 3.21.9 (#21679)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-13 14:05:41 +08:00
Enwei Jiao 49f53003e9
upgrade libcurl to fix mac build failed (#21684)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 19:03:40 +08:00
wei liu c67070ab1f
add resource group interface (#21676)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-12 18:31:41 +08:00
bigsheeper 79535931ab
Return error code when deny to read/write (#21561)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-01-06 14:31:37 +08:00
yah01 e8e2696430
avoid checking search params many times (#21516)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-06 13:15:35 +08:00
jaime 58b79eb74c
Add based on timetravel GC for snapshot KV (#21417)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-04 21:37:35 +08:00
smellthemoon bf3c02155a
Support update (#20875)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-01-04 17:21:36 +08:00
bigsheeper c187de7754
Show trigger reason when deny to read/write (#21368)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-30 18:35:32 +08:00
Jiquan Long d7156812c1
Try using ASAN in ci ut (#21089)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-12-29 15:29:31 +08:00
Ten Thousand Leaves 30b86275ac
Make BulkInsert free DataNode workload when import segments are flushed (#21253)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-28 11:11:30 +08:00
smellthemoon 1e5cd9dc6e
Modify comment (#21362)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-12-26 15:27:30 +08:00
Enwei Jiao 77de8fc746
Fix boost compile error (#21374)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-26 09:57:29 +08:00
Enwei Jiao ae41a6583f
Fix compile on M1 (#21298)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-19 15:05:24 +08:00
jaime e05ac56283
Revert build arrow with Conan (#21258)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-16 18:27:24 +08:00
liliu-z 6188960e85
Upgrade Knowhere to 1.3.6 to fix bugs (#21240)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-15 12:15:22 +08:00
yah01 cc56d58052
Fix panic while concurrent releasing payload writer/reader (#21218)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-14 16:23:22 +08:00
SimFG 655db658c8
Empty implement of the `GetLoadState` api (#21205)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-14 10:23:22 +08:00
Enwei Jiao d553b5108e
Fix jemalloc.so not found (#21106)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-13 11:27:21 +08:00
liliu-z d326c385bb
Upgrade Knowhere to 1.3.5 to fix DiskANN aio issue (#21109)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-09 22:53:20 +08:00
Xiaofan e131915207
Support 64M Rpc Limit (#20845)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-07 10:07:18 +08:00
SimFG f8cff79804
Support the graceful stop for the query node (#20851)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
liliu-z aecf2e4f4b
Upgrade knowhere to 1.3.4 (#20914)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-02 00:33:21 +08:00
xige-16 a20770c172
Delete logs that print sensitive information (#20889)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-12-01 10:35:16 +08:00
Cai Yudong 4c5ffc832c
Add param check for BruteForceSearch in segcore (#20838)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-11-30 14:53:15 +08:00
Jiquan Long 8d78d9943a
Fix term filtering failed on indexed scalar field (#20840) (#20882)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-30 10:05:15 +08:00
Enwei Jiao 9b61170bf0
Fix compile proto failed (#20820)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-28 09:47:14 +08:00
yah01 1eabfdb199
Avoid allocating and coping when fill column with default values (#20721)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-23 15:15:11 +08:00
Enwei Jiao 958e94f6f0
Use Conan as c++ package manager (#19920)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-23 10:39:11 +08:00
xige-16 34c88cea32
Fix GetDiskUsedSize not work (#20474)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 17:47:17 +08:00
xige-16 15badd0263
Fix cache disk index error (#20419)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 17:45:23 +08:00
smellthemoon f5ab719f21
timestamp decided if the pks were the same (#20166)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-21 10:55:10 +08:00
liliu-z 0ae4e5265f
Optimize some low efficient code (#20529)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-11-14 21:05:07 +08:00
liliu-z 174310a14e
Use all() api for bitset check (#20462)
Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>

Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-11-10 16:15:05 +08:00
liliu-z 0dc659a04c
Upgrade Knowhere to 1.3.3 to fix potential bugs (#20434)
Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>

Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-11-09 20:19:04 +08:00
Ten Thousand Leaves 3d25a8dc03
Improve panic error readability (#20339)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-08 15:37:03 +08:00
xige-16 4a66965df4
Delete RAW_DATA copy when load IVF_FLAT index data (#20274)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-05 17:33:05 +08:00
zhagnlu 39212bc04c
Fix get cpu num for segcore (#20120) (#20307)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-03 19:29:36 +08:00
zhagnlu cb2591d1fe
Parallel operations of disk-ann file manager (#20120) (#20122)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-03 14:39:40 +08:00
bigsheeper 37dce8a660
Make s3 client creation thread safe, prevent crashing (#20200)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-31 17:09:33 +08:00
xige-16 14968685a2
Limit disk index search list range (#20104)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-26 21:03:31 +08:00
XuanYang-cn c24244240e
Fix insert search concurrency problem (#20065)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-26 18:57:36 +08:00
xige-16 89a9b6fda0
Init storage config's SSL and IAM config (#20082)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-26 16:07:30 +08:00
Jiquan Long 377cd210c2
Make knowhere use segcore's cmake build type (#20041)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-25 14:17:37 +08:00
xige-16 32ba4fafb8
Delete Creating bucket when create core/minioChunkManager (#20049)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-25 14:15:30 +08:00
Enwei Jiao 4e5d1c2e5f
Add log to debug bitset (#19982)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-24 10:09:28 +08:00
zhagnlu 05bbb12d30
Skip build aws sdk when not need diskann (#19965) (#19966)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-10-21 18:35:29 +08:00
liliu-z ce8e4d17b3
Fix mis-copy of chunk in make_with_timestamps (#19970)
Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>

Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-10-21 16:35:29 +08:00
yah01 96cf74d53a
Add more panic info (#19922)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-20 13:33:27 +08:00
zhagnlu 02a8a07322
Fix BucketExists bug in minio chunk manager (#19548) (#19854)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-10-20 10:23:27 +08:00
xige-16 158787811e
Move assemble/disassemble func to core (#19420)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-16 21:17:25 +08:00
SimFG a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
xige-16 8c9c1672ae
Assign different storage config for indexes (#19517)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
congqixia 089912baa7
Fix segcore unit test name (#19783)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-14 14:29:24 +08:00
congqixia a20d83bccd
Assemble RAW_DATA files before storing into index.raw_data (#19771)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-14 12:03:24 +08:00
mumon b48c37cb44
Fix typo in core (#19661)
Signed-off-by: Ziyu Wang <15871035978@163.com>

Signed-off-by: Ziyu Wang <15871035978@163.com>
2022-10-12 17:05:23 +08:00
jaime 31db01b6ae
Support TTL for collection level (#19383)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
xige-16 49daee2e17
Fix lost disk index data (#19636)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-09 14:36:58 +08:00
congqixia 8c1c8000ac
Add TargetID in MsgBase and querynode v2 API check (#19627)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-08 20:26:57 +08:00
liliu-z faa9d2176a
Upgrade Knowhere to 1.3.2 to support large dim in DiskANN (#19531)
Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>

Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-09-29 19:40:55 +08:00
aoiasd a8cabbfaa9
Fix offset in search should not equal to insert barrier (#19522)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-09-28 21:04:54 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
Cai Yudong 87d78a4a85
Ignore cases when comparing metric type in segcore (#19437)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-26 17:58:52 +08:00
aoiasd 2b58bd5c0a
Optimize large memory usage of InsertRecord by using vector instead of unordered_map if InsertRecord used in sealed segment (#19245)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-09-23 17:08:51 +08:00
xige-16 d4bc00423c
Fix start milvus failed on macos (#19394)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-23 16:54:50 +08:00
SimFG 68a257458b
Implement the `hook` interceptor (#19294)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-23 09:50:52 +08:00
xige-16 428840178c
Support diskann index for vector field (#19093)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
Jenny Li d8b11121a8
Decrease build time by removing compiling openblas (#19207)
Signed-off-by: Jenny Li <jing.li@zilliz.com>

Signed-off-by: Jenny Li <jing.li@zilliz.com>
2022-09-16 17:46:47 +08:00
SimFG d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
aoiasd 58d3a49b62
Optimize large memory usage of Variant used in InsertRecord (#19197)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-09-16 10:52:47 +08:00
bigsheeper 5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector (#18636)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-16 09:56:47 +08:00
Bingyi Sun 626854cf0c
Refactor QueryCoord (#18836)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-15 18:48:32 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
Cai Yudong 7b5eb85205
Remove redundant parameter for segcore Search() (#19025)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-06 12:05:11 +08:00
Cai Yudong 765907ab77
Optimize segcore Reduce (#18902)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-06 10:55:12 +08:00
Jiquan Long a5e2d6b6fb
Refactor RootCoord (#18930)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-09-05 13:29:11 +08:00
Cai Yudong 686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935)
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-05 10:41:11 +08:00
Cai Yudong da96659569
[skip e2e] Add more testcases with different parameter combinations in test_reduce (#18967)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-01 16:59:02 +08:00
Cai Yudong 4ded45394a
[skip e2e] Optimize test_reduce (#18957)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-01 11:09:00 +08:00
Cai Yudong 9dc3bbecbd
Change the type of slice_nqs and slice_topks from int32_t[] to int64_t[] (#18867)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-29 11:36:56 +08:00
Letian Jiang ef7742507d
Remove redundant code (#18832)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-08-25 19:48:54 +08:00
Cai Yudong 305601ad25
Add API SearchOnSealedChunk() to search on sealed segment without index (#18830)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-25 18:16:55 +08:00
Cai Yudong dcf45df029
Optimize API vector_search parameter in segcore (#18827)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-25 16:16:54 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643)
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
Ji Bin f3c755602b
Adaptation for latest MinGW64 (#18699)
Signed-off-by: Ji Bin <matrixji@live.com>

Signed-off-by: Ji Bin <matrixji@live.com>
2022-08-25 10:56:53 +08:00
Cai Yudong 94ffa8a275
Upgrade to knowhere-v1.2.0 (#18746)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-23 10:20:58 +08:00
Letian Jiang c6d9d0df5e
Fix segcore benchmark (#18675)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-08-17 20:24:49 +08:00
congqixia 0db281c4ff
Fix DataType::bool cast to double (#18690)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-17 16:14:49 +08:00
Cai Yudong d878cc2ec0
Rename real_topK_per_nq_ to topks_per_nq_prefix_sum_ in SearchResult struct (#18634)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-16 00:32:48 +08:00
xige-16 111c608513
Add script to auto clang format (#18559)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 11:04:37 +08:00
xige-16 dbb96a8b70
Fix pk2offset occupy too much memory (#18325)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 10:48:38 +08:00
SimFG c267ad612d
Support Role-Based Access Control II (#18528)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-05 16:28:35 +08:00
SimFG ff0200210a
Support Role-Based Access Control (#18425)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-04 11:04:34 +08:00
Cai Yudong 7cd37fc6dd
Merge FloatSearch() and BinarySearch() into SearchOnGrowing() (#18498)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-03 11:56:36 +08:00
Enwei Jiao 283f5731d2
config from etcd (#18421)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-01 10:04:33 +08:00
zhenshan.cao a287a2b3fd
Return empty result in advance if all data filtered out (#18329) (#18438)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-28 12:36:30 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc (#18349)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-20 22:22:31 +08:00
Ten Thousand Leaves 0ebe407221
Make DataNode not panic if stale segment is not found during flush (#18322)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-20 15:06:30 +08:00
Jeng.Gwan db71c74cc7
Fix leak of memory in ParseMallocInfo while assert false (#18311)
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-19 14:26:33 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment (#18115)
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
bigsheeper 73aef14820
Improve check of purge memory and change purge place (#18271)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-15 10:22:27 +08:00
xige-16 54d17bc5da
Fix query too slow when insert multi repeated pk data (#18231)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-07-13 10:22:26 +08:00
Jeng.Gwan a926a9eb66
Fix redundant string copy of PlaceholderGroup (#18236)
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-12 15:48:25 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API (#18227)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-07-12 11:58:25 +08:00
bigsheeper 22508f36d3
Get search field id from search plan and log if loaded index when search segments (#18183)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-08 20:18:22 +08:00
zhagnlu 257da153ce
Fix core dump when nq has no topk result (#17923) (#18051)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-07-05 19:48:20 +08:00
Cai Yudong 5edeeb5fcd
Define new SubSearchResult constructor to avoid memory copy for growing (#18000)
segment search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-07-05 17:14:20 +08:00
xige-16 0702993106
Fix query not empty after delete string (#17953)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-07-04 16:54:20 +08:00
Jiquan Long 6954a5ba3e
Fix search successfully with invalid metric type (#17977)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-07-01 22:28:23 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully (#17959)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-01 10:44:18 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-29 14:20:19 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-06-28 20:04:17 +08:00
Enwei Jiao ea901393ef
fix windows ldflags (#17807)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-28 19:58:25 +08:00
bigsheeper b657e58370
Remove temporary variable to prevent memory fragmentation (#17728)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-28 16:30:23 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-24 21:12:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-24 10:34:18 +08:00
Jiquan Long 90ca61d948
Fix double-freed global variable in easylogging lib (#17730)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-23 19:12:14 +08:00
Cai Yudong e78269f450
Optimize search related interface in segcore (#17568)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-23 18:16:13 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore (#17700)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow (#17706)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-22 16:56:13 +08:00
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search (#17565)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-17 17:46:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457)
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-15 12:20:10 +08:00
Jiquan Long e751663fdc
Fix err msg when the dimension of query vectors mismatch field (#17553)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-15 10:54:09 +08:00
bigsheeper cdcdfa1ea5
Disable jemalloc and use malloc_trim instead (#17538)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-14 11:12:09 +08:00
xige-16 36ad989590
Fix segOffset grater than insert barrier when mark delete (#17444)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-10 20:02:08 +08:00
bigsheeper 2d9a52206d
Use jemalloc in QueryNode, DataNode and IndexNode (#17470)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-10 11:24:08 +08:00
xige-16 6d078ebe7a
Print more information when loading fields with different lines (#17448)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-08 20:16:06 +08:00
bigsheeper af108b5bda
Add malloc_trim after reduce search result (#17423)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-08 00:12:05 +08:00
xige-16 b5c11a216d
Alter varChar type params's name to max_length (#17409)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-07 15:58:06 +08:00
bigsheeper f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-06 21:34:05 +08:00
bigsheeper 9f786dd752
Remove unnecessary bitset flip in Search and Query (#17387)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-06 17:38:05 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS (#17316)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-02 10:48:03 +08:00
Xiaofan 9579a645c6
Support compile marisa on Macos (#17261)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-31 13:28:02 +08:00
SimFG 731870211a
Support role-based access control (only pb) (#16987)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-05-28 00:04:01 +08:00
Letian Jiang f2a27e0e64
Retry GetShardLeaders until service available or timeout (#17183)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-26 20:28:02 +08:00
xige-16 56778787be
Reverse data from scalar index (#17145)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-26 14:58:01 +08:00
xige-16 7d810ac159
Fix query empty when insert same pk after deletion (#17222)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-25 22:06:00 +08:00
Ten Thousand Leaves 5e1e7a6896
Fix an issue that causes flush hanging (#17169)
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-25 14:34:00 +08:00
xige-16 a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-24 21:56:00 +08:00
zhenshan.cao d05c24a0ff
Delete unused code (#17182)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-05-24 14:49:59 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-23 16:41:58 +08:00
Enwei Jiao d28a2db46c
move arrow from storage to core (#17061)
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
2022-05-22 20:03:58 +08:00
yah01 7746a5b742
Add NodeIds field for QuerySegmentInfo (#17121)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-05-20 18:03:58 +08:00
Jiquan Long 99b4042f8c
Disable segment small index (#16935)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-05-18 15:27:56 +08:00
Jiquan Long 77b0f290a8
Fix prefix match on sealed segment (#17050)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-05-17 20:15:55 +08:00
xige-16 935b729a0c
Fix multiple deletes not taking effect (#17033)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-16 18:39:56 +08:00
XuanYang-cn 43b36b4e5f
Make proxy update cache with NotShardLeader status code (#16940)
See also: #16926

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-05-13 11:35:53 +08:00
Jiquan Long c5149c7e85
Replace text codec of pb with binary codec (#16955)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-05-13 09:53:53 +08:00
Ten Thousand Leaves 63385b7c12
Make data queryable and data indexed states as independent flags (#16837)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-12 19:23:53 +08:00