Commit Graph

820 Commits (dad060af2e082043de56c60a5cdec107d3dad860)

Author SHA1 Message Date
Cai Yudong 94ffa8a275
Upgrade to knowhere-v1.2.0 ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-23 10:20:58 +08:00
Letian Jiang c6d9d0df5e
Fix segcore benchmark ()
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-08-17 20:24:49 +08:00
congqixia 0db281c4ff
Fix DataType::bool cast to double ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-17 16:14:49 +08:00
Cai Yudong d878cc2ec0
Rename real_topK_per_nq_ to topks_per_nq_prefix_sum_ in SearchResult struct ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-16 00:32:48 +08:00
xige-16 111c608513
Add script to auto clang format ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 11:04:37 +08:00
xige-16 dbb96a8b70
Fix pk2offset occupy too much memory ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 10:48:38 +08:00
SimFG c267ad612d
Support Role-Based Access Control II ()
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-05 16:28:35 +08:00
SimFG ff0200210a
Support Role-Based Access Control ()
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-04 11:04:34 +08:00
Cai Yudong 7cd37fc6dd
Merge FloatSearch() and BinarySearch() into SearchOnGrowing() ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-03 11:56:36 +08:00
Enwei Jiao 283f5731d2
config from etcd ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-01 10:04:33 +08:00
zhenshan.cao a287a2b3fd
Return empty result in advance if all data filtered out () ()
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-28 12:36:30 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-20 22:22:31 +08:00
Ten Thousand Leaves 0ebe407221
Make DataNode not panic if stale segment is not found during flush ()
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-20 15:06:30 +08:00
Jeng.Gwan db71c74cc7
Fix leak of memory in ParseMallocInfo while assert false ()
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-19 14:26:33 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment ()
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
bigsheeper 73aef14820
Improve check of purge memory and change purge place ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-15 10:22:27 +08:00
xige-16 54d17bc5da
Fix query too slow when insert multi repeated pk data ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-07-13 10:22:26 +08:00
Jeng.Gwan a926a9eb66
Fix redundant string copy of PlaceholderGroup ()
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-12 15:48:25 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-07-12 11:58:25 +08:00
bigsheeper 22508f36d3
Get search field id from search plan and log if loaded index when search segments ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-08 20:18:22 +08:00
zhagnlu 257da153ce
Fix core dump when nq has no topk result () ()
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-07-05 19:48:20 +08:00
Cai Yudong 5edeeb5fcd
Define new SubSearchResult constructor to avoid memory copy for growing ()
segment search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-07-05 17:14:20 +08:00
xige-16 0702993106
Fix query not empty after delete string ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-07-04 16:54:20 +08:00
Jiquan Long 6954a5ba3e
Fix search successfully with invalid metric type ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-07-01 22:28:23 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-01 10:44:18 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-29 14:20:19 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter ()
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-06-28 20:04:17 +08:00
Enwei Jiao ea901393ef
fix windows ldflags ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-28 19:58:25 +08:00
bigsheeper b657e58370
Remove temporary variable to prevent memory fragmentation ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-28 16:30:23 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-24 21:12:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-24 10:34:18 +08:00
Jiquan Long 90ca61d948
Fix double-freed global variable in easylogging lib ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-23 19:12:14 +08:00
Cai Yudong e78269f450
Optimize search related interface in segcore ()
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-23 18:16:13 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-22 16:56:13 +08:00
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-17 17:46:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load ()
So we can support DataQueryable and DataIndexed states

issue: 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-15 12:20:10 +08:00
Jiquan Long e751663fdc
Fix err msg when the dimension of query vectors mismatch field ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-15 10:54:09 +08:00
bigsheeper cdcdfa1ea5
Disable jemalloc and use malloc_trim instead ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-14 11:12:09 +08:00
xige-16 36ad989590
Fix segOffset grater than insert barrier when mark delete ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-10 20:02:08 +08:00
bigsheeper 2d9a52206d
Use jemalloc in QueryNode, DataNode and IndexNode ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-10 11:24:08 +08:00
xige-16 6d078ebe7a
Print more information when loading fields with different lines ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-08 20:16:06 +08:00
bigsheeper af108b5bda
Add malloc_trim after reduce search result ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-08 00:12:05 +08:00
xige-16 b5c11a216d
Alter varChar type params's name to max_length ()
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-07 15:58:06 +08:00
bigsheeper f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-06 21:34:05 +08:00
bigsheeper 9f786dd752
Remove unnecessary bitset flip in Search and Query ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-06 17:38:05 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-02 10:48:03 +08:00
Xiaofan 9579a645c6
Support compile marisa on Macos ()
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-31 13:28:02 +08:00
SimFG 731870211a
Support role-based access control (only pb) ()
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-05-28 00:04:01 +08:00
Letian Jiang f2a27e0e64
Retry GetShardLeaders until service available or timeout ()
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-26 20:28:02 +08:00