milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
16,023 Commits
148 Branches
122 Tags
578 MiB
Commit Graph

6076 Commits (19aec183723c952cfb6e0dee431ff5c9387f8679)

Author SHA1 Message Date
yah01 3c5ce74843
Fix LoadBalance not check segment exists (#19448) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-27 16:00:54 +08:00
yah01 9801c6dfc9
Fix data race in DataNode (#19470) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-27 15:16:55 +08:00
cai.zhang 2442335aa2
Fix indexnode ut bug (#19468) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-27 14:56:52 +08:00
XuanYang-cn f6e7b92db6
Fix read lock to write lock (#19464) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-27 11:50:57 +08:00
Ten Thousand Leaves 67b95597ed
Deprecated old RootCoord methods in Bulk Load (#19449) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-27 10:44:53 +08:00
xige-16 3fb8e80147
Rename serializeDiskIndex to getIndexFileInfo (#19416) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-27 10:32:53 +08:00
SimFG 09255e5507
Add the `indexName` param for the `indexpb.DescribeIndexRequest` (#19440) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-26 19:10:53 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-26 18:06:54 +08:00
XuanYang-cn 52c6a2706e
Fix binary results unstable (#19401) 
See also: #19338, #19366, 19367

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-26 18:02:52 +08:00
XuanYang-cn 7819297f68
Add more checks for search offset (#19388) 
See also: #19250

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-26 18:00:57 +08:00
Cai Yudong 87d78a4a85
Ignore cases when comparing metric type in segcore (#19437) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-26 17:58:52 +08:00
Enwei Jiao fc8061bb86
improve logs to debug growing segment load failed (#19419) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-09-26 17:36:54 +08:00
bigsheeper 804c18df68
Update quota params (#19351) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-09-26 16:48:53 +08:00
zhenshan.cao c15b880f0e
Remove redundant logs (#19412) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-09-26 15:40:53 +08:00
yah01 dff4d1042e
Fix can't release growing segment (#19418) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-26 14:08:52 +08:00
Enwei Jiao ee8896d81b
fix compile error (#19428) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-09-26 11:24:54 +08:00
yah01 6ba52366c5
Fix load segments can't be retried (#19414) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-26 10:54:52 +08:00
zhagnlu 59f8e96522
Change compactor policy from loading all binlogs to loading serially (#13716) (#19280) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-09-25 20:12:52 +08:00
jaime 91df8f2d6a
Use log id instead of log path of the binlog in metastore (#19123) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-25 15:56:51 +08:00
Bingyi Sun 29d2c0bd7b
Move task merge cap to config file (#19387) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-24 17:56:51 +08:00
cai.zhang 9e18645363
Optimize IndexCoord scheduler loop (#19375) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-24 13:40:52 +08:00
Jiquan Long 4e521438c3
Optimize read ddl (#19352) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-24 12:42:51 +08:00
SimFG 662c654fe3
Add the `context` param for the `hook` interceptor (#19405) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-24 09:56:51 +08:00
aoiasd 2b58bd5c0a
Optimize large memory usage of InsertRecord by using vector instead of unordered_map if InsertRecord used in sealed segment (#19245) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-23 17:08:51 +08:00
Jiquan Long 22477d4601
UnWatch channels if failed to create collection (#19390) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-23 16:56:50 +08:00
xige-16 d4bc00423c
Fix start milvus failed on macos (#19394) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-23 16:54:50 +08:00
yah01 6d6e14e67d
Fix manual balance failed with TaskStale (#19400) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-23 16:18:51 +08:00
yah01 2cfacbba8d
Fix leader observer confused with replicas (#19392) 
With 2 replicas, the same segment was loaded on 2 nodes in different
replicas, leader observer will sync the latest one (by version) to the
other one.

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-23 15:16:51 +08:00
Xiaofan 928a213e31
Fix Chunk Manager Path mis use (#19141) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-23 14:40:51 +08:00
XuanYang-cn 253fb514bd
Add Revert in DataCoord meta catalog (#19313) 
See also: #19072

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-23 10:50:51 +08:00
XuanYang-cn 432b5ddf2e
Add SyncSegments into proto (#19312) 
See also: #19072

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-23 10:22:52 +08:00
cai.zhang 1c489a85b2
Add ut for IndexCoord (#19349) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-23 09:58:51 +08:00
SimFG 68a257458b
Implement the `hook` interceptor (#19294) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-23 09:50:52 +08:00
Xiaofan 901d3fb367
Add more log when creating index, Finished/Failed index will not be counted in total index tasks (#19150) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-23 09:40:52 +08:00
cai.zhang 17f5e3c7fe
Drop segment indexes when drop partition (#19022) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-23 09:36:51 +08:00
yah01 ee9d634a43
Compaction requires the source segments indexed (#19368) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-22 18:34:50 +08:00
Jiquan Long cfd01a188c
Fix drop collection failed due to etcd txn limit. (#19362) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-22 17:36:52 +08:00
Bingyi Sun 5117017355
Remove manual balance timeout (#19358) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-22 16:52:51 +08:00
yah01 71d4a32ce6
Fix deadlock (#19340) 
The IsParentDropped() acquires read lock of meta,
which leads to recursively acquire read lock

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-22 15:48:50 +08:00
Bingyi Sun 72346f31b0
Add collectionID when checking collection load (#19322) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-22 15:08:50 +08:00
xige-16 1cd6e80c8a
Increase timeout of activate task (#19330) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-22 14:14:52 +08:00
congqixia c4761100a7
Fix SyncDistribution set Action (#19360) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-22 13:18:50 +08:00
congqixia 72ee35a78f
Add loading list filter logic for load segments (#19347) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-21 21:30:51 +08:00
xige-16 428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00
yah01 10c0ff1211
Fix Load returns incorrect error code (#19318) 
- Load parameters mismatched returns code 5
- Check whether nodes is enough to load

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-21 17:54:50 +08:00
cai.zhang fc3309b8b8
All indexes segments both write handoff (#19230) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-21 16:34:51 +08:00
Jiquan Long 42d371fd41
Make ddl follow tt mechanism (#19279) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-21 15:46:51 +08:00
yah01 97aa2bd84f
Merge load segment tasks to improve performance (#19234) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-21 14:42:51 +08:00
congqixia f24e16a66f
Handle multiple parquet RowGroup/Column (#19283) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-21 09:12:50 +08:00
Jiquan Long 5141e05c47
Add policy to seal long time idle segment (#19222) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-20 20:54:50 +08:00
XuanYang-cn 297d75fc92
Enable query pagination (#19231) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-20 20:48:50 +08:00
Bingyi Sun 8f84208c6b
Move growing segments to leaderview in GetDataDistributionResponse (#19292) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-20 19:24:51 +08:00
yah01 0306f71b49
Remove incorrect formatting, unused conditions (#19239) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-20 19:16:49 +08:00
wei liu 86769e598c
fix test unstable (#19214) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-09-20 17:04:50 +08:00
yah01 539585e91b
Check whether leader view exists (#19237) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-20 16:10:50 +08:00
Bingyi Sun 77aee8cb65
Fix data race in querycoordv2 (#19240) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-20 15:54:49 +08:00
congqixia 50ea4eeef1
Fix querynode data distribution return false positive dm channel (#19249) 
Co-authored-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2022-09-20 12:04:49 +08:00
bigsheeper 5143e2a75e
Disable quota and limits (#19242) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-09-20 09:34:50 +08:00
jaime 8ac30397dd
Disable kafka msgstream ut (#19260) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-19 16:24:51 +08:00
congqixia 3759f23f88
Add error log for load segment file failure (#19227) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-19 14:08:48 +08:00
Jenny Li d8b11121a8
Decrease build time by removing compiling openblas (#19207) 
Signed-off-by: Jenny Li <jing.li@zilliz.com>

Signed-off-by: Jenny Li <jing.li@zilliz.com>
 2022-09-16 17:46:47 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
yah01 11b352c3ae
Guarantee flushed segments are all indexed (#19062) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-16 11:32:48 +08:00
Bingyi Sun 3998752db3
Fix log print (#19204) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-16 11:12:47 +08:00
aoiasd 58d3a49b62
Optimize large memory usage of Variant used in InsertRecord (#19197) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-16 10:52:47 +08:00
bigsheeper 5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector (#18636) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-09-16 09:56:47 +08:00
XuanYang-cn ac1982c8a8
Add merger for retrieve results (#18948) 
See also: #18893

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-15 19:12:32 +08:00
Bingyi Sun 626854cf0c
Refactor QueryCoord (#18836) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-15 18:48:32 +08:00
Bingyi Sun 92857a4888
Remove useless ut. (#19210) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-09-15 16:32:33 +08:00
cai.zhang 525201decc
Mark index meta as deleted when segment is compacted (#19163) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-15 15:44:31 +08:00
XuanYang-cn 200df76864
Enable search with pagination (#19044) 
See also: #19043

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-14 20:36:32 +08:00
XuanYang-cn 086eb92ade
Enable sort for segcore RetrieveResults (#18894) 
See also: #18893

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-14 11:04:30 +08:00
SimFG 65fffa6d7d
Make sure that closing the graph is more secure (#19178) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-14 10:32:33 +08:00
congqixia 91663a6603
Fix bloom filter size not match (#19140) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-13 14:12:31 +08:00
groot b161aec95e
Support input empty string (#19111) (#19144) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-09-13 13:36:29 +08:00
Xiaofan ec48e2405a
Fix Cgo lock os thread failed (#19148) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-13 10:16:30 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00
congqixia ce9ba0dd02
Remove debug log in minio chunk manager (#19134) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-09 17:32:37 +08:00
cai.zhang e65ba6331e
Only store segmentID for flushed segments event (#19075) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-09 15:52:35 +08:00
Xiaofan eabdc2b114
Disable pulsar batch and change background flush goroutine to larger interval (#18888) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-09 11:26:35 +08:00
SimFG ceea04c274
Improve the `close` method in the graph (#19100) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-09 10:00:36 +08:00
wayblink 3927ae9952
Return all flushed segments and time-of-seal in Flush interface (#18882) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-09-09 09:58:37 +08:00
congqixia db23a256c2
Make minioChunkManager ListObject level by level when recursive is true (#19096) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-08 14:58:34 +08:00
groot ebc263f440
Not allow '$' in collection/partition/field name (#19082) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-09-08 10:16:34 +08:00
jaime 362c663b22
Remove deprecated inner compaction type (#19024) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-07 19:16:34 +08:00
Enwei Jiao 2d733bcdc1
fix goroutine leak in querycoord (#19067) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-09-07 18:56:34 +08:00
congqixia 39b847b67a
Add triggerQueue check for handoff verifyReq (#19064) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-07 16:14:35 +08:00
cai.zhang d6b0fdc944
Fill index size and fail reason when query index job (#19026) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-06 17:19:11 +08:00
Cai Yudong 7b5eb85205
Remove redundant parameter for segcore Search() (#19025) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 12:05:11 +08:00
Cai Yudong 765907ab77
Optimize segcore Reduce (#18902) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 10:55:12 +08:00
SimFG 2bafdf8c53
Fix the `Flush` privilege can't be granted (#19020) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-05 21:05:11 +08:00
cai.zhang 11b4f4e901
Fix typo for index code (#18971) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-05 17:37:11 +08:00
Jiquan Long a5e2d6b6fb
Refactor RootCoord (#18930) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
 2022-09-05 13:29:11 +08:00
Cai Yudong 686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935) 
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-05 10:41:11 +08:00
congqixia 68a6587374
Set insert&stats binlog timestamp range (#19005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-04 09:05:09 +08:00
SimFG 9cd19f5396
Return nil error when handling grpc request (#18955) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-02 21:12:59 +08:00
bigsheeper 8a45381e53
Make DescribeCollection return collectionName (#19000) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-09-02 19:20:59 +08:00
jaime 816967c000
Add metrics for etcd (#18962) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-02 17:46:59 +08:00
congqixia ff5201cbe5
Improve log of datanode compactor (#18981) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-02 14:31:06 +08:00
congqixia 6ea0fb82ab
Add InvalidNodeID logic to remove segments from offline segments (#18975) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-02 10:42:59 +08:00
congqixia 29c72cad15
Use segment statslog instead of separate statslog (#18775) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-02 10:34:59 +08:00
Xiaofan 10d1140885
Fix RocksMQ flaky test (#18954) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-01 19:46:59 +08:00
XuanYang-cn c054873908
Add limit to proto (#18950) 
See also: #18893

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-01 18:54:58 +08:00
cai.zhang 6d37b1db17
Fix bug for umarshal index model (#18959) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-09-01 17:18:57 +08:00
Cai Yudong da96659569
[skip e2e] Add more testcases with different parameter combinations in test_reduce (#18967) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 16:59:02 +08:00
Cai Yudong 4ded45394a
[skip e2e] Optimize test_reduce (#18957) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 11:09:00 +08:00
jaime 0e4e7966ec
Refine grpc server options of interceptor (#18943) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-01 09:48:58 +08:00
jaime 237d912625
Add log for the large value during etcd saving (#18229) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-01 09:47:02 +08:00
wei liu d08581d9b5
fix remove offline nodes in watchDmChannel (#18921) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-08-31 19:56:57 +08:00
congqixia 43c14055f6
Fix test compact logic and data race (#18932) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-31 12:10:57 +08:00
yah01 03a59ccbd4
Fix IndexCoord forget to fill DmChannel (#18914) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-08-31 11:28:58 +08:00
cai.zhang 34c9eae16f
Add IndexCoord ut and fix data race (#18897) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-08-31 10:23:04 +08:00
SimFG bf2bd80986
Avoid the nil point (#18925) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-31 10:20:57 +08:00
bigsheeper 5eb6127fc5
Set max read concurrency ratio to improve concurrent read performance (#18911) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-08-30 19:50:57 +08:00
XuanYang-cn 867ea63bdd
Move TopK check inside parseQueryInfo (#18892) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-30 10:32:56 +08:00
wei liu 1527aee019
fix watch dml channel remove offline node (#18835) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-08-29 16:32:57 +08:00
Xiaofan 56840da74f
Fix Segment Loader Log (#18883) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-08-29 15:30:57 +08:00
congqixia cea546ba09
Fix DefaultFactory using wrong rootPath config (#18876) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-29 12:36:55 +08:00
Cai Yudong 9dc3bbecbd
Change the type of slice_nqs and slice_topks from int32_t[] to int64_t[] (#18867) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-29 11:36:56 +08:00
Xiaofan 8e22d03cf3
Support trigger compaction everyday to eliminiate deleted data in rocksmq (#18752) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-08-29 10:56:55 +08:00
SimFG 69c0b2fb49
Improve the storage method of the grant info in the metastore (#18817) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-26 19:22:56 +08:00
yah01 d76d6f42cd
Add unit test for case of stream closed (#18848) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-08-26 19:20:55 +08:00
Enwei Jiao 096149e33f
fix GetCompactionState hang (#18829) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-26 14:22:55 +08:00
Letian Jiang ef7742507d
Remove redundant code (#18832) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-08-25 19:48:54 +08:00
yah01 63e08606e2
Fix load segment hangs forever (#18814) 
Caused if the context is timeout

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-08-25 19:34:54 +08:00
congqixia c91bb0b015
Unify rootPath in configs and ChunkManager (#18808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-25 19:32:53 +08:00
Enwei Jiao 232f48784e
fix restart querycoord blocking issue (#18801) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-25 18:42:54 +08:00
Cai Yudong 305601ad25
Add API SearchOnSealedChunk() to search on sealed segment without index (#18830) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-25 18:16:55 +08:00
jaime ef9098f84a
Refine save dropped segments into kv metastore (#18779) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-25 16:48:53 +08:00
Cai Yudong dcf45df029
Optimize API vector_search parameter in segcore (#18827) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-25 16:16:54 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643) 
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-08-25 15:48:54 +08:00
wgcn d443b5420a
add user special kafka config (#18742) 
Signed-off-by: wgcn <1026688210@qq.com>

Signed-off-by: wgcn <1026688210@qq.com>
Co-authored-by: wanggang11335 <wanggang11335@autohome.com.cn>
 2022-08-25 11:02:53 +08:00
Ji Bin f3c755602b
Adaptation for latest MinGW64 (#18699) 
Signed-off-by: Ji Bin <matrixji@live.com>

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-08-25 10:56:53 +08:00
Xiaofan 8fe3895485
change retention time to 1 day (#18749) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-08-25 10:42:53 +08:00
codeman 90de312d4b
Separate create & update user in mysql (#18785) (#18786) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-24 14:32:56 +08:00
SimFG 226bcc16eb
Support concurrent situations for the metastore (#18792) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-24 10:02:52 +08:00
wei liu 9338ad45c5
fix insert meets vChannels reuse pChannel (#18695) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-08-23 19:12:55 +08:00
wayblink 6fd83464cd
Fix watchDmChannel may be out-of-date after compaction issue and add context (#18681) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-08-23 17:44:56 +08:00
congqixia d5bb377bc7
Add semver in session (#18768) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-23 17:14:52 +08:00
jaime 909e46b6c2
Fix LessOrEqualThan method of rocksmq (#18789) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-23 16:56:52 +08:00
wei liu 5c4f854271
fix load segment hangs (#18770) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-08-23 16:52:52 +08:00
Enwei Jiao d9c7519107
refactor compaction concurency logic (#18660) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-23 15:50:52 +08:00
aoiasd dad060af2e
Fix func checkIfLoaded in proxy (#18769) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-08-23 15:14:52 +08:00
wgcn cd2622390c
remove redundant slashes (#18750) 
Signed-off-by: wanggang11335 <wanggang11335@autohome.com.cn>

Signed-off-by: wanggang11335 <wanggang11335@autohome.com.cn>
Co-authored-by: wanggang11335 <wanggang11335@autohome.com.cn>
 2022-08-23 13:24:53 +08:00
wgcn 16ccefc678
release manager Rlock (#18773) 
Signed-off-by: wanggang11335 <wanggang11335@autohome.com.cn>

Signed-off-by: wanggang11335 <wanggang11335@autohome.com.cn>
Co-authored-by: wanggang11335 <wanggang11335@autohome.com.cn>
 2022-08-23 10:56:52 +08:00
Zach 0c9a10e8f8
1. refine logging interfaces (#18692) 
2. adjust logs for query/search requests

Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>

Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-08-23 10:44:52 +08:00
SimFG ce434b496e
Implement the mysql metastore of the rbac (#18704) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-23 10:26:53 +08:00
Cai Yudong 94ffa8a275
Upgrade to knowhere-v1.2.0 (#18746) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-23 10:20:58 +08:00
Jiquan Long 641afae22f
Watch channels with start positions (#18744) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-08-22 14:42:52 +08:00
codeman 70254c3ec5
Unified catalog interface for segment (#18289) (#18290) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-20 10:24:51 +08:00
Ten Thousand Leaves fc42ee5bb3
Fix # of rows of recovering segment (#18736) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-08-19 19:50:50 +08:00
jaime 516fd928f9
Fix password verification miss cache (#18730) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-19 19:42:50 +08:00
Jiquan Long cf730f6fcd
Fix query cluster metrics (#18725) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-08-19 11:44:50 +08:00
XuanYang-cn e05e9cb9b4
Fix typo in error message (#18688) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-19 10:48:50 +08:00
XuanYang-cn a782ded0bd
Fix RHNSWPQ pqm divide by zero (#18700) 
See also: #18671

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-18 19:24:51 +08:00
congqixia b62cb82ebe
Simplify Dml-DeltaChannel mapping logic (#18702) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-18 14:24:50 +08:00
Letian Jiang c6d9d0df5e
Fix segcore benchmark (#18675) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-08-17 20:24:49 +08:00
XuanYang-cn 16dcd9718b
Fix flush panic after compaction (#18677) 
See also: #18565

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-17 17:08:49 +08:00
congqixia 0db281c4ff
Fix DataType::bool cast to double (#18690) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-17 16:14:49 +08:00
congqixia d4c54d96b0
Dedup output fields for task query (#18673) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-16 18:16:48 +08:00
aoiasd be933a75c0
Add log for NoReplicaAvailable Error in QueryCoord.GetShardLeaders (#18662) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-08-16 10:56:48 +08:00
Cai Yudong d878cc2ec0
Rename real_topK_per_nq_ to topks_per_nq_prefix_sum_ in SearchResult struct (#18634) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-16 00:32:48 +08:00
SimFG 447ce8ee48
Handle some apis about the default role (#18647) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-15 16:40:48 +08:00
XuanYang-cn 4c86cc63ba
Fix DN flowgraph delete buffer logs (#18657) 
1. Remove logs about not existing segments.
2. Group logs by timestamp.
3. Log changed segments only.
4. Pair the segments reference lock and unlock log by taskID.

Resolves: #18655

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-15 16:28:48 +08:00
aoiasd 8a70f2badb
Add ShowConfigurations for all remaining session (#18594) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-08-12 13:20:39 +08:00
codeman 174130c7d6
fix wrong operation model for user (#18614) (#18615) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-11 20:22:38 +08:00
xige-16 e40061b864
Update binlog event format (#18347) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-11 14:06:38 +08:00
codeman 6c3dbf0a07
Support meta storing on database (#17236) (#18541) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-11 12:12:38 +08:00
xige-16 111c608513
Add script to auto clang format (#18559) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-11 11:04:37 +08:00
xige-16 dbb96a8b70
Fix pk2offset occupy too much memory (#18325) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-11 10:48:38 +08:00
jaime 0e95f8a690
Refine indexcoord logs (#18589) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-11 10:00:38 +08:00
xige-16 cd04258122
Abort execute timeout search/query (#17729) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-11 09:58:41 +08:00
xige-16 29384ddbc0
Fix querycoord handle offline node with wrong nodeID (#18590) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-10 21:10:37 +08:00
SimFG 1366f130a2
Add a specific `KeyNotExistError` err for the metastore `load` interface (#18582) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-10 17:20:41 +08:00
jaime efa5dfaa7b
Fix Parse binlog path failure (#18583) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-10 17:04:37 +08:00
Jiquan Long e7b3bacbec
Refine catalog of partition & alias. (#18546) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-08-10 10:22:38 +08:00
Ten Thousand Leaves 097f144db1
Improve slow Etcd logs by specifying keys (#18547) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-08-09 21:14:37 +08:00
Ten Thousand Leaves 12023e8569
Fix a potential nullptr issue with metric (#18540) 
/kind improvement

related to: #18530

but not the root cause

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-08-09 21:12:42 +08:00
congqixia 179b496824
Add cgo worker pool for querynode (#18461) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-09 16:34:37 +08:00
bigsheeper 002f509808
Remove deprecated cacheSize param (#18566) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-08-09 15:54:36 +08:00
SimFG c267ad612d
Support Role-Based Access Control II (#18528) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-05 16:28:35 +08:00
Xiaofan b427450593
Remove Load Delta Channel Request (#18497) 
Fix #18393

remove watch delta channel task related logic

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-04 17:50:37 +08:00
jaime c345df41a7
Fix kafka ut (#18527) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-04 16:02:35 +08:00
SimFG ff0200210a
Support Role-Based Access Control (#18425) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-04 11:04:34 +08:00
jaime 4a32c1bb8e
Enable kafka ut with a mocked kafka server (#18366) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-03 19:06:35 +08:00
congqixia 28aadc988d
Use vchannel name in datanode subName (#18519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-03 17:10:36 +08:00
Cai Yudong 7cd37fc6dd
Merge FloatSearch() and BinarySearch() into SearchOnGrowing() (#18498) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-03 11:56:36 +08:00
aoiasd a7ce0eb6f6
Add ShowConfigurations to QueryNode (#18426) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-08-03 01:12:33 +08:00
jaime 6e7831470c
Revert settings of Pulsar address and refine config code (#18494) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-02 21:26:33 +08:00
XuanYang-cn 43c7c1ff03
Fix indexcheck division by zero bug (#18482) 
See also: #18479

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-02 16:04:34 +08:00
congqixia ab461c6e5e
Release collection resources when all partition released (#18488) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-02 11:40:34 +08:00
congqixia 318cdbebe9
Add heap for dmlChannels to get next channel (#18451) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-01 16:36:33 +08:00
congqixia f0fe8dae0b
Close Node/Segment detector when close ShardCluster (#18476) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-01 13:50:33 +08:00
Enwei Jiao 283f5731d2
config from etcd (#18421) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-01 10:04:33 +08:00
jaime c9174d55ba
Refine merge operation during compacting phase (#18399) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-28 14:52:31 +08:00
zhenshan.cao a287a2b3fd
Return empty result in advance if all data filtered out (#18329) (#18438) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-28 12:36:30 +08:00
wei liu a3ec44cd37
support dynamic change log level through http (#18430) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-07-28 11:36:30 +08:00
wayblink c198e6c2d9
Add some debug log to explore ocassionally channel name mismatch issue in create collection (#18414) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-07-27 13:06:30 +08:00
congqixia e9d777b336
Fix ParseSegmentIDBinlog panicks with bad input (#18413) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-26 19:32:30 +08:00
yah01 9621b6615d
Limit the concurrency level for single load request (#18400) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-07-26 15:18:30 +08:00
congqixia 128c66f301
Fix segment DmChannel in querycoord for compatibility to 2.0.2 (#18397) 
Reload segment dm channel if empty channel found

Patch originally authored by @czs007
Unit test added by @congqixia

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-25 14:14:31 +08:00
congqixia 45746bb736
Add ShowConfiguration proto definition (#18364) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-23 17:14:28 +08:00
xing.zhao d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-07-22 22:10:28 +08:00
wgcn f192c6c298
fix the search metric of querynode (#18367) 
Signed-off-by: wanggang11335 <wanggang11335@autohome.com.cn>

fix the search metric of querynode

Co-authored-by: wanggang11335 <wanggang11335@autohome.com.cn>
 2022-07-22 17:38:29 +08:00
jaime 3ec147acfe
Upgrade confluent kafka go version to 1.9.1 (#18293) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-22 13:28:28 +08:00
jaime 9672eae62c
Unify metastore catalog interface (#17772) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-22 10:20:29 +08:00
congqixia f37e07b0f8
Use Get method of request in GetMetrics to avoid panicking (#18356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-21 20:52:27 +08:00
SimFG 74fd28df9c
Refine some datanode code (#18351) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-07-21 15:52:27 +08:00
Xiaofan a6e1ddb53d
Fix node is not assigned to replica (#18350) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-21 12:02:27 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc (#18349) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-20 22:22:31 +08:00
Ten Thousand Leaves cd2f64d30c
Fix a nil pointer issue (#18343) 
/kind bug

issue: #18270
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-20 17:56:30 +08:00
congqixia d90308e9a0
Handle topic not exist in rqm & mqtt seek (#18314) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-20 15:42:29 +08:00
Ten Thousand Leaves 0ebe407221
Make DataNode not panic if stale segment is not found during flush (#18322) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-20 15:06:30 +08:00
wei liu a79c5e2366
fix sync problem during reblance (#18332) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-07-20 15:02:30 +08:00
congqixia be6ca6358d
Use serialized size before compression in log meta (#18336) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-19 21:30:30 +08:00
bigsheeper c038cd7a48
Fix addSegment error, avoid the risk of mem leaks (#18309) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-19 20:58:28 +08:00
Jeng.Gwan db71c74cc7
Fix leak of memory in ParseMallocInfo while assert false (#18311) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-19 14:26:33 +08:00
cai.zhang 315df1ff15
Reduce the timer time of assigning index task scheduler (#18319) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-19 14:24:29 +08:00
Jiquan Long 70246f82fd
Reduce chunk row size (#18320) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-19 13:50:28 +08:00
congqixia f7bd054adb
Set EnableGarbageCollection to true by default (#18305) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-18 16:34:28 +08:00
Xiaofan c97563590c
fix missing handling channels while a node down (#18250) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-18 13:06:28 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment (#18115) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-18 09:58:28 +08:00
congqixia f0846fb79b
Handles DropPartitionMsg in datanode flowgraph (#18292) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-15 17:12:27 +08:00
Xiaofan 6d82ef8c20
fix garbage collector err handling (#18277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-15 13:54:26 +08:00
bigsheeper d278213e4f
Check load request both in impl and task (#18276) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-15 10:24:28 +08:00
bigsheeper 73aef14820
Improve check of purge memory and change purge place (#18271) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-15 10:22:27 +08:00
yah01 bb2ad23e55
GetQuerySegmentInfo returns sealed loaded segments (#18255) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-07-14 16:44:27 +08:00
cai.zhang 93d9dfad1a
Ignore task when IndexNode load empty index meta (#18266) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-14 15:08:28 +08:00
cai.zhang a47a414dce
Fix bug for IndexNode pod kill test (#18177) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-13 16:16:25 +08:00
xige-16 54d17bc5da
Fix query too slow when insert multi repeated pk data (#18231) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-07-13 10:22:26 +08:00
congqixia e5fe46124b
Fix isCompacting flag lost after segment clone (#18248) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-13 10:08:28 +08:00
Zach 84d73bd618
add package name in log contents (#18251) 
Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-07-12 23:02:26 +08:00
Jeng.Gwan a926a9eb66
Fix redundant string copy of PlaceholderGroup (#18236) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-12 15:48:25 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API (#18227) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-07-12 11:58:25 +08:00
bigsheeper 1da970af18
Remove useless statsService in QueryNode (#17395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-11 15:12:25 +08:00
congqixia a9aff3a678
Allocate ID for each insertMsg (#18217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-11 14:38:24 +08:00
yah01 08612ac56c
Fix QueryCoord unit test stucks (#18197) 
Signed-off-by: yah <yang.cen@zilliz.com>
 2022-07-11 12:44:25 +08:00
Xiaofan 653dce4948
Fix pulsar error check (#18207) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-11 10:56:25 +08:00
congqixia e61f61a9bd
Dedup insert msgs by MsgID in mqtt MsgPack (#18180) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-08 20:42:22 +08:00
bigsheeper 22508f36d3
Get search field id from search plan and log if loaded index when search segments (#18183) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-08 20:18:22 +08:00
cai.zhang 90faf4a212
Fix bug for IndexCoord recycles index files (#18168) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-08 16:54:22 +08:00
bigsheeper 0fb5e9c1a9
Remove load collection check in Proxy (#18166) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-08 16:12:25 +08:00
Ten Thousand Leaves ecc250c20e
Improve logs to better cover segment flush lifecycle (#18157) 
/kind improvements

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-08 14:50:21 +08:00
zhenshan.cao 72910d65e0
Fix inconsistency in DataCoord (#18170) (#18171) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-08 13:48:21 +08:00
XuanYang-cn 53734bef58
Add seek msgID in DataNode (#17881) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-07-08 10:18:28 +08:00
wayblink 43ae6791e6
Make VChannelInfo read compatible with 2.0.2 (#17995) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-07-07 21:58:22 +08:00
codeman 5fda8261d0
Fix: Unable to list newly created credential users (#18117) (#18153) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-07-07 21:16:22 +08:00
cai.zhang 0c883b215b
Fix bug for IndexCoord ut (#18148) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 18:52:21 +08:00
cai.zhang 44e59d6318
Fix bug for clean etcd in QueryNCoord (#18145) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 18:32:21 +08:00
XuanYang-cn d9e8231728
Filter sealed segments (#18065) 
- Remove FilterThreshold in DataNode
- Alter filter logic in flowgraph DDNode

See also: #17943

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-07-07 18:26:20 +08:00
cai.zhang 58a9a0b79a
Add license for new files (#18138) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 17:44:21 +08:00
cai.zhang c1a47c8941
Use struct channel instead of bool chan (#18140) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 16:20:22 +08:00
congqixia 543197c2be
Fix handoff segment unit test uses wrong node id (#18137) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-07 15:22:21 +08:00
cai.zhang 0ad15a22c8
IndexCoord handle events correctly (#17878) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 14:44:21 +08:00
congqixia 8f1ba6a0bb
Concurrent load segment stats log when initialize dataSyncService (#18125) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-07 14:32:21 +08:00
Jiquan Long 6df47c7faf
Change pulsar compression type to zstd-faster (#18121) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-07 11:16:22 +08:00
Enwei Jiao ca88e37c98
fix receive messages before waitToFinish (#18101) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-07-07 10:00:21 +08:00
Ten Thousand Leaves d4a1e94f32
Fix flush failure caused by empty stale segments (#18097) 
/bug

issue: #18096
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-06 18:10:21 +08:00
SimFG 79523ff8d4
Fix the wrong log tag in the test case (#18103) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-07-06 15:38:20 +08:00
Zach 1c9647ff31
Merging multiple shard query/search requests into one rpc request if QueryNode is the leader of multiple shards of Collection (#17588) 
Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-07-06 15:06:21 +08:00
Xiaofan a9b1d71a8f
change the etcd compareAndSwap interface (#18068) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-06 13:54:21 +08:00
Jiquan Long 44024c1b1d
Replace pulsar compression type zstd with lz4 (#18085) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-05 22:06:21 +08:00
bigsheeper 696d747afa
Check if loaded in Proxy when LoadCollection (#18074) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-05 21:16:20 +08:00
zhagnlu 257da153ce
Fix core dump when nq has no topk result (#17923) (#18051) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-07-05 19:48:20 +08:00
Cai Yudong 5edeeb5fcd
Define new SubSearchResult constructor to avoid memory copy for growing (#18000) 
segment search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-07-05 17:14:20 +08:00
bigsheeper 7d6624fcad
Improve flow graph node name and log msgID in QueryCoord (#18043) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-05 16:46:23 +08:00
zhenshan.cao 749558d55b
Fix bug: get correct number of cpu (#17973) (#18030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-05 16:40:20 +08:00
congqixia 63230f0280
Fix data race in ShardCluster.SyncReplicas (#18054) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-05 13:16:20 +08:00
congqixia d652fa567a
Return NotShardLeader error code when dml channel is not watched (#18055) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-05 13:08:19 +08:00
Xiaofan b31f937301
Fix unsubscribe with subscription not exist (#18042) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-05 10:12:20 +08:00
Bingyi Sun 1fd3ded815
Add querycoord v2 dir and proto (#18032) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-07-04 16:56:20 +08:00
xige-16 0702993106
Fix query not empty after delete string (#17953) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-07-04 16:54:20 +08:00
yah01 39db27865f
Refine log, remove huge logs (#18036) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-07-04 16:34:20 +08:00
zhenshan.cao 1647bebaf4
Count growing segment number to estimate cpu usage (#17820) (#18027) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-04 15:10:20 +08:00
zhenshan.cao 9bd35dd84e
Refine logs to reduce print (#17759) (#18028) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-04 11:10:20 +08:00
Jiquan Long 6954a5ba3e
Fix search successfully with invalid metric type (#17977) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-01 22:28:23 +08:00
yah01 4def7047b4
Fix the scheduler enqueues concurrent load tasks (#17950) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

related to #17850
 2022-07-01 19:54:19 +08:00
congqixia 566f5b5766
Make querynode handle segmentChangeInfo with multiple vchannel (#18009) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-01 19:20:20 +08:00
yah01 66a04450ad
Fix LoadBalance Sync all segments to shard leaders (#18002) 
Signed-off-by: yah <yang.cen@zilliz.com>
 2022-07-01 19:18:24 +08:00
yah01 27eca2881a
Clean segments as releasing collection (#17932) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-07-01 19:16:25 +08:00
zhenshan.cao f33b090819
Fix bug: datarace in proxy unittest (#17990) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-01 17:56:21 +08:00
Jiquan Long 47c9e77155
Fix querycoord ut data race (#17980) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-01 10:58:18 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully (#17959) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-01 10:44:18 +08:00
codeman 6ab850bedf
Fix update credential failure (#17963) (#17970) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-30 21:34:18 +08:00
congqixia f4177f4434
Remove growing according to offline as well (#17968) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-30 20:14:19 +08:00
cai.zhang 01fc411566
Handle etcd compacted error (#17886) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-30 18:54:19 +08:00
bigsheeper cdbd75d4dd
Log delta channel timetick in DataNode (#17947) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-30 17:44:18 +08:00
zhenshan.cao 0baeb609dc
Fix bugs in monitoring metrics (#17894) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-30 17:26:19 +08:00
Xiaofan a803f9e4b0
Fix Wait Group Race in Msgstream (#17931) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-30 10:38:18 +08:00
yah01 ef6859a4d6
Refactor handoff handler (#17806) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-29 23:26:19 +08:00
codeman 95a3830dc6
fix password comparison too slow (#17874) (#17897) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-29 20:02:18 +08:00
XuanYang-cn ff3c4ab2a8
Fix compaction panic (#17840) 
See also: #17823

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-29 16:54:17 +08:00
Bingyi Sun 24b3ad22d7
Fix msgstream close block. (#17880) 
issue: #17513
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-29 16:24:20 +08:00
congqixia c179986e6c
Fix shard cluster segments data race (#17903) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-29 16:20:19 +08:00
congqixia 106f19a7b8
Fix querycoord cannot update deltaChannel info (#17901) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-29 15:38:17 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-29 14:20:19 +08:00
jaime 762e3b78a6
support configurable sasl mechanisms for kafka (#17542) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-29 10:32:17 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao ea901393ef
fix windows ldflags (#17807) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-28 19:58:25 +08:00
xing.zhao 3904a78211
Remove genReqID function in import manager bacause the request ID is useless (#17853) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-06-28 18:20:18 +08:00
bigsheeper b657e58370
Remove temporary variable to prevent memory fragmentation (#17728) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-28 16:30:23 +08:00
cai.zhang 899a9bf86d
IndexNode still writes finished state to ETCD when task was deleted (#17761) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-28 12:08:17 +08:00
cai.zhang b883d1f5dd
Make sure indexes are successfully dropped when DropCollection (#17780) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-28 11:24:21 +08:00
bigsheeper 6b79d4f7bc
Log segment type as possible in QueryNode (#17846) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-28 10:16:17 +08:00
XuanYang-cn 6b72f9013f
Fix message type of show partitions (#17821) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-28 10:06:17 +08:00
yah01 797218a8ad
Fix handoff isn't processed correctly (#17738) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-27 22:08:17 +08:00
cai.zhang ef0fc37ad7
Fixed the bug that IndexCoord lost some meta information (#17818) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-27 21:52:17 +08:00
wayblink 46e0e2658b
Move datacoord related methods from meta to globalMetaBroker (#17812) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-27 21:04:17 +08:00
congqixia 6840486efc
Fix load progress complete before syncReplica (#17828) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-27 20:52:16 +08:00
Ten Thousand Leaves f0b036a35a
Move bulk load segment lock happen early (#17612) 
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-27 13:56:17 +08:00
congqixia 68d385137a
Fix: Miss channel info when load growing segment (#17789) (#17797) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: zhagnlu <1542303831@qq.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-06-27 12:02:16 +08:00
Bingyi Sun 9d2e5aacbc
Fix missing dml nodeIDs when recovering from old version (#17800) 
issue: #17416
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-26 18:08:16 +08:00
congqixia 5471e35ccc
Wait last version search to release growing safely (#17713) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-25 18:48:15 +08:00
yah01 8388478ef3
SyncReplicaSegments syncs all segments (#17774) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-24 23:24:15 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
bigsheeper 230404499d
Check duplication before adding channel info in DataCoord (#17742) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-24 16:30:15 +08:00
shaoyue 96b97f7f58
RESTful api use same port as metrics & healthcheck (#17732) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-24 15:34:14 +08:00
Bingyi Sun 05273309e8
Drop empty sealed segment (#17708) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-24 15:16:14 +08:00
xige-16 3a0f9d41df
Print log after segment search done (#17769) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-24 14:10:14 +08:00
yah01 94a5122044
Fix LoadBalance removes source nodes from DmChannel (#17744) 
Only NodeDown LoadBalance should remove source nodes

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-24 12:40:14 +08:00
wayblink 1a336bbbf1
Fix: WatchDmChannelsRequest can be too large to save in etcd (#17722) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-24 10:54:15 +08:00
SimFG 6189a117a4
Remove the `WithOptions` for avoiding the log object being cloned continuously (#17746) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-24 10:36:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-24 10:34:18 +08:00
cai.zhang e8f53af749
When the task is canceled, reduce the load of node (#17735) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-23 19:22:15 +08:00
Jiquan Long 90ca61d948
Fix double-freed global variable in easylogging lib (#17730) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-23 19:12:14 +08:00
Cai Yudong e78269f450
Optimize search related interface in segcore (#17568) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-23 18:16:13 +08:00
XuanYang-cn 21b52bfcf2
Enhance channel manager ut (#17682) 
Resolves: #17676

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-23 18:08:14 +08:00
codeman 294f7e4408
fix: log datapb.SegmentInfo is too large #17685 (#17727) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-23 14:36:13 +08:00
xige-16 8ff1a2c4c2
Print expr when proxy receive qyery req (#17662) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-23 13:42:13 +08:00
Jiquan Long 216e2f80aa
Add msg id to log in query path (#17677) 
/kind improvement

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-23 10:46:13 +08:00
cai.zhang 1fbdafc943
IndexCoord does not assign task to IndexNode prematurely (#17717) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-23 10:40:13 +08:00
SimFG 50d5d04552
Refine some codes about the datanode (#17720) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-23 09:58:14 +08:00
SimFG f4c6a6734e
Fix rpc retry policy not taking effect (#17673) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-22 22:22:13 +08:00
SimFG 3fadf4c5d7
Return the unavailable error to make the client's retry policy take effect (#17719) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-22 21:56:13 +08:00
codeman 6cb00aa5bb
fix: log datapb.SegmentInfo is too large #17685 (#17693) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-22 21:46:13 +08:00
yah01 b4f21259ef
Fix SegmentChangeInfo is not set properly (#17701) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-22 21:08:13 +08:00
Xiaofan 6fdf88f452
Fix compaction selection policy (#17690) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-22 19:00:14 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore (#17700) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow (#17706) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-22 16:56:13 +08:00
bigsheeper 8cf54137cf
Add nil check in reduce and add collection rwmutex when release segments (#17698) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 16:12:14 +08:00
yah01 a077bad84a
Make updating replica atomic, balance plan idempotent (#17686) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-22 13:40:13 +08:00
Xiaofan 7d51b65260
Estimate varchar length more reasonablly (#17688) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-22 13:02:13 +08:00
XuanYang-cn 12158432bb
Clear segment cache when closing flowgraph (#17671) 
See also: #17537

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-22 10:16:14 +08:00
Enwei Jiao 746aeea35b
fix typo (#17668) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-21 18:54:13 +08:00
cai.zhang 4efdd0929c
Get index status based on segment generation time (#17659) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-21 18:44:12 +08:00
XuanYang-cn e43f487f02
Make LoadCollection compatible (#17669) 
See also: #17656

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-21 17:26:15 +08:00
yah01 1e87b54542
Add the offlines nodes previously crashed to offline nodes channel (#17663) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-21 16:08:13 +08:00
Jiquan Long 0282f69eb4
Add msg id to log in search path. (#17654) 
issue: #17513
Add msg id to log in search path,
which should help us to trace a detailed request more easier.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-21 13:30:12 +08:00
congqixia 37464a281d
Fix datasorter swap vector logic (#17661) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-21 11:38:12 +08:00
SimFG eb1f0bc805
Add the consistency check tool (#17136) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-21 10:28:12 +08:00