4136009a9a
Support load segments/channels in parallel ( #20036 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 13:55:33 +08:00
1f170ce1ba
Refine kafka consumer ( #19846 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 13:23:30 +08:00
581e1d8c35
Fill MsgBase ALL ( #19912 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
a3f84f2ea3
Remove dependency for dropped segment checkpoint in querynode ( #19831 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-17 18:47:25 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
51eb12a034
Fix QueryNode panics ( #19541 )
...
Applying delete messages during the segment was compacted,
the QueryNode would panic.
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-29 18:02:54 +08:00
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
9dcee37e1c
Skip canceled tasks ( #19513 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-28 19:58:54 +08:00
fc8061bb86
improve logs to debug growing segment load failed ( #19419 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-09-26 17:36:54 +08:00
72ee35a78f
Add loading list filter logic for load segments ( #19347 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-21 21:30:51 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
626854cf0c
Refactor QueryCoord ( #18836 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-15 18:48:32 +08:00
d08581d9b5
fix remove offline nodes in watchDmChannel ( #18921 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-08-31 19:56:57 +08:00
bf2bd80986
Avoid the nil point ( #18925 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-31 10:20:57 +08:00
6fd83464cd
Fix watchDmChannel may be out-of-date after compaction issue and add context ( #18681 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-08-23 17:44:56 +08:00
b427450593
Remove Load Delta Channel Request ( #18497 )
...
Fix #18393
remove watch delta channel task related logic
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-04 17:50:37 +08:00
ab461c6e5e
Release collection resources when all partition released ( #18488 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-02 11:40:34 +08:00
68d385137a
Fix: Miss channel info when load growing segment ( #17789 ) ( #17797 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: zhagnlu <1542303831@qq.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-06-27 12:02:16 +08:00
294f7e4408
fix: log datapb.SegmentInfo is too large #17685 ( #17727 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-06-23 14:36:13 +08:00
6cb00aa5bb
fix: log datapb.SegmentInfo is too large #17685 ( #17693 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-06-22 21:46:13 +08:00
aa4b42c74b
Remove wait before release collection/partition ( #17623 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-06-17 19:00:11 +08:00
eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC ( #17441 )
...
Resolves : #17233 #16047
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-16 12:00:10 +08:00
4e83f37899
Improve load performance ( #17273 )
...
Load segment's fields concurrently
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-06-02 17:56:03 +08:00
8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel ( #17308 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-02 16:56:04 +08:00
f55fe6be3c
Merge historical replica and streaming replica in QueryNode ( #17217 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-31 13:42:03 +08:00
c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster ( #16856 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-30 19:50:04 +08:00
08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks ( #17022 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-23 16:41:58 +08:00
70825a35cf
Make querynode LoadSegments interface idempotent ( #17109 )
...
This PR makes the following changes:
* separate LoadSegmentsTask into two phases: PreExecute and Execute
* filters out segments that are already loaded in PreExecute phase
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-20 09:15:57 +08:00
b95681770b
Add deferred clean up for ShardCluster ( #17094 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-18 21:51:56 +08:00
94f0951f0d
Fix query lock logic in query shard ( #17034 )
...
Previously query shard locks the querylock in collectionReplica before any search/query
The lock range is too large and easy to cause dead lock
This PR makes following changes:
- Rename collectionReplica to metaReplica which is more reasonable
- Make release collection operation cancels waiting search/query request
- Reduce the queryLock to collection level
- Add some unit tests for timeout & released case
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-16 18:23:55 +08:00
a8b81e215b
Fix deadlock in release replica ( #16925 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-11 21:57:53 +08:00
34833680da
Add lock before query release collection from replica ( #16880 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-10 18:09:53 +08:00
92b6293be4
Fix QueryNode log level ( #16604 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-07 10:27:51 +08:00
8c004fa981
Fix replicaID is allow zero from watch channel task ( #16788 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-05 18:11:51 +08:00
1a29337e6b
Print unflushed segmentIDs instead of whole SegmentInfo ( #16301 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-04-29 19:13:47 +08:00
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
69252f812d
Implement memory replica in Proxy, QueryNode and QueryCoord ( #16470 )
...
Related to #16298 #16291 #16154
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-20 16:15:41 +08:00
4f4a417c8f
Unify in log print ( #16368 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-04-06 14:09:35 +08:00
d2424fc50a
Print msgID instead of loadInfo when load segment ( #16238 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-28 23:13:27 +08:00
ffdad82b9c
Set load info in reschedule tasks ( #16065 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-16 11:43:21 +08:00
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
caf9cbfcd4
Fix load failure and remove parition release related code ( #16038 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-14 23:18:02 +08:00
6a66faa2fb
Explictly specify Load operation meta ( #16027 )
...
Fix #15996
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-14 09:50:01 +08:00
9ad5c14a4a
Return error when create message stream failed in QueryNode ( #15317 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-09 15:17:59 +08:00
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig ( #15843 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
12eb8c3da5
Add prometheus metrics for QueryNode ( #15649 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-02 14:49:55 +08:00
6a48071a5c
Set BaseMsg for LoadSegmentRequest, prevent crashing in log printing ( #15725 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-02-26 20:09:54 +08:00
a18d6b46a4
Fix watch dml channel fail because of no collection meta ( #15436 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-10 13:19:46 +08:00
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig ( #15501 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
4bccc85534
Use MsgChannelConfig in GlobalParams for all components ( #15163 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
Xiaofan
smellthemoon
congqixia
SimFG
yah01
Enwei Jiao
Bingyi Sun
wei liu
wayblink
codeman
xige-16
bigsheeper
zhenshan.cao
Letian Jiang
Jiquan Long
Cai Yudong