5d3aa2a496
fix: [Cherry-pick] Check latest leader exists before using it ( #31500 ) ( #31546 )
...
Cherry-pick from master
pr: #31500
See also #31495
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-23 11:51:06 +08:00
7ee7d484cc
fix: Transfer l0 segment to new delegator after balance ( #31332 )
...
issue: #30186
pr: #31319
during channel balance, after new delegator loaded, instead of syncing
l0 segment's location to new delegator, we should load l0 segment on new
delegator, and release the old l0 segment, then start to release old
delegator.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-18 20:49:04 +08:00
06b191b164
fix: Balance channel stuck forever due to logic dead lock ( #31202 )
...
issue: #30816
cause balance channel will stuck until leader view catch up the current
target, then start to unsub the old delegator. which make sure that the
new delegator can provide search before release old delegator. but
another logic in segment_checker skip loading segment during balance
channel. so during balance channel, if query node crash, new delegator
can't catch up target forever, then stuck forever.
This PR remove the rule that skip loading segment during balance channel
to avoid the logic dead lock here.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-13 15:05:04 +08:00
efe8cecc88
enhance: refactor segment dist manager interface ( #31073 )
...
issue: #31091
This PR add `GetByFilter` interface in segment dist manager, instead of
all kind of get func
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 16:29:01 +08:00
2a047103d6
fix: Dirty sealed segment won't release after channel balance ( #31095 )
...
issue: #31074
This PR fix dirty sealed segment doesn't release after channel balance,
dirty sealed segment means segment doesn't exist in targets.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-07 16:23:01 +08:00
565fc3a019
enhance: Skip generate load segment task ( #29724 )
...
issue: #29814
if channel is not subscribed yet, the generated load segment task will
be remove from task scheduler due to the load segment task need to be
transfer to worker node by shard leader.
This PR skip generate load segment task when channel is not subscribed
yet.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-12 18:56:58 +08:00
c4ddfff2a7
enhance: make Load process traceable in querycoord ( #29806 )
...
See also #29803
This PR:
- Add trace span for collection/partition load
- Use TraceSpan to generate Segment/ChannelTasks when loading
- Refine BaseTask trace tag usage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 09:58:49 +08:00
e98c62abbb
enhance: refactor leader_observer to leader_checker ( #29454 )
...
issue: #29453
sync distribution by rpc will also call loadSegment/releaseSegment,
which may cause all kinds of concurrent case on same segment, such as
concurrent load and release on one segment.
This PR add leader_checker which generate load/release task to correct
the leader view, instead of calling sync distribution by rpc
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-05 15:54:55 +08:00
45e6801ce4
feat: Add checker activation service interfaces ( #28850 )
...
issue: #28610
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-04 17:38:37 +08:00
8514a39d1a
feat: Add checker activation ( #28611 )
...
issue: https://github.com/milvus-io/milvus/issues/28610
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-24 18:08:24 +08:00
852be152de
Change task sourceID to stringer interface ( #27965 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-27 01:08:12 +08:00
e0222b2ce3
refine target manager code style ( #27883 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 00:44:12 +08:00
635efdf170
Schedule loading L0 segments first ( #27593 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-19 11:14:06 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
c45c32fad4
Set task reason for collection released ( #26962 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 15:15:17 +08:00
5602b22531
refine checker code style ( #26759 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-01 11:57:01 +08:00
949c320185
remove pull target from qc recover ( #26775 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-01 11:17:01 +08:00
1748c54fd7
skip load/release segment when more than one delegator exist ( #25718 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-24 19:01:01 +08:00
7d00020c9e
Reduce DataScope to historical for segment release task ( #25489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-12 09:12:28 +08:00
b68fa2049b
Fix querycoord segment checker nil reference ( #25290 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-03 18:48:24 +08:00
68ae199a9f
load segment with target version, avoid read redundant segment ( #24929 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 11:48:45 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
afd874b736
enhance segment balance by considering global rowCount(##22914) ( #23056 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-04-03 14:16:25 +08:00
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
21ba8182ee
Refine task create errors ( #22745 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-14 18:51:53 +08:00
d4fccdd135
Add reason for spawning task ( #22549 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-06 10:13:50 +08:00
73c44d4b29
resource group impl ( #21609 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
eaa5cfdcb5
fix leader observer sync logic ( #20478 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-11 11:43:06 +08:00
c5cd92d36e
update target ( #19296 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
4412cfcaaf
reduce querycoord unnecessary panic ( #19925 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-10-28 17:15:32 +08:00
1c71844b8d
Add license header ( #19678 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:39:22 +08:00
377f856833
Fix balance may confuse leader observer ( #19435 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-28 12:10:54 +08:00
539585e91b
Check whether leader view exists ( #19237 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-20 16:10:50 +08:00
50ea4eeef1
Fix querynode data distribution return false positive dm channel ( #19249 )
...
Co-authored-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2022-09-20 12:04:49 +08:00
626854cf0c
Refactor QueryCoord ( #18836 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-15 18:48:32 +08:00
congqixia
wei liu
Bingyi Sun
yah01
SimFG
jaime
MrPresent-Han
yah01
Enwei Jiao