b5f039a221
fix: Assertion all async invocations in test case ( #29737 )
...
Resolves : #29736
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-07 15:54:47 +08:00
e98c62abbb
enhance: refactor leader_observer to leader_checker ( #29454 )
...
issue: #29453
sync distribution by rpc will also call loadSegment/releaseSegment,
which may cause all kinds of concurrent case on same segment, such as
concurrent load and release on one segment.
This PR add leader_checker which generate load/release task to correct
the leader view, instead of calling sync distribution by rpc
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-05 15:54:55 +08:00
10bb2431d8
test: add checker unittests ( #28954 )
...
issue: https://github.com/milvus-io/milvus/issues/28610
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 10:56:33 +08:00
45e6801ce4
feat: Add checker activation service interfaces ( #28850 )
...
issue: #28610
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-04 17:38:37 +08:00
8514a39d1a
feat: Add checker activation ( #28611 )
...
issue: https://github.com/milvus-io/milvus/issues/28610
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-24 18:08:24 +08:00
86ec6f4832
fix load index for stopping node ( #28047 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-03 07:58:18 +08:00
5d2eba2c2f
Set qcv2 index task priority to Low ( #28117 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-02 23:22:17 +08:00
852be152de
Change task sourceID to stringer interface ( #27965 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-27 01:08:12 +08:00
e0222b2ce3
refine target manager code style ( #27883 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 00:44:12 +08:00
635efdf170
Schedule loading L0 segments first ( #27593 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-19 11:14:06 +08:00
cd5f03f80c
Add var-name sub linter in revive ( #27424 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-07 10:09:31 +08:00
a8ce1b6686
Refine QueryCoord stopping ( #27371 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-27 16:27:27 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
c45c32fad4
Set task reason for collection released ( #26962 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 15:15:17 +08:00
2101f2d289
fix unstable checker id due to go map iteration( #26943 ) ( #26944 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-10 10:11:16 +08:00
758aad705d
Fix checker using default interval after manual check ( #26953 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-09 08:29:16 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
5602b22531
refine checker code style ( #26759 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-01 11:57:01 +08:00
949c320185
remove pull target from qc recover ( #26775 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-01 11:17:01 +08:00
065d1a962e
Add sourceID output for task.String and fill reduce channel reason ( #26447 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-18 13:50:19 +08:00
889424b3f9
Fix load index with empty file list ( #26236 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-09 18:39:16 +08:00
6f89620a43
remove pull target rpc from lock ( #26054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-04 10:31:06 +08:00
a3e22786ed
Move meta store to kv catalog ( #25915 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-07-31 13:57:04 +08:00
1748c54fd7
skip load/release segment when more than one delegator exist ( #25718 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-24 19:01:01 +08:00
76e03fe6d3
Set reason for balance, index checker generated tasks ( #25865 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 17:07:00 +08:00
1045c88102
Support replace indexed field in QueryCoord ( #25747 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-19 21:22:58 +08:00
6534396b3d
enable config different interval for different checker ( #25514 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-19 16:50:57 +08:00
5afea0e5bf
Fix querycoord crash ( #25638 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-17 16:56:35 +08:00
7d00020c9e
Reduce DataScope to historical for segment release task ( #25489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-12 09:12:28 +08:00
b68fa2049b
Fix querycoord segment checker nil reference ( #25290 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-03 18:48:24 +08:00
68ae199a9f
load segment with target version, avoid read redundant segment ( #24929 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 11:48:45 +08:00
e962a8ba31
Limit the frequency of debug logs ( #25009 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-20 14:14:41 +08:00
4387f36897
make etcdKV private ( #24778 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-13 10:52:38 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
67cf23d050
Fix panic while balancing releasing collection ( #24003 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-11 12:47:20 +08:00
ed81eaa963
Make CollectionObserver trigger checker more frequently during load procedure ( #23928 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-08 14:06:41 +08:00
240c5625cd
Fix nil pointer access ( #23919 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-05-08 10:08:39 +08:00
b517bc9e6a
refine balance mechanism including:( #23454 ) ( #23763 ) ( #23791 )
...
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-05-04 12:22:40 +08:00
cbfe7a45ef
fix pull target ( #23491 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:30:32 +08:00
b5a73e6d1a
Add OWNERS files for querycoordv2 sub pkgs ( #23489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-18 15:52:30 +08:00
dbbd703667
fix balance generate unexpected task ( #23299 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-11 14:38:30 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
afd874b736
enhance segment balance by considering global rowCount(##22914) ( #23056 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-04-03 14:16:25 +08:00
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
1f718118e9
Dynamic load/release partitions ( #22655 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
3d8f0156c7
Refine scheduler & executor of QueryCoord ( #22761 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-16 17:43:55 +08:00
21ba8182ee
Refine task create errors ( #22745 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-14 18:51:53 +08:00
d4fccdd135
Add reason for spawning task ( #22549 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-06 10:13:50 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
73c44d4b29
resource group impl ( #21609 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
6fb3542f2a
enable auto balance paramter( #21504 ) ( #21507 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-01-06 14:45:35 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
f8cff79804
Support the graceful stop for the query node ( #20851 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
2ecdb4ba4a
Etcd config source support TLS ( #20874 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
a82c235599
Scheduler should returns err if the queue is full ( #20465 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-11 17:55:05 +08:00
eaa5cfdcb5
fix leader observer sync logic ( #20478 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-11 11:43:06 +08:00
c5cd92d36e
update target ( #19296 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
4112c667d0
Refine task's priority ( #20364 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-07 14:53:02 +08:00
4412cfcaaf
reduce querycoord unnecessary panic ( #19925 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-10-28 17:15:32 +08:00
8bfa55e560
Fix memory & goroutine leak ( #20152 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-28 14:55:32 +08:00
c61a4d1243
Fix panic due to double stop ( #19849 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 12:13:28 +08:00
1c71844b8d
Add license header ( #19678 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:39:22 +08:00
9dcee37e1c
Skip canceled tasks ( #19513 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-28 19:58:54 +08:00
377f856833
Fix balance may confuse leader observer ( #19435 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-28 12:10:54 +08:00
539585e91b
Check whether leader view exists ( #19237 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-20 16:10:50 +08:00
50ea4eeef1
Fix querynode data distribution return false positive dm channel ( #19249 )
...
Co-authored-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2022-09-20 12:04:49 +08:00
626854cf0c
Refactor QueryCoord ( #18836 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-15 18:48:32 +08:00
congqixia
wei liu
Bingyi Sun
yah01
SimFG
MrPresent-Han
Enwei Jiao
yah01
yiwangdr
jaime
yihao.dai