8780d65b66
fix: Use channel cp as the dml&start position for import segments ( #30107 )
...
This PR discontinuing the subscription to the mq and, instead, employing
the channel checkpoint as the DML and starting position for the import
segments.
issue: https://github.com/milvus-io/milvus/issues/30106
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-22 14:36:55 +08:00
d8025177fa
fix: return correct compaction plan count by datacoord ( #29980 )
...
issue: #29943
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-19 21:06:55 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
75e6b65c60
enhance: Use ChannelManger interface in Server ( #29629 )
...
See also: #29447
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-08 17:46:47 +08:00
010d8362ad
fix: datanode L0 segment num wrong ( #29050 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 11:20:44 +08:00
cbf0f9c527
enhance: change cp metric to absolute unix ts ( #29328 )
...
See also #29327
Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 11:04:45 +08:00
8a63e53421
enhance: Add http method to control datacoord garbage collection ( #29052 )
...
See also #29051
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 19:26:39 +08:00
58dbb7872a
fix: forbid balancing level zero segments ( #29168 )
...
related #29128
Signed-off-by: yah01 <yah2er0ne@outlook.com>
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 14:30:39 +08:00
ad866d2889
feat: integrate storagev2 into index build process ( #28995 )
...
issue: https://github.com/milvus-io/milvus/issues/28994
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-13 17:24:38 +08:00
ef18e6a532
enhance: Use a non-blocking method to trigger compaction when saveBinlogPath is executed ( #28941 )
...
/kind improvement
issue: #28924
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-12 10:16:39 +08:00
3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them ( #28869 )
...
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-11 17:52:37 +08:00
5bac7f7897
fix: Fix L0 compaction in datacoord ( #28814 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
321c5c32e3
fix: Separate schedule and check results loop ( #28692 )
...
This PR:
- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.
- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.
- Skips meta change when meta's already changed, avoid panic
- Remove not inuse injectDone(bool) parameter
See also: #28628 , #28209
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-29 10:50:29 +08:00
606ec77b66
enhance: Unify levelzero segment config in DN ( #28720 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-28 18:04:26 +08:00
5025c10e8b
enhance: Remove channel cp lag metrics when DropChannel ( #28784 )
...
See also #28765
Remove metric when Drop channel grpc execute succeed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 14:24:25 +08:00
8a4cfb7d6a
enhance: add l0 metric and fix datacoord no need drop l0 segment when flush ( #28373 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-24 15:58:24 +08:00
4fedff6d47
feat: integrate storage v2 into the write path ( #28440 )
...
#28378
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
73f2bab454
enhance:add some log when create client and get component states ( #28160 )
...
/kind improvement
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-22 09:12:22 +08:00
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition ( #28456 )
...
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
b1fbc1f4e7
fix: datacoord l0 segment bugs ( #28490 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-21 16:48:22 +08:00
13a5b9f64a
fix: query l0 segment bugs ( #28558 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-20 17:26:23 +08:00
40d5c902b6
Enable getting multiple segments in plan result ( #28350 )
...
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.
- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
a153950b10
Change channel to Interface ( #27839 )
...
This PR changes `*channel` into RWChannel interface
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:16:18 +08:00
5b45a138b1
disable auto balance when old node exists ( #28191 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
1d4be0d257
Adjust datacoord for L0 Delta ( #28021 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-06 15:26:16 +08:00
7e0bda8b4c
remove binlog/delta log from getRecoveryInfoV2 ( #27895 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-30 14:28:15 +08:00
2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 ( #27483 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-23 21:44:09 +08:00
49516d44b4
Add ctx parameter and log tracer for watch and selectNodes ( #27809 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-20 04:22:11 +08:00
49b3a12804
Return newly defined merr instead of grpc unimplemented err ( #27751 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-18 15:32:11 +08:00
efe8984eca
Fix Flush error message typo ( #27729 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-17 17:18:09 +08:00
e3f2122618
Expose metrics of stanby coordinators ( #27698 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-10-16 15:04:09 +08:00
c882e9c87b
Fix compatibility of Flush ( #27543 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-13 18:01:39 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
cb71a3e235
rm dependency to rc when getting recovery info( #25363 ) ( #27405 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-10-09 18:51:32 +08:00
6539a5ae2c
Refine DataCoord status ( #27262 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
773080df24
Fix miss to handle the error of allocating ts ( #27264 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 18:43:23 +08:00
fd73213539
Increase FlushChannel timeout to 3min ( #27180 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-18 19:49:22 +08:00
a6b98740b7
Replace all status with only error string ( #27125 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-18 14:05:28 +08:00
c162c6a4c8
Increase FlushChannel retry times ( #27140 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-16 00:41:20 +08:00
fe5781e2b8
Maintain compatibility of GetFlushState with old client ( #27077 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-14 19:57:20 +08:00
dd2cb1d44a
Flush by flush channels ( #26859 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-12 21:07:19 +08:00
0901b76732
Avoid the panic when the status of rpc response is nil ( #26910 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-07 19:23:15 +08:00
d7b6f3a9e8
Enhance FlushAll/GetFlushAllState API ( #26802 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-09-06 10:35:48 +08:00
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
bed034a44b
Protect DataCoord from calculating segment lines by stale log entries num ( #26523 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-22 14:16:22 +08:00
3c62acde05
Add coll and segState lebels to DataCoordNumStoredRows monitoring ( #26299 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 15:52:17 +08:00
8d13717cac
Fill Collection start position timestamp in WatchInfo ( #26370 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-16 09:05:32 +08:00
830f06783a
Reduce log volumes of DC&DN ( #26060 )
...
See also: #26057
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-02 17:13:11 +08:00
7613c7842c
Fix datacoord crash ( #25631 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-17 16:54:34 +08:00
yihao.dai
Bingyi Sun
smellthemoon
XuanYang-cn
aoiasd
congqixia
yah01
SimFG
yah01
wei liu
Xiaofan
wayblink
MrPresent-Han
jaime