321c5c32e3
fix: Separate schedule and check results loop ( #28692 )
...
This PR:
- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.
- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.
- Skips meta change when meta's already changed, avoid panic
- Remove not inuse injectDone(bool) parameter
See also: #28628 , #28209
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-29 10:50:29 +08:00
40d5c902b6
Enable getting multiple segments in plan result ( #28350 )
...
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.
- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map ( #25846 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
b75dcf90c0
Fix Flush hang after SyncSegments timeout ( #24953 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-06-20 10:20:41 +08:00
f57fe6d70b
Add compaction log ( #24976 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-19 14:18:41 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
4a90490a67
Fix the `segment not found` error ( #22772 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
0ce70b4f10
fix compact task data race ( #22299 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-21 16:10:26 +08:00
1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency ( #22143 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:00:33 +08:00
e081eb287d
Refine flowgraph nodes for readability ( #19864 )
...
- delete compaction tasks from executor while dropping
See also: #19780
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-20 16:39:29 +08:00
d9c7519107
refactor compaction concurency logic ( #18660 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-23 15:50:52 +08:00
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
8ab9d769de
Fix logs of compaction_executor ( #15365 )
...
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-25 13:29:40 +08:00
87394e29b6
Fix compaction ut datarace ( #15267 )
...
Fixes : #15241
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-18 17:49:39 +08:00
fefeaa2e1d
[skip e2e]Fix compaction executor typo ( #13646 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-17 19:20:46 +08:00
8bbaae04ff
Decrease compaction parallel ( #12707 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-03 18:57:33 +08:00
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode ( #12563 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
8a8ebed23d
Fix DN GC bug ( #12258 )
...
Fixes : #12249
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
c4bb73dc74
Complete drop collection procedure in DN ( #11552 )
...
drop, cancel compaction, flush all buffers and release fg
See also: #11426
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
3cd0584a04
Add compaction in datanode ( #11353 )
...
See also: #9530
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-08 19:49:07 +08:00
XuanYang-cn
congqixia
jaime
Xiaofan
SimFG
wei liu
aoiasd
Enwei Jiao
bigsheeper
godchen
Bingyi Sun