aad3ed3835
fix: [cherry-pick]Skip changing meta if nodeID not match with channel ( #31672 )
...
See also: #31648
pr: #31665 , #31694
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-10 15:09:18 +08:00
0066c016b6
enhance: Skip submit empty l0 tasks in DC ( #31280 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-19 10:13:14 +08:00
8c43c5b6cb
fix: get compaction failure when datanode is actually alive ( #31353 )
...
didn't mark the compact as failure if it's simply an rpc error when
GetCompactionPlansResults
see #31352
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-18 10:01:36 -07:00
196f0c1e1d
fix: Skip invalid compaction plan ( #31045 )
...
See also #31044
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-06 21:36:59 +08:00
def72947c7
fix: Trigger l0 compaction when l0 views don't change ( #30729 )
...
Trigger l0 compaction when l0 views don't change
So that leftover l0 segments would be compacted in the end.
1. Refresh LevelZero plans in comactionPlanHandler, remove the meta
dependency
of compaction trigger v2
2. Add ForceTrigger method for CompactionView interface
3. rename mu to taskGuard
4. Add a new TriggerTypeLevelZeroViewIDLE
5. Add an idleTicker for compaction view manager
See also: #30098 , #30556
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-05 16:37:00 +08:00
2867f50fcc
fix: Clear DN unkown compaction tasks ( #30850 )
...
If DC restarted, those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.
See also: #30137
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-01 11:31:00 +08:00
cdc5ce5d6f
fix: Donot set LogPath when executing compaction ( #30537 )
...
Compaction would copy logPaths from comapctFrom segA to compactTo segB,
and previous code would copy the logPath directly, causing there're
full-logPaths-of-segA in compactTo segB's meta. So, for the next
compaction of segB, if segA has been GCed, Download would report error
"The sperified key not found".
This PR makes sure compactTo segment's meta contains logID only. And
this PR also refines CompleteComapctionMutation, increasing some
readability and merge two methods into one.
See also: #30496
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-28 19:03:01 +08:00
b74264881c
enhance: Refine compaction interfaces to support major compaction ( #30632 )
...
Refine compaction interfaces in datacoord, support compaction result
with more than one segment. Prepare for major compaction.
related: #30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-02-19 20:52:50 +08:00
f2985d8454
fix: compact operation on datacoord meta should preform as a transcation ( #29775 )
...
issue: #29691
Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-26 16:59:00 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
cb18f18c1d
fix: compacted segment status was flushing instead flushed and L0 segment trigger gc slowly ( #29587 )
...
relate: https://github.com/milvus-io/milvus/issues/29492
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-09 10:52:49 +08:00
75e6b65c60
enhance: Use ChannelManger interface in Server ( #29629 )
...
See also: #29447
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-08 17:46:47 +08:00
f1b6ccf305
enhance: compaction use ChannelManager interface ( #29530 )
...
Rewrite compaction_test.go
See also: #29447
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-02 18:08:49 +08:00
ae180d1628
enhance: Change ChannelManager to interface ( #29300 )
...
Rewrite cluster test
issue: #28854
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-25 19:24:46 +08:00
67ab0e424b
fix: Clean the compaction plan info to avoid the object leak ( #29365 )
...
issue: #29296
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 12:00:43 +08:00
3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them ( #28869 )
...
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-11 17:52:37 +08:00
5bac7f7897
fix: Fix L0 compaction in datacoord ( #28814 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
9b371067d2
feat: Add Compaction views and triggers ( #27906 )
...
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00
40d5c902b6
Enable getting multiple segments in plan result ( #28350 )
...
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.
- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
a153950b10
Change channel to Interface ( #27839 )
...
This PR changes `*channel` into RWChannel interface
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:16:18 +08:00
22c089894e
Add scheduler ( #27938 )
...
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-07 03:18:18 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
4f6122dd4a
enable ctx traceId for assignsegment on dc( #26972 ) ( #27108 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 09:53:22 +08:00
9598a8b236
Fix timeout task never release queue ( #26593 )
...
See also: #26413 , #26566
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-30 11:12:27 +08:00
95f91e6e35
Fix segments meta not consistent between catalog and in-memory ( #25799 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: czs007 <zhenshan.cao@zilliz.com>
2023-07-21 14:21:05 +08:00
b9189b9f41
Organize mocks from types.go ( #25466 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-07-14 10:12:31 +08:00
70c4ddc6c5
Replace LoadWithPrefix with HasPrefix ( #25471 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-11 19:38:28 +08:00
268368031c
Remove redundant kv mocks ( #25420 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-07 19:38:26 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
4a90490a67
Fix the `segment not found` error ( #22772 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
3d6bded115
Fix Row Number Mismatch ( #22307 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-27 10:41:46 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency ( #22143 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:00:33 +08:00
7adabe09f1
Fix Alloc timstamp failure blocks compaction queue forever ( #22039 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-07 19:02:31 +08:00
73d8509600
Improve ListSegments performance for datacoord catalog ( #21477 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
5e49b095f5
Fix compaction handler may double release and block forever ( #21015 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-06 20:09:18 +08:00
dffcd974ff
Make sure that the segment path matches the segment id ( #20677 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-18 15:35:09 +08:00
11efa0bb5f
Fix compacted segment not dropped ( #20128 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 17:15:32 +08:00
d8ca56e084
Fix empty segment handoff fails ( #20073 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-26 16:49:31 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
a0ce533727
Add a SyncSegments to sync meta between DN and DC ( #19298 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-27 16:02:53 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
362c663b22
Remove deprecated inner compaction type ( #19024 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-07 19:16:34 +08:00
43c14055f6
Fix test compact logic and data race ( #18932 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-31 12:10:57 +08:00
XuanYang-cn
Xiaofan
congqixia
wayblink
chyezh
smellthemoon
aoiasd
SimFG
jaime
MrPresent-Han
Jiquan Long
yiwangdr
Bingyi Sun
yah01
Enwei Jiao
cai.zhang