b74264881c
enhance: Refine compaction interfaces to support major compaction ( #30632 )
...
Refine compaction interfaces in datacoord, support compaction result
with more than one segment. Prepare for major compaction.
related: #30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-02-19 20:52:50 +08:00
f2985d8454
fix: compact operation on datacoord meta should preform as a transcation ( #29775 )
...
issue: #29691
Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-26 16:59:00 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
cb18f18c1d
fix: compacted segment status was flushing instead flushed and L0 segment trigger gc slowly ( #29587 )
...
relate: https://github.com/milvus-io/milvus/issues/29492
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-09 10:52:49 +08:00
75e6b65c60
enhance: Use ChannelManger interface in Server ( #29629 )
...
See also: #29447
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-08 17:46:47 +08:00
f1b6ccf305
enhance: compaction use ChannelManager interface ( #29530 )
...
Rewrite compaction_test.go
See also: #29447
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-02 18:08:49 +08:00
ae180d1628
enhance: Change ChannelManager to interface ( #29300 )
...
Rewrite cluster test
issue: #28854
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-25 19:24:46 +08:00
67ab0e424b
fix: Clean the compaction plan info to avoid the object leak ( #29365 )
...
issue: #29296
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 12:00:43 +08:00
3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them ( #28869 )
...
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-11 17:52:37 +08:00
5bac7f7897
fix: Fix L0 compaction in datacoord ( #28814 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
9b371067d2
feat: Add Compaction views and triggers ( #27906 )
...
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00
40d5c902b6
Enable getting multiple segments in plan result ( #28350 )
...
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.
- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
a153950b10
Change channel to Interface ( #27839 )
...
This PR changes `*channel` into RWChannel interface
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:16:18 +08:00
22c089894e
Add scheduler ( #27938 )
...
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-07 03:18:18 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
4f6122dd4a
enable ctx traceId for assignsegment on dc( #26972 ) ( #27108 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 09:53:22 +08:00
9598a8b236
Fix timeout task never release queue ( #26593 )
...
See also: #26413 , #26566
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-30 11:12:27 +08:00
95f91e6e35
Fix segments meta not consistent between catalog and in-memory ( #25799 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: czs007 <zhenshan.cao@zilliz.com>
2023-07-21 14:21:05 +08:00
b9189b9f41
Organize mocks from types.go ( #25466 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-07-14 10:12:31 +08:00
70c4ddc6c5
Replace LoadWithPrefix with HasPrefix ( #25471 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-11 19:38:28 +08:00
268368031c
Remove redundant kv mocks ( #25420 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-07 19:38:26 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
4a90490a67
Fix the `segment not found` error ( #22772 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
3d6bded115
Fix Row Number Mismatch ( #22307 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-27 10:41:46 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency ( #22143 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:00:33 +08:00
7adabe09f1
Fix Alloc timstamp failure blocks compaction queue forever ( #22039 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-07 19:02:31 +08:00
73d8509600
Improve ListSegments performance for datacoord catalog ( #21477 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
5e49b095f5
Fix compaction handler may double release and block forever ( #21015 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-06 20:09:18 +08:00
dffcd974ff
Make sure that the segment path matches the segment id ( #20677 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-18 15:35:09 +08:00
11efa0bb5f
Fix compacted segment not dropped ( #20128 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 17:15:32 +08:00
d8ca56e084
Fix empty segment handoff fails ( #20073 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-26 16:49:31 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
a0ce533727
Add a SyncSegments to sync meta between DN and DC ( #19298 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-27 16:02:53 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
362c663b22
Remove deprecated inner compaction type ( #19024 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-07 19:16:34 +08:00
43c14055f6
Fix test compact logic and data race ( #18932 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-31 12:10:57 +08:00
096149e33f
fix GetCompactionState hang ( #18829 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-26 14:22:55 +08:00
c924f73105
Refactor for IndexCoord to support cloud ( #18643 )
...
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
d9c7519107
refactor compaction concurency logic ( #18660 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-23 15:50:52 +08:00
70254c3ec5
Unified catalog interface for segment ( #18289 ) ( #18290 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
ed94ecf847
Segments can't be compacted when they have reference lock ( #17649 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-20 21:56:12 +08:00
54aeb077e9
Refine compaction selection poliy ( #17486 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-15 23:14:10 +08:00
dba6a3f4b0
[skip e2e]Fix license missing issue in datacoord ( #14427 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-28 15:43:51 +08:00
wayblink
chyezh
smellthemoon
aoiasd
XuanYang-cn
SimFG
jaime
congqixia
MrPresent-Han
Jiquan Long
yiwangdr
Bingyi Sun
yah01
Xiaofan
Enwei Jiao
cai.zhang
codeman
Xiangyu Wang