c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs ( #30941 )
...
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.
issue: https://github.com/milvus-io/milvus/issues/30004
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
a434d33e75
feat: Add import scheduler and manager ( #29367 )
...
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
073de855b3
enhance:remove some unused code ( #29827 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-14 10:38:51 +08:00
9fc5f1176c
fix: Drop segment meta info with prefix ( #29856 )
...
If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 12:22:51 +08:00
1d4be0d257
Adjust datacoord for L0 Delta ( #28021 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-06 15:26:16 +08:00
2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 ( #27483 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-23 21:44:09 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
f9c47bbc1e
Fix drop index with large txn exceed etcd limit ( #25622 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-17 21:04:21 +08:00
7bd9eceb15
Remove unused KV functions ( #25972 )
...
Remove RevertAlterSegmentsAndAddNewSegment
Remove AlterIndex
Remove AlterSegmentsAndAddNewSegment
Remove AlterSegmentIndex
Remove IndexCoordCatalog
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-07-28 10:13:03 +08:00
bccdef1ad7
Fix segment meta only record last flushed binlog ( #25707 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-07-19 15:28:57 +08:00
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
b22d6d2a5d
Avoid update all binlogs every a segment is updated ( #25526 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-13 14:08:29 +08:00
70c4ddc6c5
Replace LoadWithPrefix with HasPrefix ( #25471 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-11 19:38:28 +08:00
3112dad356
Remove kv mocks ( #25426 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-08 13:20:26 +08:00
268368031c
Remove redundant kv mocks ( #25420 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-07 19:38:26 +08:00
54acb5b725
Remove alter segment interface ( #25382 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-06 20:46:25 +08:00
fe22720ff3
Avoid rewriting all binlogs every time a segment is updated ( #25339 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-06 18:20:26 +08:00
5659a3b728
Ignore log number check for compaction ( #24835 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-13 20:04:38 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
c84bdcea49
merge stats log when segment flushing or compacting ( #23570 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC ( #22559 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
2023-03-09 14:13:52 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
f2575e5fa8
Add unconvert & durationcheck linters and fix issues ( #22161 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-15 17:22:34 +08:00
a2435cfc4f
[Cherry-Pick]Add a segment metrics for count number of binlog files ( #22103 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-13 10:24:33 +08:00
9fd9bed2b9
Remove collection meta after GC finished ( #21595 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-12 09:55:42 +08:00
e127cf7b99
Reset indexpb for upgrade ( #21620 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-11 14:35:40 +08:00
73d8509600
Improve ListSegments performance for datacoord catalog ( #21477 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
5d044c4f29
Add check for stats log and bin log number ( #21264 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-26 16:35:30 +08:00
9d8f422cb8
Reduce maxFieldNum from 256 to 64 ( #21104 )
...
issue: #21077
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-14 10:39:22 +08:00
f8cff79804
Support the graceful stop for the query node ( #20851 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
482986cdf6
Revert "Fix fake segment is never handled ( #20935 )" ( #20945 )
...
This reverts commit 8f449f2996
2022-12-02 11:41:16 +08:00
8f449f2996
Fix fake segment is never handled ( #20935 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-01 20:03:16 +08:00
c373b44c2c
Verify and adjust segment # of rows before saving and passing SegmentInfo ( #20837 )
...
issue: #18120
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-01 16:31:16 +08:00
dffcd974ff
Make sure that the segment path matches the segment id ( #20677 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-18 15:35:09 +08:00
f3ef430c2e
Keep individual segment's updates in a single Etcd operation ( #20525 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-15 16:45:07 +08:00
cd19d99ad7
Add channel level checkpoint ( #20350 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
9751bec04c
Split large Etcd operations while altering segment info ( #20278 )
...
/kind bug
issue: #20275
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-03 11:01:35 +08:00
d00028edba
Mark segment as fake in flush path ( #20141 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 19:29:33 +08:00
11efa0bb5f
Fix compacted segment not dropped ( #20128 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 17:15:32 +08:00
def5972e01
fix duplicated handoff request ( #20108 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 15:15:32 +08:00
d8ca56e084
Fix empty segment handoff fails ( #20073 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-26 16:49:31 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
yihao.dai
smellthemoon
wei liu
aoiasd
Xiaofan
SimFG
congqixia
Enwei Jiao
Filip Haltmayer
Jiquan Long
Bingyi Sun
yah01
jaime
wayblink
cai.zhang
Ten Thousand Leaves