3254a14319
fix: [Cherry-pick] Cleanup write buffer when flowgraph released ( #31377 )
...
Cherry-pick from master
pr: #31376
See also #30137
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-18 23:47:06 +08:00
c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs ( #30941 )
...
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.
issue: https://github.com/milvus-io/milvus/issues/30004
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
8780d65b66
fix: Use channel cp as the dml&start position for import segments ( #30107 )
...
This PR discontinuing the subscription to the mq and, instead, employing
the channel checkpoint as the DML and starting position for the import
segments.
issue: https://github.com/milvus-io/milvus/issues/30106
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-22 14:36:55 +08:00
1ee016709d
fix: Unstable `TestDataSyncService/TestStartStop` unit test ( #29291 )
...
fix #29290
Change EXPECT `NotifyCheckpointUpdated` call to `Maybe` expectation
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-19 23:16:42 +08:00
51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode ( #28570 )
...
/kind enhancement
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
4fedff6d47
feat: integrate storage v2 into the write path ( #28440 )
...
#28378
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
bed7467f20
enhance: Remove commented code and fix naming issue ( #28450 )
...
This PR removes all the commented code and files from PR #28320
For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-16 00:22:20 +08:00
0b905078e7
Use writebuffer, sync manager refactory in datanode ( #28320 )
...
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-15 15:24:18 +08:00
f4341f254d
Not convert legacy error code to new merr ( #28232 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 14:42:20 +08:00
a21042dde7
Fix flushManager.isFull is too slow ( #28141 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-11-03 14:42:17 +08:00
82b2edc4bd
Replace manual composed grpc call with Broker methods ( #27676 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-13 09:55:34 +08:00
56c94cdfa7
Add channel manager in DataNode ( #27308 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-08 21:37:33 +08:00
5c5f9aa05e
Enhance newDataSyncService ( #27277 )
...
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
- config.maxQueueLength, config.maxParallelish
See also: #27207
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-27 11:07:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
916528f51a
Add initCtx in getDataSyncService ( #27199 )
...
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-20 16:03:23 +08:00
dd2cb1d44a
Flush by flush channels ( #26859 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-12 21:07:19 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy ( #26713 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 14:37:47 +08:00
08fd28b30b
Only do gracefully stop when DN Stop ( #26399 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-21 11:16:20 +08:00
fc12d3997c
Rename newTimeTickManager to newTimeTickSender ( #25415 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-07-12 14:56:28 +08:00
af1d84e5e1
Support sync when cp is lagged ( #24989 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-20 10:00:41 +08:00
bfae6b49af
Remove datanode timetick mq, use rpc to report instead ( #23156 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-06-14 14:16:38 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
93bc805933
Enhance ID allocator in DataNode ( #22905 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
148a024e05
Add tickle for datacoord watch event ( #21193 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:20:34 +08:00
d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel ( #21917 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-13 16:38:33 +08:00
2b982b5d8f
Fix DataNode ut never meet condition ( #22093 )
...
See also: #22079
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-02-10 11:04:32 +08:00
6a722396bd
Integration test framework ( #21283 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
4b4944ecee
Support sync all segments while close ( #21421 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 14:49:36 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
039e9ce4bb
Refine msgstream interface ( #20832 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:55:17 +08:00
bf2107ecf5
Fix a regression of local storage ( #20653 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:19:10 +08:00
2785f98de6
Refactor segment replica in DN ( #19815 )
...
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-18 15:33:26 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
3049dbb5a5
Fix exclusive consumer conflict in import ( #19632 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 10:49:21 +08:00
d64f318a80
Add collection schema in watch channel info ( #19663 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-10 01:37:21 +08:00
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
12158432bb
Clear segment cache when closing flowgraph ( #17671 )
...
See also: #17537
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-22 10:16:14 +08:00
eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC ( #17441 )
...
Resolves : #17233 #16047
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-16 12:00:10 +08:00
5e1e7a6896
Fix an issue that causes flush hanging ( #17169 )
...
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-25 14:34:00 +08:00
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
ca129d4308
Support configurable msgstream ( #16131 )
...
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
78557ca6d7
Replace minio kv with minio chunk manager ( #15936 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
congqixia
yihao.dai
wayblink
Bingyi Sun
yah01
Jiquan Long
XuanYang-cn
SimFG
Enwei Jiao
Xiaofan
yah01
jaime
aoiasd
groot
Ten Thousand Leaves
godchen
xige-16
xiyichan