86a36b105a
enhance: Tidy compaction executor ( #33778 )
...
Move compaction executor to compaction pacakge.
issue: https://github.com/milvus-io/milvus/issues/32451
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-14 14:34:01 +08:00
4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM ( #33554 )
...
See also: #33547
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 14:33:52 +08:00
0382628668
enhance: Add more tracing for l0 compactor ( #33435 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-03 10:19:49 +08:00
bcaacf6fe6
enhance: Load BF from storage instead of memory during L0 compaction ( #32913 )
...
To decouple compaction from shard, loading BF from storage instead of
memory during L0 compaction in datanode.
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-17 17:25:36 +08:00
4ae7cabb04
fix: Remove channel when create flowgraph timeout ( #33014 )
...
See also #33013
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-14 10:07:33 +08:00
6ef677f79e
fix: Remove metrics after flowgraph closed ( #32515 )
...
See also #32403
`fg_buffer_size` was decreased after metrics removed in flowgraph
ddnode, which make metrics value negative.
This PR move remove metrics logic into `dataSyncService.Close`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-23 17:33:23 +08:00
d9efea2fea
fix: Cleanup write buffer when flowgraph released ( #31376 )
...
See also #30137
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-19 01:33:05 +08:00
c6665c2a4c
test: support multiple data/querynodes in integration test ( #30618 )
...
issue: https://github.com/milvus-io/milvus/issues/29507
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
277849a915
enhance: separate serializer logic from sync task ( #29413 )
...
See also #27675
Since serialization segment buffer does not related to sync manager can
shall be done before submit into sync manager. So that the pk statistic
file could be more accurate and reduce complex logic inside sync
manager.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-26 10:40:47 +08:00
51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode ( #28570 )
...
/kind enhancement
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
4fedff6d47
feat: integrate storage v2 into the write path ( #28440 )
...
#28378
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
18dc6b61ce
enhance: fix LevelZero segment sync logic ( #28482 )
...
See also #27675
- Fix LevelZero segment cannot be flushed
- Add level option for syncTask
- Invoke `AddSegment` when new LevelZero segment is allocated
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-17 21:46:20 +08:00
bed7467f20
enhance: Remove commented code and fix naming issue ( #28450 )
...
This PR removes all the commented code and files from PR #28320
For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-16 00:22:20 +08:00
0b905078e7
Use writebuffer, sync manager refactory in datanode ( #28320 )
...
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-15 15:24:18 +08:00
0aa90de141
Reduce the goroutine in flowgraph to 2 ( #28233 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-13 10:50:17 +08:00
82b2edc4bd
Replace manual composed grpc call with Broker methods ( #27676 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-13 09:55:34 +08:00
56c94cdfa7
Add channel manager in DataNode ( #27308 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-08 21:37:33 +08:00
80eb5434d3
Fix var-naming caused by old PRs ( #27501 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-07 14:43:32 +08:00
8394b3a1ec
Block creating new error from status reason ( #27426 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-07 11:29:32 +08:00
5c5f9aa05e
Enhance newDataSyncService ( #27277 )
...
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
- config.maxQueueLength, config.maxParallelish
See also: #27207
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-27 11:07:25 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
916528f51a
Add initCtx in getDataSyncService ( #27199 )
...
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-20 16:03:23 +08:00
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek ( #26686 )
...
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
b9bc6681ae
Use atomic.bool for the isWatchFailed param in the tickler ( #26558 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-23 13:22:27 +08:00
08fd28b30b
Only do gracefully stop when DN Stop ( #26399 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-21 11:16:20 +08:00
3be4ac4022
Fix datanode/datacoord continuous restart ( #26470 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-20 21:20:24 +08:00
f371ec0316
Add data sync service close log for watch stuck debugging ( #26332 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-16 10:57:32 +08:00
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
bfae6b49af
Remove datanode timetick mq, use rpc to report instead ( #23156 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-06-14 14:16:38 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
c84bdcea49
merge stats log when segment flushing or compacting ( #23570 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
345855c984
Make datanode load statslog lazy if SkipBFStatsLog is true ( #23877 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-06 15:12:39 +08:00
b23723535c
Add log in AddImportSegment.getChannelLatestMsgID ( #23480 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-04-18 17:50:31 +08:00
695cc769df
Fix watch event timeout when IO Pool busy ( #23293 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-04-17 11:02:30 +08:00
b90fa5f459
Refine codes of datanode buffer ( #23168 )
...
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-04-10 18:42:30 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
1a485044a9
fix NumConsumers metric not work ( #23167 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-04-05 08:18:28 +08:00
4008ee6d60
Remove datanode ddNode forward delta logic ( #23005 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-27 14:18:00 +08:00
93bc805933
Enhance ID allocator in DataNode ( #22905 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
60cd548bf5
Rename concurrency package to conc ( #22453 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-28 14:19:47 +08:00
b758c305a7
Make Future generic ( #22411 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-27 12:07:47 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
9167773eff
Remove needless kv at dataSyncService ( #22206 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-16 16:36:39 +08:00
148a024e05
Add tickle for datacoord watch event ( #21193 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:20:34 +08:00
d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel ( #21917 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-13 16:38:33 +08:00
6a722396bd
Integration test framework ( #21283 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
4b4944ecee
Support sync all segments while close ( #21421 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 14:49:36 +08:00
f595500383
Move sync delete policy from deleteNode to insertBufferNode ( #21152 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:15:21 +08:00
yihao.dai
XuanYang-cn
congqixia
yiwangdr
smellthemoon
wayblink
Bingyi Sun
yah01
jaime
SimFG
Enwei Jiao
aoiasd
yah01
Xiaofan