b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
0fe5e90e8b
enhance: Remove import v1 ( #31403 )
...
Remove all code and logic related to import v1.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-22 15:29:09 +08:00
a434d33e75
feat: Add import scheduler and manager ( #29367 )
...
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
7ce876a072
fix: Decoupling importing segment from flush process ( #30402 )
...
This pr decoups importing segment from flush process by:
1. Exclude the importing segment from the flush policy, this approch
avoids notifying the datanode to flush the importing segment, which may
not exist.
2. When RootCoord call Flush, DataCoord directly set the importing
segment state to `Flushed`.
issue: https://github.com/milvus-io/milvus/issues/30359
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-02-03 13:01:12 +08:00
1d4be0d257
Adjust datacoord for L0 Delta ( #28021 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-06 15:26:16 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
4f6122dd4a
enable ctx traceId for assignsegment on dc( #26972 ) ( #27108 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 09:53:22 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
f4e72cb170
remove sync segmentLastExpire every time when assigning( #25271 ) ( #25316 ) ( #25557 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-24 14:11:07 +08:00
268368031c
Remove redundant kv mocks ( #25420 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-07 19:38:26 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
748a935a11
Modify segment seal policy with number of statslog files ( #21957 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-03 17:07:52 +08:00
f1daef22a1
Fix failed bulkload segment marked as sealed ( #21799 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-31 12:41:53 +08:00
73d8509600
Improve ListSegments performance for datacoord catalog ( #21477 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
d401608899
Add a segment seal policy by number of binlog files ( #21263 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-20 14:09:25 +08:00
2390095232
Fix load uses compacted segments' binlogs ( #20655 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-11-17 20:37:10 +08:00
22fb1d3b93
Remove dup mock_kv.go and mock_txn_kv.go ( #20267 )
...
Replace them with mockery generated mocks in kv/mocks
See also: #19289
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-03 14:41:35 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
31db01b6ae
Support TTL for collection level ( #19383 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
91df8f2d6a
Use log id instead of log path of the binlog in metastore ( #19123 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-25 15:56:51 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
70254c3ec5
Unified catalog interface for segment ( #18289 ) ( #18290 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
f0b036a35a
Move bulk load segment lock happen early ( #17612 )
...
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-27 13:56:17 +08:00
05273309e8
Drop empty sealed segment ( #17708 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-06-24 15:16:14 +08:00
f31be941bb
Fix bulkload row count bug for multi datanodes ( #17384 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-06-07 16:56:07 +08:00
e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated ( #16609 )
...
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-25 11:07:47 +08:00
f4caa94ea8
[skip e2e]Fix segment manager typo ( #13644 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-17 22:34:42 +08:00
0c39e0370c
Move segment statistics update to datanode tt loop ( #13233 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-15 10:53:16 +08:00
405b3cd932
Use the same lint rules with golint in revive ( #13324 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-14 15:31:07 +08:00
06ced4d1d1
Add DropVirtualChannel for DataCoord ( #12361 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-29 22:35:41 +08:00
5c8978e5d6
Fix drop segment bug ( #11709 )
...
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-12 17:31:11 +08:00
54b40da4c5
Add segment drop logic ( #11621 )
...
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-12 00:22:42 +08:00
548634ff05
[skip ci]Update license for dc segmentinfo ( #10588 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 19:44:37 +08:00
f18e0628ae
Make NewMeta private ( #9057 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-02 23:09:56 +08:00
c152cf2b6b
Check error in segment_manager_test.go ( #8706 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-09-27 22:42:02 +08:00
664aeb09c6
Add ut and comments for datacoord meta ( #7560 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-08 11:35:59 +08:00
5b22697ec4
Add segment manager unit tests ( #7525 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-07 13:59:58 +08:00
f5451d12f3
Refactor datacoord allocator ( #7226 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-23 17:59:51 +08:00
3cc210da26
Add flush interval ( #6996 )
...
Time tick message is sent about per 0.2 seconds. DataCoord will call
`FlushSegment` too frequently. We add a min interval for segment flush. A
segment will called `FlushSegment` only once between the interval.
issue: #6995
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-19 14:08:10 +08:00
31182e52a9
Fix expire allocation bug ( #7053 )
...
issue: #7052
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-12 10:48:08 +08:00
fc74a0f578
Auto create new segments when allocating rows more than ( #6665 )
...
max number of rows per segment
If user insert too much rows in a request. Now we will return a failed
response. Maybe auto creating new segments to hold that much rows is a
better way.
issue: #6664
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-23 21:58:33 +08:00
83be910baa
Refactor segment manager ( #6441 )
...
issue: #6440
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-12 17:24:25 +08:00
017484e093
Refactor meta in data coord ( #6300 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-07 14:02:01 +08:00
f6761bbc33
Refactor datacoord segment status ( #6039 )
...
* Refactor datacoord segment status
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove lastExpireTime
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-24 14:20:10 +08:00
46f86a0354
Return sealed segments when calling flush ( #6011 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-23 16:56:11 +08:00
c863a193de
Rename filename from dataservice to datacoordinator ( #5946 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 10:42:07 +08:00
SimFG
yihao.dai
aoiasd
jaime
MrPresent-Han
Enwei Jiao
Bingyi Sun
congqixia
yah01
Xiaofan
XuanYang-cn
codeman
Ten Thousand Leaves
groot
godchen
Xiangyu Wang
sunby