f12e368a76
fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 ( #35694 )
...
See also: [#35701 ](https://github.com/milvus-io/milvus/issues/35701 )
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-27 10:36:59 +08:00
6e9fbd1630
enhance: Solve channel unbalance on datanode ( #34984 )
...
issue: #33583
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.
This PR refine the balance policy to solve channel unbalance on datanode
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 17:35:47 +08:00
314f4d995b
enhance: Tidy dc channel manager ( #34515 )
...
See also: #34518
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-09 18:26:12 +08:00
5e39aa9272
enhance: Make channel meta able to writer 200k plus segments ( #33279 )
...
See also: #33125
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-28 12:33:42 +08:00
40e5c0be70
fix: Enable to balance released standby channel ( #32986 )
...
See also: #32879
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-14 11:51:32 +08:00
bc015df962
fix: [skip-e2e] Fix unstable policy_test ( #32883 )
...
See also: #32849
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-09 15:39:40 +08:00
0b0d279515
fix: [skip-e2e] Fix AvgAssignByCount unstable ( #32850 )
...
See also: #32849
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-08 19:09:35 +08:00
b1eacb2ae8
feat: datacoord/node watch based on rpc ( #32036 )
...
issue: https://github.com/milvus-io/milvus/issues/25309
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-07 15:49:30 +08:00
7e830f6185
fix: [skip-e2e] TestAvgReassignPolicy ut ( #32450 )
...
See also: #32448
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-19 15:11:20 +08:00
83da08c388
enhance: Use map instead of slice to maintain channel info ( #32273 )
...
See also #32165
`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.
This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 15:57:19 +08:00
623939c9f5
enhance: Remove not in use policies ( #29448 )
...
THe results don't meet our requirements, and the code hasn't been
maintained for a long time.
See also: #29447
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-28 10:38:46 +08:00
87465f07a7
Fix unstable ConsistencyHashPolicy ut ( #28375 )
...
Fixes : #28372
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 19:06:20 +08:00
a153950b10
Change channel to Interface ( #27839 )
...
This PR changes `*channel` into RWChannel interface
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:16:18 +08:00
14c8a90517
Fix auto balance block channel reassign after datanode restart ( #28275 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-09 19:00:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
77c9e33e70
support dml channel balancer on datacoord ( #22324 ) ( #22377 ) ( #22692 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-03-20 10:01:56 +08:00
148a024e05
Add tickle for datacoord watch event ( #21193 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:20:34 +08:00
e25f987a5c
Fix bug that calculated an extra DataNode of register policy ( #21927 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-06 10:33:54 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
0700e56008
Increase MaxWatchDuration and make it configurable ( #20884 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-29 17:19:14 +08:00
641afae22f
Watch channels with start positions ( #18744 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-08-22 14:42:52 +08:00
42f643e727
Make DataNode release rather than delete when reassign ( #17293 )
...
1. Reassgin now will assign to the original Node if no other nodes
avaliable
2. Make AddNode balance async: ToRealse + Reassign
See also: #16114 , #17270
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-02 13:56:04 +08:00
3eb074a9b0
Update some comments for data coord with some minor code refactoring. ( #15686 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-22 13:15:51 +08:00
169af67eb1
Fix segments not removed from flushingCache ( #10738 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-27 16:06:24 +08:00
2de3131bb3
Change default balance policy ( #10563 )
...
issue: #10198
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-26 19:40:44 +08:00
23c7691d59
[skip ci]Update license for dc policy ( #10590 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 19:48:23 +08:00
044b06421a
Use etcd to watch channel on datanode ( #7115 )
...
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-14 15:44:34 +08:00
be5c492e5a
Add datacoord unit tests ( #7197 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-20 17:50:12 +08:00
ee38d14f5d
Fix channel lost after data node crash ( #6545 )
...
If we start up 2 data nodes and one of them crashes. We expect that all
channels of crashed node will be allcoated to the alive node. But now we
discover that these channels are lost after data node crash. The reason
is we pass a NodeInfo with empty channel info. We fix it and improve log
print.
issue: #6501
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-15 16:38:31 +08:00
a8e5fd2024
Refactor cluster in dataservice ( #6356 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-12 11:03:52 +08:00
d6259d2dd6
Refactor policy ( #6003 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-23 14:28:08 +08:00
c863a193de
Rename filename from dataservice to datacoordinator ( #5946 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 10:42:07 +08:00
XuanYang-cn
wei liu
yiwangdr
congqixia
SimFG
jaime
MrPresent-Han
aoiasd
cai.zhang
Enwei Jiao
Ten Thousand Leaves
Jiquan Long
sunby