3626f49025
fix: make sure balance candidate is alway pushed back ( #29702 )
...
See also #29699
Querycoord panicked when tried to pop from an empty heap. We assume the
heap shall not be empty, but in some branch, the candidate is never
pushed back.
This PR put pop & push in a closure and adds a defer call to push item
back.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-05 10:08:47 +08:00
336fce0582
enhance: Rewrite gen segment plan based on assign segment ( #29574 )
...
issue: #29582
This PR rewrite gen segment plan logic based on assign segment in
`score_based_balancer`
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-04 11:10:44 +08:00
6cbf9c489d
enhance: Rewrite gen stopping segment plan based on assign segment ( #29473 )
...
`AssignSegment` method defines how to assign segment to nodes, but
score_based_balance implement another assign logic in
`genStoppingSegmentPlan`
This PR rewrite gen stopping segment plan based on assign segment.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-26 14:26:56 +08:00
820ee692fc
enhance: Add config for querycoord auto balance channel ( #29231 )
...
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-18 10:00:40 +08:00
008bae675d
enhance: Skip balance segment when channel need be balanced ( #29116 )
...
issue: #28622
After we support balance segment with growing segment count #28623 , if
we balance segment and channel at same time, some segments need to be
rebalanced after balance channel finish.
This PR skip balance segment when channel need be balanced.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-14 16:44:43 +08:00
58dbb7872a
fix: forbid balancing level zero segments ( #29168 )
...
related #29128
Signed-off-by: yah01 <yah2er0ne@outlook.com>
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 14:30:39 +08:00
2f0c7a6544
fix: forbid balancing level zero segments ( #29130 )
...
we can't balance the L0 segments
related #29128
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-12 20:38:38 +08:00
a67fc08865
fix: balance_unstable_view unit test ( #29127 )
...
fix : #29126
Allow unstable output channel balance plan
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:02:39 +08:00
42e538b683
enhance: enable balance channel in querycoord ( #28469 )
...
issue: #23726
/kind improvement
1. enable auto balance channel between nodes in querycoord
2. make `genSegmentPlan` reuse the `AssignSegment` logic
3. make `genChannelPlan` reuse the `AssignChannel` logic
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-11 14:18:37 +08:00
911a915798
feat: enable balance based on growing segment row count ( #28623 )
...
issue: #28622
query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-27 14:58:26 +08:00
852be152de
Change task sourceID to stringer interface ( #27965 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-27 01:08:12 +08:00
e0222b2ce3
refine target manager code style ( #27883 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 00:44:12 +08:00
41124f281a
Remove parser dependency ( #27514 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-08 15:05:31 +08:00
a8ce1b6686
Refine QueryCoord stopping ( #27371 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-27 16:27:27 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
1a8cf5c415
Organize all mockery generation commands in Makefile ( #26826 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
213db490bd
Use pointer receiver for large struct ( #26668 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-30 10:24:29 +08:00
6f89620a43
remove pull target rpc from lock ( #26054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-04 10:31:06 +08:00
a3e22786ed
Move meta store to kv catalog ( #25915 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-07-31 13:57:04 +08:00
7d00020c9e
Reduce DataScope to historical for segment release task ( #25489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-12 09:12:28 +08:00
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
4387f36897
make etcdKV private ( #24778 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-13 10:52:38 +08:00
675821c79d
refine log and process for distribution handler ( #24239 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-05-22 19:37:25 +08:00
b517bc9e6a
refine balance mechanism including:( #23454 ) ( #23763 ) ( #23791 )
...
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-05-04 12:22:40 +08:00
5244020336
ban auto balance channel ( #23725 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-26 19:26:39 +08:00
6653e2c3b0
fix balance channel ( #23631 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-25 10:22:37 +08:00
3933080511
skip to balance redundant segment ( #23490 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:32:32 +08:00
cbfe7a45ef
fix pull target ( #23491 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:30:32 +08:00
b5a73e6d1a
Add OWNERS files for querycoordv2 sub pkgs ( #23489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-18 15:52:30 +08:00
dbbd703667
fix balance generate unexpected task ( #23299 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-11 14:38:30 +08:00
9f127dae47
enable balance channel ( #23227 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-07 19:06:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
afd874b736
enhance segment balance by considering global rowCount(##22914) ( #23056 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-04-03 14:16:25 +08:00
127867b873
Add ratedgroup for some info/warning log ( #23095 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-31 15:22:23 +08:00
74da53c027
fix update load percentage ( #23054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-03-30 10:48:23 +08:00
93bc805933
Enhance ID allocator in DataNode ( #22905 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
1f718118e9
Dynamic load/release partitions ( #22655 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
21ba8182ee
Refine task create errors ( #22745 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-14 18:51:53 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
c3e8ad3629
fix balance generate reduce task ( #22236 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-21 19:06:27 +08:00
73c44d4b29
resource group impl ( #21609 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
5f3d3dc4fc
generate-mockery for querycoordv2, querynode and rootcoort ( #21714 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-16 13:59:42 +08:00
fb42466c65
Use opentelemetry ( #21509 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
6a29a964df
Fix queryCoord panic during query node down ( #21400 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-28 10:17:30 +08:00
d16b7c3c2d
Add more logs for debugging load balance ( #21389 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-27 15:57:30 +08:00
f8cff79804
Support the graceful stop for the query node ( #20851 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
2ecdb4ba4a
Etcd config source support TLS ( #20874 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
8283d32ac4
Only balance segement in targets ( #20709 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-21 16:21:11 +08:00
congqixia
wei liu
yah01
Xiaofan
yah01
SimFG
Enwei Jiao
Bingyi Sun
jaime
yiwangdr
MrPresent-Han
XuanYang-cn
yihao.dai
smellthemoon