efcaa07fe9
Implement Stringer for task related enums ( #25860 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 13:57:00 +08:00
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map ( #25846 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
1045c88102
Support replace indexed field in QueryCoord ( #25747 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-19 21:22:58 +08:00
224515eaa3
Add segment dist containing condition for loading segment ( #25736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-19 15:02:58 +08:00
228c0df3f7
Skip record ChannelNotFound error while loading segment ( #25669 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-18 11:53:18 +08:00
49655d2f13
fix panic due to lack of indexInfo on querynode( #25362 ) ( #25613 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-17 19:23:19 +08:00
948d1f1f4a
Handle errors by merr for QueryCoord ( #24926 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-17 14:59:34 +08:00
b62c82af22
fix set target version in loading sealed segment ( #25603 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-14 20:00:31 +08:00
c56a26a556
Add SetMetricType and set metric only for shard leader ( #25285 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-04 10:20:25 +08:00
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
c7dc1c067a
Assign metric type in load segment request ( #24917 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-16 16:50:41 +08:00
51404abe93
Add missing indexInfo in watchDmChannelsRequest ( #24696 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-06-14 14:40:39 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
5f7099a9bd
Fix search failed due to metric type mismatch ( #24691 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-06 21:40:35 +08:00
b0ce4cff83
Assign metric type in search request when user has not explicitly assigned ( #24591 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-01 18:10:32 +08:00
b09e7aeaf7
support detailed task metrics( #23414 ) ( #24507 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-05-30 14:59:28 +08:00
1a3dca9b5e
Fix dynamic partitions loading ( #24112 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-18 09:17:23 +08:00
7e0d1492c7
Load delete from channel checkpoint ( #23961 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 19:10:41 +08:00
b517bc9e6a
refine balance mechanism including:( #23454 ) ( #23763 ) ( #23791 )
...
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-05-04 12:22:40 +08:00
6f4ed517de
add growing segment index ( #23615 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
cbfe7a45ef
fix pull target ( #23491 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:30:32 +08:00
b5a73e6d1a
Add OWNERS files for querycoordv2 sub pkgs ( #23489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-18 15:52:30 +08:00
dbbd703667
fix balance generate unexpected task ( #23299 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-11 14:38:30 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
afd874b736
enhance segment balance by considering global rowCount(##22914) ( #23056 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-04-03 14:16:25 +08:00
ae0f467c02
Fix segment/channel may be re-loaded/subscribed ( #22969 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-27 18:28:00 +08:00
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
93bc805933
Enhance ID allocator in DataNode ( #22905 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
1f718118e9
Dynamic load/release partitions ( #22655 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
3d8f0156c7
Refine scheduler & executor of QueryCoord ( #22761 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-16 17:43:55 +08:00
21ba8182ee
Refine task create errors ( #22745 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-14 18:51:53 +08:00
1a4732bb19
Use new errors to handle load failures cache ( #22672 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-10 17:15:54 +08:00
d4fccdd135
Add reason for spawning task ( #22549 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-06 10:13:50 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
73c44d4b29
resource group impl ( #21609 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
5f3d3dc4fc
generate-mockery for querycoordv2, querynode and rootcoort ( #21714 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-16 13:59:42 +08:00
2146af1fb2
Return insufficient memory error when load failed ( #21574 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-01-10 20:35:39 +08:00
d16b7c3c2d
Add more logs for debugging load balance ( #21389 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-27 15:57:30 +08:00
396a85c926
Fix the number of executing tasks may break the limit ( #21318 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-20 20:33:27 +08:00
fc10c74005
Use channel-cp as seekPosition when FromDmlCPLoadDelete ( #21110 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:17:22 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
f8cff79804
Support the graceful stop for the query node ( #20851 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
ddd29ea6ab
Optimize scheduler, increase merge tasks probability ( #20922 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-01 16:25:16 +08:00
2ecdb4ba4a
Etcd config source support TLS ( #20874 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
060649b8aa
Refactor task scheduler and executor ( #20828 )
...
Make the performance able to scale out
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-30 13:57:15 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
cc371d6801
Task/Action won't finish util the RPC returned ( #20669 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-17 17:55:09 +08:00
1177ed1a33
fix duplicate watch channel task ( #20667 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-17 15:05:08 +08:00
2d806788fb
Add version for subscribed channel ( #20585 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-15 13:21:07 +08:00
a82c235599
Scheduler should returns err if the queue is full ( #20465 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-11 17:55:05 +08:00
eaa5cfdcb5
fix leader observer sync logic ( #20478 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-11 11:43:06 +08:00
a68cec85af
Refine QueryCoordV2 metrics ( #20461 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-10 15:01:04 +08:00
f925fa7661
fix check segment stale task ( #20401 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-08 19:43:05 +08:00
c5cd92d36e
update target ( #19296 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
4112c667d0
Refine task's priority ( #20364 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-07 14:53:02 +08:00
c08f11420f
Fix Release tasks may end with incorrect status ( #20263 )
...
If the resource has been removed, the release tasks should end with
Succeeded status, not stale
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-03 15:01:35 +08:00
4f6b87311e
Refine request merger ( #20189 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-01 14:27:34 +08:00
0b939c5735
Check whether ubsubscribe/release task is stale ( #20165 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-01 12:33:35 +08:00
4412cfcaaf
reduce querycoord unnecessary panic ( #19925 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-10-28 17:15:32 +08:00
581e1d8c35
Fill MsgBase ALL ( #19912 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
7cf26011e6
Add more logs for checkpoint of channel ( #19885 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 13:19:27 +08:00
c61a4d1243
Fix panic due to double stop ( #19849 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 12:13:28 +08:00
b0404cbbab
Remove dropped segments' info ( #19839 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-17 19:41:26 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
146a3342c3
Use task's ID as the MsgID instead of source ID ( #19751 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-13 15:21:23 +08:00
1c71844b8d
Add license header ( #19678 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:39:22 +08:00
e6c7286bee
Remove task timeout ( #19583 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-30 19:48:55 +08:00
2e9a08aecf
Use segment's position as the delta position ( #19577 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-30 17:20:55 +08:00
9dcee37e1c
Skip canceled tasks ( #19513 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-28 19:58:54 +08:00
377f856833
Fix balance may confuse leader observer ( #19435 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-28 12:10:54 +08:00
dff4d1042e
Fix can't release growing segment ( #19418 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-26 14:08:52 +08:00
6ba52366c5
Fix load segments can't be retried ( #19414 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-26 10:54:52 +08:00
29d2c0bd7b
Move task merge cap to config file ( #19387 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-09-24 17:56:51 +08:00
2cfacbba8d
Fix leader observer confused with replicas ( #19392 )
...
With 2 replicas, the same segment was loaded on 2 nodes in different
replicas, leader observer will sync the latest one (by version) to the
other one.
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-23 15:16:51 +08:00
428840178c
Support diskann index for vector field ( #19093 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
97aa2bd84f
Merge load segment tasks to improve performance ( #19234 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-21 14:42:51 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
626854cf0c
Refactor QueryCoord ( #18836 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-15 18:48:32 +08:00
congqixia
yah01
MrPresent-Han
wei liu
yihao.dai
jaime
Gao
foxspy
XuanYang-cn
Enwei Jiao
Xiaofan
SimFG
smellthemoon
Bingyi Sun
xige-16