7b91fa3db8
fix: Make leader checker generate leader task instead of segment task ( #30258 )
...
See also #30150
For leader view distribution with offline nodes, a release task can
never be sent to querynode due to targetNode online check logic. Even
the request is dispatched, normal release task does not have "force"
flag when calling `delegator.ReleaseSegment`.
This PR adds a new type of querycoord task: LeaderTask, the
responsibility of which is to rectify leader view distribtion.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-21 11:08:51 +08:00
f69f65ff68
fix: Leader checker can't remove segment from leader view ( #30151 )
...
issue: #30150
This PR fix three problems:
1. leader checker use wrong node id when generate release task, which
cause the release task finished immediately
2. the release request generated by leader_checker doesn't set the
`force` flag, the operation to clean leader view on delegator will fail.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-20 18:58:58 +08:00
c68c128e47
fix: level 0 segments not loaded ( #29908 )
...
the recent changes move the level 0 segments list to a new proto field,
which leads to the QueryCoord can't see the level 0 segments, handle the
new changes
fix #29907
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 14:40:53 +08:00
9702cef2b5
feat: Support multiple vector search ( #29433 )
...
issue #25639
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
a3cb8e2625
fix: Add atomic method to get collection target ( #29577 )
...
Related to #29575
Add `getCollectionTarget` method which is atomic when scope is
`CurrentTargetFirst` or `NextTargetFirst`
Also return error when executor finds no channel in target manager
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-29 09:04:46 +08:00
1d6bcd1ded
enhance: speed up loading with many deletions ( #29455 )
...
the executor always fetches the latest segment info, so we could consume
from the latest checkpoint, which could save much time while deleted
many entities
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-25 20:58:45 +08:00
a0e1a1eb31
feat: support enable/disable mmap for index ( #29005 )
...
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
0a87724f18
enhance: remove merger for load segments ( #29062 )
...
remove merger as now QueryNode could load segments concurrently
fix #29063
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 10:48:37 +08:00
fab52d167b
fix: may miss stream delta while loading ( #28871 )
...
we consume the delta data from the lastest channel checkpoint while
loading segment,
this works well without level 0 segments, but now it may lead to miss
some delta data,
so we have to consume from the current target's channel checkpoint
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-05 17:34:45 +08:00
ece592a42f
Deliver L0 segments delete records ( #27722 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
dc89730a50
Support collection-level mmap control ( #26901 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
852be152de
Change task sourceID to stringer interface ( #27965 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-27 01:08:12 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
168e82ee10
Fix panic while handling with the nil status ( #27040 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-15 10:09:21 +08:00
edde3cf1c7
Add tracer for querycoord tasks ( #27058 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-14 09:59:19 +08:00
54c0e64059
Fix search on empty segments set bug ( #26136 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-08-08 11:17:08 +08:00
2180ef180c
Record only failed task error ( #26033 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-01 10:11:05 +08:00
2b9ec565bb
Fix task executor can not dedup same task ( #25901 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-26 08:51:02 +08:00
a669440ee9
Allow querycoord executor load sealed segment with no index ( #25902 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 18:47:01 +08:00
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map ( #25846 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
1045c88102
Support replace indexed field in QueryCoord ( #25747 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-19 21:22:58 +08:00
49655d2f13
fix panic due to lack of indexInfo on querynode( #25362 ) ( #25613 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-17 19:23:19 +08:00
b62c82af22
fix set target version in loading sealed segment ( #25603 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-14 20:00:31 +08:00
c56a26a556
Add SetMetricType and set metric only for shard leader ( #25285 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-04 10:20:25 +08:00
c7dc1c067a
Assign metric type in load segment request ( #24917 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-16 16:50:41 +08:00
51404abe93
Add missing indexInfo in watchDmChannelsRequest ( #24696 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-06-14 14:40:39 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
5f7099a9bd
Fix search failed due to metric type mismatch ( #24691 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-06 21:40:35 +08:00
b0ce4cff83
Assign metric type in search request when user has not explicitly assigned ( #24591 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-06-01 18:10:32 +08:00
7e0d1492c7
Load delete from channel checkpoint ( #23961 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 19:10:41 +08:00
b517bc9e6a
refine balance mechanism including:( #23454 ) ( #23763 ) ( #23791 )
...
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-05-04 12:22:40 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
1f718118e9
Dynamic load/release partitions ( #22655 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
3d8f0156c7
Refine scheduler & executor of QueryCoord ( #22761 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-16 17:43:55 +08:00
1a4732bb19
Use new errors to handle load failures cache ( #22672 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-10 17:15:54 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
2146af1fb2
Return insufficient memory error when load failed ( #21574 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-01-10 20:35:39 +08:00
d16b7c3c2d
Add more logs for debugging load balance ( #21389 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-27 15:57:30 +08:00
396a85c926
Fix the number of executing tasks may break the limit ( #21318 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-20 20:33:27 +08:00
fc10c74005
Use channel-cp as seekPosition when FromDmlCPLoadDelete ( #21110 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:17:22 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
060649b8aa
Refactor task scheduler and executor ( #20828 )
...
Make the performance able to scale out
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-30 13:57:15 +08:00
cc371d6801
Task/Action won't finish util the RPC returned ( #20669 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-17 17:55:09 +08:00
c5cd92d36e
update target ( #19296 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
4f6b87311e
Refine request merger ( #20189 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-01 14:27:34 +08:00
7cf26011e6
Add more logs for checkpoint of channel ( #19885 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 13:19:27 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
1c71844b8d
Add license header ( #19678 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:39:22 +08:00
e6c7286bee
Remove task timeout ( #19583 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-30 19:48:55 +08:00
2e9a08aecf
Use segment's position as the delta position ( #19577 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-30 17:20:55 +08:00
congqixia
wei liu
yah01
xige-16
yah01
SimFG
Bingyi Sun
MrPresent-Han
yihao.dai
Gao
jaime
Enwei Jiao
Xiaofan