a2502bde75
enhance: replica manager enhancement ( #31496 )
...
issue: #30647
- ReplicaManager manage read only node now, and always do persistent of
node distribution of replica.
- All segment/channel checker using ReplicaManager to get read-only node
or read-write node, but not ResourceManager.
- ReplicaManager promise that only apply unique querynode to one replica
in same collection now (replicas in same collection never hold same
querynode at same time).
- ReplicaManager promise that fairly node count assignment policy if
multi replicas of collection is assigned to one resource group.
- Move some parameters check into ReplicaManager to avoid data race.
- Allow transfer replica to resource group that already load replica of
same collection
- Allow transfer node between resource groups that load replica of same
collection
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-05 04:57:16 +08:00
0944a1f790
enhance: Refactor channel dist manager interface ( #31119 )
...
issue: #31091
This PR add GetByFilter interface in channel dist manager, instead of
all kind of get func
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-02 10:23:14 +08:00
ff4237bb90
enhance: add hostname into node info ( #30673 )
...
issue: https://github.com/milvus-io/milvus/issues/30647
- Address may be reused in k8s environment. Using hostname can be
better.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-15 10:45:06 +08:00
efe8cecc88
enhance: refactor segment dist manager interface ( #31073 )
...
issue: #31091
This PR add `GetByFilter` interface in segment dist manager, instead of
all kind of get func
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 16:29:01 +08:00
c886aa29ff
enhance: Use `ListIndexes` instead of `DescribeIndex` for qc broker ( #31122 )
...
See also #31103
Since querycoord need index meta information from datacoord only, broker
shall use `ListIndexes` to skip segment index building check logic in
datacoord
This PR is also related to #30538 , in which DescribeIndex caused lots of
memory usage and lead to OOM eventually
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 21:43:03 +08:00
c4ddfff2a7
enhance: make Load process traceable in querycoord ( #29806 )
...
See also #29803
This PR:
- Add trace span for collection/partition load
- Use TraceSpan to generate Segment/ChannelTasks when loading
- Refine BaseTask trace tag usage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 09:58:49 +08:00
1b90630633
Fix the target updated before version updated to cause data missing ( #28250 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 11:36:22 +08:00
dc89730a50
Support collection-level mmap control ( #26901 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
e0222b2ce3
refine target manager code style ( #27883 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 00:44:12 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
a8ce1b6686
Refine QueryCoord stopping ( #27371 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-27 16:27:27 +08:00
24354b166c
Fix unit test failed when run single test ( #27348 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 09:23:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
941a383019
Fix failed to load collection with more than 128 partitions ( #26763 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-02 00:09:01 +08:00
7d61355ab0
Refactor log for Query ( #26310 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-14 18:57:32 +08:00
6f89620a43
remove pull target rpc from lock ( #26054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-04 10:31:06 +08:00
a3e22786ed
Move meta store to kv catalog ( #25915 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-07-31 13:57:04 +08:00
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation ( #25812 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
172db82b23
Release collection if all partition released ( #25706 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-19 16:54:58 +08:00
948d1f1f4a
Handle errors by merr for QueryCoord ( #24926 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-17 14:59:34 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
39d31f8bbf
Trigger checker while waiting collection/partition released ( #24523 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-30 17:41:28 +08:00
d8ad09b710
Fix sync partition context deadline exceeded ( #24479 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-29 20:55:28 +08:00
1a3dca9b5e
Fix dynamic partitions loading ( #24112 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-18 09:17:23 +08:00
a53beba14f
Move release collection metrics to job ( #24079 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-05-17 11:17:22 +08:00
30415e1b83
Fix metric QueryCoordNumCollections ( #24053 ) ( #24107 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-05-15 16:33:22 +08:00
4b26c0afb3
Add collection in querynode when sync new partitions ( #23825 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-01 16:36:38 +08:00
6b29a991a5
Fix releasing non-loaded partition blocks forever ( #23623 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-23 19:16:32 +08:00
cbfe7a45ef
fix pull target ( #23491 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:30:32 +08:00
b5a73e6d1a
Add OWNERS files for querycoordv2 sub pkgs ( #23489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-18 15:52:30 +08:00
bed8d6892e
Protect the QueryCoord meta from stale data migrated from old version ( #23412 )
...
Fix #23411
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-14 14:42:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
1f718118e9
Dynamic load/release partitions ( #22655 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
e768437681
Correct usage of Timer and Ticker ( #22228 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-02-23 18:59:45 +08:00
87a4ddc7e2
fix rg e2e ( #22187 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-16 10:48:34 +08:00
73c44d4b29
resource group impl ( #21609 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
837e3162d7
Fix ready notifiers leak when collection released ( #21712 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-14 21:55:41 +08:00
c66ce4aeba
Add warning comments ( #21697 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-13 16:43:41 +08:00
2146af1fb2
Return insufficient memory error when load failed ( #21574 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-01-10 20:35:39 +08:00
2ecdb4ba4a
Etcd config source support TLS ( #20874 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
d37ebe538b
Record load latency for loaded collection ( #20538 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-15 10:11:07 +08:00
7325b3e1c3
Substitute traceid for msgid in rpc ( #20450 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-14 15:29:06 +08:00
c71c6378ff
Clear stale replicas ( #20456 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-10 17:01:03 +08:00
a68cec85af
Refine QueryCoordV2 metrics ( #20461 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-10 15:01:04 +08:00
6e9820441f
Close the no-use chan in the querycoord node ( #20385 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-08 15:55:02 +08:00
c5cd92d36e
update target ( #19296 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
d7ebb25701
skip handoff event on unloaded partition ( #20306 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 17:05:06 +08:00
chyezh
wei liu
congqixia
yah01
SimFG
Enwei Jiao
Bingyi Sun
yihao.dai
yah01
Jiquan Long
jaime
zhenshan.cao
smellthemoon