cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
e55d6506e3
enhance: Remove frequent observe log ( #39413 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-20 11:01:10 +08:00
657550cf06
fix: Fix slow dist handle and slow observe ( #38566 )
...
1. Provide partition&channel level indexing in the collection target.
2. Make `SegmentAction` not wait for distribution.
3. Remove scheduler and target manager mutex.
4. Optimize logging to reduce CPU overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:17:00 +08:00
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
d3c174b0f1
enhance: Accelerate observe collection ( #38028 )
...
1. A collection should observe the channel only once.
2. A collection should check the CollectionLoadPercent for updates only
once.
3. Skip saving coll/partition meta if there are no changes, primarily to
accelerate collection observation after recovery.
issue: https://github.com/milvus-io/milvus/issues/37630
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 14:14:45 +08:00
e76802f910
enhance: refine querycoord meta/catalog related interfaces to ensure that each method includes a ctx parameter ( #37916 )
...
issue: #35917
This PR refine the querycoord meta related interfaces to ensure that
each method includes a ctx parameter.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-25 11:14:34 +08:00
6d86b9022e
enhance: Provide secondary index critria when filter leaderview ( #37777 )
...
Related to #37630
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 10:12:30 +08:00
75676fbd11
fix: Fix dynamic release partition may fail search/query request ( #35919 )
...
issue: #33550
cause concurrent issue may occur between remove parition in target
manager and sync segment list to delegator. when it happens, some
segment may be released in delegator, and those segment may also be
synced to delegator, which cause delegator become unserviceable due to
lack of necessary segments, then search/query fails.
this PR make sure that all write access to target_manager will be
executed in serial to avoid the concurrent issues.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 18:47:03 +08:00
09ef3f1b4f
fix: Make sure querycoord observers started once ( #35811 )
...
Related to #35809
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 14:45:00 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection ( #34947 )
...
issue: #34946
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-02 17:20:15 +08:00
b284b81a47
fix: Check partition in current target when observing partition load status ( #34282 )
...
See also #34234
`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.
This PR check current target contains the partition to load when
advancing loading percentage to 100.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 17:40:07 +08:00
ecee7d90d4
enhance: try to speed up the loading of small collections ( #33570 )
...
- issue: #33569
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-07 08:25:53 +08:00
b044e5503e
enhance:Improve load speed ( #32898 )
...
fix #32897
add memory check when load collection
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-11 10:29:31 +08:00
4cdf6c3c41
fix: Check partition nil before observe load progress ( #32659 )
...
See also #32441 #32615
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-28 16:29:25 +08:00
d7ff1bbe5c
enhance: Make querycoordv2 collection observer task driven ( #32441 )
...
See also #32440
- Add loadTask in collection observer
- For load collection/partitions, load task shall timeout as a whole
- Change related constructor to load jobs
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-22 10:39:22 +08:00
c4806b69c4
enhance: Refactor leader view manager interface ( #31133 )
...
issue: #31091
This PR add GetByFilter interface in leader view manager, instead of all
kind of get func
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 15:13:36 +08:00
797847904c
enhance: Change some frequency log to rated level ( #29720 )
...
This PR change some frequency log to rated level
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 16:30:50 +08:00
e98c62abbb
enhance: refactor leader_observer to leader_checker ( #29454 )
...
issue: #29453
sync distribution by rpc will also call loadSegment/releaseSegment,
which may cause all kinds of concurrent case on same segment, such as
concurrent load and release on one segment.
This PR add leader_checker which generate load/release task to correct
the leader view, instead of calling sync distribution by rpc
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-05 15:54:55 +08:00
514e279f3a
enhance: Remove useless log in collection observer ( #29554 )
...
This PR removed the useless log in collection observer
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-28 17:16:47 +08:00
1b90630633
Fix the target updated before version updated to cause data missing ( #28250 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 11:36:22 +08:00
e0222b2ce3
refine target manager code style ( #27883 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 00:44:12 +08:00
93a877f55e
Make qcv2 target&leader observer execute in parallel ( #27844 )
...
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false
See also #27494
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-24 10:14:11 +08:00
42c475a0e0
remove useless log in querycoord ( #27362 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-11 10:13:34 +08:00
eca79d149c
Add ctx control for observer manual check methods ( #27531 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-09 11:07:33 +08:00
a8ce1b6686
Refine QueryCoord stopping ( #27371 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-27 16:27:27 +08:00
941a383019
Fix failed to load collection with more than 128 partitions ( #26763 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-02 00:09:01 +08:00
949c320185
remove pull target from qc recover ( #26775 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-01 11:17:01 +08:00
6f89620a43
remove pull target rpc from lock ( #26054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-04 10:31:06 +08:00
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation ( #25812 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
2bed5bc461
Fix false load success for partition load timeout but collection remains ( #25379 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-06 19:10:25 +08:00
68ae199a9f
load segment with target version, avoid read redundant segment ( #24929 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 11:48:45 +08:00
ed81eaa963
Make CollectionObserver trigger checker more frequently during load procedure ( #23928 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-08 14:06:41 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
74da53c027
fix update load percentage ( #23054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-03-30 10:48:23 +08:00
1f718118e9
Dynamic load/release partitions ( #22655 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
4005508880
Fix load collection blocks when memory is limited. ( #21990 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-02-06 16:07:53 +08:00
b0b0929c6c
Change CreateAt to UpdatedAt when checking timeout ( #21769 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-01-18 17:29:43 +08:00
defb7660c8
Add refresh option for LoadCollection/LoadPartitions interfaces ( #21776 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-18 16:41:44 +08:00
b91bb5a729
Fix load timeout after next target updates ( #21759 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-01-17 15:15:46 +08:00
c8f89907b6
Fix current target may be updated to an invalid target ( #21742 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-17 11:41:51 +08:00
b04eb17088
Fix load timeout ( #21156 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-12-12 18:43:23 +08:00
27219ce9c0
Fix load progress bug ( #21069 )
...
issue: #21023
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-12-08 18:41:20 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
0297ab1a46
Make progress if any channel/segment was loaded on node ( #20775 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-06 18:29:19 +08:00
c49f20ea94
Load should base on the accurate loaded number ( #20999 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-12-06 17:13:18 +08:00
d37ebe538b
Record load latency for loaded collection ( #20538 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-15 10:11:07 +08:00
c5cd92d36e
update target ( #19296 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
c61a4d1243
Fix panic due to double stop ( #19849 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 12:13:28 +08:00
1c71844b8d
Add license header ( #19678 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:39:22 +08:00
congqixia
yihao.dai
Zhen Ye
tinswzy
wei liu
jaime
SimFG
Xiaofan
yah01
Bingyi Sun
Ten Thousand Leaves
yah01
smellthemoon
Enwei Jiao