c22a2cebb6
fix: split stream query result to avoid grpc response too large error ( #36090 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-13 15:07:09 +08:00
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
416a2cf507
fix: query iterator lack results( #33137 ) ( #33422 )
...
related: #33137
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
59a7a46904
enhance: Merge query stream result for reduce delete task ( #32855 )
...
relate: https://github.com/milvus-io/milvus/issues/32854
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-27 18:15:43 +08:00
dd9919a7dc
fix: two-phase retrieval on lru-segment ( #32945 )
...
issue: #31822
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-15 17:53:34 +08:00
1d48d0aeb2
enhance: use different value to get related data size according to segment type ( #33017 )
...
issue: #30436
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-14 14:59:33 +08:00
1c84a1c9b6
fix: lru related issue fixup patch ( #32916 )
...
issue: #32206 , #32801
- search failure with some assertion, segment not loaded and resource
insufficient.
- segment leak when query segments
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-10 19:17:30 +08:00
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
c002745902
enhance: retrieve output fields after local reduce ( #32346 )
...
issue: #31822
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-25 09:49:26 +08:00
90bed1caf9
enhance: add the related data size for the read apis ( #31816 )
...
issue: #30436
origin pr: #30438
related pr: #31772
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
1f7f3993a1
fix: Validate PlaceholderGroups before combine them ( #32016 )
...
See also #32015
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-09 11:33:17 +08:00
b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
c6665c2a4c
test: support multiple data/querynodes in integration test ( #30618 )
...
issue: https://github.com/milvus-io/milvus/issues/29507
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00
8fc4ebfa11
fix: empty MetricType ( #30216 )
...
issue: #30102 #30225
we should read MetricType from SearchResult,
because query node never
1. read metricType from LoadMeta
2. store to collection
3. set SearchRequest.MetricType
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 15:33:02 +08:00
f2c0ead51a
enhance: Add trace span for scheduling read tasks in QueryNode ( #30265 )
...
This PR adds a trace span for search/query task scheduling duration
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 16:59:00 +08:00
0d4e781f69
fix: the system rejects all queries and never recovers if enabled read rate limit ( #30061 )
...
fix #30060
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 23:30:55 +08:00
60e88fb833
fix: Restore the MVCC functionality. ( #29749 )
...
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:
1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.
issue: #29656
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
9e0163e12f
enhance: use GPU pool for gpu tasks ( #29678 )
...
- this much improve the performance for GPU index
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-04 17:50:46 +08:00
b251c3a682
enhance: add ctx for HandleCStatus and callers ( #29517 )
...
See also #29516
Make `HandleCStatus` print trace id for better logging
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-27 16:10:47 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
6ff4ffa6c2
Fix scheduler may panic during close ( #27431 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 16:53:26 +08:00
258e1ccd66
Refine querynode scheduler lifetime ( #26915 )
...
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:21:26 +08:00
6539a5ae2c
Refine DataCoord status ( #27262 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
33e3e78937
add metrics for query reduce ( #27201 )
...
/kind improvement
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-19 14:53:22 +08:00
e107d0794c
support complex delete expression ( #25752 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
5dae6a654a
Protect segment from being released while query/search ( #26322 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 18:38:17 +08:00
ca1349708b
Remove time travel ralted testcase ( #26119 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-10 18:53:17 +08:00
f33a89387f
enable config shard level cost metrics whether contains worker's cost ( #26132 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-09 11:37:15 +08:00
302897f866
refine look aside balance logic ( #25837 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
bcd3abf644
Separate segments/shards reduce latency metrics ( #25475 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-11 18:18:28 +08:00
dc1cb54404
[Fixup] unexpected read_task_ready_len metric ( #25349 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-07-06 20:02:29 +08:00
d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy ( #24839 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-07-03 18:24:25 +08:00
f7ecbff986
Make merged task channel buffered ( #25242 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-06-30 14:58:22 +08:00
9b91519c88
fix search cost metrics on replica selection ( #25059 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 19:02:44 +08:00
9250423288
Fix QueryNode metrics ( #24916 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-16 16:02:39 +08:00
75adf1df98
fix group nq metrics ( #24918 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-15 19:50:39 +08:00
e40d95e093
add search/query request cost metrics on qn ( #24413 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-15 14:24:38 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
44e5daae3a
Fix querynode read quota ( #24412 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-06-08 17:24:36 +08:00
31880ab427
Use conc.PoolOption instead of ants.Option ( #24585 )
...
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-01 15:52:31 +08:00
1c64d0618c
Revert "Consume threads of the number of nq ( #24410 )" ( #24554 )
...
This reverts commit 71a7fef5c5
2023-05-31 16:41:33 +08:00
71a7fef5c5
Consume threads of the number of nq ( #24410 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-26 17:33:27 +08:00
eab94489ba
Refine the merge algorithm ( #23767 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-27 18:26:35 +08:00
f0ababb420
Use merr in querynodev2 ( #23652 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-04-26 17:06:34 +08:00
f8ff97fe29
Fix travel timestamp set unique leads to disable task merge ( #23605 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-25 19:04:34 +08:00
4fe363c4b2
Add KNN cgo pool ( #23526 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-19 23:56:31 +08:00
aoiasd
jaime
wei liu
Chun Han
Jiquan Long
SimFG
chyezh
Bingyi Sun
congqixia
yiwangdr
PowderLi
yah01
zhenshan.cao
yah01
Enwei Jiao
chyezh
smellthemoon