Commit Graph

46 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
congqixia cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Zhen Ye bb8d1ab3bf
enhance: make new go package to manage proto (#39114)
issue: #39095

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
Zhen Ye 07c1f43d95
fix: the retrieve plan on heap is used after free when reduce (#38840)
issue: #38731

Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 17:50:51 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
Gao 8977454311
enhance: support recall estimation (#38017)
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 20:40:48 +08:00
Zhen Ye c6dcef7b84
enhance: move segcore codes of segment into one package (#37722)
issue: #33285

- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-29 10:22:36 +08:00
Gao 1d61b604e1
enhance: support retry search when topk is reduced and result not enough (#35645)
issue: #35576 

This pr is to cover those cases when queryHook optimize search params
and make the result size insufficient, add retry search mechanism and
add related metrics for alarming.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-23 19:19:30 +08:00
Chun Han df7ae08851
fix: iterator cursor progress too fast(#36179) (#36180)
related: #36179

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 11:45:13 +08:00
congqixia 3bc7d63be9
fix: overwrite correct selection when pk duplicated (#35826)
Related to #35505

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 10:27:08 +08:00
Chun Han 9d0aa5c202
fix: empty result when having only one subReq(#36098) (#36128)
related: #36098

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-10 14:25:07 +08:00
Chun Han e480b103bd
feat: supporing hybrid search group_by (#35982)
related: #35096

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
Chun Han bfd9d86fe9
feat: support groupby size on go-layer(#33544) (#33845)
related: #33544

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
congqixia f7f9a729c9
enhance: Pre-allocate space for reduce data structure (#35118)
Grow slice & map.growWork may cause a lot when segment number is large
for big K query. This PR pre-allocate space for reduce methods to avoid
this cost.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-31 10:35:49 +08:00
wei liu c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
wei liu b49862d4f3
enhance: Optimize grow slice cost during query (#34253)
issue: #32252 

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
congqixia 3fdaae8792
fix: Return record with largest timestamp for entires with same PK (#33936)
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
Chun Han 416a2cf507
fix: query iterator lack results(#33137) (#33422)
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
Jiquan Long dd9919a7dc
fix: two-phase retrieval on lru-segment (#32945)
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-15 17:53:34 +08:00
Jiquan Long 1f58cda957
enhance: add more trace for search & query (#32734)
issue: https://github.com/milvus-io/milvus/issues/32728

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 13:03:29 +08:00
Bingyi Sun fecd9c21ba
feat: LRU cache implementation (#32567)
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
Jiquan Long c002745902
enhance: retrieve output fields after local reduce (#32346)
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-25 09:49:26 +08:00
Xiaofan dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly (#32089)
fix https://github.com/milvus-io/milvus/issues/32059

this pr fix two issues:

offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 16:01:18 +08:00
SimFG 90bed1caf9
enhance: add the related data size for the read apis (#31816)
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
zhenshan.cao 089c805e0a
enhance:Refactor hybrid search (#32020)
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
SimFG b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
PowderLi 8fc4ebfa11
fix: empty MetricType (#30216)
issue: #30102 #30225

we should read MetricType from SearchResult,
because query node never
1. read metricType from LoadMeta
2. store to collection
3. set SearchRequest.MetricType

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 15:33:02 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749)
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
MrPresent-Han 9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983)
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-05 15:50:47 +08:00
MrPresent-Han bd3bde82f0
fix iterator lose data for duplicted result(#29406) (#29451)
related: #29406

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-25 23:28:47 +08:00
MrPresent-Han f93ad6471c
enable query reduce on querynodes for shortcut return(#27306) (#27837)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-10-30 15:08:11 +08:00
yah01 be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 11:39:20 +08:00
xige-16 488b423e1b
Fix high cpu usage caused by proto.size (#27054)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-09-17 20:55:21 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-06 17:43:14 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
Enwei Jiao 7d61355ab0
Refactor log for Query (#26310)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-14 18:57:32 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-10 14:11:15 +08:00
wei liu f33a89387f
enable config shard level cost metrics whether contains worker's cost (#26132)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-09 11:37:15 +08:00
wei liu 302897f866
refine look aside balance logic (#25837)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
Jiquan Long 8d193a3e56
Skip unnecessary reduce phase during search (#25166)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-06-27 19:22:45 +08:00
wei liu e40d95e093
add search/query request cost metrics on qn (#24413)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-15 14:24:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
Jiquan Long ad7f3d4c3e
Support Count(*) on querynodev2 (#23321)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-04-11 15:40:31 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
yah01 081572d31c
Refactor QueryNode (#21625)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00