Commit Graph

100 Commits (a2f0092e39d70efcef57b5e1e0dddd032cf7bcc7)

Author SHA1 Message Date
Chun Han 903450f5c6
enhance: add ts support for iterator(#22718) (#36572)
related: #22718

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-10-16 18:51:23 +08:00
Chun Han df7ae08851
fix: iterator cursor progress too fast(#36179) (#36180)
related: #36179

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 11:45:13 +08:00
Ted Xu 363004fd44
enhance: simplify reduction on single search result (#36334)
See: #36122

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-20 11:59:10 +08:00
zhagnlu 3107701fe8
enhance: optimize retrieve on dynamic field (#35580)
#35514

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-22 14:24:56 +08:00
wei liu c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
congqixia e2f40fc2a8
fix: Check legacy guarantee ts when skipping alloc ts (#34981)
See also #34980

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 10:17:45 +08:00
Jiquan Long a2ac84bd64
feat: record the duration waiting in the proxy queue (#34744)
fix: https://github.com/milvus-io/milvus/issues/34743

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-23 14:23:52 +08:00
wei liu b49862d4f3
enhance: Optimize grow slice cost during query (#34253)
issue: #32252 

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
aoiasd 186757e622
enhance: support mark error as user error (#33498)
relate: https://github.com/milvus-io/milvus/issues/33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
Chun Han 416a2cf507
fix: query iterator lack results(#33137) (#33422)
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
Jiquan Long 9f81290c63
fix: try best to get enough query results (#33178)
issue: https://github.com/milvus-io/milvus/issues/33137

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-21 11:57:51 +08:00
SimFG 8594b55ad5
enhance: add `max insert request size` and `must use partition key` configs (#32433)
issue: https://github.com/milvus-io/milvus/issues/30577
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:31:20 +08:00
Xiaofan dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly (#32089)
fix https://github.com/milvus-io/milvus/issues/32059

this pr fix two issues:

offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 16:01:18 +08:00
SimFG 90bed1caf9
enhance: add the related data size for the read apis (#31816)
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
Cai Yudong 00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
SimFG b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
Chun Han c3264ca3e3
feat: support segment pruner (#31003)
related: #30376
2024-03-22 13:57:06 +08:00
congqixia 9b3005f1be
enhance: Avoid create schema helper for each read task (#30981)
See also #30806

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-04 19:39:00 +08:00
cai.zhang 40ca98f57f
enhance: Skip timestamp allocation when search/query consistency level is eventually (#29773)
issue: #29772 
1. Skip timestamp allocation when search/query consistency level is
eventually.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-21 09:52:59 +08:00
aoiasd c863b82476
enhance: Return parse expression failed error with reason (#30548)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-20 16:02:52 +08:00
congqixia 8e8ac213aa
enhance: Utilize partition key optimization in reQuery (#30253)
See also #30250

This PR add requery flag in query task. When reQuery flag is true, query
task shall skip partition name conversion and use pre-calculated
partitionIDs passed from search task.

TODO: hybrid search does not have partition id information. we shall
apply same logic for hybrid search later.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 11:05:07 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427)
issue:https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749)
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
congqixia 4f8c540c77
enhance: cache collection schema attributes to reduce proxy cpu (#29668)
See also #29113

The collection schema is crucial when performing search/query but some
of the information is calculated for every request.

This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:28:46 +08:00
MrPresent-Han bd3bde82f0
fix iterator lose data for duplicted result(#29406) (#29451)
related: #29406

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-25 23:28:47 +08:00
yah01 bf633bb5d7
enhance: refine the retry error (#28573)
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-30 18:34:32 +08:00
Jiquan Long 6913283716
Fill output fields for count request (#27967)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-10-30 10:00:12 +08:00
yihao.dai f9c630247d
Construct plan directly when search with vector output (#27928)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-26 19:30:10 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 11:39:20 +08:00
xige-16 488b423e1b
Fix high cpu usage caused by proto.size (#27054)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-09-17 20:55:21 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
xige-16 1e5836221a
Fix CollectionNotExists when search and retrieve vector (#26524)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-22 17:06:22 +08:00
Enwei Jiao ca1349708b
Remove time travel ralted testcase (#26119)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-10 18:53:17 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-10 14:11:15 +08:00
Jiquan Long 6578a27fe7
Support empty expression (#26153)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-10 08:37:15 +08:00
Jiquan Long 5c1f79dc54
Push down the limit operator to segcore (#25959)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-01 20:29:05 +08:00
smellthemoon 9614e61f14
Fix collection and channel not match (#25859)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-08-01 17:33:06 +08:00
zhenshan.cao c4f009073c
Make query set guarantee ts based on default consistency level (#25578)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-07-17 09:58:38 +08:00
SimFG 69d274d233
Improve the operation log (#25589)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-07-14 16:08:31 +08:00
chyezh d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy (#24839)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-07-03 18:24:25 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
chyezh ccf3f0066f
[Pick] Enable max result window limit (#24986)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-06-25 14:42:43 +08:00
Enwei Jiao d143682d7d
Refactor logs in proxy package. (#24936)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-19 13:28:41 +08:00
wei liu 46f7d903a3
enable look aside balancer on replica selection (#24791)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-16 18:38:39 +08:00
wei liu a3437e0ab5
refactor replica selection and retry policy on channel (#24367)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-13 10:20:37 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
chyezh 1593278f9d
add independent nq limit option (#24632)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-06-07 10:38:36 +08:00