d55d9d6e1d
fix: change pymilvus version for hybridsearch-groupby( #36407 ) ( #36451 )
...
related: #36407
---------
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 14:29:13 +08:00
df7ae08851
fix: iterator cursor progress too fast( #36179 ) ( #36180 )
...
related: #36179
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 11:45:13 +08:00
eb23e23cd2
enhance: refine parameter relationship for hybridsearch_group_by( #35096 ) ( #36289 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-20 14:55:11 +08:00
3b10085f61
enhance: Optimize workload based replica selection policy ( #36181 )
...
issue: #35859
This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.
if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.
if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-20 12:33:11 +08:00
363004fd44
enhance: simplify reduction on single search result ( #36334 )
...
See: #36122
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-20 11:59:10 +08:00
e84c4f2a92
fix: to fill default value when nullable and default value are both enable ( #36030 )
...
#36003
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-19 17:35:11 +08:00
dcd904d2fa
fix: keep inner topK to avoid exceeding efSearch ( #36284 )
...
issue: #https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-16 11:01:25 +08:00
e486442ca6
enhance: Disallow the keywords as a field name or dynamic field name ( #36101 )
...
issue: #35873
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:19:14 +08:00
b8b4aea4f5
enhance: restrict max group size( #33544 ) ( #36223 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-14 15:45:08 +08:00
d07338b4b9
enhance: Ignore index check for non-loaded vector field ( #36170 )
...
Related to #35996
For `Field Partial Load` feature, Milvus shall ignore index check for
non-loaded vector field.
Also, this PR unifies the logic of index check for load collection and
load partitions tasks.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 11:01:08 +08:00
2d05b7f219
fix: to add check in partition key field when set nullable==true ( #36218 )
...
#36213
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 10:53:08 +08:00
9d8d332c88
fix: Fix improper use of offset in HybridSearch ( #36244 )
...
issue :https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-13 22:05:15 +08:00
bd658a6510
enhance: Enable dynamic update replica selection policy ( #35860 )
...
issue: #35859
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 17:05:15 +08:00
06a706e5f0
fix: fix ReadWrite privilege group deny all global API ( #36144 )
...
issue: #35471
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 10:33:08 +08:00
5e5e87cc2f
enhance: rename some params and reduce default bitmapCardinalityLimit… ( #36138 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-12 12:09:08 +08:00
da227ff9a1
feat: Support create collection with functions ( #35973 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
e7ea1d7a04
enhance: improve log encoding performance on proxy nodes ( #36123 )
...
See #36122
This PR is designed to enhance log performance through two improvements:
1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-11 14:51:07 +08:00
29ddb10e8c
enhance: timer instead of ticker ( #36133 )
...
https://github.com/milvus-io/milvus/issues/36132
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-10 15:23:06 +08:00
89bf226f0b
feat: support keyword text match ( #35923 )
...
fix : #35922
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-10 15:11:08 +08:00
51870aef08
enhance: Use stats Handler to record request/response size metrics ( #36107 )
...
Related to #36102
This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
91d23ecbe1
fix: memory leak in proxy meta cache ( #36075 )
...
issue: #36074
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:33:05 +08:00
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
21b135c7c2
fix: not append valid data when transfer to insert record ( #36027 )
...
fix not append valid data when transfer to insert record and add a tiny
check when in groupBy field.
#35924
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-06 14:53:04 +08:00
8b043f58dc
fix: Fix DB limiter nodes are mistakenly cleaned up ( #35991 )
...
This issue only occurs for a short time right after a table is created.
To avoid this, we simply reduce the frequency of cleaning up invalid
limiter nodes.
issue: https://github.com/milvus-io/milvus/issues/35933
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-05 17:35:04 +08:00
32e55a02ea
fix: Fix privilege group hasn't been register for validate ( #35937 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 15:35:04 +08:00
66ed289a85
enhance: Fix typo of clustering key not loaded msg ( #35948 )
...
Related to #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 10:49:03 +08:00
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
325f1987d9
enhance: rewrite index params for compatibility ( #35788 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 19:41:03 +08:00
4641fd9195
enhance: make search groupby stop when reaching topk groups ( #35814 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-02 18:25:03 +08:00
9d80137698
fix: Check clustering key skip load behavior ( #35865 )
...
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 11:17:02 +08:00
311f860676
enhance: support to drop the role which is related the privilege list ( #35727 )
...
- issue: #35545
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 15:17:00 +08:00
3c8941a88e
enhance: Print log only when rate limit updates ( #35806 )
...
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 15:07:00 +08:00
985d84d3ee
enhance: Add skip load validation for create collection task ( #35737 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:05:08 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
56b3490a98
fix: Fix rate wasn't limited to the expected value ( #35699 )
...
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.
issue: https://github.com/milvus-io/milvus/issues/35675 ,
https://github.com/milvus-io/milvus/issues/35702
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-28 14:13:00 +08:00
bfd9d86fe9
feat: support groupby size on go-layer( #33544 ) ( #33845 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
3e1052f889
enhance: use the msg position obj when getting replicate channel position ( #35606 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 10:28:59 +08:00
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
9fff07936a
fix: Re-read value after `once` initialization ( #35642 )
...
Related to #35641
See also #35271
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 17:24:58 +08:00
3107701fe8
enhance: optimize retrieve on dynamic field ( #35580 )
...
#35514
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-22 14:24:56 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
bc2bbdc82f
enhance: Forbid access log print error message with line break ( #34491 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-20 17:36:55 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2( #35350 ) ( #35470 )
...
related: #35350
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:16:55 +08:00
e09dc3be58
enhance: Mark query node as read only after suspend ( #35492 )
...
issue: #34985 #35493
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 14:02:54 +08:00
14ef88f423
fix: Fix proxy panic after Import related api failed ( #35540 )
...
issue: #35539
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 15:22:55 +08:00
a570567644
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress ( #35472 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 14:18:54 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
fe20366b5c
enhance: Remove duplicated schema helper creation in proxy ( #35489 )
...
Related to PRs of #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-15 19:18:53 +08:00
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
Chun Han
wei liu
Ted Xu
smellthemoon
zhenshan.cao
cai.zhang
congqixia
zhagnlu
aoiasd
CharlesFeng
Jiquan Long
jaime
yihao.dai
SimFG
Zhen Ye