3c8941a88e
enhance: Print log only when rate limit updates ( #35806 )
...
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 15:07:00 +08:00
985d84d3ee
enhance: Add skip load validation for create collection task ( #35737 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:05:08 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
56b3490a98
fix: Fix rate wasn't limited to the expected value ( #35699 )
...
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.
issue: https://github.com/milvus-io/milvus/issues/35675 ,
https://github.com/milvus-io/milvus/issues/35702
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-28 14:13:00 +08:00
bfd9d86fe9
feat: support groupby size on go-layer( #33544 ) ( #33845 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
3e1052f889
enhance: use the msg position obj when getting replicate channel position ( #35606 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 10:28:59 +08:00
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
9fff07936a
fix: Re-read value after `once` initialization ( #35642 )
...
Related to #35641
See also #35271
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 17:24:58 +08:00
3107701fe8
enhance: optimize retrieve on dynamic field ( #35580 )
...
#35514
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-22 14:24:56 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
bc2bbdc82f
enhance: Forbid access log print error message with line break ( #34491 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-20 17:36:55 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2( #35350 ) ( #35470 )
...
related: #35350
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:16:55 +08:00
e09dc3be58
enhance: Mark query node as read only after suspend ( #35492 )
...
issue: #34985 #35493
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 14:02:54 +08:00
14ef88f423
fix: Fix proxy panic after Import related api failed ( #35540 )
...
issue: #35539
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 15:22:55 +08:00
a570567644
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress ( #35472 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 14:18:54 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
fe20366b5c
enhance: Remove duplicated schema helper creation in proxy ( #35489 )
...
Related to PRs of #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-15 19:18:53 +08:00
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
0400eb91c1
enhance: remove the invalid log ( #35465 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-14 18:00:53 +08:00
f13b21c041
fix: error message when input is not aligned ( #35322 )
...
fix: https://github.com/milvus-io/milvus/issues/35321
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 09:56:28 +08:00
a2b517523d
enhance: Add in-memory cache for casbin enforcer result ( #35271 )
...
See also #35270
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 18:48:15 +08:00
c22b0a636e
fix: delete by expr failed at retry progress ( #35241 )
...
issue: #35240
delete by expr shard the same err object between channels, so if one
channel's query failed, it will fail all channel, which will break
channel level retry policy, and make delete operation failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 11:18:16 +08:00
c64a078458
enhance: Support proxy/delegator qn client pooling ( #35194 )
...
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
6106a48acb
fix: upsert result use the previous pk ( #34672 )
...
#34668
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-31 15:25:51 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
3c7f73137e
fix: disallow expr when partition key isolation is enabled ( #35031 )
...
issue: #34336
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 2889481ce9a14230e9c7f1c8f9c3c1decde77e03)
2024-07-29 14:21:50 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
783f9d9c33
fix: Unify hook singleton implementation in proxy ( #34887 )
...
Related to #34885
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
94d8f48b85
enhance: Use auto detection proto interface to fetch grpc size ( #35024 )
...
Related to #34763
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 14:23:47 +08:00
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs ( #34763 )
...
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
d93c51d5f6
enhance: change GetVChannels rpc into GetPChannelInfo ( #34940 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 13:29:45 +08:00
e2f40fc2a8
fix: Check legacy guarantee ts when skipping alloc ts ( #34981 )
...
See also #34980
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 10:17:45 +08:00
a2ac84bd64
feat: record the duration waiting in the proxy queue ( #34744 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-23 14:23:52 +08:00
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
88c38a3756
fix: `execute` method should return an error when result is a failure ( #34813 )
...
- issue: #34812
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:13:40 +08:00
fbad77f22a
fix: fix ut test fail ( #34832 )
...
#34831
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-19 17:51:40 +08:00
e4e18cb8c3
enhance: Add dynamic cgo pool for proxy CGO call ( #34768 )
...
Related to #34705
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-19 15:11:41 +08:00
c8bf6c8a46
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log ( #34738 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-18 15:17:41 +08:00
f1b2f7b640
enhance: refactor bitmap index and internal hybrid index ( #34450 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-18 10:39:42 +08:00
7306d2d115
fix: fix metaCache cleanup issue when listPolicy failed ( #34449 )
...
issue: #34667
---------
Signed-off-by: balloon1995 <hszoe1995@outlook.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-07-16 10:03:38 +08:00
b5225df21c
enhance: Use unified requery flag when set metrics ( #34632 )
...
See also #34601
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 23:03:38 +08:00
d8e68cbb97
enhance: use the insert rate limit for the upsert request ( #34615 )
...
- issue: #34614
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-12 16:17:35 +08:00
5bb0d21e32
fix: Restful API use deprecate error code cause access log panic. ( #34576 )
...
relate: https://github.com/milvus-io/milvus/issues/34578
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-12 10:13:35 +08:00
104d0966b7
feat: support partition key isolation ( #34336 )
...
issue: #34332
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
07b94b4615
enhance: support upsert autoid==true ( #30342 )
...
related with: #29258
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-11 16:53:35 +08:00
ae6d6f91e6
enhance: change autoindex default metric type ( #34261 )
...
issue: #34304
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 19:52:24 +08:00
60be454db0
enhance: add disk quota and max collections into db properties ( #34368 )
...
issue: #34385
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
233b8486ec
fix: generate gin logs failed cause access logs panic ( #34142 )
...
relate: https://github.com/milvus-io/milvus/issues/34141
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-05 16:30:10 +08:00
18f2458385
fix: fix bitmap supported type ( #34350 )
...
#34314
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-05 15:50:10 +08:00
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation ( #34421 )
...
issue: #34420
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
c16a704934
fix: [restful v2] count(*) & hook ( #34369 )
...
issue: #31224 #34374
for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]
add hook about grpc call
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-04 19:38:09 +08:00
10b3ce24ba
enhance: use configed max topk for iterator when input topk exceeds ( #34290 )
...
related: #34290
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-04 14:24:19 +08:00
fa8d641ce6
fix: update check for sparse hnsw index ( #33713 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-02 21:56:09 +08:00
b32dae1883
enhance: Refine max length exceeded error message ( #34300 )
...
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 21:16:07 +08:00
b49862d4f3
enhance: Optimize grow slice cost during query ( #34253 )
...
issue: #32252
This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
186757e622
enhance: support mark error as user error ( #33498 )
...
relate: https://github.com/milvus-io/milvus/issues/33492
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
96dcee5dff
fix:load major compaction partial result( #34051 ) ( #34052 )
...
related: #34051
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
9c2eeff427
enhance: refine error message for invalid limit parameter ( #34178 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 22:14:04 +08:00
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call ( #34102 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
fd922d921a
enhance: Add nilness linter and fix some small issues ( #34049 )
...
Add `nilness` for govet linter and fixed some detected issues
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 14:52:03 +08:00
d08cb885ca
enhance: enable flush rate limiter of collection level ( #33837 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
298e50b834
enhance: check index with data type ( #33880 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
0d20303e54
fix: fix binary vector data size ( #33750 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
- fix byte size wrong for binary vectors
- fix the expect/actual error msg
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-18 21:39:59 +08:00
a789c60380
enhance: autoindex for multi data type ( #33868 )
...
issue: #22837
contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
188ee433b9
enhance: Refine name rule check error msg ( #33815 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 10:37:58 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
3540eee977
enhance: Support L0 import ( #33514 )
...
issue: https://github.com/milvus-io/milvus/issues/33157
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
0a3d456688
enhance: add restful api to trigger component stop ( #32076 )
...
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-07 10:35:54 +08:00
b9bafc76b4
enhance: access log support print with consistency level ( #33503 )
...
relate: https://github.com/milvus-io/milvus/issues/33563
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 17:13:50 +08:00
25080bb455
enhance: Print `UseDefaultConsistency` param in read requests ( #33617 )
...
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 10:27:50 +08:00
0cf225fa75
enhance: access log support restful api ( #33155 )
...
relate: https://github.com/milvus-io/milvus/issues/31823
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-31 11:13:44 +08:00
416a2cf507
fix: query iterator lack results( #33137 ) ( #33422 )
...
related: #33137
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
589d4dfd82
enhance: optimize bitmap index ( #33358 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
b13932bb55
enhance: Enable database level replica num and resource groups for loading collection ( #33052 )
...
issue: #30040
This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups
User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-29 10:59:43 +08:00
a59ed65c48
doc: [skip-e2e] add comments for task queue ( #33388 )
...
Add comments for task queue in internal/proxy/task_scheduler.go.
Signed-off-by: ArenaSu <704427617@qq.com>
2024-05-27 20:51:42 +08:00
af71116499
fix: Fix frequent 'failed to get request info' log ( #33334 )
...
issue: https://github.com/milvus-io/milvus/issues/33333
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-27 14:27:41 +08:00
cb99e3db34
enhance: add the includeCurrentMsg param for the Seek method ( #33326 )
...
/kind improvement
- issue: #33325
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 10:31:41 +08:00
2964f60edc
enhance: the proxy metric in the query request ( #33307 )
...
/kind improvement
issue: #33306
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-24 14:19:41 +08:00
592d701617
fix: Fix global rate limit is not working ( #33335 )
...
If the request is limited by rate limiter, limiter should not "Cancel".
This is because, if limited, tokens are not deducted; instead, "Cancel"
operation would increase the token count.
issue: https://github.com/milvus-io/milvus/issues/31705
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:05:47 +08:00
1b4e28b97f
enhance: Check by proxy rate limiter when delete get data by query. ( #30891 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-23 20:03:40 +08:00
155cb40fc9
fix: Fix printing type of request ( #33289 )
...
issue: https://github.com/milvus-io/milvus/issues/31705
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 14:19:40 +08:00
9ff023ee35
fix: Fix filtering by partition key fails for importing data ( #33274 )
...
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.
issue: https://github.com/milvus-io/milvus/issues/33237
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 11:13:40 +08:00
dd0c6d6980
fix: the panic when db isn't existed in the rate limit interceptor ( #33244 )
...
issue: #33243
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-22 15:57:39 +08:00
13fdaea9f0
fix: accesslog writer cache close cause deadlock ( #33261 )
...
relate: https://github.com/milvus-io/milvus/issues/33260
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-22 14:55:39 +08:00
f336b2d672
fix: Check schema without vector field in proxy ( #33211 )
...
Related to #33199
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 14:33:39 +08:00
9f81290c63
fix: try best to get enough query results ( #33178 )
...
issue: https://github.com/milvus-io/milvus/issues/33137
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-21 11:57:51 +08:00
ec98de3ad4
fix: reset the quota value when init the limiter ( #33111 )
...
issue: #33107
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-20 10:35:38 +08:00
5c6de473bb
fix: check array field data is nil before getting the type ( #33114 )
...
issue: https://github.com/milvus-io/milvus/issues/33074
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-17 17:29:37 +08:00
1671c7898a
enhance: sync milvus proto version ( #33094 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-16 18:51:34 +08:00
b45798107a
enhance: add nullable in Field, check valid_data and fill data ( #32086 )
...
1. add nullable in model.Field
help to read nullable accurately.
2. check valid_data
a. if user pass default_value or the field is nullable, the length of
valid_data must be num_rows.
b. if passed valid_data, the length of passed field data must equal to
the number of 'true' in valid_data.
c. after fill default_value, only nullable field will still has
valid_data.
3. fill data in two situation
a. has no default_value, if nullable,
will fill nullValue when passed num_rows not equal to expected num_rows.
b. has default_value,
will fill default_value when passed num_rows not equal to expected
num_rows.
c. after fill data, the length of all field will equal to passed
num_rows.
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-16 11:57:35 +08:00
875ad88d84
enhance: Support access log use stdout with write cache and auto flush as time ( #32213 )
...
relate: https://github.com/milvus-io/milvus/issues/28948
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-16 10:05:34 +08:00
6fc00e42d7
fix: Fix binlog import ( #33027 )
...
binlog import should only import into one partition.
issue: https://github.com/milvus-io/milvus/issues/32807
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-14 21:45:32 +08:00
4fc7915c70
enhance: unify data generation test APIs ( #32955 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-14 14:33:33 +08:00
dc058eaf61
fix: should init access logger before init server ( #32976 )
...
relate: https://github.com/milvus-io/milvus/issues/32968
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-14 10:47:32 +08:00
230c88b1de
enhance: improve reduce performance on very large topk ( #32871 )
...
fix #32870
reduce cpu usage for reduce on large k
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-13 11:55:31 +08:00
8a9a42198d
fix: Proxy crash due to shard leader cache data race ( #32971 )
...
issue: #32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash
This PR fixed that use leaderMut when try to access shard leader cache.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-11 14:32:12 +08:00
c35797c399
enhance: expose DescribeDatabase api in proxy ( #32732 )
...
issue: #32707
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
8963615b33
enhance: add the cost response for the rest api ( #32620 )
...
issue: https://github.com/milvus-io/milvus/issues/30436
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-08 11:53:29 +08:00
ba02d54a30
enhance: update shard leader cache when leader location changed ( #32470 )
...
issue: #32466
this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00
53874ce245
fix: Fix cannot specify partition name in binlog import ( #32730 )
...
issue: https://github.com/milvus-io/milvus/issues/32807
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-07 17:19:30 +08:00
1f58cda957
enhance: add more trace for search & query ( #32734 )
...
issue: https://github.com/milvus-io/milvus/issues/32728
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 13:03:29 +08:00
7da1ca9efb
enhance: check the partition num when creating collection with partition key ( #32670 )
...
issue: #30577
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-07 10:43:29 +08:00
48fb8310ba
fix: skip to check auto id when inserting data ( #32775 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/32591
related pr: #32592
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-07 10:17:30 +08:00
02bf47f295
fix: Validate num of rows for insert field data with schema ( #32770 )
...
See also #32769
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 10:07:29 +08:00
600db9d99e
enhance: check field data type of input ( #32777 )
...
issue: #32769
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-06 21:15:29 +08:00
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
42d0412e93
enhance: Add channelCPs in FlushResponse ( #32044 )
...
#32609
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-04-30 09:45:27 +08:00
0359139299
enhance: return the auth collection list when using the `show collection` api ( #32576 )
...
issue: #32550
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-29 19:09:34 +08:00
d900e68440
fix: fix GetShardLeaders return empty node list ( #32685 )
...
issue: #32449
to avoid GetShardLeaders return empty node list, this PR add node list
check in both client side and server side.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-29 14:19:26 +08:00
09cd56d44f
enhance: add the skip auto id and partition key check config ( #32592 )
...
/kind improvement
issue: #32591
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-29 10:29:26 +08:00
07720f1a95
enhance: expose alter database api in proxy ( #32639 )
...
issue: #30040
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 18:15:31 +08:00
9a719ec89e
fix: skip the empty partition name in the rate limit interceptor ( #32647 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-28 11:01:28 +08:00
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation ( #32611 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
04f355a802
enhance: Enable alter database props in rootcoord ( #32458 )
...
issue: #30040
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-25 10:53:25 +08:00
dcc15e3ed8
enhance: Remove `TestRateLimiter/tests_refresh_rate_by_config` case ( #32545 )
...
See also #32313
This case always fail since the watch config logic is removed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-24 10:13:24 +08:00
365e50b63e
fix: revert add range search params check in proxy ( #32366 )
...
no need to check params in empty segment.
#30365
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-04-23 17:41:23 +08:00
3119a7b23f
fix: Remove watching config event in rate limiter ( #32313 )
...
Remove watching config event in rate limiter to prevent object leaks.
Instead, reset limiter rates periodically.
issue: https://github.com/milvus-io/milvus/issues/32312
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-23 16:19:24 +08:00
d0ffb71a17
enhance: forbid delete with always true expression ( #32472 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-22 11:09:23 +08:00
8594b55ad5
enhance: add `max insert request size` and `must use partition key` configs ( #32433 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:31:20 +08:00
31a29a2451
enhance: avoid the repeated metric info in the proxy ( #32380 )
...
issue: #30577
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:21:20 +08:00
02f17b842a
fix: fix incomplete hybrid search result when nq > 1 ( #32177 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-17 17:09:32 +08:00
84f05ba66e
enhance: make the auth error message more suitable ( #32253 )
...
/issue: #32252
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-16 16:47:19 +08:00
1af084ea6b
enhance: Make datanode exit and case `TestProxy` faster ( #32218 )
...
/kind improvement
issue: #32219
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-16 10:49:20 +08:00
48fe977a9d
enhance: declarative resource group api ( #31930 )
...
issue: #30647
- Add declarative resource group api
- Add config for resource group management
- Resource group recovery enhancement
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-15 08:13:19 +08:00
88c6828d6c
fix: failed to raise metric_type not match error ( #32202 )
...
issue: https://github.com/milvus-io/milvus/issues/32176
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-12 16:19:18 +08:00
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
fb376fd1e6
enhance: Refine varchar length exceeds max length error message ( #31521 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-12 15:01:19 +08:00
43a9be2cce
enhance: support access log dynamic config and prepare for support resutful interface access log ( #29638 )
...
relate: https://github.com/milvus-io/milvus/issues/29637
https://github.com/milvus-io/milvus/issues/31823
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-12 10:37:27 +08:00
0bfe130991
fix: fix metric about ProxySearchVectors ( #32152 )
...
issue :https://github.com/milvus-io/milvus/issues/25639
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-11 16:13:17 +08:00
52ae47c850
enhance: gather materialized view search info once per request ( #31996 )
...
issue: #29892
This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-11 15:21:19 +08:00
2fdf1a6e76
feat: [Sparse Float Vector] added some integration tests ( #31062 )
...
add some integration tests for sparse float vector support
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-10 19:57:18 +08:00
dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly ( #32089 )
...
fix https://github.com/milvus-io/milvus/issues/32059
this pr fix two issues:
offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 16:01:18 +08:00
90bed1caf9
enhance: add the related data size for the read apis ( #31816 )
...
issue: #30436
origin pr: #30438
related pr: #31772
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
a63cf8625c
enhance: support to invalid the database meta cache ( #32078 )
...
issue: #32077
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 06:45:18 +08:00
420baacb6d
enhance: use the rate log in the privilege interceptor ( #32037 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-09 16:19:17 +08:00
a0a4ec8b67
enhance: make range search param check message more meaningful ( #32006 )
...
Issue: #31970
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-09 16:17:26 +08:00
089c805e0a
enhance:Refactor hybrid search ( #32020 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
8fda3cbeda
enhance: check fp16/bf16 nan or inf value ( #31840 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-09 14:19:27 +08:00
7e0d03420e
enhance: Remove duplicated err check code ( #32001 )
...
The `err != nil` check is duplicated here and shall be removed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-08 21:59:17 +08:00
00438f408f
enhance: Unify data type check APIs for go ( #31887 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
1af2ee903c
fix: Check parse QueryInfo error before using it ( #31836 )
...
See also milvus-io/milvus-sdk-go#693 #31735
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-03 11:29:14 +08:00
92c84ccc3f
enhance: ban groupby on hybrid-search( #29968 ) ( #31810 )
...
related: #29968
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-02 15:47:18 +08:00
bd853be8c7
enhance: Add db label for some usual metrics ( #30956 )
...
issue: #31782
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
ac26908cc4
enhance: Remove the storage info report ( #31772 )
...
issue: #30436
origin pr: #30438
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-01 20:50:59 -07:00
417c678754
enhance: Retry on inconsistent requery result ( #31713 )
...
Make retry on ErrInConsistentRequery in proxy rather than in every SDK.
issue: https://github.com/milvus-io/milvus/issues/31642
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-30 20:53:19 +08:00
b99c46246c
enhance: ban groupby on binary vector( #31134 ) ( #31659 )
...
related: #31134
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-28 15:19:10 +08:00
congqixia
Zhen Ye
yihao.dai
Chun Han
SimFG
zhagnlu
aoiasd
wei liu
Jiquan Long
smellthemoon
Patrick Weizhi Xu
balloon1995
Gao
jaime
PowderLi
Buqian Zheng
cqy123456
wayblink
ArenaSu
Bingyi Sun
Cai Yudong
Xiaofan
zhenshan.cao