2a4c00de9d
enhance: Decouple shard client manager from shard cache ( #37371 )
...
issue: #37115
the old implementation update shard cache and shard client manager at
same time, which causes lots of conor case due to concurrent issue
without lock.
This PR decouple shard client manager from shard cache, so only shard
cache will be updated if delegator changes. and make sure shard client
manager will always return the right client, and create a new client if
not exist. in case of client leak, shard client manager will purge
client in async for every 10 minutes.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-12 10:30:28 +08:00
da227ff9a1
feat: Support create collection with functions ( #35973 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
91d23ecbe1
fix: memory leak in proxy meta cache ( #36075 )
...
issue: #36074
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:33:05 +08:00
9d80137698
fix: Check clustering key skip load behavior ( #35865 )
...
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 11:17:02 +08:00
311f860676
enhance: support to drop the role which is related the privilege list ( #35727 )
...
- issue: #35545
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 15:17:00 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
ba02d54a30
enhance: update shard leader cache when leader location changed ( #32470 )
...
issue: #32466
this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
40ca98f57f
enhance: Skip timestamp allocation when search/query consistency level is eventually ( #29773 )
...
issue: #29772
1. Skip timestamp allocation when search/query consistency level is
eventually.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-21 09:52:59 +08:00
d87726e4c7
enhance: Don't expire ShardLeaders cache actively ( #29879 )
...
issue: #29772
The shardLeaders cache does not actively expire, update the cache when
search/query fails.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-26 14:13:01 +08:00
38746dfc2b
fix: Remove useless lock which cause porxy meta cache recursive lock ( #30203 )
...
relate: https://github.com/milvus-io/milvus/issues/30193
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-23 23:56:55 +08:00
a81d2b4780
enhance: refine proxy metacache for concurrent safe ( #29872 )
...
relate: https://github.com/milvus-io/milvus/issues/29675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-22 14:28:55 +08:00
4f8c540c77
enhance: cache collection schema attributes to reduce proxy cpu ( #29668 )
...
See also #29113
The collection schema is crucial when performing search/query but some
of the information is calculated for every request.
This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:28:46 +08:00
bcf8f27aa7
enhance: refine proxy meta cache partition logic ( #29315 )
...
See also #29113
- Unify partition info refresh logic
- Prevent parse partition names for each partition key search request
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 10:02:43 +08:00
20fc90c591
enhance: find collection schema from cache ( #28782 )
...
issue: #28781 #28329
1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
b4f86ea55e
Construct all success status with merr ( #27226 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 10:57:23 +08:00
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
e8f1b1736e
Remove log.Error(err.error())-style log ( #26783 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-01 13:09:01 +08:00
f9823e039f
fix describe rg with non exist collection ( #26227 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-10 18:43:16 +08:00
4e1b65d38f
Refine the rbac cache update process ( #26150 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-07 11:59:07 +08:00
f915f1739a
[Fixup] Short-term fix metacache data race ( #25802 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-04 17:47:07 +08:00
9614e61f14
Fix collection and channel not match ( #25859 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-08-01 17:33:06 +08:00
b9189b9f41
Organize mocks from types.go ( #25466 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-07-14 10:12:31 +08:00
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
3827ac30bc
Remove load cache ( #23287 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 10:36:41 +08:00
32be374c33
Fix MetaCache ut internal not changed ( #23276 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-07 13:20:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
ed60b434fa
fix unstable ut ( #23070 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-03-28 19:06:01 +08:00
6b5dfa6db2
deprecate shard cache immediately if query failed ( #22779 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-03-17 16:25:55 +08:00
69cc68caff
support first or random strategy ( #22766 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-03-16 14:25:55 +08:00
3ffa6d5302
Mark cache deprecated instead of removing it ( #22674 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-10 10:19:53 +08:00
d9908ff231
Fix TestMetaCache_ExpireShardLeaderCache unit test ( #22651 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-09 18:25:52 +08:00
1a1a5caafa
update shard leader cache periodically ( #22595 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-03-08 15:13:51 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
13b508103e
refine mock querycoord ( #22198 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-16 15:38:34 +08:00
73c44d4b29
resource group impl ( #21609 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
ce3e0fcf9e
Fixbug: proxy meta_cache partially update ( #21232 ) ( #21242 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-12-14 20:45:23 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
0c9a10e8f8
1. refine logging interfaces ( #18692 )
...
2. adjust logs for query/search requests
Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-23 10:44:52 +08:00
ff0200210a
Support Role-Based Access Control ( #18425 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-04 11:04:34 +08:00
2763efc9b0
Add cache of grpc client of ShardLeader in proxy ( #17301 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-06-02 12:16:03 +08:00
9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID ( #16882 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-19 10:13:56 +08:00
127dd34b37
Make proxy use roundrobin to choose replica ( #17063 )
...
Fixes : #17055
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-05-17 22:35:57 +08:00
a1480aaf76
Clear shard leaders cache when releasing ( #16942 )
...
See also: #16926
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-05-17 11:11:56 +08:00
wei liu
aoiasd
jaime
congqixia
SimFG
cai.zhang
PowderLi
yah01
chyezh
smellthemoon
yiwangdr
yah01
yihao.dai
Enwei Jiao
zhenshan.cao
Zach
XuanYang-cn