Commit Graph

725 Commits (817b616eb4c22d433ff46ffeb65a8f40761df073)

Author SHA1 Message Date
sthuang 8c35064d01
enhance: [2.5] clean up logs of empty op key (#39394)
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39335
related: https://github.com/milvus-io/milvus/issues/39334

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-20 12:15:10 +08:00
Zhen Ye 0391b8c7d3
fix: update rg failed when alter database (#39356)
issue: #36933
pr: #39355

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-17 10:37:03 +08:00
yihao.dai 6773fb10a8
enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900)
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38403

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 17:53:01 +08:00
SimFG 4cb47b3f67
feat: [2.5] root privileges can be customized (#39324)
- issue: #39184
- pr: #39191

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-16 14:27:11 +08:00
yihao.dai 9b916f2cf8
enhance: [2.5] Skip creating partition rate limiters when not enable (#38911)
issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38816

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 22:19:02 +08:00
yihao.dai 8bf0f0b921
fix: [2.5] Fix rootcoord meta mutex contention (#38902)
RootCoord meta uses copy-on-write, allowing the removal of unnecessary
copies.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38799

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:11:08 +08:00
sthuang 4a47f760b3
fix: [2.5] rbac custom group privilege level check (#39164) (#39224)
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-14 16:50:59 +08:00
Zhen Ye 54036bcafd
enhance: add broadcast operation for msgstream (#39119)
issue: #38399
pr: #39040

- make broadcast service available for msgstream by reusing the
architecture streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:11:00 +08:00
SimFG 7c8db6f61f
fix: [2.5] use the object heap to keep the min ddl ts order (#39195)
- issue: #39002
- pr: #39118

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-14 14:23:05 +08:00
SimFG 68734fe4f7
enhance: [2.5] extract the task step execution process (#39196)
- issue: #38322
- pr: #38323

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-13 21:58:58 +08:00
tinswzy 39fa00fc1a
enhance: add intent ctx to rootcoord init (#38948)
issue: #38439 
cherry pick "enhance: add intent ctx to rootcoord init"

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-01-11 23:54:57 +08:00
Zhen Ye 95809ca767
enhance: make new go package to manage proto (#39128)
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
Zhen Ye 6f1febe881
enhance: move streaming coord from datacoord to rootcoord (#39009)
issue: #38399
pr: #39007

We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-07 17:56:56 +08:00
sthuang 147c6b099b
fix: [2.5] expand privilege group when list policy in rootcoord (#38760)
cherry-pick from: https://github.com/milvus-io/milvus/pull/38758
related: https://github.com/milvus-io/milvus/issues/38757

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-26 15:34:49 +08:00
sthuang a4901ef7ec
fix: [2.5] fix privilege group list and list collections (#38738)
cherry-pick from: https://github.com/milvus-io/milvus/pull/38684
related: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:06:50 +08:00
jaime 11bedf5e76
fix: Revert "Expose metrics of stanby coordinators (#27698)" (#38621)
issue: #38608
pr: #38620

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-20 18:04:47 +08:00
jaime 78438ef41e
fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555)
issue: #35563

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-19 00:38:45 +08:00
sthuang de8858931a
fix: rbac revoke check if there is same privilege in other privilege groups granted (#38558)
related: https://github.com/milvus-io/milvus/issues/38557

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-18 18:46:45 +08:00
yihao.dai 1aa31e2a9b
fix: Fix inaccurate general count (#38524)
Checking general count should only count healthy collections and
partitions.

issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 23:24:45 +08:00
yihao.dai 01fa2f8da5
fix: Fix compile error (#38510)
/kind improvement

related: #37559

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 12:16:19 +08:00
jaime 28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests (#35589)
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 11:02:45 +08:00
SimFG 2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559)
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
SimFG fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178)
- issue: #38177

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
Xianhui Lin f2d79b3d3a
enhance: altercollection validation logic (#38419)
altercollection validation logic
issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 21:42:42 +08:00
Xianhui Lin d0a8110a7a
enhance: alterdatabase support delete property (#38380)
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 17:04:43 +08:00
wei liu a118ca14a7
fix: Fix role be dropped when grant still exist. (#38342)
issue: #38325
the old impl only to check grant in default db before drop role, which
may cause role be dropped when grant still exist.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-11 11:24:42 +08:00
Xianhui Lin db05d4f976
enhance: alterindex & altercollection supports altering properties (#37437)
enhance : 

1. alterindex delete properties
We have introduced a new parameter deleteKeys to the alterindex
functionality, which allows for the deletion of properties within an
index. This enhancement provides users with the flexibility to manage
index properties more effectively by removing specific keys as needed.
2. altercollection delete properties
We have introduced a new parameter deleteKeys to the altercollection
functionality, which allows for the deletion of properties within an
collection. This enhancement provides users with the flexibility to
manage collection properties more effectively by removing specific keys
as needed.

3.support altercollectionfield
We currently support modifying the fieldparams of a field in a
collection using altercollectionfield, which only allows changes to the
max-length attribute.
Key Points:
- New Parameter - deleteKeys: This new parameter enables the deletion of
specified properties from an index. By passing a list of keys to
deleteKeys, users can remove the corresponding properties from the
index.

- Mutual Exclusivity: The deleteKeys parameter cannot be used in
conjunction with the extraParams parameter. Users must choose one
parameter to pass based on their requirement. If deleteKeys is provided,
it indicates an intent to delete properties; if extraParams is provided,
it signifies the addition or update of properties.

issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-11 10:20:42 +08:00
congqixia fec31fedce
enhance: Move `SyncCreatedPartition` step to proxy (#38296)
Related to #38275

This PR move sync created partition step to proxy to avoid potential
logic deadlock when create partition happens with target segment change.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-10 12:30:43 +08:00
jaime 8ed019735c
enhance: add disk stats within system metrics (#38033)
issue: ##36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 16:32:41 +08:00
tinswzy 5768dbbb5d
enhance: refine pular related mq interfaces (#38007)
issue: #35917 
Refines the pulsar-related mq APIs to allow the ctx to be passed down

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
cai.zhang 73aa95f596
fix: Add version to the proxy cache to resolve concurrency issues (#38067)
issue: #36989

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-04 18:06:39 +08:00
sthuang d822edb9a8
enhance: add list aliases privilege into public role (#38176)
related issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-03 23:42:39 +08:00
yihao.dai db2ed2c044
fix: Fix inaccurate partition num metric (#37996)
The partition number has already been incremented in
`ChangePartitionState`, so there is no need to increment it again in
`AddPartition`.

issue: https://github.com/milvus-io/milvus/issues/37630

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-03 21:24:40 +08:00
sthuang a5e0a56a8e
fix: move grant/revoke v2 params check from rootcoord to proxy (#38130)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues #38042: The interface "grant_v2" does not support empty
collectionName while the error says it does

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 19:48:37 +08:00
smellthemoon db453c063d
enhance: no need database check in restful sdk (#38078)
#38077 
remove the check for two reason
1. server will do the same to make sure use the correct database;
2. each req has an additional overhead of calling the proxy to check
database.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-12-02 17:04:38 +08:00
sthuang 23dc313c44
fix: fix grant/revoke v2 meta and unclear error messages (#38110)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues:
#37974: better error messages for grant v2 interface
#37903: fix meta built-in privilege group object name
#37843: better error messages for custom privilege group interface 
#38002: fix built-in privilege group meta to pass proxy interceptor
check
#38008: fix revoke v2 to support revoking v1 granted privileges

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 11:36:39 +08:00
yihao.dai 7fb0c281f2
fix: Fix checkGeneralCapacity slowly (#37976)
Cache the general count to speed up `checkGeneralCapacity`.

issue: https://github.com/milvus-io/milvus/issues/37630

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-01 18:28:38 +08:00
jaime a9080d8d96
enhance: the actual number of databases should equal the config value (#38006)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-29 14:44:36 +08:00
SimFG 49ee46ec1d
enhance: support to config the default db properties (#38035)
- issue: #38034

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-27 10:04:34 +08:00
SimFG 2208b7c2ef
fix: the too long default root password does not take effect (#37983)
- issue: #36987

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-26 17:24:35 +08:00
sthuang 19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
tinswzy e247ff9ee7
enhance: refine rootcoord/metatable interfaces to ensure that each method includes a ctx parameter (#37846)
issue: #35917
Before enhancing log trace information, it's necessary to pass the
context to the method entry point.
This PR first refine the rootcoord/metatable interfaces to ensure that
each method includes a ctx parameter.

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-21 19:30:32 +08:00
cai.zhang c07f056b17
fix: Use the ID to retrieve the real name when collectionName is empty (#37859)
issue: #36989

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-21 14:28:32 +08:00
SimFG 3bbbf3cd95
enhance: improve the lock key of the create/drop collection task (#37759)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-19 15:50:32 +08:00
Xiaofan 33bfb25c73
enhance: refine meta cache log and logic (#37318)
related to #36989
add more logs in proxy meta cache and make it clearer

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
sthuang 2d72ad33f2
enhance: RBAC built in privilege groups (#37720)
issue: #37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-18 20:38:39 +08:00
jaime 12ed40e125
fix: some missing metrics for WebUI restful API (#37747)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-18 14:00:32 +08:00
congqixia b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib (#37708)
Related to #35020

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
sthuang ff00a12805
enhance: RBAC custom privilege group ut coverage (#37558)
issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 20:40:25 +08:00