Commit Graph

714 Commits (4966230d86438e12c5f1a4315714345a3108e1c7)

Author SHA1 Message Date
SimFG d7623ab635
enhance: extract the task step execution process (#38323)
- issue: #38322

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-03 12:36:54 +08:00
tinswzy 11f8f4a378
enhance: add intent ctx to rootcoord init (#38439)
issue: #35917 
add intent ctx to rootcoord initialization process

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
sthuang 94955e5292
fix: expand privilege group when list policy in rootcoord (#38758)
related: https://github.com/milvus-io/milvus/issues/38757

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-26 15:38:50 +08:00
sthuang 6bc799061e
fix: fix privilege group list and list collections (#38684)
related: #37031
* built-in privilege group privileges in listPrivilegeGroups() should be
the same as in milvus.yaml
* collections granted by collection level built-in privilege group
should be list in showCollections()

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:00:51 +08:00
jaime 5afd0c0a2b
fix: Revert "Expose metrics of stanby coordinators (#27698)" (#38620)
issue: #38608

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-23 11:46:57 +08:00
jaime 78438ef41e
fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555)
issue: #35563

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-19 00:38:45 +08:00
sthuang de8858931a
fix: rbac revoke check if there is same privilege in other privilege groups granted (#38558)
related: https://github.com/milvus-io/milvus/issues/38557

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-18 18:46:45 +08:00
yihao.dai 1aa31e2a9b
fix: Fix inaccurate general count (#38524)
Checking general count should only count healthy collections and
partitions.

issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 23:24:45 +08:00
yihao.dai 01fa2f8da5
fix: Fix compile error (#38510)
/kind improvement

related: #37559

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 12:16:19 +08:00
jaime 28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests (#35589)
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 11:02:45 +08:00
SimFG 2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559)
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
SimFG fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178)
- issue: #38177

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
Xianhui Lin f2d79b3d3a
enhance: altercollection validation logic (#38419)
altercollection validation logic
issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 21:42:42 +08:00
Xianhui Lin d0a8110a7a
enhance: alterdatabase support delete property (#38380)
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 17:04:43 +08:00
wei liu a118ca14a7
fix: Fix role be dropped when grant still exist. (#38342)
issue: #38325
the old impl only to check grant in default db before drop role, which
may cause role be dropped when grant still exist.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-11 11:24:42 +08:00
Xianhui Lin db05d4f976
enhance: alterindex & altercollection supports altering properties (#37437)
enhance : 

1. alterindex delete properties
We have introduced a new parameter deleteKeys to the alterindex
functionality, which allows for the deletion of properties within an
index. This enhancement provides users with the flexibility to manage
index properties more effectively by removing specific keys as needed.
2. altercollection delete properties
We have introduced a new parameter deleteKeys to the altercollection
functionality, which allows for the deletion of properties within an
collection. This enhancement provides users with the flexibility to
manage collection properties more effectively by removing specific keys
as needed.

3.support altercollectionfield
We currently support modifying the fieldparams of a field in a
collection using altercollectionfield, which only allows changes to the
max-length attribute.
Key Points:
- New Parameter - deleteKeys: This new parameter enables the deletion of
specified properties from an index. By passing a list of keys to
deleteKeys, users can remove the corresponding properties from the
index.

- Mutual Exclusivity: The deleteKeys parameter cannot be used in
conjunction with the extraParams parameter. Users must choose one
parameter to pass based on their requirement. If deleteKeys is provided,
it indicates an intent to delete properties; if extraParams is provided,
it signifies the addition or update of properties.

issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-11 10:20:42 +08:00
congqixia fec31fedce
enhance: Move `SyncCreatedPartition` step to proxy (#38296)
Related to #38275

This PR move sync created partition step to proxy to avoid potential
logic deadlock when create partition happens with target segment change.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-10 12:30:43 +08:00
jaime 8ed019735c
enhance: add disk stats within system metrics (#38033)
issue: ##36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 16:32:41 +08:00
tinswzy 5768dbbb5d
enhance: refine pular related mq interfaces (#38007)
issue: #35917 
Refines the pulsar-related mq APIs to allow the ctx to be passed down

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
cai.zhang 73aa95f596
fix: Add version to the proxy cache to resolve concurrency issues (#38067)
issue: #36989

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-04 18:06:39 +08:00
sthuang d822edb9a8
enhance: add list aliases privilege into public role (#38176)
related issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-03 23:42:39 +08:00
yihao.dai db2ed2c044
fix: Fix inaccurate partition num metric (#37996)
The partition number has already been incremented in
`ChangePartitionState`, so there is no need to increment it again in
`AddPartition`.

issue: https://github.com/milvus-io/milvus/issues/37630

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-03 21:24:40 +08:00
sthuang a5e0a56a8e
fix: move grant/revoke v2 params check from rootcoord to proxy (#38130)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues #38042: The interface "grant_v2" does not support empty
collectionName while the error says it does

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 19:48:37 +08:00
smellthemoon db453c063d
enhance: no need database check in restful sdk (#38078)
#38077 
remove the check for two reason
1. server will do the same to make sure use the correct database;
2. each req has an additional overhead of calling the proxy to check
database.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-12-02 17:04:38 +08:00
sthuang 23dc313c44
fix: fix grant/revoke v2 meta and unclear error messages (#38110)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues:
#37974: better error messages for grant v2 interface
#37903: fix meta built-in privilege group object name
#37843: better error messages for custom privilege group interface 
#38002: fix built-in privilege group meta to pass proxy interceptor
check
#38008: fix revoke v2 to support revoking v1 granted privileges

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 11:36:39 +08:00
yihao.dai 7fb0c281f2
fix: Fix checkGeneralCapacity slowly (#37976)
Cache the general count to speed up `checkGeneralCapacity`.

issue: https://github.com/milvus-io/milvus/issues/37630

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-01 18:28:38 +08:00
jaime a9080d8d96
enhance: the actual number of databases should equal the config value (#38006)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-29 14:44:36 +08:00
SimFG 49ee46ec1d
enhance: support to config the default db properties (#38035)
- issue: #38034

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-27 10:04:34 +08:00
SimFG 2208b7c2ef
fix: the too long default root password does not take effect (#37983)
- issue: #36987

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-26 17:24:35 +08:00
sthuang 19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
tinswzy e247ff9ee7
enhance: refine rootcoord/metatable interfaces to ensure that each method includes a ctx parameter (#37846)
issue: #35917
Before enhancing log trace information, it's necessary to pass the
context to the method entry point.
This PR first refine the rootcoord/metatable interfaces to ensure that
each method includes a ctx parameter.

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-21 19:30:32 +08:00
cai.zhang c07f056b17
fix: Use the ID to retrieve the real name when collectionName is empty (#37859)
issue: #36989

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-21 14:28:32 +08:00
SimFG 3bbbf3cd95
enhance: improve the lock key of the create/drop collection task (#37759)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-19 15:50:32 +08:00
Xiaofan 33bfb25c73
enhance: refine meta cache log and logic (#37318)
related to #36989
add more logs in proxy meta cache and make it clearer

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
sthuang 2d72ad33f2
enhance: RBAC built in privilege groups (#37720)
issue: #37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-18 20:38:39 +08:00
jaime 12ed40e125
fix: some missing metrics for WebUI restful API (#37747)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-18 14:00:32 +08:00
congqixia b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib (#37708)
Related to #35020

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
sthuang ff00a12805
enhance: RBAC custom privilege group ut coverage (#37558)
issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 20:40:25 +08:00
sthuang 70605cf5b3
enhance: Support custom privilege group for RBAC (#37087)
issue: #37031

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 08:44:28 +08:00
yihao.dai ebc3c82761
fix: Fix subscription leak (#37382)
Close (unsubscribe) the msg stream after completing the
`PreCreatedTopic` check to prevent backlog issue.

issue: https://github.com/milvus-io/milvus/issues/36021

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-08 16:16:26 +08:00
Zhen Ye cae9e1c732
fix: drop collection failed if enable streaming service (#37444)
issue: #36858

- Start channel manager on datacoord, but with empty assign policy in
streaming service.
- Make collection at dropping state can be recovered by flusher to make
sure that
 milvus consume the dropCollection message.
- Add backoff for flusher lifetime.
- remove the proxy watcher from timetick at rootcoord in streaming
service.

Also see the better fixup: #37176

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-07 10:26:26 +08:00
SimFG f1dd55e0c0
enhance: improve rootcoord task scheduling policy (#37352)
- issue: #30301

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-06 14:48:26 +08:00
jaime 9d16b972ea
feat: add tasks page into management WebUI (#37002)
issue: #36621

1. Add API to access task runtime metrics, including:
  - build index task
  - compaction task
  - import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
  - sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
SimFG 1cc9cb49ad
enhance: allow to delete data when disk quota exhausted (#37134)
- issue: #37133

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-25 16:47:29 +08:00
jaime 4746f47282
feat: management WebUI homepage (#36822)
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
congqixia 30121a5a0d
fix: Rectify delete buffer row count quota value (#37060)
Related to #37057

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-23 10:15:29 +08:00
congqixia 70339820fa
fix: Check whether new collection name is alias (#36981)
Related to #36963

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-21 10:25:25 +08:00
XuanYang-cn 0516624ab0
enhance: Log when limit writing by deletion (#36954)
See also: #36953

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-18 17:25:25 +08:00
Bingyi Sun 6851738fd1
fix: fix `make generate-mockery` panic with go1.22 (#36830)
https://github.com/milvus-io/milvus/issues/36831
Fix `make generate-mockery` panic.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-17 12:11:31 +08:00