Commit Graph

419 Commits (730b8604f2c45e07c7c6eb9e592a753ec3b0c426)

Author SHA1 Message Date
SimFG a945345110
fix: [2.5] use the different msg type for the OperatePrivilegeV2 api (#40193)
- issue: #40178
- pr: #40192

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-03 10:20:01 +08:00
congqixia 709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117)
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
Chun Han 2677f0b834
fix: cancel sub contexts casade when http request timeout(#40030) (#40060)
related: #40030
pr: https://github.com/milvus-io/milvus/pull/40059

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-02-22 11:39:53 +08:00
Xianhui Lin f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876)
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
jaime ddc5b299ad
enhance: expose more metrics data (#39466)
issue: #36621 #39417
pr: #39456
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-02-07 11:48:45 +08:00
congqixia 8a0f0f4099
enhance: [2.5] Add dsl log field for HybridSearch (#39584) (#39598)
Cherry-pick from master
pr: #39584
The dsl fields are separated in sub request structs and cannot be easily
printed before. This PR adds a log helper to print the dsl expressions
of HybridSearchRequest.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-07 09:58:43 +08:00
Xianhui Lin 3b3959c625
enhance: [2.5]Add schema update time verification for insert and upsert to use cache (#39382)
enhance: Add schema update time verification for insert and upsert to
use cache
issue: https://github.com/milvus-io/milvus/issues/39093
Related to pr: https://github.com/milvus-io/milvus/pull/39096

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-21 10:19:11 +08:00
sthuang 4a47f760b3
fix: [2.5] rbac custom group privilege level check (#39164) (#39224)
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-14 16:50:59 +08:00
sthuang 3059fae40d
fix: [2.5] fix restore rbac empty meta crash (#39143)
cp from master: https://github.com/milvus-io/milvus/pull/39141
related: https://github.com/milvus-io/milvus/issues/38985

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-13 12:04:58 +08:00
Zhen Ye 95809ca767
enhance: make new go package to manage proto (#39128)
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
jaime 0693634f62
enhance: add db name in replica description (#38673)
issue: #36621
pr: #38672

Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:43:04 +08:00
SimFG 2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559)
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
Gao 8977454311
enhance: support recall estimation (#38017)
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 20:40:48 +08:00
Xianhui Lin db05d4f976
enhance: alterindex & altercollection supports altering properties (#37437)
enhance : 

1. alterindex delete properties
We have introduced a new parameter deleteKeys to the alterindex
functionality, which allows for the deletion of properties within an
index. This enhancement provides users with the flexibility to manage
index properties more effectively by removing specific keys as needed.
2. altercollection delete properties
We have introduced a new parameter deleteKeys to the altercollection
functionality, which allows for the deletion of properties within an
collection. This enhancement provides users with the flexibility to
manage collection properties more effectively by removing specific keys
as needed.

3.support altercollectionfield
We currently support modifying the fieldparams of a field in a
collection using altercollectionfield, which only allows changes to the
max-length attribute.
Key Points:
- New Parameter - deleteKeys: This new parameter enables the deletion of
specified properties from an index. By passing a list of keys to
deleteKeys, users can remove the corresponding properties from the
index.

- Mutual Exclusivity: The deleteKeys parameter cannot be used in
conjunction with the extraParams parameter. Users must choose one
parameter to pass based on their requirement. If deleteKeys is provided,
it indicates an intent to delete properties; if extraParams is provided,
it signifies the addition or update of properties.

issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-11 10:20:42 +08:00
congqixia fec31fedce
enhance: Move `SyncCreatedPartition` step to proxy (#38296)
Related to #38275

This PR move sync created partition step to proxy to avoid potential
logic deadlock when create partition happens with target segment change.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-10 12:30:43 +08:00
tinswzy 5768dbbb5d
enhance: refine pular related mq interfaces (#38007)
issue: #35917 
Refines the pulsar-related mq APIs to allow the ctx to be passed down

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
cai.zhang 73aa95f596
fix: Add version to the proxy cache to resolve concurrency issues (#38067)
issue: #36989

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-04 18:06:39 +08:00
sthuang a65d395ecd
fix: grant v2 collection name and privilege error typo (#38195)
issue: https://github.com/milvus-io/milvus/issues/38042

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-04 16:14:38 +08:00
wei liu e09f431891
fix: grant ManualCompact api doesn't work (#38096)
issue: #38086
cause ManualCompact api pass collection id in request, but RBAC requires
to check collection name, so grant ManualCompact api doesn't work.

This PR refine the ManualCompact api to accpet collection name in
request.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-03 10:36:38 +08:00
sthuang a5e0a56a8e
fix: move grant/revoke v2 params check from rootcoord to proxy (#38130)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues #38042: The interface "grant_v2" does not support empty
collectionName while the error says it does

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 19:48:37 +08:00
sthuang 23dc313c44
fix: fix grant/revoke v2 meta and unclear error messages (#38110)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues:
#37974: better error messages for grant v2 interface
#37903: fix meta built-in privilege group object name
#37843: better error messages for custom privilege group interface 
#38002: fix built-in privilege group meta to pass proxy interceptor
check
#38008: fix revoke v2 to support revoking v1 granted privileges

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 11:36:39 +08:00
jaime 7bbfe86bcd
enhance: add list index and segment index retrieval API for WebUI (#37861)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-22 16:58:34 +08:00
sthuang 19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
Xiaofan 33bfb25c73
enhance: refine meta cache log and logic (#37318)
related to #36989
add more logs in proxy meta cache and make it clearer

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
jaime 12ed40e125
fix: some missing metrics for WebUI restful API (#37747)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-18 14:00:32 +08:00
jaime 1d06d4324b
fix: Int64 overflow in JSON encoding (#37657)
issue: ##36621

- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
jaime 1e8ea4a7e7
feat: add segment/channel/task/slow query render (#37561)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
congqixia f5b06a3c9f
enhance: Invalidate collection cache when release collection (#37577)
Related to #37395

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-12 10:16:29 +08:00
sthuang 70605cf5b3
enhance: Support custom privilege group for RBAC (#37087)
issue: #37031

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 08:44:28 +08:00
yihao.dai 994f52fab8
fix: Revert "enhance: Support db for bulkinsert (#37012)" (#37420)
This reverts commit 6e90f9e8d9.

issue: https://github.com/milvus-io/milvus/issues/31273

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-07 17:02:25 +08:00
jaime f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
jaime 9d16b972ea
feat: add tasks page into management WebUI (#37002)
issue: #36621

1. Add API to access task runtime metrics, including:
  - build index task
  - compaction task
  - import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
  - sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
yihao.dai b45cf2d49f
enhance: Add max length check for csv import (#37077)
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.

issue: https://github.com/milvus-io/milvus/issues/34150

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
yihao.dai 6e90f9e8d9
enhance: Support db for bulkinsert (#37012)
issue: https://github.com/milvus-io/milvus/issues/31273

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:31:39 +08:00
Gao 1d61b604e1
enhance: support retry search when topk is reduced and result not enough (#35645)
issue: #35576 

This pr is to cover those cases when queryHook optimize search params
and make the result size insufficient, add retry search mechanism and
add related metrics for alarming.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-23 19:19:30 +08:00
jaime 4746f47282
feat: management WebUI homepage (#36822)
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
XuanYang-cn 2e7670908f
enhance: Track complex delete rates (#36957)
See also: #36953

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-18 17:47:30 +08:00
congqixia e4292569c7
enhance: Unify InboundLabel case using metrics.Label (#36613)
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 14:15:17 +08:00
cai.zhang ecb2b242e2
enhance: Add sorted for segment info (#36469)
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
Ted Xu e7ea1d7a04
enhance: improve log encoding performance on proxy nodes (#36123)
See #36122

This PR is designed to enhance log performance through two improvements:

1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-11 14:51:07 +08:00
congqixia 51870aef08
enhance: Use stats Handler to record request/response size metrics (#36107)
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
jaime 24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935)
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
Zhen Ye 99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406)
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
SimFG 3e1052f889
enhance: use the msg position obj when getting replicate channel position (#35606)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 10:28:59 +08:00
congqixia 2fbc628994
feat: Support field partial load collection (#35416)
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
wei liu 14ef88f423
fix: Fix proxy panic after Import related api failed (#35540)
issue: #35539

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 15:22:55 +08:00
wei liu 1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444)
issue: #35443

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
SimFG b2cc4b0776
feat: add the rbac msg and send them to the replicate channel (#35392)
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
congqixia de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00