7bbfe86bcd
enhance: add list index and segment index retrieval API for WebUI ( #37861 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-22 16:58:34 +08:00
19572f5b06
enhance: RBAC new grant/revoke privilege ( #37785 )
...
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843 ,
https://github.com/milvus-io/milvus/issues/37842 ,
https://github.com/milvus-io/milvus/issues/37887
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
33bfb25c73
enhance: refine meta cache log and logic ( #37318 )
...
related to #36989
add more logs in proxy meta cache and make it clearer
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
12ed40e125
fix: some missing metrics for WebUI restful API ( #37747 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-18 14:00:32 +08:00
1d06d4324b
fix: Int64 overflow in JSON encoding ( #37657 )
...
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
1e8ea4a7e7
feat: add segment/channel/task/slow query render ( #37561 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
f5b06a3c9f
enhance: Invalidate collection cache when release collection ( #37577 )
...
Related to #37395
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-12 10:16:29 +08:00
70605cf5b3
enhance: Support custom privilege group for RBAC ( #37087 )
...
issue: #37031
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 08:44:28 +08:00
994f52fab8
fix: Revert "enhance: Support db for bulkinsert ( #37012 )" ( #37420 )
...
This reverts commit 6e90f9e8d9https://github.com/milvus-io/milvus/issues/31273
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-07 17:02:25 +08:00
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI ( #37344 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
b45cf2d49f
enhance: Add max length check for csv import ( #37077 )
...
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
6e90f9e8d9
enhance: Support db for bulkinsert ( #37012 )
...
issue: https://github.com/milvus-io/milvus/issues/31273
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:31:39 +08:00
1d61b604e1
enhance: support retry search when topk is reduced and result not enough ( #35645 )
...
issue: #35576
This pr is to cover those cases when queryHook optimize search params
and make the result size insufficient, add retry search mechanism and
add related metrics for alarming.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-23 19:19:30 +08:00
4746f47282
feat: management WebUI homepage ( #36822 )
...
issue: #36784
1. Implement an embedded web server for WebUI access.
2. Complete the homepage development.
Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353 ">
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
2e7670908f
enhance: Track complex delete rates ( #36957 )
...
See also: #36953
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-18 17:47:30 +08:00
e4292569c7
enhance: Unify InboundLabel case using metrics.Label ( #36613 )
...
Previous label case broken by #36107 , this PR make all inbound label
using label constants from metrics package.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 14:15:17 +08:00
ecb2b242e2
enhance: Add sorted for segment info ( #36469 )
...
issue: #33744
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
e7ea1d7a04
enhance: improve log encoding performance on proxy nodes ( #36123 )
...
See #36122
This PR is designed to enhance log performance through two improvements:
1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-11 14:51:07 +08:00
51870aef08
enhance: Use stats Handler to record request/response size metrics ( #36107 )
...
Related to #36102
This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
3e1052f889
enhance: use the msg position obj when getting replicate channel position ( #35606 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 10:28:59 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
14ef88f423
fix: Fix proxy panic after Import related api failed ( #35540 )
...
issue: #35539
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 15:22:55 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
783f9d9c33
fix: Unify hook singleton implementation in proxy ( #34887 )
...
Related to #34885
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs ( #34763 )
...
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
88c38a3756
fix: `execute` method should return an error when result is a failure ( #34813 )
...
- issue: #34812
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:13:40 +08:00
c8bf6c8a46
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log ( #34738 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-18 15:17:41 +08:00
b5225df21c
enhance: Use unified requery flag when set metrics ( #34632 )
...
See also #34601
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 23:03:38 +08:00
104d0966b7
feat: support partition key isolation ( #34336 )
...
issue: #34332
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
186757e622
enhance: support mark error as user error ( #33498 )
...
relate: https://github.com/milvus-io/milvus/issues/33492
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
96dcee5dff
fix:load major compaction partial result( #34051 ) ( #34052 )
...
related: #34051
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
3540eee977
enhance: Support L0 import ( #33514 )
...
issue: https://github.com/milvus-io/milvus/issues/33157
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
25080bb455
enhance: Print `UseDefaultConsistency` param in read requests ( #33617 )
...
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 10:27:50 +08:00
2964f60edc
enhance: the proxy metric in the query request ( #33307 )
...
/kind improvement
issue: #33306
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-24 14:19:41 +08:00
1b4e28b97f
enhance: Check by proxy rate limiter when delete get data by query. ( #30891 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-23 20:03:40 +08:00
9ff023ee35
fix: Fix filtering by partition key fails for importing data ( #33274 )
...
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.
issue: https://github.com/milvus-io/milvus/issues/33237
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 11:13:40 +08:00
6fc00e42d7
fix: Fix binlog import ( #33027 )
...
binlog import should only import into one partition.
issue: https://github.com/milvus-io/milvus/issues/32807
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-14 21:45:32 +08:00
c35797c399
enhance: expose DescribeDatabase api in proxy ( #32732 )
...
issue: #32707
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
ba02d54a30
enhance: update shard leader cache when leader location changed ( #32470 )
...
issue: #32466
this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00
53874ce245
fix: Fix cannot specify partition name in binlog import ( #32730 )
...
issue: https://github.com/milvus-io/milvus/issues/32807
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-07 17:19:30 +08:00
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
07720f1a95
enhance: expose alter database api in proxy ( #32639 )
...
issue: #30040
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 18:15:31 +08:00
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation ( #32611 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
8594b55ad5
enhance: add `max insert request size` and `must use partition key` configs ( #32433 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:31:20 +08:00
jaime
sthuang
Xiaofan
congqixia
yihao.dai
Gao
XuanYang-cn
cai.zhang
Ted Xu
Zhen Ye
SimFG
wei liu
Patrick Weizhi Xu
aoiasd
Chun Han
Bingyi Sun