326370c1be
enhance: add disk quota and max collections into db properties ( #34386 )
...
issue: https://github.com/milvus-io/milvus/issues/34385
pr: #34368
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
0992f10694
enhance: improve check health ( #34265 )
...
issue: https://github.com/milvus-io/milvus/issues/34264
pr: #33800
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:18:07 +08:00
766f47c9ca
fix: [2.4] reset the RootCoordQuotaStates metric before recording this metric ( #33601 )
...
- issue: #33539
- pr: #33553
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-04 20:17:51 +08:00
cbeb6c169d
fix: [2.4] quotaCenter collect metrics failed because the wrong param ( #33399 )
...
- issue: #33397
- pr: #33398
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 17:05:42 +08:00
37b2f90c90
enhance: [2.4] the panic when db isn't existed in the rate limit interceptor ( #33308 )
...
issue: #33243
pr: #33244
1. fix: the panic when db isn't existed in the rate limit interceptor
#33244
2. enhance: check the auth in some rest v2 api #33256
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-23 17:25:41 +08:00
31a29a2451
enhance: avoid the repeated metric info in the proxy ( #32380 )
...
issue: #30577
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:21:20 +08:00
7add7248e2
fix: remove useless log of quota center ( #32307 )
...
/kind improvement
issue: #31272
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-17 16:17:19 +08:00
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
9111ecab82
enhance: Rootcoord's stop may block in quota_center's stop ( #31447 )
...
this PR fixed that rootcoord's stop may block in quota_center' stop
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 21:09:18 +08:00
dc2cdbe387
enhance: add more metrics ( #31271 )
...
/kind improvement
fix : #31272
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
af315539d4
fix: Remove time tick delay metrics when nodes go offline ( #30833 )
...
See also #30832
This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode
Also add unit tests for this case
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
e41fd6fbde
enhance: Move proxy client manager to util package ( #28955 )
...
issue: #28898
This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
c16fc854d1
Fix confusing quota log ( #28262 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-13 15:38:18 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
6ee5fedfef
Fix lock not applied in quota center ( #27133 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-15 19:11:25 +08:00
7624c2b949
Improve rated log in insertBufferNode ( #26788 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 09:59:10 +08:00
c6024a32f5
Add rate limit ratio monitoring ( #26709 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-03 20:47:02 +08:00
b6effd7345
Disable deny writing when the growing segment size exceeds the watermark ( #26163 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-08 12:43:07 +08:00
74ad7c4601
Fix monitoring metrics ( #25549 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-16 18:48:33 +08:00
d63323d117
Add rate limit and deny write in upsert ( #25351 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-07-11 11:20:34 +08:00
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
0d51f55c48
enable config each collection has different rate limit ( #24139 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-21 15:02:42 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
44e5daae3a
Fix querynode read quota ( #24412 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-06-08 17:24:36 +08:00
0fb9131973
Improve rate limiter log ( #24459 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-30 11:23:28 +08:00
1471da846d
Fix panic due to collection not exist in quota effect ( #24320 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-24 20:53:30 +08:00
6cfb4559d2
fix deny writing use wrong errcode ( #24152 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-05-17 14:23:23 +08:00
7384d83d2c
Support rate limit based on growing segments size ( #24121 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-17 09:57:22 +08:00
4c956fab73
enable config collection level rate limit ( #24012 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-05-12 18:13:26 +08:00
5a9a86ba74
Fix quota center assignment to entry in nil map ( #23819 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-29 18:22:38 +08:00
537e068fb9
fix qupta_center calculate negative memory factor for rate limit ( #23750 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-28 11:32:35 +08:00
017b15af6a
Fix disk quota when there's no binlog file ( #23776 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-28 11:10:35 +08:00
ed8836cd15
Add disk quota at the collection level ( #23704 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-26 21:52:36 +08:00
4fb8919a97
support collection level rate limit ( #22767 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-25 15:54:35 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
6e66c10114
Fix check time tick lag with no channel ( #21737 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-16 18:49:43 +08:00
79535931ab
Return error code when deny to read/write ( #21561 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-01-06 14:31:37 +08:00
aa203acfb3
Low IndexCoord weight ( #21548 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-06 14:21:37 +08:00
c187de7754
Show trigger reason when deny to read/write ( #21368 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-30 18:35:32 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
51acba7d99
Add role name to ttDelay metric ( #20239 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-02 10:55:34 +08:00
bddd47c60f
Prevent limiter burst overflow ( #20153 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-28 19:05:32 +08:00
b949fa0ea9
Increase default ttProtection max delay ( #20125 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-27 17:37:32 +08:00
f0c616050a
Reduce and improve quota center log ( #20058 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-25 19:39:30 +08:00
581e1d8c35
Fill MsgBase ALL ( #19912 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
db0486c870
Fix missing import limit and rename functions in quotaCenter ( #19907 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-19 21:43:30 +08:00
99df2d9b54
Add binlogSize and ttDelay metrics monitor ( #19872 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-19 15:13:26 +08:00
jaime
SimFG
Jiquan Long
wei liu
congqixia
yihao.dai
smellthemoon
aoiasd
yah01
cai.zhang
Enwei Jiao