9b916f2cf8
enhance: [2.5] Skip creating partition rate limiters when not enable ( #38911 )
...
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38816
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 22:19:02 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
d0a8110a7a
enhance: alterdatabase support delete property ( #38380 )
...
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 17:04:43 +08:00
1cc9cb49ad
enhance: allow to delete data when disk quota exhausted ( #37134 )
...
- issue: #37133
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-25 16:47:29 +08:00
30121a5a0d
fix: Rectify delete buffer row count quota value ( #37060 )
...
Related to #37057
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-23 10:15:29 +08:00
0516624ab0
enhance: Log when limit writing by deletion ( #36954 )
...
See also: #36953
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-18 17:25:25 +08:00
1bd3228635
enhance: Ignore db not found error in quota center ( #36821 )
...
In quota center, ignore the "DB not found error" to prevent it from
affecting the rate limiting of other databases.
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-15 15:55:22 +08:00
116d7c18cc
fix: Use metrics values instead of hardcode mock values ( #36782 )
...
Resolves #36781
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-11 20:09:20 +08:00
8593c4580a
enhance: Add delete buffer related quota logic ( #35918 )
...
See also #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 11:39:03 +08:00
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
b0ac04d104
enhance: set database properties to restrict read access ( #35745 )
...
issue: #35744
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
56b3490a98
fix: Fix rate wasn't limited to the expected value ( #35699 )
...
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.
issue: https://github.com/milvus-io/milvus/issues/35675 ,
https://github.com/milvus-io/milvus/issues/35702
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-28 14:13:00 +08:00
d8aa01bc1a
fix: inconsistent meta view causes rate limit invalid ( #35665 )
...
issue: #35663
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 11:21:04 +08:00
b5022b0647
enhance: add metric to record disk quota ( #35305 )
...
https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:26:16 +08:00
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 )
...
fix: https://github.com/milvus-io/milvus/issues/35177
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:14:13 +08:00
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states ( #34960 )
...
/kind improvement
fix : #34961
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 11:39:46 +08:00
67324eb809
enhance: Add l0 segment entry num quota ( #34733 )
...
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
60be454db0
enhance: add disk quota and max collections into db properties ( #34368 )
...
issue: #34385
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
44d7e03e56
fix: reset the RootCoordQuotaStates metric before recording this metric ( #33553 )
...
- issue: #33539
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-03 21:09:47 +08:00
1973ebbc42
fix: quotaCenter collect metrics failed because the wrong param ( #33398 )
...
- issue: #33397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 18:17:42 +08:00
dd0c6d6980
fix: the panic when db isn't existed in the rate limit interceptor ( #33244 )
...
issue: #33243
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-22 15:57:39 +08:00
31a29a2451
enhance: avoid the repeated metric info in the proxy ( #32380 )
...
issue: #30577
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:21:20 +08:00
7add7248e2
fix: remove useless log of quota center ( #32307 )
...
/kind improvement
issue: #31272
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-17 16:17:19 +08:00
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
9111ecab82
enhance: Rootcoord's stop may block in quota_center's stop ( #31447 )
...
this PR fixed that rootcoord's stop may block in quota_center' stop
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 21:09:18 +08:00
dc2cdbe387
enhance: add more metrics ( #31271 )
...
/kind improvement
fix : #31272
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
af315539d4
fix: Remove time tick delay metrics when nodes go offline ( #30833 )
...
See also #30832
This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode
Also add unit tests for this case
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
e41fd6fbde
enhance: Move proxy client manager to util package ( #28955 )
...
issue: #28898
This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
c16fc854d1
Fix confusing quota log ( #28262 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-13 15:38:18 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
6ee5fedfef
Fix lock not applied in quota center ( #27133 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-15 19:11:25 +08:00
7624c2b949
Improve rated log in insertBufferNode ( #26788 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 09:59:10 +08:00
c6024a32f5
Add rate limit ratio monitoring ( #26709 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-03 20:47:02 +08:00
b6effd7345
Disable deny writing when the growing segment size exceeds the watermark ( #26163 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-08 12:43:07 +08:00
74ad7c4601
Fix monitoring metrics ( #25549 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-16 18:48:33 +08:00
d63323d117
Add rate limit and deny write in upsert ( #25351 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-07-11 11:20:34 +08:00
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
0d51f55c48
enable config each collection has different rate limit ( #24139 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-21 15:02:42 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
44e5daae3a
Fix querynode read quota ( #24412 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-06-08 17:24:36 +08:00
0fb9131973
Improve rate limiter log ( #24459 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-30 11:23:28 +08:00
1471da846d
Fix panic due to collection not exist in quota effect ( #24320 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-24 20:53:30 +08:00
6cfb4559d2
fix deny writing use wrong errcode ( #24152 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-05-17 14:23:23 +08:00
7384d83d2c
Support rate limit based on growing segments size ( #24121 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-17 09:57:22 +08:00
4c956fab73
enable config collection level rate limit ( #24012 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-05-12 18:13:26 +08:00
5a9a86ba74
Fix quota center assignment to entry in nil map ( #23819 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-29 18:22:38 +08:00
537e068fb9
fix qupta_center calculate negative memory factor for rate limit ( #23750 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-28 11:32:35 +08:00
yihao.dai
Zhen Ye
Xianhui Lin
SimFG
congqixia
XuanYang-cn
jaime
Jiquan Long
wei liu
smellthemoon
aoiasd