milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,271 Commits
152 Branches
122 Tags
588 MiB
Commit Graph

52 Commits (5f3601a6a5c43d13ab9f0478b90c343203abbaa1)

Author SHA1 Message Date
congqixia b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib (#37708) 
Related to #35020

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-18 10:46:31 +08:00
SimFG 1cc9cb49ad
enhance: allow to delete data when disk quota exhausted (#37134) 
- issue: #37133

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-25 16:47:29 +08:00
yihao.dai 1bd3228635
enhance: Ignore db not found error in quota center (#36821) 
In quota center, ignore the "DB not found error" to prevent it from
affecting the rate limiting of other databases.

/kind improvement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-15 15:55:22 +08:00
jaime 24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935) 
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-04 14:39:10 +08:00
jaime b0ac04d104
enhance: set database properties to restrict read access (#35745) 
issue: #35744

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-29 13:17:01 +08:00
jaime d8aa01bc1a
fix: inconsistent meta view causes rate limit invalid (#35665) 
issue: #35663

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-28 11:21:04 +08:00
jaime 60be454db0
enhance: add disk quota and max collections into db properties (#34368) 
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 18:22:17 +08:00
SimFG 31a29a2451
enhance: avoid the repeated metric info in the proxy (#32380) 
issue: #30577

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-19 10:21:20 +08:00
SimFG c012e6786f
feat: support rate limiter based on db and partition levels (#31070) 
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-04-12 16:01:19 +08:00
wei liu 9111ecab82
enhance: Rootcoord's stop may block in quota_center's stop (#31447) 
this PR fixed that rootcoord's stop may block in quota_center' stop

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-01 21:09:18 +08:00
congqixia af315539d4
fix: Remove time tick delay metrics when nodes go offline (#30833) 
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
 2024-02-28 10:10:56 +08:00
wei liu e41fd6fbde
enhance: Move proxy client manager to util package (#28955) 
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-20 19:22:42 +08:00
yah01 b8674811cf
fix: data race in ProxyClientManager (#29206) 
this PR changed the ProxyClientManager to thread-safe
fix #29205

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-14 18:22:39 +08:00
wei liu fe1eeae2aa
enhance: Use mockery to replace manual mock code (#29074) 
issue: #29043
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-13 10:46:44 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
wei liu cd7f811d40
fix rootcoord unstable ut (#26460) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-30 09:42:27 +08:00
congqixia b9850ce5c0
Fix copylocks linter errors (#26217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 20:41:15 +08:00
yihao.dai b6effd7345
Disable deny writing when the growing segment size exceeds the watermark (#26163) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-08 12:43:07 +08:00
yiwangdr b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
smellthemoon d63323d117
Add rate limit and deny write in upsert (#25351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-07-11 11:20:34 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
wei liu 0d51f55c48
enable config each collection has different rate limit (#24139) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-21 15:02:42 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
yihao.dai 1471da846d
Fix panic due to collection not exist in quota effect (#24320) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-24 20:53:30 +08:00
wei liu 6cfb4559d2
fix deny writing use wrong errcode (#24152) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-17 14:23:23 +08:00
yihao.dai 7384d83d2c
Support rate limit based on growing segments size (#24121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-17 09:57:22 +08:00
wei liu 4c956fab73
enable config collection level rate limit (#24012) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-12 18:13:26 +08:00
yihao.dai 5a9a86ba74
Fix quota center assignment to entry in nil map (#23819) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-29 18:22:38 +08:00
wei liu 537e068fb9
fix qupta_center calculate negative memory factor for rate limit (#23750) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-28 11:32:35 +08:00
yihao.dai 017b15af6a
Fix disk quota when there's no binlog file (#23776) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-28 11:10:35 +08:00
yihao.dai ed8836cd15
Add disk quota at the collection level (#23704) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-26 21:52:36 +08:00
wei liu 4fb8919a97
support collection level rate limit (#22767) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-25 15:54:35 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
wei liu 13b508103e
refine mock querycoord (#22198) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-16 15:38:34 +08:00
yah01 6e66c10114
Fix check time tick lag with no channel (#21737) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-16 18:49:43 +08:00
bigsheeper 79535931ab
Return error code when deny to read/write (#21561) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-01-06 14:31:37 +08:00
bigsheeper c187de7754
Show trigger reason when deny to read/write (#21368) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-30 18:35:32 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
bigsheeper f0c616050a
Reduce and improve quota center log (#20058) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-10-25 19:39:30 +08:00
bigsheeper db0486c870
Fix missing import limit and rename functions in quotaCenter (#19907) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-10-19 21:43:30 +08:00
bigsheeper 99df2d9b54
Add binlogSize and ttDelay metrics monitor (#19872) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-10-19 15:13:26 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
bigsheeper 3a5aaeb7ad
Fix read queue metrics and memory protection (#19787) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-10-14 18:05:24 +08:00
bigsheeper f1e330a997
Add read result rate protection (#19728) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-10-13 14:57:24 +08:00
bigsheeper e6c76aba0e
Force deny writing if disk quota exceeded (#19680) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-10-12 10:03:22 +08:00
bigsheeper 804c18df68
Update quota params (#19351) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-09-26 16:48:53 +08:00