07bb071aaf
enhance: add the unmarshal method for the rbac msg ( #35692 )
...
/kind improvement
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 07:23:00 +08:00
b0ac04d104
enhance: set database properties to restrict read access ( #35745 )
...
issue: #35744
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
4d2f96c760
enhance: support bitmap mmap ( #35399 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-27 16:34:59 +08:00
098c10922b
enhance: add some check when create collection ( #35629 )
...
not support nullable==true in vector type. check it when create
collection.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-25 16:23:05 +08:00
50fcfe8ef1
enhance: add nan and inf check ( #35683 )
...
fix #35594
add float check on files
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-25 15:22:57 +08:00
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data ( #35498 )
...
#35458
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-24 01:40:58 +08:00
9dc1311f38
enhance: add the param to control whether to include the current msg ( #35656 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-23 15:56:57 +08:00
e8e3544a11
enhance: add hit segment num metrics for queryHook ( #35577 )
...
issue: #35576
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-23 12:49:02 +08:00
ba6db117e3
enhance: add some integration test in null ( #35599 )
...
#31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-21 17:44:56 +08:00
cf242f9e09
fix: fix dynamic update config doesn't works for some param ( #35572 )
...
issue: #35570
milvus support config cache to spped up config access, but only evict
param's cache when param has been updated. but milvus's param may rely
on other param's value, let's say ParamsA relys on paramsB, when paramsB
updated, it will evict paramB's cache, but the paramA's cache still keep
the old value.
This PR evict all config cache to solve the above issue, cause dynamic
update config won't be much frequetly.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-21 11:02:56 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
41646c8439
feat: integrate new deltalog format ( #35522 )
...
See #34123
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-20 19:06:56 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2( #35350 ) ( #35470 )
...
related: #35350
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:16:55 +08:00
4d69898cb2
enhance: support single pchannel level transaction ( #35289 )
...
issue: #33285
- support transaction on single wal.
- last confirmed message id can still be used when enable transaction.
- add fence operation for segment allocation interceptor.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-19 21:22:56 +08:00
853d2f33e0
enhance: Trim space during parsing string list from string ( #35546 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 20:48:55 +08:00
2736a8b88c
enhance: Update Makefile to generate mockery ( #35517 )
...
Some mockery cmd is out-of-date and fail to work. This PR update these
commands to match current pkg.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 18:44:53 +08:00
ead37326f0
enhance: Add backup/restore privilege to admin group ( #35515 )
...
issue: #35443
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 18:42:54 +08:00
a570567644
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress ( #35472 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 14:18:54 +08:00
f4a91e135b
enhance: Allow empty sparse row ( #34700 )
...
issue: #29419
* If a sparse vector with 0 non-zero value is inserted, no ANN search on
this sparse vector field will return it as a result. User may retrieve
this row via scalar query or ANN search on another vector field though.
* If the user uses an empty sparse vector as the query vector for a ANN
search, no neighbor will be returned.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-08-16 14:14:54 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
c6ae7d4d56
enhance: adding the msgchannel section in generated yaml ( #35466 )
...
See #32168
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-14 18:32:53 +08:00
f99a761d87
enhance: Support dynamic field in SchemaHelper ( #35461 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-14 18:04:59 +08:00
6c135e5ea9
fix: key of DataCoordTicklePath is wrong (issue: #35259 ) ( #35263 )
...
issue: [35259](https://github.com/milvus-io/milvus/issues/35259 )
Signed-off-by: Reid Chan <reidchan@apache.org>
2024-08-13 17:40:20 +08:00
16b0aee97f
enhance: timetick interceptor optimization ( #35287 )
...
issue: #33285
- remove redundant goroutine by using insepctor.
- remove the coutinous non-message timetick persistence
- periodically push the time tick forward without persistent timetick
message.
- add 'message type filter' deliver filter.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-12 18:58:25 +08:00
c4a240acb1
enhance: memory value are displayed in MB ( #35292 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-09 14:14:25 +08:00
344dc6a9f8
enhance: enable to set load config in cluster level ( #35169 )
...
issue: #35170
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
c725416288
enhance: move streaming proto into pkg ( #35284 )
...
issue: #33285
- move streaming related proto into pkg.
- add v2 message type and change flush message into v2 message.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-07 10:34:16 +08:00
b5022b0647
enhance: add metric to record disk quota ( #35305 )
...
https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:26:16 +08:00
885d891106
Change minio use virtual config to not panic ( #35327 )
...
minio use virtual should not panic if empty. User may use other object
storage instead
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-07 09:58:15 +08:00
a9352a073b
enhance: change gpu default mem pool size ( #34159 )
...
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-08-06 10:24:15 +08:00
14051fed7d
enhance: streaming service client ( #34656 )
...
issue: #33285
- implement streaming service client.
- implement producing and consuming service client by streaming coord
client and streaming node client.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-05 21:38:15 +08:00
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord ( #35139 )
...
issue: #35138
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
a6903701a4
enhance: Remove stackerror imported unexpected ( #35256 )
...
`stackerror` pkg is imported by accident and could be replaced by
cockroachdb errors lib. This PR removes import line and fix this
problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 14:48:16 +08:00
9871966415
enhance: segment alloc interceptor ( #34996 )
...
#33285
- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-04 07:40:15 +08:00
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection ( #34947 )
...
issue: #34946
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-02 17:20:15 +08:00
db06b86594
enhance: Sync otlp dependency version to fix security issue ( #35192 )
...
Related to #34434
otelgrpc DoS vulnerability due to unbound cardinality metrics
https://github.com/milvus-io/milvus/security/dependabot/91
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:22:20 +08:00
f466129924
enhance: add segment state on data coord bin log size metrics ( #35153 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:48:14 +08:00
3655ab10b2
fix: evict paramtable cache miss ( #34771 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-02 11:52:14 +08:00
c64a078458
enhance: Support proxy/delegator qn client pooling ( #35194 )
...
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
3b735b4b02
enhance: Refine param init for MmapDirPath ( #35181 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 11:12:14 +08:00
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 )
...
fix: https://github.com/milvus-io/milvus/issues/35177
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:14:13 +08:00
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
7721a28584
fix: Fix unstable TestDispatchToVchannels ut ( #35163 )
...
issue: https://github.com/milvus-io/milvus/issues/35150
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-01 14:56:13 +08:00
b4d0f4df0a
enhance: update param documents ( #35152 )
...
See: #35161
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-01 12:38:13 +08:00
e9d61daa3f
enhance: Reduce delegator memory overloaded factor to 0.1 ( #35092 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-01 10:21:50 +08:00
dfda9f0478
enhance: Add depguard rules to ban deprecated proto lib ( #35140 )
...
See also #34394 #34252
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 10:01:49 +08:00
SimFG
jaime
Zhen Ye
zhagnlu
smellthemoon
Xiaofan
Gao
wei liu
Ted Xu
congqixia
Chun Han
Buqian Zheng
Reid Chan
Jiquan Long
presburger
cai.zhang
yihao.dai
aoiasd