a570567644
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress ( #35472 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 14:18:54 +08:00
f4a91e135b
enhance: Allow empty sparse row ( #34700 )
...
issue: #29419
* If a sparse vector with 0 non-zero value is inserted, no ANN search on
this sparse vector field will return it as a result. User may retrieve
this row via scalar query or ANN search on another vector field though.
* If the user uses an empty sparse vector as the query vector for a ANN
search, no neighbor will be returned.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-08-16 14:14:54 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
c6ae7d4d56
enhance: adding the msgchannel section in generated yaml ( #35466 )
...
See #32168
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-14 18:32:53 +08:00
f99a761d87
enhance: Support dynamic field in SchemaHelper ( #35461 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-14 18:04:59 +08:00
6c135e5ea9
fix: key of DataCoordTicklePath is wrong (issue: #35259 ) ( #35263 )
...
issue: [35259](https://github.com/milvus-io/milvus/issues/35259 )
Signed-off-by: Reid Chan <reidchan@apache.org>
2024-08-13 17:40:20 +08:00
16b0aee97f
enhance: timetick interceptor optimization ( #35287 )
...
issue: #33285
- remove redundant goroutine by using insepctor.
- remove the coutinous non-message timetick persistence
- periodically push the time tick forward without persistent timetick
message.
- add 'message type filter' deliver filter.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-12 18:58:25 +08:00
c4a240acb1
enhance: memory value are displayed in MB ( #35292 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-09 14:14:25 +08:00
344dc6a9f8
enhance: enable to set load config in cluster level ( #35169 )
...
issue: #35170
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
c725416288
enhance: move streaming proto into pkg ( #35284 )
...
issue: #33285
- move streaming related proto into pkg.
- add v2 message type and change flush message into v2 message.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-07 10:34:16 +08:00
b5022b0647
enhance: add metric to record disk quota ( #35305 )
...
https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:26:16 +08:00
885d891106
Change minio use virtual config to not panic ( #35327 )
...
minio use virtual should not panic if empty. User may use other object
storage instead
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-07 09:58:15 +08:00
a9352a073b
enhance: change gpu default mem pool size ( #34159 )
...
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-08-06 10:24:15 +08:00
14051fed7d
enhance: streaming service client ( #34656 )
...
issue: #33285
- implement streaming service client.
- implement producing and consuming service client by streaming coord
client and streaming node client.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-05 21:38:15 +08:00
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord ( #35139 )
...
issue: #35138
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
a6903701a4
enhance: Remove stackerror imported unexpected ( #35256 )
...
`stackerror` pkg is imported by accident and could be replaced by
cockroachdb errors lib. This PR removes import line and fix this
problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 14:48:16 +08:00
9871966415
enhance: segment alloc interceptor ( #34996 )
...
#33285
- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-04 07:40:15 +08:00
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection ( #34947 )
...
issue: #34946
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-02 17:20:15 +08:00
db06b86594
enhance: Sync otlp dependency version to fix security issue ( #35192 )
...
Related to #34434
otelgrpc DoS vulnerability due to unbound cardinality metrics
https://github.com/milvus-io/milvus/security/dependabot/91
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:22:20 +08:00
f466129924
enhance: add segment state on data coord bin log size metrics ( #35153 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:48:14 +08:00
3655ab10b2
fix: evict paramtable cache miss ( #34771 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-02 11:52:14 +08:00
c64a078458
enhance: Support proxy/delegator qn client pooling ( #35194 )
...
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
3b735b4b02
enhance: Refine param init for MmapDirPath ( #35181 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 11:12:14 +08:00
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 )
...
fix: https://github.com/milvus-io/milvus/issues/35177
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:14:13 +08:00
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
7721a28584
fix: Fix unstable TestDispatchToVchannels ut ( #35163 )
...
issue: https://github.com/milvus-io/milvus/issues/35150
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-01 14:56:13 +08:00
b4d0f4df0a
enhance: update param documents ( #35152 )
...
See: #35161
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-01 12:38:13 +08:00
e9d61daa3f
enhance: Reduce delegator memory overloaded factor to 0.1 ( #35092 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-01 10:21:50 +08:00
dfda9f0478
enhance: Add depguard rules to ban deprecated proto lib ( #35140 )
...
See also #34394 #34252
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 10:01:49 +08:00
6695c6d0a3
enhance: add channel num for queryHook optimization ( #35104 )
...
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-31 18:59:49 +08:00
ce3f836876
fix: compaction task not be cleaned correctly ( #34765 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:21:56 +08:00
39d25938c5
enhance: Avoid panic due to nil schema ( #35063 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/discussions/25620
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-30 20:18:32 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
972752258a
enhance: Support otlp http exporter ( #35053 )
...
See also #35052
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 17:43:49 +08:00
0e41f104c5
enhance: use the collection id to group msg pack in the msg dispatcher ( #34871 )
...
- issue: #34870
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-29 11:45:49 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
4a3d98d88c
enhance: Fast close msgdispatcher target ( #34803 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/34075
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-29 00:25:49 +08:00
5584ae2e14
enhance: add metrics for restful apis ( #34969 )
...
issue: #34968
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-29 00:13:47 +08:00
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time ( #34949 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
804ec24c02
fix:fix retrieve raw data from bitmap array index ( #34848 )
...
#34795
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-27 01:53:47 +08:00
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs ( #34763 )
...
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
63f0154dfb
fix: enable milvus.yaml check ( #34567 )
...
See #32168
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-07-25 18:05:46 +08:00
bef06e5acf
enhance: add streaming coord and node grpc service register ( #34655 )
...
issue: #33285
- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 12:19:44 +08:00
4f6cbfd520
enhance: specialized immutable and mutable message ( #34951 )
...
issue: #33285
- add specialized mutable and immutable message, make type safe.
- add version based constructor and type.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 11:57:45 +08:00
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states ( #34960 )
...
/kind improvement
fix : #34961
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 11:39:46 +08:00
166fc902b0
enhance: Limit collection's normal balance speed ( #34810 )
...
issue: #34798
after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-24 19:11:44 +08:00
a4d7da4973
feat: add ipv6 support ( #32960 )
...
related with #34917
This PR updates the func utils get IP address function to support
getting a IPv6 address if it is available and return it. I've tested
that this works locally when running inside a docker container that has
an IPv6 network address.
This PR should address
https://github.com/milvus-io/milvus/discussions/20217
Thank you for review!
Signed-off-by: David Pichler <david.pichler@grainger.com>
Co-authored-by: David Pichler <david.pichler@grainger.com>
2024-07-23 15:39:43 +08:00
f6c6e98ec5
enhance: recover stack info when non-cgo thread crash ( #34865 )
...
issue: #34864
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-23 14:25:44 +08:00
wei liu
Buqian Zheng
SimFG
Ted Xu
congqixia
Reid Chan
chyezh
smellthemoon
Jiquan Long
Xiaofan
presburger
cai.zhang
yihao.dai
jaime
aoiasd
Gao
wayblink
PowderLi
zhagnlu
David Pichler