8af187f673
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics( #30376 ) ( #34477 )
...
related: #30376
fix: paritionIDs lost when no setting paritions
enhance: refine metrics for segment prune
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-08 19:54:15 +08:00
ae6d6f91e6
enhance: change autoindex default metric type ( #34261 )
...
issue: #34304
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 19:52:24 +08:00
145f4f919b
fix: Add `nbits` parameter check for IVF_PQ ( #34451 )
...
See also #34426
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-08 11:16:10 +08:00
43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs ( #34438 )
...
Check if the segment exists during FlushSegments and add some key logs
in write path.
issue: https://github.com/milvus-io/milvus/issues/34255
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-06 08:50:11 +08:00
60be454db0
enhance: add disk quota and max collections into db properties ( #34368 )
...
issue: #34385
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
18f2458385
fix: fix bitmap supported type ( #34350 )
...
#34314
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-05 15:50:10 +08:00
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation ( #34421 )
...
issue: #34420
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
0817802db8
enhance: use the key lock and concurrent map in the msg dispatcher client ( #34278 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-05 15:11:31 +08:00
ba04981a43
enhance: implement wal managerment on streaming node ( #34153 )
...
issue: #33285
- add lifetime control for wal.
- implement distributed-safe wal manager on streaming node.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-05 14:29:42 +08:00
7611128e57
enhance: wal adaptor implementation ( #34122 )
...
issue: #33285
- add adaptor to implement walimpls into wal interface.
- implement timetick sorted and filtering scanner.
- add test for wal.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 15:23:08 +08:00
c6e2900cfc
enhance: add the tick log for the tt msgstream seek method ( #34397 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-04 10:18:09 +08:00
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize ( #34201 )
...
See also: #34136
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-04 10:16:17 +08:00
9db897985b
enhance: refine misleading param name for bloom filter parallel factor ( #34334 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-04 10:06:10 +08:00
0fd0fcfe1d
enhance: Fix lint issues & sdk testcase ( #34399 )
...
Some lint issue is not detect due to recent static check pipeline issue.
This PR fixes these problem and Go milvusclient testcases.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-03 19:42:10 +08:00
94fb5805d5
fix: Use raw parameter value to perform CAS ( #34343 )
...
See also #34342
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-03 16:00:08 +08:00
8165961223
enhance: Optimeize clustering compaction ( #34313 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:14:08 +08:00
32c943f1bc
enhance: [skip e2e] update the version of MV ( #34378 )
...
issue: #29892
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-03 14:28:16 +08:00
34bec2ea5e
enhance: add metrics for segment prune latnecy( #30376 ) ( #34094 )
...
related: #30376
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-03 10:04:07 +08:00
fa8d641ce6
fix: update check for sparse hnsw index ( #33713 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-02 21:56:09 +08:00
3563136c2a
enhance: timetick interceptor implementation ( #34238 )
...
issue: #33285
- optimize the message package
- add interceptor package to achieve append operation intercepting.
- add timetick interceptor to attach timetick properties for message.
- add timetick background task to send timetick message.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-02 14:42:08 +08:00
a5be322ab2
enhance: Don't set the timeout for SyncSegments ( #34226 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 14:28:07 +08:00
ff51c7e628
fix: Fix panic due to failed to seek ( #34229 )
...
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.
issue: https://github.com/milvus-io/milvus/issues/34221
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-01 16:08:12 +08:00
186757e622
enhance: support mark error as user error ( #33498 )
...
relate: https://github.com/milvus-io/milvus/issues/33492
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
73ffc1b424
Refine clustering_compaction_task retry mechanism ( #34194 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:22:09 +08:00
dda70aa81b
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 ( #34184 )
...
See also: #31933
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-27 19:36:06 +08:00
d2bc4a53be
enhance: implement rmq and pulsar as wal ( #34046 )
...
issue: #33285
- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call ( #34102 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
381d43717d
enhance: update new mmap config parmeters version ( #34143 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34145
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-25 21:20:05 +08:00
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
d08cb885ca
enhance: enable flush rate limiter of collection level ( #33837 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
ca7ef26e4b
fix: sync part stats task cannot be finished( #30376 ) ( #34027 )
...
related: #30376
also: refine log output for query_coord task by rephrasing action string
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-24 10:16:02 +08:00
259a682673
enhance: async search and retrieve in cgo ( #33228 )
...
issue: #30926 , #33132
related pr: #33133
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-22 09:38:02 +08:00
958ecd500b
feat: adding grouped virtual resource allocator ( #33669 )
...
See #33559
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-21 15:58:01 +08:00
cc77363b66
enhance: Set maxPartitionNum default value to 1024 ( #33949 )
...
See also #30059
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-20 09:11:59 +08:00
2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
298e50b834
enhance: check index with data type ( #33880 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
2f6f964bc8
enhance: [skip e2e] modify gc configuration document ( #33946 )
...
issue: #31740
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-19 09:57:59 +08:00
a789c60380
enhance: autoindex for multi data type ( #33868 )
...
issue: #22837
contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
3fdaae8792
fix: Return record with largest timestamp for entires with same PK ( #33936 )
...
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
b72026c859
enhance: Add rbac support on describe/alter database api ( #33803 )
...
issue: #32707
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 10:07:57 +08:00
d43ec4db0b
enhance: support array bitmap index ( #33527 )
...
#32900
---------
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:51:58 +08:00
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
c689ef4822
fix: Remove eviction operations out of lock ( #33834 )
...
See also #33823
`EvictCacheValueByFormat` may be block by on going `CASCacheValue` and
cause possible deadlock
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-13 21:15:55 +08:00
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33611 )
...
issue:#33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 17:57:56 +08:00
be3559e022
enhance: proxy check hnsw with sparse is legal ( #33697 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33653
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-12 20:13:55 +08:00
Chun Han
Gao
congqixia
yihao.dai
jaime
zhagnlu
SimFG
chyezh
XuanYang-cn
wei liu
cai.zhang
Patrick Weizhi Xu
Buqian Zheng
aoiasd
wayblink
cqy123456
Ted Xu
smellthemoon