75d7978a18
enhance: pass partition key scalar info if enable for vector mem index ( #39123 )
...
issue: #34332
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-16 14:33:03 +08:00
1f6fd54146
fix: [skip e2e] Make test case assigner result deterministic ( #39317 )
...
Related to #39296
The case initialized with {100:8 ,101: 16}. After first assignment, the
slots become {100:8, 101:8} and the following result is not stable.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 14:25:02 +08:00
8c4ba70a4c
fix: enable to build index with single segment ( #39233 )
...
fix https://github.com/milvus-io/milvus/issues/39232
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 11:01:06 +08:00
bca2a62b78
enhance: Handle PutOrRef collection schema failure error ( #39310 )
...
Related to previous pr #39279
When NewCollection returns nil, the error shall be returned and handled
by caller
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 10:13:06 +08:00
57e5652f1a
enhance: Log error instead of panicking if load lock wait timeout ( #39308 )
...
Related to #39205
Previous PR #39206
This PR change wait timeout behavior to log error and return to avoid
making other collection read failure in only some collections have
deadlock
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 02:31:02 +08:00
a5a83a0904
fix: Fix consume blocked due to too many consumers ( #38455 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 21:37:01 +08:00
657550cf06
fix: Fix slow dist handle and slow observe ( #38566 )
...
1. Provide partition&channel level indexing in the collection target.
2. Make `SegmentAction` not wait for distribution.
3. Remove scheduler and target manager mutex.
4. Optimize logging to reduce CPU overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:17:00 +08:00
38881bf591
enhance: prevent multiple query nodes from causing excessive occupancy of a single node, leading to GPU memory overflow ( #39276 ) ( #38617 )
...
issue: #39276
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2025-01-15 20:15:01 +08:00
0df2c75b77
fix: Fix rootcoord meta mutex contention ( #38799 )
...
RootCoord meta uses copy-on-write, allowing the removal of unnecessary
copies.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 20:11:08 +08:00
82bdf9a6a8
fix: Add index param duplication check ( #39289 )
...
Related to #39288
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-15 19:23:01 +08:00
eb63334312
enhance: Add try-catch and return CStatus for NewCollection ( #39279 )
...
Related to #28795
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-15 19:17:01 +08:00
d2834a1812
enhance: Add logs for check health failed ( #39208 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-15 17:31:00 +08:00
7addeea70c
fix: wrong streaming mockery package name ( #39260 )
...
issue: #39095
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-15 15:51:00 +08:00
ce41778fe6
enhance: Optimize GetLocalDiskSize and segment loader mutex ( #38599 )
...
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 15:45:01 +08:00
e501025bba
enhance: simplify compaction tasks to reduce their memory overhead ( #39121 )
...
See #39080
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-15 14:51:00 +08:00
f5234c3c11
enhance: Remove mutex from SegmentManger in querynode ( #39051 )
...
Remove mutex from SegmentManger in querynode to prevent mutex
contention.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 12:45:00 +08:00
e8f76cd2d9
fix: unstable ut in leader_vew_manager.go file ( #39161 )
...
issue: #38672
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-15 12:26:59 +08:00
6d45dd5666
fix: Add scalar index engine version for compatibility ( #39204 )
...
issue: #39203
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-15 12:25:00 +08:00
3a6408b237
fix: Record a map to avoid repeatedly traversing the CompactionFrom ( #38925 )
...
issue: #38811
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-15 10:02:58 +08:00
ed31a5a4bf
enhance: fix inconsistenty of alias and db for query iterator( #39045 ) ( #39216 )
...
related: #39045
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-01-15 09:48:59 +08:00
55dcac375c
[automated] Update Knowhere Commit ( #39263 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-01-15 02:52:59 +08:00
ec2e77b5d7
enhance: Reduce memory usage of BF in DataNode and QueryNode ( #38129 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 01:59:01 +08:00
272d95ad79
enhance: Reduce mutex contention in datacoord meta ( #38219 )
...
1. Using secondary index to avoid retrieving all segments at
`GetSegmentsChanPart`.
2. Perform batch SetAllocations to reduce the number of times the meta
lock is acquired.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 01:15:02 +08:00
5bf1b2b929
feat: Support Int8Vector in go ( #38990 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-14 20:43:06 +08:00
d89768f9e0
enhance: Unify LoadStateLock RLock & PinIf ( #39206 )
...
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-14 18:38:59 +08:00
da07993082
enhance: report auth log to hook when grpc authenticate failed ( #38943 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-14 17:47:04 +08:00
fd84ed817c
enhance: add broadcast operation for msgstream ( #39040 )
...
issue: #38399
- make broadcast service available for msgstream by reusing the
architecture streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:14:59 +08:00
da1b786ef8
enhance: Utilize "find0" in segment.find_first ( #39229 )
...
Related to #39003
Previous PR #39004 has to clone & flip bitset due to bitset does not
support find0 operator. #39176 added this feature so clone & flip could
be removed now.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-14 14:14:58 +08:00
3e788f0fbd
enhance: record memory size (uncompressed) item for index ( #38770 )
...
issue: #38715
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field `MemSize` (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
5e38f01e5b
enhance: update knowhere version ( #39212 )
...
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-01-14 10:21:05 +08:00
3447ff7310
enhance: [bitset] extend op_find() to be able to search both 0 and 1 ( #39176 )
...
issue: #39124
`bitset::find_first()` and `bitset::find_next()` now accept one more
parameter, which allows to search for `0` bit instead of `1` bit
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2025-01-14 09:50:58 +08:00
702347bbfd
fix: Fix incorrect memory estimation for small segments ( #38813 )
...
Skip estimation index memory logic for segments without index file.
issue: https://github.com/milvus-io/milvus/issues/37630
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-13 21:49:12 +08:00
a00ba861a4
fix: Fix in filter search result is empty if pk type is varchar ( #39106 )
...
https://github.com/milvus-io/milvus/issues/39107
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-13 16:14:58 +08:00
accc9e7fbf
fix: fail to get empty index num rows ( #39155 )
...
#39125
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-13 16:04:58 +08:00
5c5948cb70
fix: rbac custom group privilege level check ( #39164 )
...
related: https://github.com/milvus-io/milvus/issues/39086
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-13 11:24:59 +08:00
5f94954bb4
fix: data race when accessing field_ when retrieving ( #39151 )
...
issue: #39148
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-13 11:23:04 +08:00
cc5d59392a
fix: channel unbalance during stopping balance progress ( #38971 )
...
issue: #38970
cause the stopping balance channel still use the row_count_based policy,
which may causes channel unbalance in multi-collection case.
This PR impl a score based stopping balance channel policy.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-13 11:21:06 +08:00
640a49ffb6
fix: fix chunk cache madvise when sparse raw data is mmaped ( #39145 )
...
instead of marking as not supported,
`ChunkedSparseFloatColumn::DataByteSize` can simply use the impl of
super class.
issue: https://github.com/milvus-io/milvus/issues/39158
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-01-13 10:34:57 +08:00
3cd74037db
fix: restore rbac with empty meta panic ( #39141 )
...
related: https://github.com/milvus-io/milvus/issues/38985
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-13 10:02:57 +08:00
2a02bbe3ee
enhance: Use template to remove unittest duplication ( #39144 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-13 09:58:57 +08:00
032292a432
feat: support phrase match query ( #38869 )
...
The relevant issue: https://github.com/milvus-io/milvus/issues/38930
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-12 20:24:58 +08:00
357eaf0d71
fix: use the object heap to keep the min ddl ts order ( #39118 )
...
issue: #39002
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-10 18:16:58 +08:00
826b726c86
fix: Prevent leader checker from generating excessive duplicate leader tasks ( #39000 )
...
issue: #39001
Background:
Segment Load Version: Each segment load request assigns a timestamp as
its version. When multiple copies of a segment are loaded on different
QueryNodes, the leader checker uses this version to identify the latest
copy and updates the routing table in the leader view to point to it.
Delegator Router Version: When a delegator builds a route to a QueryNode
that has loaded a segment, it also records the segment's version.
Router Table Update Logic: If the leader checker detects that the
version of a segment in the routing table does not match the version in
the worker, it updates the routing table to point to the QueryNode with
the latest version. Additionally, it updates the segment's load version
in the QueryNode during this process.
Issue:
When a channel is undergoing load balancing, the leader checker may sync
the routing table to a new delegator. This sync operation modifies the
segment's load version, which invalidates the routing in the old
delegator. Subsequently, the leader checker updates the routing table in
the old delegator, breaking the routing in the new delegator. This cycle
continues, causing repeated updates and inconsistencies.
Fix:
This PR introduces two changes to address the issue:
1. Use NodeID to verify whether the delegator's routing table needs an
update, avoiding unnecessary modifications.
2. Ensure compatibility by using the latest segment's load version as
the version recorded in the routing table.
These changes resolve the cyclic updates and prevent the leader checker
from generating excessive duplicate tasks, ensuring routing stability
across delegators during load balancing.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-10 14:12:57 +08:00
b8fca4f5c1
fix: Clustering compaction ignoring deltalogs ( #39132 )
...
See also: #39131
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-10 14:07:05 +08:00
4355b485e5
enhance: remove compaction parallelism control ( #39081 )
...
See #39080
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-10 13:23:00 +08:00
e5eb1159e2
fix: Refine the err msg of alter index invalid ( #39137 )
...
fix: Refine the err msg of alter index invalid
issue: https://github.com/milvus-io/milvus/issues/38517
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-10 12:08:57 +08:00
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
f03a85725a
enhance: add db name in replica ( #38672 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:40:59 +08:00
92a2d608ac
fix: Bulk insert failed when the nullable/default_value field is not exist ( #39063 )
...
#39036
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-09 19:27:03 +08:00
3bcdd92915
enhance: add broadcast for streaming service ( #39020 )
...
issue: #38399
- Add new rpc for transfer broadcast to streaming coord
- Add broadcast service at streaming coord to make broadcast message
sent automicly
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-09 16:24:55 +08:00
d6206ad2de
fix: Remove duplicated Macro definition ( #39076 )
...
Issue: #39102
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-09 15:26:56 +08:00
8abf6c9149
fix: build text index when loading field data ( #39070 )
...
fix: https://github.com/milvus-io/milvus/issues/39053
may fix https://github.com/milvus-io/milvus/issues/38644 which could be
caused by https://github.com/milvus-io/milvus/issues/39053
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-09 15:24:56 +08:00
aceb972963
enhance: update the expr static web page ( #39082 )
...
issue: #39083
/kind improvement
Three new functions of the static web page:
1. The input box can be expanded and scrolled if it exceeds the maximum
size
2. Input history
3. It will simply check whether the quotation marks and brackets appear
in pairs
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-08 21:14:57 +08:00
5a4bafccc6
enhance: Add resource group api on Restful ( #38885 )
...
issue: #38739
enable to manage resource group by Restful
- CreateResourceGroup
- DescribeResourceGroup
- UpdateResourceGroup
- ListResourceGroup
- DropResourceGroup
- TransferReplica
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-08 19:40:57 +08:00
a89ac861da
fix: not add index name in alter index properties ( #38979 )
...
#38967
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-08 15:02:55 +08:00
f0dae81494
fix: set iterative filter hint to false when no expr specified ( #39033 )
...
issue: https://github.com/milvus-io/milvus/issues/39013
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-08 12:56:56 +08:00
134952b6c5
enhance: refactor the streaming node log ( #39052 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-08 11:30:56 +08:00
47e7ea241e
enhance: Add log for case which target not update as expected ( #38944 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-01-07 17:45:03 +08:00
c5a7000a92
enhance: move streaming coord from datacoord to rootcoord ( #39007 )
...
issue: #38399
We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-07 17:42:57 +08:00
3dc95153b7
fix: build break under debug mode ( #38790 )
...
See #38435
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-07 17:36:56 +08:00
182cac03e5
enhance: Use bitset or instead of bitwise set ( #39037 )
...
Related to #39003
Copying bitset value bit by bit is slow and CPU heavy, this PR utilizes
bitset operator "|=" to accelerate this procedure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-07 15:02:56 +08:00
84f8047a86
fix: Fix Milvus build error ( #39008 )
...
Issue: #39005
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-07 14:22:56 +08:00
3739446a33
enhance: refine array view to optimize memory usage( #38736 ) ( #38808 )
...
related: #38736
700m data, array_length=10
non-mmap_offsets_uint64: 2.0G
mmap_offsets_uint64: 1.1G
mmap_offsets_uint32: 880MB
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-01-07 13:26:55 +08:00
72f5b85c05
enhance: Accelerate `find_first` by utilizing bitset simd methods ( #39004 )
...
Related to #39003
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-07 10:34:54 +08:00
8165044b6d
fix: fix query incorrect in case of concurrent delete ( #38991 )
...
#38961
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-06 15:14:54 +08:00
f0cddfd160
fix: Fix panic caused by removing directory ( #38622 )
...
https://github.com/milvus-io/milvus/issues/38604
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-06 10:54:54 +08:00
7128e36eb0
enhance: Use mvcc timestamp as guarantee ts if set ( #38980 )
...
When MvccTimestamp is set, it could be used as guarantee timestamp
directly instead of new ts allocated by scheduler reducing the waiting
time when delegator has tsafe lag
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-05 19:02:54 +08:00
bd280d6f98
fix: expression with capital AND and OR are not recognized ( #38927 )
...
fix https://github.com/milvus-io/milvus/issues/38864
AND, OR and NOT is not recognized in milvus parser
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-05 18:55:00 +08:00
9672eee78a
enhance: Increase the buffer capacity of notifyIndexChan to support concurrency ( #38957 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-05 18:53:01 +08:00
11bfc93683
[automated] Update Knowhere Commit ( #38993 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-01-04 01:16:53 +08:00
c6a7f48123
enhance: Fix using wrong pool for warmup ( #38941 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-03 17:50:54 +08:00
af08b5b311
enhance: Update Knowhere version ( #38942 )
...
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-01-03 14:28:53 +08:00
4245c5bed1
fix: text match panics when enable_match is set be false ( #38950 )
...
fix: https://github.com/milvus-io/milvus/issues/38949
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-03 14:20:55 +08:00
d7623ab635
enhance: extract the task step execution process ( #38323 )
...
- issue: #38322
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-03 12:36:54 +08:00
d9ebb70b10
fix: search req transfer correctly ( #38955 )
...
issue: #37899
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-03 10:04:54 +08:00
aa0a87eda7
fix: Block warmup submit if pool full in sync mode ( #38690 )
...
https://github.com/milvus-io/milvus/issues/38692
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-02 15:04:58 +08:00
d3a5282eaa
enhance: add param for tuning max VARCHAR length and restore limit to 65535 ( #38884 )
...
issue: #38882
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 8e740e004151dd2c11918aad2857c1c8d1bd98f5)
2025-01-02 14:42:53 +08:00
907fc24f85
enhance: support null expr ( #38772 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-02 14:16:54 +08:00
accfacf012
fix: [skip e2e]update webui logic ( #38755 )
...
issue: #36621
* update collection detail page logic
* fix data page segments list
* remove useless js file
2024-12-31 18:25:00 +08:00
11f8f4a378
enhance: add intent ctx to rootcoord init ( #38439 )
...
issue: #35917
add intent ctx to rootcoord initialization process
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
6b105837b4
enhance: add some apis in Restful ( #38733 )
...
add some apis in Restful #38709
- alter/drop collection properties
- alter/drop index properties
- alter collection field properties
- refresh load
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-12-31 11:28:51 +08:00
3822819942
enhance: Remove an undefined behavior in index writer ( #38657 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-31 10:42:52 +08:00
af0315f190
fix: Release compaction task lock when return function ( #38856 )
...
issue: #38851
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-30 22:44:49 +08:00
cb6eca8e91
fix: drop partition can not be successful if load failed ( #38793 )
...
fix #38649
when partition load failed, the partition drop will also fail due to the
wrong error message
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-12-30 19:42:52 +08:00
ba3c2e6fb1
fix: Only generate the index_null_offset file when the field support null value ( #38833 )
...
issue: #38832
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-12-30 18:02:52 +08:00
07c1f43d95
fix: the retrieve plan on heap is used after free when reduce ( #38840 )
...
issue: #38731
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 17:50:51 +08:00
56c5b66619
fix: the close operation of rmq consumer is not sync ( #38734 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 16:04:51 +08:00
dd81a13821
fix: used wrong label for query received_nq metrics ( #37987 )
...
issue: #37982
Signed-off-by: Reid Chan <reidchan@apache.org>
2024-12-30 11:10:52 +08:00
2557e3f2a9
enhance: Initialize field id to avoid negative number ( #38789 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-27 18:00:50 +08:00
4ba0ed3178
enhance: speed up streaming barrier timetick ( #38787 )
...
issue: #38399
- use last allocate but not last confirmed id to make barrier.
- move the barrier logic into the timetick allocator.
- try to sync up local allocator and remote allocator when first barrier
check not pass to speed up.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-27 15:08:50 +08:00
19052ef3e5
enhance: Add buffered writer to reduce fwrite syscall ( #38570 )
...
Related to previous PR #38157
If mmapped row is too small, frequent fwrite call still cost too much
cpu time for context switching. This PR add buffered write to avoid this
bad case with extra buffer per variable field.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-27 12:20:50 +08:00
4df444ef25
fix: Aviod add negative missing count ( #38748 )
...
See also: #34665
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-26 18:58:56 +08:00
94955e5292
fix: expand privilege group when list policy in rootcoord ( #38758 )
...
related: https://github.com/milvus-io/milvus/issues/38757
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-26 15:38:50 +08:00
85f462be1a
enhance: speed up search iterator stage 1 ( #37947 )
...
issue: #37548
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-12-26 10:32:49 +08:00
f49d618382
fix: Querycoord will trigger unexpected balance task after restart ( #38630 )
...
issue: #38606
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-25 19:30:48 +08:00
bc15ad24f2
fix: sealed segment get empty index params when brute force search for bm25 ( #38707 )
...
relate: https://github.com/milvus-io/milvus/issues/38236
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-25 19:06:51 +08:00
363d7f31ef
fix: report error when hints not supported ( #38717 )
...
issue: #38705
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-25 19:02:56 +08:00
c7ea09a8be
enhance: return exception type name when segcore return unkonwn exception ( #38326 )
...
relate: https://github.com/milvus-io/milvus/issues/38265
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-25 18:58:50 +08:00
25f0c82ceb
fix: Fix update loading collection's load config doesn't work ( #38595 )
...
issue: #38594
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-25 18:02:51 +08:00
6bc799061e
fix: fix privilege group list and list collections ( #38684 )
...
related: #37031
* built-in privilege group privileges in listPrivilegeGroups() should be
the same as in milvus.yaml
* collections granted by collection level built-in privilege group
should be list in showCollections()
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:00:51 +08:00
9c3f59dbbe
fix: Prevent balancer from overloading the same QueryNode ( #38719 )
...
issue: #38718
The balancer calculates the workload of executing tasks as an ongoing
score for target nodes. However, a logic issue arises when
GetSegmentTaskDelta or GetChannelTaskDelta is called with
collectionID=-1, which incorrectly returns zero.
Due to the incorrect global score, the executing task's workload is not
properly reflected for each collection. Consequently, each collection
submits its own balance task, leading to the balancer assigning
excessive tasks to the same QueryNode.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-25 16:36:49 +08:00
acc8fb7af6
enhance: eliminate compile warnings (part2) ( #38535 )
...
See #38435
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-25 15:30:50 +08:00
5395ec19ad
enhance: add mem size for index file and fallback to multiply with serialized size ( #38716 )
...
issue: #38715
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 14:46:49 +08:00
c731357538
enhance: Add missing delete metrics ( #38634 )
...
Add 2 counter metrics:
- Total delete entries from deltalog:
milvus_datanode_compaction_delete_count
- Total missing deletes: milvus_datanode_compaction_missing_delete_count
See also: #34665
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-25 10:24:50 +08:00
285289d5b0
enhance: implement kafka for wal ( #38598 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00
05d9f8562f
fix: fix complie error generated by conflict pr ( #38729 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:14:49 +08:00
69a9fd6ead
enhance: enable rmq for streaming ( #38669 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-24 20:24:48 +08:00
2b53b0905e
fix: Fix 0 read count during import ( #38694 )
...
issue: https://github.com/milvus-io/milvus/issues/38693
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-24 20:06:49 +08:00
c458dec206
fix: [skip e2e]update webui collection and query pages ( #38700 )
...
issue: #36621
* enhance collection detail page list columns
* enhance query page segments state detail logic
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
2024-12-24 17:36:48 +08:00
636e107e44
fix: Remove sync task after finished ( #38681 )
...
Related to #38680
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-24 13:34:48 +08:00
d718bbb4de
fix: fix altercollectionfield interceptor dbname ( #38662 )
...
fix altercollectionfield interceptor dbname
issue: https://github.com/milvus-io/milvus/issues/37436
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-23 18:24:48 +08:00
bb5f38e574
enhance: Return collection not loaded rather than not found on querynode ( #38593 )
...
issue: #38586
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-23 14:36:48 +08:00
2356425530
fix: Support hybrid search with expression template ( #38623 )
...
issue: #38616
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-23 14:32:51 +08:00
205231b9c7
fix: Decode unicode for json key in expression ( #38651 )
...
issue: #38626
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-23 13:40:49 +08:00
5afd0c0a2b
fix: Revert "Expose metrics of stanby coordinators ( #27698 )" ( #38620 )
...
issue: #38608
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-23 11:46:57 +08:00
984a605d47
enhance: use lazy initializing client for streaming node ( #38400 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-23 11:36:47 +08:00
f499ca47a0
fix: [skip e2e]update webui ( #36621 ) ( #38633 )
...
issue: #36621
* add collection detail page
* enhance query page segments & channels state display
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
2024-12-23 11:00:49 +08:00
407035c994
[automated] Update Knowhere Commit ( #38641 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-22 00:44:46 +08:00
cce25ecdbc
[automated] Update Knowhere Commit ( #38635 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-21 00:50:46 +08:00
90de37e60a
enhance: field stringtype maxlength raise to 1M ( #38592 )
...
enhance: field stringtype maxlength raise to 1M
issue: https://github.com/milvus-io/milvus/issues/37436
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-20 14:32:45 +08:00
ca7ec23198
enhance: Use partitionID when delete by partitionKey ( #38231 )
...
When delete by partition_key, Milvus will generates L0 segments
globally. During L0 Compaction, those L0 segments will touch all
partitions collection wise. Due to the false-positive rate of segment
bloomfilters, L0 compactions will append false deltalogs to completed
irrelevant partitions, which causes *partition deletion amplification.
This PR uses partition_key to set targeted partitionID when producing
deleteMsgs into MsgStreams. This'll narrow down L0 segments scope to
partition level, and remove the false-positive influence
collection-wise.
However, due to DeleteMsg structure, we can only label one partition to
one deleteMsg, so this enhancement fails if user wants to delete over 2
partition_keys in one deletion.
See also: #34665
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-20 11:18:46 +08:00
c0b855dc75
fix: ChannelManager concurret Release and Watch bug ( #38590 )
...
See also: #38589
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-19 22:50:47 +08:00
8fcb33c21d
fix:fix delete record assert failed ( #38580 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-19 18:22:47 +08:00
3d360c0624
fix: SyncSegments rpc always failed ( #38578 )
...
miss the patch due to code branching
previous pr: #38032
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
2024-12-19 15:40:45 +08:00
b537a72309
fix: interted index out of range ( #38577 )
...
issue: #38546 , #38486
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-19 15:20:47 +08:00
306e5e6898
enhance: clean compaction task in compactionHandler ( #38170 )
...
issue: #35711
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-12-19 12:36:47 +08:00
78438ef41e
fix: revert optimize CPU usage for CheckHealth requests ( #35589 ) ( #38555 )
...
issue: #35563
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-19 00:38:45 +08:00
134d06f7e6
fix: need pass parameter to listPrivilegeGroups ( #38562 )
...
issue: https://github.com/milvus-io/milvus/issues/38563
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-12-18 20:28:51 +08:00
de8858931a
fix: rbac revoke check if there is same privilege in other privilege groups granted ( #38558 )
...
related: https://github.com/milvus-io/milvus/issues/38557
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-18 18:46:45 +08:00
a348122758
fix: Support get segments from current segments view ( #38512 )
...
issue: #38511
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-18 18:00:54 +08:00
06d410b70f
enhance: update knowhere version ( #38544 )
...
related: #37730
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-12-18 16:52:45 +08:00
87056be748
fix: fix snapshot or size when query ( #38549 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-18 16:42:45 +08:00
894c203f27
enhance: allow hyphen in partition name ( #38494 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-18 15:04:45 +08:00
6ee94d00b9
fix:fix calculate arrow nest type and add ut ( #38527 )
...
#37767
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-18 11:54:44 +08:00
ffd3c5d2f5
[automated] Update Knowhere Commit ( #38542 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-12-18 01:24:47 +08:00
7a05b5bbea
fix: Fix stats task wrong RootPath when upload binlog ( #38539 )
...
issue: #38336
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-18 00:18:45 +08:00
1aa31e2a9b
fix: Fix inaccurate general count ( #38524 )
...
Checking general count should only count healthy collections and
partitions.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 23:24:45 +08:00
4d5572d7ed
enhance: [skip e2e] refine display format ( #38537 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 22:30:44 +08:00
decdfdae10
fix: growing-groupby-crush( #38533 ) ( #38538 )
...
related: #38533
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-12-17 21:05:12 +08:00
f0096ec292
fix: Fix IsMmapSupported for scalar index ( #38135 )
...
https://github.com/milvus-io/milvus/issues/38134
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 20:30:44 +08:00
9afcc5bc5c
fix:fix incorrect dir operations when create or load inverted index ( #38359 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 20:06:45 +08:00
93fba1d5ab
fix: access log remove new log not the old and skip rotate empty log ( #38309 )
...
relate: https://github.com/milvus-io/milvus/issues/38293
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-17 19:16:44 +08:00
29e620fa6d
fix: sync task still running after DataNode has stopped ( #38377 )
...
issue: #38319
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-17 18:06:44 +08:00
d0a7e98a27
fix:remove incorrect assert for delete query ( #38509 )
...
#38472
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 17:48:44 +08:00
8916fbf122
fix: use new path for streamingnode recovery info ( #38516 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 17:10:45 +08:00
d3c174b0f1
enhance: Accelerate observe collection ( #38028 )
...
1. A collection should observe the channel only once.
2. A collection should check the CollectionLoadPercent for updates only
once.
3. Skip saving coll/partition meta if there are no changes, primarily to
accelerate collection observation after recovery.
issue: https://github.com/milvus-io/milvus/issues/37630
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 14:14:45 +08:00
d4dab3c62f
enhance: Reduce segmentManager lock granularity ( #37836 )
...
Use a channel level key lock for segments in segmentManager.
issue: https://github.com/milvus-io/milvus/issues/37633 ,
https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 14:12:52 +08:00
dd4f33ae19
fix: Fix chunked segment can not warmup using mmap ( #38492 )
...
issue: #38410
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 13:42:45 +08:00
33aecb0655
fix: build break on target test_cpp under OSX ( #38479 )
...
See: #38434
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-17 13:38:45 +08:00
01fa2f8da5
fix: Fix compile error ( #38510 )
...
/kind improvement
related: #37559
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-17 12:16:19 +08:00
Gao
congqixia
Spade A
yihao.dai
presburger
wei liu
Zhen Ye
Ted Xu
jaime
cai.zhang
Chun Han
sre-ci-robot
Cai Yudong
aoiasd
Buqian Zheng
Alexander Guzhva
Bingyi Sun
smellthemoon
sthuang
SimFG
XuanYang-cn
Xianhui Lin
zhagnlu
foxspy
Patrick Weizhi Xu
MimoN
tinswzy
Xiaofan
Reid Chan