c46fccb3e9
fix: Prevent dispatcher merging if curTs is 0 ( #34562 ) ( #34626 )
...
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.
issue: https://github.com/milvus-io/milvus/issues/34255
pr: https://github.com/milvus-io/milvus/pull/34562
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-12 15:19:36 +08:00
7f3a2a278f
enhance: Refine default histogram metric bucket ( #34627 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-12 14:07:36 +08:00
d3d192064f
enhance: Avoid assign too much segment/channels to new querynode ( #34096 ) ( #34461 )
...
issue: #34095
pr: #34096
When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.
This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 14:22:15 +08:00
0bfa1a72bb
fix: Indexnode stuck at stopping progress cause by wrong lifetime control ( #34560 )
...
issue: #34557
pr: #34558
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 13:46:13 +08:00
cfb5390201
fix: [cherry-pick] [restful v2] count(*) & hook ( #34447 )
...
issue: #31224 #34374
pr: #34369(master) #34433(2.4) #34444(2.4.5)
for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]
add hook about grpc call
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-08 10:40:13 +08:00
a85e067099
fix: [2.3] Check partition in current target when observing partition load status ( #34282 ) ( #34321 )
...
Cherry-pick from master
pr: #34282
See also #34234
`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.
This PR check current target contains the partition to load when
advancing loading percentage to 100.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 15:54:08 +08:00
dbbddca582
enhance: [2.3] Refine max length exceeded error message ( #34300 ) ( #34324 )
...
Cherry-pick from master
pr: #34300
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 15:30:16 +08:00
252cf97e6d
fix: [2.3] Implement singleflight for segcore ChunkCache ( #34250 ) ( #34284 )
...
Cherry-pick from master
pr: #34250
See also #34249
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 10:30:08 +08:00
7a441c39cd
enhance: Optimize grow slice cost during query ( #34256 )
...
issue: #32252
pr: #34253
This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.
Additionally, Upgrade google.golang.org/protobuf from version 1.31 to
1.33 to address the growing slice overhead during the proto unmarshal
repeated field process, as referenced in
[#protobuffer/protobuf-go/](86bdc4705a
2024-07-01 16:46:08 +08:00
9de5b15cbb
fix: [cherry-pick]Fix datacoord's properties overwriten ( #34147 )
...
issue: #34144
pr: #34146
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-27 14:04:05 +08:00
36a1b8c30b
enhance: revert dead try on uploading codecov ( #34132 )
...
This reverts commit 56d39fa92b
2024-06-25 11:51:40 +08:00
c566edc053
fix: Fix backup channel meta is empty ( #34115 )
...
issue: https://github.com/milvus-io/milvus/issues/34061
pr: https://github.com/milvus-io/milvus/pull/32044
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-25 11:48:03 +08:00
bff0bd74b0
enhance:[Cherry-Pick] Refine varchar length exceeds max length error message ( #31521 ) ( #34034 )
...
pr: https://github.com/milvus-io/milvus/pull/31521
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-24 14:50:03 +08:00
00b02ee6ae
enhance: [2.3] try to speed up the loading of small collections ( #33863 )
...
- issue: #33569
- pr: #33570
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-22 11:46:04 +08:00
ce7bceece9
fix: [2.3] Check nodeID wildcard when removing pkOracle ( #33895 ) ( #34022 )
...
Cherry-pick from master
pr: #33895
See also #33894
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 17:08:02 +08:00
9157980232
fix: [2.3] Return record with largest timestamp for entires with same PK( #33936 ) ( #34026 )
...
Cherry-pick from master
pr: #33936
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-20 19:56:00 +08:00
b5bcf8a767
[automated] Bump milvus version to v2.3.18 ( #33973 )
...
Bump milvus version to v2.3.18
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-19 13:54:00 +08:00
ae6d95475d
enhance: Refine name rule check error msg ( #33815 ) ( #33901 )
...
pr: #33815
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-19 10:58:00 +08:00
578c38a2f7
enhance: Bump milvus version to v2.3.18 ( #33909 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 16:30:00 +08:00
aea3cfefce
fix: [2.3] Prevent use captured iteration variable partitionID ( #33912 )
...
Cherry-pick from master
pr: #33906
See also #33902
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-17 23:14:00 +08:00
2e01ae9141
fix: filter results less than topk ( #33930 )
...
issue: #33933
Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-06-17 22:54:01 +08:00
deb2c7854d
enhance: add restful api to trigger component stop ( #32076 ) ( #33798 )
...
issue: #32698
pr: #32076
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 20:46:01 +08:00
518c86236b
fix: Fix insert not limited when set collection.insertRate.max.mb=0 ( #33725 )
...
issue: #33722
pr: #31070
/kind bug
---------
Signed-off-by: xige-16 <xige2016@gmail.com>
Signed-off-by: gexi <gexi@didiglobal.com>
Co-authored-by: gexi <gexi@didiglobal.com>
2024-06-17 20:07:59 +08:00
963f601a96
enhance:[Cherry-pick] Check by proxy rate limiter when delete get data by query. ( #30891 ) ( #33794 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
pr: https://github.com/milvus-io/milvus/pull/30891
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-17 19:36:00 +08:00
284e79cf3a
enhance: Execute bloom filter apply in parallel to speed up process delete ( #33870 )
...
issue: #33610
pr: #33611 #33793
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 12:06:04 +08:00
c62e092463
enhance: [2.3] Make applyDelete work in paralell in segment level ( #32291 ) ( #33841 )
...
Cherry-pick from master
pr: #32291
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-13 23:09:56 +08:00
56d39fa92b
enhance:[skip e2e] make more retries on codecov uploader ( #33701 )
...
pr: #33683
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-12 20:40:01 +08:00
9d6af07107
fix: [2.3] Write back dbid modification for nonDB id collection ( #33641 ) ( #33695 )
...
Cherry-pick from master
pr: #33641
See also #33608
Make `fixDefaultDBIDConsistency` also write back collection dbid
modification when nonDB id collection is found.
This fix shall prevent dropped collections of this kind show up again
after dropping and restart.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-11 10:38:01 +08:00
636be02777
enhance: [2.3] Print UseDefaultConsistency param in read requests ( #33617 ) ( #33645 )
...
Cherry-pick from master
pr: #33617
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-06 14:19:56 +08:00
168724b2cd
fix: [2.3] Use milliseconds in apply pk latency metrics for upsert ( #33580 )
...
Cherry pick from master
pr: #29987
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-04 16:43:47 +08:00
6d0e167f69
fix: Fix exception info is missing ( #33393 ) ( #33395 )
...
Replace based std::exception to prevent "object slicing"
issue: https://github.com/milvus-io/milvus/issues/33392
pr: https://github.com/milvus-io/milvus/pull/33393
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-04 15:19:47 +08:00
84bc1781f8
[automated] Bump milvus version to v2.3.17 ( #33526 )
...
Bump milvus version to v2.3.17
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-31 18:55:50 +08:00
cc1bf63336
enhance: Bump milvus version to v2.3.17 ( #33509 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-31 17:29:45 +08:00
0d4ee287e1
fix: query iterator lack results( #33137 ) ( #33468 )
...
related: #33137
pr: https://github.com/milvus-io/milvus/pull/33422
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-31 13:54:07 +08:00
e94fa9e2ae
test: add partition key verify in import test ( #33347 )
...
pr: https://github.com/milvus-io/milvus/pull/33253
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-27 14:47:43 +08:00
d40929857c
fix:[cherry-pick]Panic if ProcessActiveStandBy returns error ( #33371 )
...
pr:#33369
issue:#33368
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-05-27 10:13:41 +08:00
df4a1c2185
[automated] Bump milvus version to v2.3.16 ( #33296 )
...
Bump milvus version to v2.3.16
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-23 10:09:41 +08:00
078f252aa9
fix: [cherry-pick][restful v2]role operations need dbName ( #33292 )
...
issue: #33220
master pr: #33283
2.4 pr: #33291
use dbName as part of privilege entity, so
1. grant / revoke a privilege need dbName
2. we can describe the privileges of the role which belong to one
special database
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-23 09:57:41 +08:00
653af7108c
fix: Break the dead loop of downloading binlog during compaction ( #33246 )
...
issue: #33245
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-22 17:29:40 +08:00
1007e6d562
enhance: [2.3] check the auth in some rest v2 api ( #33254 )
...
/kind improvement
pr: #33256
link 2.3 proto:
https://github.com/milvus-io/milvus-proto/blob/2.3/proto/milvus.proto
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-22 14:43:39 +08:00
0ae4c8a11c
enhance: Bump milvus & proto version to v2.3.16 ( #33127 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 16:39:40 +08:00
eb98f20316
enhance: update knowhere version ( #33215 )
...
/kind branch-feature
Fix diskann PQ train dead loop
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-05-21 16:37:44 +08:00
e325ec0a35
enhance: [2.3]Try LatestMessageID when checkpoint unmarshal fails ( #33159 )
...
Cherry-pick from master
pr: #33158
See also #33122
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 14:13:40 +08:00
76b7c23a66
fix: try best to get enough query results ( #33177 )
...
issue: https://github.com/milvus-io/milvus/issues/33137
pr: https://github.com/milvus-io/milvus/pull/32567
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-21 14:07:45 +08:00
23e7155a48
fix: avoid memoryleak in rendezvousFlushManager ( #33112 )
...
issue: https://github.com/milvus-io/milvus/issues/33110
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-05-20 22:19:40 +08:00
7a617e2921
doc: [skip e2e] change milvus docker image version to v2.3.15 ( #33186 )
...
pr: https://github.com/milvus-io/milvus/pull/33170
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-05-20 17:05:39 +08:00
f848e82971
enhance: [2.3] Add param item to ignore bad message id in checkpoint ( #33128 )
...
Cherry-pick from master
pr: #33123
See also #33122
This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 11:31:39 +08:00
b073407119
fix: check array field data is nil before getting the type ( #33119 )
...
issue: https://github.com/milvus-io/milvus/issues/33074
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-18 17:18:49 +08:00
62848d3aed
enhance: Make sure every segment only trigger flush task once by channel.flushTs. ( #33034 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-17 18:57:37 +08:00
bbf3a27472
fix: avoid concurrent sync and segment tasks ( #33087 )
...
issue: https://github.com/milvus-io/milvus/issues/33089
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-17 14:27:42 +08:00
yihao.dai
edward.zeng
wei liu
PowderLi
congqixia
xige-16
sammy.huang
aoiasd
SimFG
sre-ci-robot
Min Tian
Chun Han
zhuwenxing
wayblink
cai.zhang
Gao
Jiquan Long
zhenshan.cao
pingliu
Bingyi Sun