080ed6d96a
fix: Fix datanode cannot watch channel ( #35136 )
...
Panic when `checkWatchedList` failed, this can avoid unnoticed quit of
the channel watch goroutine.
issue: https://github.com/milvus-io/milvus/issues/35135
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 15:24:17 +08:00
20dca130c6
enhance: [cherry-pick] Retry on incomplete query result ( #35061 )
...
This PR cherry-picks the following PRs:
1. Return specific error codes when encountering incomplete requery
results error. https://github.com/milvus-io/milvus/pull/31343
2. Retry on incomplete requery result in proxy.
https://github.com/milvus-io/milvus/pull/31713
issue: https://github.com/milvus-io/milvus/issues/34820
pr: https://github.com/milvus-io/milvus/pull/31343 ,
https://github.com/milvus-io/milvus/pull/31713
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 15:22:16 +08:00
ff7c1a79ee
enhance: Reduce delegator memory overloaded factor to 0.1 ( #35092 ) ( #35165 )
...
pr: #35092
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 15:12:16 +08:00
199560da8e
enhance: Skip manual stopped component during health check ( #34953 ) ( #35122 )
...
pr: #34953
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.
to solve this, we make `healthz` API skip the manual stopped component.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 14:52:17 +08:00
f0402e0c65
fix: Fix deadlock on compaction when stopping datanode ( #35199 )
...
Stop compaction tasks in parallel to avoid interdependencies and prevent
deadlocks.
issue: https://github.com/milvus-io/milvus/issues/35198
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 11:00:17 +08:00
6ea7147321
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 ) ( #35193 )
...
pr: https://github.com/milvus-io/milvus/pull/35185
fix: https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-05 10:26:17 +08:00
eb079c99a7
[automated] Update Builder image changes ( #35247 )
...
Update Builder image changes
See changes:
c5212a42b6sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:24:33 +08:00
c5212a42b6
enhance: upgrade conan version ( #35217 )
...
https://github.com/milvus-io/milvus/pull/35215
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:37:45 +08:00
2df654274f
enhance: docker-compose first then try to use docker compose ( #35228 )
...
https://github.com/milvus-io/milvus/issues/35209
https://github.com/milvus-io/milvus/pull/35208
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:37:25 +08:00
0a122533d0
enhance: change autoindex default metric type ( #34328 )
...
issue: #34304
pr: #34261
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-02 16:22:20 +08:00
923278b75d
enhance: the datacoord gc should fast quitable ( #35057 )
...
issue: #35049
pr: #35050
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-01 14:32:13 +08:00
289336a617
enhance: Avoid panic due to nil schema ( #35063 ) ( #35065 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/discussions/25620
pr: https://github.com/milvus-io/milvus/pull/35063
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-30 20:19:49 +08:00
55802b368d
enhance: [skip e2e] enlarge timeout of UT for go up to 90 mins ( #35036 )
...
pr: #35035
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-29 15:53:57 +08:00
256817da2d
feat: record the number of times milvus turns into force-deny-writing states ( #34960 ) ( #34990 )
...
/kind improvement
fix: https://github.com/milvus-io/milvus/issues/34961
pr: #34960
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 15:45:46 +08:00
fb2ca5d9e5
[automated] Bump milvus version to v2.3.20 ( #34966 )
...
Bump milvus version to v2.3.20
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-24 19:21:46 +08:00
93e457645d
[automated] Update Builder gpu image changes ( #34878 )
...
Update Builder gpu image changes
See changes:
abc7c84699sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yellow-shine <sammy.huang@zilliz.com>
2024-07-24 16:25:47 +08:00
a6c3056df3
fix: fix parse plan proto failed for search type ( #34945 )
...
pr: #34944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-24 14:59:45 +08:00
fe786fff57
enhance: [2.3] init the hook when creating the proxy object ( #34936 )
...
- issue: #34885
- pr: #34887
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-24 13:11:43 +08:00
a6ac268c6d
[automated] Update Builder image changes ( #34875 )
...
Update Builder image changes
See changes:
abc7c84699sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-24 09:45:45 +08:00
a5deaa1d4b
enhance: Bump milvus verions to v2.3.20 ( #34915 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-23 14:57:43 +08:00
193669b55f
fix: Segment may bounce between delegator and worker ( #34889 )
...
issue: #34595
pr: #34830
pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.
This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 10:47:44 +08:00
acee8cfa92
fix: Avoid generate reduce channel task before balance channel finish ( #34721 )
...
relate: #29841 #29838
pr: #30087
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 10:33:45 +08:00
464a667e8a
fix: Avoid segment lack caused by deduplicate segment task ( #34882 )
...
issue: #34781
pr: #34782
when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.
this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-22 16:05:42 +08:00
910af16d33
enhance: Update Knowhere version ( #34825 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-22 10:17:43 +08:00
abc7c84699
enhance: Upgrade go version from 1.20 to 1.21 ( #34759 )
...
issue https://github.com/milvus-io/milvus/issues/32982
related pr in master:
pr: https://github.com/milvus-io/milvus/pull/33047
https://github.com/milvus-io/milvus/pull/33150
https://github.com/milvus-io/milvus/pull/33176
https://github.com/milvus-io/milvus/pull/33351
https://github.com/milvus-io/milvus/pull/33202
https://github.com/milvus-io/milvus/pull/33192
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-07-22 10:09:42 +08:00
08c6ab8cfb
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34791 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: https://github.com/milvus-io/milvus/pull/34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:39:41 +08:00
3d9ec2ab23
fix: Discard compaction plan when closing dataSyncService ( #34737 )
...
issue: https://github.com/milvus-io/milvus/issues/34736
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 09:49:42 +08:00
8e34c4cf8f
fix:[cherry-pick] Fix compatible ascending index typo ( #34712 )
...
issue: https://github.com/milvus-io/milvus/issues/34707
pr: https://github.com/milvus-io/milvus/pull/34708
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-07-16 18:55:38 +08:00
64dc848109
enhance: Bump milvus version to v2.3.19 ( #34677 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 16:51:38 +08:00
6cd3a68f29
[automated] Bump milvus version to v2.3.19 ( #34669 )
...
Bump milvus version to v2.3.19
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-15 11:25:39 +08:00
79c0c78a07
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34602 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-13 10:25:40 +08:00
b5ba5832d3
fix: [cherry-pick] Remove flushed segment in segment manager generated through import ( #34650 )
...
issue: #34648
master pr: #34649
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-12 23:15:14 +08:00
3c44248105
fix: [2.3] support set up knowhere-build-pool-size on querynode ( #34647 )
...
Cherry-pick from master
pr: #30922
Related: #29650
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-12 19:27:36 +08:00
c46fccb3e9
fix: Prevent dispatcher merging if curTs is 0 ( #34562 ) ( #34626 )
...
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.
issue: https://github.com/milvus-io/milvus/issues/34255
pr: https://github.com/milvus-io/milvus/pull/34562
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-12 15:19:36 +08:00
7f3a2a278f
enhance: Refine default histogram metric bucket ( #34627 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-12 14:07:36 +08:00
d3d192064f
enhance: Avoid assign too much segment/channels to new querynode ( #34096 ) ( #34461 )
...
issue: #34095
pr: #34096
When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.
This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 14:22:15 +08:00
0bfa1a72bb
fix: Indexnode stuck at stopping progress cause by wrong lifetime control ( #34560 )
...
issue: #34557
pr: #34558
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 13:46:13 +08:00
cfb5390201
fix: [cherry-pick] [restful v2] count(*) & hook ( #34447 )
...
issue: #31224 #34374
pr: #34369(master) #34433(2.4) #34444(2.4.5)
for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]
add hook about grpc call
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-08 10:40:13 +08:00
a85e067099
fix: [2.3] Check partition in current target when observing partition load status ( #34282 ) ( #34321 )
...
Cherry-pick from master
pr: #34282
See also #34234
`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.
This PR check current target contains the partition to load when
advancing loading percentage to 100.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 15:54:08 +08:00
dbbddca582
enhance: [2.3] Refine max length exceeded error message ( #34300 ) ( #34324 )
...
Cherry-pick from master
pr: #34300
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 15:30:16 +08:00
252cf97e6d
fix: [2.3] Implement singleflight for segcore ChunkCache ( #34250 ) ( #34284 )
...
Cherry-pick from master
pr: #34250
See also #34249
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-02 10:30:08 +08:00
7a441c39cd
enhance: Optimize grow slice cost during query ( #34256 )
...
issue: #32252
pr: #34253
This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.
Additionally, Upgrade google.golang.org/protobuf from version 1.31 to
1.33 to address the growing slice overhead during the proto unmarshal
repeated field process, as referenced in
[#protobuffer/protobuf-go/](86bdc4705a
2024-07-01 16:46:08 +08:00
9de5b15cbb
fix: [cherry-pick]Fix datacoord's properties overwriten ( #34147 )
...
issue: #34144
pr: #34146
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-27 14:04:05 +08:00
36a1b8c30b
enhance: revert dead try on uploading codecov ( #34132 )
...
This reverts commit 56d39fa92b
2024-06-25 11:51:40 +08:00
c566edc053
fix: Fix backup channel meta is empty ( #34115 )
...
issue: https://github.com/milvus-io/milvus/issues/34061
pr: https://github.com/milvus-io/milvus/pull/32044
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-25 11:48:03 +08:00
bff0bd74b0
enhance:[Cherry-Pick] Refine varchar length exceeds max length error message ( #31521 ) ( #34034 )
...
pr: https://github.com/milvus-io/milvus/pull/31521
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-24 14:50:03 +08:00
00b02ee6ae
enhance: [2.3] try to speed up the loading of small collections ( #33863 )
...
- issue: #33569
- pr: #33570
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-22 11:46:04 +08:00
ce7bceece9
fix: [2.3] Check nodeID wildcard when removing pkOracle ( #33895 ) ( #34022 )
...
Cherry-pick from master
pr: #33895
See also #33894
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 17:08:02 +08:00
9157980232
fix: [2.3] Return record with largest timestamp for entires with same PK( #33936 ) ( #34026 )
...
Cherry-pick from master
pr: #33936
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-20 19:56:00 +08:00
b5bcf8a767
[automated] Bump milvus version to v2.3.18 ( #33973 )
...
Bump milvus version to v2.3.18
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-19 13:54:00 +08:00
yihao.dai
wei liu
Jiquan Long
sre-ci-robot
yellow-shine
Gao
chyezh
zhagnlu
SimFG
congqixia
foxspy
shaoting-huang
xige-16
cai.zhang
edward.zeng
PowderLi
aoiasd