36278fe33b
[automated] Bump milvus version to v2.3.21 ( #35459 )
...
Bump milvus version to v2.3.21
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-14 11:16:53 +08:00
20e26588af
fix: enable limiter for restful server( #35350 ) ( #35354 )
...
related: #35350
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-13 15:36:21 +08:00
032096150d
enhance: Bump milvus version to v2.3.21 ( #35419 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-13 11:44:20 +08:00
bfd20d5f99
fix: delete by expr failed at retry progress ( #35241 ) ( #35421 )
...
issue: #35240
pr: #35241
delete by expr shard the same err object between channels, so if one
channel's query failed, it will fail all channel, which will break
channel level retry policy, and make delete operation failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-12 16:16:20 +08:00
8a7c00d2c7
fix: datacoord binlog size metrics include dropped segment ( #35338 ) ( #35420 )
...
issue: #35337
pr: #35338
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-12 14:46:21 +08:00
f75b886096
fix: [2.3] Use k locations only for basic BF test location ( #35390 )
...
Cherry-pick from master
pr: #35380
Related to #35379
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-09 21:37:16 +08:00
e5681e5b9c
enhance: make delegator delete buffer holding all delete from cp ( #29626 ) ( #35074 )
...
See also #29625
pr: #29626
This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
- holds cacheBlock slice
- `Put` method append new delete data into last block
- when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
- For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-08-09 18:48:18 +08:00
d8bfc077a4
enhance: add metric to record disk quota ( #35305 ) ( #35320 )
...
https://github.com/milvus-io/milvus/issues/35177
pr: https://github.com/milvus-io/milvus/pull/35305
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-08 10:06:17 +08:00
621ca8b564
fix: comparision operations between incompatible operands ( #35264 ) ( #35308 )
...
fix: https://github.com/milvus-io/milvus/issues/34139
pr: https://github.com/milvus-io/milvus/pull/35264
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 14:44:21 +08:00
6027647c00
fix: error message when input is not aligned ( #35322 ) ( #35324 )
...
fix: https://github.com/milvus-io/milvus/issues/35321
pr: https://github.com/milvus-io/milvus/pull/35322
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 14:32:22 +08:00
52fcd3f48b
fix: [2.3] Reduce duplicate PKs in segcore ( #34267 ) ( #35291 )
...
issue: https://github.com/milvus-io/milvus/issues/34247
pr: https://github.com/milvus-io/milvus/pull/34267
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-07 14:30:17 +08:00
b59ba81349
feat: [skip e2e] effective way to use cache ( #35233 )
...
issue: https://github.com/milvus-io/milvus/issues/34876
https://github.com/milvus-io/milvus/pull/35109
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-06 10:12:27 +08:00
080ed6d96a
fix: Fix datanode cannot watch channel ( #35136 )
...
Panic when `checkWatchedList` failed, this can avoid unnoticed quit of
the channel watch goroutine.
issue: https://github.com/milvus-io/milvus/issues/35135
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 15:24:17 +08:00
20dca130c6
enhance: [cherry-pick] Retry on incomplete query result ( #35061 )
...
This PR cherry-picks the following PRs:
1. Return specific error codes when encountering incomplete requery
results error. https://github.com/milvus-io/milvus/pull/31343
2. Retry on incomplete requery result in proxy.
https://github.com/milvus-io/milvus/pull/31713
issue: https://github.com/milvus-io/milvus/issues/34820
pr: https://github.com/milvus-io/milvus/pull/31343 ,
https://github.com/milvus-io/milvus/pull/31713
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 15:22:16 +08:00
ff7c1a79ee
enhance: Reduce delegator memory overloaded factor to 0.1 ( #35092 ) ( #35165 )
...
pr: #35092
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 15:12:16 +08:00
199560da8e
enhance: Skip manual stopped component during health check ( #34953 ) ( #35122 )
...
pr: #34953
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.
to solve this, we make `healthz` API skip the manual stopped component.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 14:52:17 +08:00
f0402e0c65
fix: Fix deadlock on compaction when stopping datanode ( #35199 )
...
Stop compaction tasks in parallel to avoid interdependencies and prevent
deadlocks.
issue: https://github.com/milvus-io/milvus/issues/35198
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 11:00:17 +08:00
6ea7147321
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 ) ( #35193 )
...
pr: https://github.com/milvus-io/milvus/pull/35185
fix: https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-05 10:26:17 +08:00
eb079c99a7
[automated] Update Builder image changes ( #35247 )
...
Update Builder image changes
See changes:
c5212a42b6sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:24:33 +08:00
c5212a42b6
enhance: upgrade conan version ( #35217 )
...
https://github.com/milvus-io/milvus/pull/35215
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:37:45 +08:00
2df654274f
enhance: docker-compose first then try to use docker compose ( #35228 )
...
https://github.com/milvus-io/milvus/issues/35209
https://github.com/milvus-io/milvus/pull/35208
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:37:25 +08:00
0a122533d0
enhance: change autoindex default metric type ( #34328 )
...
issue: #34304
pr: #34261
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-02 16:22:20 +08:00
923278b75d
enhance: the datacoord gc should fast quitable ( #35057 )
...
issue: #35049
pr: #35050
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-01 14:32:13 +08:00
289336a617
enhance: Avoid panic due to nil schema ( #35063 ) ( #35065 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/discussions/25620
pr: https://github.com/milvus-io/milvus/pull/35063
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-30 20:19:49 +08:00
55802b368d
enhance: [skip e2e] enlarge timeout of UT for go up to 90 mins ( #35036 )
...
pr: #35035
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-29 15:53:57 +08:00
256817da2d
feat: record the number of times milvus turns into force-deny-writing states ( #34960 ) ( #34990 )
...
/kind improvement
fix: https://github.com/milvus-io/milvus/issues/34961
pr: #34960
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 15:45:46 +08:00
fb2ca5d9e5
[automated] Bump milvus version to v2.3.20 ( #34966 )
...
Bump milvus version to v2.3.20
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-24 19:21:46 +08:00
93e457645d
[automated] Update Builder gpu image changes ( #34878 )
...
Update Builder gpu image changes
See changes:
abc7c84699sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yellow-shine <sammy.huang@zilliz.com>
2024-07-24 16:25:47 +08:00
a6c3056df3
fix: fix parse plan proto failed for search type ( #34945 )
...
pr: #34944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-24 14:59:45 +08:00
fe786fff57
enhance: [2.3] init the hook when creating the proxy object ( #34936 )
...
- issue: #34885
- pr: #34887
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-24 13:11:43 +08:00
a6ac268c6d
[automated] Update Builder image changes ( #34875 )
...
Update Builder image changes
See changes:
abc7c84699sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-24 09:45:45 +08:00
a5deaa1d4b
enhance: Bump milvus verions to v2.3.20 ( #34915 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-23 14:57:43 +08:00
193669b55f
fix: Segment may bounce between delegator and worker ( #34889 )
...
issue: #34595
pr: #34830
pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.
This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 10:47:44 +08:00
acee8cfa92
fix: Avoid generate reduce channel task before balance channel finish ( #34721 )
...
relate: #29841 #29838
pr: #30087
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 10:33:45 +08:00
464a667e8a
fix: Avoid segment lack caused by deduplicate segment task ( #34882 )
...
issue: #34781
pr: #34782
when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.
this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-22 16:05:42 +08:00
910af16d33
enhance: Update Knowhere version ( #34825 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-22 10:17:43 +08:00
abc7c84699
enhance: Upgrade go version from 1.20 to 1.21 ( #34759 )
...
issue https://github.com/milvus-io/milvus/issues/32982
related pr in master:
pr: https://github.com/milvus-io/milvus/pull/33047
https://github.com/milvus-io/milvus/pull/33150
https://github.com/milvus-io/milvus/pull/33176
https://github.com/milvus-io/milvus/pull/33351
https://github.com/milvus-io/milvus/pull/33202
https://github.com/milvus-io/milvus/pull/33192
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-07-22 10:09:42 +08:00
08c6ab8cfb
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34791 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: https://github.com/milvus-io/milvus/pull/34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:39:41 +08:00
3d9ec2ab23
fix: Discard compaction plan when closing dataSyncService ( #34737 )
...
issue: https://github.com/milvus-io/milvus/issues/34736
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 09:49:42 +08:00
8e34c4cf8f
fix:[cherry-pick] Fix compatible ascending index typo ( #34712 )
...
issue: https://github.com/milvus-io/milvus/issues/34707
pr: https://github.com/milvus-io/milvus/pull/34708
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-07-16 18:55:38 +08:00
64dc848109
enhance: Bump milvus version to v2.3.19 ( #34677 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 16:51:38 +08:00
6cd3a68f29
[automated] Bump milvus version to v2.3.19 ( #34669 )
...
Bump milvus version to v2.3.19
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-15 11:25:39 +08:00
79c0c78a07
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34602 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-13 10:25:40 +08:00
b5ba5832d3
fix: [cherry-pick] Remove flushed segment in segment manager generated through import ( #34650 )
...
issue: #34648
master pr: #34649
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-12 23:15:14 +08:00
3c44248105
fix: [2.3] support set up knowhere-build-pool-size on querynode ( #34647 )
...
Cherry-pick from master
pr: #30922
Related: #29650
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-12 19:27:36 +08:00
c46fccb3e9
fix: Prevent dispatcher merging if curTs is 0 ( #34562 ) ( #34626 )
...
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.
issue: https://github.com/milvus-io/milvus/issues/34255
pr: https://github.com/milvus-io/milvus/pull/34562
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-12 15:19:36 +08:00
7f3a2a278f
enhance: Refine default histogram metric bucket ( #34627 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-12 14:07:36 +08:00
d3d192064f
enhance: Avoid assign too much segment/channels to new querynode ( #34096 ) ( #34461 )
...
issue: #34095
pr: #34096
When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.
This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 14:22:15 +08:00
0bfa1a72bb
fix: Indexnode stuck at stopping progress cause by wrong lifetime control ( #34560 )
...
issue: #34557
pr: #34558
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 13:46:13 +08:00
cfb5390201
fix: [cherry-pick] [restful v2] count(*) & hook ( #34447 )
...
issue: #31224 #34374
pr: #34369(master) #34433(2.4) #34444(2.4.5)
for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]
add hook about grpc call
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-08 10:40:13 +08:00
sre-ci-robot
Chun Han
congqixia
wei liu
Jiquan Long
yihao.dai
yellow-shine
Gao
chyezh
zhagnlu
SimFG
foxspy
shaoting-huang
xige-16
cai.zhang
edward.zeng
PowderLi