c69af44472
enhance: [24]Add more info in logs ( #36732 )
...
pr: #36535
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-10 17:51:25 +08:00
e8c40e94ad
enhance: use new implementation of e2e pipeline ( #36687 )
...
https://github.com/milvus-io/milvus/issues/36348
https://github.com/milvus-io/milvus/pull/36347
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-10-10 17:11:21 +08:00
eaa948752b
enhance: [Cherry-Pick] UpdateSegmentsInfo should update remaining segment info even if some one not exist ( #36729 )
...
pr: https://github.com/milvus-io/milvus/pull/36726
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-10 15:11:20 +08:00
02bd916fa4
fix: fix: fix json in [] expr bug ( #36722 )
...
pr: #36721
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-10-09 21:53:20 +08:00
9cb5396cf6
enhance: Use common gc config ( #36668 ) ( #36670 )
...
Use the GC config from `common` and remove the GC config from
`queryNode`.
issue: https://github.com/milvus-io/milvus/issues/36667
pr: https://github.com/milvus-io/milvus/pull/36668
related pr: https://github.com/milvus-io/milvus/pull/34949
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 19:49:20 +08:00
3a80d1f602
enhance: [2.4] Add streaming forward policy switch for delegator ( #36330 ) ( #36712 )
...
Cherry pick from master
pr: #36330
Related to #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-09 17:41:20 +08:00
05f96f5298
fix: [24]raise l0 compaction memory ratio to 0.5 ( #36691 )
...
5 percent of free memory is too less for l0 compaction. This pr will
raise it to 50 percent.
See also: #36614
pr: #36690
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-09 17:19:24 +08:00
93a4574a35
Add buildIndex state for import job ( #36705 )
...
issue: https://github.com/milvus-io/milvus/issues/36698
pr: https://github.com/milvus-io/milvus/pull/35868 ,
https://github.com/milvus-io/milvus/pull/36699
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 16:57:20 +08:00
0ff8e13232
fix: [24]Remove neighbors if compactTo is unindexed ( #36503 ) ( #36694 )
...
See also: #36360
pr: #36503
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-09 15:43:20 +08:00
bcc661dbd0
fix: rocksmq consume too slow if the channel is full ( #36618 )
...
issue: #36569
pr: #36617
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-09 11:59:31 +08:00
6e93601b2c
enhance: [2.4] remove warnings when there is no partition key set ( #36679 )
...
issue: #36673
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit db60f7cf2a
2024-10-09 09:35:20 +08:00
f7302a78fb
enhance: [cherry-pick]Convert unincode to ascii to improving expression parsing efficiency ( #36676 )
...
issue: #36672
master pr: #36675
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-09 09:23:24 +08:00
a081bf394d
[automated] Update Pytest image changes ( #36689 )
...
Update Pytest image changes
See changes:
f10806650asre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-09 08:51:19 +08:00
1955738ab8
enhance: [2.4] Produce messages of multiple topics in parallel ( #36344 ) ( #36462 )
...
Cherry-pick from master
pr: #36344
Related to #36343
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-08 18:31:20 +08:00
9073f6281e
fix: [2.4] Add defer Unpin when error happens ( #36620 ) ( #36665 )
...
Cherry-pick from master
pr: #36620
Resolves : #36619
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-08 18:29:20 +08:00
03fba95284
fix: [2.4] improve mmap related code in Column.h ( #36521 )
...
pr: #36183
issue: #36182
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-08 18:27:19 +08:00
f10806650a
test: update pymilvus version and test case ( #36590 )
...
pr: #36589
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-08 17:25:24 +08:00
da2d3fb430
enhance: enable manual compaction for collections without indexes ( #36581 )
...
issue: #36576
pr: #36577
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 14:51:20 +08:00
bf671786d9
enhance: decrease CPU overhead when calculating index file size ( #36580 )
...
issue: #36578
pr: #36579
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 14:29:25 +08:00
86004d852e
enhance: Reopen chunk cache cpp ut ( #33622 ) ( #33623 )
...
issue: https://github.com/milvus-io/milvus/issues/33210
pr: https://github.com/milvus-io/milvus/pull/33622
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-08 11:21:22 +08:00
29c1d5042a
fix: Fix import segments leak in segment manager ( #36602 ) ( #36631 )
...
Directly add import segments from the meta, eliminating the dependency
on the segment manager.
issue: https://github.com/milvus-io/milvus/issues/34648
pr: https://github.com/milvus-io/milvus/pull/36602
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-08 10:15:22 +08:00
2428adea3b
enhance: Enable balance on querynode with different mem capacity ( #36466 ) ( #36625 )
...
issue: #36464
pr: #36466
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 18:11:18 +08:00
385bfc2639
enhance: [2.4] Unify InboundLabel case using metrics.Label ( #36613 ) ( #36616 )
...
Cherry pick from master
pr: #36613
Previous label case broken by #36107 , this PR make all inbound label
using label constants from metrics package.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 16:19:17 +08:00
f8dfd0de53
fix: Skip unnecessary query node health check in proxy ( #36491 ) ( #36553 )
...
issue: #36490
pr: #36491
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 14:27:16 +08:00
4120320074
enhance: make TransferChannel/TransferSegment idempotent ( #36489 ) ( #36552 )
...
issue: #36488
pr: #36489
when call TransferChannel/TransferSegment, querycoord will generate and
submit balance task to scheduler, if segment/channel's task already
exist in scheduler, submit task will failed.
to make TransferChannel/TransferSegment idempotent, we should skip to
submit if task already exist in scheduler.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 14:25:26 +08:00
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows ( #36519 ) ( #36588 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
pr: https://github.com/milvus-io/milvus/pull/36519
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-30 10:03:18 +08:00
58a763c529
enhance: [2.4] avoid to create many timer object in the target ( #36573 )
...
/kind improvement
- pr: #36570
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:27:16 +08:00
9a07c1bca9
fix:fix term expr overflow bug ( #36534 )
...
pr: #36525
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-28 17:43:16 +08:00
6335c8bf69
fix: [cherry-pick] Record the nodeID before assigning tasks ( #36493 )
...
issue: #33744
master pr: #36371
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:29:21 +08:00
132e4c3ba1
fix: [cherry-pick]Fix data race for cluerting compaction ( #36499 )
...
issue: #36438
master pr: #36440
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize ( #36574 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-27 18:15:19 +08:00
e5a6c5b31d
fix: [cherry-pick]Check string array max length after type matching ( #36497 )
...
issue: #36029
master pr: #36449
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-27 11:15:14 +08:00
e34fa0461b
fix: port listen racing in mix or standalone mode ( #36459 )
...
issue: #36441
pr: #36442
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:01:15 +08:00
74af00ba8c
fix: Segment unbalance after many times load/release ( #36537 ) ( #36543 )
...
issue: #36536
pr: #36537
query coord use `segmentTaskDeleta/channelTaskDelta` to measure the
executing workload for querynode in scheduler, and we maintains the
`segmentTaskDeleta/channelTaskDelta` by `scheulder.Add(task)` and
`scheduler.remove(task)`, but `scheduler.remove(task)` has been called
in unexpected way, which cause a wrong
`segmentTaskDeleta/channelTaskDelta` value and affect the segment assign
logic, causes segment unbalance.
This PR moves to compute the `segmentTaskDeleta/channelTaskDelta` when
access, to avoid the wrong value affect.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 20:57:14 +08:00
07e1bc8c08
enhance: [2.4] get msg type from the msg header to reduce the Unmarshal usage ( #36454 )
...
/kind improvement
- pr: #36409
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-26 16:41:15 +08:00
6604bbda8f
enhance: [2.4] update the expr version and format the expr http response ( #36467 )
...
/kind improvement
- pr: #36406
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-26 14:27:20 +08:00
ad5d24be65
enhance: Optimize workload based replica selection policy ( #36181 ) ( #36384 )
...
issue: #35859
pr: #36181
This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.
if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.
if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 11:19:14 +08:00
975a9797a2
enhance: Enable dynamic update loaded collection's replica ( #36417 )
...
issue: #35821
pr: #35822
After collection loaded, if we need to increase/decrease collection's
replica, we need to release and load it again.
milvus offers 4 solution to update loaded collection's replica, this PR
aims to dynamic change the replica number without release, and after
replica number changed, milvus will execute load replica or release
replica in async, and the replica loaded status can be checked by
getReplicas API.
Notice that if set too much replicas than querynode can afford，the new
replica won't be loaded successfully until enough querynode joins.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 10:43:15 +08:00
720f591c04
[automated] Bump milvus version to v2.4.12 ( #36494 )
...
Bump milvus version to v2.4.12
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-25 14:55:14 +08:00
bdc59f3b15
fix: Fix cornor case that segment can't be move out from stopping node ( #36431 ) ( #36475 )
...
issue: #36426
pr: #36431
the old constriant requires only segment on current target can be
balanced, which is wrong, and caused that segment can't be move out from
stopping node, if it's only exist in next target.
by design, stopping balance need to move out all segment on it by
balance task, thus the unfair old constriant should be removed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-25 10:13:18 +08:00
a00523f0fd
fix: metric type error when the collection has two vec field ( #36473 )
...
- issue: #36395
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-24 20:03:14 +08:00
fa6354f6df
enhance: [skip e2e][2.4] Add unittest for reducing duplicated pk from multi segments ( #36433 ) ( #36460 )
...
Cherry-pick from master
pr: #36433
Related to #35505 #36362
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 18:09:14 +08:00
b92daa1532
fix: iaccurate size estimation for encoded array data ( #36379 )
...
issue: #36029
pr: #36373
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-23 21:17:13 +08:00
7218a58477
test: [Cherry Pick]Add partial load tests ( #36432 )
...
related issue: #36037
pr: #36427
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-09-23 17:11:13 +08:00
a35d99eabf
fix: [2.4] long buffering causes mq to be unable to receive messages. ( #36425 )
...
- issue: #36397
- pr: #36420
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:17 +08:00
2b796b180b
enhance: Bump milvus & proto version to v2.4.12 ( #36376 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 14:51:13 +08:00
e083bf39f8
fix: [2.4] Return all compactTo segments after support split ( #36361 ) ( #36429 )
...
Cherry-pick from master
pr: #36361
Related to #36360
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 14:39:12 +08:00
7ac3ef9acb
enhance: Update Knowhere version ( #36402 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-09-20 19:25:11 +08:00
e8e40a0000
enhance: use pull request number as image tag in ci ( #36285 )
...
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-09-20 16:15:30 +08:00
210adbaffc
fix: [2.4] Wait check node id goroutine in case of data race ( #36302 ) ( #36377 )
...
Cherry-pick from master
pr: #36302
Resolves : #36301
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-20 13:03:12 +08:00
XuanYang-cn
yellow-shine
aoiasd
zhagnlu
yihao.dai
congqixia
Zhen Ye
Patrick Weizhi Xu
cai.zhang
sre-ci-robot
Buqian Zheng
nico
jaime
wei liu
SimFG
yanliang567
foxspy