8f7019468f
fix: starve lock caused by slow GetCompactionTo method when too much segments ( #30963 )
...
issue: #30823
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-05 10:04:59 +08:00
52540fecb2
enhance: Add back load memory factor when esitmating memory resource ( #30994 )
...
Segment load memory usage is underestimated due to removing the load
memroy factor. This PR adds it back to protect querynode OOM during some
extreme memory cases.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-04 19:54:59 +08:00
9b3005f1be
enhance: Avoid create schema helper for each read task ( #30981 )
...
See also #30806
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-04 19:39:00 +08:00
bd063a352d
test: add test cases for mmap ( #30942 )
...
issue: https://github.com/milvus-io/milvus/issues/30940
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-03-04 19:33:01 +08:00
71adafa933
enhance: adding a streaming deserialize reader for binlogs ( #30860 )
...
See #30863
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-04 19:31:09 +08:00
4b0c3dd377
enhance: index meta use independent rather than global meta lock ( #30869 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
3dc5e38240
[automated] Update Knowhere Commit ( #30989 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-04 16:34:59 +08:00
fa73520b57
fix: Make datacoord client retry on index api ( #30654 )
...
issue: #20553
This PR add retry on all interface which belong to indexcoord in milvus
2.2 and. move to data coord in milvus 2.3, to prevent meet
`unimplemented` error during rolling upgrade from milvus 2.2 to 2.3.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-04 15:32:59 +08:00
882b50c5c1
[automated] Update Pytest image changes ( #30852 )
...
Update Pytest image changes
See changes:
7f78e9d40dsre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-04 09:55:00 +08:00
f6ff2588cd
enhance: Optimize DescribeIndex to reduce lock contention ( #30939 )
...
issue: #29313
issue: #30443
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-03 19:00:59 +08:00
e39f46aa38
fix: use GetBinaryField incorrectly ( #30969 )
...
issue: #30980 #22837
2.3 pr: #30873
should use GetBFloat16Field, GetBfloat16Field rather than GetBinaryField
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-03 17:58:59 +08:00
d98a5e441f
test: add restful v2 test ( #30984 )
...
add restful v2 test
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-03-02 11:03:03 +08:00
d8164c43d2
test: update test cases ( #30777 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-03-02 11:01:13 +08:00
0a2c255630
enhance: Reduce the memory usage of the timeTickSender ( #30968 )
...
In the cache of the timeTickSender, retain only the latest stats instead
of storing stats for every time tick.
issue: https://github.com/milvus-io/milvus/issues/30967
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-02 10:13:01 +08:00
29f44f840a
enhance: refine groupBy error msg( #29968 ) ( #30920 )
...
related: #29968
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-01 18:53:03 +08:00
a434d33e75
feat: Add import scheduler and manager ( #29367 )
...
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
85de56e894
fix: Clean kafka default configuration ( #30924 )
...
issue: #30917
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-03-01 18:17:03 +08:00
7783098ddd
feat: support lazy load on querycoord ( #30372 )
...
https://github.com/milvus-io/milvus/issues/30361
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-01 18:15:29 +08:00
f3c56c83ab
fix: Fix binlog_io metric name conflict ( #30689 )
...
follow: #29725
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-03-01 18:13:02 +08:00
8b25ecffff
fix: add pk sort for same distance of hybridsearch ( #30901 )
...
#29923
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-01 18:09:02 +08:00
41f1e6dcad
enhance:Add integration test for searching when coord down ( #30717 )
...
issue: #29772
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 17:07:00 +08:00
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
ee8d6f236c
enhance: make the watch dm channel request better compatibility ( #30952 )
...
issue: #30938
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-01 16:07:37 +08:00
dd957cf9e3
enhance: add configurable memory index load predict memory usage factor ( #30561 )
...
related pr: https://github.com/milvus-io/milvus/pull/30475
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-01 15:23:00 +08:00
cc46d6bafc
fix: segment version doesn't update as expected ( #30951 )
...
issue: #30950
due to segment version doesn't update as expected.
This PR will update segment version until segment become loaded
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-01 14:21:10 +08:00
6387403639
fix: Prevent clone when selecting segments from meta ( #30928 )
...
See also #30538
Previously the `SelectSegments` changed to clone all return value
preventing possible update to returned info.
Since meta is implemented following COW rules, this shall not happen and
any update on segment shall have copy before it.
This PR:
- Remove clone for read-only Get segment info
- Add Segment Operator abstraction for changing segment
- Implemnt COW for updating MaxRowNum
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-01 13:59:02 +08:00
385dec3b69
fix: index attr caches wrong result variable ( #30960 )
...
See also #30757 #30756
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-01 13:17:00 +08:00
2180e2cfae
fix: wrong segment binlog path cause load segment failed #30726 ( #30959 )
...
DataCoord GetSegmentInfo should return binlog info with logpath instead
logid when segment merge child segment's
binlog.
relate: https://github.com/milvus-io/milvus/issues/30366
https://github.com/milvus-io/milvus/issues/30165
https://github.com/milvus-io/milvus/issues/30550
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-01 13:09:01 +08:00
6f184d354f
enhance: Refine describe collection error message when collection not found ( #30766 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-01 12:53:00 +08:00
2867f50fcc
fix: Clear DN unkown compaction tasks ( #30850 )
...
If DC restarted, those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.
See also: #30137
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-01 11:31:00 +08:00
72ac2b1e93
fix:Update tencent cloud version ( #30847 )
...
Update tencent cloud version for updating token when expired.
issue: #30846
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 11:09:00 +08:00
4869aaeb94
fix: [restful v2] align search api with milvus client ( #30946 )
...
issue: #30688
fix: [restful v2] align search api with milvus client #30688
fix: [restful v2] DescribeIndex return Response{ Status: IndexNotFound
}, nil #30722
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-01 11:01:01 +08:00
21b41e96fc
fix: Add hybridsearch result cases ( #30834 )
...
#30694
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-01 10:59:01 +08:00
8addd75481
fix: Mmap could be enabled on ScaNN index ( #30914 )
...
issue: https://github.com/milvus-io/milvus/issues/30899
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-01 10:56:59 +08:00
3633923bb7
enhance: clean invalid pipline excluded segment info ( #30429 )
...
relate: https://github.com/milvus-io/milvus/issues/30281
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-01 10:43:01 +08:00
f3d1c75499
enhance: optimize access log formatter and add bench test ( #30829 )
...
relate: https://github.com/milvus-io/milvus/issues/28948
https://github.com/milvus-io/milvus/issues/30806
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-01 10:41:00 +08:00
36d78e3dd0
fix: Use localStorage path to check disk cap ( #30944 )
...
See also #30943
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-01 10:17:01 +08:00
a5a4ca8459
enhance: Remove debug log ( #30955 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 10:02:59 +08:00
17a2fd048e
feat: support set up knowhere-build-pool-size on querynode( #29650 ) ( #30922 )
...
related: #29650
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-29 18:15:00 +08:00
d930666b3e
test: Add a test for issue #30607 ( #30933 )
...
related issue: #30607
and update some test for groupby
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-02-29 18:05:00 +08:00
0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases ( #30317 )
...
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
issue: #30310
also see pr: #30306
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
816ed671aa
fix: alter_index should return error if index not found ( #30786 )
...
issue: https://github.com/milvus-io/milvus/issues/30932
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-29 15:55:01 +08:00
dcfc35316f
fix: underscore matching not work ( #30826 )
...
issue: https://github.com/milvus-io/milvus/issues/30721
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-29 10:41:00 +08:00
b023bd09b3
test: skip bulk insert test for one pod standalone ( #30780 )
...
skip bulk insert test for one pod standalone
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-29 10:31:08 +08:00
e59bc1e526
enhance: use latest milvus proto, add unimplemented server ( #30889 )
...
issue: #30647
Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 10:13:00 +08:00
50a78b682e
fix: set proxy.http.acceptTypeAllowInt64: true as default ( #30720 )
...
issue: #30680
also let the parameter item to be refreshable
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
3775464b7c
enhance: Support varchar autoid for bulkinsertV1 ( #30896 )
...
This PR is a supplement to PR
https://github.com/milvus-io/milvus/pull/30377 .
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-02-28 21:20:59 +08:00
095cdbed59
Update all contributors
...
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-02-28 12:01:10 +00:00
cdc5ce5d6f
fix: Donot set LogPath when executing compaction ( #30537 )
...
Compaction would copy logPaths from comapctFrom segA to compactTo segB,
and previous code would copy the logPath directly, causing there're
full-logPaths-of-segA in compactTo segB's meta. So, for the next
compaction of segB, if segA has been GCed, Download would report error
"The sperified key not found".
This PR makes sure compactTo segment's meta contains logID only. And
this PR also refines CompleteComapctionMutation, increasing some
readability and merge two methods into one.
See also: #30496
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-28 19:03:01 +08:00
4082315bd0
enhance: Add `ParseCTraceContext` util function for tracing ( #30883 )
...
See also #29803
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 18:59:00 +08:00
chyezh
congqixia
elstic
Ted Xu
jaime
sre-ci-robot
wei liu
cai.zhang
PowderLi
zhuwenxing
nico
yihao.dai
MrPresent-Han
groot
Bingyi Sun
wayblink
zhagnlu
SimFG
aoiasd
XuanYang-cn
yanliang567
Jiquan Long
sre-ci-robot