Commit Graph

19866 Commits (648d5661ca8771fadc664f427ac330b083b8734e)

Author SHA1 Message Date
Alexander Guzhva 648d5661ca
enhance: Upgrade bitset for ARM SVE (#32718)
issue: #32826 
improve ARM SVE performance for `internal/core/src/bitset`

Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83

Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-22 11:37:40 +08:00
XuanYang-cn 819a624753
fix: Return error when startup Delete/AddNode fail (#33193)
See also: #33151, #33149

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-22 11:17:38 +08:00
Alexander Guzhva f20becb725
fix: Download and install cmake for the current platform, not x86_64 only (#32548)
issue #32476 

tested on x86_64 and aarch64. I'm not sure what needs to be done on some
exotic architectures.

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-05-22 11:15:39 +08:00
wei liu 303470fc35
fix: Clean offline node from resource group after qc restart (#33232)
issue: #33200 #33207
pr#33104 causes the offline node will be kept in resource group after qc
recover, and offline node will be assign to new replica as rwNode, then
request send to those node will fail by NodeNotFound.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-22 10:03:40 +08:00
Xiaofan 3d105fcb4d
enhance: Remove l0 delete cache (#32990)
fix #32979
remove l0 cache and build delete pk and ts everytime. this reduce the
memory and also increase the code readability

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-21 22:53:40 +08:00
SimFG e18d5aceb6
enhance: add config to control whether to init public role permissions (#33165)
issue: #33164

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-21 22:39:46 +08:00
cai.zhang ed39a38953
enhance: Reduce the frequency of logs describing indexing failures (#33212)
issue: #33001 #33102

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-21 19:27:39 +08:00
congqixia 12e8c6c583
enhance: Try LatestMessageID when checkpoint unmarshal fails (#33158)
See also #33122

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 16:59:39 +08:00
sammy.huang 7ab7e3a004
feat: support arm-based image build and pull request (#33219)
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-05-21 16:54:38 +08:00
yihao.dai 017fd7bc25
enhance: Select L2 segments in L0Compaction as well (#32991)
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-21 16:13:39 +08:00
wei liu 33bd6eed28
fix: Clean offline node from replica after qc recover (#33213)
issue: #33200 #33207

pr#33104 remove this logic by mistake, which cause the offline node will
be kept in replica after qc recover, and request send to offline qn will
go a NodeNotFound error.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-21 15:41:39 +08:00
Cai Yudong cb480d17c8
fix: Fix SparseFloatVector data parse error for parquet (#33187)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-21 15:09:39 +08:00
XuanYang-cn 2d6f12d48b
fix: channel manager's goroutine run order (#33118)
See also: #33117

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-21 14:35:39 +08:00
congqixia f336b2d672
fix: Check schema without vector field in proxy (#33211)
Related to #33199

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 14:33:39 +08:00
wei liu 2013d97243
enhance: Enable to dynamic update balancer policy in querycoord (#33037)
issue: #33036
This PR enable to dynamic update balancer policy without restart
querycoord.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-21 14:29:39 +08:00
Xiaofan f681c4b034
enhance: remove describe index in rootcoord broker (#33206)
fix #33205
remove the dependency between datacoord and rootcoord

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-21 14:11:39 +08:00
congqixia f31a20faad
fix: [Backport] Mark channel checkpoint dropped prevent cp lag metrics leakage (#32454) (#33198)
Cherry-pick from 2.3
pr: #32454
See also #31506 #31508

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 11:59:39 +08:00
Jiquan Long 9f81290c63
fix: try best to get enough query results (#33178)
issue: https://github.com/milvus-io/milvus/issues/33137

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-21 11:57:51 +08:00
Bingyi Sun 0f8c6f49ff
enhance: mmap load raw data if scalar index does not have raw data (#33175)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-21 11:53:39 +08:00
XuanYang-cn b3bcc107bb
fix: Remove L0 compactor in completedCompactor (#33169)
See also: #33168

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-21 11:35:38 +08:00
aoiasd f8929cc36a
fix: can't generate traceID when use noop exporter (#33191)
relate: https://github.com/milvus-io/milvus/issues/33190

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-21 10:39:39 +08:00
smellthemoon 89ad3eb0ca
enhance: reduce memory when read field (#33195)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-20 22:25:38 +08:00
jaime 0d99db23b8
fix: metrics leak on the coord nodes (#33075)
issue: #32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-20 22:03:39 +08:00
shaoting-huang d27db99697
enhance: upgrade amazonlinux2023 builder image go version to 1.21 (#33176)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]

issue: https://github.com/milvus-io/milvus/issues/32982

Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. This PR is mainly for upgrading images Golang
version from 1.20 to 1.21.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-20 21:11:39 +08:00
congqixia 7eeb120aab
enhance: Add lint rules for client pkg and fix problems (#33180)
See also #31293

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 20:47:38 +08:00
pingliu a1b5d7d6d3
doc: [skip-e2e] change milvus docker image version to v2.4.1 (#33170)
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-05-20 16:21:38 +08:00
shaoting-huang c35eaaa358
enhance: upgrade images golang version from 1.20 to 1.21 (#33150)
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]

issue: https://github.com/milvus-io/milvus/issues/32982

Go 1.21 introduces several improvements and changes over Go 1.20, which
is quite stable now. This PR is mainly for upgrading images Golang
version from 1.20 to 1.21.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-20 15:01:43 +08:00
congqixia f76d16780b
enhance: Refine channel mgr v2 implementation (#33156)
Related to #25309

- Remove ctx from struct
- Add ctx parameters for internal check logic methods
- Add Waitgroup to make sure worker goroutine quit before close returns

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 14:13:38 +08:00
sre-ci-robot 555df49d25
[automated] Update Pytest image changes (#33126)
Update Pytest image changes
See changes:
0d0eda24f8
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-20 11:43:37 +08:00
congqixia c2ac692008
enhance: Add param item to ignore bad message id in checkpoint (#33123)
See also #33122

This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 11:29:37 +08:00
SimFG ec98de3ad4
fix: reset the quota value when init the limiter (#33111)
issue: #33107
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-20 10:35:38 +08:00
wei liu a7f6193bfc
fix: query node may stuck at stopping progress (#33104)
issue: #33103 
when try to do stopping balance for stopping query node, balancer will
try to get node list from replica.GetNodes, then check whether node is
stopping, if so, stopping balance will be triggered for this replica.

after the replica refactor, replica.GetNodes only return rwNodes, and
the stopping node maintains in roNodes, so balancer couldn't find
replica which contains stopping node, and stopping balance for replica
won't be triggered, then query node will stuck forever due to
segment/channel doesn't move out.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-20 10:21:38 +08:00
sre-ci-robot c6e2dd05fc
[automated] Update Knowhere Commit (#33147)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-20 01:51:37 +08:00
zhagnlu d669fbcf46
enhance: support bitmap index for scalar type (#32902)
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-19 21:49:38 +08:00
Xiaofan 5cc38aa9f8
Update README.md 2024-05-18 22:23:00 +08:00
Xiaofan 5e6da3e994
Update README.md 2024-05-18 22:22:30 +08:00
sre-ci-robot c81be9ed70 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-05-17 12:00:59 +00:00
yihao.dai 32560263fa
enhance: Query slot for compaction task (#32881)
Query slot of compaction in datanode, and transfer the control logic for
limiting compaction tasks from datacoord to the datanode.

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-17 18:19:38 +08:00
Bingyi Sun 5c6de473bb
fix: check array field data is nil before getting the type (#33114)
issue: https://github.com/milvus-io/milvus/issues/33074

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-17 17:29:37 +08:00
yihao.dai bcaacf6fe6
enhance: Load BF from storage instead of memory during L0 compaction (#32913)
To decouple compaction from shard, loading BF from storage instead of
memory during L0 compaction in datanode.

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-17 17:25:36 +08:00
nico 0d0eda24f8
test: modify some test cases (#33096)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-05-17 15:07:37 +08:00
Ted Xu a9c7ce72b8
enhance: enable stream writer in compactions (#32612)
See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-05-17 15:05:37 +08:00
Cai Yudong b560602885
enhance: Store SparseFloatVector into parquet as JSON string (#33101)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-17 15:01:37 +08:00
Yinzuo Jiang 2cc50d80a3
fix: add openblas in install_deps.sh (#33065)
Install openblas using apt or yum in scripts/install_deps.sh, update
documentations and fix some typos related to build and installation.

issue: #33056, #33066

Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-05-17 14:53:37 +08:00
smellthemoon 225f4a6134
enhance: use the only MaxEtcdTxnNum (#33070)
#33071

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-17 14:27:42 +08:00
congqixia 1ef975d327
enhance: Update latest sdk update to client pkg (#33105)
Related to #31293
See also milvus-io/milvus-sdk-go#704 milvus-io/milvus-sdk-go#711 
milvus-io/milvus-sdk-go#713 milvus-io/milvus-sdk-go#721
milvus-io/milvus-sdk-go#732 milvus-io/milvus-sdk-go#739 
milvus-io/milvus-sdk-go#748

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-17 10:39:37 +08:00
wei liu f1c9986974
enhance: Skip return data distribution if no change happen (#32814)
issue: #32813

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-17 10:11:37 +08:00
sre-ci-robot 5b27a0cd2c Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-05-16 12:01:03 +00:00
smellthemoon 1671c7898a
enhance: sync milvus proto version (#33094)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-16 18:51:34 +08:00
binbin be87ae6424
test: update pymilvus version (#33079)
issue: #17614

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-05-16 14:53:34 +08:00