Cherry-pick from master
pr: #32291
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #32505
See also #32165
Change `LeaderViewFilter` to interface to provided map key to avoid
iterating all key-values in LeaderViewManager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #32515
See also #32403
`fg_buffer_size` was decreased after metrics removed in flowgraph
ddnode, which make metrics value negative.
This PR move remove metrics logic into `dataSyncService.Close`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #32441
See also #32440
- Add loadTask in collection observer
- For load collection/partitions, load task shall timeout as a whole
- Change related constructor to load jobs
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #32358
See also #31632
Release conc.Pool after usage to clean worker and stop background purge
and ticktock.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #32225
master pr: #32355
1. v1 can only accept one vector, but v2 accept list of vectors
2. cannot get dbName from AliasReq #31978
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
Cherry-pick from master
pr: #32273
See also #32165
`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.
This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to https://github.com/milvus-io/milvus/issues/32165
Avoid iterating through all replicas/collections if possible. Iteration
is expensive when there are large number of replicas/collections.
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Cherry-pick from master
pr: #32227
Related to #32165
Add collection to partitionIDs mapping to avoid interation on all
partitions loaded when trying to get all partitions with collection id
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #32029
pr: #32175
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
…#31621)
Tidy the following test codes
- Remove channel in newTestServer
- Remove newTestServerWithMeta
- Remove newTestServer2
- Remove testDataCoordBase
- Use the same func for handleTTmsg and handleRPCTTmsg
See also: #31620
pr: #31621
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #32107
pr: #32124
during rolling upgrade progress, skip to load l0 segment on old version
query node
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #31176
master pr: #32144
cannot get dbName correctly while describe alias
https://github.com/milvus-io/milvus/issues/31978
return a valid json string even if the user doesn't have the whole
privileges to describe collection
https://github.com/milvus-io/milvus/issues/31635
rename IndexParam.IndexConfig to IndexParam.Params
FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string
Signed-off-by: PowderLi <min.li@zilliz.com>
Cherry-pick from master
pr: #32172
See also #27675#26177
Make memory check evict memory buffer until memory water level is safe.
Also make `EvictBuffer` wait until sync task done.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Issue: https://github.com/milvus-io/milvus/issues/32129
pr: #32137
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Cherry-pick from master
pr: #32069
See also #32066
This PR make coordinator register successful and let
`ProcessActiveStandBy` run async. And roles may receive stop signal and
notify servers.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #31091
pr: #31133
This PR add GetByFilter interface in leader view manager, instead of all
kind of get func
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
SimFG
Gao
yihao.dai
congqixia
XuanYang-cn
zhagnlu
PowderLi
yiwangdr
Jiquan Long
wei liu
Chun Han
Cai Yudong
zhenshan.cao
Alexander Guzhva
jaime
Xiaofan