32d5c5b3c3
fix: add bitset sve support macro protect ( #32464 )
...
pr: #32463
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-29 14:01:32 +08:00
d05926f7ac
fix: remove sve flags ( #32275 )
...
pr: #32270
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-29 14:01:32 +08:00
018a784989
enhance: speed up GetByCollection/AndNode (2.4 patch) ( #32234 )
...
Related to https://github.com/milvus-io/milvus/issues/32165
Avoid iterating through all replicas/collections if possible. Iteration
is expensive when there are large number of replicas/collections.
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-15 19:53:21 +08:00
9e3099e20e
enhance: [2.4] Maintain collection-patitions mapping in qc meta ( #32227 ) ( #32249 )
...
Cherry-pick from master
pr: #32227
Related to #32165
Add collection to partitionIDs mapping to avoid interation on all
partitions loaded when trying to get all partitions with collection id
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-15 18:15:19 +08:00
2726d639f9
fix: fix collectionInfo leak in datacoord ( #32243 )
...
issue: #32029
pr: #32175
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 17:21:19 +08:00
0df0972eb4
enhance: support disable search optimization ( #32142 )
...
pr: https://github.com/milvus-io/milvus/pull/32141
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-04-15 17:17:20 +08:00
0a3a483d02
enhance: [cherry-pick]Use channel manager interface in server_test (… ( #32211 )
...
…#31621)
Tidy the following test codes
- Remove channel in newTestServer
- Remove newTestServerWithMeta
- Remove newTestServer2
- Remove testDataCoordBase
- Use the same func for handleTTmsg and handleRPCTTmsg
See also: #31620
pr: #31621
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-15 11:57:20 +08:00
e50599ba10
fix: Skip to load l0 segment on old version query node ( #32131 )
...
issue: #32107
pr: #32124
during rolling upgrade progress, skip to load l0 segment on old version
query node
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 11:23:23 +08:00
b68af208bc
fix: Use pk from binlog during import ( #32118 ) ( #32194 )
...
During binlog import, even if the primary key's autoID is set to true,
the primary key from the binlog should be used instead of being
reassigned.
issue: https://github.com/milvus-io/milvus/discussions/31943 ,
https://github.com/milvus-io/milvus/issues/28521
pr: https://github.com/milvus-io/milvus/pull/32118
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-15 10:31:19 +08:00
5f7edc7304
fix: lack good results for insufficient ef( #29883 ) ( #32151 )
...
related: #29883
pr: https://github.com/milvus-io/milvus/pull/32080
see also: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-13 22:13:23 +08:00
9cb640fe78
fix: Fix import hanging and improve logging output ( #32166 ) ( #32167 )
...
Fix import hanging when the previous import task failed, and improve
parquet import logging outout.
issue: https://github.com/milvus-io/milvus/issues/31834
pr: https://github.com/milvus-io/milvus/pull/32166
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-13 22:03:23 +08:00
3946368e82
fix: restful v2 ( #32160 )
...
issue: #31176
master pr: #32144
cannot get dbName correctly while describe alias
https://github.com/milvus-io/milvus/issues/31978
return a valid json string even if the user doesn't have the whole
privileges to describe collection
https://github.com/milvus-io/milvus/issues/31635
rename IndexParam.IndexConfig to IndexParam.Params
FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-13 21:55:29 +08:00
3a1e0c3b18
fix: put inverted index into local storage ( #32209 ) ( #32210 )
...
issue: https://github.com/milvus-io/milvus/issues/32154
pr: #32209
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-13 21:53:18 +08:00
e00772a098
fix: [2.4] Wait StandBy server ready for testcase ( #32216 ) ( #32230 )
...
Cherry-pick from master
pr: #32216
See also #32069
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-13 12:03:23 +08:00
9613d368c9
fix: fix estimate float16 field data size wrong ( #32196 )
...
issue: #32192
pr: #32193
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-12 16:29:26 +08:00
e18ddfc06d
enhance: [Cherry-pick] Make write buffer memory check do until safe ( #32172 ) ( #32201 )
...
Cherry-pick from master
pr: #32172
See also #27675 #26177
Make memory check evict memory buffer until memory water level is safe.
Also make `EvictBuffer` wait until sync task done.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-12 16:03:20 +08:00
cbac23a8ff
fix: failed to raise metric_type not match error ( #32203 )
...
issue:https://github.com/milvus-io/milvus/issues/32176
pr:https://github.com/milvus-io/milvus/pull/32202
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-12 15:57:19 +08:00
e495073e4b
fix: Use correct ts to avoid exclude segment list leak ( #32191 )
...
issue: #31990
pr: #31991
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-12 15:11:19 +08:00
5457829660
fix: [Cherry-pick] Make `ResourceGroup.nodes` concurrent safe ( #32159 ) ( #32200 )
...
Cherry-pick from master
pr: #32159
See also #32158
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-12 15:05:20 +08:00
6d7c6db49c
fix: fix incomplete hybrid search result when nq > 1 ( #32178 )
...
pr: https://github.com/milvus-io/milvus/pull/32177
issue: https://github.com/milvus-io/milvus/issues/25639
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-12 10:31:19 +08:00
a72a9cd9cb
fix: [2.4] [Cherry-pick] dynamically resolve whether SVE is available for bitset ( #32139 )
...
Issue: https://github.com/milvus-io/milvus/issues/32129
pr: #32137
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-12 10:01:20 +08:00
5b45debb28
enhance: refine sync memory watermark configuration ( #32138 )
...
pr: #32140
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-11 20:07:24 +08:00
7f683000e9
fix: [2.4] Make coordinator `Register` not blocked on ProcessActiveStandby( #32069 ) ( #32132 )
...
Cherry-pick from master
pr: #32069
See also #32066
This PR make coordinator register successful and let
`ProcessActiveStandBy` run async. And roles may receive stop signal and
notify servers.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-11 17:31:19 +08:00
329e73b58b
fix: partitionIDs should be union of partitionIDs of each subquery ( #32147 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-11 15:41:19 +08:00
246ef454dc
enhance: Add binlog import intergration test ( #32112 ) ( #32146 )
...
issue: https://github.com/milvus-io/milvus/issues/28521
pr: https://github.com/milvus-io/milvus/pull/32112
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-11 15:17:20 +08:00
d11309b37f
enhance: Support $meta key for json import ( #32013 ) ( #32121 )
...
During JSON import:
1. Allow the specification of the $meta key
2. Prohibit duplicated keys within the $meta field, for instance,
`{"id": 1, "vector": [], "x": 6, "$meta": {"x": 8}}`
issue: https://github.com/milvus-io/milvus/issues/31835
pr: https://github.com/milvus-io/milvus/pull/32013
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-11 15:15:28 +08:00
4ea2a98fa3
enhance: [2.4] support to invalid the database meta cache ( #32092 )
...
issue: #32077
pr: #32078
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-11 15:01:20 +08:00
ada0e8c931
fix: reduce didn't handle offset without limit and reduceStopForBest … ( #32088 )
...
fix https://github.com/milvus-io/milvus/issues/32059
pr: https://github.com/milvus-io/milvus/pull/32089
this pr fix two issues:
offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 21:21:07 -07:00
35f39593aa
enhance: [2.4] add the db id for the describe collection response ( #32123 )
...
/kind improvement
issue: #32110
pr: #32114
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-11 10:25:19 +08:00
94e35793c6
enhance: Refactor leader view manager interface ( #31133 ) ( #32127 )
...
issue: #31091
pr: #31133
This PR add GetByFilter interface in leader view manager, instead of all
kind of get func
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-11 10:01:23 +08:00
a70cc920ed
enhance: Remove useless logic about FromShardLeader ( #32134 )
...
issue: #32047
pr: #32029
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 21:49:18 +08:00
c8aacc6d59
feat: [2.4] [Sparse Float Vector] added some integration tests ( #32093 )
...
add some integration tests for sparse float vector support
#29419
pr: https://github.com/milvus-io/milvus/pull/31062
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-10 21:47:29 +08:00
bb2ffd47f6
enhance: Add db label for some usual metrics ( #32024 )
...
issue: https://github.com/milvus-io/milvus/issues/31782
pr: #30956
cherry pick: #30956 #32003 #31715
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2024-04-10 20:59:20 +08:00
fcef1c54da
fix: Check exclude segment before add new growing segment ( #31803 ) ( #32115 )
...
issue: #31479 #31797
pr: #31803
milvus will add released segment to excluded info, and filter out it's
stream data in filter_node. but for data buffered in insert_node's
channel, if it belongs to growing segment which already be released,
then it will all the growing segment back again.
This PR maintain `excluded segments` in delegator, and check excluded
segment before new growing segment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 19:03:20 +08:00
c85cd01557
fix: [2.4] Validate PlaceholderGroups before combine them ( #32016 ) ( #32042 )
...
Cherry-pick from master
pr: #32016
See also #32015
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-10 16:57:26 +08:00
e3821f5a48
enhance: [Cherry-pick] Remove duplicated err check code ( #32001 ) ( #32079 )
...
Cherry-pick from master
pr: #32001
The `err != nil` check is duplicated here and shall be removed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-10 15:43:18 +08:00
8f99144565
enhance: [cherry-pick]check fp16/bf16 nan or inf value ( #31842 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/31840
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-10 10:43:27 +08:00
4a4935f7b2
enhance: [2.4] use the rate log in the privilege interceptor ( #32038 )
...
/kind improvement
pr: #32037
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-09 16:37:17 +08:00
8475a63d72
fix: Check stale should check leader task's leader ( #31995 )
...
issue: #30816
pr: #31962
check stale rules for leader task:
for reduce leader task, it should keep executing until leader's node
become offline.
for grow leader task,it should keep executing until leader's node become
stopping.
This PR check leader node's stopping state for grow leader task
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-09 15:33:25 +08:00
4d7ae52d62
fix: [Cherry-Pick] delegator filter out all partition's delete msg when loading segment ( #31588 )
...
May cause deleted data queryable a period of time.
relate: https://github.com/milvus-io/milvus/issues/31484
pr: https://github.com/milvus-io/milvus/pull/31585
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-09 15:21:24 +08:00
4c07304790
enhance: Refactor hybrid search ( #31742 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368
pr :https://github.com/milvus-io/milvus/pull/32020
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 10:15:18 +08:00
39d988cf8d
enhance: Use an individual buffer size parameter for imports ( #31833 ) ( #31937 )
...
Use an individual buffer size parameter for imports and set buffer size
to 64MB.
issue: https://github.com/milvus-io/milvus/issues/28521
pr: https://github.com/milvus-io/milvus/pull/31833
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-08 21:05:17 +08:00
1cfc681461
enhance:remove openblas from conan ( #31413 )
...
issue: #30683
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-08 19:17:17 +08:00
215d314118
fix: Return err for conc.Future in sync manager ( #31790 ) ( #31936 )
...
Should not return `err, nil` when using conc.Future, as the error will
be lost/ignored when using `AwaitAll` to wait for the future.
issue: https://github.com/milvus-io/milvus/issues/31788
pr: https://github.com/milvus-io/milvus/pull/31790
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-08 14:41:16 +08:00
3df7f57e4d
enhance: Remove deprecated DataCoordNumStoredRowsCounter metric ( #31778 ) ( #31935 )
...
pr: https://github.com/milvus-io/milvus/pull/31778
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-08 14:31:16 +08:00
4b430097dd
enhance: Unify data type check APIs for go ( #31945 )
...
cherry-pick from master
pr: #31887
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-08 14:11:17 +08:00
958f933810
fix: [Cherry-pick] Check collection nil before check load status ( #31850 ) ( #31897 )
...
Cherry-pick from master
pr: #31850
See also #31849
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-08 10:01:16 +08:00
312d570855
fix: crash on arm64 ( #31951 )
...
issue: #31953
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-08 02:15:15 +08:00
7580fa8e75
fix: [cherry-pick]fix some fp16/bf16 code missing in segcore ( #31841 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/31771
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-07 14:27:22 +08:00
841f9e4f80
enhance: [Cherry-pick] performance improvements for the bitset ( #31754 )
...
Issue: #31752
see also: #30454
pr: #31753
This PR improves the performance for bitset utilities (introduced in PR
https://github.com/milvus-io/milvus/pull/30454 ), including varchar
filtering
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-06 05:19:21 +08:00
zhagnlu
yiwangdr
congqixia
wei liu
Gao
XuanYang-cn
yihao.dai
Chun Han
PowderLi
Jiquan Long
Cai Yudong
zhenshan.cao
Alexander Guzhva
jaime
SimFG
Xiaofan
Buqian Zheng
cqy123456
aoiasd
presburger