Commit Graph

8557 Commits (454984aa4e963a6cafe67f73292df29e07cc6a94)

Author SHA1 Message Date
sre-ci-robot 454984aa4e
[automated] Update Knowhere Commit (#32181)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-12 01:25:19 +08:00
jaime 371e6d2c1a
enhance: refine sync memory watermark configuration (#32140)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-11 20:07:24 +08:00
Alexander Guzhva b5455d176e
fix: dynamically resolve whether SVE is available for bitset (#32137)
Issue: #32129 
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-11 19:27:18 +08:00
congqixia b9a487608a
fix: Make `ResourceGroup.nodes` concurrent safe (#32159)
See also #32158

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-11 17:53:18 +08:00
zhenshan.cao 0bfe130991
fix: fix metric about ProxySearchVectors (#32152)
issue :https://github.com/milvus-io/milvus/issues/25639

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-11 16:13:17 +08:00
Patrick Weizhi Xu 52ae47c850
enhance: gather materialized view search info once per request (#31996)
issue: #29892 

This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-11 15:21:19 +08:00
Ted Xu dc5ea6f17c
feat: adding binlog streaming writer (#31537)
See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-04-11 10:33:20 +08:00
yihao.dai 273df98e20
enhance: Add binlog import intergration test (#32112)
issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-11 10:31:18 +08:00
SimFG 789e014c74
enhance: add the db id for the describe collection response (#32114)
/kind improvement
issue: #32110

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 21:23:20 +08:00
jaime d4fd6c7283
enhance: add db label on binlog size metrics (#32003)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-10 21:01:20 +08:00
wei liu 1a98ce39f5
enhance: Remove useless logic about FromShardLeader (#32029)
issue: #32047

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 20:11:19 +08:00
Buqian Zheng 2fdf1a6e76
feat: [Sparse Float Vector] added some integration tests (#31062)
add some integration tests for sparse float vector support

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-10 19:57:18 +08:00
congqixia 25a1c9ecf0
fix: Make coordinator `Register` not blocked on ProcessActiveStandby (#32069)
See also #32066

This PR make coordinator register successful and let
`ProcessActiveStandBy` run async. And roles may receive stop signal and
notify servers.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-10 18:49:18 +08:00
chyezh a3d6110957
fix: ut failure (#32120)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 17:30:48 +08:00
yihao.dai 1b5554c8cb
enhance: Support $meta key for json import (#32013)
During JSON import:
1. Allow the specification of the $meta key
2. Prohibit duplicated keys within the $meta field, for instance,
`{"id": 1, "vector": [], "x": 6, "$meta": {"x": 8}}`

issue: https://github.com/milvus-io/milvus/issues/31835

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-10 17:27:17 +08:00
chyezh 0be67e7f99
fix: ut failure (#32119)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 17:23:27 +08:00
Xiaofan dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly (#32089)
fix https://github.com/milvus-io/milvus/issues/32059

this pr fix two issues:

offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 16:01:18 +08:00
wei liu df208d538c
fix: Check exclude segment before add new growing segment (#31803)
issue: #31479 #31797

milvus will add released segment to excluded info, and filter out it's
stream data in filter_node. but for data buffered in insert_node's
channel, if it belongs to growing segment which already be released,
then it will all the growing segment back again.

This PR maintain `excluded segments` in delegator, and check excluded
segment before new growing segment.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 15:29:17 +08:00
Chun Han f3f2a5a7e9
fix: evicted segments in the serverlss mode(#31959) (#31961)
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-10 15:15:19 +08:00
wei liu c4806b69c4
enhance: Refactor leader view manager interface (#31133)
issue: #31091
This PR add GetByFilter interface in leader view manager, instead of all
kind of get func

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 15:13:36 +08:00
XuanYang-cn aad3ed3835
fix: [cherry-pick]Skip changing meta if nodeID not match with channel (#31672)
See also: #31648
pr: #31665, #31694

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-10 15:09:18 +08:00
SimFG 90bed1caf9
enhance: add the related data size for the read apis (#31816)
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
chyezh c9faa6d936
enhance: add more metrics for cache and search (#31777)
issue: #30931

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
SimFG a63cf8625c
enhance: support to invalid the database meta cache (#32078)
issue: #32077
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 06:45:18 +08:00
SimFG 420baacb6d
enhance: use the rate log in the privilege interceptor (#32037)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-09 16:19:17 +08:00
Cai Yudong a0a4ec8b67
enhance: make range search param check message more meaningful (#32006)
Issue: #31970

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-09 16:17:26 +08:00
wei liu 177ddda47f
fix: Check stale should check leader task's leader id (#31962)
issue: #30816

check stale rules for leader task:
1. for reduce leader task, it should keep executing until leader's node
become offline.
2. for grow leader task,it should keep executing until leader's node
become stopping.

This PR check leader node's stopping state for grow leader task

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-09 15:33:25 +08:00
aoiasd 5b693c466d
fix: delegator filter out all partition's delete msg when loading segment (#31585)
May cause deleted data queryable a period of time.
relate: https://github.com/milvus-io/milvus/issues/31484
https://github.com/milvus-io/milvus/issues/31548

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-09 15:21:24 +08:00
zhenshan.cao 089c805e0a
enhance:Refactor hybrid search (#32020)
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
cqy123456 8fda3cbeda
enhance: check fp16/bf16 nan or inf value (#31840)
issue:https://github.com/milvus-io/milvus/issues/22837

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-09 14:19:27 +08:00
cai.zhang 1b767669a4
enhance: Throw error instead of crash when index cannot be built (#31844)
issue: #27589

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-04-09 11:51:18 +08:00
yiwangdr 1cd15d9322
test: support segment release in integration test (#31190)
issue: #29507

Notice that api_testonly.go files should be guarded by compiler tag
`test`, so that production build rules don't compile them and these APIs
don't get misused.

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-09 11:39:17 +08:00
congqixia 1f7f3993a1
fix: Validate PlaceholderGroups before combine them (#32016)
See also #32015

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-09 11:33:17 +08:00
congqixia 7e0d03420e
enhance: Remove duplicated err check code (#32001)
The `err != nil` check is duplicated here and shall be removed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-08 21:59:17 +08:00
yihao.dai 49d109de18
enhance: Use an individual buffer size parameter for imports (#31833)
Use an individual buffer size parameter for imports and set buffer size
to 64MB.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-08 21:07:18 +08:00
chyezh 73adf2a5cc
fix: use stateful lock to avoid load and release on LocalSegment concurrently (#31606)
issue: #31605

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 17:09:16 +08:00
chyezh 7b400252ff
fix: add configuration disk capacity config for lru and fix some bug (#31977)
issue: #30361

- Add configurable disk capacity limit

- fix bitset reset logic

- make insert record reinsert after clear

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
Cai Yudong 00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
cqy123456 aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. (#31771)
issue:https://github.com/milvus-io/milvus/issues/22837

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-07 14:13:16 +08:00
yihao.dai d6cdcf74db
fix: Return err for conc.Future in sync manager (#31790)
Should not return `err, nil` when using conc.Future, as the error will
be lost/ignored when using `AwaitAll` to wait for the future.

issue: https://github.com/milvus-io/milvus/issues/31788

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-06 11:36:57 -07:00
yihao.dai 23ba2a5388
enhance: Remove deprecated DataCoordNumStoredRowsCounter metric (#31778)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-07 00:25:15 +08:00
Alexander Guzhva cae5722229
enhance: performance improvements for the bitset (#31753)
Issue: #31752

This PR improves the performance for bitset utilities (introduced in PR
#30454), including varchar filtering

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-04-06 05:19:22 +08:00
zhagnlu b2669e26dc
fix:reduce thread pool test time (#31893)
#31877

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-05 10:05:12 +08:00
chyezh a2502bde75
enhance: replica manager enhancement (#31496)
issue: #30647 

- ReplicaManager manage read only node now, and always do persistent of
node distribution of replica.

- All segment/channel checker using ReplicaManager to get read-only node
or read-write node, but not ResourceManager.

- ReplicaManager promise that only apply unique querynode to one replica
in same collection now (replicas in same collection never hold same
querynode at same time).

- ReplicaManager promise that fairly node count assignment policy if
multi replicas of collection is assigned to one resource group.

- Move some parameters check into ReplicaManager to avoid data race.

- Allow transfer replica to resource group that already load replica of
same collection

- Allow transfer node between resource groups that load replica of same
collection

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-05 04:57:16 +08:00
zhagnlu d6d3b01a04
fix:remove thread pool timeout test because of high load cpu (#31879)
#31877

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-03 15:55:38 +08:00
Jiquan Long 03e0db109e
fix: udpate Cargo.lock (#31859)
issue: #31681

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-03 14:18:23 +08:00
congqixia 49b8ee4339
fix: Make FlushTs Sync Policy apply to all buffers (#31839)
See also #30552

FlushTS policy was orignally designed to flushed/L0 segments only, but
in some edge case, new growing segment buffer would by-pass flush
request and hold a buffer before flush ts, which caused flush timeout

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-03 11:47:13 +08:00
congqixia 1af2ee903c
fix: Check parse QueryInfo error before using it (#31836)
See also milvus-io/milvus-sdk-go#693 #31735

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-03 11:29:14 +08:00
congqixia c2aad513c0
fix: Check collection nil before check load status (#31850)
See also #31849

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-03 10:07:13 +08:00
congqixia 56e371c478
fix: Check replica exists before get latest leader (#31848)
See also #31847

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-03 10:05:22 +08:00