Commit Graph

21748 Commits (ba312427f2e96a15e4a12261d245215a47f8a03e)

Author SHA1 Message Date
sre-ci-robot ba312427f2
[automated] Update Knowhere Commit (#39696)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-02-08 01:36:43 +08:00
sparknack 2d9bef44d4
fix: sparse: add inverted_index_algo and dim_max_score_ratio config (#39358)
issue: #39332

Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
2025-02-07 16:40:44 +08:00
yihao.dai 8288c06e3d
fix: Fix flowgraph leak (#39656)
Consider lag targets when determining whether to remove the dispatcher
manager.

issue: https://github.com/milvus-io/milvus/issues/39642

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-07 16:16:44 +08:00
Xianhui Lin 82f9689711
enhance: Add schema update time verification for insert and upsert to use cache (#39096)
enhance: Add schema update time verification for insert and upsert to
use cache
issue: https://github.com/milvus-io/milvus/issues/39093

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-02-07 14:10:45 +08:00
jaime 8a4ac8cccd
enhance: expose more metrics data (#39456)
issue: #36621 #39417
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-02-07 11:50:50 +08:00
Zhen Ye a9e0e0a852
enhance: broadcast with event-based notification (#39522)
issue: #38399

- broadcast message can carry multi resource key now.
- implement event-based notification for broadcast messages
- broadcast message use broadcast id as a unique identifier in message
- broadcasted message on vchannels keep the broadcasted vchannel now.
- broadcasted message and broadcast message have a common broadcast
header now.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-07 11:14:43 +08:00
congqixia 0bac4d3f2a
fix: [GoSDK] Unify AnnRequest creation and fix nil map (#39665)
Related to #39559

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-07 10:34:44 +08:00
congqixia d7c07d16d3
fix: [GoSDK] Use client config db name when connect (#39662)
Related to #34137

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-07 10:26:43 +08:00
aoiasd 74890dabc9
enhance: skip load bm25 sparse row data (#39078)
Skip load row data of bm25 sparse field, because it's forbidden to
return.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-06 18:40:44 +08:00
Ted Xu 427b6a4c94
enhance: reduce stats task cost by skipping ser/de (#39568)
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-02-06 17:14:45 +08:00
aoiasd 2b4caba76e
fix: check utf-8 format for varchar with analyzer open (#39299)
relate: https://github.com/milvus-io/milvus/issues/39285

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-06 17:11:51 +08:00
Zhen Ye 5669016af0
enhance: erase the rpc level when wal is located at same node (#38858)
issue: #38399

- Make the wal scanner interface same with streaming scanner.
- Use wal if the wal is located at current node.
- Otherwise fallback the old logic.

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-05 22:25:10 +08:00
Gao c1794cc490
enhance: update knowhere version and IsAdditionalScalarSupported interface (#39573)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-02-05 19:51:10 +08:00
cai.zhang 1d54ff157f
fix: Restore the compacting state for stats task during recovery (#39459)
issue: #39333

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-02-05 17:11:12 +08:00
Ted Xu f007465942
fix: unstable UT pack_writer_test (#39641)
See #39601

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-02-05 16:39:11 +08:00
wei liu 05ac4041aa
enhance: use rated logger for high frequency log in dist handler (#39452)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-02-05 15:31:10 +08:00
yihao.dai f0b7446e6b
enhance: Remove unnecessary collection and partition label from the metrics (#39536)
/kind improvement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-05 11:01:10 +08:00
Zhen Ye b1cee78a55
fix: segment stats may be inconsistent after wal closing (#39593)
issue: #38399

- The stats may be kept after wal closing if the growing segment is not
dirty.
- Change the error handling of wal open to avoid redundant manager api
call.

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-05 10:53:10 +08:00
Zhen Ye a816a03351
fix: unsafe concurrent consuming api of rocksmq (#39544)
issue: #38966

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-05 09:51:10 +08:00
congqixia b3791a6f90
fix: Use param item formatter to avoid SetConfig to overlay (#39597)
Related to #39596

When updating the build param configuration, the `Formatter` could be
used to do so and completed avoid touching the `overlay` config items

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-27 10:51:07 +08:00
congqixia f7b7bf69d1
enhance: Add dsl log field for HybridSearch (#39584)
The dsl fields are separated in sub request structs and cannot be
easiliy printed before. This PR adds a log helper to print the dsl
expressions of HybridSearchRequest.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-26 15:57:06 +08:00
sthuang 5f4bad6b23
fix: metastore privilege name check with privilege name all (#39476)
related: https://github.com/milvus-io/milvus/issues/39365

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-26 12:17:20 +08:00
congqixia e61a841806
enhance: [GoSDK] Support return recall in search result (#39580)
Related to #37899

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-24 19:05:06 +08:00
sthuang c4ae9f4ece
feat: introduce third-party milvus-storage (#39418)
related: https://github.com/milvus-io/milvus/issues/39173

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-24 17:21:13 +08:00
Cai Yudong f32830e016
feat: Support restful & go sdk for Int8Vector (#39278)
Issue: #38666

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-24 17:07:07 +08:00
Cai Yudong 5730b69e56
feat: Enable more VECTOR_INT8 unittest (#39569)
Issue: #38666

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-24 17:03:07 +08:00
Zhen Ye c84a0748c4
enhance: add rw/ro streaming query node replica management (#38677)
issue: #38399

- Embed the query node into streaming node to make delegator available
at streaming node.
- The embedded query node has a special server label
`QUERYNODE_STREAMING-EMBEDDED`.
- Change the balance strategy to make the channel assigned to streaming
node as much as possible.

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-24 16:55:07 +08:00
zhagnlu 8117d59f85
fix:fix GetValueFromConfig for bool type (#39526)
#39525

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-24 16:17:05 +08:00
congqixia 6d8441ad7e
enhance: Use mockery pkg config for datacoord&datanode (#39567)
Related to #38339

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-24 14:25:06 +08:00
junjiejiangjjj 16cbdfb3b1
feat: Add Text Embedding Function (#36366)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-01-24 14:23:06 +08:00
smellthemoon 47d280d974
fix: not enable rate limiter for restful v1 (#39553)
issue: #39556

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-24 14:05:13 +08:00
yihao.dai 5fb597b37b
fix: Remove frequently updating metric to avoid mutex contention (#38775)
issue: https://github.com/milvus-io/milvus/issues/37630

Reduce the frequency updating metrics to avoid holding the mutex for
long periods.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-24 10:31:07 +08:00
sre-ci-robot cd068f2c3a
[automated] Bump milvus version to v2.5.4 (#39548)
Bump milvus version to v2.5.4
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-01-24 10:29:15 +08:00
congqixia 05b6ea1351
enhance: Refine error msg for schema & index checking (#39533)
The error message was malformated or missing some meta info, say field
name. This PR recitfies some message format and add field name in error
message when type param check fails.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-24 09:57:06 +08:00
zhuwenxing 4cf268bcf1
test: bump deps in requirements (#39530)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-01-24 09:53:05 +08:00
laurazhao0611 a5217e9f36
test: add database milvus client testcases (#39505)
/kind improvement

---------

Signed-off-by: laurazhao0611 <laurazhao@zilliz.com>
Co-authored-by: laurazhao0611 <laurazhao@zilliz.com>
2025-01-23 18:45:07 +08:00
congqixia 844df76cc0
enhance: Rectify run_clang_format grep command (#39534)
Previously the grep with regex does not work and failed to match lots of
.cpp files

This PR:
- use "-E" flag to use regex match
- commit the fixed result of current cpp code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-23 17:07:05 +08:00
Ted Xu 56659bacbb
enhance: make serialization be part of sync task to support file format change (#38946)
See #38945

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-23 15:49:05 +08:00
zhuwenxing 176ef631bc
test: enable partition key isolatio testcases in ci (#39531)
related: https://github.com/milvus-io/milvus/issues/39407

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-01-23 14:33:06 +08:00
yihao.dai e0b26260f2
enhance: enable task delta cache (#39307)
When there are many segment tasks in the querycoord scheduler, the
traversal in `GetSegmentTaskDelta` checks becomes time-consuming. This
PR adds caching for segment deltas.

issue: https://github.com/milvus-io/milvus/issues/37630

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
2025-01-23 14:31:16 +08:00
ThreadDao e02fb41db8
test: add go-sdk cases for database properties (#39484)
issue: #33419 
- Add test cases for database properties
- Fixed typo AlterDatabaseProperies to AlterDatabaseProperties
- Rename UsingDatabase to UseDatabase
- Rename ListDatabases to ListDatabase

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2025-01-23 14:29:16 +08:00
yihao.dai 38f813bed3
enhance: Read metadata concurrently to accelerate recovery (#38403)
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.

issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-23 14:27:27 +08:00
yihao.dai 154edd237d
enhance: Skip creating partition rate limiters when not enable (#38816)
issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-23 14:25:17 +08:00
Spade A 547c686027
fix: fix assignment operator in AssertInfo to comparison operator (#39347)
fix: #39346

Remove the problem line as it's redundant.

---------

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-23 14:23:18 +08:00
congqixia 70c53f787e
enhance: [GoSDK] Pass `UseDefaultConsistency` flag for query option (#39515)
If the flag is not passed, the behaviors will not be expected when user
not passing consistency level.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-23 14:21:14 +08:00
Cai Yudong 7476eb3625
feat: Support bulk insert for Int8Vector (#39499)
Issue: #38666

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-23 10:19:06 +08:00
SimFG f070af67f7
fix: deny to set the mmap param for the alter index api when enable auto index (#39518)
- issue: #39517

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-23 10:01:04 +08:00
XuanYang-cn bb8cc6eb85
enhance: Add configs for compaction schedule (#39010)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-22 16:31:10 +08:00
congqixia 1a1ed07cfa
enhance: Skip update index metrics if index dropped (#39458)
Related to #39457

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-22 11:19:04 +08:00
laurazhao0611 41352e40e4
test: add search iterator cases and alter collection properties (#39406)
/kind improvment

---------

Signed-off-by: laurazhao0611 <laurazhao@zilliz.com>
Co-authored-by: laurazhao0611 <laurazhao@zilliz.com>
2025-01-22 10:41:04 +08:00