Commit Graph

21948 Commits (4bebca64161a962484ed73853ab43f4f9306ec01)

Author SHA1 Message Date
XuanYang-cn 4bebca6416
enhance: Replace currRows with NumOfRows ()
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-10 12:16:03 +08:00
Xiaofan 1c9d43ee9f
fix: rockmq race condition ()
fix 

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-10 10:54:04 +08:00
cai.zhang d6a650bd14
fix: Skip executing stats for zero segment ()
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-09 21:14:02 +08:00
smellthemoon faae8ee518
fix: store wrong offset when build tantivy in nullable field ()


Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-03-09 09:34:04 +08:00
sre-ci-robot 2c287dfd10 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2025-03-09 00:01:58 +00:00
Xiaofan 73bda08ec9
doc: json storage format ()
the design doc for the json storage improvemnet

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-08 09:48:02 +08:00
congqixia c348e61bdb
fix: [GoSDK] move metaheader to client in case of race ()
See failure run in 

This PR:
- move metaheader map to client struct from config
- set default value for field schema

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-07 17:06:02 +08:00
congqixia a56b24054f
enhance: [GoSDK] Support alter/drop index properties ()
Related to 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-06 21:40:04 +08:00
shaoyue 2cd03a20d4
enhance: [skip e2e] upgrade go toolchain to 1.22.7, remove unnecessary binary ()
part of: 

Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2025-03-06 21:34:04 +08:00
Bingyi Sun 37b118d55d
fix: Skip loading primary key if index has raw data ()
issue: https://github.com/milvus-io/milvus/issues/39907

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-06 17:46:02 +08:00
congqixia 7fbeb5624e
enhance: Avoid convert body byte slice to string in httpserver ()
The convertion of byte slice to string may copy the underline data which
may cause extra memory and cpu time for httpserver

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-06 16:28:02 +08:00
congqixia 5c5273f95e
fix: Pass Knapsnak ptr to avoid compact multiple times ()
Related to 

The small segments may be put into bucket twice due to value parameter
of Knapsnap.packWith

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-06 15:42:03 +08:00
nico 80213c0f6f
test: update teardown collection after rename ()
Signed-off-by: nico <cheng.yuan@zilliz.com>
2025-03-06 10:36:09 +08:00
nico ed61abb391
test: update test cases ()
Signed-off-by: nico <cheng.yuan@zilliz.com>
2025-03-06 10:34:14 +08:00
congqixia 16bc2e8087
enhance: [GoSDK] support update ts caching policy ()
Related to 

This PR add update timestamp check and retry policy according to the
design of the related issue

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-05 21:36:08 +08:00
congqixia 5b0881c93d
enhance: [GoSDK] Add DescribeReplica API & sync rbac v2 ()
Related to  

This PR:
- Add DescribeReplica API
- Add unified RBAC v2 API names(AddPrivilegesToGroup,
RemovePrivilegesFromGroup, GrantPrivilegeV2, RevokePrivilegeV2)
- Mark old ones deprecated

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-05 21:34:11 +08:00
sthuang e0ec1aceeb
fix: skip storage v2 unstable ut for now ()
related: 

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-05 20:30:39 +08:00
congqixia fde80bc8b7
enhance: Remove debug log in rg handler v2 ()
Remove debug log in resource group handler

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-05 19:18:00 +08:00
Spade A 3db56560fb
fix: fix concurrent issues in null offset ()
issue: 
This issue fixes these two concurrent issues:
1. element in null_offset is used to set bitset where the size of bitset
is initialized by tantivy document count. However, there may still be
some documents that are not committed in tantivy but are null in
null_offset. So array out of range occurs.
2. null_offset can be read and write concurrently but there's no
synchronization protection.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-03-05 17:48:00 +08:00
Ted Xu 878ce56079
fix: correct memory size estimation on arrays ()
See: 

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-03-05 16:54:09 +08:00
Zhen Ye 1637cf5664
enhance: better logging for grpc resolver ()
issue: 

- better logging for grpc resolver
- remove the redundant streaming node manage client when streaming
service is disable

Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-05 15:12:01 +08:00
Ted Xu 96952ad3c5
fix: compaction task cannot be genereted if size greater than max size ()
See: 

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-05 14:40:01 +08:00
yihao.dai 004a1875dc
enhance: Introduce batch subscription in msgdispatcher ()
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.

issue: https://github.com/milvus-io/milvus/issues/39862

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-05 14:38:02 +08:00
Bingyi Sun be4d09561b
fix: Fix missing null or non-exist key in json index ()
issue: 

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-05 11:48:02 +08:00
sthuang 63a7c4570e
feat: storage v2 sync ()
related: 

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-05 11:22:15 +08:00
SimFG a3755cf409
fix: improve error handling and unit tests for InitMetaCache function ()
- issue: 

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-05 11:08:13 +08:00
yanliang567 4437dae479
test: Add search iterator with external filter func neg tests ()
related issue: https://github.com/milvus-io/milvus/issues/39985

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-03-05 10:58:00 +08:00
junjiejiangjjj b2e630b1a1
feat: Support TEI serving and support int8 embedding ()


Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-05 10:18:00 +08:00
sthuang a6396ddda5
enhance: storage v2 clang format version 12 ()
change storage v2 clang-format version from 14 to 12 to match with
Milvus.
related: 

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-04 21:52:40 +08:00
congqixia c02763d49a
fix: [GoSDK] Pass only valid data for nullable column ()
Related to 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-04 20:58:04 +08:00
yanliang567 7d35d023a3
test: Remove the old get index params method ()
related issue: https://github.com/milvus-io/milvus/issues/40156
remove the old get index params method and update the new one

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-03-04 17:12:06 +08:00
Zhen Ye 9ca5088f62
fix: duplicate consuming from stream for invisble segment ()
issue: 

Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-04 15:54:00 +08:00
sthuang d77756cf2d
fix: fix storage v2 cgo mem leak ()
related: 

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-04 14:16:00 +08:00
XuanYang-cn 837ac295fa
enhance: Remove iterators in datanode ()
Iterators are long deprecated, but sort are still using it. This PR
unifies stats task with the latest compaction common functions and
remove the usage of iterators.

1. Rename `datanode/compaction` to `datanode/compactor`
2. Add `internal/compaction` and move some compaction commons into it.
3. Replace `DeltalogIterators` with `ComposeDeleteFromDeltalogs`
4. Remove `datanode/iterators`

See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-04 12:14:00 +08:00
Bingyi Sun 7040ba1c12
enhance: make json path index support term filter ()
issue: 

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-04 11:56:02 +08:00
congqixia f2ea4d6370
fix: Add duplication param check for create index ()
Related to 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-04 11:42:02 +08:00
zhuwenxing 45256c41d6
test: fix text match ()
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-03-03 21:26:00 +08:00
congqixia 16d468c875
fix: [GoSDK] Return role without grants ()
Related to 

Previousy DescribeRole returns only roles with grants, this PR add
select role action to check role existence.

Also added database properties related option

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-03 18:05:59 +08:00
Chun Han 16aa123185
fix: restful drop db properties failure() ()
related: 

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-03-03 15:39:59 +08:00
Zhen Ye f47ab31f23
enhance: remove redundant resource key watch operation, just keep consistency of wal ()
issue: 
related PR: 

- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-03 14:40:05 +08:00
nico 679a145e8b
test: update drop collection in teardown method ()
Signed-off-by: nico <cheng.yuan@zilliz.com>
2025-03-03 10:36:00 +08:00
sthuang de02a3ebcc
feat: Storage v2 binlog packed record reader and writer ()
related: 

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-03 10:24:02 +08:00
Zhen Ye 8eb662b4dc
enhance: add more metrics for async cgo component ()
issue: 

Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-03 09:56:03 +08:00
sre-ci-robot 6a57a1973f
[automated] Update Knowhere Commit ()
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-03-03 01:11:58 +08:00
cai.zhang 5a810400b5
enhance: Optimize Task Scheduling to Enable Concurrent Execution ()
issue:  

2.5 pr: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-02 18:38:00 +08:00
zhagnlu 7a17fb68ec
enhance: add monitor metric for retrieve raw data ()


Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-02 18:30:01 +08:00
Zhen Ye 944f018375
enhance: [skip e2e] only use mixcoord for scripts ()
issue: 

Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-02 18:05:58 +08:00
zhagnlu 8c19e5c4a7
enhance: decrease delete record dump snapshot limit ()


Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-02 17:55:59 +08:00
yihao.dai c368113233
fix: Fix task delta cache data race ()
issue: https://github.com/milvus-io/milvus/issues/40258

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-02 16:52:09 +08:00
Zhen Ye 2ff657f2d9
fix: wal may panics when context canceled ()
issue: 

- wal may panics when context canceled
- scanner may data race when closing

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-28 17:41:58 +08:00