Commit Graph

71 Commits (2.6)

Author SHA1 Message Date
Bingyi Sun 5cd2d99799
enhance: Revert "feat: encode cluster id in auto id (#44324)" (#44426)
This reverts commit 7af1594103
2025-09-17 17:56:01 +08:00
Bingyi Sun 7af1594103
feat: encode cluster id in auto id (#44324)
https://github.com/milvus-io/milvus/issues/44326
prev:
`[physical_ts][logical_ts]`
after
`[sign_bit][cluster_id][physical_ts][logical_ts]`

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-09-17 16:56:01 +08:00
Bingyi Sun e2eb8562f1
feat: Auto add namespace field data if namespace is enabled (#44198)
issue: #44011

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-09-09 16:17:56 +08:00
Bingyi Sun e3ecacca9e
feat: Add namespace prop (#43962)
issue: https://github.com/milvus-io/milvus/issues/44011
namespace is an alias for tenant. if this property is enabled, milvus
will add a __namespace_id field.
Modifications in the future will use this property to do compaction and
search.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-09-03 12:57:53 +08:00
congqixia aa4ef9c996
feat: Support enabling dynamic schema on existing collection (#44151)
Related to #44150

This PR make enabling `dynamic schema` feature for an existing
collection possible.

This related API is to reuse `AlterCollection` and underhood its
redirected to `adding nullable json field`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-09-02 15:51:52 +08:00
zhagnlu fc876639cf
enhance: support json stats with shredding design (#42534)
#42533

Co-authored-by: luzhang <luzhang@zilliz.com>
2025-09-01 10:49:52 +08:00
congqixia d3fa305785
enhance: Add grpc metadata header for client request time (#44059)
Related to #44058

This PR:
- Add common grpc metadata key for client request time
- Add gosdk & milvus inteceptor related logic for this attribute
- Bump go sdk version

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-08-27 14:27:49 +08:00
Xiaofan bd31b32167
fix: hybridsearch should support offset param in restful api (#43586)
Add support of offset param for reqeustful. api and refine some constant
usage related #43556

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-07-28 22:15:36 +08:00
Spade A faeb7fd410
feat: impl StructArray -- create schema, insert, and retrieve data (#42855)
Ref https://github.com/milvus-io/milvus/issues/42148

https://github.com/milvus-io/milvus/pull/42406 impls the segcore part of
storage for handling with VectorArray.
This PR:
1. impls the go part of storage for VectorArray
2. impls the collection creation with StructArrayField and VectorArray
3. insert and retrieve data from the collection.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
2025-07-27 01:30:55 +08:00
Chun Han 001619aef9
feat: supporing load priority for loading (#42413)
related: #40781

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-06-17 15:22:38 +08:00
Bingyi Sun cc5ac1c220
enhance: Support cast function for json index (#41949)
issue: #41948

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-06-05 19:42:32 +08:00
groot 1574673a8c
enhance: Alter collection description (#41558)
issue: https://github.com/milvus-io/milvus/issues/41557

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2025-05-12 14:16:55 +08:00
cai.zhang 640f526301
fix: Update current scalar index version to compatible tantivy different versions (#41141)
issue: #40823

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-04-27 20:44:39 +08:00
SimFG 91d40fa558
fix: Update logging context and upgrade dependencies (#41318)
- issue: #41291

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-23 10:52:38 +08:00
Chun Han 016920b023
fix: solve incompitable problem for none-encoding index(#40838) (#41369)
related: #40838

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-04-20 22:56:44 +08:00
Xianhui Lin 3bc24c264f
enhance: Add json key inverted index in stats for optimization (#38039)
Add json key inverted index in stats for optimization
https://github.com/milvus-io/milvus/issues/36995

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-04-10 15:20:28 +08:00
smellthemoon cb1e86e17c
enhance: support add field (#39800)
after the pr merged, we can support to insert, upsert, build index,
query, search in the added field.
can only do the above operates in added field after add field request
complete, which is a sync operate.

compact will be supported in the next pr.
#39718

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-04-02 14:24:31 +08:00
SimFG 16efcda5c4
feat: support to deny dll according to database property (#40764)
- issue: #40762

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-23 11:18:28 +08:00
congqixia cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Bingyi Sun b59555057d
feat: support json index (#36750)
https://github.com/milvus-io/milvus/issues/35528

This PR adds json index support for json and dynamic fields. Now you can
only do unary query like 'a["b"] > 1' using this index. We will support
more filter type later.

basic usage:
```
collection.create_index("json_field", {"index_type": "INVERTED",
    "params": {"json_cast_type": DataType.STRING, "json_path":
'json_field["a"]["b"]'}})
```

There are some limits to use this index:
1. If a record does not have the json path you specify, it will be
ignored and there will not be an error.
2. If a value of the json path fails to be cast to the type you specify,
it will be ignored and there will not be an error.
3. A specific json path can have only one json index.
4. If you try to create more than one json indexes for one json field,
sdk(pymilvus<=2.4.7) may return immediately because of internal
implementation. This will be fixed in a later version.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-02-15 14:06:15 +08:00
SimFG f070af67f7
fix: deny to set the mmap param for the alter index api when enable auto index (#39518)
- issue: #39517

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-01-23 10:01:04 +08:00
cai.zhang 6d45dd5666
fix: Add scalar index engine version for compatibility (#39204)
issue: #39203

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-15 12:25:00 +08:00
SimFG 2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559)
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
Gao 8977454311
enhance: support recall estimation (#38017)
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 20:40:48 +08:00
Gao 994fc544e7
enhance: support iterative filter execution (#37363)
issue: #37360

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 11:32:44 +08:00
congqixia 24c6a4bb29
enhance: [GoSDK] support Doc-in-doc-out APIs (#37590)
Related to #35853

This PR contains following changes:

- Add function and related proto and helper functions
- Remove the insert column missing check and leave it to server
- Add text as search input data
- Add some unit tests for logic above

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-12 10:58:28 +08:00
smellthemoon 6e880d19a8
enhance: add search params in search request in restful (#36304)
https://github.com/milvus-io/milvus/issues/36321

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-23 16:51:12 +08:00
aoiasd 139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
aoiasd da227ff9a1
feat: Support create collection with functions (#35973)
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
cai.zhang 2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054)
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
jaime b0ac04d104
enhance: set database properties to restrict read access (#35745)
issue: #35744

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
zhagnlu 42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data (#35498)
#35458

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-24 01:40:58 +08:00
SimFG 731d45abbe
enhance: provide more general configuration to control mmap behavior (#35359)
- issue: #35273

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
congqixia 2fbc628994
feat: Support field partial load collection (#35416)
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
wei liu 853d2f33e0
enhance: Trim space during parsing string list from string (#35546)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 20:48:55 +08:00
Gao 6695c6d0a3
enhance: add channel num for queryHook optimization (#35104)
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-31 18:59:49 +08:00
zhagnlu f1b2f7b640
enhance: refactor bitmap index and internal hybrid index (#34450)
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-18 10:39:42 +08:00
SimFG d8e68cbb97
enhance: use the insert rate limit for the upsert request (#34615)
- issue: #34614

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-12 16:17:35 +08:00
Patrick Weizhi Xu 104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
wei liu 9b37d3f517
enhance: Enable setting the replica number and resource group during collection creation (#34403)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 10:20:13 +08:00
Gao ae6d6f91e6
enhance: change autoindex default metric type (#34261)
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 19:52:24 +08:00
jaime 60be454db0
enhance: add disk quota and max collections into db properties (#34368)
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
Buqian Zheng fa8d641ce6
fix: update check for sparse hnsw index (#33713)
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-02 21:56:09 +08:00
Gao a789c60380
enhance: autoindex for multi data type (#33868)
issue: #22837 

contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
cai.zhang 27cc9f2630
enhance: Support analyze data (#33651)
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
zhagnlu 589d4dfd82
enhance: optimize bitmap index (#33358)
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
wei liu b13932bb55
enhance: Enable database level replica num and resource groups for loading collection (#33052)
issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-29 10:59:43 +08:00
SimFG 4031abd2fa
enhance: change default partition num to 16 when using partition key (#32950)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-13 14:19:31 +08:00
SimFG c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
Buqian Zheng 2fdf1a6e76
feat: [Sparse Float Vector] added some integration tests (#31062)
add some integration tests for sparse float vector support

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-10 19:57:18 +08:00