868723340c
enhance: [2.4] the list collection and database api ( #32602 )
...
issue: https://github.com/milvus-io/milvus/issues/32550
pr: #32576
/kind improvement
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-29 14:21:26 +08:00
5da6db9252
enhance: [2.4] add the skip auto id and partition key check config ( #32597 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/32591
pr: #32592
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-29 10:19:31 +08:00
6178355e2f
fix: Remove watching config event in rate limiter ( #32313 ) ( #32314 )
...
Remove watching config event in rate limiter to prevent object leaks.
Instead, reset limiter rates periodically.
issue: https://github.com/milvus-io/milvus/issues/32312
pr: https://github.com/milvus-io/milvus/pull/32313
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-28 11:35:28 +08:00
52cc1ed9d8
enhance: [2.4] make the auth error message more suitable ( #32298 )
...
issue: #32252
pr: #32253
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-16 17:13:19 +08:00
d565d64941
enhance: [2.4] make datanode exit and case `TestProxy` faster ( #32293 )
...
/kind improvement
issue: #32219
pr: #32218
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-16 16:37:20 +08:00
cbac23a8ff
fix: failed to raise metric_type not match error ( #32203 )
...
issue:https://github.com/milvus-io/milvus/issues/32176
pr:https://github.com/milvus-io/milvus/pull/32202
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-12 15:57:19 +08:00
6d7c6db49c
fix: fix incomplete hybrid search result when nq > 1 ( #32178 )
...
pr: https://github.com/milvus-io/milvus/pull/32177
issue: https://github.com/milvus-io/milvus/issues/25639
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-12 10:31:19 +08:00
329e73b58b
fix: partitionIDs should be union of partitionIDs of each subquery ( #32147 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-11 15:41:19 +08:00
4ea2a98fa3
enhance: [2.4] support to invalid the database meta cache ( #32092 )
...
issue: #32077
pr: #32078
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-11 15:01:20 +08:00
ada0e8c931
fix: reduce didn't handle offset without limit and reduceStopForBest … ( #32088 )
...
fix https://github.com/milvus-io/milvus/issues/32059
pr: https://github.com/milvus-io/milvus/pull/32089
this pr fix two issues:
offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 21:21:07 -07:00
c8aacc6d59
feat: [2.4] [Sparse Float Vector] added some integration tests ( #32093 )
...
add some integration tests for sparse float vector support
#29419
pr: https://github.com/milvus-io/milvus/pull/31062
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-04-10 21:47:29 +08:00
bb2ffd47f6
enhance: Add db label for some usual metrics ( #32024 )
...
issue: https://github.com/milvus-io/milvus/issues/31782
pr: #30956
cherry pick: #30956 #32003 #31715
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2024-04-10 20:59:20 +08:00
e3821f5a48
enhance: [Cherry-pick] Remove duplicated err check code ( #32001 ) ( #32079 )
...
Cherry-pick from master
pr: #32001
The `err != nil` check is duplicated here and shall be removed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-10 15:43:18 +08:00
8f99144565
enhance: [cherry-pick]check fp16/bf16 nan or inf value ( #31842 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/31840
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-04-10 10:43:27 +08:00
4a4935f7b2
enhance: [2.4] use the rate log in the privilege interceptor ( #32038 )
...
/kind improvement
pr: #32037
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-09 16:37:17 +08:00
4c07304790
enhance: Refactor hybrid search ( #31742 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368
pr :https://github.com/milvus-io/milvus/pull/32020
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 10:15:18 +08:00
4b430097dd
enhance: Unify data type check APIs for go ( #31945 )
...
cherry-pick from master
pr: #31887
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-08 14:11:17 +08:00
b2a79a0570
enhance: add more metrics ( #31271 ) ( #31861 )
...
/kind improvement
fix : #31272
pr: #31271
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-05 10:09:22 +08:00
b83e48ed11
fix: [Cherry-pick] Check parse QueryInfo error before using it ( #31836 ) ( #31864 )
...
Cherry-pick from master
pr: #31836
See also milvus-io/milvus-sdk-go#693 #31735
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-03 16:07:15 +08:00
8b8a5b4ac4
enhance: ban groupby on hybrid-search( #29968 ) ( #31812 )
...
related: #29968
pr: https://github.com/milvus-io/milvus/pull/31810
Ban groupby on hybrid search for the time being because the semantic
meaning is unclear
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-02 15:47:18 +08:00
4f3bfe24a6
fix: proxy ctx cancel before revoking session ( #31611 )
...
issue: https://github.com/milvus-io/milvus/issues/31219
pr: https://github.com/milvus-io/milvus/pull/31596
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-01 15:37:13 +08:00
f3216bfe18
enhance: ban groupby on binary vector( #31134 ) ( #31735 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/31659
See also: https://github.com/milvus-io/milvus/issues/31134
Currently, don't support brute force search iterator for binary_vector
so group_by in such cases will fail and to avoid inconsistent behavior,
we ban groupby on binary vector for the time being.
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-01 14:13:12 +08:00
bc4a9a1abb
enhance: Retry on inconsistent requery result ( #31713 ) ( #31734 )
...
Make retry on ErrInConsistentRequery in proxy rather than in every SDK.
issue: https://github.com/milvus-io/milvus/issues/31642
pr: https://github.com/milvus-io/milvus/pull/31713
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-30 20:53:19 +08:00
3e3a92fc89
enhance: Add restful api for devops to execute rolling upgrade ( #29998 ) ( #31645 )
...
issue: #29261
pr: #29998
This PR Add restful api for devops to execute rolling upgrade, including
suspend/resume balance and manual transfer segments/channels.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-29 16:15:11 +08:00
d37e1fdd9f
fix: predict inverted index resource usage more reasonably ( #31615 ) ( #31641 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/31617
pr: #31615
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-27 21:07:11 +08:00
b8f849e98e
enhance: Support auto index for scalar index ( #31593 )
...
issue: #29309
master pr: #31255
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-27 14:57:10 +08:00
b0cbddae8d
fix: minio ssl compatible issue ( #31618 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: https://github.com/milvus-io/milvus/pull/31607
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-03-27 14:41:20 +08:00
17eb0c276c
fix: add some check fieldata dim to avoid error data input ( #31478 )
...
issue:https://github.com/milvus-io/milvus/issues/30138
pr: https://github.com/milvus-io/milvus/pull/31564
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-03-25 19:29:08 +08:00
f1a108c97b
enhance: Add max file num limit and max file size limit for import ( #31497 ) ( #31542 )
...
The max number of import files per request should not exceed 1024 by
default (configurable).
The import file size allowed for importing should not exceed 16GB by
default (configurable).
issue: https://github.com/milvus-io/milvus/issues/28521
pr: https://github.com/milvus-io/milvus/pull/31497
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 14:33:07 +08:00
8858f8d612
enhance: Return more fields in import progress response ( #31539 ) ( #31561 )
...
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.
issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521
pr: https://github.com/milvus-io/milvus/pull/31539
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 14:27:07 +08:00
1e0bf5acd2
enhance: Remove import v1 ( #31403 ) ( #31535 )
...
Remove all code and logic related to import v1.
issue: https://github.com/milvus-io/milvus/issues/28521
pr: https://github.com/milvus-io/milvus/pull/31403
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-24 21:51:07 +08:00
99774548f2
enhance: [Cherry-pick] Add AllPartitionsID const to replace InvalidPartitionID ( #31438 ) ( #31515 )
...
Cherry-pick from master
pr: #31438
"-1" as `InvalidPartitionID` previously used as All partition place
holder in delete cases. It's confusing and hard to maintain when a const
var has more than one meaning.
This PR add `AllPartitionsID` to replace these usages in delete
scenarios.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-22 16:37:08 +08:00
a0535edb67
enhance: Support MinIO TLS connection ( #31396 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: https://github.com/milvus-io/milvus/pull/31292
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
0dbc0f2c11
fix: Hybrid search doesn't expire shard leader cache ( #31381 )
...
issue: #31351
pr: #31380
This PR fixed that search/hybrid_search doesn't expire shard leader
cache when send request to query node failed, which make every request
keep trying to connect a offline query node
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-19 10:05:05 +08:00
1885d176cf
fix: Return specific error code when encountering incomplete requery results ( #31343 ) ( #31359 )
...
During requery, segments may change (e.g., due to compaction), so we
need to return specific error codes when encountering incomplete requery
results. Clients can then retry to avoid this issue.
issue: https://github.com/milvus-io/milvus/issues/29656
pr: https://github.com/milvus-io/milvus/pull/31343
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-18 19:43:06 +08:00
555d96ff05
fix: Unstable ut TestGetClientFailed ( #31296 ) ( #31335 )
...
issue: #31295
pr: #31335
This PR fix unstable ut TestGetClientFailed
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-18 13:51:04 +08:00
7e3ac66617
fix: fix wrong guaranteeTs for hybrid search ( #31329 )
...
pr: #31302
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-18 12:29:04 +08:00
ca8eee2c47
fix: Set node unreachable when get shard client failed ( #31277 )
...
issue: #30531
cause get client from `shardClientMgr`, doesn't means query node is
unavailable. because of the ref counter policy in `shardClientMgr`,
which will clean the client, if no collection use qn as shard leader.
This PR fix that set node unreachable when get shard client failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 10:23:03 +08:00
056e9f0cf5
fix: unmarshal nil when check search params ( #31139 )
...
#31020
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-03-15 09:15:04 +08:00
3c80083f51
feat: [Sparse Float Vector] add sparse vector support to milvus components ( #30630 )
...
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.
https://github.com/milvus-io/milvus/issues/29419
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 14:32:54 -07:00
b5c67948b7
enhance: Enhance and modify the return content of ImportV2 ( #31192 )
...
1. The Import APIs now provide detailed progress information for each
imported file, including details such as file name, file size, progress,
and more.
2. The APIs now return the collection name and the completion time.
3. Other modifications include changing jobID to jobId and other similar
adjustments.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-13 19:51:03 +08:00
87b3c25b15
fix: Fix binlog import ( #31205 )
...
1. File type validation is omitted during binlog import.
2. System fields are appended to the schema during binlog import.
issue: https://github.com/milvus-io/milvus/issues/28521
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-13 10:35:04 +08:00
de2c95d00c
enhance: Constraint dynamic field as key-value format ( #31183 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-12 12:45:03 +08:00
6a83f16871
feat: Support for multiple forms of JSON ( #31052 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-11 19:55:02 +08:00
8cb06acfed
feat: Replacing the current import API with the v2 implementation ( #31046 )
...
Replacing the current import API v1 implementation with the v2
implementation.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-10 12:23:02 +08:00
3574bdf858
enhance: ban range-search iteration for search-group-by ( #30824 )
...
related: #30033
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-08 14:17:00 +08:00
eb0cfe3e84
fix: set CollectionID field for AlterCollection request ( #30814 )
...
issue: https://github.com/milvus-io/milvus/issues/30800
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-07 14:49:02 +08:00
007fab183c
enhance: Skip partition key name check if feature not enabled ( #31067 )
...
Skip partition key naming & hash value pre process if collection schema
does not have partition key
The PR removes mislead warning when collection has no partition key
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-06 19:33:00 +08:00
d0eeea4b44
fix: reduce incorrectly for group-by with offset( #30828 ) ( #30882 )
...
related: #30828
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-06 16:47:00 +08:00
a88c896733
enhance: purge client infos periodically ( #31037 )
...
https://github.com/milvus-io/milvus/issues/31007
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-06 12:50:59 +08:00
SimFG
yihao.dai
zhenshan.cao
Xiaofan
Buqian Zheng
jaime
congqixia
cqy123456
Cai Yudong
Jiquan Long
Chun Han
wei liu
cai.zhang
groot
zhagnlu
smellthemoon
Bingyi Sun