Commit Graph

1468 Commits (23e2a0e78b0c65635e8f17b71e39efb97ba85f10)

Author SHA1 Message Date
Buqian Zheng fa8d641ce6
fix: update check for sparse hnsw index (#33713)
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-02 21:56:09 +08:00
congqixia b32dae1883
enhance: Refine max length exceeded error message (#34300)
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 21:16:07 +08:00
wei liu b49862d4f3
enhance: Optimize grow slice cost during query (#34253)
issue: #32252 

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
aoiasd 186757e622
enhance: support mark error as user error (#33498)
relate: https://github.com/milvus-io/milvus/issues/33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
Chun Han 96dcee5dff
fix:load major compaction partial result(#34051) (#34052)
related: #34051

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
jaime 9c2eeff427
enhance: refine error message for invalid limit parameter (#34178)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 22:14:04 +08:00
cqy123456 51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call (#34102)
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
jaime 9630974fbb
enhance: move rocksmq from internal to pkg module (#33881)
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
wayblink f9a0f7bb25
Add an option to enable/disable vector field clustering key (#34097)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
congqixia fd922d921a
enhance: Add nilness linter and fix some small issues (#34049)
Add `nilness` for govet linter and fixed some detected issues

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 14:52:03 +08:00
jaime d08cb885ca
enhance: enable flush rate limiter of collection level (#33837)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
Patrick Weizhi Xu b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161)
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
cqy123456 298e50b834
enhance: check index with data type (#33880)
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
Gao 0d20303e54
fix: fix binary vector data size (#33750)
issue: https://github.com/milvus-io/milvus/issues/22837

- fix byte size wrong for binary vectors
- fix the expect/actual error msg

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-18 21:39:59 +08:00
Gao a789c60380
enhance: autoindex for multi data type (#33868)
issue: #22837 

contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
wei liu 188ee433b9
enhance: Refine name rule check error msg (#33815)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 10:37:58 +08:00
wayblink a1232fafda
feat: Major compaction (#33620)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
yihao.dai 3540eee977
enhance: Support L0 import (#33514)
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
wei liu 0a3d456688
enhance: add restful api to trigger component stop (#32076)
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-07 10:35:54 +08:00
aoiasd b9bafc76b4
enhance: access log support print with consistency level (#33503)
relate: https://github.com/milvus-io/milvus/issues/33563

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 17:13:50 +08:00
congqixia 25080bb455
enhance: Print `UseDefaultConsistency` param in read requests (#33617)
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 10:27:50 +08:00
aoiasd 0cf225fa75
enhance: access log support restful api (#33155)
relate: https://github.com/milvus-io/milvus/issues/31823

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-31 11:13:44 +08:00
Chun Han 416a2cf507
fix: query iterator lack results(#33137) (#33422)
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
zhagnlu 589d4dfd82
enhance: optimize bitmap index (#33358)
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
wei liu b13932bb55
enhance: Enable database level replica num and resource groups for loading collection (#33052)
issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-29 10:59:43 +08:00
ArenaSu a59ed65c48
doc: [skip-e2e] add comments for task queue (#33388)
Add comments for task queue in internal/proxy/task_scheduler.go.

Signed-off-by: ArenaSu <704427617@qq.com>
2024-05-27 20:51:42 +08:00
yihao.dai af71116499
fix: Fix frequent 'failed to get request info' log (#33334)
issue: https://github.com/milvus-io/milvus/issues/33333

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-27 14:27:41 +08:00
SimFG cb99e3db34
enhance: add the includeCurrentMsg param for the Seek method (#33326)
/kind improvement
- issue: #33325

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 10:31:41 +08:00
SimFG 2964f60edc
enhance: the proxy metric in the query request (#33307)
/kind improvement
issue: #33306

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-24 14:19:41 +08:00
yihao.dai 592d701617
fix: Fix global rate limit is not working (#33335)
If the request is limited by rate limiter, limiter should not "Cancel".
This is because, if limited, tokens are not deducted; instead, "Cancel"
operation would increase the token count.

issue: https://github.com/milvus-io/milvus/issues/31705

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:05:47 +08:00
aoiasd 1b4e28b97f
enhance: Check by proxy rate limiter when delete get data by query. (#30891)
relate: https://github.com/milvus-io/milvus/issues/30927

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-23 20:03:40 +08:00
yihao.dai 155cb40fc9
fix: Fix printing type of request (#33289)
issue: https://github.com/milvus-io/milvus/issues/31705

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 14:19:40 +08:00
yihao.dai 9ff023ee35
fix: Fix filtering by partition key fails for importing data (#33274)
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 11:13:40 +08:00
SimFG dd0c6d6980
fix: the panic when db isn't existed in the rate limit interceptor (#33244)
issue: #33243

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-22 15:57:39 +08:00
aoiasd 13fdaea9f0
fix: accesslog writer cache close cause deadlock (#33261)
relate: https://github.com/milvus-io/milvus/issues/33260

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-22 14:55:39 +08:00
congqixia f336b2d672
fix: Check schema without vector field in proxy (#33211)
Related to #33199

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 14:33:39 +08:00
Jiquan Long 9f81290c63
fix: try best to get enough query results (#33178)
issue: https://github.com/milvus-io/milvus/issues/33137

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-21 11:57:51 +08:00
SimFG ec98de3ad4
fix: reset the quota value when init the limiter (#33111)
issue: #33107
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-20 10:35:38 +08:00
Bingyi Sun 5c6de473bb
fix: check array field data is nil before getting the type (#33114)
issue: https://github.com/milvus-io/milvus/issues/33074

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-17 17:29:37 +08:00
smellthemoon 1671c7898a
enhance: sync milvus proto version (#33094)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-16 18:51:34 +08:00
smellthemoon b45798107a
enhance: add nullable in Field, check valid_data and fill data (#32086)
1. add nullable in model.Field
   help to read nullable accurately.
2. check valid_data
a. if user pass default_value or the field is nullable, the length of
valid_data must be num_rows.
b. if passed valid_data, the length of passed field data must equal to
the number of 'true' in valid_data.
c. after fill default_value, only nullable field will still has
valid_data.
3. fill data in two situation
    a. has no default_value, if nullable,
will fill nullValue when passed num_rows not equal to expected num_rows.
    b. has default_value,
will fill default_value when passed num_rows not equal to expected
num_rows.
c. after fill data, the length of all field will equal to passed
num_rows.
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-16 11:57:35 +08:00
aoiasd 875ad88d84
enhance: Support access log use stdout with write cache and auto flush as time (#32213)
relate: https://github.com/milvus-io/milvus/issues/28948

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-16 10:05:34 +08:00
yihao.dai 6fc00e42d7
fix: Fix binlog import (#33027)
binlog import should only import into one partition.

issue: https://github.com/milvus-io/milvus/issues/32807

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-14 21:45:32 +08:00
Cai Yudong 4fc7915c70
enhance: unify data generation test APIs (#32955)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-14 14:33:33 +08:00
aoiasd dc058eaf61
fix: should init access logger before init server (#32976)
relate: https://github.com/milvus-io/milvus/issues/32968

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-14 10:47:32 +08:00
Xiaofan 230c88b1de
enhance: improve reduce performance on very large topk (#32871)
fix #32870
reduce cpu usage for reduce on large k

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-13 11:55:31 +08:00
wei liu 8a9a42198d
fix: Proxy crash due to shard leader cache data race (#32971)
issue: #32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash

This PR fixed that use leaderMut when try to access shard leader cache.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-11 14:32:12 +08:00
wei liu c35797c399
enhance: expose DescribeDatabase api in proxy (#32732)
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
SimFG 8963615b33
enhance: add the cost response for the rest api (#32620)
issue: https://github.com/milvus-io/milvus/issues/30436

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-08 11:53:29 +08:00
wei liu ba02d54a30
enhance: update shard leader cache when leader location changed (#32470)
issue: #32466

this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00