Commit Graph

19352 Commits (9dbd67879f4294be2db6ac319a640ca554f4e289)

Author SHA1 Message Date
Jiquan Long e2f35954d4
enhance: support pattern matching on json field (#30779)
issue: https://github.com/milvus-io/milvus/issues/30714

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
sammy.huang ed1197ea50
enhance: Revert[skip e2e] bump milvus version for scripts/standalone_embed.sh (#30892)
This reverts commit cdb3f61fc8.

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-02-28 16:55:06 +08:00
Jiquan Long 16b785e149
enhance: optimize the memory usage and speed up loading variable length data (#30787)
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
Jiquan Long 4459078e0b
fix: wrong num_entities used when mmap variable length data (#30848)
https://github.com/milvus-io/milvus/issues/30728

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
Guillaume Moutier d15db7596d
fix: fix permissions on /milvus for OpenShift compatibility (#30775)
OpenShift runs Pods with a random uid and gid 0.
As Milvus needs to write into the /milvus directory, this fix modifies
the group permissions to allow the root group (gid 0) to write into it.
This is a simple solution that allows to run Milvus on OpenShift without
adding security constraints, use special ServiceAccounts or whatever.
This should also not impact any other kubernetes platform as it's a
simple group permission change, nothing regarding the userid.

Note: Other actions are necessary to properly deploy the full stack
(Minio, Pulsar,...) on OpenShift. I will document them in the helm chart
project.

Fixes #25565

Signed-off-by: Guillaume Moutier <guillaume.moutier@gmail.com>
2024-02-28 16:29:05 +08:00
SimFG 229fc4f755
enhance: retry to read when the s3 get the unexpect eof error (#30861)
/kind improvement
issue: #30877

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-02-28 16:28:53 +08:00
congqixia a115b731ed
enhance: fix old pr cpp format issue (#30894)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
Buqian Zheng f658dd5faa
enhance: update knowhere version to 60a5c9c (#30788)
/kind improvement

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-02-28 14:18:55 +08:00
zhuwenxing d2c286c536
test: skip building index when field already has index (#30820)
skip building index when field already has index

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-28 14:08:58 +08:00
wei liu 545e8de401
fix: promote leader task failed when segment only exist on current target (#30794)
issue: #30150

`checkLeaderTaskStale` will check segment whether exist on next current
for leaderTask's growing action, which will cause promote leader task
failed when segment only exist on current target

This PR will check segment for both current or next target.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-28 13:14:59 +08:00
wei liu 382b7485a6
enhance: Correct misleading nodeID in GetComponentStates's log (#30731)
This PR corrects the misleading nodeId in GetComponentStates's log

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-28 13:13:12 +08:00
Cai Yudong 8a219e0102
feat: Support knowhere trace using OpenTelemetry (#30750)
Issue: #21508

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
smellthemoon a4f3e01a3a
fix: add the range search params check in proxy (#30423)
if check in Segcore, will not do the it when not insert data.
so, check "radius" and "range_filter" in proxy.
related with #30365

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-28 11:24:58 +08:00
sre-ci-robot 6e9f3ea531
[automated] Update Knowhere Commit (#30744)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-28 10:50:57 +08:00
sammy.huang 74c82b48b4
enhance: [skip e2e] disable macos code checker (#30874)
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-02-28 10:46:57 +08:00
chyezh 77477d6340
fix: wrong context passing into NewClient, error handling lost in session_util (#30817)
issue: #30799

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-28 10:40:09 +08:00
PowderLi 6548182a4a
fix: [restful]vector field cannot be empty while insert (#30871)
issue: #30870 

did a check whether the vector field is empty in advance

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-28 10:36:57 +08:00
congqixia af315539d4
fix: Remove time tick delay metrics when nodes go offline (#30833)
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
yah01 57397b1307
enhance: add new LRU cache impl (#30360)
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
zhuwenxing f87a3a13d0
test: remove concurrent test in ci (#30862)
remove concurrent test in ci

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-27 19:38:56 +08:00
PowderLi 19911a411f
fix: [restful v2] bug list (#30740)
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797

enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-27 16:48:56 +08:00
zhuwenxing 7f78e9d40d
test: use jax.numpy to generate brain float16 datatype (#30825)
JAX is a more lightweight lib that can also generate bfloat16 datatype

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-27 12:48:55 +08:00
sammy.huang 7a49cf2104
enhance: [skip e2e] How to build Milvus v2.3.4 arm image to support ky10 sp3 (#30813)
issue: #30843

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-27 10:52:57 +08:00
Xiaofan 0661d33879
enhance: change frequent log to debug (#30782)
change the "pipeline fetch insert msg" log to debug

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-02-27 10:30:56 +08:00
Jiquan Long 3e82d21ca1
enhance: reduce 1x memory copy when loading json (#30753)
/kind improvement

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-27 10:18:55 +08:00
congqixia 637dcffb6b
fix: Disk resource is not requested for index loaded with disk (#30757)
See also #30756

This PR:
- Request disk resource when index type, version loaded with disk
- Add attribute cache for index utility
- Add `typeutil.Pair`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-27 08:50:56 +08:00
sre-ci-robot bea4c3a418 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-02-26 12:00:49 +00:00
foxspy e1e87d572b
fix: compatibility for diskann cache param (#30119)
patch search cache param from index configs when index meta could not
get the search cache size key
#issue: #30113

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-02-26 16:54:55 +08:00
cqy123456 b31d1a1eb5
enhance: add addition index params for raft index (#30179)
issue: https://github.com/milvus-io/milvus/issues/29230

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-02-26 14:50:54 +08:00
zhagnlu a0531b72aa
fix: reorder hybridsearch result for L2 metric (#30739)
#30694

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-26 14:18:55 +08:00
Bingyi Sun ece9d273a7
enhance: some patches for #30636 (#30664)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-26 11:42:55 +08:00
smellthemoon 9815cf50c9
fix: close rocksmq and natsmq when shutdown (#30706)
related: #30703 #30355

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-26 11:38:55 +08:00
wei liu befe0e21fd
fix: Set indexInfo when try to set segment to leader view (#30758)
issue: #30150
see also: #30258

cause `SyncDataDistribution` will try to load delta for segment. if miss
indexInfo in request, sync action will failed due to lack of index info.

This PR set indexinfo when try to set segment to leader view

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-26 11:02:55 +08:00
Xiaofan d970a9fc6c
enhance: refine error log of json parse error (#30671)
add more error info when json format does not match.
relate to https://github.com/milvus-io/milvus/issues/29837

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-02-24 12:32:14 +08:00
sre-ci-robot 58260864b5 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-02-23 12:00:50 +00:00
zhuwenxing f0bff1e1a8
test: add hybrid search in checker for test (#30341)
add hybrid search in checker for test

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-23 10:46:52 +08:00
wei liu 6dd7297178
fix: Skip generate balance task when target not ready (#30724)
issue: #30723

This PR skip generate balance task when collection's target isn't ready.
also refine the check stale logic in query coord's scheduler, if channel
exist in current or next target, task won't be canceled.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-23 10:32:53 +08:00
chyezh 950624d8d3
fix: nil ptr is used as nil interface in grpc client (#30754)
issue: #30715

- Bug: Set nil struct pointer to describe nil interface.
Panic with segment violation when calling method on this nil struct
pointer.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-23 10:26:53 +08:00
congqixia 536e082f5a
enhance: Use defer to clean legacy growing segment (#30770)
See also #30735 #30735

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-23 10:02:55 +08:00
congqixia 90e802e441
enhance: Add unit test for level-zero segment deltalog request resource (#30718)
See also #30670 #30704

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-23 10:00:54 +08:00
zhagnlu c5363c70db
fix: fix upsert using wrong field to compute partition key (#30772)
#30607

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-23 09:54:53 +08:00
cai.zhang 16b4c9a79e
fix: Skip filling segmentID in indexBuildCh to prevent flush blocked (#30747)
issue: #30580

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-22 20:40:53 +08:00
Jiquan Long e2330f02f8
fix: pattern match use incorrect raw data (#30764)
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
Jiquan Long 375190e76e
fix: cpp format check not work (#30767)
fix: #30765

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:40:53 +08:00
aoiasd e5353ad049
enhance: access log should get get client info by get method (#30502)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-22 14:58:52 +08:00
congqixia f3b7a8892f
fix: Release loaded growing if WatchDmlChannel fail (#30735)
See also #30734

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-21 22:12:53 +08:00
Ted Xu 12acaf3e4f
enhance: Adding a generic stream payload reader (#30682)
See: #30404

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-02-21 17:10:52 +08:00
MrPresent-Han 77eb6defb1
feat: support groupby on growing and non-indexed sealed egment(#30307) (#30644)
related: #30308

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-21 14:02:53 +08:00
zhagnlu 18aac076de
fix: move test from NEON to X86 (#30324)
#26137

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 11:58:53 +08:00
yiwangdr c6665c2a4c
test: support multiple data/querynodes in integration test (#30618)
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00