69a82acc46
enhance: [Cherry-pick] Set delete scope for LoadSegment streaming data ( #30245 ) ( #30367 )
...
Cherry pick from master
pr: #30245
See also #29474
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-02 16:05:06 +08:00
73df0b872e
fix: [2.3] add more requests to the database interceptor ( #30453 )
...
issue: https://github.com/milvus-io/milvus/issues/30368
pr: #30452
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-02-02 16:03:06 +08:00
fba28f0e74
enhance:[skip e2e] clean up ( #30449 )
...
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-02 09:09:06 +08:00
910e68f2d3
enhance: [skip e2e]allow github actions runner run on ubuntu-latest os ( #30435 )
...
pr: #29525
---------
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 19:45:09 +08:00
462608fe04
enhance: [skip e2e]correct a shell syntax error ( #30247 )
...
pr: #30248
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 14:31:04 +08:00
2a415200db
enhance: [skip e2e]extract cache stuff into a independent composite action ( #30426 )
...
pr: #29499
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 14:27:05 +08:00
d33a37c1dd
enhance: move network mode of container into builder.sh file ( #30420 )
...
pr:#30419
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 10:31:04 +08:00
3036c19867
fix: can't not get search_cache_budget_gb in create index ( #30353 )
...
issue:https://github.com/milvus-io/milvus/issues/30375
pr: https://github.com/milvus-io/milvus/pull/30119
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-31 15:49:03 +08:00
028721db25
enhance: optimize the loading strategy ( #29910 ) ( #30348 )
...
as we have the pool size limit so we don't need to limit the concurrency
manually
pr: #29910
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-31 15:25:04 +08:00
df2dc7617c
enhance: hotfix container network ( #30390 )
...
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-31 09:35:04 +08:00
eddcc362cf
enhance: hotfix docker-compose.yml ( #30382 )
...
PR: #30378
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-30 20:45:03 +08:00
3e994242d6
fix: panic with datanode negetive wait group counter ( #30136 )
...
issue: #29170
pr: #30135
Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-30 18:07:03 +08:00
21c944beaa
enhance: add basic information of milvus into metrics ( #29666 )
...
add basic build information and runtime component dependency into
metrics.
issue: #29664
pr: #29665
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:49:04 +08:00
9ab2ce0767
enhance: [Cherry-pick] Opt vector dimension mismatch error message ( #30316 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/29928
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-29 14:47:03 +08:00
5ec13ecc76
[automated] Bump milvus version to v2.3.7 ( #30328 )
...
Bump milvus version to v2.3.7
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-29 09:47:03 +08:00
06e06aae05
enhance: [skip e2e]change to a better name ( #30330 )
...
pr: #30329
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-29 09:45:08 +08:00
77e123762f
enhance: add graceful stop timeout to avoid node stop hang under extreme cases ( #30320 )
...
1. add coordinator and proxy graceful stop timeout to 5s.
3. add other work node graceful stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
4. change the order of datacoord component while stop.
5. `LivenessCheck` do not perform graceful shutdown now.
issue: https://github.com/milvus-io/milvus/issues/30310
pr: #30317
also see: https://github.com/milvus-io/milvus/pull/30306
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-27 08:45:02 +08:00
650dcc512e
fix: dead lock while getting configs ( #30319 )
...
issue: https://github.com/milvus-io/milvus/issues/30295
pr: #30318
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-26 20:15:01 +08:00
26df754514
enhance: Bump milvus version to 2.3.7 ( #30297 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-26 11:07:02 +08:00
e0f987ee9b
enhance: Allows proactive warming up of chunk cache ( #30182 ) ( #30289 )
...
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
pr: https://github.com/milvus-io/milvus/pull/30182
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-26 09:57:01 +08:00
2c4d0605ef
enhance: add a weight for growing row count when balancing segments ( #30293 )
...
Cherry-pick from master
pr: #30271
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-26 09:17:03 +08:00
d182a51653
fix: [Cherry-pick] Use correct pools for all CGO methods in segments pkg ( #30275 )
...
Cherry-pick from master
pr: #30274
See also #30273
This PR:
- Rename confusing `LoadIndexInfo` to `UpdateIndexInfo` for LocalSegment
- Use `DynamicPool` instead of `LoadPool` for `UpdateSealedSegmentIndex`
- Fix cgo call missing pool control
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 19:49:01 +08:00
1a54571c10
enhance: [Cherry-pick] Add trace span for scheduling read tasks in QueryNode ( #30266 )
...
Cherry-pick from master
pr: #30265
This PR adds a trace span for search/query task scheduling duration
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 15:39:01 +08:00
9e8eb2aa51
fix: Revert leader checker related check ( #30262 )
...
See also #30150
PR reverted: #29984 #30152
Currently this scenario could not be covered by ut/it/e2e test cases
Revert it for now
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 12:39:02 +08:00
e3114b6a4d
enhance: [2.3] Utilize partition key optimization in reQuery ( #30255 )
...
Partial cherry-pick from master due to code branching
pr: #30253
See also #30250
This PR add requery flag in query task. When reQuery flag is true, query
task shall skip partition name conversion and use pre-calculated
partitionIDs passed from search task.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 11:05:07 +08:00
44c2ae130b
enhance: [skip e2e]increase timeout for image build ( #29083 ) ( #30239 )
...
pr: #29083
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-24 15:37:02 +08:00
285eade31c
enhance: [skip e2e]increase timeout for building gpu image ( #30237 )
...
pr: #30238
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-24 15:35:13 +08:00
95cd6f20d0
fix: [2.3] wrong format expr for the delete rest api ( #30218 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/30092
pr: #30217
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:05 +08:00
44545e2fc4
[automated] Bump milvus version to v2.3.6 ( #30234 )
...
Bump milvus version to v2.3.6
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-24 09:49:00 +08:00
efea282111
feat: [Pick] Support tencent cloud object storage for milvus ( #30210 )
...
issue: #30162
master pr: #30163
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 16:07:01 +08:00
35e4165722
enhance: [2.3] make Load process traceable in querynode & segcore ( #30187 )
...
Cherry-pick from master, modified some files since branching
pr: #29858
See also #29803
This PR:
- Add trace span for LoadIndex & LoadFieldData in segment loader
- Add TraceCtx parameter for Index.Load in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-23 15:58:57 +08:00
e0beea7fb6
test: update pymilvus version and test cases ( #30209 )
...
issue: #17614
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-01-23 14:40:55 +08:00
4d0a6dbc25
fix: written file size is over the int32 range and raises error ( #30057 ) ( #30207 )
...
we sum the total data size in int32, which could lead to an overflow
error
related #30056
pr: #30057
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-23 13:50:56 +08:00
9bd94c4fab
fix: the system rejects all queries and never recovers if enabled read rate limit ( #30061 ) ( #30196 )
...
fix #30060
pr: #30061
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-23 10:37:00 +08:00
565d37ced8
enhance: Bump version to 2.3.6 ( #30184 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-22 19:04:55 +08:00
0e71923408
enhance: enable converting segcore error to merr ( #29914 ) ( #30178 )
...
this converts the segcore error to merr if possible
pr: #29914
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:56:55 +08:00
1cc5a613d5
enhance: adjust the GPU pool size ( #29937 ) ( #30177 )
...
according to benchmark, the GPU pool size with 6 performs best
pr: #29937
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:55:04 +08:00
c8a129756f
enhance: filter out the not needed collections while listing ( #29690 ) ( #30180 )
...
this improves performance while many collections exist resolve #29631
pr: #29690
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:52:55 +08:00
6aaccdd5f4
feat: support general capacity restrict for cloud-side resoure contro… ( #30017 )
...
related: #29844
pr: #https://github.com/milvus-io/milvus/pull/29845
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 16:18:56 +08:00
2465d86138
enhance: [2.3] support related privilege for grant api ( #30154 )
...
/kind improvement
pr: #30153
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-22 14:42:55 +08:00
ce318f3286
enhance: make the error of parsing expression to `ParameterInvalid` ( #29681 ) ( #29795 )
...
before this, the error is unexpected error
pr: #29681
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 13:36:55 +08:00
917a4d74f3
fix: Use channel cp as the dml&start position for import segments ( #30107 ) ( #30133 )
...
This PR discontinuing the subscription to the mq and, instead, employing
the channel checkpoint as the DML and starting position for the import
segments.
issue: https://github.com/milvus-io/milvus/issues/30106
pr: https://github.com/milvus-io/milvus/pull/30107
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-22 13:32:55 +08:00
a8d9b0ccba
enhance: optimize the loading index performance ( #29894 ) ( #30018 )
...
this utilizes concurrent loading
pr: #29894
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 13:12:56 +08:00
bac1a1355b
fix: [Cherry-pick] collection properties not saved for alter collection ( #30145 ) ( #30156 )
...
Cherry-pick from master
pr: #30145
Resolves : #30144
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-22 10:08:55 +08:00
b95f0cc0a1
enhance: Add a counter monitoring for the rate-limit requests ( #30109 ) ( #30132 )
...
Add a counter monitoring metric for the ratelimited rpc requests with
labels: proxy nodeID, rpc request type, and state.
issue: https://github.com/milvus-io/milvus/issues/30052
pr: https://github.com/milvus-io/milvus/pull/30109
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-21 14:44:59 +08:00
3dc2585d9b
enhance: support dataType: array & json ( #30077 )
...
issue: #30075
master pr: #30076
deal with the array<?> field data correctly
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-21 14:00:56 +08:00
b2997eb881
fix: Leader checker can't remove segment from leader view ( #30152 )
...
issue: #30150
pr: #30151
This PR fix three problems:
1. the load request generated by leader checker doesn't set load scope
2. leader checker use wrong node id when generate release task, which
cause the release task finished immediately
3. the release request generated by leader_checker doesn't set the force
flag, the operation to clean leader view on delegator will fail.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-20 18:58:58 +08:00
079ddbfc01
enhance: [Cherry-pick] Shuffle candidates before channel assignment ( #30066 ) ( #30089 )
...
Cherry-pick from master
pr: #30066
Shuffle candidates to reduce scenario that some channel allocated into
same node
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-19 12:06:54 +08:00
0700434c58
fix: patching search cache param when index meta does not hold one ( #30116 )
...
patch search cache param from index configs when index meta could not
get the search cache size key
issue: #30113
pr: #30119
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-01-19 11:50:56 +08:00
be1470a654
enhance: [2.3] Add load/release partitions to replicate msg stream ( #30001 )
...
/kind improvement
pr: #28399
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-18 22:50:55 +08:00
congqixia
SimFG
sammy.huang
cqy123456
yah01
chyezh
xige-16
sre-ci-robot
jaime
yihao.dai
Bingyi Sun
cai.zhang
nico
MrPresent-Han
PowderLi
wei liu
foxspy