cc2bc3f8f2
enhance: [Cherry-Pick] access log should get get client info by get method ( #30503 )
...
https://github.com/milvus-io/milvus/pull/30502
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-04 18:57:07 +08:00
7e7722ed43
enhance: [skip e2e] set logrus log level to reduce output error logs ( #30478 )
...
issue: #30295
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-02-04 17:23:06 +08:00
f2310ab4ce
enhance: [Cherry-pick] Use dynamic pool for `NewLoadIndexInfo` ( #30489 ) ( #30497 )
...
Cherry-pick from master
pr: #30489
See also #30445
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-04 16:39:06 +08:00
ad4a53d225
enhance: [Cherry-Pick] Fix some access log bugs ( #30496 )
...
pr: https://github.com/milvus-io/milvus/pull/30409
https://github.com/milvus-io/milvus/pull/29680
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-04 16:37:07 +08:00
cea5396c33
enhance: Bump milvus & milvus-proto version to v2.3.8 ( #30492 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-04 12:57:05 +08:00
3c5ff624f8
fix: [pick]Only use bound indexnodes in bound mode ( #30462 )
...
master pr: #30461
issue: #30463
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-03 21:59:05 +08:00
655e235230
enhance: calculate the accuracy memory usage while loading segment ( #30473 ) ( #30475 )
...
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.
Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data
pr: #30473
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-03 13:01:12 +08:00
20608287b9
fix: Decoupling importing segment from flush process ( #30402 ) ( #30439 )
...
This pr decoups importing segment from flush process by:
1. Exclude the importing segment from the flush policy, this approch
avoids notifying the datanode to flush the importing segment, which may
not exist.
2. When RootCoord call Flush, DataCoord directly set the importing
segment state to `Flushed`.
issue: https://github.com/milvus-io/milvus/issues/30359
pr: https://github.com/milvus-io/milvus/pull/30402
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-02-03 12:59:14 +08:00
f50799b7fd
fix: proxy may never setup if the port binded ( #30035 ) ( #30416 )
...
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service resolve #30034
pr: #30035
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-02 16:21:06 +08:00
692dcebac6
enhance: support varchar autoid when bulkinsert（#30377） ( #30448 )
...
related pr: #30377
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-02 16:11:08 +08:00
69a82acc46
enhance: [Cherry-pick] Set delete scope for LoadSegment streaming data ( #30245 ) ( #30367 )
...
Cherry pick from master
pr: #30245
See also #29474
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-02 16:05:06 +08:00
73df0b872e
fix: [2.3] add more requests to the database interceptor ( #30453 )
...
issue: https://github.com/milvus-io/milvus/issues/30368
pr: #30452
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-02-02 16:03:06 +08:00
fba28f0e74
enhance:[skip e2e] clean up ( #30449 )
...
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-02 09:09:06 +08:00
910e68f2d3
enhance: [skip e2e]allow github actions runner run on ubuntu-latest os ( #30435 )
...
pr: #29525
---------
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 19:45:09 +08:00
462608fe04
enhance: [skip e2e]correct a shell syntax error ( #30247 )
...
pr: #30248
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 14:31:04 +08:00
2a415200db
enhance: [skip e2e]extract cache stuff into a independent composite action ( #30426 )
...
pr: #29499
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 14:27:05 +08:00
d33a37c1dd
enhance: move network mode of container into builder.sh file ( #30420 )
...
pr:#30419
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-01 10:31:04 +08:00
3036c19867
fix: can't not get search_cache_budget_gb in create index ( #30353 )
...
issue:https://github.com/milvus-io/milvus/issues/30375
pr: https://github.com/milvus-io/milvus/pull/30119
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-31 15:49:03 +08:00
028721db25
enhance: optimize the loading strategy ( #29910 ) ( #30348 )
...
as we have the pool size limit so we don't need to limit the concurrency
manually
pr: #29910
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-31 15:25:04 +08:00
df2dc7617c
enhance: hotfix container network ( #30390 )
...
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-31 09:35:04 +08:00
eddcc362cf
enhance: hotfix docker-compose.yml ( #30382 )
...
PR: #30378
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-30 20:45:03 +08:00
3e994242d6
fix: panic with datanode negetive wait group counter ( #30136 )
...
issue: #29170
pr: #30135
Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-30 18:07:03 +08:00
21c944beaa
enhance: add basic information of milvus into metrics ( #29666 )
...
add basic build information and runtime component dependency into
metrics.
issue: #29664
pr: #29665
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:49:04 +08:00
9ab2ce0767
enhance: [Cherry-pick] Opt vector dimension mismatch error message ( #30316 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/29928
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-29 14:47:03 +08:00
5ec13ecc76
[automated] Bump milvus version to v2.3.7 ( #30328 )
...
Bump milvus version to v2.3.7
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-29 09:47:03 +08:00
06e06aae05
enhance: [skip e2e]change to a better name ( #30330 )
...
pr: #30329
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-29 09:45:08 +08:00
77e123762f
enhance: add graceful stop timeout to avoid node stop hang under extreme cases ( #30320 )
...
1. add coordinator and proxy graceful stop timeout to 5s.
3. add other work node graceful stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
4. change the order of datacoord component while stop.
5. `LivenessCheck` do not perform graceful shutdown now.
issue: https://github.com/milvus-io/milvus/issues/30310
pr: #30317
also see: https://github.com/milvus-io/milvus/pull/30306
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-27 08:45:02 +08:00
650dcc512e
fix: dead lock while getting configs ( #30319 )
...
issue: https://github.com/milvus-io/milvus/issues/30295
pr: #30318
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-26 20:15:01 +08:00
26df754514
enhance: Bump milvus version to 2.3.7 ( #30297 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-26 11:07:02 +08:00
e0f987ee9b
enhance: Allows proactive warming up of chunk cache ( #30182 ) ( #30289 )
...
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
pr: https://github.com/milvus-io/milvus/pull/30182
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-26 09:57:01 +08:00
2c4d0605ef
enhance: add a weight for growing row count when balancing segments ( #30293 )
...
Cherry-pick from master
pr: #30271
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-26 09:17:03 +08:00
d182a51653
fix: [Cherry-pick] Use correct pools for all CGO methods in segments pkg ( #30275 )
...
Cherry-pick from master
pr: #30274
See also #30273
This PR:
- Rename confusing `LoadIndexInfo` to `UpdateIndexInfo` for LocalSegment
- Use `DynamicPool` instead of `LoadPool` for `UpdateSealedSegmentIndex`
- Fix cgo call missing pool control
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 19:49:01 +08:00
1a54571c10
enhance: [Cherry-pick] Add trace span for scheduling read tasks in QueryNode ( #30266 )
...
Cherry-pick from master
pr: #30265
This PR adds a trace span for search/query task scheduling duration
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 15:39:01 +08:00
9e8eb2aa51
fix: Revert leader checker related check ( #30262 )
...
See also #30150
PR reverted: #29984 #30152
Currently this scenario could not be covered by ut/it/e2e test cases
Revert it for now
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 12:39:02 +08:00
e3114b6a4d
enhance: [2.3] Utilize partition key optimization in reQuery ( #30255 )
...
Partial cherry-pick from master due to code branching
pr: #30253
See also #30250
This PR add requery flag in query task. When reQuery flag is true, query
task shall skip partition name conversion and use pre-calculated
partitionIDs passed from search task.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 11:05:07 +08:00
44c2ae130b
enhance: [skip e2e]increase timeout for image build ( #29083 ) ( #30239 )
...
pr: #29083
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-24 15:37:02 +08:00
285eade31c
enhance: [skip e2e]increase timeout for building gpu image ( #30237 )
...
pr: #30238
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-24 15:35:13 +08:00
95cd6f20d0
fix: [2.3] wrong format expr for the delete rest api ( #30218 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/30092
pr: #30217
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:05 +08:00
44545e2fc4
[automated] Bump milvus version to v2.3.6 ( #30234 )
...
Bump milvus version to v2.3.6
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-24 09:49:00 +08:00
efea282111
feat: [Pick] Support tencent cloud object storage for milvus ( #30210 )
...
issue: #30162
master pr: #30163
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 16:07:01 +08:00
35e4165722
enhance: [2.3] make Load process traceable in querynode & segcore ( #30187 )
...
Cherry-pick from master, modified some files since branching
pr: #29858
See also #29803
This PR:
- Add trace span for LoadIndex & LoadFieldData in segment loader
- Add TraceCtx parameter for Index.Load in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-23 15:58:57 +08:00
e0beea7fb6
test: update pymilvus version and test cases ( #30209 )
...
issue: #17614
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-01-23 14:40:55 +08:00
4d0a6dbc25
fix: written file size is over the int32 range and raises error ( #30057 ) ( #30207 )
...
we sum the total data size in int32, which could lead to an overflow
error
related #30056
pr: #30057
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-23 13:50:56 +08:00
9bd94c4fab
fix: the system rejects all queries and never recovers if enabled read rate limit ( #30061 ) ( #30196 )
...
fix #30060
pr: #30061
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-23 10:37:00 +08:00
565d37ced8
enhance: Bump version to 2.3.6 ( #30184 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-22 19:04:55 +08:00
0e71923408
enhance: enable converting segcore error to merr ( #29914 ) ( #30178 )
...
this converts the segcore error to merr if possible
pr: #29914
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:56:55 +08:00
1cc5a613d5
enhance: adjust the GPU pool size ( #29937 ) ( #30177 )
...
according to benchmark, the GPU pool size with 6 performs best
pr: #29937
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:55:04 +08:00
c8a129756f
enhance: filter out the not needed collections while listing ( #29690 ) ( #30180 )
...
this improves performance while many collections exist resolve #29631
pr: #29690
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:52:55 +08:00
6aaccdd5f4
feat: support general capacity restrict for cloud-side resoure contro… ( #30017 )
...
related: #29844
pr: #https://github.com/milvus-io/milvus/pull/29845
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 16:18:56 +08:00
2465d86138
enhance: [2.3] support related privilege for grant api ( #30154 )
...
/kind improvement
pr: #30153
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-22 14:42:55 +08:00
aoiasd
jaime
congqixia
cai.zhang
yah01
yihao.dai
smellthemoon
SimFG
sammy.huang
cqy123456
chyezh
xige-16
sre-ci-robot
Bingyi Sun
nico
MrPresent-Han