Cherry-pick from master
pr: #30072
issue: #30074
This PR fix that management restful api in proxy doesn't register to
http service
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: wei liu <wei.liu@zilliz.com>
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.
Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data
pr: #30473
Signed-off-by: yah01 <yang.cen@zilliz.com>
This pr decoups importing segment from flush process by:
1. Exclude the importing segment from the flush policy, this approch
avoids notifying the datanode to flush the importing segment, which may
not exist.
2. When RootCoord call Flush, DataCoord directly set the importing
segment state to `Flushed`.
issue: https://github.com/milvus-io/milvus/issues/30359
pr: https://github.com/milvus-io/milvus/pull/30402
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service resolve#30034
pr: #30035
Signed-off-by: yah01 <yang.cen@zilliz.com>
1. add coordinator and proxy graceful stop timeout to 5s.
3. add other work node graceful stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
4. change the order of datacoord component while stop.
5. `LivenessCheck` do not perform graceful shutdown now.
issue: https://github.com/milvus-io/milvus/issues/30310
pr: #30317
also see: https://github.com/milvus-io/milvus/pull/30306
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
pr: https://github.com/milvus-io/milvus/pull/30182
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Cherry-pick from master
pr: #30274
See also #30273
This PR:
- Rename confusing `LoadIndexInfo` to `UpdateIndexInfo` for LocalSegment
- Use `DynamicPool` instead of `LoadPool` for `UpdateSealedSegmentIndex`
- Fix cgo call missing pool control
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #30150
PR reverted: #29984#30152
Currently this scenario could not be covered by ut/it/e2e test cases
Revert it for now
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Partial cherry-pick from master due to code branching
pr: #30253
See also #30250
This PR add requery flag in query task. When reQuery flag is true, query
task shall skip partition name conversion and use pre-calculated
partitionIDs passed from search task.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master, modified some files since branching
pr: #29858
See also #29803
This PR:
- Add trace span for LoadIndex & LoadFieldData in segment loader
- Add TraceCtx parameter for Index.Load in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #30150
pr: #30151
This PR fix three problems:
1. the load request generated by leader checker doesn't set load scope
2. leader checker use wrong node id when generate release task, which
cause the release task finished immediately
3. the release request generated by leader_checker doesn't set the force
flag, the operation to clean leader view on delegator will fail.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #30066
Shuffle candidates to reduce scenario that some channel allocated into
same node
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
patch search cache param from index configs when index meta could not
get the search cache size key
issue: #30113
pr: #30119
Signed-off-by: xianliang <xianliang.li@zilliz.com>
issue: #29677#29838
pr: #29999
during get shard leaders, if qeurynode doesn't ack the heartbeat than
10s, querycoord will treat it as unavailable, and won't return shard
leader on it. but when querynode has a full cpu usage, it's easily to
stuck for more than 10s without ack the heartbeat, which cause no shard
leader to search/query.
This PR remove heartbeat lag logic during get shard leaders
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #29453
pr: #29452
sync distribution by rpc will also call loadSegment/releaseSegment,
which may cause all kinds of concurrent case on same segment, such as
concurrent load and release on one segment.
This PR add leader_checker which generate load/release task to correct
the leader view, instead of calling sync distribution by rpc
---------
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
cherry pick from master
pr: #29810#30055
`SampleFraction` shall be float and all `C.CString` shall be freed
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
chyezh
congqixia
wayblink
foxspy
aoiasd
cai.zhang
yah01
yihao.dai
smellthemoon
SimFG
cqy123456
xige-16
Bingyi Sun
MrPresent-Han
PowderLi
wei liu