Cherry-pick from master
pr: #29806
See also #29803
This PR:
- Add trace span for collection/partition load
- Use TraceSpan to generate Segment/ChannelTasks when loading
- Refine BaseTask trace tag usage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #29726
See also: #29113
Add a new utitliy function in `pkg/util/typetuil` to pre-allocate field
data slice capacity acoording to search limit. This shall avoid copying
the data during `AppendFieldData` when previous slice is out of space.
And shall also save CPU time during high paylog.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
pr: #29856
If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #29709#291712
pr: #29836
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:
1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.
issue: #29656
pr: #29749
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions
fix: #29766
pr: #29767
Signed-off-by: yah01 <yah2er0ne@outlook.com>
PR: https://github.com/milvus-io/milvus/pull/29797
enhance: We aim to cancel the previously initiated job or workflow
exclusively during the Pull Request (PR) stage
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
- this much improve the performance for GPU index
- this also reduce 1x copy while parsing index meta
pr: #29678
Signed-off-by: yah01 <yang.cen@zilliz.com>
Once a role is granted to a user, the user should automatically possess
the privilege information associated with that role.
issue: #29710
pr: #29747
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Cherry-pick from master
pr: #29668
See also #29113
The collection schema is crucial when performing search/query but some
of the information is calculated for every request.
This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #29494
pr: #29496
1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly
4. diff from old restful, named HandlersV1
Signed-off-by: PowderLi <min.li@zilliz.com>
this protect the loading index from failure, and speed up the loading
progress
pr: #29715
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
issue: #29582
pr: #29574
This PR rewrite gen segment plan logic based on assign segment in
`score_based_balancer`
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #29672
master pr: #29703
the storage account need privileges of actions
`Microsoft.Storage/storageAccounts/blobServices/containers/blobs/*` at
least
Signed-off-by: PowderLi <min.li@zilliz.com>
Cherry pick from master
pr: #29577
Related to #29575
Add `getCollectionTarget` method which is atomic when scope is
`CurrentTargetFirst` or `NextTargetFirst`
Also return error when executor finds no channel in target manager
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #29523
pr: #29525
readable shard leader should still be the old one during channel
balance, if the new shard leader is not ready.
This PR fixed that query coord choose wrong shard leader during balance
channel
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #28952
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #29548
See also #29526
Previous PR removed flushed segment info from request, which causes
pipeline failing to exclude flushed segment info
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #29517
See also #29516
Make `HandleCStatus` print trace id for better logging
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #29526
`WatchDmChannel` only need growing segment info, this PR removes fetch
segmentInfos when fill watch dml channel request.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
the executor always fetches the latest segment info, so we could consume
from the latest checkpoint, which could save much time while deleted
many entities
pr: #29455
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Cherry-pick from master
pr: #29464
Add log for releasing segment created during load process when load
error happens
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
wei liu
yah01
chyezh
sre-ci-robot
jaime
congqixia
zhenshan.cao
sammy.huang
XuanYang-cn
Jiquan Long
PowderLi
smellthemoon
cai.zhang
zhuwenxing
nico
SimFG
foxspy
MrPresent-Han