1. Using secondary index to avoid retrieving all segments at
`GetSegmentsChanPart`.
2. Perform batch SetAllocations to reduce the number of times the meta
lock is acquired.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38399
- make broadcast service available for msgstream by reusing the
architecture streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #39003
Previous PR #39004 has to clone & flip bitset due to bitset does not
support find0 operator. #39176 added this feature so clone & flip could
be removed now.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38715
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field `MemSize` (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #39124
`bitset::find_first()` and `bitset::find_next()` now accept one more
parameter, which allows to search for `0` bit instead of `1` bit
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
issue: #38970
cause the stopping balance channel still use the row_count_based policy,
which may causes channel unbalance in multi-collection case.
This PR impl a score based stopping balance channel policy.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
instead of marking as not supported,
`ChunkedSparseFloatColumn::DataByteSize` can simply use the impl of
super class.
issue: https://github.com/milvus-io/milvus/issues/39158
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #39001
Background:
Segment Load Version: Each segment load request assigns a timestamp as
its version. When multiple copies of a segment are loaded on different
QueryNodes, the leader checker uses this version to identify the latest
copy and updates the routing table in the leader view to point to it.
Delegator Router Version: When a delegator builds a route to a QueryNode
that has loaded a segment, it also records the segment's version.
Router Table Update Logic: If the leader checker detects that the
version of a segment in the routing table does not match the version in
the worker, it updates the routing table to point to the QueryNode with
the latest version. Additionally, it updates the segment's load version
in the QueryNode during this process.
Issue:
When a channel is undergoing load balancing, the leader checker may sync
the routing table to a new delegator. This sync operation modifies the
segment's load version, which invalidates the routing in the old
delegator. Subsequently, the leader checker updates the routing table in
the old delegator, breaking the routing in the new delegator. This cycle
continues, causing repeated updates and inconsistencies.
Fix:
This PR introduces two changes to address the issue:
1. Use NodeID to verify whether the delegator's routing table needs an
update, avoiding unnecessary modifications.
2. Ensure compatibility by using the latest segment's load version as
the version recorded in the routing table.
These changes resolve the cyclic updates and prevent the leader checker
from generating excessive duplicate tasks, ensuring routing stability
across delegators during load balancing.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #38399
- Add new rpc for transfer broadcast to streaming coord
- Add broadcast service at streaming coord to make broadcast message
sent automicly
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #39083
/kind improvement
Three new functions of the static web page:
1. The input box can be expanded and scrolled if it exceeds the maximum
size
2. Input history
3. It will simply check whether the quotation marks and brackets appear
in pairs
Signed-off-by: SimFG <bang.fu@zilliz.com>
Related to #38923
This PR:
- Check whether `os.Stat` config file error is io.ErrNotExist
- Panic when get config return error during Milvus initialization
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38399
We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #39003
Copying bitset value bit by bit is slow and CPU heavy, this PR utilizes
bitset operator "|=" to accelerate this procedure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
yihao.dai
Cai Yudong
congqixia
aoiasd
junjiejiangjjj
Zhen Ye
Buqian Zheng
Alexander Guzhva
sre-ci-robot
Bingyi Sun
smellthemoon
sthuang
wei liu
Spade A
zhuwenxing
SimFG
XuanYang-cn
Ted Xu
Xianhui Lin
wt
ThreadDao
jaime
Gao
yanliang567