issue: #43794
Fix the issue where L0 segments were not correctly selecting appropriate
delegators during loading, which could cause load failures or incorrect
delegator assignments.
Changes include:
- Add special handling for L0 segments in delegator selection logic
- Find delegators that are missing the L0 segment for direct loading
- Fallback to existing serviceable delegator selection when no suitable
delegator is found for L0 segments
- Add comprehensive test coverage for L0 segment loading scenarios
- Test delegator selection when some delegators are missing segments
- Test fallback behavior when all delegators already have the segment
- Test error handling when no delegators are available
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
pr: #43833
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Alexander Guzhva <alexanderguzhva@gmail.com>
issue: #43040
pr: #42665
This patch introduces a disk file writer that supports Direct IO.
Currently, it is exclusively utilized during the QueryNode load process.
Below is its parameters:
1. `common.diskWriteMode` This parameter controls the write mode of the
local disk, which is used to write temporary data downloaded from remote
storage. Currently, only QueryNode uses 'common.diskWrite*' parameters.
Support for other components will be added in the future.
The options include 'direct' and 'buffered'. The default value is
'buffered'.
2. `common.diskWriteBufferSizeKb` Disk write buffer size in KB, only
used when disk write mode is 'direct', default is 64KB.
Current valid range is [4, 65536]. If the value is not aligned to 4KB,
it will be rounded up to the nearest multiple of 4KB.
3. `common.diskWriteNumThreads` This parameter controls the number of
writer threads used for disk write operations. The valid range is [0,
hardware_concurrency]. It is designed to limit the maximum concurrency
of disk write operations to reduce the impact on disk read performance.
For example, if you want to limit the maximum concurrency of disk write
operations to 1, you can set this parameter to 1.
The default value is 0, which means the caller will perform write
operations directly without using an additional writer thread pool. In
this case, the maximum concurrency of disk write operations is
determined by the caller's thread pool size.
Both parameters can be updated during runtime.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Cherry-pick from master
pr: #43299
Related to #43031
Previous pr: #43064
Since old version may create collection with invalidate collection name,
milvus shall allow some API to let user notice such collection still
exists.
This patch removes collection name validation from `DescribeCollection`
call, letting user know that such collection still exists.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43557
In 2.5 branch, L0 segments must be loaded before other segments. If an
L0 segment has been garbage collected but is still in the target list,
the load operation would keep failing, preventing other segments from
being loaded.
This patch adds a segment existence check for L0 segments in
getSealedSegmentDiff. Only L0 segments that actually exist will be
included in the load list.
Changes:
- Add checkSegmentExist function parameter to SegmentChecker constructor
- Filter L0 segments by existence check in getSealedSegmentDiff
- Add unit tests using mockey to verify the fix behavior
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
During the rolling upgrade from 2.5 to 2.6, the 2.5 coordinator detects
newly started 2.6 proxies. However, 2.6 proxies do not sync timetick,
which leads to timetick delay. This PR ignores 2.6 proxies to prevent
ttDelay during the upgrade process.
issue: https://github.com/milvus-io/milvus/issues/43518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #42860
pr: #42786
Fix channel node allocation when QueryNode count is not a multiple of
channel count. The previous algorithm used simple division which caused
uneven distribution with remainders.
Key improvements:
- Implement smart remainder distribution algorithm
- Refactor large function into focused helper functions
- Support two-phase rebalancing (release then allocate)
- Handle edge cases like insufficient nodes gracefully
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #43408
Related to #43407
When `MultiSaveAndRemove` like ops contains same key in saves and
removal keys it may cause data lost if the execution order is save first
than removal.
This PR make all the kv execute removal first then save the new values.
Even when same key appeared in both saves and removals, the new value
shall stay.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Previous code uses diskSegmentMaxSize if and only if all of the
collection's vector fields are indexed with DiskANN index.
When introducing sparse vectors, since sparse vector cannot be indexed
with DiskANN index, collections with both dense and sparse vectors will
use maxSize instead.
This PR changes the requirments of using diskSegmentMaxSize to all dense
vectors are indexed with DiskANN indexs, ignoring sparse vector fields.
See also: #43193
pr: #43194
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Cherry-pick from master
pr: #43321
Related to #43262
This patch fixes following logic bug:
- When multiple chunks are loaded and size cannot be divided by 8, just
appending uint8_t as bitmap will cause null bitmap dislocation
- `null_bitmap_data()` points to start of whole row group, which may not
stand for current `arrow::Array`
The current solutions is:
- Reorganize the null_bitmap with currect size & offset
- Pass `array->offset()` in tuple to info the current offset
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42900
pr: #43179
Upd: also handles Inf and NaN values, and the division by zero case for
fp32 and fp64
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Alexander Guzhva
liliu-z
wei liu
sthuang
sparknack
Bingyi Sun
congqixia
yihao.dai
Xianhui Lin
zhagnlu
Spade A
Xiaofan
aoiasd
Chun Han
SimFG
XuanYang-cn
foxspy
cai.zhang