issue: #43088
issue: #43038
The current loading process:
* When loading an index, we first download the index files into a list
of buffers, say A
* then constructing(copying) them into a vector of FieldDatas(each file
is a FieldData), say B
* assembles them together as a huge BinarySet, say C
* lastly, copy into the actual index data structure, say D
The problem:
* We can see that, after each step, we don't need the data in previous
step.
* But currently, we release the memory of A, B, C only after we have
finished constructing D
* This leads to a up to 4x peak memory usage comparing with the raw
index size, during the loading process
* This PR allows timely releasing of B after we assembled C. So after
this PR, the peak memory usage during loading will be up to 3x of the
raw index size.
I will create another PR to release A after we created B, that seems
more complicated and need more work.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Realted #43520
Datanode may have memory leakage when reader returns error. In
previously mention issue, datanodes got OOM killed due to continueous
error in read path.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
related: #42417
- update the isValidUsername function to accept dots and hyphens in
addition to letters, digits, and underscores
- this change improves compatibility with common username formats and
addresses feedback in issue #42417
Signed-off-by: Jean-Francois Weber-Marx <jfwm@hotmail.com>
Signed-off-by: Jean-Francois Weber-Marx <jf.webermarx@criteo.com>
issue: #42884
Fixes an issue where delete records for a segment are lost from the
delete buffer if `load segment` execution on the delegator is too slow,
causing `syncTargetVersion` or other cleanup operations to clear them
prematurely.
Changes include:
- Introduced `Pin` and `Unpin` methods in `DeleteBuffer` interface and
its implementations (`doubleCacheBuffer`, `listDeleteBuffer`).
- Added a `pinnedTimestamps` map to track timestamps protected from
cleanup by specific segments.
- Modified `LoadSegments` in `shardDelegator` to `Pin` relevant segment
delete records before loading and `Unpin` them afterwards.
- Added `isPinned` check in `UnRegister` and `TryDiscard` methods of
`listDeleteBuffer` to skip cleanup if corresponding timestamps are
pinned.
- Added comprehensive unit tests for `Pin`, `Unpin`, and `isPinned`
functionality, covering basic, multiple pins, concurrent, and edge
cases.
This ensures the integrity of delete records by preventing their
premature removal from the delete buffer during segment loading.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #43522
Currently, passing partial schema to storage v2 packed reader may
trigger SEGV during clustering compaction unit test.
This patch implement `NeededFields` differently in each `RecordReader`
imlementation. For now, v2 will implemented as no-op. This will be
supported after packed reader support this API.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43072, #43289
- manage the schema version at recovery storage.
- update the schema when creating collection or alter schema.
- get schema at write buffer based on version.
- recover the schema when upgrading from 2.5.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
1. Modify the binlog reader to stop reading a fixed 4096 rows and
instead use the calculated bufferSize to avoid generating small binlogs.
2. Use a fixed bufferSize (32MB) for the Parquet reader to prevent OOM.
issue: https://github.com/milvus-io/milvus/issues/43387
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
currently we multiplied the requesting size when adding to loading, but
did not do so when estimating projected usage.
issue: #43088
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #43113
This PR:
- Change member of FieldIndex from `FieldMeta &` to needed `DataType`
and dim member resolving dangling reference after schema change
- Add double check after acquiring lock to reduce multiple assignment
- Change `auto schema` to `auto& schema` to reduce schema copy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The root cause of the issue lies in the fact that when a sealed segment
contains multiple row groups, the get_cells function may receive
unordered cids. This can result in row groups being written into
incorrect cells during data retrieval.
Previously, this issue was hard to reproduce because the old Storage V2
writer had a bug that caused it to write row groups larger than 1MB.
These large row groups could lead to uncontrolled memory usage and
eventually an OOM (Out of Memory) error. Additionally, compaction
typically produced a single large row group, which avoided the incorrect
cell-filling issue during query execution.
related: https://github.com/milvus-io/milvus/issues/43388,
https://github.com/milvus-io/milvus/issues/43372,
https://github.com/milvus-io/milvus/issues/43464, #43446, #43453
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
This patch makes `PackedReader` return EOF when try to calling
`ReadNext` after closing it.
This behavior make importv2.binlog reader could retry after EOF reached
and act normally.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR fill default value for `PackedBinlogRecordWriter` timestamp
range so target segment meta will contains correct timestamp range
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43434
- the segment start position can be carried by other segment sync
operation. so the sync start position operation can happens before
insert.
- TODO: It's a wired design should be removed.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #41435
this is to prevent AI from thinking of our exception throwing as a
dangerous PANIC operation that terminates the program.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/41435
turns out we have per file binlog size in golang code, by passing it
into segcore we can support eviction in storage v1
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Realted to #43407
When `MultiSaveAndRemove` like ops contains same key in saves and
removal keys it may cause data lost if the execution order is save first
than removal.
This PR make all the kv execute removal first then save the new values.
Even when same key appeared in both saves and removals, the new value
shall stay.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41435
issue: https://github.com/milvus-io/milvus/issues/43038
This PR also:
1. removed ERROR state from ListNode
2. CacheSlot will do reserveMemory once for all requested cells after
updating the state to LOADING, so now we transit a cell to LOADING
before its resource reservation
3. reject resource reservation directly if size >= max_size
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #42995
- don't balance the wal if the producing-consuming lag is too long.
- don't balance if the rebalance is set as false.
- don't balance if the wal is balanced recently.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #43117, #42966, #43373
- also fix channel balance may not work at 2.6.
- fix error lost at delete path
- add mvcc into s/q log
- change the log level for TestCoordDownSearch
Signed-off-by: chyezh <chyezh@outlook.com>
Previous code uses diskSegmentMaxSize if and only if all of the
collection's vector fields are indexed with DiskANN index.
When introducing sparse vectors, since sparse vector cannot be indexed
with DiskANN index, collections with both dense and sparse vectors will
use maxSize instead.
This PR changes the requirments of using diskSegmentMaxSize to all dense
vectors are indexed with DiskANN indexs, ignoring sparse vector fields.
See also: #43193
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
fix: https://github.com/milvus-io/milvus/issues/43354
The current implementation of stdsort index is not supported for
std::string. Remove the code.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Correct read and buffer size to 64MB to prevent OOM during clustering
compaction.
issue: https://github.com/milvus-io/milvus/issues/43310
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Related to #43262
This patch fixes following logic bug:
- When multiple chunks are loaded and size cannot be divided by 8, just
appending uint8_t as bitmap will cause null bitmap dislocation
- `null_bitmap_data()` points to start of whole row group, which may not
stand for current `arrow::Array`
The current solutions is:
- Reorganize the null_bitmap with currect size & offset
- Pass `array->offset()` in tuple to info the current offset
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Ref https://github.com/milvus-io/milvus/issues/42053
This PR enable ngram to support more kinds of matches such as prefix and
postfix match.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Related to #43250
Use FieldIDList to check missing field. If column is missing, return
empty resultset
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41570
Fix issue where growing and sealed segments could be searched
simultaneously, causing inflated count(*) results. This was caused by
logic introduced in PR #42009 that made sealed segments readable before
target version advancement.
Changes include:
- Fix conditional filtering logic in PinReadableSegments to prevent
sealed segments from becoming readable prematurely
- Use target version filter for full results (ratio=1.0) to ensure
sealed segments only become readable after target advancement
- Use query view segment list filter for partial results (ratio<1.0) to
maintain backward compatibility
- Simplify target version setting in AddDistributions to prevent
premature segment readability
- Add logging for redundant growing segments during sync
- Add comprehensive unit tests covering the duplicate segment scenario
This fix ensures count(*) queries return accurate results by preventing
the same segment from being counted in both growing and sealed states.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/42900
@sunby Unfortunately, it is not that easy to fix as it was thought in
#43177
Upd: also handles `Inf` and `NaN` values, and the division by zero case
for `fp32` and `fp64`
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
issue: #42995
- The consuming lag at streaming node will be reported to coordinator.
- The consuming lag will trigger the write limit and deny by quota
center.
- Set the ttProtection by default.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #43113
When schema change happens, insert shall not happen, otherwise:
- Data race may happen causing insertion failure
- Inconsistent data schema
This PR add shared_lock prevent this data race.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43107
- Add checkLoadConfigChanges() to apply load config during startup
- Call config check in startQueryCoord() after restart
- Skip auto-updates for collections with user-specified replica numbers
- Add is_user_specified_replica_mode field to preserve user settings
- Add comprehensive unit tests with mockey
Ensures existing collections use latest cluster-level config after
restart.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #39178
This PR add logs for segment schema change operations.
Also fixes the nit comments from PR #42490
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
- Introduce dynamic buffer sizing to avoid generating small binlogs
during import
- Refactor import slot calculation based on CPU and memory constraints
- Implement dynamic pool sizing for sync manager and import tasks
according to CPU core count
issue: https://github.com/milvus-io/milvus/issues/43131
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
use meta to get chunked column memory size to avoid getting cells
actually from storage.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #43074
- fix: panic when logging a old message should be skipped, #43074
- fix: make the ack of broadcaster idompotent, #43026
- fix: lost dropping collection when upgrading, #43092
- fix: panic when DropPartition happen after DropCollection, #43027,
#43078
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #43099
Previously `cache_raw_data_to_disk_common` used `milvus::DataType`
template typename, which shall be `knowhere::bf16` or other actual
datatype.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42833
- also fix the error metric for async cgo.
- also make sure the roles can be seen when node startup, #43041.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #43040
This patch introduces a disk file writer that supports Direct IO.
Currently, it is exclusively utilized during the QueryNode load process.
Below is its parameters:
1. `common.diskWriteMode`
This parameter controls the write mode of the local disk, which is used
to write temporary data downloaded from remote storage.
Currently, only QueryNode uses 'common.diskWrite*' parameters. Support
for other components will be added in the future.
The options include 'direct' and 'buffered'. The default value is
'buffered'.
2. `common.diskWriteBufferSizeKb`
Disk write buffer size in KB, only used when disk write mode is
'direct', default is 64KB.
Current valid range is [4, 65536]. If the value is not aligned to 4KB,
it will be rounded up to the nearest multiple of 4KB.
3. `common.diskWriteNumThreads`
This parameter controls the number of writer threads used for disk write
operations. The valid range is [0, hardware_concurrency].
It is designed to limit the maximum concurrency of disk write operations
to reduce the impact on disk read performance.
For example, if you want to limit the maximum concurrency of disk write
operations to 1, you can set this parameter to 1.
The default value is 0, which means the caller will perform write
operations directly without using an additional writer thread pool.
In this case, the maximum concurrency of disk write operations is
determined by the caller's thread pool size.
Both parameters can be updated during runtime.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Related to #43028
This PR:
- Add mutex prevent concurrent load segment & schema change
- Add schema verison field in load meta
- Update schema in PutOrRef if schema verison is larger
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Ref #42053
This is the first PR for optimizing `LIKE` with ngram inverted index.
Now, only VARCHAR data type is supported and only InnerMatch LIKE
(%xxx%) query is supported.
How to use it:
```
milvus_client = MilvusClient("http://localhost:19530")
schema = milvus_client.create_schema()
...
schema.add_field("content_ngram", DataType.VARCHAR, max_length=10000)
...
index_params = milvus_client.prepare_index_params()
index_params.add_index(field_name="content_ngram", index_type="NGRAM", index_name="ngram_index", min_gram=2, max_gram=3)
milvus_client.create_collection(COLLECTION_NAME, ...)
```
min_gram and max_gram controls how we tokenize the documents. For
example, for min_gram=2 and max_gram=4, we will tokenize each document
with 2-gram, 3-gram and 4-gram.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue: #42404
Add syncedByCoord field to ensure delegator only becomes serviceable
after coordinator sync, preventing unreliable service state when memory
is insufficient.
Issue: When memory is low, delegator may become serviceable before
current target is ready, but segments can be released at any time,
making the serviceable state unreliable.
Changes include:
- Add syncedByCoord field to track coordinator sync status
- Update Serviceable() to require both data readiness and coord sync
- Set syncedByCoord=true in SyncTargetVersion
- Add comprehensive test coverage
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #41609
- add env `MILVUS_NODE_ID_FOR_TESTING` to set up a node id for milvus
process.
- add env `MILVUS_CONFIG_REFRESH_INTERVAL` to set up the refresh
interval of paramtable.
- Init paramtable when calling `paramtable.Get()`.
- add new multi process framework for integration test.
- change all integration test into multi process.
- merge some test case into one suite to speed up it.
- modify some test, which need to wait for issue #42966, #42685.
- remove the waittssync for delete collection to fix issue: #42989
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42860
Fix channel node allocation when QueryNode count is not a multiple of
channel count. The previous algorithm used simple division which caused
uneven distribution with remainders.
Key improvements:
- Implement smart remainder distribution algorithm
- Refactor large function into focused helper functions
- Support two-phase rebalancing (release then allocate)
- Handle edge cases like insufficient nodes gracefully
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #39173
This PR
- Close packed reader after sort
- Release arrow.Record preventing memory leakage
- Invoke `pack_reader->Close()` for CloseReader
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix issues in end-to-end tests:
1. **Split column groups based on schema**, rather than estimating by
average chunk row size. **Ensure column group consistency within a
segment**, to avoid errors caused by loading multiple column group
chunks simultaneously.
2. **Use sorted segmentId** when generating the stats binlog path, to
ensure consistent and correct file path resolution.
3. **Determine field IDs as follows**:
For multi-column column groups, retrieve the field ID list from
metadata.
For single-column column groups, use the column group ID directly as the
field ID.
related: #39173fix: #42862
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
In corner cases where there are many fields but only a small number of
rows to import, the default preallocated IDs may be insufficient. To
address this, consider the number of fields when preallocating IDs.
issue: https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #42098
Fix concurrent access issue by adding proper locking around
ReleasePartition operation to prevent race conditions when releasing
partitions on the same collection.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42856
Data under mmapped growing segment shall be treated respecting
growingMmap setting. Otherwise, varchar datatype could be treated with
logic error.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42918
Previously the `CollectionTtlTimestamp` could be overflowed when the
guarantee_ts==1, which means using `Eventually` consistency level.
This patch use task timestamp, allocated by scheduler, to generate ttl
timestamp ignore the potential very small timestamp being used.
Also add overflow check for ttl timestamp calculated.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR moves the deltalog file count check inside hasTooManyDeletions
check. Unifies the logic on checking if a segment has too many deletions
including: delta log count, deleted rows ratio and deltalog size.
This change removes several uncessary traverse through segment's binlogs
and deltalogs. And add more clear trigger logs
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Related to #42856
Default value will be missing after segment get sorted/compacted. This
PR is a temp workaround since in long term default value shall be filled
with storage engine instead.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42489
Since load list works as hint after cachelayer implemented, the related
check logic could be removed to keep code logic clean.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
In storage v2, "narrow" column group could have group id not mapped
schema, which causing loading fails or resource estimation result
inaccurate.
This PR handles this case by mapping binlog from index instead of vice
versa.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42098#42404
Fix critical issue where concurrent balance segment and balance channel
operations cause delegator view inconsistency. When shard leader
switches between load and release phases of segment balance, it results
in loading segments on old delegator but releasing on new delegator,
making the new delegator unserviceable.
The root cause is that balance segment modifies delegator views, and if
these modifications happen on different delegators due to leader change,
it corrupts the delegator state and affects query availability.
Changes include:
- Add shardLeaderID field to SegmentTask to track delegator for load
- Record shard leader ID during segment loading in move operations
- Skip release if shard leader changed from the one used for loading
- Add comprehensive unit tests for leader change scenarios
This ensures balance segment operations are atomic on single delegator,
preventing view corruption and maintaining delegator serviceability.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
ref https://github.com/milvus-io/milvus/issues/42626
This PR makes text match index and json key stats index be loaded based
on mmap config.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Ref #42626
This path tidy up path for scalar index including path for loading index
from remote storage and temporary path for buliding index.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Related to #42773
Growing segment fills all known meta into `InsertRecord` data, which
cause even the field is missing, the field data will still exists.
This PR update the logic while finish loading growing segment to check
field empty or not instead of existence.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42661
Add NotStopped() and IsWorking() methods to shardDelegator for better
state management and error handling.
Changes include:
- Add instance state checking methods with proper error messages
- Replace lifetime package calls with delegator instance methods
- Add comprehensive unit tests for state transitions and error cases
- Improve error reporting with channel name for better debugging
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #41858
This PR add log while debugging schema mismatch between pymilvus cache
and proxy schema.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
`null_bitmap_data()` returns raw pointer of null bitmap of Array. While
after slicing, this bitmap is not rewritten due to zero copy
implementation, so the current start pos maybe non-zero while
FillFieldData generating column `valid_data` array.
This PR add `offset` param for `FillFieldData` method, and force all
invocation pass correct offset of `null_bitmap_data` ptr.
Also update milvus-storage commit fixing reader failed to return data
when buffer size smaller than row group size problem.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42735
Load field list shall work as hint after tiered storage impl, so the
load list compare is meaningless and block load with empty list after
adding a new field.
This PR totally moves the check logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42561
Move the exclude nodes clearing logic from ExecuteWithRetry to
selectNode after shard leader cache refresh to ensure proper retry
behavior:
- Remove premature exclude clearing in ExecuteWithRetry that happened
before shard leader cache update
- Add exclude clearing logic in selectNode after refreshing shard leader
cache when all replicas are excluded
- Ensure multiple retries can properly update shard leader cache and
clear exclude list when needed
- Add comprehensive tests for edge cases including empty shard leaders
and mixed serviceable node scenarios
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42723
Previous PR #42684 permit insert msg transformation but insertCodec did
not adapt the same skip logic, whic causes panicking.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42649
- the sync operation of different pchannel is concurrent now.
- add a option to notify the backlog clear automatically.
- make pulsar walimpls can be recovered from backlog exceed.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42663
Fix panic issue when processing VchannelInfo messages from older
coordinator versions that don't have DeleteCheckpoint field.
Changes:
- Add null safety check for DeleteCheckpoint before accessing methods
- Maintain backward compatibility with legacy message formats
- Improve seek position selection logic for both old and new versions
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #41858#41951#42084
When insert msg consumer (pipeline/flowgraph) have newer schema than
insertMsg, it have to adapter the insert msg used old schema(missing
newly added field)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42702
This patch add wait logic for `CheckerController` and nil check for
channel checker in case of panicking during server/testcase stop
procedure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42640
The search/query plan holded a reference to schema, which could be
destructed after schema change. This PR make plan hold a shared ptr to
it fixing dangling reference problem under concurrent read & schema
change.
This PR also remove field binlog check for loading index for old segment
with old schema may have binlog lack.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
1. Optimize the import process: skip subsequent steps and mark the task
as complete if the number of imported rows is 0.
2. Improve import integration tests:
a. Add a test to verify that autoIDs are not duplicated
b. Add a test for the corner case where all data is deleted
c. Shorten test execution time
3. Enhance import logging:
a. Print imported segment information upon completion
b. Include file name in failure logs
issue: https://github.com/milvus-io/milvus/issues/42488,
https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Ref https://github.com/milvus-io/milvus/issues/42148
This PR mainly enables segcore to support array of vector (read and
write, but not indexing). Now only float vector as the element type is
supported.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue: #42028
W20250522 09:52:55.785657 12779 ChunkedSegmentSealedImpl.cpp:1752]
[SERVER][generate_interim_index][CGO_LOAD][]fail to generate binlog
index, because bad optional access
After the cachelayer is added, num_rows_ can not be obtained before
interim index generated , and an external parameter pass is required
Signed-off-by: foxspy <xianliang.li@zilliz.com>
Defer clone and decompress operations until just before removing from
meta, instead of eagerly applying them to all segments in advance.
issue: https://github.com/milvus-io/milvus/issues/42592
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #42165
Implement dynamic task execution capacity calculation based on QueryNode
CPU core count instead of static configuration for better resource
utilization.
Changes include:
- Add CpuCoreNum() method and WithCpuCoreNum() option to NodeInfo
- Implement GetTaskExecutionCap() for dynamic capacity calculation
- Add QueryNodeTaskParallelismFactor parameter for tuning
- Update proto definition to include cpu_core_num field
- Add unit tests for new functionality
This allows QueryCoord to automatically adjust task parallelism based on
actual hardware resources.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42629
This PR handles converting default value to int8/int18 scalar with int32
default value definition
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42028
- limit the concurrency of zstd compression.
- zstd.go modified from
`github.com/apache/arrow/go/v17/parquet/compress/ztsd.go`
- may be related to #42129
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #39333#41570
Fix delete record missing issue introduced in PR #39552 L0 refactoring:
- Use delete checkpoint as consume start position when deleteCP <
channelCP
- Add logging when delete checkpoint is used instead of seek position
- Prevent delete record loss when deleteCP is earlier than default
channelCP
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue #42492
- streamingcoord start before old rootcoord.
- streaming balancer will check the node session synchronously to avoid
redundant operation when cluster startup.
- ddl operation will check if streaming enabled, if the streaming is not
enabled, it will use msgstream.
- msgstream will initialize if streaming is not enabled, and stop when
streaming is enabled.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #39173
Like logic in #41919, storage v2 fs shall use complete paths with
bucketName prefix to be compatible with its definition. This PR fills
bucket name from config when creating reader for compaction tasks.
NOTE: the bucket name shall be read from task params config for
compaction task pooling.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
`nullable` flag is crucial for serde logic of v2 writer, missing this
flag causes logic bug for v2 nullalbe data.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42098#42404
related to: ##42009 #41937
Implement new method to handle partition removal from next target
without directly modifying current target.
Changes include:
- Add RemovePartitionFromNextTarget method and deprecate RemovePartition
- Update target_observer to use new method for ReleasePartition
operations
- Add unit tests and mock methods for new functionality
This ensures that all changes to next target will propagates to
delegator's query view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Increase insert buffer size from 16MB to 64MB, while keeping delete
buffer size at 16MB.
issue: https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Each cell of a field should be written to its own mmap file, rather than
writing all cells of the field into a single mmap file.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
the manager's logging lambda should not capture the pipeline object
this creates a circular reference between the manager and the pipeline
object, making it impossible for both to be GC-ed.
issue: https://github.com/milvus-io/milvus/issues/42581
Signed-off-by: Buqian Zheng <buqianzheng@Buqians-MacBook-Air.local>
Co-authored-by: Buqian Zheng <buqianzheng@Buqians-MacBook-Air.local>
Previous PR: #42017 did not work due to following updated points by this
PR:
- Initialize the `name_map`, which not touched at all before
- Trim the thread name under 15 characters to fit syscall limit
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42530
The cluster id is missing when drop worker drop causing redoing task on
report duplicated task error.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42498
- fix: sealed segment cannot be flushed after upgrading
- fix: get mvcc panic when upgrading
- ignore the L0 segment when graceful stop of querynode.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Remove the unlimited logID mechanism and switch to redundantly
allocating a large number of IDs.
issue: https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #42492
- consider the old RO query node (not streaming node) when balancing
channel.
- querynode graceful stop can be done if there's only L0 segment exists.
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #42489
See also #41435
This PR's main target is to make partial load field list work as caching
layer warmup policy hint. If user specify load field list, the fields
not included in the list shall use `disabled` warmup policy and be able
to lazily loaded if any read op uses them.
The major changes are listed here:
- Pass load list to segcore and creating collection&schema
- Add util functions to check field shall be proactively loaded
- Adapt storage v2 column group, which may lead to hint fail if columns
share same group
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Remove the hardcoded batchSize of 100,000 and instead trigger a write
every 64MB based on actual data size. This prevents sort stats from
generating excessively large binlog files.
issue: https://github.com/milvus-io/milvus/issues/42400
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Remove the 'goccy/go-json' library, which was inadvertently introduced,
and revert to using the standard internal JSON handling.
Changes include:
- Removed dependency on 'github.com/goccy/go-json' in go.mod and go.sum.
- Replaced import of 'goccy/go-json' with 'internal/json' in
'internal/querycoordv2/task/scheduler.go'.
This correction ensures the project continues to use the intended JSON
processing libraries and avoids unnecessary external dependencies.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #42457
Replace unsafe ExpectedCalls modification with atomic.Int32 state
tracking to avoid race conditions in concurrent test execution. Changes
include:
- Use atomic counters instead of direct mock ExpectedCalls manipulation
- Add RunAndReturn with atomic state transitions for thread safety
- Remove github.com/samber/lo dependency
This prevents data race when mock framework and test goroutines access
ExpectedCalls concurrently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #39173#39718
In storage v2, the `lack_bin_rows` cannot be used since field id is not
column group id, which will not be matched forever.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42176
Remove the mutual exclusion constraints between channel and segment
balance tasks to allow them to run concurrently.
Changes include:
- Remove permitBalanceChannel() and permitBalanceSegment() methods from
RoundRobinBalancer
- Update ChannelLevelScoreBalancer, MultiTargetBalancer,
RowCountBasedBalancer, and ScoreBasedBalancer to remove constraint
checks
- Allow segment balance tasks to proceed even when channel balance tasks
are running
- Update test cases to reflect new behavior where balance tasks no
longer block each other
This change improves the efficiency of load balancing by removing
unnecessary coordination overhead between different types of balance
operations.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #39173
This PR
- Handle storage v2 log path in local storage mode on querynode
- Ignore field info check when append index for loaded sealed segment
when using storage v2
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41976
- make drop partition message as a broadcast message.
- add gc when drop partition message is acked.
- add a call back to handle the broadcast message when ack.
- the ack operation of broadcast message will retry until success.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #40532
- start timeticksync at rootcoord if the streaming service is not
available
- stop timeticksync if the streaming service is available
- open a read-only wal if some nodes in cluster is not upgrading to 2.6
- allow to open read-write wal after all nodes in cluster is upgrading
to 2.6
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42162
- enhance: add read ahead buffer size issue #42129
- fix: rocksmq consumer's close operation may get stucked
- fix: growing segment from old arch is not flushed after upgrading
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42147
- error of sync task should be returned if error is returned to avoid
checkpoint is push forward.
- fix up node id checker of UpdateChannelCheckpoint in streaming.
Signed-off-by: chyezh <chyezh@outlook.com>
This PR fixes a minor typo in a log message in the `PreExecute` method
of `internal/datanode/index/task_index.go`.
Corrected "dimesion" to "dimension".
Signed-off-by: hckex <33862757+hckex@users.noreply.github.com>
Related to #42084
Embedding node cached schema when created, causing schema mismatch after
schema change. This PR make embeddingNode use schema from metacache,
which will be updated.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/41690
This commit implements partial search result functionality when query
nodes go down, improving system availability during node failures. The
changes include:
- Enhanced load balancing in proxy (lb_policy.go) to handle node
failures with retry support
- Added partial search result capability in querynode delegator and
distribution logic
- Implemented tests for various partial result scenarios when nodes go
down
- Added metrics to track partial search results in querynode_metrics.go
- Updated parameter configuration to support partial result required
data ratio
- Replaced old partial_search_test.go with more comprehensive
partial_result_on_node_down_test.go
- Updated proto definitions and improved retry logic
These changes improve query resilience by returning partial results to
users when some query nodes are unavailable, ensuring that queries don't
completely fail when a portion of data remains accessible.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Return `false` in the `Process()` function for `executing` or
`pipelining` state `l0Compaction`. This prevents the `l0Compaction` task
from being removed from the `CompactionInspector`'s executing queue,
thereby avoiding concurrent execution of `l0Compaction` and `Stats`.
issue: https://github.com/milvus-io/milvus/issues/42008
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Related to #40756
Large nq will naturally increase query time, which causing lots of slow
log when user NQ numbers are very large.
This PR make slow search counts span per nq (using avg val) to decide
whether one request is slow or not.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39718
The absent nullable field shall be filled at server-side in nullable
design. While the implementation here was buggy causing the feature was
not able to serve.
This PR make proxy fill the field data in correct format so that field
data with absent column(s) will be accepted.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR fixes a minor typo in the README file of the datanode module.
Corrected "imformation" to "information".
Signed-off-by: hckex <33862757+hckex@users.noreply.github.com>
Buqian Zheng
wei liu
sthuang
Shuyoou
Spade A
zhagnlu
yihao.dai
congqixia
Jean-Francois Weber-Marx
Zhen Ye
foxspy
cai.zhang
junjiejiangjjj
XuanYang-cn
Chun Han
sparknack
aoiasd
Bingyi Sun
Xianhui Lin
Ted Xu
Alexander Guzhva
PjJinchen
tinswzy
groot
rhys
cqy123456
Xiaowei Shi
hckex