issue: #43858
Reduce CompactionDropToleranceInSeconds from 24 hours to 1 hour to
improve memory efficiency and faster task metadata cleanup. Changes
include:
- Update default value from 86400s (24h) to 3600s (1h) in
component_param.go
- Update corresponding configuration in milvus.yaml
- Faster cleanup of completed compaction task metadata
- Reduce memory footprint by shorter retention period
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/44011
namespace is an alias for tenant. if this property is enabled, milvus
will add a __namespace_id field.
Modifications in the future will use this property to do compaction and
search.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
related: #42417
- Add NameValidationAllowedChars and RoleNameValidationAllowedChars
configuration parameters to specify additional characters allowed
respectively in (generic) names and a role names
- All validations in validateName method is moved to a the new method
validateNameWithCustomChars which is called by both validateName
and ValidateRoleName while specifying characters allowed
Signed-off-by: Jean-Francois Weber-Marx <jfwm@hotmail.com>
Signed-off-by: Jean-Francois Weber-Marx <jf.webermarx@criteo.com>
issue: https://github.com/milvus-io/milvus/issues/44105
- I have added support to set this property
**queued.max.messages.kbytes** in kafka consumers from the user side.
- It limits the size (in KB) of the consumer’s local message queue
(buffer) where messages are temporarily stored after being fetched from
Kafka but before your application actually processes them
---------
Signed-off-by: Nischay Yadav <Nischay.Yadav@ibm.com>
issue: #43785
- pulsar client will print log into milvus logger now.
- pulsar client open the metric by default.
- upgrade the pulsar client to v0.15.1, and use offical repo.
- the fixing of milvus-io/pulsar-client-go is already covered by
official v0.15.1.
Signed-off-by: chyezh <chyezh@outlook.com>
- Add enable configuration for all model service providers
- Migrate environment variables from MILVUSAI_* to MILVUS_* prefix with
backward compatibility
- Unify model service enable/disable logic using configuration
- Add tests for environment variable parsing with fallback support
#35856
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
issue: #42416
- Rename the InsertMetric into ModifiedMetric.
- Add L0 control configuration.
- Add some L0 current state collect.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #41435
After introducing the caching layer's lazy loading and eviction
mechanisms, most parts of a segment won't be loaded into memory or disk
immediately, even if the segment is marked as LOADED. This means
physical resource usage may be very low. However, we still need to
reserve enough resources for the segments marked as LOADED. Thus, the
logic of resource usage estimation during segment loading, which based
on physcial resource usage only for now, should be changed.
To address this issue, we introduced the concept of logical resource
usage in this patch. This can be thought of as the base reserved
resource for each LOADED segment.
A segment’s logical resource usage is derived from its final evictable
and inevictable resource usage and calculated as follows:
```
SLR = SFPIER + evitable_cache_ratio * SFPER
```
it also equals to
```
SLR = (SFPIER + SFPER) - (1.0 - evitable_cache_ratio) * SFPER
```
`SLR`: The logical resource usage of a segment.
`SFPIER`: The final physical inevictable resource usage of a segment.
`SFPER`: The final physical evictable resource usage of a segment.
`evitable_cache_ratio`: The ratio of a segment's evictable resources
that can be cached locally. The higher the ratio, the more physical
memory is reserved for evictable memory.
When loading a segment, two types of resource usage are taken into
account.
First is the estimated maximum physical resource usage:
```
PPR = HPR + CPR + SMPR - SFPER
```
`PPR`: The predicted physical resource usage after the current segment
is allowed to load.
`HPR`: The physical resource usage obtained from hardware information.
`CPR`: The total physical resource usage of segments that have been
committed but not yet loaded. When one new segment is allow to load,
`CPR' = CPR + (SMR - SER)`. When one of the committed segments is
loaded, `CPR' = CPR - (SMR - SER)`.
`SMPR`: The maximum physical resource usage of the current segment.
`SFPER`: The final physical evictable resource usage of the current
segment.
Second is the estimated logical resource usage, this check is only valid
when eviction is enabled:
```
PLR = LLR + CLR + SLR
```
`PLR`: The predicted logical resource usage after the current segment is
allowed to load.
`LLR`: The total logical resource usage of all loaded segments. When a
new segment is loaded, `LLR` should be updated to `LLR' = LLR + SLR`.
`CLR`: The total logical resource usage of segments that have been
committed but not yet loaded. When one new segment is allow to load,
`CLR' = CLR + SLR`. When one of the committed segments is loaded, `CLR'
= CLR - SLR`.
`SLR`: The logical resource usage of the current segment.
Only when `PPR < PRL && PLR < PRL` (`PRL`: Physical resource limit of
the querynode), the segment is allowed to be loaded.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
#43574#43604#43431#43603
Fix wp metrics not registered bug;
Update the version dependent on wp to v0.1.2-rc1;
improve advanced reader with concurrent prefetch blks;
add the segment rolling policy based on the number of blocks;
improve concurrent compaction
release lock failed bug
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
1. Use blocking memory allocation to wait until memory becomes available
2. Perform memory allocation at the file level instead of per task
3. Limit Parquet file reader batch size to prevent excessive memory
consumption
4. Limit import buffer size from 20% to 10% of total memory
issue: https://github.com/milvus-io/milvus/issues/43387,
https://github.com/milvus-io/milvus/issues/43131
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
1. Modify the binlog reader to stop reading a fixed 4096 rows and
instead use the calculated bufferSize to avoid generating small binlogs.
2. Use a fixed bufferSize (32MB) for the Parquet reader to prevent OOM.
issue: https://github.com/milvus-io/milvus/issues/43387
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #41435
issue: https://github.com/milvus-io/milvus/issues/43038
This PR also:
1. removed ERROR state from ListNode
2. CacheSlot will do reserveMemory once for all requested cells after
updating the state to LOADING, so now we transit a cell to LOADING
before its resource reservation
3. reject resource reservation directly if size >= max_size
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #42995
- don't balance the wal if the producing-consuming lag is too long.
- don't balance if the rebalance is set as false.
- don't balance if the wal is balanced recently.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42995
- The consuming lag at streaming node will be reported to coordinator.
- The consuming lag will trigger the write limit and deny by quota
center.
- Set the ttProtection by default.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
- Introduce dynamic buffer sizing to avoid generating small binlogs
during import
- Refactor import slot calculation based on CPU and memory constraints
- Implement dynamic pool sizing for sync manager and import tasks
according to CPU core count
issue: https://github.com/milvus-io/milvus/issues/43131
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #43040
This patch introduces a disk file writer that supports Direct IO.
Currently, it is exclusively utilized during the QueryNode load process.
Below is its parameters:
1. `common.diskWriteMode`
This parameter controls the write mode of the local disk, which is used
to write temporary data downloaded from remote storage.
Currently, only QueryNode uses 'common.diskWrite*' parameters. Support
for other components will be added in the future.
The options include 'direct' and 'buffered'. The default value is
'buffered'.
2. `common.diskWriteBufferSizeKb`
Disk write buffer size in KB, only used when disk write mode is
'direct', default is 64KB.
Current valid range is [4, 65536]. If the value is not aligned to 4KB,
it will be rounded up to the nearest multiple of 4KB.
3. `common.diskWriteNumThreads`
This parameter controls the number of writer threads used for disk write
operations. The valid range is [0, hardware_concurrency].
It is designed to limit the maximum concurrency of disk write operations
to reduce the impact on disk read performance.
For example, if you want to limit the maximum concurrency of disk write
operations to 1, you can set this parameter to 1.
The default value is 0, which means the caller will perform write
operations directly without using an additional writer thread pool.
In this case, the maximum concurrency of disk write operations is
determined by the caller's thread pool size.
Both parameters can be updated during runtime.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
congqixia
sparknack
zhagnlu
wei liu
Bingyi Sun
Jean-Francois Weber-Marx
nish112022
Chun Han
Zhen Ye
junjiejiangjjj
cqy123456
cai.zhang
aoiasd
Ted Xu
sthuang
Buqian Zheng
tinswzy
yihao.dai
PjJinchen