c340f387cf
enhance: [cherry-pick] Change the fixed value to a ratio for clustering segment size ( #35075 )
...
issue: #34495
master pr: #35076
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-31 10:32:00 +08:00
935a117396
enhance: [2.4] Support otlp http exporter ( #35053 ) ( #35073 )
...
Cherry-pick from master
pr: #35053
See also #35052
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 21:00:08 +08:00
86edca8c1b
fix: support auto index for array ( #35095 )
...
/kind branch-feature
pr: #34450
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: Zhagnlu <lu.zhang@zilliz.com>
2024-07-30 17:57:50 +08:00
b3bc7f3985
enhance: Limit collection's normal balance speed ( #34810 ) ( #34987 )
...
issue: #34798
pr: #34810
after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 10:13:46 +08:00
77ae127a62
fix: check collection health(queryable) fail for releasing collection ( #34948 )
...
issue: #34946
pr: #34947
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-25 10:25:57 +08:00
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time ( #34950 )
...
issue: #34703
master pr: #34949
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
7540714c1b
enhance: [2.4] Add l0 segment entry num quota ( #34733 ) ( #34837 )
...
Cherry-pick from master
pr: #34733
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-22 12:05:43 +08:00
c0c3c5f528
enhance: [cherry-pick] refine clustering compaction configs and logs ( #34818 )
...
issue: #30633
pr: #34784
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:27:41 +08:00
07bc1b6717
enhance: Seal by total growing segments size ( #34692 ) ( #34779 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
pr: https://github.com/milvus-io/milvus/pull/34692
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 18:25:50 +08:00
0e226502e4
enhance: [2.4] pick default root password and log level pr ( #34777 )
...
default root password
- issue: #33058
- pr: #34752
set log level
- issue: #34756
- pr: #34757
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-18 13:45:43 +08:00
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic ( #34625 )
...
issue: #34544
pr: #34581
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 09:55:43 +08:00
83fc26c31a
fix: [cherry-pick] compaction task not be cleaned correctly ( #34766 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
pr: #34765
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 22:17:42 +08:00
cf701a9bf0
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34600 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
10c04f33c7
enhance: [2.4] Add param item for segmentFlushInterval ( #34629 ) ( #34663 )
...
Cherry-pick from master
pr: #34629
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:31:40 +08:00
a60e2a65ff
enhance: change autoindex default metric type ( #34277 )
...
issue: #34304
pr: #34261
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 10:52:14 +08:00
d3e94f9861
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl ( #34377 )
...
issue: #32995
pr: #33405
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.
WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.
In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.
Block BF construct time {"time": "54.128131ms"}
Block BF size {"size": 3021578}
Block BF Test cost {"time": "55.407352ms"}
Basic BF construct time {"time": "210.262183ms"}
Basic BF size {"size": 2396308}
Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.
Block BF TestLocation cost {"time": "529.97183ms"}
Basic BF TestLocation cost {"time": "3.197430181s"}
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-05 17:04:10 +08:00
0f1915ef24
fix: DataNode might OOM by estimating based on MemorySize ( #34203 )
...
See also: #34136
pr: #34201
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-04 15:24:10 +08:00
a1a0a56f86
enhance: async search and retrieve in cgo ( #34200 )
...
issue: #33132
pr: #33133
other pr: #33228 , #34084 , #33946
- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 13:02:09 +08:00
bc1746f96c
enhance: [cherry-pick] Optimize clustering compaction ( #34313 ) ( #34398 )
...
issue: #30633
master pr: #34313
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-04 09:52:09 +08:00
7000cec365
enhance: [Cherry-pick] Merge query stream result for reduce delete task ( #32855 ) ( #34281 )
...
relate: https://github.com/milvus-io/milvus/issues/32854
pr: https://github.com/milvus-io/milvus/pull/32855
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-03 18:08:09 +08:00
945f0106f6
fix: [2.4] Use raw parameter value to perform CAS ( #34373 )
...
Cherry-pick from master
pr: #34343
See also #34342
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-03 17:20:10 +08:00
18668aaace
enhance:[Cherry-Pick] change access log write cache default config ( #34352 )
...
pr: https://github.com/milvus-io/milvus/pull/34351
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-03 15:10:09 +08:00
88550081c3
enhance: [skip e2e][2.4] update the version of MV ( #34380 )
...
issue: #29892
master PR: https://github.com/milvus-io/milvus/pull/34378
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 92f3f4521a
2024-07-03 14:28:16 +08:00
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations ( #34360 )
...
This PR cherry-picks the following commits:
- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232
issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232
---------
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-03 09:53:37 +08:00
ad545b6fa6
enhance: refine misleading param name for bloom filter parallel factor ( #34335 )
...
pr: #34334
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-02 19:24:09 +08:00
bc8ca593bd
enhance: [Cherry-Pick] Update access log ( #34295 )
...
Support restful api,print with consistency level and add new write
cache.
relate: https://github.com/milvus-io/milvus/issues/31823
pr: https://github.com/milvus-io/milvus/pull/33155
https://github.com/milvus-io/milvus/pull/33503
https://github.com/milvus-io/milvus/pull/32213
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-02 15:30:15 +08:00
c924b0b502
enhance: [cherry-pick] Refine index code and support analyze data ( #34311 )
...
This PR primary picks up the support analyzing functionality, including
the following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33651
- refine indexnode code: https://github.com/milvus-io/milvus/pull/33458
- related fixes:
- https://github.com/milvus-io/milvus/pull/33832
- https://github.com/milvus-io/milvus/pull/33161
issue: #30633
master prs: #33651 , #33458 , #33832 , #33161
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-02 09:50:39 +08:00
14a11e379c
enhance: Refactor Compaction to enable persistence( #33265 ) ( #34268 )
...
pr : #33265
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-07-01 19:32:07 +08:00
4ebdccd160
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 ( #34185 )
...
See also: #31933
pr: #34184
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-01 18:22:08 +08:00
0992f10694
enhance: improve check health ( #34265 )
...
issue: https://github.com/milvus-io/milvus/issues/34264
pr: #33800
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:18:07 +08:00
85aa929749
enhance: Set the timeout for SyncSegments based on the number of segments ( #34228 )
...
issue: #32809
master pr: #34226
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-28 14:18:06 +08:00
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher ( #33420 ) ( #34186 )
...
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
- https://github.com/milvus-io/milvus/pull/33664
- https://github.com/milvus-io/milvus/pull/33829
- https://github.com/milvus-io/milvus/pull/34056
- https://github.com/milvus-io/milvus/pull/34156
issue: #32809
master pr: #33420 , #33664 , #33829 , #34056 , #34156
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-27 11:24:05 +08:00
07a05db51b
fix: [2.4] Wrap init segcore tracing with golang timeout ( #33494 ) ( #34191 )
...
Cherry-pick from master
pr: #33494
See also #33483
Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 19:44:05 +08:00
3e51256fb4
enhance: [cherry-pick]update new mmap config parmeters version ( #34145 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34143
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-25 21:24:05 +08:00
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard ( #34157 )
...
This PR cherry-picks the following commits:
- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932
issue: https://github.com/milvus-io/milvus/issues/32809
pr: https://github.com/milvus-io/milvus/pull/32881 ,
https://github.com/milvus-io/milvus/pull/32913 ,
https://github.com/milvus-io/milvus/pull/33138 ,
https://github.com/milvus-io/milvus/pull/33858 ,
https://github.com/milvus-io/milvus/pull/33932
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 20:22:03 +08:00
f5344abdaf
enhance: [cherry-pick]growing segment support mmap ( #34110 )
...
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633 ,
https://github.com/milvus-io/milvus/pull/33951 ,
https://github.com/milvus-io/milvus/pull/33993
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-25 14:52:07 +08:00
60695bdb44
enhance: [2.4] Set maxPartitionNum default value to 1024 ( #33950 )
...
Cherry-pick from master
pr: #33949
See also #30059
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 18:42:00 +08:00
5fc1370f6f
enhance: [2.4] autoindex for multi data type ( #33867 )
...
issue: #22837
pr: https://github.com/milvus-io/milvus/pull/33868
- opensource autoindex support
- metric type check for different data types
- autoindex data type for search param
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-14 23:26:00 +08:00
fd1c7b1a1c
enhance: enable flush rate limiter of collection level ( #33864 )
...
pr: #33837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-14 16:27:57 +08:00
62bd51ec6d
fix: [2.4] Compare config value then swap when caching param value ( #33785 ) ( #33797 )
...
Cherry-pick from master
pr: #33785
See also #33784
This PR change the behavior of `SetCacheValue` of config manager:
- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value
With this implementation, concurrent caching & eviction shall always
have current output:
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 |get | |old value|null |
|t1 |CAS OK | |old value|old value|
|t2 | |update |new value|old value|
|t3 | |eviction|new value|null |
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 |get | |old value|null |
|t1 | |update |new value|null |
|t2 |CAS fail| |old value|null |
|t3 | |eviction|new value|null |
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 | |update |new value|null |
|t1 |get | |new value|null |
|t2 |CAS OK | |new value|new value|
|t3 | |eviction|new value|null |
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 | |update |new value|null |
|t1 |get | |new value|null |
|t2 | |eviction|new value|null |
|t3 |CAS OK | |new value|new value|
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 | |update |new value|null |
|t1 | |eviction|new value|null |
|t2 |get | |new value|null |
|t3 |CAS OK | |new value|new value|
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-13 17:51:57 +08:00
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33793 )
...
issue: #33610
pr: #33792
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 14:14:04 +08:00
54feef30e7
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33752 )
...
issue: #33610
pr: #33611
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-12 17:45:58 +08:00
f664b51ebe
enhance: [2.4] try to speed up the loading of small collections ( #33746 )
...
- issue: #33569
- pr: #33570
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-11 15:07:55 +08:00
26db471968
enhance: [2.4]Add consts of MsgDispatcher to configs ( #33680 )
...
See also: #33676
pr: #33679
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-07 14:21:58 +08:00
58a7111599
enhance: [cherry-pick] add autoindex mapping for binary/sparse datatype ( #33625 )
...
issue: #22837
pr: #33624
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-06-06 10:33:52 +08:00
b9b76ee9a4
fix: [2.4]Use localStorage path to check disk cap for indexnode ( #33450 ) ( #33505 )
...
Cherry-pick from master
pr: #33450
See also #30943 #30944
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-03 10:21:47 +08:00
7384bfe3f8
fix: use seperate warmup pool and disable warmup by default ( #33348 ) ( #33349 )
...
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool
issue: https://github.com/milvus-io/milvus/issues/32772
pr: https://github.com/milvus-io/milvus/pull/33348
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-28 19:27:43 +08:00
36172492f7
enhance: Decrease bloom filter fp rate to reduce delete impact ( #33301 ) ( #33329 )
...
pr: #33301
when milvus process delete record, it need to find record's corresponded
segment by bloom filter, and higher bloom filter fp rate will cause
delete record forwards to wrong segments.
This PR Decrease bloom filter's default fp to 0.001.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-24 10:27:41 +08:00
d6bc95de55
enhance: [2.4] Add param item to ignore bad message id in checkpoint ( #33123 ) ( #33249 )
...
Cherry-pick from master
pr: #33123 #33158
See also #33122
This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-23 15:27:41 +08:00
2a38f5b598
enhance: [2.4] add config to control whether to init public role permissions ( #33174 )
...
issue: #33164
pr: #33165
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-21 22:39:46 +08:00
cai.zhang
congqixia
Jiquan Long
wei liu
jaime
wayblink
yihao.dai
SimFG
Gao
XuanYang-cn
chyezh
aoiasd
Patrick Weizhi Xu
zhenshan.cao
cqy123456
foxspy