Commit Graph

205 Commits (9a13b9822f1ecb4922e92a30c3b6e114bd3c4509)

Author SHA1 Message Date
congqixia 7b38274cce
fix: Use mutex protection when accessing pks of l0 segments (#31505)
See also #31504

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-22 14:11:06 +08:00
Chun Han c3264ca3e3
feat: support segment pruner (#31003)
related: #30376
2024-03-22 13:57:06 +08:00
Ted Xu 09281a07f4
enhance: disk cache to enable size based eviction (#31469)
See #31262

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 18:57:07 +08:00
Ted Xu c13c96e321
feat: adding cache scavenger (#31264)
See #31262

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 14:33:06 +08:00
chyezh 9f9ef8ac32
enhance: transfer resource group and dbname to querynode when load (#30936)
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-21 11:59:12 +08:00
groot c81909bfab
enhance: Support MinIO TLS connection (#31311)
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292

Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
chyezh 8e293dc1ce
enhance: add resource usage estimate for segment interface (#31050)
issue: #30931

- move resource estimate function outside from segment loader.
- add load info and collection to base segment.
- add resource usage method for sealed segment.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-19 11:53:05 +08:00
Bingyi Sun bdc70dfc6a
feat: Add global mmap enable configuration (#31267)
https://github.com/milvus-io/milvus/issues/31279

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:17:10 +08:00
Bingyi Sun 17b4938985
fix: mmap does not work on index load (#31228)
issue: https://github.com/milvus-io/milvus/issues/31101

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-17 23:27:03 +08:00
congqixia 08aba2e05f
fix: Remove `QueryNodeEntitiesSize` after segment/collection released (#31290)
See also #31289

This PR:
- Set collection level `QueryNodeEntitiesSize` to zero if all segment
released
- Delete `QueryNodeEntitiesSize` metrics value after collection ref is
zero

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-15 15:43:04 +08:00
congqixia b1ff8e2749
fix: Fix data race on loadSealedSegmentFields opts (#31230)
See also #31229

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-14 10:43:04 +08:00
Buqian Zheng 3c80083f51
feat: [Sparse Float Vector] add sparse vector support to milvus components (#30630)
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.

https://github.com/milvus-io/milvus/issues/29419

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 14:32:54 -07:00
congqixia 57a5e44b88
enhance: Add id,type filter shortcut for QueryNode segment manager (#31179)
See also #31143

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 20:15:03 +08:00
Bingyi Sun 7b7187b465
fix: fix mmap load (#31171)
issue: https://github.com/milvus-io/milvus/issues/31101

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-13 10:49:04 +08:00
congqixia 1d96239137
enhance: Add ctx trace for segment load `prepare` (#31203)
Add ctx paramter to `prepare` and add trace id in underline logs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 10:01:07 +08:00
wei liu 5016031dc7
enhance: add metrics for load segment progress (#31005)
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
zhagnlu a65a9ce8a5
fix: fix oom because of wrong load mode for index's raw data (#31130)
#30962

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-07 12:14:26 -08:00
Bingyi Sun fd17a5f050
fix: check collection lazy load prop using schema (#30992)
issue: https://github.com/milvus-io/milvus/issues/30361

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-06 16:19:01 +08:00
congqixia 30398d4b71
enhance: Fix misleading log content & possible nil panic (#31021)
- Change load field log from "dy pool" to "load pool"
- Also defer delete when there is no error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-05 16:33:00 +08:00
congqixia 52540fecb2
enhance: Add back load memory factor when esitmating memory resource (#30994)
Segment load memory usage is underestimated due to removing the load
memroy factor. This PR adds it back to protect querynode OOM during some
extreme memory cases.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-04 19:54:59 +08:00
chyezh dd957cf9e3
enhance: add configurable memory index load predict memory usage factor (#30561)
related pr: https://github.com/milvus-io/milvus/pull/30475

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-01 15:23:00 +08:00
wei liu cc46d6bafc
fix: segment version doesn't update as expected (#30951)
issue: #30950

due to segment version doesn't update as expected.
This PR will update segment version until segment become loaded

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-01 14:21:10 +08:00
congqixia 385dec3b69
fix: index attr caches wrong result variable (#30960)
See also #30757 #30756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-01 13:17:00 +08:00
congqixia 4082315bd0
enhance: Add `ParseCTraceContext` util function for tracing (#30883)
See also #29803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 18:59:00 +08:00
Cai Yudong 8a219e0102
feat: Support knowhere trace using OpenTelemetry (#30750)
Issue: #21508

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
yah01 57397b1307
enhance: add new LRU cache impl (#30360)
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
congqixia 637dcffb6b
fix: Disk resource is not requested for index loaded with disk (#30757)
See also #30756

This PR:
- Request disk resource when index type, version loaded with disk
- Add attribute cache for index utility
- Add `typeutil.Pair`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-27 08:50:56 +08:00
congqixia 90e802e441
enhance: Add unit test for level-zero segment deltalog request resource (#30718)
See also #30670 #30704

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-23 10:00:54 +08:00
congqixia 1346b57433
enhance: Add deltalog expansion rate in segment loader (#30704)
See also #30191

It turned out that in auto id and batch delete scenario actual memory
size of deltalog maybe way larger than deltalog file size. This PR add a
configurable expansion rate for deltalog memory usage to prevent
out-of-memory panicking during loading deltalogs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-21 11:26:52 +08:00
congqixia e5a16050ce
fix: Update disk usage metrics after segment released (#30702)
See also #30701

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-20 22:14:53 +08:00
yah01 b74673c147
enhance: calculate the accuracy memory usage while loading segment (#30473)
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.

Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-20 14:52:51 +08:00
congqixia ed754dc58c
enhance: Check Level-zero segment memory usage as well (#30670)
Related to #30191

When loading segment, segment loader shall check memory usage for
current loading task. Previously l0 segment was ignored but level zero
segment may actually cost lots of memory.

This PR adds back memory resource check for Level zero segment loading.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-20 14:34:51 +08:00
congqixia 5143b8c793
enhance: Use dynamic pool for `NewLoadIndexInfo` (#30489)
See also #30445

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-04 14:19:05 +08:00
yah01 878c4c9463
enhance: limit the max pool size to 16 (#30371)
according to our benchmark, concurrency level 16 is enough to fully
utilize the object storage network bandwidth

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-31 14:13:06 +08:00
PowderLi 8fc4ebfa11
fix: empty MetricType (#30216)
issue: #30102 #30225

we should read MetricType from SearchResult,
because query node never
1. read metricType from LoadMeta
2. store to collection
3. set SearchRequest.MetricType

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 15:33:02 +08:00
xige-16 e9fdd2475d
fix: fix searchPlan metricType modified concurrently (#30227)
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-26 14:03:09 +08:00
congqixia 405877c8cd
fix: Use correct pools for all CGO methods in segments pkg (#30274)
See also #30273

This PR:
- Rename confusing `LoadIndexInfo` to `UpdateIndexInfo` for LocalSegment
- Use `DynamicPool` instead of `LoadPool` for `UpdateSealedSegmentIndex`
- Fix cgo call missing pool control

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-26 10:09:00 +08:00
yihao.dai c02fb64ad6
enhance: Allows proactive warming up of chunk cache (#30182)
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.

issue: https://github.com/milvus-io/milvus/issues/30181

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-25 19:55:39 +08:00
SimFG aa7014a360
enhance: move the cgo code in the pkg dir to interal dir (#30261)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-25 15:15:01 +08:00
smellthemoon e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873)
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
yah01 9a3837212c
enhance: add index after load succeeded (#30015)
this avoids a corner case: after load index failed, this index can be
never loaded as it has been added into the segment's index map

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 15:06:53 +08:00
yah01 8f083e45cb
enhance: enable converting segcore error to merr (#29914)
this converts the segcore error to merr if possible

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 11:28:53 +08:00
chyezh d300bc7bcb
fix: querynode num entity metric is broken by illegal label (#29948)
issue: #29766
also see pr: #29825

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:23:00 +08:00
Bingyi Sun e1258b8cad
feat: integrate storagev2 into loading segment (#29336)
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
jaime 90984a2cab
enhance: Support read hardware metrics for cgroupv2 (#29850)
issue: https://github.com/milvus-io/milvus/issues/29846

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-12 10:30:51 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427)
issue:https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
yah01 26e900180e
fix: the insert count is zero after set the pointer to nil (#29870)
this leads to the EntitiesNum metric would be never reduced

fix: #29766

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 14:56:50 +08:00
yah01 44fe06f198
enhance: skip loading duplicated index (#29715)
this protect the loading index from failure, and speed up the loading
progress

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 11:52:49 +08:00
congqixia d6429933a7
enhance: make Load process traceable in querynode & segcore (#29858)
See also #29803

This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 21:58:51 +08:00
yah01 d357139064
fix: the entities num metric may be contributed more than once (#29767)
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions

fix: #29766

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-10 10:00:51 +08:00