cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
3e5f349c9e
fix: Segments return both growing&sealed result ( #39787 )
...
Previous PR #38311
Logic bug found in #38348
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-13 10:10:55 +08:00
f0b7446e6b
enhance: Remove unnecessary collection and partition label from the metrics ( #39536 )
...
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-05 11:01:10 +08:00
f5234c3c11
enhance: Remove mutex from SegmentManger in querynode ( #39051 )
...
Remove mutex from SegmentManger in querynode to prevent mutex
contention.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 12:45:00 +08:00
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
7c55649585
enhance: Refine querynode collection number metrics ( #38350 )
...
Related to #37630
Previously the loaded collection metrics was calculated via scanning all
loaded segment in segment manager, which is slow and buggy
implementation.
This PR:
- Move collection num metrics to collection manager
- Remove deprecated loaded partition metrics update logic
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-11 11:36:43 +08:00
b8e3795374
enhance: Add secondary index for querynode segment manager ( #38311 )
...
Related to #37630
Add secondary index with vchannel to reduce `GetBy` rlock holding time
when segment number is large.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-10 18:18:42 +08:00
f43527ef6f
enhance: Batch forward delete when using DirectForward ( #37076 )
...
Relatedt #36887
DirectFoward streaming delete will cause memory usage explode if the
segments number was large. This PR add batching delete API and using it
for direct forward implementation.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-24 10:39:28 +08:00
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
d60e628aed
enhance: Avoid use concrete segment type in segments interfaces ( #34521 )
...
See also #34519
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 10:18:12 +08:00
9b69601dfb
fix: load operation when segment is on releasing ( #31340 )
...
issue: #30857
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-14 15:35:56 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
370562b4ec
fix: fix partition loaded num metric ( #33316 )
...
issue: https://github.com/milvus-io/milvus/issues/32108
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-24 15:31:42 +08:00
1c84a1c9b6
fix: lru related issue fixup patch ( #32916 )
...
issue: #32206 , #32801
- search failure with some assertion, segment not loaded and resource
insufficient.
- segment leak when query segments
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-10 19:17:30 +08:00
40728ce83d
enhance: Add `metautil.Channel` to convert string compare to int ( #32749 )
...
See also #32748
This PR:
- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 19:13:35 +08:00
641f702f64
fix: add request resource timeout for lazy load, refactor context usage in cache ( #32709 )
...
issue: #32663
- Use new param to control request resource timeout for lazy load.
- Remove the timeout parameter of `Do`, remove `DoWait`. use `context`
to control the timeout.
- Use `VersionedNotifier` to avoid notify event lost and broadcast,
remove the redundant goroutine in cache.
related dev pr: #32684
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-07 16:33:30 +08:00
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
73adf2a5cc
fix: use stateful lock to avoid load and release on LocalSegment concurrently ( #31606 )
...
issue: #31605
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 17:09:16 +08:00
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug ( #31977 )
...
issue: #30361
- Add configurable disk capacity limit
- fix bitset reset logic
- make insert record reinsert after clear
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
8e661f791a
fix: lazy load index data in cache ( #31094 )
...
issue: https://github.com/milvus-io/milvus/issues/31571
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:43:07 +08:00
09281a07f4
enhance: disk cache to enable size based eviction ( #31469 )
...
See #31262
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 18:57:07 +08:00
c13c96e321
feat: adding cache scavenger ( #31264 )
...
See #31262
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 14:33:06 +08:00
57a5e44b88
enhance: Add id,type filter shortcut for QueryNode segment manager ( #31179 )
...
See also #31143
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 20:15:03 +08:00
7b7187b465
fix: fix mmap load ( #31171 )
...
issue: https://github.com/milvus-io/milvus/issues/31101
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-13 10:49:04 +08:00
57397b1307
enhance: add new LRU cache impl ( #30360 )
...
- remove the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache
related #30361
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
d357139064
fix: the entities num metric may be contributed more than once ( #29767 )
...
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions
fix : #29766
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-10 10:00:51 +08:00
a0cec4047a
fix: make the entity num metric accurate ( #29643 )
...
fix #29642
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-05 18:24:47 +08:00
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition ( #28456 )
...
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
13a5b9f64a
fix: query l0 segment bugs ( #28558 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-20 17:26:23 +08:00
ece592a42f
Deliver L0 segments delete records ( #27722 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
50672f0d60
Skip empty segment while query/search ( #27711 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-15 16:10:04 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
c6116d1819
Remove segment to LocalSegment type assertion ( #26931 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-09 10:35:16 +08:00
95dcf7fa06
Fix memory leak when Put duplicated segments ( #26693 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-30 11:37:00 +08:00
1cf6e00fa6
Improve segment manager interface ( #26637 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 15:46:27 +08:00
7b3a1dc93f
Fix load index panic ( #26457 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-21 09:56:20 +08:00
3b11ad2695
fix dead lock in concurrent search/release segment ( #26435 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 15:28:18 +08:00
5dae6a654a
Protect segment from being released while query/search ( #26322 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 18:38:17 +08:00
b47a72bfcf
fix set dirty segment distribution to leader view ( #26180 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation ( #25812 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
d216f9abda
Clear collection meta after all channels/segments released ( #25486 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
7b5013664d
Support graceful stop ( #25226 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-30 17:28:23 +08:00
400364483d
Improve predict the resource usage ( #23968 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-11 15:33:24 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
congqixia
yihao.dai
Zhen Ye
wayblink
Bingyi Sun
Ted Xu
yah01
yah01
aoiasd
SimFG
Xiaofan
wei liu
jaime