c3264ca3e3
feat: support segment pruner ( #31003 )
...
related: #30376
2024-03-22 13:57:06 +08:00
9f9ef8ac32
enhance: transfer resource group and dbname to querynode when load ( #30936 )
...
issue: #30931
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-21 11:59:12 +08:00
3c80083f51
feat: [Sparse Float Vector] add sparse vector support to milvus components ( #30630 )
...
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.
https://github.com/milvus-io/milvus/issues/29419
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 14:32:54 -07:00
1d96239137
enhance: Add ctx trace for segment load `prepare` ( #31203 )
...
Add ctx paramter to `prepare` and add trace id in underline logs.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 10:01:07 +08:00
cc46d6bafc
fix: segment version doesn't update as expected ( #30951 )
...
issue: #30950
due to segment version doesn't update as expected.
This PR will update segment version until segment become loaded
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-01 14:21:10 +08:00
57397b1307
enhance: add new LRU cache impl ( #30360 )
...
- remove the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache
related #30361
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
90e802e441
enhance: Add unit test for level-zero segment deltalog request resource ( #30718 )
...
See also #30670 #30704
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-23 10:00:54 +08:00
ed754dc58c
enhance: Check Level-zero segment memory usage as well ( #30670 )
...
Related to #30191
When loading segment, segment loader shall check memory usage for
current loading task. Previously l0 segment was ignored but level zero
segment may actually cost lots of memory.
This PR adds back memory resource check for Level zero segment loading.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-20 14:34:51 +08:00
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
13aa174b8a
enhance: add log when release segment created for load failure ( #29464 )
...
Add log for releasing segment created during load process when load
error happens
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-26 00:46:46 +08:00
dc89730a50
Support collection-level mmap control ( #26901 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
0a750408d0
Skip delta logs have been applied ( #26971 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-19 16:21:23 +08:00
2a5d574a0d
Fix querynodev2 concurrent load logic ( #26959 )
...
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done
Make delegator control concurrency for each same segment
Related to #26908
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 07:41:18 +08:00
c8f9f22c4a
Fix segment loader return false success ( #26926 )
...
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded
This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-08 16:41:16 +08:00
b47a72bfcf
fix set dirty segment distribution to leader view ( #26180 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
889424b3f9
Fix load index with empty file list ( #26236 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-09 18:39:16 +08:00
7fec0d61cc
Refine test chunkmanager of querynode v2 ( #25888 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-08-01 10:33:04 +08:00
3735a097e6
Refine the memeory usage prediction ( #25889 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-25 17:41:01 +08:00
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE ( #25708 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
d216f9abda
Clear collection meta after all channels/segments released ( #25486 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
400364483d
Improve predict the resource usage ( #23968 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-11 15:33:24 +08:00
884f7904a1
Re-calculate segment entry num when binlog field is zero ( #23883 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-06 14:24:39 +08:00
6f4ed517de
add growing segment index ( #23615 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
967a97b9bd
Support json & array types ( #23408 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
dc0eb433ae
Add IoPoolSize config ( #23360 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-13 14:50:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
826c1fb850
Support mmap on QueryNodeV2 ( #23073 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-28 21:30:05 +08:00
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
Chun Han
chyezh
Buqian Zheng
congqixia
wei liu
yah01
Bingyi Sun
yah01
SimFG
aoiasd
Cai Yudong
xige-16
foxspy
Enwei Jiao
jaime