976b6fc0e4
enhance: change opendal as compile configurable ( #30384 )
...
#30373
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-20 19:16:52 +08:00
a27c0e86fd
enhance: reduce many I/O operations while loading disk index ( #30189 )
...
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file
this optimized this to open only once and continiously write all data.
This also makes it concurrent to load the files from object storage
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-25 15:23:02 +08:00
0907d76253
enhance: pass partition key scalar info if enabled when build vector index ( #29931 )
...
issue: #29892
Pass optional scalar IVF offsets to Cardinal
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
a77693aa19
enhance: convert the `GetObject` util to async ( #30166 )
...
This makes it much easier to use
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 19:20:57 +08:00
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
aef483806d
enhance: improve the segcore logs ( #29372 )
...
- remove the streaming logging
- refine existing logs
fix #29366
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-23 21:52:43 +08:00
36f69ea031
feat: integrate storagev2 in building index of segcore ( #28768 )
...
issue: https://github.com/milvus-io/milvus/issues/28655
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 16:48:54 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only ( #27017 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-14 14:05:20 +08:00
5634ba777d
add new threadpool with various priority to avoid deadlock( #25781 ) ( #26028 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 09:31:07 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
247f117096
Refine segcore code style ( #24999 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-20 14:16:46 +08:00
c80d8f82ce
Reduce memory usage for diskann index ( #23600 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-05-06 10:32:39 +08:00
ef57ba3bac
Remove easylogging and replace by GLog ( #22979 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-27 18:22:02 +08:00
9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData ( #22726 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-03-20 10:41:56 +08:00
bdd6bc7695
Re-format cpp code ( #22513 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-02 15:55:49 +08:00
5cae49e28d
Store index files in slices to reduce memory peaks ( #22369 ) ( #22455 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-03-01 19:43:47 +08:00
15badd0263
Fix cache disk index error ( #20419 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 17:45:23 +08:00
cb2591d1fe
Parallel operations of disk-ann file manager ( #20120 ) ( #20122 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-03 14:39:40 +08:00
02a8a07322
Fix BucketExists bug in minio chunk manager ( #19548 ) ( #19854 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-10-20 10:23:27 +08:00
158787811e
Move assemble/disassemble func to core ( #19420 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-16 21:17:25 +08:00
8c9c1672ae
Assign different storage config for indexes ( #19517 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
49daee2e17
Fix lost disk index data ( #19636 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-09 14:36:58 +08:00
428840178c
Support diskann index for vector field ( #19093 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
zhagnlu
yah01
Patrick Weizhi Xu
Bingyi Sun
yah01
foxspy
Enwei Jiao
MrPresent-Han
xige-16
cai.zhang