4bd426dbe7
fix: Fix minio latency monitoring for get operation ( #28510 )
...
see also: https://github.com/milvus-io/milvus/issues/28509
Currently Minio latency monitoring for get operation only collects the
duration of getting object (which just returns an io.Reader and does not
really read from minio), this pr will correct this behavior.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-28 10:00:27 +08:00
9658367a3c
Refine chunk manager errors ( #27590 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 12:18:15 +08:00
6c3f29d003
Identify service providers based on addresses ( #27907 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-10-25 17:28:10 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
4feb3fa7c6
support azure ( #26398 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager ( #26194 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
33c2012675
Add more metrics ( #25081 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
9288020da3
Fix bug for batch delete files on gcp of minio ( #23052 ) ( #23083 ) ( #23090 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-09 16:22:30 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
32581e6452
Support aliyun oss as object storage with ak or IAM ( #22376 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-03-09 16:33:52 +08:00
90a5aa6265
Refine errors, re-define error codes ( #22501 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-09 15:47:52 +08:00
ea4ab9e4a4
Replace ErrorList with multiErrors ( #22532 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-03 14:59:49 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
5936723904
Refine Read method of MinioChunkManager ( #22235 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-21 16:22:26 +08:00
6004867853
Add bucket name in error message ( #21887 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-10 17:44:31 +08:00
908023a06e
Add more test on file reader ( #21082 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-09 11:55:19 +08:00
eb7ef01b9a
Fix bug for ListWithPrefix from minio with recursive is false ( #21059 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-12-08 15:47:18 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
766957e75a
Recuce memory consumption in MinioChunkManager ( #20284 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-03 12:19:35 +08:00
9fac1476f3
Add support for GCS(GoogleCloudStorage) with IAM ( #20164 )
...
go mod tidy
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-11-01 11:07:35 +08:00
89c9cb3680
Fix scan may break GC limitation ( #19670 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-15 15:13:24 +08:00
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
928a213e31
Fix Chunk Manager Path mis use ( #19141 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
ce9ba0dd02
Remove debug log in minio chunk manager ( #19134 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-09 17:32:37 +08:00
db23a256c2
Make minioChunkManager ListObject level by level when recursive is true ( #19096 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-08 14:58:34 +08:00
c91bb0b015
Unify rootPath in configs and ChunkManager ( #18808 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-25 19:32:53 +08:00
d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection ( #18092 )
...
Signed-off-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
2022-07-22 22:10:28 +08:00
0ad15a22c8
IndexCoord handle events correctly ( #17878 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-07-07 14:44:21 +08:00
04f836f516
Fix s3 delete file ( #17574 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-16 18:14:10 +08:00
76eaa3fc50
Support access s3 through AWS IAM role ( #17292 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-02 19:42:03 +08:00
5355153805
Minio Error is not Handled gracefully ( #17003 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-16 19:25:55 +08:00
2be46a01c0
Reduce the number of retries and add error log ( #16754 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-05-05 09:31:51 +08:00
4ef2df8cb9
Fix an issue where query coord calls MinioChunkManager during init ( #16629 )
...
It causes Milvus not runnable with local storage.
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-25 11:11:46 +08:00
4781db8a2a
Add datanode import ( #16414 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-12 22:19:34 +08:00
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
478f6ca11e
Add cache for vector chunk ( #15912 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-26 22:05:26 +08:00
fcf0887d92
Add and implement chunk manager interface. ( #15541 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-24 23:53:53 +08:00
effd4cc461
[skip e2e]Update license for storage miniochunk ( #14193 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-24 17:32:20 +08:00
0aa8e65501
[skip ci] Fix grammar mistake in minio_chunk_manager ( #13417 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
2021-12-15 11:53:47 +08:00
31df0faaf5
[skip ci]Add chunk manager comment ( #7833 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-09-17 14:49:50 +08:00
8cc24e09c9
Make users get vectors without using local storage ( #7134 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-08-18 16:30:11 +08:00
db94d7771f
Read vector from disk ( #6707 )
...
* Read vector from disk
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* go fmt
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix git action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix test error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix caculate error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change var name
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix len error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused code
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change bytes to float method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change float to bytes method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-07-24 09:25:22 +08:00
b9f5949680
Add vector chunk manager ( #6613 )
...
* add file manager
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* add file manager
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* add vector file manager
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* rename
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* go fmt
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix ut error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix memory error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-07-20 15:06:09 +08:00
yihao.dai
yah01
zhenshan.cao
SimFG
PowderLi
zhagnlu
xige-16
cai.zhang
jaime
shaoyue
yah01
congqixia
Enwei Jiao
Xiaofan
Ten Thousand Leaves
xing.zhao
godchen
XuanYang-cn
jaime