milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
17,402 Commits
127 Branches
113 Tags
553 MiB
Commit Graph

353 Commits (d3eacb2563e58ab320701959100e0ceb8f063489)

Author SHA1 Message Date
Xiaofan 928a213e31
Fix Chunk Manager Path mis use (#19141) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-23 14:40:51 +08:00
congqixia f24e16a66f
Handle multiple parquet RowGroup/Column (#19283) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-21 09:12:50 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
groot b161aec95e
Support input empty string (#19111) (#19144) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-09-13 13:36:29 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00
congqixia ce9ba0dd02
Remove debug log in minio chunk manager (#19134) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-09 17:32:37 +08:00
congqixia db23a256c2
Make minioChunkManager ListObject level by level when recursive is true (#19096) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-08 14:58:34 +08:00
congqixia 68a6587374
Set insert&stats binlog timestamp range (#19005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-04 09:05:09 +08:00
congqixia c91bb0b015
Unify rootPath in configs and ChunkManager (#18808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-25 19:32:53 +08:00
codeman 70254c3ec5
Unified catalog interface for segment (#18289) (#18290) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-20 10:24:51 +08:00
xige-16 e40061b864
Update binlog event format (#18347) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-11 14:06:38 +08:00
jaime efa5dfaa7b
Fix Parse binlog path failure (#18583) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-10 17:04:37 +08:00
jaime 6e7831470c
Revert settings of Pulsar address and refine config code (#18494) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-02 21:26:33 +08:00
jaime c9174d55ba
Refine merge operation during compacting phase (#18399) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-28 14:52:31 +08:00
congqixia e9d777b336
Fix ParseSegmentIDBinlog panicks with bad input (#18413) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-26 19:32:30 +08:00
xing.zhao d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-07-22 22:10:28 +08:00
Xiaofan 6d82ef8c20
fix garbage collector err handling (#18277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-15 13:54:26 +08:00
cai.zhang 0ad15a22c8
IndexCoord handle events correctly (#17878) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 14:44:21 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
congqixia 37464a281d
Fix datasorter swap vector logic (#17661) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-21 11:38:12 +08:00
shaoyue 04f836f516
Fix s3 delete file (#17574) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-16 18:14:10 +08:00
yah01 70f8bea4b4
Avoid growing slice as deserializing binlogs (#17421) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-08 11:46:06 +08:00
shaoyue 76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
Enwei Jiao d28a2db46c
move arrow from storage to core (#17061) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-22 20:03:58 +08:00
Xiaofan 5355153805
Minio Error is not Handled gracefully (#17003) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:25:55 +08:00
Letian Jiang 72bbe40254
Make PayloadReader read column data in batch (#16826) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-10 11:37:52 +08:00
cai.zhang 2be46a01c0
Reduce the number of retries and add error log (#16754) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-05-05 09:31:51 +08:00
Letian Jiang 5b2b917987
Benchmark go/cgo parquet payload readers (#16736) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-29 15:59:47 +08:00
xige-16 515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
jaime 68b1b82faf
Remove DataKV interface (#16692) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-28 21:03:47 +08:00
yah01 7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment (#16514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-25 15:57:47 +08:00
Ten Thousand Leaves 4ef2df8cb9
Fix an issue where query coord calls MinioChunkManager during init (#16629) 
It causes Milvus not runnable with local storage.

issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:11:46 +08:00
congqixia 9769426e84
Use fmt.Errorf instead of string concat in local_chunk_manager.go (#16442) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-19 18:39:39 +08:00
Ten Thousand Leaves e76a8c5ec2
Update compile options and config for embedded Milvus (#16472) 
/kind enhancement

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-14 19:57:34 +08:00
godchen 4781db8a2a
Add datanode import (#16414) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-12 22:19:34 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
Xiaofan 801eeffbcc
Replace cgo parquet reader to go parquet reader (#16199) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-30 15:21:28 +08:00
Xiaofan b6b3c9863c
Fix incompatibility of stats log (#16277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-30 10:15:28 +08:00
godchen 478f6ca11e
Add cache for vector chunk (#15912) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-26 22:05:26 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
yah01 d4225117ee
Enable Zstd compression for binlog (#15840) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-14 16:00:01 +08:00
Jiquan Long 3121619758
Chunk manager support scalar data (#16010) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-11 14:39:59 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong 92c8e32ebd
Let MemoryKV.Load return error when key not exist (#15814) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-02 18:51:55 +08:00
Cai Yudong 503724be19
Optimize CMakeLists.txt under internal/core (#15770) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-01 10:31:55 +08:00
godchen 2e4391e6c5
Add lru cache (#15733) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-28 21:43:55 +08:00
Cai Yudong adca79facc
Fix centos build error (#15760) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-25 18:49:54 +08:00
godchen fcf0887d92
Add and implement chunk manager interface. (#15541) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-24 23:53:53 +08:00
jaime 307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
Ji Bin d83f69bb21
Support windows compliation (#15448) 
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS

issue: #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-02-07 23:43:46 +08:00
congqixia 4369e08f2a
Fix storage memory leak caused by runtime.SetFinalizer (#15100) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-10 17:13:35 +08:00
congqixia 998ebcce4e
Fix file not close when ReadAll returns error in local chunk manager (#15068) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-10 11:53:34 +08:00
godchen 384ceea223
Close payload writer when error occurs (#15013) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-07 18:27:23 +08:00
congqixia b22e945513
[skip e2e] Add comment for NewInsertCodec (#14907) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-06 18:04:03 +08:00
XuanYang-cn befb5d77e7
[skip e2e]Update license for cwrapper test (#14785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-06 09:37:30 +08:00
shaoyue c5e3efa874
[skip e2e] Add comment for payload (#14803) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-01-05 13:53:35 +08:00
shaoyue 756904dd3e
[skip e2e] Add coment for payloadwriter (#14724) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-01-04 10:49:27 +08:00
XuanYang-cn 99fd72cfb6
[skip e2e]Update license for cwrapper payload (#14626) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 09:11:39 +08:00
XuanYang-cn 3afe467d86
[skip e2e]Update license for cwrapper parquet (#14625) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 09:09:22 +08:00
XuanYang-cn c7c147b26c
[skip e2e]Update license for cwrapper cmake (#14627) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-30 20:05:26 +08:00
shaoyue f9dc87ba6c
[skip e2e] Add comment for data_codec (#14571) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-30 10:44:05 +08:00
XuanYang-cn 7bfde0aa73
[skip e2e]Update license for storage vector chunk m (#14540) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-29 22:42:14 +08:00
Xiaofan 029b153259
Remove data directory for test (#14363) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 10:08:47 +08:00
XuanYang-cn dd860a76cf
[skip e2e]Update license for storage util (#14453) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 20:11:55 +08:00
XuanYang-cn 758dc8b70f
[skip e2e]Update license for storage unsafe (#14452) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 20:03:56 +08:00
XuanYang-cn 647307ea64
[skip e2e]Update license for storage types (#14451) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 19:57:10 +08:00
XuanYang-cn 9b335e74a2
[skip e2e]Update license for storage stats (#14349) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 09:44:06 +08:00
XuanYang-cn e4ae7a6bef
[skip e2e]Update license for storage print binlog (#14348) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 09:43:49 +08:00
XuanYang-cn 1567aa3f53
[skip e2e]Update license for storage payload (#14347) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 09:43:26 +08:00
congqixia 20cc6843a4
[skip e2e] Add comment for FieldData interface (#14351) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-28 09:34:15 +08:00
edward.zeng 57ec25c246
[skip e2e] Refine storage binlog iterator comment (#14309) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-27 15:46:03 +08:00
edward.zeng 94e65eaaa5
[skip e2e] Refine storage binlog writer comment (#14312) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-27 15:42:03 +08:00
jaime 28dfccb6ea
[skip e2e] Fix golint in data_codec (#14212) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-24 19:56:28 +08:00
XuanYang-cn 7ce71870ce
[skip e2e]Update license for storage event test (#14191) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 17:34:21 +08:00
XuanYang-cn effd4cc461
[skip e2e]Update license for storage miniochunk (#14193) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 17:32:20 +08:00
XuanYang-cn c29cf1bc21
[skip e2e]Update license for storage localchunck (#14192) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 17:30:32 +08:00
shaoyue e0a2f1ce82
[skip e2e] Add comment for IndexCodec (#14139) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-24 13:31:30 +08:00
shaoyue 94265639fe
[skip e2e] Add comment for PayloadWriter (#14138) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-24 13:29:03 +08:00
XuanYang-cn 107e3f8971
[skip e2e]Update license for storage reader (#14041) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 13:08:05 +08:00
jaime d623ea5c37
[skip e2e] Fix goint error in payload (#14109) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-23 22:07:38 +08:00
jaime 0114708c95
[skip e2e] Fix grammar mistake in data_codec (#14108) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-23 22:05:51 +08:00
jaime 846e86b478
[skip e2e] Fix grammar mistake in event_data (#14107) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-23 22:03:56 +08:00
shaoyue d4f9365a3d
[skip e2e] Add comment for internal/storage/data_codec.go (#14030) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-23 14:30:34 +08:00
XuanYang-cn e580bb9ccb
[skip e2e]Update license for storage event writer (#14043) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:07:10 +08:00
XuanYang-cn 78c7ec7300
[skip e2e]Update license for storage event (#14042) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:05:21 +08:00
XuanYang-cn 6eb0de1640
[skip e2e]Update license for storage writer (#14040) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:03:28 +08:00
XuanYang-cn bccf65ec67
[skip e2e]Update license for storage datacodec (#14039) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:01:36 +08:00
XuanYang-cn a666cb4f76
[skip e2e]Update license for storage sorter (#14038) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 11:59:21 +08:00
shaoyue cd410c6069
[skip e2e] Add comment for internal/storage.PayloadWriter (#13993) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-22 19:33:39 +08:00
shaoyue ec91866a5c
[skip e2e] Add comment for internal/storage.Int8FieldData (#13992) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-22 19:31:51 +08:00
shaoyue 770adc17ae
Support library link in centos7 (#13974) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-22 19:22:19 +08:00
JackLCL b79d751977
[skip ci]Fix error log output format in data_codec.go (#13382) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-22 09:59:04 +08:00
shaoyue 8998e1d29b
[skip e2e] Add comment for PayloadReader (#13899) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-21 21:06:28 +08:00
XuanYang-cn d2c83c24f4
[skip e2e]Update license for storage iterator (#13890) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-21 18:48:57 +08:00
shaoyue bcb951b0de
[skip e2e] Add comment for event writer (#13800) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-20 23:18:53 +08:00
jaime 40a0c76e73
[skip e2e] Fix grammar mistake in event_data (#13805) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-20 22:49:00 +08:00