milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
20,338 Commits
139 Branches
115 Tags
576 MiB
Commit Graph

464 Commits (1f55061717d8e9e29ac571c1bcd0a7836b7c4d2f)

Author SHA1 Message Date
Xiaofan 633a749880
Recude IndexCodec Load Memory (#20621) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-11-18 10:47:08 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 18:59:09 +08:00
shaoyue 09ea38615e
Fix gcp oauth token not cached (#20380) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-11-08 12:43:02 +08:00
Xiaofan 766957e75a
Recuce memory consumption in MinioChunkManager (#20284) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-11-03 12:19:35 +08:00
shaoyue 9fac1476f3
Add support for GCS(GoogleCloudStorage) with IAM (#20164) 
go mod tidy

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-11-01 11:07:35 +08:00
Xiaofan 2bfecf5b4e
Refine bloomfilter and memory usage (#20168) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-31 17:41:34 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
yah01 89c9cb3680
Fix scan may break GC limitation (#19670) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-15 15:13:24 +08:00
jaime 1919353f02
Fix parse segment id error from delta log (#19566) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-30 14:18:55 +08:00
congqixia 838a633584
Add ctx parameter for ChunkManager methods (#19546) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-29 16:18:56 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-26 18:06:54 +08:00
Xiaofan 928a213e31
Fix Chunk Manager Path mis use (#19141) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-23 14:40:51 +08:00
congqixia f24e16a66f
Handle multiple parquet RowGroup/Column (#19283) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-21 09:12:50 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
groot b161aec95e
Support input empty string (#19111) (#19144) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-09-13 13:36:29 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00
congqixia ce9ba0dd02
Remove debug log in minio chunk manager (#19134) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-09 17:32:37 +08:00
congqixia db23a256c2
Make minioChunkManager ListObject level by level when recursive is true (#19096) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-08 14:58:34 +08:00
congqixia 68a6587374
Set insert&stats binlog timestamp range (#19005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-04 09:05:09 +08:00
congqixia c91bb0b015
Unify rootPath in configs and ChunkManager (#18808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-25 19:32:53 +08:00
codeman 70254c3ec5
Unified catalog interface for segment (#18289) (#18290) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-20 10:24:51 +08:00
xige-16 e40061b864
Update binlog event format (#18347) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-08-11 14:06:38 +08:00
jaime efa5dfaa7b
Fix Parse binlog path failure (#18583) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-10 17:04:37 +08:00
jaime 6e7831470c
Revert settings of Pulsar address and refine config code (#18494) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-02 21:26:33 +08:00
jaime c9174d55ba
Refine merge operation during compacting phase (#18399) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-28 14:52:31 +08:00
congqixia e9d777b336
Fix ParseSegmentIDBinlog panicks with bad input (#18413) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-26 19:32:30 +08:00
xing.zhao d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-07-22 22:10:28 +08:00
Xiaofan 6d82ef8c20
fix garbage collector err handling (#18277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-15 13:54:26 +08:00
cai.zhang 0ad15a22c8
IndexCoord handle events correctly (#17878) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 14:44:21 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
congqixia 37464a281d
Fix datasorter swap vector logic (#17661) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-21 11:38:12 +08:00
shaoyue 04f836f516
Fix s3 delete file (#17574) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-16 18:14:10 +08:00
yah01 70f8bea4b4
Avoid growing slice as deserializing binlogs (#17421) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-08 11:46:06 +08:00
shaoyue 76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
Enwei Jiao d28a2db46c
move arrow from storage to core (#17061) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-22 20:03:58 +08:00
Xiaofan 5355153805
Minio Error is not Handled gracefully (#17003) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:25:55 +08:00
Letian Jiang 72bbe40254
Make PayloadReader read column data in batch (#16826) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-10 11:37:52 +08:00
cai.zhang 2be46a01c0
Reduce the number of retries and add error log (#16754) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-05-05 09:31:51 +08:00
Letian Jiang 5b2b917987
Benchmark go/cgo parquet payload readers (#16736) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-29 15:59:47 +08:00
xige-16 515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
jaime 68b1b82faf
Remove DataKV interface (#16692) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-28 21:03:47 +08:00
yah01 7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment (#16514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-25 15:57:47 +08:00
Ten Thousand Leaves 4ef2df8cb9
Fix an issue where query coord calls MinioChunkManager during init (#16629) 
It causes Milvus not runnable with local storage.

issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:11:46 +08:00
congqixia 9769426e84
Use fmt.Errorf instead of string concat in local_chunk_manager.go (#16442) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-19 18:39:39 +08:00
Ten Thousand Leaves e76a8c5ec2
Update compile options and config for embedded Milvus (#16472) 
/kind enhancement

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-14 19:57:34 +08:00
godchen 4781db8a2a
Add datanode import (#16414) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-12 22:19:34 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
Xiaofan 801eeffbcc
Replace cgo parquet reader to go parquet reader (#16199) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-30 15:21:28 +08:00
Xiaofan b6b3c9863c
Fix incompatibility of stats log (#16277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-30 10:15:28 +08:00
godchen 478f6ca11e
Add cache for vector chunk (#15912) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-26 22:05:26 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
yah01 d4225117ee
Enable Zstd compression for binlog (#15840) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-14 16:00:01 +08:00
Jiquan Long 3121619758
Chunk manager support scalar data (#16010) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-11 14:39:59 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong 92c8e32ebd
Let MemoryKV.Load return error when key not exist (#15814) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-02 18:51:55 +08:00
Cai Yudong 503724be19
Optimize CMakeLists.txt under internal/core (#15770) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-01 10:31:55 +08:00
godchen 2e4391e6c5
Add lru cache (#15733) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-28 21:43:55 +08:00
Cai Yudong adca79facc
Fix centos build error (#15760) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-25 18:49:54 +08:00
godchen fcf0887d92
Add and implement chunk manager interface. (#15541) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-24 23:53:53 +08:00
jaime 307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
Ji Bin d83f69bb21
Support windows compliation (#15448) 
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS

issue: #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-02-07 23:43:46 +08:00
congqixia 4369e08f2a
Fix storage memory leak caused by runtime.SetFinalizer (#15100) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-10 17:13:35 +08:00
congqixia 998ebcce4e
Fix file not close when ReadAll returns error in local chunk manager (#15068) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-10 11:53:34 +08:00
godchen 384ceea223
Close payload writer when error occurs (#15013) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-07 18:27:23 +08:00
congqixia b22e945513
[skip e2e] Add comment for NewInsertCodec (#14907) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-06 18:04:03 +08:00
XuanYang-cn befb5d77e7
[skip e2e]Update license for cwrapper test (#14785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-06 09:37:30 +08:00
shaoyue c5e3efa874
[skip e2e] Add comment for payload (#14803) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-01-05 13:53:35 +08:00
shaoyue 756904dd3e
[skip e2e] Add coment for payloadwriter (#14724) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-01-04 10:49:27 +08:00
XuanYang-cn 99fd72cfb6
[skip e2e]Update license for cwrapper payload (#14626) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 09:11:39 +08:00
XuanYang-cn 3afe467d86
[skip e2e]Update license for cwrapper parquet (#14625) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 09:09:22 +08:00
XuanYang-cn c7c147b26c
[skip e2e]Update license for cwrapper cmake (#14627) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-30 20:05:26 +08:00
shaoyue f9dc87ba6c
[skip e2e] Add comment for data_codec (#14571) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-30 10:44:05 +08:00
XuanYang-cn 7bfde0aa73
[skip e2e]Update license for storage vector chunk m (#14540) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-29 22:42:14 +08:00
Xiaofan 029b153259
Remove data directory for test (#14363) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 10:08:47 +08:00
XuanYang-cn dd860a76cf
[skip e2e]Update license for storage util (#14453) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 20:11:55 +08:00
XuanYang-cn 758dc8b70f
[skip e2e]Update license for storage unsafe (#14452) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 20:03:56 +08:00
XuanYang-cn 647307ea64
[skip e2e]Update license for storage types (#14451) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 19:57:10 +08:00
XuanYang-cn 9b335e74a2
[skip e2e]Update license for storage stats (#14349) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 09:44:06 +08:00
XuanYang-cn e4ae7a6bef
[skip e2e]Update license for storage print binlog (#14348) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 09:43:49 +08:00
XuanYang-cn 1567aa3f53
[skip e2e]Update license for storage payload (#14347) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-28 09:43:26 +08:00
congqixia 20cc6843a4
[skip e2e] Add comment for FieldData interface (#14351) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-28 09:34:15 +08:00
edward.zeng 57ec25c246
[skip e2e] Refine storage binlog iterator comment (#14309) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-27 15:46:03 +08:00
edward.zeng 94e65eaaa5
[skip e2e] Refine storage binlog writer comment (#14312) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-27 15:42:03 +08:00
jaime 28dfccb6ea
[skip e2e] Fix golint in data_codec (#14212) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-24 19:56:28 +08:00
XuanYang-cn 7ce71870ce
[skip e2e]Update license for storage event test (#14191) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 17:34:21 +08:00
XuanYang-cn effd4cc461
[skip e2e]Update license for storage miniochunk (#14193) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 17:32:20 +08:00
XuanYang-cn c29cf1bc21
[skip e2e]Update license for storage localchunck (#14192) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 17:30:32 +08:00
shaoyue e0a2f1ce82
[skip e2e] Add comment for IndexCodec (#14139) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-24 13:31:30 +08:00
shaoyue 94265639fe
[skip e2e] Add comment for PayloadWriter (#14138) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-24 13:29:03 +08:00
XuanYang-cn 107e3f8971
[skip e2e]Update license for storage reader (#14041) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 13:08:05 +08:00
jaime d623ea5c37
[skip e2e] Fix goint error in payload (#14109) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-23 22:07:38 +08:00
jaime 0114708c95
[skip e2e] Fix grammar mistake in data_codec (#14108) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-23 22:05:51 +08:00
jaime 846e86b478
[skip e2e] Fix grammar mistake in event_data (#14107) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-23 22:03:56 +08:00
shaoyue d4f9365a3d
[skip e2e] Add comment for internal/storage/data_codec.go (#14030) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-23 14:30:34 +08:00
XuanYang-cn e580bb9ccb
[skip e2e]Update license for storage event writer (#14043) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:07:10 +08:00
XuanYang-cn 78c7ec7300
[skip e2e]Update license for storage event (#14042) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:05:21 +08:00
XuanYang-cn 6eb0de1640
[skip e2e]Update license for storage writer (#14040) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:03:28 +08:00
XuanYang-cn bccf65ec67
[skip e2e]Update license for storage datacodec (#14039) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 12:01:36 +08:00
XuanYang-cn a666cb4f76
[skip e2e]Update license for storage sorter (#14038) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-23 11:59:21 +08:00
shaoyue cd410c6069
[skip e2e] Add comment for internal/storage.PayloadWriter (#13993) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-22 19:33:39 +08:00
shaoyue ec91866a5c
[skip e2e] Add comment for internal/storage.Int8FieldData (#13992) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-22 19:31:51 +08:00
shaoyue 770adc17ae
Support library link in centos7 (#13974) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-22 19:22:19 +08:00
JackLCL b79d751977
[skip ci]Fix error log output format in data_codec.go (#13382) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-22 09:59:04 +08:00
shaoyue 8998e1d29b
[skip e2e] Add comment for PayloadReader (#13899) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-21 21:06:28 +08:00
XuanYang-cn d2c83c24f4
[skip e2e]Update license for storage iterator (#13890) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-21 18:48:57 +08:00
shaoyue bcb951b0de
[skip e2e] Add comment for event writer (#13800) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-20 23:18:53 +08:00
jaime 40a0c76e73
[skip e2e] Fix grammar mistake in event_data (#13805) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-20 22:49:00 +08:00
jaime 99737d30e2
[skip e2e] Fix golint in data_codec (#13804) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-20 22:47:07 +08:00
jaime 7723d7b571
[skip e2e] Fix golint in payload (#13803) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-20 22:45:15 +08:00
shaoyue 83a59d2545
[skip e2e]Add comment for payload (#13799) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-20 20:00:53 +08:00
Xieql 35ab21372b
[skip e2e] Improve annotation (#13722) 
Signed-off-by: Xieql <727092844@qq.com>
 2021-12-20 19:08:38 +08:00
congqixia 65faf03923
[skip e2e] Add comment for Int64Stats (#13663) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-17 22:04:43 +08:00
congqixia 0e30ec25f3
[skip e2e] Add comment for descriptorEvent (#13662) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-17 22:02:42 +08:00
godchen 9576ec4dcb
[skip e2e]Fix binlog writer comment (#13576) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:42:50 +08:00
godchen ecd3dc5830
[skip e2e]Remove useless event header check func (#13573) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:00:14 +08:00
shaoyue 43fbb0fd16
[skip e2d] Fix typo in StatsReader comment (#13623) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-17 18:47:56 +08:00
godchen 7e56f08747
Add payload bytes interface. (#13467) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-16 16:35:42 +08:00
shaoyue 1f2c098fe7
[skip ci]Add comment for PayloadReader (#13486) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-16 09:37:22 +08:00
shaoyue 0e24520192
[skip ci] Add comment for index codec (#13464) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-15 21:13:23 +08:00
jaime a91d2118b6
[skip ci] Fix grammar mistake in event_data (#13419) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-15 12:03:58 +08:00
jaime dec2eba610
[skip ci] Fix grammar mistake in local_chunk_manager (#13418) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-15 11:56:07 +08:00
jaime 0aa8e65501
[skip ci] Fix grammar mistake in minio_chunk_manager (#13417) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2021-12-15 11:53:47 +08:00
congqixia 89a26b2679
Fix redundant init declaration in storage/util.go (#13381) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-15 11:13:10 +08:00
shaoyue a54d1ec674
[skip ci]add comment for EventWriter (#13369) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-14 20:05:09 +08:00
shaoyue 18fbc167f8
[skip ci] Add comment for payload (#13367) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-14 20:03:20 +08:00
shaoyue 41190c23f1
[skip ci] Add comment for EventTypeCode (#13368) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-14 19:55:17 +08:00
Xiangyu Wang 405b3cd932
Use the same lint rules with golint in revive (#13324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-14 15:31:07 +08:00
JackLCL 5b9282fdc3
[skip ci]Fix error log output format in data_ciodec.go (#13226) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-13 13:22:14 +08:00
godchen cec42cb19a
Make arrow use jemalloc (#13029) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-09 14:17:36 +08:00
godchen febdda90f4
Change binlog writer close behavior (#13046) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-09 12:37:06 +08:00
godchen 5c44ee8c0f
Close binlog event when error occur (#12995) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-08 21:11:39 +08:00
XuanYang-cn 383f14999d
Make Arrow compilation easier (#13005) 
replace all the bundled libraries with arrow_bundled_dependencies

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-08 20:55:05 +08:00
godchen 0c60df4b3b
[skip ci]Fix comment typo (#13017) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-08 20:51:06 +08:00
godchen 6845b0e986
[skip ci]Fix comment typo (#13016) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-08 20:49:14 +08:00
godchen 9363dbf7b2
[skip ci]Fix comment typo (#13015) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-08 20:47:24 +08:00
zhenshan.cao 69259d9294
Call ReleaseUnused of memorypool in arrow (#12893) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-08 15:53:05 +08:00
shaoyue 5045ba208e
[skip ci] Add comment for storage.StatsWriter (#12965) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-08 15:13:32 +08:00
shaoyue d5aa125c49
[skip ci] Add comment for StatsReader (#12870) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-07 14:53:13 +08:00
shaoyue ccf957e1a8
[skip ci] Add comment for PayloadReader (#12869) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-07 14:51:21 +08:00
godchen 513afd9e42
Update arrow version (#12644) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-03 18:59:33 +08:00
congqixia 21a6ce8647
Remove redundant conversion in event_writer.go (#12590) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-02 19:15:33 +08:00
shaoyue 490f81b932
[skip ci] Add comment for DataDefinitionCodec (#12616) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-02 18:57:32 +08:00
shaoyue ffda2e5945
[skip ci] Fix comment for PayloadReader (#12621) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-02 18:07:33 +08:00
shaoyue 0d92f61d56
[skip ci] Add comment for storage.ChunManger methods (#12524) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-01 15:45:40 +08:00
shaoyue ee89ca83e1
[skip ci] Add comment for internal/storage/data_codec.go (#12520) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-12-01 15:39:57 +08:00
congqixia 001752eb91
Fix zero length slice declaration in storage/util.go (#12412) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-30 09:57:44 +08:00
XuanYang-cn 48b45d82e5
Add ut for binlog_io to 100 coverage (#12283) 
Make DN ut coverage upto 90%
Resolves: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-26 17:43:17 +08:00
XuanYang-cn 6bf0c4326d
Fix compaction generates wrong insert binlogs (#12204) 
This PR
- changes Get() to GetRow() in BinlogIterator
- removes Get() and Length() of FieldData
- adds ut in BinlogIterator for floatvector and binaryvector
- fix a log bug

Fixes: #12146

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-23 17:09:15 +08:00
congqixia 8c107ab15f
Set segment DroppedAt when compacted (#12182) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-22 20:11:14 +08:00
godchen 9d5bcd3e3a
Close event and binlog reader (#12173) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-22 17:27:14 +08:00
Bingyi Sun a3d4cbdd4c
Convert c array to a golang slice (#12066) 
issue: #11974
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-22 10:05:14 +08:00
shaoyue 1587214676
[skip ci] Add comment for exported items in data_codec.go (#12089) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-11-18 19:15:27 +08:00
XuanYang-cn 9bd3664672
Fix compaction bug (#11816) 
This PR fixes:
- timeout in compaction
- BinlogIterator using rawID as PKField bug

See also: #11757

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 10:07:13 +08:00
Xiangyu Wang 12f50cb22c
[skip ci]Update OWNERS files (#11898) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-16 15:41:11 +08:00
Cai Yudong b3d85adb6e
Use API HandleCStatus to handle CStatus returned by CGo in storage (#11782) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-15 13:27:09 +08:00
bigsheeper 93149c5ad9
Load growing segment in query node (#11664) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-11-12 18:27:10 +08:00
Xiangyu Wang 95ac530f5c
[skip ci]Improve import statements (#11715) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-12 11:14:44 +08:00
Xiangyu Wang b8dcaa31bb
[skip ci]Improve import statements (#11714) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-12 11:12:47 +08:00
Xiangyu Wang 0b4d7dc7f6
[skip ci]Improve import statements (#11713) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-12 11:10:47 +08:00
godchen 863f1bb34e
Fix multi delete data not effect (#11422) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-09 15:01:17 +08:00
shaoyue e00a4242df
[skip ci] Add comments (#11408) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-11-08 11:31:00 +08:00
dragondriver 82b510eda2
[skip ci] Fix golint warning of binlog_iterator.go (#11091) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-02 21:45:53 +08:00
godchen a0a3a889e7
Add common endian for global use (#11092) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-02 18:16:32 +08:00
XuanYang-cn 8704293215
Fix storage Value not exposed problem (#10755) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-27 22:32:23 +08:00
zhenshan.cao 373a4684b9
[skip ci]Fix golint error in data_codec (#10697) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-26 22:32:44 +08:00
cai.zhang 5b42a3223c
Increase compatibility for EstimateMemorySize interface (#10603) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-10-26 15:34:21 +08:00
XuanYang-cn cd06f50645
Remove schema in delete codec (#10517) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-24 09:59:10 +08:00
Xiangyu Wang 99ca6e4ba2
Improve comment in storage (#10448) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-10-22 20:45:12 +08:00
Xiangyu Wang 99d57beff8
Improve comment in storage (#10447) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-10-22 17:29:11 +08:00
godchen ffc0c07610
Change delete data primary key to int64 (#10438) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 15:37:12 +08:00
godchen e7738243b9
Fix stats log nil when not int64 field (#10417) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 12:51:11 +08:00
congqixia c3518b8c81
Fix nil stats from DeserializeStats (#10255) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 14:26:35 +08:00
sunby db53eccec9
Add binlog iterators (#10173) 
issue: #9530
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-20 11:52:35 +08:00
godchen 8daeb0d519
Change bloom filter use pk (#10193) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-19 20:18:47 +08:00
XuanYang-cn 2255fe0b45
Change deserialize deltelog from 1 blob to blobs (#10085) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-19 10:28:35 +08:00
XuanYang-cn 75dccad4fa
[skip ci]Fix const golint in data_codec (#9793) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-13 18:59:26 +08:00
XuanYang-cn 2659807826
[skip ci]Fix golint in data_codec (#9792) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-13 18:57:44 +08:00
Cai Yudong a63ef91c74
Fix static-check (#9776) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-13 13:22:33 +08:00
godchen 59ab0e441c
Add bloom filter for stats (#9630) 
* Add bloom filter for stats

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* trigger GitHub actions

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-13 10:22:33 +08:00
dragondriver 1f224c4b2e
Optimize the ut of storage utils (#9740) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-12 19:47:08 +08:00
dragondriver f85271cf3f
Estimate memory size by descriptor event (#9688) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-12 17:00:34 +08:00
dragondriver abb8c2b9ed
Fix the way that binlog store the original size (#9681) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-11 21:02:37 +08:00
dragondriver 7daa319dc2
[skip ci] Rename EstimateMemorySize to GetBinlogSize (#9651) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-11 18:20:30 +08:00
dragondriver 9a7a060484
Store original memory size of binlog file to extra information (#9628) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-11 17:28:30 +08:00
dragondriver 1bc4b36617
Estimate the memory size of binlog file (#9612) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-11 14:10:48 +08:00
dragondriver dedf745b76
Rename IndexParamsFile to IndexParamsKey (#9563) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-09 19:27:02 +08:00
dragondriver 28a90f9e7a
Store index parameters to descriptor event (#9451) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-09 16:08:56 +08:00
congqixia 74233fcfd8
[skip ci] Fix golint for binlog_writer.go (#9388) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-07 22:13:09 +08:00
zhenshan.cao b76e9de5ab
[skip ci] Fix golint error in event_reader.go (#9156) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-03 20:56:05 +08:00
zhenshan.cao 6d2efe13b6
[skip ci] fix golint error in binlog_reader.go (#9114) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-03 08:53:55 +08:00
zhenshan.cao 5e78500519
[skip ci] Fix golint error in print_binlog (#9042) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-01 23:41:23 +08:00
dragondriver 818cf3ffa0
Split blob into several string rows when index file is large (#8919) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-30 17:57:01 +08:00
dragondriver ca5e9ebe55
Binlog parser tool supports index file (#8914) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-30 15:47:32 +08:00
dragondriver cf8600077f
Refactor the index file format (#8514) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-29 09:52:12 +08:00
godchen c23236c50e
Fix chunk manager ci lint (#8779) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-28 20:46:04 +08:00
godchen af173dd2a0
Add delete codec (#8736) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-28 14:30:02 +08:00