milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
14,574 Commits
140 Branches
117 Tags
580 MiB
Commit Graph

5158 Commits (bb6cd4b4848b09f6f2f9706147e43abf9ceaa968)

Author SHA1 Message Date
yah01 63a71e247a
Loading segments concurrency level fallbacks to the value that won't cause OOM (#16172) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-24 13:39:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot 478890a700
modify import proto (#16155) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-23 19:29:25 +08:00
Letian Jiang 321105cc01
Add column-based insert interface in segcore (#16100) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-23 19:05:25 +08:00
groot 1c4b949a1d
modify import rpc interface (#16129) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-22 15:11:24 +08:00
groot 2078b24dca
Prepare for import (#16083) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-21 15:47:23 +08:00
Jiquan Long f8d9bc919d
Unify interface of vector index & scalar index. (#15959) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-21 14:23:24 +08:00
XuanYang-cn ad41e2a082
Tidy getQueryChannel for readability (#16096) 
- Remove useless codes in query_coord.go
- Tidy slice initialization in LoadCollection task
- Add ut for getQueryChannel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-21 11:25:24 +08:00
shaoyue 97e5d77953
implements restful entities api (#15916) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-03-21 11:21:23 +08:00
XuanYang-cn 15a3fe41c5
[skip e2e]Fix comments of exposed functions in QC (#16094) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-21 11:17:24 +08:00
Xiaofan 787d4f8bfd
Refine prometheus metrics and etcd log (#16084) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-18 15:51:22 +08:00
yah01 31c5b286da
Fix wrong unittest, check whether loading collection will cause OOM (#16068) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-18 13:17:22 +08:00
godchen 78557ca6d7
Replace minio kv with minio chunk manager (#15936) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-17 18:03:23 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
Cai Yudong d65cf5309f
Remove useless LogMgr (#16080) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-16 21:19:21 +08:00
edward.zeng d4ad785b76
Bump go version in documentation (#15979) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2022-03-16 18:51:21 +08:00
Ten Thousand Leaves a3a29fed06
Add entry point for embedded Milvus. (#16008) 
/kind feature

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-16 16:51:22 +08:00
Bingyi Sun 256ccb8f65
Fix rootcoord goroutine leak (#16071) 
issue: #15658
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-03-16 16:47:21 +08:00
Jiquan Long ffdad82b9c
Set load info in reschedule tasks (#16065) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-16 11:43:21 +08:00
jaime cf4c014b2e
Remove collect query result by mq (#15988) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-16 10:19:21 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
Jiquan Long c984c4ea73
Set load meta info in load balance task (#16060) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-15 20:37:21 +08:00
congqixia b61e8201e7
Refactor querynode historical replica (#15992) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-15 19:15:20 +08:00
Xiaofan c51c8c0a74
Fix get flused segment errror handling (#15155) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-15 18:51:21 +08:00
Bingyi Sun d3d5a41b76
Fix memory leak casued by unclosed channel in msgstream (#16051) 
issue: #16045
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-03-15 15:55:21 +08:00
zhenshan.cao 0bd07daae3
Add grpc healthy check (#16050) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 15:05:21 +08:00
jaime 97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-15 14:45:22 +08:00
XuanYang-cn d4a0c5eba6
Alter comments in parallel.go for readability (#16032) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-15 10:26:01 +08:00
xige-16 09b2c46b61
Support create collection with VarChar field (#15904) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-14 23:20:02 +08:00
bigsheeper caf9cbfcd4
Fix load failure and remove parition release related code (#16038) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-14 23:18:02 +08:00
congqixia fab0263616
Use RmqFactory in DataNode unit test (#16039) 
Use Rocksmq instead of Pulsar in DataNode unittest
- Avoid pulsar seek problem caused timeout issue
- Remove third party component dependency in unit test

See also: #15986

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-14 19:22:01 +08:00
yah01 d4225117ee
Enable Zstd compression for binlog (#15840) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-14 16:00:01 +08:00
congqixia 6a66faa2fb
Explictly specify Load operation meta (#16027) 
Fix #15996

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-14 09:50:01 +08:00
jaime 29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
groot 89356ebcb9
Modify import rpc interfaces (#16006) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-11 17:13:59 +08:00
yah01 7d2934e4c2
Add methods to compress/decompress small blocks (#15980) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-11 14:55:59 +08:00
Jiquan Long 3121619758
Chunk manager support scalar data (#16010) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-11 14:39:59 +08:00
yah01 4f3d5fb6eb
Make load operation parallel improve performance (#15928) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-11 14:28:01 +08:00
Letian Jiang 8f52e5b6c7
Add unit tests in segcore (#15960) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-10 16:33:59 +08:00
Ten Thousand Leaves d5bae6710f
Make all logs configurable for standalone/embedded Milvus. (#15926) 
Resolves: #15919, #15708
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-10 10:35:59 +08:00
groot 5fdef607d5
Add rpc interfaces for import (#15930) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-09 18:33:59 +08:00
Xiaofan d013134b44
Enhance error message when coordinator not healthy (#15950) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-09 15:57:58 +08:00
XuanYang-cn c2fe12bc97
Handle error in topic creation of rocksmq (#15922) 
Reopen for: #14429

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-09 15:47:59 +08:00
congqixia 6513be5c44
Extend watch/release channel logic in DataNode (#15925) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-09 15:39:58 +08:00
bigsheeper 9ad5c14a4a
Return error when create message stream failed in QueryNode (#15317) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-09 15:17:59 +08:00
顾炎 4f8c08f912
fix checkSegmentSize overflows int64 (#15858) 
Signed-off-by: guyan <632722781@qq.com>
 2022-03-09 13:29:58 +08:00
yah01 565524ed3c
Add zstd compressor in util (#15779) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-08 18:17:58 +08:00
zhenshan.cao dff08dbf47
Refactor querynode unittest (#15929) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-08 17:39:58 +08:00
Letian Jiang 74f66dce3b
Fix passing empty binlog content in compaction (#15909) (#15927) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-08 16:34:01 +08:00
bigsheeper 6fec7ed6fb
Add search benchmark in QueryNode (#15854) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-08 10:15:59 +08:00
yah01 7124e91d53
Refactor assigning loading segments task in QueryCoord (#15895) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-03-07 19:23:58 +08:00
congqixia a5c3f4eef4
Add CompareFailErr for CAS operation error check (#15893) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-07 18:01:58 +08:00
cai.zhang 2512dff261
Fix misuse of timerecord (#15868) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-04 15:45:56 +08:00
Letian Jiang 08d12e62f3
Add ByteSlice Method for embed etcd (#15844) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-04 15:14:01 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
Cai Yudong 57b97ccb9c
Let search fail as if one query_node return error result (#15855) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 10:43:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
congqixia 54862a31b0
Extend ChannelWatchState in protobuf (#15847) 
Related #15846
/kind enhancement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-03 18:43:56 +08:00
Letian Jiang 2be217bcf4
Specify appropriate timestamp in initPKBloomFilter (#15823) (#15826) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-03 17:33:56 +08:00
congqixia 3f34bc1ab7
Ban DataType_String in CreateCollection (#15838) 
Fix #15836
/kind enhancement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-03 16:57:56 +08:00
cai.zhang 88527fc0a9
Fix metrics label conflict (#15839) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-03 16:05:57 +08:00
朱亚光 27a7875a15
add array hashKeys empty check (#15796) 
fix #15791 https://github.com/milvus-io/milvus/issues/15791
Signed-off-by: zhuyaguang <zhuyaguang1368@163.com>
 2022-03-03 00:33:56 +08:00
cai.zhang 9f4c59e621
Add prometheus metrics for RootCoord (#15684) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-02 21:11:57 +08:00
cai.zhang 09a27d1620
Add prometheus metrics for indexnode (#15640) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-02 20:49:56 +08:00
Cai Yudong 92c8e32ebd
Let MemoryKV.Load return error when key not exist (#15814) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-02 18:51:55 +08:00
Letian Jiang 442e791128
Add ByteSlice Method for memkv (#15774) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-02 17:29:56 +08:00
XuanYang-cn 63eec7ffc8
Refactor logs in binlog_io to reduce output (#15827) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-02 16:49:55 +08:00
cai.zhang d183748484
Add some prometheus metrics for proxy (#15582) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-02 16:23:55 +08:00
Xiaofan bd7d294c6d
Refine log level in data (#15821) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-02 15:35:55 +08:00
bigsheeper 12eb8c3da5
Add prometheus metrics for QueryNode (#15649) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-02 14:49:55 +08:00
Xiaofan 827389c704
Refine tso log print (#15813) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-02 14:31:55 +08:00
zhenshan.cao f1aec49a74
Fixbug: building index return nil by mistake when error not occur (#15809) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-02 09:41:56 +08:00
XuanYang-cn e394ba8c88
Refactor rocksmq comments and logs for readability (#15801) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-01 18:55:55 +08:00
Cai Yudong 529098fd87
[skip ci] Update knowhere/CMakeLists.txt (#15804) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-01 17:55:55 +08:00
Jiquan Long 001567640e
Fix shallow copy of typeutil.AppendFieldData (#15790) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-01 15:13:54 +08:00
Cai Yudong 503724be19
Optimize CMakeLists.txt under internal/core (#15770) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-01 10:31:55 +08:00
Nemo a77e0562b2
Make C++ log level configurable and add log configuration while building embedded milvus. (#15743) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-01 10:15:55 +08:00
godchen 2e4391e6c5
Add lru cache (#15733) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-28 21:43:55 +08:00
bigsheeper 13177a90af
Add prometheus metrics for DataNode (#15650) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-02-28 19:11:55 +08:00
Bingyi Sun d6857f4b24
Remove globalZapWrapper in logutil.go (#15726) 
issue: #15720

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-02-28 17:23:54 +08:00
bigsheeper e1bfd14db1
Add Prometheus metrics for QueryCoord (#15606) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: jingkl <34296482+jingkl@users.noreply.github.com>
 2022-02-28 16:51:55 +08:00
Bingyi Sun b41992eda1
Fix querycoord ut bug (#15752) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-02-28 11:27:54 +08:00
yah01 0cfb6a85be
Make compaction not generate empty segment (#15707) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-28 10:17:54 +08:00
XuanYang-cn e8edaa02fa
Refine code of DataNode binlogIO (#15759) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-28 10:09:54 +08:00
yah01 867cf620ee
Add Extract() method to take out C pointer from BytesConverter without releasing it (#15614) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-27 09:47:53 +08:00
bigsheeper 6a48071a5c
Set BaseMsg for LoadSegmentRequest, prevent crashing in log printing (#15725) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-02-26 20:09:54 +08:00
Cai Yudong adca79facc
Fix centos build error (#15760) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-25 18:49:54 +08:00
yah01 d67c801511
Add bytes methods for RocksdbKV (#15749) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-02-25 17:19:53 +08:00
yah01 1a9933a7c3
Add methods to load/save bytes for MinIOKV (#15732) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-02-25 16:21:53 +08:00
Nemo cdcb3627b4
Make Etcd log level configurable. (#15737) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-02-25 15:03:53 +08:00
congqixia a05df77e96
Remove unused ValueKV (#15748) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-25 14:47:52 +08:00
Letian Jiang b56ec7ea97
Add ByteSlice Method for Etcd-kv (#15738) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-02-25 11:25:53 +08:00
godchen fcf0887d92
Add and implement chunk manager interface. (#15541) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-02-24 23:53:53 +08:00
xige-16 1e8e3eb912
Fix wrong flushSegmentInfos in watchdmChannel request (#15740) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-02-24 19:23:52 +08:00
XuanYang-cn 6f85b49a4f
Add more compaction test cases (#15715) 
Test merge compaction with 2 segments with the same PK and only 1
valid deletion.

This test can varify the compaction behavior of DataNode for the
dup PK in different segment cases.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-24 11:27:53 +08:00
shaoyue 83fdc1da7c
Add web server in proxy with sample handler (#15647) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-02-23 14:37:52 +08:00
congqixia b0923f1299
Fix DataCoord panics if message queue service quits before it (#15702) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-23 11:09:52 +08:00
Cai Yudong 54b8b24151
Rename variable names for better readibility (#15700) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-22 22:15:52 +08:00
Bingyi Sun ce9662c140
Fix data race in test (#15690) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-02-22 17:43:52 +08:00
Cai Yudong 9a6ee61731
Change all fields in Expr to const to avoid modification (#15663) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-22 14:45:51 +08:00
Nemo 3eb074a9b0
Update some comments for data coord with some minor code refactoring. (#15686) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-02-22 13:15:51 +08:00
xige-16 782f350da5
Modify parameter spelling of task queue (#15581) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-02-22 10:25:51 +08:00
cai.zhang b435c422c1
Add prometheus metrics for IndexCoord (#15638) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-02-21 17:15:51 +08:00
XuanYang-cn d011073269
Fix DataNode panic when compact empty segment (#15580) 
remove the TODO in compactor

See also: #15573

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-18 18:47:51 +08:00
zhenshan.cao 142848fcc3
Abandon using protobuf to pass binaryset parameter (#15626) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-02-18 18:39:50 +08:00
Jenny Li 83f1358408
Update codecov.yml (#15639) 
Signed-off-by: Jenny Li <jing.li@zilliz.com>
 2022-02-18 18:29:50 +08:00
XuanYang-cn 23dfbf1f34
Fix eventmanager retry interval (#15631) 
See also: #15608

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-18 17:31:49 +08:00
Bingyi Sun 2e676a4b78
Extract segments info into a new struct. (#15537) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-02-18 15:27:50 +08:00
congqixia 5751759c85
Implement mix compaction logic (#15542) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-18 14:51:49 +08:00
Bingyi Sun daf25bcb81
Add log when remove keys in garbage collector. (#15453) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-02-18 11:59:49 +08:00
Nemo dc65b1aeea
Switch to knowhere v1.0.1. Also switch to forked gorocksdb v0.0.1 (#15603) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-02-17 19:05:49 +08:00
congqixia c7f55c2e34
Make SessionWatch keep watch even no Rewatch func when find ErrCompacted (#15497) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-15 15:07:48 +08:00
yah01 684110bc9a
Avoid to copy during converting C bytes to Go bytes (#15551) 
* Avoid to copy during converting C bytes to Go bytes

Signed-off-by: yah01 <yah2er0ne@outlook.com>

* Add BytesConverter to achieve no-copy converting C bytes to Go bytes

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-15 14:17:48 +08:00
congqixia 7daed4ee47
Return flush state true when segment in Flushing state (#15530) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-10 16:55:46 +08:00
Xiaofan d499ffc421
Fix confusing logs in rootcoord (#15431) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-10 13:51:47 +08:00
Xiaofan a18d6b46a4
Fix watch dml channel fail because of no collection meta (#15436) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-10 13:19:46 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-09 18:55:46 +08:00
jaime 87dd4e8f92
Fix clean up data dir failed for TestEtcdRestartLoad (#15506) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-02-09 18:37:45 +08:00
jaime cb9c63d039
Remove index dir (#15467) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-02-09 16:07:45 +08:00
Cai Yudong 8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-09 14:41:45 +08:00
jaime 307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
guoxiangzhou f43d96b58d
[skip e2e] Modify code comments according to the naming conventions (#15173) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-02-09 10:19:56 +08:00
guoxiangzhou a157487f8c
[skip e2e] Modify code comments according to the naming conventions (#15172) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-02-09 10:18:02 +08:00
guoxiangzhou c7e93f95aa
[skip e2e] Modify code comments according to the naming conventions (#15171) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-02-09 10:14:05 +08:00
congqixia 2263425cde
Fix EntriesNum is not passed in flush task (#15482) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-09 10:05:46 +08:00
xige-16 55cdc5aa35
Get indexInfo and segment size by queryCoord (#14207) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-02-08 21:57:46 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
XuanYang-cn 43b6c295a4
Put logs of compaction elapse in defer (#15485) 
Before this PR, time elapse are logged after `compact()` finished
normally. No elapse will be logged with any errors.

This PR put elapse logs in defer once time-consuming steps are finished.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-08 19:15:45 +08:00
XuanYang-cn 46dea0f006
Add elapse logs in compaction (#15414) 
See also: #15413

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-08 14:11:45 +08:00
Ji Bin d83f69bb21
Support windows compliation (#15448) 
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS

issue: #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-02-07 23:43:46 +08:00
JunSung Park f6873d3dc1
Implement automatic item expiration on compaction (#14418) 
* Implement automatic item expiration on compaction

Signed-off-by: JunSung Park <writtic@gmail.com>

* Change entityExpiration value comparison logic

Signed-off-by: JunSung Park <writtic@gmail.com>

* Add logging for the number of expired entity

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default value of CompactionEntityExpiration to off

Signed-off-by: JunSung Park <writtic@gmail.com>

* Fix errors for code checker

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default entityExpiration to max value

Signed-off-by: JunSung Park <writtic@gmail.com>
 2022-02-07 22:45:46 +08:00
XuanYang-cn 5eef9886b3
Add more specific log when exiting BackGroundGC goroutine (#15273) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-07 16:55:44 +08:00
congqixia d830791e4f
Respect SegmentIDs in querynode.GetSegmentInfo (#15455) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-07 15:35:44 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
congqixia b4bfe58fd9
[skip e2e] Fix component name for IndexCoordComponent comment (#15138) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-29 22:39:44 +08:00
XuanYang-cn eef295465a
Fix timetick inconsistency in drop collection (#15408) 
When rootcoord drops a collection, it'll produce a `drop-collection DDL` msg
and one last timetick into DML channels of this collection.
So that when DataNode receives this msg, DN can release the resources for the specific collection.

Before this PR, RootCoord produced these two msgs with an older timestamp, generated
before many time-consuming RPCs.

Once these RPCs spend more time than timetick producing interval, the timetick of these 2 msgs are
older to the channel current timestamp, causing in-consistency in time, thus
making msgstream fail to consume the last `drop-collection DDL` msg.

This PR generates a new timestamp for `drop-collection DDL` msg and timetick msg after those time-consuming
RPCs

Fixes: #15406

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-28 20:49:41 +08:00
Filipe Caixeta dee1961b4d
Make SearchResults and QueryResults return collectionID (#15377) 
* Make SearchResults and QueryResults return collectionID

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>

* add return collection name from QueryRequest and SearchRequest

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>

* remove comment

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>
 2022-01-27 22:55:41 +08:00
Xiaofan 885ecaf222
Refine time sync logic log (#15251) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-01-27 11:25:41 +08:00
congqixia 273890a91e
Fix datanode flush manager injection unit test (#15358) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-27 10:23:40 +08:00
xige-16 b9d1396258
Delete ut's useless param for genCollectionSchema (#15119) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-26 14:21:40 +08:00
xige-16 6336e2326c
Set triggerCondition when queryCoord reload loadbalanceTask meta (#15380) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-25 17:26:13 +08:00
XuanYang-cn 8ab9d769de
Fix logs of compaction_executor (#15365) 
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-25 13:29:40 +08:00
zhenshan.cao 3f48a40ab0
Load index meta in every retry (#15376) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-25 13:28:05 +08:00
Bingyi Sun 515fe962d5
Fix subscription path (#15373) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-01-25 13:27:44 +08:00
xige-16 3a024307f4
Release collection first when drop collection (#15370) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-24 21:51:22 +08:00
zhenshan.cao 5bc6ec7250
Fixbug: QueryNode search exceptions will cause memory leak (#15305) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-24 21:19:39 +08:00
Bingyi Sun 3987574448
Unsubscribe topic in DataCoord (#15353) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-01-24 19:47:38 +08:00
zhenshan.cao 6bc2efe429
Fixbug: IndexNode should panic when save meta failed to MetaKV (#15347) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-24 17:18:46 +08:00
bigsheeper 390cdd2515
Remove tmporary debug log in core (#15330) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-24 10:03:41 +08:00
xige-16 1cd2363bd8
Delete usless params SkipQueryChannelRecovery (#15289) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-21 21:43:40 +08:00
xige-16 57383c9f76
Delete Useless timetick channel (#15296) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-21 14:31:39 +08:00
Bingyi Sun c877f33084
Downgrade pulsar client (#15325) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-01-20 22:25:38 +08:00
xige-16 c132302129
Clean flowGraph if watchChannel failed (#15303) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-20 10:01:38 +08:00
congqixia 1e61112b5a
Add retry logic in pulsar consumer unsubscribe (#15284) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-20 01:23:37 +08:00
xige-16 bbb672cb00
Remove state service in querynode (#15281) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-18 18:43:37 +08:00
XuanYang-cn 87394e29b6
Fix compaction ut datarace (#15267) 
Fixes: #15241

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-18 17:49:39 +08:00
Cai Yudong 356efee492
Rename timetickSync APIs and variables for better readability (#15264) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-18 14:47:36 +08:00
cai.zhang f564ea7fc7
RootCoord set healthy state after registered service (#15261) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-01-18 12:09:37 +08:00
Cai Yudong 1b1d4e502b
Remove session from timetickSync (#15255) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-18 00:07:37 +08:00
bigsheeper 7f775096c5
Unsubscribe when close pulsar consumer (#15221) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-17 18:53:36 +08:00
xige-16 408a6e9e58
Fix collectResultLoop hang after search timeout (#15250) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-17 18:11:36 +08:00
xige-16 dfc6670f0f
Unsubscribe channel after query node down (#15230) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-17 17:37:37 +08:00
zhenshan.cao 871e093dc8
Call release memory in case error occured (#15237) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-17 14:43:38 +08:00
Jiquan Long 675e6d352b
QueryNodes send search/query results by rpc (#15223) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-17 14:41:35 +08:00
yah01 fd8d452cfd
remove unused fieldIDs and fieldTypes (#15229) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-01-16 17:37:35 +08:00
Xiaofan f36e02dfbb
Add more log when tt failed (#15227) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-01-15 18:53:34 +08:00
Xiaofan f617c8887b
Time Tick Failed need log (#15225) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-01-14 23:55:34 +08:00
zhenshan.cao 1601a61b78
Move logic of checking available port to service (#15222) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-14 21:17:34 +08:00
zhenshan.cao f14df1d2db
Initilize listener in service instead of ParamTable (#15220) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-14 18:29:34 +08:00
xige-16 cc5bee4684
Stop handoff if segment has been compacted (#15181) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-13 21:49:35 +08:00
Jiquan Long 2819f6cc6f
Release memory to OS after index is built (#15196) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-13 20:27:34 +08:00
congqixia 2528b6850d
Refactor flush manager injection to reduce goroutine number (#15180) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-13 18:49:34 +08:00
xige-16 384d3169d3
Don't remove nodeInfo when querycoord panic at loadBalanceTask (#15189) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-13 17:59:34 +08:00
XuanYang-cn 008e08a996
Add flowgraph manager and event manager (#15102) 
1. Add flowgraph manager to manager start and release of dataSyncService
2. Add event manager to manager etcd watchInfo event
  - Make put event able to retry
  - Make delete event able to terminate the retry loop

See also: #14604, #14300

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-13 14:21:34 +08:00
xige-16 4e956ee10c
Panic when recover querynode failed (#15161) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-12 17:43:37 +08:00
bigsheeper 0c98f21d4d
Remove global sealed segments in historical (#15177) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-12 17:01:38 +08:00
cai.zhang 3254d36609
[skip e2e]Add comment for msgIndexCoordIsUnhealthy (#15146) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-01-12 11:15:36 +08:00
cai.zhang 745d5f5a8a
[skip e2e]Add comment for newGrpcClient (#15144) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-01-12 11:13:37 +08:00
cai.zhang 89c9073ab3
[skip e2e]Add comment for msgIndexNodeIsUnhealthy (#15142) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-01-12 11:11:36 +08:00
cai.zhang 4daf0fb6b5
[skip e2e]Add comment for errIndexNodeIsUnhealthy (#15141) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-01-12 11:09:36 +08:00
Jiquan Long 3bf34f6326
[skip e2e] Add comment for baseTaskQueue (#15170) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-12 10:59:37 +08:00
Jiquan Long 5d91fb3c70
[skip e2e] Add comment for MetaCache (#15169) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-12 10:57:41 +08:00
bigsheeper ebed1a68ff
Add log for segcore search (#15159) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-11 18:07:34 +08:00
XuanYang-cn 6e847ff194
Make dn handleChannelInfo log more consist (#14936) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-11 09:37:35 +08:00
Cai Yudong 951249c6c1
Use KnowhereConfig for indexnode and querynode (#15112) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-11 09:35:34 +08:00
zhenshan.cao 543e348730
[skip e2e]Update license (#15131) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-11 00:09:34 +08:00
zhenshan.cao 0f7a04fc8c
[skip e2e]Update license (#15129) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-11 00:03:38 +08:00
zhenshan.cao 6d36fc60cf
[skip e2e]Update license (#15132) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-11 00:01:36 +08:00
zhenshan.cao e5e739ec26
[skip e2e]Update license (#15127) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-10 23:59:43 +08:00
zhenshan.cao 2962ceeb65
[skip e2e]Update license (#15126) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-10 23:57:40 +08:00
xige-16 99315ce00e
Optimize recentlyModified's lock (#15120) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-10 23:17:36 +08:00
jaime 828b9d399f
[skip e2e] Update license (#15123) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2022-01-10 22:45:37 +08:00
jaime 96dde53240
[skip e2e] Update license (#15122) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
 2022-01-10 22:43:41 +08:00
Jiquan Long 7e74db0d65
[skip e2e] Add comment for insertRepackFunc (#15115) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-10 22:41:46 +08:00
Jiquan Long 5ab8d99ee9
[skip e2e] Add comment for lastTick (#15114) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-10 22:39:54 +08:00
Jiquan Long ff60bb8bfc
[skip e2e] Add comment for GetCollectionSchema (#15113) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-10 22:37:58 +08:00
bigsheeper 018f32847f
[skip e2e] Add comment for createSearchPlan (#15111) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-10 22:36:06 +08:00
bigsheeper 4174daee80
[skip e2e] Add comment for newPartition (#15110) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-10 22:34:09 +08:00