godchen
78557ca6d7
Replace minio kv with minio chunk manager ( #15936 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
Ji Bin
3cd28420f1
Support compile under windows ( #15786 )
...
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
* using gofrs/flock instead of syscall.Flock
* using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`
issue #7706
Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-17 17:17:22 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
jaime
97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API ( #15971 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-15 14:45:22 +08:00
congqixia
fab0263616
Use RmqFactory in DataNode unit test ( #16039 )
...
Use Rocksmq instead of Pulsar in DataNode unittest
- Avoid pulsar seek problem caused timeout issue
- Remove third party component dependency in unit test
See also: #15986
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-14 19:22:01 +08:00
jaime
29975a7a26
Remove Consume() API from mqstream interface ( #15886 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-11 20:09:59 +08:00
groot
89356ebcb9
Modify import rpc interfaces ( #16006 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-03-11 17:13:59 +08:00
groot
5fdef607d5
Add rpc interfaces for import ( #15930 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-03-09 18:33:59 +08:00
congqixia
6513be5c44
Extend watch/release channel logic in DataNode ( #15925 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-09 15:39:58 +08:00
Letian Jiang
74f66dce3b
Fix passing empty binlog content in compaction ( #15909 ) ( #15927 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-03-08 16:34:01 +08:00
Jiquan Long
f71651e294
Support column-based insert data in message stream ( #15802 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-04 15:09:56 +08:00
Cai Yudong
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig ( #15843 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
jaime
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Letian Jiang
2be217bcf4
Specify appropriate timestamp in initPKBloomFilter ( #15823 ) ( #15826 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-03-03 17:33:56 +08:00
cai.zhang
88527fc0a9
Fix metrics label conflict ( #15839 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-03-03 16:05:57 +08:00
XuanYang-cn
63eec7ffc8
Refactor logs in binlog_io to reduce output ( #15827 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-03-02 16:49:55 +08:00
Xiaofan
bd7d294c6d
Refine log level in data ( #15821 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 15:35:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
yah01
0cfb6a85be
Make compaction not generate empty segment ( #15707 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-28 10:17:54 +08:00
XuanYang-cn
e8edaa02fa
Refine code of DataNode binlogIO ( #15759 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-28 10:09:54 +08:00
XuanYang-cn
6f85b49a4f
Add more compaction test cases ( #15715 )
...
Test merge compaction with 2 segments with the same PK and only 1
valid deletion.
This test can varify the compaction behavior of DataNode for the
dup PK in different segment cases.
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-24 11:27:53 +08:00
XuanYang-cn
d011073269
Fix DataNode panic when compact empty segment ( #15580 )
...
remove the TODO in compactor
See also: #15573
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-18 18:47:51 +08:00
XuanYang-cn
23dfbf1f34
Fix eventmanager retry interval ( #15631 )
...
See also: #15608
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-18 17:31:49 +08:00
congqixia
5751759c85
Implement mix compaction logic ( #15542 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-18 14:51:49 +08:00
Xiaofan
be42627ab2
Move Logutil to Util dir ( #15447 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
Cai Yudong
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig ( #15501 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
congqixia
2263425cde
Fix EntriesNum is not passed in flush task ( #15482 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-09 10:05:46 +08:00
Cai Yudong
1f337891e1
Rename base_param to service_param, rename global_param to component_param ( #15463 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-08 20:57:47 +08:00
XuanYang-cn
43b6c295a4
Put logs of compaction elapse in defer ( #15485 )
...
Before this PR, time elapse are logged after `compact()` finished
normally. No elapse will be logged with any errors.
This PR put elapse logs in defer once time-consuming steps are finished.
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-08 19:15:45 +08:00
XuanYang-cn
46dea0f006
Add elapse logs in compaction ( #15414 )
...
See also: #15413
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-08 14:11:45 +08:00
JunSung Park
f6873d3dc1
Implement automatic item expiration on compaction ( #14418 )
...
* Implement automatic item expiration on compaction
Signed-off-by: JunSung Park <writtic@gmail.com>
* Change entityExpiration value comparison logic
Signed-off-by: JunSung Park <writtic@gmail.com>
* Add logging for the number of expired entity
Signed-off-by: JunSung Park <writtic@gmail.com>
* Set default value of CompactionEntityExpiration to off
Signed-off-by: JunSung Park <writtic@gmail.com>
* Fix errors for code checker
Signed-off-by: JunSung Park <writtic@gmail.com>
* Set default entityExpiration to max value
Signed-off-by: JunSung Park <writtic@gmail.com>
2022-02-07 22:45:46 +08:00
XuanYang-cn
5eef9886b3
Add more specific log when exiting BackGroundGC goroutine ( #15273 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-07 16:55:44 +08:00
Cai Yudong
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable ( #15434 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components ( #15163 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
congqixia
273890a91e
Fix datanode flush manager injection unit test ( #15358 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-27 10:23:40 +08:00
XuanYang-cn
8ab9d769de
Fix logs of compaction_executor ( #15365 )
...
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-25 13:29:40 +08:00
congqixia
1e61112b5a
Add retry logic in pulsar consumer unsubscribe ( #15284 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-20 01:23:37 +08:00
XuanYang-cn
87394e29b6
Fix compaction ut datarace ( #15267 )
...
Fixes : #15241
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-18 17:49:39 +08:00
bigsheeper
7f775096c5
Unsubscribe when close pulsar consumer ( #15221 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-01-17 18:53:36 +08:00
yah01
fd8d452cfd
remove unused fieldIDs and fieldTypes ( #15229 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-01-16 17:37:35 +08:00
congqixia
2528b6850d
Refactor flush manager injection to reduce goroutine number ( #15180 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-13 18:49:34 +08:00
XuanYang-cn
008e08a996
Add flowgraph manager and event manager ( #15102 )
...
1. Add flowgraph manager to manager start and release of dataSyncService
2. Add event manager to manager etcd watchInfo event
- Make put event able to retry
- Make delete event able to terminate the retry loop
See also: #14604 , #14300
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-13 14:21:34 +08:00
XuanYang-cn
6e847ff194
Make dn handleChannelInfo log more consist ( #14936 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-11 09:37:35 +08:00
Cai Yudong
b6a48817b0
Add MinioConfig in GlobalParams for all components ( #15099 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 17:29:34 +08:00
XuanYang-cn
45f1f97ac4
Fix deltalog savebinlogpath save log ( #15092 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-10 15:45:34 +08:00
Cai Yudong
7efb02a4c5
Use PulsarConfig in GlobalParams for all components ( #15046 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-09 23:45:33 +08:00
Cai Yudong
f8f61fb116
Use BaseParams etcd configuration for all components ( #15012 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-07 16:15:26 +08:00
Xieql
8b06aac7a9
[skip e2e] Improve annotation ( #14976 )
...
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2022-01-07 13:57:49 +08:00
zhenshan.cao
a5f8bdeb42
Fix bug: data race in MsgFactory ( #14839 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-06 14:57:20 +08:00
Xieql
25b00230c8
[skip e2e] Improve annotation ( #14908 )
...
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2022-01-06 14:15:20 +08:00