Xiaofan
192151bc11
Refine Data Log ( #19996 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
smellthemoon
581e1d8c35
Fill MsgBase ALL ( #19912 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
congqixia
0095869d5b
Keep segment statslog if no insert applied ( #19910 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-20 14:33:27 +08:00
XuanYang-cn
2785f98de6
Refactor segment replica in DN ( #19815 )
...
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-18 15:33:26 +08:00
congqixia
a1cdc55bcb
Make newSegment transfer state after SaveBinlogPath succuess ( #19858 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-18 14:07:25 +08:00
SimFG
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
congqixia
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
jaime
91df8f2d6a
Use log id instead of log path of the binlog in metastore ( #19123 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-25 15:56:51 +08:00
SimFG
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
bigsheeper
5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector ( #18636 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-16 09:56:47 +08:00
congqixia
68a6587374
Set insert&stats binlog timestamp range ( #19005 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-04 09:05:09 +08:00
congqixia
29c72cad15
Use segment statslog instead of separate statslog ( #18775 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 10:34:59 +08:00
congqixia
c91bb0b015
Unify rootPath in configs and ChunkManager ( #18808 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-25 19:32:53 +08:00
Ten Thousand Leaves
0ebe407221
Make DataNode not panic if stale segment is not found during flush ( #18322 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-20 15:06:30 +08:00
congqixia
be6ca6358d
Use serialized size before compression in log meta ( #18336 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-19 21:30:30 +08:00
Ten Thousand Leaves
ecc250c20e
Improve logs to better cover segment flush lifecycle ( #18157 )
...
/kind improvements
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-08 14:50:21 +08:00
congqixia
12b3a29db8
Fix segments info in drop virtual channel request is not set ( #17641 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-06-20 13:40:12 +08:00
congqixia
55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking ( #17247 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-27 16:20:00 +08:00
zhenshan.cao
0a953948af
Simplify monitoring metrics ( #16687 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-27 23:03:47 +08:00
Xiaofan
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
godchen
78557ca6d7
Replace minio kv with minio chunk manager ( #15936 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
cai.zhang
88527fc0a9
Fix metrics label conflict ( #15839 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-03-03 16:05:57 +08:00
Xiaofan
bd7d294c6d
Refine log level in data ( #15821 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 15:35:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
congqixia
273890a91e
Fix datanode flush manager injection unit test ( #15358 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-27 10:23:40 +08:00
congqixia
2528b6850d
Refactor flush manager injection to reduce goroutine number ( #15180 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-13 18:49:34 +08:00
XuanYang-cn
45f1f97ac4
Fix deltalog savebinlogpath save log ( #15092 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-10 15:45:34 +08:00
Xieql
8b06aac7a9
[skip e2e] Improve annotation ( #14976 )
...
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2022-01-07 13:57:49 +08:00
Xieql
c488c41dfb
[skip e2e] Improve annotation ( #14912 )
...
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2022-01-06 13:38:08 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components ( #13885 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
congqixia
2de5cf7762
[skip e2e] Improve log in flush manager ( #13689 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-20 12:22:43 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs ( #13606 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
XuanYang-cn
dbf757e08b
Change genKey into JoinIDPath ( #12959 )
...
1. Simplify genKey(), make genKey a rpc call
2. Unify the usage of JoinIDPath
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-09 11:09:06 +08:00
JackLCL
e51ef53632
[skip ci]Fix error log output format in flush_manager.go ( #12857 )
...
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-07 14:38:15 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode ( #12563 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
congqixia
e813867652
Implement Drop mode for flush manager ( #12469 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-01 10:11:39 +08:00
XuanYang-cn
8a8ebed23d
Fix DN GC bug ( #12258 )
...
Fixes : #12249
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
congqixia
59d19384dc
Fix flush manager injection logic with multiple segments ( #12260 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-24 18:41:16 +08:00
XuanYang-cn
7dbb4d10e5
Fix DeltaLogs empty yet appending a deltaInfo bug ( #11984 )
...
- Add more logs for startpositions
Fixes : #11970
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-17 17:57:11 +08:00
congqixia
5e550a8793
Add close for flushManager ( #11874 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-17 10:39:15 +08:00
congqixia
579416e3d3
Apply retry for flush manager execution ( #11808 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-15 17:19:10 +08:00
XuanYang-cn
c4bb73dc74
Complete drop collection procedure in DN ( #11552 )
...
drop, cancel compaction, flush all buffers and release fg
See also: #11426
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
congqixia
756be81bce
Add injectFlush method for flushManager interface ( #10659 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-28 19:16:31 +08:00
congqixia
ae2a301662
Add injection logic for FlushManager ( #10580 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-25 20:17:34 +08:00
godchen
d2e0943f21
Fix delete node flush error ( #10549 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-25 18:49:20 +08:00
XuanYang-cn
cd06f50645
Remove schema in delete codec ( #10517 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-24 09:59:10 +08:00
congqixia
ed4e516ce7
Save stats and delta logs from DataNode ( #10159 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 14:34:35 +08:00
congqixia
5737e0075f
Apply flush manager logic in datanode ( #10142 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 11:04:34 +08:00
congqixia
90f88c6559
Add flush manager structure ( #9986 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-18 10:38:35 +08:00