Commit Graph

50 Commits (c6151ad3515c75619f1c691af950420182bdc4d9)

Author SHA1 Message Date
Xiaofan 192151bc11
Refine Data Log (#19996)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
smellthemoon 581e1d8c35
Fill MsgBase ALL (#19912)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
congqixia 0095869d5b
Keep segment statslog if no insert applied (#19910)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-20 14:33:27 +08:00
XuanYang-cn 2785f98de6
Refactor segment replica in DN (#19815)
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-18 15:33:26 +08:00
congqixia a1cdc55bcb
Make newSegment transfer state after SaveBinlogPath succuess (#19858)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-18 14:07:25 +08:00
SimFG a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
congqixia 838a633584
Add ctx parameter for ChunkManager methods (#19546)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
jaime 91df8f2d6a
Use log id instead of log path of the binlog in metastore (#19123)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-25 15:56:51 +08:00
SimFG d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
bigsheeper 5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector (#18636)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-16 09:56:47 +08:00
congqixia 68a6587374
Set insert&stats binlog timestamp range (#19005)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-04 09:05:09 +08:00
congqixia 29c72cad15
Use segment statslog instead of separate statslog (#18775)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 10:34:59 +08:00
congqixia c91bb0b015
Unify rootPath in configs and ChunkManager (#18808)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-25 19:32:53 +08:00
Ten Thousand Leaves 0ebe407221
Make DataNode not panic if stale segment is not found during flush (#18322)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-20 15:06:30 +08:00
congqixia be6ca6358d
Use serialized size before compression in log meta (#18336)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-19 21:30:30 +08:00
Ten Thousand Leaves ecc250c20e
Improve logs to better cover segment flush lifecycle (#18157)
/kind improvements

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-08 14:50:21 +08:00
congqixia 12b3a29db8
Fix segments info in drop virtual channel request is not set (#17641)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-06-20 13:40:12 +08:00
congqixia 55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking (#17247)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-27 16:20:00 +08:00
zhenshan.cao 0a953948af
Simplify monitoring metrics (#16687)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-27 23:03:47 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
godchen 78557ca6d7
Replace minio kv with minio chunk manager (#15936)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
cai.zhang 88527fc0a9
Fix metrics label conflict (#15839)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-03-03 16:05:57 +08:00
Xiaofan bd7d294c6d
Refine log level in data (#15821)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 15:35:55 +08:00
bigsheeper 13177a90af
Add prometheus metrics for DataNode (#15650)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
congqixia 273890a91e
Fix datanode flush manager injection unit test (#15358)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-27 10:23:40 +08:00
congqixia 2528b6850d
Refactor flush manager injection to reduce goroutine number (#15180)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-13 18:49:34 +08:00
XuanYang-cn 45f1f97ac4
Fix deltalog savebinlogpath save log (#15092)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-10 15:45:34 +08:00
Xieql 8b06aac7a9
[skip e2e] Improve annotation (#14976)
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2022-01-07 13:57:49 +08:00
Xieql c488c41dfb
[skip e2e] Improve annotation (#14912)
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2022-01-06 13:38:08 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components (#13885)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
congqixia 2de5cf7762
[skip e2e] Improve log in flush manager (#13689)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-20 12:22:43 +08:00
congqixia 1dcd06cc78
Add log file info for all kind of segment binlogs (#13606)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
XuanYang-cn dbf757e08b
Change genKey into JoinIDPath (#12959)
1. Simplify genKey(), make genKey a rpc call
2. Unify the usage of JoinIDPath

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-09 11:09:06 +08:00
JackLCL e51ef53632
[skip ci]Fix error log output format in flush_manager.go (#12857)
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-07 14:38:15 +08:00
congqixia 8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
congqixia e813867652
Implement Drop mode for flush manager (#12469)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-01 10:11:39 +08:00
XuanYang-cn 8a8ebed23d
Fix DN GC bug (#12258)
Fixes: #12249

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
congqixia 59d19384dc
Fix flush manager injection logic with multiple segments (#12260)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-24 18:41:16 +08:00
XuanYang-cn 7dbb4d10e5
Fix DeltaLogs empty yet appending a deltaInfo bug (#11984)
- Add more logs for startpositions

Fixes: #11970

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-17 17:57:11 +08:00
congqixia 5e550a8793
Add close for flushManager (#11874)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-17 10:39:15 +08:00
congqixia 579416e3d3
Apply retry for flush manager execution (#11808)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-15 17:19:10 +08:00
XuanYang-cn c4bb73dc74
Complete drop collection procedure in DN (#11552)
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
congqixia 756be81bce
Add injectFlush method for flushManager interface (#10659)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-28 19:16:31 +08:00
congqixia ae2a301662
Add injection logic for FlushManager (#10580)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-25 20:17:34 +08:00
godchen d2e0943f21
Fix delete node flush error (#10549)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-25 18:49:20 +08:00
XuanYang-cn cd06f50645
Remove schema in delete codec (#10517)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-24 09:59:10 +08:00
congqixia ed4e516ce7
Save stats and delta logs from DataNode (#10159)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 14:34:35 +08:00
congqixia 5737e0075f
Apply flush manager logic in datanode (#10142)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 11:04:34 +08:00
congqixia 90f88c6559
Add flush manager structure (#9986)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-18 10:38:35 +08:00