74c73fa75f
fix datanode ( #5635 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
7064937dc3
fix insertBuffer node ( #5632 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0c362a8831
fix bug and add unit test for insert buffer ( #5630 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3c70675313
data node, save binlog to minIO, and let data service save these meta ( #5618 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
91ef35bad4
Change receive msg logic ( #5605 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0a82c6381f
add auto flush for data node ( #5609 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3a5c8c4d3a
Add seek position to WatchDmChannels response ( #5601 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
a948b71e06
Support to send dd msg to all dml channels ( #5597 )
...
* send dd msg to dml channel
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add dml_channels.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix race
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:04:48 +08:00
e7521afed5
Change SaveBinlogPath ( #5576 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
03a2052343
Update DataNode recovery design ( #5578 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
aa8a038305
Rename SegmentAllocator to SegmentManager ( #5559 )
...
Add numRows to segmentStatus and Rename SegmentAllocator to
SegmentManager. Remove SegmentAllocStats.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
275881dbdb
Remove address param ( #5556 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:46 +08:00
41794ec3a6
Change SaveBinLogPath proto & func ( #5551 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
cb6ae3678a
fix master service ut ( #5555 )
...
* fix master unitttest
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* tt
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
339a71aa0a
Add design for datanode flowgraph recovery ( #5562 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
e96d39bf6a
refactor datanode flowgraph ( #5538 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
f1ccbb8f9a
Fix init session in dataservice ( #5522 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
5966076880
fix data service ( #5535 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
31b400a9f7
Refactor the workflow of receiving search result from query node ( #5527 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
54ab03e28f
Fix queryservice connect error ( #5526 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
fca3eb7126
Add implementation of getMinTsStatistics function ( #5524 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
62eaa0390a
Add dataservice metrics with grpc prometheus middleware ( #5523 )
...
* Add dataservice metrics with grpc prometheus middleware
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* register server metrics
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
38b1f7dabe
Change the logic of getting vchannel position ( #5502 )
...
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
31c4a4f5ed
Simplify dd buffer & add lock ( #5513 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
6ea11f67cb
Dock channelsTimeTicker, channelsMgr with master service ( #5509 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
2cc330b117
Remove proxy service from master ( #5510 )
...
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove time sync
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service from distributed/masterservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix unittest
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix static-check
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
68bf983f21
Add Collection ID in ddNode and filter logic ( #5496 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
1f3c9d5dc7
Add statistics function to task scheduler ( #5500 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
6c02ae4eb1
Remove Old msgposition logic ( #5487 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
c213dc468a
Add dqlChannelsMgr ( #5488 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
9a54f1f72d
Watch channel if needed when allocating segmnets ( #5482 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
0b41031ce4
Add channelsTimeTicker to proxy ( #5462 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
a2dd16472d
Implement random reassign and add test case ( #5474 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
8657251f41
DataNode stores segment msgpack positions ( #5472 )
...
* DataNode stores segment msgpack positions
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix typo caused deadlock
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Use MsgPack EndPositions for flush pos
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
43534ef63f
Add unit tests in dataservice ( #5478 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
7a8b34ee9f
Remove not using code in DataNode ( #5476 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
e7368c9447
Add OWNERS file
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-15 15:59:04 +08:00
a14c35274f
Fix bugs in dataservice ( #5457 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
55eac3ee7f
Complete Flush procedure in DataNode ( #5454 )
...
* Complete Flush proceduer in DataNode
Resolves : #5220
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* fix unittest
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
59a3d83f97
Replace insertChannelsMap with channelsMgr ( #5453 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
607a8ffa1f
Update WatchDmChannels signature ( #5447 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
cfdd98b70f
Fix recovery branch after rebasing master ( #5444 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
07c6a4a669
Add dataservice register and discovery ( #5435 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
e3956ad13f
Change SaveBinLogPaths field2Path to repeated ( #8 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
ef563fda8f
Updata flushe procedure
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
c53afee616
DataNode scales flowgraph
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
b9cf4c1095
Fix recovery ( #5417 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
81ac20143b
Add persistency of datanode cluster ( #5387 )
...
We save channels registered in datanode in etcd and restore the cluster info after
restarting.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
3662b3f0e5
Refactor dataservice ( #5400 )
...
Pick up grpc calls to grpc_handler.go and refactor code format
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:01 +08:00
neza2017
godchen
sunby
Cai Yudong
congqixia
XuanYang-cn
dragondriver
Xiangyu Wang
congqixia