1727edb5d3
fix proxy node ( #5686 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:06:13 +08:00
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
e7fd3aa926
Flowgraph recovery ( #5640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
f794ceb0cc
Add segment info rand logic in test case ( #5639 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
e9b282c0be
Use ElementsMatch to test slice equality ( #5641 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
e57e2f77de
Add flush monitor and unit test ( #5622 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
ac19711d74
Destroy DataNode when drop collections ( #5638 )
...
* Destroy DataNode when drop collections
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* golanci-lint
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
f054fc9be2
Add GetRecoveryInfo in dataservice ( #5629 )
...
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
74c73fa75f
fix datanode ( #5635 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
7064937dc3
fix insertBuffer node ( #5632 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0c362a8831
fix bug and add unit test for insert buffer ( #5630 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3c70675313
data node, save binlog to minIO, and let data service save these meta ( #5618 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
91ef35bad4
Change receive msg logic ( #5605 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0a82c6381f
add auto flush for data node ( #5609 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3a5c8c4d3a
Add seek position to WatchDmChannels response ( #5601 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
a948b71e06
Support to send dd msg to all dml channels ( #5597 )
...
* send dd msg to dml channel
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add dml_channels.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix race
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:04:48 +08:00
e7521afed5
Change SaveBinlogPath ( #5576 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
03a2052343
Update DataNode recovery design ( #5578 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
aa8a038305
Rename SegmentAllocator to SegmentManager ( #5559 )
...
Add numRows to segmentStatus and Rename SegmentAllocator to
SegmentManager. Remove SegmentAllocStats.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
275881dbdb
Remove address param ( #5556 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:46 +08:00
41794ec3a6
Change SaveBinLogPath proto & func ( #5551 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
cb6ae3678a
fix master service ut ( #5555 )
...
* fix master unitttest
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* tt
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
339a71aa0a
Add design for datanode flowgraph recovery ( #5562 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
e96d39bf6a
refactor datanode flowgraph ( #5538 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
f1ccbb8f9a
Fix init session in dataservice ( #5522 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
5966076880
fix data service ( #5535 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
31b400a9f7
Refactor the workflow of receiving search result from query node ( #5527 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
54ab03e28f
Fix queryservice connect error ( #5526 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
fca3eb7126
Add implementation of getMinTsStatistics function ( #5524 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
62eaa0390a
Add dataservice metrics with grpc prometheus middleware ( #5523 )
...
* Add dataservice metrics with grpc prometheus middleware
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* register server metrics
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
38b1f7dabe
Change the logic of getting vchannel position ( #5502 )
...
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
31c4a4f5ed
Simplify dd buffer & add lock ( #5513 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
6ea11f67cb
Dock channelsTimeTicker, channelsMgr with master service ( #5509 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
2cc330b117
Remove proxy service from master ( #5510 )
...
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove time sync
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* remove proxy service from distributed/masterservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix unittest
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix static-check
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
68bf983f21
Add Collection ID in ddNode and filter logic ( #5496 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
1f3c9d5dc7
Add statistics function to task scheduler ( #5500 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
6c02ae4eb1
Remove Old msgposition logic ( #5487 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
c213dc468a
Add dqlChannelsMgr ( #5488 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
9a54f1f72d
Watch channel if needed when allocating segmnets ( #5482 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
0b41031ce4
Add channelsTimeTicker to proxy ( #5462 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
a2dd16472d
Implement random reassign and add test case ( #5474 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
8657251f41
DataNode stores segment msgpack positions ( #5472 )
...
* DataNode stores segment msgpack positions
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix typo caused deadlock
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Use MsgPack EndPositions for flush pos
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
43534ef63f
Add unit tests in dataservice ( #5478 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
7a8b34ee9f
Remove not using code in DataNode ( #5476 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
e7368c9447
Add OWNERS file
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-15 15:59:04 +08:00
a14c35274f
Fix bugs in dataservice ( #5457 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
55eac3ee7f
Complete Flush procedure in DataNode ( #5454 )
...
* Complete Flush proceduer in DataNode
Resolves : #5220
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* fix unittest
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
59a3d83f97
Replace insertChannelsMap with channelsMgr ( #5453 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
607a8ffa1f
Update WatchDmChannels signature ( #5447 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
neza2017
sunby
XuanYang-cn
congqixia
godchen
Cai Yudong
dragondriver
Xiangyu Wang