0300b682f8
Remove deprecated interfaces in data coordinator ( #5929 )
...
* Remove derecated interfaces in data coordinator
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove RegisterNode in data coordinator
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 17:40:05 +08:00
9246c663fc
Add start position in segment ( #5753 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:39:01 +08:00
5bcb94294c
Set start positions in datanode ( #5757 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
f054fc9be2
Add GetRecoveryInfo in dataservice ( #5629 )
...
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
3a5c8c4d3a
Add seek position to WatchDmChannels response ( #5601 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
e7521afed5
Change SaveBinlogPath ( #5576 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
41794ec3a6
Change SaveBinLogPath proto & func ( #5551 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
38b1f7dabe
Change the logic of getting vchannel position ( #5502 )
...
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
607a8ffa1f
Update WatchDmChannels signature ( #5447 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
ef563fda8f
Updata flushe procedure
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
c53afee616
DataNode scales flowgraph
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
81ac20143b
Add persistency of datanode cluster ( #5387 )
...
We save channels registered in datanode in etcd and restore the cluster info after
restarting.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
3662b3f0e5
Refactor dataservice ( #5400 )
...
Pick up grpc calls to grpc_handler.go and refactor code format
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:01 +08:00
50b3d39370
Add GetRecoveryInfo interface and proto ( #5598 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-07 14:16:36 +08:00
4aa6a8fe18
Dataservice use lastest grpc design and saves ddl/dml pos pair ( #5348 )
...
Dataservice use lastest grpc design and saves ddl/dml pos pair
* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)
See also #5220
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-05-21 11:38:58 +00:00
02f2348500
Save segment allocation status to meta ( #5184 )
...
To ensure segments being flushed correctly, we need to load
the segment allocation status from meta after starting the service.
2021-05-21 10:30:41 +00:00
c1efa53779
Remove redundant fields in SegmentInfo and related structure ( #5280 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-05-21 06:51:25 +00:00
38f5b0826f
Add SaveBinlogPaths grpc in dataservice ( #5277 )
...
* Add SaveBinlogPaths grpc in dataservice
* fix merged compile error
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-05-20 11:34:45 +08:00
82ccd4cec0
Rename module ( #4988 )
...
* Rename module
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
751c9d4e81
Refactor meta.go
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-09 09:55:04 +08:00
4c98f4efbf
Update binlog meta proto
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-20 09:50:42 +08:00
f3649f0419
Refactor interface and proto
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
ca916a5c47
Add ci nightly
...
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2021-03-11 14:14:29 +08:00
aeb3038545
Remove GetCount in data service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-08 10:49:44 +08:00
1aafe86ffe
Change MsgPosition Logic
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-04 11:19:48 +08:00
1b616fb874
Add persistent_segmentinfo
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-03 18:55:00 +08:00
f3aad3a71c
Change SegmentInfo
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-02 18:53:10 +08:00
ca9c2f5c12
Add GetCount and implement GetCollectionStatistics
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-02 14:25:58 +08:00
3afa7dd50d
Fix GetSegmentStates
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-02 10:58:39 +08:00
dbfe68512d
Add dbID to data service proto
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-27 10:33:17 +08:00
1fe099775d
Refactor data service
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-25 15:17:17 +08:00
e89e89b894
Add main for masterservice
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-25 12:03:44 +08:00
54fcd486bb
Refactor data service.
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-25 13:45:56 +08:00
9fe672ab86
Enchance datanode interface
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-24 21:20:11 +08:00
50e2369000
Change name and tag to ID
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 19:36:09 +08:00
90a0b8b2f5
Update proto
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-22 11:44:02 +08:00
ef79b8d8c7
Add client and grpc service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-22 11:07:07 +08:00
377ee8687d
Modify flush logic
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 09:36:40 +08:00
2ec573d2f9
Recover delete docs
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-20 09:36:50 +08:00
fb493cc235
Update interface name of GetIndexStates
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-19 18:32:57 +08:00
918e279869
Set msg stream
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-19 16:14:13 +08:00
76a7684d96
Add data service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-19 12:10:49 +08:00
d22c2cd203
Add proto for query service loading collection
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-18 19:32:08 +08:00
37688559f1
Fix multi rowdata writenode crash bug
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-16 15:06:19 +08:00
67e6ea8051
Add data service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-15 17:09:41 +08:00
6e36f10a26
Fix rocksdb illegal instruction in some machine
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-14 14:24:14 +08:00
sunby
XuanYang-cn
congqixia
congqixia
Xiangyu Wang
godchen
ThreadDao
zhenshan.cao
neza2017
cai.zhang
xige-16
yukun