milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,115 Commits
156 Branches
125 Tags
591 MiB
Commit Graph

1103 Commits (81d3546b7ce83f62850001795071a1a7a8bbe682)

Author SHA1 Message Date
cai.zhang 81d3546b7c Remove indexnode in indexservice when indexnode crashed (#5719) 
* Remove indexnode in indexservice when indexnode crashed

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:36:36 +08:00
cai.zhang 561897c3d0 Load nodes when indexservices started (#5740) 
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong 17b19233dc add BroadcastMany, send DdMsg to collection channels only (#5734) 
* add BroadcastMany, send DdMsg to collection channels only

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix time tick of masterservice

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix time tick unittest of masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* skip dataservice ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SendDdXXXReq interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back unecessary change

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:36:36 +08:00
sunby ba9952e780 Fix unit test in dataservice (#5745) 
After a unit test, the msg stream will be closed. But in pulsar go
client, the connection between client and server will be reconnected.
Then we rebuild the msgstream immediately and produce a message, this
message will be sent to the previous connection which cause timeout of
test. We append the channelName wil a random value.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:36:36 +08:00
zhenshan.cao 89c0acd8b8 Fix bug: describe_collection not return fail reason (#5742) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
zhenshan.cao f74ea3beee [skip ci] Remove ProxyService (#5738) 
* [skip ci] Remove ProxyService

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Change helm branch

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: paramstable crashed

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:35 +08:00
sunby 8a0335be44 Fix dd node filter bug (#5736) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:24:49 +08:00
zhenshan.cao 33a43f48c8 [skip ci] Fix bug: Search timeout and refactor client init logic (#5735) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:24:48 +08:00
XuanYang-cn ab2fd34a2f fix double messages bug in datanode recovery (#5733) 
* fix double messages bug in datanode recovery

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* remove debug log

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:14:29 +08:00
neza2017 ac9dde7352 fix seek on pulsar msgstream (#5726) 
* fix seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* test seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* msg-seek test test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add pure pulsar client seek test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:14:29 +08:00
Cai Yudong fd07d544c3 [skip ci]add timerecord for debug (#5729) 
* add timerecord for debug

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update comments

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:14:29 +08:00
zhenshan.cao 6ec2074da4 [skip ci] Fix bug: concurrent map read and map write (#5713) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:14:29 +08:00
xige-16 f2e551e9a0 Add asConsumer before seek (#5706) 
* add asConsumer before seek

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* get dmChannel from master

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* use pChannel instead in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-15 16:14:27 +08:00
zhenshan.cao 0ddd13aeb3 [skip ci] Fix bug: tick timestamp always zero (#5711) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia 7d9f299ff6 Change default unregister policy (#5710) 
* Change default unassign policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix gofmt

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia a3788a99c5 Fix unregister node info has no channel (#5699) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia a3679d5540 Add vchannel buffer for cluster (#5691) 
* Add channel buffer for cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Change default register policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
XuanYang-cn 46699c4c2b fix datanode dropcollection error (#5690) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:06:13 +08:00
neza2017 351d87055a fix ttmsgstream (#5689) 
* fix msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix asconsumer

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia c172af78bd Add dataservice state check (#5687) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
neza2017 1727edb5d3 fix proxy node (#5686) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby 189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
XuanYang-cn e7fd3aa926 Flowgraph recovery (#5640) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia f794ceb0cc Add segment info rand logic in test case (#5639) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby e9b282c0be Use ElementsMatch to test slice equality (#5641) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia e57e2f77de Add flush monitor and unit test (#5622) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
XuanYang-cn ac19711d74 Destroy DataNode when drop collections (#5638) 
* Destroy DataNode when drop collections

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* golanci-lint

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby f054fc9be2 Add GetRecoveryInfo in dataservice (#5629) 
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017 74c73fa75f fix datanode (#5635) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017 7064937dc3 fix insertBuffer node (#5632) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017 0c362a8831 fix bug and add unit test for insert buffer (#5630) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017 3c70675313 data node, save binlog to minIO, and let data service save these meta (#5618) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen 91ef35bad4 Change receive msg logic (#5605) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017 0a82c6381f add auto flush for data node (#5609) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby 3a5c8c4d3a Add seek position to WatchDmChannels response (#5601) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
Cai Yudong a948b71e06 Support to send dd msg to all dml channels (#5597) 
* send dd msg to dml channel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby e7521afed5 Change SaveBinlogPath (#5576) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia d0d845805b Change WatchDmChannelsRequest proto (#5577) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby aa8a038305 Rename SegmentAllocator to SegmentManager (#5559) 
Add numRows to segmentStatus and Rename SegmentAllocator to
SegmentManager. Remove SegmentAllocStats.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen 275881dbdb Remove address param (#5556) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:46 +08:00
congqixia 41794ec3a6 Change SaveBinLogPath proto & func (#5551) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
neza2017 cb6ae3678a fix master service ut (#5555) 
* fix master unitttest

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* tt

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn e96d39bf6a refactor datanode flowgraph (#5538) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby f1ccbb8f9a Fix init session in dataservice (#5522) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
neza2017 5966076880 fix data service (#5535) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
dragondriver 31b400a9f7 Refactor the workflow of receiving search result from query node (#5527) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-15 15:59:04 +08:00
godchen 54ab03e28f Fix queryservice connect error (#5526) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
dragondriver fca3eb7126 Add implementation of getMinTsStatistics function (#5524) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia 62eaa0390a Add dataservice metrics with grpc prometheus middleware (#5523) 
* Add dataservice metrics with grpc prometheus middleware

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* register server metrics

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby 38b1f7dabe Change the logic of getting vchannel position (#5502) 
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00