96cb24e566
fix session util ( #5780 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 18:53:55 +08:00
324f5d1df7
Fix memory leak in indexservice ( #5779 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 18:25:55 +08:00
ebe5073749
Update proto
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:56:09 +08:00
9246c663fc
Add start position in segment ( #5753 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:39:01 +08:00
1d4ff7c8d7
Add waitgroup for starting server ( #5705 )
...
* Add waitgroup for starting server
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
* [skip ci] Just for skip ci
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-06-15 16:39:01 +08:00
cf8953a347
Add log for indexservice ( #5754 )
...
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-06-15 16:39:01 +08:00
fceba63a33
Add segment and channel msgs in retrieve results ( #5750 )
...
* Fix query hang bug (#5709 )
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Add segment and channel msgs in retrieve results
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-15 16:38:59 +08:00
5c69c6902d
Refactor proxy channelTimeTick and fix search timeout ( #5760 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:36 +08:00
0f4bd50ce3
Support ETCD endpoints array ( #5755 )
2021-06-15 16:36:36 +08:00
5bcb94294c
Set start positions in datanode ( #5757 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
7beab6eddc
Fix new miniokv retry logic ( #5749 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:36:36 +08:00
dea0a4e522
Idempotent of FlushSegment ( #5746 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
81d3546b7c
Remove indexnode in indexservice when indexnode crashed ( #5719 )
...
* Remove indexnode in indexservice when indexnode crashed
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-06-15 16:36:36 +08:00
561897c3d0
Load nodes when indexservices started ( #5740 )
...
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-06-15 16:36:36 +08:00
17b19233dc
add BroadcastMany, send DdMsg to collection channels only ( #5734 )
...
* add BroadcastMany, send DdMsg to collection channels only
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix master unittest
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix time tick of masterservice
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix time tick unittest of masterservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* skip dataservice ut
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix format issue
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update timeticksync
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update timeticksync
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update SendDdXXXReq interface
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update master unittest
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* roll back unecessary change
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* enable ut
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:36:36 +08:00
ba9952e780
Fix unit test in dataservice ( #5745 )
...
After a unit test, the msg stream will be closed. But in pulsar go
client, the connection between client and server will be reconnected.
Then we rebuild the msgstream immediately and produce a message, this
message will be sent to the previous connection which cause timeout of
test. We append the channelName wil a random value.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:36:36 +08:00
89c0acd8b8
Fix bug: describe_collection not return fail reason ( #5742 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:36 +08:00
f74ea3beee
[skip ci] Remove ProxyService ( #5738 )
...
* [skip ci] Remove ProxyService
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
* Change helm branch
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
* Fix bug: paramstable crashed
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:35 +08:00
8a0335be44
Fix dd node filter bug ( #5736 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:24:49 +08:00
33a43f48c8
[skip ci] Fix bug: Search timeout and refactor client init logic ( #5735 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:24:48 +08:00
ab2fd34a2f
fix double messages bug in datanode recovery ( #5733 )
...
* fix double messages bug in datanode recovery
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* remove debug log
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:14:29 +08:00
ac9dde7352
fix seek on pulsar msgstream ( #5726 )
...
* fix seek on pulsar msgstream
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* test seek on pulsar msgstream
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* msg-seek test test
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* add pure pulsar client seek test
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:14:29 +08:00
fd07d544c3
[skip ci]add timerecord for debug ( #5729 )
...
* add timerecord for debug
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update comments
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:14:29 +08:00
6ec2074da4
[skip ci] Fix bug: concurrent map read and map write ( #5713 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:14:29 +08:00
f2e551e9a0
Add asConsumer before seek ( #5706 )
...
* add asConsumer before seek
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* get dmChannel from master
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* use pChannel instead in queryservice
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-15 16:14:27 +08:00
0ddd13aeb3
[skip ci] Fix bug: tick timestamp always zero ( #5711 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:06:13 +08:00
7d9f299ff6
Change default unregister policy ( #5710 )
...
* Change default unassign policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix gofmt
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:13 +08:00
a3788a99c5
Fix unregister node info has no channel ( #5699 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:13 +08:00
f1bb02f724
Stop indexnode when milvus process stops ( #5694 )
...
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-06-15 16:06:13 +08:00
a3679d5540
Add vchannel buffer for cluster ( #5691 )
...
* Add channel buffer for cluster
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Change default register policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:13 +08:00
46699c4c2b
fix datanode dropcollection error ( #5690 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:06:13 +08:00
351d87055a
fix ttmsgstream ( #5689 )
...
* fix msgstream
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix asconsumer
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:06:13 +08:00
c172af78bd
Add dataservice state check ( #5687 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:13 +08:00
1727edb5d3
fix proxy node ( #5686 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:06:13 +08:00
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
e7fd3aa926
Flowgraph recovery ( #5640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
f794ceb0cc
Add segment info rand logic in test case ( #5639 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
e9b282c0be
Use ElementsMatch to test slice equality ( #5641 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
e57e2f77de
Add flush monitor and unit test ( #5622 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
ac19711d74
Destroy DataNode when drop collections ( #5638 )
...
* Destroy DataNode when drop collections
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* golanci-lint
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
f054fc9be2
Add GetRecoveryInfo in dataservice ( #5629 )
...
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
74c73fa75f
fix datanode ( #5635 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
7064937dc3
fix insertBuffer node ( #5632 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0c362a8831
fix bug and add unit test for insert buffer ( #5630 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3c70675313
data node, save binlog to minIO, and let data service save these meta ( #5618 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
91ef35bad4
Change receive msg logic ( #5605 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0a82c6381f
add auto flush for data node ( #5609 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3a5c8c4d3a
Add seek position to WatchDmChannels response ( #5601 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
a948b71e06
Support to send dd msg to all dml channels ( #5597 )
...
* send dd msg to dml channel
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add dml_channels.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix race
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:04:48 +08:00
e7521afed5
Change SaveBinlogPath ( #5576 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
neza2017
sunby
zhenshan.cao
cai.zhang
yukun
Cai Yudong
XuanYang-cn
congqixia
xige-16
godchen