milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,588 Commits
156 Branches
125 Tags
589 MiB
Commit Graph

7588 Commits (ec22185ff4a660018f9d5c86b5ae9ceed06660f1)

Author SHA1 Message Date
neza2017 71ed21f9bf
Pass smoke test () 
* pass smoke test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-16 16:59:50 +08:00
bigsheeper d6f7fbf19f
release collection synchronously () 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-16 15:31:57 +08:00
bigsheeper bf39644457
Fix channels in query node and fix search () 
* fix add channel error

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix tSafe vChannel key

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-16 12:05:56 +08:00
godchen 1c6786f85c
Add blob info () 
* Add blob info

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-16 12:03:57 +08:00
xige-16 97049ce293
use dataservice's getRecoveryInfo to get load and watch info () 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-16 11:09:56 +08:00
Cai Yudong 587ccc0557
Optimize dml_channels () 
* update timetickSync::UpdateTimeTick

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* remove ProduceAll and BroadcastAll

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 21:11:58 +08:00
yukun 5c18138a6b
Fix retrieve_collection return nil vchannel () 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-15 20:07:30 +08:00
bigsheeper 58bf92a407
Add vChannel,pChannel in query node and get vChannel from master in query service () 
* add vChannel,pChannel in query node and get vChannel from master in query service

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-06-15 20:06:10 +08:00
sunby d997dd20af
Modify GetRecoveryInfo logic () 
To satisfy QueryNode's request, GetRecoveryInfo interface should return
unflushed segments with start position. Because of using the same code
for getting seek position in QueryNode and DataNode before, we add a
flag to differentiate.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 19:23:55 +08:00
XuanYang-cn 75f2b3c2ba
Add Unittest for allocator () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 19:11:55 +08:00
godchen c2ecce61c0
Make pulsar client singleton () 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 19:03:56 +08:00
neza2017 96cb24e566
fix session util () 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 18:53:55 +08:00
sunby 324f5d1df7
Fix memory leak in indexservice () 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 18:25:55 +08:00
zhenshan.cao ebe5073749 Update proto 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:56:09 +08:00
sunby 9246c663fc Add start position in segment () 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:39:01 +08:00
cai.zhang 1d4ff7c8d7 Add waitgroup for starting server () 
* Add waitgroup for starting server

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>

* [skip ci] Just for skip ci

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:39:01 +08:00
cai.zhang cf8953a347 Add log for indexservice () 
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:39:01 +08:00
yukun fceba63a33 Add segment and channel msgs in retrieve results () 
* Fix query hang bug ()

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add segment and channel msgs in retrieve results

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-15 16:38:59 +08:00
zhenshan.cao 5c69c6902d Refactor proxy channelTimeTick and fix search timeout () 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array () 2021-06-15 16:36:36 +08:00
XuanYang-cn 5bcb94294c Set start positions in datanode () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
congqixia 7beab6eddc Fix new miniokv retry logic () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:36:36 +08:00
XuanYang-cn dea0a4e522 Idempotent of FlushSegment () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
cai.zhang 81d3546b7c Remove indexnode in indexservice when indexnode crashed () 
* Remove indexnode in indexservice when indexnode crashed

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:36:36 +08:00
cai.zhang 561897c3d0 Load nodes when indexservices started () 
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong 17b19233dc add BroadcastMany, send DdMsg to collection channels only () 
* add BroadcastMany, send DdMsg to collection channels only

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix time tick of masterservice

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix time tick unittest of masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* skip dataservice ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SendDdXXXReq interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back unecessary change

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:36:36 +08:00
sunby ba9952e780 Fix unit test in dataservice () 
After a unit test, the msg stream will be closed. But in pulsar go
client, the connection between client and server will be reconnected.
Then we rebuild the msgstream immediately and produce a message, this
message will be sent to the previous connection which cause timeout of
test. We append the channelName wil a random value.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:36:36 +08:00
zhenshan.cao 89c0acd8b8 Fix bug: describe_collection not return fail reason () 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
zhenshan.cao f74ea3beee [skip ci] Remove ProxyService () 
* [skip ci] Remove ProxyService

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Change helm branch

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: paramstable crashed

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:35 +08:00
sunby 8a0335be44 Fix dd node filter bug () 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:24:49 +08:00
zhenshan.cao 33a43f48c8 [skip ci] Fix bug: Search timeout and refactor client init logic () 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:24:48 +08:00
XuanYang-cn ab2fd34a2f fix double messages bug in datanode recovery () 
* fix double messages bug in datanode recovery

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* remove debug log

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:14:29 +08:00
neza2017 ac9dde7352 fix seek on pulsar msgstream () 
* fix seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* test seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* msg-seek test test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add pure pulsar client seek test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:14:29 +08:00
Cai Yudong fd07d544c3 [skip ci]add timerecord for debug () 
* add timerecord for debug

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update comments

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:14:29 +08:00
zhenshan.cao 6ec2074da4 [skip ci] Fix bug: concurrent map read and map write () 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:14:29 +08:00
xige-16 f2e551e9a0 Add asConsumer before seek () 
* add asConsumer before seek

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* get dmChannel from master

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* use pChannel instead in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-06-15 16:14:27 +08:00
zhenshan.cao 0ddd13aeb3 [skip ci] Fix bug: tick timestamp always zero () 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia 7d9f299ff6 Change default unregister policy () 
* Change default unassign policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix gofmt

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia a3788a99c5 Fix unregister node info has no channel () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
cai.zhang f1bb02f724 Stop indexnode when milvus process stops () 
Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia a3679d5540 Add vchannel buffer for cluster () 
* Add channel buffer for cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Change default register policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
XuanYang-cn 46699c4c2b fix datanode dropcollection error () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:06:13 +08:00
neza2017 351d87055a fix ttmsgstream () 
* fix msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix asconsumer

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia c172af78bd Add dataservice state check () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
neza2017 1727edb5d3 fix proxy node () 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby 189ac881f3 Fix bugs () 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
XuanYang-cn e7fd3aa926 Flowgraph recovery () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia f794ceb0cc Add segment info rand logic in test case () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby e9b282c0be Use ElementsMatch to test slice equality () 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia e57e2f77de Add flush monitor and unit test () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00