Commit Graph

349 Commits (6df613f64874bdd58a2ff28c2ffeaf061f60d5c0)

Author SHA1 Message Date
sunby c863a193de
Rename filename from dataservice to datacoordinator (#5946)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 10:42:07 +08:00
godchen 8643127e99
Change grpc retry (#5925)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-22 09:46:06 +08:00
sunby 7f2d7c1da3
Rename dataservice to datacoordinator (#5912)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 18:22:13 +08:00
sunby 0300b682f8
Remove deprecated interfaces in data coordinator (#5929)
* Remove derecated interfaces in data coordinator

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove RegisterNode in data coordinator

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 17:40:05 +08:00
Cai Yudong 92e429d812
Rename IndexService to IndexCoord (#5932)
* rename package indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix queryservice static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move distributed/indexservice to distributed/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move internal/indexservice to internal/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename MasterComponent to RootCoordComponent

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for queryservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for dataservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for datanode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for proxynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for querynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix rebase issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-21 17:28:03 +08:00
dragondriver 2642415bc3
Redefine the auto_id and primary_key (#5903)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-21 11:42:18 +08:00
sunby da1f9f9241
Rename dataservice to datacoordinator (#5904)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 11:40:15 +08:00
xige-16 e4c51aae36
Add recovery logic for querynode and queryservice (#5843)
* merge milvus/recovery2

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add recovery logic in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug smoke case

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add etcd to querynode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix release partition error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug querynode down and recovery

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add log

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix showCollection

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* skip smoke test search without insert

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-19 11:45:09 +08:00
Cai Yudong 5a1f7e4feb
rename masterservice to rootcoord (#5875)
* rename masterservice to rootcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename package

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename masterservice to rootcoord under distributed

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update scripts

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable more unittests

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-18 21:30:08 +08:00
bigsheeper 710ac1a4f1
remove query node id env dependency and fix start script (#5874)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-06-18 19:00:15 +08:00
congqixia 2f8bccda8d
Improve help message (#5870)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-18 16:16:06 +08:00
dragondriver 676a30e418
Use channelsMgr to manage search message stream (#5819)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-18 10:33:58 +08:00
neza2017 b2908c4780
Release collection (#5839)
* relese collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* gen-proto

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* release collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* release collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-17 17:45:56 +08:00
Cai Yudong ae3daff5e4
Rename Master to RootCoord (#5830)
* rename master_service.go to root_coordinator.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coordinator under cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord under cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix metrics unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-17 16:47:57 +08:00
godchen b7bf26b486
Update grpc trace (#5817)
* Update grpc trace

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove useless code in mod

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-17 14:17:56 +08:00
congqixia 5ce06de3d1
Add context usage and fix defer issue (#5796)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-16 18:35:59 +08:00
zhenshan.cao 5c69c6902d Refactor proxy channelTimeTick and fix search timeout (#5760)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:36 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
Cai Yudong 17b19233dc add BroadcastMany, send DdMsg to collection channels only (#5734)
* add BroadcastMany, send DdMsg to collection channels only

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix time tick of masterservice

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix time tick unittest of masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* skip dataservice ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SendDdXXXReq interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back unecessary change

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:36:36 +08:00
zhenshan.cao f74ea3beee [skip ci] Remove ProxyService (#5738)
* [skip ci] Remove ProxyService

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Change helm branch

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: paramstable crashed

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:35 +08:00
zhenshan.cao 33a43f48c8 [skip ci] Fix bug: Search timeout and refactor client init logic (#5735)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:24:48 +08:00
congqixia 7d9f299ff6 Change default unregister policy (#5710)
* Change default unassign policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix gofmt

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:13 +08:00
sunby 189ac881f3 Fix bugs (#5676)
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
Cai Yudong a948b71e06 Support to send dd msg to all dml channels (#5597)
* send dd msg to dml channel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:04:48 +08:00
godchen 275881dbdb Remove address param (#5556)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:46 +08:00
godchen 54ab03e28f Fix queryservice connect error (#5526)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia 62eaa0390a Add dataservice metrics with grpc prometheus middleware (#5523)
* Add dataservice metrics with grpc prometheus middleware

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* register server metrics

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
Cai Yudong 2cc330b117 Remove proxy service from master (#5510)
* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove time sync

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service from distributed/masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby a14c35274f Fix bugs in dataservice (#5457)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn 55eac3ee7f Complete Flush procedure in DataNode (#5454)
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
bigsheeper cdbc6d2c94
Refactor query node and query service (#5751)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 12:41:40 +08:00
sunby 50b3d39370
Add GetRecoveryInfo interface and proto (#5598)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-07 14:16:36 +08:00
zhenshan.cao 03e3b4beb4
Add log to startup process (#5594)
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-04 16:29:35 +08:00
Cai Yudong 2ba93deb5e
Support master get segment info channel name from config file (#5606)
* optimize msgstream unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* get data service segment info channel name from config file directly

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-04 16:28:34 +08:00
Xiangyu Wang afa5da7df0
Add query rpc into proto (#5592)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-04 12:02:34 +08:00
zhenshan.cao b0b8f58192
Fix bug: search timeout (#5557)
* Fix bug: search timeout

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Add log and fix unittest bug

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-03 14:58:34 +08:00
Xiangyu Wang ca0993c274 Update OWNERS
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-30 18:58:47 +08:00
cai.zhang a606ab9c67
Add power-off restart logic for IndexService and IndexNode (#5395)
Add power-off restart logic for IndexService and IndexNode


Signed-off-by: <xiaocai2333> <cai.zhang@zilliz.com>
2021-05-27 14:24:29 +00:00
godchen 6e3a0d2a8e
Add queryservice grpc retry (#5437)
Add queryservice grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-27 10:44:36 +00:00
godchen 220c12e12f
Add indexservice grpc retry (#5425)
Add indexservice grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
2021-05-27 07:12:13 +00:00
godchen 279d77020a
Change proxy grpc retry (#5431)
Change proxy grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
2021-05-27 07:10:39 +00:00
godchen 9aa1fd25fa
Add indexnode grpc retry (#5418)
Add indexnode grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-27 02:30:11 +00:00
neza2017 f80dbab6ec
watch proxynode (#5424)
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
2021-05-26 12:14:30 +00:00
godchen 5fd92ec80c
Add querynode grpc retry (#5428)
Add querynode grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-26 11:37:34 +00:00
godchen bfa683ce79
Change dataservice grpc retry (#5411)
Change dataservice grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
2021-05-26 10:55:05 +00:00
godchen 5f0f840782
Remove datanode client code (#5427)
Remove datanode client code.

Signed-off-by: godchen qingxiang.chen@zilliz.com
2021-05-26 10:41:37 +00:00
godchen 200801271d
Fix GetSession error (#5401)
Fix GetSession error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-25 12:15:46 +00:00
godchen 17cabfbed3
Add datanode client retry (#5394)
Add datanode client retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-25 07:47:08 +00:00
neza2017 c28c34e852
let master use session.ServerID as nodeID (#5385)
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-05-25 07:06:05 +00:00
Xiangyu Wang 88b42304ef
Add dummy RPC (#5376)
Adding a dummy RPC makes it easier to develop new RPCs in the future.
When developing a new function, the SDK is often implemented last. 
At this time, this is helpful for debugging.

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-25 06:44:43 +00:00
Cai Yudong dab536a6cf
Rename some field names in master for better code readability (#5389)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-25 06:03:06 +00:00
Cai Yudong 4058350e30
update AddIndex not return error when fail (#5372)
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-25 03:42:23 +00:00
Cai Yudong 39f8c9dd56
Save msg start position and end position when AddSegment and AddIndex (#5358)
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-24 06:19:52 +00:00
neza2017 e326eaad1e
master client,reconnect if grpc failed (#5360)
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-05-24 01:44:49 +00:00
godchen 0eee7e5252
Change session (#5329)
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-21 11:28:52 +00:00
Cai Yudong 761350e44d
Refactor timetick msg handling between proxy node and master (#5321)
Proxy node send channel timetick msg to master, master reduce timetick
msg for each channel and send the min timetick to msg stream.

Resolves: #5276 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-21 08:08:12 +00:00
neza2017 c0daf8e41d
set timestamp on dd requests (#5298)
- set dd request's time tick on master
- send time tick into dd channel

See also: #5291

Signed-off-by: yefu.chen yefu.chen@zilliz.com
2021-05-20 06:14:14 +00:00
congqixia 38f5b0826f
Add SaveBinlogPaths grpc in dataservice (#5277)
* Add SaveBinlogPaths grpc in dataservice

* fix merged compile error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-05-20 11:34:45 +08:00
yukun a7bb701f73
Add Retrieve method grpc definition (#5294)
See also: #5253 

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-05-19 10:45:15 +00:00
godchen e224d1e725
Add SessionManager (#5288)
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-19 10:36:05 +00:00
neza2017 49f6542b1d
query master service's with timestamp (#5275)
let meta support snapshot
so collection could query meta with timestamp

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-05-18 06:18:02 +00:00
Cai Yudong 001795ee70
Optimize CreateIndex routine in master service (#5238)
Remove CreateIndexTask struct and indexTaskQueue

Resolves: #5228 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-15 10:08:08 +00:00
Cai Yudong 08bb1b2ec3
Resend unsuccessful ddMsg when master start (#5214)
Resend unsuccessful ddMsg when master start

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-14 21:26:06 +08:00
Cai Yudong 5f0006d0f9
Improve code coverage for distributed/datanode (#5133)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-08 06:07:04 +00:00
Cai Yudong d66a5acca3
improve code coverage for distributed/dataservice (#5123)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
improve code coverage for distributed/dataservice
2021-05-07 07:43:52 +00:00
xige-16 1e7205fa03
fix querynode's port not set correctly when conflict with other node (#5066)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-05-07 12:03:53 +08:00
XuanYang-cn cebdfb34f7
Enable multiple datanode (#5068)
Now it's able to start multiple datanode, however, they will process the same insert data.
Further changes are needed to truly enable multiple datanode:

While registering, dataservice need to return different insert channels for different datanode.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-04-28 18:43:48 +08:00
godchen 060f7ca0d2
Add get index build progress proto (#5049)
Add get index build progress proto.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-04-27 07:46:45 +00:00
neza2017 262c484cec
the return struct of "DescribeIndex" should contain "filed_name" (#5055)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-04-27 02:30:55 +00:00
neza2017 d7c0a72611
index name is unique in collection scope (#5046)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-04-26 15:53:47 +08:00
neza2017 ada58b5bde
add unit test for master's grpc server, the coverage of package "internal/distributed/masterservice" increase from 46.6% to 82.9% (#5019)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-04-24 09:23:35 +00:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988)
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
sunby b79a408491 Remove unused config item in dataservice and datanode
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-19 19:28:45 +08:00
cai.zhang 9ccaff8005 Add copyright
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-04-19 19:28:11 +08:00
XuanYang-cn 6deba1faa5 Add license in datanode package
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-04-19 15:16:33 +08:00
cai.zhang 8a037a404e Add Copyright for indexnode and indexservice
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-04-19 13:50:12 +08:00
bigsheeper 71b8f309b2 Add licence to query node and query service
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-04-19 13:47:10 +08:00
neza2017 83647e8503 Add copyright in master service module
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-04-19 11:12:56 +08:00
dragondriver dc7db73251 Add copyright annotations to proxynode and proxyservice
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-04-19 10:09:43 +08:00
sunby 751c9d4e81 Refactor meta.go
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-09 09:55:04 +08:00
Xiangyu Wang 87a1e0b662 Reorganize msgstream
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-02 13:48:25 +08:00
zhenshan.cao efbf4452c5 Add log for port changed or proto unmarshal error
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-30 20:19:30 +08:00
Xiangyu Wang 72917467ec Change singlenode logs output
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-03-30 10:52:42 +08:00
neza2017 71614313b0 Fix master service
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-29 09:50:52 +08:00
neza2017 eca1e25470 Add msg id at master service
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-26 15:13:33 +08:00
bigsheeper 1634d75980 Release flowgraph properly
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-03-22 12:49:50 -05:00
dragondriver 49c6eeb052 Add useful debug log in proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-03-22 19:28:43 +08:00
zhenshan.cao c2734fa55f Fix bug and enchance system
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-22 16:36:10 +08:00
neza2017 6eb45c92eb Fix flush segment
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-13 17:05:36 +08:00
groot 47d9471498 Change gracefultime to 1 second
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-03-13 14:42:53 +08:00
cai.zhang 012f5f360e Add init log logic for index and proxy
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-03-12 19:47:37 +08:00
godchen f3649f0419 Refactor interface and proto
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
ThreadDao ca916a5c47 Add ci nightly
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2021-03-11 14:14:29 +08:00
xige-16 09ae985daa Fix wrong error code in master_service_test.go
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-10 22:06:22 +08:00
sunby 800e31ebe4 Fix grpc startup in query node
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-10 15:49:23 +08:00
del-zhenwu b1763c3690 Disable xdist in test cases
Signed-off-by: del-zhenwu <zhenxiang.li@zilliz.com>
2021-03-10 15:27:26 +08:00
neza2017 7b2d67242d Fix describe segment if index not exist
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-10 14:45:35 +08:00
cai.zhang 60b97f9b65 Replace log of indexnode and indexservice
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-03-10 09:56:09 +08:00
quicksilver 455cc59dbb Update publish-test-images.yaml
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-09 13:55:35 +08:00
neza2017 55fc3f56aa Set grpc max recv mag as math.MaxInt32
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-08 15:49:42 +08:00
neza2017 014c4fe8ce Add config of minSegmentSizeToEnableIndex
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-08 15:46:51 +08:00
sunby aeb3038545 Remove GetCount in data service
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-08 10:49:44 +08:00
ThreadDao 7a3223dc81 tags collection stats
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2021-03-08 10:09:48 +08:00
neza2017 3ef18e86f0 Add 0331 tag for test_partition
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-06 17:47:11 +08:00
neza2017 2835bcdc4e Update masterservice unittest
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-05 20:41:34 +08:00
quicksilver 74154a11a4 Fix deploy error during regression stage
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-05 16:52:45 +08:00
BossZou 1efcf605e2 Enlarge Proxy node max size of reveivable grpc message
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-03-05 11:16:23 +08:00
sunby ddddd65d10 Delete internal/errors package
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-05 10:15:27 +08:00
bigsheeper 447a15207e Add zap log to query node
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-03-05 09:21:35 +08:00
dragondriver 801396e3ef Disable register link logic, use proxy node directly
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-03-04 22:27:12 +08:00
sunby dec0b40a66 Add log panic in log module
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-04 16:01:30 +08:00
godchen 3ba0dc2f44 Change doc
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-04 10:35:28 +08:00
bigsheeper d3db2b0d82 Add zap log in query service
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-27 10:11:52 +08:00
godchen 7b9fdd7f29 Add opentracing
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-26 17:44:24 +08:00
xige-16 50a5f97421 Delete componentStateResponse in queryService.proto
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-26 15:26:21 +08:00
neza2017 4ea27335fd Ignore index param from create_collection
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-26 14:40:53 +08:00
XuanYang-cn 10374f15d7 Update log in datanode
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-26 10:13:36 +08:00
Cai Yudong 84110d2684 Add tests/benchmark and tests/python_test using new python SDK
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2021-02-25 17:35:36 +08:00
cai.zhang 1d899513dd Unify service name in config file
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-24 18:46:19 +08:00
bigsheeper 08a020798e Release collection and partitions
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-24 17:24:51 +08:00
neza2017 e396c3f4ad Add zap log
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-24 16:25:40 +08:00
XuanYang-cn fd562f9f9c Update doc: add proxy graph
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-24 09:48:17 +08:00
zhenshan.cao 7c5e2d89f6 Read node port from config
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-23 18:08:17 +08:00
xige-16 d5379a82a8 Fix load binlog error when segment not flushed
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-23 14:13:33 +08:00
dragondriver b506754364 Add workflow of drop index in index service
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-23 11:57:18 +08:00
zhenshan.cao bbfcbbdd68 Fix ci: paramstable and startup logic refactor
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-23 11:40:30 +08:00
sunby ea1d9ea99a Replace log in dataservice
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-23 09:58:06 +08:00
BossZou 7ac1821cf1 Add RocksMQ throughout test
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
2021-02-20 18:30:37 +08:00
yukun 346b0a9549 Add unittest for RmqTtMsgStream
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-02-20 15:58:19 +08:00
neza2017 1d3fdebcf4 Add drop index for proxy
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-20 15:38:44 +08:00
cai.zhang 4c5a898e9a Add masterservice docker
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-20 11:15:19 +08:00
dragondriver 6ebb03791f Fix drop index interface
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-20 10:37:11 +08:00
zhenshan.cao b3891ca199 Correct docker file
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-20 10:15:37 +08:00
xige-16 450ea631bc Fix queryService assign queryChannel failure
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-18 16:26:02 +08:00
neza2017 85c1404393 Refactor master:index ids
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-11 08:41:59 +08:00
xige-16 70fe5233cf Watch queryChannel on query node
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-09 17:09:26 +08:00
neza2017 e2d8358cb4 Refactor master:index ids
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-09 13:11:55 +08:00
neza2017 4035083188 Support Index name
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-08 14:49:12 +08:00
groot dc3736281a Integrate message stream
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-08 14:30:54 +08:00
sunby eba5b9ae27 Fix queryservice client init bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-07 13:53:40 +08:00
groot b034740c6c Make paramstable initialize once by using do_once
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-06 13:39:15 +08:00
sunby 0111cba8ae Fix startup bugs in some modules
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-06 10:28:53 +08:00
XuanYang-cn dd72dfcbd1 Fix queryservice start
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-05 17:57:41 +08:00
dragondriver e6aec3fd08 Hardcode the peer id of proxy node
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-05 14:09:55 +08:00
godchen 0f620ab149 Fix invalidate metacache error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-05 11:49:13 +08:00
dragondriver e384222803 Fix the logic of node id allocator
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-05 11:25:52 +08:00
neza2017 bea1535425 Add call of release collection when drop collection
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-02-05 11:05:37 +08:00
dragondriver a48590a2df Add query service logic to Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-02-05 10:29:14 +08:00
XuanYang-cn a64c4efde9 Refactor collectionReplica in datanode
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-04 20:31:23 +08:00
godchen c2ca2c276f Fix response check error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-04 19:34:35 +08:00