milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,480 Commits
149 Branches
120 Tags
577 MiB
Commit Graph

388 Commits (223c330ed5be7b711cae899383ee26dff444ff8d)

Author SHA1 Message Date
dragondriver 676a30e418
Use channelsMgr to manage search message stream (#5819) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-18 10:33:58 +08:00
neza2017 b2908c4780
Release collection (#5839) 
* relese collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* gen-proto

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* release collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* release collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-17 17:45:56 +08:00
Cai Yudong ae3daff5e4
Rename Master to RootCoord (#5830) 
* rename master_service.go to root_coordinator.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coordinator under cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord under cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix metrics unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-17 16:47:57 +08:00
godchen b7bf26b486
Update grpc trace (#5817) 
* Update grpc trace

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove useless code in mod

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-17 14:17:56 +08:00
congqixia 5ce06de3d1
Add context usage and fix defer issue (#5796) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-16 18:35:59 +08:00
zhenshan.cao 5c69c6902d Refactor proxy channelTimeTick and fix search timeout (#5760) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
Cai Yudong 17b19233dc add BroadcastMany, send DdMsg to collection channels only (#5734) 
* add BroadcastMany, send DdMsg to collection channels only

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix time tick of masterservice

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix time tick unittest of masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* skip dataservice ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SendDdXXXReq interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back unecessary change

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:36:36 +08:00
zhenshan.cao f74ea3beee [skip ci] Remove ProxyService (#5738) 
* [skip ci] Remove ProxyService

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Change helm branch

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: paramstable crashed

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:35 +08:00
zhenshan.cao 33a43f48c8 [skip ci] Fix bug: Search timeout and refactor client init logic (#5735) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:24:48 +08:00
congqixia 7d9f299ff6 Change default unregister policy (#5710) 
* Change default unassign policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix gofmt

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby 189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
Cai Yudong a948b71e06 Support to send dd msg to all dml channels (#5597) 
* send dd msg to dml channel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen 275881dbdb Remove address param (#5556) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:46 +08:00
godchen 54ab03e28f Fix queryservice connect error (#5526) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia 62eaa0390a Add dataservice metrics with grpc prometheus middleware (#5523) 
* Add dataservice metrics with grpc prometheus middleware

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* register server metrics

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
Cai Yudong 2cc330b117 Remove proxy service from master (#5510) 
* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove time sync

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service from distributed/masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby a14c35274f Fix bugs in dataservice (#5457) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn 55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
bigsheeper cdbc6d2c94
Refactor query node and query service (#5751) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 12:41:40 +08:00
sunby 50b3d39370
Add GetRecoveryInfo interface and proto (#5598) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-07 14:16:36 +08:00
zhenshan.cao 03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
Cai Yudong 2ba93deb5e
Support master get segment info channel name from config file (#5606) 
* optimize msgstream unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* get data service segment info channel name from config file directly

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-04 16:28:34 +08:00
Xiangyu Wang afa5da7df0
Add query rpc into proto (#5592) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-04 12:02:34 +08:00
zhenshan.cao b0b8f58192
Fix bug: search timeout (#5557) 
* Fix bug: search timeout

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Add log and fix unittest bug

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-03 14:58:34 +08:00
Xiangyu Wang ca0993c274 Update OWNERS 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-30 18:58:47 +08:00
cai.zhang a606ab9c67
Add power-off restart logic for IndexService and IndexNode (#5395) 
Add power-off restart logic for IndexService and IndexNode


Signed-off-by: <xiaocai2333> <cai.zhang@zilliz.com>
 2021-05-27 14:24:29 +00:00
godchen 6e3a0d2a8e
Add queryservice grpc retry (#5437) 
Add queryservice grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-27 10:44:36 +00:00
godchen 220c12e12f
Add indexservice grpc retry (#5425) 
Add indexservice grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-27 07:12:13 +00:00
godchen 279d77020a
Change proxy grpc retry (#5431) 
Change proxy grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-27 07:10:39 +00:00
godchen 9aa1fd25fa
Add indexnode grpc retry (#5418) 
Add indexnode grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-27 02:30:11 +00:00
neza2017 f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
godchen 5fd92ec80c
Add querynode grpc retry (#5428) 
Add querynode grpc retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-26 11:37:34 +00:00
godchen bfa683ce79
Change dataservice grpc retry (#5411) 
Change dataservice grpc retry.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-26 10:55:05 +00:00
godchen 5f0f840782
Remove datanode client code (#5427) 
Remove datanode client code.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-26 10:41:37 +00:00
godchen 200801271d
Fix GetSession error (#5401) 
Fix GetSession error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-25 12:15:46 +00:00
godchen 17cabfbed3
Add datanode client retry (#5394) 
Add datanode client retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-25 07:47:08 +00:00
neza2017 c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00
Xiangyu Wang 88b42304ef
Add dummy RPC (#5376) 
Adding a dummy RPC makes it easier to develop new RPCs in the future.
When developing a new function, the SDK is often implemented last. 
At this time, this is helpful for debugging.

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-25 06:44:43 +00:00
Cai Yudong dab536a6cf
Rename some field names in master for better code readability (#5389) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-25 06:03:06 +00:00
Cai Yudong 4058350e30
update AddIndex not return error when fail (#5372) 
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-25 03:42:23 +00:00
Cai Yudong 39f8c9dd56
Save msg start position and end position when AddSegment and AddIndex (#5358) 
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-24 06:19:52 +00:00
neza2017 e326eaad1e
master client,reconnect if grpc failed (#5360) 
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-24 01:44:49 +00:00
godchen 0eee7e5252
Change session (#5329) 
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-21 11:28:52 +00:00
Cai Yudong 761350e44d
Refactor timetick msg handling between proxy node and master (#5321) 
Proxy node send channel timetick msg to master, master reduce timetick
msg for each channel and send the min timetick to msg stream.

Resolves: #5276 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-21 08:08:12 +00:00
neza2017 c0daf8e41d
set timestamp on dd requests (#5298) 
- set dd request's time tick on master
- send time tick into dd channel

See also: #5291

Signed-off-by: yefu.chen yefu.chen@zilliz.com
 2021-05-20 06:14:14 +00:00
congqixia 38f5b0826f
Add SaveBinlogPaths grpc in dataservice (#5277) 
* Add SaveBinlogPaths grpc in dataservice

* fix merged compile error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-20 11:34:45 +08:00
yukun a7bb701f73
Add Retrieve method grpc definition (#5294) 
See also: #5253 

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-05-19 10:45:15 +00:00
godchen e224d1e725
Add SessionManager (#5288) 
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-19 10:36:05 +00:00
neza2017 49f6542b1d
query master service's with timestamp (#5275) 
let meta support snapshot
so collection could query meta with timestamp

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-18 06:18:02 +00:00
Cai Yudong 001795ee70
Optimize CreateIndex routine in master service (#5238) 
Remove CreateIndexTask struct and indexTaskQueue

Resolves: #5228 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-15 10:08:08 +00:00
Cai Yudong 08bb1b2ec3
Resend unsuccessful ddMsg when master start (#5214) 
Resend unsuccessful ddMsg when master start

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-14 21:26:06 +08:00
Cai Yudong 5f0006d0f9
Improve code coverage for distributed/datanode (#5133) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-08 06:07:04 +00:00
Cai Yudong d66a5acca3
improve code coverage for distributed/dataservice (#5123) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
improve code coverage for distributed/dataservice
 2021-05-07 07:43:52 +00:00
xige-16 1e7205fa03
fix querynode's port not set correctly when conflict with other node (#5066) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-05-07 12:03:53 +08:00
XuanYang-cn cebdfb34f7
Enable multiple datanode (#5068) 
Now it's able to start multiple datanode, however, they will process the same insert data.
Further changes are needed to truly enable multiple datanode:

While registering, dataservice need to return different insert channels for different datanode.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-04-28 18:43:48 +08:00
godchen 060f7ca0d2
Add get index build progress proto (#5049) 
Add get index build progress proto.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-27 07:46:45 +00:00
neza2017 262c484cec
the return struct of "DescribeIndex" should contain "filed_name" (#5055) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-27 02:30:55 +00:00
neza2017 d7c0a72611
index name is unique in collection scope (#5046) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-26 15:53:47 +08:00
neza2017 ada58b5bde
add unit test for master's grpc server, the coverage of package "internal/distributed/masterservice" increase from 46.6% to 82.9% (#5019) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-24 09:23:35 +00:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
sunby b79a408491 Remove unused config item in dataservice and datanode 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-19 19:28:45 +08:00
cai.zhang 9ccaff8005 Add copyright 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-04-19 19:28:11 +08:00
XuanYang-cn 6deba1faa5 Add license in datanode package 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-19 15:16:33 +08:00
cai.zhang 8a037a404e Add Copyright for indexnode and indexservice 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-04-19 13:50:12 +08:00
bigsheeper 71b8f309b2 Add licence to query node and query service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-04-19 13:47:10 +08:00
neza2017 83647e8503 Add copyright in master service module 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-19 11:12:56 +08:00
dragondriver dc7db73251 Add copyright annotations to proxynode and proxyservice 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-04-19 10:09:43 +08:00
sunby 751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
Xiangyu Wang 87a1e0b662 Reorganize msgstream 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-02 13:48:25 +08:00
zhenshan.cao efbf4452c5 Add log for port changed or proto unmarshal error 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-30 20:19:30 +08:00
Xiangyu Wang 72917467ec Change singlenode logs output 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-03-30 10:52:42 +08:00
neza2017 71614313b0 Fix master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-29 09:50:52 +08:00
neza2017 eca1e25470 Add msg id at master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-26 15:13:33 +08:00
bigsheeper 1634d75980 Release flowgraph properly 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-22 12:49:50 -05:00
dragondriver 49c6eeb052 Add useful debug log in proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-22 19:28:43 +08:00
zhenshan.cao c2734fa55f Fix bug and enchance system 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-22 16:36:10 +08:00
neza2017 6eb45c92eb Fix flush segment 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-13 17:05:36 +08:00
groot 47d9471498 Change gracefultime to 1 second 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-03-13 14:42:53 +08:00
cai.zhang 012f5f360e Add init log logic for index and proxy 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-12 19:47:37 +08:00
godchen f3649f0419 Refactor interface and proto 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-12 14:22:09 +08:00
ThreadDao ca916a5c47 Add ci nightly 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-03-11 14:14:29 +08:00
xige-16 09ae985daa Fix wrong error code in master_service_test.go 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-03-10 22:06:22 +08:00
sunby 800e31ebe4 Fix grpc startup in query node 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-10 15:49:23 +08:00
del-zhenwu b1763c3690 Disable xdist in test cases 
Signed-off-by: del-zhenwu <zhenxiang.li@zilliz.com>
 2021-03-10 15:27:26 +08:00
neza2017 7b2d67242d Fix describe segment if index not exist 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-10 14:45:35 +08:00
cai.zhang 60b97f9b65 Replace log of indexnode and indexservice 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-03-10 09:56:09 +08:00
quicksilver 455cc59dbb Update publish-test-images.yaml 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-09 13:55:35 +08:00
neza2017 55fc3f56aa Set grpc max recv mag as math.MaxInt32 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-08 15:49:42 +08:00
neza2017 014c4fe8ce Add config of minSegmentSizeToEnableIndex 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-08 15:46:51 +08:00
sunby aeb3038545 Remove GetCount in data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 10:49:44 +08:00
ThreadDao 7a3223dc81 tags collection stats 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-03-08 10:09:48 +08:00
neza2017 3ef18e86f0 Add 0331 tag for test_partition 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-06 17:47:11 +08:00
neza2017 2835bcdc4e Update masterservice unittest 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-03-05 20:41:34 +08:00
quicksilver 74154a11a4 Fix deploy error during regression stage 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-05 16:52:45 +08:00
BossZou 1efcf605e2 Enlarge Proxy node max size of reveivable grpc message 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-03-05 11:16:23 +08:00
sunby ddddd65d10 Delete internal/errors package 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-05 10:15:27 +08:00
bigsheeper 447a15207e Add zap log to query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-03-05 09:21:35 +08:00
dragondriver 801396e3ef Disable register link logic, use proxy node directly 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-03-04 22:27:12 +08:00
sunby dec0b40a66 Add log panic in log module 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-04 16:01:30 +08:00
godchen 3ba0dc2f44 Change doc 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-04 10:35:28 +08:00
bigsheeper d3db2b0d82 Add zap log in query service 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-27 10:11:52 +08:00
godchen 7b9fdd7f29 Add opentracing 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-26 17:44:24 +08:00
xige-16 50a5f97421 Delete componentStateResponse in queryService.proto 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-26 15:26:21 +08:00
neza2017 4ea27335fd Ignore index param from create_collection 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-26 14:40:53 +08:00
XuanYang-cn 10374f15d7 Update log in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-26 10:13:36 +08:00
Cai Yudong 84110d2684 Add tests/benchmark and tests/python_test using new python SDK 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-02-25 17:35:36 +08:00
cai.zhang 1d899513dd Unify service name in config file 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-24 18:46:19 +08:00
bigsheeper 08a020798e Release collection and partitions 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-24 17:24:51 +08:00
neza2017 e396c3f4ad Add zap log 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-24 16:25:40 +08:00
XuanYang-cn fd562f9f9c Update doc: add proxy graph 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-24 09:48:17 +08:00
zhenshan.cao 7c5e2d89f6 Read node port from config 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-23 18:08:17 +08:00
xige-16 d5379a82a8 Fix load binlog error when segment not flushed 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-23 14:13:33 +08:00
dragondriver b506754364 Add workflow of drop index in index service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-23 11:57:18 +08:00
zhenshan.cao bbfcbbdd68 Fix ci: paramstable and startup logic refactor 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-23 11:40:30 +08:00
sunby ea1d9ea99a Replace log in dataservice 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-23 09:58:06 +08:00
BossZou 7ac1821cf1 Add RocksMQ throughout test 
Signed-off-by: BossZou <yinghao.zou@zilliz.com>
 2021-02-20 18:30:37 +08:00
yukun 346b0a9549 Add unittest for RmqTtMsgStream 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-02-20 15:58:19 +08:00
neza2017 1d3fdebcf4 Add drop index for proxy 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-20 15:38:44 +08:00
cai.zhang 4c5a898e9a Add masterservice docker 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-20 11:15:19 +08:00
dragondriver 6ebb03791f Fix drop index interface 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-20 10:37:11 +08:00
zhenshan.cao b3891ca199 Correct docker file 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-20 10:15:37 +08:00
xige-16 450ea631bc Fix queryService assign queryChannel failure 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-18 16:26:02 +08:00
neza2017 85c1404393 Refactor master:index ids 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-11 08:41:59 +08:00
xige-16 70fe5233cf Watch queryChannel on query node 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-09 17:09:26 +08:00
neza2017 e2d8358cb4 Refactor master:index ids 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-09 13:11:55 +08:00
neza2017 4035083188 Support Index name 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-08 14:49:12 +08:00
groot dc3736281a Integrate message stream 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-08 14:30:54 +08:00
sunby eba5b9ae27 Fix queryservice client init bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-07 13:53:40 +08:00
groot b034740c6c Make paramstable initialize once by using do_once 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-06 13:39:15 +08:00
sunby 0111cba8ae Fix startup bugs in some modules 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-06 10:28:53 +08:00
XuanYang-cn dd72dfcbd1 Fix queryservice start 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-05 17:57:41 +08:00
dragondriver e6aec3fd08 Hardcode the peer id of proxy node 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 14:09:55 +08:00
godchen 0f620ab149 Fix invalidate metacache error 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-05 11:49:13 +08:00
dragondriver e384222803 Fix the logic of node id allocator 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 11:25:52 +08:00
neza2017 bea1535425 Add call of release collection when drop collection 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-05 11:05:37 +08:00
dragondriver a48590a2df Add query service logic to Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-05 10:29:14 +08:00
XuanYang-cn a64c4efde9 Refactor collectionReplica in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 20:31:23 +08:00
godchen c2ca2c276f Fix response check error 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-04 19:34:35 +08:00
dragondriver e0e8e1605e Implement the release-related task and load-related task in Proxy 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-02-04 17:47:19 +08:00
sunby 7270d18c98 Fix timesync startup bugs 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-04 17:31:02 +08:00
xige-16 aaac839075 Joint debug query service success with other components 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-02-04 17:09:18 +08:00
XuanYang-cn 560d2350ae Fix datanode watchDmChannel bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 16:09:34 +08:00
zhenshan.cao b89e5a3240 Add GetQuerySegmentInfo 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-04 14:37:12 +08:00
bigsheeper c6c99ef32b Implement getSegmengInfo function 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-04 11:40:14 +08:00
XuanYang-cn bc7ad02df4 Fix datanode init bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 10:25:01 +08:00
godchen 6e7e0b748a Send timetick to insert channel 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-03 20:04:29 +08:00
sunby e7588d3250 Implement getSegmentInfo interfaces 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-03 19:45:52 +08:00
zhenshan.cao 1b616fb874 Add persistent_segmentinfo 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-03 18:55:00 +08:00
bigsheeper 01e9dc8e3f Remove collection name 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 11:52:19 +08:00
sunby f3aad3a71c Change SegmentInfo 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 18:53:10 +08:00
bigsheeper 5e781b9370 Remove field name in query node and segCore 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 10:10:07 +08:00
sunby 45b99c0cf3 Save indexName and indexID in IndexCodec 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 19:56:11 +08:00
neza2017 899d31e72d Change interface: add index id and index name params 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-02 20:09:46 +08:00
sunby ca9c2f5c12 Add GetCount and implement GetCollectionStatistics 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 14:25:58 +08:00
neza2017 5de89422ed Support multi index 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-02-02 10:09:10 +08:00
XuanYang-cn 90d471b133 Refactor main of queryservice and querynode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-02 09:52:42 +08:00
godchen 94dfab5b21 Check collection whether exists before drop 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-02-01 10:53:13 +08:00
godchen abcc565311 Add proxy metacache 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-31 14:55:36 +08:00
sunby 3a67dda06c Add NewDataService in main 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-30 11:49:48 +08:00
dragondriver f660caca0c Refactor the client interface of proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-29 17:29:31 +08:00
Xiangyu Wang 99cef4b5c8 Add Distributed main entrypoint 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-01-29 17:08:31 +08:00
bigsheeper 73d2b6a101 Get index param from minio and filter by vector fields 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-29 15:22:24 +08:00
neza2017 bdf84f08ab Add watcher_test and fix some bugs 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-29 14:40:19 +08:00
zhenshan.cao ee0faddb07 Refactor proxy 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-29 09:27:26 +08:00
dragondriver 7957170220 Dock Proxy with other service component 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-28 20:51:44 +08:00
sunby bcd7368533 Fix get component states 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-28 15:20:24 +08:00
sunby 5ac463c9ca Add datanode client in data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-28 11:24:41 +08:00
XuanYang-cn 4acaa97039 Add unittests for datanode and queryservice 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-27 20:00:47 +08:00
bigsheeper 68cd15af63 Get index info from master 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-27 14:41:56 +08:00
sunby b4bcaaf847 Fix data service startup bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-27 11:40:10 +08:00
XuanYang-cn 05b013c52a Fix datanode bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-27 11:37:26 +08:00
xige-16 947976dbec Add test for query service’s load function 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-27 11:34:16 +08:00
bigsheeper 4f914a2c30 Add docker file and main function of query node 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-27 09:50:52 +08:00
neza2017 4d37a329da Fix data service init 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 19:25:17 +08:00
neza2017 d1b23cb8c5 Add debug info 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 19:24:09 +08:00
sunby f632ed0272 Fix data service init 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 18:45:12 +08:00
zhenshan.cao e83ac41bca Add component for index 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-26 18:01:32 +08:00
neza2017 8c54af292e Add dd handler 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 17:55:15 +08:00
XuanYang-cn 0b1c4f0420 Add building datanode cmd in Makefile 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-26 17:40:26 +08:00
sunby 9f72633fd9 Add dd handler 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 17:06:13 +08:00
neza2017 2ea6a5f9ee Add make dataservice in Makefile 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 16:41:10 +08:00
sunby b4997a37a2 Add make dataservice in Makefile 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 16:36:26 +08:00
neza2017 b6a788373b Fix proxy server 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 16:30:36 +08:00
neza2017 fd0227b69a Add logic of get time tick 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-26 15:23:42 +08:00
sunby 3b08162059 Add main function 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-26 15:14:49 +08:00
xige-16 7c210310dc Add load function for query service 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-26 15:13:20 +08:00
dragondriver ad244dc4e8 Add initial interface to proxy service client 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-26 15:09:31 +08:00
dragondriver d972b75aaa Add GetTimeTickChannel, GetComponentStates to proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-26 14:55:57 +08:00
XuanYang-cn b445587e11 Add datanode main 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-26 14:46:54 +08:00
cai.zhang 346bb1ba49 Delete some unused code and some debug logs 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-26 14:31:31 +08:00
dragondriver c6a6b1436c Add soft time tick in proxy service 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-01-26 13:41:41 +08:00
cai.zhang f940cc455a Add dockerfile for index 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-26 09:38:40 +08:00
neza2017 7f3aa92d10 Add docker file for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 15:58:04 +08:00
bigsheeper 7554246ace Add segment seeking and use real client 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-25 18:33:10 +08:00
sunby 7c0c835b49 Init params and add segment start/end position 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-25 15:16:28 +08:00
XuanYang-cn 5015a1035d Add logic of get meta from MasterService for datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-25 17:10:32 +08:00
neza2017 1fe099775d Refactor data service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 15:17:17 +08:00
neza2017 e89e89b894 Add main for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 12:03:44 +08:00
sunby 54fcd486bb Refactor data service. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-25 13:45:56 +08:00