5417fee45f
fix releasePartition not work after loadCollection ( #6025 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-23 17:44:12 +08:00
46f86a0354
Return sealed segments when calling flush ( #6011 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-23 16:56:11 +08:00
7dbf33b788
Update component names in config files ( #5989 )
...
* rename master/indexService in .yaml
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename proxyNode to proxy in config files
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move master.yaml to root_coord.yaml, move proxy_node.yaml to proxy.yaml
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move proto/query_service.proto to proto/query_coord.proto
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move proxy_node_manager.go to proxy_manager.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add query_coord.pb.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update ci
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 19:08:03 +08:00
2af1c27811
Rename queryservice to querycoordinator ( #5931 )
...
* Rename queryservice to querycoordinator
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix param table
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* code format
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix unittest
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* service to coordinator
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* code format
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* set querycoord in ci
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-22 16:44:09 +08:00
d5f4ee6f44
Rename proto master/index_service/proxy_service ( #5973 )
...
* rename proto proxy_service to proxy
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename proto index_service to index_coord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename proto master to root_coord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 16:14:09 +08:00
c863a193de
Rename filename from dataservice to datacoordinator ( #5946 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 10:42:07 +08:00
b2eb596c23
Change search result to columnBased and return target_entry ( #5896 )
...
* Change search result to columnBased and return target_entry
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
Co-authored-by: fluorinedog <fluorinedog@gmail.com>
* synchronize the latest pymilvus-orm version
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Change pymilvus-orm version to 40
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
Co-authored-by: fluorinedog <fluorinedog@gmail.com>
2021-06-21 20:18:13 +08:00
7f2d7c1da3
Rename dataservice to datacoordinator ( #5912 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 18:22:13 +08:00
0300b682f8
Remove deprecated interfaces in data coordinator ( #5929 )
...
* Remove derecated interfaces in data coordinator
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove RegisterNode in data coordinator
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 17:40:05 +08:00
2642415bc3
Redefine the auto_id and primary_key ( #5903 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-21 11:42:18 +08:00
958280ebb8
Add support for time travel ( #5897 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-21 10:42:10 +08:00
255e3959af
support time travel ( #5894 )
...
* support time travel
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
* lint
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-19 17:38:11 +08:00
e4c51aae36
Add recovery logic for querynode and queryservice ( #5843 )
...
* merge milvus/recovery2
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add recovery logic in queryservice
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* debug smoke case
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add etcd to querynode
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix release partition error
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix load balance error
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* debug querynode down and recovery
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add log
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix showCollection
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* skip smoke test search without insert
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-19 11:45:09 +08:00
676a30e418
Use channelsMgr to manage search message stream ( #5819 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-18 10:33:58 +08:00
b2908c4780
Release collection ( #5839 )
...
* relese collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* gen-proto
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* release collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* release collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-17 17:45:56 +08:00
abc0faeb49
add default time tick ( #5827 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-17 14:57:57 +08:00
97049ce293
use dataservice's getRecoveryInfo to get load and watch info ( #5790 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-16 11:09:56 +08:00
9246c663fc
Add start position in segment ( #5753 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:39:01 +08:00
fceba63a33
Add segment and channel msgs in retrieve results ( #5750 )
...
* Fix query hang bug (#5709 )
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Add segment and channel msgs in retrieve results
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-15 16:38:59 +08:00
5bcb94294c
Set start positions in datanode ( #5757 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
f74ea3beee
[skip ci] Remove ProxyService ( #5738 )
...
* [skip ci] Remove ProxyService
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
* Change helm branch
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
* Fix bug: paramstable crashed
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-15 16:36:35 +08:00
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
f054fc9be2
Add GetRecoveryInfo in dataservice ( #5629 )
...
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
3a5c8c4d3a
Add seek position to WatchDmChannels response ( #5601 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
a948b71e06
Support to send dd msg to all dml channels ( #5597 )
...
* send dd msg to dml channel
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add dml_channels.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix race
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 16:04:48 +08:00
e7521afed5
Change SaveBinlogPath ( #5576 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
41794ec3a6
Change SaveBinLogPath proto & func ( #5551 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
38b1f7dabe
Change the logic of getting vchannel position ( #5502 )
...
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
607a8ffa1f
Update WatchDmChannels signature ( #5447 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
ef563fda8f
Updata flushe procedure
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
c53afee616
DataNode scales flowgraph
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
81ac20143b
Add persistency of datanode cluster ( #5387 )
...
We save channels registered in datanode in etcd and restore the cluster info after
restarting.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
3662b3f0e5
Refactor dataservice ( #5400 )
...
Pick up grpc calls to grpc_handler.go and refactor code format
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:01 +08:00
bec9f2c182
Split segcore and plan proto for future feature ( #5767 )
...
* Split segcore and plan proto for future feature
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
* lint
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-15 14:43:57 +08:00
cdbc6d2c94
Refactor query node and query service ( #5751 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 12:41:40 +08:00
68f703f983
Add Query impl in proxynode ( #5644 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-07 17:21:36 +08:00
50b3d39370
Add GetRecoveryInfo interface and proto ( #5598 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-07 14:16:36 +08:00
afa5da7df0
Add query rpc into proto ( #5592 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-04 12:02:34 +08:00
9a90313390
Support GetEntityByIDs in CGo, fix segcore bugs ( #5563 )
...
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-04 10:38:34 +08:00
99699b2021
Support to show collections which are loaded into memory ( #5567 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-03 19:09:33 +08:00
e76830ab09
Support column-based insert request in proxy ( #5560 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-03 15:03:34 +08:00
4af2cacb28
Refactor IndexService and IndexNode: remove redundant serviceID ( #5491 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-05-31 10:32:30 +08:00
ca0993c274
Update OWNERS
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-30 18:58:47 +08:00
fce792b8bf
Add historical and streaming module in querynode ( #5469 )
...
* add historical and streaming
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* fix GetSegmentInfo
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* pass regression test
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2021-05-28 10:26:30 +08:00
a606ab9c67
Add power-off restart logic for IndexService and IndexNode ( #5395 )
...
Add power-off restart logic for IndexService and IndexNode
Signed-off-by: <xiaocai2333> <cai.zhang@zilliz.com>
2021-05-27 14:24:29 +00:00
5ac8b8acc6
Add comments to proto ( #5452 )
...
Specify which fields must be set by SDK
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-05-27 08:34:31 +00:00
88b42304ef
Add dummy RPC ( #5376 )
...
Adding a dummy RPC makes it easier to develop new RPCs in the future.
When developing a new function, the SDK is often implemented last.
At this time, this is helpful for debugging.
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-25 06:44:43 +00:00
4aa6a8fe18
Dataservice use lastest grpc design and saves ddl/dml pos pair ( #5348 )
...
Dataservice use lastest grpc design and saves ddl/dml pos pair
* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)
See also #5220
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-05-21 11:38:58 +00:00
02f2348500
Save segment allocation status to meta ( #5184 )
...
To ensure segments being flushed correctly, we need to load
the segment allocation status from meta after starting the service.
2021-05-21 10:30:41 +00:00
xige-16
sunby
Cai Yudong
yukun
dragondriver
zhenshan.cao
FluorineDog
neza2017
XuanYang-cn
congqixia
bigsheeper
Xiangyu Wang
cai.zhang
congqixia