4f7c3e7d3a
Save binlog path in SegmentInfo ( #6590 )
...
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-19 13:00:12 +08:00
86ca96a02e
Fix master branch unittest error ( #7083 )
...
Resolves : #7080
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-13 14:00:14 +08:00
6c5be83ea1
Refactor NewEtcdKV API ( #7075 )
...
This PR mainly did the following 3 things:
- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision
The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.
Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.
Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.
Resolves : #6955
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-13 11:04:09 +08:00
e1b3273d1c
Add Datanode watch etcd channel ( #6965 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-13 10:50:09 +08:00
5e78fd6a97
[skip ci]Update approvers ( #7063 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-08-12 19:36:09 +08:00
9d81a90402
Fix mem leak of flush cache in DataNode ( #6862 )
...
Resolves : #6858
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-11 14:24:09 +08:00
47df4a2e57
Fix DataNode BackGroudGC problem ( #6551 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-03 09:59:24 +08:00
7307332fab
Add nil check ( #6832 )
...
* add nil check
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* use grpc functions
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* use grpc functions
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* fix indexCoord crash
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-07-29 20:11:23 +08:00
db94d7771f
Read vector from disk ( #6707 )
...
* Read vector from disk
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* go fmt
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix git action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix test error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix caculate error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change var name
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix len error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused code
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change bytes to float method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change float to bytes method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-07-24 09:25:22 +08:00
dca6c6afde
Add unittest for BackGroundGC in DataNode ( #6522 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-07-15 10:05:55 +08:00
f0a68704e2
Add unittests for getChannelName in DataNode ( #6346 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-07-08 15:52:40 +08:00
49a0528d53
Fix trace ( #6218 )
...
* Fix trace
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix go fmt
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-30 16:18:13 +08:00
357af89d55
Add unittest in segment_replica ( #6197 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-30 10:26:12 +08:00
769cbb292a
Update doc in datanode ( #6194 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-29 17:34:13 +08:00
39614aa8eb
Add rocksmq_path in config and paramtable ( #6099 )
...
* Add rocksmq_path in config and paramtable
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Add rocksdbPath in NewRmsFactory
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Change rdb default path to /vat/lib/milvus/rdb_data
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-25 19:44:11 +08:00
a09a3a1905
rename service to coord ( #6020 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-23 16:14:08 +08:00
b640627a38
Fix datanode restart bug ( #5976 ) ( #6004 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-23 12:26:10 +08:00
7dbf33b788
Update component names in config files ( #5989 )
...
* rename master/indexService in .yaml
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename proxyNode to proxy in config files
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move master.yaml to root_coord.yaml, move proxy_node.yaml to proxy.yaml
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move proto/query_service.proto to proto/query_coord.proto
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move proxy_node_manager.go to proxy_manager.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add query_coord.pb.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update ci
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 19:08:03 +08:00
d5f4ee6f44
Rename proto master/index_service/proxy_service ( #5973 )
...
* rename proto proxy_service to proxy
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename proto index_service to index_coord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename proto master to root_coord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 16:14:09 +08:00
7f2d7c1da3
Rename dataservice to datacoordinator ( #5912 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 18:22:13 +08:00
688a587f4f
Add metrics in datanode ( #5926 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-21 18:08:15 +08:00
0300b682f8
Remove deprecated interfaces in data coordinator ( #5929 )
...
* Remove derecated interfaces in data coordinator
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove RegisterNode in data coordinator
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 17:40:05 +08:00
92e429d812
Rename IndexService to IndexCoord ( #5932 )
...
* rename package indexservice to indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename indexservice to indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix queryservice static-check
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move distributed/indexservice to distributed/indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move internal/indexservice to internal/indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename indexservice to indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename MasterComponent to RootCoordComponent
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for queryservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for dataservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for datanode
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for proxynode
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for querynode
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename IndexService to IndexCoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename IndexService to IndexCoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix rebase issue
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-21 17:28:03 +08:00
441300140d
Tidy replica ( #5920 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-21 16:00:22 +08:00
da1f9f9241
Rename dataservice to datacoordinator ( #5904 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 11:40:15 +08:00
3cc0ab1345
Remove panic in NewDataSyncService ( #5888 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-19 15:18:06 +08:00
ab7f642740
Add alias in paramtable ( #5878 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-19 12:38:06 +08:00
33f5225968
Use physical channel to init msg stream ( #5815 )
...
* Use physical channel to init msg stream
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix search collection error check (#1 )
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: bigsheeper <954206947@qq.com>
2021-06-18 16:02:05 +08:00
18eb27aa7a
Tidy replica in datanode ( #5805 )
...
- Remove not using functions
- Remove `initNode` logic in ParamTable
- Delete collection.go and collection_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-16 19:03:57 +08:00
75f2b3c2ba
Add Unittest for allocator ( #5781 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 19:11:55 +08:00
0f4bd50ce3
Support ETCD endpoints array ( #5755 )
2021-06-15 16:36:36 +08:00
5bcb94294c
Set start positions in datanode ( #5757 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
dea0a4e522
Idempotent of FlushSegment ( #5746 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
8a0335be44
Fix dd node filter bug ( #5736 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:24:49 +08:00
ab2fd34a2f
fix double messages bug in datanode recovery ( #5733 )
...
* fix double messages bug in datanode recovery
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* remove debug log
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:14:29 +08:00
46699c4c2b
fix datanode dropcollection error ( #5690 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:06:13 +08:00
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
e7fd3aa926
Flowgraph recovery ( #5640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
ac19711d74
Destroy DataNode when drop collections ( #5638 )
...
* Destroy DataNode when drop collections
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* golanci-lint
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
74c73fa75f
fix datanode ( #5635 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
7064937dc3
fix insertBuffer node ( #5632 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0c362a8831
fix bug and add unit test for insert buffer ( #5630 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
3c70675313
data node, save binlog to minIO, and let data service save these meta ( #5618 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0a82c6381f
add auto flush for data node ( #5609 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
275881dbdb
Remove address param ( #5556 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 16:04:46 +08:00
e96d39bf6a
refactor datanode flowgraph ( #5538 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
31c4a4f5ed
Simplify dd buffer & add lock ( #5513 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
68bf983f21
Add Collection ID in ddNode and filter logic ( #5496 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
6c02ae4eb1
Remove Old msgposition logic ( #5487 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
XuanYang-cn
congqixia
Xiangyu Wang
bigsheeper
godchen
yukun
Cai Yudong
neza2017