93777e0c9d
Handle flowgraph releasing properly ( #16169 )
...
See also: #15846
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-20 16:45:41 +08:00
289e468a7a
Make bulk load fully work ( #16512 )
...
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
eae0747f14
Fix bug: data race caused by read and write Params.DataNodeCfg.NodeID ( #16478 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-13 20:09:34 +08:00
4781db8a2a
Add datanode import ( #16414 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-12 22:19:34 +08:00
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
45be3deb3f
Implement RC, DC, DN calling path for import. ( #16321 )
...
/kind feature
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-01 11:33:28 +08:00
0d54697d42
Implement ReportImport related logic. ( #16312 )
...
/kind feature
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-31 13:51:28 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
ca129d4308
Support configurable msgstream ( #16131 )
...
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
2078b24dca
Prepare for import ( #16083 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-21 15:47:23 +08:00
fab0263616
Use RmqFactory in DataNode unit test ( #16039 )
...
Use Rocksmq instead of Pulsar in DataNode unittest
- Avoid pulsar seek problem caused timeout issue
- Remove third party component dependency in unit test
See also: #15986
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-14 19:22:01 +08:00
6513be5c44
Extend watch/release channel logic in DataNode ( #15925 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-09 15:39:58 +08:00
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig ( #15843 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable ( #15434 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
4bccc85534
Use MsgChannelConfig in GlobalParams for all components ( #15163 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
1e61112b5a
Add retry logic in pulsar consumer unsubscribe ( #15284 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-20 01:23:37 +08:00
7f775096c5
Unsubscribe when close pulsar consumer ( #15221 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-01-17 18:53:36 +08:00
008e08a996
Add flowgraph manager and event manager ( #15102 )
...
1. Add flowgraph manager to manager start and release of dataSyncService
2. Add event manager to manager etcd watchInfo event
- Make put event able to retry
- Make delete event able to terminate the retry loop
See also: #14604 , #14300
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-13 14:21:34 +08:00
7efb02a4c5
Use PulsarConfig in GlobalParams for all components ( #15046 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-09 23:45:33 +08:00
f8f61fb116
Use BaseParams etcd configuration for all components ( #15012 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-07 16:15:26 +08:00
69087ff8dd
Support embedded etcd ( #14333 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
891c202b73
Use GlobalParamTable for all components ( #13885 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
0c39e0370c
Move segment statistics update to datanode tt loop ( #13233 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-15 10:53:16 +08:00
8a8ebed23d
Fix DN GC bug ( #12258 )
...
Fixes : #12249
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
ee0f753f7a
Fix datarace between GetComponentStates and Register ( #11935 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-19 13:57:12 +08:00
9186d5527e
Forward delete msg ( #11210 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-05 11:59:02 +08:00
225b2c66a0
Fix datanode ut using register bug ( #10885 )
...
Resolves : #10881
See also: #8058
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-01 11:01:49 +08:00
7ce7cb7a5e
Add a timed flush trigger mechanism ( #10197 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-20 15:02:36 +08:00
880056e664
[skip ci]Update license for datesync ( #9959 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:01:38 +08:00
044b06421a
Use etcd to watch channel on datanode ( #7115 )
...
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-14 15:44:34 +08:00
c90a97f95a
Fix Watch Channel canceled due to revision compacted ( #9788 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-13 17:02:33 +08:00
f0bedc7046
Add ut for data_node.go ( #9489 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-11 18:50:30 +08:00
5e7648fc7c
Check data node test error ( #9532 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-09 15:45:15 +08:00
937dbf4279
Add flushed segments ( #9476 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-08 19:09:12 +08:00
3be6672753
Add flush channel for delete_node ( #8762 )
...
Resolves : #8761
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-28 18:22:16 +08:00
d374ea71ba
Unify liveness check in datacoord and datanode ( #8480 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-28 16:10:04 +08:00
e4e762f2e8
Reduce go unittest time cost ( #8503 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-09-26 12:50:24 +08:00
2aa9deda1c
Improve logic when watch dm channel fails ( #8386 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-23 18:31:55 +08:00
6469d83072
Add datanode etcd alive check ( #8363 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-23 14:29:54 +08:00
f5173b595f
Add a channel prefix for all channels ( #8166 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-09-23 10:53:53 +08:00
06cee85a39
Add data node unit tests ( #7622 )
...
Issue: #6357
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-09-09 16:40:00 +08:00
89748ac639
Expose metrics of data cluster ( #7177 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-09-01 10:13:15 +08:00
881d0af1fc
Add sync logic in TestWatchChannel ( #7188 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-20 17:48:13 +08:00
86ca96a02e
Fix master branch unittest error ( #7083 )
...
Resolves : #7080
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-13 14:00:14 +08:00
e1b3273d1c
Add Datanode watch etcd channel ( #6965 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-13 10:50:09 +08:00
47df4a2e57
Fix DataNode BackGroudGC problem ( #6551 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-03 09:59:24 +08:00
dca6c6afde
Add unittest for BackGroundGC in DataNode ( #6522 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-07-15 10:05:55 +08:00
f0a68704e2
Add unittests for getChannelName in DataNode ( #6346 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-07-08 15:52:40 +08:00
441300140d
Tidy replica ( #5920 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-21 16:00:22 +08:00
ab7f642740
Add alias in paramtable ( #5878 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-19 12:38:06 +08:00
33f5225968
Use physical channel to init msg stream ( #5815 )
...
* Use physical channel to init msg stream
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix search collection error check (#1 )
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: bigsheeper <954206947@qq.com>
2021-06-18 16:02:05 +08:00
18eb27aa7a
Tidy replica in datanode ( #5805 )
...
- Remove not using functions
- Remove `initNode` logic in ParamTable
- Delete collection.go and collection_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-16 19:03:57 +08:00
dea0a4e522
Idempotent of FlushSegment ( #5746 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
ac19711d74
Destroy DataNode when drop collections ( #5638 )
...
* Destroy DataNode when drop collections
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* golanci-lint
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
0a82c6381f
add auto flush for data node ( #5609 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
6c02ae4eb1
Remove Old msgposition logic ( #5487 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
7a8b34ee9f
Remove not using code in DataNode ( #5476 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
55eac3ee7f
Complete Flush procedure in DataNode ( #5454 )
...
* Complete Flush proceduer in DataNode
Resolves : #5220
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* fix unittest
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
c53afee616
DataNode scales flowgraph
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
fc630bc1c8
Correct auto-flush behaviour ( #5282 )
...
Before this PR, DataNode considered auto-flush a valid flush
complete. It's wrong. So I open this PR to correct this behaviour
in DataNode.
Now binlog paths from auto-flush will be buffered in replica,
waiting until the manul flush to save into etcd all together.
See also: #5220 , #5268
A follow up job of #5271
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-05-18 11:45:00 +00:00
82ccd4cec0
Rename module ( #4988 )
...
* Rename module
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
51cae2c9ba
add datanode unittest to improve code coverage ( #4982 )
...
add datanode unittest to improve code coverage
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-04-21 18:41:37 +08:00
6deba1faa5
Add license in datanode package
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-04-19 15:16:33 +08:00
91f5c48037
Fix datanode goroutine leak
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-30 09:47:27 +08:00
ca3187d173
Fix data race in datanode tests
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-24 16:35:38 +08:00
51f669f1cb
Flush segments concurrently & change etcd meta
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-23 18:50:13 +08:00
f3649f0419
Refactor interface and proto
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
09ae985daa
Fix wrong error code in master_service_test.go
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-03-10 22:06:22 +08:00
7b2d67242d
Fix describe segment if index not exist
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-10 14:45:35 +08:00
455cc59dbb
Update publish-test-images.yaml
...
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-09 13:55:35 +08:00
2835bcdc4e
Update masterservice unittest
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-05 20:41:34 +08:00
7b9fdd7f29
Add opentracing
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-02-26 17:44:24 +08:00
10374f15d7
Update log in datanode
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-26 10:13:36 +08:00
fd9e42d3ed
Update msgstream/unmarshal interface
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-02-03 17:30:10 +08:00
abffdbff62
Remove partition tag
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-03 16:44:38 +08:00
73a8e7de84
Fix datanode insertchannel refresh bug
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-03 15:18:05 +08:00
99cef4b5c8
Add Distributed main entrypoint
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-01-29 17:08:31 +08:00
73d2b6a101
Get index param from minio and filter by vector fields
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-29 15:22:24 +08:00
bdf84f08ab
Add watcher_test and fix some bugs
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-29 14:40:19 +08:00
f088467dde
Add watcher_test and fix some bugs
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-28 10:14:31 +08:00
b597a637f3
Add datanode metatable tests
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-29 12:03:10 +08:00
b445587e11
Add datanode main
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-26 14:46:54 +08:00
9fe672ab86
Enchance datanode interface
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-24 21:20:11 +08:00
50e2369000
Change name and tag to ID
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 19:36:09 +08:00
ec27788628
Add segment statistics updates logic for datanode
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-21 09:55:25 +08:00
6a13386393
Add DataNode package
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-19 11:37:16 +08:00
XuanYang-cn
Ten Thousand Leaves
zhenshan.cao
godchen
xige-16
xiyichan
groot
congqixia
Cai Yudong
jaime
bigsheeper
Xiaofan
Bingyi Sun
dragondriver
sunby
neza2017
Xiangyu Wang
quicksilver