milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
14,358 Commits
130 Branches
115 Tags
556 MiB
Commit Graph

426 Commits (2e708a291024c100cc760c89093948530f946978)

Author SHA1 Message Date
Bingyi Sun e3e4dc7153
Improve GetVChanPosistion and fix unit tests (#11888) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-16 21:41:50 +08:00
Xiangyu Wang 12f50cb22c
[skip ci]Update OWNERS files (#11898) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-16 15:41:11 +08:00
congqixia 7e43b568ee
Add dropped ts and apply when gc scan dropped segments (#11838) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-16 14:23:21 +08:00
shaoyue 6322eb31e7
[skip ci] Add comments for exported things in policy.go (#11876) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2021-11-16 13:21:16 +08:00
Bingyi Sun b8c5239784
Add compaction retention config (#11793) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-16 11:47:10 +08:00
Bingyi Sun 8b60aa8c4e
Skip single compaction if there's no delta log in a segment (#11806) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-16 11:23:11 +08:00
Bingyi Sun 902e856a06
[skip ci]Remove redundancy log (#11732) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-15 10:25:09 +08:00
congqixia d38043f235
Filter with partition id in GetVChanPosition (#11746) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-13 08:45:09 +08:00
Xiaofan 71e814f796
Remove common.yaml and component.yaml (#11661) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-11-12 21:25:08 +08:00
bigsheeper 93149c5ad9
Load growing segment in query node (#11664) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-11-12 18:27:10 +08:00
Bingyi Sun 5c8978e5d6
Fix drop segment bug (#11709) 
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-12 17:31:11 +08:00
Bingyi Sun 54b40da4c5
Add segment drop logic (#11621) 
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-12 00:22:42 +08:00
Bingyi Sun 002b854590
Move compaction and garbage switch to milvus.yaml (#11654) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-11 23:38:42 +08:00
Bingyi Sun 7a3873960b
Fix watch channel with nil seek position (#11650) 
issue: #11649
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-11 19:32:43 +08:00
Bingyi Sun 5574a104c9
Add triggerInfo in compactionTask (#11517) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-11 15:54:42 +08:00
godchen 5357e301f4
Fix retrieve search error (#11501) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-11 00:54:45 +08:00
Bingyi Sun f01063a020
[skip ci]Add comment for GetCompactionState (#11599) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-11 00:00:45 +08:00
Bingyi Sun 1104f752f5
[skip ci]Add comment for ManualCompaction (#11598) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-10 23:58:57 +08:00
Bingyi Sun adea19352c
[skip ci]Add comment for CompleteCompaction (#11597) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-10 23:57:38 +08:00
congqixia 3c9610cb1b
Add garbage collector for DataCoord (#11554) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-10 19:03:38 +08:00
Xiaofan 5f08b67156
Move node.yaml to component.yaml (#11363) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-11-10 14:27:37 +08:00
Bingyi Sun c3f0c5a3ff
Add compaction interfaces in proxy (#11431) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-09 14:47:02 +08:00
sunby 05681364c9
Fix some review comments in compaction. related pr #9906 (#11399) 
issue: #9904
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2021-11-08 21:45:00 +08:00
sunby db6ecd58f2
Fix UpdateFlushSegmentsInfo bug (#11407) 
issue: #7313
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2021-11-08 14:16:56 +08:00
congqixia 0c4e09e341
Add comment for datacoord Server.startDataNodeTtLoop (#11033) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 23:01:02 +08:00
sunby c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
sunby a55019825b
Fix stale segments selector (#11051) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-01 22:30:04 +08:00
godchen 7108612398
[skip ci]Fix allocator typo (#10998) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-01 16:54:07 +08:00
godchen 0970aea7e7
[skip ci]Fix allocator typo (#10997) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-01 16:52:19 +08:00
godchen d0ee395de5
[skip ci]Fix allocator typo (#10996) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-01 16:50:33 +08:00
congqixia 54738d2ac5
[skip ci] Fix lint for datacoord/channel_store.og (#10964) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-30 21:08:00 +08:00
congqixia 2b07876827
[skip ci] Fix lint for AvgAssignRegisterPolicy (#10965) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-30 21:03:52 +08:00
Xiaofan 2df2d488e1
Fix Unit test fatal fail (#10933) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-30 10:24:38 +08:00
congqixia 390fad4eb7
Fix LatestPosition option conflict with earliest patch (#10907) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-29 21:30:49 +08:00
godchen ad2cfdd99f
[skip ci]Fix datanode typo error (#10880) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-29 13:04:04 +08:00
godchen 7071ded008
[skip ci]Fix datanode typo error (#10879) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-29 12:03:38 +08:00
Xiaofan 93f205b49e
Rename param names (#10853) 
* Rename param names

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

* Rename configs

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-28 23:16:40 +08:00
congqixia c019e80dbb
Change SeekPosition to earliest of all segments (#10771) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-28 14:30:26 +08:00
Xiaofan 6fddcb3925
Fix Session checker for proxy (#10737) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-27 21:58:33 +08:00
congqixia 169af67eb1
Fix segments not removed from flushingCache (#10738) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-27 16:06:24 +08:00
congqixia 5c0b5c42ca
Fix merge caused compile error (#10693) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-26 21:46:21 +08:00
sunby 7fbb469081
Change channel's field to upper case to log details (#10567) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-26 20:15:41 +08:00
sunby 2de3131bb3
Change default balance policy (#10563) 
issue: #10198
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-26 19:40:44 +08:00
sunby 9b8e1c657e
Split watch operations to many transactions (#10655) 
issue: #10633
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-26 19:38:20 +08:00
godchen 095a2378ca
Add datanode ut (#10433) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-25 22:52:27 +08:00
godchen 3030454eb9
[skip ci]Fix datacoord package name missing (#10608) 2021-10-25 21:21:40 +08:00
XuanYang-cn 102bb36440
[skip ci]Update license for dc cluster (#10593) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 20:23:16 +08:00
XuanYang-cn fe955b888f
[skip ci]Update license for dc meta (#10592) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 20:21:18 +08:00
XuanYang-cn acb4dd2069
[skip ci]Update license for dc channel manager (#10591) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 20:19:25 +08:00
XuanYang-cn 23c7691d59
[skip ci]Update license for dc policy (#10590) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 19:48:23 +08:00
XuanYang-cn e5a88374bd
[skip ci]Update license for dc server (#10589) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 19:46:28 +08:00
XuanYang-cn 548634ff05
[skip ci]Update license for dc segmentinfo (#10588) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 19:44:37 +08:00
xige-16 9ebabdac79
Add handoffTask (#10330) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-10-24 22:39:09 +08:00
sunby 4036830b2d
Fix modify segment without clone (#10498) 
issue: #10430
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-23 13:31:09 +08:00
congqixia 285c3f63e3
Trim segmentinfo binlog for VChaninfo usage (#10425) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-22 14:35:13 +08:00
godchen d6bf0c928f
Get stats delta path from datacoord (#10362) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 14:31:13 +08:00
groot 0a1529cf7e
Fix a typo (#10339) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-21 14:16:37 +08:00
godchen c2fa2e940d
Check cluster error (#10116) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-21 10:57:03 +08:00
congqixia 7ce7cb7a5e
Add a timed flush trigger mechanism (#10197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 15:02:36 +08:00
groot 76b053a11c
Fix a typo (#10192) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-20 12:46:42 +08:00
sunby 6d5671add8
[skip ci]Add comment for NewBgChecker (#10224) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 21:00:35 +08:00
sunby 0bad693cd1
[skip ci]Add comment for NewReassignPolicy (#10223) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 20:58:42 +08:00
sunby 7c74c0fda1
[skip ci]Add comment for NewAssignPolicy (#10222) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 20:56:48 +08:00
sunby 600768882c
[skip ci]Add comment for NewDeregisterPolicy (#10133) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 14:44:35 +08:00
sunby 09435300b6
[skip ci]Add comment for NewConsistentHashChannelPolicyFactory (#10131) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 14:42:42 +08:00
sunby 9847063a40
[skip ci]Add comment for NewRegisterPolicy (#10132) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 14:40:42 +08:00
congqixia 6a886d63f2
Support stats and delta log SaveBinlog logic (#10156) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 14:32:41 +08:00
zhenshan.cao 32a029fbed
[skip ci]Fix golint error in datacoord (#10135) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-18 21:32:49 +08:00
groot 07599e6a35
[skip ci] Fix a typo (#10106) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-18 17:54:41 +08:00
congqixia b028dc3918
[skip ci] Fix golint issue for datacoord/channel_store.go (#9995) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-15 20:44:38 +08:00
congqixia 1e11a11c27
[skip ci] Fix golint issue for datacoord/util.go (#9992) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-15 20:40:53 +08:00
zwd1208 08ca551a75
[skip ci]Change etcd to lowercase (#9983) 
Signed-off-by: Weida Zhu <weida.zhu@zilliz.com>
 2021-10-15 18:58:37 +08:00
godchen 3f713a1104
Check server test error (#9954) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-15 17:12:35 +08:00
sunby bfc9e47e2c
Fix data race in datacoord test (#9936) 
issue: #9935
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-15 15:56:35 +08:00
Xiaofan a0b98dacda
Skip reconsume tt msgstream when datacoord restart (#9684) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-15 11:46:33 +08:00
sunby 003a7ed7cc
[skip ci]Add comment for MarshalLogObject (#9920) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 21:26:33 +08:00
sunby 3a6f6d399f
[skip ci]Add comment for ChannelOpTypeNames (#9919) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 21:24:41 +08:00
sunby 2617b0f2a9
[skip ci]Improve code style (#9918) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 21:22:45 +08:00
congqixia 641e472b73
[skip ci] Add comment for ChannelPolicyFactoryv1 (#9913) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-14 20:28:33 +08:00
congqixia 6b8ff60de3
Close event channel when watch event found error (#9819) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-14 19:20:35 +08:00
congqixia ec6e8e4127
Hide liveChn into sessionutil and fix liveness init order (#9860) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-14 16:40:35 +08:00
sunby 044b06421a
Use etcd to watch channel on datanode (#7115) 
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 15:44:34 +08:00
dragondriver 72d3d470ba
Use the server id as identifier in system topology graph (#9825) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-10-14 11:12:38 +08:00
sunby 26a365c06d
Print more information when handling SaveBinlogPathsRequest (#9803) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-13 20:54:33 +08:00
zhenshan.cao b122fda325
[skip ci]Clarify the errors of DataCoord (#9771) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-13 11:02:34 +08:00
sunby e811a66527
Fix save binlog path timeout bug (#9573) 
issue: #9559
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-12 23:46:35 +08:00
sunby aba9aa450d
[skip ci]Add comment for SetFlushTime (#9668) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-11 21:06:37 +08:00
sunby 6fc64b836a
[skip ci]Add comment for SetBinlogs (#9667) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-11 21:04:42 +08:00
congqixia 02dc2a2453
Add GetVChanPosition new cases ut (#9544) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-09 20:57:33 +08:00
sunby 1e0c3cbcb9
[skip ci]Add comment for SetCurrentRows (#9526) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-09 00:16:56 +08:00
sunby 7834649dbc
[skip ci]Add comment for AddAllocation (#9525) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-09 00:15:13 +08:00
sunby 292aeeac63
[skip ci]Add comment for SetAllocations (#9524) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-09 00:13:21 +08:00
godchen 937dbf4279
Add flushed segments (#9476) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-08 19:09:12 +08:00
sunby 006303285f
Fix a nil pointer bug and improve comments (#7625) 
Issue: #7624
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-08 19:05:25 +08:00
congqixia d828a63c94
[skip ci] Add comment for SegmentsInfo AddBinglogPaths (#9386) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-08 17:31:50 +08:00
congqixia 2711b53d57
Apply collection start position (#9459) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-08 14:51:13 +08:00
sunby e7e481444b
Check error returned by server stop (#9015) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-08 10:09:02 +08:00
sunby f705281884
Check error returned by ExpireAllocations (#9016) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-08 10:07:02 +08:00
sunby 21655d54be
[skip ci]Add comment for AddSegmentBinlogs (#9389) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 18:32:50 +08:00
sunby ad15e786cc
[skip ci]Add comment for SetStartPosition (#9399) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:59:31 +08:00
sunby d5081783c3
[skip ci]Add comment for SetDmlPosition (#9398) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:57:47 +08:00
sunby d41cdbb909
[skip ci]Add comment for SetState (#9397) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:55:53 +08:00
sunby cc794354b5
[skip ci]Add comment for SetRowCount (#9394) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:54:12 +08:00
sunby 82ff4563b3
[skip ci]Add comment for SegmentInfoOption (#9393) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:52:17 +08:00
sunby 41633aee0a
[skip ci]Add comment for ShadowClone (#9391) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:50:14 +08:00
sunby ad6b06ed85
[skip ci]Add comment for Clone (#9390) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:48:00 +08:00
sunby 6c0b38eb5a
[skip ci]Add comment for SetExpireTime (#9395) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:45:29 +08:00
godchen d08370d06b
[skip ci]Fix meta go comment typo (#9378) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-10-07 10:36:03 +08:00
godchen 5c14f96dc4
[skip ci]Fix cluster go comment typo (#9377) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-10-07 10:32:13 +08:00
congqixia 5031f09073
[skip ci] Add detail comment for SegmentsInfo (#9332) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-06 17:20:07 +08:00
sunby 5467240fca
[skip ci]Add comment for SetFlushTime (#9295) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-05 22:36:48 +08:00
sunby a1a69a8fef
[skip ci]Add comment for SetBinlogs (#9293) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-05 22:34:49 +08:00
sunby 88b4439fdf
[skip ci]Add comment for package datacoord (#9014) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-05 22:33:01 +08:00
sunby a51533de7f
[skip ci]Add comment for SetCurrentRows (#9294) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-05 22:28:38 +08:00
congqixia 768535f31c
Add positionProvider implementation assertion (#9261) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-05 20:58:04 +08:00
congqixia 423f6584da
[skip ci] Add comment for Datacoord SegmentsInfo allocation (#9260) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-05 15:29:03 +08:00
zhenshan.cao 45d6b8dcd0
[skip ci] Fix golint error in datacoord (#9184) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-04 17:16:54 +08:00
zhenshan.cao 5fcae9d956
[skip ci] Fix golint error in datacoord (#9191) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-04 13:20:51 +08:00
godchen 2792f85211
Handle server error (#9032) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-03 22:17:56 +08:00
congqixia e20445987e
Add Datacoord Server implementation Assertion (#9143) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-03 19:44:11 +08:00
sunby 68b26326d3
[skip ci]Modify comment for Init in param table (#9142) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-03 17:51:55 +08:00
sunby c54599ae6f
[skip ci]Modify comment for SetState (#9141) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-03 17:50:03 +08:00
sunby 7448904c8a
[skip ci]Modify comment in param table (#9140) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-03 17:38:14 +08:00
godchen f18e0628ae
Make NewMeta private (#9057) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-02 23:09:56 +08:00
congqixia a5cd484ff2
Add Manager implemenation assertion (#9038) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-02 14:45:56 +08:00
congqixia 2044e5eacb
Add ClusterStore implementation assertion (#9037) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-01 23:48:54 +08:00
sunby 6f08430cb8
Check error in server test (#9012) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-01 21:00:46 +08:00
sunby 332025aace
Check error in server_test.go (#9011) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-01 20:58:10 +08:00
Xiaofan 2c2b800ff4
Change Log Name with NodeId (#8721) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-01 08:52:50 +08:00
congqixia a11f9b7eb5
Add log for Flush to print return segments with CollectionID (#8928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-30 18:12:25 +08:00
congqixia 1cb2e2153f
[skip ci] Fix AllocationPolicy golint (#8883) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-29 20:52:09 +08:00
zhenshan.cao d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-29 20:26:00 +08:00
sunby 2ea7c0af25
Check error in datacoord server test (#8802) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-28 22:04:06 +08:00
sunby eee075b10e
Check error in server test (#8801) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-28 21:54:14 +08:00
congqixia d374ea71ba
Unify liveness check in datacoord and datanode (#8480) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-28 16:10:04 +08:00
sunby 745221e87d
Check error in segment_manager.go (#8705) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 22:46:12 +08:00
sunby e5bc8e7050
Check error in MoveSegmentBinlogs (#8704) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 22:44:18 +08:00
sunby c152cf2b6b
Check error in segment_manager_test.go (#8706) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 22:42:02 +08:00
congqixia ffbfaf7f76
Apply collection start position when open segment (#8697) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-27 20:34:03 +08:00
sunby d1dead500c
Check error in allocator test (#8621) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 10:46:08 +08:00
sunby 75767d58b3
Remove useless codes in datacoord (#8619) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-26 21:54:07 +08:00
Xiaofan d38adda8c2
Rename log file based on node id (#8517) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:18:11 +08:00
dragondriver 019f18ddf9
Expose created time and updated time to datacoord metrics (#8172) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-26 17:50:07 +08:00
zhenshan.cao 5390136cd6
[skip ci] Fix golint error of services.go in datacoord (#8524) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-24 21:33:55 +08:00
congqixia ca62b1e149
Use clone instead of ShadowClone when update pb data (#8461) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-24 12:53:54 +08:00
Xiaofan f5173b595f
Add a channel prefix for all channels (#8166) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-23 10:53:53 +08:00
sunby 3fc1b9cf9f
Change copyright (#8318) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-22 20:12:08 +08:00
sunby 82dd762ef0
Change copyright (#8317) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-22 19:59:59 +08:00
cai.zhang 79b7dcc75a
Add trace log for insert (#8301) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-09-22 19:33:54 +08:00
Xiaofan 0c35cd57b0
Unify param table to the same naming format (#8254) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-20 23:27:52 +08:00
congqixia 21f15d66bb
Add option for DataNode creation (#8240) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-18 19:10:07 +08:00
Xiangyu Wang c314b2b4a8
Remove unused label rules (#7929) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-18 18:19:51 +08:00
Yusup ccb57ba4b2
add collection alias (#7255) 
Signed-off-by: Yusup <yusup@lsgrep.com>
 2021-09-18 11:13:51 +08:00
congqixia 266db4c883
Add readme for datacoord (#8141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-17 23:57:55 +08:00
congqixia 01c8341a2e
[skip ci]Add comments for SegmentManager (#8087) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-16 19:55:49 +08:00
congqixia 2bca69596e
Add some comment for SegmentsInfo (#7995) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-15 21:07:53 +08:00
congqixia c76f91233f
Put errors together (#7958) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-15 21:05:52 +08:00
sunby eed856d626
Change error message to lower case (#7890) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-14 17:33:48 +08:00
sunby 9c2fb0bccb
Recorrect comment for Allocation (#7887) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-14 17:27:55 +08:00
congqixia 85d73358cc
Remove ctx from LongTermChecker (#7792) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-13 16:50:57 +08:00
congqixia e2c15d5b01
[skip ci]Add comment for ClusterStore (#7785) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-13 15:59:55 +08:00
congqixia d3be316bc2
Add Comment for datacoord cluster (#7676) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-10 11:26:00 +08:00
Xiangyu Wang 42aafdc407
Remove unused msg from msg.go (#7612) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-09 19:56:00 +08:00
sunby 49b6c3101d
Remove datacoord useless code (#7673) 
Issue: #7624
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-09 19:08:03 +08:00
congqixia c96b19a640
Add datacoord metricsinfo unit test (#7595) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-09 10:16:00 +08:00
dragondriver 9b21b3b5fd
Initialize test framework of proxy (#7563) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-09 10:06:29 +08:00
congqixia 664aeb09c6
Add ut and comments for datacoord meta (#7560) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-08 11:35:59 +08:00
congqixia 5b22697ec4
Add segment manager unit tests (#7525) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-07 13:59:58 +08:00
bigsheeper b7103357fd
Load failed if collection beyonds the memory limit (#7434) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-07 11:35:18 +08:00
congqixia fb4e23bc79
Add datacoord server unit tests (#7499) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-06 17:02:41 +08:00
congqixia 1f590d6c26
Add datacoord services related unit tests (#7467) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-06 11:12:42 +08:00
dragondriver 960741ee16
Support more detailed metrics information (#7472) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 19:35:47 +08:00
dragondriver 1db573731b
Cache the system information metrics (#7436) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 17:15:26 +08:00
dragondriver 3721cbfea1
Proxy metrics integrates with other coordinators (#7400) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 17:35:00 +08:00
dragondriver 89748ac639
Expose metrics of data cluster (#7177) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 10:13:15 +08:00
congqixia 0281bd8d22
Change context to correct package (#7331) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-27 15:37:56 +08:00
Xiangyu Wang c62f23e201
[skip ci]Remove neza2017 from reviewers (#7324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-27 00:27:36 +08:00
sunby 4d56d53c0b
Add a tool to move binlog path to SegmentInfo (#7243) 
issue: #7214
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-24 15:51:51 +08:00
Xiaofan 58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-24 09:45:51 +08:00
congqixia f5451d12f3
Refactor datacoord allocator (#7226) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-23 17:59:51 +08:00
congqixia be5c492e5a
Add datacoord unit tests (#7197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-20 17:50:12 +08:00
sunby 6e34f4c7f1
Add a seal policy which restrict the lifetime of a segment (#7172) 
issue: #7164
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-20 15:42:12 +08:00
sunby 3cc210da26
Add flush interval (#6996) 
Time tick message is sent about per 0.2 seconds. DataCoord will call
`FlushSegment` too frequently. We add a min interval for segment flush. A
segment will called `FlushSegment` only once between the interval.

issue: #6995
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 14:08:10 +08:00
sunby 4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
sunby 9551dc3b7e
Fix segment auto flush bug (#7085) 
issue: #7084
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-16 11:00:09 +08:00
XuanYang-cn 6c5be83ea1
Refactor NewEtcdKV API (#7075) 
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 11:04:09 +08:00
sunby 5c383d1d02
Add a checker to listen for the timetick. (#7061) 
issue: #7060
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-12 19:58:08 +08:00
Xiangyu Wang 5e78fd6a97
[skip ci]Update approvers (#7063) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-12 19:36:09 +08:00
sunby 31182e52a9
Fix expire allocation bug (#7053) 
issue: #7052
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-12 10:48:08 +08:00
sunby 454620dfaa
Add unit test in datacoord (#6985) 
issue: #6984
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-04 16:59:25 +08:00
sunby 23bffe399d
Enhance log print in data coordinator (#6681) 
issue: #6680
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-28 11:43:22 +08:00
zhenshan.cao 8a1a841011
FixBug: Incorrect handling of time synchronization (#6814) 
* FixBug: proxy send wrong sync timetick and DataCoord did not filter channel

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-07-27 14:34:48 +08:00
sunby aa20d88f6f
Fix datacoord stop bug (#6642) 
issue: #6641
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-24 09:25:14 +08:00
sunby fc74a0f578
Auto create new segments when allocating rows more than (#6665) 
max number of rows per segment

If user insert too much rows in a request. Now we will return a failed
response. Maybe auto creating new segments to hold that much rows is a
better way.

issue: #6664
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-23 21:58:33 +08:00
sunby 63387c83ef
Remove collection check when saving binlog (#6658) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-23 21:23:22 +08:00
sunby b98b30b96f
Fix datacoord_test.go failed (#6558) 
Create a mock rootcoord when testing datacoord to hide the network layer
issue: #6557
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-19 10:41:15 +08:00
sunby ee38d14f5d
Fix channel lost after data node crash (#6545) 
If we start up 2 data nodes and one of them crashes. We expect that all
channels of crashed node will be allcoated to the alive node. But now we
discover that these channels are lost after data node crash. The reason
is we pass a NodeInfo with empty channel info. We fix it and improve log
print.

issue: #6501
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-15 16:38:31 +08:00
groot 6b001fd76a
fix debug error in IDE (#6539) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-07-15 14:27:54 +08:00
sunby 83be910baa
Refactor segment manager (#6441) 
issue: #6440
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-12 17:24:25 +08:00
sunby a8e5fd2024
Refactor cluster in dataservice (#6356) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-12 11:03:52 +08:00
sunby 017484e093
Refactor meta in data coord (#6300) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-07 14:02:01 +08:00
sunby 836a45ec26
Remove sending newSegmentMsg to rootcoord (#6301) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-06 09:24:05 +08:00
congqixia 76fd9ad339
Fix restart node logic (#6265) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-07-02 16:48:30 +08:00
sunby 07b0989628
Add GetFlushedSegments in data coordinator (#6253) 
Send segment flush message by grpc and add a new interface
`GetFlushedSegments`

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-02 11:16:20 +08:00
neza2017 6019c193fc
add grpc interface to process NewSegment And FlushSegmentCompleted (#6247) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-07-01 14:58:17 +08:00
congqixia 580e3a57cf
stop add new node into cluster before validated (#6241) 
* stop add new new into cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix startup & test cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-07-01 14:32:16 +08:00
congqixia 3119ce2a6c
Stop create session in watch op (#6222) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-30 16:50:20 +08:00
congqixia ae072b4f1d
Add candidate management for datacoord dn cluster (#6196) 
* Add candidate management for datacoord dn cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix test cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-30 10:20:15 +08:00
sunby dd3ba9b2b3
Add rev by one when start watch event (#6152) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-29 14:52:14 +08:00
congqixia c49ce4ddc2
Add datacoord Server comment and improve para (#6182) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-29 10:46:13 +08:00
congqixia b27e6b52bf
Force cluster refresh for each dn change event (#6161) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-28 13:28:14 +08:00
yukun 39614aa8eb
Add rocksmq_path in config and paramtable (#6099) 
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 19:44:11 +08:00
sunby 25d3fead9f
Change error type to error functions in data coord (#6076) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-24 19:05:42 +08:00
godchen 3785bb6330
Change retry times (#6083) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-24 19:05:06 +08:00
congqixia f6761bbc33
Refactor datacoord segment status (#6039) 
* Refactor datacoord segment status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove lastExpireTime

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-24 14:20:10 +08:00
yukun c8bf9078ae
Change pchannels to vchannel in retrieveResults (#6054) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-24 13:40:03 +08:00
sunby 46f86a0354
Return sealed segments when calling flush (#6011) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 16:56:11 +08:00
congqixia d6259d2dd6
Refactor policy (#6003) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-23 14:28:08 +08:00
sunby a529410b38
Fix data race in internal/datacoord/meta.go (#6005) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 12:20:06 +08:00
sunby fe8432016d
Refactor data coordinator (#6008) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 12:10:12 +08:00
godchen fd07a44dba
Fix error in retry refactor (#6000) 
* Fix error in retry refactor

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 11:48:06 +08:00
godchen 99be4c09a5
change retry (#5996) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 09:24:10 +08:00
sunby 0978b93a21
Refactor data coordinator (#5982) 
Rename variable name and make error msg more clear

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 18:24:08 +08:00
sunby d2c8462dac
Rename params in data_coord.yaml (#5968) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 16:18:21 +08:00
Cai Yudong d5f4ee6f44
Rename proto master/index_service/proxy_service (#5973) 
* rename proto proxy_service to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto index_service to index_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto master to root_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-22 16:14:09 +08:00
sunby c863a193de
Rename filename from dataservice to datacoordinator (#5946) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 10:42:07 +08:00