milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,060 Commits
127 Branches
113 Tags
553 MiB
Commit Graph

297 Commits (e0291df43d744f3cd2a7b12c8d2c6d0779b9c76d)

Author SHA1 Message Date
godchen 937dbf4279
Add flushed segments (#9476) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-08 19:09:12 +08:00
Cai Yudong b9f5a6bb1e
Rename some variables in unittest (#9484) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-08 17:47:25 +08:00
Cai Yudong ec9ec97cec
Fix typo (#9482) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-08 17:45:35 +08:00
Cai Yudong 78755263e2
Update ddnode test (#9477) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-08 17:43:46 +08:00
congqixia a1651104c3
[skip ci] Fix datanode Cache comment typo (#9483) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-08 17:22:54 +08:00
Cai Yudong 3dac2611b6
Remove redundant variable load (#9452) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-08 14:49:08 +08:00
Cai Yudong 12d640a023
Update filterSegmentByPK (#9450) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-08 12:04:56 +08:00
XuanYang-cn a1f9e46e44
[skip ci]Update comment in datanode package (#9428) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-07 23:50:52 +08:00
XuanYang-cn 9526d22dc7
[skip ci]Add comment for normal2FlushedSegment (#9431) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-07 23:46:37 +08:00
godchen 445f5426d1
Check data node error (#9303) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-07 22:16:56 +08:00
congqixia 908b7df41d
Add comment and implementation assertion for ddnode (#9333) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-07 19:54:56 +08:00
godchen cd4205e447
[skip ci]Fix allocator comment typo (#9363) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-10-06 23:32:27 +08:00
XuanYang-cn e7e37448d2
[skip ci]Add comment for flowgraph insertBufferNode (#9361) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 23:30:21 +08:00
XuanYang-cn 5bb832e6cc
[skip ci]Add comment on flowgraph dmstream inputnode (#9359) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 23:02:08 +08:00
XuanYang-cn 27143c6095
[skip ci]Add comment on flowgraph ddNode (#9358) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:51:58 +08:00
XuanYang-cn 7b14b0e63d
[skip ci]Add comment on readBinary (#9355) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:28:11 +08:00
XuanYang-cn 65a76b9bb5
[skip ci]Add comment in insertbuffernode (#9354) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:20:05 +08:00
godchen 83228c35ad
Add flowgraph insert buffer node error check (#9310) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-06 21:06:23 +08:00
XuanYang-cn e7aa5eebd6
[skip ci]Update comment of cache in datanode (#9313) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 07:08:24 +08:00
XuanYang-cn cb6d853b56
[skip ci]Add README.md in datanode (#9311) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-05 23:12:03 +08:00
XuanYang-cn f49e35a7fb
[skip ci]Update datanode param table comment (#9308) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-05 22:54:03 +08:00
godchen e6ee4f39bb
Add flow graph dmstream input node test error check (#9213) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-04 17:37:57 +08:00
godchen 3eee4e4259
Add segment replica test error check (#9212) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-04 17:36:06 +08:00
godchen 83f256c31b
Remove datanode mock test error (#9211) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-04 17:34:16 +08:00
congqixia becf6c8f7d
Add DataNode implementation Assertion (#9144) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-03 19:46:05 +08:00
congqixia 109186db9c
[skip ci] Add comment for datanode meta_service.go (#9036) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-01 22:25:24 +08:00
Xiaofan 2c2b800ff4
Change Log Name with NodeId (#8721) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-01 08:52:50 +08:00
congqixia 6bbf0a8f8f
Read datanode insertbuf from env (#8942) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-30 17:45:07 +08:00
congqixia 10a6f7f146
Defer buf removal from insertBuf after SaveBinlog succeeds (#8930) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-30 17:42:11 +08:00
zhenshan.cao d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-29 20:26:00 +08:00
congqixia db944cd09c
Refactor flowgraph and related invocation (#8770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-29 20:19:59 +08:00
Cai Yudong d94361fdcc
Add parameter partitionID for API filterSegments (#8813) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-29 10:27:58 +08:00
XuanYang-cn 3be6672753
Add flush channel for delete_node (#8762) 
Resolves: #8761

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 18:22:16 +08:00
congqixia d374ea71ba
Unify liveness check in datacoord and datanode (#8480) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-28 16:10:04 +08:00
XuanYang-cn b150f57927
No buffer also means flushed (#8686) 
Resolves: #8684

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 11:18:03 +08:00
Xiaofan 585989935a
Revert the logger change (#8677) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-27 17:37:57 +08:00
godchen 51151b7b95
Fix delete node ut error (#8674) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-27 17:07:58 +08:00
godchen 77a773dff0
Change segment replica struct (#8608) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-27 14:38:00 +08:00
congqixia e7117f67ee
Add BroadcastMark for Msgstream returning MessageIDs (#8654) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-27 14:10:09 +08:00
zhenshan.cao b71bffe26f
Replace ChannelID with ShardName in DML Request (#8539) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-27 10:01:59 +08:00
Xiaofan d38adda8c2
Rename log file based on node id (#8517) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:18:11 +08:00
XuanYang-cn 1307782093
[skip-ci]Fix golint in param_table in datanode (#8602) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:57:59 +08:00
XuanYang-cn ebd894d8b7
Refactor buffer size calculation (#8565) 
This PR changes the datanode insertBufSize config
from 32000 number of rows to 16MB

Resolves: #7741

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:55:59 +08:00
XuanYang-cn b376107ab3
[skip-ci]Fix golint in segmentRep (#8601) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:51:58 +08:00
XuanYang-cn b13170f83e
[skip-ci]Fix golint in flowgraphnode.go (#8600) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:49:59 +08:00
groot e4e762f2e8
Reduce go unittest time cost (#8503) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-26 12:50:24 +08:00
Cai Yudong b2e5627061
Rename insertMsg to flowGraphMsg in datanode flow graph (#8546) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-26 10:43:57 +08:00
Cai Yudong 28d305148a
Update ddnode insert (#8536) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-26 09:43:57 +08:00
XuanYang-cn 3aabc1d3ee
[skip-ci]Fix golint in type_def.go (#8464) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 21:43:56 +08:00
sunby 816bac82ec
Add root path in minio and replace etcd.rootPath in datanode (#8272) 
issue: #7905
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-24 20:50:23 +08:00
XuanYang-cn e4f0d4bab6
[skip-ci]Fix golint in flow_graph_message.go (#8463) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 20:46:04 +08:00
XuanYang-cn 2539dd4593
[skip-ci]Fix golint in data_node.go (#8462) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 20:44:05 +08:00
XuanYang-cn faeb1bf50f
Add comment on exposed struct (#8348) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 20:15:54 +08:00
XuanYang-cn 2b82482679
Fix golint and ruleguard (#8347) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 19:14:01 +08:00
XuanYang-cn a830c28bd8
Fix golint in datanode (#8345) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 19:05:55 +08:00
congqixia 2aa9deda1c
Improve logic when watch dm channel fails (#8386) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 18:31:55 +08:00
SS 58a178fdf5
[skip ci] Fix grammar in comment (#8382) 
Signed-off-by: sida shen <sida@sidas-MacBook-Pro.local>

Co-authored-by: sida shen <sida@sidas-MacBook-Pro.local>
 2021-09-23 16:50:16 +08:00
XuanYang-cn d816a81ee3
Make FlushSegments async (#8159) 
Resolves: #8122

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 16:03:54 +08:00
congqixia 6469d83072
Add datanode etcd alive check (#8363) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 14:29:54 +08:00
Xiaofan f5173b595f
Add a channel prefix for all channels (#8166) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-23 10:53:53 +08:00
godchen 050e27cbcf
[skip ci]Add delete node comment (#8323) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-22 20:02:09 +08:00
Xiangyu Wang c314b2b4a8
Remove unused label rules (#7929) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-18 18:19:51 +08:00
groot 334297b49d
[skip ci] Update OWNERS files (#8211) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-18 15:30:37 +08:00
congqixia bdfb8768d4
Enable gosec (#8202) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-18 14:45:50 +08:00
XuanYang-cn 46c72c57c2
Add BufferData in insertbufferNode (#8138) 
Optimize bufferInsertMsg param to avoid a second transfer
of endposition

See also: #8058, #7741

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-18 14:25:50 +08:00
Cai Yudong d10c6abc9e
Apply for msgstream from pool when creating collection (#7738) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 09:13:50 +08:00
dragondriver daf001f683
Expose created time and updated time to datanode metrics (#8173) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-17 21:32:47 +08:00
XuanYang-cn 6c434b4801
Improve codes in insertbuffernode (#8054) 
Add error handling in insertbuffernode

See also: #7624, #7684

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-17 16:27:56 +08:00
XuanYang-cn 82d4d19337
Remove not in use segmentCheckPoints (#8039) 
Resolves: #8037

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-17 16:25:55 +08:00
XuanYang-cn 3db76e8e59
[skip-ci]Add comment in data sync service (#8002) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-15 21:25:49 +08:00
XuanYang-cn 376d39fa02
[skip-ci]Add comment for Cache and allocator (#7924) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-15 10:33:37 +08:00
dragondriver a872a4357c
Add more test cases for proxy implementation (#7685) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-11 11:36:22 +08:00
XuanYang-cn ce33855652
[skip-ci]Fix datanode comment typo (#7689) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-10 14:46:00 +08:00
Cai Yudong 4e23ed9a18
Revert apply physical channels from pool (d5f7e358 and 3b960969) (#7687) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-10 13:00:00 +08:00
sunby 06cee85a39
Add data node unit tests (#7622) 
Issue: #6357
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-09 16:40:00 +08:00
XuanYang-cn 64aad49959
Add ut in datasync service to 88% (#7615) 
See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-09 15:36:01 +08:00
congqixia 69794fd32d
Refactor insertBufferNode and add unit tests (#7621) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-09 15:00:00 +08:00
XuanYang-cn 762060e670
Add ut for segmentReplica to 100% (#7589) 
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-09 10:14:00 +08:00
XuanYang-cn 074b610811
Add ut for dmstream_input_node to 100% (#7593) 
See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-08 19:12:00 +08:00
XuanYang-cn a5767e05c6
Add ut for flowgraph_delete_node to 100% (#7550) 
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-08 10:41:59 +08:00
XuanYang-cn 29756c6ce8
Add ut for flow_graph_dd_node to 100% (#7511) 
This PR:
- Change mutex to sync.Map
- Add ut for flow_graph_dd_node.go to 100%

See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-07 15:41:59 +08:00
dragondriver 960741ee16
Support more detailed metrics information (#7472) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 19:35:47 +08:00
dragondriver 89748ac639
Expose metrics of data cluster (#7177) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 10:13:15 +08:00
Cai Yudong 3b9609692b
Apply for msgstream from pool when creating collection (#7377) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 18:35:58 +08:00
Cai Yudong f4b53df727
Revert "Apply for msgstream from pool when creating collection (#7323)" (#7376) 
This reverts commit e6de86a433.
 2021-08-31 11:07:58 +08:00
Cai Yudong e6de86a433
Apply for msgstream from pool when creating collection (#7323) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 09:47:58 +08:00
congqixia 65f168eac1
Fix datanode corner cases (#7336) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-30 10:03:58 +08:00
godchen c333af0dcf
Add query node segment filter (#7303) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-28 10:12:00 +08:00
godchen b4c55f6b3c
Add delete node (#7261) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-25 11:41:52 +08:00
congqixia daae05641a
Construct bloom filter when inserting (#7268) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-25 10:29:52 +08:00
Xiaofan 58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-24 09:45:51 +08:00
congqixia 881d0af1fc
Add sync logic in TestWatchChannel (#7188) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-20 17:48:13 +08:00
sunby 4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
XuanYang-cn 86ca96a02e
Fix master branch unittest error (#7083) 
Resolves: #7080

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 14:00:14 +08:00
XuanYang-cn 6c5be83ea1
Refactor NewEtcdKV API (#7075) 
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 11:04:09 +08:00
congqixia e1b3273d1c
Add Datanode watch etcd channel (#6965) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-13 10:50:09 +08:00
Xiangyu Wang 5e78fd6a97
[skip ci]Update approvers (#7063) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-12 19:36:09 +08:00
XuanYang-cn 9d81a90402
Fix mem leak of flush cache in DataNode (#6862) 
Resolves: #6858

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-11 14:24:09 +08:00
XuanYang-cn 47df4a2e57
Fix DataNode BackGroudGC problem (#6551) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-03 09:59:24 +08:00
bigsheeper 7307332fab
Add nil check (#6832) 
* add nil check

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* use grpc functions

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* use grpc functions

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix indexCoord crash

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-29 20:11:23 +08:00