milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
15,548 Commits
141 Branches
117 Tags
549 MiB
Commit Graph

5748 Commits (928a213e31f4cc8abe2a79a915d3a4fff2e61875)

Author SHA1 Message Date
yah01 66a04450ad
Fix LoadBalance Sync all segments to shard leaders (#18002) 
Signed-off-by: yah <yang.cen@zilliz.com>
 2022-07-01 19:18:24 +08:00
yah01 27eca2881a
Clean segments as releasing collection (#17932) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-07-01 19:16:25 +08:00
zhenshan.cao f33b090819
Fix bug: datarace in proxy unittest (#17990) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-01 17:56:21 +08:00
Jiquan Long 47c9e77155
Fix querycoord ut data race (#17980) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-01 10:58:18 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully (#17959) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-01 10:44:18 +08:00
codeman 6ab850bedf
Fix update credential failure (#17963) (#17970) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-30 21:34:18 +08:00
congqixia f4177f4434
Remove growing according to offline as well (#17968) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-30 20:14:19 +08:00
cai.zhang 01fc411566
Handle etcd compacted error (#17886) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-30 18:54:19 +08:00
bigsheeper cdbd75d4dd
Log delta channel timetick in DataNode (#17947) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-30 17:44:18 +08:00
zhenshan.cao 0baeb609dc
Fix bugs in monitoring metrics (#17894) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-30 17:26:19 +08:00
Xiaofan a803f9e4b0
Fix Wait Group Race in Msgstream (#17931) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-30 10:38:18 +08:00
yah01 ef6859a4d6
Refactor handoff handler (#17806) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-29 23:26:19 +08:00
codeman 95a3830dc6
fix password comparison too slow (#17874) (#17897) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-29 20:02:18 +08:00
XuanYang-cn ff3c4ab2a8
Fix compaction panic (#17840) 
See also: #17823

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-29 16:54:17 +08:00
Bingyi Sun 24b3ad22d7
Fix msgstream close block. (#17880) 
issue: #17513
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-29 16:24:20 +08:00
congqixia c179986e6c
Fix shard cluster segments data race (#17903) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-29 16:20:19 +08:00
congqixia 106f19a7b8
Fix querycoord cannot update deltaChannel info (#17901) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-29 15:38:17 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-29 14:20:19 +08:00
jaime 762e3b78a6
support configurable sasl mechanisms for kafka (#17542) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-29 10:32:17 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao ea901393ef
fix windows ldflags (#17807) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-28 19:58:25 +08:00
xing.zhao 3904a78211
Remove genReqID function in import manager bacause the request ID is useless (#17853) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-06-28 18:20:18 +08:00
bigsheeper b657e58370
Remove temporary variable to prevent memory fragmentation (#17728) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-28 16:30:23 +08:00
cai.zhang 899a9bf86d
IndexNode still writes finished state to ETCD when task was deleted (#17761) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-28 12:08:17 +08:00
cai.zhang b883d1f5dd
Make sure indexes are successfully dropped when DropCollection (#17780) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-28 11:24:21 +08:00
bigsheeper 6b79d4f7bc
Log segment type as possible in QueryNode (#17846) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-28 10:16:17 +08:00
XuanYang-cn 6b72f9013f
Fix message type of show partitions (#17821) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-28 10:06:17 +08:00
yah01 797218a8ad
Fix handoff isn't processed correctly (#17738) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-27 22:08:17 +08:00
cai.zhang ef0fc37ad7
Fixed the bug that IndexCoord lost some meta information (#17818) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-27 21:52:17 +08:00
wayblink 46e0e2658b
Move datacoord related methods from meta to globalMetaBroker (#17812) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-27 21:04:17 +08:00
congqixia 6840486efc
Fix load progress complete before syncReplica (#17828) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-27 20:52:16 +08:00
Ten Thousand Leaves f0b036a35a
Move bulk load segment lock happen early (#17612) 
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-27 13:56:17 +08:00
congqixia 68d385137a
Fix: Miss channel info when load growing segment (#17789) (#17797) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: zhagnlu <1542303831@qq.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-06-27 12:02:16 +08:00
Bingyi Sun 9d2e5aacbc
Fix missing dml nodeIDs when recovering from old version (#17800) 
issue: #17416
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-26 18:08:16 +08:00
congqixia 5471e35ccc
Wait last version search to release growing safely (#17713) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-25 18:48:15 +08:00
yah01 8388478ef3
SyncReplicaSegments syncs all segments (#17774) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-24 23:24:15 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
bigsheeper 230404499d
Check duplication before adding channel info in DataCoord (#17742) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-24 16:30:15 +08:00
shaoyue 96b97f7f58
RESTful api use same port as metrics & healthcheck (#17732) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-24 15:34:14 +08:00
Bingyi Sun 05273309e8
Drop empty sealed segment (#17708) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-24 15:16:14 +08:00
xige-16 3a0f9d41df
Print log after segment search done (#17769) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-24 14:10:14 +08:00
yah01 94a5122044
Fix LoadBalance removes source nodes from DmChannel (#17744) 
Only NodeDown LoadBalance should remove source nodes

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-24 12:40:14 +08:00
wayblink 1a336bbbf1
Fix: WatchDmChannelsRequest can be too large to save in etcd (#17722) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-24 10:54:15 +08:00
SimFG 6189a117a4
Remove the `WithOptions` for avoiding the log object being cloned continuously (#17746) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-24 10:36:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-24 10:34:18 +08:00
cai.zhang e8f53af749
When the task is canceled, reduce the load of node (#17735) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-23 19:22:15 +08:00
Jiquan Long 90ca61d948
Fix double-freed global variable in easylogging lib (#17730) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-23 19:12:14 +08:00
Cai Yudong e78269f450
Optimize search related interface in segcore (#17568) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-23 18:16:13 +08:00
XuanYang-cn 21b52bfcf2
Enhance channel manager ut (#17682) 
Resolves: #17676

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-23 18:08:14 +08:00
codeman 294f7e4408
fix: log datapb.SegmentInfo is too large #17685 (#17727) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-23 14:36:13 +08:00
xige-16 8ff1a2c4c2
Print expr when proxy receive qyery req (#17662) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-23 13:42:13 +08:00
Jiquan Long 216e2f80aa
Add msg id to log in query path (#17677) 
/kind improvement

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-23 10:46:13 +08:00
cai.zhang 1fbdafc943
IndexCoord does not assign task to IndexNode prematurely (#17717) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-23 10:40:13 +08:00
SimFG 50d5d04552
Refine some codes about the datanode (#17720) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-23 09:58:14 +08:00
SimFG f4c6a6734e
Fix rpc retry policy not taking effect (#17673) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-22 22:22:13 +08:00
SimFG 3fadf4c5d7
Return the unavailable error to make the client's retry policy take effect (#17719) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-22 21:56:13 +08:00
codeman 6cb00aa5bb
fix: log datapb.SegmentInfo is too large #17685 (#17693) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-06-22 21:46:13 +08:00
yah01 b4f21259ef
Fix SegmentChangeInfo is not set properly (#17701) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-22 21:08:13 +08:00
Xiaofan 6fdf88f452
Fix compaction selection policy (#17690) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-22 19:00:14 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore (#17700) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow (#17706) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-22 16:56:13 +08:00
bigsheeper 8cf54137cf
Add nil check in reduce and add collection rwmutex when release segments (#17698) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 16:12:14 +08:00
yah01 a077bad84a
Make updating replica atomic, balance plan idempotent (#17686) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-22 13:40:13 +08:00
Xiaofan 7d51b65260
Estimate varchar length more reasonablly (#17688) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-22 13:02:13 +08:00
XuanYang-cn 12158432bb
Clear segment cache when closing flowgraph (#17671) 
See also: #17537

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-22 10:16:14 +08:00
Enwei Jiao 746aeea35b
fix typo (#17668) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-21 18:54:13 +08:00
cai.zhang 4efdd0929c
Get index status based on segment generation time (#17659) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-21 18:44:12 +08:00
XuanYang-cn e43f487f02
Make LoadCollection compatible (#17669) 
See also: #17656

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-21 17:26:15 +08:00
yah01 1e87b54542
Add the offlines nodes previously crashed to offline nodes channel (#17663) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-21 16:08:13 +08:00
Jiquan Long 0282f69eb4
Add msg id to log in search path. (#17654) 
issue: #17513
Add msg id to log in search path,
which should help us to trace a detailed request more easier.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-21 13:30:12 +08:00
congqixia 37464a281d
Fix datasorter swap vector logic (#17661) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-21 11:38:12 +08:00
SimFG eb1f0bc805
Add the consistency check tool (#17136) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-21 10:28:12 +08:00
shaoyue 32a3ed0791
Add e2e test for RESTful APIs, fix json encoding for on proto_oneof fields (#17362) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-20 22:06:12 +08:00
cai.zhang ed94ecf847
Segments can't be compacted when they have reference lock (#17649) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 21:56:12 +08:00
cai.zhang 3610cc77bf
Fix bug for align segments (#17652) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 20:16:12 +08:00
cai.zhang f123d657ed
Fix bug for dropping index (#17635) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 15:22:13 +08:00
Xiaofan e36fce1fd4
Add log for mqtt msgstream seek (#17640) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-20 15:18:13 +08:00
wayblink f2bd910df5
[skip e2e]Fix log mistake: WatchDmChannels -> WatchDeltaChannels (#17643) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-20 15:10:12 +08:00
congqixia 12b3a29db8
Fix segments info in drop virtual channel request is not set (#17641) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-20 13:40:12 +08:00
Xiaofan d70f483806
Support rocksmq dynamic lru cache size (#17626) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-20 10:56:12 +08:00
Xiaofan 516d91470f
Fix Delta log loss after compaction (#17634) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-19 20:12:11 +08:00
congqixia 50cecc65ed
Fix mock querynode server session not revoked (#17229) 
Revoke mock querynode server session when it's stopped
This PR reduces the running time of TestLoadBalanceIndexedSegmentsAfterNodeDown from 60+ seconds to less than 1+ seconds
Also related to #17212 #17215

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-18 18:22:12 +08:00
Xiaofan 144f7a2d80
Refine meta, remove redundant proxy data (#17631) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-18 12:50:11 +08:00
Letian Jiang b15c24a554
Acquire reference lock of compacted from segments during index building (#17619) 
- Accquire reference lock during index building
- Keep dropped segment record

Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-18 01:24:11 +08:00
groot 7c69f4b338
Fix bulkload failed bug caused by dropping collection (#17624) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-17 21:14:11 +08:00
groot 371a44f018
Fix a bug of bulkload (#17620) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-17 20:52:11 +08:00
congqixia aa4b42c74b
Remove wait before release collection/partition (#17623) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-17 19:00:11 +08:00
wayblink 074ec3060a
Support return dropped segments info in GetSegmentInfo rpc (#17617) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-17 18:24:12 +08:00
cai.zhang 141a3bbe00
Mark deleted before drop index (#17490) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-17 18:08:12 +08:00
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search (#17565) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-17 17:46:10 +08:00
congqixia 785a5a757f
Use segment version instead of ref cnt (#17609) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-17 17:38:12 +08:00
congqixia cc3ecc4bd5
Make querycoord channel allocator respect context (#17552) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-17 16:02:12 +08:00
Jiquan Long 0e29c37499
Forbid system fields in user schema (#17613) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-17 14:10:11 +08:00
yah01 f5fa93aa0b
Return err if failed to assign segments/channels to nodes (#17616) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-17 12:42:11 +08:00
SimFG 2008f60e50
Simplify the implementations of the ToPhysicalChannel and the ConvertChannelName func (#17605) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-17 10:22:11 +08:00
yah01 0f87763682
Fix LoadBalance may not update the shard leader (#17608) 
This happens probably with concurrent updating replicas,
some goroutines modify the nodes list of replicas,
and the others modify the shard leaders of replicas

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-17 08:58:10 +08:00
cai.zhang 53a49a9255
GetIndexState only get the state generated call CreateIndex (#17549) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-16 20:12:11 +08:00
yah01 3f42f5f345
Set the task state to TaskFailed if error occurs (#17598) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-16 19:18:11 +08:00
Jiquan Long 8e2797fe30
Print error when failed to broadcast message (#17595) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-16 18:32:10 +08:00
shaoyue 04f836f516
Fix s3 delete file (#17574) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-16 18:14:10 +08:00
Letian Jiang be4077421e
Filter duplicated sealed segments which are both online and offline (#17593) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-16 17:42:11 +08:00
Xiaofan 1f6fbf91b2
Fix pulsar unsubsribe issue (#17562) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-16 17:28:11 +08:00
Jiquan Long 2f66531fdf
Fix wrong IP distances (#17590) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-16 16:34:11 +08:00
yah01 7d5c8c5f38
Fix bug not remove offline node (#17560) 
The LoadBalance task won't remove the offline node if the node never load/watch any segment/dmchannel

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-16 16:00:10 +08:00
groot b5e62023ea
Return error when bulkload pending list hit limit (#17570) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-16 13:02:10 +08:00
wayblink eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC (#17441) 
Resolves: #17233 #16047

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-16 12:00:10 +08:00
Xiaofan 54aeb077e9
Refine compaction selection poliy (#17486) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
cai.zhang ea5041aec2
Acquiring the segment reference lock on task level (#17544) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-15 21:38:10 +08:00
Jiquan Long 875b6f88b0
Add channel information to flow graph node (#17349) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-15 21:04:10 +08:00
Ten Thousand Leaves 51a9e54b21
Fix a bug that causes infinite loop (#17550) 
This will cause AddNode() to keep its lock forever so DataCoord would just hang

issue: #17366
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 20:50:10 +08:00
xige-16 98b32f3eba
Improve time record log in search/query (#17545) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-15 20:48:10 +08:00
xige-16 dd483beefd
Delete the log that prints a lot of deleteMsg.timestamps (#17543) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-15 20:46:14 +08:00
Bingyi Sun d4fdfab818
Fix VDC-510 (#17554) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-15 19:58:10 +08:00
xing.zhao 76523b313b
Fix bug #17403 by double check collmeta prefix and alias prefix (#17548) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-06-15 19:14:10 +08:00
SimFG e825215c44
Remove the redundant 'for' in the TaskCondition (#17558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-15 16:32:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
Jiquan Long e751663fdc
Fix err msg when the dimension of query vectors mismatch field (#17553) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-15 10:54:09 +08:00
congqixia f9553970f9
Add BindContext function for querycoord task scheduler (#17531) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-14 17:12:09 +08:00
congqixia ed3a1164dd
Handle wrapped grpc error in funcutil.IsGrpcError (#17547) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-14 17:10:10 +08:00
groot ab9b186f15
Change bulkload bufsize for row-based parser (#17473) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-14 16:18:09 +08:00
bigsheeper cdcdfa1ea5
Disable jemalloc and use malloc_trim instead (#17538) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-14 11:12:09 +08:00
xing.zhao e9dcda161f
Add aliases return value in describeCollection task (#17516) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: 赵星 <zhaoxing@zhaoxings-MacBook-Pro.local>
 2022-06-13 19:22:10 +08:00
congqixia a0d3f37c0d
Sync grpc DefaultKeepAliveTimeout to milvus.yaml (#17519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-13 16:18:09 +08:00
XuanYang-cn e6225d923b
Cancel timers while adding new timer (#17511) 
See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-13 14:46:08 +08:00
congqixia d1de8cabdd
Fix %w verb for error wrapping in grpcclient (#17484) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-13 12:48:08 +08:00
xige-16 16f0815184
Add time record log on search/query execution path (#17497) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-13 11:42:09 +08:00
xige-16 36ad989590
Fix segOffset grater than insert barrier when mark delete (#17444) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-10 20:02:08 +08:00
Enwei Jiao a5b008acec
ignore getReplica's error when handle rebalanceTask (#17469) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-06-10 14:50:08 +08:00
jaime b64a467311
Revert "Use unified catalog interface to reconstruct metastore (#17042)" (#17480) 
This reverts commit 218326bafe.

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-10 13:10:08 +08:00
bigsheeper 2d9a52206d
Use jemalloc in QueryNode, DataNode and IndexNode (#17470) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-10 11:24:08 +08:00
XuanYang-cn 2b5340581c
Skip remove if reassigns to the original node (#17450) 
Fix ut race

See also: #15966, #17432

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-10 10:54:07 +08:00
Xiaofan 66f26943f8
Add Err code when task rollback (#17472) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-09 21:08:07 +08:00
Xiaofan 05a918e0ab
Refine logs (#17438) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-09 21:04:07 +08:00
congqixia 97a871cc82
Make querycoord segment allocator respect context (#17452) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-09 20:14:06 +08:00
Letian Jiang dfaed5acdd
Add QueryServiceAvailable field in ShowCollections (#17456) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-09 18:20:07 +08:00
yah01 a38790f586
Predict the loading memory usage more accurate (#17451) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-09 18:16:07 +08:00
Jiquan Long 2ca81620ea
Reduce lock operations when get dml stream (#17468) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-09 17:34:09 +08:00
groot 9210299706
Fix bulkload OOM issue (#17461) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-09 16:52:07 +08:00
xige-16 6d078ebe7a
Print more information when loading fields with different lines (#17448) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-08 20:16:06 +08:00
wayblink 25883840b2
Remove delete log in datanode because it may influence datanode performance (#17415) 
Resolves #16984

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-08 19:10:07 +08:00
jaime dbef4c6ec2
Align kafka consume from latest with puslar and rocksmq (#17433) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-08 19:00:07 +08:00
XuanYang-cn 4f66bcbb22
Fix DataNode processes event out of order (#17440) 
The probability is low so very unlikly to reproduce

See also: #15966

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-08 17:24:07 +08:00
Xiaofan 6426e3d037
Split Etcd save operation into small txns (#17386) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-08 15:08:07 +08:00
yah01 70f8bea4b4
Avoid growing slice as deserializing binlogs (#17421) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-08 11:46:06 +08:00
XuanYang-cn 1dbfebf827
Add debug log in removing assignment (#17427) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-08 10:44:06 +08:00
bigsheeper af108b5bda
Add malloc_trim after reduce search result (#17423) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-08 00:12:05 +08:00
groot f31be941bb
Fix bulkload row count bug for multi datanodes (#17384) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-06-07 16:56:07 +08:00
xige-16 b5c11a216d
Alter varChar type params's name to max_length (#17409) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-07 15:58:06 +08:00
Xiaofan a9ec1f8864
Fix compatibility of upgrade (#17406) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-07 14:52:06 +08:00
congqixia de8f2f76c5
Fix shardLeader cache with ErrConnect check (#17402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-07 12:20:06 +08:00
jaime 218326bafe
Use unified catalog interface to reconstruct metastore (#17042) 
Co-authored-by: kejiang ke.jiang@zilliz.com
Co-authored-by: yun.zhang yun.zhang@zilliz.com

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-07 12:00:07 +08:00
bigsheeper f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 21:34:05 +08:00
zhenshan.cao 98e95275fe
Fix several code issues caused by carelessness (#17385) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-06 18:58:05 +08:00
bigsheeper 9f786dd752
Remove unnecessary bitset flip in Search and Query (#17387) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 17:38:05 +08:00
yah01 a2d2ad88bd
Make assigning segments faster (#17377) 
This improve the Load performance,
and let the LoadBalance fails fast, which allows us to retry it timely

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-06 16:52:05 +08:00
congqixia ac6394d0fd
Fix container/list removal in iteration (#17382) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-06 16:34:05 +08:00
Xiaofan af994b5e1c
Fix etcd watch error (#17364) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-06 16:26:06 +08:00
bigsheeper 45febac298
Extract lease of cProto in QueryNode (#17373) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 15:56:05 +08:00
XuanYang-cn c70af73005
Fix datacoord set wrong state for node registering (#17376) 
Fix datacoord datarace

See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-06 15:10:05 +08:00
bigsheeper 48cf63a2d7
Wait for flow graph to exit (#17341) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 19:48:04 +08:00
shaoyue 76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
SimFG 0986c29d7f
proxy tls support one-way authentication (#17348) 
Signed-off-by: xiyichan <2863768433@qq.com>

Co-authored-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>

Co-authored-by: xiyichan <2863768433@qq.com>
 2022-06-02 18:56:04 +08:00
Ten Thousand Leaves f5e63177d2
Fix an issue with bulk load with multiple DataNodes (#17324) 
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-02 18:54:04 +08:00
yah01 4e83f37899
Improve load performance (#17273) 
Load segment's fields concurrently

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-02 17:56:03 +08:00
shaoyue 7c52a8c535
Add RESTful API for new methods, fix err when receive empty boy (#17260) 
create collection schema use struct instead of protobuf

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 17:40:04 +08:00
xige-16 8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel (#17308) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-02 16:56:04 +08:00
zhenshan.cao 5f44e45480
Add metrics on QueryNode and Proxy (#17328) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 16:06:03 +08:00
Jiquan Long adf3b14027
Refine channels management in Proxy. (#17334) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-02 15:34:04 +08:00
bigsheeper 7a59a80506
Remove segmentPtr lock and use collection lock instead (#17303) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 14:00:04 +08:00
XuanYang-cn 42f643e727
Make DataNode release rather than delete when reassign (#17293) 
1. Reassgin now will assign to the original Node if no other nodes
   avaliable
2. Make AddNode balance async: ToRealse + Reassign

See also: #16114, #17270

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-02 13:56:04 +08:00
yah01 cc69c5cdd3
Make Cluster interface's methods called outside public (#17315) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-02 13:16:05 +08:00
congqixia c76b4ade01
Fix data race in LoadSegmentsTask (#17342) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-02 12:52:03 +08:00
zhenshan.cao 2763efc9b0
Add cache of grpc client of ShardLeader in proxy (#17301) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 12:16:03 +08:00
jaime 5698bf4236
Add sasl configurations for kafka (#17323) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-02 12:12:03 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS (#17316) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-02 10:48:03 +08:00
bigsheeper e0cbacba59
Check ignorable error to prevent unnecessary panic (#17317) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 10:12:03 +08:00
yah01 f5bd519e49
Add retry mechanism for NodeDown LoadBalance (#17306) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-01 20:00:03 +08:00
xige-16 63a31ccb38
Remove error return value when removeSegment (#17310) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-01 13:18:02 +08:00
Jiquan Long b4f797506d
Fix err msg of invalid expression (#17296) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-01 10:00:07 +08:00
Jiquan Long 72f454e054
Fix 16961, make creating dml stream idempotent (#17302) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-31 22:58:02 +08:00
XuanYang-cn d920e5c915
Fix DataNode panic while allocating IDs (#17294) 
See also: #17270

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-31 18:02:03 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence (#16782) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
groot affcc2acad
Fix bulkload entity number bug (#17278) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-31 15:40:04 +08:00
bigsheeper f55fe6be3c
Merge historical replica and streaming replica in QueryNode (#17217) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-31 13:42:03 +08:00
Xiaofan 9579a645c6
Support compile marisa on Macos (#17261) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-31 13:28:02 +08:00
Jiquan Long 6bd786bf50
Reduce default seal propotion (#17287) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-31 12:02:04 +08:00
cai.zhang f1bad98f00
Fix indexcoord ut (#17282) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 12:00:09 +08:00
congqixia c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster (#16856) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:50:04 +08:00
congqixia cecbd40a22
Fix Data race of loadType in querynode/collection.go (#17275) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:16:02 +08:00
bigsheeper 11834a0948
Improve query log in QueryNode (#17268) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-30 17:18:03 +08:00
zhenshan.cao b0d54ed0af
Rename Client to GrpcClient in mock package (#17230) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-30 17:16:08 +08:00
Jiquan Long ac2cadc83b
Refine segment type logs to string (#17248) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-30 15:30:01 +08:00
yah01 b09359b12f
Remove useless collection ID in error message (#17269) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-30 14:40:01 +08:00
SimFG 731870211a
Support role-based access control (only pb) (#16987) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-28 00:04:01 +08:00
congqixia 55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking (#17247) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-27 16:20:00 +08:00
zhenshan.cao ad9276a440
Add metrics for proxy and querynode (#17205) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-27 14:12:01 +08:00
xige-16 e196caf604
Force reclamation of memory after loading (#17244) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-27 12:50:00 +08:00
xige-16 3a63d6c98e
Fix load timeout in chaos (#17241) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 22:28:03 +08:00
Letian Jiang f2a27e0e64
Retry GetShardLeaders until service available or timeout (#17183) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-26 20:28:02 +08:00
yah01 5872c5afb6
Fix updating shard leaders may lost some modifications (#17218) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-26 17:14:02 +08:00
xige-16 56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
groot 3928da6493
Refine calc distance function (#17161) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-26 14:34:01 +08:00
xige-16 7d810ac159
Fix query empty when insert same pk after deletion (#17222) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-25 22:06:00 +08:00
Jiquan Long e44afc7aa3
Fix ut: wait service ready after run Proxy server (#17208) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 18:56:00 +08:00
congqixia 7409bfc56d
Make allocateNode run async in case of block offline event (#17185) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-25 18:53:59 +08:00
Jiquan Long c12ee2f5cb
Fix flow_graph_insert_buffer_node.go code style (#17216) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 17:03:59 +08:00
congqixia e06d6d7cc6
Returns collection instead of copy to make leaderMutex work (#17206) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-25 15:20:00 +08:00
yah01 de0ba6d495
Fix GetQuerySegmentInfo() returns incorrect result after LoadBalance (#17190) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-25 15:17:59 +08:00
Ten Thousand Leaves 5e1e7a6896
Fix an issue that causes flush hanging (#17169) 
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-25 14:34:00 +08:00
congqixia 37d7d7baf8
Fix node state data race in querycoord (#17198) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-25 10:43:59 +08:00
Jiquan Long 4293b640b2
Remove confusing log in segment id assigner (#17195) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 10:26:00 +08:00
Jiquan Long 75ca64f8c7
Refine task type logs to string (#17196) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 03:08:00 +08:00
xige-16 a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-24 21:56:00 +08:00
zhenshan.cao ec1103ca27
Simplify the merge logic of searchTask (#17194) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 21:27:59 +08:00
bigsheeper a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph (#17096) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-24 21:11:59 +08:00
Bingyi Sun ffaead6ad9
Add load meta in hand off task. (#17179) 
issue: #16842

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-24 18:24:00 +08:00
congqixia a0cef3a0a2
Fix sessionutil log always datanode (#17184) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-24 16:05:59 +08:00
zhenshan.cao d05c24a0ff
Delete unused code (#17182) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 14:49:59 +08:00
zhenshan.cao 2512e668f7
Update timestmap according to gracefultime for Bounded Consistency (#17171) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 12:05:59 +08:00
zhenshan.cao ea8e1623cd
Acquire nq from placeholdergroup incase of nq is zero (#17168) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 10:05:58 +08:00
zhenshan.cao cce533c246
Fix unitest fail, change default index slice to 16 (#17162) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-23 17:44:25 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
cai.zhang 3685ff9e0c
Mark indexing task to retry when minio occurs error (#17153) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-23 15:47:58 +08:00
Xiaofan 65269a4744
Remove rocksmq reader (#17149) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-23 13:55:59 +08:00
jaime a20e0dfc74
Refactor kafka msgstream implementation (#17069) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-05-23 12:39:58 +08:00
Enwei Jiao d28a2db46c
move arrow from storage to core (#17061) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-22 20:03:58 +08:00
Bingyi Sun 86728490a2
Fix partition not found (#17132) 
issue: #16842

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-20 19:37:57 +08:00
Jiquan Long e756fdccd4
Fix pool of lexer and parser (#17129) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-20 18:47:57 +08:00
yah01 7746a5b742
Add NodeIds field for QuerySegmentInfo (#17121) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-20 18:03:58 +08:00
congqixia 90ee23df22
Fix shardLeader cache concurrent access (#17120) 
Fix  write map without mutex control
Also GetShards returns a copy of leader list instead of original one

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-20 14:33:57 +08:00
ownbylichaobao 2f8a7e7793
Support tls/ssl connection to etcd (#7703) (#17012) 
Signed-off-by: lichaobao <1527563274@qq.com>
 2022-05-20 12:29:19 +08:00
groot d525e95550
Bulkload not allow duplicate numpy file (#17105) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-20 10:27:56 +08:00
Letian Jiang 70825a35cf
Make querynode LoadSegments interface idempotent (#17109) 
This PR makes the following changes:
* separate LoadSegmentsTask into two phases: PreExecute and Execute
* filters out segments that are already loaded in PreExecute phase

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-20 09:15:57 +08:00
congqixia 599763d9bf
Fix replicas info is not removed after release (#17111) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 20:05:57 +08:00
cai.zhang e05cfe194e
Fix data race for indexcoord test (#17057) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-19 17:25:56 +08:00
Jiquan Long 276ba69819
Replace plan parser with antlr (#17090) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-19 17:15:57 +08:00
yah01 dcfe472586
Fix LoadBalance doesn't save the modification to replicas' shards (#17064) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-19 16:51:57 +08:00
congqixia ae2864639e
Fix replica freeAll order (#17106) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 14:53:56 +08:00
congqixia 7eb6fcfb34
Limit the query node timetick log rate (#17065) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 10:19:56 +08:00
bigsheeper 9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-19 10:13:56 +08:00
congqixia b95681770b
Add deferred clean up for ShardCluster (#17094) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-18 21:51:56 +08:00
SimFG cf08b5aa11
Fix 'TestChannelManager_StateTransfer' ut when etcd is slow (#17082) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-18 17:19:57 +08:00
Jiquan Long 99b4042f8c
Disable segment small index (#16935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-18 15:27:56 +08:00
yah01 33c855dcd2
Fix LoadBalance doesn't remove the source nodes from segment (#17051) 
If the triggerCondition isn't NodeDown, the removing won't happen.

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-18 11:55:56 +08:00
XuanYang-cn 127dd34b37
Make proxy use roundrobin to choose replica (#17063) 
Fixes: #17055

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-17 22:35:57 +08:00
cai.zhang b37b87eb97
The GetIndexFilePaths interface judges whether the index is complete (#17052) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-17 20:41:56 +08:00
Jiquan Long 77b0f290a8
Fix prefix match on sealed segment (#17050) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-17 20:15:55 +08:00
congqixia 5282397c1b
Fix VectorChunkManager cache enable parameter (#17058) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-17 17:37:55 +08:00
cai.zhang d6bb86dd46
Return buildID when creating index if the index alreay exists (#17045) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-17 15:21:57 +08:00
xige-16 c2f1d65fa1
Print err info when insert msg not aligned (#16305) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-17 13:01:57 +08:00
yah01 960d35e517
Add MockCluster, make unit tests reliable (#17032) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-17 12:57:56 +08:00
XuanYang-cn a1480aaf76
Clear shard leaders cache when releasing (#16942) 
See also: #16926

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-17 11:11:56 +08:00
Xiaofan 5355153805
Minio Error is not Handled gracefully (#17003) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:25:55 +08:00
Xiaofan 327bd6abbe
Support Txn interface in rockskv (#17027) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:17:55 +08:00
xige-16 935b729a0c
Fix multiple deletes not taking effect (#17033) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-16 18:39:56 +08:00
congqixia ffff291018
Fix Handoff log in shardCluster service (#17039) 
Fix #17035

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-16 18:29:56 +08:00
congqixia 94f0951f0d
Fix query lock logic in query shard (#17034) 
Previously query shard locks the querylock in collectionReplica before any search/query
The lock range is too large and easy to cause dead lock

This PR makes following changes:
- Rename collectionReplica to metaReplica which is more reasonable
- Make release collection operation cancels waiting search/query request
- Reduce the queryLock to collection level
- Add some unit tests for timeout & released case

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-16 18:23:55 +08:00
Bingyi Sun 59bc0a7000
Add some log. (#16978) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-16 17:15:55 +08:00
yah01 e38c6f6c44
Fix load the same segments multiple times for manual LoadBalance (#16921) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-16 15:41:56 +08:00
Xiaofan 338edcd3cc
Load binlog for different field in parallel (#17005) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-15 16:11:54 +08:00
SimFG ea2c395a38
Fix the comment info in task_test.go (#16982) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-15 09:03:54 +08:00
jaime 328ab7ecf6
Fix kafka client hang after killing broker pod of parition leader (#16983) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-05-14 23:05:54 +08:00
congqixia 23f7ef931c
Remove not used QueryService (#16973) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-13 20:27:54 +08:00
congqixia ec66ac69ac
Simplify chunk manager in QueryShard (#16976) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-13 19:55:54 +08:00
yah01 a382133a8a
Add new node into the replica which has the most of offline nodes (#16907) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-13 18:31:54 +08:00
Xiaofan 52fc9a558c
Fix valid error message (#16953) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-13 18:19:53 +08:00
Jiquan Long ca3ecec5b5
Fix ut data race (#16939) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-13 17:43:53 +08:00
groot 168935f4cf
Fix bulkload bug for string primary key (#16958) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-13 16:07:54 +08:00
Letian Jiang f0dee539bb
Improve query shard log (#16950) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-13 13:45:53 +08:00
XuanYang-cn 43b36b4e5f
Make proxy update cache with NotShardLeader status code (#16940) 
See also: #16926

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-13 11:35:53 +08:00
Jiquan Long c5149c7e85
Replace text codec of pb with binary codec (#16955) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-13 09:53:53 +08:00
Ten Thousand Leaves 63385b7c12
Make data queryable and data indexed states as independent flags (#16837) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-12 19:23:53 +08:00
congqixia ae717bf991
Fix channelUnsubscribe data race and logic (#16946) 
- Add a RWMutex for container/list which is not goroutine-safe
- Fix the element in list is never removed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-12 18:09:53 +08:00
xige-16 08ad77c71b
Delete all repeated primary keys (#16863) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-12 14:09:53 +08:00
Letian Jiang b3eb2b1d0d
Support deltaLog loading on growing segment (#16903) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-12 11:59:53 +08:00
congqixia a8b81e215b
Fix deadlock in release replica (#16925) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-11 21:57:53 +08:00
congqixia a0614a3ff8
Use NodeIds instead of NodeID in segment info (#16912) 
The NodeID field is deprecated in current design
Query Node segment detector shall read NodeIds instead of NodeID

Fix #16911

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-11 17:39:53 +08:00
xige-16 9e6d9fb7d8
Print collectionName when load failed (#16878) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-11 09:47:53 +08:00
Jiquan Long be688d894a
Forbid multiple indexes on same field (#16875) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-10 21:07:53 +08:00
exia 4ee78eb5ad
proxy support tls (#16706) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-05-10 20:05:53 +08:00
congqixia 5c98329f7c
Return error when no replica available (#16886) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-10 19:47:53 +08:00
Xiaofan 000c5ff3de
Fix msgstream unsubscription (#16883) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-10 19:43:52 +08:00
congqixia 34833680da
Add lock before query release collection from replica (#16880) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-10 18:09:53 +08:00
Ten Thousand Leaves 216c45fbd6
Add tests where segments load delta logs (#16825) 
issue: #16821

/kind improvements

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-10 17:59:52 +08:00
yah01 2d0f908dba
Fix updating segments' NodeIds correctly after LoadBalance (#16854) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-10 15:47:53 +08:00
Letian Jiang 72bbe40254
Make PayloadReader read column data in batch (#16826) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-10 11:37:52 +08:00
Ten Thousand Leaves 5afdf3d49f
Move locks back to meta table methods (#16848) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-09 20:47:52 +08:00
Bingyi Sun 76bfd41952
Refine DataNode logs. (#16849) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-09 16:25:52 +08:00
Xiaofan 62658dcda6
Fix pulsar unscubsribe fail because of consumer not found (#16839) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-09 12:07:52 +08:00
cai.zhang b0c9f25352
Sort out the log of IndexCoord (#16810) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-09 12:05:52 +08:00
xige-16 8fcf349609
Fix create index failed after drop index (#16836) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-09 11:35:53 +08:00
congqixia c7935d1a86
Add legacySegments for load balance segments rc track (#16831) 
- Add legacySegments in ShardCluster tracking old segment inUse
- Apply node id check for segment online/inUse check
- Add unit test simulating load balance case

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-07 18:25:52 +08:00
Jiquan Long ac389f3e93
Limit and translate pattern match operations (#16818) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-07 16:31:52 +08:00
Ten Thousand Leaves 6b2d3c3ec1
Make each dataNode only accept one bulk load task simultaneously (#16820) 
issue: #16674

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-07 14:05:52 +08:00
Xiaofan 92b6293be4
Fix QueryNode log level (#16604) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-07 10:27:51 +08:00
Enwei Jiao 67ce206cf0
make SegmentMaxLifetime configurable (#16783) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-06 23:41:51 +08:00
Ten Thousand Leaves 31ddff2056
Some minor fixes and improvements (#16814) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-06 21:35:51 +08:00
bigsheeper 64d2793fe9
Fix wrong result topks when reduce search result (#16815) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-06 21:17:51 +08:00
Ten Thousand Leaves a8e1c8fa9d
Make duplicate flush recalls success and other logic update (#16755) 
Latest logic:
1) Duplicate flush calls on same segments will not result in errors (same as the original design)
2) `FlushSegments` now still flushes stale segments even if non-stale segments failed to get flushed

issue: #16749

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-06 17:49:51 +08:00
Jiquan Long 98ceb162aa
Support antlr as plan parser (#16696) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: xaxys <tpnnghd@163.com>

Co-authored-by: xaxys <tpnnghd@163.com>
 2022-05-06 17:43:51 +08:00