milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
15,100 Commits
149 Branches
120 Tags
577 MiB
Commit Graph

5494 Commits (641fb969584bc88da48a1c6e6a1d03b753cf369f)

Author SHA1 Message Date
groot b5e62023ea
Return error when bulkload pending list hit limit (#17570) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-16 13:02:10 +08:00
wayblink eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC (#17441) 
Resolves: #17233 #16047

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-16 12:00:10 +08:00
Xiaofan 54aeb077e9
Refine compaction selection poliy (#17486) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
cai.zhang ea5041aec2
Acquiring the segment reference lock on task level (#17544) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-15 21:38:10 +08:00
Jiquan Long 875b6f88b0
Add channel information to flow graph node (#17349) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-15 21:04:10 +08:00
Ten Thousand Leaves 51a9e54b21
Fix a bug that causes infinite loop (#17550) 
This will cause AddNode() to keep its lock forever so DataCoord would just hang

issue: #17366
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 20:50:10 +08:00
xige-16 98b32f3eba
Improve time record log in search/query (#17545) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-15 20:48:10 +08:00
xige-16 dd483beefd
Delete the log that prints a lot of deleteMsg.timestamps (#17543) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-15 20:46:14 +08:00
Bingyi Sun d4fdfab818
Fix VDC-510 (#17554) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-15 19:58:10 +08:00
xing.zhao 76523b313b
Fix bug #17403 by double check collmeta prefix and alias prefix (#17548) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-06-15 19:14:10 +08:00
SimFG e825215c44
Remove the redundant 'for' in the TaskCondition (#17558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-15 16:32:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
Jiquan Long e751663fdc
Fix err msg when the dimension of query vectors mismatch field (#17553) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-15 10:54:09 +08:00
congqixia f9553970f9
Add BindContext function for querycoord task scheduler (#17531) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-14 17:12:09 +08:00
congqixia ed3a1164dd
Handle wrapped grpc error in funcutil.IsGrpcError (#17547) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-14 17:10:10 +08:00
groot ab9b186f15
Change bulkload bufsize for row-based parser (#17473) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-14 16:18:09 +08:00
bigsheeper cdcdfa1ea5
Disable jemalloc and use malloc_trim instead (#17538) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-14 11:12:09 +08:00
xing.zhao e9dcda161f
Add aliases return value in describeCollection task (#17516) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: 赵星 <zhaoxing@zhaoxings-MacBook-Pro.local>
 2022-06-13 19:22:10 +08:00
congqixia a0d3f37c0d
Sync grpc DefaultKeepAliveTimeout to milvus.yaml (#17519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-13 16:18:09 +08:00
XuanYang-cn e6225d923b
Cancel timers while adding new timer (#17511) 
See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-13 14:46:08 +08:00
congqixia d1de8cabdd
Fix %w verb for error wrapping in grpcclient (#17484) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-13 12:48:08 +08:00
xige-16 16f0815184
Add time record log on search/query execution path (#17497) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-13 11:42:09 +08:00
xige-16 36ad989590
Fix segOffset grater than insert barrier when mark delete (#17444) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-10 20:02:08 +08:00
Enwei Jiao a5b008acec
ignore getReplica's error when handle rebalanceTask (#17469) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-06-10 14:50:08 +08:00
jaime b64a467311
Revert "Use unified catalog interface to reconstruct metastore (#17042)" (#17480) 
This reverts commit 218326bafe.

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-10 13:10:08 +08:00
bigsheeper 2d9a52206d
Use jemalloc in QueryNode, DataNode and IndexNode (#17470) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-10 11:24:08 +08:00
XuanYang-cn 2b5340581c
Skip remove if reassigns to the original node (#17450) 
Fix ut race

See also: #15966, #17432

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-10 10:54:07 +08:00
Xiaofan 66f26943f8
Add Err code when task rollback (#17472) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-09 21:08:07 +08:00
Xiaofan 05a918e0ab
Refine logs (#17438) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-09 21:04:07 +08:00
congqixia 97a871cc82
Make querycoord segment allocator respect context (#17452) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-09 20:14:06 +08:00
Letian Jiang dfaed5acdd
Add QueryServiceAvailable field in ShowCollections (#17456) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-09 18:20:07 +08:00
yah01 a38790f586
Predict the loading memory usage more accurate (#17451) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-09 18:16:07 +08:00
Jiquan Long 2ca81620ea
Reduce lock operations when get dml stream (#17468) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-09 17:34:09 +08:00
groot 9210299706
Fix bulkload OOM issue (#17461) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-09 16:52:07 +08:00
xige-16 6d078ebe7a
Print more information when loading fields with different lines (#17448) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-08 20:16:06 +08:00
wayblink 25883840b2
Remove delete log in datanode because it may influence datanode performance (#17415) 
Resolves #16984

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-08 19:10:07 +08:00
jaime dbef4c6ec2
Align kafka consume from latest with puslar and rocksmq (#17433) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-08 19:00:07 +08:00
XuanYang-cn 4f66bcbb22
Fix DataNode processes event out of order (#17440) 
The probability is low so very unlikly to reproduce

See also: #15966

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-08 17:24:07 +08:00
Xiaofan 6426e3d037
Split Etcd save operation into small txns (#17386) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-08 15:08:07 +08:00
yah01 70f8bea4b4
Avoid growing slice as deserializing binlogs (#17421) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-08 11:46:06 +08:00
XuanYang-cn 1dbfebf827
Add debug log in removing assignment (#17427) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-08 10:44:06 +08:00
bigsheeper af108b5bda
Add malloc_trim after reduce search result (#17423) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-08 00:12:05 +08:00
groot f31be941bb
Fix bulkload row count bug for multi datanodes (#17384) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-06-07 16:56:07 +08:00
xige-16 b5c11a216d
Alter varChar type params's name to max_length (#17409) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-07 15:58:06 +08:00
Xiaofan a9ec1f8864
Fix compatibility of upgrade (#17406) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-07 14:52:06 +08:00
congqixia de8f2f76c5
Fix shardLeader cache with ErrConnect check (#17402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-07 12:20:06 +08:00
jaime 218326bafe
Use unified catalog interface to reconstruct metastore (#17042) 
Co-authored-by: kejiang ke.jiang@zilliz.com
Co-authored-by: yun.zhang yun.zhang@zilliz.com

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-07 12:00:07 +08:00
bigsheeper f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 21:34:05 +08:00
zhenshan.cao 98e95275fe
Fix several code issues caused by carelessness (#17385) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-06 18:58:05 +08:00
bigsheeper 9f786dd752
Remove unnecessary bitset flip in Search and Query (#17387) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 17:38:05 +08:00
yah01 a2d2ad88bd
Make assigning segments faster (#17377) 
This improve the Load performance,
and let the LoadBalance fails fast, which allows us to retry it timely

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-06 16:52:05 +08:00
congqixia ac6394d0fd
Fix container/list removal in iteration (#17382) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-06 16:34:05 +08:00
Xiaofan af994b5e1c
Fix etcd watch error (#17364) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-06 16:26:06 +08:00
bigsheeper 45febac298
Extract lease of cProto in QueryNode (#17373) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 15:56:05 +08:00
XuanYang-cn c70af73005
Fix datacoord set wrong state for node registering (#17376) 
Fix datacoord datarace

See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-06 15:10:05 +08:00
bigsheeper 48cf63a2d7
Wait for flow graph to exit (#17341) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 19:48:04 +08:00
shaoyue 76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
SimFG 0986c29d7f
proxy tls support one-way authentication (#17348) 
Signed-off-by: xiyichan <2863768433@qq.com>

Co-authored-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>

Co-authored-by: xiyichan <2863768433@qq.com>
 2022-06-02 18:56:04 +08:00
Ten Thousand Leaves f5e63177d2
Fix an issue with bulk load with multiple DataNodes (#17324) 
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-02 18:54:04 +08:00
yah01 4e83f37899
Improve load performance (#17273) 
Load segment's fields concurrently

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-02 17:56:03 +08:00
shaoyue 7c52a8c535
Add RESTful API for new methods, fix err when receive empty boy (#17260) 
create collection schema use struct instead of protobuf

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 17:40:04 +08:00
xige-16 8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel (#17308) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-02 16:56:04 +08:00
zhenshan.cao 5f44e45480
Add metrics on QueryNode and Proxy (#17328) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 16:06:03 +08:00
Jiquan Long adf3b14027
Refine channels management in Proxy. (#17334) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-02 15:34:04 +08:00
bigsheeper 7a59a80506
Remove segmentPtr lock and use collection lock instead (#17303) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 14:00:04 +08:00
XuanYang-cn 42f643e727
Make DataNode release rather than delete when reassign (#17293) 
1. Reassgin now will assign to the original Node if no other nodes
   avaliable
2. Make AddNode balance async: ToRealse + Reassign

See also: #16114, #17270

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-02 13:56:04 +08:00
yah01 cc69c5cdd3
Make Cluster interface's methods called outside public (#17315) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-02 13:16:05 +08:00
congqixia c76b4ade01
Fix data race in LoadSegmentsTask (#17342) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-02 12:52:03 +08:00
zhenshan.cao 2763efc9b0
Add cache of grpc client of ShardLeader in proxy (#17301) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 12:16:03 +08:00
jaime 5698bf4236
Add sasl configurations for kafka (#17323) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-02 12:12:03 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS (#17316) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-02 10:48:03 +08:00
bigsheeper e0cbacba59
Check ignorable error to prevent unnecessary panic (#17317) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 10:12:03 +08:00
yah01 f5bd519e49
Add retry mechanism for NodeDown LoadBalance (#17306) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-01 20:00:03 +08:00
xige-16 63a31ccb38
Remove error return value when removeSegment (#17310) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-01 13:18:02 +08:00
Jiquan Long b4f797506d
Fix err msg of invalid expression (#17296) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-01 10:00:07 +08:00
Jiquan Long 72f454e054
Fix 16961, make creating dml stream idempotent (#17302) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-31 22:58:02 +08:00
XuanYang-cn d920e5c915
Fix DataNode panic while allocating IDs (#17294) 
See also: #17270

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-31 18:02:03 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence (#16782) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
groot affcc2acad
Fix bulkload entity number bug (#17278) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-31 15:40:04 +08:00
bigsheeper f55fe6be3c
Merge historical replica and streaming replica in QueryNode (#17217) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-31 13:42:03 +08:00
Xiaofan 9579a645c6
Support compile marisa on Macos (#17261) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-31 13:28:02 +08:00
Jiquan Long 6bd786bf50
Reduce default seal propotion (#17287) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-31 12:02:04 +08:00
cai.zhang f1bad98f00
Fix indexcoord ut (#17282) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 12:00:09 +08:00
congqixia c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster (#16856) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:50:04 +08:00
congqixia cecbd40a22
Fix Data race of loadType in querynode/collection.go (#17275) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:16:02 +08:00
bigsheeper 11834a0948
Improve query log in QueryNode (#17268) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-30 17:18:03 +08:00
zhenshan.cao b0d54ed0af
Rename Client to GrpcClient in mock package (#17230) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-30 17:16:08 +08:00
Jiquan Long ac2cadc83b
Refine segment type logs to string (#17248) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-30 15:30:01 +08:00
yah01 b09359b12f
Remove useless collection ID in error message (#17269) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-30 14:40:01 +08:00
SimFG 731870211a
Support role-based access control (only pb) (#16987) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-28 00:04:01 +08:00
congqixia 55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking (#17247) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-27 16:20:00 +08:00
zhenshan.cao ad9276a440
Add metrics for proxy and querynode (#17205) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-27 14:12:01 +08:00
xige-16 e196caf604
Force reclamation of memory after loading (#17244) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-27 12:50:00 +08:00
xige-16 3a63d6c98e
Fix load timeout in chaos (#17241) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 22:28:03 +08:00
Letian Jiang f2a27e0e64
Retry GetShardLeaders until service available or timeout (#17183) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-26 20:28:02 +08:00
yah01 5872c5afb6
Fix updating shard leaders may lost some modifications (#17218) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-26 17:14:02 +08:00
xige-16 56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
groot 3928da6493
Refine calc distance function (#17161) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-26 14:34:01 +08:00
xige-16 7d810ac159
Fix query empty when insert same pk after deletion (#17222) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-25 22:06:00 +08:00
Jiquan Long e44afc7aa3
Fix ut: wait service ready after run Proxy server (#17208) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 18:56:00 +08:00
congqixia 7409bfc56d
Make allocateNode run async in case of block offline event (#17185) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-25 18:53:59 +08:00
Jiquan Long c12ee2f5cb
Fix flow_graph_insert_buffer_node.go code style (#17216) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 17:03:59 +08:00
congqixia e06d6d7cc6
Returns collection instead of copy to make leaderMutex work (#17206) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-25 15:20:00 +08:00
yah01 de0ba6d495
Fix GetQuerySegmentInfo() returns incorrect result after LoadBalance (#17190) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-25 15:17:59 +08:00
Ten Thousand Leaves 5e1e7a6896
Fix an issue that causes flush hanging (#17169) 
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-25 14:34:00 +08:00
congqixia 37d7d7baf8
Fix node state data race in querycoord (#17198) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-25 10:43:59 +08:00
Jiquan Long 4293b640b2
Remove confusing log in segment id assigner (#17195) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 10:26:00 +08:00
Jiquan Long 75ca64f8c7
Refine task type logs to string (#17196) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 03:08:00 +08:00
xige-16 a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-24 21:56:00 +08:00
zhenshan.cao ec1103ca27
Simplify the merge logic of searchTask (#17194) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 21:27:59 +08:00
bigsheeper a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph (#17096) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-24 21:11:59 +08:00
Bingyi Sun ffaead6ad9
Add load meta in hand off task. (#17179) 
issue: #16842

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-24 18:24:00 +08:00
congqixia a0cef3a0a2
Fix sessionutil log always datanode (#17184) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-24 16:05:59 +08:00
zhenshan.cao d05c24a0ff
Delete unused code (#17182) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 14:49:59 +08:00
zhenshan.cao 2512e668f7
Update timestmap according to gracefultime for Bounded Consistency (#17171) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 12:05:59 +08:00
zhenshan.cao ea8e1623cd
Acquire nq from placeholdergroup incase of nq is zero (#17168) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 10:05:58 +08:00
zhenshan.cao cce533c246
Fix unitest fail, change default index slice to 16 (#17162) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-23 17:44:25 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
cai.zhang 3685ff9e0c
Mark indexing task to retry when minio occurs error (#17153) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-23 15:47:58 +08:00
Xiaofan 65269a4744
Remove rocksmq reader (#17149) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-23 13:55:59 +08:00
jaime a20e0dfc74
Refactor kafka msgstream implementation (#17069) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-05-23 12:39:58 +08:00
Enwei Jiao d28a2db46c
move arrow from storage to core (#17061) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-22 20:03:58 +08:00
Bingyi Sun 86728490a2
Fix partition not found (#17132) 
issue: #16842

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-20 19:37:57 +08:00
Jiquan Long e756fdccd4
Fix pool of lexer and parser (#17129) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-20 18:47:57 +08:00
yah01 7746a5b742
Add NodeIds field for QuerySegmentInfo (#17121) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-20 18:03:58 +08:00
congqixia 90ee23df22
Fix shardLeader cache concurrent access (#17120) 
Fix  write map without mutex control
Also GetShards returns a copy of leader list instead of original one

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-20 14:33:57 +08:00
ownbylichaobao 2f8a7e7793
Support tls/ssl connection to etcd (#7703) (#17012) 
Signed-off-by: lichaobao <1527563274@qq.com>
 2022-05-20 12:29:19 +08:00
groot d525e95550
Bulkload not allow duplicate numpy file (#17105) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-20 10:27:56 +08:00
Letian Jiang 70825a35cf
Make querynode LoadSegments interface idempotent (#17109) 
This PR makes the following changes:
* separate LoadSegmentsTask into two phases: PreExecute and Execute
* filters out segments that are already loaded in PreExecute phase

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-20 09:15:57 +08:00
congqixia 599763d9bf
Fix replicas info is not removed after release (#17111) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 20:05:57 +08:00
cai.zhang e05cfe194e
Fix data race for indexcoord test (#17057) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-19 17:25:56 +08:00
Jiquan Long 276ba69819
Replace plan parser with antlr (#17090) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-19 17:15:57 +08:00
yah01 dcfe472586
Fix LoadBalance doesn't save the modification to replicas' shards (#17064) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-19 16:51:57 +08:00
congqixia ae2864639e
Fix replica freeAll order (#17106) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 14:53:56 +08:00
congqixia 7eb6fcfb34
Limit the query node timetick log rate (#17065) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-19 10:19:56 +08:00
bigsheeper 9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-19 10:13:56 +08:00
congqixia b95681770b
Add deferred clean up for ShardCluster (#17094) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-18 21:51:56 +08:00
SimFG cf08b5aa11
Fix 'TestChannelManager_StateTransfer' ut when etcd is slow (#17082) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-18 17:19:57 +08:00
Jiquan Long 99b4042f8c
Disable segment small index (#16935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-18 15:27:56 +08:00
yah01 33c855dcd2
Fix LoadBalance doesn't remove the source nodes from segment (#17051) 
If the triggerCondition isn't NodeDown, the removing won't happen.

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-18 11:55:56 +08:00
XuanYang-cn 127dd34b37
Make proxy use roundrobin to choose replica (#17063) 
Fixes: #17055

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-17 22:35:57 +08:00
cai.zhang b37b87eb97
The GetIndexFilePaths interface judges whether the index is complete (#17052) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-17 20:41:56 +08:00
Jiquan Long 77b0f290a8
Fix prefix match on sealed segment (#17050) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-17 20:15:55 +08:00
congqixia 5282397c1b
Fix VectorChunkManager cache enable parameter (#17058) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-17 17:37:55 +08:00
cai.zhang d6bb86dd46
Return buildID when creating index if the index alreay exists (#17045) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-17 15:21:57 +08:00
xige-16 c2f1d65fa1
Print err info when insert msg not aligned (#16305) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-17 13:01:57 +08:00
yah01 960d35e517
Add MockCluster, make unit tests reliable (#17032) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-05-17 12:57:56 +08:00
XuanYang-cn a1480aaf76
Clear shard leaders cache when releasing (#16942) 
See also: #16926

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-17 11:11:56 +08:00
Xiaofan 5355153805
Minio Error is not Handled gracefully (#17003) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:25:55 +08:00
Xiaofan 327bd6abbe
Support Txn interface in rockskv (#17027) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:17:55 +08:00