milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,270 Commits
156 Branches
125 Tags
589 MiB
Commit Graph

836 Commits (13877a07ff030981b515115be2ecb3729ed42595)

Author SHA1 Message Date
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
SimFG 0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
Jiquan Long 61c7b0990d
Workaround fix ChannelManager holding mutex too long (#26870) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-06 14:29:40 +08:00
wei liu 0e2085b77f
fix dc standby to active (#26810) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-06 10:41:49 +08:00
wayblink d7b6f3a9e8
Enhance FlushAll/GetFlushAllState API (#26802) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-06 10:35:48 +08:00
Enwei Jiao 73f4347ac6
Fix code checker error (#26844) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 15:51:48 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
cai.zhang d7cd1f2a6d
Fix bug for get index state with compaction segment (#26822) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:29:48 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
wei liu 5af7559904
fix revoke ic session when stop dc (#26680) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-30 14:57:01 +08:00
XuanYang-cn 9598a8b236
Fix timeout task never release queue (#26593) 
See also: #26413, #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-30 11:12:27 +08:00
yah01 213db490bd
Use pointer receiver for large struct (#26668) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 10:24:29 +08:00
yah01 bed034a44b
Protect DataCoord from calculating segment lines by stale log entries num (#26523) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-22 14:16:22 +08:00
SimFG 3be4ac4022
Fix datanode/datacoord continuous restart (#26470) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-20 21:20:24 +08:00
yihao.dai 63b86b32a6
Add server id validation interceptor (#26395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-17 20:20:20 +08:00
yihao.dai 3c62acde05
Add coll and segState lebels to DataCoordNumStoredRows monitoring (#26299) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-17 15:52:17 +08:00
congqixia 8d13717cac
Fill Collection start position timestamp in WatchInfo (#26370) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 09:05:32 +08:00
congqixia f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
cai.zhang 6e1f284485
Refine index log (#26191) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-11 11:14:39 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
zhenshan.cao 7619db1edc
Fast GC compacted dropped segment (#26207) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-08-09 19:39:15 +08:00
wayblink 21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact (#26203) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-08 21:17:21 +08:00
congqixia 21e5c1792e
Add unit test for channel store reload & metrics (#26133) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 14:51:09 +08:00
XuanYang-cn 830f06783a
Reduce log volumes of DC&DN (#26060) 
See also: #26057

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-02 17:13:11 +08:00
congqixia 0727e017c4
Add Promethues suite test utility (#26046) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-02 10:05:05 +08:00
XuanYang-cn 35f3e263cd
Fix lock usage in DataCoord compaction (#26032) 
See also: #26009

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-01 08:55:04 +08:00
congqixia 7f51655ce3
Add unit test for DataNodeNum metrics value (#25985) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-31 16:59:03 +08:00
cai.zhang eaf53c7b39
Refine describe index that doesn't wait compacted segment indexed (#25899) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-28 11:15:02 +08:00
xige-16 d7cd1c849d
Fix indexnode and datanode num metric (#25919) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-07-28 10:23:02 +08:00
xige-16 6f18587f35
Fix small segment compaction (#21327) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-07-26 14:49:01 +08:00
aoiasd b533c68632
Forbid update checkpoint without msgID (#25694) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-25 10:43:04 +08:00
Zhihong Yu 150947a5f4
Conditionally log warning when DropSegment fails (#25774) 
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
 2023-07-25 10:39:01 +08:00
cai.zhang 760a2d9aa7
Support AllocTimestamp api for Milvus (#25784) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-25 10:05:00 +08:00
MrPresent-Han f4e72cb170
remove sync segmentLastExpire every time when assigning(#25271) (#25316) (#25557) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-24 14:11:07 +08:00
congqixia 3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia 37636b2f4d
Add compaction worker max parallel config item (#25834) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:17:00 +08:00
cai.zhang b15e34db21
Add contraint for compaction based indexed segments (#25709) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-23 21:31:00 +08:00
aoiasd 4cb6351f87
update isDelete of index task meta when delete task from index builder (#25717) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-21 17:15:00 +08:00
Jiquan Long 95f91e6e35
Fix segments meta not consistent between catalog and in-memory (#25799) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: czs007 <zhenshan.cao@zilliz.com>
 2023-07-21 14:21:05 +08:00
congqixia 1045c88102
Support replace indexed field in QueryCoord (#25747) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-19 21:22:58 +08:00
Jiquan Long bccdef1ad7
Fix segment meta only record last flushed binlog (#25707) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-07-19 15:28:57 +08:00
congqixia 8d343bf75a
Make compaction rpc timeout and parallel maxium configurable (#25672) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 14:25:20 +08:00
Bingyi Sun 7613c7842c
Fix datacoord crash (#25631) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-17 16:54:34 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wei liu 012fd1152a
fix datacoord consume datanode tt metrics (#25524) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:26:29 +08:00
congqixia dbfade77c5
Retry on SyncSegment failure (#25540) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-13 14:12:29 +08:00
Bingyi Sun b22d6d2a5d
Avoid update all binlogs every a segment is updated (#25526) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-13 14:08:29 +08:00
Xiaofan dbf0130803
Faster garbage collect on compacted data (#25088) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-07-12 10:14:29 +08:00
Bingyi Sun 70c4ddc6c5
Replace LoadWithPrefix with HasPrefix (#25471) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-11 19:38:28 +08:00
Bingyi Sun 268368031c
Remove redundant kv mocks (#25420) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-07 19:38:26 +08:00
Bingyi Sun 54acb5b725
Remove alter segment interface (#25382) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-06 20:46:25 +08:00
Bingyi Sun fe22720ff3
Avoid rewriting all binlogs every time a segment is updated (#25339) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-06 18:20:26 +08:00
wei liu cba0feb119
add coordinator broker, to unify rootcoord api access (#25187) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-03 18:28:25 +08:00
congqixia 597a4d9227
Treat small segment without index as sealed (#25237) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-02 19:50:23 +08:00
Xiaofan 63b6a4a639
use single instance ppol (#25159) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-28 14:54:45 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
yiwangdr c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
Xiaofan 72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
Xiaofan f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
cui fliter 35cb0b5ba6
fix function name in comment (#24879) 
Signed-off-by: cui fliter <imcusg@gmail.com>
 2023-06-14 16:20:38 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
yiwangdr 37c02c9927
clean up meta interface (#24729) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-08 14:18:35 +08:00
cai.zhang 93ea9c4925
Support return pending index rows when describe index (#24588) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-06-01 18:14:32 +08:00
yihao.dai 19164ea753
Make binlog size metric to collect healthy binlog only (#24557) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-31 16:45:29 +08:00
congqixia 6375236533
Make SegmentInfo.size atomic (#24457) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-29 11:31:28 +08:00
Ikko Eltociear Ashimine eb9ef23cca
Fix typo in handler.go (#24125) 
postitions -> positions

Signed-off-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
 2023-05-17 14:25:23 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
yihao.dai fe0a1bc2d9
Fix panic caused by wrong logic of getting unindexed segments (#24044) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-15 20:59:28 +08:00
congqixia f6dec281c4
Add Statistic entity id log for datacoord (#24073) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-12 18:11:21 +08:00
Enwei Jiao 7e645dabf1
Add trace log for datacoord (#24046) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-05-12 09:53:21 +08:00
congqixia 5aa9db0d38
Add collection level auto compaction enabled config (#24013) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-10 17:45:20 +08:00
smellthemoon 146050db82
Fix some wrong ut (#23990) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-05-10 09:31:19 +08:00
yihao.dai fc010ee86b
Improve the metric of stored binlog size to the level of collection and segment (#23859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-07 20:52:38 +08:00
wayblink 899702f13c
Implement GetIndexStatistics interface (#23603) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-06 10:34:39 +08:00
MrPresent-Han b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
Xiaofan 016311ad48
Fix Has Collection Failed on datacoord (#23710) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-04-27 09:50:34 +08:00
yihao.dai ed8836cd15
Add disk quota at the collection level (#23704) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-26 21:52:36 +08:00
XuanYang-cn 72485c9eeb
Remove index check to trigger compaction (#23657) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-04-24 20:26:34 +08:00
congqixia 4a22af6e1a
Unwatch channel in watch buffer (#23548) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-20 10:34:31 +08:00
wei liu cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
congqixia d83654c33f
Add Close method for ChannelManager in datacoord (#23493) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-18 17:54:31 +08:00
zhenshan.cao 4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
congqixia ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-14 09:54:28 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
zhenshan.cao dc5abe086c
Revert "Restrain gc files by collection (#22504)" (#23380) 
This reverts commit fa86de530d.

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-12 19:38:28 +08:00
cai.zhang 9288020da3
Fix bug for batch delete files on gcp of minio (#23052) (#23083) (#23090) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-09 16:22:30 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
yihao.dai 17d3190fcc
Fix DescribeCollection failed with undefined msg type (#23192) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-03 19:52:24 +08:00
smellthemoon ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
Ten Thousand Leaves 1596abf6e8
Add lock for segment manager during bulk insert (#23111) 
/kind bug

issue: #23106

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-03-31 20:16:24 +08:00
congqixia 127867b873
Add ratedgroup for some info/warning log (#23095) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-31 15:22:23 +08:00