milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
17,934 Commits
146 Branches
120 Tags
573 MiB
Commit Graph

282 Commits (22bb9c199b884dbcc05f14f6feb007081b072f73)

Author SHA1 Message Date
cqy123456 376642b933
reset diskann min dim limit (#26344) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-08-15 16:45:34 +08:00
chyezh 4c4b2903f9
[Fixup] nats log configuration, open nats log at info level by deafult (#26168) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-08-14 14:21:32 +08:00
Enwei Jiao a3c176045d
Fix kafka producer init many times (#26314) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-13 16:59:30 +08:00
Gao b6fcbb0998
Support ScaNN index (#26099) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-11 14:21:29 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
xige-16 1971d98897
Add disk metric info (#25675) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-11 10:35:42 +08:00
wei liu 05c87af146
fix unexpected score in look aside balancer (#26213) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-10 19:45:16 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
MrPresent-Han 3421956afa
modify default value for sync max parallel to mitigate oom on dn(#26763) (#26231) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-10 10:23:15 +08:00
wei liu f33a89387f
enable config shard level cost metrics whether contains worker's cost (#26132) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-09 11:37:15 +08:00
congqixia 767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 (#26179) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 10:05:14 +08:00
yihao.dai b6effd7345
Disable deny writing when the growing segment size exceeds the watermark (#26163) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-08 12:43:07 +08:00
xige-16 d1d0169fa3
Delete useless config (#26173) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-08 10:23:08 +08:00
PowderLi a7eecb1be0
support high-level RESTFUL API, listen on the same port as grpc. (#25108) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-08-08 10:15:07 +08:00
congqixia 241117dd6d
Use last end position in pursuit mode (#26144) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-05 16:55:06 +08:00
SimFG d2649b63db
Delete the user-role mapping info when deleting the user (#25988) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-04 18:37:08 +08:00
wayblink 775f03ed5d
Add default value for mq.type (#26064) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-04 18:35:07 +08:00
congqixia fcde5b8d7f
Add pursuit mode for mqtt msgstream (#26098) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-04 12:33:06 +08:00
wei liu 518b6310a2
refine retry times on replica (#26043) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-03 15:55:09 +08:00
MrPresent-Han 47392b0a0f
support metrics mutex to monitor cost of locks(#26102) (#26103) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 15:03:06 +08:00
MrPresent-Han 5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
congqixia 0727e017c4
Add Promethues suite test utility (#26046) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-02 10:05:05 +08:00
yah01 9c55a7f422
Add worker num as one of load resource (#26045) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-01 21:47:06 +08:00
yah01 2180ef180c
Record only failed task error (#26033) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-01 10:11:05 +08:00
congqixia 8b5e276361
Use different cgo pool for SQ and other operations (#26021) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-01 09:19:05 +08:00
zhagnlu 833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-27 19:49:02 +08:00
congqixia 33cfb2ba2f
Fix MsgPack from mq msgstream ts not set (#25926) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-27 13:57:12 +08:00
xige-16 6f18587f35
Fix small segment compaction (#21327) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-07-26 14:49:01 +08:00
MrPresent-Han b5e79e7f34
add metrics for assign-segment on proxy(#24570) (#25907) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-26 14:15:01 +08:00
chyezh e24a8b3606
[Feature] Add benchmark and retention configuration for nmq (#25768) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-07-25 19:33:01 +08:00
wei liu 302897f866
refine look aside balance logic (#25837) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 18:51:01 +08:00
congqixia 0bc03ede0d
Add eventlog pkg and support grpc streaming event observation (#25812) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-25 17:23:01 +08:00
wei liu 2ff469771c
fix delegator insert/delete cost metrics (#25871) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 10:01:00 +08:00
jaime 8e5d6193f9
Add a timeout config for bulkinsert request (#25789) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-07-24 14:23:00 +08:00
MrPresent-Han f4e72cb170
remove sync segmentLastExpire every time when assigning(#25271) (#25316) (#25557) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-24 14:11:07 +08:00
Bingyi Sun 9f31fc9a31
Add broker timeout config item. (#25855) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-07-24 14:07:00 +08:00
congqixia 3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia 37636b2f4d
Add compaction worker max parallel config item (#25834) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:17:00 +08:00
cai.zhang b15e34db21
Add contraint for compaction based indexed segments (#25709) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-23 21:31:00 +08:00
cqy123456 0f3350ded4
set knowhere thread pool size in index node (#25804) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-07-21 18:50:59 +08:00
wei liu 32827f538a
add metrics for delegator insert/delete cost (#25733) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-21 15:30:59 +08:00
congqixia 1045c88102
Support replace indexed field in QueryCoord (#25747) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-19 21:22:58 +08:00
congqixia fc6f0f97a6
Add skip level for rated log (#25765) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-19 20:43:02 +08:00
wei liu 6534396b3d
enable config different interval for different checker (#25514) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-19 16:50:57 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
aoiasd 3545b1a608
Refine rocksmq (#25031) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-18 20:18:57 +08:00
Enwei Jiao 9fe62cb5f3
Fix kafka producer panic by nil (#25691) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-18 18:24:57 +08:00
congqixia 8d343bf75a
Make compaction rpc timeout and parallel maxium configurable (#25672) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 14:25:20 +08:00
Cai Yudong 73512c72fd
Remove calc_distance (#25663) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-18 14:23:20 +08:00
congqixia 9a61c0291b
Disable error verbose for zap logger (#25688) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 12:53:21 +08:00
aoiasd 77a9553c3f
Add some log when node being reachable or unreachable (#25572) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-18 10:35:20 +08:00
yah01 948d1f1f4a
Handle errors by merr for QueryCoord (#24926) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-17 14:59:34 +08:00
wei liu 51c348158b
fix use pChannel in seek position (#25611) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-17 09:18:33 +08:00
yihao.dai f19621fc8c
Support refund the tokens to limiter (#25598) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-07-14 15:58:32 +08:00
Bingyi Sun c51922eb0b
Use sync.Map to reduce lock granularity to make create collection faster (#25485) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-13 21:46:31 +08:00
wayblink 50454bec4c
Allow empty minio.rootPath (#25534) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 14:46:29 +08:00
MrPresent-Han ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-12 20:22:29 +08:00
yah01 205a7c430a
Fix QueryNode panic while inserting to released segment (#25495) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-12 19:48:29 +08:00
SimFG f9e2d00f91
Prevent `exclusive consumer` exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
yihao.dai 4c93495587
Add segment size metric in querynode (#25406) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-07-12 14:26:28 +08:00
Xiaofan dbf0130803
Faster garbage collect on compacted data (#25088) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-07-12 10:14:29 +08:00
congqixia bcd3abf644
Separate segments/shards reduce latency metrics (#25475) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-11 18:18:28 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
yah01 8b06941da3
Reduce the load memory usage predict factor to 2 (#25469) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-11 13:06:28 +08:00
smellthemoon d63323d117
Add rate limit and deny write in upsert (#25351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-07-11 11:20:34 +08:00
Cai Yudong 5aad0ceec3
HNSW support binary vector (#25417) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-10 18:18:28 +08:00
Xiaofan b437195e17
Set compaction default value to be true (#25424) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-07-08 11:04:25 +08:00
wei liu 342cfcad46
decrease default value of group max nq (#25380) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-07 15:40:25 +08:00
wei liu c84496a1ba
fix sq count metrics (#25287) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:46:26 +08:00
wei liu 951c312ef9
refine grafana metrics for replica selection (#25319) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-05 17:00:25 +08:00
congqixia efdd71c640
Make cgo pool size larger than worker pool size (#25318) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-05 16:56:26 +08:00
foxspy 31173727b2
growing segment index memory opt & get vector bugfix (#25272) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-07-05 00:04:25 +08:00
chyezh d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy (#24839) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-07-03 18:24:25 +08:00
Gao de6e4817a2
Support dynamic tuning config (#25152) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-07-03 15:18:24 +08:00
smellthemoon b30517d303
Enlarge timeout to prevent health check failure (#25173) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-06-29 17:10:23 +08:00
yihao.dai 37fe3393d1
Remove const to enable move semantics and improve schema util functions (#25193) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-29 11:04:22 +08:00
congqixia 1357ef7043
Merge get worker process for same nodeID (#25185) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-28 18:34:24 +08:00
wei liu 3a222e976d
fix consumer close may block forever (#25120) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-27 14:04:45 +08:00
SimFG a1e6428bc7
Fix the kafka panic when sending the message to a closed channel (#25116) (#25117) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-06-27 10:42:44 +08:00
wei liu 17796743dd
add metrics for replica selection (#25037) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-27 09:52:44 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
wei liu 7b999b42bd
enable config policy on replica selection (#25067) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-25 19:46:44 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
chyezh ccf3f0066f
[Pick] Enable max result window limit (#24986) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-06-25 14:42:43 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
congqixia 6a1eff3487
Fix kafka consumer double close (#25051) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 14:16:44 +08:00
congqixia 00f8f236d7
Revert close kafka internal producer (#25072) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 12:38:43 +08:00
congqixia a3630e17a0
Refine grpc Status handling: retry legacy only for Unimplemented (#25041) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 10:30:43 +08:00
wei liu 0d51f55c48
enable config each collection has different rate limit (#24139) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-21 15:02:42 +08:00
PowderLi 3f4356df10
fix the spelling of `field` (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yihao.dai 9f5756f16e
Add cluster validation interceptor to resolve the ip-reuse issue (#25001) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-21 11:36:46 +08:00
congqixia 4fda4f3951
Close kafka internal consumer&producer properly (#24998) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-20 21:14:41 +08:00
Xiaofan af1d84e5e1
Support sync when cp is lagged (#24989) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-20 10:00:41 +08:00
cai.zhang c9e456c6eb
Remove metric_type check and fix some minor bugs (#24921) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-06-19 09:54:41 +08:00
cai.zhang c81bdda55a
Hide autoindex params details (#24851) (#24935) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-06-16 14:30:40 +08:00
cqy123456 a519213316
Update knowhere version, update diskann api and generate cache nodes in build process (#24898) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-06-16 14:20:39 +08:00
yihao.dai c73219a54d
Limit the number of concurrent sync tasks and allow only one sync task for the same segment (#24881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 14:14:39 +08:00
MrPresent-Han ce5cb3c0c5
[skip e2e] modify default used balancer for 2.3 (#24937) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-06-16 14:00:45 +08:00
smellthemoon db31e88a73
Add length check when insert and upsert (#24759) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-06-15 10:24:38 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00