milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,721 Commits
142 Branches
117 Tags
569 MiB
Commit Graph

1521 Commits (81b197267aa8ce5a28d9370017a3115b69b468d2)

Author SHA1 Message Date
XuanYang-cn 5c5f9aa05e
Enhance newDataSyncService (#27277) 
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
  - config.maxQueueLength, config.maxParallelish

See also: #27207

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-27 11:07:25 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
XuanYang-cn 676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
wei liu 9433a24f5d
fix component not exit when liveness check failed (#27236) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-22 19:13:25 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia 16b35e07b3
Fix `TestSessionSuite/TestKeepAliveRetryActiveCancel` unit test logic (#27231) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-20 18:59:23 +08:00
yah01 9baff1b81c
Rename error to match the nameing rules (#27214) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:59:24 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
PowderLi 4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
Enwei Jiao 8e17bc3013
Use merr to handle segcore errors (#27137) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-18 14:07:22 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
yah01 7bb20fa394
Fix the double nil return values from RPC call (#27101) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-14 19:01:19 +08:00
congqixia c32a9a1d4a
Fix grpcclient use needReset as needRetry for cancel&timeout code (#27091) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-14 17:47:26 +08:00
congqixia f0d0651989
Do not reset connection immediately if grpc code is `Canceled` or `DeadlineExceeded` (#27014) 
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 15:01:18 +08:00
yihao.dai dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
Jiquan Long 802a6a86c0
Fix printing too many warnings (#27015) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-12 13:39:17 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
congqixia adfb5298c6
Refine `TestSessionProcessActiveStandBy` unit test logic (#26980) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 18:13:17 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
SimFG 0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
wei liu 0e2085b77f
fix dc standby to active (#26810) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-06 10:41:49 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
chyezh 0530fd80c9
[Fixup] remove nats from default (#26791) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-09-04 10:01:04 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
groot 1b1bafaff1
Bulkinsert read varchar by batch (#26199) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-08-30 14:23:06 +08:00
yah01 dd4bc5b6a0
Fix data race in gRPC client (#26574) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-28 18:26:28 +08:00
congqixia 145387fdcb
Bump proto go-api to v2.3.0 (#26561) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-23 20:18:23 +08:00
jaime b82fc8d6e7
[skip e2e] Add connection log for etcd and grpc (#26536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-22 21:20:24 +08:00
yah01 bed034a44b
Protect DataCoord from calculating segment lines by stale log entries num (#26523) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-22 14:16:22 +08:00
chyezh 85886f039e
[Fixup] msg context access data race when msgdispatcher (#26463) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-08-21 19:16:22 +08:00
XuanYang-cn 08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
wayblink 71133e5eaa
Check and reset if grpc client serverID mismatch with session (#26448) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-18 22:50:20 +08:00
yihao.dai 63b86b32a6
Add server id validation interceptor (#26395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-17 20:20:20 +08:00
wei liu 74133a3996
refine retry on grpc (#26360) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-16 19:36:18 +08:00
SimFG 8b54914287
Add go.mod file for the milvus/pkg package (#26359) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-16 14:49:32 +08:00
Enwei Jiao 78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
Gao b6fcbb0998
Support ScaNN index (#26099) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-11 14:21:29 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
congqixia 2b367b6bb0
Fix sessionutil Liveness check blcok in watch forever (#26248) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-10 14:07:16 +08:00
congqixia e13b6b88eb
Replace deprecated grpc WithInsecure option (#26226) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 13:37:16 +08:00
congqixia 767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 (#26179) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 10:05:14 +08:00
wei liu 0e103efc5d
disable retry in grpc connect and grpc sendMsg (#26095) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-03 15:23:07 +08:00
congqixia 7dfc8fbf0a
Fix data race on keepAliveCancel (#26087) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-02 18:55:07 +08:00
congqixia 8b11636e72
Cancel previous ctx for session retry keepalive (#26050) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-02 12:09:05 +08:00
groot c2693ea284
Fix a crash bug of bulkinsert for dynamic field (#25982) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-28 15:25:03 +08:00
XuanYang-cn 84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
zhagnlu 833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-27 19:49:02 +08:00
groot a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
SimFG 69d274d233
Improve the operation log (#25589) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-14 16:08:31 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
wayblink 587237a3c9
Fix dead loop in session (#25451) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 18:02:29 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
yah01 cd29b863d0
Fix data race in session (#25354) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 14:52:25 +08:00
wayblink b7ecb7f56b
Disable retryKeepAlive when LivenessCheck's Context close (#25161) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-27 17:08:45 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
wayblink b752a29995
Add timeout for keepalive in session (#25077) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-26 12:30:44 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
yihao.dai 9f5756f16e
Add cluster validation interceptor to resolve the ip-reuse issue (#25001) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-21 11:36:46 +08:00
SimFG 0c3f92d7d7
Improve the panic code about the rootcoord/session/rocksmq (#24859) (#25024) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-06-21 11:24:42 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
yiwangdr 4387f36897
make etcdKV private (#24778) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-13 10:52:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
Enwei Jiao d3af451d92
Upgrade golangci-lint (#24707) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-06-07 19:34:36 +08:00
congqixia d0c2fa5d19
Fix retryKeepAlive assertion panic (#24667) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-07 10:08:36 +08:00
chyezh f97127ae55
add nats mq wrappers (#24445) 
bug fixup, configurable natsmq, add unittest, pass e2e.



move natsmq to pkg project

Signed-off-by: chyezh <ye.zhen@zilliz.com>
Co-authored-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-07 10:00:37 +08:00
xige-16 732fe54775
Support partition Key (#24047) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-06 10:24:34 +08:00
wayblink 5fb5b072ae
Retry keepalive when keepalive channel close (#24581) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-01 16:14:35 +08:00
groot 3022e37298
Fix bug of bulkinsert for dynamic field (#24569) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-06-01 14:14:31 +08:00
groot 5358da2a10
Fix a crash bug for backup/restore (#24529) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-31 14:45:30 +08:00
groot 0d0e97ab7f
Fix crash issue of bulkinsert by invalid numpy array file (#24480) (#24528) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-31 14:43:39 +08:00
groot 3cb75997a3
Fix a bug of bulkinsert dynamic schema (#24405) (#24527) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-31 14:41:41 +08:00
xige-16 fe18109aab
Clean tmp disk data when start milvus (#24404) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-05-26 15:07:26 +08:00
wei liu ce03248b1a
simplify logic circle in search (#24110) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-23 16:01:26 +08:00
groot 97cd0409e4
Support dynamic field for bulkinsert (#24265) (#24304) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-23 14:17:25 +08:00
Jiquan Long bd343550a5
Support to manage connections (#24224) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-05-19 12:51:23 +08:00
MrPresent-Han 7744573d3d
support parms for import maxfilesize(#24191) (#24192) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-05-18 18:59:27 +08:00
groot b89f06dd19
Support JSON for bulkinsert (#24130) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-18 16:33:23 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia 74bba2320a
Fix session stop/goingStop stuck after connection lost (#24131) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 14:51:22 +08:00
Enwei Jiao a933b595c4
Add trace log for grpc client (#23951) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-05-09 10:10:41 +08:00
wayblink 899702f13c
Implement GetIndexStatistics interface (#23603) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-06 10:34:39 +08:00
Enwei Jiao 967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
wei liu cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00