Commit Graph

263 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
Cai Yudong 73512c72fd
Remove calc_distance (#25663)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-18 14:23:20 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
SimFG d246defa4d
Return error response when an unimplemented request is received (#24545)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-05-31 20:24:31 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
SimFG ef82e0f1a4
Fix the error when a role removes an already deleted user (#24049) (#24109)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-05-16 10:15:22 +08:00
smellthemoon 146050db82
Fix some wrong ut (#23990)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-05-10 09:31:19 +08:00
wayblink 899702f13c
Implement GetIndexStatistics interface (#23603)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-05-06 10:34:39 +08:00
SimFG 5cd21893c8
Fix superusers' password verification problem (#23733)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-04-26 21:16:34 +08:00
wayblink b7cb34b966
Add blank GetIndexStatistics interface (#23610)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-04-23 14:50:32 +08:00
zhenshan.cao 03ec804e68
Change default shard number to 1 (#23586)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-04-21 07:08:32 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
yihao.dai 1f718118e9
Dynamic load/release partitions (#22655)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-20 14:55:57 +08:00
smellthemoon 29b3690c83
Upsert raise error (#22533)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-03-06 14:29:51 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
Enwei Jiao 0851e05014
Gernate milvus.yaml by code (#22003)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-23 11:37:46 +08:00
wei liu 13b508103e
refine mock querycoord (#22198)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-16 15:38:34 +08:00
congqixia 5986106037
Make paramtable init only once (#21782)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
jaime 1dcd9eeb79
Support rename collection (#21693)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-19 14:13:43 +08:00
wayblink 6a722396bd
Integration test framework (#21283)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
bigsheeper 2146af1fb2
Return insufficient memory error when load failed (#21574)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-01-10 20:35:39 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
smellthemoon bf3c02155a
Support update (#20875)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-01-04 17:21:36 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-16 15:59:23 +08:00
SimFG 63cd4132a6
Implement the `GetLoadState` api (#21257)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-16 14:39:24 +08:00
SimFG 4a5c282b62
Add the superuser config (#21090)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-09 16:13:20 +08:00
Xiaofan e131915207
Support 64M Rpc Limit (#20845)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-07 10:07:18 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
congqixia 14c1182418
Add healthy check for InvalidateCollectionMetaCache (#20368)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-07 16:55:01 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
groot cd3b3d3da6
Only allow one file for row-based import (#20234)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-02 10:23:35 +08:00
groot bee66631e3
Refine bulkload (#19671)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-10-27 16:21:34 +08:00
wayblink dbd60757c9
Support pass start_ts and end_ts through Import API (#19810)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-20 12:15:27 +08:00
SimFG a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
jaime 8e7a4733ec
Add GetVersion API for the client (#19694)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-14 09:59:24 +08:00
Eng Zer Jun 438790744d
test: use `T.Setenv` to set env vars in tests (#19649)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-11 18:43:23 +08:00
yah01 2925d8a1ac
Remove QueryCoordV1 (#19681)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:11:23 +08:00
jaime 31db01b6ae
Support TTL for collection level (#19383)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
zhenshan.cao 0ac4bc32a5
Add logic of autoindex (#19570)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-10-08 15:38:58 +08:00
SimFG 7d54e35fa5
Add the `GetLoadingProgress` RPC (#19491)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-01 09:02:55 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
SimFG d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
bigsheeper 5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector (#18636)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-16 09:56:47 +08:00
Jiquan Long a5e2d6b6fb
Refactor RootCoord (#18930)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-09-05 13:29:11 +08:00
SimFG 9cd19f5396
Return nil error when handling grpc request (#18955)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-02 21:12:59 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643)
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
jaime 516fd928f9
Fix password verification miss cache (#18730)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-19 19:42:50 +08:00
SimFG 447ce8ee48
Handle some apis about the default role (#18647)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-15 16:40:48 +08:00
SimFG 1366f130a2
Add a specific `KeyNotExistError` err for the metastore `load` interface (#18582)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-10 17:20:41 +08:00
SimFG c267ad612d
Support Role-Based Access Control II (#18528)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-05 16:28:35 +08:00
SimFG ff0200210a
Support Role-Based Access Control (#18425)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-04 11:04:34 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment (#18115)
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
bigsheeper 0fb5e9c1a9
Remove load collection check in Proxy (#18166)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-08 16:12:25 +08:00
codeman 5fda8261d0
Fix: Unable to list newly created credential users (#18117) (#18153)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-07-07 21:16:22 +08:00
bigsheeper 696d747afa
Check if loaded in Proxy when LoadCollection (#18074)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-05 21:16:20 +08:00
zhenshan.cao f33b090819
Fix bug: datarace in proxy unittest (#17990)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-01 17:56:21 +08:00
codeman 95a3830dc6
fix password comparison too slow (#17874) (#17897)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-06-29 20:02:18 +08:00
cai.zhang f123d657ed
Fix bug for dropping index (#17635)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-20 15:22:13 +08:00
Jiquan Long adf3b14027
Refine channels management in Proxy. (#17334)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-02 15:34:04 +08:00
zhenshan.cao 2763efc9b0
Add cache of grpc client of ShardLeader in proxy (#17301)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-06-02 12:16:03 +08:00
congqixia c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster (#16856)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-30 19:50:04 +08:00
bigsheeper 9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-19 10:13:56 +08:00
xige-16 515d0369de
Support string type in segcore (#16546)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
groot bc5e9ec929
Add list import tasks api (#16605)
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-25 17:37:46 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
yah01 804fbb6e27
Fix panic: Set Base of GetReplicas request (#16578)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-22 14:47:42 +08:00
congqixia 69252f812d
Implement memory replica in Proxy, QueryNode and QueryCoord (#16470)
Related to #16298 #16291 #16154
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-20 16:15:41 +08:00
Ten Thousand Leaves 289e468a7a
Make bulk load fully work (#16512)
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
codeman 3a1b2cedd2
create default root user for authentication (#16545) (#16549)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-20 13:03:40 +08:00
codeman f5e1ded854
fix #16504 (#16523)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-19 16:35:39 +08:00
Ten Thousand Leaves 442c73a544
Support passing channel names to DataNode (#16424)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-12 13:25:34 +08:00
codeman 4e12271d79
Support login with username and password (#15656) (#16341)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-11 19:49:34 +08:00
godchen bb7a0766fe
Add dependency factory (#16204)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
groot 2078b24dca
Prepare for import (#16083)
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-21 15:47:23 +08:00
Xiaofan 787d4f8bfd
Refine prometheus metrics and etcd log (#16084)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-18 15:51:22 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
congqixia 3f34bc1ab7
Ban DataType_String in CreateCollection (#15838)
Fix #15836
/kind enhancement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-03 16:57:56 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
Cai Yudong 8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Jiquan Long 675e6d352b
QueryNodes send search/query results by rpc (#15223)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-01-17 14:41:35 +08:00
Cai Yudong 6323907566
Use CommonConfig in GlobalParams for all components (#15106)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 19:03:35 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-07 16:15:26 +08:00
guoxiangzhou 5fccbe9764
[skip e2e] Modify log text according to the naming conventions (#14289)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-31 11:39:31 +08:00
guoxiangzhou a87a79a65a
[skip e2e] Modify log text according to the naming conventions (#14287)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-31 11:37:35 +08:00
guoxiangzhou 451178203e
[skip e2e] Modify log text according to the naming conventions (#14283)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-31 11:33:41 +08:00
guoxiangzhou 4f1ee13795
[skip e2e] Modify log text according to the naming conventions (#14280)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-31 11:31:22 +08:00
guoxiangzhou 62f002f9df
[skip e2e] Modify log text according to the naming conventions (#14278)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-30 16:47:01 +08:00
guoxiangzhou 437b4c0f29
[skip e2e] Modify log text according to the naming conventions (#14277)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-30 16:44:09 +08:00
guoxiangzhou 0769c1f757
[skip e2e] Modify log text according to the naming conventions (#14275)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-30 16:41:22 +08:00
zhenshan.cao 32a5126705
Add the limitation of the traceable interval to timeTravel (#14498)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-12-29 21:29:45 +08:00
Xiaofan 69087ff8dd
Support embedded etcd (#14333)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
guoxiangzhou 4abc4c7e1a
[skip e2e] Modify log text according to the naming conventions (#14274)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-29 14:09:21 +08:00
guoxiangzhou 3705ac7142
[skip e2e] Modify log text according to the naming conventions (#14292)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-29 14:07:22 +08:00
guoxiangzhou 4671a88fbb
[skip e2e] Modify log text according to the naming conventions (#14291)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2021-12-27 15:21:13 +08:00