Commit Graph

204 Commits (2.5)

Author SHA1 Message Date
SimFG d2649b63db
Delete the user-role mapping info when deleting the user (#25988)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-04 18:37:08 +08:00
XuanYang-cn 1fff8d2c66
Fix restore redrop enormous dup collections (#26029)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-03 10:37:06 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-07-11 15:18:28 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
xige-16 732fe54775
Support partition Key (#24047)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-06 10:24:34 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
Jiquan Long 3cb9a44ed8
Add license to rootcoord module (#22301)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-02-22 11:37:45 +08:00
cai.zhang 66b3566ac1
Update component state to healthy after start (#22118)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-10 17:40:32 +08:00
Xiaofan f1daef22a1
Fix failed bulkload segment marked as sealed (#21799)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-31 12:41:53 +08:00
congqixia 5986106037
Make paramtable init only once (#21782)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
jaime 1dcd9eeb79
Support rename collection (#21693)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-19 14:13:43 +08:00
groot 0af53d1e2e
Fix bulkinsert report bug (#21778)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-01-18 10:33:43 +08:00
Ten Thousand Leaves 23ceb9c638
Remove index checking phase for bulk insert (#21578)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-09 21:15:38 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-04 16:37:35 +08:00
Ten Thousand Leaves 30b86275ac
Make BulkInsert free DataNode workload when import segments are flushed (#21253)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-28 11:11:30 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-09 14:31:21 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
groot 33a102c6fc
Refine bulkinsert (#20986)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-12-07 14:53:18 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
Xiaofan af66a0b621
Fix Time Tick session2ChanTs map use the same nodeID (#20537)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-14 14:41:11 +08:00
groot b847c425e1
Fix listImportTasks bug (#20301)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-07 17:11:02 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
XuanYang-cn 22fb1d3b93
Remove dup mock_kv.go and mock_txn_kv.go (#20267)
Replace them with mockery generated mocks in kv/mocks

See also: #19289

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-03 14:41:35 +08:00
Jiquan Long 89541a6f68
Revert parallel ddl (#20118)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-28 13:25:34 +08:00
Jiquan Long 7a13b1c329
Fix panic when stop rootcoord (#20019)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-25 09:51:30 +08:00
jaime 59bcbf0cf6
Support check health for the service level (#19779)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-18 13:39:26 +08:00
SimFG a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
jaime 31db01b6ae
Support TTL for collection level (#19383)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
Jiquan Long bdd6171231
Make creating collection/partition follow tt mechanism (#19628)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-09 12:32:57 +08:00
wayblink 54e2c079c4
MEP-30:Support basic coordinators active-standby mechanism #16901 (#19166)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-09-29 18:35:02 +08:00
Ten Thousand Leaves 67b95597ed
Deprecated old RootCoord methods in Bulk Load (#19449)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-27 10:44:53 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
Jiquan Long 4e521438c3
Optimize read ddl (#19352)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-24 12:42:51 +08:00
Jiquan Long 42d371fd41
Make ddl follow tt mechanism (#19279)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-21 15:46:51 +08:00
SimFG d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
Jiquan Long a5e2d6b6fb
Refactor RootCoord (#18930)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-09-05 13:29:11 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643)
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
SimFG 226bcc16eb
Support concurrent situations for the metastore (#18792)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-24 10:02:52 +08:00
Jiquan Long 641afae22f
Watch channels with start positions (#18744)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-08-22 14:42:52 +08:00
codeman 70254c3ec5
Unified catalog interface for segment (#18289) (#18290)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
congqixia b62cb82ebe
Simplify Dml-DeltaChannel mapping logic (#18702)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-18 14:24:50 +08:00
aoiasd 8a70f2badb
Add ShowConfigurations for all remaining session (#18594)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-08-12 13:20:39 +08:00
codeman 6c3dbf0a07
Support meta storing on database (#17236) (#18541)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-11 12:12:38 +08:00
SimFG ff0200210a
Support Role-Based Access Control (#18425)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-04 11:04:34 +08:00
jaime 9672eae62c
Unify metastore catalog interface (#17772)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-07-22 10:20:29 +08:00
Letian Jiang b15c24a554
Acquire reference lock of compacted from segments during index building (#17619)
- Accquire reference lock during index building
- Keep dropped segment record

Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-06-18 01:24:11 +08:00
cai.zhang 141a3bbe00
Mark deleted before drop index (#17490)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-17 18:08:12 +08:00
cai.zhang 53a49a9255
GetIndexState only get the state generated call CreateIndex (#17549)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-16 20:12:11 +08:00
cai.zhang ea5041aec2
Acquiring the segment reference lock on task level (#17544)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-15 21:38:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457)
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-15 12:20:10 +08:00
groot ab9b186f15
Change bulkload bufsize for row-based parser (#17473)
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-06-14 16:18:09 +08:00
jaime b64a467311
Revert "Use unified catalog interface to reconstruct metastore (#17042)" (#17480)
This reverts commit 218326bafe.

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-06-10 13:10:08 +08:00
jaime 218326bafe
Use unified catalog interface to reconstruct metastore (#17042)
Co-authored-by: kejiang ke.jiang@zilliz.com
Co-authored-by: yun.zhang yun.zhang@zilliz.com

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-06-07 12:00:07 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence (#16782)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-05-31 16:36:03 +08:00
bigsheeper 9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-19 10:13:56 +08:00
Ten Thousand Leaves 63385b7c12
Make data queryable and data indexed states as independent flags (#16837)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-12 19:23:53 +08:00
Ten Thousand Leaves 5afdf3d49f
Move locks back to meta table methods (#16848)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-09 20:47:52 +08:00
xige-16 7cc995d085
Prohibit creating diff indexes with same index_name on a collection (#16774)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-05 21:33:51 +08:00
Ten Thousand Leaves 1acd256481
Add DataQueryable and DataIndexed states for bulk load tasks (#16725)
issue: #16607

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-05 21:17:50 +08:00
groot bc5e9ec929
Add list import tasks api (#16605)
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-25 17:37:46 +08:00
Ten Thousand Leaves e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated (#16609)
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-25 11:07:47 +08:00
Ten Thousand Leaves 242f08fb67
Refine import task related logic (#16601)
issue: #16600

/kind bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-24 11:29:45 +08:00
codeman 91e84ffedf
fix: not create user root if exists, optimize log & imports (#16554)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-21 19:57:42 +08:00
Ten Thousand Leaves 289e468a7a
Make bulk load fully work (#16512)
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
Xiaofan fce8f6cfab
Remove Timetick channel (#16449)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-17 23:47:36 +08:00
jaime ed3c29eae7
Support Kafka (#16149)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-12 19:47:33 +08:00
godchen bb7a0766fe
Add dependency factory (#16204)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
Ten Thousand Leaves 1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. (#16374)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-06 15:33:32 +08:00
Ten Thousand Leaves 2bd0a9526a
Implement task expiring logic and make all deadlines configurable. (#16355)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-03 11:37:29 +08:00
Ten Thousand Leaves 45be3deb3f
Implement RC, DC, DN calling path for import. (#16321)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-01 11:33:28 +08:00
Ten Thousand Leaves 0d54697d42
Implement ReportImport related logic. (#16312)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-31 13:51:28 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-30 21:11:28 +08:00
Ten Thousand Leaves 40c703dacd
Support check index state in root coord, for bulk load feature. (#16198)
issue: #15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-28 16:41:28 +08:00
Ten Thousand Leaves a23ca745bb
Add Etcd support for Import Tasks. (#16164)
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-25 11:03:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131)
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
groot 1c4b949a1d
modify import rpc interface (#16129)
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-22 15:11:24 +08:00
groot 2078b24dca
Prepare for import (#16083)
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-21 15:47:23 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
Cai Yudong 356efee492
Rename timetickSync APIs and variables for better readability (#15264)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-18 14:47:36 +08:00
Cai Yudong 6323907566
Use CommonConfig in GlobalParams for all components (#15106)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 19:03:35 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-09 23:45:33 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-07 16:15:26 +08:00
Xiaofan 69087ff8dd
Support embedded etcd (#14333)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components (#13885)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
XuanYang-cn 33f326d576
[skip e2e]Update license for rc root_coord (#13746)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-20 17:45:37 +08:00
cai.zhang 9f23fc7f2a
Register the service when the component state is healthy (#13248)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2021-12-15 11:47:10 +08:00
Cai Yudong 829077ad0c
Optimize reSentDdMsg logic (#13378)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-15 08:59:08 +08:00
Xiangyu Wang 405b3cd932
Use the same lint rules with golint in revive (#13324)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-14 15:31:07 +08:00
JackLCL 3ddcec79b4
[skip ci]Fix error log output format in root_coord_test.go (#13130)
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-10 10:17:30 +08:00
cai.zhang 342200ce13
Estimate the memory size of the index before building the index (#12973)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2021-12-09 14:19:40 +08:00