milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
20,058 Commits
153 Branches
123 Tags
549 MiB
Commit Graph

178 Commits (88cca9d90f6f3b64d8eaa0a376d9c0042087ce46)

Author SHA1 Message Date
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-04 14:25:38 +08:00
XuanYang-cn 22fb1d3b93
Remove dup mock_kv.go and mock_txn_kv.go (#20267) 
Replace them with mockery generated mocks in kv/mocks

See also: #19289

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-03 14:41:35 +08:00
Jiquan Long 89541a6f68
Revert parallel ddl (#20118) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-10-28 13:25:34 +08:00
Jiquan Long 7a13b1c329
Fix panic when stop rootcoord (#20019) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-10-25 09:51:30 +08:00
jaime 59bcbf0cf6
Support check health for the service level (#19779) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-18 13:39:26 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
jaime 31db01b6ae
Support TTL for collection level (#19383) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-10 20:31:22 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-10-10 15:55:22 +08:00
Jiquan Long bdd6171231
Make creating collection/partition follow tt mechanism (#19628) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-10-09 12:32:57 +08:00
wayblink 54e2c079c4
MEP-30:Support basic coordinators active-standby mechanism #16901 (#19166) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-09-29 18:35:02 +08:00
Ten Thousand Leaves 67b95597ed
Deprecated old RootCoord methods in Bulk Load (#19449) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-27 10:44:53 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-26 18:06:54 +08:00
Jiquan Long 4e521438c3
Optimize read ddl (#19352) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-24 12:42:51 +08:00
Jiquan Long 42d371fd41
Make ddl follow tt mechanism (#19279) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-21 15:46:51 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
Jiquan Long a5e2d6b6fb
Refactor RootCoord (#18930) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
 2022-09-05 13:29:11 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643) 
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-08-25 15:48:54 +08:00
SimFG 226bcc16eb
Support concurrent situations for the metastore (#18792) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-24 10:02:52 +08:00
Jiquan Long 641afae22f
Watch channels with start positions (#18744) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-08-22 14:42:52 +08:00
codeman 70254c3ec5
Unified catalog interface for segment (#18289) (#18290) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-20 10:24:51 +08:00
congqixia b62cb82ebe
Simplify Dml-DeltaChannel mapping logic (#18702) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-18 14:24:50 +08:00
aoiasd 8a70f2badb
Add ShowConfigurations for all remaining session (#18594) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-08-12 13:20:39 +08:00
codeman 6c3dbf0a07
Support meta storing on database (#17236) (#18541) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-11 12:12:38 +08:00
SimFG ff0200210a
Support Role-Based Access Control (#18425) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-04 11:04:34 +08:00
jaime 9672eae62c
Unify metastore catalog interface (#17772) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-22 10:20:29 +08:00
Letian Jiang b15c24a554
Acquire reference lock of compacted from segments during index building (#17619) 
- Accquire reference lock during index building
- Keep dropped segment record

Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-18 01:24:11 +08:00
cai.zhang 141a3bbe00
Mark deleted before drop index (#17490) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-17 18:08:12 +08:00
cai.zhang 53a49a9255
GetIndexState only get the state generated call CreateIndex (#17549) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-16 20:12:11 +08:00
cai.zhang ea5041aec2
Acquiring the segment reference lock on task level (#17544) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-15 21:38:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
groot ab9b186f15
Change bulkload bufsize for row-based parser (#17473) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-14 16:18:09 +08:00
jaime b64a467311
Revert "Use unified catalog interface to reconstruct metastore (#17042)" (#17480) 
This reverts commit 218326bafe.

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-10 13:10:08 +08:00
jaime 218326bafe
Use unified catalog interface to reconstruct metastore (#17042) 
Co-authored-by: kejiang ke.jiang@zilliz.com
Co-authored-by: yun.zhang yun.zhang@zilliz.com

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-07 12:00:07 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence (#16782) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
bigsheeper 9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-19 10:13:56 +08:00
Ten Thousand Leaves 63385b7c12
Make data queryable and data indexed states as independent flags (#16837) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-12 19:23:53 +08:00
Ten Thousand Leaves 5afdf3d49f
Move locks back to meta table methods (#16848) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-09 20:47:52 +08:00
xige-16 7cc995d085
Prohibit creating diff indexes with same index_name on a collection (#16774) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-05 21:33:51 +08:00
Ten Thousand Leaves 1acd256481
Add DataQueryable and DataIndexed states for bulk load tasks (#16725) 
issue: #16607

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-05 21:17:50 +08:00
groot bc5e9ec929
Add list import tasks api (#16605) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-25 17:37:46 +08:00
Ten Thousand Leaves e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated (#16609) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:07:47 +08:00
Ten Thousand Leaves 242f08fb67
Refine import task related logic (#16601) 
issue: #16600

/kind bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-24 11:29:45 +08:00
codeman 91e84ffedf
fix: not create user root if exists, optimize log & imports (#16554) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-21 19:57:42 +08:00
Ten Thousand Leaves 289e468a7a
Make bulk load fully work (#16512) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 14:03:40 +08:00
Xiaofan fce8f6cfab
Remove Timetick channel (#16449) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-17 23:47:36 +08:00
jaime ed3c29eae7
Support Kafka (#16149) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-12 19:47:33 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves 1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. (#16374) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-06 15:33:32 +08:00
Ten Thousand Leaves 2bd0a9526a
Implement task expiring logic and make all deadlines configurable. (#16355) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-03 11:37:29 +08:00
Ten Thousand Leaves 45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
Ten Thousand Leaves 0d54697d42
Implement ReportImport related logic. (#16312) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-31 13:51:28 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
Ten Thousand Leaves 40c703dacd
Support check index state in root coord, for bulk load feature. (#16198) 
issue: #15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-28 16:41:28 +08:00
Ten Thousand Leaves a23ca745bb
Add Etcd support for Import Tasks. (#16164) 
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-25 11:03:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot 1c4b949a1d
modify import rpc interface (#16129) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-22 15:11:24 +08:00
groot 2078b24dca
Prepare for import (#16083) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-21 15:47:23 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
Cai Yudong 356efee492
Rename timetickSync APIs and variables for better readability (#15264) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-18 14:47:36 +08:00
Cai Yudong 6323907566
Use CommonConfig in GlobalParams for all components (#15106) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 19:03:35 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
Xiaofan 69087ff8dd
Support embedded etcd (#14333) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 14:35:21 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components (#13885) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
XuanYang-cn 33f326d576
[skip e2e]Update license for rc root_coord (#13746) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-20 17:45:37 +08:00
cai.zhang 9f23fc7f2a
Register the service when the component state is healthy (#13248) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-15 11:47:10 +08:00
Cai Yudong 829077ad0c
Optimize reSentDdMsg logic (#13378) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-15 08:59:08 +08:00
Xiangyu Wang 405b3cd932
Use the same lint rules with golint in revive (#13324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-14 15:31:07 +08:00
JackLCL 3ddcec79b4
[skip ci]Fix error log output format in root_coord_test.go (#13130) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-10 10:17:30 +08:00
cai.zhang 342200ce13
Estimate the memory size of the index before building the index (#12973) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-09 14:19:40 +08:00
Bingyi Sun a05a74fca2
Improve proxy manager in rootcoord (#12303) 
issue: #11275
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-29 17:33:41 +08:00
Cai Yudong cbbeb2a383
Add debug log when proxy idle for a long time (#12319) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-27 13:57:16 +08:00
Cai Yudong 858e95f377
Move dmlChannel and deltaChannel into timeticksync (#12254) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-25 10:07:15 +08:00
Cai Yudong d4c297b1a8
Enhance dml channel operations (#12143) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-21 21:33:13 +08:00
dragondriver ee0f753f7a
Fix datarace between GetComponentStates and Register (#11935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-19 13:57:12 +08:00
Cai Yudong 173e7b3808
Fix golint warnings for rootcoord/root_coord.go (#11871) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-16 14:25:17 +08:00
godchen 5357e301f4
Fix retrieve search error (#11501) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-11 00:54:45 +08:00
Cai Yudong a2ad88237d
Add deltaChannels in rootcoord core (#11097) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-03 21:04:14 +08:00
Cai Yudong 66b9684fe5
Update ListCollectionPhysicalChannels and ListCollectionVirtualChannels (#11007) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-02 15:50:30 +08:00
congqixia ed2b8d67c0
Reduce info saved in SnapshotMeta (#10288) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-21 14:04:36 +08:00
Cai Yudong c7566345eb
Simplify meta_snapshot interface (#9778) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-13 15:54:33 +08:00
Yusup 49198dbce1
Disable dropping collection via alias (#8850) 
Signed-off-by: Yusup <yusup@lsgrep.com>
 2021-10-08 17:37:53 +08:00
congqixia 88e319acda
Save start position in collection meta (#8682) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-27 18:10:00 +08:00
Cai Yudong 177a465913
Clean golint warnings for rootcoord (#8383) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-23 15:10:00 +08:00
Cai Yudong a62bce360b
Replace MarshalTextString with Marshal in rootcoord (#8341) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-23 10:37:54 +08:00
Yusup bdfb78e936
Add aliases to DescribeCollection (#8268) 
Signed-off-by: Yusup <yusup@lsgrep.com>
 2021-09-22 16:20:48 +08:00
Yusup ccb57ba4b2
add collection alias (#7255) 
Signed-off-by: Yusup <yusup@lsgrep.com>
 2021-09-18 11:13:51 +08:00
godchen b3f965c7d4
Add unrecoverable error for retry. (#7828) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-18 10:49:51 +08:00
Cai Yudong d10c6abc9e
Apply for msgstream from pool when creating collection (#7738) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 09:13:50 +08:00
congqixia 675b71cc30
Replace metaSnapshot with suffixSnapshot (#7975) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-15 22:05:49 +08:00
zhenshan.cao e5e4b24954
Fix bug: describe collection return wrong shard number (#7841) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-14 11:59:47 +08:00
Cai Yudong 4e23ed9a18
Revert apply physical channels from pool (d5f7e358 and 3b960969) (#7687) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-10 13:00:00 +08:00
zhenshan.cao 29e5138274
Fix unittest: timeout caused by excessive use of resources (#7653) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-09 17:39:30 +08:00
zhenshan.cao b0524adbd8
Support specified shard number when create collection (#7482) 
Add shards_num to CollectionInfo

Modify dmlChannelNum config

Compatible with old meta

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-08 15:00:00 +08:00
Cai Yudong d15d3fa0ac
Improve rootcoord codecov (#7518) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-07 11:16:37 +08:00
zhenshan.cao 86afde4598
Fix unittest for rootcoord, should filter out timetick msg first (#7490) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-06 11:52:41 +08:00