milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
14,990 Commits
127 Branches
113 Tags
553 MiB
Commit Graph

748 Commits (37464a281dfd4aa9b4aaa2d4e109747ba8c13e42)

Author SHA1 Message Date
zhenshan.cao 0a953948af
Simplify monitoring metrics (#16687) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-27 23:03:47 +08:00
XuanYang-cn 35b7267edb
Add ut for task_policies (#16663) 
See also: #16652

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-27 16:55:46 +08:00
congqixia 0bf176a7e1
Add SyncReplicaSegments interface for QueryNode (#16672) 
Add SyncReplicaSegments for LoadCollection/LoadPartition child task
This API notifies related ShardCluster to record segment Loaded
Preventing the Etcd events is delayed, which will cause some segment not search

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-27 10:41:46 +08:00
zhenshan.cao a3f7bb5f2d
Remove the collectionID label from metrics (#16571) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-26 17:29:45 +08:00
XuanYang-cn fbc7fe1cdc
Fix proxy unable to update cache (#16646) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-26 11:27:53 +08:00
groot bc5e9ec929
Add list import tasks api (#16605) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-25 17:37:46 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-24 22:03:44 +08:00
Aivin V. Solatorio 04fffb08ef
Support arithmetic operations on numerical fields for scalar filtering (#16520) 
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
 2022-04-24 16:43:45 +08:00
yah01 804fbb6e27
Fix panic: Set Base of GetReplicas request (#16578) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-22 14:47:42 +08:00
Ten Thousand Leaves 6f75d02c65
Disable knowhere logging for embedded Milvus and some other tweaks (#16496) 
/kind enhancement

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 17:23:46 +08:00
congqixia 69252f812d
Implement memory replica in Proxy, QueryNode and QueryCoord (#16470) 
Related to #16298 #16291 #16154
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-20 16:15:41 +08:00
Ten Thousand Leaves 289e468a7a
Make bulk load fully work (#16512) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 14:03:40 +08:00
codeman 3a1b2cedd2
create default root user for authentication (#16545) (#16549) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-20 13:03:40 +08:00
codeman f5e1ded854
fix #16504 (#16523) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-19 16:35:39 +08:00
Ten Thousand Leaves cf9b9aba79
Fix a bug where RPC calls before proxy finishes initialization crashes Milvus (#16474) 
/kind bug

issue: #16473
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-14 20:01:34 +08:00
Ten Thousand Leaves 442c73a544
Support passing channel names to DataNode (#16424) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-12 13:25:34 +08:00
codeman 4e12271d79
Support login with username and password (#15656) (#16341) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-11 19:49:34 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves 1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. (#16374) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-06 15:33:32 +08:00
XuanYang-cn 78200009a3
Impl GetReplicas in Proxy (#16390) 
See also: #16298

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-06 14:57:31 +08:00
bigsheeper 7161a7760c
Fix segIDAssigner spelling mistake (#16370) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-06 14:11:31 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
XuanYang-cn e9090a62ab
Rearrange search/queryTask for readability (#16325) 
`searchTask` has 683 lines of code, `queryTask` has 485 lines of code.
`task.go` contains 4k+ codes including `searchTask` and `queryTask`.

It was so much pain navigating codes of searchTask and queryTask though task.go,
task.go and task_test.go are literaly unreadable.

This PR moves
1. 650+ lines of code of `searchTask` from `task.go` to `task_search.go`.
2. 1.4k+ lines of test code of `searchTask` from `task_test.go` to
   `task_search_test.go`.
3. 450+ lines of code of `queryTask` from `task.go` to `task_query.go`.
4. 200+ lines of test code of `queryTask from `task_test.go to
   `task_query_test.go`.

This PR also rearrange methods positions of `searchTask` and
`queryTask`:
-  Putting the most important methods `PreExecute`, `Execute`, and
   `PosExecute` at the beginning of the file.
-  Moves interface methods `ID`, `SetID`, `Type`, `BeginTs`, etc.
   that nobody cares about to the bottom of the file.

See also: #16298

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-01 18:59:29 +08:00
yah01 c005f07ccc
Modify proto to add GetReplicas() interface (#16328) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-01 16:15:29 +08:00
Ten Thousand Leaves 45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
XuanYang-cn 58addbf9f4
Add GetShardLeaders rpc in querycoord proto (#16299) 
See also: #16298

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-31 16:39:29 +08:00
XuanYang-cn a0bb758be5
Refine log in proxy searchTask (#16263) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-30 22:09:28 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
Jiquan Long cd0f3cbcd3
Refine log (#16269) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-29 22:43:28 +08:00
zhenshan.cao 11cea5aad0
Refine proxy log (#16246) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-29 02:29:27 +08:00
congqixia 2c03287e79
Improve log level in proxy QueryTask (#16234) 
Use warn instead of debug in error case

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-28 23:11:26 +08:00
Jiquan Long 973ea406ca
Improve log of show collections (#16233) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-28 21:47:27 +08:00
Ten Thousand Leaves 40c703dacd
Support check index state in root coord, for bulk load feature. (#16198) 
issue: #15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-28 16:41:28 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot 1c4b949a1d
modify import rpc interface (#16129) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-22 15:11:24 +08:00
groot 2078b24dca
Prepare for import (#16083) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-21 15:47:23 +08:00
Jiquan Long f8d9bc919d
Unify interface of vector index & scalar index. (#15959) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-21 14:23:24 +08:00
Xiaofan 787d4f8bfd
Refine prometheus metrics and etcd log (#16084) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-18 15:51:22 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
jaime cf4c014b2e
Remove collect query result by mq (#15988) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-16 10:19:21 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
jaime 97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-15 14:45:22 +08:00
xige-16 09b2c46b61
Support create collection with VarChar field (#15904) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-14 23:20:02 +08:00
jaime 29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
groot 89356ebcb9
Modify import rpc interfaces (#16006) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-11 17:13:59 +08:00
groot 5fdef607d5
Add rpc interfaces for import (#15930) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-09 18:33:59 +08:00
cai.zhang 2512dff261
Fix misuse of timerecord (#15868) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-04 15:45:56 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
Cai Yudong 57b97ccb9c
Let search fail as if one query_node return error result (#15855) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 10:43:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
congqixia 3f34bc1ab7
Ban DataType_String in CreateCollection (#15838) 
Fix #15836
/kind enhancement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-03 16:57:56 +08:00
朱亚光 27a7875a15
add array hashKeys empty check (#15796) 
fix #15791 https://github.com/milvus-io/milvus/issues/15791
Signed-off-by: zhuyaguang <zhuyaguang1368@163.com>
 2022-03-03 00:33:56 +08:00
cai.zhang d183748484
Add some prometheus metrics for proxy (#15582) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-02 16:23:55 +08:00
Nemo 3eb074a9b0
Update some comments for data coord with some minor code refactoring. (#15686) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-02-22 13:15:51 +08:00
xige-16 782f350da5
Modify parameter spelling of task queue (#15581) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-02-22 10:25:51 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-09 18:55:46 +08:00
Cai Yudong 8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-09 14:41:45 +08:00
guoxiangzhou a157487f8c
[skip e2e] Modify code comments according to the naming conventions (#15172) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-02-09 10:18:02 +08:00
guoxiangzhou c7e93f95aa
[skip e2e] Modify code comments according to the naming conventions (#15171) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-02-09 10:14:05 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Filipe Caixeta dee1961b4d
Make SearchResults and QueryResults return collectionID (#15377) 
* Make SearchResults and QueryResults return collectionID

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>

* add return collection name from QueryRequest and SearchRequest

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>

* remove comment

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>
 2022-01-27 22:55:41 +08:00
xige-16 408a6e9e58
Fix collectResultLoop hang after search timeout (#15250) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-01-17 18:11:36 +08:00
Jiquan Long 675e6d352b
QueryNodes send search/query results by rpc (#15223) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-17 14:41:35 +08:00
Jiquan Long 3bf34f6326
[skip e2e] Add comment for baseTaskQueue (#15170) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-12 10:59:37 +08:00
Jiquan Long 5d91fb3c70
[skip e2e] Add comment for MetaCache (#15169) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-12 10:57:41 +08:00
Jiquan Long 7e74db0d65
[skip e2e] Add comment for insertRepackFunc (#15115) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-10 22:41:46 +08:00
Jiquan Long 5ab8d99ee9
[skip e2e] Add comment for lastTick (#15114) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-10 22:39:54 +08:00
Jiquan Long ff60bb8bfc
[skip e2e] Add comment for GetCollectionSchema (#15113) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-10 22:37:58 +08:00
Cai Yudong 6323907566
Use CommonConfig in GlobalParams for all components (#15106) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 19:03:35 +08:00
shaoyue 2e8dd74c7e
[skip e2e] Add comment for proxy task (#15089) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-01-10 18:33:46 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
guoxiangzhou 1807ba534d
[skip e2e] Modify log text according to the naming conventions (#14969) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-01-07 18:41:23 +08:00
guoxiangzhou 8ec3c69447
[skip e2e] Modify log text according to the naming conventions (#14970) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2022-01-07 18:39:22 +08:00
Jiquan Long 92b466800f
[skip e2e] Add implementation assertion for taskQueue (#15023) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-07 17:53:28 +08:00
Jiquan Long 149e049f04
[skip e2e] Add implementation assertion for Cache (#15022) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-07 17:51:33 +08:00
Jiquan Long 72e3352366
[skip e2e] Add comment for GetPartitionInfo (#15021) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-07 17:49:43 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
Jiquan Long 6a8d8f43c4
[skip e2e] Add comment for GetPartitions (#14954) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-06 23:32:00 +08:00
Jiquan Long 5eb346fd9e
[skip e2e] Add license for id_cache (#14953) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-06 23:30:10 +08:00
Jiquan Long 691e4d954d
[skip e2e] Add implementation assertion for channelsTimeTicker (#14952) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-06 23:24:16 +08:00
Jiquan Long a18744346a
[skip e2e] Add comment for GetPartitionID (#14872) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-05 23:19:21 +08:00
Jiquan Long 95fd0d1bf4
[skip e2e] Add comment for newChannelsTimeTicker (#14871) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-05 21:01:30 +08:00
Jiquan Long 3e9340e3f6
[skip e2e] Add comment for channelsTimeTicker (#14870) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-05 19:07:20 +08:00
congqixia 36a43e647d
[skip e2e] Add comment for GetCollectionInfo of proxy meta cache (#14824) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-05 14:27:42 +08:00
congqixia 6400263f3c
[skip e2e] Improve SearchTask log format (#14753) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-04 14:15:39 +08:00
Jiquan Long 77ef791b6e
[skip e2e] Add comment for minFloat32 (#14752) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-04 14:13:44 +08:00
Jiquan Long b2fbf66f2f
[skip e2e] Add comment for getMinTick (#14751) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-04 14:11:51 +08:00
Jiquan Long deff7de3ac
[skip e2e] Add comment for newChannelsMgrImpl (#14750) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-01-04 14:09:52 +08:00
congqixia cd43b12a77
[skip e2e] Add comment for meta_cache.go (#14746) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-04 14:07:56 +08:00
guoxiangzhou 5fccbe9764
[skip e2e] Modify log text according to the naming conventions (#14289) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2021-12-31 11:39:31 +08:00
guoxiangzhou a87a79a65a
[skip e2e] Modify log text according to the naming conventions (#14287) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2021-12-31 11:37:35 +08:00
guoxiangzhou 451178203e
[skip e2e] Modify log text according to the naming conventions (#14283) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2021-12-31 11:33:41 +08:00
guoxiangzhou 4f1ee13795
[skip e2e] Modify log text according to the naming conventions (#14280) 
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
 2021-12-31 11:31:22 +08:00
Jiquan Long 4fe0e5e457
[skip e2e] Add comment for definitionLoop (#14637) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-30 20:45:21 +08:00
Jiquan Long 700546708c
[skip e2e] Add comment for GetCollectionInfo (#14636) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-30 20:43:26 +08:00
Jiquan Long 2cf0a551b4
[skip e2e] Add comment for getMinTsStatistics (#14635) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-30 20:41:33 +08:00
Xiaofan 98e4ff33a8
Fix RocksDB Slow (#14614) 2021-12-30 20:08:15 +08:00