milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
15,430 Commits
133 Branches
115 Tags
559 MiB
Commit Graph

1082 Commits (1bb2fdfbe85fd9493ed0f76f0ddcebdefae95353)

Author SHA1 Message Date
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search (#17565) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-17 17:46:10 +08:00
SimFG 2008f60e50
Simplify the implementations of the ToPhysicalChannel and the ConvertChannelName func (#17605) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-17 10:22:11 +08:00
cai.zhang 53a49a9255
GetIndexState only get the state generated call CreateIndex (#17549) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-16 20:12:11 +08:00
Xiaofan 1f6fbf91b2
Fix pulsar unsubsribe issue (#17562) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-16 17:28:11 +08:00
Xiaofan 54aeb077e9
Refine compaction selection poliy (#17486) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
congqixia ed3a1164dd
Handle wrapped grpc error in funcutil.IsGrpcError (#17547) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-14 17:10:10 +08:00
groot ab9b186f15
Change bulkload bufsize for row-based parser (#17473) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-14 16:18:09 +08:00
congqixia a0d3f37c0d
Sync grpc DefaultKeepAliveTimeout to milvus.yaml (#17519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-13 16:18:09 +08:00
congqixia d1de8cabdd
Fix %w verb for error wrapping in grpcclient (#17484) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-13 12:48:08 +08:00
yah01 a38790f586
Predict the loading memory usage more accurate (#17451) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-09 18:16:07 +08:00
groot 9210299706
Fix bulkload OOM issue (#17461) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-09 16:52:07 +08:00
xige-16 b5c11a216d
Alter varChar type params's name to max_length (#17409) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-07 15:58:06 +08:00
Xiaofan a9ec1f8864
Fix compatibility of upgrade (#17406) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-07 14:52:06 +08:00
congqixia de8f2f76c5
Fix shardLeader cache with ErrConnect check (#17402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-07 12:20:06 +08:00
Xiaofan af994b5e1c
Fix etcd watch error (#17364) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-06 16:26:06 +08:00
XuanYang-cn c70af73005
Fix datacoord set wrong state for node registering (#17376) 
Fix datacoord datarace

See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-06 15:10:05 +08:00
bigsheeper 48cf63a2d7
Wait for flow graph to exit (#17341) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 19:48:04 +08:00
shaoyue 76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
SimFG 0986c29d7f
proxy tls support one-way authentication (#17348) 
Signed-off-by: xiyichan <2863768433@qq.com>

Co-authored-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>

Co-authored-by: xiyichan <2863768433@qq.com>
 2022-06-02 18:56:04 +08:00
Ten Thousand Leaves f5e63177d2
Fix an issue with bulk load with multiple DataNodes (#17324) 
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-02 18:54:04 +08:00
yah01 4e83f37899
Improve load performance (#17273) 
Load segment's fields concurrently

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-06-02 17:56:03 +08:00
zhenshan.cao 2763efc9b0
Add cache of grpc client of ShardLeader in proxy (#17301) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-02 12:16:03 +08:00
jaime 5698bf4236
Add sasl configurations for kafka (#17323) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-06-02 12:12:03 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence (#16782) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
Jiquan Long 6bd786bf50
Reduce default seal propotion (#17287) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-31 12:02:04 +08:00
congqixia c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster (#16856) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-30 19:50:04 +08:00
zhenshan.cao b0d54ed0af
Rename Client to GrpcClient in mock package (#17230) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-30 17:16:08 +08:00
SimFG 731870211a
Support role-based access control (only pb) (#16987) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-28 00:04:01 +08:00
Ten Thousand Leaves 5e1e7a6896
Fix an issue that causes flush hanging (#17169) 
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-25 14:34:00 +08:00
congqixia a0cef3a0a2
Fix sessionutil log always datanode (#17184) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-24 16:05:59 +08:00
zhenshan.cao 2512e668f7
Update timestmap according to gracefultime for Bounded Consistency (#17171) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-24 12:05:59 +08:00
zhenshan.cao cce533c246
Fix unitest fail, change default index slice to 16 (#17162) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-05-23 17:44:25 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
ownbylichaobao 2f8a7e7793
Support tls/ssl connection to etcd (#7703) (#17012) 
Signed-off-by: lichaobao <1527563274@qq.com>
 2022-05-20 12:29:19 +08:00
groot d525e95550
Bulkload not allow duplicate numpy file (#17105) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-20 10:27:56 +08:00
bigsheeper 9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-19 10:13:56 +08:00
Xiaofan 5355153805
Minio Error is not Handled gracefully (#17003) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-16 19:25:55 +08:00
groot 168935f4cf
Fix bulkload bug for string primary key (#16958) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-13 16:07:54 +08:00
exia 4ee78eb5ad
proxy support tls (#16706) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-05-10 20:05:53 +08:00
Bingyi Sun 76bfd41952
Refine DataNode logs. (#16849) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-05-09 16:25:52 +08:00
Enwei Jiao 67ce206cf0
make SegmentMaxLifetime configurable (#16783) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-06 23:41:51 +08:00
Jiquan Long 98ceb162aa
Support antlr as plan parser (#16696) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: xaxys <tpnnghd@163.com>

Co-authored-by: xaxys <tpnnghd@163.com>
 2022-05-06 17:43:51 +08:00
groot 8736372fd2
Fix bulkload bugs (#16760) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-06 11:21:50 +08:00
Ten Thousand Leaves 1acd256481
Add DataQueryable and DataIndexed states for bulk load tasks (#16725) 
issue: #16607

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-05 21:17:50 +08:00
Xiaofan 333b439a6c
Support config grpc log level (#16618) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-05 10:35:50 +08:00
shaoyue fa97f86cda
Clean up not used id_cache related code (#16655) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-05-05 09:59:51 +08:00
Xiaofan 337ad53613
Support config small index nprobe and nlist (#16731) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-05 09:49:50 +08:00
cai.zhang 2be46a01c0
Reduce the number of retries and add error log (#16754) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-05-05 09:31:51 +08:00
Cai Yudong 6a62ff18bf
Support easylogging config for segcore and knowhere (#16751) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-05-03 08:39:49 +08:00
Jiquan Long 2fe8677cbf
Enable dimension check in Proxy when create index request received (#16718) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 18:01:49 +08:00
shaoyue 2f49db9d3c
Implements remaining rest APIs (#16387) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-04-29 15:15:47 +08:00
xige-16 515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
groot 21b5973253
Refine list import task api (#16690) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-28 17:21:47 +08:00
yah01 206b554d49
Sync the distribution of segments to shard leaders (#16685) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-28 16:01:48 +08:00
Xiaofan ed51ffd50f
Fix Compact Entity ttl param (#16665) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-28 11:51:47 +08:00
congqixia 0bf176a7e1
Add SyncReplicaSegments interface for QueryNode (#16672) 
Add SyncReplicaSegments for LoadCollection/LoadPartition child task
This API notifies related ShardCluster to record segment Loaded
Preventing the Etcd events is delayed, which will cause some segment not search

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-27 10:41:46 +08:00
yah01 80ae6de323
Add comments for concurrency package (#16654) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-26 11:33:45 +08:00
groot bc5e9ec929
Add list import tasks api (#16605) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-25 17:37:46 +08:00
yah01 7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment (#16514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-25 15:57:47 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-24 22:03:44 +08:00
groot a6a3b69d91
Report bulk load state (#16555) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-21 21:37:42 +08:00
codeman 91e84ffedf
fix: not create user root if exists, optimize log & imports (#16554) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-21 19:57:42 +08:00
Ten Thousand Leaves 6f75d02c65
Disable knowhere logging for embedded Milvus and some other tweaks (#16496) 
/kind enhancement

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 17:23:46 +08:00
Ten Thousand Leaves 289e468a7a
Make bulk load fully work (#16512) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 14:03:40 +08:00
codeman 3a1b2cedd2
create default root user for authentication (#16545) (#16549) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-20 13:03:40 +08:00
Xiaofan f367249b9e
Fix storage type config key (#16542) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-19 17:47:39 +08:00
codeman f5e1ded854
fix #16504 (#16523) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-19 16:35:39 +08:00
Xiaofan fce8f6cfab
Remove Timetick channel (#16449) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-17 23:47:36 +08:00
godchen 4781db8a2a
Add datanode import (#16414) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-12 22:19:34 +08:00
jaime ed3c29eae7
Support Kafka (#16149) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-12 19:47:33 +08:00
codeman 4e12271d79
Support login with username and password (#15656) (#16341) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-11 19:49:34 +08:00
congqixia 994bb829d3
Use fmt.Errorf instead of string concat in schema.go (#16404) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-11 09:47:37 +08:00
Cai Yudong 71cd7ba67a
Add configuration common.indexSliceSize (#16438) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-04-08 20:29:33 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves 1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. (#16374) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-06 15:33:32 +08:00
XuanYang-cn 78200009a3
Impl GetReplicas in Proxy (#16390) 
See also: #16298

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-06 14:57:31 +08:00
Ten Thousand Leaves 2bd0a9526a
Implement task expiring logic and make all deadlines configurable. (#16355) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-03 11:37:29 +08:00
groot be8d9a8b6b
Support import numpy file (#16348) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-03 11:27:29 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
Letian Jiang 028c5cb882
Modify grpc interface for replica Search/Query in QueryNode (#16326) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-04-02 14:15:31 +08:00
XuanYang-cn 5494f3c318
Remove offLineNodes from watch states check (#16347) 
This PR also
- adds more information in log
- makes Session able to logged by zap.Any/zap.String

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-01 22:33:29 +08:00
yah01 c005f07ccc
Modify proto to add GetReplicas() interface (#16328) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-04-01 16:15:29 +08:00
Jiquan Long fd589baca7
Integrates marisa trie index (#16192) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-01 15:31:29 +08:00
Ten Thousand Leaves 45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
groot bd241cd28b
Add performance test case for import (#16296) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-01 10:07:28 +08:00
XuanYang-cn 58addbf9f4
Add GetShardLeaders rpc in querycoord proto (#16299) 
See also: #16298

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-31 16:39:29 +08:00
congqixia f529f8c1fc
Use fmt.Errorf instead of str concat in func.go (#16315) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-31 15:03:28 +08:00
Jiquan Long ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
groot ffa06c77b6
Import util functions (#16237) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-30 16:25:30 +08:00
congqixia 18a3e9f265
Add grpc interface for replica Search/Query in QueryNode (#16197) 
Resolves #16195
Add Search and Query grpc interface in query proto and types/types.go

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-30 12:03:27 +08:00
godchen 478f6ca11e
Add cache for vector chunk (#15912) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-26 22:05:26 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
Cai Yudong f4ebd3a9ce
Upgrade to knowhere v1.1.0 (#16186) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-25 13:49:25 +08:00
Ten Thousand Leaves a23ca745bb
Add Etcd support for Import Tasks. (#16164) 
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-25 11:03:25 +08:00
bigsheeper 2fcf7b9760
Fix GetMemoryCount, get min as total memory (#15931) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-03-24 20:41:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot 1c4b949a1d
modify import rpc interface (#16129) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-22 15:11:24 +08:00
Jiquan Long f8d9bc919d
Unify interface of vector index & scalar index. (#15959) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-21 14:23:24 +08:00
Xiaofan 787d4f8bfd
Refine prometheus metrics and etcd log (#16084) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-18 15:51:22 +08:00