168935f4cf
Fix bulkload bug for string primary key ( #16958 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-13 16:07:54 +08:00
4ee78eb5ad
proxy support tls ( #16706 )
...
Signed-off-by: xiyichan <2863768433@qq.com>
2022-05-10 20:05:53 +08:00
76bfd41952
Refine DataNode logs. ( #16849 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-05-09 16:25:52 +08:00
67ce206cf0
make SegmentMaxLifetime configurable ( #16783 )
...
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
2022-05-06 23:41:51 +08:00
98ceb162aa
Support antlr as plan parser ( #16696 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-05-06 17:43:51 +08:00
8736372fd2
Fix bulkload bugs ( #16760 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-06 11:21:50 +08:00
1acd256481
Add DataQueryable and DataIndexed states for bulk load tasks ( #16725 )
...
issue: #16607
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-05 21:17:50 +08:00
333b439a6c
Support config grpc log level ( #16618 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-05 10:35:50 +08:00
fa97f86cda
Clean up not used id_cache related code ( #16655 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-05-05 09:59:51 +08:00
337ad53613
Support config small index nprobe and nlist ( #16731 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-05 09:49:50 +08:00
2be46a01c0
Reduce the number of retries and add error log ( #16754 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-05-05 09:31:51 +08:00
6a62ff18bf
Support easylogging config for segcore and knowhere ( #16751 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-05-03 08:39:49 +08:00
2fe8677cbf
Enable dimension check in Proxy when create index request received ( #16718 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 18:01:49 +08:00
2f49db9d3c
Implements remaining rest APIs ( #16387 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-04-29 15:15:47 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
21b5973253
Refine list import task api ( #16690 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-28 17:21:47 +08:00
206b554d49
Sync the distribution of segments to shard leaders ( #16685 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-28 16:01:48 +08:00
ed51ffd50f
Fix Compact Entity ttl param ( #16665 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-28 11:51:47 +08:00
0bf176a7e1
Add SyncReplicaSegments interface for QueryNode ( #16672 )
...
Add SyncReplicaSegments for LoadCollection/LoadPartition child task
This API notifies related ShardCluster to record segment Loaded
Preventing the Etcd events is delayed, which will cause some segment not search
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-27 10:41:46 +08:00
80ae6de323
Add comments for concurrency package ( #16654 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-26 11:33:45 +08:00
bc5e9ec929
Add list import tasks api ( #16605 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-25 17:37:46 +08:00
7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment ( #16514 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-25 15:57:47 +08:00
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
a6a3b69d91
Report bulk load state ( #16555 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-21 21:37:42 +08:00
91e84ffedf
fix: not create user root if exists, optimize log & imports ( #16554 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-21 19:57:42 +08:00
6f75d02c65
Disable knowhere logging for embedded Milvus and some other tweaks ( #16496 )
...
/kind enhancement
issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 17:23:46 +08:00
289e468a7a
Make bulk load fully work ( #16512 )
...
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
3a1b2cedd2
create default root user for authentication ( #16545 ) ( #16549 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-20 13:03:40 +08:00
f367249b9e
Fix storage type config key ( #16542 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-19 17:47:39 +08:00
f5e1ded854
fix #16504 ( #16523 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-19 16:35:39 +08:00
fce8f6cfab
Remove Timetick channel ( #16449 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-17 23:47:36 +08:00
4781db8a2a
Add datanode import ( #16414 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-12 22:19:34 +08:00
ed3c29eae7
Support Kafka ( #16149 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-12 19:47:33 +08:00
4e12271d79
Support login with username and password ( #15656 ) ( #16341 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-11 19:49:34 +08:00
994bb829d3
Use fmt.Errorf instead of string concat in schema.go ( #16404 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-11 09:47:37 +08:00
71cd7ba67a
Add configuration common.indexSliceSize ( #16438 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-04-08 20:29:33 +08:00
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. ( #16374 )
...
/kind feature
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-06 15:33:32 +08:00
78200009a3
Impl GetReplicas in Proxy ( #16390 )
...
See also: #16298
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-06 14:57:31 +08:00
2bd0a9526a
Implement task expiring logic and make all deadlines configurable. ( #16355 )
...
/kind feature
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-03 11:37:29 +08:00
be8d9a8b6b
Support import numpy file ( #16348 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-03 11:27:29 +08:00
99984b88e1
Support delete varChar value ( #16229 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
028c5cb882
Modify grpc interface for replica Search/Query in QueryNode ( #16326 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-02 14:15:31 +08:00
5494f3c318
Remove offLineNodes from watch states check ( #16347 )
...
This PR also
- adds more information in log
- makes Session able to logged by zap.Any/zap.String
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-01 22:33:29 +08:00
c005f07ccc
Modify proto to add GetReplicas() interface ( #16328 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-01 16:15:29 +08:00
fd589baca7
Integrates marisa trie index ( #16192 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-04-01 15:31:29 +08:00
45be3deb3f
Implement RC, DC, DN calling path for import. ( #16321 )
...
/kind feature
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-01 11:33:28 +08:00
bd241cd28b
Add performance test case for import ( #16296 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-01 10:07:28 +08:00
58addbf9f4
Add GetShardLeaders rpc in querycoord proto ( #16299 )
...
See also: #16298
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-03-31 16:39:29 +08:00
f529f8c1fc
Use fmt.Errorf instead of str concat in func.go ( #16315 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-31 15:03:28 +08:00
ba37531456
Add support for loading multiple indexes ( #16138 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-30 21:11:28 +08:00
ffa06c77b6
Import util functions ( #16237 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-30 16:25:30 +08:00
18a3e9f265
Add grpc interface for replica Search/Query in QueryNode ( #16197 )
...
Resolves #16195
Add Search and Query grpc interface in query proto and types/types.go
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-03-30 12:03:27 +08:00
478f6ca11e
Add cache for vector chunk ( #15912 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-26 22:05:26 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
f4ebd3a9ce
Upgrade to knowhere v1.1.0 ( #16186 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-25 13:49:25 +08:00
a23ca745bb
Add Etcd support for Import Tasks. ( #16164 )
...
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-25 11:03:25 +08:00
2fcf7b9760
Fix GetMemoryCount, get min as total memory ( #15931 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-24 20:41:25 +08:00
ca129d4308
Support configurable msgstream ( #16131 )
...
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
1c4b949a1d
modify import rpc interface ( #16129 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-22 15:11:24 +08:00
f8d9bc919d
Unify interface of vector index & scalar index. ( #15959 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-21 14:23:24 +08:00
787d4f8bfd
Refine prometheus metrics and etcd log ( #16084 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-18 15:51:22 +08:00
3cd28420f1
Support compile under windows ( #15786 )
...
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
* using gofrs/flock instead of syscall.Flock
* using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`
issue #7706
Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-17 17:17:22 +08:00
d4ad785b76
Bump go version in documentation ( #15979 )
...
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2022-03-16 18:51:21 +08:00
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
0bd07daae3
Add grpc healthy check ( #16050 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 15:05:21 +08:00
d4a0c5eba6
Alter comments in parallel.go for readability ( #16032 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-03-15 10:26:01 +08:00
09b2c46b61
Support create collection with VarChar field ( #15904 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-14 23:20:02 +08:00
29975a7a26
Remove Consume() API from mqstream interface ( #15886 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-11 20:09:59 +08:00
89356ebcb9
Modify import rpc interfaces ( #16006 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-03-11 17:13:59 +08:00
7d2934e4c2
Add methods to compress/decompress small blocks ( #15980 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-03-11 14:55:59 +08:00
d5bae6710f
Make all logs configurable for standalone/embedded Milvus. ( #15926 )
...
Resolves : #15919 , #15708
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-10 10:35:59 +08:00
5fdef607d5
Add rpc interfaces for import ( #15930 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-03-09 18:33:59 +08:00
565524ed3c
Add zstd compressor in util ( #15779 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-03-08 18:17:58 +08:00
f71651e294
Support column-based insert data in message stream ( #15802 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-04 15:09:56 +08:00
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig ( #15843 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
92c8e32ebd
Let MemoryKV.Load return error when key not exist ( #15814 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-02 18:51:55 +08:00
e394ba8c88
Refactor rocksmq comments and logs for readability ( #15801 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-03-01 18:55:55 +08:00
001567640e
Fix shallow copy of typeutil.AppendFieldData ( #15790 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-01 15:13:54 +08:00
d6857f4b24
Remove globalZapWrapper in logutil.go ( #15726 )
...
issue: #15720
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-02-28 17:23:54 +08:00
867cf620ee
Add Extract() method to take out C pointer from BytesConverter without releasing it ( #15614 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-27 09:47:53 +08:00
cdcb3627b4
Make Etcd log level configurable. ( #15737 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-25 15:03:53 +08:00
fcf0887d92
Add and implement chunk manager interface. ( #15541 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-24 23:53:53 +08:00
83fdc1da7c
Add web server in proxy with sample handler ( #15647 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-02-23 14:37:52 +08:00
c7f55c2e34
Make SessionWatch keep watch even no Rewatch func when find ErrCompacted ( #15497 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-15 15:07:48 +08:00
684110bc9a
Avoid to copy during converting C bytes to Go bytes ( #15551 )
...
* Avoid to copy during converting C bytes to Go bytes
Signed-off-by: yah01 <yah2er0ne@outlook.com>
* Add BytesConverter to achieve no-copy converting C bytes to Go bytes
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-15 14:17:48 +08:00
be42627ab2
Move Logutil to Util dir ( #15447 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig ( #15501 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
307a8ce535
Support compile and run on Mac ( #15491 )
...
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
2022-02-09 14:27:46 +08:00
55cdc5aa35
Get indexInfo and segment size by queryCoord ( #14207 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-02-08 21:57:46 +08:00
1f337891e1
Rename base_param to service_param, rename global_param to component_param ( #15463 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-08 20:57:47 +08:00
f6873d3dc1
Implement automatic item expiration on compaction ( #14418 )
...
* Implement automatic item expiration on compaction
Signed-off-by: JunSung Park <writtic@gmail.com>
* Change entityExpiration value comparison logic
Signed-off-by: JunSung Park <writtic@gmail.com>
* Add logging for the number of expired entity
Signed-off-by: JunSung Park <writtic@gmail.com>
* Set default value of CompactionEntityExpiration to off
Signed-off-by: JunSung Park <writtic@gmail.com>
* Fix errors for code checker
Signed-off-by: JunSung Park <writtic@gmail.com>
* Set default entityExpiration to max value
Signed-off-by: JunSung Park <writtic@gmail.com>
2022-02-07 22:45:46 +08:00
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable ( #15434 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
4bccc85534
Use MsgChannelConfig in GlobalParams for all components ( #15163 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
515fe962d5
Fix subscription path ( #15373 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-25 13:27:44 +08:00
3987574448
Unsubscribe topic in DataCoord ( #15353 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-24 19:47:38 +08:00
1cd2363bd8
Delete usless params SkipQueryChannelRecovery ( #15289 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-21 21:43:40 +08:00
c877f33084
Downgrade pulsar client ( #15325 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-20 22:25:38 +08:00
1e61112b5a
Add retry logic in pulsar consumer unsubscribe ( #15284 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-20 01:23:37 +08:00
groot
exia
Bingyi Sun
Enwei Jiao
Jiquan Long
Ten Thousand Leaves
Xiaofan
shaoyue
cai.zhang
Cai Yudong
xige-16
yah01
congqixia
codeman
godchen
jaime
XuanYang-cn
Letian Jiang
bigsheeper
Ji Bin
edward.zeng
zhenshan.cao
yah01
JunSung Park