Commit Graph

179 Commits (index)

Author SHA1 Message Date
Ten Thousand Leaves 2bd0a9526a
Implement task expiring logic and make all deadlines configurable. (#16355)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-03 11:37:29 +08:00
godchen 478f6ca11e
Add cache for vector chunk (#15912)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-26 22:05:26 +08:00
Ten Thousand Leaves a23ca745bb
Add Etcd support for Import Tasks. (#16164)
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-25 11:03:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131)
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
Ten Thousand Leaves d5bae6710f
Make all logs configurable for standalone/embedded Milvus. (#15926)
Resolves: #15919, #15708
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-10 10:35:59 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Cai Yudong 92c8e32ebd
Let MemoryKV.Load return error when key not exist (#15814)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-02 18:51:55 +08:00
Nemo cdcb3627b4
Make Etcd log level configurable. (#15737)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-25 15:03:53 +08:00
shaoyue 83fdc1da7c
Add web server in proxy with sample handler (#15647)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-02-23 14:37:52 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
Cai Yudong 8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-08 20:57:47 +08:00
JunSung Park f6873d3dc1
Implement automatic item expiration on compaction (#14418)
* Implement automatic item expiration on compaction

Signed-off-by: JunSung Park <writtic@gmail.com>

* Change entityExpiration value comparison logic

Signed-off-by: JunSung Park <writtic@gmail.com>

* Add logging for the number of expired entity

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default value of CompactionEntityExpiration to off

Signed-off-by: JunSung Park <writtic@gmail.com>

* Fix errors for code checker

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default entityExpiration to max value

Signed-off-by: JunSung Park <writtic@gmail.com>
2022-02-07 22:45:46 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
Bingyi Sun 515fe962d5
Fix subscription path (#15373)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-25 13:27:44 +08:00
Bingyi Sun 3987574448
Unsubscribe topic in DataCoord (#15353)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-24 19:47:38 +08:00
xige-16 1cd2363bd8
Delete usless params SkipQueryChannelRecovery (#15289)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-21 21:43:40 +08:00
xige-16 dfc6670f0f
Unsubscribe channel after query node down (#15230)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-17 17:37:37 +08:00
zhenshan.cao 1601a61b78
Move logic of checking available port to service (#15222)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-14 21:17:34 +08:00
zhenshan.cao f14df1d2db
Initilize listener in service instead of ParamTable (#15220)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-14 18:29:34 +08:00
Cai Yudong 951249c6c1
Use KnowhereConfig for indexnode and querynode (#15112)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-11 09:35:34 +08:00
zhenshan.cao e5e739ec26
[skip e2e]Update license (#15127)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-10 23:59:43 +08:00
Cai Yudong 6323907566
Use CommonConfig in GlobalParams for all components (#15106)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 19:03:35 +08:00
Cai Yudong b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 17:29:34 +08:00
Cai Yudong ee60e95310
Use RocksmqConfig in GlobalParams (#15083)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 14:51:34 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-09 23:45:33 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-07 16:15:26 +08:00
shaoyue 0b74c62aed
[skip e2e] Add comment for global param (#14987)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-01-07 13:27:38 +08:00
XuanYang-cn 06ae5b19a3
Remove not inuse configs (#14836)
- segmentBinlgoSubPath
- collectionBinlogSubPath
- flushStreamPosSubPath
- statsStreamPosSubPath

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-06 14:49:20 +08:00
shaoyue 8b70ae0507
[skip e2e] Add comment for global param (#14926)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-01-06 13:51:21 +08:00
shaoyue 00d06f2562
[skip e2e] Add comment for base table (#14925)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-01-06 13:49:27 +08:00
guoxiangzhou 2836b1c670
[skip e2e] Modify code comments according to the naming conventions (#14896)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-01-06 13:09:49 +08:00
Cai Yudong d01931d8d9
[skip e2e] Remove useless configuration rootcoord.timeout (#14852)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-05 19:05:25 +08:00
shaoyue bb7d0fb2ad
[skip e2e] Add comment for base table (#14804)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-01-05 13:56:05 +08:00
shaoyue b6b62c025e
[skip e2e] Add comment for base table (#14723)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-01-04 10:47:27 +08:00
guoxiangzhou 915f11b1cd
[skip e2e] Modify log text according to the naming conventions (#14716)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-01-04 10:29:18 +08:00
guoxiangzhou d00d5bd651
[skip e2e] Modify log text according to the naming conventions (#14713)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-01-04 10:27:22 +08:00
guoxiangzhou 0f477092b0
[skip e2e] Modify log text according to the naming conventions (#14715)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-01-04 10:25:18 +08:00
guoxiangzhou d6ec5a0170
[skip e2e] Modify log text according to the naming conventions (#14712)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-01-04 10:21:18 +08:00
zhenshan.cao 32a5126705
Add the limitation of the traceable interval to timeTravel (#14498)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-12-29 21:29:45 +08:00
Xiaofan 69087ff8dd
Support embedded etcd (#14333)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
Cai Yudong d34bb26414
Set rolname when init GlobalParamTable (#14422)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-28 23:12:46 +08:00
Cai Yudong 30382d8bf9
Let rootcoord share proxy.TimeTickInterval (#14326)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-27 21:38:45 +08:00
zhenshan.cao 9a2c91d326
Fix bug: refresh config after session init (#14194)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-12-24 22:22:17 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components (#13885)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
shaoyue 9672c4d7d0
[skip e2e] Add comment for ConvertRangeToIntRange (#13898)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-21 21:04:25 +08:00
Cai Yudong fe4830046c
Update base_param (#13863)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-21 15:53:51 +08:00
Cai Yudong 9d6ee4ab75
Remove useless var from base_param (#13603)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-17 19:09:23 +08:00
Cai Yudong cf50e9ac55
Use GrpcServerConfig and GrpcClientConfig for distributed components (#13469)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-17 10:23:15 +08:00
Cai Yudong 77b2112249
Rename param to base_param, rename basetable to base_table in package paramtable (#13407)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-15 12:05:55 +08:00
Cai Yudong 024c1ac412
Let standalone share same LogCfg with distributed components (#13332)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-14 18:31:25 +08:00
shaoyue c77ed0417f
[skip ci] Add comment for basetable (#12694)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-03 17:09:43 +08:00
Xiaofan d37fdf9379
Remove component.yaml (#11959)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-17 14:35:10 +08:00
Xiaofan 71e814f796
Remove common.yaml and component.yaml (#11661)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-12 21:25:08 +08:00
Xiaofan 3cd47cd89e
Remove common.yaml and component.yaml (#11662)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-12 18:53:11 +08:00
Xiaofan 5f08b67156
Move node.yaml to component.yaml (#11363)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-10 14:27:37 +08:00
BIKI DAS c41bc0dede
Combined function params (#10868)
Signed-off-by: Biki-das <bikid475@gmail.com>
2021-10-29 11:06:37 +08:00
BIKI DAS b1d6d52c36
Combined appends into a single call (#10865)
Signed-off-by: Biki-das <bikid475@gmail.com>
2021-10-29 10:58:38 +08:00
Xiaofan 93f205b49e
Rename param names (#10853)
* Rename param names

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

* Rename configs

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-28 23:16:40 +08:00
Xiaofan fce23538a8
Remote Common yaml (#10616)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-28 21:24:41 +08:00
BIKI DAS d55d3d63eb
Improved empty string tests (#10429)
Signed-off-by: Biki-das <bikid475@gmail.com>
2021-10-27 14:44:24 +08:00
zhenshan.cao 49e23ebc85
[skip ci]Fix golint error in basetable (#10698)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-10-26 22:35:05 +08:00
cai.zhang aff5cd9598
Supprot to read the configuration of minio from environment variables (#10534)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-10-26 15:10:03 +08:00
dragondriver 977e98d21b
[skip ci] Fix golint warning of paramtable (#9991)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-15 20:38:58 +08:00
zhenshan.cao bbafd05911
[skip ci]Fix golint error in paramtable (#9851)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-10-14 14:04:34 +08:00
Xiaofan 2c2b800ff4
Change Log Name with NodeId (#8721)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-01 08:52:50 +08:00
congqixia 6bbf0a8f8f
Read datanode insertbuf from env (#8942)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-30 17:45:07 +08:00
Xiaofan 93fc10f8a6
Remove channel&&common.yaml (#8716)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-09-27 22:32:02 +08:00
Xiaofan d38adda8c2
Rename log file based on node id (#8517)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-09-26 21:18:11 +08:00
Xiaofan 31ec399edb
Move channel configuration from channel.yaml to milvus.yaml (#8420)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-09-26 19:46:17 +08:00
congqixia 06d4faf771
Fix G109 from gosec, prevents overflow (#7896)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-14 18:35:48 +08:00
Cai Yudong 27dcf698d3
Support set segcore chunk_size via config file (#7635)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-11 14:40:01 +08:00
groot 89815a64d7
Update unittest name of paramtable (#7526)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-09-07 15:20:40 +08:00
dragondriver 960741ee16
Support more detailed metrics information (#7472)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-09-03 19:35:47 +08:00
Cai Yudong 7edd96989f
Improve codecov for paramtable and typeutil (#7451)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-02 22:18:10 +08:00
Xiaofan 58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-08-24 09:45:51 +08:00
dragondriver b3f10ae5cc
Fix 6990, enhance the log when parsing grpc recv/send parameters (#6993)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-08-12 16:16:09 +08:00
dragondriver ad77a6e0d4
Fix #6859, increase the MaxCallRecvMsgSize and MaxCallSendMsgSize of grpc client (#6861)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-08-04 13:03:24 +08:00
yukun 39614aa8eb
Add rocksmq_path in config and paramtable (#6099)
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-25 19:44:11 +08:00
Cai Yudong 7dbf33b788
Update component names in config files (#5989)
* rename master/indexService in .yaml

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proxyNode to proxy in config files

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move master.yaml to root_coord.yaml, move proxy_node.yaml to proxy.yaml

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move proto/query_service.proto to proto/query_coord.proto

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move proxy_node_manager.go to proxy_manager.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add query_coord.pb.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-22 19:08:03 +08:00
xige-16 2af1c27811
Rename queryservice to querycoordinator (#5931)
* Rename queryservice to querycoordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix param table

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix unittest

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* service to coordinator

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* code format

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* set querycoord in ci

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-22 16:44:09 +08:00
sunby 7f2d7c1da3
Rename dataservice to datacoordinator (#5912)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 18:22:13 +08:00
XuanYang-cn ab7f642740
Add alias in paramtable (#5878)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-19 12:38:06 +08:00
Cai Yudong 3b0951e3e6
Combine standalone binary into milvus (#5866)
* rename service to coord for cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update docs

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update variable name

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize roles.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* support milvus run standalone

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* remove cmd/standalone

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proxynode to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for compoments

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add comments for codacy check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* set helm chart branch

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-18 15:20:08 +08:00
yukun e9a8d1c404
Add vChannels in proxy for query results (#5802)
* Fix proxynode for new retrieve logic

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove querynodenum from proxynode and querynode

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove QueryNodeIDList from proxy

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-16 20:15:59 +08:00
Cai Yudong f28403b99b
clean proxy service (#5812)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-16 18:19:57 +08:00
Cai Yudong 0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
XuanYang-cn cebdfb34f7
Enable multiple datanode (#5068)
Now it's able to start multiple datanode, however, they will process the same insert data.
Further changes are needed to truly enable multiple datanode:

While registering, dataservice need to return different insert channels for different datanode.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-04-28 18:43:48 +08:00
Xiangyu Wang 82ccd4cec0
Rename module (#4988)
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
sunby fad94fb576
Remove unsed configs (#4990)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-22 03:04:59 +00:00
XuanYang-cn e281f6a4a1 Remove useless configs in config files
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-04-13 10:04:39 +08:00
cai.zhang 1d899513dd Unify service name in config file
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-24 18:46:19 +08:00
zhenshan.cao 7c5e2d89f6 Read node port from config
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-23 18:08:17 +08:00
zhenshan.cao bbfcbbdd68 Fix ci: paramstable and startup logic refactor
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-23 11:40:30 +08:00
xige-16 7c210310dc Add load function for query service
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-26 15:13:20 +08:00
cai.zhang f940cc455a Add dockerfile for index
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-26 09:38:40 +08:00
xige-16 511aa419fa Refactor query service
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-22 14:28:06 +08:00
cai.zhang fb493cc235 Update interface name of GetIndexStates
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-19 18:32:57 +08:00